Zhegu at SemEval-2023 Task 9: Exponential Penalty Mean Squared Loss for Multilingual Tweet Intimacy Analysis
Year: 2023

Abstract:
We present the system description of our team Zhegu in SemEval-2023 Task 9 Multilingual Tweet Intimacy Analysis. We propose EPM (Exponential Penalty Mean Squared Loss) for the purpose of enhancing the ability of learning difficult samples during the training process. Meanwhile, we also apply several methods (frozen Tuning & contrastive learning based on Language) on the XLM-R multilingual language model for fine-tuning and model ensemble. The results in our experiments provide strong faithful evidence of the effectiveness of our methods. Eventually, we achieved a Pearson score of 0.567 on the test set.

heading: 1 Introduction
text: Text intimacy is considered as the essential component in emotional communication and social relationships (Pei and Jurgens, 2020; Hovy and Yang, 2021; Sullivan, 2013). At the same time, intimacy exists in any language in the world. However, both data sets and methods of this area are still rare. Valuable research on the multilingual regression task is necessary to facilitate the sentiment analysis of language. To further investigate the intimacy in text, Pei et al. propose the MINT, a new Multilingual intimacy analysis dataset. In this case, we describe the model system of our team Zhegu in SemEval-2023 Task 9.
Since the mean of intimacy is different from the "positive" or "negative" in sentiment analysis, it is difficult to have a clear definition. There exist many hard samples, which can not be learned by the model easily. Lin et al. propose Focal loss for learning the hard samples for model only in classification. Therefore, we present EPM (Exponential Penalty Mean Squared Loss) for the regression task, which set a penalty term for hard samples during calculating the training loss to accelerate the
∗*Corresponding author
model focus on those hard samples. We also construct a language-based contrastive loss function to draw closer the distance of the text in the same language and distinguish in the different languages. The test set in this task includes four new languages text data additionally, which do not exist in the training data. In order to achieve better Zero-shot inference on new languages data, we also adopt the frozen tuning to fine tune the model, which means freezing some parameters of the model for training. All in all, the main contributions of our paper are as follows.
• We first propose EPM, which is an exponential penalty mean squared loss function for better learning hard sample in regression.
• We introduce a language-based contrastive loss for the multilingual problem.
• We apply frozen tuning for increasing the capability of model generalization.
All methods we mentioned above will be presented in detail in the Section 3 and will be validated in Section 4. In Section 5, we will give the conclusion of our research and future work.

heading: 2 Dataset
text: The data sets of Task 9 is divided into the labeled set and the unlabeled testing set. Some samples of the labeled set are shown in Table 1. The value of intimacy is range from 1 to 5. The training set contains 9,491 labeled data of six languages, including English (en), Chinese (zh), French (fr), Italian (it), Portuguese (po) and Spanish (sp). There are a total of 13,697 unlabeled data in the test set of ten languages with additional four new languages, including Arabic (ar), Dutch (du), Hindi (hi) and Korean (ko). Details of the data sets can refer to Table 2.
According to Table 2, the amount of text data for each language is relatively balanced to each other
318
in both the training set and test set. Meanwhile, the distribution of the number of text data with different labeled values in the labeled set is shown in Figure 1. It is obvious that the distribution shows
the decreasing trend, principally like the long-tail distribution. The number of those data with 1.0 is close to 1400, which is nearly one hundred times of the value of 5.0.

heading: 3 Methodology
text: In this section, we describe in detail the methods we used on task 9. We first introduce a loss function
for regression task we proposed, and we named it EPM. After that, we present contrastive loss and frozen tuning. The reasons for their ability to improve model performance will be described in detail. And We take XLM-RoBERTa-large (Conneau et al., 2019) as the backbone of our model.

heading: 3.1 EPM
text: We denote the training data D = [(x1, y1), (x2, y2), ..., (xN , yN )], yi ∈ [1.0, 5.0]. N represents the number of the training set. And the model function is
ŷi = f(xi) = wxi (1)
where ŷi is the prediction of the model and w is the parameters. The calculation of MSE (Mean Squared Error) loss in regression is
LMSE(yi, ŷi) = 1
N
N∑
i=1
(yi − ŷi)2 (2)
The MSE loss function is used to measure the difference between the label and the prediction. And we define our EPM loss function as
LEPM (yi, ŷi) = 1
N
N∑
i=1
(yi − ŷi)2e|yi−ŷi| (3)
EPM adds a penalty term e|yi−ŷi| on side of MSE, which can further penalize those samples with huge deviations. EPM will degrade to the standard MSE loss function when ŷi equals to yi. The derivative of the MSE loss function with respect to w is
∂(LMSE) ∂w = 2 N
N∑
i=1
(ŷi − yi)xi (4)
The derivative of the EMP loss function with respect to w is
∂LEPM ∂w = 1 N
N∑
i=1
e|ŷi−yi|[2(ŷi − yi) + (ŷi − yi)2]xi
(5)
= ∂(LMSE)
∂w e|ŷi−yi| + LEPMxi, ŷi ≥ yi (6)
∂LEPM ∂w = 1 N
N∑
i=1
e|ŷi−yi|[2(ŷi − yi)− (ŷi − yi)2]xi
(7)
= ∂(LMSE)
∂w e|ŷi−yi| − LEPMxi, ŷi < yi (8)
The value of Equation 5 equals to Equation 7 when ŷi equals to yi. We can easily know that the derivative function of EPM is continuous by referred to Equation A. The EPM derivative function reveals the e|yi−ŷi| penalty term amplifies the gradient isometrically during the process of back-propagation. And (ŷi − yi)2 penalizes the gradient once again. The larger |yi − ŷi|, the larger the penalty for that one hard sample. Focal loss (Lin et al., 2017) also has capacity to penalize hard samples for classification tasks. Nevertheless, EPM does not reduce the attention of the learning of simple samples, while Focal loss does. For those easy samples that they could be easy to learn by the model, ŷi converges to yi so much that LEPM highly approximates to LMSE .

heading: 3.2 Contrastive Loss
text: Contrastive learning can help model to distinguish the negative samples and converge the positive samples by constructing appropriate positive and negative samples. In this section, we introduce a language-based contrastive loss.
In contrastive learning, the construction of positive and negative samples is crucial. According to the analysis of data in the previous Section 2, the labeled set contains text data from six languages. For the multilingual task, we take XLM-R (Conneau et al., 2019) as the backbone of our model. We believe that the distance between those text data from the same language should be closer than those from different languages. Based on the above analysis, the construction of the positive and negative samples of contrastive learning in our paper are based on the languages. The process of contrastive learning for a batch of training data is shown in Figure 2.
Let vi be the sentence vector obtained by inputting xi to the model, li denote the language of xi, li ∈ {zh, en, fr, it, po, sp}. For a batch of data (x1, x2, ..., xM ), (v1, v2, ..., vM ) are the sentence vectors. M is batch size. Then KL (KullbackLeibler) divergence is calculated between every two samples to get the matrix KL ∈ RM×M . The reason of adopting KL divergence rather than cosine similarity is that cosine similarity usually is
used to calculate the similarity of the meaning between the text, while KL divergence obtains the distance of two distributions. KL divergence is regarded as the metric appropriately because the different texts in the same language do not contain the same meaning. Denote the contrastive learning loss function as LCL.
LCL = − 1
2 ln
∑ li=lj
eKL[i][j]/τ ∑
li ̸=lj e KL[i][j]/τ , i, j ∈ [1,M ] (9)
where τ is temperature hyper-parameter. The final loss function of the model is the sum of EPM loss and contrastive loss. The model learns the objective function of regression while converging the distribution of the text data from the same language and distinguishing from the different languages.
L = LEMP + LCL (10)

heading: 3.3 Frozen Tuning
text: The test set contains text data from four languages additionally, which the training set do not have. The model of this task is required to have the capability of generalization for inferring the new language text data as possible. During the training process, the parameters of the model are fitted to the training set containing only six languages. To enhance the generalization of our model on new language text data, the Frozen Tuning is used in our paper. We named the parameters of the standard XLM-R General parameters, which is pre-trained on the general pre-trained corpus. And we also named the parameters of fine-tuned XLM-R with the training set Domain parameters. Frozen Tuning freezes some of the parameters of the XLM-R during fitting in the training set, as shown in Figure 3. The languages of the pre-trained corpus of XLM-R contain all the languages in the training set and the test set. The frozen parameters in the pre-training model remain the same generalization capacity for all the languages, while the fine-tuned parameters can be considered as the classifier learned on the training set. Meanwhile, the updatable parameters of the model during the training process can be reduced by frozen tuning to improve the generalization performance.

heading: 4 Experiments & Results
text: In this section, we describe the experimental procedures and results in detail. The evidence of those methods we mentioned above will be provided by results and analysis.

heading: 4.1 Data Split
text: The labeled data and the unlabeled test set are provided in Task 9. Two strategies of dividing the validation set for model selection were adopted in our experiments.
Dev1 : We directly sample one-fifth of the labeled data as the validation set and the remaining data as the training set.
Dev2 : In order to simulate the distribution of the test set as much as possible, the validation set should also contain unseen languages’ data. Therefore, we first sample two languages text data from the labeled data containing six languages. Subsequently, we concatenate one-fifth of the remaining four languages’ data and two unseen languages data as the final validation set.

heading: 4.2 Performance & Analysis
text: All experiments in this paper were conducted on the XLM-R pre-trained languages model. The detailed hyperparameters can be found in Table 4 in the Appendix. The final results are shown in Table 3. Figure 4 shows the PCC (Pearson correlation coefficient) of the training set with using different loss function. For fair comparison, we used the MSE loss function as the reference group. With the help of the EPM, the model could converge faster than MSE on the training data. Meanwhile, the fluctuation of PCC of EPM in the optimization process is much smoother than MSE. And the results of the validation sets in Table 3 show that EPM not only let the model converge faster, but also can obtain better performances than MSE.
Figure 5 presents the PCC of the training data using EPM and MSE loss functions respectively
when we conduct contrastive learning. The model also converges normally when we combine EPM and language-based contrastive learning. Although slightly slower optimization in the early stage of training, the model still can converge on the training set. The reason of slower optimization is that the model optimizes both LEPM and LCL at the same time. Nevertheless, the PCC of the training set suddenly drops or even becomes negative in the middle of the training process when the final loss is LMSE + LCL. This phenomenon is caused by the gradient disappearance during the back-propagation process. In the training process, the regression and the contrastive learning task are optimized at the same time. And it may make two gradients of LMSE and LCL add up to a small amount and cause disappearing of the gradient. However, the penalty term of EPM could amplify the gradient, which is able to avoid the gradient vanishing.
Table 3 presents EPM brings improvement both on the validation and test sets. And the improve-
ment on the seen languages and unseen languages are nearly 2% and 7%. Meanwhile, the contrast learning based on languages outperforms other methods on zero-shot prediction of the unseen languages, which obtains the promotion of 2% on the basis of EPM. Although the effectiveness is not much as both EPM and contrastive learning, Frozen tuning also brings a surprising improvement of 1% in the unseen languages’ data, which may cause by the reduction of the training parameters. It is difficult to optimize LEPM and LCL simultaneously when freeze some layers of the pre-trained model. Finally, the best result is obtained by multi excellent models ensemble, which selected by the validation set. The decent improvement is 7% in the seen languages and 5% improvement of the unseen languages compared to baseline. The results strongly provide faithful evidence to our methods.

heading: 5 Conclusion
text: We introduce the system description of our team Zhegu in SemEval-2023 Task 9. We first propose a loss function, we named EMP (Exponential Penalty Mean Squared Loss), which could increase the gradient of those hard samples by adding the penalty term. EPM could be used in any regression task. And we conduct a contrastive learning task based on languages to reduce the distance of those in the text from the same languages. The experiments prove the effectiveness of our proposed methods. Meanwhile, EPM could avoid the disappearance of gradient than MSE when we conduct contrastive learning. Frozen tuning is also useful for improving the performance. In future work, the effectiveness of EPM should also be proved in others natural language regression tasks.

heading: Future Work
text: In future work, the effectiveness of EPM should also be proved in others natural language regression tasks