ZBL2W at SemEval-2023 Task 9: A Multilingual Fine-tuning Model with Data Augmentation for Tweet Intimacy Analysis
Year: 2023

Abstract:
This paper describes our system used in the SemEval-2023 Task 9 Multilingual Tweet Intimacy Analysis. There are two key challenges in this task: the complexity of multilingual and zero-shot cross-lingual learning, and the difficulty of semantic mining of tweet intimacy. To solve the above problems, our system extracts contextual representations from the pretrained language models, XLM-T, and employs various optimization methods, including adversarial training, data augmentation, ordinal regression loss and special training strategy. Our system ranked 14th out of 54 participating teams on the leaderboard and ranked 10th on predicting languages not in the training data. Our code is available on Github 1.

heading: 1 Introduction
text: A theoretical approach to understanding the level of intimacy, human emotion, and social connections in communication is called intimacy in language. Quantifying the intimacy expressed in language plays a key role in revealing important social norms in various context (Pei and Jurgens, 2020). SemEval-2023 Task 9 aims at mining intimacy in tweets. Pei et al. (2023) provides a new multilingual intimacy analysis dataset covering 13,372 tweets in 10 languages. Text intimacy is labeled on a continuum scale, in the range of [1, 5]. Among them, the participants are only given the training set in six languages (English, Spanish, Italian, Portuguese, French, and Chinese), and the model performance will be evaluated on the test set of ten languages, including 4 languages not in the training set (Hindi, Arabic, Dutch and Korean).
Given that the SemEval-2023 Task 9 is a complex challenge that integrates language transfer and
*These authors contributed equally to this work. †Corresponding author. 1https://github.com/HelloHank-Hub/Multilingual-Tweet-
Intimacy-Analysis
semantic mining, we propose the intimacy prediction system based on pre-trained representations and the fusion of multiple optimization methods. First, we chose XLM-T (Barbieri et al., 2021) as the encoder of the model, which is pre-trained on millions of tweets in multiple languages. Then, according to the form of annotated dataset, we change the calculation method of model loss and adjust the training strategy of the model to make it more suitable for the intimacy prediction of zero-shot language. Finally, we introduce adversarial training and data augmentation to enhance the model generalization. Our system obtained Pearson correlation scores of 0.7029 on seen language section and 0.4359 on unseen language section.

heading: 2 Related Work
text: Intimacy computing is an emerging field of natural language processing. Pei et al. (2023) propose to encode social messages of intimacy through topics and other more subtle cues such as swear words. A new computational framework for studying the expression of intimacy in language is introduced, accompanied by multilingual datasets, MINT, and deep learning models.
With the development of pre-training technology, multi-lingual pre-train model based methods are becoming the new trend to solve cross-lingual tasks. Conneau and Lample (2019) extend generative pre-training to multiple languages and show the effectiveness of cross-lingual pre-training for the first time. Knowledge distillation methods (Hinton et al., 2015) have been widely used in crosslanguage model research, but DistillBert (Sanh et al., 2019) and MiniLM (Wang et al., 2020) do not perform well for intimacy tasks. Our work builds on the work of Barbieri et al. (2021), which takes into account the unique expression of tweets and the generalization of multiple languages.
770

heading: 3 Methodology

heading: 3.1 System Architecture
text: The overall architecture of our final system is illustrated in Figure 1. In order to mine the semantic features related to intimacy in multilingual texts, we chose XLM-T as the encoder of the whole system. The input to the XLM-T was randomly obtained from tweets in different languages, and the last hidden state of the output was converted into a semantic pooling vector through the pooling layer. Then, the pooled vector and the output [CLS] vector were concatenated and fed to the feedforward network consisting of two layers of linear networks to predict the final results. Finally, we trained the model using the ordinal regression loss.

heading: 3.2 Adversarial Training
text: Adversarial training (Goodfellow et al., 2015) is an efficient regularization technique for classifiers to increase robustness to small, approximately worstcase perturbations. In Task 9, we introduced the Fast Gradient Method (FGM) (Miyato et al., 2016), a novel method for adversarial training, enhancing the generalization of the model in the intimacy analysis.
According to FGM, we applied tiny perturbations to sentence embeddings, and the adversarial perturbation radv on s is defined as:
radv = ϵ · g/∥g∥2 where g = ∇sL(s, y) (1)
where ϵ is a hyperparameter controlling the strength of the adversarial perturbations.
We employed an overall loss function to combine the information learned from the original and adversarial samples:
L = L(s, y) + Ladv(s+ radv, y) (2)

heading: 3.3 Ordinal Regression
text: Though intimacy in MINT is represented as a real number after post-processing, the original annotation comes from Likert Scale (Pei et al., 2023), which is a close-ended, forced-choice scale, whose options listed from 1 to 5 have strong inner order. That inspired us to introduce ordinal regression into our model.
Ordinal regression dedicates to learn a rule to predict labels from an ordinal scale by ensuring that predictions farther from the true label receive a greater penalty than those closer to the right label (Pedregosa et al., 2017; Rennie and Srebro, 2005).
If we have k ordinal labels Y = {y1, . . . , yk}, these labels are separated by k+1 boundary values A = {α0 . . . αk}, to be more generalized, extend the bound to infinity, i.e. α0 = −∞ and αk = +∞. The probability of label yi could be defined as:
p(yi) = Π(αi)−Π(αi−1), 1 ≤ i ≤ k (3)
where Π is the Cumulative Distribution Function (CDF), and we used sigmoid in our code. To describe the error between ground-truth and predicted labels, we used a "inter-boundary" method, i.e.
p(yi) = Π(αi−z)−Π(αi−1−z), 1 ≤ i ≤ k (4)
where z = z(x) is the model’s predict value based on input dataset X , and the ordinal regression loss function is defined as:
loss(z(x);Y,A) = − k∑
i=1
log p(yi). (5)
z(x) starts up with a random number, and A could be treated as a hyperparameter.

heading: 3.4 Data Augmentation
text: In this task, data augmentation is crucial considering of the zero-shot cross-lingual prediction target, and also the dataset is relatively small for large pretrained language models, we employed two levels of data augmentation in our system.

heading: 3.4.1 Word-level: Substitution
text: By tweaking words, faint noises are introduced to the model, which may challenge the training process and improve model robustness. We used the bert-base-multilingual-uncased (Devlin et al., 2018) as a masked language model to randomly mask out and substitute word-pieces in tweets (Jiao et al., 2019). But thinking of the essence of tweets-concise, we refrained from deleting or inserting random words since which may drastically alter the literal meaning.

heading: 3.4.2 Sentence-level: Translation
text: Although XLM-T is pre-trained using multi-lingual corpus and XLM has a translation language modeling (TLM) pre-train task, considering zero-shot learning is an extreme case in transfer learning which may only work well under the ideal situation, in other words, such theory is only feasible unless additional information has been exploited during training (Socher et al., 2013), we should provide unseen language samples.
In such semantic comprehension tasks, there is no doubt that the richer semantic information is, the better the model performs (Xu et al., 2022). By calling Azure Translator API2, we randomly chose tweets from the original dataset and translated them into (a) 6 seen languages, to enrich the original training set; (b) 4 unseen languages, to provide additional information for training.
Level 1 and level 2 could be applied separately or simultaneously, after which, an enhanced dataset was obtained. During training, a portion of augmented data was chosen and mixed up with the original dataset.
2https://azure.microsoft.com/products/cognitiveservices/translator/

heading: 3.5 Training Strategy
text: In the model training stage, we tried to randomly split 10% of the training set as the validation set to fine-tune the model, and we chose one language from the six seen languages and added them all to the validation set to verify the model’s predictive performance for text intimacy in unseen languages. In order to unleash the potential of the model’s unseen language prediction and eliminate the diversity of intimacy conveyed in different languages, we proposed the strategy of multilingual cross-validation training. During the training process, the composition of the validation set and the training set was changed every 13 epochs. The six languages were each used as the unseen language in turn, with a total of 78 epochs for training across all languages. All parameters in XLM-T were not frozen during training.

heading: 4 Experiments

heading: 4.1 Dataset and Evaluation
text: The intimacy dataset, MINT (Pei et al., 2023), used in this competition covers 13,372 in 10 languages tweets sampled from 2018 to 2022. The label distribution of datasets are shown in Figure 3.
As mentioned in the official task description, we use Pearson’s r to evaluate the level of linear correlation between predictions and ground-truth. The Pearson correlation coefficient is calculated by dividing the covariance of the two variables by the product of their standard deviations:
r = ∑ (xi − x̄i) (yi − ȳi)√∑
(xi − x̄i)2 ∑ (yi − ȳi)2 (6)

heading: 4.2 Experimental Settings
text: We implemented the intimacy analysis model based on XLM-T (Barbieri et al., 2021) with Pytorch (Paszke et al., 2019) and the Huggingface Transformers library (Wolf et al., 2020). The AdamW
optimizer was used for model training. Before starting to train the model, we processed the data, specifically, we filtered user tags and some invalid characters. We kept the URL, which had a positive effect on intimacy analysis. During the training phase, we evaluated the performance of the model every 100 steps and retained the parameters of the model that performed best on the validation set. The hyperparameters settings adopted are shown in Table 2. All models were trained on NVIDIA Geforce GTX 3090 GPU.

heading: 4.3 Results and Discussions

heading: 4.3.1 Adversarial Training Analysis
text: We set up the experiment to verify the effectiveness of adversarial training, as shown in Table 1. From the experimental results, we can observe that the performance of the model’s intimacy prediction is improved after adding FGM. It shows that adversarial training can improve the robustness of the model.

heading: 4.3.2 Ordinal Regression Analysis
text: The results in Table 1 demonstrate the effect of the ordinal regression loss. Compared with MSE loss, the system performance is significantly improved by using ordinal regression loss, which further confirms that using ordinal regression as the loss function to help model capturing the interval label order rather than fit the continuous value will benefit more for predicting multiple-level discrete ordinal labels. At the same time, it can also be observed from the results that finer-grained interval division can improve the performance of the model.

heading: 4.3.3 Data Augmentation Analysis
text: In order to analyze whether data augmentation will improve the generalization of the model, we designed three experiments 6 languages, 4 languages and 4 languages (Substitution) to compare our model, and the details are shown in Table 1. Translation augmentation and word-level noise on seen language data can bring observable benefits to the model. However, the model’s performance will suffer if noise is added while expanding unseen language data. This may be because using training sets to extend unseen languages introduces the distribution error, and adding perturbations to the data will cause the error to progressively grow.

heading: 4.3.4 Training Strategy Analysis
text: Since the text features of intimacy between different languages are not the same, the model will have deviations in learning performance due to the
different text language combinations in the training set during model training. As the results in the Table 1, compared with the single combination of training set languages, it is better to alternately use combination methods of different languages in one training process, which proves that the training strategy we designed is more reasonable.

heading: 5 Conclusion and Future Work
text: In this paper, we present an intimacy analysis system by deploying various optimization methods, including adversarial training, data augmentation, ordinal regression loss and special training methods, to SemEval-2023 Task 9.
Compared with the method of sharing the same task in multiple languages, building a multi-task model for different languages is a more worthy of analysis. Multilingual research based on pretraining is more dependent on the diversity of languages contained in the pre-training corpus. For low-resource languages, the performance of the model will get stuck in a bottleneck. In the future, we will further explore the relationships between different linguistic features and build a stronger multilingual multitasking system for the intimacy analysis.

heading: Acknowledgement
text: We thank our anonymous reviewers for their helpful comments. This work was supported by grant from the National Social Science Fund of China (Grant No. 21BXW047).

heading: Future Work
text: In the future, we will further explore the relationships between different linguistic features and build a stronger multilingual multitasking system for the intimacy analysis