lazybob at SemEval-2023 Task 9: Quantifying Intimacy of Multilingual Tweets with Multi-Task Learning
Year: 2023

Abstract:
This study presents a systematic method for analyzing the level of intimacy in tweets across ten different languages, using multi-task learning for SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis. The system begins with the utilization of the official training data, and then we experiment with different finetuning tricks and effective strategies, such as data augmentation, multi-task learning, etc. Through additional experiments, the approach is shown to be effective for the task. To enhance the model’s robustness, different transformerbased language models and some widely-used plug-and-play priors are incorporated into our system. Our final submission achieved a Pearson R of 0.6160 for the intimacy score on the official test set, placing us at the top of the leader board among 45 teams.

heading: 1 Introduction
text: Intimacy is a fundamental social aspect of language. Understanding the dynamics of intimacy in conversation is important for a variety of practical applications. Research on intimacy in conversations between people, particularly in the context of chance encounters, such as the phenomenon of the "stranger on the train," can provide valuable insights into human communication and relationships (Griffin, 2006; West et al., 2010; Farber, 2003). Studying the content on different social platform can help researchers to better understand how people establish and develop relationships through conversation. Social workers can also benefit from an understanding of how intimacy is established and maintained in conversation to help clients build stronger relationships with others (Labov, 1972; Brown and Levinson, 1978).
The multilingual tweet intimacy task was hosted by the University of Michigan and Snap Inc. This task focuses on predicting the intimacy of tweets in 10 languages (Pei et al., 2022), where 6 lan-
guages with training data (English, Spanish, Portuguese, Italian, French, Chinese), and 4 zero-shot languages (Korean, Arabic, Hindi, Dutch). Our submission achieved the first place overall score in these 10 languages.
In this paper, we demonstrate the following contributions: (1) We designed a novel multi-language and multi-task training framework that can simultaneously train data from all languages. (2) Based on 10 million crawled Twitter, book, movie, and Reddit data in various languages, we conducted domain pre-training of multiple pre-training models to enhance the model’s prediction ability in that domain. (3) During the training process, we also adopted contrastive learning strategies such as group-layerwise learning rate decay (GLLRD), adversarial weight perturbation (AWP) to enhance the semantic understanding and robustness of language models. (4) For zero-shot language prediction, we tried bidirectional translation data augmentation to enhance the model’s prediction ability for small languages in this task.

heading: 2 Background
text: Sociologists and linguists have conducted many studies on how people use language to measure the relationships between individuals. By analyzing conversations, researchers can estimate the relationship between people, such as friendly, neutral, or adversarial relationship (Clark and Schunk, 1980; Weber, 2008; Locher and Graham, 2010). Recently, some scholars have proposed computational models to evaluate the intimacy of a conversation, which greatly reduces the workload of linguistic research (Pei and Jurgens, 2020).
By training on a large amount of annotated data, the proposed model can learn to better understand the semantics of language and accurately predict the intimacy level in multiple languages. This task is significant for comprehending how language is utilized to convey emotions and relationships, and
927
has practical applications in fields such as psychology, social work, and marketing.

heading: 3 System Overview
text: To better understand the content and semantics of the training and testing sets, which are drawn from Twitter posts and comments, we use large language models trained specifically on tweets, for language encoding. We also use additional datasets from Twitter and Reddit, as well as some from books and movies to pretrain language models. At the same time, we used multiple fine-tuning tricks to improve the model’s performance. To make the distribution of the predicted intimacy scores are close to the distribution of the true annotation scores, we designed a multi-task training mode to adjust the distribution of the predicted labels. We also used model fusion methods to generate more robust results. The following section introduce the detail of language model, pretraining method, fine tuning tricks, multi-task setting and ensemble method, etc. were used in this paper.

heading: 3.1 Language Models
text: Considering the source and language of the data, we tested about many pretrained language models in different strategy to understand the semantics of the tex. All of the pretrained models mentioned can be downloaded from Hugging Face 1. This section introduces some effective pretrained models that were used, along with their basic information and advantages.
Due to the fact that the task dataset came from Twitter, was multilingual, and included zero-shot languages, we mainly used large multilingual models that has been pretrained on tweets. Additionally, we used different models for Chinese and Spanish according to related work (Pei et al., 2022). To enhance the performance of predicting zero-shot languages, we used a data augmentation strategy by using translation. Both the Hindi and Korean which shown a very poor performance in task paper are consider additional training using translation.
XLM-RoBERTa 2: XLM-R is a multilingual language model that builds on the RoBERTa architecture and outperforming BERT on many multilingual NLP tasks, including language modeling, text classification, and machine translation (Devlin et al., 2018; Conneau et al., 2019). It performs well
1https://huggingface.co/models?sort=downloads 2https://huggingface.co/xlm-roberta-large
on text encoding and generation on many different languages, including low-resource languages that have little training data available. The XLMRoBERTa model is pre-trained on a large corpus of text from more than 100 languages. It is trained using a self-supervised learning approach, which means it learns to represent language by predicting masked words and the order of sentences in a given text.
DeBERTa 3: The main advantage of DeBERTa over BERT is that it uses disentangled attention, which allows the model to focus on different aspects of the input sequence in a more fine-grained way (He et al., 2020). Based on many previous tasks, DeBERTa can capture complex relationships and dependencies between words and characters, especially on English text, resulting in improved performance on a wide range of NLP tasks.
Twitter-XLM-Roberta 4: XLM-T trained start from XLM-Roberta base and continue pre-training on a large corpus of Twitter in multiple languages (Barbieri et al., 2021). For text augmented by translation strategy, we applied the latest version of this model which trained only by English datasets 5. The Twitter-XLM-Roberta model has performed very well on Twitter-related corpora. This is likely due to the fact that it was pre-trained using a large amount of data in domain.
TwHin-BERT 6: TwHin-BERT is based on the BERT architecture and is designed to encode the meaning of tweets in multiple languages, taking into account the specific linguistic and social features of tweets (Zhang et al., 2022). The main advantage of TwHin-BERT compared to other BERTrelated models is its ability to capture the social and linguistic features that are unique to tweets, such as hashtags, emoticons, and slang words. These features are often used to convey sentiment, humor, or irony in tweets, and can significantly impact the meaning of a tweet.

heading: 3.2 External data and Pretrained Models
text: In development stage, we realized the domianbased data can improve the model preformance. We select TwHin-BERT, which shown a best performance in validation stage, as a base model to
3https://huggingface.co/microsoft/deberta-v3-large 4https://huggingface.co/cardiffnlp/twitter-xlm-roberta-
base-sentiment 5https://huggingface.co/cardiffnlp/twitter-roberta-basesentiment-latest 6https://huggingface.co/Twitter/twhin-bert-large
pretrain a specific model for this task. The external data are processed and provided by the organizer (Pei and Jurgens, 2020). However, due to the time and cost, we only sample some part data ( 10M out of 80.5M) from the external data to pretrain the TwHin-BERT model. The external data we used to pretrain include 1M English tweet 7 and 9M from Reddit, movies and books (Pei and Jurgens, 2020; Louviere et al., 2015; Kiritchenko and Mohammad, 2017). We also test the performance of this self-trained model. In the following section, the self-pretrained model is referred as "IntimacyTwHin-Bert".

heading: 3.3 Fine-tuning Strategies
text: In this task, we applied some popular deep learning fine-tuning techniques to enhance the model robustness, such as weight sampling, constrastive learning, dropout, attention mechanism, etc. The performance and effectiveness of different trick are also test in the section of Experimental setup.

heading: 3.3.1 Weighted Random Sampler
text: In deep learning, a weighted random sample is used for giving more importance to certain samples (He and Garcia, 2009). The advantage of using a weighted random sample over a simple random sample is that it allows the model to focus more on interested samples. It also can help to mitigate the negtive effects of noisy or irrelevant data. By assigning lower weights to such examples, the model is less likely to be biased.

heading: 3.3.2 Attention
text: In this task, we applied attention on text embedding output from a pretrained model (Vaswani et al., 2017). A set of attention weights is calculated from the output embedding using attention mechanism. It learns to assign weights to different parts of the input data based on their importance or relevance to the task at hand. Attention focus on different parts of the input data based on their relevance or importance, enabling the model to achieve better accuracy.

heading: 3.3.3 Multi-sample Dropout
text: Dropout is a regularization technique that is commonly used to prevent overfitting in neural networks (Srivastava et al., 2014). It randomly drops some nodes in a neural network during training to prevent the network from relying too heavily on
7https://developer.twitter.com/
any single feature. However, dropout technique shows less effective for regression tasks than for classification tasks. In this task we set a dropout with a rate of 0 for regression tasks (Bahdanau et al., 2014).
For classification problem in multi-task learning, we incorporate the multi-sample dropout layer technique into the output of pretrained language models to improve the model generalization. Multisample dropout is a regularization technique which can accelerate training convergence and improve the model generalization compared to the network structure with a traditional dropout layer (Inoue, 2019). Four dropout layers with a dropout rate equals 0.4 were applied to the pooling layer output from the pretrained model.

heading: 3.3.4 Adversarial weight perturbation
text: Adversarial weight perturbation (AWD) is a contrastive learning technique that aims to improve the quality of learned representations by adding perturbations to the weights of the neural network. The basic idea is to introduce small and designed perturbations to the weights of the neural network in order to make the learned representations more robust to adversarial attacks (Wu et al., 2020; Liu et al., 2020; Tian et al., 2019). This technique usually can applied to improved performance on downstream tasks.

heading: 3.3.5 Group-layer wise learning rate decay
text: Group-layer wise learning rate decay (GLLRD) is a technique used in contrastive learning to adjust the learning rate of different layers in a neural network during training (Tenney et al., 2019; Tan and Le, 2019). It divide the layers of the neural network into different groups based on their depth or complexity, and then apply a different learning rate schedule to each group. The advantage of using GLLRD is ro prevent overfitting and improve the training efficiency and effectiveness of the neural network.

heading: 3.4 Ensemble
text: In the final version of our submission, we utilized a combination of fusion strategies. The version that we ultimately submitted involved merging 25 sets of results from the Multi-learning Task, all of which were trained using Twhin-BERT. We based this decision on our previous experimentation and validation scores obtained during the evaluation stage. Furthermore, we substituted the Chinese
results by training the model using XLM-R, while the Spanish results were replaced with a model that was trained using XLM-T.

heading: 4 Experimental setup
text: In this competition, we applied 3 strategy with different languages models and datasets to predict the intimacy score of tweets in 10 languages. Based on the organizers’ description, this task is more like a classification task since the initial task of the annotators was to label the data into one of the five levels of intimacy on a scale of 1-5. However, the true training data provided is the average value of multiple types of annotation data, which is not a standard Multi-level classification.
Considering the final evaluation metric for this competition is Pearson correlation coefficient (Pearson R) (Yeager, 2019), we consider this task as a regression problem for prediction. The goal of the task is to predict a continuous value of intimacy score rather than a discrete class label, and the Pearson r evaluates the correlation between the predicted values and the true values.
The three strategies are described as below. Table 1 shows the overall performance using different strategies. Table 2 shows the performance in each single language with different languages models. TwHin-BERT-large trained using multi-task learning provide the best performance all over all the method.
Strategy 1 (Single task on official data): The first strategy we used is to treat this task as a pure regression problem, using the raw data provided by the organizer for training. Since the data was already cleaned, we only performed some basic text processing on features unique to tweets. We then added an attention layer and fine-tuned the model using various language models.
Strategy 2 (Single task on augmented data): Due to limited size of our training dataset, we perform data augmentation through Google Translation Cloud API 8 to translate all data into English. We then fine-tuned a well-performing English pretrained language model using the translated data. This approach allowed the model to better handle the linguistic features, particularly in zero-shot tasks.
Strategy 3 (Multi task with Pseudo labels): The label distribution from 1 to 5 is unbalanced and has very few labels greater than 3 (See Table 8https://cloud.google.com/translate
3 in Appendix). When we treat this task as a pure regression problem, we used Mean Squared Error (MSE) as the loss function and Pearson R as the evaluation metric. This can lead to a general problem caused by the convergence of the MSE loss function, where the model will pursue a similar average of the true and predicted values for each batch to achieve a higher Pearson R score during validation.
To address this issue, we firstly tried removing the weighted sampler in the batch sampling process. However, this strategy did not work and actually had a negative impact. Therefore, we ultimately adopted a multi-task learning approach to train the model. We converted the official labels into pseudo-labels based on their specific numerical values. Specifically, we divided the labels into four categories based on four intervals: [1-2), [2, 3), [3, 4), and [4,5]. In Multi-task learning, we trained the regression with MSE loss and classification with cross entropy loss at a same time. In validation stage, we still only applied the Pearson R as the metric. Figure 1 shows the label prediction using multi task learning is closer to the annotated distribution rather than single task distribution.

heading: 5 Result and Discussion
text: In conclusion, Twhin-BERT is the best language model that can understand the semantic of competition tweets, both in zero-shot and trained languages. Data augmentation by translation helps improve the accuracy of prediction of zero-shot languages, such as Dutch and Arabic. Multi-task learning help to amend the distribution of label prediction and improve the prediction accuracy. Contrastive learning strategies can slight improve the performance of
label prediction but it requires a lot more effort on training.
Additionally, we also tried other approaches which not demonstrated in details. For instance, we tried training each language separately, but due to the small size of the training data, the validation score was relatively low. We also tried another multi-task learning approach, where each language was treated as a separate task and trained separately as a regression problem. However, the performance of this approach was even worse than that of the single task, probably because of the lack of training data. Finally, to ensure the prediction performance for zero-shot languages, we also tried translating all of the training data into Korean and Hindi to improve the performance of these two languages. However, the results were not as good as using the multi-language strategy directly, possibly due to the noise introduced by translation.

heading: 6 Conclusion and Future Work
text: Overall, research on intimacy in conversations between people can provide valuable insights into social relationships. This work developed a pretrained and multilingual language model to determine the intimacy level of tweets. The model perform well on both trained language and zero-shot languages. In the future, research can consider adding feature engineering and tuning more deep learning tricks to improve the performance of the model prediction. Additionally, this study demonstrated that pretraining a model with domain knowledge can improve its robustness, so it is important to consider pretraining the model with relevant content before using it to predict intimacy scores for specific types of conversations.

heading: Future Work
text: In the future, research can consider adding feature engineering and tuning more deep learning tricks to improve the performance of the model prediction. Additionally, this study demonstrated that pretraining a model with domain knowledge can improve its robustness, so it is important to consider pretraining the model with relevant content before using it to predict intimacy scores for specific types of conversations