Arizonans at SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis with XLM-T
Year: 2023

Abstract:
This paper presents the systems and approaches of the Arizonans team for SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis. We finetune the Multilingual RoBERTa model trained with about 200M tweets, XLM-T. Our final model ranked 9th out of 45 overall, 13th in seen languages, and 8th in unseen languages.

heading: 1 Introduction
text: Intimacy is an integral part of human relationships and an essential component of language (Pei and Jurgens, 2020). Pei and Jurgens also propose that intimacy can be modeled with computational methods and that analyzing textual intimacy can potentially reveal information about social norms in different contexts.
With social media taking an increasingly large part in people’s lives, intimacy has found itself a new platform for display. In these digital social contexts’, Twitter is an important source for analyzing textual intimacy. To further promote computational modeling of textual intimacy, Pei et al. (2023) make use of publicly available Twitter data to create and annotate a textual intimacy dataset called MINT from the following 6 languages: English, Spanish, French, Portuguese, Italian, and Chinese. They also annotate a smaller set of tweets in Dutch, Hindi, Korean, and Arabic to allow for zero-shot prediction experiments. SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis (Pei et al., 2023), asks participants to predict the intimacy of tweets in the aforementioned 10 languages.
In this paper, we explain our approach to intimacy analysis: finetuning XLM-T, the Multilingual RoBERTa model trained with about 200M tweets. To improve performance we experimented with different hyperparameters and changed the training data label distribution by duplicating tweets with certain labels. We discuss the issues we ran into while working on our model, such as the imbalanced range of intimacy scores in the provided
training dataset. Our final submission ranked 9th out of 45 overall, 13th in seen languages, and 8th in unseen languages. We release our code at https: //github.com/beyzabozdag/tweet-intimacy.

heading: 2 Background
text: Intimacy is an essential part of language, however resources on textual intimacy analysis remain rare (Pei et al., 2023). The first textual intimacy dataset was annotated by Pei and Jurgens (2020) with 2,397 English questions collected mostly from social media posts and fictional dialogues. However, Pei et al. (2023) claim that models trained over phrases in the question structure might not generalize well to text in other forms of languages.
To work on a more generalizable dataset and promote computational modeling of textual intimacy, Pei et al. (2023) proposed their new dataset MINT, which is a Multilingual intimacy analysis dataset covering 13,384 tweets in 10 languages including English, French, Spanish, Italian, Portuguese, Korean, Dutch, Chinese, Hindi, and Arabic. Each tweet in the dataset has a label between 1 and 5 as an intimacy score; 1 indicating “Not intimate at all’ and 5 indicating “Very intimate”. These scores were generated by taking the mean of all labels given by different annotators.
SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis asks participants to predict the intimacy of tweets in the 10 languages in MINT. They benchmark some of the large multilingual pretrained language models such as XLM-T (Barbieri et al., 2021), XLM-R (Conneau et al., 2019), BERT (Devlin et al., 2018), DistillBERT (Sanh et al., 2019) and MiniLM (Wang et al., 2020), and report that distilled models generally perform worse than other normal models (Pei et al., 2023). The XLM-T model which is a Multilingual RoBERTa model trained with about 200M tweets, is the bestperforming model in the task organizers’ reports, and the model we experiment with.
1656

heading: 3 Methodology
text: We decided that using the best-performing model in the baseline models listed in Pei et al. (2023) should be the starting point and therefore commenced our experiments with XLM-T. We used the https: //github.com/cardiffnlp/xlm-t quick prototyping Google Colab notebook for fine-tuning.
Our initial goal was to first reach the baseline performance for each language as reported in the task paper (Pei et al., 2023), and then improve from that point. Since there were no data available in the zero-shot languages (Hindi, Korean, Dutch, Arabic), we shifted our focus to training individually on the 6 languages; English, Spanish, Portuguese, Italian, French, and Chinese.
Once we confirmed that we could attain close results to the baseline XLM-T scores for the abovementioned 6 languages by training the model on each language and testing on that language, we started training the model on all available languages making use of XLM-T’s multilingual capacity.
After exploring hyperparameters on the validation data to decide which hyperparameter settings performed the best, we trained a model on the combined training and validation data to make use of all of the provided data.

heading: 4 Experimental Setup
text: The data provided by the task organizers include 9491 annotated tweets in English, Spanish, Portuguese, Italian, French, and Chinese, with almost an equal number of samples from each language, as shown in Table 1. The two languages with the most samples (Chinese and Portuguese) have a total of 1596 tweets, and the language with the least samples (Italian) has a total of 1532 tweets.
To test each individual language in the dataset, we split a given language into 60% train, 20%
validation, and 20% test, setting the random state to 42. This required creating two text files for each split (train, validation, test); one for labels and one for tweets, with the order of labels matching the order of tweets. We set the learning rate to 2e−5, the number of epochs to 10, batch size to 32, max length to 512, and weight decay to 0.01. We performed all evaluations with sklearn.feature_selection.r_regression from the scikit-learn library (version 1.2.1), which computes Pearson’s r between the model predictions and the annotated labels.
Table 2 shows that we were able to reach results close to the baseline performance. We then kept the learning rate and the batch size fixed, and observed the effects of different epochs on the accuracy of the model. Figure 1 shows that performance increased until epoch 20, beyond which there were no further gains. We thus picked 20 epochs for tuning our final model for the test set, training on all provided tweets and labels (not just the training
data we had split out).

heading: 5 Results
text: According to the released evaluation results our submitted model had an accuracy of 0.7129227988 for seen languages, and 0.4441965257 for unseen languages, placing our model 13th and 8th respectively out of 45 submissions, and 9th overall.
For seen languages, our model performed as expected with some improvements for Spanish, Italian, and French relative to the baselines stated by Pei et al. (2023). Table 3 shows that out of all the seen languages, our model performed the worst in Portuguese and English. This is interesting since English and Portuguese are the top two most frequent languages in the 198M tweet corpus which the Twitter-based language model XLM-T is finetuned on (Barbieri et al., 2021)
Another interesting result is the unseen (zeroshot) languages. Our model surpasses the baseline for both Arabic, and Hindi, placing our model on 3rd and 4th for the respective languages.

heading: 5.1 Error Analysis
text: Using the training dataset, we examined the errors of our model. Table 4 shows some errors, along with the difference between our model’s predictions and the human judgments.
Figure 2 shows that our model’s errors are largest for the tweets that have actual intimacy scores within the 4-5 range. Figure 3 explains this trend since the dataset provided by the task organizers has far fewer examples of tweets with intimacy labels in the 4-5 range. We tried addressing this problem by augmenting our dataset with duplicates
of the tweets with intimacy scores in the 4-5 range, but this resulted in a drop in worse performance in our experiments on our train/validation/test split.
An error analysis we would have liked to perform was to examine the whether our models were worse on the samples where humans had greater disagreement. Intimacy is a subjective matter, so labeling can differ greatly from annotator to annotator, in which case the average of the labels might not accurately reflect annotator judgments of intimacy. However, the task organizers were unable to release the individual annotator judgments to us for this analysis.

heading: 6 Conclusion
text: In our work, we have shown that intimacy prediction on multilingual tweets with the XLM-T model performs well in the languages available in the training dataset, and also the zero-shot languages
Dutch and Arabic. Since the model makes the most errors in predicting higher intimacy scores, future work with balanced training datasets in terms of the number of instances of tweets in different ranges of labels would likely yield more accurate predictions. We also would have liked to incorporate the item-level agreement information into our training process to use as weights, and put more emphasis on labels that the annotators agreed more on. This could help the model make more accurate predictions, and be swayed less by the subjective nature of intimacy perception.

heading: Future Work
text: Since the model makes the most errors in predicting higher intimacy scores, future work with balanced training datasets in terms of the number of instances of tweets in different ranges of labels would likely yield more accurate predictions. We also would have liked to incorporate the item-level agreement information into our training process to use as weights, and put more emphasis on labels that the annotators agreed more on. This could help the model make more accurate predictions, and be swayed less by the subjective nature of intimacy perception