Year: 2023

Abstract:
Nowadays, intimacy is a fundamental aspect of how we relate to other people in social settings. The most frequent way in which we can determine a high level of intimacy is in the use of certain emoticons, curse words, verbs, etc. This paper presents the approach developed to solve SemEval 2023 task 9: Multiligual Tweet Intimacy Analysis. To address the task, a transferlearning approach was conducted by finetuning various pre-trained language models. Since the dataset supplied by the organizer was highly imbalanced, our main strategy to obtain high prediction values was the implementation of round-trip translation for oversampling and a random approach for undersampling on the training set. Our final submission achieved an overall Pearson’s r of 0.497.

heading: 1 Introduction
text: Intimacy is defined as the set of thoughts and feelings that human beings keep private. There are many ways of expressing a thought and not all are equally intimate, so it is important to know what determines the degree of intimacy of a particular Tweet. To detect this type of language, SemEval 2023 proposed the task 9: Multiligual Tweet Intimacy Analysis (Pei et al., 2023).
The aim of the proposed task is to quantify the intimacy expressed in a set of tweets written in six languages: Chinese, English, Spanish, Portuguese, French and Italian.
We addressed the problem using regression (Specht et al., 1991) and neural networks (Zhou et al., 2015). We utilized different pre-trained models such as RoBERTa (Liu et al., 2019) or BERT (Devlin et al., 2018), available on the Hugging Face platform. Given the strong imbalance in the data, we have focused on applying different oversampling (Liu et al., 2007) and undersampling (Bach et al., 2019) strategies.
The main contributions addressed in this work
are as follows:
• Analysis of sampling methods to mitigate
bias in data.
• Benchmarking of different pre-trained
multilingual and monolingual models.
The rest of this paper is organized as follows: in Section 2 we explain the dataset, the meaning of the labels and data distribution. Therefore, we refer to other studies that contribute to our approach. Section 3 discusses the approaches applied to achieve better model performance. In Section 4, we explain the libraries used. In Section 5, the results obtained in test phase are shown. Finally, conclusions and future works are described in Section 6.

heading: 2 Background
text: The training dataset consists of 9491 tweets written in six languages. The distribution of each of the languages is balanced. Figure 1 shows an example of a dataset row.
The main fields are:
• Text. Represents the tweet to be analyzed.
• Label. Determines the level of intimacy
assigned to the text with a range [1-5], where 1 represents the lowest level and 5 represents the highest level of intimacy.
• Language. It's the language of the text.
In Figure 2, the dataset's distribution is depicted for each language, along with the distribution of intimacy levels across languages. Four intervals have been considered to study the distribution and focus on the problem. We can see how the number
758
of tweets between languages is balanced. However, at lower levels of intimacy we have more examples than at the higher levels. This distribution is repeated across all the languages of the dataset. Therefore, we decided to apply the same techniques over the whole dataset without distinction between languages.
There are not many published studies in the field of intimacy analysis. However, there are many studies related to the classification of levels of aggression and violence in texts and sentiment analysis that can help to better understand our problem and as a starting point to address it. In (Arias and Fabian, 2022) a study on the automatic detection of levels of psychological violence against women in written virtual expressions was conducted. (Zhao et al., 2017) proposed a framework called Weakly-supervised Deep Embedding (WDE), which employs review ratings to train a sentiment classifier.

heading: 3 System Overview
text: This section describes the development of our approach and outlines the steps taken to achieve the results.
Since only one training dataset was provided, for the experimental environment we decided to split the original dataset into a training/validation dataset (80%) and a test dataset (20%).

heading: 3.1 Text pre-processing
text: Several techniques were employed to simplify the text and remove any potential noise. The main techniques that were applied include:
• Converting all characters to lower case
• Removing url and links
• Removing "@user"
For this study, we considered that both hashtags and emojis were relevant to the analysis of
1 https://huggingface.co/xlm-roberta-base
intimacy, so they were not removed from the tweets. Regarding emojis, we consider their usage to be quite prevalent in highly intimate texts and believe that they should be taken into consideration. It was decided to expand the vocabulary of the Transformer models so that they would recognize emojis as one more token without treating them as unknown tokens. The result of a tweet before and after applying the text preprocessing can be seen in Figure 3.

heading: 3.2 Data distribution
text: As previously stated, one of the primary concerns with the training dataset is the significant imbalance in the data. Most of examples across all languages had an intimacy level below 2.5, and as the level of intimacy increased, the number of examples reduced significantly. Different versions of resampling were implemented on the training dataset to improve the value of the Pearson's r on our test dataset.
The same pre-trained model was fine-tuned to test the performance of the different resampling approaches. Specifically, the multilingual xlmroberta-base 1 (Conneau et al., 2019) was the selected model.

heading: 3.2.1 Oversampling
text: Oversampling consists of modifying the distribution of the data by increasing the number of cases of the minority class.
For this task, the round-trip translation (Xie et al., 2019) was used to perform oversampling. In this way, it is possible to have tweets with the same meaning and level of intimacy but with a different structure. An example of paraphrase is shown in Figure 4.
The proposed approaches for implementing
oversampling were:
• O1. Oversampling on tweets with label
values > 2.5.
• O2. Oversampling on tweets with label
values > 3.
• O3. Oversampling on tweets with label
values > 4.
• O4. Oversampling with different ranges.
Tweets whose labels belongs to the value range [1 - 2.5] were not oversampled. Tweets labeled in the value range (2.5 - 4] were duplicated. Tweets whose labels belong to the value range (4 - 5] were repeated three times.
For approaches O1, O2, and O3, tweets meeting the specified criterion were duplicated. Table 1 shows the results achieved for all oversampling techniques using the pre-trained xlm-roberta-base model.

heading: 3.2.2 Undersampling
text: Undersampling is used to reduce the number of instances belonging to the majority class. Random undersampling (RUS) was proposed in (Prusa et al. 2015) to randomly reduce the examples of the majority class. To apply some of the undersampling techniques, the balanced datasets with the oversampling techniques were used. The approaches for undersampling were:
• U1. Undersampling on tweets with label
values < 2.5 from the original dataset.
• U2. Undersampling on tweets with label
values < 2.5 on O1 dataset
• U3: Undersampling on tweets with label
values < 2.5 on O1 dataset.
• U4. Undersampling on tweets with label
values < 2.5 on O4 dataset.
The ratio selected to remove the examples was 50% for U1, U2 and U4 approaches, and 70% for
2 https://huggingface.co/bert-base-uncased 3 https://huggingface.co/mrm8488/bert-spanish-cased-finetuned-ner 4 https://huggingface.co/bert-base-chinese
U3 approach. Table 2 shows a comparison of the results achieved using the different undersampling approaches with the pre-trained xlm-roberta-base model.

heading: 3.3 A model for each language
text: In the previous section, the experimentation and results achieved using different resampling approaches have been described. The experiments were conducted using a single multilingual model to assess the impact of the balancing techniques on the original dataset. Multilingual models generally perform satisfactorily for all languages. However, monolingual models tend to perform better in their own languages because they have been specifically trained in those languages. For this experiment, we decided to test the performance with the oversampling O1 dataset for each of the languages using a pre-trained monolingual model.
The pre-trained monolingual models used for
this experiment were:
5 https://huggingface.co/dbmdz/bert-base-italian-xxl-uncased 6 https://huggingface.co/neuralmind/bert-large-portuguese-cased 7 https://huggingface.co/Jean-Baptiste/camembert-ner
Table 3 shows the results obtained for each of
the languages using the O1 training dataset.

heading: 3.4 Multilingual and monolingual
text: Table 3 shows that the monolingual models performed better in English and Spanish. However, the monolingual models performed worse compared to the multilingual model for the other languages.
For this reason, a multilingual model with the worst performing languages (Chinese, French, Portuguese and Italian) was fine-tuned. The monolingual models which outperformed the multilingual model (English and Spanish) were used. The overall outcome was nearly identical to that of the multilingual model, exhibiting a meager increase of merely 0.04 in Pearson's r compared to the xlm-roberta-base model.

heading: 4 Experimental setup
text: Some of the libraries used for the experimentation were: “NumPy” (Harris et al., 2020), a library for advanced natural language processing; “NLTK” (Loper and Bird, 2002), a tool-kit to work with human language information; “Keras”, a neuralnetwork library; “scikit-learn”, which contains simple and efficient tools for predictive data analysis; and “Pandas” (McKinney, 2010), used for manipulating data. The metric used is Pearson's r from the scipy library as requested by the competition.
For fine-tuning the models, the hyperparameters used were: batch size of 16, learning rate of 5e-5, max length of 128 and weight decay of 0.01, 10 epochs using early stopping with a patience of 3.

heading: 5 Results
text: As can be seen in the previous sections, oversampling O4 was the best performing and most computationally efficient technique. Therefore, we decided to try out other pre-trained multilingual models: xlm-roberta-large8 (Conneau et al., 2019), cardiffnlp/twitter-xlm-roberta-base-sentiment 9 , bert-base-multilingual-uncased 10 (Devlin et al., 2018) and Twitter/twhin-bert-base11 (Zhang et al., 2022). Table 4 shows the results achieved by the different multilingual models.
8 https://huggingface.co/xlm-roberta-large 9 https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment
In the test phase, four new languages were included: Arabic, Korean, Hindi and Dutch. To predict the level of intimacy of these tweets with our models, they were translated into English.
Finally, the predictions obtained by Twitter/twhin-bert-base model and the O4 technique were submitted. The Pearson’s r achieved for the official test dataset was 0.497. We were ranked 36th in the overall intimacy score. In Table 5, the results and overall position obtained for each language are shown.

heading: 6 Conclusions
text: In this paper, the approach of the I2C Group to address Task 9: Multiligual Tweet Intimacy Analysis has been described. Fine-tuning of several pre-trained monolingual and multilingual language models was performed and different resampling techniques were applied.
The best results in the development phase were obtained by using a range oversampling (O4) technique. In this phase, the Pearson’s r achieved was 0.827 but, in the evaluation phase, the value of Pearson’s r was only 0.497. We think that the reason for the difference between the two values
10 https://huggingface.co/bert-base-multilingual-uncased 11 https://huggingface.co/Twitter/twhin-bert-base
could be due to the incorporation of new languages into the test dataset.
For future works, we would like to add additional features to the tweets such as the number and type of emoticons, the use of certain adjectives, etc. We think that the intimacy language vocabulary is very specific, and this could help the prediction models.

heading: Acknowledgments
text: This paper is part of the I+D+i Project titled “Conspiracy Theories and hate speech online: Comparison of patterns in narratives and social networks about COVID-19, immigrants, refugees and LGBTI people [NON-CONSPIRA-HATE!]”, PID2021-123983OB-I00, funded by MCIN/AEI/10.13039/501100011033/ and by “ERDF A way of making Europe”.

heading: Future Work
text: For future works, we would like to add additional features to the tweets such as the number and type of emoticons, the use of certain adjectives, etc. We think that the intimacy language vocabulary is very specific, and this could help the prediction models