DUTH at SemEval-2023 Task 9: An Ensemble Approach for Twitter Intimacy Analysis
Year: 2023

Abstract:
This work presents the approach developed by the DUTH team for participating in the SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis. Our results show that preprocessing techniques do not affect the learning performance for the task of multilingual intimacy analysis. In addition, we show that fine-tuning a transformer-based model does not provide advantages over using the pre-trained model to generate text embeddings and using the resulting representations to train simpler and more efficient models such as Multilayer Perceptron (MLP). Finally, we utilize an ensemble of classifiers, including three MLPs with different architectures and a CatBoost model, to improve the regression accuracy.

heading: 1 Introduction
text: Intimacy analysis is concerned with identifying and analyzing the level of emotional closeness between individuals using natural language (Pei and Jurgens, 2020). This form of analysis has a variety of applications in domains such as social media, healthcare and marketing, providing valuable insights into user behaviour, sentiment and preferences (Prager, 2005).
Despite the significance of intimacy as a reflection of social norms (Pei and Jurgens, 2020; Prager, 2005; Pei et al., 2022), there is limited availability of resources for analyzing it using machine learning techniques. To address this issue, Pei et al. (2022) processed and prepared a multilingual dataset consisting of tweets in ten languages, including English, Spanish, French, Portuguese, Italian, Chinese, Hindi, Korean, Dutch, and Arabic. This dataset has been made publicly available to evaluate the performance of computational tools that analyse intimacy.
This paper presents an ensemble intimacy regression scheme based on an extensive experimental study of several regressor models (Breiman, 2001;
Chen and Guestrin, 2016; Prokhorenkova et al., 2018; Haykin, 1994) and text-to-feature representations (Ramos et al., 2003; Le and Mikolov, 2014; Conneau et al., 2020; Barbieri et al., 2022). Our study shows that training simple models using text embeddings generated by transformer-based models leads to effective and rapid training and accurate results. Our models achieve state-of-the-art performance without fine-tuning a huge and trainingcostly transformer-based model. We carefully optimized the components of our final ensemble, considering the influence of the imbalanced regression task. Our team achieved a rank of 7 in the overall score, 25 using the languages seen during training, and 2 on cross-language transfer learning.
The remainder of this work is structured as follows. The proposed system is described in Section 2. Section 3 reports on our experiments. Finally, conclusions and future directions are discussed in Section 4.

heading: 2 System Description
text: The primary objective of SemEval-2023 Task 9 was to assign a continuous value within the range of [1, 5] to multilingual tweet texts, representing their intimacy level. The higher the value, the higher the intimacy level. This section presents a detailed description of our system, elucidating the text-tofeature generation methods used. In addition, we provide statistics about the given dataset.

heading: 2.1 Dataset
text: The task organisers released the complete data in the training and testing sets. The training set comprised 9,491 texts, each annotated with a continuous value reflecting the level of intimacy. The test set consisted of 13,697 texts from ten different languages mentioned in the Introduction. The training set consisted of six languages: English, Spanish, Portuguese, Italian, French, and Chinese. Table 1 illustrates the number of samples per language on
1225
both the training and testing sets. Notably, the training and testing sets exhibit balanced examples with respect to language. However, we observed a high level of skewness in the intimacy level per language, resulting in an imbalanced regression task. Figure 1 presents the distribution of the intimacy level on the training set per language. Our analysis shows that, in all cases, the distribution of texts per language is skewed, indicating a challenging learning task.
The majority of observations exhibit a low intimacy score, implying that most tweets are not intimate. To mitigate the effect of the skewed distribution, we employed the LogCosh loss function to update a model’s weights whenever applicable. We selected LogCosh over conventional criteria such as MSE and MAE, empirically and based on its robustness to outliers.

heading: 2.2 Pre-Processing
text: We follow the techniques presented in (Symeonidis et al., 2018) for pre-processing. We convert all tweets to lowercase and remove multiple whitespaces. Then, we examined the impact of various pre-processing techniques on regression accuracy. These techniques include removing HTML tags, converting numbers and mathematical symbols to words, replacing @user with atUser, removing Unicode characters by language, replacing repetitions, contractions, elongated words, URL addresses, hashtags and stopwords, as well as the application of stemming and/or lemmatization.
Our experimental results, as we will see later, indicate that pre-processing techniques have minimal impact on the regression accuracy for multilingual tweet intimacy analysis.

heading: 2.3 Text-to-Feature Representation
text: For representing tweets as feature vectors, we consider the following methods:
• CountVectorizer is a widely used method for converting text documents into numerical features based on token counts. This method ignores the order of words in the document, focusing only on their frequency.
• TF-IDF is another widely used technique for document representation. It measures the importance of a term in a document by assigning a higher weight to words that appear more frequently in the document, improving the overall representation.
• Doc2Vec (Le and Mikolov, 2014) is a neural network model that learns a vector representation of documents. It captures the meaning of words within a document, providing a more accurate analysis at the document level than CountVectorizer and TF-IDF.
• XLM-R (Conneau et al., 2020) is a pre-trained transformer that models multilingual representations. It is designed for cross-lingual language understanding and achieves state-ofthe-art performance in various multilingual tasks.
• XLM-T (Barbieri et al., 2022), which is based on XLM-R, is a transformer model designed explicitly for multilingual tweet-based
datasets. XLM-T has been shown to outperform other state-of-the-art approaches in tasks such as sentiment analysis.
Our experimental study, as we will see later, indicates that training on transformer-based embeddings results in significant improvements with respect to regression accuracy compared to traditional methods.

heading: 2.4 Machine Learning Models
text: The system implementation uses Python 3.10 and relies on several open-source frameworks, including scikit-learn, PyTorch, CatBoost, XGBoost and transformers. The following regression models were employed:
• Linear Regression is a simple baseline that models the relationship between a dependent variable and one or more independent variables.
• k-Nearest Neighbors is a lazy learning method, identifying the k training instances closest to a given sample.
• Random Forest (Breiman, 2001) is an ensemble approach that combines multiple decision trees to predict the value of a given sample.
• XGBoost (Chen and Guestrin, 2016) is a popular gradient-boosting algorithm by iteratively training a sequence of weak decision trees and adjusting the weights for the wrong predicted instances.
• CatBoost (Prokhorenkova et al., 2018) is another gradient-boosting algorithm using a combination of ordered boosting, random permutations and gradient-based sampling.
• MLP (Haykin, 1994) is a simple feedforward neural network consisting of multiple layers of interconnected nodes. Unless stated otherwise, the architecture of the MLP consists of one hidden layer with 64 units.
Our experimental results, as we will present after, indicate that both the CatBoost and MLP regressors attain state-of-the-art performance in multilingual tweet intimacy analysis. Notably, the performance achieved by these models is comparable to the finetuning of large transformer-based models.

heading: 2.5 Evaluation Measures
text: The evaluation measure the task organisers consider is the Pearson correlation coefficient (r) between the predicted and actual sentiment levels. Additionally, given that we deal with a regression task, the Mean Squared Error (MSE) and Mean Absolute Error (MAE) were calculated as commonly used measures for evaluating model performance.

heading: 3 Experiments
text: This section presents the results from utilizing the pre-processing techniques, vectorizers, and machine learning models discussed in Section 2. Additionally, we compare our findings with the corresponding reported results obtained from fine-tuned transformers. Unless stated otherwise, all experiments used 5-fold cross-validation after shuffling the dataset to improve generalisation.

heading: 3.1 Impact of Pre-processing Techniques
text: We begin our experimental study by assessing the influence of pre-processing on regression accuracy. We select CatBoost as the learning algorithm, a model known for its fast training and high predictive accuracy in related tasks, and the CountVectorizer which enables rapid text conversion to feature representations.
From Table 2, it is evident that pre-processing techniques have minimal impact on learning performance. While some techniques, such as numbers to words and Unicode removal, have demonstrated a slightly higher Pearson’s r correlation coefficient, the difference amounts to only approximately 2%. This observation holds by applying combinations of pre-processing techniques, which are not included in the manuscript due to space limitations. Thus, we assert that pre-processing techniques have little significance in the context of intimacy analysis, in contrary to the previous
study of Symeonidis et al. (2018). Consequently, we proceeded with experiments using raw text data.

heading: 3.2 Encoders and Machine Learning Model Evaluation
text: We continue our experimental study using the raw data and 5-fold cross-validation without applying pre-processing techniques. We utilize the vectorizers along with the machine learning models mentioned in Section 2. We have chosen not to finetune a transformer-based model due to their timeconsuming training.
Figure 2 illustrates the average Pearson’s r for each considered model and vectorizer. It is evident that transformer-based embeddings, specifically XLM-R and XLM-T, exhibit a significant improvement compared to CountVectorizer, TFIDF, and Doc2Vec. It is worth noting that Doc2Vec fails to achieve convergence, which we attribute to the nature of our multilingual task.
To calculate the relative improvement (RI), we consider the MLP model and use the following formula:
RI = ((rnew − rold)/rold)× 100, (1) where rnew denotes the Pearson’s r for the transformer-based embeddings and rold is the Pearson’s r for the TF-IDF vectorizer. The results show a remarkable improvement of approximately 54.5% and 76.5% for XLM-R and XLM-T, respectively, compared to TF-IDF in terms of performance gain. Comparing the relative improvement between XLM-R and XLM-T, the latter achieves approximately 14.2% higher r.
Regarding model-specific results, we observe that MLP and CatBoost outperform the other models in all considered cases.
For completeness, Table 3 presents the MSE and MAE obtained for each model and vectorizer to present sufficiently the Regression experimental results. The highest-performing model per vectorizer is denoted in bold, and the second-best model is underlined. The results confirm that MLP and CatBoost outperform the other models and that the XLM-T model performs better than the XLM-R model.

heading: 3.3 Cross-language Transfer Learning
text: Based on the findings in the previous section, we have opted to use MLP and CatBoost for evaluating the cross-language transfer learning task using XLM-T. For this task, we have randomly selected a subset of the given dataset to act as the validation set. Moreover, we exclude the text from a single language it has never seen during training. With the cross-language task, we ensure the generalization and robustness of the considered approach.
Figure 3 presents the obtained Pearson’s r by excluded language. Overall, both MLP and CatBoost perform similarly across all languages, with MLP outperforming CatBoost in all cases. Interestingly, both classifiers achieve their highest performance score in English, which may be due to the large English corpus of the pre-trained model. The lowest
accuracy scores are obtained for Chinese, which is a more challenging language for natural language processing due to its complex character set and grammar.

heading: 3.4 Final Ensemble Model and Results
text: In the previous experimental sections, we demonstrated that combining XLM-T with MLP or CatBoost outperforms other approaches. In our final solution, we employ an ensemble consisting of three MLPs and the CatBoost model. MLP1, which was considered for all previous experiments, has a single hidden layer with 64 units. MLP2 has three hidden layers of size 256, 168, and 64. MLP3 also has three hidden layers of size 1024, 128, and 64.
All models are trained by optimizing the LogCosh loss function due to the imbalanced nature of the task, as mentioned in Section 2. The dataset is randomly split into 80% for training and 20% for testing. After training, the results indicate that MLP3 > MLP2 > MLP1 > CatBoost in terms of Pearson’s r. For the final solution, a weighted voting approach is utilized using the formula
Prediction = 0.1MLP1 + 0.3MLP2 + 0.5MLP3 + 0.1CatBoost (2)
which is found based on a small grid search, which involves tuning the weights assigned to each model and ensuring that their sum equals one. The selection of optimal weighting is based on the performance of the ensemble method measured by the highest r observed in the test set.
The results of the organizer’s test set across all languages, as well as the comparison with fine-tuned transformers (obtained from (Pei et al., 2022)), are presented in Table 4 for a Pearson’s r. XLM-T and XLM-R appear to perform relatively well across all languages, with Hindi being
the only language with significantly lower performance. Distill, MiniLM and BERT perform somewhat worse, with BERT showing poor performance across all languages except for English and Spanish. Interestingly, our model’s performance, consisting of an ensemble of models trained with the embeddings generated by the pre-trained XLM-T, is consistently high across all languages, outperforming all other models except for XLM-T in some cases.
r

heading: 4 Conclusion
text: Twitter provides a platform for users to express personal opinions and share emotions, making it an attractive research area for analyzing intimacy and social norms. This work presents an approach for Twitter sentiment analysis that involves training simple and efficient models using embeddings generated by pre-trained transformers. Our approach is comparable to fine-tuning large transformer models and proposes that the complex and challenging task of fine-tuning transformer models can be replaced with more straightforward and highly efficient models trained using embeddings derived from pre-trained transformers. This conclusion could potentially lead to significant improvements in the efficiency and effectiveness of natural language processing and machine learning applications. A limitation of the presented method for Intimacy analysis is that it does not address the issue of imbalanced regression tasks, which could compromise the quality of the results. Future work in this domain could appropriately address the imbalanced regression task to achieve higher-quality outcomes.

heading: Future Work
text: Future work in this domain could appropriately address the imbalanced regression task to achieve higher-quality outcomes