SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis
Year: 2023

Abstract:
Intimacy is an important social aspect of language. Computational modeling of intimacy in language could help many downstream applications like dialogue systems and offensiveness detection. Despite its importance, resources and approaches on modeling textual intimacy remain rare. To address this gap, we introduce MINT, a new Multilingual intimacy analysis dataset covering 13,372 tweets in 10 languages including English, French, Spanish, Italian, Portuguese, Korean, Dutch, Chinese, Hindi, and Arabic along with SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis. Our task attracted 45 participants from around the world. While the participants are able to achieve overall good performance on languages in the training set, zero-shot prediction of intimacy in unseen languages remains challenging. Here we provide an overview of the task, summaries of the common approaches, and potential future directions on modeling intimacy across languages. All the relevant resources are available at https: //sites.google.com/umich.edu/ semeval-2023-tweet-intimacy.

heading: 1 Introduction
text: Intimacy has long been viewed as a primary dimension of human relationships and interpersonal interactions (Maslow, 1981; Sullivan, 2013; Prager, 1995). Existing studies suggest that intimacy is an essential social component of language and can be modeled with computational methods (Pei and Jurgens, 2020). Recognizing intimacy can also serve as an important benchmark to test the ability of computational models to understand social information (Hovy and Yang, 2021).
Despite the importance of intimacy in language, resources on textual intimacy analysis remain rare. Pei and Jurgens (2020) annotated the first textual intimacy dataset containing 2,397 English questions, collected mostly from social media posts and fictional dialogues. However, such question phrases
are often used primarily for interrogative situations, and, as such, models trained over the dataset may not generalize well to other types of text.
To further promote computational modeling of textual intimacy, we introduce a new Multilingual textual intimacy dataset (MINT). The training data in MINT covers tweets in 6 languages, including English, Spanish, French, Portuguese, Italian, and Chinese, which are languages used by over 3 billion people on Earth, in The Americas, Europe, and Asia. A total of 12,000 tweets are annotated for the six languages. To test the model generalizability under zero-shot settings, we also include small test sets for Dutch, Korean, Hindi, and Arabic (500 tweets for each), which are spoken by over 0.8 billion people around the world.
We benchmarked a series of large multilingual pre-trained language models including XLM-T (Barbieri et al., 2021), XLM-R (Conneau et al., 2019), BERT (Devlin et al., 2018), DistillBERT (Sanh et al., 2019) and MiniLM (Wang et al., 2020). We found that distilled models generally perform worse than other normal models, while an XLM-R model trained over the Twitter dataset (XLM-T) performs the best on seven of the ten languages. While the pre-trained language models are able to achieve promising performance, zero-shot prediction of unseen languages remains challenging, especially for Korean and Hindi.
Based on MINT, we organized SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis, which attracted 45 teams from more than 20 countries. Participants primarily built their systems based on multilingual pre-trained language models like XLM-T (Barbieri et al., 2021), mBERT (Devlin et al., 2018), and XLM-R (Conneau et al., 2019) due to the multilingual nature of this task. However, participants also used a variety of modelbuilding techniques to potentially improve performance, with data augmentation methods like translation, external label generation, and word-level
2235
substitution being among the most heavily used. Overall 24 out of 45 teams are able to achieve >0.7 Pearson’s r on the training languages and 22 teams are able to achieve >0.4 Pearson’s r on the unseen languages, which beat the strong XLM-T baseline.

heading: 2 Task description
text: Intimacy is one of the most fundamental dimensions of human relationships (Prager, 1995). Intimacy has long been used as the index for not only relationships but also the interactions between people (Hinde, 1981). Language plays a central role in interpersonal interactions as it allows people to communicate various types of information, share emotions and build connections (Hartley, 2002). One of the most prominent forms of interaction involving language is self-disclosure, which refers to the process of sharing personal experiences or emotions about themselves (Cozby, 1973). Due to the importance of self-disclosure in interpersonal relationships, existing studies on intimacy in language generally focus on self-disclosure. Those studies usually consider intimacy as the primary dimension of self-disclosure, where deep disclosures of personal emotions or relationships are considered as intimate (Jourard and Lasakow, 1958; Snell et al., 1988). Datasets and NLP models are also built to automatically analyze self-disclosure in communications, especially on social media (Bak et al., 2012, 2014). However, self-disclosure is only part of the language that people use in daily communications. Solely studying intimacy in the context of self-disclosure overlooks many types of language interactions which do not involve self-disclosure. For example, Pei and Jurgens (2020) build the first dataset of English questions and found that questions can also have various levels of intimacy.
While the question intimacy dataset provides resources to study intimacy in language, it only focuses on questions, missing the variation of intimacy in other types of texts. Moreover, the expressions of intimacy depend on the specific types of language. Different languages may have different expressions of intimacy. Therefore, to support further studies on modeling intimacy in different languages, we present SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis.
Multilingual Tweet Intimacy Analysis is a task to predict the intimacy of tweets in different languages. Defining intimacy in language is challenging as intimacy is a natural concept (Prager, 1995).
In our task, we focus on perceived intimacy by asking annotators to give their subjective judgment of tweet intimacy. We draw a diverse pool of native speakers for each language from Prolific.co to participate in our annotation task. Each annotator is asked to answer “How intimate do you think the given tweet is?” using a 1-5 likert scale, which finally leads to 13,372 tweets in 10 languages annotated with an intimacy label.
The training data contains labeled intimacy for six languages: English, French, Spanish, Italian, Portuguese, and Chinese. Pre-trained language models (PLMs) have achieved huge advances in recent years and have shown promising zero-shot abilities for downstream tasks in different languages (Conneau and Lample, 2019; Devlin et al., 2018). To encourage new studies on understanding intimacy in language as well as understanding zeroshot ability of PLMs, we also include four other languages without training data (Dutch, Hindi, Korean, and Arabic). The participants are asked to build models using the six training languages that can predict tweet intimacy from 1 (not intimate at all) to 5 (very intimate). The final model performance is evaluated on the test set in all ten languages and Pearson’s r is used as the final evaluation metric.1

heading: 3 Data
text: We choose Twitter as the source of our dataset because Twitter is a public social media platform that naturally includes multilingual text data and, from our analysis, a fair amount of intimate texts. In this section, we introduce the data collection and annotation process for MINT.

heading: 3.1 Sampling
text: We use tweets sampled from 2018 to 2022. We use the lang_id key in the tweet object to select English and Chinese tweets. For other languages, we use fastText (Joulin et al., 2016b,a) for language identification2 and assign language labels when the model confidence is larger than 0.8. All the mentions of unverified users are replaced with a special token “@user” during pre-processing to remove noise from random and very infrequent usernames.
1An alternative evaluation was considered, namely using the mean r across each language’s performance. However, this approach could allow models to use varying scales across languages, leading to lower comparability of the scores for texts in different languages.
2https://fasttext.cc/docs/en/ language-identification.html
We fine-tune XLM-T, a multilingual RoBERTa model adapted to the Twitter domain (Barbieri et al., 2021) over the annotated question intimacy dataset (Pei and Jurgens, 2020). The fine-tuned model attained a Pearson’s r of 0.80 for English questions3 and we use it to estimate the intimacy of all the collected tweets in 10 languages. Then, in the second step, we split the tweets into 5 buckets based on the estimated intimacy and up-sampled relatively more intimate tweets. We did bucket sampling for 1,000 English tweets and randomly sampled another 1,000 English tweets as well as all the tweets for the rest of the languages4.

heading: 3.2 Annotation
text: We recruited annotators from Prolific.co and paid them $15 USD per hour for their annotations. We set a “first language” requirement during annotator pre-screening. For example, an annotator must meet the requirement of “Spanish as the first language” to annotate the intimacy of Spanish tweets. Intimacy is annotated using a 5-point Likert scale where 1 indicates “Not intimate at all” and 5 indicates “Very intimate”. The annotators are asked “How intimate do you think the given tweet is?” (translated into the corresponding language when tweets in a certain language are presented) and are encouraged to apply their own subjective judgment.
In pilot annotations, we explored 7-point likert scales as well as Best-Worst-Scaling (BWS) similar to Pei and Jurgens (2020) and calculated the Krippendorff’s α to measure inter-annotator agreement (IAA). We found that the 5-point Likert scale annotations (α = 0.38) achieve IAA similar to BWS (α = 0.36) and have higher IAA than a 7-point likert scale (α = 0.25). For each language in the training set, we collected annotations for 2,000 tweets. Because they annotated the intimacy of tweets, the annotators could see sexual or potentially offensive content during annotation. Therefore, we required annotators to be at least 18 years old to work on our task. Each annotator was explicitly notified
3Pei and Jurgens (2020) report a Pearson’s r of 0.82 using RoBERTa-base. Given that XLM-T is pre-trained on tweets, a Pearson’s r of 0.8 is reasonable.
4We intended to do bucket sampling for all the data, however, due to an issue in the pre-processing, we were only able to do it for 1,000 tweets. We conducted further analyses for the potential effect of this error. We found that the distribution of the final annotated intimacy scores are not changed much, while the fine-tuned XLM-T only achieved a Pearson’s r of 0.43 on the random sample, suggesting that the model trained on Reddit questions may not be reliable enough to detect intimacy in tweets.
about the potential for sexual or offensive content and they signed a consent form before starting the annotations.
Each tweet was annotated by 7 annotators and each annotator was shown 50 tweets. After the annotation, each annotator was required to complete a post-study survey about their demographics including gender, age, religion, country, educational background, and occupation. For tweets that were not in the target language or that did not make sense (e.g. random characters), the annotators were instructed to annotate them as Invalid Tweet. We used POTATO (Pei et al., 2022) to set up all the annotation interfaces.

heading: 3.3 Quality control
text: Annotating textual intimacy is challenging because of the subjective nature of intimacy perception and potential individual rating bias. We designed a series of quality control procedures throughout the annotation process: (1) we conducted 10 pilot studies on Prolific.co and revised our annotation procedures according to attained IAA and participant feedback; (2) annotation guidelines for each language were carefully translated by native translators,5 which prompts the annotators to think about intimacy in their own languages; (3) all instructions in the recruitment phase were written in the annotator’s indicated first language in the recruitment phase, which could potentially remove potential spam annotators in crowdsourcing platforms; (4) we randomly inserted two attention test questions6 to identify potential spammers; (5) the annotators were balanced by sex (based on Prolific’s built-in feature) and were also generally diverse regarding other demographics (e.g. the annotators are from 73 unique countries and regions), which allowed us to collect more population-representative ratings.

heading: 3.4 Post processing
text: We first removed annotations from users who failed the attention test. No more than 2 annotators per language were removed in this step, except for Hindi (26 removed), Korean (7 removed), and Arabic (4 removed). To remove potential noise in the
5Chinese, Spanish, Dutch, French, Korean, Portuguese, Hindi, and Italian guidelines were translated by native speakers at Snap Inc. and the University of Michigan. The Arabic guideline was translated by one expert translator and one expert proofreader recruited from Upwork.com;both were paid $13/h.
6“This is a test question, please select N” where N was a random number between 1-5.
crowdsourcing setting, similar to trimmed mean (Millsap and Maydeu-Olivares, 2009), we removed one highest score and one lowest score for tweets with at least five labels. After all the processing, we kept the tweets with at least two valid scores. For external test languages (i.e. Dutch, Hindi, Korean, and Arabic), we only kept tweets with a relatively low label diversity (i.e. standard deviation lower than 1) to ensure a good golden test set for the zeroshot setting7. The final intimacy score is calculated as the mean score of all the remaining labels for each tweet.

heading: 3.5 Annotation result
text: The final dataset includes 13,372 tweets annotated with the textual intimacy score. Table 2 shows the final statistics for the annotated data. We attained moderate inter-annotator-agreement, similar to previous work (Pei and Jurgens, 2020). Given the subjective nature of intimacy perception, we believe that such an IAA score is promising. To further verify the quality of the annotations, we conduct a split-half-reliability test (SHR; Johnson and Penny, 2022): randomly splitting labels into two groups and calculating the Pearson correlation between the aggregated scores from the two groups. All the SHR scores are above 0.63 with an average of 0.68, suggesting that the final aggregated scores are reliable. Figure 5 shows the intimacy distribution of the final dataset. The final dataset for English, Spanish, French, Italian, Portuguese, and
740%, 15%, 17%, 16% of tweets are removed for Hindi, Dutch, Korean, and Arabic respectively.
Chinese is split into training, validation, and test sets following a ratio of 7:1:2, and all the tweets are held as the test set for Arabic, Dutch, Korean, and Hindi.

heading: 4 Baseline Models
text: We benchmark several baseline models on the tweet intimacy prediction task. We compare the following multilingual pre-trained language models:
1. BERT (Devlin et al., 2018): multilingual BERT model.
2. XLM-R (Conneau et al., 2019): multilingual RoBERTa model.
3. XLM-T (Barbieri et al., 2021): Multilingual RoBERTa model trained over 200M tweets.
4. DistillBERT (Sanh et al., 2019): Multilingual distilled BERT model.
5. MiniLM (Wang et al., 2020): Multilingual MiniLM model.
All the models are trained with 10 epochs and the best-performing model is selected based on the dev set8. We train each model with 5 different random seeds and report the mean score. The learning rate is set as 0.001 and the batch size is 64. We use AdamW as the optimizer (Loshchilov and Hutter, 2017).
Table 3 shows the performance of the baselines. We found that XLM-T achieved the best performance over 7 languages, suggesting that domainspecific language model training is beneficial for our tweet intimacy analysis task.
For zero-shot tasks, while XLM-T still performs the best on Hindi and Arabic, XLM-R and MiniLM achieved the best result on Dutch and Korean, respectively. Moreover, the zero-shot performance is generally lower compared with the tasks with
8We evaluate the model performance every 500 steps and choose the best model.
in-domain training, suggesting that the zero-shot task is challenging. We encourage the Task participants to explore different strategies to improve the zero-shot intimacy prediction performance.

heading: 5 Results
text: This task received final submissions from 45 teams. Each team was allowed to submit multiple times but only the final submission was scored on the test set. Of the 45 teams, 22 submitted a system description paper. Here, we summarize the submissions’ approaches by the following aspects: base pretrained language model, data augmentation, and other heuristic techniques.

heading: 5.1 Overall performance
text: Among the 45 teams, 37 submissions are able to beat the M-BERT baseline and 18 participants are
beating the strong XLM-t baseline. The best solution (LAZYBOB (Yuan and Chen, 2023)) achieves 0.616 Pearson’s r on the test set. The highest singlelanguage performance is achieved on Spanish by KING001 (Pearson’s r = 0.784) and the overall correlation on the training languages is generally above 0.7, suggesting that the models are able to accurately predict intimacy in tweets, once fine-tuned on the labeled dataset. However, the overall performance on the unseen languages (Hindi, Dutch, Korean, Arabic) is relatively lower, especially on Hindi and Korean, which might be caused by the different distributions of the test data. Despite this, participants are able to achieve better performances on these unseen languages than the already very strong XLM-T baseline. Detailed descriptions of each system is available in the system papers. Table 5 shows the full leaderboard on this task.

heading: 5.2 Base pre-trained language models
text: Given that predicting the intimacy for unseen languages is an important part of this task and the recent advances of multilingual pretrained language models, most of the participants are using PLMs as the base architecture of their solutions. Participants primarily used three types of PLMs including XLM-t (Barbieri et al., 2021), XLM-R Conneau and Lample, 2019, and TwHin-BERT as their base PLMs. Some participants also explored m-BERT
(Devlin et al., 2018), mDeBERTa, XLNET, and mluke. Some participants (e.g., JELENASTEAM (Lazi and Vujnovi, 2023)) reported experimenting with linguistic features as well as word-level representations but did not use them as the final submissions because of their low performance and difficulty to do cross-lingual tasks.

heading: 5.3 In-domain LM fine-tuning
text: Existing studies suggest that domain-focused language model fine-tuning could help to improve downstream task performances (Gururangan et al., 2020). As in-domain LM fine-tuning requires extensive computational resources, only 2 out of the 22 teams conduct in-domain LM fine-tuning. OPI (Dadas, 2023) fine-tuned 155M tweets over XLMR with mask-language-modeling and achieved the best performance on 6 languages. Moreover, for participants using XLM-T, the XLM model finetuned on twitter data generally achieved better per-
formance than participants using other base PLMs, suggesting that in-domain LM fine-tuning is still a simple yet effective approach to improve intimacy prediction in a multilingual setting.

heading: 5.4 Data augmentation
text: Given the multilingual nature of this task, data augmentation is actively used by participants. Fourteen out of the 22 participating teams used at least some form of data augmentation method for their final submissions.
Translation As part of the task is to predict in-
timacy in unseen languages, translation is widely used by participants as a way of data augmentation. Participants report performance improvements in ablation studies. Some participants (e.g., CHAPAT (Chavan and Patwardhan, 2023)) also tried to translate unseen languages back to English but only found marginal performance gain.
Externally labeled dataset Three out of 22 participating teams leveraged the question intimacy dataset (Pei and Jurgens, 2020) which contains 2397 questions from Reddit, Twitter, books, and movies annotated with intimacy scores from -1 to 1. Ablation studies were not reported in the system paper, therefore, the effectiveness of adding the question intimacy data remains unclear. UZH_CLYP (Michail et al., 2023) uses ChatGPT to generate labels for each language and shows performance improvement, suggesting that this approach may be helpful. OPI (Dadas, 2023) also leveraged pseudolabelling (Lee et al., 2013) and achieved the best performance on 6 languages.
Rebalancing Due to the skewed distribution of the labeled data, participants also explored methods to either directly rebalance the training set or leverage learning algorithms to dynamically weight different samples. LAZYBOB (Yuan and Chen, 2023) used weighted random sampler (He and Garcia, 2009) to allow the model to focus more on interesting examples. SEA_AND_WINE (Chen et al., 2023b) designed an ad-hoc sampling strategy during data augmentation to balance the positive and negative samples. WADER (Suri et al., 2023) designed a weakly-labeling framework that includes distribution-based sampling and difference-based sampling methods to attain a more balanced training set.
Others Participants also explored other methods of data augmentation. HULAT (Segura-Bedmar, 2023) explored synonym replacement using EDA and ZBL2W (Zhang et al., 2023) experimented with word-level substitution. However, it is unclear whether these methods are helpful, due to the lack of ablation studies.

heading: 5.5 Ensemble
text: Many participants chose to combine multiple models’ outputs as the final scores. The participants mostly explored two ways of ensembling: (1) finetune the same multilingual pre-trained model with different seeds and ensemble them for the final submission or (2) use separate models for different languages. Participants using ensemble methods generally performed better than others. The top 4 submissions (LAZYBOB (Yuan and Chen, 2023), UZH_CLYP (Michail et al., 2023), OPI (Dadas, 2023) and TMN (GLAZKOVA, 2023)) all used ensembles as part of their solution.

heading: 5.6 Other methods
text: While most of the systems generally focus on the methodologies above, some also explored other methods to improve their model performance. SEA_AND_WINE (Chen et al., 2023b), ZHEGU (He and Zhang, 2023), and ODA_SRIB (Kumar et al., 2023) explored different loss functions other than the standard MSE loss. All report performance gain with their specially designed loss function. Despite the regression nature of this task, ODA_SRIB explores binary classification problems with soft labels and optimizes the model with a BCE loss. Through ablation studies, ODA_SRIB found that a binary classification setting with BCE loss attained better performance than a regression with MSE loss.
Besides modifying the loss function, adversarial methods are also adopted by some participants: ODA_SRIB (Kumar et al., 2023) and LAZYBOB (Yuan and Chen, 2023) found that Adversarial Weight Perturbation (AWP) improves the final prediction performance.

heading: 6 Discussion
text: Intimacy is an important dimension of human relationships and language. Predicting intimacy across languages is a difficult task, because language is embedded in different cultures with different perceptions of intimacy. To address this issue, we built
MINT, the multilingual intimacy dataset. Forty-five participating teams have submitted their solutions to our multilingual tweet intimacy prediction task. Overall the participants are able to achieve relatively high performance (Pearson’s rs > 0.7) on languages that are in the training set. While the performance on the unseen languages is relatively lower, many participants are able to improve their performance over the already-strong XLM-T baseline.
We observe a clear trend that most of the participants focused on the data augmentation and ensemble methods, with less effort spent on traditional feature engineering. Moreover, despite the fact that in-domain LM fine-tuning has been found to be helpful for downstream tasks (Gururangan et al., 2020), only a very limited number of participants considered in-domain LM fine-tuning as part of their solution. This is potentially due to the computational resources required for large-scale fine-tuning.
Despite recent advances in prompting, only one participant explored soft prompts, suggesting that the regression task remains challenging for prompting large language models. Our dataset provides a valuable multilingual resource for studying prompting regression tasks in the context of understanding social information. We encourage future researchers to explore this direction.
As a first step towards understanding textual intimacy in language, in this task, we only released aggregated labels from all the annotators. However, the perception of intimacy may vary with factors like gender, age, and culture. We are planning to release detailed background information of all the annotators. This will allow a potential follow task to predict intimacy ratings that incorporate annotators’ background information.

heading: 7 Conclusion
text: In this paper, we present SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis along with MINT, the first multilingual textual intimacy analysis containing 13,372 tweets in ten languages (English, French, Spanish, Italian, Portuguese, Korean, Dutch, Chinese, Hindi, and Arabic). We benchmarked a series of multilingual pre-trained language models. Our task attracted participation from 45 teams and 18 teams are able to beat the already strong XLM-T baseline. Our overview indicates that data augmentation, external labels, and ensem-
ble methods are commonly used by the participants, and led to good performance. Overall, participants generally achieved good performance on training languages with an average Pearson’s r above 0.7, showing that the current pretrained language models, once fine-tuned, are able to accurately predict intimacy in various languages. However, the overall performance remains low on unseen languages, suggesting that zero-shot intimacy analysis remains a challenging task. Further research is needed to better analyze textual intimacy across languages.

heading: Acknowledgment
text: Our multilingual tweet intimacy annotation has been approved by the IRB office of the University of Michigan (HUM00214259). This work was supported in part by the National Science Foundation under Grant No. IIS-2143529. We thank Jane Im, Minje Choi, Anubha Singh, Isra Salah, Abdulrahman Alhourani and Maria Casamor Vidal for help translating the task guidelines. We thank all the annotators on the Prolific platform. We thank all the participating teams for their contributions to this task.

heading: Future Work
text: We encourage future researchers to explore this direction. As a first step towards understanding textual intimacy in language, in this task, we only released aggregated labels from all the annotators. However, the perception of intimacy may vary with factors like gender, age, and culture. We are planning to release detailed background information of all the annotators. This will allow a potential follow task to predict intimacy ratings that incorporate annotators’ background information