MIND at SemEval-2023 Task 11: From Uncertain Predictions to Subjective Disagreement
Year: 2023

Abstract:
This paper describes the participation of the research laboratory MIND, at the University of Milano-Bicocca, in the SemEval 2023 task related to Learning With Disagreements (LeWi-Di). The main goal is to identify the level of agreement/disagreement from a collection of textual datasets with different characteristics in terms of style, language, and task. The proposed approach is grounded on the hypothesis that the disagreement between annotators could be grasped by the uncertainty that a model, based on several linguistic characteristics, could have on the prediction of a given gold label.

heading: 1 Introduction
text: Nowadays, natural language processing models are consistently adopted to address several tasks, such as Abusive and Offensive Language Detection (Pradhan et al., 2020; Kaur et al., 2021) and Hate Speech Detection (Alkomah and Ma, 2022). The state-of-the-art works address both general and specific targets (e.g., women, LGBTQ, immigrants) from a unimodal and multimodal perspective, also considering several languages ((Fersini et al., 2022; Magnossao de Paula et al., 2021; Chakravarthi et al., 2022; Lee et al., 2022; Muaad et al., 2022). However, although these tasks are subjective, the majority of the proposed works are based on the assumption that a unique perception and interpretation exists for each instance. However, this assumption does not reflect what happens in the real world, where multiple readers could have different points of view and understanding, resulting in a sort of disagreement. Such disagreement, which typically reflects the subjectivity of the task, can be due to linguistic ambiguity or different beliefs originated from different socio-cultural aspects.
To this purpose, many researchers have proposed corpora that aim at capturing all the distinctive interpretations of a given text by preserving the
annotations disagreement instead of just aggregating them in a single gold label. The information brought by the annotators’ disagreement has been mainly exploited in three different ways: (1) to improve the quality of the dataset by removing those instances characterized by a disagreement between annotators (Beigman Klebanov and Beigman, 2009), (2) to weight the instances during the training phase (Dumitrache et al., 2019) or (3) to directly train a machine learning model from disagreement, without considering any aggregated label (Uma et al., 2021b; Fornaciari et al., 2021). An important contribution in this field has been given by the SemEval-2021 Task 12 - Learning with Disagreements (Le-Wi-Di) (Uma et al., 2021a), whose aim was to provide a unified testing framework from disagreement for interpreting language and classifying images.
In this paper, we address the Learning with Disagreements (Le-Wi-Di) task at SemEval-2023 Task 11 (Leonardellli et al., 2023), where the main goal is to model the disagreement between annotators on different types of textual messages. In particular, we proposed a straightforward strategy that given the probability of the hard-label prediction, it creates an optimal soft-label mapping.
The paper is organized as follows. Detail about the shared task and the related datasets are reported in Section 2. In Section 3 an overview of the state of the art is provided. In Section 4 the proposed approach is detailed focusing on the prediction model, linguistic characteristics, preprocessing and post-processing operations. In Section 5 the results achieved by the proposed models are reported. Finally, conclusions and future research directions are summarized in Section 7.

heading: 2 Task Description
text: The main goal of the SemEval-2023 Task 11 - Learning With Disagreements (Le-Wi-Di) (Leonardellli et al., 2023), is to develop methods
556
able to capture the agreement/disagreement among the annotators toward a specific sentence label. The organizers encourage the usage of shared features between the available datasets for the definition of a model that is able to generalize over potentially different domains. Given the subjectivity of the proposed tasks, and therefore the absence of a golden label, the evaluation is based on two different strategies: (i) hard evaluation based on the F1 measure estimated on the prediction capabilities on the hard label and (ii) soft evaluation that considers how well the model’s probabilities reflect the level of agreement among annotators using the Cross-Entropy as a performance metric.

heading: 2.1 Dataset
text: The organizers of Task 11 provided 4 benchmark datasets with different characteristics, in terms of types (social media posts and conversations), languages (English and Arabic), goals (misogyny, hate-speech, offensiveness detection) and annotation methods (experts, specific demographics groups and general crowd). The datasets available for the challenge (Akhtar et al., 2021; Almanea and Poesio, 2022; Curry et al., 2021; Leonardelli et al., 2021a) are summarized in Table 1.
All datasets relate to hate speech detection problems and are characterized by a different number of annotators ranging from 2 to 7. Most of the datasets are composed of tweets, except ConvAbuse which reports dialogues between a user and two conversational agents. Regarding the hard label, there is a significant imbalance over all the datasets. For what concerns the soft label, the ArMIS dataset
shows a balanced distribution while we can still observe a disproportion for the remaining datasets. A summary of the available instances of the datasets is reported in Table 2, where dataset-specific attributes are available such as information about the annotators and the task for which the text has been collected.

heading: 3 Related Work
text: Machine learning models require to be trained with a significant amount of data, which usually needs to be labeled by human annotators. In order to collect such data in an efficient and cost-effective way, many researchers have relayed on crowd-sourcing platforms. Since the collected labels typically show disagreement among the annotators, in many cases, the gold label is obtained by a majority voting mechanism, that also involve the intervention of a domain expert or more complex approaches based on annotators commonalities or sample weights (Akhtar et al., 2021). Moreover, disagreement can also derive from overlapping labels, subjectivity, or annotator error (Uma et al., 2022). Nowadays, datasets with multiple annotations have become increasingly common and disagreement information have been used as additional information for training predictive models (Basile et al., 2021; Leonardelli et al., 2021b). In (Wich et al., 2020) the authors model the annotators behavior in a graph structure in order to represent their frequency of disagreement with each other and take advantage of information collected through community detection algorithms to train classifiers. In (Fornaciari et al., 2021) the authors implement a multitask learning
model to predict agreement as an auxiliary task in addition to the standard classification task, which improves the performance even in less subjective tasks such as part-of-speech tagging. The approach proposed by (Davani et al., 2022) models each annotator’s labels individually: the authors propose a multi-annotator architecture that uses a multitask approach to separately model each annotator’s perspectives. The authors model uncertainty in predictions while maintaining high performances in terms of accuracy and efficiency. Finally, an important contribution to the field is represented by the shared task organized at SemEval-2021 (Uma et al., 2021a). The approaches proposed by the participants (Osei-Brefo et al., 2021) were based on a Sharpness-Aware Minimization (SAM), i.e., an optimization technique that is robust with respect to noisy labels, and on a neural network layer called softmax-Crowdlayer, which was specifically designed to learn from crowd-sourced annotations. Both approaches were able to improve the performance of the state-of-the-art Wide Residual Network and Multi-layer Perception models.

heading: 4 Proposed Approach
text: The proposed approach is grounded on the hypothesis that the disagreement between annotators could be grasped by the uncertainty that a model, based on several linguistic characteristics, could have on the prediction of a given gold (hard) label. This hypothesis reflects the uncertainty that a human annotator might have when annotating a text with disagreement. To validate this hypothesis, the proposed approach maps the probability values returned by a classifier trained to predict the hard label into agreement/disagreement values. The main two steps of the proposed approach are synthesized in Figure 1 and described as follows:
• Hard label (HL) training. A classifier is trained to predict if a given message has to be classified as positive or negative as hard label. The probability distributions over all the sam-
ples in the training dataset are used to compute the optimal classification threshold according to the Youden’s J statistics (Youden, 1950). The Youden’s J statistics, which is a linear combination of sensitivity and specificity, is maximized evaluating several cut off of the decision threshold of the classifier to obtain the optimal value. The proposed strategy, which basically optimizes the Area Under the Curve (AUC), allowed us to select the best classification threshold for the hard label and partially overcome the imbalance of the dataset labels.
• Soft label (SL) representative estimation. The model designed to predict the hard label is used to determine representative probability values that can be associated to each soft label. In particular, focusing on the positive label, the sub-samples of the training data sharing the same soft label (e.g. SL = 0.8) are used to obtain the corresponding probability distributions from the hard label model. Such probability distributions are then used to compute, for the positive class, the corresponding mean value subsequantly used as representatives for determining the corresponding SL labels.
Once the model for the hard label prediction has been trained and the soft label representative identification has been performed, we can move to the inference phase. In particular, when an unseen sample has to be classified, the hard label model is exploited. For the hard label prediction, the most probable class is selected according to the comparison of the probability distribution and the optimal cut-off obtained via the Youden’s statistics. For the soft label prediction, the probability of the positive class is compared to the representative mean values estimated in the previous step, and assigned to the closest one. This allows us to automatically map the probability of the hard label to the corresponding soft label.
In order to accomplish the above-mentioned
steps, we developed a system based on four main constituents subsequently described.

heading: 4.1 Hard Label Classifier
text: The aim of the proposed approach is to investigate a potential relationship between classifier predictions on hard labels and annotator agreement. To this purpose, a simple cost-effective neural network has been adopted. The model, trained to predict the hard label, is composed of the following layers:
• Input: layer with length that varies according to the combination of language characteristics described in subsection 4.2;
• Hidden: dense layer with 256 internal neurons, with LeakyReLU activation function and dropout equal to 0.2;
• Output: dense layer with 1 output neuron, with sigmoid activation function able to predict the hard label.

heading: 4.2 Language Characteristics
text: The input of the proposed system has been defined to consider multiple aspects of the language of each dataset.
The first descriptor of each text is represented by the sentence encoding of the text itself. In order to guarantee a representation that could model
multiple tasks, we adopted the Universal Sentence Encoder (USE) (Cer et al., 2018). The objective function, based on the binary cross-entropy loss, is minimized using the Adam (Kingma and Ba, 2014) optimization technique. Due to limited computational resources, we selected a training phase of 10 epochs with a batch size of 20 instances. The optimal classification threshold has been selected through Youden’s J statistic (Youden, 1950), which allows selecting the best tradeoff between specificity and sensitivity.
Additional metadata has been considered in order to enhance the neural network discrimination capabilities. The selected metadata are strictly related to the nature of the proposed tasks (i.e., hate speech, misogyny, abusive and offensive language detection). In particular, such metadata are based on the assumption that hateful messages are characterized by specific emotions, such as anger and disgust, and a peculiar writing style, i.e., long sentences with a consistent number of special characters and uppercase. Moreover, since the majority of the proposed datasets are a collection of tweets, metadata representing online users’ pragmatics have been considered (e.g. related to the use of hashtags). The selected metadata are summarized as follows:
• Emotion: emotion-related features have been derived through the NRC Emotion Lexicon (Mohammad and Turney, 2013). A value for each of the eight emotions (anger, fear, expectation, trust, surprise, sadness, joy, and disgust) has been computed and stored into an 8-dimensional vector.
• Hashtag Disagreement: for each textual message, a score representing a measure of disagreement related to the contained hashtags is computed. In particular, given a text s and the corresponding hashtag set Hs = {hs1, ...hsn}, the Hashtag Disagreement (HD) score is computed as follows:
HD(s) = 1
|Hs|
|Hs|∑
i=1
1
Ni
|T (hsi)|∑
j=1
A+j −A−j Aj Nij
where:
– T (hsi) dentoes the set of training messages containing the hashtag hsi – Aj represents the number of annotators that labelled the training sample j containing hsi – A+j is the number of annotators that labelled the training sample j containing hsi as positive – A−j denotes the number of annotators that labelled the training sample j containing hsi as negative – Nij represents the number of occurrences of hashtag hsi in the training sample j
– Ni is the total number of occurrences of hashtag hsi in the training dataset.
The HD score associated with each text is bounded in the interval [-1; 1], where negative values denote a high correlation between the hashtag and the negative label, while positive values represent a high correlation between the hashtag and the positive label.
• Language pragmatics: for each textual message, features representing the presence of a few special characters (i.e., @, #, ! and ") and emoji are measured. In particular, given a textual message s, the number of occurrences of each of the above-mentioned pragmatic elements is estimated and represented through
a 5-dimensional vector. The above-mentioned special characters have been selected as proxies of hateful content. In fact, from a preliminary analysis of the datasets it has been observed that users using a lot of exclamation marks strengthen their point of view towards hateful content. Moreover, hateful tweets usually include hashtags and mentions to directly and explicitly address specific groups or gain visibility by mentioning specific trends. Finally, users usually make use of quotes both to report sentences they’re willing to comment on or to highlight the usage of sarcasm.
• Stylometric features: for each textual message, two stylometric characteristics have been considered. In particular, the number of words contained in each message and the percentage of uppercase characters have been considered as stylometric features, originating a 2-dimensional vector. This choice is motivated by the fact that hateful messages are frequently characterized by a longer text and the presence of uppercase to emphasize the exasperation related to a given topic.

heading: 4.3 Soft Label Mapping
text: The mechanism to predict the level of agreement of a given message is based on the assumption that samples with similar values of agreement are characterized by a similar probability distribution related to the hard label. In other words, the overall assumption is that a model trained on the hard label will predict a similar probability value for samples with the same value of agreement. In particular, we designed a mapping strategy that is able to associate the probability prediction related to the hard label with a given representative value denoting the corresponding soft label. The proposed mapping strategy is based on the following steps:
1. the training samples are grouped in subsets according to their value of agreement, i.e., the corresponding soft label (e.g. SL = 0.8), and given as input to a neural network that is able to predict the probability distribution of the corresponding hard labels. In particular, the training dataset has been split in order to create subsets of samples with the same agreement value (i.e. with the same soft labels). Then for each sample, the probability distribution for the hard label has been computed, while keeping track of the assigned subset;
2. for each of the above-mentioned subsets, the mean of the classifier prediction values are computed considering the positive class label;
3. given a new sample, its prediction probability of being predicted as positive by the hard label model is mapped to the closest mean value among the ones available for each level of agreement estimated at the previous step.

heading: 4.4 Prediction Refinement
text: An additional post-processing operation has been performed in order to improve the predictions given by the soft mapping. In particular, the goal is to adjust the soft label prediction towards the number of annotators that labelled a given sample. To this purpose, the soft label prediction has been shifted to the closest plausible value in accordance with the number of annotators.
This step was particularly effective for the dataset with different numbers of annotators (i.e., ConvAbuse) while resulting in a basic rounding operation for the other datasets. The post-processing steps to refine the soft label prediction allowed us to achieve an improvement of the performance up to 4.28 in terms of cross-entropy.

heading: 5 Results
text: In order to provide the label predictions for the test set, several input configurations have been investigated related to the input layer of the hard label classifier. In particular, the following models have been considered during the training phase:
• M1: For each dataset, the classification model has been trained using only the sentence encoder (USE) as input, without considering any additional information. The resulting input used to train the M1 has a size of 512 descriptors given by the USE encoder.
• M2: For each dataset, the classification model has been trained using the embeddings given by the sentence encoder, coupled with emotion-related features and hashtag disagreement score. The resulting input used to train M2 consists of 512 descriptors given by the USE, 8 dimensions for capturing the emotions of the message and 1 dimension for denoting the hashtag disagreement.
• M3: For each dataset, the classification model has been trained using the embeddings
given by the sentence encoder, together with emotion-related features, hashtag disagreement, language pragmatics and stylometric features. The resulting input used to train M3 consists of a 528-dimension vector.
• M4: A single model has been trained using all the available datasets to take advantage of hateful content commonalities among them. The classification model M4 has been trained using only the sentence encoder (USE) as input, without considering any additional information.
The results achieved by the proposed approaches on development and test data are summarized in Tables 3 and 4 respectively 1. The achieved results highlight that the exploitation of the considered metadata, i.e., emotion, hashtag disagreement score, language pragmatics and stylometric features, allows a better representation of disagreement resulting in an improvement of the CrossEntropy values in all the considered datasets. However, since this representation emphasizes the uncertainty in the model predictions it also results in a decreasing performance in terms of F1 measure on the hard label. It is important to note that while M4 brings an overall improvement (with respect to the other models) when considering the Cross-Entropy measure, we can not observe any difference in terms of F1-Micro measure.

heading: 6 Considerations about the evaluation metrics

heading: 6.1 Performance evaluation
text: The state of the art about performance evaluation is characterized by two main types of evaluation metrics (Uma et al., 2021b) in order to evaluate how well the model performs when (i) all items are treated equally and (ii) the items are weighted depending on disagreement. The most frequent hard label measures in the first scenario are the percentage accuracy and the class-weighted F1. An alternative is based on the crowd-truth weighted f-measure (CT-F1). This approach is based on the intuition that items characterized by a large value of disagreement should be weighted less than items characterized by a large value of agreement. Regarding the soft label predictions, many approaches have been discussed (Uma et al., 2021b)
1Only the M1 and the M2 model predictions have been submitted for the challenge partecipation
to capture (i) the similarity in the class distributions between the predicted labels and the ones given by the annotators or (ii) the ability of the model to reproduce human uncertainty in the prediction. The first type of evaluation is based on the assumption that the label distribution obtained by the annotation process is representative of the ambiguity of each item. To measure the label distribution similarity two approaches have been proposed:
• cross-entropy (CE): to capture how confident the model is with respect to humans and the reasonableness of the distribution over alternative categories.
• Jensen-Shannon Divergence (JSD): a standard symmetric method to measure the similarity between two probability distributions based on the Jensen-Shannon Divergence.
The second type of evaluation assesses the model’s capability to capture the disagreement between annotators via normalized entropy. This approach is based on the assumption that the entropy of the annotation distribution represents how confusing the annotators find an item.

heading: 6.2 Comparison of Evaluation Metrics
text: Despite both F1 and CT-F1 are considered valid to rank systems based on their performances, F1 measure is more suitable with class imbalance, while the accuracy measure results in a rank strictly associated with the majority class. Regarding the soft labels’ evaluation, several authors highlighted a few limitations of the crossentropy measure (Uma et al., 2021b,a). The first weakness of cross-entropy is related to its unbound
nature. Moreover, since the Cross-Entropy measure is asymmetrical, i.e. it tends to have large values when close to the boundaries of the distribution, and it depends on the intrinsic entropy of the ground truth distribution, a few behaviours result to be unintended. Additionally, in a few cases, Cross-Entropy assumes values that do not reflect the quality of the best prediction. For instance, considering a Ground Truth (GT) distribution of [0.8, 0.2], a prediction of [0.95, 0.05] will result in a cross-entropy value of 0.32 while a prediction of [0.6, 0.4], despite more distant from the real value, results in a better cross-entropy value that is equal to 0.2959. For these reasons, other metrics such as the JansenShannon Divergency of the Wasserstein Distance appear to be more suitable for a soft-label evaluation being positive, symmetric and satisfying the triangle inequality. Moreover, the Wasserstein metric may calculate the distance between two distributions even if they are not in the same probability space, while the Jansen-Shannon Divergency, when the distributions are supported on non-overlapping domains, could fail (Kolouri et al., 2019).

heading: 7 Conclusions and Future Work
text: The proposed approach represents a preliminary attempt to deepen the relation between model predictions and samples’ disagreement. The achieved results highlight a correlation between metadata and disagreement predictions: considering the average performances, the inclusion of emotion and lexical-related metadata resulted in an improvement of the performance in terms of disagreement detection. As future research direction, the correlation between annotators’ agreement and both emo-
tion and lexical-based features will be deepened in order to achieve a representation more suitable for the model to address the task of disagreement prediction. Despite the poor improvement introduced by the universal model, the data provided by the organizer resulted to be referred to tasks too different to be assimilated into a single classification model. As a future study, it would be interesting to integrate additional datasets collected for the same tasks in order to realize task-related models. Finally, more complex architectures (e.g. transformer-based) will be implemented to improve prediction capabilities.

heading: Acknowledgment
text: The work of Paolo Rosso was done in the framework of the FairTransNLP-Stereotypes research project on Fairness and Transparency for equitable NLP applications in social media: Identifying stereotypes and prejudices and developing equitable systems (PID2021-124361OB-C31) funded by MCIN / AEI / 10.13039 / 501100011033 and by ERDF, EU A way of making Europe.
The work of Elisabetta Fersini has been partially funded by the European Union – NextGenerationEU under the National Research Centre For HPC, Big Data and Quantum Computing - Spoke 9 - Digital Society and Smart Cities (PNRR-MUR).

heading: Future Work
text: As future research direction, the correlation between annotators’ agreement and both emo-
tion and lexical-based features will be deepened in order to achieve a representation more suitable for the model to address the task of disagreement prediction. Despite the poor improvement introduced by the universal model, the data provided by the organizer resulted to be referred to tasks too different to be assimilated into a single classification model. As a future study, it would be interesting to integrate additional datasets collected for the same tasks in order to realize task-related models. Finally, more complex architectures (e. g. transformer-based) will be implemented to improve prediction capabilities