Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?

Authors: urja  khurana
eric  nalisnick
antske  fokkens
swabha  swayamdipta
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 15:32:06

Abstract: Subjective tasks in NLP have been mostly relegated to objective standards, where the gold label is decided by taking the majority vote. This obfuscates annotator disagreement and the inherent uncertainty of the label. We argue that subjectivity should factor into model decisions and play a direct role via calibration under a selective prediction setting. Specifically, instead of calibrating confidence purely from the model’s perspective, we calibrate models for subjective tasks based on crowd worker agreement. Our method, Crowd-Calibrator, models the distance between the distribution of crowd worker labels and the model’s own distribution over labels to inform whether the model should abstain from a decision. On two highly subjective tasks, hate speech detection and natural language inference, our experiments show Crowd-Calibrator either outperforms or achieves competitive performance with existing selective prediction baselines. Our findings highlight the value of bringing human decision-making into model predictions.

Main paper: 
 introduction: 
 Natural language is inherently subjective, leading to subjectivity in classification tasks (Aroyo & Welty, 2015; Plank, 2022; Cabitza et al., 2023; Jamison & Gurevych, 2015; Pavlick & Kwiatkowski, 2019 ). Yet, in natural language processing, most such tasks are treated as if there exists a single ground truth. The conventional setup consists of a small number of annotators labeling each sample and taking the majority vote determines the final label. However, this setup dismisses subjectivity in implications (for e.g., in NLI; Pavlick & Kwiatkowski, 2019) , and removes minority voices (for e.g., in safety-critical applications like hate speech detection; Prabhakaran et al., 2021; Sap et al., 2022) . While there are many cases for which humans are more likely to agree with each other (Jiang et al., 2021a; Salminen et al., 2019) , there are also cases where there is a lack of consensus (Khurana et al., 2022) . In such cases, models, rather than predicting a single label, must make decisions that reflect potential disagreements. Selective prediction frameworks (Geifman & El-Yaniv, 2017; Chow, 1957) allow for this kind of model abstention. In this paper, we argue that selective prediction is an ideal fit for subjective tasks.Ideally, we would want a model's confidence, i.e. softmax probabilities, for its prediction to reflect human disagreement. However, neural models with a large number of parameters tend to be overconfident (Guo et al., 2017; Chen et al., 2023) . A straightforward way to make a model aware of human variation in the label is to use soft labels (Jamison & Gurevych, 2015; Uma et al., 2020; 2021b) . Instead of a model predicting one label (hard label), we want the model to output the human label distribution. However, most available NLP datasets only provide 3-5 annotations per sample, which limits the distributions that the model can learn. We need many more annotations per sample to learn the crowd's beliefs. crowd disagreement. Most calibrators either rely on the model's confidence or have a separate calibrator that outputs if a model is correct. For subjective tasks, there are limited ways of calibrating a model as mostly there is no one correct label. Our CROWD-CALIBRATOR determines if the model's output distribution is close to the human judgment distribution. If that is the case, then the model makes a prediction. If the model is far from the human distribution, then the model abstains.We apply CROWD-CALIBRATOR to two subjective NLP tasks: hate speech detection and natural language inference (NLI), a task for which an abundance of human subjectivity data is available (Nie et al., 2020b) . We show the potential of our setup by comparing it to selective prediction baselines such as MaxProb and Kamath et al. (2020) . Our method is competitive with these baselines for hate speech detection and outperforms them for NLI. We also show that our setup beats baselines on cross-dataset evaluation for both tasks. Our method is beneficial for in-domain and out-of-domain datasets when we have access to a small set of samples with many annotations per instance (∼ 100) and for out-of-domain datasets when we have access to individual annotator samples (> 2000) . Our work restates the potential of using selective prediction for subjective tasks as a research direction. 
soft labels for hate speech detection: 
 Our method experiments with using soft labels as a candidate to better calibrate models for subjective tasks. Soft labels-where the model directly learns the human label distributionhave been used extensively in prior work on subjective task prediction (Jamison & Gurevych, 2015; Uma et al., 2020; 2021b) . Based on this, we ask whether soft labels could also help improve model calibration, compared to hard labels. Moreover, these calibrated confidence estimates could be used to make the model abstain from any decision (Chow, 1957; Geifman & El-Yaniv, 2017) when there is greater annotator disagreement.A majority vote over the annotation count of a sample x: y h = arg max l∈Y x ( f req(l)), results in hard labels (Y h ) as the ground truth, used in training with the Cross-Entropy loss (CE hard ). However, for subjective tasks, majority voting might result in loss of vital information about the annotator disagreement and unjustified high model confidences, necessitating soft labels. Soft labels (Y s ) are the probability distribution over the classes of the annotator judgments for a sample. This can be done by e.g. normalizing the votes for each class or taking the softmax (Uma et al., 2020) when the number of annotators per sample is low. We illustrate the distinction between hard and soft labels in Figure 1 . Training. Given a model f that takes as input text x and predicts the soft label y s , the objective involves minimizing the distance between two probability distributions: the model distribution output ŷs and the soft label y s . This has been done by extending the Cross-Entropy Loss to measuring how distant ŷs and y s are from each other (Peterson et al., 2019; Uma et al., 2020) , using KL Divergence, Jensen-Shannon Divergence (JSD; the symmetric version of KL Divergence) or Mean Squared Error (Uma et al., 2021b) .Evaluating. Unlike hard evaluation metrics such as accuracy or F1-score, evaluating soft label prediction requires distance-specific metrics (Basile et al., 2021) . Uma et al. (2021b) suggest using the soft losses mentioned as soft metrics. Baan et al. (2022) suggest using the Total Variation Distance (TVD).We first investigate the utility of soft labels for the task of hate speech detection, where we have few annotations per example. If samples with more disagreement would then get lower confidences,1 we can then use these confidences as a way to prevent a model from predicting when there is no clear consensus.Data. We combine two widely used datasets: HateXplain (HX; Mathew et al., 2021) and the Measuring Hate Speech Corpus (MHSC; Sachdeva et al., 2022) for our experiments. Both provide the raw annotations for each sample. MHSC has one to five annotations per instance and a binary hate speech category. The HX dataset has three annotations per instance and consists of three classes: hate speech, offensive, neutral. In Appendix H we apply CrowdTruth (Dumitrache et al., 2018) as a dataset analysis. To match the datasets, we merge the hate speech and offensive classes in HX, essentially reducing the task to identifying offensive labels. The hard labels are derived by majority voting and the soft labels by using softmax over the annotator votes per class (in line with Figure 1 ), due to the low annotations per instance. We split the dataset into train (41832 samples), validation (5230 samples), and test (5230 samples).Model. We use RoBERTa-large (Liu et al., 2020) 2 initialized with five different random seeds. We follow the original hyperparameters as described in Appendix C. Model selection is based on JSD when training with JSD loss and macro F1 when training with CE hard .Metrics. We compare the performance of models trained with CE hard and JSD using macro F1, soft metrics (JSD and TVD), and confidence calibration metrics. Soft labels could provide better confidence estimates of the model's empirical performance. To measure this, we use Expected Calibration Error (ECE; Naeini et al., 2015) and Brier Score (BS; Brier, 1950) . For ECE, the confidence scores of the model are binned and for each bin, the accuracy is calculated. The average difference between the calculated and expected accuracy is taken over all bins. This metric is sensitive to the bin size. BS measures the mean squared error between the model confidence and its predictions. In NLP, only the calibration of the top confidence score is usually measured. This is a much weaker notion as it ignores the other dimensions (Vaicenavicius et al., 2019) .Figure 2 shows the results3 . As expected, using soft labels with few annotations per instance leads to (slightly) better performance on soft and calibration metrics (JSD, TVD, ECE, and BS). There is not much difference between the Macro F1-scores for the hard and soft models, in line with previous research. When looking at the confidence distributions for both the hard and soft loss models, there is not a lot of difference in general. In Figure 3 , we visualize the confidence distributions on samples for two types of agreement: perfect agreement, a sample where all annotators agree with each other on the assigned label, and disagreement: a sample where at least one annotator assigns a different label than the rest. In general, when training with CE hard (Figure 3a ), we see a strongly unimodal confidence distribution for samples with perfect agreement. The skew on disagreement samples is less but still similar to perfect agreement. For the soft loss (Figure 3b ), there is more mass distributed around lower confidences, especially for disagreement samples. Ideally, we would want fewer high confidences. When we disentangle correct and incorrect predictions for both perfect agreement (PA) and disagreement (DIS) samples, we see that with the hard loss (Figure 3c ), regardless of correctness, for both agreement types most of the density is around the higher confidences. For the soft loss (Figure 3d ), the confidences are more dispersed, especially for perfect agreement. Correct samples with perfect agreement are denser around higher confidences and incorrect perfect agreement samples are denser around lower confidences. This difference in distribution based on correctness for perfect agreement is not as notable for disagreement samples, which is more spread out but still has more mass in higher regions. We want disagreement samples to have generally lower confidence scores but our analysis shows that this is generally not the case. Our notion of disagreement in these experiments is based on 3 -5 annotators, which can be limited in learning the crowd distribution. This motivates us to find a different method to make models more aware of human subjectivity. 
selective prediction: 
 The selective prediction framework (Geifman & El-Yaniv, 2017; Chow, 1957) aims to reduce errors made by a model by giving it the option to abstain. When dealing with subjective tasks, we argue that this is an attractive setup to align model output accordingly; abstain when dealing with highly subjective input and let predictions with more agreement, hence clear(er) correct answers, through. This setup consists of a base model f (x) that outputs a softmax confidence distribution ŷ that we want to calibrate accordingly and a decision function h(x) on top, which determines if f (x) predicts or abstains. Existing decision functions can either be confidence-based or a separate calibrator model: MaxProb This method makes use of confidence thresholding. The highest confidence max( ŷ) is compared to a threshold t. If the confidence is higher than t, the model predicts, but if the confidence is lower than t, the model abstains. We can apply MaxProb to f (x) trained with either a hard loss with majority vote labels ( ŷmaj ) or a soft loss ( ŷs ).Originally, this setup uses a separate calibrator that predicts if the base model is correct or not for the task of Question Answering in the case of domain shift where the base model is only trained on in-domain data. The calibrator, a random forest classifier, is trained on both in and out-of-domain data. We draw an analogy between their in-& out-of-domain setup with perfect and disagreement in our task. In our case, the base model is only trained on samples with perfect agreement, m PA , and the calibrator h is trained on both disagreement and perfect agreement samples and outputs h(x) = z; if m PA is correct or not. If z = 1, only then does the prediction go through, otherwise, the model abstains. We train the calibrators with a random forest from XGBoost (Chen & Guestrin, 2016) . We use the following input features: hidden state of [CLS] of base model's last layer (Corbière et al., 2019; Zhang et al., 2021; Zhang & Choi, 2023) , base model's softmax probabilities, and n-grams from the calibration data. 
crowd-calibrator: 
 Most of the calibration research has focused on the alignment between a model's confidence and its predictive performance. However, for subjective tasks, we want this to extend to reflecting human uncertainty as well, i.e. we want a soft calibration approach. Instead of focusing on the correctness of the model, we ask if the softmax confidence distribution of a model is close to the crowd distribution for a given sample.We propose CROWD-CALIBRATOR (Figure 4 ), a soft calibration approach where we calibrate a model according to how close its confidence is to the crowd opinion. If the model confidence is far off from the crowd distribution, we want the model to abstain, i.e. not make a prediction. If the model confidence is close to the crowd distribution, we let the model predict. When the base model is overconfident on a sample where humans tend to disagree and thus there is no clear label, we can prevent the model from making a prediction. Our setup consists of three components: the base model that we are calibrating, the crowd estimator that estimates the crowd judgment, and the soft calibration. Our base model m maj , trained to predict the conventional majority vote label, is the model we want to calibrate. This is the same model trained with the hard loss in Section 2, outputting the softmax confidence distribution ŷmaj (also marked as ŷbase in Figure 4 inference).With our CROWD ESTIMATOR we want to predict the human annotation distribution considering we want to calibrate our base model m maj according to human judgments.However, we do not have this information available at test time. Our goal with the CROWD ESTIMATOR4 is to estimate the crowd (dis)agreement of a given sample x.The hate speech datasets that we have used until now are limited in terms of modeling the crowd. While there might be many annotators, the number of coders per instance is low (1 -5) and many annotators have only seen a handful of samples. As there are no hate speech datasets with more than 5 annotations, we need an alternative to estimating the crowd. We do have access to hate speech datasets where a decent amount of annotators have seen a handful of samples: the Gab Hate Corpus (GHC; Kennedy et al., 2022) and the Dynamically Generated Hate Speech Dataset (DGHS; Vidgen et al., 2021) . We select individual annotators that have annotated more than 2000 samples and train different individual MLPs m a i (single layer, hidden size 512). We use the hidden state of [CLS] of base model's last layer as the input feature to predict if the annotator would perceive something as hate speech. To combine the individual annotator predictions into one crowd distribution ŷcrowd , we employ three strategies: taking the individual predicted labels and creating a distribution over them with softmax (Label Dist), averaging over the confidence scores (Avg. Conf.) of the MLPs, and Weighted Scoring (Equation 1) where we take the weighted (fraction of voters r c ) distance score between the average confidence distribution of voters (M c ) for a particular class c and sum over all classes C.EQUATIONFor NLI, we have enough instance-level annotations with ChaosNLI (Nie et al., 2020b) . This dataset consists of 4500 re-annotated samples from SNLI (Bowman et al., 2015) , MNLI (Williams et al., 2018) , and αNLI (Bhagavatula et al., 2019) 5 . Each sample gets 100 annotations, which makes it ideal to train our CROWD ESTIMATOR on. Our base model, RoBERTa-large, is trained on SNLI and MNLI with the majority vote. Instead of training separate annotator classifiers, we train a small MLP regressor (2-layered MLP with hidden sizes of 100 and the hidden state of [CLS] of the base model's last layer as input feature) that outputs soft labels directly. The further process is the same as shown in Figure 4 .Now that we have both ŷmaj and ŷcrowd , we want to measure their distance. Thus, we experiment with different distance metrics (KL divergence (KL), JSD, and TVD; see Appendix D for formulae). If the model's output distribution is far off from the human probability distribution, the model will abstain, otherwise the model's prediction, argmax( ŷmaj ) will go through. We also experiment with adding the entropy (+E) of ŷmaj to the entire score to remove instances where both ŷcrowd and ŷmaj are close to each other but have low confidence.6 We evaluate our soft calibrator through the lens of how well it abstains (does the calibrator let the correct samples through)7 and how conservative it is (does it let through sufficient samples to achieve good performance).Cov@Acc= Since the model abstains, we focus on coverage: the fraction of samples the base model predicts. The coverage changes as we adjust our threshold for the distance score or model confidence. By increasing the confidence or reducing the distance score, the coverage decreases and we expect accuracy to increase. Hence we compare coverage at a fixed accuracy value to evaluate how conservative a technique is at that point.We yield corresponding accuracies for each coverage when thresholding. Following Geifman & El-Yaniv (2017), we calculate the area under the accuracy-coverage curve (AUC) to quantify how much the calibrator abstains and how correctly it abstains over the entire trajectory. Similarly, to measure if a calibration method improves calibration as coverage decreases, we compute the area under the coverage-BS curve (AUBS).AUROC Following Shrivastava et al. (2023) , we want to know if the thresholding or confidence has good differentiating power between correct and incorrect predictions, which is why we plot the fraction of wrong samples let through against the fraction of correct samples let through and calculate the area under this curve. 
results: 
 We discuss the results of our soft calibrator obtained for both hate speech and NLI8 . We also explore the soft calibrator's capabilities on other unseen datasets for both tasks. For all results, we highlight the best performance in blue and the worst performance in red . All results are averaged over five seeds.9 We present the results obtained in Table 1 , where we use a combined test set of perfect agreement and disagreement with 3200 samples. We also experimented with GPT-4 but did not yield competitive results, which we present in Appendix A. To verify our hate speech results, we also experiment with Online Misogyny in Appendix E, obtaining similar results. The upper part shows the results using MaxProb with RoBERTa-large models trained on different types of data ( ŷmaj , ŷPA ), soft loss ( ŷso f t ), or using better-calibrated confidences with Temperature Scaling (TS) (Guo et al., 2017) . We also show MaxProb when training with only perfect agreement samples ( ŷPA ) for completeness with Kamath et al. ( 2020)'s base model. The lower part shows the best-performing results using our proposed CROWD-CALIBRATOR with RoBERTa-large as a base model and combining individual annotator models for the CROWD-ESTIMATOR.We see that ŷmaj performs the best in general. The Kamath et al. (2020) baseline performs closely, with the same AUC and slightly better AUROC (∆0.0023). However, Kamath et al. (2020) has a more complex setup in front of MaxProb. MaxProb on soft labels ŷso f t performs worse than ŷmaj in terms of AUC and AUROC, highlighting the findings discussed in Section 2.3. For CROWD-CALIBRATOR, averaging over the annotator confidences and using JSD as a distance metric with entropy yields the best results. It also gets the highest Cov@Acc=90. The best AUC is 0.9290 (-∆0.12%) and AUROC is 0.7790 (-∆0.17%).We present results obtained on unseen datasets in Table 2 . We apply the best-performing aggregation strategies to the Davidson (5000 random samples) (Davidson et al., 2017) , Founta (2500 negative and 2500 positive random sampled) (Founta et al., 2018) 2020) on other datasets. In all cases, the best-performing method is a variant of our proposed setup. For Davidson and HateCheck, we show the results when only using DGHS annotators, and for Founta when only using GHC annotators (indicated with a **) as these give the best results. This reveals the sensitivity of our method to the individual annotators on which it is training. For the original test set, the Gab annotators were more useful instead.We show the results obtained on the test of ChaosNLI (312 samples) in Table 3 . We only compare with MaxProb since Kamath et al. (2020) has a complex setup but, compared to MaxProb, yields comparable or worse results for hate speech. We see that with TVD and entropy, we can beat the baseline with an increase of 4.65% in AUC and 8.37% in AUROC.cov@acc=0.8 ↑ cov@acc=0.9 ↑ AUC ↑ AUROC ↑ AUBS ↓ We also apply our method to other datasets where it beats the baseline: the test set of the ANLI (Nie et al., 2020a ) (3200 samples) dataset and the test set of WANLI (5000 samples) (Liu et al., 2022) . We show these results in Table 4 . For ANLI, adding the entropy decreases performance but MaxProb still performs the worst out of all. For WANLI too, our soft calibration method with JSD and entropy gets the highest performance and beats MaxProb. The specific strength of the calibrator seems to be that it maintains higher coverage when targeting higher accuracy.  
related work: 
 Learning from disagreement. Recent research has advocated incorporating the notion of subjectivity into our models and NLP pipeline (Aroyo & Welty, 2015; Cabitza et al., 2023; Plank, 2022; Plank et al., 2014; Nie et al., 2020b; Chklovski & Mihalcea, 2003) . A direct way is to move away from hard labels to soft labels (Uma et al., 2021a; Peterson et al., 2019; Wu et al., 2023; Jamison & Gurevych, 2015; Fornaciari et al., 2021) . Such labels can also be used to model an annotator's uncertainty for a label's distribution (Liu et al., 2019) . Other approaches focusing on annotation disagreement range from a probabilistic setup (Raykar et al., 2010) to modeling the annotators themselves (Rodrigues & Pereira, 2018; Davani et al., 2022; Guan et al., 2018) . Zhou et al. (2022) explore distribution estimation methods. Utilizing disagreement for hate speech detection systems has had limited attention. Leonardelli et al. (2021) investigate the use of different data splits based on (dis)agreement for offensive language and Al Kuwatly et al. ( 2020) group annotators based on their demographic to investigate their biases for hate speech classification. Davani et al. (2022) show the benefit of using individual annotator information but require data where individual annotators have labeled many samples. This is something that many datasets including the ones we used in the paper-the HateXplain, Measuring Hate Speech Corpus, and NLI datasets-do not have. Hence, we could not empirically compare to their approach.Calibration. Investigating if a model is well-calibrated has been gaining traction (Desai & Durrett, 2020; Jiang et al., 2021b; Corbière et al., 2019; Baan et al., 2022; Ulmer et al., 2022; Nalisnick et al., 2018; Hendrycks & Gimpel, 2016) . Guo et al. (2017) show how modern neural networks are miscalibrated. Recalibration can be done post-hoc (Guo et al., 2017; Gal & Ghahramani, 2016; Jiang et al., 2021b) or through a hybrid human-model approach (Kerrigan et al., 2021) , amongst others. Efforts are being made to understand the calibration of LLMs, through their verbal (Tian et al., 2023; Krause et al., 2023) and probabilistic confidence (Shrivastava et al., 2023) . Shrivastava et al. (2023) show how using a mixture of linguistic and probabilistic confidences leads to better calibration. Vidgen et al. (2020) use a Bayesian approach to recalibrate models for abusive language and show that uncalibrated models are not in line with annotators.Selective prediction. Selective prediction/learning to abstain is a relaxation of learning to defer (Madras et al., 2018) : when a model determines when to defer to an expert by modeling that expert's knowledge. Learning to abstain/reject, on the other hand, does not model the expert, implicitly assuming a fallback mechanism with uniformly better accuracy than the rejection threshold. Learning to defer has been extended to multiple experts (Verma et al., 2023) . In NLP (Xin et al., 2021) , there has been a lot of focus on selective prediction for Question Answering (Kamath et al., 2020; Rodriguez et al., 2019; Zhang et al., 2021; Zhang & Choi, 2023; Varshney & Baral, 2023; Cole et al., 2023; Yoshikawa & Okazaki, 2023) . 
discussion and conclusion: 
 We propose CROWD-CALIBRATOR, a soft calibration method for subjective tasks that refrains from predicting if its confidence is far from the crowd label distribution. We show the utility of our method by applying it to two complementary data scenarios: the realworld scenario with only access to a large number of individual annotations and the ideal scenario with a large number of sample annotations. Our experiments show that CROWD-CALIBRATOR outperforms the MaxProb baseline for the task of NLI, both on the respective test set (albeit small) and two other datasets. For hate speech, our method is competitive with the baselines on the respective test set and outperforms baselines on unseen datasets.Our proposed setup clearly works with fewer samples but requires a high amount of sample annotations (∼ 100) to estimate the crowd distribution. This is less the case when we estimate the crowd distribution from individual annotators. Yet the number of annotators varied across tasks, with fewer annotators available for hate speech (24) than for NLI (100). Therefore, it is clearly beneficial to have a high amount of instance-level annotations, as also seen in Gruber et al. (2024) . In future work, investigating the number of annotators and its effect on our method poses an obvious and interesting direction. Thus applying our method to a novel task means that we need at least a small amount of multiply annotated data reflecting different annotator judgments. As such data is not easily available for many subjective NLP tasks, this is a limitation that needs to be considered.Takeaways. While our method shows sensitivity to the number of annotations, in both data scenarios we see notable results for unseen data. Our method offers flexibility in the choice of the distance metric and aggregation strategy. Which combination works best is largely dataset-dependent. The various settings we experiment with are not exhaustive options and can easily be replaced by other metrics or strategies. Generally, JSD and TVD gave the best results in combination with aggregating through label distribution or averaged confidences. In all cases, it is beneficial to add entropy to the score of the metric as it prevents the model from making decisions when both the model and crowds are uncertain.In line with other work advocating for more subjectivity-informed model behavior, we hope our findings encourage more research in the direction of subjectivity-based selective prediction and better design of datasets and model setup for subjective tasks. If we really want to understand the crowd and create more human-aligned applications, we need many more instance-level annotations from a diverse set of annotators with a high individual annotation count.KL(P||Q) = ∑ x∈X P(x)log( P(x) Q(x) ) H(Q) = -∑ x∈X Q(x) log(Q(x)) KL(P||Q) + H(Q) = [ ∑ x∈X P(x)log( P(x) Q(x) )] + [-∑ x∈X Q(x) log(Q(x))] = [ ∑ x∈X P(x)(log(P(x)) -log(Q(x)))] -∑ x∈X Q(x) log(Q(x)) = ∑ x∈X P(x) log(P(x)) -P(x) log(Q(x)) -∑ x∈X Q(x) log(Q(x)) = ∑ x∈X P(x) log(P(x)) -P(x) log(Q(x)) -Q(x) log(Q(x)) 
 
 Appendix: For each experiment, we give the hyperparameters and the dataset sizes used. All of our results are aggregated over 5 random seeds and our experiments are done on a Titan 6000.To train our models we follow the original hyperparameters. We train for 10 epochs, with a learning rate of 1e -5 , weight decay of 0.1, and 6% warmup steps. We use a training batch size of 8. We train these models on 41832 samples and validate on 5230. The results shown in Section 2.3 are on the test set with 5230 samples.We use the same original hyperparameters for the RoBERTa-large base model that is now only trained on 32316 perfect agreement samples and validated on 4039 perfect agreement samples from MHSC + HX. Our Random Forest calibrator, trained with XGBoost has a learning rate of 0.07, max depth of 5, and 500 parallel trees. The calibrator is trained on a mixture of perfect agreement samples (2020) and the rest disagreement samples to bring the total to around 7000 samples to balance according to correctness and agreement.Individual Hate Speech Annotators Each annotator dataset is split into 80% training, 10% validation, and 10% test. For each annotator, we train an MLP, that is a single layer MLP with a hidden size of 512, for the rest using the default parameters in scikit-learn.ChaosNLI ChaosNLI comes with 4500 samples from which we remove the αNLI samples since those have different classes. We train on 2490 samples, validate on 311 samples, and test on 312 samples. We use a two-layer MLP regressor with hidden sizes of 100, for the rest using the default parameters in scikit-learn.The selective prediction results for hate speech on MHSC + HX are done on 3200 samples with mixed perfect and disagreement.To calibrate our base model m in a soft fashion, we need distance metrics to determine the proximity between the base model's predicted distribution ŷbase (also referred to as ŷmaj when the base model m maj is trained to predict the majority vote) and the crowd distribution ŷcrowd . We experiment with the following widely-used distance metrics:KL DIVERGENCE (KL, Equation 2) measures the difference of one reference distribution from the other, however, it is not symmetric. If the other distribution is the reference distribution, we will get a different output.KL( ŷbase , ŷcrowd ) = ŷcrowd • ( ŷcrowdlog ŷbase )(2)JENSEN-SHANNON DIVERGENCE (JSD, Equation 3) measures the distance between the two distributions and is a symmetrical version of the KL divergence.EQUATIONTOTAL VARIATION DISTANCE (TVD, Equation 4) is a way to measure the distance between two distributions by measuring the absolute difference in probabilities.TVD( ŷbase , ŷcrowd ) = 1 2 || ŷbase -ŷcrowd || 1 (4)To verify the results achieved on the hate speech dataset, we apply our CROWD-CALIBRATOR setup to a similar subjective task: misogyny detection. This dataset (Guest et al., 2021) has 6383 samples with 2-3 annotations for the majority of its samples (6259), with the rest having 4 -16 annotations per sample. We train on 5096 samples and validate on 646 samples. This dataset is imbalanced, where most of the samples are non-misogynistic in comparison to misogynistic.To train the CROWD ESTIMATOR, we use the EDOS dataset (Kirk et al., 2023) , for sexism detection. We train individual MLPs just like for the hate speech dataset for annotators that have more than 3000 annotations. This gives us 14 individual annotators, which is less than what we had for the hate speech dataset.Our results on the respective test set can be found in Table 7 . Due to the imbalanced nature of the dataset, we balance out the test set to get a better view, where there are 72 misogynistic samples and 100 non-misogynistic.cov@acc=0.85 ↑ cov@acc=0.9 ↑ cov@acc=0.95 ↑ AUC ↑ AUROC ↑ AUBS ↓ We observe results similar to hate speech detection, with competitive performance.We experiment with different loss functions that would minimize the difference between the annotator probability distribution y s and the model's output distribution ŷs : Cross Entropy Loss (CE so f t , Equation 5), Jensen Shannon Divergence (JSD, Equation 3), and Mean Squared Error (MSE, Equation 7). We refer to these losses as soft losses. For comparison to the conventional setup, we also use Cross Entropy Loss in the conventional setup and refer to this loss as a hard loss (CE hard ), where the objective is to predict the hard label y h . EQUATIONTo measure the performance and calibration of the model, we employ different metrics.Hard evaluation metrics. We use these metrics to measure if the model gets its predictions right in the conventional way: macro F1 and accuracy.Soft evaluation metrics. To measure if the probabilities that the model outputs are similar to the annotator label distribution, we use CE so f t , KL, and JSD.Calibration metrics. To measure if the confidence the model outputs reflects its empirical performance, we measure the Expected Calibration Error (ECE).The hard and soft evaluation results can be found in Figure 5 . Here, we see that for all metrics, the hard loss is outperformed by other losses. This is even the case for hard metrics.The superior performance of the soft losses highlights how beneficial it is for models to learn the human label variation instead of the majority vote.To evaluate the losses in terms of calibration, we show the Expected Calibration Error in Table 8 . Generally, the hard loss has the worst (highest) ECE out of all and tends to be more overconfident. The soft losses have confidences closer to their empirical performance and tend to be rather underconfident. The JSD loss comes the closest to reflecting the actual performance of the model, having the lowest ECE. We therefore continue our experiments with only the JSD loss as a soft loss. To showcase the standard deviation and variation of our results, we look at the improvement or deterioration for the metrics we apply when using CROWD-CALIBRATOR compared to MaxProb on the base model trained with majority labels. We take the score achieved with CROWD-CALIBRATOR and subtract the score of MaxProb ŷmaj to get the difference (∆) in performance, for each seed individually. After that, we take the average and calculate the standard deviation.We show the mean variation in Tables 9 and 11 for hate speech and Tables 10 and 12 for NLI. The subscript is the standard deviation (±). A positive mean variation means that in general, our CROWD-CALIBRATOR performs better than the baseline MaxProb. A negative mean variation means generally worse performance.For completeness, we also show the difference in performance between the two baselines of MaxProb and Kamath et al. (2020) . -0.0029 0.0209 0.0041 0.0230 -0.0199 0.0288 -0.0019 0.0033 -0.0027 0.0087 Weighted Scoring -JSD+E -0.0091 0.0066 0.0025 0.0091 -0.0332 0.0180 -0.0027 0.0013 -0.0044 0.0029 GHC -Avg. Conf. -JSD+E -0.0089 0.0085 0.0040 0.0085 -0.0118 0.0324 -0.0014 0.0009 -0.0017 0.0058 GHC -Label Dist. -JSD+E -0.0151 0.0051 -0.0080 0.0116 -0.0062 0.0130 -0.0011 0.0009 -0.0054 0.0038 Table 9 : Difference (∆) in increase (+) or decrease (-) of metric score when calibrating with CROWD-CALIBRATOR compared to MaxProb ŷmaj for the in-domain hate speech test set. We present the mean difference, with the standard deviation ± as subscript.∆ cov@acc=0.8 ↑ ∆ cov@acc=0.9 ↑ ∆ AUC ↑ ∆ AUROC ↑ With model confidence for both hard and soft labels as output we mean the softmax scores outputted by the model. In this paper, we do not consider model confidence in the output distribution it predicts as a soft label.Using the HuggingFace Transformers library(Wolf et al., 2019).In Appendix F we show results when training with other soft losses.Our Crowd Estimator MLPs (both tasks) are trained with scikit-learn(Pedregosa et al., 2011).We filter out samples from αNLI as it has different classes. See Appendix C for details.Adding the entropy to the KL divergence does not result in canceling out the entropy already present in the equation and reducing down to the cross entropy. We write out the proof in Appendix B.Note that using the notion of correctness does not mean that we assume a single ground truth, we view it as observing a single sample of the underlying crowd distribution.Code released here: https://github.com/urjakh/crowd-calibratorFor the variation and standard deviation in change of performance with CROWD-CALIBRATOR compared to the baseline, see Appendix G. 


Decision: This is a solid paper, on the topic of modelling subjective tasks and means of calibrating models to better reflect the aggregate human (crowd) distribution. The problem is novel, the work is well motivated, and the empirical evaluation convincing. Reviewers question how realistic it is to have datasets with rich crowd annotations, and thus the practical applicability of this work may be limited. However, this is not a reason to dismiss the paper: this line of work has the potential to influence future dataset annotation activities & release. The author responses helped to clarify a number of questions from reviewers (e.g., Va5r, who had several concerns, which were well addressed and the reviewer raised their score.)

Reviews: review: paper_summary: paper_summary: The author proposes a novel method to calibrate models for subjective NLP classification tasks when crowdsourced annotations are available (rather than each instance only having a single ground truth). Here, "calibrate" refers to abstaining from making a prediction when the model is uncertain.

Specifically, a base model is trained on aggregated labels using the usual cross entropy loss (RoBERTa is used in this paper), then a classifier is trained for each worker. The worker classifier is formed by passing the embedding of the CLS token from the base model through a single hidden layer MLP. Subsequently, the output distributions from all worker models are aggregated into a crowd distribution. The distance between this crowd distribution and the base model's output distribution is calculated, and if the distance exceeds a threshold, the model chooses not to output a prediction. The author proposes several methods for how to aggregate worker distributions into a crowd distribution and how to calculate the distance. If the dataset only contains crowdsourced annotations but without worker information, then a crowd model is trained directly to fit each instance's label distribution, and the same method is used to calculate the distance between the crowd distribution and the base model's output distribution.

The author tested the proposed method and other baselines on the tasks of hate speech detection and natural language inference (NLI). The results showed competitive or superior performance.
strength_weakness: strength_weakness: The author only conducted experiments on two tasks, where the NLI dataset has far more annotations per instance than typical ones, and the proposed model did not show superior performance to the baseline on the hate speech datasets. Additionally, in Table 2, the author chose different groups of workers to train worker models for each dataset, which also shows that the proposed model is quite sensitive to the choice of workers. In this case, the author should conduct more comprehensive testing of the proposed model on more datasets. For instance, a dataset for detecting online misogyny [1], and a dataset for multilabel emotion classification [2]. Moreover, comparisons should be made with some highly relevant works, such as [3], which also proposed training a model for each worker and discussed the calculation of uncertainty.

* [1] An Expert Annotated Dataset for the Detection of Online Misogyny (EACL 2021)
* [2] GoEmotions: A Dataset of Fine-Grained Emotions (ACL 2020)
* [3] Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (ACL 2022)

Although the paper primarily focuses on subjective NLP tasks, the chosen evaluation metrics are still based on the assumption that each instance has only one ground truth, such as accuracy in Cov@Acc=0.9. In fact, while reading the paper, I initially thought the evaluation would be about the ground truth being a distribution, the model also outputs a distribution, and then evaluating the performance by calculating the distance between the two distributions.

Another confusing point is "confidence." For a typical softmax output, like [0.4, 0.6], we would consider it represents p(y=0)=0.4, p(y=1)=0.6, or say the prediction is y=1 with confidence=0.6. However, in subjective tasks, if the ground truth is considered a distribution, then [0.4,0.6] just represents the predicted distribution by the model, which does not contain any information about confidence. It means the model might be very certain that the distribution is [0.4,0.6] or very uncertain. The unknown confidence can be calculated through other means, such as in the proposed method, if the base model's output distribution is very close to the crowd model's output, it is considered high confidence. However, in the paper, the term confidence seems to be interchangeably used with the model's output, as mentioned in the paragraph above Figure 4:
> "If the model confidence is far off from the crowd distribution, we want the model to abstain, i.e. not make a prediction."

But in Sec 4.1.2 it is mentioned:
> "If the model’s output distribution is far off from the human probability distribution, the model will abstain."

Clarifying these concepts more clearly could potentially reduce reader confusion.
questions: questions: It’s not clear to me how the Weighted Scoring is calculated. Could the authors provide a formula for it?
paper_summary: paper_summary: This paper proposes a new method called Crowd-Calibrator to calibrate models for subjective tasks leveraging disagreements among annotators. The method is based on the soft-label approach, to which it adds the idea of abstaining from the the selective prediction framework.  If a model's softmax output distribution is close to the human judgment distribution, the model makes a prediction; else the model abstains. The results suggest that the proposed method achieve better results than the baselines for the NLI domain, and for unseen texts in the hate speech domain.
strength_weakness: strength_weakness: 1. Some of the evaluation questions are not properly justified. E.g., given that this is supposed to be a new soft-label method, but is it not compared with a standard soft-label model? Also, why isn't CE_soft used as a soft evaluation metric, given that it has become semi-standard in NLP (which is not to say it doesn't have its problems!)?

2. Some of the claims are a bit strong: e.g., in the abstract, 'we argue that that subjectivity should play a role in model decisions' is a bit strong given how many papers how come out in the last five-six years making the same point?

3. A number of methods which involve modelling the individual annotators have been proposed, most famously the Deep Learning from Crowd approach (which is not cited), and more recently the approach by Davani et al (which is), it would have been interesting to see why this particular approach was chosen.
questions: questions: Questions:

1. Why didn't you use CE_soft as an evaluation metric?

2. Why didn't you compare your method with one of the standard soft-label methods (e.g., Peterson et al 2019/Uma et al 2020)?

References:

- there are many important papers making the point that disagreement is not necessarily going back at least to the work by Chklovsky and Mihalcea (2003) for wordsense disambiguation and Poesio et al (2006) for coreference, but the following papers at least should be cited if the paper is accepted:

Aroyo & Welty (2015). Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine, Vol. 36 No. 1

Rodrigues & Pereira (2018). Deep learning from crowds. In AAAI'18/IAAI'18/EAAI'18: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, Pages 1611–1618
paper_summary: paper_summary: The paper introduces Crowd-Calibrator, a soft calibration method for subjective tasks (e.g. hate speech detection or NLI) that prevents a model to output prediction if its model confidence is far different than crowd label distribution. Experiment results show that the proposed method outperforms a selective prediction baseline (majority baseline + confidence thresholding) on unseen dataset.
strength_weakness: strength_weakness: While the authors provide thorough comparison on the various distance metrics and aggregation methods, it is hard to get the main takeaway from the overall results. Which distance metric (KL, TVD, JSD) should be used when applying this method? Does it depends on the nature of the datasets (# of annotators, task, etc). What are the effects of adding entropy (+E)?
questions: questions: - Please provide definition/formula of the distance metrics used to better understand the results.
- How sensitive is the method on the nature of the task? Do you have any opinion on that? I guess one possible small experiments is to use two task in an almost identical setup (the same number of annotators) to isolate different variations of the training data you used.
paper_summary: paper_summary: The authors propose a soft-labeling approach, along with the ability to abstain, to predictive modeling of subjective natural language tasks. They evaluate their model on two datasets HateXplain (HX)  Measuring Hate Speech Corpus (MHSC) combined into a single dataset against several baselines and another abstain model. They show that the baselines outperfom their model on in-domain test data, except when their abstention coverages is adjusted to be 90%, but that their model outperforms the others on out-of-domain data.
strength_weakness: strength_weakness: - in-domain results are underwhelming, not so much because the authors's model only occasionally outperform the others but because of the inconsistency at different levels of coverage. Perhaps more test data is needed.
questions: questions: Soft-labeling is a broader context than just modeling the responses of a population of raters. It can, for instance, also be used to model a rater's own level certainly. It might be good to clarify this. See also "Learning to Predict Population-Level Label Distributions" Liu et al. 2019 for another early example the soft-labeling approach.

Comments: title: Thanks for response.
comment: nfm
Good reply, still happy to accept the paper
Thanks for the new results and the clarification. I've raised my score.
Thanks for the clarification. I will keep my original score.