Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?
Authors: Urja Khurana
Eric Nalisnick
Antske Fokkens
Swabha Swayamdipta
Venue: COLM (2024)
Keywords: calibration, human subjectivity, disagreement, hate speech detection
TL;DR: We propose Crowd-Calibrator, a soft calibration method, where we calibrate models for subjective tasks according to whether their prediction is close to what the crowd thinks.

Abstract: Subjective tasks in NLP have been mostly relegated to objective standards, where the gold label is decided by taking the majority vote. This obfuscates annotator disagreement and the inherent uncertainty of the label. We argue that subjectivity should factor into model decisions and play a direct role via calibration under a selective prediction setting. Specifically, instead of calibrating confidence purely from the model’s perspective, we calibrate models for subjective tasks based on crowd worker agreement. Our method, Crowd-Calibrator, models the distance between the distribution of crowd worker labels and the model’s own distribution over labels to inform whether the model should abstain from a decision. On two highly subjective tasks, hate speech detection and natural language inference, our experiments show Crowd-Calibrator either outperforms or achieves competitive performance with existing selective prediction baselines. Our findings highlight the value of bringing human decision-making into model predictions.

Published as a conference paper at COLM 2024
CROWD-CALIBRATOR: Can Annotator Disagreement Inform Calibration in Subjective Tasks?
Urja Khurana
Eric Nalisnick Antske Fokkens Swabba Swayamdipta' Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam
Department of Computer Science, Johns Hopkins University
Thomas Lord Dept. of Computer Science, University of Southern California [u. khurana, antske. fokkens @veu. nl, nalisnick@jhu. edu, swabhas@usc. edu
Abstract
Subjective tasks in NLP have been mostly relegated to objective standards, where the gold label is decided by taking the majority vote. This obfus- cates annotator disagreement and the inherent uncertainty of the label. We argue that subjectivity should factor into model decisions and play a direct role via calibration under a selective prediction setting. Specifically, instead of calibrating confidence purely from the model's perspective, we calibrate models for subjective tasks based on crowd worker agreement. Our method, CROWD-CALIBRATTN bution of crowd worker labels and the model's own distribution over labels to inform whether the model should abstain from a decision. On two highly subjective tasks, hate speech detection and natural language inference, our experiments show CROWD-CAL CALI-CAATBRATTRRR either outperforms or achieves competitive performance with existing selective prediction baselines. Our findings highlight the value of bringing human decision-making into model predictions.
models the distance between the distri-
1 Introduction
Natural language is inherently subjective, leading to subjectivity in classification tasks (Aroyo Welty, 2015; Plank, 2022; Cabitza al., 2023; Jamison & Gurevych, 2015; Pavlick & Kwiatkowski, 2019). Yet, in natural language processing, most such tasks are treated as there exists a single ground fruth. The conventional setup consists of a small number of annotators labeling each sample and taking the majority vote determines the final label. However, this setup dismisses subjectivity in implications (for e.g., in NLI; Pavlick & Kwiatkowski, 2019), and removes minority voices (for e.g., in safety-critical applications like hate speech detection; Prabhakaran et al., 2021; Sap et al., 2022). While there are many cases for which humans more likely to agree with each other (Jiang et al., 2021a; Salminen et al., 2019), there are also cases where there is a lack of consensus (Khurana et al., 2022). In such cases, models, rather than predicting single label, must make decisions that reflect potential disagreements. Selective prediction frameworks (Geifman & EI- El-nannvv, 2017; Chow, 1957) allow for this kind of model abstention. In this paper, we argue that selective prediction is an ideal fit for subjective tasks.
Ideally, we would want a model's confidence, i.e softmax probabilities, for its prediction to reflect human disagreement. However, neural models with a large number of parameters tend to be overconfídent (Guo et al., 2017; Chen al., 2023). A straightforward way to make model aware of human variation in the label is to use soft labels (Jamison & Gurevych, 2015; Uma et al., 2020; 2021b). Instead of a model predicting one label (hard label), we want the model to output the human label distribution. However, most available NLP datasets only provide 3-5 annotations per sample, which limits the distributions that the model can learn. We need many more annotations per sample to learn the crowd's beliefs.
How can we still have our model make human subjectivity-aware predictions? We propose CROWD-CALIBRATUN method that calibrates models for subjective tasks according to

---

Published as a conference paper at COLM 2024
crowd disagreement. Most calibrators either rely on the model's confidence or have separate calibrator that outputs if model is correct. For subjective tasks, there are limited ways of calibrating a model as mostly there is no one correct label. Our CROWD-CALIBAATON determines if the model's output distribution is close to the human judgment distribution. If that is the case, then the model makes a prediction. If the model is far from the human distribution, then the model abstains.
We apply CROWD-CALIBRATON to two subjective NLP tasks: hate speech detection and natural language inference (NLD), a task for which an abundance of human subjectivity data is available (Nie et al., 2020b). We show the potential of our setup by comparing it to selective prediction baselines such as MaxProb and Kamath et al. (2020). Our method is competitive with these baselines for hate speech detection and outperforms them for NILI We also show that our setup beats baselines on cross-dataset evaluation for both tasks. Our method beneficial for in-domain and out-of-domain datasets when we have access small set of samples with many annotations per instance (~ 100) and for out-of-domain datasets when we have access to individual annotator samples (> 2000). Our work restates the potential of using selective prediction for subjective tasks research direction.
Soft Labels for Hate Speech Detection
Our method experiments with using soft labels as a candidate to better calibrate models for subjective tasks. Soft have been used extensively in prior work on subjective task prediction (Jamison & Gurevych, 2015; Uma et al., 2020; 2021b). Based on this, we ask whether soft labels could also help improve model calibration, compared to hard labels. Moreover, these calibrated confidence estimates could be used to make the model abstain from any decision (Chow, 1957; Geifman El- Yaniv, 2017) when there is greater annotator disagreement.
the model directly learns the human label distribution-
2.1 Background
A majority vote over the annotation count of a sample x: y,, = arg maxie); (fre results in hard labels (Y₁,) as the ground truth, used in training with the Cross-Entropy loss (CEhard). However, for subjective tasks, majority voting might result in loss of vital information about the annotator disagreement and unjustified high model confidences, necessitating soft labels. Soft labels (Ys) are the probability distribution over the classes of the annotator judgments for sample. This can be done by e.g. normalizing the votes for each class or ta taking the tmfx (Uma et al., 2020) when the number of annotators per sample is low. We illustrate the distinction between hard and soft labels in Figure 1.
ANNOTAT ONS
HAND LABEL
"They are just bunch of idials nuining if ad"
Amottatons
Judgments ts
hate" hate" I'hans"
"hate"
nommaiaayctes
SOFT LABEL [0.731, 0.269]
votee :
3, 'not nate"
Figure 1: Example showing the difference between hard and a soft label.
Training. Given model f that takes input text and predicts the soft label ys, the objective involves minimizing the distance between two probability distributions: the model distribution output 9s and the soft label ys- This has been done by extending the Cross- Entropy Loss to measuring how distant is and ys are from each other (Peterson et al., 2019; Uma et al., 2020), using KL Divergence, Jensen-Shannon Divergence (JSD; the symmetric version of KL Divergence) or Mean Squared Error (Uma et al., 2021b).
Evaluating. Unlike hard evaluation metrics such as accuracy or F1-score, evaluating soft label prediction requires distance-specific metrics (Basile et al., 2021). Uma et al. (2021b) suggest using the soft losses mentioned as soft metrics. Baan et al. (2022) suggest using the Total Variation Distance (TVD).
2

---

Published as a conference paper at COLM 2024
2.2 Experimental Setup
We first investigate the utility of soft labels for the task of hate speech detection, where we have few annotations per example. If samples with more disagreement would then get lower confidences, 1 we can then use these confidences as way to prevent model from predicting when there is no clear consensus.
Data. We combine two widely used datasets: HateXplain (HX; Mathew et al., 2021) and the Measuring Hate Speech Corpus (MHSC; Sachdeva et al., 2022) for our experiments. Both provide the raw annotations for each sample. MHSC has one to five annotations per instance and binary hate speech category. The HX dataset has three annotations per instance and consists of three classes: hate speech, offensive, neutral. In Appendix H we apply CrowdTrruth (Dumitrache et al., 2018) as a dataset analysis. To match the datasets, we merge the hate speech and offensive classes in HX, essentially reducing the task to identifying offensive labels. The hard labels are derived by majority voting and the soft labels by using soft over the annotator votes per class (in line with Figure 1), due to the low annotations per instance. We split the dataset into train (41832 samples), validation (5230 samples), and test (5230 samples).
Model. We use RoBERTa-large (Liu et al., 2020)² initialized with five different random seeds. We follow the original hyperparameters as described in Appendix C. Model selection is based on JSD when training with JSD loss and macro F1 when training with CE
Metrics. We compare the performance of models trained with Chaand and JSD using macro F1, soft metrics (JSD and TVD), and confidence calibration metrics. Soft labels could provide better confidence estimates of the model's empirical performance. To measure this, we use Expected Calibration Error (ECE; Naeini et al., 2015) and Brier Score (BS; Brier, 1950). For ECE, the confidence scores of the model are binned and for each bin, the accuracy is calculated. The average difference between the calculated and expected accuracy taken over all bins. This metric is sensitive to the bin size. BS measures the mean squared error between the model confidence and its predictions. confidence score is usually measured. This is a much weaker notion as it ignores the other dimensions (Vaicenavicius et al., 2019).
NLP, only the calibration of the top
2.3 Soft vs. Hard Labels Results
Figure 2 shows the results?, As expected, using soft labels with few annotations per instance leads to (slightly) better performance on soft and calibration metrics (JSD, TVD, ECE, and BS). There is not much difference between the Macro Fl-scores for the hard and soft models, in line with previous research.
Macro F1
JSD 1
TVD
ECE
BS 4
e 0.820
0.08
0.18
0.050
0.125
0.815
0.06
0.16
0.120
CEnwd
CEnad
15D
CCardd
JSD
CCrurd
JSD
CC
JSD
Figure 2: Results of training RoBERTa-large using hard (green, left) vs. soft labels (orange, right). Arrows indicate if higher or lower value is better.
When looking at the confidence distributions for both the hard and soft loss models, there is not a lot of difference in general. In Figure 3, we visualize the confidence distributions
With model confidence for both hard and soft labels as output we mean the softmax scores outputted by the model. In this paper, we do not consider model confidence in the output distribution it predicts soft label.
Using the HuggingFacc ace Transformers library (Wolf et al., 2019).
I'n Appendix we show results when training with other soft losses.
3

---

Published as a conference paper at COLM 2024
samples for two types of agreement: perfect agreement, sample where all annotators agree with each other on the assigned label, and disagreement: a sample where at least one annotator assigns different label than the rest. In general, when training with Chard (Figure 3a), we see a strongly unimodal confidence distribution for samples with perfect agreement. The skew on disagreement samples is less but still similar to perfect agreement. For the soft loss (Figure 3b), there is more mass distributed around lower confidences, espe- cially for disagreement samples. Ideally, we would want fewer high confidences. When we disentangle correct and incorrect predictions for both perfect agreement (PA) and disagree- ment (DIS) samples, we see that with the hard loss (Figure 3c), regardless of correctness, for both agreement types most of the density is around the higher confidences. For the soft loss (Figure 3d), the confidences are more dispersed, especially for perfect agreement. Correct samples with perfect agreement are denser around higher confidences and incorrect perfect agreement samples are denser around lower confidences. This difference in distribution based on correctness for perfect agreement is not notable for disagreement samples, which is more spread out but still has more mass in higher regions.
Donstty
20
15
Disagreement
Agreement
Perfect Agreemment
Disagreement
20
Dis Insorrect
10.0
7.5
Dis Incomrect
PA Connect
10
PA Incureec
5.0
25
0.0
0.6
0.8
1.0
0.6
0.8
0.6
0.8
1.0
0.8
0.8
Confidence
Confidence
Confidence
Confidence
(a)
Confidences with (b) Confidences with (c) Entangling data and (d) Entangling data and model uncertainty with model uncertainty with hard loss training.
hard loss training.
soft loss training.
soft loss training-
Figure 3: Visualizations of the confidence distributions for both perfect agreement and disagreement samples separately in the test set. Figure (a) shows the distribution for models trained with Chaard and Figure (b) shows it for models trained with JSD. In Figures (c) - (d), we visualize the confidence distributions for four different scenarios: (Perfect Agreement, Disagreement) x [Correct, Incorrect).
We want disagreement samples to have generally lower confidence scores but our analysis shows that this is generally not the case. Our notion of disagreement in these experiments is based on 3 - 5 annotators, which can be limited in learning the crowd distribution. This motivates us to find different method to make models more aware of human subjectivity.
3 Selective Prediction
The selective prediction framework (Geifman & El-Yaniv, 2017; Chow, 1957) aims to reduce errors made by model by giving it the option to abstain. When dealing with subjective tasks, we argue that this is an attractive setup to align model output accordingly; abstain when dealing with highly subjective input and let predictions with more agreement, hence clear(er) correct answers, through. This setup consists of a base model f(x) that outputs softmax confidence distribution ÿ that we want to calibrate accordingly and decision function h(x) on top, which determines if f(x) predicts or abstains. Existing decision functions can either be confidence-based or separate calibrator model:
MaxProb This method makes use of confidence thresholding. The highest confidence max (g) is compared to a threshold f. If the confidence is higher than f, the model predicts, but if the confidence is lower than f, the model abstains. We can apply MaxProb to fx) trained with either a hard loss with majority vote labels (Dmaj) or soft loss (fs).
Kamath et al. (2020) Originally, this setup base model is correct or not for the task of Question Answering in the case of domain shift where the base model is only trained in-domain data. The calibrator, random forest
separate calibrator that predicts if the
4

---

Published as a conference paper at COLM 2024
classifier, is trained on both in and out-of-domain data. We draw analogy between their in- out-of-domain setup with perfect and disagreement in our task. In our case, the base model is only trained samples with perfect agreement, PA and the calibrator h is trained on both disagreement and perfect agreement samples and outputs h(x) = z; mpp is correct not. If z = only then does the prediction go through, otherwise, the model abstains. We train the calibrators with random forest from XGBoost (Chen & Guestrin, 2016). We use the following input features: hidden state of [CLS] of base model's last layer (Corbière et al., 2019; Zhang et al., 2021; Zhang & Choi, 2023), base model's softmax probabilities, and n-grams from the calibration data.
CROWD-CALIBRATOR
Most of the calibration research has focused on the alignment between a model's confidence and its predictive performance. However, for subjective tasks, we want this extend to reflecting human uncertainty as well, i.e. we want soft calibration approach. Instead of focusing on the correctness of the model, we ask the softmax confidence distribution of model close to the crowd distribution for given sample.
We propose CROWD-CALIBRATE
(Figure 4), a soft calibration approach where we calibrate model according to how close its confidence is to the crowd opinion. If the model confidence far off from the crowd distribution, we want the model to abstain, i.e. not make prediction. If the model confidence is close to the crowd distribution, we let the model predict. When the base model is overconfident on a sample where humans tend to disagree and thus there is no clear label, can prevent the model from making a prediction.
TRAINING
INFERENCE
BASE MOCEL
lol get outta here
TFx
YCC
77 maj rostu
maj
RASE MODE maij
GALIBRATION
we'r re gonna find you
score Crona wood
base
GROWD ESS MATOR
abstain, predict score
threshold
gonna you
wo''r gonna you
we'ro gonna
PSTIMATOR
Use difterntt aggregation stratogles distibution output distribution from the predicted crowd output distribution?
Thin
771
-Sas
aggrogate
Scrowd
Ya,
7?1
Figure 4: CROWD-CALIBRATOR our proposed soft calibration approach where we calibrate a model according to human subjectivity. We only let the model predict if there high agreement and thus clear(er) correct answer. For NLI we directly train our crowd estimator to predict the crowd distribution as we only have access to the label distribution and not individual annotators.
4.1 Pipeline
Our setup consists of three components: the base model that we are calibrating, the crowd estimator that estimates the crowd judgment, and the soft calibration. Our base model majji trained to predict the conventional majority vote label, the model we want calibrate. This is the same model trained with the hard loss in Section 2, outputting the softmax confidence distribution mmaj (also marked as Passe Figure inference).
4.1.1 CROWD ESTIMATOR
With our CROWD ESTIMATOR we want to predict the human annotation distribution considering we want to calibrate base model Mmjj according to human judgments.

---

Published as a conference paper at COLM 2024
However, we do not have this information available at test time. Our goal with the CROWD ESTIMATOR is to estimate the crowd (dis)agreement of given sample x.
The hate speech datasets that we have used until now are limited in terms of modeling the crowd. While there might be many annotators, the number of coders per instance is low 51- and many annotators have only seen a handful of samples. As there are no hate speech datasets with more than 5 annotations, we need alternative to estimating the crowd. We do have access to hate speech datasets where decent amount of annotators have seen handful of samples: the Gab Hate Corpus (GHC; Kennedy et al., 2022) and the Dynamically Generated Hate Speech Dataset (DGHS; Vidgen et al., 2021). We select individual annotators that have annotated more than 2000 samples and train different individual MLPs ma; (single layer, hidden size 512). We use the hidden state of [CLS] of base model's last layer as the input feature to predict the annotator would perceive something as hate speech. To combine the individual annotator predictions into one crowd distribution Ferowd, we employ three strategies: taking the individual predicted labels and creating distribution over them with softmax (Label Dist), averaging over the confidence scores (Avg. Conf.) of the MLPs, and Weighted Scoring (Equation 1) where we take the weighted (fraction of voters re) distance score between the average confidence distribution of voters (Mc) for a particular class c and sum over all classes C.
WS = Loec Te score
[I EnEM,Ba)) .an),,nm .Pmmi
(1)
For NLI, we have enough instance-level annotations with ChaosNLI (Nie et al., 2020b). This dataset consists of 4500 re-annotated samples from SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), and aNLI (Bhagavatula et al., 2019)5. Each sample gets 100 an- notations, which makes ideal to train our CROWD ESTIMATOR on. Our base model, RoBERTa-large, is trained on SNLI and MNLI with the majority vote. Instead of training separate annotator classifiers, we train a small MLP regressor (2-layered MLP with hidden sizes of 100 and the hidden state of [CLS] of the base model's last layer as input feature) that outputs soft labels directly. The further process is the same as shown in Figure 4.
4.1.2 Calibration
Now that we have both Umaj and Ferowd, we want to measure their distance. Thus, we experi- ment with different distance metrics (KL divergence (KL), JSD, and TVD; see Appendix for formulae). Tf the model's output distribution is far off from the human probability distribu- tion, the model will abstain, otherwise the model's prediction, argmax(0 will go through. We also experiment with adding the entropy (+E) of mmaj to the entire score to remove instances where both Pcrowd and 9muj are close to each other but have low confidence.
4.2 Metrics
We evaluate our soft calibrator through the lens of how well it abstains (does the calibrator let the correct samples through)? and how conservative it (does let through sufficient samples to achieve good performance).
Cov@Acc= Since the model abstains, we focus on coverage: the fraction of samples the base model predicts. The coverage changes as we adjust our threshold for the distance score or model confidence. By increasing the confidence or reducing the distance score, the coverage decreases and we expect accuracy to increase. Hence we compare coverage at fixed accuracy value to evaluate how conservative technique is at that point.
(Pedregosa et al., 2011). 5e filter out samples from aNLI as it has different classes. See Appendix C for details.
*Our Crowd Estimator MLPs (both tasks) are trained with
6 Adding the entropy to the KI. divergence does not result in canceling out the entropy already present the equation and reducing down to the cross entropy. We write out the proof in Appendix B.
7 Note that using the notion of correctness does not mean that we assume single ground truth, view it as observing single sample of the underlying crowd distribution.

---

Published as a conference paper at COLM 2024
AUC and AUBS We yield corresponding accuracies for each coverage when thresholding. Following Geifman & El-Yaniv (2017), we calculate the area under the accuracy-coverag curve (AÚC) to quantify how much the calibrator abstains and how correctly it abstains over the entire trajectory. Similarly, to measure if a calibration method improves calibration as coverage decreases, we compute the area under the coverage-Bi BB curve (AUBS).
AUROC Following Shrivastava et al. (2023), want to know if the thresholding, or confidence has good differentiating power between correct and incorrect predictions, which is why we plot the fraction of wrong samples let through against the fraction of correct samples let through and calculate the area under this curve.
Results
We discuss the results of our soft calibrator obtained for both hate speech and NL1 We also explore the soft calibrator's capabilities on other unseen datasets for both tasks. For all results, we highlight the best performance in blue and the worst performance red All results are averaged over five seeds. 9
5.1 Hate Speech: Calibrating from Individual Annotators
We present the results obtained in Table 1, where we use a combined test set of perfect agreement and disagreement with 3200 samples. We also experimented with GPT-4 but did not yield competitive results, which we present in Appendix A. To verify our hate speech results, we also experiment with Online Misogyny in Appendix E, obtaining similar results.
Cov@Acc= 1
0.95
AUC
AUROC1
AUBS |
OURS MAXPROB
UPA
0.8733
0.6890
0.4422
0.9277
0.7775
0.0638
Pinj looft
0.8757
0.6934
0.4839
0.9302
0.7807
0.0592
0.8799
0.6772
0.3241
0.9184
0.7574
0.0692
Pmuj TS
0.8751
0.6937
0,4834
0.9301
0.7807
0.0567
Kamath et al. (2020)
0.8683
0.6866
0.4775
0.9302
0.7830
0.0597
Label Dist. JSD+R
0.8571
0.6717
0.4663
0.9281
0.7713
0.0601
Label Dist. - TVD+E
0.8319
0.6712
0.4624
0.9270
0.7663
0,0605
Avg. Conf. JSD+E
0.8728
0.6975
0.4640
0.9283
0.7780
0.0609
Weighted Scoring- JSD+F
0.8666
0.6959
0.4507
0.9275
0.7763
0.0616
GHC Avg. Conf. JSD+E
0.8668
0.6974
0.4721
0.9288
0.7790
0.0602
GHC Label Dist. JSD+E
0.8606
0.6834
0.4777
0.9290
0.7754
0.0591
Table 1: Calibration results on the combined HX + MHSC hate speech test set for calibration. The upper part shows the results using MaxProb with RoBERTa-large models trained on different types of data (fmaj. PA)), soft loss (t)) using better-calibrated confidences with Temperature Scaling (TS) (Guo et al., 2017). We also show MaxProb when training with only perfect agreement samples (A))) for completeness with Kamath et al. (2020)'s 's base model. The lower part shows the best-performing results using our proposed CROWD-CALIBRATR with RoBERTa-large as a base model and combining individual annotator models for the CROWD-ESTIMATOR,
We see that maaj performs the best in general. The Kamath et al. (2020) baseline performs closely, with the same AUC and slightly better AUROC (40.0023). However, Kamath et al. (2020) has a more complex setup in front of MaxProb. MaxProb on soft labels s performs worse than maj in terms of AUC and AUROC, highlighting the findings discussed in Section 2.3. For CROWD-CALIBRATOS averaging over the annotator confidences and
&Code released here: https: thub. com/urjakh, crowd caldd-ttobbrrrr 9or the variation and standard deviation in change of performance with CROWDD-CALIBRATI compared to the baseline, see Appendix G.
7

---

Published as a conference paper at COLM 2024
using JSD as distance metric with entropy yields the best results. It also gets the highest Cov@Acc=90. The best AUC is 0.9290 (-A0.12%) and AUROC is 0.7790 (-A0.17%)
We present results obtained on unseen datasets in Table 2. We apply the best-performing aggregation strategies to the Davidson (5000 random samples) (Davidson et al., 2017), Founta (2500 negative and 2500 positive random sampled) (Founta et al., 2018), and full HateCheck (Röttger et al., 2021) datasets.
DAVIDSON
FOUNTA
HATECHICK
AUC 1
AUROCT
AUBS
AUC
AUROCT
AUBS 1
AUC'T
AUROCT
AUBS
MaxProb 8msj
0.6980
0.5692
0.3006
0.7695
0.6543
02195
0,5702
0.5857
0.3973
Kamath et al. (2020)
0.6723
0.5368
0.2691
0.7696
0.6457
0.2181
0.5294
0.6066
0:4412
Avg. Conf. TVD+E"
0.7197
0.6171
0.2269
0.7890*
0.6622
0.1981
0.7256
0.8248*
0.2502
Weighted 1SD+E*
0.2388
0.1964
0.6870*
0.7477
0.2895
DGHS Tabel Dist. TVD+E
0.7144
0.6088
0.2310
0.7756
0.6525
0.2116
0.7172
0L8303
0.2619
Table 2: Calibration results on unseen hate speech datasets: Davidson, Founta, and Hate- Check. We compare our CROWD-CALIBRATOR with the best-performing baselines. We the results for the Aug. Conf. TVD+E and Weighted Score -SD+E aggregation strategies using DGHS annotators for Davidson and HateCheck and GHC annotators for Founta as these sets of annotators give the best results. "paired t-test with p-value < 0.05.
Here see that our CROWD-CALIBRATON can outperform both MaxProb and Kamath et al. (2020) on other datasets. In all cases, the best-performing method is a variant of our proposed setup. For Davidson and HateCheck, we show the results when only using DGHS annotators, and for Founta when only using GHC annotators (indicated with a **) as these give the best results. This reveals the sensitivity of our method the individual annotators on which it is training. For the original test set, the Gab annotators were more useful instead.
5.2 NLI: Calibrating from Crowd Distributions
We show the results obtained on the test of ChaosNLI (312 samples) in Table 3. We only compare with MaxProb since Kamath et al. (2020) has a complex setup but, compared to MaxProb, yields comparable worse results for hate speech. We see that with TVD and entropy, we can beat the baseline with an increase of 4.65% in AUC and 8.37% in AUROC.
cov@acc=0.8
cov8acc=0.91
AUC 1
AUROC1
AUBS 1
MaxProb ýmaj
0.6244
0.0846
0.8114
0.6720
0.1147
KI.
0.61677
0.2455*
0.8226
0.6774
0
K
0.7276
0.3603*
0.8540
0.7400
0.0846*
TVD
0.7128
0.3545*
0.8468*
0.7325*
0.0802*
TVD+E
0.7596*
0.3558*
0.8579*
0.7557
0.0841
JSD
0.6833
0.2609
0.8304
0.7027
0.0862*
JSD+E
0.7045*
0.2801*
0.8435*
0.7262*
0.0925*
Table 3: Calibration results on the test set of ChaosNLL We compare the best-performing selective prediction baseline (MaxProb maaj) based on performance on the hate speech sets. For CROWD- CAW.D-RRR,RRTOOR, we show the results for different distance metric variations and include entropy into the score or not. *paired t-test with p-value 0.05.
We also apply our method to other datasets where it beats the baseline: the test set of the ANLI (Nie et al., 2020a) (3200 samples) dataset and the test set of WANLI (5000 samples) (Liu et al., 2022). We show these results in Table 4. For ANLI, adding the entropy decreases performance but MaxProb still performs the worst out of all. For WANLI too, our soft calibration method with JSD and entropy gets the highest performance and beats MaxProb. The specific strength of the calibrator seems to that maintains higher coverage when targeting higher accuracy.
8

---

Published as a conference paper at COLM 2024
ANLI
WANLI
cov@acc=
AUC
AUROCT 1
AUBS
L
0.7
0.75
0.8
AUC 1
AUROC 1
AUBS
MaxIProb Dmnj
0.2809
0.4503
0.4300
0.6446
0.3807
0.1099
0.7259
0.6288
0.1711
KI.
0.3345'
0.5285
0.3400
KL+E
0.3026
0.4877
0.4037
0.6177
0.3364
0.1450
0.7276
0.6217
0.1658
TVD
0.3266*
0.5185*
0.3583
TVD+F
0.3011
0.4810*
0.4072*
0.6398
0.3623
0.1424
0.7309
0.6290
0.1646
JSD
0.3336*
0.5248*
0.3461
JSD+E
0.2930
0.4674
0.4176
0.6450
0.3989
0.1878
0.7350
0.6346
0.1638
Table 4: Calibration results on other unseen NLI datasets. "paired t-test with p-value < 0.05
5.3
Qualitative Examples A Offensive Content
Though a full manual qualitative analysis is out of the scope of this paper, we illustrate BROWD-CALIBRATOR with some examples of its output in Table 5. We selected examples that either have very small scores (obvious pass-throughs) or large ones (obvious abstains) according to
so that they are representative of various thresholds. As such, for hate speech, the shown examples are obvious cases of the ground truth when let through but rather debatable when abstained. For NLI, misclassification is prevented. These examples indicate that our approach may be useful for manual analysis of the data, e.g. retrieving cases of agreement between the base model and BROWD-CALIBRATE but different ground truth can be quick tool to find (some) annotation mistakes.
Bingj
CROWD-CALIBRATOR
Ground Truth
will stck yo d'ckk and call you f"gg'l This is amazing! Kudos for that women for being so strong
hale
hale
nout-hate
non-hate
Imagine the kid was black in white class, liberal lunatics would go to war with the white race <user>> Men r trash tho right
hute
non-
non-hate
hate
woman walks with her big purse. a beautiful female passing by crowd
nentral
neutral
bicycles stationed while a group people socialize. peo-
entailment
x
contradiction
ple talk near bunch of parked motorcycles.
Table 5: Examples of what
abstains (X) on and lets the base model maj predict (V) for both hate speech and NLI (premise, hypothesis).
6 Related Work
Learning from disagreement. Recent research has advocated incorporating the notion of subjectivity into our models and NLP pipeline (Aroyo & Welty, 2015; Cabitza et al., 2023; Plank, 2022; Plank et al., 2014; Nie et al., 2020b; Chklovski & Mihalcea, 2003). A direct way is to move away from hard labels to soft labels (Uma et al., 2021a; Peterson et al., 2019; Wu et al., 2023; Jamison & Gurevych, 2015; Fornaciari et al., 2021). Such labels can also be used to model approaches focusing on annotation disagreement range from probabilistic setup (Raykar et 2010) to modeling the annotators themselves (Rodrigues & Pereira, 2018; Davani et al., 2022; Guan et al., 2018). Zhou et al. (2022) explore distribution estimation methods. Utilizing disagreement for hate speech detection systems has had limited attention. Leonardelli et al. (2021) investigate the use of different data splits based on (dis)agreement for offensive language and Al Kuwatly et al. (2020) group annotators based on their demographic to investigate their biases for hate speech classification. Davani et al. (2022) show the benefit of using individual annotator information but require data where individual annotators have labeled many samples. This is something that many datasets including the ones used in the paper-the HateXplain, Measuring Hate Speech Corpus, and NLI datasets not have. Hence, we could not empirically compare to their approach.
annotator's 's uncertainty for label's distribution (Liu et al., 2019). Other

---

Published as a conference paper at COLM 2024
Calibration. Investigating if model is well-calibrated has been gaining traction (Desai & Durrett, 2020; Jiang et al., 2021b; Corbière et al., 2019; Baan et al., 2022; Ulmer et al., 2022; Nalisnick et al., 2018; Hendrycks & Gimpel, 2016). Guo et al. (2017) show how modern neural networks are miscalibrated. Recalibration can be done post-hoc (Guo et al., 2017; Gal & Ghahramani, 2016; Jiang et al., 2021b) or through hybrid human-model approach (Kerrigan et al., 2021), amongst others. Efforts are being made to understand the calibration of LLMs, through their verbal (Tian et al., 2023; Krause al., 2023) and probabilistic confidence (Shrivastava et al., 2023). Shrivastava et al. (2023) show how using a mixture of linguistic and probabilistic confidences leads better calibration. Vidgen et al. (2020) use Bayesian approach to recalibrate models for abusive language and show that uncalibrated models are not in line with annotators.
Selective prediction. Selective prediction/learriinn to abstain is relaxation of learning to defer (Madras et al., 2018): when a model determines when to defer to an expert by modeling that expert's knowledge. Learning to abstain /reain//rejct, the expert, implicitly assuming a fallback mechanism with uniformly better accuracy than the rejection threshold. Learning to defer has been extended to multiple experts (Verma et al., 2023). In NLP (Xin et al., 2021), there has been a lot of focus on selective prediction for Question Answering (Kamath al., 2020; Rodriguez al., 2019; Zhang et al., 2021; Zhang & Choi, 2023; Varshney & Baral, 2023; Cole et al., 2023; Yoshikawa & Okazaki, 2023).
the other hand, does not model
7 Discussion and Conclusion
We propose CROWD-CALIBRATON a soft calibration method for subjective tasks that refrains from predicting if its confidence is far from the crowd label distribution. We show the utility of our method by applying it to two complementary data scenarios: the real- world scenario with only access to a large number of individual annotations and the ideal scenario with a large number of sample annotations. Our experiments show that CROWD- CALIBRATOR outperforms the MaxProb baseline for the task of NLI, both on the respective test set (albeit small) and two other datasets. For hate speech, our method is competitive with the baselines on the respective test set and outperforms baselines on unseen datasets.
Our proposed setup clearly works with fewer samples but requires a high amount of sample annotations (~ 100) estimate the crowd distribution. This is less the case when we estimate the crowd distribution from individual annotators. Yet the number of annotators varied across tasks, with fewer annotators available for hate speech (24) than for NLI (100). Therefore, it is clearly beneficial to have a high amount of instance-level annotations, as also seen in Gruber et al. (2024). In future work, investigating the number of annotators and its effect on our method poses an obvious and interesting direction. Thus applying our method to a novel task means that we need at least a small amount of multiply annotated data reflecting different annotator judgments. As such data is not easily available for many subjective NLP tasks, this is a limitation that needs to be considered.
Takeaways. While our method shows sensitivity to the number of annotations, in both data scenarios we see notable results for unseen data. Our method offers flexibility in the choice of the distance metric and aggregation strategy. Which combination works best is largely dataset-dependent. The various settings we experiment with are not exhaustive options and can easily be replaced by other metrics or strategies. Generally, JSD and TVD gave the best results in combination with aggregating through label distribution or averaged confidences. In all cases, it is beneficial to add entropy to the score of the metric as it prevents the model from making decisions when both the model and crowds are uncertain.
In line with other work advocating for more subjectivity- hope our findings encourage more research in the direction of subjectivity-based selective prediction and better design of datasets and model setup for subjective tasks. we really want to understand the crowd and create more human-aligned applications, we need many more instance-level annotations from a diverse set of annotators with a high individual annotation count.
model behavior, we
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We thank Rajeev Verma for his helpful feedback on the draft version. We are also grateful to the anonymous reviewers for their valuable feedback that helped improve the paper. The figures and affiliation emojis have been designed using resources from flaticon.com, the tulip created by smaahiccons, crab by mihimihi, and the rest of the emojis by freepik. This research was (partially) funded by the Hybrid Intelligence Center, a 10-year programme funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, as well as by awards from the Allen Institute for Al and Intel Labs.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777, 2023.
Hala Al Kuwatly, Maximilian Wich, and Georg Groh. Identifying and measuring an- notator bias based on annotators' demographic characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pp. 184-190, Online, November 2020. Association for Computational Linguistics. doi:
URL
https://aclantthologs org/2020.002.2 a1w-1.21.
Lora Aroyo and Chris Welty. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15-244 2015.
Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. Stop measuring calibra- tion when humans disagree. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 1892-1915, Abu Dhabi, United Arab Emirates, December 2022. Associa- tion for Computational Linguistics. doi: 10.18653/v1 http:////2222.mmppmai... https://aclaantholog. org/2022. emnlp-main. 124.
URL
Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. We need to consider disagreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pp. 15-21, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1 w 22021.ppp-....... URL https://acllanthooogy org/ 2021 1ppf-1.3.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtz- man, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive common- sense reasoning. In International Conference Learning Representations, 2019.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, PP. 632-642. 642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/ / 115-1075. URL https: httttp:://aclaathoooog. org/D15-
Glenn W Brier. Verification of forecasts expressed terms of probability. Monthly weather review, 78(1):1-3, 1950.
Federico Cabitza, Andrea Campagner, and Valerio Basile. Toward perspectivist turn in ground truthing for predictive computing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6860-6868, 2023.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, PP. 785-794 2016.
11

---

Published as a conference paper at COLM 2024
Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. A close look into the calibration of pre-trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1343-1367, Toronto, Canada, July 2023. URL
Association for Computational Linguisties. doi: 2..1663//112223.acha https://aclanthhooogs org/ acl -l1g..5..
Timothy Chklovski and Rada Mihalcea. Exploiting agreement and disagreement of hu- man annotators for word sense disambiguation. In Recent Advances in Natural Language Processing, 2003.
Chi-Keung Chow. An optimum character recognition system using decision functions. IRE
Transactions on Electronic Computers, (4):247-254, 1957.
Jeremy Cole, Michael Zhang, Dan Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Ja- cob Eisenstein. Selectively answering ambiguous questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 530-543, 2023.
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Ad- dressing failure prediction by learning model confidence. Adoances in Neural Information Processing Systems, 32, 2019.
Aida Mostafazadeh Davani, Mark Diaz, and Vinodkumar Prabhakaran. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Compulational Linguistics, 10:92-110, 2022. doi: 10.1162/tacl a 00449.
URL https:: https://aclantholooy) org/2022. tacl-1.6.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th Interna- fional AAAI Conference on Web and Social Media, ICWSM '17, pp. 512-515, 2017.
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 295-302, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/vi
2020.emmlp-main.21 URL https: hatps/////caaatoooggoyy or org/2020 emnlp-main.22 21.
Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, and Chris Welty. Crowdtruth 2.0: Quality metrics for crowdsourcing with disagreement. In 1st Work- shop Subjectivity, Ambiguity and Disagreement in Crowdsourcing, and Short Paper 1st Workshop on Disentangling the Relation Befween Crowdsourcing and Bias Management, SAD+ CrowdBias 2018, pp. 11-18. CEUR-WS, 2018
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. Beyond black & white: Leveraging annotator disagreement via soft-label multi- task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pp. 2591-2597, Online, June 2021. Association for Computational Linguistics. doi: 00.18653/v1/2001 naacl-main.200 204. URL https: /ttbpp:://ccai h/tas//thllalalooooooooooo org/2021111.... naacl-main 204.
Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. Large scale crowdsourcing and characterization of twitter abusive behavior. In Proceedings of the international AAAI conference on web and social media, volume 12, 2018.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger
(eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR. URL htwph:tt/////tt /pwp://///socccco/...eoe mlr wwr.prress/v4... 16....1.... html.
Yonatan Geifman and Ran El-yv..v. Selective classification for deep neural networks.
Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at COLM 2024
Cornelia Gruber, Katharina Hechinger, Matthias Assenmacher, Göran Kauermann, and Bar- bara Plank. More labels or cases? assessing label variation in natural language inference. In Valentina Pyatkin, Daniel Fried, Elias Stengel-Eskin, Elias Stengel-Eskin, Alisa Liu, and Sandro Pezzelle (eds.), Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language, pp. 22-32, Malta, March 2024. Association for Computational Linguistics. URL pettp:://aclantholl.e owg.orgg//// 2024. wwwmplicittiiim... 1-1.2.
Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. Who said what: Modeling individual labelers improves classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Ella Guest, Bertie Vidgen, Alexandros Mittos, Nishanth Sastry, Gareth Tyson, and Helen Margetts. An expert annotated dataset for the detection of online misogyny. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1336-1350, Online, April 2021. Association for Computational Lin- guistics, doi: (0118653vv12000 eac.-mmam..n1le URL https://aclanthoooggy org/2021. eacl-main. 114.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. In International Conference on Learning Represen- fations, 2016.
Emily Jamison and Iryna Gurevych. Noise or additional information? leveraging crowd- source annotation item agreement for natural language tasks. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Language Processing, pp. 291-297, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 11116653/v1/115//1
URL https: / aal.atoooaaaa.ooo.ao http:/tthhooggoog/////o//g org/D15- 1035.
Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R Brubaker. Under- standing international perceptions of the severity of harmful content online. Plo5 one, 16 (8):e0256762, 2021a.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 2021b. doi: 10.1162/ tacl URL https: //acl:: www.acollnnootoooooooooooooooooe www.hology.org/222211 org/2021 tacl-1.55 17.
Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5684-5696, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/ 18653/v1/200 wll2222.acllmain..... main.cm..m.
URL http:/::///alathhologgg/o www.///.....././oo acl acl-mainn.55 503.
Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Nortillo-Wightmen Elaine Gon- zalez, et al. Introducing the gab hate corpus: defining and applying bate-based rhetoric to social media posts at scale. Language Resources and Evaluation, pp. 1-30, 2022.
Gavin Kerrigan, Padhraic Smyth, and Mark Steyvers. Combining human predictions with model probabilities via confusion matrices and calibration. Advances in Neural Information Processing Systems, 34:44221,4444411111 34:4421 4434, 2021.
Urja Khurana, Ivar Vermeulen, Eric Nalisnick, Marloes Van Noorloos, and Antske Fokkens. Hate speech criteria: A modular approach to task-specific hate speech definitions. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pp. 176-191, Seattle, Washington (Hybrid), July 2022. Association for Computational Linguistics. doi: 10. org/2022.2222 woah-1.17.
18653//v12002.woallll w.11 URL WRR.httts://aaaaathollo.
Hannah Kirk, Wenjie Yin, Bertie Vidgen, and Paul Röttger. SemEval-2023 task 10: Ex- plainable detection of online sexism. In Atul Kr. Ojha, A. Seza Doğruöz, Giovanni
13

---

Published as a conference paper at COLM 2024
Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, and Elisa Sartori (eds.), Proceedings of the 17th International Workshop on Semantic Evaluation (SemEoul-2023), pp. 2193-2210, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: URL htttps://aclanttii
org/ 2023. semeval-1.380
Lea Krause, Wondimagegnhue Tufa, Selene Baez Santamaria, Angel Daza, Urja Khu- rana, and Piek Vossen. Confidently wrong: Exploring the calibration and expres- sion of (un)certainty of large language models in multilingual setting. In Albert Gatt, Claire Gardent, Liam Cripwell, Anya Belz, Claudia Borg, Aykut Erdem, and Erkut Erdem (eds.), Proceedings of the Workshop on Multimodal, Multilingual Natural Lan- guage Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pp. 1-9, Prague, Czech Republic, September 2023. Association for Computational Linguistics. URL https://aclaanthol.gs org/2023 mmmng-11......
Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. Agreeing to disagree: Annotating offensive language datasets with annotators' disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10528- Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653 http///112222..mmlll... .822. URL httpss::/aalaanthlllgg//// org/ 2021 wwm..emnnppmm....m.emmmmmmmmmlmn wem.p-mmiin.......mmmmmmmmmm 822.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: Worker and Al collaboration for natural language inference dataset creation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6826-6847, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.fmn /2tp222..iningseennn.a.. 508. URL
orgee..rrr///// org/2022. findings emnlp. 508.
Tong Liu, Akash Venkatachalam, Pratik Sanjay Bongale, and Christopher Homan. Learning to predict population-level label distributions. In Companion Proceedings of The 2019 World Wide Web Conference, pp. 1111-1120, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2020. URL httpp:///ppeereeeewwewee////////
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving fairness and accuracy by learning to defer. Advances in Neural Information Processing Systems, 31, 2018.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp- 14867-14875, 2021.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshmi- narayanan. Do deep generative models know what they don't 't know? In International Conference on Learning Representations, 2018
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ppp 4885-4901, Online, July 2020a. Association for Computational Linguistics. doi: 10.16653/v1/2020 acl-main.441. URL ttttp:://aclanthol.og
2020 acl -maim.44.. 441.
Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9131-9143, Online, November 2020b.
14

---

Published as a conference paper at COLM 2024
Association for Computational Linguistics. doi: (0.18653/112202.mmmm/// org/ 1amllpmmain.344
URL
Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677-694, 2019. doi: 10.1162/
tacl 00293. URL https: //acl anthology htttoo//
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhoffer R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky.
Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Barbara Plank. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Yoav Goldbberg Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 10671- 10682, Abu Dhabi, United Arab Emirates, December 2022, Association for Computational Linguistics. doi: 2022. www.e-mannmnnn.... 731.
http:/2022.mmmmmmmmmmm..... URL
org/
Barbara Plank, Dirk Hovy, and Anders Segaard. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 507-511, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.315/v1/P14-2005 URL https: / haclss//oolataoooggo.ooooo org/ P14-2083.
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. On releasing annotator-level labels and information in datasets. In Proceedings of the Joint 15th Lin- guistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pp. 133-138, Punta Cana, Dominican Republic, November 2021. Associa- tion for Computational Linguistics. doi: 1018653/v1/222.aaw.111 URL https: //aclanthology, org/2021. 1aw-1......1.111 1.
Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of machine learning research, 11(4), 2010.
Filipe Rodrigues and Francisco Pereira. Deep learning from crowds. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Pedro Rodriguez, Shi Feng, Mohit lyyer, He He, and Jordan Boyd-Graber. Quizbowl: The case for incremental question answering. Xiv preprint ar Xiv:1904.04492, 2019.
Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. HateCheck: Functional tests for hate speech detection models. In Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Pa- pers), pp. 41-58, Online, August 2021. Association for Computational Linguistics. doi: 10.18653 /ttp///11221111llll/om/
URL https: hww.s//hoolaaalooooo.oooooo 2021 http//2211...11oggee.. 4.
Pratik Sachdeva, Renata Barreto, Geoff Bacon, Alexander Sahn, Claudia von Vacano, and Chris Kennedy. The measuring hate speech corpus: Leveraging rasch measurement theory for data perspectivism. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LKEC2022, pp. 83-94, Marseille, France, June 2022. European Language Resources Association. URL https: ptttps://aclanthottt aclaathooootoyyy..oo orgg/022 nlperspectives-1.111
Joni Salminen, Hind Almerekhi, Ahmed Mohamed Kamel, Soon-gyo Jung, and Bernard Jansen. Online hate ratings vary by extremes: A statistical analysis. In Proceedings of the 2019 conference human information interaction and retrieval, pp. 213-217, 2019.
15

---

Published as a conference paper at COLM 2024
Maarten Sap, Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational I Linguistics: Human Language Technologies, pp. pp.5884-555 188445906, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 11188633v1120000 naacl-main.431. URL htttps//acclantholog. org/2022. naacl-main. 10mmai..3111 431.
Vaishnavi Shrivastava, Percy Liang, and Ananya Kumar. Llamas know what gpts don't show: Surrogate models for confidence estimation. arXiv preprint arXtt:2311.088// 2023.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 5433-5442, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 100.0..emmpmmmi.3300. /aaps://acllas hatpss/////laaaolooggyy org/ 2023. momnlp-maing 330.
URL https:
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Xiir:2307.00228. 2023.
Dennis Ulmer, Jes Frellsen, and Christian Hardmeier. Exploring predictive uncertainty and calibration in NLP: A study on the impact of method & data scarcity. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2707-2735, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v vv1 2022..indings-emmllp 198. URL https: https://aclantholo.g org/ Windinng--enii 198.
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. A case for soft loss functions. In Proceedings of the AAAI Conference on Human Computation and Crowodsourcing, volume 8, pp. 173-177, 2020.
Alexandra Uma, Tommaso Fornaciari, Anca Dumitrache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. Semeval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEoul-2021) pp. 338-30--00 338,347,0 2021a.
Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385-1470, 2021b.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd international conference on artificial intelligence and statistics, pp. 3459-3467. PMLR, 2019.
Neeraj Varshney and Chitta Baral. Post-abstention: Towards reliably re-attempting the abstained instances in QA. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: LLnng Papers), pp. 967-982, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/11/2003acccooo https://aclanthholgg/ htw.aaantthooooorgg//2/. www./r/////// acl 15.
URL
Rajeev Verma, Daniel Barrejón, and Eric Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 11415-11434. PMLR, 2023.
Bertie Vidgen, Scott Hale, Sam Staton, Tom Melham, Helen Margetts, Ohad Kammar, and Marcin Szymczak. Recalibrating classifiers for interpretable abusive content detection. In Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pp. 132-138, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653 v1 /2000.nn/pcss- 1.14. URL https::/aallanntoooo....oo org//200.....//00.o..o/ wrtp://.00//..no..osoo.. nlpcss-1.14 1.14.
16

---

Published as a conference paper at COLM 2024
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1667-1682, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 org/2021 acl-long. 132.
acl-long.132. URL
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018.
Association for Computational Linguistics. doi: 10.18653 /1/1///111111111111.1 URL https: wwwptntooooooooooo//eeea 1101.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arv preprint arXiv:1910.08771, 2019.
Ben Wu, Yue Li, Yida Mu, Carolina Scarton, Kalina Bontcheva, and Xingyi Song. Don't waste single annotation: improving single-label classifiers through soft labels. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5347-5355, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ v1/2023..vvvmmmv 211//22..nniings--eee.... findings-emmlp.355. URL https:
//aclanthology wwwpanthhooog..ogg/222222 wwg.oo2g./......ooo Findings-emnla 355. 355.
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing ( (Volume 1: Long Papers), pp. 1040-1051, Online, August 2021. Association for Computational Linguistics. doi: 10. 111188631112000 /2l.a//.......oooggggg... 84. URL https://aclaanthology org/2021 acl long...g.... 84.
Hiyori Yoshikawa and Naoaki Okazaki. Selective-LAMA: Selective prediction for confidence-aware evaluation of language models. Andreas Vlachos and Isabelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EACI. 2023, pp. 2017-2028, Dubrovnik, Croatia, May 2023. Association for Computational Linguis- tics. doi: 10.18653 / v1 22122223.. 1023..indinggseac..155 URL https:://allanthology/ or2/2023.
findings-eacl. 150.
Michael Zhang and Eunsol Choi. Mitigating temporal misalignment by discarding outdated facts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14213-14226, 2023.
Shujian Zhang, Chengyue Gong, and Eunsol Choi. Knowing more about questions can help: Improving calibration in question answering. In Findings of the Association for Computational IamaalLLiggee Linguistics: ACL-IJCNLLP 2021, pp. 1958-1970, Online, August 2021. As- sociation for Computational Linguistics. doi: 10.186....//1 (1653vvl/// http//222...innnasaeeeee https://aclanes https::/aclantholoogy org/2021, findings-aci acl. 172.
URL
Xiang Zhou, Yixin Nie, and Mohit Bansal. Distributed NLI: Learning predict hu- man opinion distributions for language reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 972-987, Dublin, Treland, May 2022. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/2222 http://w112222.fddinn----- aic.. URL nttps://aclanthologyy org.:/gg..rrg//2/// findings-acl, 79.
17

---

Published as a conference paper at COLM 2024
A Results with LLMs
We use GPT-4 (gpt-4-1 1oppt---116-ppp 1014--116-ppeeee 10--preveew) (Achiam et al., 2023) to investigate if open-source LLMs are useful in detecting hate speech and if their linguistic and model confidence aligned with human subjectivity. Following Tian et al. (2023), we prompt the model to detect if given sample is hate speech and if can express its verbal confidence (GPT-4 V) through score between 0 and 1. We also take the log probabilities (GPT-4 L) of the label the LLM predicts.
We use the following prompt, following Tian et al. (2023):
GPT-4 Prompt
Classify with one of the following labels: (a) not hate speech, (b) hate speech, and the probability that it is correct (0.0 to 1.0) for the following sentence. Give ONLY the label and probability, no other words explanation.
Follow this format:
Label: <most likely label, either or b, as short possible; not complete sentence, just the label!> Probability: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability! The sentence is: """(sentence)""".
We show the results in Table 6, which clearly show that GPT-4, with both sources of confidence, not able to get close to the simplest baseline of MaxProb Imaj not even reaching an accuracy of 0.95 when using verbal confidences.
Cov@Acc= t
0.85
0.90
0.95
AUC f
AUBS 4
Imaj
0.8757
0.6934
0.4839
0.9302
0.0592
GPT-4V
0.4936
0.2993
0.8320
0.1390
G
0.5870
0.4516
0.2512
0.8756
0.0875
Table 6: Calibration results on the combined HX + MHSC hate speech test set for calibration. The upper part shows the results using MaxProb with RoBERTa-large when training on the majority vote. We compare this to GPT-4 when using verbal confidence estimates and the actual log probs for the label.
We also attempted using LLAMA-2 7B Chat (Touvron et al., 2023) but due to the strict safety instructions, it refused to classify hate speech.
B Equation KL Divergence with Entropy
As score to calibrate our model with, we initially used only the KL Divergence between the crowd observation P and the model softmax distribution Q: KL(P]IQ). This, however, does not take into account when both P and Q have low confidence. To filter out such cases, we add the entropy of Q: H(Q). When both and Q are close to each other and there is high confidence, the difference score will be small. If both are close to each other but there is low confidence, the difference score will still be larger due to the added entropy. We write out the entire equation below:
18

---

Published as a conference paper at COLM 2024
P(x)
KI((PII)) Σ
Pox)log((
Q(x)
xEX
H(Q)
Σ x
Q(x) log (Q(x))
KL(PIIQ)+ Σ P(x)log( P(x)
[
Σ
Q(x) (()))
Q(x)
xeX
x
11Σ (()) log (o())(()))) log)(((((())
xeX
XEX
Σ P(x) log((()))
Σ Q(x) (((x)) XCX
XCX
Σ log()))(P(())) P(x) (o)) Q(x) log (x)))
xeX
C
Training Details
For each experiment, we give the hyperparameters and the dataset sizes used. All of our results are aggregated over random seeds and our experiments are done on Titan 6000.
RoBERTa-large on MHSC + HX (soft and hard) To train our models we follow the original hyperparameters. We train for 10 epochs, with a learning rate of le weight decay of 0.1, and 6% warmup steps. We training batch size of We train these models 41832 samples and validate on 5230. The results shown in Section 2.3 are on the test set with 5230 samples.
Kamath et al. (2020) We use the same original hyperparameters for the RoBERTa-large base model that is now only trained on 32316 perfect agreement samples and validated on 4039 perfect agreement samples from MHSC HX. Our Random Forest calibrator, trained with XGBoost has a learning rate of 0.07, max depth of 5, and 500 parallel trees. The calibrator is trained on mixture of perfect agreement samples (2020) and the rest disagreement samples to bring the total to around 7000 samples to balance according to correctness and agreement.
Individual Hate Speech Annotators Each annotator dataset is split into 80% training, 10% validation, and 10% test. For each annotator, we train MLP, that is a single layer MLP with hidden size of 512, for the rest using the default parameters in scikit-lears
ChaosNLI ChaosNLI comes with 4500 samples from which we remove the aNLI samples since those have different classes. We train on 2490 samples, validate on 311 samples, and test on 312 samples. We use a two-layer MLP regressor with hidden sizes of 100, for the rest using the default parameters scikit-learn.
Selective Prediction MHSC HX The selective prediction results for hate speech MHSC + HX are done on 3200 samples with mixed perfect and disagreement.
D Distance Metrics Formulae
To calibrate our base model m in soft fashion, we need distance metrics to determine the proximity between the base model's predicted distribution Jbuse (also referred to as p when the base model majj is trained to predict the majority vote) and the crowd distribution Feoos. We experiment with the following widely-used distance metrics:
KL DIVERGENCE (KL, Equation 2) measures the difference of one reference distribution from the other, however, it is not symmetric. If the other distribution is the reference
19

---

Published as a conference paper at COLM 2024
distribution, we will get different output.
KLlgtuser Berowd)
Dcrowd (Feroud - log Piase)
(2)
JENSEN-SHANNON DIVERGENCE (JSD, Equation 3) measures the distance between the two distributions and is a symmetrical version of the KL divergence.
JSD(ybase baser Dcrowd) ) 1 KL "Dbase + Ierood yybase
1
9base
+ !erowd
+ KL
Berowd
(3)
2
TOTAL VARIATION DISTANCE (TVD, Equation 4) is way to measure the distance between two distributions by measuring the absolute difference in probabilities.
TVD Mubser baser Berowd)
119base Derowdl
Results for Online Misogyny
To verify the results achieved on the hate speech dataset, we apply our CROWD-CALIBRATOR setup to a similar subjective task: misogyny detection. This dataset (Guest et al., 2021) has 6383 samples with 2-3 annotations for the majority of its samples (6259), with the rest having annotations per sample. We train on 5096 samples and validate on 646 samples. This dataset is imbalanced, where most of the samples are non-misogynin in comparison to misogynistic.
To train the CROWD ESTIMATOR, use the EDOS dataset (Kirk et al., 2023), for sexism detection. We train individual MLPs just like for the hate speech dataset for annotators that have more than 3000 annotations. This gives us 14 individual annotators, which is less than what we had for the hate speech dataset.
Our results the respective test set can be found in Table 7. Due to the imbalanced nature of the dataset, we balance out the test set to get better view, where there are 72 misogynistic samples and 100 non-misogynistic.
cov@acc=0.85
cov@acc=0.91
cov@acc=0.95 1
AUC 1
AUROC 1
AUBS
MaxProb Amaj
0.8600
0.7087
0.4950
0.9244
0.7796
0.06672
Label Dist. TVD-E
0.8675
0.7163
0.4975
0,9254
0.7870
0.06585
Label Dist. JSD+F
0.8575
0..0755
0.4913
0.9249
0.7844
0.06604
Label Dist - KL+E
0.8775
0.7163
0.4975
0.9252
0.7857
0.06587
Avg. Conf. TVD+E
0.8729
0.7063
0.4650
0,9244
0.7871
0.06672
Table 7: Calibration results on the test set of the Online Misogyny dataset. We compare with
the best-performing selective prediction baseline (MaxProb Imaj). For CROWD-CALIBRATOR we show the results for different distance metric variations and include entropy in the score.
We observe results similar to hate speech detection, with competitive performance.
Comparing Distance Metrics for Soft Labels
We experiment with different loss functions that would minimize the difference between the annotator probability distribution ys and the model's output distribution gs: Cross Entropy Loss (CEgoft, Equation 5), Jensen Shannon Divergence (JSD, Equation 3), and Mean Squared Error (MSE, Equation 7). We refer to these losses as saft losses. For comparison to the conventional setup, we also use Cruss Entropy Loss in the conventional setup and refer to this loss hard loss (CFnard), where the objective is to predict the hard label y/1-
20

---

Published as a conference paper at COLM 2024
Accuracy
Macro F1 T
Cross Entropy
0./2
0.71
1.25
Soore
0.70
0.70
0.69
1.00
JSD CE CErord MSE
KL 4
JSD
0.75
Model
Score
0.100
0.50
RoBERTe-base
RoBERTa-large
0.075
0.25
JSD
CEwodEnand MSE
JSD
CEsor CEnari MSE
Figure 5: Results on the HateXplain dataset for both RoBERTa-base and RoBERTa-large when training with hard and soft losses (x-axis). Each plot is a different metric.
(Esoft(ys:33)
Σ ys log (0s.))
(5)
CEhard (yhr (yrr 11:) Σ yhu log(Una)
(6)
MSE(Ms, (yss 81)2
E1 Evaluation
To measure the performance and calibration of the model, we employ different metrics.
Hard evaluation metrics. We use these metrics to measure if the model gets its predictions right the conventional way: macro and accuracy.
Soft evaluation metrics. To measure if the probabilities that the model outputs are similar to the annotator label distribution, we use CEgoft, KL, and JSD.
Calibration metrics. To measure the confidence the model outputs reflects its empirical performance, we measure the Expected Calibration Error (ECE).
F2
Results
The hard and soft evaluation results can be found in Figure 5. Here, we see that for all metrics, the hard loss is outperformed by other losses. This is even the case for hard metrics. The superior performance of the soft losses highlights how beneficial is for models to learn the human label variation instead of the majority vote.
To evaluate the losses in terms of calibration, we show the Expected Calibration Error in Table 8. Generally, the hard loss has the worst (highest) ECE out of all and tends to be more overconfident. The soft losses have confidences closer to their empirical performance and tend to be rather underconfident. The JSD loss comes the closest to reflecting the actual performance of the model, having the lowest ECE. We therefore continue our experiments with only the JSD loss as soft loss.
21

---

Published as a conference paper at COLM 2024
Chard
CBaftt
JSD
MSE
RoBERTa-base RoBERIa-large
9.67
4.55
3.21
4.65
10.63
5.50
3.73
5.30
Table ECE for different losses on the HateXplain dataset, averaged over five runs.
G Variation in Change of Performance with CROWD-CALIBRATOR
To showcase the standard deviation and variation of our results, we look at the improvement or deterioration for the metrics we apply when using CROWD-CALIRRATON compared to MaxProb the base model trained with majority labels. We take the score achieved with CROWD-CALIBRATORR and subtract the score of MaxProb Pmnji to get the difference (A) in performance, for each seed individually. After that, we take the average and calculate the standard deviation.
We show the mean variation in Tables 9 and 11 for hate speech and Tables 10 and 12 for NLI The subscript is the standard deviation (+). A positive mean variation means that in general, our CROWD-CALIBRNN mean variation means generally worse performance.
performs better than the baseline MaxPrro.. A negative
For completeness, we also show the difference in performance between the two baselines of MaxProb and Kamath et al. (2020).
Cov@Acc=
0.90
0.95
AUC
A AUROC 1
Kamath et al. (2020)
-0.0074a.1197
-0.0068p.m69
-0.00644.m28
0.0000p.0044
0.0022a.0086
Label Dist. - JSD+E
-0.0186p.0124
-0.0217
-0.0176g.0205
-0.0021
-0.0095g.00sz
Label Dist. - TVD+E
-0.0238p.018s
-0.0222a.0117
-0.0215a.00ss
-0.00310.0015
-0.0145a.0066
Avg. Conf. JSD+EE
-0.00290.0209
0.0041o.0200
-0.01996.02ss
-0.00190.003s
-0.0027..008
Weighted Scoring E
-0.0091 p.006
0.0025p.0991
-0.03322.0180
-0.0027
-0.00444.0022
GHC Avg. Conf. JSD+E
-0.0089
0.0040p.005
-0.0118p.0224
-0.00144.0009
-0.0017a.0089
GHC Label Dist. J
-0.0151
-0.0080p.0116
-0.00622.0130
-0.00114.0009
-0.0054g.0088
Table 9: Difference (A) in increase (+) decrease (-) of metric
when calibrating with
CROWD-CALIBRATOR compared to MaxProb 9maj for the in-domain hate speech test set. We present the mean difference, with the standard deviation + as subscript.
A cov@acc=0.8 1
A cov@acc=0.91
A AUC t
A AUROC1
KL
-0.0077
0.1609p.047
0.0111a.0133
0.0054p.0236
KL +
0.10324.0672
0.2756p.0216
0.0426u.05s8
0.0680p.0123
TVD
0,0885b.0615
0.2699b.0666
0.0354..0133
0.0605p.0202
TVD+E
0.1353g.0561
0.271 20.060
0.04650.0046
0.0837
JSD
0.0590a.0668
0.1763p.0445
0.0190u.m143
0.0307p.02n
JSD+E
0.0801
0.1955p.0513
0.0321
0.0542p.0066
Table 10: Difference (A) in increase (+) or decrease (-) of metric score when calibrating with CROWD- CROWD-CALIBRATOR compared to MaxProb maaj for the in-domain NLI test set. We present the mean difference, with the standard deviation subscript.
22

---

Published as a conference paper at COLM 2024
DAVIDSON
FOUNTA
HATECHECK
AAUC
AAUROC 1
AAUC
AAUROCT
AAUC
AAUROC 1
Kamnah et (2020)
g.02577
-0.0324o.55a1
a.0
0.0086p.054
+0.447
0.0509uuss4
Avg. Conf, TVD+F**
0.02176.005
0.04
0.0202ames
0.00350.005
0.1554umes
0.2691amss
Weighted Score JSD+EM
0.01290.0011
0.0270a0185
0.0210
0.0099buur7
0.1168u.009
0.1921am
DGHS Label Dist. TVD+E
0.0164g,003
0.0396guzaz2
0.0061qquus
0.0032a0118
0.147Igumss
0.2747aum2
Table 11: Difference (A) increase (+) or decrease (-) of metric score when calibrating with CROWD-CALIBRATOR compared to MaxProb Dmaj for unseen hate speech datasets. We present the mean difference, with the standard deviation HateCheck, we show the results when only using DGHS annotators and for Founta when only using GHC annotators (indicated with **)
subscript. For Davidson and
ANLI
WANLI
cov
AUC f
A AUROCT
0.7
A 0.75
A
A AUC
A AUROC
KL
0.0535p.00ss
0.0782a.0216
-0.5097.1309
-0.337810343
-0.1034u.ww22
-0.0567
-0.09060.0136
KL+F
0.0217
0.0350p.0199
-0.026800445
-0.04424.0661
0.0352ac485
0.00180.0081
-0.007 1.0071a.008
TVD
0.0456p.01o7
0.0682a.2212
-0.2387
-0.27766.0624
-0.0920n
-0.0324p.006s
-0.0525g.0136
TVD+E
0.02026.0071
0.0007.0005
-0.0048
-0.0184p.0662
0.0325g..825
0.00
0.000266 0098
JSD
0.0527 0.0093
0.07440.0201
-0,49085
-0.33944.0047
-0.1026600668
-0,04988.0065
-0.073420.0199
JSD+E
0.0121g.0056
0.017la.0057
0.0004p.0066
0.0182g.099
0.0779qu758
0.0092au02s
0.0059
Table 12: Difference (A) in increase (+) or decrease (-) of metric score when calibrating with BOWDDCALIBBATOR compared to MaxProb Mmaj for unseen NLI datasets. We present the mean difference, with the standard deviation + as subscript.
H Analysis of Hate Speech Datasets
To measure the agreement and understand the dataset dynamic, we look at dataset char- acteristics of different hate speech datasets where we have access to who annotated what and each instance receives more than one annotation: HX (original in Figure 6 and binary version in Figure 7), MHSC (Figure 8), and GHC (Figure 9). The original HX dataset has three classes, normal, offensive, and hale speech. We then reduce it to binary classification task. We plot the amount comments annotated for each annotator and two metrics from CrowdTruuth (Dumitrache et al., 2018): sentence quality and annotator / /oorker quality, which show the reliability and agreement in dataset. Sentence quality describes the agreement of annotators for given input and worker quality describes how much an annotator agrees with other annotators.
For all datasets, we see a similar story. The majority of the annotators have not rated many instances, except for GHC where many annotators have annotated more than 2000 comments. In general, we see many high-quality workers and relatively smaller group of low-quality sentences. For HX, worker quality is lower when we keep offensive and hate speech separated.
23

---

Published as a conference paper at COLM 2024
8000
Average:
Average: 0.68
120
100
Sentences
Workers
+0
4000
20
2000
0.25
0.00
0.25
0.50
0.75
1.00
Sentence Quality Score
Quality Score
(a) Comments annotated by each annotator.
(b) Unit Quality Scores.
(c) Worker Quality Scores,
Figure 6: Dataset characteristics of HX dataset.
60
Average: 0.88
175
10000
Avarage: 0.81
Sentencess
Workers
40
#ArrstMor
7500
75
5000
2500
1.0
0.4
0.6
0.8
1.0
Comments/omozated
Sentenoe Quality Soore
Worker Quality Score
(a) Comments annotated by each annotator.
(b) Unit Quality Scores.
(c) Worker Quality Scores.
Figure 7: Dataset characteristics of HX dataset when reduced to binary classes.
Average:
2000
Average: 0.89
800
Sentences
10000
Workers
400
1000
200
0
15
25
0.00
Sentance Quality Score
0.25
0.50
0.75
1.00
0.00
Worker Quality Score
0.25
0.50
0.75
1.00
Comments annotated by
(b) Unit Quality Scores.
(c) Worker Quality Scores.
each annotator.
Figure 8: Dataset characteristics of MHSC dataset.
Average: 0.87
Average:
Sentencess
16000
Workers
7.5
10000
5.0
2.5
0.00
0.25
0.50
0.75
1.00
0.6
0.7
0.8
Comments Annotated
Sentence Quelity Score
Worker Quality Score
(a) Comments annotated by each annotator.
(b) Unit Quality Scores.
(c) Worker Quality Scores.
Figure 9: Dataset characteristics of GHC dataset.
24


Reviews:
[{"review_id": "kDrYRHjap4", "reviewer": "Reviewer_Va5r", "summary": "The author proposes a novel method to calibrate models for subjective NLP classification tasks when crowdsourced annotations are available (rather than each instance only having a single ground truth). Here, \"calibrate\" refers to abstaining from making a prediction when the model is uncertain.\n\nSpecifically, a base model is trained on aggregated labels using the usual cross entropy loss (RoBERTa is used in this paper), then a classifier is trained for each worker. The worker classifier is formed by passing the embedding of the CLS token from the base model through a single hidden layer MLP. Subsequently, the output distributions from all worker models are aggregated into a crowd distribution. The distance between this crowd distribution and the base model's output distribution is calculated, and if the distance exceeds a threshold, the model chooses not to output a prediction. The author proposes several methods for how to aggregate worker distributions into a crowd distribution and how to calculate the distance. If the dataset only contains crowdsourced annotations but without worker information, then a crowd model is trained directly to fit each instance's label distribution, and the same method is used to calculate the distance between the crowd distribution and the base model's output distribution.\n\nThe author tested the proposed method and other baselines on the tasks of hate speech detection and natural language inference (NLI). The results showed competitive or superior performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and your appreciation of us tackling a crucial issue through a novel approach.\n\n**Additional Datasets:**  We chose our datasets specifically based on their complementary properties (also noted by j6sj) the most common real-world scenario with only limited annotators per sample (HS) and the ideal but rare scenario with many annotators per (NLI). \n\nThanks for the dataset suggestions; these are similar to HS in having limited annotators per sample. We experimented with the suggested Online Misogyny dataset, and obtained results similar to HS: our Crowd-Calibrator slightly outperforms the MaxProb baseline in terms of AUC (baseline: 0.9205, ours: 0.9228) and AUROC (baseline: 0.7623, ours: 0.7741). We will add this in our paper. \n\n**Comparison with Davani et al. (DwD)** Davani et al.’s approach requires data where individual annotators have labeled many samples (as highlighted by their GoEmotions experiments). Most datasets including the HateXplain, Measuring Hate Speech Corpus, and NLI datasets do not follow this setting. For the NLI datasets, we only have access to the distribution of labels per sample. Hence, we could not empirically compare to DwD; we will clarify this in our paper.\n\n**Chosen evaluation metric**. We evaluate with widely used metrics for calibration (Brier’s Score) and selective prediction (Coverage@Accuracy, AUC, and AUROC) since our proposed setup is a soft approach for selective prediction and improved calibration. This does not mean we assume there is a single ground truth; rather we are observing a single sample from the underlying crowd distribution. Our crowd-calibrator abstains such that it does not predict samples that are far off from the crowd and where they both have high entropy. \n\n**Usage of “confidence”:** Thank you, we will update our paper with the term “softmax scores”.  \n\n**Weighted Scoring formula.** \nThe weighted scoring we used follows: \n\n$r_{non-hs} \\cdot \\text{score}(AvgConfidence_{non-hs}, \\hat{y}\\_{maj}) + r_{hs} \\cdot \\text{score}(AvgConfidence_{hs}, \\hat{y}_{maj})$\n\nwhere $r_c$ is the ratio of votes for the particular class $c$. We will include this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Annotator disagreement in subjective NLP tasks is a prevalent issue that affects many applications.\n\nThe paper introduces a novel approach to calibrate models for subjective tasks, emphasizing the value of integrating human annotator insights into the model's decision-making process.", "reasons_to_reject": "The author only conducted experiments on two tasks, where the NLI dataset has far more annotations per instance than typical ones, and the proposed model did not show superior performance to the baseline on the hate speech datasets. Additionally, in Table 2, the author chose different groups of workers to train worker models for each dataset, which also shows that the proposed model is quite sensitive to the choice of workers. In this case, the author should conduct more comprehensive testing of the proposed model on more datasets. For instance, a dataset for detecting online misogyny [1], and a dataset for multilabel emotion classification [2]. Moreover, comparisons should be made with some highly relevant works, such as [3], which also proposed training a model for each worker and discussed the calculation of uncertainty.\n\n* [1] An Expert Annotated Dataset for the Detection of Online Misogyny (EACL 2021)\n* [2] GoEmotions: A Dataset of Fine-Grained Emotions (ACL 2020)\n* [3] Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (ACL 2022)\n\nAlthough the paper primarily focuses on subjective NLP tasks, the chosen evaluation metrics are still based on the assumption that each instance has only one ground truth, such as accuracy in Cov@Acc=0.9. In fact, while reading the paper, I initially thought the evaluation would be about the ground truth being a distribution, the model also outputs a distribution, and then evaluating the performance by calculating the distance between the two distributions.\n\nAnother confusing point is \"confidence.\" For a typical softmax output, like [0.4, 0.6], we would consider it represents p(y=0)=0.4, p(y=1)=0.6, or say the prediction is y=1 with confidence=0.6. However, in subjective tasks, if the ground truth is considered a distribution, then [0.4,0.6] just represents the predicted distribution by the model, which does not contain any information about confidence. It means the model might be very certain that the distribution is [0.4,0.6] or very uncertain. The unknown confidence can be calculated through other means, such as in the proposed method, if the base model's output distribution is very close to the crowd model's output, it is considered high confidence. However, in the paper, the term confidence seems to be interchangeably used with the model's output, as mentioned in the paragraph above Figure 4:\n> \"If the model confidence is far off from the crowd distribution, we want the model to abstain, i.e. not make a prediction.\"\n\nBut in Sec 4.1.2 it is mentioned:\n> \"If the model’s output distribution is far off from the human probability distribution, the model will abstain.\"\n\nClarifying these concepts more clearly could potentially reduce reader confusion.", "questions_to_authors": "It’s not clear to me how the Weighted Scoring is calculated. Could the authors provide a formula for it?", "extra_scores": {}}, {"review_id": "n6dgQ7Jh6D", "reviewer": "Reviewer_j6sj", "summary": "This paper proposes a new method called Crowd-Calibrator to calibrate models for subjective tasks leveraging disagreements among annotators. The method is based on the soft-label approach, to which it adds the idea of abstaining from the the selective prediction framework.  If a model's softmax output distribution is close to the human judgment distribution, the model makes a prediction; else the model abstains. The results suggest that the proposed method achieve better results than the baselines for the NLI domain, and for unseen texts in the hate speech domain.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your insightful comments and positive review! We answer your questions below. \n\n**Comparison to standard soft-label methods.** We compare our results to standard soft-label methods (Peterson et al. 2019/Uma et al. 2020) which use human disagreement to create soft labels $\\hat{y}\\_{soft}$ in Table 1. To make it compatible with abstention, we apply MaxProb. However, the results are far more conservative compared to using hard labels $\\hat{y}\\_{maj}$.  \n\n**Related Work.** Thank you for the pointers; we will definitely cite these. Deep Learning from Crowd (DLC) is shown to be outperformed by learning soft labels with soft losses in Uma et al. 2020 and Uma et al. 2021, hence we did not include this method. Davani et al’s approach requires data where individual annotators have labeled many samples. This is something that many datasets and the ones we used in the paper---the HateXplain, Measuring Hate Speech Corpus, and NLI datasets—do not have. For the NLI datasets, we do not know who annotated what, we only have access to the distribution of labels per sample. Hence, we could not empirically compare to DwD; we will clarify this in our paper.\n\n**$CE_{soft}$ as an evaluation metric.** We evaluated the soft label model in Section 2.3 with Jensen-Shannon Divergence (JSD) as it is strictly more principled since unlike $CE_{soft}$ it is symmetrical. When evaluating for selective prediction, we do not use $CE_{soft}$ since our approach aims to improve calibration and selective prediction for which we use the widely used metrics. \n\n**Strong Claims.** Thank you for pointing this out, we agree and did not intend to suggest we are the first to point out that subjectivity should play a role, but we do want to restate its importance. We will make sure to add the references.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research is aware of much of the relevant literature and was carried out in a methodological sound way.\n\n2. The proposed method is interesting and, as far as I know, novel, and achieves good results on some datasets.\n\n3. The comparative  analysis of model confidence when trained using various hard and soft loss functions in Section 2.3 is, as far as I know, novel and  very interesting.", "reasons_to_reject": "1. Some of the evaluation questions are not properly justified. E.g., given that this is supposed to be a new soft-label method, but is it not compared with a standard soft-label model? Also, why isn't CE_soft used as a soft evaluation metric, given that it has become semi-standard in NLP (which is not to say it doesn't have its problems!)?\n\n2. Some of the claims are a bit strong: e.g., in the abstract, 'we argue that that subjectivity should play a role in model decisions' is a bit strong given how many papers how come out in the last five-six years making the same point?\n\n3. A number of methods which involve modelling the individual annotators have been proposed, most famously the Deep Learning from Crowd approach (which is not cited), and more recently the approach by Davani et al (which is), it would have been interesting to see why this particular approach was chosen.", "questions_to_authors": "Questions:\n\n1. Why didn't you use CE_soft as an evaluation metric?\n\n2. Why didn't you compare your method with one of the standard soft-label methods (e.g., Peterson et al 2019/Uma et al 2020)?\n\nReferences:\n\n- there are many important papers making the point that disagreement is not necessarily going back at least to the work by Chklovsky and Mihalcea (2003) for wordsense disambiguation and Poesio et al (2006) for coreference, but the following papers at least should be cited if the paper is accepted:\n\nAroyo & Welty (2015). Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine, Vol. 36 No. 1\n\nRodrigues & Pereira (2018). Deep learning from crowds. In AAAI'18/IAAI'18/EAAI'18: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, Pages 1611–1618", "extra_scores": {}}, {"review_id": "cIc0BsybBf", "reviewer": "Reviewer_efvr", "summary": "The paper introduces Crowd-Calibrator, a soft calibration method for subjective tasks (e.g. hate speech detection or NLI) that prevents a model to output prediction if its model confidence is far different than crowd label distribution. Experiment results show that the proposed method outperforms a selective prediction baseline (majority baseline + confidence thresholding) on unseen dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive review and constructive suggestions! \n\n**Main takeaway:** Our results show that our method is beneficial for seen and unseen datasets when we have access to a small set of samples that have many annotations per instance (~100). Our results also show that our method can work on unseen datasets when we have access to individual annotator samples (> 2000). We will clarify these as the main takeaways of our paper.\n\n**Questions w.r.t. scoring strategies:** Our method offers flexibility in the choice of the distance metric, KL-divergence, JSD, and TVD (for which we will add the formulae in the Appendix) and aggregation strategy. In general, JSD and TVD give the best results in combination with the aggregation strategies of label distribution or averaged confidences. In all cases, it is beneficial to add entropy to the score of the metric as it prevents the model from still making decisions when both the model and crowds are uncertain of their response, while distance metrics alone would allow the model to make a prediction. \n\n**Sensitivity:** There does appear to be some sensitivity to the nature of the task, particularly the amount of annotations we can learn from for the Crowd Estimator. We see better performance when the Crowd Estimator learns from 100 annotations per sample in the case of NLI in comparison to estimating the crowd from 24 annotators. We’ll reflect on this in our discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well-executed experimental setup. I appreciate the authors in experimenting on different baselines, select the best one, and have evaluation on seen and unseen dataset.\n- I also like that authors conduct experiments on different data situation, i.e. where there are few number vs. enough number of annotators. The setup is closer to real-world setup where we don't always have enough annotations/annotators which can give good insights for real-world experimentation.\n- Paper is well-written and easy to follow.", "reasons_to_reject": "While the authors provide thorough comparison on the various distance metrics and aggregation methods, it is hard to get the main takeaway from the overall results. Which distance metric (KL, TVD, JSD) should be used when applying this method? Does it depends on the nature of the datasets (# of annotators, task, etc). What are the effects of adding entropy (+E)?", "questions_to_authors": "- Please provide definition/formula of the distance metrics used to better understand the results.\n- How sensitive is the method on the nature of the task? Do you have any opinion on that? I guess one possible small experiments is to use two task in an almost identical setup (the same number of annotators) to isolate different variations of the training data you used.", "extra_scores": {}}, {"review_id": "Q5OQGmr2Ys", "reviewer": "Reviewer_jzn7", "summary": "The authors propose a soft-labeling approach, along with the ability to abstain, to predictive modeling of subjective natural language tasks. They evaluate their model on two datasets HateXplain (HX)  Measuring Hate Speech Corpus (MHSC) combined into a single dataset against several baselines and another abstain model. They show that the baselines outperfom their model on in-domain test data, except when their abstention coverages is adjusted to be 90%, but that their model outperforms the others on out-of-domain data.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review, constructive suggestions, and appreciation for finding the combination of soft label with abstention novel.\n\n**Inconsistency.** Our method especially works better when we aim to achieve higher accuracy, as the coverage is then higher than the baseline. The only case where this trend is broken is our Hate Speech dataset, where for Cov@Acc=0.9 our method gives the best coverage, but not for Cov@Acc=0.95. Additional experiments on the Online Misogyny task (as suggested by Var5, containing roughly similar amount of annotators) show slightly better performance than MaxProb with our Crowd Estimator for (baseline: 0.9205, ours: 0.9228) and AUROC (baseline: 0.7623, ours: 0.7741) and a somewhat similar pattern with more coverage for a higher accuracy. For Cov@Acc=0.85: baseline: 0.8875, ours: 0.8844 and Cov@Acc=0.9: baseline: 0.7094, ours: 0.7125. For Cov@Acc=0.95: baseline: 0.5125, ours: 0.5125.   \n\n**Soft-labeling for raters.** Thank you for pointing this out, we will clarify this in the paper and cite Liu et al. 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The combination of soft-label with an abstention metric is novel. It was interesting to see how performance decreased as coverage increased!\n+ out-of-domain performance is impresssive.\n+ The paper is well-written.", "reasons_to_reject": "- in-domain results are underwhelming, not so much because the authors's model only occasionally outperform the others but because of the inconsistency at different levels of coverage. Perhaps more test data is needed.", "questions_to_authors": "Soft-labeling is a broader context than just modeling the responses of a population of raters. It can, for instance, also be used to model a rater's own level certainly. It might be good to clarify this. See also \"Learning to Predict Population-Level Label Distributions\" Liu et al. 2019 for another early example the soft-labeling approach.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is a solid paper, on the topic of modelling subjective tasks and means of calibrating models to better reflect the aggregate human (crowd) distribution. The problem is novel, the work is well motivated, and the empirical evaluation convincing. Reviewers question how realistic it is to have datasets with rich crowd annotations, and thus the practical applicability of this work may be limited. However, this is not a reason to dismiss the paper: this line of work has the potential to influence future dataset annotation activities & release. The author responses helped to clarify a number of questions from reviewers (e.g., Va5r, who had several concerns, which were well addressed and the reviewer raised their score.)
Decision: Accept