Don't waste a single annotation: improving single-label classifiers through soft labels
Authors: Ben Peng Wu
Yue Li
Yida Mu
Carolina Scarton
Kalina Bontcheva
Xingyi Song
Venue: EMNLP 2023 Findings (2023)
Keywords: Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels

Abstract: In this paper, we address the limitations of the common data annotation and training methods for objective single-label classification tasks. Typically, when annotating such tasks annotators are only asked to provide a single label for each sample and annotator disagreement is discarded when a final hard label is decided through majority voting. We challenge this traditional approach, acknowledging that determining the appropriate label can be difficult due to the ambiguity and lack of context in the data samples. Rather than discarding the information from such ambiguous annotations, our soft label method makes use of them for training. Our findings indicate that additional annotator information, such as confidence, secondary label and disagreement, can be used to effectively generate soft labels. Training classifiers with these soft labels then leads to improved performance and calibration on the hard label test set.

Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels
Ben Wu, Yue Li, Yida Mu, Carolina Scarton, Kalina Bontcheva and Xingyi Song Department of Computer Science, The University of Sheffield, Sheffield, UK bpwul@shef of fiell.elleeo ac. uk
Abstract
2019), superior calibration, enhanced performance (Fornaciari al., 2021) and even enable less than one-shot learning (Sucholutsky and Schonlau, 2021).
This paper addresses limitations the com- mon data annotation and training methods for objective single-label classification tasks. Typ- ically, such tasks annotators are only asked to provide single label for each sample and annotator disagreement is discarded when a final hard label decided through majority vot- ing. We challenge this traditional approach, ac- knowledging that determining the appropriate label can be difficult due the ambiguity and lack of context in the data samples. Rather than discarding the information from such ambigu- soft label method makes use of them for training. Our findings indicate that additional annotator information, such as confidence, secondary label and disagreement, can be used to effectively generate soft labels. Training classifiers with these soft labels then leads to improved performance and calibration on the hard label lest set.
This paper's primary focus is on exploring effec- tive ways for improving classification performance using soft labels. Our experimental findings indi- cate that confidence-bast labelling significantly enhances model performance. Neverthcless, the interpretation of confidence scores can also pro- foundly influence model capability. Given the vari- ability in confidence levels among different anno- tators (Lichtenstein and Fischhoff, 1977), aligning these disparate confidence levels emerges as the central research question of this paper.
annotations,
To address this challenge, propose novel method for generating enhanced soft labels by leveraging annotator agreement align confidence levels. Our contributions include:
We demonstrate how classification perfor- mance can be improved by using soft labels generated from annotator confidence and sec- ondary labels. This presents solution to the challenge of generating high-quality soft la- bels with limited annotator resources.
1 Introduction
Reliable, human- annotated data crucial for train- ing and ovaluation of classification models, with the quality of annotations directly impacting the models' classification performance. Traditionally. in order to ensure high quality annotated data, mul- tiple annotators are asked to judge each individual data instance, and the final 'gold standard' hard label is determined by majority vote.
We propose Bayesian approach to leverag- ing annotator agreement way of aligning individual annotators' confidence scores.
We introduce a novel dataset to facilitate re- search on the use of soft labels in Natural Language Processing.
However, this hard label approach tends to ig- nore valuable information from the annotation pro- cess, failing to capture the uncertainties and in- tricacies in real-world data (Uma et al., 2021). An emerging alternative approach that addresses these limitations is the use of soft labels through techniques such Knowledge Distillation (Hin- ton al., 2015), Label Smoothing (Szegedy et al., 2016). Confidence-based Labeling (Collins al., 2022), and Annotation Aggregation (Uma al., 2020). These soft label approuches demonstrate potential for improved robustness (Peterson et al.,
2 Related Work
Current rescarch typically interprets annotator dis- agreement in two primary ways, either by captur- ing diverse beliefs among annotators, or by assum- ing a single ground truth label exists despite dis- agreement (Rottger et al., 2022: Uma et al., 2021). This paper focuses on situations where the latter
Dataset can he found at: PateNLL/dontt-ae http://ootwwwttteeeeeeee wast-w www.iiinnntltttion annotation
com/

---

viewpoint more applicable. Thus, we focus on traditional "hard" evaluation metries such as FI- score which rely on gold-label, despite the emer- gence of alternative, non-aggregated evaluation ap- proaches (Basile et al., 2021; Baan et al., 2022; Basile et al., 2020). This is made possible because we evaluate on high-agreement test sets, where the 'true' label is fairly certain.
Input
Annolator
Primery Label
Confidence
Secondary Label
Text
Annotator
Class 0
0.8
Class 2
Text
Annotator
Class
Bayasian Confidence Calibration
Calibrated Confidence
Input
Primary Label
Secondary
Text
Annotator
Class 0
0.7
Cass 2
Aggregation of annotator disagreement gener- ally falls into two categories: aggregating labels into a one-hot hard label (Dawid and Skene, 1979;
Text
Annotator
Class
0.85
Soft label conversion
Text
Input
Annotator
Soft Label
Hovy et al., 2013; Jamison and Gurevych, 2015;
Text
Annotator
0.7. 0.3.0
0.375,
Beigman and Beigman Klebanov, 2009), or model- ing disagreement as probability distribution with soft labels (Sheng et al., 2008; Uma et al., 2020; Pe-
0.425.
Annotator
0:05, 0.05,
0.175,
0.025
Merge annotation
terson al., 2019; Davaní al., 2022; Rodrigues and Pereira, 2018: Fornaciari et al., 2021).
Figure Soft label conversion pipeline soft label (Section 3.2)
Similar to Collins et al. (2022), our study ex- plores how soft labels can be generated from small pool of annotators, using additional informa- tion such as their self-reported confidence. This has benefits over traditional hard/soft label aggregation, which requires extensive annotator resources and/or reliance on potentially unreliable crowd-sourced annotators (Snow et al., 2008; Dumitrache et al., 2018; Poesio et al., 2019; Nie et al., 2020).
3.1 Bayesian Confidence Calibration
To calibrate confidence levels across annotators. we use annotator agreement proxy for reliabil- ity. For each annotator, we consider the level of agreement obtained (across all their annotations) when they have expressed a particular confidence score, and use this to re-weight their confidence.
The process comprises two steps:
First, we compute the probability of the primary label (lim) according to the confidence level (cim).
3 Methodology
This step is agnostic to the identity of the annotator.
In order to generate soft labels, our methodology requires annotators to provide confidence scores. Figure shows how each annotator provides both a primary class label and a confidence rating that represents their certainty. This ranges from 0 to 1, with representing 100% confidence. In addition, annotators can also provide an optional 'secondary' class label. This represents their selection of the second most probable class. Thus, formally, the annotation of the text x; by an annotator am con- sists of primary label lim, its confidence rating Cams and an optional secondary label 12 We use Yi to denote the text's true label.
p(u: = 4im Cim) = P(Cim (m P(lim)
(1)
P(cmm)
Where P(lm) is the prior, P(Cim|lim) is the like- lihood of the confidence score assigned to the pri- mary label and P(cim) = P(cim lim P(lim) + P(Cm P(am//laam (mmm) P(-lim) is the marginal probability. In this paper, we just make simple assumption Plimm) 1/C and P(-lim) (C- 1)/C, where C is the total number of possible classes, We also assume that P(cam
Cim and P(cmml -y:) =
Cim-
In the second step, using information about agreement, we compute the calibrated probability of the primary label (lim) given the specife annota- (am).
Overall, there are three steps to generating soft labels as shown in Figure 1:
1. Annotator confidences are calibrated using our Bayesian method (Section 3.1)
P(y = 4am am)) P(am lim ) P(lim) P(am)
(2)
2. Annotations are converted to soft labels (Sec-
tion 3.2)
Where our updated prior P(Imn) P(.)) calcu-
3. Annotator soft labels are merged into final
lated from Equation P(am (im) ) is the likelihood that an annotator assigns the label 4m matching the true label, yi, and P(am) = P(am y/: P().) + P(a.l-y:)P(-5))
"In practice, annotators can provide this directly as a per- centage or chnose from a likert-style numerical rating (c.g. 1-5) that is then converted to (-1 scale.

---

We calculate this likelihood based on annotator disagreement:
datasets release the confidence scores which anno- tators provided alongside their class labels (anno- tators are denoted by a corresponding anonymous ID).
am O 00m Omn, li)
P(aml))) Count(am O 70m: 7mm li)
(3)
Count(am (-mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm am, li) is the number of sam- ples that involves annotator am and any other an- notator(s) -Om- where both Om and at least one other annotator provided the label li. Count N 70 mm li) the number of samples that in- volves annotator am and other annotator(s) -am- where at least one annotator provided the label li. Similarly, we calculate:
4.1.1 COVID-19 Disinformation Corpus (CDS)
CDS (Song al., 2021) includes 1,480 debunks of COVID-19-related disinformation from various countries. The debunks are classified into ten topic categories (e.g., public authority, conspiracies and prominent actors). The number of annotators per instance ranges from one to six. Each annotator has provided only one first-choice topic class and their confidence score for each annotated debunk (0 Cim 9).
P(aml (y;) Count(am mmm. Om, li) Count(am 70m: 70 -li)
(4)
Count(am U-am, Om, It) is the number of sam- ples that involves annotator Om and other annota- tor(s) -m where am annotated the sample as li but others did not. Counl(am -0 7m -li) is the number of samples that involves annotator Om and other annotator mm where -0 did not annotate the sample as label li.
4.1.2 VaxxHesitancy
VaxxHesitancy (Mu et al., 2023) consists of 3,221 tweets annotated for stance towards the COVID- 19 vaccine. Each instance is categorised into pro- vaccine, anti- vaccine, vaccine-hesitant, or irrele- vant. The number of annotators per tweet ranges from one to three. Annotators provide a first- choice stance category and confidence score (1 Cim S 5).
We also employ fallback mechanisms to miti- gate cases where annotator agreement cannot be reliably estimated. If the number of counts (< 3 in our experiments) is insufficient calculate pos- terior probability, we fall back to using the prior confidence score.
VaxxHesitaney Additional Annotation As our aim is to investigate how additional information provided by annotators could impact classification performance, we also explore the integration of secondary label for instances where annotators have expressed uncertainty about their primary la- bel choice.
3.2
Soft Label Conversion & Merging Annotations
Once we have calibrated confidence, P(ys(am)) we assign this probability to the primary class 4, and assign ((ss (m)) to the secondary class (?. Thus, in Figure 1, Annotator I's soft label consists of 0.7 for their primary class and 0.3 for their secondary class. If secondary class label is not provided by the annotator, just uniformly distribute the confidence level to the other classes. In Annotator 2's case, this means 0.85 for their primary class and 0.05 for the three remaining classes.
As none of the original datasets had such sec- ondary labels, we undertake additional round of data annotation, based the original annotation guidelines. We introduce two new tasks in this data annotation round: 1) For all instances (train test set) exhibiting low confidence (less than 4), we op- tionally request that annotators provide a 'second stance' label. We guide annotators to propose this if they believe it to be appropriate, even if it wasn't 't their primary choice. Consequently, we add 569 additional sccond-choie stances. 2) We assign third annotator to all instances annotated by two annotators. As a result, we obtain majority vote for the majority of annotated tweets. This major- ity vote can be employed for hard label training in subsequent experiments.
Once we have generated soft labels for each an- notator, we merge these into a final soft label by taking the mean of each class.
4 Experiments
4.1 Datasets
We experiment with two datasets: VaxxHesitancy (Mu et al., 2023) and the COVID-19 Disinfor- mation Corpus (CDS) (Song et al., 2021). Both

---

VaxxHesitancy fold) FI Maero
CDS fold)
Labels Used
ECE Calibration
Labels Used
F1 Macro
ECE Calibration
Test-set only
Hard Label
1 and 2 1 and 2
66.9 68,7
0.262
65.2 67.7
0.182
Soft Label
0.192
0.161
Train-Test-set
Train- Test-set
Without confidence
Hard Label
I and
70.2
0.237
1
67.1
0.211
Dawid Soft Label
71.7
0.162
1
19.6
Soft Label (primury label)
71.0
0.171
68.2
0.160
Label (primary and secondary)
and
70.7
0.188
With confidence
Hard
73.1
Hard Label (label smoothed 0.1)
and
68.8
0.167
68.8
0.096
Soft Label (from primary label)
73.7
0.107
71.1
0.096
Soft Label (primary and secondary)
and
74.5
0.106
With confidence bayes calibration
Bayesian Soft Label
I and
75.2
0.099
70.4
0.118
Table I: Evaluation results for the CDS and VaxxHesitancy datasets. The 'Labels Used' column indicates whether the hard/soft labels are generated using only the primary label (1), or considering the secondary label as well (2). For the CDS dataset, labels are generated using only primary since no secondary labels are available.
4.1.3 Dataset Split
and the uniform prior distribution over this class (Szegedy et al., 2016).
We construct the test sets to contain instances where annotators reach agreement on every in- stance with high confidence scores. For VaxxH- esitancy, we follow the original train-test split by including instances whose confidence scores are larger than three in the test set. For CDS, the test set has debunks that are labelled by more than one annotator with confidence scores larger than six (on their original 10 point scale). Given the limited size of this subset, data with only one annotation but very high confidence scores also included in the CDS test set. Summary of the statistics is in the Appendix Table 2.
Soft label w/o annotator confidence We explore generating soft labels using only annotator dis- agreement. In this approach, we assign 0.7 to the primary stance, 0.3 to the secondary stance label (if available), evenly distribute the remaining probability among all other classes.
4.3 Experimental Setup
We conduct 5-fold validaaai where each fold contains the entire training set, and 4/5 folds the test set. We also investigate a second scenario in which we perform 5-fold cross-validation only on the test set. These two scenarios allow us to investigate the performance of soft labels when the level of annotator agreement differs.
4.2 Baselines & Ablations
We compare our methods against variety of hard/soft label aggregation strategies, which make use of annotator confidennces/eecondary labels to varying degrees.
These two scenarios are motivated by the fact that the train-test splits of our datasets contain an uneven distribution of samples: low annotator agreement samples were placed in the train set and high-agreement samples in the test set. This is nec- essary to enable evaluation against gold-standard test set. However, for generating soft labels, we want to use mixture of high-agreement and low- agreement annotations, so we include a portion of the original test set for training.
Hard label w/o confidence We employ majority voting for hard labels. In the absence of consensus, a class category is chosen at random.
Hard label with confidence: For each w.. we ag- gregate d; estimate single hard label Åi by giving different weights to lam according to the annotator confidence Cim-
Dawid-Skene Soft label:We utilise an enhanced Dawid-Skens model (Passonneau and Carpenter, 2014) as an alternative to majority voting, and use the model's confusion matrices to generate soft labels. This model only relies on class labels does not make use of additional information.
We perform experiments using Pytorch (Paszke et al., 2019) and the Transformers library from HuggingFace (Wolf al., 2020). We fine-tune COVID-Twitter BERT V2, a BERT large uncased model that has been pre-trained on COVID-19 data (Müller et al., 2023). We fine-tune for 20 epochs with lcarning rate as 2c-5 (1 epoch warm-up fol- lowed by linear decay) and batch size as 8, with
Label Smoothing Soft Label: For each class, we use a mixture of its one-hot hard label vector

---

A optimizer (Loshchilov and Hutter, 2017). We use cross-entropy loss. The model performance is evaluated with macro-F1 due to the imbalanced datasets, and we use expected calibration error (ECE) (Naeini et al., 2015) to measure model cali- bration.
to collect such information, it can be greatly beneficial for the creation of effective soft labels.
Bayesian calibration outperforms other meth- ods on the VaxxHesitanncy dataset. By incorpo- rating the full annotation information, i.e., confi- dence, secondary label, and annotator agreement, Bayesian soft label method achieves 75.2 FI Macro score on the VaxxHesitancy dataset, improv- ing upon 74.5 from the soft label stance and 2. In addition. Bayesian soft label also has the best confidence alignment (ECE) score.
For both the VaxxHesitancy and CDS datasets, we harmonise the range of confidence scores. In the case of the VaxxHesitancy dataset, this involves converting a confidence score of I, 0.9, and so forth, down to being converted 0.6. Similarly, for the CDS dataset, confidence score 10 is converted to 1, 9 to 0.95, and so on, with also becoming 0.6. 3
However, on the CDS dataset, despite outper- forming hard labels, our Bayesian method fails to improve upon soft labels. Its adjustments soft labels results in fall from 71.1 to 70.4. We be- lieve this is due to the characteristics of the CDS dataset, which has 10 possible classes (vs the 4 of VaxxHesitancy), an increased range of possible confidences (1-9), as well as fewer overall sam- ples. Because there is less annotator overlap, this greater range of options makes it more difficult to accurately estimate annotator reliability on per- confidence score level. This reveals a direction in which our Bayes methodcould be improved, as it is currently reliant on sufficient overlap between an individual annotator and their peers.
5 Results
Soft labels improve classification performance across all of our scenarios. Table presents the results on the VaxxHesitancy and CDS datasets. Soft labels surpass hard labels for both datasets, with and without confidence, well for the test-only and train + test set scenarios. In the case of test-set only, soft labels achieve 68.7 F-1 Macro hard label's 66.9 (VaxxHesitancy) and 67.7 vs 65.2 (CDS). As previously mentioned, the test set comprises of only high-agreement samples, so this indicates that soft labels are beneficial for learn- ing even when there is not a lot of disagreement between annotators and they are relatively certain.
Conclusion
We demonstrate the benefits of using soft labels over traditional hard labels in classification tasks. We propose novel Bayesian method for annota- tion confidence calibration, and efficiently utilising all available annotation information, outperforming other methods for the VaxxHesitancy dataset. The performance improvements offered by soft labels suggests the importance of collecting additional information from annotators during the annotation process, with annotator confidence being particu- larly important.
Combining confidence scores and secondary la- bels generates better soft labels. Using annota- tors' self-reported confidence scores helps to im- prove soft labels, as shown by the Fl and cali- bration improvements between soft labels in the 'with' and 'without confidence' settings (Table 1). Alternative approaches such Dawid Skene able outperform soft labels when confidence scores are not available (71.7 vs 71.0 on VaxxH- esitancy). However, once confidence information is introduced, soft labels significantly improves and outperforms alternatives, achieving 73.7 (from 71.0) on VaxxHesitancy and 71.1 (from 68.2 on CDS).
Acknowledgments
This work has been co-funded by the UK's innovation agency (Innovate UK) grant 10039055 (approved under the Horizon Europe Programme as vera.ai. EU grant agreement 101070093) the European Union under action number 2020-EU-IA-0282 and agreement number
Furthermore, for the VaxxHesitancy dataset, once secondary label information is included, clas- sification performance is further improved from 73.7 to 74.5. This suggests that more consideration should be taken during the data annotation stage
We manually tested different confidencee conversion scales and this conversion yields the best classification performance. See Appendix 3 for an alternative conversion strategy.
INEACCEFICT//2220022886 (EDMO Ire-
"https://ww.vvraal

---

land).3. Ben Wu is supported by an EPSRC Doctoral Training Partnership Grant and Yue Li is supported by a Sheffilld-hhna Scholarships Council PhD Scholarship.
at any time. For quality and ethics reasons, volun- teer annotators were recruited from our university, rather than via Mechanical Turk.
Limitations
References
Our dataset construction introduces secondary la- bels that are not provided by the same annotators as those who created the original dataset, which may not accurately reflect the choices that the original annotators would have made.
Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1892-1915, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
As discussed in the Results section, the CDS dataset was more challenging due number of class labels and number of samples. This caused the Dawid Skene model to perform poorly. This issue may be alleviated using Laplace Smoothing, but we did not explore this due to time constraints.
Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio. and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the Ist Workshop on Benchmarking: Past, Present and Future, pages 15-21, Online. Association for Com- putational Linguistics.
Another important limitation of our Bayesian method is its assumption that an individual annota- tor's level of agreement with their peers a good proxy for their reliability. This leaves it vulner- able to situations where there is high agreement between poor annotators.
Valerio Basile al. 2020. It's the end of the gold stan- dard as we know it. on the impact of pre-aggregation on the evaluation of highly subjective tasks. In CEUR WORKSHOP PROCEEDINGS, volume 2776, pages 31-40. CEUR-WS.
Even though our soft label method is effective, it is not compared against 'traditional' soft labels, which are constructed by aggregating many annota- tor labels per sample, since this would necessitate the large-scale annotation of the two datasets by many users, which we are trying to avoid as our goal is to reduce the amount of annotators and an- notation effort required.
Eyal Beigman and Beata Beigman Klebanov. 2009. Learning with annotation noise. In Proceedings of the Joint Conference of the 47th Anmual Meeting of the ACL and the 4th International Joint Confer- ence Natural Language Processing of the AFNLP, pages 280-287, Suntec, Singapore, Association for Computational Linguistics.
Katherine M Collins, Umang Bhatt, and Adrian Weller. 2022. Eliciting and learning with soft labels from every annotator. In Proceedings of the AAAI Con- ference on Human Computation and Crowdsourcing, volume 10, pages 40-52.
Finally, we observed high variance across folds during vross-validation. We believe this was due to the small size of test set as well as the pres- ence of 'hard-to-llassify" samples in certain folds. These were samples where annotators relied on multimodal information to come to a decision (e.g. viewing a video embedded in the tweet). Our model is only provided with text, and so struggles on these samples.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics. 10:92-110.
Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error- rates using the em algorithm. Journal of the Royal Statistical Society: Series C. (Applied Statistics). 28( 1):20228.
Ethics Statement
Our work has received ethical approval from the Ethics Committee of our university and complies with the research policies of Twitter and follows es- tablished protocols for the data annotation process. The human annotators were recruited and trained following our university's ethics protocols, includ- ing provision of an information sheet, a consent form, and the ability to withdraw from annotation
Anea Dumitrache, Lora Aroyo, and Chris Welty. 2018. Crowdsourcing semantic label propagation in relation classification. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 16-21, Brussels, Belgium. Association for Computa- tional Linguisties.
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy, and Massimo Poesio. 2021.
hhttps://edmohube

---

Beyond black & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 25591-2597, Online. Association for Computational Linguisties.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Joshua C Peterson, Ruairidh M Battleday. Thomas L Griffiths, and Olga Russakovsky. 2019. Human un- certainty makes classification more robust. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617-9626.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint 1503.01503.02 02531.
Dirk Hovy, Taylor King-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computarional Linguisticx: Human Language Technologies, pages 1120-1130 Atlanta, Georgia. Association for Computational Linguistics.
Massimo Poesio, Jon Chamberlain, Silviu Paun, Jun- tao Yu, Alexandra Uma, and Udo Kruschwitz. 2019, A crowdsourced corpus multiple judgments and disagneement on anaphoric interpretation. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1778-1788 Min- neapolis, Minnesota. Association for Computational Linguistics.
Emily Jamison and Iryna Gurevych. 2015, Noise or additional information? leveraging crowdsource an- notation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 291-297, Lishon, Portugal. Association for Compu- tational Linguistics.
Filipe Rodrigues and Francisco Pereira. 2018. Deep learning from crowds. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Paul Rottger. Bertie Vidgen, Dirk Hovy, and Janet Pier- rehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 175-190, Seattle, United States. Association for Computational Linguistics.
Sarah Lichtenstein and Baruch Fischhoff. 1977. Do those who know more also know more about how much they know? Organizational behavior and human performance, 20(2):159-188
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.05100
Victor Sheng. Foster Provost, and Panagiotis Ipeiro- tis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 614-622
Yida Mu, Mali Jin, Charlic Grimshaw, Carolina Scarton, Kalina Bontcheva, and Xingyi Song. 2023. Vaxxhe- sitancy: A dataset for studying hesitancy towards covid-19 vaccination on twitter. arXiv preprint arXiv:2301.06660.
Martin Müller, Marcel Salathé, and Per E. Kummervold. 2023. Covid-twitte twitter-bertt A natural language pro- cessing model to analyse covid-19 content on twitter. Frontiers in Artificial Intelligence,
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast - but is it good? evaluating non-exper annotations for natural lan- guage tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Process- ing, pages 254-263, Honolulu, Hawai. Association for Computational Linguistics.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated proba- bilities using bayesian binning. Proceedings of the AAAI conference on artificial intelligence, volume 29,
Xingyi Song, Johann Petrak, Ye Jiang, Iknoor Singh, Diana Maynard, and Kalina Bontcheva. 2021. Classi- fication aware neural topic model for covid-19 disin- formation categorisation. PloS one, 16(2):e0247086.
Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on nat- ural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131-9145 -13, Online. Association for Computational Linguistics.
Ili Sucholutsky and Matthias Schonlau. 2021. Less than one' -hott learning: Learning n classes from m< n samples. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9739- 9746.
Rebecca J. Passonneau and Bob Carpenter. 2014, The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311- 326.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking

---

the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 281 888--882.
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2020. case for soft loss functions. In Proceedings of the AAAT Conference on Human Computation and Crowdsourcing, volume pages 177-177.
Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385-1470.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- iez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
A Appendix
Table 2 shows information about the composition of the two datasets used in our experiments.
Vexitasat
Train
Traiu
Number items
2,790
431
965
515
Avg unnotations per
1.25
1.38
1.58
Avg confidence score
4.16
4.63
6.73
8.53
Table 2: Summary of the datasets
Table 3 shows the effect of changing the annota- for confidence conversion scale from the one pre- sented in the main section of this paper (9: 1.0 1:0.6) an alternative one (9: 1.0, 8: 0.9 1: 0.1). By comparing between the two columns, we can see that this change leads to drop of FI point for the resulting soft labels. This highlights the importance of selecting good initial conversion scale.

---

Covid Misinfo (5 fold)
1-0.1 conversion
1-0.6 conversion
Uses confidence
Macro
Macro
E'periments test-set only
Hard Label
yes
65.18
(same left) 67.71
Soft Label
yes
66.51
Experiments train test-set
Label (majority vote, lies broken by confidence)
yes
69.02
Soft Label
yes
70.59
71.11
Experiments with confidence
Label
n1
67.13
(same left)
Soft Label
no
68.15
(same as left)
Dawid Soft Label
19.58
(same left)
Experiments with confidence
Hard Label
yes
68.53
69.02
Hard Label (label smoothed 0.1)
yes
68.69
68.75
Hard Label (label smoothed 0.3)
yes
68.
68.44
Soft Label
yes
70.59
71.11
Experiments with bayesian calibration
Soft with Bayesian
yes
69.75
70.40
Table 3: CDS Results with different confidence conversions.


Reviews:
[{"review_id": "3QoTDaWRzC", "reviewer": "Reviewer_N6xf", "summary": "", "questions": "See weakness.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions!\n\n1. \nWe will make our dataset publicly available alongside the paper so that researchers can replicate/build upon our results. The full dataset is currently provided in the supplementary material to maintain anonymity, however we will publish the dataset in an open data repository upon acceptance. \n\n2. \nWe agree that the Methodology section could be made simpler and will work towards improving notation for the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper challenges the conventional practice of discarding annotator disagreement and ambiguity in single-label classification tasks. By proposing the use of soft labels derived from these uncertain annotations, the approach maximizes the utilization of available data. This innovative method acknowledges that labeling ambiguity often stems from complex real-world situations and the lack of contextual cues in the data, making it crucial to extract as much information as possible from annotations.\n\n2. The paper addresses a well-recognized limitation in data annotation and training methodologies for single-label classification tasks. By introducing a novel soft label method, the approach provides a practical solution to tackle the challenges posed by ambiguous labels.\n\n", "reasons_to_reject": "1. Insufficient evidence or validation might undermine the credibility of the approach, raising concerns about the reliability of the results and conclusions. The authors did not provide any source to the dataset which might not be publically available.\n\n2. The Methodology section could be made simpler and simpler by using simpler notations.", "questions_to_authors": "See weakness.", "paper_topic_and_main_contributions": "The paper presents a method for generating improved soft labels by leveraging consensus among annotators to align their confidence levels. \nThe main contributions are:\n1. Authors utilize the Bayesian approach to align individual annotators' confidence scores.\n2. Authors present an innovative dataset designed to support research focused on the application of soft labels in the field of Natural Language Processing.\n3. Paper provides a solution to the issue of generating high-quality soft labels with limited annotator resources.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}, {"review_id": "uxg0iuCyfC", "reviewer": "Reviewer_MMN4", "summary": "", "questions": "A The authors have utilized only two datasets but there is a slew of other datasets which does fit the bill of the requirements, any reason as to why? Datasets such as Social Bias Inference Courpus/GoEMotions/Toxic Ratings.\nB. How is the confidence calculated by the annotators? Is it a self rating?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful questions and feedback! \n\n**B: Confidence scores**\n\nConfidence scores are self-reported by annotators, who were provided with definitions. For example, in VaxxHesitancy, a confidence of 3 is “Pretty confident about the annotation (I’m pretty sure about the annotation, but there might be in high chance that other annotators may label it in a different category.)”\n\nWhen generating soft labels, these are converted to values between 0 and 1, in the manner described in Line 279.  \n\n\n**A: Datasets** \n\nIn order to generate soft labels, we require data that contains individual annotator confidence scores. This was a surprisingly limiting restriction, and we hope that our paper will encourage researchers to gather and publish this information.  \n\nFor the datasets that you mention: \n- GoEmotions and Toxic Comment Classification do not offer annotators a way to express their confidence. (Toxic Comments does have a Rating set which evaluates toxicity based on score. However, this is only a test set, with no training data provided).\n- Social Bias Inference (SBIC) provides annotator confidence information. However, the range of possible confidences is too small to generate meaningful soft labels. (Most classes are only annotated with the options  ‘yes, maybe, no’ or ‘yes, no’. Only one class is annotated with the options ‘yes, probably, probably not, not’).\n- We conducted a preliminary experiment which confirms this. When trained on hard labels, BERT base uncased achieves  83.9 F1 on the SBIC ‘Intent’ class. However, training with soft labels results in a performance drop to 72.5, demonstrating that the annotation-style of this dataset is unsuitable. \n\nAdditionally, while ‘soft label’ datasets do exist (such as ChaosNLI), these are created by aggregating annotator hard labels (and so individual annotator confidence is unavailable).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is a short paper of 4 pages and authors have definitely fit in lot of important context about their methods. \n2. Their findings are stress tested with various baselines that are standard in the domain. ", "reasons_to_reject": "Included in questions for author to justify. ", "questions_to_authors": "A The authors have utilized only two datasets but there is a slew of other datasets which does fit the bill of the requirements, any reason as to why? Datasets such as Social Bias Inference Courpus/GoEMotions/Toxic Ratings.\nB. How is the confidence calculated by the annotators? Is it a self rating?", "paper_topic_and_main_contributions": "The authors introduce an important direction in the domain of modeling annotators. The paper proposes a novel method for generating enhanced soft labels that takes into account the variability in confidence levels among different annotators and the paper investigates how annotator confidence and secondary labels can be used to generate soft labels when the number of annotators is low.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JFAHcP7MrX", "reviewer": "Reviewer_qiyT", "summary": "", "questions": "- Some notation is unclear. E.g., Eq. 1 & 2: is it assumed that we calculate the probability of _estimated_ y (i.e., y hat) in Eq. 1 and the probability of _true_ y in Eq. 2? If so, why in line 134 is there the probability of _estimated_ y (i.e., y hat) again?\n- The experimental setup is unclear to me. As far as I get it from Section 4, 4/5 of the test set was included in the training set. Does it mean the test set was involved in the training? \n\n+ see above.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough feedback and review! \n\n**1. Weak Supervision Baselines**\n\nWe agree that there are many exciting weak supervision techniques that could also be used as benchmarks. However, we highlight that the Majority Vote baseline we use remains very competitive for our task. In the WRENCH paper you reference, the authors recommend ‘Majority Voting’ and ‘MeTaL’ as the two best-performing weak supervision methods for classification. In Table 12, which focuses on textual classification with Bert/Roberta (similar to our task scenario), Majority Voting (both hard and soft) outperforms MeTaL labels (both hard and soft). This suggests that MV is an appropriate baseline. \n\n**2. Dataset size**\n\nDataset size is definitely a limitation of our findings. However, as discussed in our response to Reviewer MMN4, we were limited by the lack of publicly available datasets that offered individual annotator confidence scores. This lack is also what motivated our additional annotation of the VaxxHesitancy set. \nWe hope that if published, our paper will 1) provide a useful dataset resource for the research community, and 2) encourage the community to gather confidence scores during annotation and make them publicly available alongside aggregated labels. \n\n**3. Results analysis** \n\nOur results analysis was constrained by the 4 page short paper limit. If accepted, we will use the additional 1 page to further contextualise and discuss our results (including discussion of variance and experimental setup discussed below). \n\n**4. Notation** \n\nWe agree the notation should be made clearer. The y hat we calculate from Eq 1 is used twice in Eq 2. Once as the prior in the numerator i.e. p(lim); once for calculating the marginal probability in the denominator i.e. p(am).    \n\n**5. Experimental Setup** \n\nWe do use portions of the test set for training. (For each round of cross-validation, we take 4  / 5 folds from the test set for training (along with the entire train set) and evaluate on the remaining fold)\n\nThis is motivated by the fact that the original VaxxHesitancy train-test split distributed samples unevenly: low annotator agreement samples were placed in the train set and high-agreement samples in the test set. For generating soft labels, we want to train with a mixture of high-agreement and low-agreement annotations, so we include a portion of the original test set for training.     \n\n\n**6. Numbers from the Appendix** \n\nWe apologise for this mistake. Here we accidentally reference values from Table 3 (Appendix), which uses the 1-0.1 confidence scale. The main paper uses the 1-0.6 scale, so the values should be 71.1 and 70.4 (Table 1). However, the point demonstrated remains unchanged, as the trend is consistent across confidence scales. (In both cases, on the CDS dataset, soft labels perform better without calibration).\n\n**7.Cross-validation Variance** \n\nHigh variance between folds was due to the presence of ‘hard-to-classify’ samples in certain folds. These were samples where annotators relied on multimodal information to come to a decision (e.g. viewing a video embedded in the tweet). \n\nAs expected, model performance was poor on these samples, leading to a lower overall f1 score for those folds. However, we used the same seed to generate folds across all experiments, so this variation between folds is consistent across different settings. \n\n\n**8. Hyperparameters** \n\nWe did not perform a hyperparameter search for this study. Instead, we relied on our empirical knowledge to determine the hyperparameters. We appreciate the reviewer’s concern regarding this matter. Since this work is still in progress and has been submitted as a short paper, we will take this reviewer feedback into account for our subsequent experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Annotation improvement is an interesting and important task which could be relevant in many domains. \n- The method is well motivated and theoretically sound. \n- The article is well written and easy to follow.", "reasons_to_reject": "- The method is compared towards simple and old baselines. More recent baselines are needed (e.g., weakly supervised baselines, among which there are methods that use soft labels as well, see [1]). \n- All datasets used in the experiments are small. There is no guarantee that the method performs well on larger datasets. \n- More result analysis is needed. \n- Line 335: I couldn't find these numbers in Table 1. \n- Line 361: If the authors observed high variance in runs, this is worth entering in Table 1 and discussing. \n- No hyperparameter search space is provided. Also, not clear how the best values were retrieved given that there is no development set. \n\n[1] Zhang et al. 2021. WRENCH: A Comprehensive Benchmark for Weak Supervision.", "questions_to_authors": "- Some notation is unclear. E.g., Eq. 1 & 2: is it assumed that we calculate the probability of _estimated_ y (i.e., y hat) in Eq. 1 and the probability of _true_ y in Eq. 2? If so, why in line 134 is there the probability of _estimated_ y (i.e., y hat) again?\n- The experimental setup is unclear to me. As far as I get it from Section 4, 4/5 of the test set was included in the training set. Does it mean the test set was involved in the training? \n\n+ see above.", "paper_topic_and_main_contributions": "The paper proposes a new method to improve the annotations by replacing the hard labels with soft ones generated using additional annotator information (e.g., confidence, disagreement, and the secondary label).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Zhang et al. 2021. WRENCH: A Comprehensive Benchmark for Weak Supervision.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Line 151: typo\n- Line 163: typo"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a method for improving soft labels by aligning annotator confidence levels using a Bayesian approach. It also presents a novel dataset for soft label research in NLP.

*Reasons To Accept:*
1. Challenges conventional practice by utilizing uncertain annotations for soft labels, maximizing data utilization.
2. Addresses a recognized limitation in data annotation for single-label classification.
3. Innovative dataset creation supports future research.

*Reasons To Reject:*
1. Lack of dataset source accessibility raises concerns about evidence and validation.
2. Methodology could benefit from simplification for better clarity.

**Summary:**

The paper introduces an innovative method for generating soft labels, addressing a significant issue in the field of data annotation for single-label classification tasks. It challenges conventional practices and provides valuable contributions. However, concerns regarding dataset accessibility, method complexity, and comparison against more recent baselines need attention. The reviewers' confidence levels vary, indicating some uncertainty, but overall, the paper shows promise and potential for acceptance with revisions.
Decision: Accept-Findings