ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation
Authors: Xinpeng Wang
Barbara Plank
Venue: EMNLP 2023 Main (2023)
Keywords: Active Learning
Human Label Variation
Multi-task Learning
TL;DR: In active learning, a model with annotator-specific heads improves performance while reducing the annotation budget by 70%, leveraging group and individual uncertainty based on dataset properties.

Abstract: Label aggregation such as majority voting is commonly used to resolve annotator disagreement in dataset creation. 
However, this may disregard minority values and opinions.
Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation.
Active learning, as an annotation cost-saving strategy, has not been fully explored in the context of learning from disagreement. 
We show that in the active learning setting, a multi-head model performs significantly better than a single-head model in terms of uncertainty estimation.
By designing and evaluating acquisition functions with annotator-specific heads on two datasets, we show that group-level entropy works generally well on both datasets. 
Importantly, it achieves performance in terms of both prediction and uncertainty estimation comparable to full-scale training from disagreement, while saving 70\% of the annotation budget.

ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation
Xinpeng Wang and Barbara Plank
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany Munich Center for Machine Learning (MCML), Munich, Germany (xinpeng, bplank]ecis. 1mu.de
Abstract
New Annotation
Add
Label aggregation such majority voting is commonly used to resolve annotator disagree- in dataset creation. However, this may disregard minority values and opinions. Recent studies indicate that learning from individual annotations outperforms learning from aggre- gated labels, though they require consider- able amount of annotation. Active learning, an annotation cost-saving strategy, has not been fully explored in the context of learning from disagreement. We show that in the active learning setting, multi-head model performs significantly better than single-head model in terms of uncertainty estimation. By design- ing and evaluating acquisition functions with menotatto-sppcifcc heads on two datasets, we show that group-level entropy works generally well on both datasets. Importantly, it achieves performance in terms of both prediction and uncertainty estimation comparable to full-scale training from disagreement, while saving 70% of the annotation budget.
Labelled
Data
Who ask?
Train
Unlabelled
Muli-head Model
Figure For each sample that needs he labelled, our model actively selects specific annotators for annota- tions learn from the label variation.
2020; Fornaciari al., 2021) and learning from the hard label individual annotators (Cohn and Specia, 2013; Rodrigues and Pereira, 2018; Davani et al.. 2022).
In recent work, Davani al. (2022) shows that modelling the individual annotators by adding annotator-specife classification heads in multi- task setup outperforms the traditional approach that learns from a majority vote. However, training such model needs huge amount of data with multiple annotations to model the opinions and beliefs of the individual annotators.
1 Introduction
An important aspect of creating a dataset is asking for multiple annotations and aggregating them in or- der to derive single ground truth label. Aggregat- ing annotations, however, implies single golden ground truth, which is not applicable to many sub- jective tasks such as hate speech detection (Oves- dotter Alm, 2011). human's 's judgement on sub- jective tasks can be influenced by their perspective and beliefs or cultural background (aanndWWaass (Waseem et al., 2021; Sap et al., 2022). When addressing disagree- ment in annotation, aggregating them by majority vote could result in the viewpoints of the minority being overlooked (Suresh and Guttag, 2019).
On another line, Active Learning (AL) is frame- work that allows learning from limited labelled data by querying the data to be annotated. this paper, we propose to take the best of both worlds: active learning and human label variation, to mitigate the high cost of the annotation budget needed for train- ing the model. In particular, active learning setting, where the multi-head model actively selects the annotator and the sample be labelled. Our results show this effectively reduces annotation costs while at the same time allowing for modelling individual perspectives.
propose novel
In order to address this issue, many works have been proposed to directly learn from the annotation disagreements in subjective tasks. There are two major approaches to achieving that: learning from the soft label (Peterson et al., 2019; Uma et al.,

---

Key Findings We made several key observations:
tor, following Davani et al., 2022. In the fine-tuning stage, annotations are fed the corresponding an- notator heads, adding their losses to the overall loss. During testing, the F1 score is calculated by com- paring the majority votes of the annotator-specific heads with the majority votes of the annotations.
The multi-head model works significantly bet- ter than the single-head model on uncertainty estimation.
The use of group-level entropy is generally recommended. Individual-level entropy meth- ods perform differently depending on the dataset properties.
3.1 Multi-head Acquisition Functions
We study five acquisition functions for the multi- head model. Since our model learns directly from the annotation, wc care about which annotator should give the label. So query the instance- annotation pair (24,, 2/¹) with its annotator ID a. In this way, our data duplicated by the number of annotations available.
The multi-head model achieves performance comparable to full-scale training with only around 30% annotation budget.
2 Related Work
2.1 Learning from Disagreement
There growing body of work that studies irrec- oncilable differences between annotations (Plank et al., 2014; Aroyo and Welty, 2015; Pavlick and Kwiatkowski, 2019; Uma et al., 2021). One line of research aims at resolving the variation by ag- gregation or filtering (Reidsma and Carletta, 2008;
Random Sampling (Rand.) We conduct random sampling as a baseline acquisition method where we randomly sample K (data, annotation, anno- tator ID) pairs from the unlabeled data pool U at each active learning iteration.
Individual-level Entropy (Indi.) Intuitively, the annotator-specific heads model the corre- sponding annotators. Therefore, we can cal- culate the entropy of the classification head to measure the specific annotator's uncertainty. Given the logits 2a = =[ =A] of the head a, the entropy is calculated as following: Fpndi(" [x) = - -' (x) log(p? (x)), where pf (x) 22 Then we choose the (in- stance, annotator) pair with the highest entropy: argmaX,eU inds (p*|z), where U denotes the unlabeled set and A denotes the annotator pool. We compute entropy only for the remaining annotators who have not provided annotations for the instance.
Beigman Klcbanov et al., 2008; Hovy et al., 2013; Gordon et al., 2021). Another line of research tries embrace the variance by directly learning from the raw annotations (Rodrigues and Pereira, 2018; Peterson et al., 2019; Fornaciari et al., 2021; 1; Da- vani et al., 2022), which is the focus of our paper.
2.2 Active Learning
active lcarning, many different methods for se- lecting data have been proposed to save annotation cost, such as uncertainty sampling (Lewis. 1995) based on entropy (Dagan and Engelson, 1995) or approximate Bayesian inference (Gal and Ghahra- mani, 2016). Other approaches focus the diver- sity and informativeness of the sampled data (Sener and Savarese, 2017; Gissin and Shalev-Shh Shwarv-SSwarrz. 2019; Zhang and Plank, 2021). Herde et al. (2021) proposed a probabilistic active learning framework in a multi-annotator setting, where the disagree- ment is attributed to errors. Recent work by Baum- ler et al. (2023) accepted the disagreement in the active learning setting, and they showed improve- ment the passive learning setting using single- head model. Our work shows the advantage of the melti-head model and compares with traditional single-head active learning methods.
Group-level Entropy (Group) Instead of look- ing the individual's uncertainty, we can also query the data by considering the group-level un- certainty. One way to represent the uncertainty of the group on a sample is to calculate the en- tropy based on the aggregate of each annotator- specific head's output. Therefore, we normal- ize and sum the logits of each head at the group level: Zgroup
[E1,-.., 2n] Σ# zhorm and calculate the group-level entropy as follows: H (x)
Σi= Pc(x)) log(p<())), where pi(z) = softmax(2 We then query the data with the highest uncertainty.
3 Method
Multi-head Model We use a multi-heas model where each head corresponds to one unique annota-
Vote Variance (Vote) Another way to measure
the uncertainty among a group is by measuring

---

HS-Brexit
HS-Brexit
GHC
Pearson Correlation
0.6
0.4
0.4
0.4
0.3
Ent.
Ent.
0.2
DAL
BALD
BALD
BALD
BALD
Multi.
Rand.
0.0
Rand.
0.1
0.2
Rand.
Rounds
Rounds
Rounds
0.7
Mm
Pearson Correlation
0.6
0.4
0.5
0.4
0.4
0.2
Enit.
0.0
DALL
0.0
DAL
0.2
Rond. Multi.
0.2
0.1
Multi.
Figure 2: Comparsion of the Multi-head model and Single-Majority (upper row) and Single-Annotation (bottom row). Results are averaged over 4 runs. All the methods have the same annotation cost the seed dataset and the queried batch each round.
the variance of the votes. Given the prediction / of classification head h. we calculate the vote variance: Var = н (r
(Rand.), Max-Entropy (Ent.; Dagan and Engel- son, 1995), Bayesian Active Learning by Disagree- ment (BALD; Houlsby et al., 2011) and Discrimi- native Active Learning (DAL; Gissin and Shalev- Shwartz, 2019). We compare them with the multi- head approach with random sampling which has average performance among the five multi-hest head acquisition methods we investigated.
(t 12)2, where = 1 E E This approach can be applied to bi- nary classification or regression problems.
Mixture of Group and Indi. Entropy (Mix.)
We also consider a variant which combines the group and individual-level entropy by simply adding the two: Hmiz Hinds + H ggroup.
Two different single-head model approaches considered: Learning from the Majority Vote (Single- and Learning from Raw annota- tions
4 Experiments
the first setting, all annotators' annotations are queried, and the major- ity vote is used train the model. In the second setting, we train the model with individual annota- tions without aggregation, following the repeated labelling approach by Sheng et al. (2008).
Dataset We selected two distinct hate speech datasets for our experiments: Hate Speech on Brexit (HS-Brexit) (Akhtar et al., 2021) and Gab Hate Corpus (GHC) (Kennedy et al., 2022). We split the raw annotation dataset according to the split of the aggregated version dataset provided. The HS-Brexit dataset includes 1,120 English tweets relating to Brexit and immigration, where total of six individuals were involved in anno- tating each tweet. As each tweet contains all six annotations, we refer HS-Brexit as densely anno- tated. GHC, 27.665 social-media posts were col- lected from the public corpus of Gab.com (Gaffney, 2018). From a set of 18 annotators, each instance gets at least three annotations. Therefore, GHC is sparsely annotated. Both datasets contain binary labels y [0, 1] and have almost the same posi- tive/negative raw annotation ratio (0.15).
Experimental Setup We follow the setup of Da- vani et (2022) for modelling and evaluation. We initialize the layers before the heads with a BERT- base model (Devlin et al., 2019). To balance train- ing data, we do oversampling following Kennedy et al. (2022). Moreover, we use class weights on the loss function for multi-head model training, which makes it more stable. It is not used for the single-heat head model as degrades performance.
To evaluate the model, first report the Fl score against the majority vote. Secondly, we also compute individual F1 scores, measuring annotator- specific heads against annotator labels. Thirdly and importantly, we are interested to gauge how well the model can predict the data uncertainty
Single-head Model Baselines We implement four acquisition methods for single-head model active learning for comparison: Random sampling

---

HS- Brexxtt
HS-Brexit
Breeitt
Pearson Correaation
0.30
0.0
F Indit
0.4
6.24
0.3
leal. Mix
Indi
Indi.
0.2
Mix
Mic
Group
Group
0.1
8.0
Relands
Rounnds
Rounds
0.5
Correlat
0.40
0.55
0.4-
0.36
Iedi:
0.3-
6.45
Rand.
0.2
0.20
Mix
Greup
Group
0.1
Votte
Vatee
20
10
20
Figure 3: Comparison of multi-heaa head acquisition functions. Results are averaged over 4 runs. Group-level entropy method (Group) performs generally well both datasets all three metrics. Individual-level uncertainty (Indi.) only performs well on GHC.
by calculating the Pearson correlation between the model's uncertainty and the annotation disagree- ment measured by the variance of the annotations on the same instance. For the single-head model, we use Prediction Softmax Probability proposed by Hendrycks and Gimpel (2017) as the uncer- tainty estimation of the model. For the multi-head model, follow Davani et al. (2022) and calculate the variance of the prediction of the heads as the model's uncertainty.
from: Label Diversity First and Sample Diversity First. In Label Diversity First, we query from all the available annotators to prioritize the label diver- sity single sample. In Sample Diversity Fisrat approach, we only randomly choose one of the an- notators for annotation. Given the same annotation budget for each annotation round, Label Diversity First would query fewer samples but anno- tations than Sample Diversity First approach. In our preliminary result, Label Diversity First shows stronger performance in general. Therefore, we adopt this approach for the following experiments.
5 Result
Single-head Multi-head Model Figure shows the comparison of the multi-head model and the single-head during the active learning process the upper row, we compare the multi-head ap- proach with single-majority approach on majority Fl score and uncertainty estimation. In terms of predicting the majority vote, the multi-head model performs on par with the best-performing single- head method on both datasets, such BALD. For uncertainty estimation measured against annotator disagreement, the multi-head model outperforms the single-head model by large margin.
Comparison of Multi-head acquisition functions To compare different strategies to query for anno- tations, we compare the five proposed acquisition functions from Section 3.1 in Fig 3. Group per- forms generally well on both datasets. We also trend here that HS Brexit favours acquisition function based on group-level uncertainty (Vote), while individual-levw uncertainty (Indi.) works better on GHC dataset. For HS-Brexit, Group is the best pessooeefonmng method based on the major- ity Fl score. When evaluated on raw annotation (F1 indi. score). both vote variant and group-level entropy perform well. For uncertainty estimation, random sampling is slightly better than group-level entropy approach. On the GHC dataset, both Indi. and Group perform well uncertainty estimation and raw annotation prediction. However, we don't 't see an obvious difference between all the acquisi- tion functions on the majority vote Fl score.
We have the same observation when comparing with single-ane iiglle-nmotation model, shown in the bottom row. Therefore, we recommend using multi-head model in subjective task where humans may dis- agree and uncertainty estimation is important.
Label Diversity vs. Sample Diversity When it comes group-level uncertainty based acquisi- tion functions (Group and Vote), we tested two approaches to determine which annotator to query
Annotation Cost In terms of saving annotation
cost, we see that the Fl score slowly goes into a

---

plateau after around 25 rounds on both datasets in Fig 3, which is around 30% usage of the overall dataset (both datasets are fully labelled at around 90 rounds). For example, Vote achieves the majority Fl score of 52.3, which is 94% of the performance (55.8) of the full-scale training (round 90).
References
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? perspective-aware mod- els to identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106.1586.
Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. A/ Magazine, 36(1):15-24.
6 Conclusion
We presented an active learning framework that em- braces human label variation by modelling the an- notator with annotator-specific classification heads, which are used to estimate the uncertainty at the individual annotator level and the group level. We first showed that a multi-head model is a better choice over a single-head model in the active learn- ing setting, especially for uncertainty estimation. We then designed and tested five acquisition func- tions for the annotator-heads model on two datasets. We found that group-level entropy works generally well on both datasets and is recommended. Depend- ing on the dataset properties, the individual-level entropy method performs differently.
Connor Baumler, Anna Sotnikova, and Hal Daumé III. 2023. Which examples should be multiply amno- tated? active learning when annotators may disagree. In Findings of the Association for Computational Lin- guisties: ACL 2023, pages 10352- -1337, Toronto, Canada. Association for Computational Linguistics.
Beata Beigman Klebanov, Eyal Beigman, and Daniel Diermeier. 2008. Analyzing disagreements. In Col- ing 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics, pages 2-7, Manchester, UK. Coling 2008 Organizing Commit- tec.
Trevor Cohn and Lucia Specia. 2013. Modelling an- notator bias with multi-task Gaussian processes: An application to machine translation quality estimation. In Proceedings of the Sist Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 32 42, Sofia, Bulgaria. Asso- ciation for Computational Linguistics.
Limitations
The multi-heate approach is only viable when the annotator IDs are available during the active learn- ing process since we need to ask the specific an- notator for labelling. Furthermore, the annotators should remain available for period of time in or- der provide enough annotations to be modelled by the specific head successfully. Note that we here simulate the AL setup. The multi-head approach is good at estimating the uncertainty based on the annotators it trains on, however, whether the uncer- tainty can still align with yet another pool of people is still an open question.
Ido Dagan and Sean P Engelson. 1995. Committee- based sampling for training probabilistic classifiers. In Machine Learning Proceedings 1995, pages 150- 157. Elsevier.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Further analysis is needed to understand why GHC and HS-Brexit favour different acquisition functions. Besides the difference between dense and sparse annotation, factors such as diversity of the topics covered and annotator-specific speciffcc annota- tion statics are also important, which we leave as future work.
Tommaso Forncciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2591-2597, Online. Association for Computational Linguistics.
Acknowledgements
We thank the anonymous reviewers for their feed- back. This research is supported by the ERC Con- solidator Grant DIALECT 101043235.

---

Gavin Gaffney Gaffney, 2018. Pushshift gab corpus.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguis- rics, 7:677-694.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncer- tainty in deep learning. In international conference on machine learning, pages 1050-1059. PML.R.
Joshua C Peterson, Ruairidh M Battleday: Thomas I. Griffiths, and Olga Russakovsky. 2019. Human un- certainty makes classification more robust. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617-9626.
Daniel Gissin and Shai Shalev-Shwartz. 2019. Dis- criminative active learning. arXiv preprint
arXiv:1907.06347.
Barbara Plank, Dirk Hovy, and Anders Søgaard. 20 14. Linguistically debatable or just plain wrong? Proceedings of the 52nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 507-511.
Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tat- sunori Hashimoto, and Michael S Bernstein. 2021. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHII Conference on Human Factors in Computing Systems, pages 1-14.
Dennis Reidsma and Jean Carletta. 2008. Squibs: Reli- ability measurement without limits. Computational Linguistics, 34(3):319-326.
Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution exam-
Filipe Rodrigues and Francisco Pereira. 2018. Deep learning from crowds. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
ples in neural networks. In International Conference on Learning Representations.
Maarten Sap, Swabha Swayamdipta, Laura Vianna. Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884-5906, Seattle, United States. Association for Computational Linguistics.
Marek Herde, Daniel Kottke, Demis Huseljic, and Bern- hard Sick. 2021. Multi-annotator probabilistic ac- tive learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 10281-10288 IEEE.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. 2011. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745.
Ozan Sener and Silvio Savarese. 2017. Active learn- ing for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 00489.
Dirk Hovy, Taylor Berg- Kirkpatrk,,,,,, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages |/20-1 130, Atlanta, Georgia. Association for Computational Linguistics.
Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008, Get another label? improving data quality and data mining using multiple, noisy label- ers. In Proceedings of the 14th ACM SIGKDD Inter- national Conference Data Mining, KDD '08, page 614-622, New York, NY, USA. Association for Computing Machinery.
Knowledge Discovery and
Brendan Kennedy,
Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo- Wightman, Elaine Gonzalez, et al. 2022. Introducing the gab hate corpus: defining and applying hate-based rhetoric to social media posts at scale. Language Resources and Evaluation, pages 1-30.
Harini Suresh and John V Guttag, 2019. A framework for understanding unintended consequences of ma- chine learning. arXiv preprint 1901. 2(8).
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2020. A case for soft loss functions. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 173-177
David D Lewis. 1995. A sequential algorithm for train- ing text classifiers: Corrigendum and additional data. In Acm Sigir Forum, volume 29, pages 13-19, ACM New York, NY, USA.
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 12:1385-1470.
Cecilia Ovesdotter Alm. 2011. Subjective natural lan- guage problems: Motivations, applications, charac- terizations, and implications.
Proceedings of the 49th Annual Meeting of the Association for Compu- tational Linguistics: Human Language Technologies. pages 107-112, Portland, Oregon, USA. Association for Computational Linguistics.
Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021. Disembodied machine learning: On the illusion of objectivity nlp. arXiv preprimi 2rX.

---

Mike Zhang and Barbara Plank. 2021. Cartography ac- tive learning. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 395- 406, Punta Cana, Dominican Republic. Association for Computational Linguistics.
A Data Preparation
In contrast to the traditional active learning ap- proach, which only selects instances from the unla- beled data pool, our approach also considers which annotator should give the annotation since the anno- tation should be fed to the annotator-specifff heads, To address this issue, reform the dataset by split- ting one instance with N annotations into N (data, annotation) pairs with its annotator ID. When lecting the batch from the populated data pool, the (data, annotation) pair given to the classification head corresponding to its annotator ID.
B Active Learning Setting
Since our work focus on raw annotation, we the seed dataset size and the query batch size based on the annotation budget. We set the annotation budget for HS-Brexit as (60, 60) for both seed data size and query data size respectively. For GHC, we set it as (200, 200).
C Training Setup
We list our training parameter Table 1. We halve the learning rate when the Fl score decreases at evaluation.
Parameter
Value
Weight Decay
0.01
Optimizer
A
Adam
le-6
Adam 31
0.9
Adam B2
0.99
Gradient Clipping
0.0
Peak Learning Rate
2c-5
Batch size
32
Early stopping
5
Table Parameter used for training.


Reviews:
[{"review_id": "0Ligz4lBgF", "reviewer": "Reviewer_LA7Z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback and your acknowledgement of the novelty of our method. \nWe appreciate the suggestion to conduct additional experiments on other language models (LLMs) to further strengthen the conclusions of our study. \nWe follow the model choice (BERT) in the prior works in active learning [1] and learning from disagreement [2]. \nGiven that the BERT/RoBERTa are the most common model choices for text classification and the similar architecture choice between BERT and RoBERTa, we only test our method on BERT. \nWe will include RoBERTa model result in the camera-ready version\n\n[1]: [Active Learning for BERT: An Empirical Study](https://aclanthology.org/2020.emnlp-main.638) (Ein-Dor et al., EMNLP 2020)\n\n[2] [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://aclanthology.org/2022.tacl-1.6) (Mostafazadeh Davani et al., TACL 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is an interesting and novel direction in active learning. The authors conduct comprehensive evaluation and analysis.", "reasons_to_reject": "More experiments on other LLMs to justify the conclusions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores to incorporate annotator disagreement in the active learning process. Based on the experiments on two subjective NLP tasks, it shows that modelling hard labels from each annotator through a multi-head layer could improve the performance with a smaller annotation budget.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WXs1q6xQac", "reviewer": "Reviewer_vzgW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The paper ultimately details a report that combines tools already presented in existing work. The acquisition functions, the active learning approaches as well as the idea to use annotator-specific heads to model disagreement are all present in the existing literature.**\n\nA1: No, we would like to clarify that the acquisition functions we proposed are new. Traditional active learning acquisition functions only consider the output from a single classification head, to which we compare, please see the next answer. Our method explores how to connect active learning and annotator-specific head modelling and design new active learning acquisition functions for that. Therefore, we propose different acquisition functions in two main categories: **group level** and **individual level**. This enables the AL algorithm to specifically query the individual annotator's annotation which previous AL methods cannot achieve. \n\n**Q2: Missing baselines (just AL without head-per-annotator setup)**\n\nA2: We do have included this baseline for comparison. In Figure 2, we compare the multi-head model approach (Multi.) with the single-head  model (without head-per-annotator setup) with 4 different acquisition functions (Ent., DAL, BALD, Rand.). In the upper row, we compare to a  single-head model that learns only from the majority vote. In the bottom row, we compare the multi-head model with the single-head model that leanrs from the raw annotations. \n\nWe show that **explicitly modelling each annotator** substantially benefits the learning in terms of majority  vote F1 score, uncertainty estimation and individual annotator modelling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper does show that the active learning scenario does benefit from the head-per-annotator setup shown to work for regular fine-tuning. \n\n2. The authors do conduct experiments w.r.t different settings(e.g. dense vs sparse multi-annotations, different acquisition functions, different AL methods, etc)", "reasons_to_reject": "1. The paper ultimately details a report that combines tools already presented in existing work. The acquisition functions, the active learning approaches as well as the idea to use annotator-specific heads to model disagreement are all present in the existing literature. Hence, the paper ultimately presents a new application of existing tools.\n\n2. I also feel that the paper in its current state is missing important baselines (such as just AL without head-per -annotator setup).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper seeks to improve the model performance when trained on human-annotated data, with multiple annotators per sample. It builds on existing work that shows that exposing the model to annotator uncertainty (instead of aggregated labels) improves downstream performance.\n\nThe authors show that combining this setup with an active learning method gets the best of both worlds - a model that is capable of taking uncertainty into account and thus performs better, as well as being sample efficient.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u8PsbMkBCN", "reviewer": "Reviewer_9qCH", "summary": "", "questions": "A) You claim your method can select which annotators to query. How do you do that if you are using the group-level entropy? Or are you able to select annotators only for individual-level entropy?\n\nB) How do you compare different setups? I see you use the F1 score, but I wonder if you mention at which step is F1 score calculated. I guess it is at the end of the AL loop, but could you confirm that, and if you mentioned that in the text, please correct me.\n\nC) How do you calculate individual F1 scores for the Single-Annotation model? Does that model have access to the information about the label's annotator? If not, that could explain a rather large performance gap between your methods on the Multi-Head model and methods on the Single-Annotation model, as seen in Figure 2. Please comment on why you used the individual F1 score to compare Multi-head and Single-Annotation models in Figure 2. It does not seem fair to use scoring that considers the label's annotators when one model is explicitly trained to model annotators, and another model has no information about the annotators but is just trained on multiple labels for the same instance. Or does the Single-Annotation model have access to the information about the label's annotators during training, and if yes, how?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:  It would be better to make it clear that entropy depends on a datapoint $H\\left(p^a \\mid x\\right)$**\n\nA1: Thanks for this notation suggestion. We will add this in the camera-ready version.\n\n**Q2: Adding up the logits from different heads doesn’t make a lot of sense since they can be on different scales.  A simple fix could be to normalize the logits of different heads before adding them up.**\n\nA2: We appreciate this valuable suggestion. We conducted additional experiments on summing the normalised logits as the group entropy. \nWe normalize the logits vector of each annotator heads before adding them up.\nMajority vote F1 score and Pearson Correlation results on HS-BREXIT are listed below.\n\n__Majority vote F1__\n| AL iteration | 0 | 10 | 20 | 30 | 40 | 50 | 60 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Group_Normalized | 0.10 | **0.55** | **0.61** | 0.54 | **0.65** | 0.63 | **0.67** |\n| Group | 0.11 | 0.21 | 0.41 | 0.48 | 0.54 | 0.55 | 0.53 |\n| Random | **0.12** | 0.34 | 0.47 | **0.62** | 0.61 | **0.65** | **0.67** |\n| Vote | **0.12** | 0.44 | 0.54 | 0.53 | 0.64 | 0.59 | 0.60 |\n\n__Pearson's Correlation__\n| AL iteration | 0 | 10 | 20 | 30 | 40 | 50 | 60 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Group_Normalized | **0.07** | 0.37 | 0.53 | 0.52 | 0.60 | 0.59 | 0.61 |\n| Group | 0.02 | 0.34 | 0.54 | 0.53 | 0.46 | 0.55 | 0.50 |\n| Random | -0.01 | **0.48** | **0.59** | **0.59** | **0.62** | **0.61** | **0.64** |\n| Vote | 0.02 | 0.35 | 0.51 | 0.51 | 0.51 | 0.56 | 0.56 | \n\n__Individual F1__\n| AL iteration | 0 | 10 | 20 | 30 | 40 | 50 | 60 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Group_Normalized | 0.20 | **0.30** | **0.30** | **0.31** | 0.30 | **0.31** | **0.30** |\n| Group | **0.21** | 0.27 | **0.30** | 0.30 | 0.30 | 0.28 | 0.29 |\n| Random | **0.21** | 0.28 | 0.28 | 0.29 | 0.28 | 0.28 | 0.27 |\n| Vote | 0.20 | 0.28 | 0.29 | 0.30 | 0.30 | 0.29 | 0.28 |\n\nWe do observe improved performance of using the normalised logits compared to the raw logits in all the three metrics . In terms of majority F1 score, it performs the best among all the baselines at the early stage and on par with the Rand. baseline at later stage. In terms of uncertainty estimation, Rand. is still the best performing method. When measured against individual F1, normolized version Group entropy performs the best.  We will include the results as well as the GHS dataset results in the final version.\n\n**Q3:  Vote variance only works when there are only two classes.**\n\nA3: This is a very good observation. We will mention this restrict in the paper since we only study hate speech detection task which only requires two classes. Other metrics should be considered such as disagreement score when there are more classes. \n\n**Q4: How to select  annotators when using group-level entropy?**\n\nA4: We select all the available annotators. In our preliminary experiments, we tried two approaches when using group-level entropy: \n1. select all the annotators 2. randomly select one annotator. \n\nIn our preliminary results, the first approach performs better. \nOur methods only query the specific annotators when using the individual-level entropy as an acquisition function. \nOur results show that individual-level entropy benefits more in GHC dataset than in HS-BREXIT. \nWe will make this point clear in the final paper. \n\n\n**Q5: At which step is the F1 score calculated？ Is it at the end of the AL loop?**\n\nWe take the best F1 score during each AL loop. \n\n**Q6: How do you calculate individual F1 score for the single-annotation model?  Does the single-annotation model have access to the information about the label’s annotators?**\n\nThe single-annotation model does not have access to the annotators id during the training. \nThe goal of the experiment is to confirm **how large the gap is** between explicit annotator modelling (annotator-heads) and implicit modelling (training on raw annotation). \n\nPrevious work [1] only compared the multi-head model with a single-head model that was trained on the majority vote on the majority vote F1 metrics (where the multi-head model has access to annotator id while the single-head model does not) . Therefore, we want to know if a single-head model trained on the raw annotation can match the performance of the multi-head model on individual F1 metric. \n\nWe think this is an important baseline to compare to (Reviewer 2 explicitly asks for this comparison), which tells us how much benefits we can get by explicitly modelling the annotators. \n\n[1] [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://aclanthology.org/2022.tacl-1.6) (Mostafazadeh Davani et al., TACL 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea is interesting. Letting the model choose not just what instances to label but also what annotators should label them is an exciting approach to active learning.\n- Creating models that are not only optimized to model the annotators' opinions but are also good at modeling annotators' confidence is an important research direction.", "reasons_to_reject": "- In the paragraphs Individual-level entropy and Group-level entropy, authors represent entropy with $H$. It would be better to make it clear that entropy depends on a datapoint, e.g., $H(p^a|x)$ , so that formulas like $\\text{argmax}_{x \\in U, a \\in A} H^a$ became more understandable.\n- The authors' formulation of Group-level entropy adds up the logits of all the heads to calculate the entropy. However, since each head uses its own projection to calculate logits, it does not make a lot of sense to add them since they can be on a different scale. For example, if one head's logits are of a much higher magnitude than the logits of the other heads, the larger logits could take over, and the Group-level entropy would equal the entropy of that one head. It would be safer to calculate the joint entropy of the posterior probabilities of the model. If you could assume independence of the heads, it is just the sum of the entropies of each of the heads, although I wonder if it is a valid assumption since we expect to see some agreement between annotators. If independence can not be assumed, it is more challenging to calculate joint entropy since one needs to calculate joint distributions. The problem is that the approach has no theoretical grounding and can be easily overwhelmed and overtaken by a single head. A simple fix could be to normalize the logits of different heads before adding them up.\n- In the paragraph Vote variance, authors explain how they calculate vote variance from the predictions of the classification heads. But the given formulation makes sense only for binary classification, i.e., $y^h \\in $ {$0,1$}, because calculating the average vote $\\mu$ with more than two classes wouldn't make sense since you are averaging categorical variables. Since all other methods are more general, i.e., work with more classes than Vote variance, it would be important to mention it.", "questions_to_authors": "A) You claim your method can select which annotators to query. How do you do that if you are using the group-level entropy? Or are you able to select annotators only for individual-level entropy?\n\nB) How do you compare different setups? I see you use the F1 score, but I wonder if you mention at which step is F1 score calculated. I guess it is at the end of the AL loop, but could you confirm that, and if you mentioned that in the text, please correct me.\n\nC) How do you calculate individual F1 scores for the Single-Annotation model? Does that model have access to the information about the label's annotator? If not, that could explain a rather large performance gap between your methods on the Multi-Head model and methods on the Single-Annotation model, as seen in Figure 2. Please comment on why you used the individual F1 score to compare Multi-head and Single-Annotation models in Figure 2. It does not seem fair to use scoring that considers the label's annotators when one model is explicitly trained to model annotators, and another model has no information about the annotators but is just trained on multiple labels for the same instance. Or does the Single-Annotation model have access to the information about the label's annotators during training, and if yes, how?", "paper_topic_and_main_contributions": "The authors propose using a multi-head model, in which each head models one annotator for active learning. They also present multiple AL methods that should go well with their multi-head model. Authors show their methods are competitive against other popular methods for single-head models. Their model is also better at modeling individual opinions. Lastly, they use the Pearson correlation between the model's uncertainty and the disagreement between annotators and show that their model's uncertainty is by far better at modeling human uncertainty than the single-head models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Title case the title.\n\n- Line 104 – use \\citet{}\n\n- Line 259 – variant -> variance"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes an active learning approach in the case of annotator disagreement. It takes the multihead approach that's now popular in passive learning, and experiment with both known and novel (eg Vote/Mix) data selection procedures. They find that active learning is effective, across three datasets.

On the positive side, this is a timely paper addressing an important problem, and the proposed approach is sensible and works reasonably well.

On the negative side, it is unfortunate that there is not a clear winner between group-level and individual-level annotation.
Decision: Accept-Main