Confidence-based Ensembling of Perspective-aware Models
Authors: Silvia Casola
Soda Marem Lo
Valerio Basile
Simona Frenda
Alessandra Teresa Cignarella
Viviana Patti
Cristina Bosco
Venue: EMNLP 2023 Main (2023)
Keywords: Data perspectivism
irony
hate speech
confidence
ensemble
TL;DR: We propose a technique to ensemble models that encode different perspectives, using their individual confidence, and it works for irony and hate speech classification.

Abstract: Research in the field of NLP has recently focused on the variability that people show in selecting labels when performing an annotation task. Exploiting disagreements in annotations has been shown to offer advantages for accurate modelling and fair evaluation. In this paper, we propose a strongly perspectivist model for supervised classification of natural language utterances. Our approach combines the predictions of several perspective-aware models using key information of their individual confidence to capture the subjectivity encoded in the annotation of linguistic phenomena.
We validate our method through experiments on two case studies, irony and hate speech detection, in in-domain and cross-domain settings. The results show that confidence-based ensembling of perspective-aware models seems beneficial for classification performance in all scenarios.  In addition, we demonstrate the effectiveness of our method with automatically extracted perspectives from annotations when the annotators' metadata are not available.

Confidence-based Ensembling of Perspective-aware Models
Silvia Casola', Soda Marem Lo¹, Valerio Basile', Simona Frenda" 1.2 Alessandra Teresa Cignarella Viviana Patti and Cristina Bosco! University of Turin, Italy Pacqua-tech, Turin, Italy
Abstract
points of view on observed phenomena (Abercrom-
bie et al., 2022; Mastromattei al., 2022).
Research in the field of NLP has recently fo- cused on the variability that people show in selecting labels when performing an annotation task. Exploiting disagreements in annotations has been shown to offer advantages for accu- modelling and fair evaluation. In this paper, we propose strongly perspectivist model for supervised classification of natural language utterances. Our approach combines the pre- dictions of several perspective-awan models using key information of their individual confi- dence to capture the subjectivity encoded in the annotation of linguistic phenomena. We vali- date our method through experiments on two case studies, irony and hate speech detection, in-domain and cross-domann settings. The results show that confidence- based ensembling of perspective-awe aware models seems beneficial for classification performance in all scenarios. In addition, we demonstrate the effectiveness of our method with automatically extracted per- spectives from annotations when the annotators' metadata not available.
This paper proposes a strongly perspectivist model for the supervised classification of natural language utterances. While method is agnostic to the specific phenomenon that is the subject of classification, we expect it to work well when the focus is on pragmatic language phenomena. The reason for this expectation lies in the subjectivity of the annotation process for this level of analysis, which our model aims to model and exploit for a better classification.
To prove it, we tested this methodology two largely explored pragmatic phenomena NLP: irony and hate speech detection. Irony is a figura- tive language device that conveys the opposite or an extension of the literal meaning; hate speech pro- duces the incitement of hate and violence against specific target (individuals or entire communi- ties) the reinforcement of negative stereotypes (Sanguinetti et al., 2018). All experiments are on datasets in English.
To test the robustness of the proposed perspec- tivist method, we performed in-domain and cross- domain experiments, i.e., testing models fine-tuned on a dataset on data from the same set and from different sets.
1 Introduction
Human label variability (Plank, 2022), traditionally considered a source of noise in annotated data, has recently become the subject of a series of research works in Natural Language Processing. Leveraging disagreement in annotation has been found bene- ficial towards more accurate modelling of natural language phenomena (Uma et al., 2020a) and fairer evaluation (Basile et al., 2021).
We took inspiration from the observations in (Frenda et al., 2023b) about the higher confidence exhibited by perspective-aware models trained on longitudinal subsets of the disaggregated data with respect to models trained aggregated data. Start- ing from this observation, we propose a method to ensemble perspective-aware models that take model confidence into account.
These observations are particularly relevant when the focus of the study is on some latent lin- guistic phenomena at the level of pragmatics, such irony or hateful speech, because of the high degree of subjectivity involved in the annotation of these phenomena (Basile, 2020). Taking this research direction one step further, the perspec- tivist approach (Cabitza et al., 2023) aims at creat- ing models capable of encoding different human
Furthermore, prove that methodology can also be applied when the annotators' meta- data are unavailable and describe a methodology mine perspectives from annotations directly. With this approach, on the one hand, we test whether our method can be applied when no explicit demo-

---

graphic data of the annotators are available. On the other hand, we test whether the annotations can be used to identify interesting implicit discriminative dimensions that differ from those encoded in the metadata.
For modelling perspectives in hate speech detec- tion, scholars have proposed various approaches, taking into account the individual annotation of each annotator (Davani et al., 2022), group-based annotations (Akhtar et al., 2020), specific aspects of annotators such as biases (Milkowski et al., 2021)
This paper is organized as follows. In Section we present related previous works. with specific focus on methods to build perspective-aware mod- els; Section describes our perspectivist method. We perform in-domain and cross-domain experi- ments for irony (Section 4) and hate speech detec- tion (Section 5). We detail our approach to mine perspectives and the results obtained with these implicit perspectives in Section 6. The discussion and conclusive observations that emerged from our work are in Section 7.
and beliefs (Kocoń et al., 2021a), or using person- alized methods (Kocoñ et al., 2021b).
In particular, Davani et al. (2022) proposed exploit annotators' systematic disagreement and investigated multiple approaches for hate speech and emotions detection: an ensemble model, where each classifier is trained on annotator-specifie data only; a multi-labod model, where cach label corre- sponds to that assigned by individual annotator; multi-task model, where the prediction of labels from each annotator is modelled as a separate task, The latter is superior to a baseline model trained on the aggregated data. Akhtar et al. (2020), instead, divided the annotators into two groups; they took into account common personal characteristics such as ethnicity, social background, and culture and modelled the group-based perspectives in recogni- tion of hate speech.
2 Related work
The development of computational methodologies to account for the differing viewpoints of different annotators in detecting pragmatic phenomena is not new. However, most previous research focused on taking advantage of disagreement in the data to improve the classification performance (Aroyo and Welty, 2015; Plank et al., 2014; Jamison and Gurevych, 2015).
In contrast to perspectivist literature on hate speech detection, perspectivist approaches to irony detection are less explored. To our knowledge, only two disaggregated datasets for English exist on humour (Simpson et al., 2019) and irony (Frenda et al., 2023b). The first dataset was used bench- mark in the first edition of LeWiDi (Uma et al., 2021). The second, the English Perspectivist Irony Corpus (EPIC), was used to explore the agreement among annotators with similar socio-demographic traits. Based on these traits, the authors also cre- ated perspectivist models showing the variation of the perception of irony among different population segments.
Recently, attention to the disagreement among the annotators has been increasing. Uma et al. (2021) and Leonardellli et al. (2023) organized two editions of the LeWiDi (Learning with disagree- ment) shared tasks, respectively, in SemEval 2021 (Task 12) and SemEval 2023 (Task 11). The orga- nizers of both editions of the workshop remark on the concept of soft label (vs. hard label) proposed by Uma et al. (2020b), method for learning from a distribution over the label space to maximize classification performance and provide a robust evaluation metric. The paradigm of learning from disagreement takes into account the presence of dis- agreement in the annotated data, because different opinions could arise from different annotators.
Confidence-based perspectivist ensemble
Inspired by studies that demonstrate how the per- ception of irony and toxicity varies based on so- ciodemographic identities (Sap et al., 2022; Frenda et al., 2023a), we divided annotators based on their metadata (gender, age, nationality) and constructed perspective-specife datasets encoding their percep- tion of a given phenomenon (see details in Section 4.1). For each of these longitudinal splits of the training set, we fine-tuned pre-trained language model (PLM), obtaining I classification models
Data perspectivism overcomes the idea of "ground truth" in constructing datasets and creating NLP models give space different perspectives that can be encoded in the annotated corpora, focus- ing on who the annotators are. Following this new theoretical framework, two editions of the NLPer- spectives workshop! have been organized in 2022 and 2023.
Ihttps://mnpperspeecivve wnito. it/w/

---

for irony detection. While these models are trained on mostly similar texts, we expect differences to emerge in their predictions due to the different la- bels they were exposed to at training time.
from Twitter and Reddit annotated by 74 annotators coming from 5 English-speakinn countries (United States, United Kingdom, Australia, Ireland, and India) and reporting different vocio-demographic traits. We used the data split proposed in the origi- nal publication for our experiments. Starting from the entire set of EPIC data, we computed the major- ity voting among the annotations of cach instance. creating aggregated set of 2,767 instances?. We then split the data into training (80%) and test set (20%), reproducing the same distribution of tweets and comments from Reddit. The aggregated train- ing set was used to train the non per-ppesptttiiist mod- els (NP) (see Section 4.3), while the test set was used to test all the models in the in-domain setting (Table 1). After setting apart the test instances, we reproduced the longitudinal split from the EPIC paper based on the annotators' socio-demographic traits, resulting in 11 sub-sets of different sizes, each representing an annotator perspective: Fe- male (1,952), Male (2,023), Boomer (441), GenX (1,757), GenY (1,964), GenZ (1,124), UK (1,365), India (1,175), Irish (1,296), US (1,352), Australian (1,377).
Furthermore, the models will output predictions with different degrees of confidence, which is the critical information we exploit for the proposed ensemble model. Formally, for each instance at in- ference time, we obtain from the trained models predictions (4, c1), ..., (4n, Cn) where 4 {0, 1) is a binary label and 9 E [0, 1] is confidence score. We use the normalized difference between the two logits rescaled by softmax (Taha et al., 2022) as a confidence score. Specifically, we compute the confidence as:
L1 Lo Lio
if 4
Cj
Lo L1
if 4 =
Loo
where Le is the logit of class e rescaled by softmax and 4 is the model prediction.
The predieted label of our confidence-basee en- semble model is therefore computed as:
Σ
The details are reported in Appendix A (Table 8).
huiight argmar
Cke
Ck
4.2 Experimental Setting
In plain terms, the confidence scores of the models predicting the negative class are summed, as are the confidence scores of the models predicting the positive class, and the ensemble selects the label associated with the highest confidence sum.
We fine-tuned three general- purppse PLMs. We aimed test the method with well-established base- lines, and therefore selected two of the most widely used PL.Ms namely BERT (Devlin et al., 2019) and RoBERTa (Zhuang et al., 2021). Since our method implies multiple fine tunings with potential ecological and computational impact, we further checked its validity on smaller model i.e., Dis- tIBERT (Sanh et al., 2019). We downloaded pre- trained models from the Huggingface repository with identifiers bert-base-uncased. roberta-base and distilbert-base-une uncased respectively. We used a batch size of 16 and a learning rate of 5e - 5 for BERT and 5c - 6 for ROBERTA and DISTIL- BERT. We customized the model to implement Fo- cal Loss (Lin et al., 2017) to prevent overfitting in the case of unbalanced datasets. We implemented early stopping, with patience of 2 epochs on the validation loss for all the models.
We also propose a simpler version of the ensem- ble computing a confidence-basee vote:
htigh- = 4 k argmar(ci..... Cn)
In this version, the ensemble selects the label pre- dicted by the classifier with the highest confidence in its prediction.
4 Perspectivist Classification of Irony
We present an irony detection experiment with the goal of proving the informative value of annotator perspectives in the annotated data.
4.1 Data
4.3 Baselines
The dataset used for testing our methodology in irony detection is EPIC (English Perspectivist Irony Corpus) released by Frenda et al. (2023b). EPIC contains 3,000 short conversations (post and reply)
We experimentally compared our model with sev- eral baselines and other approaches to determine
Instances were removed when no majority was reached.

---

negative class
positive class
macro-average
PLM
model
prec.
rec.
prec.
rec.
F1
prec.
rec.
Fl
Acc.
NP
.833
.711
.342
.581
.425
.608
.646
.603..036
.855.000
I-ENStigh
.800
.701
.339
.614
.436
.610
.657
.608 018
1.81:.00 026
I-FiN
.833
.737
.799
.350
.568
.432
.611
.652
.61520016
.703+0020
A-ENShigh
.897
.500
.711
.307
.728
.431
.602
.659
.571 2.011
.618 2.28
BERT
A-FNNSweigh
.977
.566
.715
.307
.722
.431
.602
.659
2573
1.019
M-ENS
868
.45
.801
.349
.543
.423
.608
.644
66122.016
705 1.06
C-ENSMiR
.76
.701
.779
.335
.603
.30
.606
.652
.605
1882+.018
C-ENS
875
.743
8003
.358
.571
.439
.616
.657
.621+.021
.709 1199..020
NP
894
.658
.757
.3332
.685
.44
.613
.671
.602
.663
I-ENStigh
.73
.689
.770
.324
.597
.400
.598
.643
.595
.671 1.066
1-ENht
889
6455
.748
.321
.676
436
.605
.661
.592+.012
.652
A ENShigh
891
645
.748
.323
.683
.439
.607
.644
.593
652
DESTILBERT
A weigttght
889
.645
.747
.321
.676
.355
.605
660
.591
.65
M-ENS
8877
.712
.786
3341
.600
.455
.609
.656
.610
6900..15
C-ENSsigh
.879
.712
.786
.343
.605
438
.611
.658
.612
.690
C-ENSawighh
878
.713
.787
.344
.603
438
.611
.658
.612
.691 =.018
NP
916
.702
.793
.366
.740
-506
.651
.721
.649
.710+.022
L-ENStigh
.988
.736
.809
.384
.664
.487
.641
.700
.648 -.10
:721
L-ENSweight
.911
.723
.002
.379
.679
.486
.640
.701
.644..006
.714+.000
A-ENShigh
912
.655
.762
.30
.747
.476
.631
.701
.619
.6732.014
ROBERTA
w-ENS
913
.648
.758
.347
.752
.475
.630
.700
.6161.015
66992001
M-ENS
.977
.760
.823
.403
.649
.496
.650
.704
4.09
.738
C-ENSMig
.748
.818
.401
.680
.505
.653
.714
0.09
.7341 0000
C-ENSweigh
.758
.823
.667
.505
.654
.712
.664..011
.739
de. 2.02
Table 1: In-domain performance, i.c. models trained on EPIC and tested on the EPIC gold test set. The proposed confidence- based models (C-ENS) are marked in italics. For the macro-average F1 score and the accuracy score, we also report the standard deviation.
negative class
positive class
maaro-average
PLM
model
рrеc.
rec.
F1
ргec.
F1
рrec.
F1
Acc.
NP
.592
.641
.40
.315
.466
.478
4422+.041
511+.0ap
I-ENStigh
.611
.297
.280
.431
.445
.430a.03
.480
i-ENS
.599
315
.377
.302
.445
.458
.438
4877.014
A-ENSsigh
.445
.506
.396
.562
.460
.504
.503
.833..03
.491 #.05
BERT
A-ENSweight weight
.399
.394
.608
.473
.505
.504
.473+.016
4822.016
M-ENS
.572
.364
.399
.480
.486
.475 039
.504+.016
C-ENSsigh
.602
.589
.587
.377
.397
.490
.493
.481
.513
C-ENS
.596
.598
.590
.365
.375
.360
.481
.486
.475..088
.509
NP
.655
.600
.288
.411
.576
.47
.188
.18
.503+.007
506+..00
I-ENStigh
.588
.683
.631
.350
.269
.302
.469
.476
.466..094
.519.0017
I-ENSweight
.597
.673
.632
.372
.305
.332
.485
.489
482+.084
.57
A-EN A-ENSpigh
.604
.643
.621
.384
.351
.364
.494
.497
.492
.527..022
DISTILBERT
A A-ENS
.601
.641
.619
.383
.347
.361
.492
.494
.490
.525
M-ENS
.611
.726
.656
.381
.265
.308
.411
.46
.8221.08
.543
C-ENSwigh
.612
6676
.64
.399
.342
.363
.006
.009
.501
.543
C-ENS
608
7788
.652
.398
.301
.338
.503
.044
.495
.547 +.07.012
NP
.717
.524
.598
.486
.679
563
.602
.601
.581 1.026
.585
L-ENStigh
.693
.542
.605
.476
.699
.40
.584
.86
.5722.019
.577.000
I-FNSi
.706
.116
.592
.476
.667
.54
.591
.592
.5731.024
.576..00
A ENS
.750
.496
.591
.492
.739
.888
.621
.618
.590+.023
.593
RoBERTA
A- A-ENS
.765
.471
.578
.492
.773
.599
.629
.622
.589
.5911.00
M-ENS
.699
.645
.668
.514
.570
.536
.606
.608
.602+.019
.6154.010
C-ENSMis
.711
.620
.666
.515
.615
.560
.613
.617
.610=.015
.6184.014
C-ENSwvight
.708
.630
.665
.517
.601
.555
.612
.616
.610#.011
+.10
Table 2: Cross-domain performance, i.e., models trained on EPIC and tested on the SemEval 2018 Task 3 test set.
the source of the improvement.
4.1. We then used this dataset to fine-tune the three PLMs. This approach does exploit the annota- tors' disagreement.
Non-perspectivist model (NP) For each data in- stance, we aggregated annotators' labels through majority voting. We discarded utterances for which no majority was found, as discussed in Section
Instance-based Ensemble (I-ENS) This base-
line is an ensemble of non-perspectivist models.

---

Each model was trained on a sample of the aggre- gated dataset, which had the same cardinality as the perspective-aware ones. We predicted the final label using the two confidence- based approaches discussed in Section 3.
4.5 Cross-domain Classification of Irony
To test the robustness of our method, we tested our ensemble model in cross-domain setting. Given the models trained on the EPIC dataset that we described in Section 4, we tested them on dif- ferent irony detection dataset, with particular adaptation. In particular, we use the test set from SemEval 2018 Task "Irony Detection in English Tweets" (Van Hee et al., 2018).
Anmotator-based Ensemble (A-ENS) We wanted to quantify whether grouping annota- tors by their demographic characteristics was advantageous over using random groups. To this end, we randomly selected annotators from the entire pool for each model in the ensemble. We then created 11 sub-datasets based only on the majority voting of such annotators, which we used to train the models; we predicted the final label using our confidence-based methods. The number of annotators whose labels are used in each sub-dataset is consistent with the number of annotators in each perspectivist dataset.
With respect to the in-domain performance, the results of the cross-domain experiment are more mixed and less straightforward, as shown in Table 2. Nevertheless, the general trend confirms the ad- vantage of using disaggregated data, perspectives, and model confidence, with notable exceptions: A- nigh with BERT and the non-perspeetivist base- line (NP) with
in particular, gave particularly high results on this task. In both cases, the results are mostly due to better performance on the positive class (irony), indicating the need to better control model bias during the training of perspective-awart models. However, the C-ENS classifiers always rank in the top-three results, and the best result by far is obtained with ROBERTA paired with either C-ENS models.
Majority voting Ensemble (M-ENS) We wanted to test the contribution of using confidence information as an ensembling strategy. To this end, we compared our model to identical ensemble that computes the final label using simple majority voting strategy over the output of each classifier.
Perspectivist Hate Speech Detection
For all baselines, the three PLMs were trained with the same hyperparameters and training strategies.
Considering the promising results achieved by our method on the irony detection task presented in the previous sections, we further validated our ap- proach on a different task. Hate speech (HS), sim- ilarly to irony, is pragmatic phenomenon in lan- guage, with a high degree of subjectivity involved in its annotation (Poletto et al., 2021). Unsurpris- ingly, HS and related phenomena of undesirable language objects of several works in perspec- tivist NLP (Cercas Curry et al., 2021; Akhtar al., 2021; Dinu al., 2021). This section presents the results of experiments analogous to the one we pre- sented on irony detection on -haied data annotated with HS.
4.4 Results
Table shows the experiment results, reporting an average of 10 runs. With all three PLMs, the performance the classification provided by our ensemble models on the gold standard test set over- come the baselines and are always better than all the other models. C-ENSweight- which computes its prediction based on the weighted sum of each model's label, achieves better result, in terms of fl score, over C-ENS which selects the model with the highest confidence in the prediction. How- ever, the latter still superior to most baseline results.
5.1 Data
For all models, we notice substantial improve- ment when annotators are grouped by their charac- teristics rather than randomly, which validates our original hypothesis about the informativeness of perspectivist data. Most importantly, using confi- dence measure consistently leads to better results than simply ensembling by voting. This confirms confidence as a key piece of information for per- spectivist modelling.
The dataset used in this set of experiments is the Measuring Hate Speech corpus (MHS) (Kennedy et al., 2020)², which contains identity group targets and the annotators' sociodemographic information, as well as their estimated survey interpretation bias. difficulty, and rarity of decision. MHS contains
"nttps://huggingfaet co/ datasets/ bbbekkelyyddlabmneasrrr hate-speech

---

negative
positive class
macro-average
PLM
model
prec.
rec.
F1
prec.
rec.
F1
ргec.
rec.
Acc.
NP
.898
818
.680
806
.737
.789
.812
.796_
008
8144.010
BERT
M-ENS
.912
.804
.854
.671
.837
.745
.792
.800
.800
003
8814 003
C-ENNShigh
.912
800
.852
.668
.888
.743
.790
.819
.798
004
..03.0005
CoEigr
.913
805
.856
.674
.840
.748
.794
.833
.802
003
.816. 004
NP
906
815
.888
.680
822
.745
.793
.819
.801
.817
M-ENS
913
801
.853
.669
.841
.745
.791
.211
.799
8814 .002
DISTILBERT
G-ENShigh
911
.000
.852
.667
.837
.742
.789
.888
_797_o0s 000
.8 .812 .003
C-ENSweighh
912
801
.853
.669
.839
.744
.791
.20
.799
.813
NP
908
.802
.852
.668
.330
.740
.788
.866
.796_
811+
M-ENS
911
805
.855
.672
.836
.745
.792
.20
880
8854
ROBERTA
C-ENStigh
912
.807
.856
.675
.837
.747
.793
.822
.802
.817
C-ENSueighr
9911
.888
.856
.676
835
.747
.793
.822
.802
001
.817 000
Table 3: In-domain performance of Hate Speech detection models, i.e., models trained on MHS and tested on the MHS gold test set. The proposed confidence-basse models (C-ENS) are marked in italics. For the macro-average F1 score and the accuracy score, we also report the standard deviation.
negative class
positive class
macro-average
PLM
model
prec.
rec.
F1
prec.
rec.
F1
rec.
F1
Acc.
NP
.602
.468
.526
.441
.576
.500
.522
.522
.513
.514 1.014
M-ENS
.615
.472
.534
.451
.595
.513
.533
.533
.523
.524
BERT
C-ENS
.617
.470
.533
.452
.600
.515
.534
.535
_5241.010
.524
C-ENS
.133
.471
.533
.49
.922
.511
.531
.532
.22++.004
.22++.004
NP
.620
.493
.550
.457
-886
.513
.539
.539
.531 #.002
.532+.003
M-ENS
.629
.502
.558
.465
.594
.522
.547
.548
.400.003
.541 1.00
DISTILBERT
C-ENSSigh
.631
.515
.567
.469
.587
.521
.550
.551
.544 1.03
.545
C-ENStrrighr
.629
.507
.562
.466
.521
.548
.549
S41 1.003
.542..004
NP
.488
.542
.448
.502
.529
.530
.522..00
.523+.005
M-ENS
.613
.495
.548
.452
.505
.533
.533
.5262.005
.527..005
ROBERTA
C-ENSMig
.613
.497
.549
.452
.569
.504
.532
.533
.5262.005
.527+.007
C-ENStrripht
.613
.497
.549
.452
.569
.504
.533
.533
.5261.005
.528
Table 4: domsan performance of Hate Speech detection models, Le., models trained on MHS and tested the HatEval test set.
39,565 texts annotated by 7,912 people. The HS label in MHS has three values (no HS, weak, and strong HS) that we combine into a binary label conflating weak and strong HS. From the available
computational resources, and possibly unnec- essarily impactful the environment. Comparing the two ensemble models, C-ENSweight tends metadata, we selected 4 dimensions (education, gender. ideology, and income, see details in Table 9 in Appendix A) and aggregated the annotators by their possible values, and split the dataset randomly across the comments into training (80%), validation (10%), and test (10%) sets.
give higher scores, except for the ROBERTA PLM, where the two have the same performance.
5.3 Cross-domain Classification of Hate Speech
We further tested our models on the test set the English part of HatEval (Basile et al., 2019), popular benchmark for HS detection from the SemEval 2019 Task 5: "Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter." The HatEval dataset consists of 9,000 in- stances for the training set, 1,000 for the validation set, and 2,972 for the test set.
5.2 Results
Table 3 illustrates the results when testing the models in-domain, showing that the confidence- based models perform better than the baselines with ROBERTA, while the results are mixed with the other two PLMs. Given the dataset size which is an order of magnitude larger than EPIC-- could not include the ensemble-based baselines; adding the baseline results would have required fine-tuning 9 PLMs per run (with a grand total 540 models), which would be unfeasible given
The results in Table 4 show how the confidence- based models give the best results in cross-domain HS detection with all three PLMs. Differently from the in-domain performance, in this case, the C- ENS tends to outperform the C-ENS We attribute this trend to the difference in the annota-

---

tors' backgrounds from training to test set in the cross-domain setting. Since the socio-demographic makeup of the annotators of HatEval is differ- ent from EPIC, the C Shigh ensemble it is likely able to pick up on their intersection, while in C' E weeghtt the opinion of a broader set of annotators (possibly not intersecting with HatEval) has a stronger influence on the final prediction.
cluster
Node
6.307
5.919
6.639
5.600
5,444
Annotators
18
12
19
15
10
Female
34%
11%
31%
9%
14%
Male
15%
21%
21%
31%
13%
Australian
27%
13%
20%
27%
13%
Indian
33%
33%
20%
7%
Irish
27%
20%
33%
7%
1.3%
British
20%
0%
27%
40%
13%
US american
14%
14%
29%
21%
21%
6 Mining perspectives
Boomer
0%
0%
33%
33%
33%
GenX
36%
18%
14%
14%
18%
The method we presented so far takes advantage of some division of the annotators known in ad- vance. However, this is not always the case, for instance, when an annotation results from crowd- sourcing with anonymous contributors. Therefore, we devised a method to automatically group the annotators meaningfully to represent different per- spectives. In this section, we present the result of a similar experiment to the main one presented in Section 4, with the difference that the perspectives are mined automatically. In this setting, the data split to train the perspective-aware models depends on annotations rather than demographic traits.
Gen'Y
18%
16%
32%
21%
13%
GenZ
30%
10%
30%
30%
0%
Table 5: Description the clusters in respect to the node level at which they joined; the number of annotators; the distribution of the demographic traits among clusters.
spectively measure how dense and how similar the clusters are; thus, we opted for a trade-off between the two, minimizing their ratio, and resulting in 5 clusters described in Table 5.
Looking at the annotators' distribution, there is no evident mapping between the demographic-basee perspectives and the mined clusters, for any of the gender, nationality, and generation dimensions. This confirms hypothesis that the metadata pro- vided by EPIC do not necessarily induce the best possible split among the annotators, and therefore automatically mining perspectives could uncover latent dimension that better discriminates the anno- tators. As for gender, cluster 3 represents a small percentage of females, but male annotators are bet- ter distributed among the five clusters. Unbalanced be also when consider- ing nationality; cluster 3 hosts 40% of the British annotators, and small percentage of both Indian and Irish annotators (represented especially the first three clusters), while cluster has no British annotators at all. Similarly, focusing on age, clus- ter 0 and have no Boomer annotators, and GenZ is more represented by clusters 0. 2 cluster 3 in respect to the remaining two. Despite these dif- ferences, none of the demographic groups merge homogeneously into specific clusters.
Following Fell et al. (2021), given n annota- tors and & instances, we constructed a label ma- trix vaxk, Since each instance was labeled, on average, by 4.72 annotators only, the resulting ma- trix was highly sparse. We modelled the similar- ity among annotators in terms of inter-annotator agreement, using Krippendorff's alpha (a) (Krip- pendorff, 2011). We thus computed the pairwise agreement between annotators i, j. obtaining a sim- ilarity matrix A,j - aVV.. Vj.). from which we calculated the distance matrix D = A, used as input for the clustering algorithm.
representations
We found 82 cases in which two annotators had common annotated instances, and we decided assign 0 for all of them. In 158 cases, there was perfect agreement between pairs of annotators on a small sample of data, having as a consequence that the a was not well-defined - Checco et al. (2017) describes it as a "paradox". We set a = for these pairs. We used hierarchical agglomera- tive clustering* with Ward's linkage criterion to group the annotators, computing the full dendro- gram. Then, applied the Calinski-Harabasy (Caliński and Harabasz, 1974) and Davies-Bouldis Indexes (Davies and Bouldin, 1979) to assess the best number of clusters between 2 and 5, and 11 clusters as the perspectives. The two metrics re-
Finally, we tested the validity of the clusters by reproducing the method described in Section 3. We obtained 5 classification models, which we tested in an in-domain setting (Section 4) and in cross-domain one (Section 4.5); in both cases, we averaged the results over 10 runs. Table 6 shows that the macro-average Fl scores are always
FFrom Scikit Lear

---

negative class
positive class
maaro-averagge
PLM
model
prec.
rec.
F1
prec.
rec.
F1
prec.
F1
Acc.
C-ENSiag
887
.679
.768
.338
.651
.443
.613
.665
6054.024 (A 0000)
663 +
BERT
C- C-ENSweiie
887
.709
.787
.354
-634
.422
.620
.671
.204.017 (A .001)
.694 028
DISTILBERT
C- C-ENSaag
.87
.725
.794
.348
.590
437
.612
.657
6164..009
(A + .004)
.998 4. 012
C-ENSuug
.877
.727
.795
.350
.589
.38
.613
.658
.677.001
(A .005)
.700 0.15
ENSiig
.736
.396
6584.014
(A .033
.788+ 016
ROBERTA
C-ENSwerie
.077
.753
8233
.410
.689
.514
.658
.211
4688..05 (A .04))
.740+.014 016
Table 6: In-domain performance cluster- based models, trained on EPIC and tested on the EPIC gold test set. In brackets the delta between the cluster-based and the correspondent demographic- ensemble model.
negative class
positive cluss
macro-avcraage
PLM
model
rec.
F1
pree.
rec.
F1
prec.
rec.
F1
Acc.
C-ENSwia
.629
.404
.467
.400
.613
.470
.515
.509
-69-
A .012)
.487
BERT
C- C-ENS
.602
.438
.89
.380
.552
.437
.491
.955
463_.037
(A .012)
,483 057
C- C-ENS
.601
.694
.642
.379
.295
.325
.490
.495
484_.052
(A .017)
,336+ 012
DISTILBERT
C-ENS
.597
.730
.372
.249
.294
.484
.489
474_ 025
(A .021)
.539 015
C-ENSSugs
.75
.540
.623
.504
.711
.588
.624
.626
-05_.017
(A .005)
.608 015
ROBERTA
C-FNS
.726
.584
.646
.512
.662
.575
.619
.623
.611 008
(A + .001
.615 0.0
Table 7: Cross-domain performance of cluster-based models, trained on EPIC and tested on SemEval 2018 Task 3 test set. In brackets the delta between the cluster-based and the correspondent demographic- ensemble model.
above all baselines (Section 4.3). In-domain, the mining-based models perform rather well, with two PLMs (DISTILBERT and ROBERTA) per- forming better than their demographic-based coun- terparts, and BERT obtaining very similar scores. Cross-domain (Table 7), the results of the mining- based models are comparable to the demographic- based counterparts. The confidence- based ensem- bles with ROBERTA still perform better than their demographic- based counterparts, while the same is not true for the other two PLMs. This experi- ment shows that, by mining annotator perspectives, we can model annotator opinions based on their actual choices and look irony as phenomenon that transversely connects individuals of different demographics.
hate speech detection.
The experimental results presented in this pa- per show that our method produces more accurate classifications of irony. In the case of hate speech, the results were mixed. However, the shape of the available data used for training the models the two tasks varies greatly, in particular with respect to the number of annotators - EPIC (irony) has 74 annotators while MHS has thousands, resulting in much greater variety of individual perspectives beyond those captured in the annotator metadata. More impressively, we have shown how the method equally valid (and in many cases performing bet- ter) when the perspectives are automatically ex- tracted from the disaggregated annotations rather than explicitly provided (Section 6).
Our experiments focused on the two case studies of irony and hate speech detection, and it remains to be explored how well our approach generalizes to other pragmatic phenomena. We plan to ap- ply our methods other pragmatic phenomena, irony-adjacent and in the undesirable language area, where we expect the most impact, as well as other languages. Future research should also include a broader range of demographic profiles to ensure greater inclusivity and improve the generalizability of perspectivist models across various linguistic contexts. Finally, perspective-aware models and ensembles naturally have the potential to improve explainability: they provide layer of information that is useful for explaining the output of the classi- fiers, i.e., in terms of the human perspectives more likely to be sensitive to specific labels.
7 Conclusion
We showed how computational models of irony and hate speech more proficient when built disag- gregated data, i.e., explicitly modeling the disagree- ment between the annotators. One step further, our experiments validate the perspectivist approach of grouping annotators to better encode their shared background, either by using their metadata or by automatically grouping them with a clustering ap- proach. The main novelty of the method described in this paper is the use of the model confidence computational tool ensemble set of pre- dictions coming from different perspective-aware models. This intuition, originated by the work of Frenda et al. (2023b), was proved central in build- ing more accurate predictive models for irony and

---

Limitations
sonal information. We acknowledge the potential biases and ethical implications that may arise from the annotation process and strive to address them through careful analysis, discussion, and mitiga- tion strategies. Furthermore, we advocate for the responsible use and deployment of perspectivist models in real-world applications. We emphasize the importance of fairness and inclusivity in model training and evaluation, aiming minimize the amplification of biases controversial viewpoints present in the annotations.
While the perspectivist approach for pragmatic phenomena classification shows promising results, there are certain limitations to consider. Firstly, our methodology relies on the availability of datasets data with disaggregated labels, showing different perspectives. This may not always be feasible or easily obtainable, especially for niche or under- resourced domains or languages.
The EPIC dataset has some limitations besides being monolingual. While efforts were made to ensure fair balance in terms of the demographic profile of the annotators, the dataset is limited to five varieties of English tied to specific countries. This exclusion of other potential locations, such as New Zealand or Nigeria, as well more nuanced distinctions among language varieties, may limit the dataset's representativeness and coverage of diverse linguistic perspectives.
Acknowledgements
The work of A. T. Cignarella, V. Patti, and C. Bosco was partially funded by the International project STERHEOTYPES Studying European Racial Hoaxes and sterEOTYPES, funded by the Compagnia di San Paolo and VolksWagen Stiftung under the "Challenges for Europe' Call for Projects (CUP: B99C20000640007).
About the self-identified gender dimension (pro- vided by annotators themselves on the Prolific web application), we are aware of the wider spectrum of genders. However, this information was provided by the annotators only in binary form.
References
Gavin Abercrombie, Valerio Basile, Sara Tonelli, Ver- ena Rieser, and Alexandra Uma, editors. 2022. Pro- ceedings of the 1st Workshop on Perspectivist Ap- proaches NLP @LREC2022. European Language Resources Association, Marseille, France.
Regarding the HS data, and in particular the MHS dataset, we aggregated annotations based on a selection of dimension influenced by factors like their distribution in the data. While it would have been unfeasible to train models on very sparse per- spectives, we recognize that this may lead to their under-repreesentation in our models.
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2020. Modeling annotator perspective and polarized opin- ions to improve hate speech detection. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 8(1):151-154
Ethics Statement
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? Perspective-aware models identify opinions of Hate Speech victims in abusive language detection. CoRR, abs/2 15896....
This research paper emphasizes the importance of incorporating diverse human perspectives in prag- matic phenomena classification, such as irony and hate speech detection, within Natural Language Processing tasks. We recognize the ethical con- siderations inherent working with sensitive lan- guage data and promoting responsible research practices. The perspectivist approach in general, and this work in particular, aims at "giving voice to the few who hold a minority view" (Cabitza et al., 2023). Applied to the creation of language resources and the creation of automatic models, this principle leads to resources and models where bias is controlled factor rather than undesirable criticality.
Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. Al Magazine, 36(1):15-24
Valerio Basile. 2020. It's the end of the gold standard as we know it: Leveraging non-aggregated data for better evaluation and explanation of subjective tasks. In AIxIA 2020 - Advances in Artificial Intelligence: XIXth International Conference of the Italian As- sociation for Artificial Intelligence, Virtual Event, November 25-27, 2020, Revised Selected Papers, page 441 Berlin, Heidelberg, Springer-Vette
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women Twitter. In Proceedings of the 13th International
To ensure ethical conduct, we adopted measures to protect the privacy of annotators, and our data handling protocols are designed to safeguard per-

---

Workshop on Semantic Evaluation, pages 54-63, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
Michael Fell, Sohail Akhtar, and Valerio Basile. 2021. Mining annotator perspectives from hate speech cor- pora. In NI.4T@@ A/* IA.
Valerio Basile, Michael Fell, Tommaso Fomaaciai, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the Ist Workshop on Benchmarking: Past, Present and Future, pages 15-21, Online. Association for Com- putational Linguistics.
Simona Frenda, Soda Marem Lo, Silvia Casola, Bianca Scarlini, Cristina Marco, Valerio Basile, and Davide Bernardi. 2023a. Does anyone see the irony here? analysis of perspective-awarc model predictions in irony detection. In ECAI 2023 Workshop on Perspec- rivist Approaches to NLP.
Simona Frenda, Alessandro Pedrani, Valerio Basile, Soda Marem Lo, Alessandra Teresa Cignarella, Raf- faella Panizzon, Cristina Marco, Bianca Scarlini, Vi- viana Patti, Cristina Bosco, and Davide Bernardi. 2023b. Epic: persptpprr annotation of a cor- pus of irony. In ACI.2025.
Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, Washington DC, USA.
Tadeusz Caliñski and Jerzy Harabasz. 1974. A den- drite method for cluster analysis. Communications in Statistics-therry and Methods, 3(1):1-27.
Emily Jamison and Iryna Gurevych. 2015. Noise or additional information? leveraging crowdsource an- notation item agreement for natural language tasks, In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 291-297, Lisbon, Portugal. Association for Compu- tational Linguistics.
Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser. 2021. ConvAbuse: Data, analysis, and bench- marks for nuanced abuse detection in conversational AL In Proceedings of the 2021 Conference Empir- ical Methods in Natural Longuage Processing, pages Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Chris J Kennedy, Geoff Bacon, Alexander Sahn, and Claudia von Vacano. 2020. Constructing interval variables via faceted Rasch measurement and multi- task deep learning: A Hate Speech application. arXiv preprint arXiv:2009. 10277.
Alessandro Checco, Kevin Roitero, Eddy Maddalena, Stefamo Mizzaro, and Gianluca Demartini. 2017. Let's s agree to disagree: Fixing agreement measures for crowdsourcing. In Proceedings of the AAAI Con- ference on Human Computation and Crowdsourcing, volume 5, pages 1-20.
Jan Kocoñ, Alicja Figas, Marcin Gruza, Daria Puchalska, Tomasz Kajdanowicz. and Przemystaw Kazienko. 2021a. Offensive, aggressive, and hate speech analysis: From data-centric to human- centered approach. Information Processing & Man- agement, 58(5):002643.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110.
Jan Kocoñ, Marcin Gruza, Julita Bielaniewicz, Damian Grimling. Kamil Kanclerz, Piotr Milkowski, and Przemystaw Kazienko. 2021b. Learning personal human biases and representations for subjective tasks in Natural Language Processing. In 2021 IEFE Inter- national Conference on Data Mining (ICDM), pages 1168- 1173.
David L Davies and Donald W Bouldin. 1979. A cluster separation measure. IEEE transactions on puttern analysis and machine intelligence, P2:2-1(2):224- 227.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability. Technical report, University of PennSylvania.
Elisa Leonardellli, Gavin Abercrombie, Dina Almanea, Valerio Basile. Tommaso Fornaciari, Barbara Plank. Massimo Poesio, Verena Rieser, and Alexandra Uma. 2023. SemEval-2023 Task 11: Learning With Dis- agreements WLDWi,i), In Proceedings of the 17th International Workshop on Semantic Evaluation, Toronto, Canada. Association for Computational Lin- guistics.
Liviu P. Dinu, loan-Bogdan lordache, Ana Sabina Uban, and Marcos Zampieri. 2021. A computational ex- ploration of pejorative language social media. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 3493-3498, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. CoRR, abs/ 700110088002222

---

Michele Mastromattei, Valerio Basile, and Fabio Mas- simo Zanzotto. 2022. Change my mind: How syntax- based hate speech recognizer can uncover hidden motivations based on different viewpoints. In Pro-
learning, Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 5716-5728, Florence, Italy. Association for Computational Linguistics.
ceedings of the 1st Workshop on Perspectivist Ap- prouches to NIP @ LREC2022, pages 117-125, Mar- seille, France. European Language Resources Asso- ciation.
Abdel Aziz Taha, Leonhard Hennig, and Petr Knoth. 2022. Confidence estimation of classification based on the distribution of the neural network output layer. arXiv preprint arXiv:2210.0744
Piotr Milkowski, Marcin Gruza, Kamil Kanclerz, Prze- myslaw Kazienko, Damian Grimling, and Jan Ko- con. 2021. Personal bias in prediction of emotions elicited by textual opinions. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing: Stu- dent Research Workshop, pages 248- 259, Online. Association for Computational Linguisties.
Alexandra Uma, Tommaso Fornaciari, Anca Dumi- trache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021. SemEval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop Semantic Evaluation (SemEval-200)) 2021), pages 338-
347, Online. Association for Computational Linguis- tics.
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 16671- Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alexandra Uma. Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2020a. A case for soft loss functions. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 8(1::133-177
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy. Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2020b. A case for soft-loss functions. In Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing, pages 173-177.
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the As- sociation for Computational Linguistics ( (Volume Short Papers). pages 507-511, Baltimore, Maryland. Association for Computational Linguistics.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-201 task 3: Irony detection in En- glish tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 39-50, New Orleans. Louisiana. Association for Computational Linguistics.
Fabio Poletto, Valerio Basile, Manuela Sanguinetti. Cristina Bosco, and Viviana Patti. 2021. Resources and henchmark corpora for Hate Speech detection: a systematic review. Language Resources and Evalua- tion, 55(2))4777-523
Liu Zhuang. Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre- training approach with post-training In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218-1 1277, Huhhot, China. Chinese Informa- tion Processing Society of China.
Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Vi- viana Patti, and Marco Stranisci. 2018. An Italian Twitter corpus of hate speech against immigrants. In Proceedings of the Eleventh International Confer- ence on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Re- sources Association (ELRA).
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884-5906, Seattle, United States. Association for Computational Linguistics.
Edwin Simpson, Erik-Lån Do Dinh, Tristan Miller, and Iryna Gurevych. 2019. Predicting humorousness and metaphor novelty with Gaussian process preference

---

A Appendix
Table 8 shows the details of the different subsets of data extracted from EPIC to train the perspectivist models. The models were used to build the ensemble model described in Section 3 and the I-ENS and A-ENS baseline models presented in Section 4.3.
Dimension
Value
# Instances
Socio-demographice
Female
1,952
Self-identified female.
Gender
Male
2,023
Self-identified as male.
Boomer
441
Older than 58.
GenX
1,757
Older than 42 and younger than 57.
Age
GenY
1,964
Older than 26 and younger than 41.
GenZ
1,124
Younger than 25.
UK
1,365
With English nationality.
Indian
1,175
With Indian nationality.
Nationality
Irish
1,296
With Irish nationality.
US
1,352
With American nationality.
Australian
1,377
With Australian nationality.
Table 8: Sub-sets of data extracted from EPIC on the basis of the socio-demographic traits of annotators.
Table 9 shows the details of the different subsets of data extracted from MHS, user in Section 5,
Dimension
Value high
Train
Val.
Test
24,021
2,971
2.956
Education
low
18,455
2,236
2,308
female
23,159
2,861
2,912
Gender
male
20,072
2,517
2,465
conservative
13,536
1,612
1,655
Idcology
liberal
19,376
2,433
2,416
neutral
10,380
1,331
1,300
high
22,489
2,755
2,762
Income
low
20,921
2,619
2,635
Gold
28,387
3,498
3,503
Table 9: Sub-sets of data extracted from MHS on the basis the socio demograhpic traits of annotators.


Reviews:
[{"review_id": "AFckPOm3lp", "reviewer": "Reviewer_17tq", "summary": "", "questions": "A. Line 197, Section 4.2: What is the rationale for grouping each demographic and calling it a perspective? How can we determine if a specific demographic dimension forms a valid perspective? Assuming it is indeed a perspective, does the main contribution lie in the approach to combining these perspectives, while also claiming the importance of considering all of them?\nB.What serves as the ground truth in this context? For each of the 11 splits, was the ground truth for training derived from the majority vote of that specific split? And during testing, were the 11 splits aggregated, and then compared to the overall majority vote? \nC.The 11 splits are based on demographics. If a single text is annotated by multiple people within a split, how does the model learn from these annotations?\nD.Line 378: What criteria were used to select only those four dimensions?\nE.Are the observed performance differences statistically significant? It would be helpful to report the averages and standard deviations over repeated runs, as some reported values show less than a 0.01 difference, which might not be statistically significant.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the many insights and useful comments, to which we reply by topic below.\n\n### Splitting of perspectives\nWe agree that demographics may not always align optimally with annotators perspectives, as discussed in recent works by [Santy et al.](https://aclanthology.org/2023.acl-long.505/) and [Orlikowsi et al.](https://aclanthology.org/2023.acl-short.88/) with mixed conclusions.\nIn the ACL 2023 paper introducing the [EPIC dataset](https://aclanthology.org/2023.acl-long.774.pdf), it is shown that the demographic groups exhibit a higher inter-group agreement compared to the agreement of the whole set of annotations. The same holds for the [MHS dataset](https://aclanthology.org/2022.nlperspectives-1.11.pdf), for which Krippendorff’s alpha indicates that target groups (as defined by demographic data) exhibit higher inter-group agreement.\nThe analysis of perspectivist datasets such as BREXIT (Akhtar et al. 2019, [2020](https://ojs.aaai.org/index.php/HCOMP/article/view/7473/7260)) comes to the same conclusion. Furthermore, while the clustering approach leads to similar results with our method (see Table 6), dividing the annotators based on demographics often has an advantage (Table 1), confirming the point that demographic factors strongly influence the perspectives of the annotators on irony in language.\n\n### Ensembling\nIndeed, the ensemble classifiers presented in our work outputs a single prediction. However, this is strongly informed by the perspectivist learning made possible by a disaggregated dataset. Moreover, the comparison of perspectives-based ensembles with the two baselines (I-ENS and A-ENS) shows that the improvement of our model is motivated by taking into account annotator perspectives.\nOne reason for proceeding this way is for the sake of evaluation, in- and cross-dataset. While a fully perspectivist evaluation framework has not been established yet, we point out in the future work section how we intend to tackle this relevant issue. \n\n### Research questions\nThe main result of our paper shows that model confidence is a key information to create perspectivist modeling. The section on using clustering techniques to extract the perspectives is an addition to the main line of research, that experimentally shows that, even when demographic information is not used, model confidence can still be used to create valid perspectivist models.\n\n### Answers to the questions\nA: See above “splitting of perspectives”\n\nB, C: At training time, for each split, the ground truth is aggregated by majority vote of that specific split. To create the test sets, the majority vote is computed on all labels together.\n\nD. In the MHS corpus, those 4 dimensions are the ones annotated with categorical values. \n\nE. The result tables report the averages of 10 runs. We calculated the standard deviation, which is at most 0.06 and about 0.02 on average for the macro-average F1-score and accuracy, indicating that the runs are very stable. Thank you for this important suggestion, we will include all the figures in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Capturing the subjective viewpoints of annotators is a vital area of study, and the attempt to model these perspectives is of crucial importance.\n\n\n\n\n", "reasons_to_reject": "Main issues with the proposed method:\n\nIf the model selects a label from just one perspective, it effectively disregards all the other ten perspectives. In such cases, the confidence corresponds to the model's confidence in its prediction. However, it is possible that the model's confidence is low because it didn't perfectly learn the perspective of a particular group, resulting in that label never being selected. This can be because of not correct splitting of perspective which is such a strong assumption to assumer that each demographic values share a perspective.\nThis approach contradicts the authors' claim that all perspectives need to be taken into account.\nThe authors acknowledged in Section 6 (line 471) that splitting based on demographics might not be representative of the perspective. However, a significant portion of the work relies on this approach, and conclusions are drawn based on assumptions that the authors are aware of being potentially flawed.\nThere are two research questions in this study, but they appear to be unrelated. One question focuses on considering confidence, while the other aims to extract perspective from annotations when metadata is not available.\nThe confidence score is almost identical to getting the soft labels, which has been proposed before\nThe suggested method involves aggregating the predicted labels, going beyond the majority vote of the models, and considering the soft labels. However, it remains unclear how this approach helps to incorporate more perspectives. The low confidence of a model could stem from various reasons, one of which may be the chosen and defined perspective. Annotations within a group might not necessarily share the exact same perspective merely due to a common demographic, and this could potentially lead to challenges in obtaining accurate soft labels.\n", "questions_to_authors": "A. Line 197, Section 4.2: What is the rationale for grouping each demographic and calling it a perspective? How can we determine if a specific demographic dimension forms a valid perspective? Assuming it is indeed a perspective, does the main contribution lie in the approach to combining these perspectives, while also claiming the importance of considering all of them?\nB.What serves as the ground truth in this context? For each of the 11 splits, was the ground truth for training derived from the majority vote of that specific split? And during testing, were the 11 splits aggregated, and then compared to the overall majority vote? \nC.The 11 splits are based on demographics. If a single text is annotated by multiple people within a split, how does the model learn from these annotations?\nD.Line 378: What criteria were used to select only those four dimensions?\nE.Are the observed performance differences statistically significant? It would be helpful to report the averages and standard deviations over repeated runs, as some reported values show less than a 0.01 difference, which might not be statistically significant.\n", "paper_topic_and_main_contributions": "The paper proposes an approach to split the annotators based on their demographic values, assumes each of these values represent a perspective. It aggregates  perspective-aware models predictions while considering the models’ confidence. \n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 388 : we could not include"}, "extra_text": {}}, {"review_id": "jOrgg3qQtT", "reviewer": "Reviewer_cKGw", "summary": "", "questions": "Can you elaborate on why you chose distilBERT, BERT, and RoBERTa PLMs for fine-tuning?  Was the idea just to experiment with small, medium, and large (or larger) PLMs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the kind words and recognition of the contribution of our paper.\n\nReplying to the specific question, we aimed at testing the method with well established baselines, and therefore we selected two between the most widely used PLMs (BERT and RoBERTa), also in order to check the robustness of the method against different architectures. Since our method implies multiple fine tunings, with potential ecological and computational impact, we considered it important to further check its validity on a smaller model (DistilBERT).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and will motivate researchers to think about multiple annotation and how to use it to best represent different perspectives of different populations represented within that annotation.  To that end, it will get researchers thinking about conducting massive annotation efforts _with_ as many different populations represented as possible.", "reasons_to_reject": "Some reviewers may not find this work particularly novel.", "questions_to_authors": "Can you elaborate on why you chose distilBERT, BERT, and RoBERTa PLMs for fine-tuning?  Was the idea just to experiment with small, medium, and large (or larger) PLMs?", "paper_topic_and_main_contributions": "The work described in this paper focuses on building perspective-aware models of tasks whose annotation is highly subjective, such as irony and hate speech.  The perspective-aware models described here use the confidence scores produced by several models trained on different demographic subsets of the annotators of each data set to produce an overall confidence score and aggregate prediction.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "On line 011, \"the key\" should just be \"key\".\n\nOn line 019, \"the classification\" should just be \"classification\".\n\nOn line 083, this line would flow better as \"on the one hand, we test whether...\".\n\nOn line 107, \"the classification\" should just be \"classification\".\n\nOn line 109, \"the attention\" should just be \"attention\".\n\nOn line 124, \"raise\" should be \"arise\".\n\nOn line 125, \"Whereas, data\" should be \"Data\" because this sentence doesn't have a clause that supports the \"whereas\".\n\nOn line 164, insert \"the\" before \"English\".\n\nOn line 388, remove the \"to\" before \"include\".\n\nOn line 439, \"resulted\" should be \"resulting\"."}}, {"review_id": "X086zcv3PP", "reviewer": "Reviewer_ZE2f", "summary": "", "questions": "1. General question, but let's look at Table 4 specifically. The difference between C-ENS and M-ENS in accuracy is in the third number ( 0.528 and 0.527 respectively). What is your uncertainty for the computations? If you round until the first or the second number, then you will get 0.5 and 0.53 in both cases - not better than the baseline, but way more effort. So, how is that justified the comparison in the third digit?\n2. Why do you think you have the difference just in the third digit? \n2a  do you plan to somehow address this issue for future work? \n3. why did you choose three models from the same ''family''? you could have taken the strongest one\n4.  Will the model for the data clustering be available and ready to use?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the encouraging comments, and questions, to which we answer point by point:\n\n1, 2: We calculated the standard deviation, which is at most 0.06 and about 0.02 on average or the macro-average F1-score and accuracy, indicating that the runs are very stable. \n\n3: We aimed at testing the method with well established baselines, and therefore we selected two between the most widely used PLMs (BERT and RoBERTa), also in order to check the robustness of the method against different architectures. Since our method implies multiple fine tunings, with potential ecological and computational impact, we considered it important to further check its validity on a smaller model (DistilBERT).\n\n4: We will make the code used to cluster annotators available, in order to reproduce the full experiment.\n\nOn the topic of **reproducibility**, we are slightly surprised by the low score. All the dataset used in our study are publicly available, the main method and the baselines are documented in detail, and the source code to reproduce the results was attached to the submission as supplementary material and is in a Github repository that will be made public upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I like the idea of accounting for the annotator's demographics. There are multiple works urging us to take this path, and this work shows another dimension of how that could be incorporated into real research. In addition, the authors propose a method to get the annotator's data clustered even without demographic data, which might be useful for other researchers.\nAnother contribution is that authors train classifiers on data clustered by the annotator's demographic data, they conduct experiments on three models and in-domain and cross-domain experiments as well.", "reasons_to_reject": "See questions", "questions_to_authors": "1. General question, but let's look at Table 4 specifically. The difference between C-ENS and M-ENS in accuracy is in the third number ( 0.528 and 0.527 respectively). What is your uncertainty for the computations? If you round until the first or the second number, then you will get 0.5 and 0.53 in both cases - not better than the baseline, but way more effort. So, how is that justified the comparison in the third digit?\n2. Why do you think you have the difference just in the third digit? \n2a  do you plan to somehow address this issue for future work? \n3. why did you choose three models from the same ''family''? you could have taken the strongest one\n4.  Will the model for the data clustering be available and ready to use?", "paper_topic_and_main_contributions": "In this paper, the authors propose a way to account for the annotators' uncertainty due to different demographic features. They train several classification models on data grouped by the annotators' demographics. In addition, they also propose a way to cluster the data if no annotators' demographic data is available.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a method that accounts for annotators disagreement by grouping data based on demographic information about the annotators, for tasks where annotation is subjective (irony and hate speech detection).  The classifier aggreates confidence scores and predictions from models trained on annotation by annotators from different demographic groups. 

This paper contributes to the growing literature on accounting for annotators disagreement. The claims are overall well-supported by the experiments, however, the presentation could be improved by clarifying what is meant by perspective and how that relates to demographic groups (reflecting for instance some of the points made in response to reviewer 17tq).
Decision: Accept-Main