Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It’s Best to Relate Perspectives!
Authors: Philipp Heinisch
Matthias Orlikowski
Julia Romberg
Philipp Cimiano
Venue: EMNLP 2023 Main (2023)
Keywords: argument quality
perspectivism
inter-annotator-disagreement
LLM
recommender
argument mining
TL;DR: Argument quality can not reduce to a single majority label but has to decompose into different perspectives - we explore different model architectures to capture the interaction between individual views and common ground.

Abstract: Many annotation tasks in natural language processing are highly subjective in that there can be different valid and justified perspectives on what is a proper label for a given example.
This also applies to the judgment of argument quality, where the assignment of a single ground truth is often questionable.
At the same time, there are generally accepted concepts behind argumentation that form a common ground.
To best represent the interplay of individual and shared perspectives, we consider a continuum of approaches ranging from models that fully aggregate perspectives into a majority label to ``share nothing''-architectures in which each annotator is considered in isolation from all other annotators. 
In between these extremes, inspired by models used in the field of recommender systems, we investigate the extent to which architectures that predict labels for single annotators but include layers that model the relations between different annotators are beneficial.
By means of two tasks of argument quality classification (argument concreteness and validity/novelty of conclusions), we show that recommender architectures increase the averaged annotator-individual F1-scores up to 43% over a majority-label model.
Our findings indicate that approaches to subjectivity can benefit from relating individual perspectives.

Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!
Philipp Heinisch 1x Matthias Orlikowski Julia Romberg", and Philipp Cimiano'
'Center for Cognitive Interaction Technology (CITEC), Bielefeld University, Germany opheinisch, morlikowski cimiano / j@techfak. uni miieeeeeeeldddd Department of Social Sciences, Heinrich Heine University Düsseldorf, Germany julia, juliarromberg.. romberg@hhu......mm de
Abstract
as a problem, but rather as chance end up with more user-adaptable classifiers, giving voice to mi- noritics as well (Prabhakaran et al., 2021; Gordon et al., 2022).
Many annotation tasks in natural language pro- cessing are highly subjective in that there can be different valid and justified perspectives on what a proper label for a given example. This also applies the judgment of argument qual- ity, where the assignment of single ground truth is often questionable. At same time, there are generally accepted concepts behind argumentation that form a common ground. To best represent the interplay of individual and shared perspectives, we consider continuum of approaches ranging from models that fully aggregate perspectives into majority label to "share nothing" laiinn"--rcchhecttti -architectures in which each an- notator considered in isolation from all other annotators. In between these extremes, inspired by models used in the field of recommender systems, we investigate the extent to which architectures that include layers to model the relations between different annotators are ben- eficial for predicting single-annotato labels. By means of two tasks of argument quality classification (argument concreteness and va- lidity/novelty of conclusions), we show that recommender architectures increase the aver- aged unnotator-individual Fisscores up to 43% over majority-label model. Our findings indi- cate that approaches to subjectivity can benefit from relating individual perspectives.
In this paper, we hypothesize that machine learn- ing can best address variation human labeling by both accounting for subjective perspectives individuals and for more objective concepts that build common ground between annotators, A prime example of the interplay of individual and shared perspectives is the understanding and as- sessment of argumentation and, in particular, of argument quality (Romberg, 2022). Given the sub- jectivity that many concepts of argument quality face, it has been generally shown that its annota- tion often results in only fair to moderate inter- annotator agreements (Aharoni et al., 2014; Rinott et al., 2015; Habernal and Gurevych, 2017; Shnarch et al.. 2018).
For example, Gretz et al. (2020) determined a global value of argument quality by asking annota- tors "if they would recommend a friend to use that argument as is in speech supporting/contesting the topic". Even in more well-defined aspects of argument quality, such the sufficiency of ar- gumentative conclusion, Stab and Gurevych (2017) observed "hard cases" in which labeling depends on subjective interpretations of keywords without being able to agree on a single ground truth. How- ever, alongside "hard cases" of irreconcilable in- dividual perspectives, in assessing argument qual- ity we regularly also observe uncontroversial cases such in human annotation of the validity and nov- clty of conclusions (Heinisch et al., 2022) or the concreteness of arguments (Romberg et al., 2022). In order to verify our hypothesis, investi- gate the impact of different architectural design choices on the ability of models predict the la- bels of single annotators. To this end, we look at a continuum model architectures between such that fully suppress annotation variation by learning
1 Introduction
There is inherent subjectivity in many annotation tasks in natural language processing (Shahid et al., 2020; Kumar et al., 2021; Thorn Jakobsen et al., 2022). Recent work has criticized the widely adopted approach of viewing variation in human labeling behavior as "noise" (Plank, 2022), advocat- ing against approaches that disregard the richness of human annotations and perspectives by aggre- gating them into single label. In fact, it has been argued that disagreement should not be regarded
Authors contributed equally to this work.

---

annolalor wwwaaaoo--spccciii spocifie
PerAnnotator
mili Of comolltte
der model individual label decisions along the continuum from predicting a single label for annotators to "share nothing" architec- tures that model the label decisions of single individuals independently from each another. This framework is not limited to our use case of argument quality, but can be applied to any classification task whose annotation combines subjective and objective aspects, in order find architectural sweet spots for modeling label variation.
LLM
cls. head
prediction
specific par aotamm
SepREC
LLM
Combine
prediction
A
A
ShareREc
hybrid
LLM
combine
prediction
We perform an extensive automatic evalu- ation tuning different model types and hy- perparameters on two datasets correspond- ing to three tasks, involving vanilla large language models (LLM) as well as LLMs with annotator-speciftt classification heads and recommender models adapted to the clas- sification tasks in argument quality. Using a recommender-base architecture, increase the averaged imnotator-indvvidie Fi-scores by up to 4 points the classification of argu- ment concreteness, 18 points in the classifi- cation of conclusion validity and 19 points in the classification of conclusion novelty.
SepHeads
cls. prediction head
LLM
annolato aignoslic
Majority
prediction
LLM
cls. head
Figure 1: Overview of our different approaches model- ing (anotator- hehaviors.
We conduct qualitative case study of the be- havior of our models on controversial exam- ples in order to shed light on the differences garding the effect of (inotaator(disaagreemme differing amounts of annotated samples be- tween annotators, and annotation behavior.
from aggregated label and such that specifi- cally tailored each annotator. Along this contin- uum, focus in particular on models that include components that involve the perspectives of sin- gle anmotators (Davani et al., 2022), or are able to find and relate similar annotation behavior shared among different annotators using recommender approach (Gordon et al., 2022). Figure/ 1 gives first overview of the continuum of concepts and models, which we explain in detail in Section 3. Drawing on two tasks of argument quality, namely argument concreteness and validity/novelty conclusions, we show that architectures inspired by recommender systems perform best in learning from disagreement. We attribute this to the identi- fication of patterns in annotation behavior across individual annotators while being faithful single choices.
Extending previous work that proposed to take subjectivity into account by predicting the degree of expected label variation by Romberg (2022), our study presents the first approach in the field of gument mining to learn directly from the individual human labels. Beyond the specific contributions mentioned above, our work might encourage fu- ture research in argument mining and other fields to step away from systems that suppress valid per- spectives by relying on single aggregated label, and instead moving on towards systems that cover the multi-faceted spectrum of opinions.
Our main contributions are:
Related Work
We present novel framework! for comparing different architectural design choices in or-
2.1 Subjectivity & Modeling Individual Annotators
There is a growing body work researching sub- jectivity (Ovesdotter Alm, 2011; Rottger et al.,
'https://github. httphl...oophhhhheeeeeee Relateeerspectives-sweess

---

2022), learning with disagreement (Uma et al.,
allows user interactively aggregate them based on "a jury" inspired by the US judicial system. Their approach is based on recommender archi- tecture using "Deep & Cross Networks" (Wang et al., 2021) which we also use in our work.
2021a; Leonardelli al., 2023; Sandri et al., 2023), diversity perspectives (Abercrombie et al., 2022; Cabitza et al., 2023) and human label varia- tion (Plank, 2022). Despite the varying terminol- ogy, these works overlap in concerns that aggregat- ing labels into single "truth" is not appropriate for many tasks (Aroyo and Welty, 2015; Uma et al., 2021b; Basile al., 2021) and might not repre- sent perspectives equally (Prabhakaran al., 2021; Abercrombie et al., 2022). This line of research has produced various approaches to learning models based on individual annotations (Plank et al., 2014;
2.2 Subjectivity in Argument Mining
Argumentation and, in particular, the human un- derstanding of its quality, is often subjective, con- ditioned by variety of phenomena (e.g., van der Weide al., 2010; Esau, 2018). Only recently has the argument mining community joined other disciplines, such as social sciences and formal argu- mentation, in addressing the backing mechanisms for differing perspectives in argumentation.
Jamison and Gurevych, 2015; Akhtar et al., 2020;
Fornaciari et al., 2021; Cercas Curry et al., 2021; Plepi et al., 2022).
Ajjour et al. (2019) first examined the framing of arguments in order to appeal to specific audiences based their interests, cultural backgrounds, and socialization. Putting a special focus on moral frames, Kobbe (2020) and Alshomary et al. (2022) studied different moral belief systems that arguments are subject to, drawing on the moral foundations theory (Haidt and Joseph, 2004).
A particular way of learning from annotator- specific labels are models which learn to predict in- dividual annotators' decisions. Conceptually, these models can be seen as feature-based models of an- notation (see for an overview Paun al. 2022b) in that they model how each annotator labels indi- vidual examples. However, in contrast to standard models of annotation (e.g., Hovy et al. 2013; Pas- sonneau and Carpenter 2014; Paun et al. 2018), their goal is not to aggregate decisions to a sin- gle label before training but to learn classifiers directly from non-aggregated annotations (Paun al., 2022a). Most work in this area (Raykar et al., 2010; Albarqouni et al., 2016; Guan et al., 2018; Rodrigues and Pereira, 2018) argues for training on non-aggregated labels as a way to deal with vary- ing annotator reliability to better derive the correct labels. Among these, Chu et al. (2021) explore an idea similar our emphasis of common ground in addition to individual variation, modeling annota- tion noise in terms of both individual and common noise. Closely related to our work, more recent studies (Davani et al., 2022; Gordon al., 2022) focus on subjective tasks for which a single ground truth can not always be determined.
Going into more detail on individuals" diverse beliefs and emphasises, Kiesel et al. (2022) utilized computational methods to identify comprehen- sive taxonomy of 54 human values (Searle, 2003) embedded in arguments. In addition, the effect of storytelling on individual perceptions of argumen- tation was addressed by Falk and Lapesa (2022).
Besides the motives behind subjective reasoning. research also considered systems that include mul- tiple perspectives in the output, such as diversity of stances about some claim (Chen et al., 2019). A direct modeling of different perspectives as part of the machine learning process itself, on the other hand, has hardly been considered so far. The only contribution in this direction was made by Romberg (2022), who presented methodology for integrating subjectivity information into con- ventional text classification workflows of argument mining. While that approach involves training a separate classifier to predict a subjectivity value, this work we implement models that directly learn from the non-aggregated labels.
Davani et al. (2022) introduce a multi-annotane model (further explored in Orlikowski et al. 2023; Vitsakis et al. 2023), in particular variant using a multi-task architecture: For each annotator, there is separate classification head trained anno- tations from that annotator. All these annotator layers share pre-trained language model used to encode the input. We evaluate this architecture in our experiments. Gordon et al. (2022) present a model that also predicts individual annotations and
Methodology
We aim at the comparison of different paradigms that can be applied in order to model a classifier for predicting individual labeling decisions in sub- jective annotation tasks. To this end, we define a

---

model spectrum ranging from a purely annotator- specific model design (denying that there is any- thing learnable which is shared among the anno- tators, i.e., the objective grounding) to a purely annotator-agnostic model design which only con- siders the majority vote (denying that there is any- thing learnable which is individual for annotator, i.c. the subjective grounding).
captures the majority perspective (Majority)
3.2 Approaches between annotator-specifie and anmotator-agnostic approaches
For modeling both objective and subjective compo- nents, we compare two different architectures.
Annotator-specifee classification head The first architecture replaces the single classification head of the LLM with set of annotator-specifie clas- sification heads. This architecture equivalent to (multi-task) multi-annotator models introduced by Davani et al. (2022). Because of its separated classification heads, we refer to this architecture as SepHeads in the following to distinguish it from the other approaches modeling multiple annotators.
We also include models that exploit the charac- teristics of both sides of the coin. These hybrid models combine model components that are shared for all annotators well as model components that are specific for each annotator. In this way can explore different architectural design choices along the above mentioned continuum, varying the degree to which the shared labeling behavior of cer- tain groups of users is modeled the architecture.
Recommender--ystem inspired models The second approach, motivated by Gordon et al. (2022), explores recommender systems that incor- porate LLMs. Such systems rely on two encoder blocks, one for the text (using LLM) and one for the annotator. This results in two internal vector representations of the input pair of text and annota- tor ID. To combine these two representations, we use a neural combiner component that performs the final classification. Such recommender style architectures, by encoding annotators by their IDs, can induce representations that generalize across single annotators, thus learning commonalities the behavior of annotators that have a similar label- ing pattern. is in this sense that the recommender architectures can relate perspectives of different annotators.
The proposed model spectrum complements isting taxonomies of learning from disagreement. Uma et al. (2021b) distinguish four categories of approaches, one of them being "learning directly from crowd annotations". Within this category, our proposed spectrum allows to further differentiate how models process individual annotations.
Figure shows our broad bandwidth of ap- proaches, which covers five different approaches to classification: a majority vote model, per-annotator models, and three hybrid approaches, namely a model with -ametftor-specific classification heads, a recommender system with shared text en- coder, and recommender system with annotator- separated text encoders. These will be described in more detail in the following subsections.
In the standard recommender approach, the model contains exactly one pre-trained LLM shared among all annotators. Hence, we call this architecture ShareREC. To explore hybrid ap- proaches emphasizing the more annotator-specific component, we model an option in which each an- notator has their own seperate text encoder. This type of model is referred as SepREC hereafter.
3.1 Two poles: annotator-specific and annotator-agnostic approaches
For the pure annotator-specific and annotator- agnostic approaches, consider pre-trained LLM with standard classification head. The only difference between the two paradigms is the way the dataset preprocessed. In the case the "share nothing" annotator-speccns specific paradigm, the dataset is split annotator-wise, i.e., one split consists of all annotated instances paired with the individual annotation of exactly one annotator. Hence, hav- ing n different annotators, we fine-tune language models, assuming training data for each annota- tor. We call this variant PerAnnotator. The op- posite annotator-agnostcc paradigm considers the full dataset with maajorit--aggregatte annotations, resulting in a single amotator-aggnssee model that
We additionally experimented with further archi- tectures that fit in between ShareREC and SepREC. The methodology and results of these models can be seen in Appendix C.2.
Experiment Design
4.1 Datasets
Due to the fact that work on modeling labeling choices of single annotators is quite recent, so far

---

there are not many datasets available that have re- leased the data in a way that explicitly contains the labels of individual annotators. We base our exper- iments on two such datasets that examine different aspects of argument quality.
gree of subjectivity in the annotation of validity and novelty. While about two-thirds of the annotation samples for validity and half of the annotation sam- ples for novelty are non con-controversial, i.e., there is complete agreement among the annotators, the labels vary more in the remaining cases, which is a manifestation of their subjectivity. In light of the careful selection of annotators and due to the com- prehensive guidelines, we argue it is reasonable assume that the variations in labels are thus due individual perspectives.
CIMT Argument Concreteness Dataset (abbr. Concreteness, Romberg et al., 2022) The German German-language dataset consists of argumenta- tive text units (ATUs) extracted from public partici- pation processes related to traffic planning. These ATUs were categorized into three levels of content- related concreteness: low, intermediate, and high. While ATUs of low concreteness were defined as being vague and lacking specificity, ATUS of high concreteness should provide detailed information.
Given our interest in learning from human label variation, we required samples to be present in the training data for each of the five annotators. As this was not the case for the original split by Heinisch et al. (2022), where two annotators were only introduced in the test split, we re-partitioned the dataset. In doing so, we adhered to the original course of action by avoiding topic overlap between training and the other data, sharing eight of the total 37 topics between development and test data, and introducing seven novel topics in the test data, We also kept the proportions of the original split in terms of the gremise-conclusion pairs included. As result of the non-aggregated approach, the premise-conclusion pairs in the resulting dataset may contain fewer than three labels, as we exclude individual decisions of abstain?.
Each ATU was labeled by five different annota- tors. Released to account for individual annotation behavior, the authors applied rigorous annotation process to ensure that discrepancies were due to subjective perceptions and not due to annotation errors.
In our experiments, we use a split of the dataset that was introduced by Romberg (2022). However, in contrast to the original work, we opted not to repeated k-fold cross-validation to minimize the use of computational resources and energy con- sumption. This decision was based on the reported small deviations in results between different splits.
Table 1 provides a general overview of the two datasets' distribution amongst training (train), de- velopment (dev), and test set, well as the number of classes.
Argument Validity and Novelty Prediction Shared Task (abbr. ValNov, Heinisch et al., 2022) The shared task of the 2022 edition of the Argu- ment Mining Workshop focused on two tasks pre- dicting once the validity and once the novelty of argumentative conclusions. In the corresponding dataset, consisting of English-languee language arguments from debatepedia. .org, validity captures the ex- tent to which a conclusion is justified given its premise. Novelty captures to what extent a con- clusion contains content that is not merely para- phrase of the premise. In both tasks, annotators had binary choice but could abstain if they were unsure. total of five annotators contributed the annotation process, with each premise-conee conclusion pair labeled by exactly three annotators.
Going more into detail regarding annotator- individual labels, the proportion within the splits in ValNov - unlike in the Concreteness dataset - de- pends on the respective annotator. For this reason, Table breaks down how strongly each annotator is represented in ValNov. It shows that the distribu- tion varies greatly, with the predominant annotator covering nearly the entire dataset, while the least represented annotator has assigned labels to only 12% of the examples.
4.2 Experimental Setup
We use the transformers- by Wolf et al. (2020) implement the text processing units (pre-trained LLMs) in all models. We
While the original version of the dataset contains aggregated labels, we disaggregated these labels for the purpose of modeling human label variation. Although not originally designed as dataset allow- ing for the study of the labels of single annotators, Heinisch et al. (2022) already emphasized the de-
'hhe annotator-individual split of ValNov is in- cluded in the GitHub repository: https://githan com/ phhei Detasets/ValNov-nee.sstt

---

Dataset Classes
Train
Dev
Test
4.3
Evaluation Metries
Concreteness 3
788
113
226
How to best evaluate models that do not learn from aggregated labels is an open problem (Basile et al., 2021; Uma et al., 2021b), and various approaches to move beyond majority labels in evaluation ex- ist (Plank, 2022: Leonardelli al., 2023).
ValNov
2
746
203
525
Table I: Overview of the datasets.
Annotator
We follow studies on annotator-leve models in evaluating against individual annotator's deci- sions (Davani et al., 2022; Gordon et al., 2022; Orlikowski et al., 2023), However, instead of cal- culating scores over all individual annotations, we derive scores in two steps: We first calculate the macro-averaged F1 for each annotator separately (i.e., annotator-level scores). As second step, we take the average of annotator scores as model's score. This done for each run of our models, so that the final score is the average of the single-run model scores (i.e., annotator-average scores).
Sub-task
Split
#1
#2
#3
#1
#5
Train
739
489
204
Validity
Dev
203
180
106
75
22
Test
238
266
Train
740
642
495
204
87
Novelty
Dev
118
Test
523
446
240
206
69
Table 2: Annotator distribution among the train/dev/test
split for the ValNov dataset.
use the same LLM architecture (RoBERTa, Liu et al., 2019) for all models and settings. RoBERTa has proven to be effective in the prediction of different aspects related to argument quality (Gureke et al., 2021; Heinisch et al., 2022). We use RoBERTa base variants, namely roberta-base for the English ValNov dataset and (Minixhofer
This two-step calculation has several advantages in our setting. By first calculating per-annotator scores, the final score more appropriately reflects how well models represent each annotator. Ad- ditionally, this method allows us analyze model performance annotator-wise. Based on the range of these scores, we can investigate how much per- formance diverges between annotators.
roberta-base bbberta-base-wechee---eeee al., 2022) for the German Concreteness dataset. Thus, for all models evaluated on the same dataset, the majority of initial weights are equal. During training, we use a class-weighted cross-entropy loss. These class weights are calculated and applied for each annotator separately based on the train split (except for the Majority-mode where we use the majority label for calculating the class weights). Further choices for model components, in particular for different implementation options for the recommender architectures, were deter- mined during preliminary experiments (Appendix C). For further details on the selection of hyper- parameters, the training, and used computational resources, see Appendix A.
5 Results & Evaluation
We provide results for the five models considered along the spectrum of architectures as outlined in Figure 1. We evaluate these architectures two levels using the previously described two-step cal- culation of model scores. In Section 5.1, use the annotator-ate averrag scores order to evaluate mod- cls across individual predictions (suitable for use cases which a global indicator of performance is desired). To have a more fine-grained examination (suitable for use cases aiming to maximize argu- ment quality for individual user) we evaluate against the ovn-averaged annotator-level scores in Section 5.2. See Appendix and for full results and additional recommender configurations.
As described Section 4.1, we use fixed data splits for both datasets (Concreteness and ValNov), having the concreteness task in setting in which all annotators are equally represented and the two tasks in the ValNov dataset in a setting in which the annotated set of instances differs among the annotators (cf. Table 2). We perform ten runs of training and evaluation for each architecture type, using the same random seeds the same order (see Appendix A). We report scores based on averages over individual runs.
5.1 Annotator-average Results
The annotator-aver aveeaae results are provided in Table 3. All models outperform a naive baseline consist- ing of always predicting the most frequent label in the training set: "high" for argument concreteness. "valid" for validity prediction, and "not novel" for novelty prediction, which are the same for an- notators. There are major improvements between

---

+24.7 and +31.69 Fi-points in the concreteness task deciding between three classes and some im- provements between +7.31 and +29.22 Fi-points and between +1.17 and +21.38 Fi-points for bi- nary validity and novelty classification, respec- tively. These diminishing gains over the naive base- line reflect the increasing task difficulty from con- creteness to novelty classification (Romberg et al., 2022; Heinisch al., 2022).
tent to which the models considered can represent the perspectives not only of annotators on average, but all of them. For this, we consider the vari- ability in F1 between the user whose perspective (subjective labeling behavior) is captured best and the user whose perspective is captured the worst (i.e., the range). A large range shows that there are large differences in how well we can represent the subjectivity across users.
Looking at the different architectures along our architectural continuum, the variation regarding the Fi-scores is larger with respect to validity and nov- elty prediction than with respect to the correctness prediction task.
Table 4 shows the results per model in terms of Fi for the annotators whose behavior can be mod- elled best (max) and worst (min), respectively. The highest scores across annotators in terms of min and max are yielded by ShareREC (having Fi-scores of between 54.01 and 62.71 for concreteness, 56.12 to 71.16 and 56.80 to 68.85 for novelty), while the other models at both ends of our architectural spectrum (PerAnnotator and Majority) show poor performance, both on the best and worst annotators. In the concreteness task, where each annotator labeled each sample, we observe that models that include annotator-specifie encoders (PerAnnotator and SepREC) can successfully model the subjec- tive text reading behavior of specific annotators. Hence, the PerAnnotator-modell have a maximum Fi score of 62.88, which is only outperformed by the separated recommenders yielding the overall best single-annotator Fi-score of 67.54, success- fully combining the individual view on text for this annotator with the common ground. However, the individual annotator engagement of those mod- els (including the SepHeads model) involves the risk of overfitting individual trends - showing a comparable high standard deviation between the prediction performances (> 6). Therefore, the best result regarding the minimum annotator Fc-sc is obtained by the more conservative ShareREC, yielding a score of 54.01.
We find that all three tasks show similar patterns: the models at both ends of our architectural con- tinuum (PerAnnotator and Majority) underperform in general compared to the hybrid models. The PerAAnotato-MModee result in 51.57, 11.01, and 41.83 Fi ppoints for predicting concreteness, valid- ity, and novelty on average, respectively. Especially for the latter two tasks where the available training data for some models is reduced to minor amounts (Table 2), which are typically not sufficient to fine- tune model, this approach falls apart. However, using the Majority model does not increase the performance much in validity and novelty (+3.559 and +1.64, respectively) or even reduce the perfor- mance in the concreteness task (-0.75).
Using hybrid approach increases the perfor- mance always with only one exception (SepHeads with 50.83 in the case of concreteness classifica- tion). In all other cases, we successfully relate the different perspectives by capturing the common ground of the tasks by also incorporating (and relat- ing) the individual perspectives of the annotators. Having a medium ratio of annotator-specific param- eters (Figure 1) yields superior performance. In all three tasks, ShareRECC yields the best-averaged F1- scores with 57.83, 65.95, and 62.04 in predicting conereteness, validity, and novelty, respectively, closely followed by the SepREC with annotator- individual text encoders (57.77, 62.08 and 57.03, respectively). The model using separated heads yields Fi-scores of 50.83 (concreteness). 53.82 (va- lidity), and 47.04 (novelty).
For validity and novelty, where each annotator labeled different amount of samples, architectures featuring text encoders that are specific for single annotators fail to reliably predict the behavior of annotators having provided few samples. For ex- ample, the PerAnnotator-modeli have an Fi-score of 34.09 and 39.15 15 for validity and novelty, respec- tively, taking the most underrepresented annotator into account. This performance is worse than the majority baseline (34.62 validity and 45.24 nov- clty). Especially for annotators with sparse labels.
5.2 Minimum and Maximum Annotator-individual Results
"This annotator has the strongest correlation between label decisions and text length. Only ScPRFC. maximizes the Fi- score for this annotator.
Beyond considering only aggregated results in terms of average F,, we also investigate the ex-

---

Concreteuess
Validity
Novelt
discussing controversial examples. We consider in particular an example from the Conereteness dataset in which the full range of possible labels is provided by annotators. Table 5 shows such an example of an argumentative text unit about "bike lane blocked by cars" where two annota- tors assigned the label "high concreteness", two said it was of "intermediate concreteness", and one labeled the example with "low concreteness". Pre- dictions are taken from models trained with the first random seed (see Section 4.2).
Baseline
26.14 0.00
36.73
0.00
40.00 = 0.00
PerAnnolator
54.57 + 1.01
44.04
1 4.71
41.83 + 2.88
SepREC
37.77 + 1.18
62.08
2.25
57.03 + 2.09
ShareRECC
57.83
1.44
65.95
1.66
62.04
0.88
SepHeads
50.83
+: 2.31
53.82
5.79
47.04
+: 4.22
Majority
53.82 1: 1.00
47.63
7.76
43.47 1 2.73
Table 3: Annotator-average Fi-scores and standard de- viation of them for 10 runs.
Concreteness
Validity
Novelty
Min
Max
Min
Max
Min
Max
PorAnnotator
19.
62.88
34.09
56.06
39.15
41.48
SepREC
52.14
67.54
52.59
68.91
50.37
65.73
ShareREC
54.01
62.71
36.12
71.16
56.80
68.85
The Majority model, by definition, predicts the same label for cach annotator. The predicted value, intermediate concreteness, is plausible this case at face value: Examining further examples from the dataset shows a tendeney for short texts to be annotated as less concrete. However, for two an- notators this is an example of high concreteness, despite its brevity. Thus, there is no clear majority label in this controversial case, so this prediction misses three out of five annotators.
SepHeads
44.24
61.09
43.69
61.67
45.36
Majority
47.70
58.86
36.96
54.37
38.16
47.89
Table 4: Highest and lowest annotator-level Fcores for 10 runs.
it seems thus crucial to share the text encoding components of the architecture. Among the archi- tectures that share components, the recommender systems inspired architectures perform best with respect to predicting the labeling behavior of both under- and overrepresented annotators, yielding the highest min- and max-F; scores. Recommenders with a shared text encoder perform between 56.12 12 and 71.12 for validity and between 56.80 and 68.85 for novelty, yielding the best scores for these two tasks.
ShareRFC shows the same prediction pattern as the Majority model. This is in line with the model's s discussed tendency to predict uniform labels per example. The model stresses the common ground based on its shared text representation and picks a plausible label.
The PerAnnnotator and SepHeads models show more diversity in their predictions. This makes sense, as their architectures contain components for more variation isolated from other annotators via separate classification heads or models. This example underlines, however, that variation does not necessarily lead to more accurate representation of annotators overall. Both models, again, only manage to predict two out of five labels correctly. Thus, they are not more accurate than the uniform predictions.
When looking at the goal of catching contrastive views and opinions, for example, to detect "hard cases", further insights considering the predicted agreement reveal a weakness of the ShareREC models. REC models stress the common text understanding and learn to predict better- matchrn majoritics that are appropriate for more annotators, resulting in almost no divergent predicted labels per sample. Recommenders with mmnotator-separalt text encoders (SepRECs) model different perspec- tives much better, having predicted Fleiss' kappa inter-annotator agreement between x 0.73 (nov- elty) and 0.79 (concreteness). However, mod- eling the individual traits of annotators is challeng- ing, especially for complex tasks such as validity and novelty, explaining the overall worse perfor- mance of SepRECs in comparison to ShareRECs in these two tasks.
SepREC, in contrast to all other models, is very close predicting the actual distribution of labels. For the one annotator misses, the tendency of the label (less concrete) is correct. SepREC is thus the only model to predict the class "high concrete- ness" correctly for two annotators. In this example, the model picks up peculiarity of argument con- creteness in public participation related to traffic planning: while generally short argumentative units tend to be less concrete, they might be very con- crete to some annotators. For example, annotators might have specific knowledge about local contexts, such as that there are certain bike lanes in a city
6 Qualitative Analysis: Case Study
The evaluated models show different behaviors that result from how they incorporate individual anno- tators. These differences can best be illustrated by

---

of argument quality, without denying generally ac- cepted objective concepts of argumentation. Still, further work is needed to fully understand the rea- sons for disagreement.
controversial example: "Bike lane blocked by cars"
Human Annotations
1, 1.2
PerAnnotator
0, 1, 0, I,
SepREC
2, 1, 1.2
Sha
1, 1, 1,1
Limitations
SepHeads
1, 0, 1,
Majority
1, 1, 1.1
We have explored different architectures for model- ing human label variation in the subjectíve assess- ment of argument quality. Although tify clear patterns across languages and tasks, our experimental findings are limited to two datasets that address three different aspects of argument quality. This is due to the fact that only very few argument-mining datasets have been released in a way that explicitly contains the labels of individual annotators. While we believe that we have con- sidered the available sources regarding argument quality, we hope to expand the data basis in the future generalize findings.
Table 5: Example from the Concreteness dataset with each model's predictions. Translated from German: "Zugeparkter Radweg". 0 stands for "low concreteness", for "intermediate concreteness", and 2 for "high con-
iden-
creteness".
that are frequently blocked. This observation is in line with SepREC achieving the highest annotator- individual scores on the Concreteness dataset (cf. Table 4).
7 Conclusion & Future Work
Furthermore, the studied datasets each contain annotations from five individual annotators al- lowing to explore our SepREC models where each annotator requires its LLM. Settings with higher numbers of annotators will quickly hit boundaries of computational resources, as we need to instantiate a separate pre-trained LLM per an- notator which has to be stored on a GPU in order to train the entire model in a suitable time span (cf. limitations of ensemble method in Davani et al., 2022). Future work could explore trade-offs between performance and minimizing the loaded LLM size for each annotator.
In this work, we have proposed a general frame- work to investigate the performance of different ar- chitectures on subjective annotation tasks and sim- ilar cases where different legitimate perspectives exist the appropriate labeling decision across notators. We have, in particular, highlighted archi- tectures predicting labels for individual annotators along continuum from fully annotator-specific architectures to architectures that rely only on ag- gregated annotations that completely disregarded individual annotators. Our main focus has been on hybrid architectures along these extreme ends of the continuum that model commonalities in anno- tation behavior across individual annotators.
Our work assumes that the different labeling de- cisions annotators reflect legitimate and valid perspectives. However, it is possible that some dis- agreements are due to unreliabilities in the annota- tion process or lack of good guidelines, potentially leading to misunderstandings of the task (Uma et al., 2021b; Paun and Simpson, 2021: Plank, 2022). Issues related to unreliability of annota- tion processes have been studied exhaustively the context of crowd-sourcing annotations, and countermeasures have been proposed (see for an overview Paun et al. 2018, 2022c). However, the potential trade-offs between ensuring reliability and preserving diverse amnotation behaviors remain less clear. Nevertheless, plausible policies in- crease soundness include pilot studies, cautiously selecting annotators and attention checks. Specifi- cally to identify genuine disagreement due to sub- jectivity, Abercrombie et al. (2023) suggest consid-
Regarding different aspects of argument quality (concreteness, conclusion validity, and conclusion novelty), we have shown that such hybrid architec- tures are best suited to classify the range of opin- ions. Our results show that recommender architec- tures that an annotator-shared LLM for encod- ing the text excel in this setting in terms of max- imizing the Fi Fi-score averaged over all individual annotators up to 13% over a majority-label model. Nevertheless, further analysis going beyond this averaged score has emphasized the importance of having an ianotator-tailore text encoding to cap- ture the different reading nuances in "hard cases", resulting in variance of predicted classes.
Our work provides a starting point for including human label variation into subjective tasks in the field of argument mining, such as the assessment

---

ering the intra-annotator agreement, i.e. checking which extent annotators are consistent with them- selves. While the first three policies can only avoid noise a-priori, the latter one can be also used to filter already existing datasets in cases where anno- tators labeled instances multiple times or multiple similar instances.
variation in natural language processing with intra- annotator agreement. Preprint arXiv:2301.10684
Ehud Aharoni, Anatoly Polnarov, Tamar Lavee, Daniel Hershcovich, Ran Levy, Ruty Rinott, Dan Gutfreund, and Noam Slonim. 2014. benchmark dataset for automatic detection of claims and evidence in the context of controversial topics. In Proceedings of the First Workshop on Argumentation Mining, pages 64-68, Baltimore. Maryland. Association for Com- putational Linguistics.
Lastly, the train/dev/test splits in our datasets are constructed so that we have annotations from each annotator in each subset. Thus, our results do not apply to settings where a) (some) annota- tors contained in training are not contained in test, or, vice versa, b) (some) annotators in the test set are not included in training. However, as can been from ValNov results, annotators can be effectively learned from a few annotations using recommender systems. Hence, an application following these findings could ask a new user to annotate few examples in order to provide predictions that are tailored to this user with minimal effort due to the common ground supporting the decision. Further- more, the user-tailored predictions can be further refined by asking for more annotations. In addi- tion, models providing multiple predictions per in- stance can be used for more fine-grained auto- matic instance-labeling by providing range and distribution of labels instead of one single majority label.
Yamen Ajjour, Milad Alshomary, Henning Wachsmuth, and Benno Stein, 2019. Modeling frames in argu- mentation, In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-LJCNL) pages 2922-200 2932, Hong Kong, China. Association for Com- putational Linguistics.
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2020. Modeling annotator perspective and polarized opin- ions to improve hate speech detection. In Proceed- ings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 151-154.
Shadi Albarqouni, Christoph Baur, Felix Achilles. Vasileios Belagiannis, Stefanie Demirci, and Nassir Navab. 2016. AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images. IEEE Transactions on Medical Imaging, 35 Conference Name: IEEE Transac- tions on Medical Imaging.
Milad Alshomary, Roxanne El Baff, Timon Gurcke, and Henning Wachsmuth. 2022. The moral debater: A study on the computational generation of morally framed arguments. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8782- 8797, Dublin, Ireland. Association for Computational Linguistics.
Acknowledgements
This work has in part been funded by DFG within the project ACCEPT, which is part of the priority program "Robust Argumentation Machines" (RA- TIO), and by the VolkswagenStiftung as part of the "Bots Building Bridges (3B)" project under the "Ar- tificial Intelligence and the Society of the Future" initiative. Julia Romberg is funded by the Federal Ministry of Education and Research of Germany, project CLMTT/Patizipatiionnnttza of the funding Research (funding no. priority Responsibility for the content of this 01UU1904).
Lora Aroyo and Chris Welty. 2015, Truth Lie: Crowd Truth and the Seven Myths of Human Anno- tation. Al Magazine, 36(1):15-25 24. Number: 1.
Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the Ist Workshop on Benchmarking: Past, Present and Future, pages 15-21, Online. Association for Com- putational Linguistics.
publication lies with the authors.
References
Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023, Toward a Perspectivist Turn in Ground Truthing for Predictive Computing. Proceedings of the AAAI Conference on Artificial Intelligence, 68666668
Gavin Abercrombie, Valerio Basile, Sara Tonelli, Ver- ena Rieser, and Alexandra Uma, editors. 2022. Pro- ceedings of the 1st Workshop on Perspectivist Ap- proaches to NLP @ LREC2022. European Language Resources Association, Marseille, France.
Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser. 2021. ConvAbuse: Data, analysis, and bench- marks for nuanced abuse detection in conversational
Gavin Abercrombie, Verena Rieser, and Dirk Hovy. 2023. Consistency is key: Disentangling label

---

AL. Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing, pages 7388-7403, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Melody Guan, Varun Gulshan, Andrew Dai, and Ge- offrey Hinton. 2018. Who Said What: Modeling Individual Labelers Improves Classification. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 32(1).
Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Dallison-Burch, and Dan Roth. 2019. Seeing things from a different angle:discovering diverse perspec- tives about claims. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Pa- pers), pages 542-557 557, Minneapolis, Minnesota. As- sociation for Computational Linguistics.
Timon Gurcke, Milad Alshomary, and Henning Wachsmuth. 2021. Assessing the sufficiency of ar- guments through conclusion generation. In Proceed- ings of the 8th Workshop on Argument Mining, pages 67-77, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Ivan Habernal and Iryna Gurevych. 2017. Argumenta- tion mining in user-generated web discourse. Com- putational Linguistics, 43(1):125-11) 179.
Zhendong Chu, Jing Ma, and Hongning Wang. 2021. Learning from Crowds by Modeling Common Con- fusions. Proceedings of the AAAI Conference on Artificial Intelligence, 35(7):5332-5840, Number: 7.
Jonathan Haidt and Craig Joseph. 2004. Intuitive ethics: How innately prepared intuitions generate culturally variable virtues. Daedalus, 133(4):55-66.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110 110.
Philipp Heinisch and Philipp Cimiano. 2021. A multi- task approach to argument frame classification at vari- able granularity levels. it Information Technology, 63( I) 1755-272
Katharina Esau. 2018 Capturing citizens' values: On the role of narratives and emotions in digital partici- pation. Analyse & Kritik, 40(1):55-72
Philipp Heinisch, Anette Frank, Juri Opitz, Moritz Plenz, and Philipp Cimiano. 2022. Overview of the 2022 validity and novelty prediction shared task. In Proceedings of the 9th Workshop on Argument Min- ing, pages 84-94, Online and in Gyeongju, Republic of Korea. International Conference on Computational Linguistics.
Neele Falk and Gabriella Lapesa. 2022. Reports of personal experiences and stories in argumentation: datasets and analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 5530- 5553, Dublin, Ireland. Association for Computational Linguistics.
Dirk Hovy, Taylor Kirg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120-1130. Atlanta, Georgia. Association for Computational Linguisties.
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2591-2597, Online. Association for Computational Linguistics.
Emily Jamison and Iryna Gurevych. 2015. Noise or additional information? leveraging crowdsource an- notation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 291-297, Lisbon, Portugal. Association for Compu- tational Linguistics.
Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury learning: Integrat- ing dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Hu- man Factors in Computing Systems, CHI '22, New York, NY, USA. Association for Computing Machin- ery.
Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein. 2022. Identifying the human values behind argu- ments. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 1459-4477. Dublin, Ireland. Association for Computational Linguistics.
Shai Gretz, Roni Friedman, Edo Cohen-Karlhs As- saf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A large-scale dataset for argument quality ranking: Construction and analysis. Proceed- ings of the AAAI Conference on Artificial Intelligence, 34(05):7805-7815
Jonathan Kobbe, Ines Rehbein, Ioana Hulpus. and Heiner Stuckenschmidt. 2020. Exploring morality in argumentation. In Proceedings of the 7th Workshop on Argument Mining, pages 30-40, Online. Associa- tion for Computational Linguistics.

---

Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo. Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In Proceedings of the Seventeemth USENIX Confer- ence on Usable Privacy and Security, SOUPS 21, USA. USENIX Association.
Lectures on Human Language Technologies. pages 147-165. Springer International Publishing, Cham.
Silviu Paun, Ron Artstein, and Massimo Poesio. 2022b. Probabilistic Models of Annotation. In Silviu Paun, Ron Artstein, and Massimo Poesio, editors, Statisti- cal Methods for Annotation Analysis, Synthesis Lec- tures on Human Language Technologies, pages 105- 145. Springer International Publishing, Cham.
Elisa Leonardelli, Gavin Abercrombie, Dina Almanca, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma, and Massimo Poe- sio. 2023. SemEval 2023 task 11: Learning with disagreements (L)W. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2304-2318, Toronto, Canada. Association for Computational Linguistics.
Silviu Paun, Ron Artstein, and Massimo Poesio. 2022c. Statistical Methods for Annotation Analysis. Syn- thesis Lectures on Human Language Technologies. Springer International Publishing, Cham.
Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 1-10, Vancouver, Canada. Association for Computational Linguistics.
Comparing Bayesian Models of Annotation. Trans- actions of the Association for Computational Linguis- tics, 6:571 1-55.
Silviu Paun and Edwin Simpson. 2021. Aggregating and Learning from Multiple Annotators. Proceed- ings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Tu- torial Abstracts, pages 6-9, online. Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. Preprint arXiv: 1692.
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Process- ing, pages 10671-10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Benjamin Minixhofer, Fabian Paischer, and Navid Rek- absaz. 2022. WFCCHSEL; Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3992-4006, Seattle, United States. Association for Computational Linguistics.
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with anner-annotator agreement loss. In Proceedings of the 14th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 742-751, Gothen- burg, Sweden. Association for Computational Lin- guistics.
Matthias Orlikowski, Paul Röttger, Philipp Cimiano, and Dirk Hovy. 2023. The ecological fallacy in anno- tation: Modeling human label variation goes beyond sociodemographics. In Proceedings of the 61 An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1017- 1029. Toronto, Canada. Association for Computa- tional Linguistics.
Joan Plepi, Béla Neuendorf, Lucie Flek, and Charles Welch. 2022. Unifying data perspectivism and per- sonalization: An application to social norms. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 7391- 7402, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cecilia Ovesdotter Alm. 2011. Subjective natural lan- guage problems: Motivations, applications, charac- terizations, and implications. In Proceedings of the 49th Annual Meeting of the Association for Compu- tational Linguistics: Human Language Technologies, pages 107-112, Portland, Oregon, USA. Association for Computational Linguistics.
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133-138, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311- 326.
Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Ger- ardo Hermosillo Valadez, Charles Florin, Luca Bo- goni, and Linda Moy. 2010. Learning From Crowds. Journal of Machine Learning Research, 1 1244))12977 1322.
Silviu Paun, Ron Artstein, and Massimo Poesio. 2022a. Corpora. In Silviu
Learning from Paun, Ron Artstein, and Massimo Poesio, editors, Sta- fistical Methods for Annotation Analysis, Synthesis

---

Ruty Rinott, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence an automatic method for context dependent evidence detection. In Pro- ceedings of the 2015 Conference on Empirical Meth- ods in Natural Language Processing, pages 440-450, Lishon, Portugal. Association for Computational Lin- guistics.
Christian Stab and Iryna Gurevych. 2017. Recognizing insufficiently supported arguments in argumentative essays. In Proceedings of the 15th Conference of the European Chapter of the Association for Compu- tational Linguistics: Volume 1, Long Papers, pages 980-990, Valencia, Spain. Association for Computa- tional I Linguistics.
Terne Sasha Thorn Jakobsen, Maria Barrett, Anders Sø- gaard, and David Lassen. 2022. The sensitivity of annotator bias to task definitions in argument min- ing. In Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022, pages 44- 61, Marseille, France. European Language Resources Association.
Filipe Rodrigues and Francisco Pereira. 2018. Deep Learning from Crowds. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). Number:
Julia Romberg. 2022. Is your perspective also my per- spective? enriching prediction with subjectivity. In Proceedings of the 9th Workshop on Argument Min- ing, pages 1 115-15,, Online and in Gyeongju, Repub- lic of Korea. International Conference Computa- tional Linguistics.
Alexandra Uma, Tommaso Fornaciari, Anca Dumi- trache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 202 SemEval-200 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2022) pages 338- 347, Online. Association for Computational Linguis- tics.
Julia Romberg, Laura Mark, and Tobias Escher. 2022. corpus of German citizen contributions in mobil- ity planning: Supporting evaluation through multidi- mensional classification. In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 2874-2883, Marseille, France. European Language Resources Association.
Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021b. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385- 1470.
Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pier- rehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguis- ties: Human Language Technologies. pages 175-190, Seattle, United States. Association for Computational Linguisties.
T. L. van der Weide, F. Dignum, J. J. Ch. Meyer, H. Prakken, and G. A. W. Vreeswijk. 2010. Prac- tical reasoning using values. In Argumentation in Multi-Agent Systems, pages 79-93, Berlin, Heidel- berg. Springer Berlin Heidelberg.
Nikolas Vitsakis, Amit Parekh, Tanvi Dinkar, Gavin Abercrombie, loannis Konstas, and Verena Rieser. 2023. iLab at SemEval-2022 task 11 le-wi-di: Mod- elling disagreement or modelling perspectives? In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2022 2023),, pages 1660- 1669. Toronto, Canada. Association for Computa- tional Linguistics.
Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elis- abetta Jezek. 2023. Why don't you do it right? analysing annotators' disagreement in subjective tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- tational Linguistics, pages 2448-2441, Dubrovnik, Croatia. Association for Computational Linguistics.
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the Web Conference 2021, WWW '21 page 1785-1797, New York, NY, USA. Association forr Computing Machinery.
John R Searle. 2003. Rationality in action. MIT press.
Usman Shahid, Barbara Di Eugenio, Andrew Rojecki, and Elena Zheleva. 2020. Detecting and understand- ing moral biases in news. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 120-125, Online. As- sociation for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of the-ahr art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, Association for Computational Linguistics.
Eyal Shnarch, Carlos Alzate, Lena Dankin, Mar- tin Gleize, Yufang Hou, Leshem Choshen, Ranit Abaronov, and Noam Slonim. 2018. Will it blend? blending weak and strong labeled data in a neural network for argumentation mining. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 599,605, Melbourne, Australia, Association for Computational Linguistics.

---

A 'Trinnin Details, Hyperparameters and Computational Resources
train ShareREC for 20 epochs on concreteness and for 14 epochs on ValNov. Despite these particular choices, we note that the *REC models' develop- ment set performance only changes minimally after epoch 12-14.
In addition to the text-processing units (Section 4.2), also the training loop was implemented using the transformerr--liraay by Wolf et al. (2020). For all hyperparameters not explicitly mentioned we used default settings. Maximum sequence length is 512 tokens, with truncation and padding to the maximum length. We train with an initial learning rate of le 5 for all models. Most experiments use a batch size of 8. The only exception SepREC where we use a batch size of 2 so that the model fits on a single GPU (see below for details on used resources). Each run uses a fixed random seed, most importantly used in weight initialization of the non-pretrained layers: 2923262358,
The majority of parameters in our models be- long to the pre-trained language model. Specif- ically, RoBERTa-base has 125 million parame- ters. For ShareREC these parameters are multi- plied by the number of annotators (five for our setting, so 625 million parameters). We keep the pre-trained model's 's default output dimensionality of 768. Thus, each classification head (in Major- ity, PerAnnotator, and SepHeads) adds 768- 768 768 = 590, 592 parameters for a fully-connected layer and 768 2 + 2 = 1,838 (two classes in ValNov) 768 3 + 3 2, 307 (three classes in concreteness) for a projection layer. Accord- ingly, the added parameter counts for ShareREC and SepREC are higher based on 3 fully-connected layers in the user representation' 's dimensionality (50) and 3 fully-connected layers in the combined dimensionality of user and text.
1842330218, 827634346, 991167630, 1070299506,
171049425,
762227973,
555596930, 1010185121, 419984946
Most experiments ran on a single Nvidia GelFrce GTX 1080 Ti (12GB GPU RAM). Only the SepREC experiments ran on an Nvidia A40 (48GB GPU RAM) because of higher RAM re- quirements due to several LLM instances that need to be loaded simultaneously. Per run, training and evaluation together take on average about minutes for Majority and a single model from PerAnnotator, about minutes for SepHeads, about 10 minutes for ShareREC, and about 50 minutes for SepREC.
B Full Results for All Individual Annotators
Table 6 contains all annotator-individdaa results from all tested architectures and configurations.
C Recommender Architectures
As described in Section 4.2, all mod- cls use RoBERTa (Liu et al., 2019) to encode text, using roberta-base or roberta-base-wechse----
As already depicted in Section 3.2, our approaches relying on recommender systems contain three core components: one or more blocks processing the text standalone (text encoder), one block for pro- cessing the user-id (user encoder) and one block for combining these two encodings and returning the final classification prediction (combiner). For the sake of comparability our other approaches, we fix the text encoder RoBERTa model, having only two degrees of freedom: i) whether there is exactly one text encoder shared among all annota- tors (ShareRFCC and/ or a text encoder separately for each annotator (SepREC) and ii) whether/ how the separated text encoders are connected to each other. We study these hyperparameter settings in Appendix C.2. For our other two components, we implement several single pieces, which can be seen as additional hyperparameters. To encode the user- ids, we can opt between i) one-hot encoder or ii) a simple neural feed-forwar encoder. As options for combiner, we offer a simple feed-forward neural
(Minixhofer et al., 2022) initial weights depending on the dataset language.
ShareREC and SepREC are both based on par- allel DeepCrossNetwork (Wang et al., 2021) (see Appendix C for more context) with 3 layers, ReLU activation and 30 appended feed-forwarded fea- tures. To encode the user, both ShareREC and SepREC use a forward network with 3 layers (embedding size of 50, ReLU activation) trained with dropout of 20%.
Based on the development set performance in preliminary experiments, select different num- ber of training epochs per setting. The Majority model is trained for 10 epochs. For the PerAn- notator approach, each individual model also trained for 10 epochs. SepHeads is trained for 7 epochs. SepREC is trained for 12 epochs. We

---

Concreteness
#1
#2
#3
#4
#5
Baseline
(b) Complex: Fecdforward-Ncural-Y with 26.16 0.00
26.06 0.00
26.19 + 0.00
25.95 + 0.00
26.06 + 0.00
PerAnnotator
$9.98
49.66 2.59
62.88
2.10
2.110
50.60
SepREC
67.54 3.26
52.47 1.81
60.54 1.93
52.14
-6.17
ShareREC
60.23 + 2.05
1.77
62.71 3.08
1.89
54.37 2.39
SepHeads
61.09 + 3.30
16.09 + 3.79
52.92 + 6.06
49.81 3.59
11.24 + 6.73
Majority
38.86 + 1.23
47.70 + 1.75
58.61 + 1.42
53.08 + 1.48
50.88 + 3.25
Validity
#1 37.40 0.00
#2
#3
#4
#5 34.62 + 0.00
Baseline
38.30 + 0.00
36.53 + 0.00
36.82 1 0.00
PerAnnotator
47.59 8.21
56.06 13.32
44.16 10.34
3.75
34.09 + 1.10
SepREC
66.71 1.33
68.91
1.61
3.87
52.59 1.70
ShareREC
67.98 + 1.33
0.89
1.61
1.78
64.68 = 5.81
SepHeads
59.48 + 10.01
61.67
4.07
48.69
+5.49+9.229
Majority
50.47 + 9.17
53.24 10.26
54.57 = 10.54
42.91
36.96 4 5.06
Novelty
#1
#2
#3
#4
#5
Baseline
10.90 + 0.00
34.89 + 0.00
43.10 + 0.00
38.85 + 0.00
15.21 + 0.00
PerAnnotator
2.54
41.26 3.98
2.45
10.19
39.15 12.83
SepREC
55.93 + 2.65
2.46
6
52.22
50.37
ShareREC
56.80 1.44
63.26 1.77
5.10
2.112
68.85 = 2.44
SepHeads
49.88 6.73
48.57
45.90
45.36 5.05
45.50 + 9.63
Majority
43.35 1.82
38.16 2.63
45.96 + 2.19
42.02 = 4.77
47.89 + 4.52
Table 6: Annotator-level macro Fj averaged from 10 runs of training and evaluation the same train/test with different seeds. Separate table for concreteness, validity and novelty, respectively.
net processing the concatenation of text-encoding and user-encoding or all variations of DeepCross- Networks as proposed by Wang et (2021). We explore various settings with shared text encoder in Appendix C.I.
activation-function) and dropout of 20%
2. For Combiner:
Simple: Geedforward-Neural- with layer (no activation function) and dropout of 20%
C.1 Hyperparameter Study for ShareREC
In order explore suitable compositions of our implemented single components, apply grid search using the following selected anchors after an experimental consolidation phase:
Medium: Bertooowwwrr-Nuuua--ee with 3 layers RRel.--accivation--fnicooo and dropout of 20%
Complex: Feedforvarrd-Nerral-eee with layers (Tantl-activationn-fueetooo and dropout of 20%
1. For User-Encoding:
DeepCross: parallel DeepCrossNet- work with layers (Rel.U-acctiation:
(a) Simple:
Feedforward -Neural--ee with 1 layer (embedding size of 25, no activa- tion function) and a dropout of 20%
function), 30 appended feod-forwarded features
3 layers (embedding size of 50, ReLU-
Table presents the results showing that the

---

Combiiee
Concreteness
Validity
Novelyy
+sbared
Concreteness
Validity
Novelty
Simple
Simple
57.41 # 1:06
66.26 m 0.68
61.85 1665=10 1:35
56
61.26 2.05
55.31 3.10
Simple
Medium
56.35
13.54
49.43
0.0
57. 77 = 1.18
62.08 2:25
57.03 4: 2.09
Complex
Simple
57.30 à: 1,36
65.48
1.39
61.00
1.22
0.0
yes
56 + 1.19
66.67 0.94
60.42 +: 2.38
Complexx
Complex
Complex
Medium
58.44
07.90
0.98
$7.90
65.99
15.27
1.68
60.70
54.35
1.98
0.1
B0
57.01
36.77
+ 0.97
62.99 2.08
56.77 1.81
56.86 1: 2.41
Complex
DeapCross
07.83 1.44
65.95
1.06
62.04
57.40 + 1.00
61.40 2:27
56.29 + 2.04
Table 7: Average and standard deviation of per- amno- tator macro F1 from 10 runs for different ShareREC- hyperparmerters
Table 8: Average and standard deviation of per- annota- macro F from 10 runs for the continuum SepREC to ShareREC, À is the value mentioned in Equation 1. 0.0 equals the SepRECC-eeting.
majority of hyperparameter settings yield compara- ble strong performance (concreteness with an Fi- score of = 57, validity with an F1-score of = 66 and novelty with an Fc-sc of = 62), showing the general success of the recommender architec- ture and the importance of the text encoder. How- ever, there are some outliers: the combination of simple user representation and non-simple com- biner do not complement one another well, As observable in Table 7, the himple-user-mddune combiner-setting has the lowest and most unstable scores across tasks, especially for the most complex task of novelty prediction (49.31 points). However, the shallow combination of simple user encoding and simple combiner shows good results and is superior in validity (66.28 points), emphasizing the differences among annotators in the strongest straightforward fashion compared to all other hy- perparameter settings in ShareREC. However, this combination is outperformed by the more complex setting (complex user encoder and complex com- biner) by +1.03 points in concreteness (58.44) and by the final selected setting (complex user encoder and DeepCross-Network) by +0.19 points in nov- elty. Despite have only the third best results w.r.t concreteness (0.61 Fi-points worse than the best setting), underestimating the conflicting views in that task, this model results in stable and overall superior scores for our tasks in argument quality assessment.
weights for the ith annotator and À is hyperpa- rameter regulating the strength of this added loss. While a high positive value of leads to a tight text encoder connection, only allowing minimal annotator-specifie variations, a value next to 0 re- sults in a very loose connection. A negative value would encourage variations between the annotator- specific text encoders. This "smooth" parame- ter sharing is motivated by Heinisch and Cimiano (2021).
In addition, we explore the private-shared ap- proach by Liu et al. (2017) for our text processing unit.
Following the equation 1, we experiment with following values: À [-0.5, 0.1, 1.0, 2.0). We note that 0.5 even more annotator-spect than SepREC since it penalizes similarities between trainable weights of the separated text encoders, emphasizing the differences between annotators. While value of X = 0.1 corresponds approxi- mately to SepREC, X 2.0 emphasizes the com- mon ground regarding the text representation and is thus closer to ShareREC.
In addition, we experimented with introducing an additional shared text encoder alongside the annotator-specific text encoders (called +shared), Table 8 shows the results using complex user encoding and the DeepCross-combing While the most simple scenario in this continuum (the pure SepREC-setting) has the best Fi-score of 57.77 in the concreteness task, where all models be- have similarly (having minimal Fi-scores of 56.93 with À = 0.55,, modifying the interaction of text encoders has a higher impact in the more text- complex tasks of validity and novelty. While the impact of À is still minor, having sweet spot at x= 0.1 with Fi Fi-score between 61.26 (A -0.5) and 62.99 (A 0.1) in validity and Fc-scoses be- (1) tween 55.34 (A = 0.5) and 58.86 (A = 0.1), underlining the "common ground"-aspeca text understanding, the combination of a shared text
C.2 Between SepREC and ShareREC
We additionally explore architectures fitting be- tween ShareREC and SepREC. To this end, we introduce the option to loosely connect each text encoder to each other by adding a further loss term as in Equation! I, penalizing the models if the text encoders diverge (too much).
c EWi W2
Emskies
Hereby W; represents the set of all text encoder

---

encoder and an annotator-specific text encoder per- forms best in validity and novelty prediction (yield- ing an Fi-score of 65.97 and 60.12, respectively). However, we observed the validity task the ten- dency that this model (including shared text en- coder) relies too much on this shared encoding and propagates more the majority opinion than the other models in the SepREC-series. Hence, in terms of capturing different views validity, we recommend using the model without a shared text encoder and À 0.1, having better minimal annotator-Fi-score of 54.31 than the counterpart including the shred text encoder (52.59).
We finally remark that we observe that the im- pact of À vanishes primarily at the start of the train- ing process since the core of each text encoder is large language model which is pre-trained and thus initialized with these pre-trained weights. Hence, future work is to define dynamic À starting with a high value that is slowly decreasing.


Reviews:
[{"review_id": "DQXuNkcz6p", "reviewer": "Reviewer_QBW2", "summary": "", "questions": "How can your findings assist researchers intending to begin an annotation task?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting that our paper is clearly and well-written.  Regarding your concerns on which criteria we have used to select the best architecture, we would like to emphasize that we have opted for a standard evaluation / benchmarking setting, comparing the different architectures on established datasets in the field of argumentation (see prior work by Romberg et al. (2022) and Heinisch et al (2022) as stated in Section 4). We have opted for tasks in the area of computational argumentation and in particular focus on dimensions related to the quality of arguments. The prediction of quality of arguments has been shown to be a challenging task which is subjective, leading to substantial disagreements between annotators. The particular choice of task / datasets regarding the prediction of concreteness, validity and novelty is due to the fact that these tasks feature non-aggregated datasets, that is datasets where the individual annotations of different annotators are explicit. This is not the case for many datasets and explains why for our study we selected those.\n\n---\n\nRegarding your question, _“How can your findings assist researchers intending to begin an annotation task?”_, our findings exemplify the value of using non-aggregated labels in model development for subjective tasks. Hence, to assist researchers intending to begin an annotation task that bears subjective traits we recommend NOT merging all annotation labels into one per sample but providing single annotation decisions for each sample.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The article is very well documented.", "reasons_to_reject": "The criteria used to determine which architecture is best, however, are only loosely explained: concreteness, validity, and innovation.", "questions_to_authors": "How can your findings assist researchers intending to begin an annotation task?", "paper_topic_and_main_contributions": "The article explores the challenges of subjective annotation tasks in natural language processing and proposes several approaches to best represent individual and shared perspectives. The findings show that recommender architectures can significantly improve annotator-individual F1-scores. In this regard, the authors create and describe six architectures. Results show that the best architecture used to model human label variation in subjective annotation tasks is called ShareRec, which is a standard recommender approach that combines two representations using a neural combiner component to perform the final classification and contains exactly one pre-trained LLM shared among all annotators.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "84AyO0Rsho", "reviewer": "Reviewer_XbzC", "summary": "", "questions": "1. Why didn't you consider 'soft label' approaches? ", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comprehensive review and pointing to some missing related work. We will include these references and discuss in more depth how the architectures investigated by us relate to other architectures in the papers you mention. \n\nRegarding the novelty of and main contribution of our paper, we can confidently claim that so far there has not been a systematic comparison of different approaches along the continuum we propose on the same datasets. This is the main contribution of our paper and we believe that the results showing the superiority of the recommender-based architecture are interesting, and relevant, and novel. In particular, recommender system-inspired architectures have so far not received major consideration in the field of prediction of quality of arguments. Introducing text-separated encoders (or the variant of loose connection as proposed in Appendix C) as a novel explored point on the proposed continuum, too. In this sense, we think that our results will inspire and help others in their choice of methods and future investigations regarding the ability to predict labels for specific individuals, demographic groups, etc.\n\n---\n\nRegarding your suggestion of considering „soft labelling“ approaches, we think this is a fair suggestion and comment. Certainly, soft-labeling approaches, given that they predict a distribution of labels, could be incorporated into our continuum between approaches predicting a „majority label“ and approaches that incorporate annotator-individuals-specific layers into a model. Conceptually, a „soft label“ approach is comparable to a „majority labelling“ approach if one would use maximum a posteriori inference (MAP), that is choosing the most probable label. Alternatively, one would have to sample proportionally to the distribution. As we evaluate on the level of predictions of labels for single individuals in our framework, both methods are conceptually close in that they make aggregate non-individualized predictions that need to be extrapolated to single individuals. While we thus agree that we could have included „soft labelling“ approaches in the comparison and this could be seen indeed as a weakness of our paper, we do not expect substantially different conclusions and think that the comparison as it stands already is a significant contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Applying techniques for training models while preserving disagreement to argument quality task extends the range of applications in which such models have been tested\n\n2. I like the idea that such models range in a continuum, although see below\n\n3. The key finding - that 'hybrid' models combining annotator-specific with group-derived information achieve the best performance - is plausible and in keeping with the most successful approaches to 'learning from crowds' - see, e.g., chapter 6 of\n\nS Paun, R Artstein, M Poesio (2022). Statistical methods for annotation analysis. Springer Nature.\n\n   (although again, see below)\n\n4. The paper is very well written.", "reasons_to_reject": "1. similar taxonomies of approaches for learning from disagreeement have been proposed, and more in general comparisons between different approaches on the same have been made, in a number of previous papers, such as Uma et al 2021 that the authors cite:\n\nAlexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021b. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385– 1021.\n\n   the paper would benefit from some explanation of how the analysis proposed here differs from earlier work other than the different application\n\n2. The proposed model with separate encoders for text and annotators that then alternatively feed into a shared classifier or separate classifiers for each annotator is elegant, but highly reminiscent of a variety of 'learning from crowds' approaches also separately modelling text and annotators, it would be good to have some discussion of the advantages of this particular model.\n\n3. Strangely, the authors do not consider a class of 'hybrid' models which have been shown to be highly effective both in  Uma et al 2021 survey  and  in the recent SEMEVAL Le-Wi-Di shared task (Leonardelli et al, 2023) - namely, the 'soft label' approaches in which the annotator votes are merged but preserving the existence of differences through a probability distribution. ", "questions_to_authors": "1. Why didn't you consider 'soft label' approaches? ", "paper_topic_and_main_contributions": "This paper explores the impact  of different architectural choices in designing models that can be trained with data containing  subjective variance between annotators, looking in particular at the task of argument quality. The authors argue that such models can be organized along a continuum from models in which labels are aggregated ('annotator agnostic') to models in which every annotator is explicitly modelled ('annotator specific').\n\nThe main contributions are\n\n1. the general idea of a continuum from annotator agnostic to annotator specific models\n\n2. the application of these methods to the area of argument quality\n\n3. the finding that 'hybrid' models seem to work best in this setting. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is very well written. "}, "extra_text": {"Missing_References": "1. Multi-annotator models predate Davani et al - they have been widely  used in work on modelling with subjective tasks such as offensive language detection by, e.g.,  Akhtar and Basile\n\n2. The survey of Uma et al 2021 and the book by Paun et al 2022 discuss a number of highly successive alternative 'hybrid' approaches adding a common backup to learning from crowds models that are not mentioned here, such as\n\nZhendong Chu, Jing Ma, and Hongning Wang. Learning from crowds by modeling\ncommon confusions. Proceedings of the AAAI Conference on Artificial Intelligence, 35(7):5832–5840, May 2021. URL https://ojs.aaai.org/index.php/AAAI/article/view/16730."}}, {"review_id": "ZV2YRWLN06", "reviewer": "Reviewer_kkvu", "summary": "", "questions": "The adopted evaluation strategy computes per annotator metrics which are then averaged to get the model scores. \nA. How is this in support of training these models to take decisions? How would you use one of these models in practice? I.e., how would you interpret their output?\nB. How is noise being handled by the different methods you considered? (many annotated corpora are quite noisy)", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and honoring the quality of our study itself. Also, thank you for the references to improve our related work section.\n\nRegarding the limited scope and novelty: the goal is to explore learning from disagreement in the domain of argument quality, for which it is widely recognized that different people may perceive argument quality differently and yet validly. Despite this generally prevailing consensus, classification models in the field of argument mining have so far (except for the work of Romberg, 2022, which proposes a soft approach to consider a subjectivity score, and thus does not learn directly from the non-aggregated data) been considered from an aggregate perspective. In our study, we rely on the existing non-aggregated datasets for argument quality available at the time of submission.\n\nIn our study, we, therefore, decided to give our full attention to this particular area first, since \"learning from disagreement\" is of paramount importance to the field of argument mining, but has surprisingly received almost no attention. Nevertheless, although the domain is the same, the datasets differ in language (German vs. English) and discussion topics (urban development vs. well-known debate topics such as abortion).\n\nIt is true that we don’t invent an entirely new approach here. However, placing these different architectures in such a continuum is novel, consistently evaluated on the annotator level. We think that our results will inspire and help others in their choice of methods regarding the ability to predict labels for specific individuals. In addition, some combinations of modules and ideas are novel, too, as the SepREC (introducing individualistic text encoders) with its connection variants presented in Appendix C.\n\nThe review mentions _“different sets of annotators with varying amounts of workloads across the train and test data”_. In order to have a clear study, we rely on datasets with a not too large amount of annotators. It is true that all annotators annotated all samples in the Concreteness-dataset to cover this scenario, but the ValNov-dataset varying amounts of workloads across the train and test data, see Table 2. Here, the total annotated samples as well as the proportions in train/dev/test differ.\n\n---\n\n# Regarding the questions:\n\n> A. How is this in support of training these models to take decisions?\n\nI’m not sure whether I understand the question correctly. In training, the (class-weighted) cross-entropy loss is calculated over each annotator prediction separately (with the exception of the Majority model). This training is not tailored to the metric but rather general.\n\n> “How would you use one of these models in practice? I.e., how would you interpret their output?”\n\nOur framework has the great advantage of flexible usage. Besides being used in line with the still emerging perspective view of ground truth (see Cabitza et al., AAAI 2023: Toward a Perspectivist Turn in Ground Truthing for Predictive Computing), it can also be applied for conventional single-ground truth machine learning by aggregating the annotator-specific predictions.\n\nFurthermore, our approach can be used for cost-efficient personalized decisions: In our experiments, we could show that the recommender system-inspired architectures were capable of predicting individual preferences with just a few training samples, presumably due to the shared understanding of basic concepts of argumentation among different annotators.\n\n> “B. How is noise being handled by the different methods you considered? (many annotated corpora are quite noisy)” \n\nDisagreement or human variation in labels is often referred to as noise in the conventional sense. In our work, however, this kind of \"noise\" is the basis on which we work (following Plank, EMNLP 2022: The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation), accepting that there may be several valid labeling choices when it comes to judging the quality of arguments.\n\nHowever, when noise refers to an unclean annotation process (i.e. due to unreliable annotators), we believe that this is a very universal problem in supervised machine learning, regardless of whether aggregated or non-aggregated data is involved. In our study, we assume that the quality of the data is ensured upfront, as is the case in the datasets we use (see Section 4).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors find a bi-encoder model of the text and annotators to consistently get the best results across their evaluation settings. Reasonable baselines were used as well to make the paper a good study.", "reasons_to_reject": "Limited scope and novelty -- the authors conduct their study on two datasets from the same domain, using existing approaches. Also, to make the 'learning from multi-annotated corpora' strategies practical they should support different sets of annotators with varying amounts of workloads across the train and test data.", "questions_to_authors": "The adopted evaluation strategy computes per annotator metrics which are then averaged to get the model scores. \nA. How is this in support of training these models to take decisions? How would you use one of these models in practice? I.e., how would you interpret their output?\nB. How is noise being handled by the different methods you considered? (many annotated corpora are quite noisy)", "paper_topic_and_main_contributions": "The authors experiment with different strategies to train neural models on multi-annotated corpora covering two argument quality classification tasks. The strategies vary from training annotator independent models to 'hybrid' models that share some of their parameters between the modelled data and annotators.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The learning from disagreement field is vast. Useful background knowledge should cover agreement statistics, models for annotation aggregation, and lastly, learning from multi-annotated corpora methods. A good survey of this area with many of useful references and resources is the following book:\nPaun, Silviu, Ron Artstein, and Massimo Poesio. Statistical methods for annotation analysis. Springer Nature, 2022.\n\nOther useful resources you may want to check:\nAggregating and learning from multiple annotators, EACL 2021 tutorial\nSemEval-2023 Task 11: Learning With Disagreements (LeWiDi)\nA crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation, NAACL 2019\nComparing Bayesian models of annotation, TACL 2018"}}]

Author rebuttal:

Meta-review: This paper explores the impact of different architectural choices in designing models that can be trained with data containing subjective variance between annotators, looking in particular at the task of argument quality. The authors argue that such models can be organized along a continuum from models in which labels are aggregated ('annotator agnostic') to models in which every annotator is explicitly modeled ('annotator specific').

The shared perspective of the reviewers wrt this paper include:
* the paper is well-written and the approach is sound and explored with reasonable baselines
* the contribution of a continuum of annotator agnostic and annotator specific models and the findings on which models yield best results are interesting and valuable
* the novelty of the proposed approach is not clear 

As the approach is accepted to be quite sound and there are no scientific or technical issues raised in the reviews, I tend to base my score strongly on this. This is encouraged also by the interesting results provided and the potential application to other tasks. The issues of novelty and comparative discussions wrt existing approaches seems one that can be resolved through elaboration in a camera-ready (as done so in the author response).
Decision: Accept-Main