Prefer to Classify: Improving Text Classifier via Pair-wise Preference Learning
Authors: Jaehyung Kim
Jinwoo Shin
Dongyeop Kang
Venue: Submitted to ICLR 2023 (2023)
Keywords: NLP
text classification
annotation
disagreement
preference

Abstract: The development of largely human-annotated benchmarks has driven the success of deep neural networks in various NLP tasks. These benchmarks are collected by aggregating decisions made by different annotators on the target task. Aggregating the annotated decisions via majority is still used as a common practice, despite its inevitable limitation from simple aggregation. In this paper, we establish a novel classification framework, based on task-specific human preference between a pair of samples, which provides an informative training signal to capture fine-grained and complementary task information through pair-wise comparison. Hence, it improves the existing instance-wise annotation system by enabling better task modeling from learning the relation between samples. Specifically, we propose a new multi-task learning framework, called prefer-to-classify (P2C), to effectively learn human preferences in addition to the given classification task.
We collect human preference signals in two ways: (1) extracting relative preferences implicitly from annotation records (for free) or (2) collecting subjective preferences explicitly from (paid) crowd workers. In various text classification tasks, we demonstrate that both extractive and subjective preferences are effective in improving the classifier with our preference learning framework. Interestingly, we found that subjective preference shows more significant improvements than extractive preference, revealing the effectiveness of explicit modeling of human preferences. Our code and preference dataset will be publicly available upon acceptance.

Under review as a conference paper at ICLR 2023
PREFER TO CLASSIFY: IMPROVING TEXT CLASSIFIER VIA PAIR-WISE PREFERENCE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The development of largely human-annotatie -nnotated benchmarks has driven the success of deep neural networks in various NLP tasks. These benchmarks are collected by aggregating decisions made by different annotators on the target task. Aggregating the annotated decisions via majority is still used as common practice, despite its inevitable limitation from simple aggregation. In this paper, establish a novel classification framework, based on task sssc-spccffic human preference between a pair of samples, which provides informative training signal to capture fine- fine-grained and complementary task information through pair-wise comparison. Hence, it improves the existing instance- wise annotation system by enabling better task modeling from learning the relation between samples. Specifically, we propose a novel multi-task learning framework, called prefer-to-classify (P2C), effectively learn human preferences in addition to the given classification task. We collect human preference signals in two ways: (1) extracting relative preferences implicitly from annotation records (for free) or (2) collecting subjective preferences explicitly from (paid) crowd workers. In various text classification tasks, we demonstrate that both extractive and subjective preferences are effective in improving the classifier with our preference learning framework. Interestingly, we found that subjective preference shows more significant improvements than extractive preference, re- vealing the effectiveness of explicit modeling of human preferences. Our code and preference dataset will be publicly available upon acceptance.
INTRODUCTION
The recent success of natural language processing (NLP) systems has been driven by, among other things, the construction of largely human-annotated benchmarks, like GLUE (Wang et al., 2019) or SQuAD (Rajpurkar et al., 2016). Nevertheless, current NLP benchmarks often include problems occurring in their construction process, such as annotation artifacts (Gururangan et al., 2018) or spurious patterns (Kaushik & Lipton, 2018). To alleviate these issues, various approaches have been recently proposed to construct more robust and effective benchmarks via human-in-the-loop with model (Kiela et al., 2021; Yuan al., 2021; Liu et al., 2022) or adversarial sample mining (Kaushik et al., 2020; Nie et al., 2020; Potts et al., 2021). Despite such a careful selection of samples to annotate, is relatively under-explored how to aggregate the annotations and assign the label, fully exploit the advantage of these benchmarks.
For example, most NLP data collection still follows a long-standing custom for annotation voting called majority voting (Snow et al., 2008; Hovy et al., 2013) that aggregates multiple annotators' judgments into majority-voted ones. This labeling with majority voting, however, inevitably discards the valuable information embedded in multiple annotators' assessments and their disagreements, such as the inherent difficulty of instance (Pavlick & Kwiatkowski, 2019) or uncertainty from the task's subjectivity (Alm, 2011). 1). As the modern NLP systems extending the interest to greater variety of social issues and subjective tasks (Uma, 202 1), such as humor detection (Simpson et al., 2019) and racist language detection (Larimore et al., 2021), the capability of modeling the fine-graie ine-gnaieed, distributional opinions from multiple annotators becomes more important.
To address the limitation of the simple labeling method, various approaches have been recently proposed, such as label smoothing (Fornaciari et al., 2021: Leonardelli et al., 2021). However, they still limited by the discretized annotation space and the limited number of annotators, resulting in

---

Under review as a conference paper at ICLR 2023
Sentiment: Posilive
Preference
Distance Apnotations
Task Labuls
Preference Lace's
Expacted Calibration Entor
6.8
I came here and ordered excaptional.
70-
Text
0.6-
Accuracy
8
Toxt
Mario and Carlos cut our oxactly the property
0.2
0.0
55
Tedl Labee
Prrforette Laben
Test
(a0g)
Aco (hard)
(a) Pair-wise preference signals
(b) Alignment to human annotations
(c) Improved text classification
Figure 1: (a) Example of pair-wise preference in the sentiment classification. (b) Effect of preference learning. makes the classifier capture the fine- ane-grained task information; e.g., predictions of classifier become more aligned with human annotations. Test samples are divided into Hard, Normal, Easy based on the annotators' disagreement. (c) Improvement from the collected preference and P2C in various aspects, e.g., better accuracy and calibration. More results are presented in Section 4.2.
coarse-graine saasse-grained modeling of the task. Hence, it inspires us to investigate a new and complementary direction for capturing fine-grained task information, by relatively ordering a pair of two texts and better calibrating them with respect to the task, using human preference.
Contribution. In this paper, we establish
classification framework based human preference between pair of samples, e.g., which text is more positive for sentiment classification (see Figure (())), in addition to the task annotations. Learning with human preference has been demonstrated in multiple domains, including reinforcement learning (Christiano et al., 2017) and generative models (Ziegler al., 2019), by training the model to follow human behavior and achieve the complex goal with better modeling of the task. While this direction is under-explored in the classification regime, could provide an informative training signal by capturing the complementary task information through "pair-wise' comparison that cannot be captured with 'instance- wise" evaluation (see Figure 1(b).. Hence, it would effectively improve the classifier, as shown in Figure 1(c).
Specifically, propose novel multi-task learning framework, coined prefer-to-classe (P2C), to effectively train the model from both classification and preference learning tasks. We introduce diverse multiple preference heads beside the classification head of the model to learn from preference labels. Then, we apply a consistency regularization between them for imposing the model to have higher confidence in classification with the preferred samples. We also develop advanced sampling schemes to select more informative text pairs during the training.
To train P2C, collect types of human preference labels: extractive preference and subjective preference. Extractive preference is constructed from the existing annotation records in datasets 'without additional cost'; if one sample has been less voted than the other, we treat the latter relatively higher preference between the two samples. One may argue that the extracted preferences are somewhat artificial (yet for free) and implicit signals, as they are not obtained from direct comparison by human. To alleviate this, we also collect subjective preferences for 5,000 pairs of texts from (paid) crowd workers by directly asking them which text more preferred the task label.
We demonstrate the effectiveness of preference learning via P2C in addition to given task-specific labels, on both extractive and subjective preference labels. In six text classification datasets, P2C with extractive preference exhibited 7.59% and 4.27% relative test error reduction average, compared to the training with majority voting and the previous best method to learn using annotation records, respectively. Moreover, newly- newly-collected subjective preference labels show clear advantages the extractive ones, not only with the improvement in task performance but also with better calibration and task modeling; for example, 6.09% of expected calibration error while 9.19% from the same number of task labels. Overall, our work highlights the effectiveness of pairwise human preference for better task learning; we suggest that NLP benchmarks should include annotation records, instead of just providing majority-voted labels, or collect human preferences.
IMPROVING TEXT CLASSIFICATION VIA PREFERENCE LEARNING
In this section, we present classification. Our main idea is take advantage of the preference between input samples to
(P2C), a new preference learning framework for text
2

---

Under review as a conference paper at ICLR 2023
train the classifier by providing the complementary learning signal from pair-wise comparison that cannot be captured from the instance wise task labels. In Section 2.1, we describe the problem settings and preliminaries. Then, in Section 2.2, elaborate on the specific components of P2C for learning from human preferences. Figure 2 presents the overview of our framework.
2.1 PRELIMINARIES
Problem description. We describe the problem setup of our interest under text classification scenario with K classes. Let D denote the given training dataset consisting of tuples (x, Utaak) E D where 1- xT,] is the sequence of input tokens X;, and 1/task is the target task label. Our goal is to train a classifier fa := W Wxeex o 9d. composed with Transformer- based language model backbone Io (e.g. BERT (Devlin et al., 2019)) and a random initialized classification head W taez, to minimize the task-specific loss Lrasx such as a cross-entropy loss where p(x) Softmax (x)( The common standard to determine the target label is one-hot labeling to the majority voted one i.e. Vtack E [0. 1]" which aggregates multiple annotations on one single sample. To address the limitation of the majority voting, different labeling approaches have recently been explored by utilizing annotators' disagreements for the training, such as soft-labeling (Fornaciari et al., 2021).
Preference learning. In this paper, we use human preference between two data instances complementary learning signal to train the classifier. Specifically, the preference signals reflect the relative suitability between the two input samples concerning the given task. We assume that the human preferences, Le., preference labels, of the given dataset are available.' Then, our goal is training preference predictor to learn from the given human preferences, by predicting which one among the two input samples is more preferred. To this end, we formulate a preference learning as a supervised learning problem following the approaches in other domains such reinforcement learning and generative modeling (Christiano et al., 2017; Ziegler et al., 2019; Lee et al., 2021). Given a pair of two different input tokens (x", x²) and task label Btask, preference label yprer is additionally given; it indicates which input is preferred considering Vtask. i.e., Sprer E {0, 0.5), where I indicates x > x" (i.e., x² is preferred than x"), O indicates x > x², and 0.5 implies an equally preferable case. Each preference label is stored in dataset D quadruplet (x", x² "yaakky Upret). ). Then, predict a preference using the preference predictor he following (Bradley & Terry, 1952):
exp (lo (2taa)aa))
Pa(x' x²; ;tank]] Exe exp (x),axx)ra)
where x² > x implies that input i is preferable to input j. The underlying assumption of this model that the probability of preferring an input depends exponentially on its output. Then, the preference predictor he is trained through supervised learning with the given human preferences, by minimizing the binary cross-entropy loss as follow
Sprot
Uprot log Pyo (x² x"; [task) + -Upret)))))))) log Po [x(xx > x²: |rasx] (2)
(x" xet Meazaa (pous) D
2.2 PREFER-TO-CIASSITY (P2C)
Next, we present our specific techniques to train the classifier with preference labels: (a) multi-task learning of classification and preference learning, (b) consistency regularization between classification and preference learning, and (c) informative pair sampling using disagrecment or inconsistency.
Multi-task learning with preference labels. To effectively learn from the given preference label Upref and the task label Yrask. we train the classifier fo via a multi-task learning (Ruder, 2017; Sener & Koltun, 2018) of both classification and preference learning. Specifically,
model the preference predictor hy in Eq. upon the classifier fa similar to the case of Wtasx- The preference prediction head W prat added on the output of Transformer backbone (x(x) and task label ytask. i.e., h Blesx)
W Wpret= o (x); \taax] where fx( Wtask (x).
'See Section for different practical scenarios how preference labels collected.
"Equally preferable case is learned by imposing the same coefficients (yprwt = 1 Uprat = 0.5) in Eq. 2.
"Igo (x): (taez] means the concatenation between go and Ytask-
3

---

Under review as a conference paper at ICLR 2023
Training Samples
Classification Head
Text
Text
Consistency Regularization
Multi-task Learning
Language Model
Text N
Selecting Informative Pairs
Disagreement Modeling with Mult-prreference Heads
Figure 2: Illustration of Prefer-to- taassss (P2C) framework.
Preference modeling with diverse multi-preference heads. In addition, we introduce multiple prat 17 to fully exploit the given preference labels, through the advantage of classifier ensemble (Ganaie et al., 2021). Each preference head is independently random initialized and trained with Sprat in Eq. Since the different initialization limited to impose the diversity between (Wr(+) prediction for each preference head by maximizing KL-divergence between them (Wang et al., 2021):
preference heads (Wrlt)
prat }7=, during the training, we add regularization Laiv to encourage the diverse
Laiv
DEL (P pe (x x²: Yeack) (| P (x+,xx x Veaex))
(3)
T
where (x x"; Brask) is the predictive distribution of the preference predictor hg. i.e. (x x"; (taax) = [x] > x": Utask), Px(x" > x Utaxx]] Overall, we train the classifier with the following multi-task learning objective Laulti under hyper-parameter Aaivi
Lmuits = Lrask + Cpret + dasvLaiv
where Gpear indicate the preference learning objective with each head v)(0) and Call -E
preet
Consistency regularization between classification and preference learning. Even though multi- task learning is effective way train the model, it is still unclear whether not the model can capture the relations between the two tasks explicitly. Accordingly, we hypothesize that a more preferred instance should have higher confidence from the classifier, i.e., Py (x*) Py (xx) if x' x" with the given task label y. Hence, impose the model explicitly follows this intuition, further propose a consistency regularization between the two tasks as follow:
Lcone = Ypref ma Py Py (x²)} + (1 - Vpres) max (0, (x()x) (x('')}
(5)
Additionally, when the degree of preference explicitly provided, i.e., Uprart E [0, 1] (see Section 4.1 of extractive preference case) rather than Uprat {0, 1, 0.5), we further extend this consistency regularization with margin m which represents the degree of preference:
Lcome lppeef max 0,
(xy xx)} + 3pres) max (0, (p, x²) m)
(6)
where Ap, x²)
(xx) Py (x') We note that the previous consistency regularization Eq. becomes the special case of Eq. with 0. Overall, our training loss of the classifier follow:
Lursia Lauats + Aeous
where Acona is hyper-parameter.
Selecting informative pairs. As the number of pairs of samples (x"; x-) proportional the square of the number of training samples, it is difficult to obtain the preference label for all possible pairs, and even harder to learn from them even if we have all the preference labels. Hence, we propose the following advanced sampling scheme to maximize preference learning's effectiveness during training: (1) Disagreement based sampling, which selects pairs of instances with high variance across multiple preference predictors {haee N= and (2) inconsistency-base sampling, which seeks to reduce the mismatched pairs with high consistency loss Lcona in Eq. 5, We evaluate the effects of these sampling methods in Appendix C.
4

---

Under review as a conference paper at ICLR 2023
Table I: Examples of the collected extractive and subjective preference sets.
A: I got 3 veggies and side of fries for over 11 dollars if you like homecooked fond
B: She listened to my ideas, asked questions to get better idea about my style, and was excellent at offering advice us if were lotal pleb.
Sentiment: Positive, Extractive Preference: A
B. Subjective Preference: B > A
A: This was the best movie. Movie was entertaining but not as good as the original
The restaurant was not busy. but everything was ready for big crowd if needed. We love this place.
B:
Sentiment: Positive, Extractive Preference: No preference. Subjective Preference: > A
A: Drove 15 miles out of my way for them to tell me on the speaker at 9:58 it's closed. am pissed
B: Wow, the service was like I was staying at San Quentin for 2 to 4 years.
Sentiment: Negative, Extractive Preference: B
A, Subjective Preference: B
A: never get for the holiday.
The casino was full, with the slots hitting.
Sentiment: Neutral, Extractive Preference: A
B, Subjective Preference: B > A
COLLECTION OF PREFERENCE LABELS
In this section, we provide two different ways of collecting human preferences to prepare the ground-truth preference labels for our training: extractive and subjective preferences.
3.1 EXTRACTING IMPLICIT PREFERENCES FROM DATA ANNOTATION RECORDS
The first method is to extract human preference signals from the existing datasets without extra cost. Our high- level assumption is that annotation records of each data, which are naturally gathered during the construction of dataset, implicitly encode the preference between data samples. For example, if one sample has higher voting (9 out of 10), i.e,, less disagreement, than the other sample (6 out of 10), we assume that the former has a relatively higher preference. We call this implicit human preference as extractive preference. Since the extractive preference is derived from existing sources of data, it can be obtained for any pair of samples without additional cost.
Extractive Preference
Sentiment: Positive
Subjective Preference
came here and ordered and exceptional.
Test more posttie
Text
2ediiis thai Torl
Mario Carlos maat exactly the way we wanted the properly
Ted postive 231
Ansonation Records
Tour
Crowd werkers
Figure 3: Comparison between extractive and subjective preference labels.
3.2 COLLECTING SUBJECTIVE PREFERENCES FROM CROWD WORKERS
Although the extractive preferences are easy to obtain and intuitive, they are sub-optimal since they not obtained from the direct comparison of the pair of sentences by human labeler. Hence, to the best of our knowledge, we collect the first preference dataset for the text classification. Our dataset is collected based paired samples from DynaSent-R2 R2 dataset (Potts et al., 2021) for sentiment classification task. To be specific, we gather the subjective preference of the pairs by asking crowd workers answer "which sentence is more positive (neutral, negative)?" using Amazon's Mechanical Turk crowd-sourcing platform (Crowston, 2012). Then, each worker should select one of the two sentences or answer "No Preference". Following (Nie et al., 2020), we initially provide each pair of sentences to two crowd workers. If two workers give the same preference label, this pair is labeled with that. If they disagree, we ask third crowd worker break the tie. If they still fail to reach a consensus, this pair is labeled with "No Preference".
Under this procedure, we first gather 1,000 subjective preference labels of randomly selected pairs of sentences. Then, we dynamically collect the additional subjective preference labels to maximize the information of collected pairs, motivated by the recent dynamic benchmark constructions (Kiela et al., 2021; Nie et al., 2020). Namely, we first train the model with existing subjective preference labels. Then, we find the most informative pairs the aspect of trained model, using the disagreement- sampling introduced in Section 2.2 and query their preference labels in next stage. We select an equal number of pairs for each class to balance the label distribution. Overall, starting with 1,000 random pairs, collect the preference of 2,000 pairs each round and iterate this procedure for rounds, f.e., a total of 5,000 pairs' subjective preference labels are collected. The used interface for gathering the preference label and more detailed statistics are presented in Appendix B. Also, Table shows
5

---

Under review as a conference paper at ICLR 2023
some examples in our extractive and preference sets. Here, the advantage of subjective preference compared to extractive one could be observed by providing more accurate pair-wise preference.
4
EXPERIMENTS
In this section, validate the effectiveness of the proposed P2C with the extractive preference labels (for free) in Section 4.1 and the subjective preference labels (payed) in Section 4.2, respectively.
4.1 EXPERIMENTS WITH EXTRACTIVE PREFERENCE FROM ANNOTATION RECORDS
Datasets. To extract preferences from the annotation records, we extensively investigate the publicly available datasets providing such information and use the following six text classification datasets. All datasets have the annotation records from 5 annotators for each sample. DynaSent (Potts et al., 2021) is sentiment classification benchmark with temary (positivennegativennuuaal sentiments. is dynamically constructed through multiple iterations of training a classifier model and finding its adversarial text inputs. We use the dataset from the first round, (1) DynaSent-RI, and from the second round, (2) DynaSent-Ri R2, the dataset for experiments. Standford politeness corpus (Danescu- ficulescu-Mizil et al., 2013) is a binary classification benchmark for predicting whether the given sentence is polite impolite. Since there are two different input domains within this benchmark, we split them into two different datasets: (3) Polite- Wiki from Wikipedia, and (4) Polite-SE from StackExchange, following the original paper. Offensive agreement dataset (Leonardelli et al., 2021) is a binary classification benchmark for predicting whether the given sentence is offensive or not. Since some of the original samples are not available anymore publicly, we only utilize the available samples while keeping the setups of original dataset: (5) Offensive. (6) MultiNLI (Williams et al., 2018) is crowd-sourced collection of sentence pairs annotated with textual entailment information. As the only validation set includes the annotation records, we split it into 8:1:1 for training, validation, and test sets. Detailed description and pre-processing procedures are presented in Appendix A.
Baseline methods. We first compare the proposed P2C to a naïve training using majority voting without consideration of disagreement, denoted by (a) Vanilla. Then, since our method with extractive preference can be viewed as new way to utilize the annotators' disagreement from the annotation records, we compare P2C with a wide range of disagreement learning methods, as listed as follows; (b) Soft-labeling (Fornaciari et al., 2021): using the probabilistic distribution of annotations soft labels for training; (c) Margin (Sharmanska et al., 2016): training the model with hinge loss by setting margin proportional to the annotators' agreements: (d) Filtering (Leonardelli et al., 2021): removing the training samples with high disagreement. Following (Leonardelli et al., 2021), we discard the samples with 3 agreements among 5 annotators, and use majority voting for the others; (e) Weighting (Uma et al., 2021): using weighted cross-entropy with smaller weights for the samples with high disagreements; (f) Multi-annolator (Davani et al., 2022): training the multiple classification heads for each annotation and using its ensemble for the evaluation. Furthermore, since we train the model with pair of samples, we also consider the baseline considering pair-wise training, (g) wlase Knoweeen Distillation (CS-KD) (Yun et al., 2020): adding regularization between sample pair that forces the similar predictive distribution between the same class samples. More details about baselines are described in Appendix A.2.
Training details. All the experiments are conducted by fine-tuning RoBERTa-base (Liu et al., 2019) using Adam optimizer (Kingma & Ba, 2015) with a fixed learning rate le-5 and the default of Adam. For all text classification tasks, the model is fine-tuned using the specified disagreement learning method with batch size 16 for 20 epochs. In the case of P2C, use T 3 preference heads { (W peperr) 1T and 2-layer MLPs for each Wpret. We choose hyper-parameter from a fixed set of candidates based on the validation set: Acons; dasv E (1.0, 0.13. We sample the pair of instances with the same majority voted labels for the efficiency, and apply the consistency loss with margin (Eq, 6) by using the difference of annotation as the margin m. More details and experimental supports for the design choices can be found in Appendix A.3 and C, respectively.
We compare P2C with various disagreement learning schemes to fine-tune RoBERTa- base classifier for each dataset. Table 2 summarizes the results six text classification datasets. Remarkably, P2C consistently outperforms the baseline methods for all six datasets. To be specific, P2C exhibits 7.59% relative test error reduction compared to the vanilla method in the average.
Results.

---

Under review as a conference paper at ICLR 2023
Table Test accuracy of finetuned RoBERTa classifiers with each annotation method on 5 different text classification datasets. All the values and error bars mean and standard deviation across 5 random seeds. The best and the second best results are indicated in bold and underline, respectively.
Method
Offensive
Polite-Wiki
Polite-SE
MNLI
DynaSent-RI
DynaSent-R2
Vanilla
75.88
89.35
70.00
81.92=a70
80,43=a33
71.23
Soft-labeling
76,08+144
89.57=176
70.35
82.67-a50
81.10=aa
72.15
Margin Loss
76,67
88.51 1ow
70.51
81.41
80,42.a0z
69.27 200
Filtering
76. 11.18
89.50=aa7
68,28
82.13=a07
80.38
69.86=078
Weighting
76. 111118
89.65=1.4
68.
82.48=20.4
80.21 #041
71.81=111
Multi--nnoottor
89.88 1.82
69.39
82.61=am
81.14
71.97
CS-KD
25.75
89.65
10.10
82.32
80.63 2o27
71.81 :1067
P2C (Ours)
77.81 10.21
91.06_am4
71,21 1000
83.15_00.9
81.27
73.06
Dislaance Amnolation
Sof-lebeing
F2C (Duna)
Verill
Sorts aa
Expeceed Calbrettion Error
CS-KD
0.8
CSAD
Teat Aceuracy
P2C (Oun)
0.5
0.8
0.65
6.
Varila
Multi amotaaor P2C
0.5
DynaSent-R2
Predicted Confidence
DynaSent-R2
(a) Disagreement Prediction (L)
(b) Reliability Diagram
(c) Expected Calibration Error (L)
Figure 4: Additional experiments with P2C on DynaSent-R2. (a) Average LI distance between the predictions and the soft labels obtained from the annotation records. The lower distance (4) means better alignment with annotators. (b) Reliability diagram shows accuracy as a function of confidence. Perfect calibration is plotted by dashed diagonals (/). (c) Expected Calibration Error (ECE) to quantitatively measure the calibration of the classifier. The lower ECE (1) means better calibration.
Furthermore, compared to the previous best disagreement learning method of each dataset, P2C exhibits 4.27% relative test reduction average. These results show that extractive preferences successfully provide complementary training signals to the classifier from the pair-wise preference, and demonstrate the effectiveness of P2C training method utilize the annotation records.
Next, on DynaSent-R2, we conduct additional experiments to verify depthly how P2C improves the classifier. We first check whether the prediction of the trained model with P2C is similar the annotators' judgment. Specifically, we compare the L1 distance between the predictions and the soft labels obtained from the data annotation records Figure 4(a). We verify that P2C achieves lowest distance to the soft labels, showing the validity of our preference learning for better modeling of given task. Moreover, we verify that a calibration of the classifier is also improved as a result of pair-wise preference modeling. To be specific, which plot the expected sample accuracy as a function of confidence of classifier in Figure 4(b). We remark that the plotted identity function (dashed diagonal) implies perfect calibration (Guo et al., 2017), and our method is the closest one among the baselines. This calibration effect of P2C is also verified through the additional quantitative metric, Expected Calibration Error (ECE), in Figure 4(c). Here, we commonly adopt the temperature scaling to measure ECE following (Guo et al., 2017).
first provide reliability diagram (Yun et al., 2020),
To verify the effectiveness of each component of P2C, we perform the ablation experiments and the results are presented in Table 3. It is observable that diverse multi-preferene heads improve the effectiveness of preference labels with better modeling compared to the single preference head (2-4th rows). In addition, consistency regularization between classification and preference heads enables the classifier to fully utilize the pair-wiss training signal to solve the task, hence the performance is significantly improved (5th row). The performance is further improved by selecting the informative pairs during the training (6th row). More results are presented in Appendix C.
4.2 EXPERIMENTS WITH SUBJECTIVE PREFERENCE FROM CROWD- WOOKERS
Setups. As described in Section 3.2, we collect subjective preference labels of the samples on DynaSent-R2, and use them fine-tune RoBERTa-base. All other training setups and details
7

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation study with each component of P2C. Test accuracy of finetuned RoBERTa classifiers on DynaSent-R2 and Offensive are compared. All the values and error bars are and standard deviation across 5 random seeds. The best results are indicated bold.
Method
T
Lrack
Spret
Laiv
Leane
Sampling
DynaSent-RA
Offensive
Vamilla
71.23
75.88
Preference
/
71.84+078
75.90+1.5
3
v
/
71.922206
76.43.00.0
72.05=1300
76.67:1138
72.67
77.67440.9
P2C (Ours)
d
73.062001
77.81
Table 4: Results of finetuned RoBERTa classifiers with different ways to obtain the labels on DynaSent-R2. Ntaxx and Npret indicate the number of used task labels and preference labels, respectively. dhard and deany are the 1 distance to annotations with hard and easy samples. Here, the difficulty is defined on the disagreement between annotators. the values and error bars are mean and standard deviation across 5 random seeds. The best results are indicated in bold.
Method
Nus
Nyeat
Accoug(t)
AcCuard AcCusay (t)
ECB(I)
dhard / daszy (4)
Vanilla
7.5k
69.032129
59.332257 80.00
9.25
0.8562am / 0.4405
Task Labels
12.5k
71.17
57.86123:
/8,221 21.6
9.19
0.878 / 0.327
Extractive Preference
7.5k
5k
71.362119
61.16219:
83.1 1 12:7%
6.75gam
0.847 / 0.35 2000
Subjective Preference
7.5k
5k
71.74210
62.082091 83.01 2:107
6.09
0.828 / 0.35622os
same as described in Section 4.1 except that the consistency regularization (Eq. 5) is now used, since the explicit degree of preference is not available from the subjective preference. In this section, we mainly compare P2C with different types of the preference labels: Extructive and Subjective preference labels. But the subjective preference labels have been collected for limited number of pairs of sentences (5,000) due to the cost, they are not available for all pairs of samples unlike extractive one. Hence, we also limits the availability of extractive preference labels only for the same pairs of texts with subjective preference, to directly compare the effect of different preference labels.
Results. We first verify the effectiveness of collected subjective preference labels compared to other types of labels. To this end, we consider the scenario that the specific types of labels are additionally obtained on top of the existing task labels; task labels could be more collected with additional training samples or preference labels between the existing samples could be obtained." Table 4 summarizes the experimental results. Here, it observed that subjective preference labels is the most effective for improving the test accuracy (Accavg) along with better calibration. Remarkably, it is noticeable that the preference labels significantly improves the accuracy on relatively hard samples (AcChard) while the additional task labels are effective for the relatively easy samples.'
Furthermore, to clearly demonstrate the advantage of subjective preference labels compared to the extractive one, we conduct additional experiments in Figure 5; subjective preferences collected with multiple rounds (Npret 1,000 - 3,000 + 5,000), we evaluate the effectiveness of preference labels for all rounds Figure 5(b), by controlling the number of preference labels of extractive preference and subjective preference. Remarkably, P2C with subjective preference outperforms P2C with extractive preference for all cases with only few thousand samples. Also, the performance of P2C is further improved when we extend the number of available preference labels via pseudo-labeling as denoted in the dotted lines (See details in Appendix D). The advantage of subjective preference is clearly shown when more challenging scenario: the accuracy on the mis-predicted test samples by vanilla model, and alignments with annotations both hard and easy samples. These results indicate that obtaining the explicit preference labels directly collected from annotators are better than extracting the implicit ones from the data annotation records.
take closer look at the evaluation results on
*Here, to facilitate the experimental comparison, we assume that the annotation cost for preference labels stays the same as that for task labels, although is arguable. We define the difficulty based the disagreement of annotators, i.e., disagree indicates more difficult.
8

---

Under review as a conference paper at ICLR 2023
P3
Sudjective Exrective
L1 Distance to Annolations
Test Accurscy (%)
Number of Preference
1800
Annuracy (%)
18
1200
0.6
602-
Extractive
Subjective
Number of Preference Labels
1000
2000
5000
12
Extractvve
Subjjotho
Erery
0.3
(a) Distribution of preference labels (b) Test accuracy improvements
(c) Comparison of predictions
Figure 5: Comparison between two different types of preference labels (Extractive and Subjective) on DynaSent-R2. (a) Distribution of collected labels. x" > x' indicates that sample x is preferred than x when considering Ytask. (b) Overall test accuracy increases more preference labels are used upon the existing task labels. (c) Subjective preference shows better performance on the mis-predicted test samples by Vanilla and closer alignment to annotators' agreements.
RELATED WORKS
Preference learning from human feedback. Preference learning is about modeling the preference using a set of instances, associated with an order relation (Fürnkranz & Hüllermeier, 2010). Since it much easier for humans to make relative judgments, i.e., comparing behaviors as better or worse, preference-based learning becomes an attractive alternative; hence, the extensive research has been conducted to address this problem by proposing different techniques to learn from human judgements (Bıyık et al., 2020; Chu & Ghahramani, 2005). One of the most representative fields that adopt preference-
learning is Reinforcement Learning (RL), to learn RL algorithms from the preferences rather than the explicit design of reward function (Wirth et al., 2017). After the successful scale-up of preference- based learning with deep neural networks (Christiano et al., 2017: Lee et al., 2021), this research direction has been extensively explored in other domains such as NLPP (Stiennon et al., 2020; Ziegler et al., 2019) and computer vision (Kazemi et al., 2020), especially focused on the generation tasks, e.g., text summarization and image generation. However, iifference-based learning is yet under-explored for classification tasks, despite its great potential to provide the complementary training signals via pair-wise comparison of samples.
Labeling beyond majority voting, With the rapid advance of DNNs thanks to improvements in computational resources (Jouppi al., 2017) as well algorithmic breakthroughs (Devlin al., 2019), the existing NLP benchmarks easily become obsolete and hence suffer to keep up with the model's development. To alleviate this, various approaches have been recently explored to construct more challenging and robust benchmarks using human-in-the-loww (Nie et al., 2020) or dynamic benchmarking system (Kiela et al., 2021). But, relatively under-explored how to annotate for utilizing them maximally. Even most of the recent NLP benchmarks still follow long-standing custom for their annotation; obtaining multiple annotators' judgements on the same data instances and aggregating them with majority voting (Hovy al., 2013). This aggregation, however, has a risk of sacrificing the valuable nuances embedded the annotators' assessments and their disagreements. Hence, various approaches have been recently explored exploit this information better and successfully improve the performance DNNs in various NI.P tasks; for example, Fornaciari al. (2021) constructs soft labels from annotation records. Also, Leonardelli et al. (2021) suggests to remove the samples with high disagreement. However, these methods are still limited due to the discretization of annotation space and the limited number of annotators. Hence, it inspires us to investigate an independent and complementary direction.
6 CONCLUSION
In this paper, we introduce task-specific human preferences between pairs of samples as a new and complementary data annotation to improve the existing text classification system, which relies on instance-wise annotations. To this end, we propose a novel multi-task learning framework, called prefer-to-classify (P2C), to effectively train the classifier from both task and preference labels, and demonstrate this framework under two different cases of human preference labels. We hope that our work could motivate other researchers for better data annotation and data usage in the future, e.g., suggesting to include annotation records instead of just providing majority- labels, in order to better learn the task from disagreed annotations.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
While the advantage of the proposed method has been demonstrated, some limitations exist. For example, we have assume the availability of task labels before applying our method, one could wonder that the relative importance between task labels and preference labels for constructing the datasets from scratch, i.e., the order of priority for collecting annotations among task labels and preference labels. If the number of given task labels is too small, then the task labels can be more effective than subjective preference labels. However, we highlight that our P2C could still benefit this case with the extractive preference as shown in Table 12. Also, the gain from subjective preference labels is enlarged after enough task labels are collected, demonstrating the complementary effect of preference labels upon the task labels as we are motivated initially. Lastly, this new framework is expected to be more effective in the domains that input samples are hard to collect as it provides the additional task information, e.g., medical domain.
REPRODUCIBILITY STATEMENT
We describe the implementation details of the method in Section 4 and Appendix A.3. Also, we provide the details of the datasets and baselines in Appendix A.I and A.2, respectively. We also provide our code in the supplementary material. All the used packages are along with the code. In our experiments, we use single GPU (NVIDIA TTTAN Xp) and 8 CPU cores (Intel Xeon E5-2630 v4).
REFERENCES
Cecilia Ovesdotter Alm. Subjective natural language problems: Motivations, applications, characteri- zations, and implications. In Annual Meeting of the Association for Computational Linguistics (ACL), 2011.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Erdem Bıyik, Nicolas Huynh, Mykel J Kochenderfer. and Dorsa Sadigh. Active preference-based gaussian process regression for reward learning. arXiv preprint arXiv:2005.02575. 2020.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 34544-3455141 1952.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems (NeurIPS). 2017.
Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceedings of the International Conference on Machine Learning (ICMI), 2005.
Kevin Crowston. Amazon mechanical turk: A research tool for organizations and information systems scholars. In Shaping the future of ict research. methods and approaches, pp. 210-221. Springer, 2012.
Cristian Danescu-Niculescu-MMe Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. A computational approach to politeness with application to social factors. In Annual Meeting of the Association for Computational Linguistics (ACI.), 2013.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics (TACL), 10:92-110, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NA.CL-HI.T), 2019.
10

---

Under review as a conference paper at ICLR 2023
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies (NtAaiiieeNAAA...... (NAACI-HHIT), 2021.
Johannes Fümkranz and Eyke Hüllermeier. Preference learning and ranking by pairwise comparison. In Preference learning, pp. 65-82. Springer, 2010.
Lukas Galke and Ansgar Scherp. Bag-of of-words vs. graph vs. sequence in text classification: Ques- tioning the necessity of text- text-graphs and the surprising strength of wide mlp. In Annual Meeting of the Association for Compulational Linguistics (ACL), 2022.
Mudasir A Ganaie, Minghui Hu, et al. Ensemble deep learning: A review. arXiv preprint 1rXiv:2104.02335.. 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the International Conference Machine Learning (ICML), 2017.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACL-HLT), 2018.
Dirk Hovy, Taylor King-Kiikpatrick, Ashish Vaswani, and Eduard Hovy. Learning whom trust with mace. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2013.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture, 2017.
Divyansh Kaushik and Zachary C Lipton. How much reading does reading comprehension require? critical investigation of popular benchmarks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes data. In International Conference Learning
difference with counterfactually-aggnne Representations (ICLR), 2020.
Hadi Kazemi, Fariborz Taherkhani, and Nasser Nasrabadi. Preference-basee image generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3404-3413, 2020.
Douwe Kiela, Max Bartolo, Yixin Nie. Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in nip. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2021.
Jachyung Kim, Youngbum Hur. Sejun Park, Eunho Yang, Sung Ju Hwang. and Jinwoo Shin. Distri- bution aligning refinery of pseudo-label for imbalanced semi-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations (ICIR), 2020.
Savannah Larimore, Ian Kennedy, Breon Haskett, and Alina Arseniev- Koehler. Reconsidering annotator disagreement about racist language: Noise or signal? In Proceedings of the Ninth International Workshop Natural Language Processing for Social Media, pp. 81-90, 2021.
11

---

Under review as a conference paper at ICLR 2023
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-effiiient interactive reinforcement learning via relabeling experience and unsupervised pre- -reetraining.. In Proceedings of the Interna- tional Conference on Machine Learning (ICML), 2021.
Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. Agreeing to disagree: Annotating offensive language datasets with annotators' disagreement, In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. Wanli: Worker and ai collaboration for natural language inference dataset creation. arXiv preprint arXiv:2201.0553 05955,, 2022.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-base reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguistics, 7:677-694, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825 2-2822--2800, 2011.
Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kicla. Dynasent: A dynamic benchmark for sentiment analysis. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods Natural Language Processing (EMNLP), 2016.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 1rXiv:1006.0509, 2017.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2018
Viktoriia Sharmanska, Daniel Hernández-Lobate Jose Miguel Hernandez-Lobate and Novi Quadrianto. Ambiguity helps: Classification with disagreements in crowdsourced annotations. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016
Edwin Simpson, Erik-Lån Do Dinh, Tristan Miller, and Iryna Gurevych. Predicting humorousness and metaphor novelty with gaussian process preference learning. In Annual Meeting of the Association for Computational Linguistics (ACI., 2019.
Rion Snow, Brendan O'connor, Dan Jurafsky, and Andrew Y Ng. Cheap and fast-but is it good? evaluating non-expert annotations for natural language tasks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2008
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei. and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72: 1385-1470, 2021.
12

---

Under review as a conference paper at ICLR 2023
Alexandra Nnemamaka Uma. Making the most of crowd information: Learning and evaluation in ai tasks with disagreements. 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2019.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-aww aware experts. In International Conference on Learning Representa- tions (ICLR), 2021.
Adina Williams, Nikita Nangia, and Samuel R Bowman, A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, 2018.
Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preference-
based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-45 2017.
Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. Synthbio: A case study in human-ai collaborative curation of text datasets. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wiss predictions via self-knowledge distillation. In Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: (90:::9999.99999 2019.
13

---

Under review as a conference paper at ICLR 2023
Appendix
Prefer to Classify:
Improving Text Classifier via Pair-wise Preference Learning
A EXPERIMENTAL DETAILS
A.1 DATASETS
As described in Section 4.1, we have used the five text classification datasets obtained from the following three different sources, which release the annotation records during the construction of the datasets, Here, all datasets have the annotation records of different annotators for each data; however, the annotators can be different among data, L.e. there are more than annotators in overall. For the tokenization of given data, we commonly set the maximum length L as 256.
DynaSent (Potts al., 2021) is a sentiment classification benehmark with ternary (posi- tive/negative/neuttaa) sentiments. It is dynamically constructed through multiple iterations of training a classifier model and finding adversarial samples by involving human annotator the loop. In our experiments, we use the dataset from the first round, (1) DynaSent-RI, and the dataset from the second round, (2) DynaSent-R2 R2. DynaSent-RI comprises 80,488 training samples, 3,600 validation samples, and 3,600 test samples, respectively. DynaSent-R2 comprises 13,065 training samples, 720 validation samples, and 720 test samples. All the validation and test samples are fully balanced between the three classes. DynaSent dataset and more details of the dataset are officially available at https: / github. com/ capotts dyynasay.d...e..n
Standford politeness corpus Caanesuu-Neuuescuu Mizil et al., 2013) is binary classification benchmark for predicting whether the given sentence is polite or impolite. Since there are two different input domains within this benchmark, we split them into different datasets: (3) Polite- Wiki from Wikipedia, and (4) Polite-SE from Stack Exchange, following the original paper (Danescu- Niculescu-Mizil et al., 2013). Here, two classes: polite and impolite, are defined as the top and, respectively. bottom quartile of sentences when sorted by their politeness score. The classes are therefore balanced, with each class consisting of 1,089 samples for the Wikipedia domain and 1,651 samples for the Stack Exchange domain. We split each dataset into an 8:1:1 ratio to construct training. validation, and test datasets. The source data and more details of the dataset officially available at https:/ / www. cs cornell
-rrts aan/Pol Poll teness html.
Offensive agreement dataset (Leonardelli et al., 2021) is a binary classification benchmark for predicting whether the given sentence is offensive not. Each sentence is collected from Twitter using Twitter public APIs, based on the hashtags and keywords on three different domains: Covid-19 19, US Presidential elections and the Black Lives Matter (BLM) movement. Remarkably, some of the original samples are not available anymore due to the climination of tweets from the user-side; for example, 10,735 samples collected initially (Leonardelli al., 2021), but only 6,513 samples are now available. To address the issue of the reduced number of samples, we slightly modify the dataset keep the setups of the original paper, e.g., balanced among the classes and domains. Specifically, we gather the given splits of the dataset into the unified one and then re-split it as much be balanced as possible. This re-constructed dataset has 2,400 training samples, 400 validation samples, and 400 test samples. Also, the ratio between Covid-19, Election, and BLM is 3:3:2. The dataset is officially available with the request to authors https: ////tttthhua// com/ dhfbk/ annotators imnotttors-agreeeeenn agreement segrremmnt-dttasee dataset.
Multi-Genre Natural Language Inference (MultiNLD) (Sener & Koltun, 2018) a crowd-sourced collection of 433k sentence pairs annotated with textual entailment information: for a given premise sentence, one should classify whether the given hypothesis sentence is entailment, neutral, or contradiction to the premise (ternary classification). Since the annotation records are only available with validation set, we construct the datasets by splitting it into 8:1:1 for training, validation, and test sets. This re-constructed datasets has 15,717 training samples, 1,964 validation samples, and 1,966 test samples. The source data and more details of the dataset is officially available at https: I //cims. nyu. edu/ sbowman /mul tinli.
14

---

Under review as a conference paper at ICLR 2023
A.2 BASELINES
We first introduce some notations for clear explanation. For each sample x, there are annotation records By (x) NK where K is the number of class and Tivata (x) Σy Bly (x) is the number of votes Then, the majority voted target label is obtained by finding the most agreed labels, i.e, (x)a arg maXy Rvota (x), and simply denoted by 1/taax. Here, our goal is train classifier So = Wass o Gé. composed with Transformer-basee language model backbone Je (e.g. BERT (Devlin et al., 2019)) and random initialized classification head W taex where the prediction for is obtained with softmax, i.e., p(x) Sof tmax fe (x)) For the analysis in Figure 4, we only include four baselines with high performance based on the results in Table 2.
(1) Vanilla: as described in Section 2.1, the model fo is trained with the following training loss:
Ltrain 2ze (p(x), (tack)
(2) Soft-labeling (Fornaciari et al., 2021): instead of using majority voted label Yrasks it use the soft-labels g(x) = My(()/ (2)x///negg Nvvoa (x) with cross entropy loss:
Lerain (x((p))). Σ - (x)
(3) Margin Loss (Sharmanska et al., 2016) instead of using majority voted label and cross entropy loss, it use the soft-labels q(x) as a margin for the multi-class hinge loss:
Ltrain Σ g0 - (,))
(4) Filtering (Leonardelli et al., 2021): following the setups the original paper, we exclude the ambiguous samples that have low argeement between the annotators. Specifically, we exclude the samples with yos = since there are 5 annotators for all considered datasets.
Ciaan
[Taeea (x) >
3] Ex Blask)
(5) Weighting (Uma et al., 2021): using weighted cross entropy that down-weigh the samples with low argeement:
Ltrain w(x) Exa (p(x), Braak)
where w(x) Mysa (xooD (x).
(6) Multi-annotator (Davani et al., 2022): instead of aggregating the different annotators' annotation records, it introduce multiple classification heads W[(() task for learning from each annotator's annotation Vtak Since each annotator does not annotate all the samples, we simply sepurate the Thyose (x) annotations and train each of classification head where / = ..... Prota- For the inference of test. samples, the ensemble multiple classification heads is used.
Cirain
Σ (x)l vranx)
Tyyot (x)
where (x) = W((1) o go (x). tast
(7) CS-KD (Yun et al., 2020): for each sample the sample & within the same class, defined by majority voted label Vtaek. is also sample and the consistency regularization additionally imposed between their prediction with temperature Following the original paper, 4.
Ltrain Exo (p(x), Vrask) + Lxo (F(x),
Alll the used datasets commonly have Thoote
15

---

Under review as a conference paper at ICLR 2023
Algorithm 1
(P2C) with extractive preference labels Input: Classifier from a pre-trained language model fo. training dataset with preference labels {(x), xx,, Utask Upret) |xx , D), preference predictors (hates ) mini-batch size B, and
hyper-parameter Xcona
1: for each iteration do
2: Draw a mini-batch B = {(x,, (task,)) and the corresponding pairs with preference labels
B - [(x,, :) from DD with the inconsistency-basee sampling (see Section 2.2)
3: Obtain [o(x) by forwarding B, then calculate Lmuiti in Eq. 4
4: Obtain (x) by forwarding B and B, then calculate Ccone in Eq. 6
5: Update parameters 0 and vi(t) to minimize Ltrain = Lmilti + Acone Coons
6: end for
Algorithm 2 Pneefer-to-Classiee (P2C) with subjective preference labels
Input: Classifier from a pre-trained language model fa. original training dataset DD = {(x;, y:)} collected dataset D with preference labels {(x", x! Utaak: Uxxe x! E D) where [D]
Nprat: preference predictors (hyrn 1 mini-batch size B and hyper-parameter Acona
1: for each iteration do
2: Draw a mini-batch
{(x,, Vtasx.i) from
3: Draw an another mini-batch B {(x,, X,, Vrack i Upref. i=1 from :) B
4: Obtain fx(x) by forwarding then calculate Lmuiti in Eq. 4
5: Obtain (x) by forwarding B, then calculate Lcone in Eq. 5
6: Update parameters 0 and vi(1) to minimize Ltrain = Lmitts + Acone Lions
7: end for
where p(x) Softmax(fo(x)/7)
A.3 PREFER- TO- CLASSIFY (P2C)
In this section, we describe the details of P2C. We first note that the details slightly different between extractive preference learning (Section 4.1) and subjective preference learning (Section 4.2) due to the difference in experimental setups between them. As described in Section 4, commonly use = 3 preference heads {Wr(9) pref 12" and 2-layer MLPs with tanh activation for each W Wpret. We choose hyper-parameters from a fixed set of candidates based on the validation set; Apret (1.0,0.1) Also, we only sample the pair of instances with the same majority voted labels for the efficiency.
In case learning with extractive preference in Section 4.1, we apply the consistency regularization with margin (Eq. 6) by using the difference of annotation as the margin Specifically, we set a margin of class y between two samples x! and x" as the difference of their soft-labels my = Ly (x²²) 4y (x"), defined in Section A.2. Then, we apply the consistency regularization to all classes addition, we apply the inconsistency-ban based sampling for the experiments with extractive preference labels based on the superior experimental results, presented in Section C.
y E [0, 1|
In the case of learning with subjective preference in Section 4.2, we apply the consistency regular- ization without margin (Eq. 5) since the explicit degree of preference is not given. Also, since the number of pairs with subjective preference labels limited, we use all them training without applying sampling methods described in Section 2.2. We introduce the additional mini-batch from these pairs to optimize the model with consistency regularization. The full procedures of P2C with extractive and subjective preference described in Algorithm and 2, respectively.
16

---

Under review as a conference paper at ICLR 2023
Read the instruction given below carefully:
Read
You can set sontment topo positive,
negetivel sentence..
Mark "ble Fistecence" when nother two sentences is preferred or can't represent the given sontiment catogory.
You can ignere miner grammatical or syntactic errons.
Example
Sentimenn Type: Fostth
Sentence A: '1 got veggioe les and a side of tries for over dollars you like homrccoked food"
Sentience B: "She lster end ideas, asked questiooe get better idea about my styls, and was eccellnn an ottering sovice as were hotal pleb.
Cholces:
Gentence
Gentence
No Preference
style wwo.ittthiiee..o. allowed). Once you chocse ausuens mutryruuntion, you can Suwen hustron the buttoo ene the trnn
carefully Imput
appropriate category
Otherwise, end the bank, The estimated time minuter.
candorn misporass baponoocisti arawora detected on samples, they be entrely future studios
Question
Sentiment Esattha
original
Sentence B: restaurnnt was not buey, but everything was meddy big crowd if needed, Wi love this place.
Oulput Choices:
Sentence A
Bentence B
No Proforence
Figure 6: Interface to collect subjective preference labels from crowd workers for sentiment classifi- cation based on DynaSent-R2 (Potts et al., 2021).
B COLLECTION OF PREFERENCE LABELS
B. INTERFACE
As described in Section 3.2, gather the subjective preference of the pairs by asking crowd workers to answer "which sentence is more positive (neutral, or negative)." using Amazon's 's Mechanical Turk crowd-sourcing platform (Crowston, 2012). Figure shows the interface used to collect subjective preference labels from crowd workers for sentiment analysis based on DynaSent-R2 (Potts et al., 2021). The top provides the instructions, and then one example is shown. The whole task has 10 items per Human Interface Task (HIT). Workers were paid USS0.8 per HIT on average, and all workers were paid for their work. To improve the quality of collected preference labels, we only hire the Master workers identified as high-performing workers from Amazon's 's Mechanical Turk system.
B.2 MORE DETAILS AND ANALYSIS OF EXTRACTIVE AND SUBJECTIVE PREFERENCE SETS
Extractive preference. For formal description of the process of collecting extractive preference,
we borrow some notations introduced Section A.2. As described Section 3.1, we obtain the extractive preference label Upret by comparing the number of votes Nyu (x) with the given task label where it indicates x" > x". Similarly, we assign Upret = O when Ryye (x') Ne (x") and Mprett = 0.5 when n (x' n (x"). respectively. To reduce the noisy signal and focus on the effective pair, we only compare the samples that have the same majority voted labels, i.e., (xaat(x-)) Stasie @xaas((*) (x)). The resulting distribution of extractive preference labels for each data is presented in Figure and
Utask: if y.a (x')
Tayass (x"), then we assign Upref
Subjective preference. To address the limitation of the extractive preference from the annotation records, we collect the subjective preference labels from crowd workers using the inference in Figure 6. As described in Section 3.2, dynamically collect the subjective preference labels. Here, to increase the effectiveness of the constructed preference dataset, we query the pairs with the unique anchor sample where x' is denoted the pairs of the samples that have the same majority voted labels as in the case of extractive preference labels. Figure 5(a) shows the distribution of collected preference labels. We remark that the collected
anchor sample of given pair (x (x), x²). Also, only query
17

---

Under review as a conference paper at ICLR 2023
Number of Preference Label 10*5)
Number of Preference Label 10*5)
Number of Preference Label 10%6)
9-
6-
4-
1-
(a) Offensive
(b) Polite-Wiki
(c) Polite-SE
Figure 7: Distribution of the extractive preference labels from the annotation records.
Number of Preference Label (x 10*7)
Number of Preference Label 10^8)
Number of Preference Label 10*7)
8-
2-
6-
2
(a) MultiNLI
(b) DynaSent-RI
(c) DynaSent-R2
Figure 8: Distribution of the extractive preference labels from the annotation records.
subjective preference labels are different compared to the extractive ones: collected subjective preferences shows 40.3 correspondence with the extractive preference labels, respectively.
18

---

Under review as a conference paper at ICLR 2023
B.3 MORE EXAMPLES IN EXTRACTIVE AND SUBJECTIVE PREFERENCE SETS
In Table 5, we present more examples in our extractive and subjective preference sets on DynaSent-R2 dataset, similar to Table 1.
Table 5: More examples in our extractive and subjective preference sets DynaSent-R2.
A: The prices were great and the service was friendly, which is why I'm surprised by my overall feeling of the place.
B: entered the college apartment and was shocked, everything neat and tidy.
Sentiment: Positive, Extractive Preference:
A. Subjective Preference: A
A: The atmosphere is great, Nikko our server was personable, knowledgeable and just all-around great, and the food looking so delicious.
B: The loaded fries were to die for.
Sentiment: Positive, Extractive Preference: No preference, Subjective Preference:
A: The food was surprisingly similar to a chain restau- rant such as applebee's or Friday's. s.
The soup and pad Thai arrived soon after we or- dered.
B:
Sentiment: Neutral, Extractive Preference:
>
B, Subjective Preference: B >
A: have laptop it acts like phone.
The baby cooed laughed without understanding the gravity situation.
Sentiment: Neutral, Extractive Preference:
B, Subjective Preference:
A: Went on a Thursday night with my two boys, B:
They provided great service and did wonderful job with my greasy hair and heavy makeup.
table right away close the kitchen.
Sentiment: Positive. Extractive Preference: No Preference, Subjective Preference: B
A: We requested a new tech and that guy said was a major leak and the would need to charge $7,000.
B: Teacher warned the teenagers not to enter the short cut because trespassers will be severely punished
Sentiment: Neutral, Extractive Preference: No Preference, Subjective Preference: >
A: The food is not fresh and delicious.
B: They treat customers like they want their money.
Sentiment: Negative, Extractive Preference:
>
Subjective Preference: > A
A: After check in had to search for the room without any direction. It was horrible.
B: That food blender worked great, paper weight.
Sentiment: Negative, Extractive Preference: No Preference, Subjective Preference:
A: The menu choices were too narrow.
The waiter seem to be busy else where back and would show up a few moments after was started to wonder if he' left. It was awful.
Sentiment: Negative, Extractive Preference: No Preference, Subjective Preference: B > A
B.4 EXAMPLES OF EXTRACTIVE PREFFRENCE SETS ON DIFFERENT DATASETS
In this section, we present more examples of extractive preference sets on another datasets used in Section 4.1. Specifically, we present the examples of Polite-Si -EE dataset in Table 6 and the examples of MultiNLI in Table respectively.
19

---

Under review as a conference paper at ICLR 2023
Table 6: Examples of the collected extractive preference sets on Polite-SE dataset.
A: @ Prrcctorian: You had me into tears. I agree you
B: This is not really enough tell what can be the problem. Have you tried debug you code, for example trace something in the conditions (so the function what have to be called is called properly or not)?
but is that going to help me with my question?
Politeness: Impolite, Extractive Preference: A > B
A: Usually compilers should generate good code for such algorithms. Did you check what assembly a compiler generating?
B: Why do you want to view the code? Maybe if you give better description of what you want to do?
Politeness: Polite, Extractive Preference: No Preference
A: Please don't I downvote without stating your reason.
B: Homework? Who told you that checking the pre- condition is bad idea?
Should this perhaps be turned into wiki question?
Politeness: Impolite, Extractive Preference: B > A
A: He has said its for ASP! am I missing something?
B: Sorry but never saw any application breaking after deleting/moving folders... usually they just reconfigure and restore defaults. What exactly did break for you?
Politeness: Impolite, Extractive Preference:
No Preference
A: Could you describe problems you are having?
@den: OK, so rectangles are represented as two points. Is this vector representation or a raster?
Are you logging the response codes?
Politeness: Polite, Extractive Preference: A
Table 7: Examples of the collected extractive preference sets on MultiNLI dataset.
A: (premise) This is arguably starting to distort the practice of science itself. (hypothesis) This began to distort scientific practice.
B: (premise) This number represents the most reliable, albeit conservative, estimate of cases closed in 1999 by LSC grantees. (hypothesis) This is actual verified number of closed cases.
Textual Entailment: Entailment,
Extractive Preference:
A: (premise) Uh-huh well maybe well i've enjoyed talking to you okay bye-bye. (hypothesis) I liked talk- ing to you.
B: (premise) And then was off, the world exploding behind me. (hypothesis) The world exploded behind
Textual Entailment: Entailment, Extractive Preference: No Preference
A: (premise) am surprised though that we do have so many that are in politics down here (hypothesis) I am surprised that not many of them are in polities down
B: (premise) Outside, set in manicured gardens, are the remains of the Abbey of Holyrood. (hypothesis) The gardens containing the remains of the Abbey of Holyrood are disarray and not well-kept.
here.
Textual Entailment: Contradiction, Extractive Preference: No Preference
A: (premise) After the purge of foreigners, only a few stayed on, strictly confined to Dejima Island in Na- gasaki Bay. (hypothesis) A few foreigners were left free after the purge foreigners.
B: (premise) It sure will well good to talk to. (hypothesis) That is unlikely and this conversation has gollen nowhere.
Textual Entailment: Contradiction, Extractive Preference: > A
A: (premise) Trying Your Luck. (hypothesis) Think
B: (premise) Emeralds? (hypothesis) Are they wearing emeralds?
carefully and calculate your way to certain victory.
Textual Entailment: Neutral, Extractive Preference: B A
20

---

Under review as a conference paper at ICLR 2023
C MORE ABLATION STUDY
In this section, we provide more ablation studies on the design choices of P2C. Here, all experiments are conducted on DynaSent-R2 (Potts et al., 2021) and Offensive (Leonardelli et al., 2021) datasets with extractive preference labels, as same as we have done Section 4.1. The values and error bars are the mean and standard deviation across five random seeds. The results with the chosen design in Section 4. I are underlined.
Multiple Preference head for preference learning, In Section 2.2, we introduce multi-preferee preference heads with diversity regularization (Eq. 3) to effectively learn the given preference labels. To see the effect, we compare it with two different designs for preference heads; 1) single-preference head and 2) multi-prefe cultiippeefernce heads without diversity regularization. Remark that the other components, consistency regularization, and inconsistency- based sampling, are still applied to separately verify the effect from different designs of the preference head. As shown in Table 8, one can verify that a single preference head is not enough to exploit the given preference labels fully; hence, the empirical gain is relatively small compared to multi- multi-preefrence heads. Also, it is observable that the proposed regularization is more effective to impose the diversity than only relying on the random initialization.
Table 8: Effect of different designs for preference head.
Single-Pref Head
Multi-Pret Heads without diversity
Multi-Pret Heads with diversity
Dataset
DynaSent-R2
22.22
72.75 77.25
73.06=0.11
Offensive
77.08
77.81 -0.21
Auxiliary loss for preference learning. As described in Section 2.2, we use consistency regulariza- tion (Eq. 5 and 6) between classification and preference learning as an auxiliary loss for learning preference; specifically, consistency regularization with margin (Eq. 6) is used in Section 4.1. To clarify the effectiveness of this regularization, we compare it with 1) consistency regularization without margin (Eq. 5). We also compare it 2) soft-labeling, which also uses the annotation records to construct soft-labels instead of the preference and margin. Here, we use random sampling instead of inconsistency- based sampling since it designed explicitly for consistency regularization while using the multi- celltiipeefeence heads. Table 9 shows the results of these auxiliary losses; although consistency regularization is effective in improving the performance without margin, the gain is smaller than the consistency regularization with margin since the latter utilizes the additional knowledge about the given preference label. In addition, the result with soft-labeling validate that the gain from our consistency loss from the use of the annotation records but from the regularization that imposes the following intuition: more preferred instance should have a higher confidence from the classifier.
Table 9: Effect of different auxiliary losses to learn from the given preference labels.
Soft -labeling
Consistency without margin
Consistency with margin
Dataset
DynaSent-R2
72.29
72.40 77.54
72.67 77.67+0.99
Offensive
77.04
Sampling of pairs for preference learning. To improve the efficiency of preference learning by sampling the informative pairs during the training, we introduce two advanced sampling methods: (I) disagreement- based sampling and (2) inconsistency-based sampling in Section 2.2. Remark that the other components, consistency regularization with margin and multi-preferrnce heads, are still applied to verify the effect from different sampling methods separately. In Table 10, we compare both sampling methods to random sampling. Here, one can verify that both ways are more effective than random sampling, and the inconsistency-bar sampling is slightly better than the disagreement-be based sampling. Hence, we commonly used inconsistency-base based sampling for in Section 4.1.
21

---

Under review as a conference paper at ICLR 2023
Table 10: Effect of different sampling methods with preference learning.
Dataset
Random
Disagreement
Inconsistency
DynaSent-R2
72.67 =0,99
72.73
73.06
Offensive
77.67
77.75
+0.2
Sensitivity to Laiv. To verify the sensitivity of our method with Laiv. we conduct the experiments by introducing λaıv. coefficient of Laiv. and varying it to investigate its effect. In Table 11, one can observe that KL divergence does not dominate the entire loss until the certain level of Xaiv including the original value (Aai==1), but it can diverge with too large value (e.g., daiv 10). Hence, recommend to use the original value or investigate daiv with smaller than 1.
Table 11: Effect of diversity regularization between multi preffereeee heads with Adiv-
Dataset
Adiv = 0
Adiv =
Adiv
Adiv = 10
DynaSent-Ra R2
72.75
73.06
71.44
57.05
Offensive
77.25
77.81 10:21
65.05 75.35
22

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL EXPERIMENTAL RESULTS
Smaller training samples. Here, we validate the effectiveness of P2C with extractive preferences for the smaller training samples. Specifically, control the number of training samples (N) of the DynaSent-R2 R2 dataset from N = 250 to N = 4000, and compare our method with three representative baselines with high performance: Vanilla, Soft-labeling, and Multi- Multi-annotator. As shown in Table 12, P2C shows significant improvement, especially when the dataset size is smaller. We also remark that P2C shows consistent improvement for all cases while other baselines do not.
Table 12: Results with the smaller training samples.
Method
N 250
N 500
N 1000
N 2000
N 4000
Vanilla
54.89
60.36+2.9
63.61
66.50
68.69
Soft-labeling
57.75+2.35
60.03
62.81+145
6..6
68.17
Multi-nnnoator
57.3321321 61.392
63.00
66.19 19
68. 68.7821146
P2C (Ours)
58.9421.6
61.83
64.13
67.72.1046
69.83
Semi-supervised learning with the collected preference labels. As the collection of preference labels induce the additional cost, especially with subjective preference, it would be beneficial if there is another way to fully exploit the collected labels; hence, we validate that the gain from limited preference labels can be further enlarged with the semi-supervised learning scenario. We remark that -umi-supereieed learning is well known to be effective for better utilization of limited labeled data (Berthelot et al., 2019; Kim et al., 2020). To verify the effectiveness of this approach, we conduct the following experiments: using trained P2C models with given 5,000 preference labels, we generate pseudo labels for the preference of each pair. Here, we clarify that we generate "pseudo" preference labels (i.e., pair-wise) for unlabeled pair data for semi-supervised learning while we assume that task labels (i.c., instance-wist wiiee) for all training data are still available. Then, we train another model with P2C using these "pseudo preference labels". As summarized in Table 13, we observe the clear improvements with this approach, and the gain is much larger with subjective preference, which indicates the informativeness of these preference labels. We also remark that there is still room for improvement from the advanced semi-supervised learning approaches such as iterative refinement (Ziegler al., 2019) or confidence thresholding (Park al., 2022).
Table 13: Semi-supervised learning with the collected preference labels with pscudo-labeling method.
Method
Vanilla
Extractive Preference
Subjective Preference
Original 71.23 1.05
71.94
72.40 73.08 #-042
Semi-sup
71.23
72.1 +
Compatibility with other types of models. While we have previously used a model built over RoBERTa-base (Liu et al., 2019), the proposed P2C is not limited to the specific model. To verify this, we conduct additional experiments based on DynaSent-R2 with extractive preference labels from the annotation records. As shown in Table 14, the proposed P2C consistently improves the test accuracy of classifiers built over one-hot encoded vectors (Galke Scherp. 2022) as well as other language models (BERT-base (Devlin et al., 2019), ALBERT-base (Lan et al., 2020), and RoBERTa-large).
Table 14: Results with other types of models.
Method
One-hot encoding
BERT-base
ALBERT-base
RoBERTa-large
Vanilla
54.42+tmm 55.47 40.82
67.26 68.2649.56
62.72+ 65.00
75.62
P2C (Ours)
77.71
Different number of annotations. In addition, we validate that pair-wise human preference could be more efficient annotation than the existing instance-wise task labels. To this end, we conduct
23

---

Under review as a conference paper at ICLR 2023
additional experiments DynaSent-R2 with subjective preference labels by varying the number of additional annotations (N) upon the existing 7.5k task labels, similar to Table 4.
As shown in Table 15, it is verified that the pair-wise human preference even requires fewer annotations to achieve the same level of test accuracy; for example, requires 29.8% fewer annotations than instance- -wise task label to achieve 70.5% test accuracy. Here, es- timate the number of annotations at 70.5% test accuracy by interpolating the results with polynomial approxima- tion (Pedregosa et al., 2011).? Also, we note the pro- posed framework has an additional advantage; could provide an effective solution, especially when input data acquisition is expensive, such as medical datasetss or human-in-the-loop benchmarks.
Suttecttvvn prefersncs
710
Task labels
Test accurayy
70.0
03.0
1003
2000
2600
Number of addiiiona arnotations
Figure 9; Interpolated results of Table 15
Table 15: Results with different number of additional annotations (N).
Method
N 0
N 500
N 1000
N 1500
N 2000
N 2500
Task Labels
69.03
69.471058
70.14..0
70.33
70.58
70.75
Subjective Preference
69.03
69.99
70.33
70.61
70.89
71.09
We use degree for interpolation, and code is available at auto_examplee es/ innarr mear_modell pt att_innrrpllltionn
24


Reviews:
[{"review_id": "TnSZQT0PvM", "reviewer": "Reviewer_M4mz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a multi-task learning framework called  prefer-to-classify (P2C), which is based on pair-wise preference learning. Specifically, their work is focused on handling NLP tasks. The paper explains the overall approach, and the process of selecting informative pairs, and how the preferences are collected from human annotators using Amazon Turk.\nThey have shown experimentally  that their approach is able to outperform on baselines - they have used 7 base lines across 6 datasets. ", "summary_of_the_review": "The authors present a multi-task learning framework called  prefer-to-classify (P2C), for handling NLP tasks. They have shown experimentally  that their approach is able to outperform on baselines - they have used 7 base lines across 6 datasets. \nThere are some drawbacks in terms of comparison baselines as discussed in the weakness section.", "strength_and_weaknesses": "Overall, the paper is well written, and the authors have explained their approach and provided experimental results to show their work. \n\nHowever, there are few weaknesses. \nThere are similar approaches in the literature, studied under various titles such as active learning, learning to rank and even preference learning. The approach sounds interesting, but not necessarily novel. The authors have compared their results against 7 approaches, but it is not clear which baseline is the state-of-art approach. Further, referencing approaches from active learning and learning to rank would greatly help demonstrating the effectiveness of the work.\n\nWhile the authors have given examples for the DynaSent-R1 dataset, it is not clear what kind of questions/ preferences were obtained for other datasets. Since they are subjective in nature, it will be good to understand the nature of responses obtained on others as well. On the positive side, the results are encouraging. \n\nFinally, in terms of relevance to the conference: the paper is more of an active learning or preference learning paper, and there is less focus on representations themselves. Though the authors have used a model build over RoBERTa-base, their approach could probably work on one-hot encoded vectors too. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N1fup78Tmmm", "reviewer": "Reviewer_TEZ4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel framework that utilizes the annotation disagreements in human-annotated benchmarks and human-annotated preference to improve the model training in scenarios where annotation records are provided. ", "summary_of_the_review": "This paper proposes a simple yet effective way to utilize human annotation records and incorporate human preference to guide the model training, which may motivate better data annotation and data usage in the future. I would recommend an acceptance for this paper.", "strength_and_weaknesses": "Strength:\n  - The idea is simple yet effective.\n  - The paper is well-motivated and well-written.\n  - The empirical results are convincing.\n  - Besides simply using annotation records, the authors also employ crowd workers to provide specific preference annotation. And they show using the preference labels further improves the model performance compared to simply extracting preferences from annotation records, making the logic and motivation sound and self-explained.\n  - The paper illustrates the effectiveness of pair-wise human preference to better guide the model, which may motivate better data annotation in the future.\n\nWeakness:\n  - Since only a few datasets provide annotation record and collecting human annotation is very expensive, the application scenario of this model is very limited at this time.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3Y_yYqZApFo", "reviewer": "Reviewer_cBxH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the paper, the authors propose P2C, a framework to synthesize the classification and preference learning tasks. The authors argue that capturing the preference information from the annotators during the data labeling process would lead to better text classification task performance. They propose two methods to collect annotator's preference information: (1) Implicit preference extraction on existing annotation records; (2) Explicit preference collection of annotators by online query and they formulate preference learning as a supervised learning problem.", "summary_of_the_review": "The MTL design to combine main tasks (e.g., text classification task in the paper) and other useful tasks (e.g., preference learning task in the paper) is well motivated. However, reproduction in some cases requires huge human resources.", "strength_and_weaknesses": "Strengths\n1. The authors formulate a preference learning task as a classification task and propose a framework to simultaneously solve text classification tasks and preference learning tasks. These formulations are clean and make the difference between the various setup clear.\n2. Improved performance (with test accuracy) compared to baseline methods.\n3. The author has done a lot of experiments to verify the importance of each module in the P2C framework.\n\nWeaknesses:\n1. The description in parts of the paper needs to be improved. \n- The abstract part mentions \"we propose a new multi-task learning framework\", the introduction part mentions \"we propose a new multi-task learning method\", and the conclusion part mentions \"we propose a novel multi-task learning object\". It is not a reasonable way to introduce P2C in three ways.\n- The information expressed in Eq. 5 is that the preference learning task is a binary classification task, but in the authors’ setting, the preference learning task is a triple classification task. That is, Eq. 5 does not express an equally preferable case (i.e., = 0.5).\n2. For datasets without any remaining annotation records, the proposed framework requires huge human resources (labels for preference learning tasks) to construct labels for preference learning tasks. Is this measure of improving model performance at a huge human resource desirable?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject