PALS: Personalized Active Learning for Subjective Tasks in NLP
Authors: Kamil Kanclerz
Konrad Karanowski
Julita Bielaniewicz
Marcin Gruza
Piotr Miłkowski
Jan Kocon
Przemyslaw Kazienko
Venue: EMNLP 2023 Main (2023)
Keywords: personalization
user modeling
active learning
natural language processing
subjective NLP tasks
subjective NLP

Abstract: For subjective NLP problems, such as classification of hate speech, aggression, or emotions, personalized solutions can be exploited. Then, the learned models infer about the perception of the content independently for each reader. To acquire training data, texts are commonly randomly assigned to users for annotation, which is expensive and highly inefficient. Therefore, for the first time, we suggest applying an active learning paradigm in a personalized context to better learn individual preferences. It aims to alleviate the labeling effort by selecting more relevant training samples. In this paper, we present novel Personalized Active Learning techniques for Subjective NLP tasks (PALS) to either reduce the cost of the annotation process or to boost the learning effect. Our five new measures allow us to determine the relevance of a text in the context of learning users personal preferences. We validated them on three datasets: Wiki discussion texts individually labeled with aggression and toxicity, and on Unhealthy Conversations dataset. Our PALS techniques outperform random selection even by more than 30%. They can also be used to reduce the number of necessary annotations while maintaining a given quality level. Personalized annotation assignments based on our controversy measure decrease the amount of data needed to just 25%-40% of the initial size.

PALS: Personalized Active Learning for Subjective Tasks in NLP
Kamil Kanclerz', Konrad Karanowski", Julita Bielaniewicz', MMarrin Gruza', Piotr Milkowski', Jan Kocoñ', Przemyslaw Kazienko
Wroclaw University of Science and Technology
Wroclaw, Poland
k htw.://rre....llaaaaaaeaaaaaaaee mar.ci
piotr. piotr.mi wwt.:.iiikwwwiiiimmmmmmmm kaziekko @pwr edu. 22545330suudent, pww.: pwr edu.pl
Abstract
2022; Kocoń al., 2022: Srivastava et al., 2023; Koptyra et al., 2023), we can observe that gains strongly depend on large annotated datasets (Kan- clerz et al., 2020; Shim et al., 2021; Kocoń et al., 2021b,a). Moreover, the data annotation process is very expensive (Kocoñ et al., 2019a,b; Wierzba et al., 2021). Several lines of research, especially active learning, have developed mechanisms duce the amount of supervision required to achieve better predictive performance. They, however, fo- cused on tasks themselves, i.e., such text selection that would increase agreement among annotators. This does not seem to be adequate for subjective NLP tasks, in which we allow the model to pro- vide different decisions for different users and the same text (Kocoñ et al., 2023a; Micleszezenko- Kowszewicz et al., 2023). As result, there are active learning methods dedicated for subjective tasks and personalized reasoning.
For subjective NLP problems, such as classifi- cation hate speech, aggression, or emotions, personalized solutions can be exploited. Then, the learned models infer about the perception of the content independently for each reader. To acquire training data, texts commonly ran- domly assigned to users for annotation, which is expensive and highly inefficient. Therefore, for the first time, we suggest applying ac- tive learning paradigm a personalized con- text to better learn individual preferences. It aims to alleviate the labeling effort by selecting more relevant training samples. this paper, we present novel Personalized Active Learning techniques for Subjective NLP tasks (PALS) either reduce the cost of the annotation pro- cess to boost the learning effect. Our five new measures allow us to determine the rele- vance of text in the context of learning users' personal preferences. We validated them on three datasets: Wiki discussion texts individu- ally labeled with aggression and toxicity, and on the Unhealthy Conversations dataset. Our PALS techniques outperform random selection even by more than 30%. They can also be used to reduce the number of necessary annotations while maintaining given quality level. Per- sonalized annotation assignments based on our controversy measure decrease amount of data needed just 25%-400 of the initial size.
Therefore, we introduce new approach to ac- tive learning in NLP: PALS Personalized Active Learning for Subjective problems. Its main contri- bution is modeling user preferences (perception) in the context of a given task. This allows us to exploit existing personalized models (Kocon et al.. 2021; Kocoń et al., 2023b; Kazienko et al., 2023) and complement them with more appropriate data acquisition. The main idea of PALS is the selection of texts for annotation adapted to a given user that maximizes extraction of knowledge about the user preferences. has been achieved by means of five measures described in Sec. 4. Our measures not only improve the overall performance of personal- ized models. They also sigmificantly decrease the number of annotations needed. As a result, they reduce computing power, which saves moncy and cuts down the environmental toll. The final prod- uct of our PALS methods is the optimization of the data collection process, which allows us re- duce the cost. Less but more adequate data may provide the same training effect. Alternatively, we
1 Introduction
The data acquisition process plays a crucial role in almost all natural language processing (NLP) stud- ies. The relevant data can significantly improve the model performance. However, the low-quality data limits the model quality regardless of its advanced architecture. Modern machine lcarning systems require large amounts of data to perform satisfac- torily. Although deep learning rapidly advances state-of-the-art results on number of supervised learning tasks (Mukherjee and Awadallah, 2020; Kocoń and Maziarz, 2021; Korczyński and Kocoñ,

---

acquire more information for the same cost. Addi- tionally, the annotation process can be stopped after achieving desirable quality. To develop efficient PALS methods, we raised several questions. How many annotations should each participant evalu- ate? Should we focus more on controversial or unambiguous texts?
ral Language Processing, significant amount of literature showed that the previously mentioned Bayesian Active Learning outperformed classical uncertainty sampling. However, the newest ap- proaches (Lu et al., 2019; Prabhu et al., 2021) ex- plored AL with multiple strategies using BERT based models for binary text classification tasks and proved that the performance of BERT-based models managed to achieve even better scores. Vi- sual Active Learning (Harpale and Yang, 2008; Wang et al., 2016) a generally known domain of AL, which is explored in learning- based image clas- sification. Common implementations incorporate deep convolutional neural networks into AL. The sample selection strategy works as cost-effective improvement of the classifier through incorporat- ing the analysis of high-confidence samples from an unlabeled set for feature learning. The selection of high-confidence samples leads an automatic selection and interactively assigned pseudo labels.
Please note that currently there is no method for personalized selection of texts for annotation appropriate for subjective tasks - PALS is the first. Our main contribution in this work are: (1) pos- ing a new problem regarding the effective acquisi- tion of annotations from users in order acquire knowledge about both the task and individual user perception supporting the learning of personalized reasoning models for subjective NLP tasks, (2) our novel PALS solution five measure-batt based meth- ods, which capture the diversity in subjective text perception, (3) validation of our PALS methods on three datasets and ten classification tasks related the offensiveness of texts for different number of documents annotated by the user, (4) ablation study including the analysis of what fraction of the training set is sufficient to achieve performance comparable to a reference random selection.
The vast majority of AL work bases its mech- anisms and score comparison the generalized approach. The first attempts at developing person- alization in AL extended the Bayesian AL tech- niques (Harpale and Yang, 2008; Seifert and Gran- itzer, 2010) through computing the probability of getting any rating, instead of the probability of get- ting particular rating for text. The algorithm is aimed at selecting items which have a very high probability of getting a rating from certain user. This approach provides interesting scores, how- ever, keeping the gold standard measure. Therefore, the area of personalization remains dependent on generalization to a certain degree (Bernard et al., 2018; Kocoń et al., 2023a; Mieleszczenko- -ow.nkkooo-owwwizzzzeo et al., 2023; Ferdinan and Kocoñ, 2023). What we achieved in our work is the contrary, it is the person themselves who becomes the gold standard, through analyzing the annotations and ratings they submitted. This break- through not only may change if we analyze users' annotations the AL domain, but also in analyzing data universally.
2
Related work
Active Learning (AL) has been extensively studied and applied in numerous tasks over recent years. The costs and means of obtaining labeled datasets can be troublesome, and this fact became more prominent as the research in the domain of ma- chine learning began to develop rapidly. The need of training classifiers using the least amount of data sprouted nearly 20 years ago, yet grew stronger in recent years. The more common AL approaches use sampling based on model uncertainty using measures such margin sampling (Zhou and Sun, 2014; Prabhu et al., 2019), least model confidence (Lowell et al., 2018) and entropy (Zhu al., 2008; Zhang et al., 2017). Deep Learning in AL, also named Deep Active Learning, can be a challenging task due to the tendency of rare uncertainty during inference and the requirement of a relatively large amount of training data. However, Bayesian Deep Learning based approaches (Wang et al., 2016; Tan al., 2021; Ren et al., 2021) proved that using dropout, batch normalization, and ensembles were all useful ways of lowering uncertainty model prediction. There have been numerous promis- ing implementations of AL in the field of Natu-
generalized
3 Generalized vs. Personalized Active Learning for Text Annotation
Active learning techniques should take into account the nature of the task which they are applied. In the case of personalized classification, this task requires the model to combine the general knowl- edge about the task (nature of the text) with the

---

individual preferences of the user (individual con- tent perception). Here, this knowledge balance is crucial, as a model too focused on the user will not take into account the linguistic aspects of the text. On the other hand, a classifier mainly exploiting linguistic knowledge will not be able to make per- sonalized predictions given the user's point of view. To ensure this balance, suggest the introduction of personalized active learning techniques. They should take into account the impression that the text evokes in the user and how much knowledge about the user's preferences should be gathered. To reveal individual user beliefs, a text selected for annotation must be controversial and the en- tire set of texts assigned to the user should contain contrasting examples.
Text subjectivity can be measured both in gen- eralized and personalized ways. In the first case, we examine how diverse are the opinions that the text evokes among all annotators. The general- ized ambiguity of a text can provide information about its potential to gain knowledge about the per- sonal preferences of each annotator. However, this upproach mask detailed information about the preferences of individual people with unique points of view by treating all annotators holistically. To counteract this, knowledge of the general text con- troversy should be enriched with personalized in- ML model performance
Text sesigmment
Perspective
Rendom mechanism
Generalized AL
Medium
Persoralized AL subjective
High
User
Figure 1: Random vs. Generalized Active Learning (GAL) vs. Personalized Active Learning for Subjective tasks (PALS) used select texts be annotated. The random approach (top) does not use active learning or consider any perspective. GAL (middle) is oriented towards task itself (documents) providing the same texts to all users. In PALS our personalized strategy (bot- tom), text selection also respects the individual user perspective.
This can further improve the performance of per- sonalized architectures by reducing the number of redundant documents. Such user-specific knowl- edge also facilitates distinguishing user from another in
efficient way.
Since we focus on subjective tasks, we experi- mentally confronted our methods with the baseline (random selection) rather than with generalized approaches, which were not designed to infer for subjective problems.
formation about the user's past decisions to assess how their individual subjectivity compares with the generalized subjectivity of other users (Kan- clerz et al., 2021, 2022; Bielaniewicz et al., 2022; Milkowski al., 2022; Ngo al., 2022).
Active Learning Mechanisms
We assume that, in the case of subjective tasks, active learning methods should respect user pref- erences. To achieve it, we prepared five measures: (I) Controversy, (2) Var Ratio, (3) Ratio Distance, (4) Stranger Count, and (5) Average Minimal Con- formity. They take into account the diversity the aggregated user beliefs or focus solely on the decisions of a single individual. Texts with the greatest value of given measure are selected to be annotated by the particular user.
Generalized Active Learning (GAL) methods for text amnotation focus on selection of documents that better reflect a given NLP problem averaged over the whole user community without consid- ering any individual user preferences. As a result, every may annotate the same texts, Fig. 1. Usu- ally their goal is to maximize the inter-annotator agreement level (Lin et al., 2019). For that rea- son, they do not appear to be suitable for subjective tasks like personalized offensiveness detection.
(1) Controversy (Contr(d)) of document d is an entropy- measure, as follows:
We, however, introduce another idea Person- alized Active Learning for Subjective problems (PALS), in which texts are selected for annota- tion individually for partieular user to better cap- ture their specific content and task perception. In other words, we implement the human-centred per- spective introduced in (Kocoñ et al., 2021), which matches single users with subjective NLP tasks.
Contr(u) (0. if in V Ad
a5 Eo-o. wa
loga
otherwise
where no and n. is the number of negative and pos- itive annotations assigned to document d, respec-
tively: na the number of all annotations related to

---

3:1
Var Ratio 1 13::4) = 1/4
2:2
Var Ratio 1 (2:4)= 1/2
Co-annotators
1:5
Ratio (5:6)=
Figure 2: An example of Var Ratio metric calculation. The middle document has the highest score, meaning that is candidate for annotation.
Stranger Count:
3
2
d, nd n + n.g. 2d estimates the probability that annotation of document d assigns it to the class c. Contr(d) happens when all users assigned to the same class c, Contr(d) of users considered aggressive and 50% did not. (2) The Var Ratio (Var ar Ratio(d)) measure aims to calculate the dispersion around the class, to which the document (d) was assigned most often (mode). The main intuition is that the text is am- biguous if the majority class was assigned to the document relatively rarely. This often indicates that users annotated the document as belonging to the other classes a relatively large number of times. The Var Ratio measure is defined as follows:
nd
calcu- lation. Co-annotators people who already annotated documents with our considered black user. On the right side, have red users, who have not yet anno- tated any document with our user. The greatest Stranger Count value identifies the orange text the best to annotated our black user.
Figure 3: An example of Stranger Count
mcans that 50%
(4) The Stranger Count measure (StrangerCounl(d) u)) aims to compare the subjective preferences of the user with as large groups of unique users as possible, Fig. 3. Its value for specific document d in the context of user is equal to the difference between the number of users who annotated d ((Ua)) and the number of annotators (a Ug) whose of annotated documents (U)) contains least one common document with the set of documents annotated by u: DD, U Dal > 0:
mode
7td
V VarRut Ratioll) =
Tid
where remode is the number of users who annotated d its largest class. An example of Var Ratio usage shown in Fig.
(3) The Ratio Distance (RatioDist(d, u)) mea- sure focuses on the difference between the ratio of users who annotated document as aggres- sive/toxic (ratio(d) and the average ratio of users (arg_ratio(u)) who assigned the document to the positive (aggressive toxic) class (c 1) for doc- uments already annotated by the user In the first step, ratio(d) is calculated for each document and annotations of users who annotated d (u Ua). The number of annotations assigning to the posi- tive class (1) (1 is divided by the total number of d's annotations (1 (1.cog):
StrangerCount(d u) [TA Eeeu, (PP.nD.ssol
(5) The Average Minimal Conformity (ArgMinCon describes the frequency of u belonging the majority of users over all u's texts. Firstly, the conformity measure (Conft f(u, c)) is calculated for user u and a given class E C::
Cons(u, Edep, P
Eder, 11 (1aee)
where D,, - the set of texts annotated by u; la the label assigned by u to d; La - the d's label obtained via majority voting. The final value of the Average Minimal Conformity measure for doc- ument d averages the minimal conformity of user over all considered classes (c C)) and for all users Ua who annotated d:
ratio(d) = EnEU, [[an=1]]
EveU, ((aaC)
avg_ratio(u) EdED, ratio(d) DDa
AngMinConf(()-
arg (min(Con wetiy cEC
RatioDist(d, u) ratio(d) aug_ratio(u)

---

5 Subjective Tasks
Aggression datasets are very similar in terms of user and annotation distribution, selected two out of the three for our research: WikiDetox Aggresison and WikiDetox Toxicity. The labels of the original dataset are degrees of aggression/toxicity in the range [-3, 3]. Like other researchers, we decided to transform this into binary classification task assuming that scores below 0 mean that the text is aggressive/toxic (class=1), and scores higher than or equal 0 mean that the text is neutral (class=0).
In our experiments, We used datasets obtained within the Wikipedia Detox project and the Un- healthy Conversations dataset (Price al., 2020). Table contains their profile. Each dataset has been divided into three subsets: training, validation, and testing in the 60-20-20 proportion. However, the original split of the data means that our model is trained and tested on the same users but on different texts. In the real world, we are faced with the prob- lem of evaluating the same texts but with respect to users. For this reason, we had decided to split the dataset with respect to users suggested in (Kocon al., 2021; Milkowski al., 2021), Fig. Sec. 6.1.
5.2 Unhealthy Conversations
The Unhealthy Conversations dataset (Price et al., 2020) consists of 44k comments, the length of which does not exceed 250 characters. They were acquired from the Globe and Mail opinion articles obtained from the Simon Fraser University Opin- ion and Comments Corpus (Kolhatkar et al., 2020), Each comment is rated by at least three annota- tors, whose task was to assign at least one of the offensiveness labels: antagonize, condescending, dismissive, generalization, unfair generalization, healthy, hostile and sarcastic. Each comment was presented to annotators separately, devoid of con- text related to the article and other comments order to reduce the bias.
(0.1)
train
Users
Folds
test dev
10
Experimental Setup
past
present
fut.
fut.
2
In order to validate our personalized active learning methods, we have decided apply the following principles. The experiments that were performed on all three datasets using the binarized versions of the tasks. Class imbalance was addressed by focusing on the positive, minority class (1) F1- score (F1 -binary) metric.
Texts
Figure 4: The U-T dataset split. Only the purple parts of the dataset are pruned in order to limit the amount of knowledge about the users in the dev and test folds the model can obtain
6.1 Data preprocessing
5.1 WikiDetox: Aggression and Toxicity
In the data preprocessing, we removed users who annotated less than 40 texts and those who an- notated all texts with the same label. This duced the number of unique annotators to 3,719 for WikiDetox Aggression dataset and 4,128 for WikiDetox Toxicity. Additionally, we have removed all new line characters from the text and then, we tokenized them using XLM-RoBERTa dedicated tokenizer (Conneau et al., 2020).
To ensure the best possible quality of Wikipedia entries, their authors discuss about them on the in- ternal forum. Unfortunately, the forum has also become place where some users verbally attack others. To counteract these phenomena, Wikime- dia Research has developed methods to automati- cally detect aggressive, toxic, and offensive content (Wulczyn et al., 2017). The WikiDetox datasets were created on the basis of user comments on more than 95M articles in 2001-2005. As a result, three datasets were created. Since the Attack and
We utilize the same user-text (U-T) data partition setup in (Kocon al., 2021; Milkowski et al., 2021).
The users were first divided into 10 folds. Simi- larly, the texts were split into 4 sets: past, present,
'https: //meta.////aaa//// wi k media.orge Research: Det.ar/badeeto//aa..... Re lease

---

Output
WikiDetox: Aggression
WikiDetox: Toxicity
Unhealthy Conversations
Number of texts
115,864
159,686
44,355
Number of users
4,053
4,301
558
Number of annotations
1,365,217
2,598,289
227,975
Avg. annotations per user
336.84
371.61
387.71
Avg. annotations per text Labels
11.78
10.01
4.66
[0, 1]
{0, 1]
8 (0, 1) English
Language
English
English
Table
Dataset profiles. Class=l means the text is aggressive, toxic or unhealthy.
faturel, and future2. Such partitioning allows to test methods texts and users that are unknown by the model. The past set represents the general knowledge about all users. This is the only part of the texts that are used to train the model with anno- tations made by users from test and validation folds. The present set is the main source of knowledge about the texts, while futurel and future2 represent the validation and test sets of texts.
6.4 Experimental scenario
At the beginning, we took the entire past data from the training part. Then, at each step, we added to it one annotation from the set of present for each user, selected on the basis of our personalized ac- tive learning (PALS) methods, see Sec. 4. The procedure regarding the ordering of texts as can- didates for annotation described Alg. 1. We trained our model on the set obtained in this way. Then, it was evaluated on the future2 subset. We repeated this procedure until we reached the maxi- mum available number of 14 annotations from the present set for each user. Each of such experiments was repeated 10 times in the U-T cross-validation scenario, Fig. To examine the impact of data acquired with our PALS measures on the model performance, we confronted results with the same model but trained on randomly annotated data (baseline).
6.2 Language Model
To obtain vector representations of texts, we used the XIMMRoBERTa model. It is a multilingual version of RoBERTa transformer trained on 100 languages and 2.5 TB of preprocessed Common- Crawl data.
Due to limited resources and the wide scope of experiments, the language model was not fine- tuned for downstream tasks during the experiments. Instead, we only used it as feature extractor. The vector representation of the text is an average of the token vectors of the last transformer's 's layer. The only manual preprocessing step for the texts in the datasets was to remove new line token strings for Wiki datasets.
Algorithm The process of ordering the texts
candidates for annotation aj E A,, for user E U. The PALS measure v value my is calculated for each potential annotation aj not yet annotated by u4. Then, texts aj are sorted for given user tt by the measure values.
6.3 Classification models
As personalized classification models, we used HuBi-Simple, presented in (Kocon al., 2021). HuBi-Simple is relatively simple linear model that uses textual features and additionally learns the biases of annotators and biases single words. The annotator bias is straightforward component that allows us to capture the annotator personal bias towards the modeled NLP task. We selected this model because want personalized model help us evaluate our PALS approach data annotation. We applied the same hyperparameter settings as proposed in the original model publi- cation because our research covers two the same datasets and similar setup.
orders [0] for Us E U do text text_measures 0
4:
for aj E A,, do
5:
my < vaa)) lext leat_measures(aj] + m2
6:
end for
K = text free__maaurre.sort bas_bbyuuuussss orders[i] < K.keys() 10: end for 11: return orders

---

7 Results
WikiDetox: Aggression
0.65
0.64
In the of WikiDetox Aggression (the upper
0.63
plot in Fig. 5), the best results were achieved by the Var Ratio and Controversy selection. Average Minimal Conformity and Stranger Count demon- strated stable behavior, slightly different from the random annotations. However, the Average Mini- mal Conformity was clearly better in the range of 3-5 annotations per user. The Conformity Distance measure achieved significantly worse results than all other measures. Our experiments showed that choice of right measure allowed personalized rea- soning models to provide better results for just annotations per user compared to Random selection with 14 annotations per user.
0.62
0.61
0.60
0.59
0.58
Controversy
Ratio Ratio Distance
0.57
Stranger
0.56
Average Minimal Conformity
0.55
4
5
6
8
9
10
11
12 13 14
WikiDetox: Toxicity
0.62
0.61
0.60
The best results on the WikiDetox Toxicity data set (the middle in Fig. 5) were observed for the Controversy measure. However, the Var Ratio also performed better than the Random assignment. The Conformity Distance and Average Minimal Con- formity turned out not to be the most stable but still slightly better for random baseline, especially in the range of 3-6/8 annotations per user. Stranger Count measure performed the worst.
0.59
0.58
0.57
0.55
Controversy
Var Ratio
0.54
Stranger Count
0.53
Mirimal Conformity
0.52
5
6
13 14
For the Unhealthy Conversations dataset (the bottom plot in Fig. 5), the best results achieved by the Controversy measure. However, the Average Minimal Conformity and the Ratio Distance still performed better than the Random se- lection. Surprisingly, the Var Ratio measure, which performed very well on the WikiDetox datasets, pro- vided worse results than the Random baseline. may be related to the high variance between the labels, which affects the final measure values.
Unhealthy Conversations
0.350
0.315
0.280
0.245
score
0.210
0.175
0.140
0.105
Contreversy
Ratio
0.070
Distance
Choosing the best PALS measure for data anno-
Stranger Count
tation (Controversy) allowed our model to outper- form even by more than 30%, Fig. 6. It means that better selected training cases significantly boost quality of personalized reasoning since they deliver more information about user diversity. Moreover, using the Controversy or the Var Ratio measure allowed for achieving the similar results as for the Random assignment, but using only about 25% of the data, Fig. In other words, PALS methods boost training so that it requires much less learning samples and computing resources.
0.035
Average Conformity
Random
0.000
5
13
Number of annotations per user
Figure 5: Performance the models trained on data acquired using various active learning methods of text assignment. axis corresponds to F1 score for the aggressive toxicity class for WikiDetox data and Fl for positive class averaged over all classes for Un- healthy Comversations. Two horizontal lines indicate the reference performance (random selection) for 5 and 10 documents annotated by each user.
The detailed results on all three datasets and the Fl score values for all eight labels in the Unhealthy Conversations dataset are presented in Appendix A.

---

around certain value. similar effect is observ- able for Unhealthy Conversations but the inflection point occurs later 12 texts. We can also see that the method of selecting texts for annotation also fects the amount of data needed to reach saturation point. Improving the quality of annotated texts still in the collection process reduces not only annota- tion costs but also training time, which positively impacts the reduction of the carbon footprint emitted while training of deep learning models. Most Active Learning approaches focus on the confidence of the model on unlabeled data. This works well for standard NLP methods where the text is the only source of knowledge. In this work, show that, in personalized models, the subjectiv- ity of people and their individual approach is also an essential factor that should be taken into account when selecting data samples for annotation. By se- lecting controversial texts and annotating them by different people, we gain more knowledge about their subjectivity, and therefore we improve the model quality of inference for these people.
30.0%
+0.00%
F1 score
10.0%
0.0%
Average Mirimal Corfomilty
3 4
6
7
8
9
10
11 12 13 14
Number of annotations per user
Figure 6: Gains in FI score for Unhealthy Conversa- tions and for our two active learning measures compared to Random selection. The Y axis denotes the gain F1 for the positive class averaged over all classes.
annotations active learning
Controversy
Var Ratio
A common method of evaluating NLP models is counting metries globally on all annotations of in- dividuals or on majority votes of annotators. How- ever, this way we lose information about spe- cific individuals or groups of people for whom our model gives significantly different results. For this reason, we consider evaluating models for indi- vidual annotators and then averaging these results or comparing their distributions.
No. of annotations with random assignment 14
5
7
The least number of annotations needed a given active learning measure (Y axis) to achieve a F1 score for positive class not worse than for the Random selection (X axis); the WikiDetox: Aggression dataset.
Figure 7:
Conclusions and Future Work
In this paper, for the first time, raised complex question: how to carry out the annotation process wisely and efficiently reduce costs and/or in- crease reasoning quality for subjective tasks solved in the personalized way?
8 Discussion
By analyzing the results obtained on three datasets, we can come to the conclusion that the effective- ness of the model grows quickly with the addition of subsequent annotations for the users only up to a certain level, see Fig. 5. After it is exceeded, adding more samples to the set improves the perfor- mance only slightly, until it finally ceases to affect its performance at all. In the case of the WikiDetox Aggression dataset, the greatest increase in the effi- ciency of the model occurs in the range from to annotations per user, then the increase slows down until about 10 annotations very insignificant. For WikiDetox Toxicity, the performance increases for up to annotations per user, and then it fluctuates
Instead of commonly used random selection of texts to be annotated, propose novel paradigm: PALS personalized active learning methods suit- able for subjective NLP tasks. They allow us ef- ficiently gather information about user preferences related to text perception.
Experimental studies on three datasets and ten problems revealed that the Controversy measure appears be the best out of all 20+ measures con- sidered (only five best are described in the paper). The results show that PAL.S methods increase qual- ity of collected data and are better suited both to the task and to the user. As a result, the classifiers

---

are able improve their reasoning quality even by 30%. Alternatively, PALS methods can be used to reduce amount of data needed to reach a given per- formance; just 25%-40% is enough. Thus, models using data provided by PALS methods can learn in an extremely short time. This has a direct im- pact on reducing the costs of the entire annotation process.
properties would allow us simulate various strate- gies of personalized selection texts for annotation. This would also facilitate studies on application of reinforcement learning methods to text selec- tion, e.g. recalculation of PALS measures after each bulk data. To some extent, our results may depend on the way the data was collected. For example, in Wiki data, less offensive texts (with smaller number of offensive annotations) received less annotations in total than offensive ones. As a result, we plan to collect yet another data set with greater (possible full)) coverage of annotations for each text. Moreover, due to limited computational resources, we did not consider fine-tuning the lan- guage model, and we were not able to try other language models. In addition, in our scenario, we assume that have certain data sample from the beginning calculate the evaluated measures. The scenario considered is based on adding new anno- tations for each user proposed by our method. This is, to some extent, a suitable setup for incremental learning. However, we have not investigated any solutions dedicated to incremental learning all our models were re-trained on the full, increased dataset. Furthermore, we tested many more mea- sures (more than 20), but we have only included those in the paper that provided relatively better results. Moreover, we have not considered the di- versity the user community. Perhaps the datasets are biased towards some social groups. We also did not consider the context of data collection, e.g. the user mood, the broader context of the statement, the sequence of annotations in the case of analyz- ing the impact of the annotation order the user perception, etc. Both limitations nature of the available data the datasets did not included appropriate information. Moreover, we did not conduct broader analysis our models vulnerability to attacks. On the other hand, the quality and reliability of the data, e.g. potential mistakes in the annotation process, has not been analyzed and may have an impact the results.
In the future work, will focus the problem of text diversity, collection representativeness for the task, and reinforcement learning methods that would adjust the selection process runtime.
The code for all methods and experiments is publicly available under the MIT license.
A Acknowledgments
This work was financed by (1) the National Science Centre, Poland, project no. 202 D021/4////ST/044477 (2) Contribution to the European Research Infras- tructure CLARIN ERIC European Research In- frastructure Consortium: Common Language Re- sources and Technology Infrastructure', 2022-23 (CLARIN Q) (3) the Polish Ministry of Edu- cation and Science, CLARIN-PL: (4) the Euro- pean Regional Development Fund, as a part of the 2014-2020 Smart Growth Operational Pro- gramme, projects no. WOU.R.4..2..0000000 PO1..01.01.01-00-09922222 0615/21, and POIR.01.01.01 010000000.00--00028///2/ (5) the statutory funds of the Department of Artificial In- telligence, Wroclaw University of Science and Technology: (6) the Polish Ministry of Education and Science within the programme "International Projects Co-Funded"; (7) the European Union un- der the Horizon Europe, grant no. 101086321 (OMINO). However, the views and opinions ex- pressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor European Research Exec- utive Agency can be held responsible for them.
NOIR..0.01.01-000
due to the
10 Limitations
The problem with testing Personalized Active Learning methods for subjective NLP tasks is the small number of relevant datasets. In particular, it is important that the dataset has a large number of annotations per text, and additionally a large number of annotations made by each user. Such
References
Jürgen Bernard, Matthias Zeppelzauer, Markus Lehmann, Martin Müller, and Michael SedImair. 2018. Towards user-centered active learning algo- rithms. Computer Graphics Forum, volume 37. pages 121-132. Wiley Online Library.
/ thub. www..cco//cATT-PP... personal weeeiineppllle htwpp//eeeeeeeeeaaaeeeeeee -mmppppppppppppppp
Julita Bielaniewiez. Kamil Kanclerz, Piotr Milkowski. Marcin Gruza, Konrad Karanowski, Przemyslaw

---

Kazienko, and Jan Kocoñ. 2022. Deep-sheep: Sense of humor extraction from embeddings in the personal- ized context. In 2022 IEEF International Conference on Data Mining Workshops (ICDMW), pages 967- 974. IEEE
Jan Kocoñ, Joanna Baran, Kamil Kanclerz. Michat Ka- jstura, and Przemystaw Kazienko. 2023a. Differen- tial dataset cartography: Explainable artificial intelli- gence in comparative personalized sentiment analy- sis. In International Conference on Computational Science, pages 148-162. Springer.
Alexis Conncau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451.
Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydto, Joamna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023b. Chatgpt: Jack of all trades, master of none. Information Fusion, page 101861.
Jan Kocon, Marein Gruza, Julita Bielaniewicz. Damian Grimling. Kamil Kanclerz, Piotr Milkowski, and Przemyslaw Kazienko. 2021. Learning personal hu- man biases and representations for subjective tasks in natural language processing. In Proceedings of st IEEE International Conference Data Min- ing (IEEE ICDM 2021), pages 1168-1173, Online. IEEE
Teddy Ferdinan and Jan Kocoń. 2023. Personalized models resistant to malicious attacks for human- centered trusted ai. In The AAAI-23 Workshop on Artificial Intelligence Safety (SafeAl 2023). CEUR Workshop Proceedings.
Abhay S Harpale and Yiming Yang. 2008 Personal- ized active learning for collaborative filtering. In Proceedings of the 31st annual international ACM SIGIR conference Research and development in information retrieval, pages 91-98.
Jan Kocoń and Marek Maziarz. 2021. Mapping wordnet onto human brain connectome in emotion processing and semantic similarity recognition. Information Processing & Management, 58()):002530.
Kamil Kanclerz, Alicja Figas, Marcin Gruza, Tomasz Kajdanowicz. Jan Kocon, Daria Puchalska, and Prze- myslaw Kazienko. 2021. Controversy and confor- mity: from generalized to personalized aggressive- ness detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5915-5926, Online. Association for Computational Linguistics.
Jan Kocoń, Piotr Milkowski, and Kamil Kanclerz. 2021a. Multiemo: Multilingual, multilevel, multido- main sentiment analysis corpus of consumer reviews. In International Conference on Computational Sci- ence, pages 297-312. Springer.
Jan Kocoñ, Piotr Milkowski, Malgorzata Wierzba, Barbara Konat, Katarzyna Klessa, Arkadiusz Janz, Monika Riegel, Konrad Juszezyk, Damian Grimling, Artur Marchewka, et al. 2019a. Multilingual and language- agnossic recognition of emotions, valence and arousal in large-scale multi-domain text reviews. In Language and Technology Conference, pages 214- 231. Springer.
Kamil Kanclerz, Marcin Gruza, Konrad Karanowski, Julita Bielaniewicz, Piotr Milkowski, Jan Kocoñ, and Przemyslaw Kazienko. 2022. What if ground truth is subjective? personalized deep neural hate speech detection. In Proceedings of the 1st Workshop on Per- spectivist Approaches NLP@ LREC2022, pages 47-45.
Jan Kocoá, Piotr Milkowski, and Monika Zaśko- Zielińska. 2019b. Multi-level sentiment analysis of polemo 2.0: Extended corpus of multi-domain con- sumer reviews. In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 980-991.
Kamil Kanclerz, Piotr Milkowski, and Jan Kocoú. 2020. Cross-lingual deep neural transfer learning in senti- ment analysis. Procedia Computer Science, 176:128- 137.
Przemyslaw Kazienko, Julita Bielaniewicz, Marcin Gruza, Kamil Kanclerz, Konrad Karanowski. Piotr Milkowski, and Jan Kocoñ. 2023. neural reasoning for subjective content processing: Hate speech, emotions, and humor. Information Fu- sion, 94:43-65.
Jan Kocoń, Jarema Radom, Ewa Kaczmarz- Kaczmarz-Wawryk, Kamil Wabnic, Ada Zajaczkowska, and Monika 7aško-Zielifska. 202 Aspectemo: multi-domin corpus of consumer reviews for aspect-based senti- ment analysis. 2021 International Conference on Data Mining Workshops (ICDMW), pages 166-173. IEEE.
Jan Kocoñ, Joanna Baran, Marcin Gruza, Arkadiusz Janz, Michat Kajstura, Przemystaw Kazienko, Woj- ciech Korczyúski, Piotr Milkowski, Maciej Piasecki, and Joanna Szolomicka. 2022. Neuro-symbolic mod- els for sentiment analysis. In International con- ference on computational science, pages 667-681. Springer.
Jan Kocoñ, Alicja Figas, Marcin Gruza, Daria Puchalska, Tomasz Kajdanowicz. and Przemystaw Kazienko. 2021. Offensive, aggressive, and hate speech analysis: From data-centric to human- centered approach. Information Processing & Man- agement, 58(m)) 58(5):

---

Varada Kolhatkar, Hanhan Wu, Luca Cavasso, Emilie Francis, Kavan Shukla, and Maite Taboada. 2020. The sfu opinion and comments corpus: A corpus for the analysis of online news comments. Corpus Pragmatics, 4(2):155-190.
text classification with few labels. arXiv preprint 0rXiv:2006.15315
Anh Ngo, Agri Candri, Teddy Ferdinan, Jan Kocon, and Wojeiech Korczynski. 2022. Studemo: A non- aggregated review dataset for personalized emotion recognition. In Proceedings of the 1st Workshop on Perspectivist Approaches to NL.@@ 1REC2022. pages 46-55.
Bartiomiej Koptyra, Anh Ngo, Lukasz Radliński, and Jan Kocoñ. 2023. Clarin-emo: Training emotion recognition models using human annotation and chat- gpt. In International Conference on Computational Science, pages 365-379 Springer.
Ameya Prabhu, Charles Dognin, and Maneesh Singh. 2019. Sampling bias in deep active classification: An empirical study. arXiv preprini arXiv: (ete:: 093389.
Wojciech Korezyński and Jan Kocoñ. 2022. Compres- sion methods for transformers in multidomain senti- ment analysis. In 2022 IEEE International Confer- ence on Data Mining Workshops (ICDMW), pages 419-426. IEFE.
Sumanth Prabhu, Moosa Mohamed, and Hemant Misra. 2021. Multi-class text classification using bert-based active learning. arXiv preprint arXiv:2 14289.
Ilan Price, Jordan Gifford-Mooee Jory Flemming. Saul Musker, Maayan Roichman, Guillaume Sylvain, Nithum Thain, Lucas Dixon, and Jeffrey Sorensen. 2020. Six attributes of unhealthy conversations. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 114-124, Online. Association for Computational Linguistics.
Bill Yuchen Lin, Dong-Ho Lee, Frank F. Xu, Ouyu Lan, and Xiang Ren. 2019. AlpacaTag: An active learning- based crowd annotation framework for sequence tag- ging. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Sys- lem Demonstrations, pages 58-63, Florence, Italy. Association for Computational I Linguistics.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. 2021. survey of deep active learning. ACM Computing Surveys (CSUR), 54(9):1-44 1-0..
David Lowell, Zachary C Lipton, and Byron C. Wallace. 2018. How transferable are the datasets collected by active learners. arXiv preprint arXiv: 008004 3.
Jinghui Lu, Maeve Henchion, and Brian Mac Namee. 2019. Investigating the effectiveness of represen- tations based on word-embeddings in active learn- ing for labelling text datasets. arXiv preprint 11X.::19..003505.
Christin Seifert and Michael Granitzer. 2010. User- based active learning. In 2010 IEEE International Conference on Data Mining Workshops, pages 418- 425. IEEE
Heereen Shim, Dietwig Lowet, Stijn Luca, and Bart Van- rumste. 2021. Lets: A label-efficint training scheme for aspect-based sentiment analysis by using a pre- trained language model. JEEE Access, 9:115563 15563- 115578.
Wiktoria Mieleszczenko- -ww.skewwoowzwwzzz...czooooo Kamil Kanclerz, Julita Bielaniewicz, Marcin Oleksy, Marcin Gruza, Stanistaw Wožniak, Ewa Dzięciol, Przemyslaw Kazienko, and Jan Kocoń. 2023. Capturing human perspectives in nlp: Questionnaires. annotations, and biases. In The ECAI 2023 2nd Workshop on Perspec- tivist Approaches to NLP. CEUR Workshop Proceed-
Aarohi Srivastava. Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. Transactions on Machine Learning Research.
ings.
Piotr Milkowski, Marcin Gruza, Kamil Kanclerz, Prze- myslaw Kazienko, Damian Grimling, and Jan Ko- con. 2021. Personal bias in prediction of emotions elicited by textual opinions. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 11th International Joint Conference on Natural Language Processing: Stu- dent Research Workshop, pages 248-259, Online. Association for Computational Linguistics.
Wei Tan, Lan Du, and Wray Buntine. 2021. Diversity enhanced active learning with strictly proper scoring rules. Advances in Neural Information Processing Systems, 34.
Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. 2016. Cost-effective active learning for deep image classification. IEEE Transactions on Cir- cuits and Systems for Video Technology. 27(12)25591 2600.
Piotr Milkowski, Stanislaw Saganowski, Marcin Gruza, Przemyslaw Kazienko, Maciej Piasecki, and Jan Ko- con. 2022. Multitask personalized recognition of emotions evoked by textual content. In 2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). pages 347-352. IEEE.
Malgorzata Wierzba, Monika Riegel, Jan Kocoñ, Pi- otr Milkowski, Arkadiusz Janz, Katarzyna Klessa, Konrad Juszezyk, Barbara Konat, Damian Grimling, Maciej Piasecki, et al. 2021. Emotion norms for 6000 polish word meanings with a direct mapping to the
Subhabrata Mukherjee and Ahmed Hassan Awadal- lah. 2020. -ncertainty-aware self-training for

---

polish wordnet. Behavior Research Methods, pages 1-16.
0.50
Ellery Wulezyn, Nithum Thain, and Lucas Dixon, 2017. Ex machina: Personal attacks seen at scale. In Pro- ceedings of the 26th international conference on world wide web, pages 1391-1399.
0.45
0.40
0.35
Yexun Zhang, Wenbin Cai, Wenquan Wang, and Ya Zhang. 2017. Stopping criterion for active learn- ing with model stability. ACM Transactions on Intel- ligent Systems and Technology (TIST), 122
0.30
0.25
0.20
Var Raão
Ratio Dislance
Jin Zhou and Shiliang Sun. 2014. Improved margin sampling for active learning. In Chinese Conference on Pattern Recognition, pages 129. Springer.
0.15
Stranger
Average Minimal Conformity Random
0.10
10 11 12 13 14 Number annotations per user
Jun Zhu, Eric Xing, and Bo Zhang. 2008. Partially observed maximum entropy discrimination markov networks. Advances in Neural Information Process- ing Systems, 21.
Figure 8: Results for Unhealthy Conversations and class antagonize.
A Appendix
0.48
The results for the WikiDetox Aggression and WikiDetox Toxicity datasets shown in Tab. and Tab. 3 respectively. The results for each sepa- class in Unhealthy Conversations dataset are presented in Figures 8-14. The evaluation results averaged over classes for the Unhealthy Conver- sations dataset are presented in Tab. 4. The class- specific values are described in Tables 5-11. The gains provided by our measures in comparison to the Random assignment for WikiDetox Aggression and WikiDetox Toxicity datasets presented in Fig. 15 and Fig. 16 respectively. The least number of annotations needed for our measures to obtain equal or better results than the Random assignment for WikiDetox Toxicity and Unhealthy Conversa- tions datasets are shown in Fig. 17 and Fig. 18 respectively.
0.43
0.38
0.33
score
0.28
0.23
Contreversy
0.18
Ratio Distance
Stronger
0.13
Average Minimal Conformity Random
0.08
10 11 12 13 14 Number annotations user
Figure 9: Results for Unhealthy Conversations and class condescending.
0.30
0.25
F1 score
0.20
0.15
Controversy
Var Ralio
Distance
0.10
Stranger Count
Minimal Conformity
Random
Number of annotations per user
Figure 10: Results for Unhealthy Conversations and class dismissive.

---

Number of annotations per user
Measure
2
5
7
10
12
13
Controversy
58.29
60.28
60.59
61.93
62.31
62.38
63.15
62.54
63.13
63.23
63.04
63.85
63.01
63.62
Ratio
63.53
62.89
63.55
63.74
Ratio Distance
56
56.67
57.02
56.96
57.44
57.69
58.23
58.68
58.36
58.75
58.73
59.50
59.83
59.87
Strager Count
56.39
56.55
57.04
57.74
58.33
58.81
58.95
59.68
60.09
60.30
60.44
59.82
60.67
60.88
Average Minimal Confornity
56.89
57.41
57.55
58.71
58.97
58.77
59.06
60.09
59.92
60.47
60.05
60.89
61.20
60.78
Random
56.50
56.71
57.42
57.77
58.41
58.75
59.33
59.98
60.44
60.99
60.74
60.71
60.56
60.86
Table 2: Results WikiDetox Aggression dataset, for different number of anmotations per user. Bold indicates the best result for a given number of annotations. Underlining indicates the best result for given measure.
Number of annotations per user
Measure
2
5
7
10
12
13
Controversy
57.54
58.55
58.56
59.67
59.79
60.22
60.22
60.22
60.71
61.46
60.99
61.16
61.29
60.95
Ratio
57.04
57.55
58.62
58.95
59.72
59.76
59.30
Ratio Distance
57.15
56.62
57.04
57.58
57.34
57.50
57.92
58.22
57.27
58.44
58.23
58.23
58.28
58.61
Surangr Count
54.92
53.92
54.24
54.89
54.79
55.72
56.46
56.16
56.18
56.09
56.17
56.15
56.72
56.89
Average Minimal Conformity
56.75
56.09
57.16
56.82
57.37
57.41
$7.53
57.31
57.85
57.98
58.08
58.58
58.61
58.74
Randam
56.62
55.89
55.73
$7.05
56.65
56.57
56.97
57.69
58.08
57.54
57.77
58.00
58.11
58.56
Table 3: Results on WikiDetox Toxicity dataset, for different number of annotations per user. Bold indicates the best result for given number of annotations. Underlining indicates the best result for a given measure.
Number of annotations per user
Measure
2
7
10
12
13
Controversy
21.77
27.75
29.44
30.44
33.54
34.34
Ratio
15.05
15.91
18.06
18.73
21.13
21.20
Ratio Distance
13.65
16.20
16.82
19.83
23.10
24.89
25.90
26.72
26.95
28.22
27.90
28.35
28.87
29.89
Stranger Count
14.17
16.30
18.97
19.96
20.92
21.66
22.05
23.17
24.30
24.83
25.04
24.95
25.33
25.62
Average Minimal Conformity
18.98
20.43
22.25
22.96
23.11
23.91
26.28
25.08
25.00
27.02
27.76
29 91
28.50
28.64
Random
14.17
17.87
20.92
20.80
21.38
23.70
22.60
24.57
26.82
26.82
27.68
28. 28.38
27.93
27.33
Table 4: Results Fl averaged for all classes on Unhealthy Conversations dataset, for different number of annotations per user. Bold indicates the best result for a given number of annotations. Underlining indicates the best result for a given measure.
Number unnotations per
Measure
2
4
5
7
8
10
11
12
13
14
Controversy
33.67
37.32
41.12
42.30
44.21
44.10
44.54
45.87
46.64
47.24
48.72
48.91
47.98
48.97
Var Ratio
25.63
26.15
29.00
27.35
26.85
25.61
27.69
29.75
28.29
31.98
29.42
29 29.99
31.75
32.21
Ratio Distance
24.07
25.30
25.13
29.72
33.05
34.30
35.24
36.62
37.11
37.88
38.31
38.26
38.77
39.57
Stranger Count
25.65
28.79
32.27
33.78
34.89
35.82
37.57
Average Minimal Confomitty
22.49
25.63
28.79
79
30.84
30.72
31.89
36.02
33.97
33.86
35.97
38.41
39.96
37,52
37.27
Random
25.93
29.78
34.12
33.19
34.10
35.72
34.45
36.73
40.14
39.90
40.42
40.99
41.63
40.43
Table Results F1 for class antagonize on Unhealthy Conversations dataset, for different number of annotations
per user. Bold indicates the best result for given number of annotations. Underlining indicates the best result for given measure.
Number annotations per user
Controversy
28.67
30.88
36.69
38.73
39.78
39.33
39.78
43.26
43.45
43.36
43.80
45.31
43.76
45.81
Var Ratio
21.08
21.43
24.77
22.31
24.55
24.96
25.63
24.12
24.09
26.77
24.58
26.11
28.05
30.23
Ratio Distance
14.83
17.11
17.30
19.92
23.43
25.41
26.70
27.99
29.18
29.85
31.04
30.52
30.83
32.60
Stranger Count
21.78
25,30
27.47
29.64
28.40
30.62
31.42
31.61
33.43
33.56
33.23
33.52
34.66
34.79
Average Minimal Confommity
18.65
22.58
27.67
27.61
28.63
27.98
30.16
27.54
29.75
32.56
31.42
35.73
34,00
33.72
Random
21.18
26.01
29.73
30.02
30.56
32.59
32.44
34.05
35.78
36.10
36.52
37.53
37.81
Table 6: Results of Fl for class condescending on Unhealthy Conversations dataset, for different number of
annotations per user. Bold indicates the best result for given number of annotations. Underlining indicates the best result for given measure.

---

Number annotations
Measure
2
4
5
7
8
9
10
11
12
13
14
Controversy
12.09
15.42
15.77
18.53
20.83
21.75
22.77
24.43
23.73
24.36
25.21
25.75
26.63
25.61
Var Ratio
4.75
5.42
7.75
6.82
10.16
8.64
9.55
9.20
12.06
13.41
12.23
12.54
12.07
13:55
Ratio Distance
14.23
15.14
15.31
18.25
19.82
22.59
24.35
26,06
25.15
26.23
26,21
26.49
26.59
28.45
Stranger Count
7.08
11.12
11.18
11.30
12.45
12.50
16.70
15.27
15.98
15.73
Average Minimal Confomity
21.62
21.27
16.65
17.84
20.28
19.43
21.35
20.92
21.45
22.67
22.72
23.78
23.33
23.53
Random
5,00
6.73
9.50
9.79
11.41
12.47
10.69
13.48
16.27
15.72
15.87
17.99
16.50
16.55
Table 7: Results of F1 for class generalisation on Unhealthy Conversations dataset, for different number of annotations per user. Bold indicates the best result for given number of annotations. Underlining indicates the best result for a given measure.
Number unnotations per
Measure
2
4
5
7
8
10
11
12
13
14
Controversy
8.15
9.94
11.96
14.39
14.27
17.29
18.81
22.06
20.57
22.43
21.98
22.78
24.11
23.05
Var Ratio
3.83
3.43
4.29
3.85
5.22
5.48
5.90
6.35
7.33
8.78
9:43
8.26
9.15
10.46
Ratio Distance
14.08
16.82
16.07
19.36
19.61
22.68
23.12
23.24
23,47
25.65
24.38
25.17
25.61
26.09
Stranger Count
4.55
8.82
9.42
9.22
11.87
11.32
11.79
Average Minimal Confomitty
23.21
20.92
17.52
19.10
16.23
18:53
17.57
18.03
15.72
17.93
19.09
20.59
18.08
20.29
Random
3.59
5.22
6.90
7.04
7.62
10.05
9.21
11.35
14.73
12.47
14,00
14.98
13.29
13.20
Table Results F1 for class generalisation unfair on Unhealthy Conversations dataset, for different number of annotations per user. Bold indicates the best result for given number of annotations. Underlining indicates the best result for a given measure.
Number unnotations per
Measure
2
4
5
7
8
9
10
11
12
13
14
Controversy
9.76
10.04
16.10
19.18
21.13
19.58
20.68
23.67
23.37
24.02
25.09
27.15
24.71
26.90
Var Ratio
13.35
14.98
17.67
16.82
19.87
18.32
18.07
20.75
19.39
21.44
18.64
19.73
20.79
22.02
Ratio Distance
7.09
11.07
14.76
18.65
23.30
24.13
25.12
25.16
25.62
29.04
25.82
26.50
28.68
27.97
Stranger Count
12.27
13.03
15.74
16.89
19.96
20.39
21.07
21.44
24.20
23.82
25.69
24.71
24.02
Average Minimal Confomitty
18.90
20.70
24.01
23.05
23,64
24.07
25.90
25.44
23.60
27.17
27.82
28.69
28.95
28.00
Random
12.99
16.61
18.52
19.84
19.28
22.34
22.42
24.18
23.03
25.34
27.84
27.88
25.86
26.96
Table Results Fl for class sarcastic on Unhealthy Conversations dataset, for different number of annotations per user. Bold indicates the best result for given number of annotations. Underlining indicates the best result for given measure.
Number annotations per
Measure
2
4
7
10
13
Controversy
21.54
25.10
29.71
32,14
34.62
34.21
35,30
36,80
36.06
38.59
38.55
39.90
38.24
38.82
Var Ratio
23.09
19.98
22.31
20.02
19.34
19.87
20.59
21.68
20.69
23.68
21.74
21.50
23.36
23.12
Ratio Distance
12.76
15.73
15.20
18.00
22.00
23.99
23.41
24.95
26.65
26.11
27.70
27.10
27.50
30.05
Stranger Count
15.85
17.23
19.32
19.20
20.04
20.43
20.24
22.61
23.52
23.67
24.47
24.47
24.36
Average Minimal Conformity
16.96
21.57
22.17
28.19
26.82
26.91
32.55
Random
20.00
24.56
26.57
25.84
25.95
28.06
27.52
28.05
30.73
32.09
32.31
32.39
33.10
32.22
Table Results of for class hostile on Unhealthy Conversations dataset, for different number of annotations per user. Bold indicates the best result for given number of annotations. Underlining indicates the best result for given measure.
Number annotations per
Measure
2
7
8
10
13
Controversy
19.39
20.45
28.26
28.98
31.24
31.94
31.19
34.95
33.36
34.79
34.45
35.87
34.97
35.75
Var Ratio
14.78
13.99
17.82
14.23
17.14
17.12
18.99
19.26
20.32
21.82
21.16
21.58
23.25
23.84
Ratio Distance
6.62
9.79
10.57
12.89
18.28
19.84
20.50
21.55
22.21
21.48
22.78
24.12
24.16
24.78
Stranger Count
13.68
18.12
21.33
21.76
23.6 64
25.57
25.05
27.36
27.34
28.35
27.93
28.11
29.20
Average Minimal Conformity
14.96
20.71
24.77
23.81
28.
27.79
Random
10.51
16.16
21.11
19.92
20.72
24.67
21.50
24.16
27.08
26.12
26,84
26.87
27.33
24.96
Table II: Results of for class dismissiie on Unhealthy Conversations dataset, for different number of annotations per user. Bold indicates the best result for given number of annotations. Underlining indicates the best result for given measure.

---

0.30
0.40
Controversy
0.35
Radle Distance
0.25
Stranger
Avcrage Minimal Conformity
Random
F1 score
0.25
F1 score
0.20
0.20
0.15
Controversy
0.15
Var
Ratio Distance
0.10
0.10
Stranger
Average Minimal Conformity Random
10 11 12 13 14 Number annotations user
1
2
3
13 14 Number of annotations per user
4
5
6
7
8
9
10
11
12
Figure 14: Results for Unhealthy Conversations and class sareastic.
Figure 11: Results for Unhealthy Conversations and
class generalisation.
8.0%
Controversy Var Ralio
0.40
Controversy
Radio Distance
7.0%
Stranger
0.30
Average Conformity
E7 score gain
Random
6.0%
0.25
0.20
5.0%
0.15
4.0%
0.10
0.05
3.0%
2
3
4
7
8 9
10
11
12
13 14
0.00
Number annotations per user
2
3
13 14 Number of annotations user
4
5
6
7
9
10
11
12
Figure 15: Gains for WikiDetox: Aggression. The Y axis shows the gain of Fl score for the positive class with respect to Random selection.
Figure 12: Results for Unhealthy Conversations and class generalisation unfair.
7.0%
Controversy
Var Ratio
6.0%
5.0%
score gain
F1 score
0.28
4.0%
0.23
3.0%
Controversy
2.0%
Var Ratio
Stranger Count
1.0%
Mirimal Confomityy
Rancom
2 3
8
9
10
11
12
13
14
0.08
12 Number of annotations per user
Number annotations per
Figure 16: Gains for WikiDetox: Toxicity. The Y axis shows the gain of score for the positive class with respect to Random assignment.
Figure 13: Results for Unhealthy Conversations and class hostile.

---

annotations active learning,
Controversy Var Ratio
No. of annotations with random assignment
The least number of annotations needed for given measure (Y axis) to achieve F1 score for the positive class not less than the Random selection (X axis) for the WikiDetox: Toxicity dataset.
Figure 17:
of annotations active learning
113
Controversy
12
Average Mirimal Conformity
10
8
7
5
3
1
2
5
8
10 11
13 14
No. annotations with random assignment
Figure 18: The least number of annotations needed for a given measure (Y axis) to achieve a F1 score for the positive class averaged across all classes not less than the Random selection (X axis) for the Unhealthy Conversations dataset.


Reviews:
[{"review_id": "9z4fBBsS5w", "reviewer": "Reviewer_X3Lx", "summary": "", "questions": "A. Can you add justification for using the XLM-RoBERTa model than other pre-trained models?\n\nB. Can you explain why particular strategies worked better/worse than others?\n\nC. Do you have any results with the ensemble approach? e.g., using multiple measures to score each example.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your review. We appreciate your help and will do our best to apply your suggestions to improve the quality of the article.\n\nThe main goal of our work was to design methods for optimizing the size of the train set. while keeping the performance similar to training on the full dataset. Personalization is a crucial approach to prediction of subjective perception of text. Hence, our second contribution was proposing the active learning methods to be used in subjective NLP tasks and our paper is the first to pose this problem. The main difference between general active learning methods used for non-personalized methods and personalized ones is respecting the user context instead of assuming the gold standard for each text. This is unique in the active learning domain, thus requires our dedicated methods.\n\nHaving regard to the completeness we collected comparative results for non-personalized methods with similar setups, but it will not provide much insight related to contextualized data collection.\n\nRegarding the new texts scenario, our test set always contained texts that did not appear in the training set or the dev set. This allows us to simulate a situation when new texts appear in the system. According to the results of our experiments, the training samples provided by our methods allowed the models to train and achieve good results on the new texts in the test set. This is due to the fact that the model used the similarity of vector representations of texts, obtained from the language model. This topic is further explained in section 6.1 and in Figure 4.\nAccording to your interpretation, the Past set represents the general knowledge about all users. It is a starting knowledge source used to prepare a starting model for each of the proposed methods. In the next steps, each method was used to select one new text for each user from the Present set in every iteration of the Active Learning algorithm (see Algorithm 1). After every iteration, the model was trained on the enriched dataset and then evaluated on the same (constant) dev (Future 1) and test (Future 2) sets. The conducted evaluation leveraged the cross-validation mechanism, which guarantees the model will be tested on users, for which the only data source is the small Past set.\n\nThe performance plateau for each dataset is similar to the one achieved by the best measure for the maximum number of annotations per user. That is why we selected the 14 users as the upper limit in presenting the results of our experiments. We will add appropriate information in the camera-ready version of the paper.\n\nAnswer to Question A: We used English language models during experiments, but their impact on results was insignificant compared to the multilingual model. What is more, the XLM-RoBERTa model achieved the best results out of all multilingual models used during experiments. Moreover, the use of multilingual model shows the possibility of applying our methods to various other languages and the higher versatility of our approaches. \n\nAnswer to Question B: We will add the more detailed analysis focused on explaining the results for each of the methods on the additional page in the camera-ready version of the paper. The higher values of the Var Ratio F1-score for the WikiDetox imply that the tasks in those datasets seem to be significantly easier to learn than the ones available in the Unhealthy Conversations dataset. In the case of the more difficult task, a simple intuition based on selecting the texts which users annotated with higher variability may prevent the model from gathering the basic  knowledge about the task and its relation to the language. These basic intuitions should be gathered from the texts, which are characterized by low variability of user annotations and they are considered as easy to learn and extract the most basic patterns useful for the task learned.\n\nAnswer to Question C: As this is quite a novel topic and there are no prior works regarding this matter, we plan to leverage the ensemble architectures synthesizing knowledge obtained via various measures in the future work.\n\nMoreover, we will revise the language and correct the typos in the entire content to make it more understandable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper was easy to read. The research questions were well justified, and the intuitions were effectively presented with figures.\n- Adding a user-level dimension to the scoring measures for the active learning application is an interesting idea.", "reasons_to_reject": "- Although the paper's main contribution is to suggest the benefit of personalized active learning methods, the analysis results lack comparison with generalized active learning strategies (e.g., ablation study without user dimensions).\n- It is unclear how the authors split the data with a relative timeframe (i.e., past, present, future1, and future2).\n- As I understand, the past set was used to build a background model, and the present set was used to update the model with different measures. However, Figure 4 shows that the past set includes the same user data from the future1 (dev) and the future2 (test) sets (in different timeframes). Is this a correct interpretation? If so, it would contradict the authors' intention of using user-level stratification (Section 5) for cross-validation design and cause some data leakage in the inference stage, as the model already knows something about users in the dev and test sets.\n- What's the performance plateau for each dataset? (e.g., how far would the model reach when it is trained with the entire training dataset?)", "questions_to_authors": "A. Can you add justification for using the XLM-RoBERTa model than other pre-trained models?\n\nB. Can you explain why particular strategies worked better/worse than others?\n\nC. Do you have any results with the ensemble approach? e.g., using multiple measures to score each example.", "paper_topic_and_main_contributions": "This paper presented a user-level active learning application. Unlike the generalized active learning approach, the authors claimed that their approach could improve the model's performance on subjective/personalized tasks. The authors compared five measures to select the training samples on two toxic conversation datasets. The results showed the 'controversy' (entropy-based measure) strategy works consistently better than other active learning and baseline strategies with their classifier.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In Section 6.4 (lines 424-426), are the use of 'present' and 'pastes' swapped as a typo?\n- I think the presentations of figures can be improved for better readability. For example, each strategy can be further distinguished with different line/point styles, or the baseline (random selection) can be shown in a dotted line, etc."}}, {"review_id": "LzykSvpOyi", "reviewer": "Reviewer_66GU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your review and emphasizing the promising direction of our research. In our opinion, the annotation process for the subjective NLP tasks is often more expensive and prone to unnecessary expenses than the annotation process for the generalized NLP tasks. Therefore, the development of personalized methods for cost optimization could provide even higher cost reduction that the ones used in generalized NLP problems. \n\nIn the camera-ready version of the paper we will also work on the language to make the article more understandable and readable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\nI think it is a reasonable step of applying active learning for NLP tasks. The motivation of this paper is clear. The authors provide a clear definition of the problem formulation, along with clear explanation for their proposed measures. The experimental results look fine.", "reasons_to_reject": "\nI do not have major reasons to reject this paper. The authors have pointed out there are no prior works for this problem, so that is probably the reason why a random baseline is used. But since the goal of this paper is not to achieve the state of the art performances in a task, so it is totally fine and reasonable.", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nIn this paper, the authors explore the possibility of applying active learning technics to subjective tasks. The authors try to address an important question that how to choose text to annotate so as to not only reduce the total cost, but also improve the downstream task performances. To do this, the authors identify 5 measures, including controversy and ratio distance, as metrics to select data to be annotated. Then the authors experiment with the Wikipedia Detox project and Unhealthy Conversations dataset. Results show that PALS methods could outperform a random baseline and require fewer training samples.\n\nIn general, I think it is an interesting work. There is a growing trend that brings methods from the active learning community to NLP these days, such as choosing which data to be included in the prompt for large language models. I would rather treat this paper as very timely, although this paper is studying a relevant problem: personalized selection of annotating texts for subjective tasks. Regarding the designed measures, I think they fit into the general goal of incorporating the users' preferences. Experimental design follows the literature in active learning and experimental results could support their claims.  ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y4j7F3KeYt", "reviewer": "Reviewer_FazH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your valuable suggestions and pointing out the risk of causing psychological damage to the users during the annotation process. This aspect is especially important in the case of annotation regarding the offensiveness of the texts as the abusive content may negatively affect the user's mental well-being.\n\nIn the future work, we plan to evaluate our methods on diverse datasets with various groups of annotators.\n\nMoreover, we want to employ an experienced copy editor in order to revise the language, and correct the typos in the entire content to make it more understandable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The results show a way to reduce the number of annotations required for subjective tasks (many of which can be psychologically damaging) while maintaining or even increasing accuracy. It is also important to bring attention to the problems with labeling for subjective tasks.", "reasons_to_reject": "As mentioned in the Limitations section, testing the method on a broader variety of datasets with more diverse annotators remains as future work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors present a method to select data for annotation for subjective tasks. Because the task is subjective, the threshold for what each person considers \"acceptable\" differs, so the proposed method chooses data using active learning so that the threshold for each person is more accurately determined given fewer annotations. The method is validated on three common datasets for subjective tasks. The results show that the selected data annotations yield better accuracy, even with fewer annotations.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I recommend going through the text with a copy editor as there are several grammar errors. For example, in the abstract, line 19 is missing an apostrophe, line 23 is missing the word \"the\", and line 24 has an extra \"even\"."}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the paper was well written and the experiments are sound. Reviewer X3Lx appreciated the user-level dimension to scoring. Reviewer FazH also pointed out that the results reduce the number of annotations required for subjective tasks, while keeping the accuracy level.

Reviewer X3Lx had most notable concerns including the lack of comparisons in findings.The authors mentioned conducting some analyses but suggested that they did it find useful insights. It was still a little unclear. Reviewer 66GU mentioned that they do not have a concern about a lack of a baseline. Reviewer FazH also thought that leaving testing of the method on a broader variety of datasets with more diverse annotators as a future work.
Decision: Accept-Main