Learning to love diligent trolls: Accounting for rater effects in the dialogue safety task
Authors: Michael John Ilagan
Venue: EMNLP 2023 Findings (2023)
Keywords: chatbots
trolls
safety
automated essay scoring
latent class analysis
TL;DR: For cleaning trolled training data of chatbot safety judgments, methodology from automated essay scoring overcomes limitations of cross validation, in terms of efficiency and (no GPU needed) robustness (accurate even with majority "diligent" trolls).

Abstract: Chatbots have the risk of generating offensive utterances, which must be avoided.
Post-deployment, one way for a chatbot to continuously improve is to source utterance/label pairs from feedback by live users.
However, among users are trolls, who provide training examples with incorrect labels.
To de-troll training data, previous work removed training examples that have high user-aggregated cross-validation (CV) error.
However, CV is expensive; and in a coordinated attack, CV may be overwhelmed by trolls in number and in consistency among themselves.
In the present work, I address both limitations by proposing a solution inspired by methodology in automated essay scoring (AES):
have multiple users rate each utterance, then perform latent class analysis (LCA) to infer correct labels.
As it does not require GPU computations, LCA is inexpensive.
In experiments, I found that the AES-like solution can infer training labels with high accuracy when trolls are consistent, even when trolls are the majority.

Learning to love diligent trolls: Accounting for rater effects in the dialogue safety task
Michael John Ilagan
McGill University Montreal, QC, Canada michael. ilagan@mail www.emmmmlmmmiilll.m worggill... ll.ca
Abstract
Grader
Grader 2
Grader 3
Essay 1
0
Chatbots have the risk of generating offen- sive utterances, which must be avoided. Post- deployment, one way for chatbot to continu- ously improve is to source utterance/label pairs from feedback by live users. However, among are trolls, who provide training exam- ples with incorrect labels. To de-troll training data, previous work removed training examples that have high user-aggregated cross-validation (CV) error. However, CV is expensive; and in a coordinated attack, CV may be overwhelmed by trolls in number and in consistency among themselves. In the present work, I address both limitations by proposing solution inspired by methodology in automated essay scoring (AES): have multiple users rate each utterance, then perform latent class amalysis (LCA) in- fer correct labels. As it does not require GPU computations, LCA inexpensive. In experi- ments, I found that the AES-like solution can infer training labels with high accuracy when trolls are consistent, even when trolls are majority.
Essay 2
0
Essay 3
0
Essay 4
1
Essay 5
1
Table I: An essay x grader inter-rater matrix (denoted Z in the text) of annotations. Here, the grades are binary (e.g. pass/fail). Each essay is assigned to a random subset of the graders. Rater effects must be accounted for, so that final score can be assigned to each essay for training an automated essay scoring (AES) engine.
uttcrance/label pairs sourced from users. But due to trolls (as well as honest mistakes), some of the training pairs have incorrect labels (i.e. y" A y). We thus endeavor to de-troll these labels, pre- processing step (Song et al., 2022). Once labels are corrected (i.e. y* = 3/, the problem reduces familiar supervised learning.
Toward de-trolling the training set, two- intervention solution was recently offered by Ju et al. (2022). Intervention 1 has each user pro- vide multiple (z, w") pairs, so examples are nested within users. Intervention 2 computes the "un- trustworthiness" of cach training example via (3- fold) cross-validation (CV). training example is removed if it exceeds some threshold of disagree- ment with the crowd. The intuition is that trolls end up having a bad track record against the crowd but occasionally, by chance, provide examples worth keeping. The (xr, v²) pairs remaining then suit- able for training.'
1 Introduction
For chatbots, it is not enough to learn to gener- ate coherent utterances- many coherent utterances are undesirable, such as offensive remarks. Post- deployment, to have the chatbot continuously im- prove its judgment, one way is to source training amples from feedback by live organic users (Shus- ter et al., 2022; Xu et al., 2022). Though more realistic than crowdworkers, among organic users are trolls, whose erroneous feedback fosters bad behavior (Wolf et al., 2017). How can the chatbot learn continuously from user interactions in way that is robust to trolls?
There are two issues with this CV-based (Ju et al., 2022) solution. First, as it involves CV on large lan- guage model (LLM) computations, it is expensive. Second, CV relies upon abundance of helpers (Ju et al., 2022), good-faith users who end up
The problem can be cast dialogue safety task (Dinan al., 2019; Noever, 2018): Given input utterance æ, output its binary class label y, either safe (denoted y = 0) or unsafe (denoted = 1). The training set is then a set of observed (x, y*)
'Alternatively training pairs exceeding the threshold are kept but with their labels flipped (Ju et al., 2022).

---

agreement with each other. Thus, this approach can be overwhelmed in the event of a coordinated attack where trolls have both strength in numbers and are consistent among themselves.
making the classifier robust. Xu al. (2022) con- sidered various methods to improve chatbots in deployment from different forms of human feed- back, including free-form text. An emerging in- novation is architectures that are self-monitoring, being both generator and classifier (Arora et al., 2022; Lu et al., 2022). But as such architectures assume that training examples have correct labels, they are solutions for supervised learning, not de- trolling. Addressing the problem takes more than finding a state-of-the-ee classifier.
A promise of circumventing both issues lies in methodology for automated essay scoring (AES) (Attali and Burstein, 2006; Taghipour and Ng. 2016; Uto, 2021). Given an input essay x, an AES engine (which may incorporate an LLM) must out- put a score y, a numeric or ordinal rating of the essay's s quality. The burden of annotating a train- ing set is divided among multiple human graders, which incurs rater effects- vary on how strict or permissive they are, which adds variance to automated scoring (Wind et al., 2018; Uto and Okano, 2020).
The threat posed by trolls falls under the um- brella of robust learning (Song et al., 2022). Whereas the CV-based procedure (Ju et al., 2022) is one way to do robust learning, another is inrectation-maxmmizatte
(EM) algorithms (Do and Batzoglou, 2008; Boos and Stefanski, 2013), of which LCA fitting (Linzer and Lewis, 2011) is an instance. In statistics, EM a local optimiza- tion technique for fitting models involving latent variables, such as unobserved class labels. An ele- mentary instance of EM is the familiar k-means clustering algorithm (Do and Batzoglou, 2008). More sophisticated instances of EM are applicable to problems with untrustworthy training examples and rater effects, such as crowdworkers' tagging of images and documents (Ipeirotis et al., 2010; Raykar and Yu, 2012).
Analogously, the AES solution has two inter- ventions (Uto and Okano, 2020). Intervention 1 induces redundancy by randomly assigning mul- tiple graders (users) to each training essay (utter- ance), yielding inter-rater matrix Z as in Table 1. Intervention 2 fits rater effects model to Z, a statistical (as opposed to neural) model, which yields inference, for each unique essay (in dialogue safety: utterance), a single final score (in dialogue safety: class label) ÿ suitable for training. Such a solution is inexpensive, being free of embeddings the essays' content is not incorporated in Z (as in Table 1) and is not needed to infer ÿ. In the case of discrete ratings, the rater effects model is called a latent class analysis (LCA) model (Linzer and Lewis, 2011).
The use of AES in large-scale educational as- sessment dates back decades (Attali and Burstein, 2006). The AES literature goes deeper-as es- says may have been written in response to different prompts, some not appearing in training, and some assessments score essays in multiple criteria (Uto, 2021). But in the present work, the most basic AES setup suffices for the analogy- tion y (the class label or score) is a commonsense holistic evaluation of the safety or toxicity of x (the utterance or essay).
In the present work, my contribution is twofold. First, I establish connection between the chat- bot robust learning (e.g. Dinan et al., 2019) and the AES (e.g. Attali and Burstein, 2006) literatures. Second, conduct an examination of an AE'- solution to de-trolling the training set. The main re- sult that this solution thrives on consistency, even among trolls: when trolls are consistent among themselves and are majority of the users, inferred labels are accurate; but curiously, when trolls are majority but make only random noise, the AES-like solution is not as accurate.
annota-
While the present article is focused on toxicity or safety (Dinan et al., 2019; Noever, 2018), there are other desiderata for chatbot behavior, such as factual correctness and staying on-topic (Xu et al., 2022; Arora et al., 2022).
2 Related work
3 From ratings to predicted classes
To motivate LCA, consider Table 1, an intuitive case. By inspection, Graders 1 and 2 are in agree- ment, and Grader 3 goes in the opposite direction. 2
In open dialogue models, the capacity to improve past pre-training is desired. Dinan et al. (2019) found that challenging crowdworkers to "break" a pre-trained classifier with adversarial inputs in follow-up rounds of training was helpful toward
am not suggesting that educational assessments risk trolling, as they are not crowdsoureed. Non-troll rater effects

---

prior: 0.7
prior: 0.3
opposite directions, and which is the "correct di- rection" cannot be inferred from merely looking at Z, as in Table 1. To be able to map clusters to classes in the present work, I make the assumption that the cluster having more utterances is the safe class. Such an assumption is true of the dataset (Dinan et al., 2019) used in experiments in the next section. Furthermore, if the training examples are feedback the chatbot's own utterances (Xu al., 2022), then any decent chatbot should produce safe examples more than half the time.
Cluster A
Cluster B
likelihood
likelihood
1
0
1
user 1
0.91
0.19
user
0.14
0.86
user 2
0.35
0.65
user
0.73
0.27
:
user m
0.76
0.24
user
0.28
0.72
Figure I: A latent class analysis (LCA) model for dia- logue safety, Suppose Cluster A the safe cluster. Then User must be helper, as they have 91 %% probability of labeling a safe utterance as such, and they have an 86% probability of labeling an unsafe utterance as such, From similar reasoning. User 2 must be a troll. But if Cluster A were unsafe instead, the roles are the oppo- site.
Altogether, the AES-like solution consists of (Intervention 1) introducing redundancy, then (In- tervention 2) statistically leveraging it via LCA with the safe-as-majorit assumption (LCA+SM). The end result is corrected training pairs (x; 7)- The de-tro de-trolling solution is inexpensive because it is embeddings- and it is robust because user tendencies are accounted for, without taking for granted that the consensus is correct. Unlike the CV-based solution (Ju et al.. 2022), no untrustwor- thiness scores need be calculated, as user tenden- cies are already accounted for. No examples are removed from training the intuition is that user that reliably gives incorrect answers is perfectly good data once their labels are flipped.
Accordingly, there are two clusters: those rated "0" by Grader 3 (i.e. Essays I and 2); and those rated "0" by Graders and 2 (i.e. Essays 3 to 5). Realis- tic instances of Z are not clear-cut, it takes statistical model to apply such intuitions scale.
The LCA model depicted in Figure 1. De- spite the name, LCA deals with clusters rather than classes. According to this model, each utterance belongs to one of two clusters, A and B, with a probability distribution (the prior). Within each cluster, the distribution of the observed label (the likelihood) is user-specific: each user has some probability of labeling a Cluster utterance as safe, and some probability of labeling a Cluster B utterance as s
Experiments
4.1 Methods
Experiments were conducted to test the efficiency and robustness of the AES-like solution under dif- ferent troll scenarios, especially extreme cases of trolls being majority not covered in Ju et al. (2022). Real utterances were labeled by synthetic users. The real utterances I, with gold labels y known, were from Meta Al's single-turn safety dataset (Dinan et al., 2019). Unfortunately due to com- putational resource constraints, there was no di- rect comparison to the CV-based solution in Ju et al. (2022) Instead, the proposed Intervention 2, LCA+SM, was compared to bascline of major- ity vote (MV) on each row of Z. Both solutions leverage redundancy, though MV naively takes for granted that the consensus is correct; and both di- reetly impute classes rather than rank examples by untrustworthiness.
An EM algorithm (Linzer and Lewis, 2011), pro- vided Z, estimates all parameters (in both prior and likelihood). The intuition is that users' tendencies can be quantified due to utterances shared between them. Provided these estimates, for each utterance. the posterior distribution can be computed, which is the answer to the question, "Given the observed pattern of labels (i.e. row of Z), which cluster is more probable?" Each utterance is then assigned to the highest- probbabitt cluster. Note that LCA fit- ting is possible even amid missing values in Z, but some redundancy (from Intervention 1) is required (see Limitations section for details).
However, clusters are not yet classes. In the dialogue safety task, trolls and helpers behave in
Two Experiments followed the same 2 2 2 x
exist, and Table resembling trolling simply to serve the analogy to the dialogue sufety task.
*CV-based trustworthiness scores are a function of two hyperparumeters (one of them is the threshold) that themselves require tuning with a validation set. Thus, there are two layers of validation.
'Altermatively, the likelihood can be expressed in terms of user-specific probahility transition matrices, as in Ju et al. (2022).

---

design, varying the following factors:
supervised learning solution. For instance, bad per- formance might be blamed on a poor neural clas- sifier or on utterances being intrinsically difficult, rather than the merits of the de- de-trolling solution itself. Thus, Experiment redid the de-trolling phase with the same settings except that the metric was instead the accuracy of the imputed labels in the training set-I call this metric imputation accu- racy." The test set is thus irrelevant. Unburdened by the expense of training large language model, each scenario had 500 runs.
UNSAFE PREVALENCE: Either 10% or 30% % of the utterances were unsafe, in both training and validation sets.
TROLI_PREVALRNNEE Either 50% or 90% of the users were trolls.
When corruption of the label occurred, it was either "diligent" (y² 1 - y/ or "lazy" (randomly choose between and with equal probability).
CORRUPT ACTTON:
4.2 Results
With GPU for the supervised learning phase, Ex- periment ran for -12.1 hours. In contrast, without need for GPU, Experiment 2 ran for ~2.2 hours. Figure 2 shows the results for both Only results for UN- SAFE_ PREVALENCE=30% are shown. The patterns for UNSAFE UNSAAE_ PREVALENCE-10% are similar and can be found in the Appendix.
TROLI_ RATE: Each troll cor- rupted either 80% or 95% of their labels.
Experiment 1 had both de- de-trolling phase and supervised learning phase.
Experiments.
1. De-trolling, 200 utterances were sampled as training set, and Z was created by ran- domly assigning each utterance to 5 users out of pool of 50 users. Thus, Z had 90% of its cells missing. To incorporate honest mis- takes, helpers corrupted 5% of their labels in all conditions. Both competing methods (i.c. LCA+SM and MV) were applied to yield (x, Ø) pairs.
Trends were similar between the two Exper- iments, so describe them together. Keep mind that when CORRUPT_ "CTTUP"_ATTON="dlliiee"""" higher TROLL_ COOLL_CORRUTT_AAEE RATE means that trolls are more consistent among themselves. as they are strongly negatively correlated with ground truth; but when CORRUPT_ DEARUUTTACTIOO==lary higher TROLL_CORRUPT_
means that trolls are less consistent among themselves, due to their randomness. The overall picture is that LCA+SM leverages consistency in the user-provided labels, even malicious consistency from trolls; but LCA+SM falters when there is little consistency to leverage. Under COR- LCA+SM was highly RUPT_ACTION="dilieee SOT-ORRRRPPIRAEE-55555 and accurate when TROLL_ TROLL_ DOOUL__PREVALENCL=990 always better than the baseline; but under CORRUPT_ACTION= LCA+SM was highly inaccurate for the same com- bination, sometimes worse than the baseline. Ma- licious as trolls are, the diligent ones are just as valuable as helpers-ther should not be removed from training.
2. Supervised learning. 24 utterances were sam- pled as validation set, with gold labels known for simplicity. classifier was trained on the corrected training set, with the validation set used for early stopping. Constant across all runs, the test set was the same as in Ju et al. (2022), which had 900 safe utterances and 100 unsafe utterances.
Accordingly, the evaluation metric in Experi- ment was test set accuracy. Due to the computa- tional expense, there were only 5 runs per scenario.
To fit LCA, used the package mirt mirt in R. For training, used ParlAI (Miller al., 2017), using the same settings as in Ju et al. (2022) except for omitting CV and setting max max-train-steps=400 to keep the experiment manageable. The neural in ParlAl's Model Zoo, consisted of linear layer top of pre-trained transformer from Humeau al. (2019).
Discussion
model, called bi
How much data should Z have for LCA to reli- able? While the proportion of missing values was
While evaluating downstream accuracy is usu- ally sought, an issue in Experiment is that such a metric conflates the de-trolling solution with the
'The connotations of overfitting and over-optimism usually associated with "training set accuracy" are not applicable here. Yes, hoth LCA+SM and MV are used on the training set; but hoth are unsupervised and do not overfit.

---

Unsele prevelencer: 30%
required. Note that having a very low non-missing proportion may make fitting LCA impossible.
Ial anserecce S0%
tsll pnen comepp aolore
2:5 Yot 50%
contul aidiiiie
While the CV-based (Ju et al., 2022) solution's 's issues were not directly experimentally validated, there is reason to believe that it would have been at a disadvantage against the AES- solution. In terms of expense. CV would involve training a large language model, even at the de-trolling phase. In terms of robustness, CV can be thought of simulating "checking each other's work" or tap- ping into a consensus in a way that MV directly does, so the CV-based solution would have been likewise vulnerable to a false consensus. Scenarios where the AES-like solution faltered had little con- sensus, which would have been unfavorable to the CV-based solution as well.
08-
08-
04
181 40ORES LCA-0A
0.-
03-
0.0
3.54
0.5
176
325
120
335
als
Unsofe prevolence: 30%
Inai inviintta 50%
eisl peli 50%
solp voliin l05
Unlike Ju et al. (2022), my Experiments did not manipulate the prevalence of adversarial vs. non- adversarial utterances sampled. In the dataset (Di- nan et al., 2019), half of the utterances were adver- sarial. These prevalences would have affected the supervised learning phase but not the de- de-trolling phase, as MV and LCA+SM are embeddings free. It turns out that Experiment 2 was a purely statis- tical exercise, though one relevant to the de-trolling problem. After all, producing Z and evaluating im- putation accuracy did not reference any utterance content, so no real data was needed. Consequently, the results for imputation accuracy are widely gen- eralizable in the sense that they are irrespective of the choice of neural classifier or the intrinsic difficulty of the utterances being classified. In ad- dition, Experiment 2 can be expediently adapted to different settings (e.g. UNSAFE PREVALENCE and number of training utterances) for researchers to mess around with. But for specific applications involving a specific dataset and neural classifier, downstream accuracy becomes of interest.
dulberse
0.-
Irpudaon 022 LCK 4H
044
somppt seragrroes
cam
030
030
1:0
1/5
300.00
3.5
0.5
13 325 05s ols aio 10
Inpultiio DCCARDS
Figure 2: Results for both Experiments, for 30% unsafe prevalence. Within each Experiment, each panel a see- nario. Solid black line is the identity line-when a point is above the line, LCA+SM did better than MV. Dotted lines mark the accuracy of indiscriminately predicting safe. LCA+SM = latent class analysis with safe-as- majority assumption; MV = majority vote.
Conclusion
not manipulated in the Experiments, exploratory analysis found that larger non- non-missing proportions were more favorable: increasing the users assigned per utterance was beneficial for fixed total user pool; but for fixed number of users per utterance, increasing the user pool was detrimental. While having more data is statistically expedient, it also imposes a greater burden on the user (Shuster et al.. 2022). In these Experiments, having 10% of the inter-rater matrix filled meant that each user la- beled on average 20 utterances, which is 5x what the Ca-based approach (Ju et al., 2022) would have
The AES-like solution is inexpensive, it does not require GPU computation.
In the event of coordinated attack where trolls are majority and consistent among themselves, the AES- like solution holds up well, bearing the wis- dom that "diligent" trolls may be leveraged toward better performance. But when there is little con- sensus among users, there is little to work with, so de- deetrooling accuracy drops.

---

The AES-like solution proposed inherently has higher data requirements, which imposes bur- den on the users. Logistical and privacy concerns may be raised in light of having users "check each other's work", especially when example utterances are from users' own conversations with the chatbot. R. Philip Chalmers. 2012. mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6):1-29,
Limitations
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing -MNNLP) pages 4537-4546, Hong Kong, China. Association for Com- putational Linguistics.
To map clusters to classes, the present work as- sumed SM. MV and the CV-based (Ju et al., 2022) solution put trust in good-faith behavior of users, whereas LCA+SM gains robustness by putting trust in the chatbot's decency instead. Either way, trust has to be put somewhere. That said, even if SM were true more generally, it could still fail to hold for some batches in continual learning, if the batches are small enough.
Chuong B Do and Serafim Batzoglou. 2008. What is the expectation maximization algorithm? Nature Biotechnology, 26(8):8978899
Samuel Humeau, Kurt Shuster. Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Trans- former architectures and pre-training strategies for fast and accurate multi-sentence scoring.
Not every inter-rater matrix Z can be fitted to LCA. To make fitting possible, there are two re- quirements (Chalmers, 2012).
Panagiotis G Ipeirotis. Foster Provost, and Jing Wang. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '10 pages 64-67. 67, New York, NY, USA. Association for Computing Machinery.
(1) As each column (i.e. user) is associated with two parameters, the number of rows (i.e. utter- ances) in must be at least twice the columns.
(2) Each column must have both possible labels.
Da Ju, Jing Xu, Y-Lan Boureau, and Jason Weston, 2022. Learning from data in the mixed adversarial non- adversarial case: Finding the helpers and ignoring the trolls.
If (2) is violated by a column, that column may be deleted before fitting, as long as (1) is still met. Grave scenarios of trolling may yield many invalid columns, deletion of which would make fitting LCA impossible.
Drew A Linzer and Jeffrey B Lewis. 2011. pol CAA::: An package for polytomous variable latent class analysis. Journal of Statistical Software, 42(10).
Hua Lu, Siqi Bao, Huang He, Fan Wang. Hua Wu, and Haifeng Wang. 2022. Towards boosting the open-
Acknowledgements
domain chatbot with human feedback.
Inspiration for the present topic came from a con- versation with Nicholas Meade. The use of LCA came from a suggestion by Carl Falk.
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bor- des. Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79-84, Copenhagen, Denmark. Association for Computational Linguis- tics.
References
Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2022. Director: Generator-classinn for supervised language modeling. In Proceedings of the 2nd Conference of the Paciffc Chapter of the Association for Computaaional Linguistics and the 12th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 512-526, Online only. Association for Compu- tational Linguistics.
David Noever. 2018. Machine learning suites for online toxicity detection. CoRR, abs/1810.01869.
Vikas C Raykar and Shipeng Yu. 2012. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Re- search, 13(1):491-555
Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v.2. The Journul of Technology, Learning, and Assessment, 403):1-31.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju. Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur. and Jason Weston 2022. Blenderbot 3: a deployed
Dennis D Boos and L A Stefanski. 2013. Essential Statistical Inference: Theory and Methods. Springer, New York.

---

conversational agent that continually learns to respon- sibly engage.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2022. Learning from noisy labels with deep neural networks: survey. IEEE Transactions on Neural Networks and Learning Sys- tems, pages 1-19.
Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 1882-1891, Austin, Texas. Association for Computational Linguistics.
Unzofe
piva veca 50%
SOee DECA 50%
Masaki Uto. 2021. A review of deep-neural automated essay scoring models. Behaviormetrika, 48:459-484.
Masaki Uto and Masashi Okano. 2020. Robust neural automated essay scoring using item response theory. Artificial Intelligence Educution 2020, pages 549-561, Cham. Springer International Publishing.
omaginte
Stefamie A Wind, Edward W Wolfe, George Engelhard Jr., Peter Foltz, and Mark Rosenstein. 2018. The influence of rater effects training sets on the psy- chometric quality of scoring for writing assessments. International Journal of Testing, 18(1):27-49.
637
055
6.5
125
01s 12a
035
675
9.5
375
Iestetaccu/aCy MV
Unsale prexelence:
Marty J. Wolf, Keither W. Miller, and Frances S. Grodzinsky. 2017. Why we should have seen that coming: Comments on Microsoft's Tay "experiment," and wider implications. ACM SIGCAS Computers and Society, 47(35-44-64
Jing Xu, Megan Ung. Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. 2022. Learning new skills after deployment: Improving open-domain internet-drvee dialogue with human feedback.
corutudio siitio
AEPICTIICE BA
A Appendix
Figure 3 shows the results for UN- SAFE 10%
030
4.5
430
3.5
3.35
0.a 635 10000
1/s 100.00 035 020
ola 10
arpulakkn acOuRIS MV
Figure 3: Results for both Experiments, for 10% unsafe prevalence. Within each Experiment, each panel is see- nario. Solid black line is the identity mire-when point is above the line, LCA+SM did better than MV. Dotted lines mark the accuracy of indiscriminately predicting safe. LCA+SM = latent class analysis with safe-as- majority assumption; MV majority vote.


Reviews:
[{"review_id": "GSlyMzkB9I", "reviewer": "Reviewer_5UUg", "summary": "", "questions": "A. Line 194: What is rationale behind the setting of max-train-steps? Have you tried other values and did you see any difference?\n\nB. Why did you design two experiments? What is your conclusion when you compare the results of these two experiments?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "It is clear that Reviewer understood the present paper, based on their summary of it as well as their questions.\nAddressing their questions will certainly improve the present paper.\nThank you.\n\n# Two experiments?\n\nLet me tackle Question B first.\nThe Reviewer said:\n\n> B. Why did you design two experiments? What is your conclusion when you compare the results of these two experiments?\n\nThe two experiments have the same scenarios, but they use different metrics.\nLet me explain each one.\n\n**Experiment 1: downstream accuracy.**\nThis was done to have a metric in line with Ju et al (2022).\nTo evaluate their proposed de-trolling method, what Ju et al (2022) did was: apply it on a trolled training set, fine-tune a classifier on the resulting de-trolled training set, then compute accuracy on a test set.\nBut there's an issue---test-set accuracy conflates the performance of the de-trolling method itself with the subsequent fine-tuning (Line 283).\nOne might contend that, for instance, good results for my proposed AES (automated essay scoring)-like solution are just because it was carried by a good pretrained model or an easy discrimination task (say, the unsafe utterances are obvious insults).\n\n**Experiment 2: imputation accuracy.**\nFor a more direct evaluation, I simply computed the accuracy of the imputed labels in the training set (Line 233).\nConsequently, evaluation was disentangled from fine-tuning.\nNote that the imputation methods compared, LCA (latent class analysis) and MV (majority vote), are both unsupervised.\nCalling the metric \"training set accuracy\" (Line 233) is regrettable, as there is really no concern that the evaluation understates error.\n\nAnother asset of Experiment 2 is that because it avoids LLM fine-tuning, it is cheap. Evaluation is then more precise, done with many many more runs (Line 291).\n\nAltogether now, what can we conclude?\nFrom two separate metrics, the conclusion is the same:\nUnder the AES-like solution, diligent trolls are not a threat, even if they are a majority.\nTo see this, notice that in the diligent-troll panels (first two columns) of Figure 2 (above Line 242), the points are far above the horizontal line (marks 70% accuracy, if you predict the majority class all the time) and far above the identity line (marks equal accuracy between LCA and MV).\n\nWhy do Experiment 1 at all?\nWhile it had clear limitations, I imagine that if I had not included it, reviewers would have penalized the present paper, due to the metric's precedence in Ju et al (2022).\nI will be perfectly happy to relegate Experiment 1 to the Appendix, or remove it entirely, if the Reviewer recommends doing so.\n\n# Max train steps?\n\nNow for Question A.\nThe Reviewer said:\n\n> A. Line 194: What is rationale behind the setting of max-train-steps? Have you tried other values and did you see any difference?\n\nThe parameter `max-train-steps`, which is for fine-tuning with the imputed labels, was set to 400 (Line 195) simply due to limited GPU resources.\nNo, I have not tried other values. (Sorry, new experiments ran addressed other Reviewers, and there was not much time.)\nBut I do not think of it as a huge issue.\nNote that the two methods compared, LCA and MV, use the same setting, so neither has an unfair advantage in Experiment 1.\nIf I were to speculate, more training steps (short of overfitting or catastrophic forgetting) might benefit test-set accuracy in situations where training accuracy was good but might harm test-set accuracy where training accuracy was bad.\nIn Experiment 2, fine-tuning was avoided, so the concern of \"not enough training was done\" (or even \"there was too much\") disappears.\n\n# The merit of studying diligent trolls\n\nThe Reviewer said:\n\n> the LCA method only outperformed the baseline when the corruption action is \"diligent\", which is not very realistic in practice---it seems a lot of trolls just give random answers, corresponding to the \"lazy\" corruption action.\n\nA citation would help to convince me that diligent trolling is rare enough to be ignored.\nOn my end, I humbly offer that nonrandom adversarial labels are a perfectly respectable topic in crowdsourcing contexts, as in Wang et al (2014).\nAlso, the trolling behaviors considered in Ju et al (2022) are not all random.\n\nI must emphasize that the insight offered by the present paper---that even trolling behavior can be leveraged to benefit classification performance---is counterintuitive. \nKeep in mind that Ju et al's (2022) CV (cross-validation)-based method simulates tapping into wisdom of the crowd to de-troll data. \nMy work dares to ask, what if the consensus itself is tainted? \nOne could be forgiven for supposing that a diligent-troll majority would be a hopeless situation---a checkmate, then our chatbot devolves into Tay.\nAnd in fact, this intuition is supported in the results for MV, as seen in Figure 2 (above Line 242). \nBut it turns out, with LCA, there may still be hope. \nSo indeed, we can learn to love diligent trolls. \nI plead the Reviewer consider, would you have easily thought so after seeing Ju et al (2022) but before seeing the present paper?\n\n# Data requirements, safe-as-majority\n\nThe Reviewer said:\n\n> As the author mentioned, the proposed method has its limitations: it has high data requirements, and it just simply assumes the safe class is the majority of the utterances.\n\n**On the data requirements.** \nPerhaps the Reviewer will agree that resorting to \"more data\" is instinctive to many researchers---and to them, the present paper's results are of interest.\nMy results show that \"more data\" is not enough---MV uses the same data as LCA, but MV is more easily misled by a trolled consensus.\nThere is a wise way of using more data, and there is an unwise one.\n\n**On the safe-is-majority assumption.** \nCorrect inferences must come from somewhere: either one leverages large representative data (so assumptions are unnecessary); or one leverages strong correct assumptions (so even less-than-large data will work).\nRobust continual learning amid trolls is precisely a context where we refuse to put all our trust in data, so some assumptions are necessary.\nIn Ju et al (2022), the assumption was that most users act in good faith.\nIn the present paper, the assumption is that the chatbot has a base level of decency (Line 161).\nTo parties who have a system ready for deployment but are wary of trolls (Line 27), I suspect that the latter assumption is more palatable than the former.\n\nAll that said, I look forward to future work remedying both limitations.\nKeep in mind that I have made an effort to evaluate the AES-like solution in a way that disentagles it from fine-tuning LLMs.\nWith the basic solution established, the next step might be to reintegrate LLMs to avoid assumptions on which class is more prevalent (Line 162).\n\n----\n\n**References**\n\nJu, D., Xu, J., Boureau, Y.-L., & Weston, J. (2022). Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls. https://doi.org/10.48550/ARXIV.2208.03295\n\nWang, G., Wang, T., Zheng, H., & Zhao, B.Y. (2014). Man vs. machine: Practical adversarial detection of malicious crowdsourcing workers. In 23rd USENIX security symposium, USENIX Association, CA. https://www.usenix.org/system/files/conference/usenixsecurity14/sec14-paper-wang-gang.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a simple and straightforward method to address the problem of trolling in assessing the safety of chatbot utterances.\n- Compared with existing methods, LCA is cost-friendly and does not require GPU resources.", "reasons_to_reject": "- As the experimental results show, the LCA method only outperformed the baseline when the corruption action is \"diligent\", which is not very realistic in practice---it seems a lot of trolls just give random answers, corresponding to the \"lazy\" corruption action.\n- As the author mentioned, the proposed method has its limitations: it has high data requirements, and it just simply assumes the safe class is the majority of the utterances.", "questions_to_authors": "A. Line 194: What is rationale behind the setting of max-train-steps? Have you tried other values and did you see any difference?\n\nB. Why did you design two experiments? What is your conclusion when you compare the results of these two experiments?", "paper_topic_and_main_contributions": "Inspired by the automated essay scoring (AES) method, this paper proposes to perform latent class analysis (LCA) to infer correct labels in the scenario of de-trolling the training data of utterances generated by chatbots. The presented work is claimed to address the two limitations of the cross-validation (CV) method: being costly, and the risk of being overwhelmed by trolls. The proposed solution first generates the inter-rater matrix Z, and then fits a rater effects model to Z, so that it could infer the label of each utterance suitable for training. Experimental results show the advantage of LCA over majority vote (MV) when the corruption action is designed to be diligent.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6byTiSub4W", "reviewer": "Reviewer_U2Js", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for this review.\n\n# Recap of the experiments\n\nThe Reviewer said:\n\n> The writing of the paper is not very clear. The experiments part is suggested to be organized better.\n\nI'll take this opportunity to recap the experiments, to give myself and the Reviewer another chance at a shared understanding.\n\nWhen faced with trolls, we want to do two things.\nFirst, we want to de-troll the observed labels.\nSecond, we want to carry out supervised learning as normal, with the de-trolled data (Lines 42--44).\n\nTo evaluate two imputation methods, MV (majority vote) and LCA (latent class analysis), Experiment 1 did as follows in each run (Lines 177--220).\n\n1. From Meta's single-turn safety dataset, sample utterances with known gold labels. The result is the training set and the test set.\n2. Assign training set utterances to users, who label it safe vs. unsafe. \nSome users are trolls, while others are helpers. The scenarios (Lines 203--210) controlled how prevalent trolls were and how they behaved. \nThe result is the inter-rater matrix $Z$ (similar to Table 1, above Line 42).\n3. For each utterance (i.e. row of $Z$), impute (MV or LCA) a de-trolled label.\n4. Using the de-trolled data, do supervised learning as normal. \nMetric is test set accuracy.\n\nBut there's a limitation to using test set accuracy.\nWhat is of primary interest is the immediate quality of the imputation procedure,\nbut the metric conflates that with the quality of the fine-tuned classifier (Line 283).\nOne might contend that good results for my proposed AES (automated essay scoring)-like solution are just because it was carried by a good pretrained model or an easy discrimination task.\n\nThus, I did Experiment 2, where I evaluated the accuracy of the imputation itself (not of the classifier) in the training set.\nSteps 1--3 above are the same, but Step 4 is replaced by a comparing gold vs. imputed labels on the training set (Line 233).\nNote that \"training set accuracy\" is a regrettable name, as the imputation procedure (MV or LCA) is unsupervised---there is no worry that error is understated.\n\n# Is it trivial?\n\nThe Reviewer said:\n\n> The innovation of the algorithm is trivial.\n\nI plead the Reviewer give an explanation of this comment, preferably backed by a citation.\n\nAES (automated essay scoring) has long history (Attali & Burstein 2006), yet the reference (Uto & Okano 2020) inspiring the present paper's methodology is recent.\nI would hesitate to call it \"trivial\" in the AES context, and I would likewise hesitate to call \"trivial\" its application to trolls.\n\nI must emphasize that the insight offered by the present paper---that even trolling behavior can be leveraged to benefit classification performance---is counterintuitive. Keep in mind that Ju et al's (2022) CV (cross-validation)-based method simulates tapping into wisdom of the crowd to de-troll data. My work dares to ask, what if the consensus itself is tainted? One could be forgiven for supposing that a diligent-troll majority would be a hopeless situation---a checkmate, then our chatbot devolves into Tay. And in fact, this intuition is supported in the results for MV, as seen in Figure 2 (above Line 242). But it turns out, with LCA, there may still be hope. So indeed, we can learn to love diligent trolls. I plead the Reviewer consider, would you have easily thought so after seeing Ju et al (2022) but before seeing the present paper?\n\nClearly, Meta AI (Dinan et al 2019; Ju et al 2020) has a stake in their counter-trolling efforts working well.\nAs a lone researcher, I humbly offer the aforementioned insight as my piece of the puzzle.\nIf this insight is underappreciated in their recent work (Ju e tal 2022), wouldn't NLP researchers be interested to hear that?\n\n# Other comments\n\nThe Reviewer said:\n\n> The AES-like solution proposed has inherently higher data requirements, which might be burdensome for users.\n\nThe higher data requirements will not stop everyone worried about trolls.\nAt any rate, I must emphasize that having the multiple-raters-per-utterance redundancy allows us to tap into a more concrete (or real) consensus, avoiding the simulated consensus from CV---and doing so allows us to evaluate the imputation procedure itself, disentagled from the power of the pretrained models in the background.\nWith the merit of LCA itself established in the present paper, future work can remedy this limitation.\nI am personally enthusiastic about the prospect of integrating CV with the AES-like solution.\n\nThe Reviewer also said:\n\n> The method might raise logistical and privacy issues, especially when example utterances come from users' own conversations with the chatbot.\n\nAnonymizing data is commonplace in NLP.\nUsers can evaluate how safe or unsafe utterances are even after they are scrubbed of personal information.\n\n----\n\n**References**\n\nAttali, Y. & Burstein, J. (2006). Automated essay scoring with e-rater v.2. The Journal of Technology, Learning, and Assessment, 4 (3), 1--31.\n\nDinan, E., Humeau, S., Chintagunta, B., & Weston, J. (2019). Build it break it fix it for dialogue safety: Robustness from adversarial human attack. EMNLP 2019. https://doi.org/10.18653/v1/D19-1461\n\nJu, D., Xu, J., Boureau, Y.-L., & Weston, J. (2022). Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls. https://doi.org/10.48550/ARXIV.2208.03295\n\nUto, M. & Okano, M. (2020). Robust neural automated essay scoring using item response theory. AIED 2020. https://doi.org/10.1007/978-3-030-52237-7_44", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper establishes a connection between chatbot robust learning and the Automated Essay Scoring (AES) literature, which is innovative.\n2. The paper delves into the realm of robust learning, discussing the threat posed by trolls and the potential solutions. The paper introduces the LCA model for dialogue safety. The model provides a way to determine the probability distribution of observed labels for each user, helping in identifying trolls and helpers.\n", "reasons_to_reject": "1.  The AES-like solution proposed has inherently higher data requirements, which might be burdensome for users.\n2.  The method might raise logistical and privacy issues, especially when example utterances come from users' own conversations with the chatbot.\n3.  The innovation of the algorithm is trivial. The writing of the paper is not very clear. The experiments part is suggested to be organized better.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper offers a perspective on addressing the challenges posed by trolls in chatbot robust learning. While the approach is promising, there are inherent limitations, especially concerning data requirements and potential privacy issues.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8hzISLw29d", "reviewer": "Reviewer_VSeF", "summary": "", "questions": "* In the introduction the paper describes that it would be able to identify the cases where the trolls constitute a majority of the annotators. At the same time, in LL. 153--158 it is mentioned that the cluster with more utterances is assumed to be 'safe'. Could you please clarify whether such an assumption would be correct if the 'trolls' were the majority and labelled the text as unsafe?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review.\n\n# Reconciling two statements \n\nThe Reviewer said:\n\n> In the introduction the paper describes that it would be able to identify the cases where the trolls constitute a majority of the annotators. At the same time, in LL. 153--158 it is mentioned that the cluster with more utterances is assumed to be 'safe'. Could you please clarify whether such an assumption would be correct if the 'trolls' were the majority and labelled the text as unsafe?\n\nSure, let me clarify. The two statements in question are:\n\n1. Most utterances are safe.\n2. Most users are trolls. (So most utterances are labeled unsafe.)\n\nYes, both statements can be true at the same time, no contradiction.\nWhen I said (1), I meant precisely in terms of the gold labels.\nSorry if that was confusing.\nOf course, the users may provide observed labels that are false.\n\nLet's walk through an intuitive example.\nSuppose we have three users: Alice, Bob, and Charmaine.\nAlice and Bob completely agree with each other; but Charmaine says the opposite.\nConsequently, there'll be two clusters.\n\n* Cluster D: utterances that Alice and Bob say are safe but Charmaine says is unsafe.\n* Cluster E: utterances that Charmaine says is safe but Alice and Bob say are unsafe.\n\nFinally, suppose that Cluster E ended up larger.\nSo which of the users are lying?\nIf we knew the chatbot to have a basic decency so that most utterances are safe (you wouldn't deploy it otherwise), then we catch Alice and Bob (the majority) as lying.\nBy similar reasoning, those lying would be caught as well if they were the minority.\n\nLCA weaponizes these intuitions formally.\nIt works even if a user's labels were neither 100% true nor 100% false.\nAnd it works even if each utterance is labeled by only a subset of the users.\nEach utterance gets a probability of being unsafe.\n\nAlso, another point of clarification.\nThe Reviewer said:\n\n> the paper applies a method from automated essay scoring to identify the malevolent users\n\nI want to emphasize that the end result of the AES-like solution is not to classify users as trolls or helpers (Line 145--147).\nWhat it does to the users is compute their best-fitting likelihood parameters, as shown in Figure 1 (above Line 118), where essentially, each row is a user-specific confusion matrix.\nThere is no cut-off for \"helper\" or \"troll\" on these parameters, and none are needed.\nGiven the parameters, for each utterance, its probability of being unsafe can be computed, considering that safe is the bigger class.\n\n# Baseline model\n\nThe Reviewer said:\n\n> the paper does not provide a comparison to any baseline model\n\nTo set up the answer, when faced with trolls, there are two problems to solve (Lines 40--43).\n\n* **(P1)** Given input/output pairs with dubious veracity (i.e. trolled training set), produce input/output pairs that can be trusted (i.e. de-trolled training set).\n* **(P2)** Given input/output pairs that can be trusted (i.e. de-trolled training set), predict outputs for new inputs.\n\nNotice that P2 is just the familiar supervised classification task, which is not of immediate interest.\nWe are not asking what classifier would be great to leverage labels that we trust.\nInstead, what we care about is the solution to P1.\nWe are asking, can we arrive at labels that we can trust? Does the solution impute classes accurately?\n\nI wish the Reviewer had said more on what they were expecting as a \"baseline model\".\nDue to the P1-P2 sequence, there are two ways I can interpret it.\n\n* **(P1)** Did the Reviewer mean I failed to compare to a P1 baseline?\nI must disagree.\nThe proposed method was to (Intervention 1) have multiple-users-per-utterance redundancy, then (Intervention 2) apply LCA.\nAs a baseline, I chose (Intervention 1) having multiple-users-per-utterance redundancy, then (Intervention 2) apply MV (majority vote).\nNote that both attempt to leverage redundancy, but one does it wisely (LCA) while the other naively (MV).\n* **(P2)** Did the Reviewer mean I failed to compare to a P2 baseline?\nNo comparison was intended, as there were none of immediate interest.\nTo be clear, in Experiment 1, an LLM (same across scenarios) was fine-tuned after de-trolling the training data, and accuracy was computed on a test set.\nNote that this metric conflated P2 performance and P1 performance. \nThus, I did Experiment 2, which measured the accuracy of the imputation itself in the training set.\nOtherwise, one might contend that my good results were just carried by a good pretrained classifier or an easy discrimination task.\nNote that the metric was given the name \"training set accuracy\" (Line 233), which is regrettable---the imputation method is unsupervised, so there is no concern that the evaluation understates error.\n\nIf both of these interpretations are incorrect, I will be happy to explain if the Reviewer clarifies what \"baseline\" they were expecting.\n\n# Synthetic data\n\nThe Reviewer said:\n\n> the method is only tested on synthetic data\n\nI must emphasize that the utterances and their gold labels are from real data in Dinan et al (2019).\nFor the present paper, what I manipulated was the troll behaviors, troll prevalence, and unsafe prevalence, to see how they impact the performance of the AES-like solution.\nMonte Carlo experiments are not unprecedented, as in Howcroft & Reiser (2021).\n\nJu et al (2022) performed an experiment on real deployment data but have not made it publicly available.\n\n----\n\n**References:**\n\nDinan, E., Humeau, S., Chintagunta, B., & Weston, J. (2019). Build it break it fix it for dialogue safety: Robustness from adversarial human attack. EMNLP 2019. https://doi.org/10.18653/v1/D19-1461\n\nHowcroft, D.M. & Rieser, V. (2021). What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think. EMNLP 2021. http://doi.org/10.18653/v1/2021.emnlp-main.703\n\nJu, D., Xu, J., Boureau, Y.-L., & Weston, J. (2022). Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls. https://doi.org/10.48550/ARXIV.2208.03295", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* the paper identified an interesting problem and proposed a potential solution which leads to improved performance;\n* the proposed solution is computationally light;", "reasons_to_reject": "* the paper does not provide a comparison to any baseline model\n* the method is only tested on synthetic data", "questions_to_authors": "* In the introduction the paper describes that it would be able to identify the cases where the trolls constitute a majority of the annotators. At the same time, in LL. 153--158 it is mentioned that the cluster with more utterances is assumed to be 'safe'. Could you please clarify whether such an assumption would be correct if the 'trolls' were the majority and labelled the text as unsafe?", "paper_topic_and_main_contributions": "The paper considers the problem of identifying trolls, or live users who wrongly classify the model's performance (on purpose or by mistake). To this end, the paper applies a method from automated essay scoring to identify the malevolent users.  The method is an application of expectation-maximisation algorithm to the problem of troll identification. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5kHPJr2Vrd", "reviewer": "Reviewer_Ysnq", "summary": "", "questions": "1. How does the model scale?\n2. How does the model perform compared with large language models?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed LCA method does not require GPU computations, making it a more cost-effective solution compared to cross-validation.\nRobustness: The AES-like solution demonstrated its effectiveness, particularly when faced with coordinated attacks by trolls. The paper establishes a connection between the challenges faced in chatbot robust learning and the methodologies employed in the AES domain.", "reasons_to_reject": "1. The paper is not well represented. Both the writing and formatting could be improved.\n2. The dataset size is extremely small (around one thousand) in crowd-sourcing context. I doubt the scalability of the proposed approach.\n3. The proposed method is not compared with any basic large language model, let alone the state-of-the-art models, making the empirical advantage look in-convincing to me. \n4. The efficacy of the AES-like solution is contingent on trolls being consistent in their feedback, which might not always be the case.\nThe method's performance drops significantly when trolls provide random feedback, indicating a potential vulnerability.", "questions_to_authors": "1. How does the model scale?\n2. How does the model perform compared with large language models?", "paper_topic_and_main_contributions": "The paper addresses the challenge of \"trolls\" who provide maliciously incorrect feedback to chatbots, leading to the generation of inappropriate or offensive responses. The issue is framed as a dialogue safety task, wherein input utterances are classified as either safe or unsafe. The conventional approach to combat trolling, which uses cross-validation based on user-aggregated behavior, is critiqued for its computational expense and vulnerability to coordinated troll attacks. Instead, the author introduces a method inspired by automated essay scoring (AES). This involves having multiple users rate each utterance and then employing latent class analysis to infer the correct labels. The paper finds that this AES-inspired approach is effective, particularly when trolls consistently provide similar feedback, even if they are in the majority. When trolls provide random feedback, the method's accuracy diminishes.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rbAs1eQpU1", "reviewer": "Reviewer_UqMy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "This review was absolutely a delight to read.\nThe Reviewer's summary of the present paper was brilliant.\nI had to stop and check that I wasn't mistakenly reading a paragraph I wrote myself.\n\nThe Reviewer said:\n\n> in the experiments, they did not compare it with Ju et al 2022 mentioned in the paper, but only with Dummy Classifer (MV), which is a weak comparison. Although the results of the method in this paper may not be as good as Ju et al 2022, I think this method is a very good comparison if it is not too much worse.\n\nI confirm the Reviewer's claim that the comparison was to MV (majority vote) and not exactly the CV (cross-validation)-based method in Ju et al (2022).\nLet me explain.\n\n# Why not the CV-based method?\n\nLet's walk through Ju et al's (2022) method, with a little more detail.\n\n1. **(Intervention 1)** Use 3-fold CV with an LLM. \nThe result is CV-predicted unsafe probabilities per utterance.\nNote that doing so simulates \"checking each other's work\" or tapping into some wisdom of the crowd.\n2. **(Intervention 2)** Filter the training data by untrustworthiness scores.\nThe formulas are in Ju et al (2022), but the idea is that labels far from the CV predictions are less trustworthy.\nApply a threshold for untrustworthiness, a hyperparameter $\\beta$.\nThe result is a training set with fewer examples, retaining their original user labels.\n(It mentions the possibility of flipping the labels below the threshold instead, but they went for deletion in their experiments.)\n\nUnfortunately, Ju et al's (2022) full procedure was prohibitively difficult to reproduce.\nTheir Github code shows only up to how to get the CV predictions (Intervention 1) in ParlAI.\nBut the rest (Intervention 2) is just described in the text with some formulas, no ParlAI code, and the grid to be searched for $\\beta$ was not given.\nNote that choosing $\\beta$ using a validation set (on top of the CV predictions), as described in Ju et al (2022), would be expensive.\nImagine fine-tuning the LLM with all training examples, then again dropping the most untrustworthy example, then dropping the two most untrustworthy examples, and so on.\n\nI want to caution the Reviewer about comparing results to those reported in Ju et al (2022).\nJu et al's (2022) experiments did not have a condition where trolls were a majority and consistent with each other, which is precisely what motivated the present paper.\n\n# Why majority vote?\n\nNow let's recap my proposed AES (automated essay scoring)-like solution.\n\n1. **(Intervention 1)** Assign multiple users per utterance.\nThe result is the $Z$ matrix.\nNote that doing so, rather than simulating asking the crowd, actually takes redundant labels, as in AES.\n2. **(Intervention 2)** Apply LCA (latent class analysis).\nEach utterance is assigned a cluster, and the bigger cluster is labeled the safe class.\nThe result is a final imputed label per utterance.\n\nLCA cannot be done without Intervention 1.\nBut with Intervention 1, one might contend that it is just carried by the additional data collected.\nThus, I chose a baseline that would also benefit from Intervention 1.\nThe most obvious to me was MV (majority vote) (Ipeirotis et al 2010).\nNote that MV would also be vulnerable to a false consensus, like Ju et al's (2022) method, just with actual redundant labels rather than simulated consensus.\n\n# More experiments\n\nHaving said all that, I still wished to satiate the Reviewer's curiosity.\nWith the same settings as Experiment 2, I did new runs, for the Ju et al (2022) method, only up to producing CV predictions (i.e. Intervention 1).\nLet's call this \"CV1\", to emphasize that we have only one user per utterance (no redundancy, only one non-missing cell per row of $Z$).\nThen I did 50 runs of MV and 50 runs of LCA.\nLet's call these \"MV5\" and \"LCA5\" respectively, to emphasize that we have 5 users per utterance (with redundancy).\nBut what metric to use?\nThe metric would have to satisfy two properties.\n\n* The metric must be intrinsic (instead of downstream test set accuracy), so as to avoid expensive GPU computations.\n* The metric must be applicable to both a trustworthiness-ranking method (CV1) and a class-imputing method (MV5 and LCA5).\n\nTo arrive at such a metric, I did as follows.\n\n1. I computed untrustworthiness score equivalents for MV5 and LCA5.\nTake the squared difference between the predicted probability and the observed label.\nFor MV5, the predicted probability is just the proportion of votes for unsafe (or mean of the row in $Z$).\nNote that there are as many scores as there are non-missing cells in $Z$.\n2. For each method, I created a binary \"correctness\" variable: the class is \"correct\" if the observed label matches the gold label, and \"incorrect\" otherwise.\nAgain, there are as many binary values as there are non-missing cells in $Z$.\n3. For each method, the metric was the area under the ROC curve for predicting correctness from trustworthiness. Is the untrustworthiness predictive of whether the label was incorrect? Higher values are better.\n\nScenario means are in the table below.\nFor diligent-troll majority scenarios, CV1 was better than MV5, but LCA5 was better than both.\nBut for lazy-troll majority scenarios, MV5 was better than the other two.\n\n|unsafe_prevalence|troll_prevalence|corrupt_action|troll_corrupt_rate|lca5_auc         |mv5_auc            |cv1_auc          |\n|-----------------|----------------|--------------|------------------|-----------------|-------------------|-----------------|\n|0.1              |0.5             |diligent      |0.8               |0.974|0.641  |0.596|\n|0.3              |0.5             |diligent      |0.8               |0.994|0.658   |0.564|\n|0.1              |0.9             |diligent      |0.8               |0.938|0.079|0.431|\n|0.3              |0.9             |diligent      |0.8               |0.945  |0.084 |0.376|\n|0.1              |0.5             |lazy          |0.8               |0.990|0.951   |0.611|\n|0.3              |0.5             |lazy          |0.8               |0.992|0.958  |0.692|\n|0.1              |0.9             |lazy          |0.8               |0.723|0.754  |0.548|\n|0.3              |0.9             |lazy          |0.8               |0.699|0.763  |0.551|\n|0.1              |0.5             |diligent      |0.95              |0.988|0.479  |0.448|\n|0.3              |0.5             |diligent      |0.95              |0.999|0.491  |0.550|\n|0.1              |0.9             |diligent      |0.95              |0.997|0.005|0.321|\n|0.3              |0.9             |diligent      |0.95              |0.999|0.006|0.422|\n|0.1              |0.5             |lazy          |0.95              |0.990|0.920  |0.610|\n|0.3              |0.5             |lazy          |0.95              |0.994|0.929  |0.707 |\n|0.1              |0.9             |lazy          |0.95              |0.603 |0.623  |0.489|\n|0.3              |0.9             |lazy          |0.95              |0.619|0.632  |0.534|\n\n----\n\n**References**\n\nIpeirotis, P.G., Provost, F., Wang, J. (2010). Quality management on Amazon Mechanical Turk. ACM SIGKDD workshop on human computation, 64--67. https://www.ipeirotis.com/wp-content/uploads/2012/01/hcomp2010.pdf\n\nJu, D., Xu, J., Boureau, Y.-L., & Weston, J. (2022). Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls. https://doi.org/10.48550/ARXIV.2208.03295", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Excellent writing style, concise language and clear logic.\n- The issue of chatbots inadvertently generating offensive utterances is crucial. Improving the quality and safety of chatbot responses is pertinent to both industry and society.\n- The paper offers a novel solution inspired by automated essay scoring (AES) methodologies. The introduced latent class analysis (LCA) method, in contrast to GPU-intensive computations, is highlighted as an inexpensive solution, making it more scalable and feasible for real-world applications.", "reasons_to_reject": "In this paper, the authors repeatedly emphasize the simplicity and cheapness of this method, but in the experiments, they did not compare it with Ju et al 2022 mentioned in the paper, but only with Dummy Classifier (MV), which is a weak comparison. Although the results of the method in this paper may not be as good as Ju et al 2022, I think this method is a very good comparison if it is not too much worse.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper delves into the challenges presented by adversarial noise, specifically from \"trolls\", in corrupting data for machine learning models. Traditionally, training examples with high user-aggregated cross-validation (CV) error were omitted to counteract this \"trolling\". However, CV, being computationally expensive, can be rendered ineffective in situations of coordinated troll attacks. Addressing this, the paper introduces a novel solution inspired by automated essay scoring (AES) methods: garner ratings from multiple users for each utterance and subsequently employ latent class analysis (LCA) to deduce accurate labels. Notably, LCA, avoiding GPU-intensive computations, proves to be cost-effective.  Through experimental validation, it was demonstrated that this AES-inspired approach is adept at inferring correct training labels, even when the majority of feedback originates from coordinated trolls. The research concludes with an intriguing observation: when harnessed effectively, even \"diligent\" trolls can inadvertently improve classifier performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to recover the real harmfulness of chatbot conversations even when some users (a.k.a. trolls) intentionally provide incorrect feedback. The proposed method, based on LCA, is significantly more efficient than common cross-validation practices (but also has higher data requirements) and more robust to scenarios where trolls are the majority, provided that they are consistent in their behaviour. Hence, an interesting aspect of this paper is that it presents a somewhat counter-intuitive result. However, the paper also suffers from some weaknesses: the test data size is very limited, which somewhat puts into question the validity of the conclusions. Most crucially, the effectiveness of the method relies on somewhat strong assumptions: namely, 1) that trolls are "diligent", i.e. consistent and 2) that the safe class is in the majority of the utterances. It remains unclear how realistic these assumptions are, without any quantitative evidence from real-world scenarios in support. For these reasons, this paper might be considered for Findings.
Decision: Accept-Findings