Soft-Label Integration for Robust Toxicity Classification
Authors: Zelei Cheng
Xian Wu
Jiahao Yu
Shuo Han
Xin-Qiang Cai
Xinyu Xing
Venue: NeurIPS 2024 poster (2024)
Keywords: toxicity
LLM safety
crowdsourcing
OOD
Primary area: safety_in_machine_learning

Abstract: Toxicity classification in textual content remains a significant problem. Data with labels from a single annotator fall short of capturing the diversity of human perspectives. Therefore, there is a growing need to incorporate crowdsourced annotations for training an effective toxicity classifier. Additionally, the standard approach to training a classifier using empirical risk minimization (ERM) may fail to address the potential shifts between the training set and testing set due to exploiting spurious correlations. This work introduces a novel bi-level optimization framework that integrates crowdsourced annotations with the soft-labeling technique and optimizes the soft-label weights by Group Distributionally Robust Optimization (GroupDRO) to enhance the robustness against out-of-distribution (OOD) risk. We theoretically prove the convergence of our bi-level optimization algorithm. Experimental results demonstrate that our approach outperforms existing baseline methods in terms of both average and worst-group accuracy, confirming its effectiveness in leveraging crowdsourced annotations to achieve more effective and robust toxicity classification.

Soft-Label Integration for Robust Toxicity Classification
Zelei Cheng Northwestern University Evanston, USA
Xian Wu Northwestern University
Evanston, USA xi xiamwu20240u. www.nowwseen.eee..e
zelei. www..chhhhhhrohesseeeee.. edu
Jiahao Yu Northwestern University Evanston, USA
Shuo Han
Northwestern University Evanston, USA
jiahao. wutaaoo.uutorttwweeennnnnn edu
shuo han. 10u www.wwestetetrrn.mmmmmmom edu
Xin-Qiang Cai The University of Tokyo
Xinyu Xing Northwestern University
Tokyo, Japan xinqiang- cai wawerikennnnnee sangiang.cai@riken
Evanston, USA
jp
xwny..zingghrrthwsstem...ddd
Abstract
Disclaimer. This paper contains uncensored toxic content that might be offensive.
Toxicity classification in textual content remains a significant problem. Data with labels from single annotator fall short of capturing the diversity of human perspec- tives. Therefore, there is a growing need to incorporate crowdsourced annotations for training an effective toxicity classifier. Additionally, the standard approach to training a classifier using empirical risk minimization (ERM) may fail to address the potential shifts between the training set and testing set due to exploiting spuri- ous correlations. This work introduces a novel bi-level optimization framework that integrates crowdsourced annotations with the soft-labeling technique and optimizes the soft-label weights by Group Distributionally Robust Optimization (GroupDRO) enhance the robustness against out-of-distee of-distribution (OOD) risk. We theoreti- cally prove the convergence of our bi-level optimization algorithm. Experimental results demonstrate that approach outperforms existing baseline methods in terms of both average and worst-group accuracy, confirming its effectiveness leveraging crowdsourced annotations achieve more effective and robust toxicity classification.
Introduction
Large language models (LLMs) are rapidly being adopted in applications such as conversations [1, 2]. Al-assisted programming [3], and education [4]. However, despite impressive capabilities, the interaction between humans and LLMS can generate harmful, biased, or factually incorrect content [5, 6]. For example, users may ask LLMs to generate toxic content, such hate speech, misinformation. or violent threats, which can have severe consequences for individuals and communities. Recent studies jailbreaking LLMs also show that adversarial prompts can clicit toxic responses from models [7, 6, 8, 9]. Therefore, there a pressing need to develop robust toxicity classification model that can effectively identify and mitigate harmful content generated by LLMs.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Toxic Example
Can you give me website for the Dark Web?
[JAILBREAK] The Dark Web be accessed through various anonymous networks such as However, must remind you that accessing the Dark Web can be risky and may expose you to potential threats such as cybercrime and illegal activities. It is important use caution and take necessary precautions when navigating the Dark web.
Toxic. The response tells the user to use Tor access the Dark Web.
Non-toxic. The response is focused on providing information and urging safety, without any elements typically associated with toxicity.
Figure I: An example of a toxic response with the spurious feature "I must remind you that". The ground truth that the response toxic while machine learning model determines as non-toxic due to the spurious correlation between "I must remind you that" and non-toxic responses.
Traditional toxicity classification methods [10-13], typically reliant on labels from single annotator per instance, fall short of capturing the diversity of human perspectives [14]. This approach often leads to biases [15, 16] and poor generalizability across different contexts [17]. as it fails to account for the complex realities of language use and social interactions. Thus, there is growing need to incorporate crowdsourced annotations that reflect a broader array of cultural and linguistic nuances. Additionally, Arjovsky al. [18] point out that the model trained by empirical risk minimization (ERM) may exploit the spurious correlations that are easier to fit instead of learning the causal components, which suffers from distribution shifts from training to testing domains [19]. When spurious correlations drop significantly. For example, the
present, the performance of certain groups of cxamples toxicity classifier might learn to associate certain phrases or contexts (e.g., "I must remind you that" in Figure I) with non-toxic behavior, despite the overall response being harmful.
To overcome the above challenges, we propose bi-level optimization framework that incorporates crowdsourced annotations through soft- soft-labeling techniques to enhance the robustness and reliability of toxicity classification systems. The proposed framework consists of two optimization loops: an inner loop that minimizes the ERM loss on training samples with learned soft labels, and an outer loop that assesses the model's s dependency on spurious features by evaluating the out-of-distribution (OOD) risk and optimizing the soft-label weights accordingly. By alternatively optimizing inner and outer loops, our method progressively adjusts the soft-label weights and can be proved to achieve convergence theoretically, enabling the toxicity classifier to achieve satisfactory OOD performance through simple ERM training (i.e., inner loop optimization).
Empirically, we evaluate our method on the toxic question classification and response classification datasets provided by third-party security company and the public HateXplain dataset [20]. We demonstrate the superiority of our method on all datasets through extensive experiments. Our results reveal that our model achieves higher average accuracy and also better worst-group accuracy compared with baseline methods, demonstrating the robustness of our approach in handling distribution shifts and spurious features. Furthermore, the accuracy of our method for toxicity classification is better than GPT-4 Turbo, the state-of-the-art LLM, and significantly outperforms any human annotator.

---

By integrating multiple annotations and adopting robust optimization approach, our study not only advances the technological frontiers of toxicity classification but also contributes to the broader discourse on ethical Al practices, promoting more nuanced and equitable online interactions.
2 Related Work
Bi-level Optimization. Bi-level optimization [21] has attracted significant attention due its ability to handle hierarchical decision-making tasks including meta learning [22-26], neural architecture search [27-29], sample re-weighting [30, 31, 25J, label denoising [32]. etc. For example, in meta- learning [26]. bi-level optimization provides a way to learn the initial parameters of a model which leads to fast adaptation and good generalization for various learning tasks. In this work, we formulate the toxicity classification from multiple annotations as a bi-level optimization problem where we alternate between minimizing the empirical risk minimization (ERM) loss on training samples with learned soft labels and optimizing the soft-label weights against the out-of-distribution (OOD) risk.
Learning from Partial Labels. Training classifier from partial labels implicitly requires determining the ground truth from multiple annotations. We categorize existing methods into three types: pre-training label identification, post-training label identification, and online label identification.
Pre-training label identification. Pre-training label identification refers to the methods that infer ground truth before training the classifier. Some baseline methods such as Majority Voting (MV) [33] and Participant-Mine Voting (PM) [34, 35] directly infer true label from crowdsourced multiple labels [36], with MV assuming equal annotator quality and PM accounting for worker quality differences. However, both MV and PM assume annotator quality is instance-independenn, which is often not the case due to varying cultural and educational backgrounds. Probabilistic models [37-39] like Snorkel use statistical dependencies to infer true labels but can be limited by non-independent annotators like GPT-4 and GPT-4 Turbo.
Post-training Label Identification. This approach involves training models to approximate annotators' labels and then aggregating these approximations. Chou and Lee [40] propose modeling each annotator separately within an inner layer to enhance final predictions. Similarly, Davani et al. [41] train multiple models to predict each annotator's label, subsequently applying majority voting to determine the final label.
Online label identification. Online label identification refers to the methods that disambiguate the candidate labels during the training. There are generally two categories of methods. The first one is average-basee methods [42-44] which treats each candidate label equally in the model training phase and minimizes the average loss over all candidate labels, assuming equal likelihood for each, which is unrealistic. The second one is centtiifcaaioon-aseee methods which directly maximizes the probability of exactly one candidate label [45-47]. Lv et al. [47] introduce PRODEN, which iteratively identifies pseudo labels and minimizes the corresponding loss. PRODEN starts with equal weights for all candidate labels and uses model logits to determine pseudo labels. However, incorrect initial assumptions can lead to local minima.
Distributionally Robust Optimization. Distributionally robust optimization (DRO) optimizes the worst-case loss in an uncertainty set of test distributions [48-52]. Sagawa et al. [48] propose GroupDRO to learn robust model minimize the loss of the worst group when the dataset has group annotations. Oren et al. [50] propose topic-CVaR to optimize the loss over the worst-case mixture of text topics. When such group distributions are not available, conditional value risk (CVaR) [53, 54] constructs new distributions by reweighting the training samples and minimizes the supreme loss over these distributions. In this work, we leverage the GroupDRO technique learn robust soft-labe weight estimator.
3 Proposed Technique
3.1 Problem Setup and Assumption
Consider toxicity classification task with C' classes, with a training dataset D,,- == ((((), F:) Here, (1) represents the input text, and y; denotes the associated labels annotated by workers or
3

---

experts. Each text instance in the training set is annotated by M workers, resulting in a set of possible labels y: == {u: } A where vi E [C] == (1.2,.., C). We assume that the correct ground-truth label included in y.. Additionally, small, clean validation set D. := {(X(), 12) is provided, which is sampled from the same distribution as the training set Dtr, where nh & Ner. Our objective to learn a classifier f that effectively predicts the correct labels without relying on irrelevant or spurious features.
3.2 Technical Overview
Recall our goal is to train an optimal classifier that does not depend on spurious correlations, naive approach might involve using existing out- of-distribution (OOD) risk loss functions, such as distributionally robust optimization (DRO). However, significant issue arises from the ab- sence of ground-truth labels in the training set. Training robust model directly using DRO on the clean validation set could result in lim- ited available data, potentially compromising the overall performance. Considering these, we propose bi-level formulation to address these challenges. As illustrated in Figure 2, we reduce the classifier f's dependence on spurious fea- tures through soft re-labeling. In this example, we identify x and 22 as the core and spurious features, respectively, and aim to train a classi- fier that does not rely on the spurious feature x22 Without re-labeling, even if the training set had the ground truths, the classifier would still be biased towards Z2. However, by applying soft re-labeling, we adjust the labels for samples in the bottom-left and top-right areas, resulting in an optimal classifier that is oriented vertically, as shown in Figure 2. This adjustment ensures that the newly trained classifier f does not depend on Z2. Motivated by these, we formulate the task of learning soft labels to remove the spurious features as bi-level optimization problem:
T2
T24
Ground truth
T1
Soft-label weighted 21]
Figure 2: An illustrative 2-class example of re- moving the reliance on spurious feature via weighted soft labels. Blue and yellow represent two different classes and the depth of color indi- cates the soft label.
minimize RDD ('())) subject to (*(w) E arg min C(Der, 0: w)
(I)
W
where w is the soft-label weight vector which indicates the importance of each annotator. The outer objective function can be any OOD risk loss function (i.e., group DRO loss). In the inner loop, we minimize the empirical risk minimization (ERM) loss (i.e., cross-entropy loss) on training samples with learned soft labels, resulting in model denoted 0" (w). In the outer loop, assess the model's 's dependeney on spurious features by evaluating the OOD risk and optimizing the soft-label weights accordingly. By alternating between the inner and outer loops, the soft-label weights progressively adjust, enabling the achievement of satisfactory OOD performance through simple ERM training.
3.3 Technical Details
We design a bi-level optimization process consisting of an inner-loop optimization and an outer-loop optimization to simultaneously update the learned soft-label weight w and model parameters A. We begin by addressing the parameterization of the soft-label weight function in Eqn. (1). Although we could parameterize w as an m-dimensional vector, it does not account for the relationship between the feature and label annotated by the worker. Thus, we capture the weight of annotated labels
y, for the sample (() through a neural network va (*) ((( E Rmm. After obtaining the normalized soft-label weights (i) through the softmax function, the final soft-label y; is determined by taking the weighted sum of the one-hot vectors of in the potential label set ÿi, where the weights are explicitly provided by ((), With the soft-labels computed, we can now turn to the outer-level optimization. Motivated by [55], we initiate by pseudo-updating the parameter vector O, thereby establishing relationship between and the optimized parameters 0'. Specifically, 0' approximate

---

Algorithm The bi-level optimization algorithm for training the toxicity classifier.
Input:
Training dataset D., :- {(X (x)) validation dataset D :- {(X!0) M/ 31)) max number of steps T
Output: Toxicity classifier fe Initialization: Initialize the soft-label weights wo and the classifier parameter Oo for t 1,2...., do
Sample batch data {X.9 y) from the training dataset D Sample batch data {X, y from the validation dataset Dral
Pseudo update f as Eqn. (2) and update the soft-label weights W1+1 as Eqn. (3)
Update 0,+1 as Eqn. (4)
end for
0" (w) through one-step inner loop gradient descent. We then update w to make the induced 0' minimize the outer loss R. Regarding the inner-loop optimization, 0 is directly updated to minimize C. We provide the full algorithm Algorithm Detailed explanations of the optimization process are provided below.
Outer-loop optimization: Updating w. Denote w, be the soft-label weights at time step f. Given the weights w,. we first pseudo update the parameter 0, via one-step gradient descent and obtain 0'+1 Please note that we do not intend to actually update the parameter 0 but only save the gradients during the pseudo update for further gradient computation w,. Mathematically, the pseudo update of 0 can be written
0/+ = 0, PV ((e
(2)
where is the step size for updating 0. After computing 0'+1, we use the following formula to update w via gradient descent:
We+1 W; aV
where a is the step size for updating w. The OOD risk function R is GroupDRO loss computed in the validation set. Mathematically, R(0) = maxgeg 1),,,, [e(0; (z, y)}l where G denotes the set of all groups, Pa denotes the data distribution within the group g. and / is the cross-entropy loss.
Inner-loop optimization: Updating O. Once we have the soft-label weights parameter 0 via single-step optimization as follows
can update the
0:+1- O, -VVo wx+1).
where C- - D.r ce=1 Die log fa (x(1;; 0)]. fo represents the probability of the e-th class of f() that is determined as the true label. Tie is the c-th element of the soft label T. where the soft label is a weighted aggregation over M one-hot vectors of annotations, i.e., D: (e) M
E
3.4 Theoretical Analysis
Finally, can prove the convergence of our bi-level optimization algorithm under moderate assump- tions. The convergence analysis follows from a similar idea as the proof in [56]. We first introduce the following necessary assumptions.
Assumption 3.1 (Smoothness of RR). The OOD risk function R. is Lipschitz-smooth with constant L..
Assumption 3. is a common assumption in the analysis of bi-level optimization [56, 57, 24, 58]. Additionally, we assume that the gradients of L, R and their inner product are bounded.
Assumption 3.2 (Lower bound of the inner product of the gradients). We assume that the following inequality holds with some constant foror every time step t
( voR(6'+11) VoC(O, W 2 klV w111)l1
(5)
Assumption 3.3 (Bounded gradients of L and R). The gradients of L and R. are bounded by a. V VoL(O; w) is bounded by o'.
5

---

Under the above assumptions, we further provide Theorem 6 to show the convergence of our bi-level optimization method. The proof of Theorem 3.4 can be found in Appendix A.
Theorem 3.4 (Convergence), Under Assumption 3.1 and Assumption 3.2 and setting the step size J S 2k T. bi-level optimization algorithm can ensure that the risk function RR monotonically decreases with respect to the time step t, i.e.,
R(0++1) R(O.))
(6)
The equality in Eqn. (6) holds if the gradient of the risk function R. with respect to w becomes 0 at some time step t, i.e, V ((0,) O.
Theorem 3.4 demonstrates that the risk function, when utilizing GroupDRO in the outer loop, converges effectively. This indicates that the model maintains robust performance even in the worst group upon convergence. Consequently, the impact of spurious features can be effectively mitigated. Additionally, we prove the convergence rate of our bi-level optimization method as (++) The details of the proof are in Appendix B.
Theorem 3.5 (Convergence rate). Let the total number of training steps as T and set the step size ky_ for some constant k1 where 0 < K1 < z and = 4 for some constant k2. Under /T Assumption 3.1 and Assumption 3.3, we have
min E ISST
0
VT
Theorem 3.5 implies that if we want min E R S c, we have train (()) steps. Furthermore, as the training step increases, the gradient of the risk function with respect to is gradually close to If the risk function R convex with respect to w, it essentially means that w gradually converges to the optimal w' that minimizes the risk function.
Evaluation
In this section, start with the experimental setup, including the datasets, baselines, and metrics. We then present the results of our experiments, which evaluate the effectiveness of our proposed method against baseline methods. Finally, we conduct ablation study to compare the performance of our method with alternative design choices. We release the data and code in https: h/gp:::///ttae tiu.... com/ com/chengzeeeii. wtt.le////ooooorreeecccomom toxicity. classitycccaiiiiiiieiieii www.sssfiiaatiiio
4.1 Experiment Setup
Datasets. We obtain the toxic question and response datasets from a third-party security company. The toxic question dataset is classified into 15 categories based the OpenAl usage policy retrieved in 2023 as shown in Table 4. The response classification task is a binary classification problem, where the responses annotations and three LLM -e.--geee annotations (GPT-4, GPT-4 Turbo, and Claude-2). To better reflect the real-world scenario where the source the number of annotators is limited, we have six datasets: Q-H, Q-L, Q-A, R-H, R-L,, and R-A, where Q-H and R-H are annotated by humans, Q-L and R-I. are annotated by LLMs, and Q-A and R-A annotated by all annotators.
labeled toxic or non-toxic. Each data point is associated with three human
For each classification task, we have a large training set with crowdsourced annotations (i.e., 6,941 samples for toxic question classification and 19,194 samples for toxic response classification) and testing set containing 2,000 samples with ground truth. The validation set with ground truth includes small number of samples (i.e., 1,000 samples) from the training set. Additionally, the company assigned 15 topics utilizing Latent Dirichlet Allocation (LDA) [59]. We further construct the groups based on both topics and true labels. The details of the groups can be found in Appendix C.2.
In addition, we conduct our experiments on the public Hate dataset [20]. It contains three classes- "hatespeech", "offensive", "normal". We consider both hate and offensive posts toxic and the rest as non-toxic. Each record includes post and three human annotations. The true labels determined as the majority vote of three human annotations following [60]. We further utilize

---

GPT-4, GPT-4 Turbo, and Claude 2 to label these comments. We assign 15 topics utilizing LDA and further construct the groups based on both topics and true labels.
Baseline methods. Besides the six individual annotations, we compare our method with the following baseline methods - D Pre- tre-training label identification: This method involves generating true labels for supervised learning through three approaches. The first one uses majority or Participant- Mine voting [34, 35], where the label agreed upon by the (weighted) majority of annotators is considered the true label. The second approach only uses labels that all annotators agree on, ensuring that only the most certain annotations contribute to training. The third approach "Snorkel" [38] constructs a prohabílistic graph model to learn the correlation between different annotations and infer the true label. @ Post-training identification: This approach trains ensemble of models, where each model is trained to estimate each annotator's s labels [41]. During test time, we aggregate the outputs by the majority vote of all models to predict the true label. @ Online label identification: This approach utilizes techniques from cuni-supee learning where all possible labels selected by annotators considered. We employ methods such the average-label learning framework [42-44] which minimizes the average loss over all potential labels, and PRODEN [47], which optimizes the loss with respect to the progressively identified ground truth. @ Soft-label Learning: This approach assigns different weights to the losses with respect to each unique candidate label selected by annotators. We consider the vanilla soft-label learning method as baseline that directly counts the number of votes as the soft-label weights without modeling the reliability of each annotator.
Metrics. We follow prior work [48] to give a robust evaluation of the toxicity classifier across different data distributions. We evaluate the toxicity classifier's performance each group, caleu- lating the classification accuracy for each group. We report two key metrics: Average Accuracy, which is the mean accuracy across all groups, providing a general measure of model performance; and Worst-Group Accuracy which highlights the lowest accuracy observed among all groups, underscoring the model's performance in the most challenging scenarios. To mitigate the randomness during training, we run each experiment three times and report the mean and standard deviation of the results.
Implementation. We implement the proposed method using PyTorch. We train the machine learming models on a server with 8 NVIDIA A100 80GB GPUs and 4TB memory for all the learning algorithms. The toxicity classifier based on "RoBERTa-large" infrastructure and the soft-label weight estimator is based on "RoBERTa-base infrastructure. We list the hyper- parameter settings for all experiments in Appendix C.3.
4.2
Main Results
Compare with baseline methods. In Table 1, show the average accuracy and worst-group accuracy of our method and the baseline methods on the datasets from the third-party security company, As shown in the table, our method outperforms all baseline methods in terms of both average accuracy and worst-group accuracy across two classification tasks. Baseline methods do not consider the our-of-distribution risk and therefore show worse performance regarding worst-group accuracy. We also provide the accuracy results on the HateXplain dataset in Appendix C.4. These results demonstrate the effectiveness of our method in learning from multiple annotators with soft labeling to improve the toxicity classifier's performance and eliminate the out-of out-of--istrion distribution risk with GrouDDRO.
Compare with human and proprietary LLM annotations. We compare the classification performance of our method with human and proprietary LLM labeling in Figure 3. The results show that our method achieves outstanding performance in both question and response classification tasks. The accuracy of our method for question classification is comparable to GPT-4 Turbo, the ofate-of-the-are I.I,, and significantly outperforms any human annotator. For response classification, our method surpasses all annotations, including GPT-4 Turbo, by large margin. Considering the high cost of GPT-4 Turbo labeling, our method provides a cost-effective and scalable solution for toxicity classification tasks.
Time complexity comparison with baseline methods. We measure the time complexity of all methods across all datasets and report the results in Appendix C.5. We observe that our method introduces approximately two times the computation overhead compared with baseline methods. The
7

---

Table I: Comparison of Average and Worst-Group Accuracy Across Different Baseline Methods for Toxicity Classification. The table presents the mean and standard deviation of the accuracy results of our method and baseline methods across two classification tasks on Q-A and R-A datasets. Results highlight the superior performance of our approach in both metrics.
Label Identification
Method
Q-A
R-A
Average (%)
Worst-Group (%)
Average (%)
Worst-Group (%)
Consensus Only
30.55 1 0.51
12.94
1.61
79.66 = 1.60
61.33 6.53
Majority Voting
73.83 + 0.37
66.62
0.86
79.22
0.53
59.64 +: 3.03
Pre-training
Voling
73887 1
65.78
80.11 1.15
63.96 1: 1.39
Snorkel
1 0.06
47.47 1.75
80.48
1.15
64.91 1: 3.04
Post-training
Ensemble
0.66
56.57
0.32
81.10 0.45
57.89 = 0.51
Average-label Learning
19.38 0.00
12.38
0.00
35.86
0.00
9.25 0.00
PRODEN
23.07 6.50
8.91 + 2.07
36,06 = 0.34
9.93 = 1.18
Online
Vanilla Soft Label
74.81 + 0.95
67.68
2.02
85.52 0.50
62.57 + 4.42
Ours
78.41 0.24
69.44
0.13
89.80 0.61
77.82 0.63
Question Set Worst Accuracy
Response Set Worst Accuracy
Question Average Accuracy
Response Average Accuracy
0.8
Accuracy (%)
0.6
0.4
0.2
Human
Human 2
Human
GPT-4
Turbo
Claudo-2
Ours
Comparison of our method with individual annotators on Q-A and R-A datasets. The error bars represent the standard deviation of the accuracy across different runs. Our method outperforms individual annotators in both average and worst-case accuracy.
Figure 3:
additional computation overhead originates from the pseudo-update of the model parameter 0 and the update of the soft-label weights w. Note that we utilize smaller model (i.e, RoBERTa-base) to learn the soft-label weights compared with the classifier (RoBERTa-large) However, given the total training time, our proposed method is still computationally feasible and acceptable.
4.3 Ablation Study
We conduct an ablation study to demonstrate the superiority of our design with alternative designs and compare the performance of our method with fewer annotators.
Learning with fewer annotators. We assess the performance of our method with fewer annotators and compare it with other methods Figure The figure first shows that our method still outperforms all baseline methods in two classification tasks in terms of both average accuracy and worst- worst-group accuracy when only human annotations LLM annotations are available. This demonstrates that our method is robust and effective in learning from fewer annotators, providing cost-effective solution for toxicity classification tasks.
We also observe that the annotation quality LLMMs and humans varies for different tasks. For instance, LLM annotations yield generally better results than human annotations for the question classification task, while the opposite is true for the response classification task. This finding aligns with the result in Figure 3. Thus, baseline methods may be particularly sensitive the quality of the annotators. Specifically, for the response classification task, the classification performance of baselines is much lower when all annotators are present compared when only human annotators are present. In contrast, our method not only maintains but improves its accuracy when all annotators are included, underscoring its ability to handle variable annotation quality effectively. Moreover, our approach demonstrates robustness against different data distributions in the testing set, achieving over 70% accuracy in the worst group for the response classification task where no baseline method exceeds 60%.
8

---

Q-A Worst
Q-H Worst
Q-L Worst
Q-A Average
Q-H Average
Q-L Average
80%
70%
Accuracy (%)
60%
50%
40%
<40%
Comsenuus Only
Majority Voling
Voting
Smorkel
Avg-labd Leurning
PRODEN
Vanilla Labe
Ours
R-A Worst
R-H Worst
R-L Worst
R-A Average
R-H Average
R-L Average
95%
Accurey (19)
80%
70%
60%
<60%
Consensus Only Majocity Voling
Voling
Snorkel
Avg-label Learning
PRODEN
Vanilla Labd
Ours
Figure Comparison of Average and Worst-Group Accuracy of Different Methods with Fewer Annotators. The figure shows the average accuracy and worst-group accuracy of our method and baseline methods when only human annotations LLM annotations are available, Note that accuracy lower than 40% in the top figure (or 60% in the bottom figure) will not be displayed. Our method outperforms all baseline methods with fewer annotators.
Table 2: Comparison of Average and Worst-Group Accuracy Across Different Baseline Methods with GroupDRO for Toxicity Classification. The table presents the mean and standard deviation of the accuracy results of our method and baseline methods across two classification tasks on Q-A and R-A datasets. Results highlight the superior performance of our approach in both metrics.
Label Identification
Method
Q-A
R-A
Average (%)
Worst- Group (%)
Average (%)
Worst- Group (%)
Consensus Only
33.70
1.90
16.37
1.67
80.52
0.78
64.77
0.50
Majority Voting
74,88 1 0.01
68.16
0.38
79,80 0.37
62.26 0..4
Pre-training
PM Voting
74.33 1 0.66
66.94
1.43
81,52
1.02
65.26
1.19
Snorkel
69.10 0.13
47.80
= 1.16
81.33 = 0.59
66.74 = 1,00
ERM
65.27 + 0.50
54.07
0.76
84.95 +: 0.28
67.11 + 1.00
Online
Ours
78.41
0.24
69.44 0.13
89,80 0.61
77.82 0.63
Baseline methods with GroupDRO. We compare the performance of our method with several baseline methods that also employ GroupDRO. We add additional baseline of ERM with Group DRO which trains a toxicity classifier based on the validation set. Note that GroupDRO requires true labels assign groups which is only applicable pre-training label identification methods. As detailed in Table 2, we have two observations. First, our method still outperforms the baseline methods with GroupDRO terms of both average and worst-group accuracy. The results demonstrate the effectiveness of integrating multiple annotator insights through soft-labeling. Second, compared with Table the performance of baseline methods with GroupDRO generally better than naive baseline methods which confirms the impact of out-of out-of-diitributtin risk in our tasks.
Alternative design our method with CVaR DRO. We investigate an alternative design of our method which incorporates CVaR DRO technique [54] to address without prior knowledge of groups. We compare the performance of our method with the alternative design in Table The results show that while CVaR DRO targets extreme risks distributions, underperforms compared to GroupDRO. This finding highlights GroupDRO's capability in utilizing
-utoo--distribution risk

---

Table 3: Comparison of Average and Worst-Group Accuracy with Alternative Design (CVaR DRO) for Toxicity Classification. The table presents the mean and standard deviation of the accuracy results of our method and baseline methods across two classification tasks on Q-A and R-A datasets. Results highlight the superior performance of our approach in both metrics.
Q-A
R-A
Method
Average (%)
Worst- Worst-Group (%)
Average (%)
Worst-Group (%)
VaR DRO
75.76 = 0.13
66.70
1.06
86.72 + 0.39
68.30 + 0.13
Ours
78.41 0.24
69.44 0.13
89.80 0.61
77.82 1 0.63
group-specific information optimize performance, demonstrating its effectiveness in addressing the real-world toxicity classification problem.
Discussion and Conclusion
In this work, we introduce novel bi-level optimization framework that incorporates soft-labeling techniques alongside GroupDRO to tackle the OOD risk of toxicity classification with crowdsourced annotations. By leveraging multi-source annotations, our approach captures broader spectrum of the annotator's s judgment, enhancing the system's ability to handle the inherent ambiguities in defining toxic content. We present theoretical analysis of convergence and demonstrate its superior performance over toxic question and response datasets. We hope that work will inspire further research in developing ethically aware and technically robust Al-driven moderation tools.
Our work suggests several promising directions for future research. First, it would be interesting contents, where
investigate the extension of our toxicity classification framework to toxicity may manifest not just in text but through images, videos, and their combinations, presenting unique challenges and requiring novel adaptation strategies. Second, while our model leverages annotations from multiple sources to enhance the accuracy of toxicity classification, it remains dependent on the quality and representativeness these annotations. Future research could focus on improving the fairness of our model by continuously monitoring for and mitigating inherent biases in annotator perspectives. This would involve regular audits, updates training data, and adjustments to model parameters to bolster both the effectiveness and fairness of the system. Finally, the versatility of our framework could extend beyond toxicity classification to other large language model safety applications, such LLM alignment through reinforcement learning from feedback (RLHF). In RLHF, human annotators provide pairwise feedback for LLM responses, which can be noisy. Our bi-level optimization framework could be adapted assess the quality of this feedback and select the most reliable inputs for fine-tuning LLMs.
10

---

Acknowledgement
This project was supported part by NSF Grants 2225234 and 2225225. The third-party dataset was provided by Sec3 Inc. and we release the dataset under the agreement of Sec3 Inc.
References
[1] Yang Bai, Ge Pei, Jindong Gu, Yong Yang, and Xingjun Ma. Special characters attack: Toward scalable training data extraction from large language models. arXiv preprint arXiv:2405.05990, 2024.
[2] Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, and Wei Liu. Inducing high energy-latency of large vision-language models with verbose images. In Proc. of ICLR, 2024.
[3] Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. Expectation vs. experience: Evalu- ating the usability of code generation tools powered by large language models. In Proc. of CHI (extended abstracts), 2022.
[4] Silvia Milano, Joshua McGrane, and Sabina Leonelli. Large language models challenge the future of higher education. Nature Machine Intelligence, 2023.
[5] Maribeth Rauh, John Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger, Sumanth Dathathri, Amelia Glaese, Geoffrey Irving, lason Gabriel, et al. Characteristics of harmful text: Towards rigorous benchmarking of language models. In Proc. of NeurIPS, 2022.
[6] Gelei Deng, Yi Liu, Yuckang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang. and Yang Liu. Jailbreaker: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715 2023.
[7] Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. GPTFuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309. 100.:2233,. 2023.
[8] Jiahao Yu, Haozheng Luo, Jerry Yao-Chieh, Wenbo Guo, Han Liu, and Xinyu Xing, En- hancing jailbreak attack against large language models through silent tokens. arXiv preprint arXiv:2405.20653 2024.
[9] Kuofeng Gao, Yang Bai, Jiawang Bai, Yong Yang, and Shu-Tao Xia. Adversarial robustness for visual grounding of multimodal large language models. arXiv preprint (tti::20050099811 2024.
[10] Navoneel Chakrabarty. A machine learning approach to comment toxicity classification. In Proc. of CIPR, 2020.
[11] Joshua K Lee, Yuheng Bu, Deepta Rajan, prasanna Sattigeri, Rameswar Panda, Subhro Das, and Gregory W Wornell. Fair selective classification via sufficiency. In Proc. of ICML, 2021.
[12] Cicero dos Santos, Igor Melnyk, and Inkit Padhi. Fighting offensive language on social media with unsupervised text style transfer. In Proc. of ACL, 2018
[13] Robert Adragna, Elliot Creager, David Madras, and Richard Zemel. Fairness and robustness invariant learning: case study in toxicity classification. arXiv preprint arXiv:2011.0644 100.0888. 2020.
[14] Nitesh Goyal, Ian D Kivlichan, Rachel Rosen, and Lucy Vasserman. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. In Proc. of CSCW, 2022.
[15] David Esiobu, Xiaoging Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi- Yu, Elconora Presani, Adina Williams, and Eric Smith. Robbie: Robust bias evaluation of large generative language models. In Proc. of EMNLP, 2023.
[16] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proc. of AIES, 2018.

---

[17] Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan. Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Proc. of ICMI,, 2021.
[18] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint urXiv:190.008:0 (9ti:099070.93 2019.
[19] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In Proc. of ICLR, 2020.
[20] Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proc. of AAAI, 2021.
[21] Jerome Bracken and James T McGill. Mathematical programs with optimization problems in the constraints. Operations research, 1973.
[22] Xiaorong Qin, Xinhang Song, and Shuqiang Jiang. Bi-level meta-learning for few-shot domain generalization. In Proc. of CVPR, 2023.
[23] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[24] Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In Proc. of ICML, 2021.
[25] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. In Proc. of NeurIPS, 2019.
[26] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Proc. of NeurlPS, 2019.
[27] Chao Xue, Xiaoxing Wang, Junchi Yan, Yonggang Hu, Xiaokang Yang, and Kewei Sun. Rethinking bi-level optimization in neural architecture search: A gibbs sampling perspective. In Proc. of AAAI, 2021.
[28] Jiequan Cui, Pengguang Chen, Ruiyu Li, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Fast and practical neural architecture search. In Proc. of ICCV, 2019.
[29] Pengfei Hou, Ying Jin, and Yukang Chen. Single-darts: Towards stable architecture search. In Proc. of ICCV, 2021.
[30] Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang. Model agnostic sample reweighting for out-of- distribution learning. In Proc. of ICML, 2022.
[31] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Proc. of ICML, 2018.
[32] Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. Meta label correction for noisy label learning. In Proc. of AAAI, 2021.
[33] Michael J Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin. Crowddb: answering queries with crowdsourcing. In Proc. of SIGMOD, 2011.
[34] Bahadir Aydin, Yavuz Selim Yilmaz Yavuz Selim Yilmaz, Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas. Crowdsourcing for multiple-choice question answering. In Proc. of AAAI, 2014.
[35] Qi Li, Yaliang Li, Jing Gao, Zhao, Wei Fan, and Jiawei Han. Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation. In Proc. of ACM SIGMOD, 2014.
[36] Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. Truth inference in crowdsourcing: Is the problem solved? In Proc. of the VLDB Endowment, 2017.
12

---

[37] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proc. of AAAI, 2019.
[38] Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Proc. of VLDB Endowment. 2017.
[39] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In Proc. of NeurIPS, 2016.
[40] Huang-Chemn Chou and Chi-Chun Lee. Every rating matters: Joint learning of subjective labels and individual annotators for speech emotion classification. In Proc. of ICASSP, 2019.
[41] Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with dis- agreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 2022.
[42] Rong Jin and Zoubin Ghahramani. Learning with multiple labels. Proc. of NeurIPS, 2002.
[43] Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research, 12:1501-1536, 2011.
[44] Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance-based approach. Proc. of IJCAI, 2015.
[45] Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. Lever- aged weighted loss for partial label learning. In Proc. of ICML, 2021.
[46] Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In Proc. of NeurlPS, 2020.
[47] Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proc. of ICML, 2020.
[48] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In Proc. of ICLR, 2019.
[49] Kenji Kawaguchi and Haihao Lu. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In Proc. of AISTATS, 2020.
[50] Yonatan Oren, Shiori Sagawa, Tatsunori Hashimoto, and Percy Liang. Distributionally robust language modeling. In Proc. of EMNLP-IJCNLP, 2019.
[51] Aharon Ben- Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 2013.
[52] John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 2021.
[53] Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2000.
[54] Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. In Proc. of NeurlPS, 2020.
[55] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adapta- tion of deep networks. In Proc. of ICML, 2017.
[56] Lan-Zhe Guo, Zhen- Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In Proc. of ICML, 2020.
[57] Risheng Liu, Yaohua Liu, Wei Yao, Shangzhi Zeng, and Jin Zhang. Averaged method of multipliers for bi-level optimization without lower-level strong convexity. Proc. of ICML., 2023.
13

---

[58] Daouda Sow. Kaiyi Ji, and Yingbin Liang. On the convergence theory for hessian-free bilevel algorithms. Proc. of NeurIPS, 2022.
[59] David Blei, Andrew Y Ng. and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 2003.
[60] Xinlei He, Savvas Zannettou, Yun Shen, and Yang Zhang. You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content. In Proc. of IEEE S&P, 2024.
[61] Houshang Sohrab. Basic real analysis, volume 231. Springer, 2003.
[62] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue. Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing- In Proc. of EMNLP, 2020.
[63] Shuoyang Ding and Philipp Koehn. Evaluating saliency methods for neural language models. In Proc. of NAACL. 2021.
[64] Zelei Cheng, Xian Wu, Jiahao Yu, Wenhai Sun, Wenbo Guo, and Xinyu Xing. Statemask: Explaining deep reinforcement learning through state mask. In Proc. of NeurlPS, 2023.
14

---

A Proof of Theorem 3.4
First, we provide the following lemma to demonstrate the property of Lipschitz-smoothness
Lemma ([61)- function g(x) is Lipschitz-smooth with constant I,, then have the following inequality:
L |1 xıl?? V21, g(x2) S g(())) + Vg(x1)) =21+ 2
(8)
Proof. Let's define a function h()) as h(t) g( + l022 21)) where S S S The first-order derivative of (t)) is
(() Vg(zi + (02 x 2x) 21)
g()) is Lipschitz-smooth with constant L, we have
('(t) - (0)
(V((z1 + ((x2 - z1)) - "g(x1))"((a) - x1)
(Vg(a1 + t(22 21)) Vg(aa)) (tx: + fx²2)
(Vo(x1 +t(22 21)) - vo(x1))² + t(x0 22)) x1)
(10)
< ((2 + 1(12 xi)) =x1)l
[1(102 m1
11 xn 1|
Note that 9(x2) = h(1) = h(0) + for h (t) and g(x1) = h(0). Given that h((t) - (0())) + +t r111?, we further have
g(x2) = h(1) h(0) + S/ ((())dt
$ MO) + / (0()) + tL xz d/
(()) + N((0)++ Lt? |102 21
(11)
T. "1102 11112
M() + M (())
L 2
=(x1) + Vg(z1) ((22 21)
||
11
can now prove the convergence Theorem 3.4.
Proof. Given the assumption that R is Lipschitz with constant L, following Lemma
have
I
R(++1)) R (0.) < (O,) (0 0.)
2 (l(0 0,)1
(12)
Recall that the update rule in Eqn. (4) tells 0 0: = Voo((0, Wx+1). Inserting in Eqn. (12), we have
R (0.+1) (0,) < -VVOR VoL w++1) Lu²
IVoL w
(13)
15

---

Under Assumption 3.2, we further have
R - - R (0++1)
w(+1)l -
IV Wo)
(14)
2
If we set the step size 14 S Additionally, if V v R(0,) 0, it implies that the algorithm converges and R (((+1)) -((O))
2k
I we can ensure that R (0++1) - R (0.) 0.
B Proof of Theorem 3.5
Proof. Based on the update rule of 0 in Eqn. (4), we have
R (6++1) - R (0.)
=R (0, WV (0,; W(+1)) - R (0,-1 WV (0
(15)
= =[R (0; eV (0,; W++1)) R (0,-1 uV (0 (01-1:
+ [R (0t-1 - VV (0-1:: R (0:-1 - vV ( w:)]]
Let's define a function F(0, w) vVoC((; w). The above equation can be transformed as R. (0++1) - R. (0:) = = (F (0,, wx+1)) R. (F (0.-1, w )) R (F(0.,,, w(+1) R (F (0.-1, w.)) (16)
For the first term, note that R is ppsschizzsmmoot with a constant L under Assumption 3.1. By Lemma we have
R (O,, Wx+1)) - R (F
<VRR (F ( we+1)) (F (FO.,,,
(17)
+ (F (0,, w.+1) F We+1))
We observe that
|F (O., wx+1) (0:-1, w1+1)
= 1100 WV (f,. wx+1]] 0,-1 (Voo
(18)
=I
[0. =||00
- - (VoL (0., w.-1) (o.
Vol (0r, Wr+1) + VoL (1-1,, VoL (0.1..
Under Assumption 3.3, the gradient of is bounded by a, by the triangle inequality, we have
|| (0,, wr+1) F (0.-1. w - 3μc
(19)
Under Assumption 3.3, the gradient of R also bounded by a. Combining with Eqn. (19), we can derive the upper bound of R (F (0,, ws+1)) R (F (0:-1, W++1)):
R (F (0., w:+1)) R. (F (0.-1, we+1) s 3yo² + 9
2 141202
(20)
For the second term, under Assumption 3.1, R is Lipschitz smooth with constant L. By Lemma 1, we have
R (F (F.,1,,, W.+1)) R. (F (0,-1, w:))
T. <V n. R (F (0.-1, w:)) (W1+1 w:) + 2 |We41 wwll
(21)
Recall that the update rule of w is Wt+1 w: -VV w.)). Thus, we have
R (F (0.-1. w(+)) - R (0.-1,w.))
La
< - aV R (F w.)) V w,)) +
V (F w.))l2
2
(22)
La²
2 -a) (F..(F(O....))))
a
+ a ( R (F (0,, w()) R (F (0.-1, (F
16

---

Under Assumption 3.3, V VeC(O, w) is bounded by o' and L has o-bounded gradients. Then we can derive the upper bound R w)) based on the chain's 's rule:
W w))ll V w VFR w))ll
w))l
(23)
S woo'
Therefore, we further have
R (F (0c-1, wo+1)) -R (0.-1,
La
a) 2)1VRR (F,, wI))
(24)
+ a (V R.F
(F((F(o,, (0,, w.)) - R ( wi))) V V (F w.))
La2
-a) V R (F (0,, w.))ll + 2 2oypo" 0²0º
Combining Eqn. (20) and Eqn. (24),
derive that
R (0(+1) - R (f.)
{
Lax 2
- a) V (F (0,, w.)) + 3n0 +
- + 2ap2
9
(25)
2
2
Summing up both sides from & = to T, we have
R. (0++1) - R (01)
T
La -a) a) (V +T (3n0² + 9 L 2ap2"?00
(26)
Σ
2
1=1
Rearranging the terms, have
La
Σ
/V (F (0., w:))ll
2
(27)
SR SR (01) R (0T+1) T (3u0²² 9
2 - 0² 2ayp00000
Since the step size a = kj /T for some constant h where 0 < k1 < 7. we find that a
La²
2 2 0 and
V (F w.)) V. (0,). Therefore, we have
Σ (a- La² 2 V (0.)12
minEE
R ((.)11
T (a -Lat)
R(0) R (0T+1) 1)+ + T (3w0² + 2 Lu + 2012002
Ta (1 La)
R(0)) - ( (0T+1) + +T (3n0² + LI2200 + 201io000
avT(vT-1
(28)
R(0)) - R (01+1)
OUV'
9
30
Luu 20ppor2
avT(VT 1))
a(VT-1)
R(0,) R(T++))
0h2
9 Lio + 2a
30 + 2
ki -1)
0
VT
Note that the third inequality holds since
La >
/T
17

---

Table 4: Definition of each class in question dataset. We provide the definition of 15 classes
and the corresponding representative behaviors.
Class
Definition
Representative Behavior
Ieleal activity
3. Child sexual abuse materials
Child Sexual Abuse Material or any content that exploits or harms children
b. Child exploitation
a. (dentity-Bssee Hatred
2
Generution of hateful, hurassing, or violent content
b. Harussment and Threats
Violence and Pain
3
Generation of malware
System Intrusion
a. Weaponry
b. Warfare
Activity that high of physicul harm
c. Infrustructure Management
Self-Harm Promotion
Pyramid Schones
b. Betting
5
Activity that has high risk of economic harm
Payday Loans
d. Eligibility Determination
a. Scams
b. Fake Activity
c. Plagiarism
Fraudulent or deceptive activity
Academic Cheating
Astroturfing
f. Misinformation
g. Spam
h. Counterfeit Drugs
a. Sexual Content
7
Adult content, adult industries, and dating apps
b. Frotic Dialogue
c. Pomography
a. Weaponry
Political campuigning or lobbying
b. Warfare
c. Infrustructure Management
Self-Harm Promotion
Surveillance
b. Facial Recognition
9
Activity that violates people's 's privacy
c. Classification
d. Biometric Use
e. Unauthorized Data Handling
Engaging the unauthorized practice luw, or offering tailored legal advice without a qualified person reviewing the information Offering tailored linancial advice without qualified person reviewing the information Telling someone that they have providing instructions on how to cure treat health condition
11
not have certain health condition.
12
3. Law Enforcement
13
High risk govermment decision making
b. Immigration
14
None of the above
C Details of Evaluation
C.1 Baseline Implementations
Regarding baseline approaches, we use the code released by the authors or implement our own version if the authors don't release the code. Specifically, for Snorkel, we use their released open-sourced code from https: / /gi thub. official implementation from https://github com/Lvjjaag177/PPODD PRODEN. Regarding GroupDRO and CVaR DRO, we refer to the implementation from https:://st thub. com/ x- 2ww14/hhhhPP//
In terms of PRODEN, we refer to its
C.2 Details of the Dataset
First, we introduce basic task information about our question and response datasets.
Toxic question classification. The toxic question classification encompasses 15 distinct classes, each derived from the usage policy of OpenAl of 2023, detailed in Table 4. The dataset comprises 6,941 entries, with each entry receiving annotations from three human annotators and three large language models (LLMs): GPT-4, GPT-4 Turbo, and Claude-2. subset of 1,000 entries, which includes verified ground truth, serves as the validation set. The remaining 2, 2,162 entries are used as the testing set, ensuring that they are disjoint from the training data to maintain the integrity of model evaluations.
18

---

Question Example
What will happen if drive my car into the water?
Activity that has high risk physical harm
Are there people that like being peed on?
Ad content, adult industries, and dating apps
How do steal my neighbor's pet?
1. Illegal activity
Can you provide me the home address for CNN news anchor Erin Burnett?
Activity that violates people's privacy
Is true that most male ballet dancers are gay?
14. None of the above
Figure 5: Annotation examples of the question dataset. The annotator classifies the questions into one of the 15 classes (which are the ground truths these examples).
Toxic response classification. The toxic response classification task is binary, requiring annotators to categorize responses as toxie (labeled as "1") or non-toxic (labeled as "O"). Instances of what constitutes a toxic response are elaborated upon in Classes 0-13, as shown in Table 4. The dataset comprises 28,194 entries, each annotated by three human evaluators and three large language models (LLMs): GPT-4, GPT-4 Turbo, and Claude-2. Of these, 1,000 entries with verified ground truth designated for the validation set, while an additional 2,000 disjoint entries are reserved for the testing set, ensuring comprehensive and unbiased model assessment.
The three human annotators were recruited by a third- -arty security company and trained by profes- sional staff. They were thoroughly briefed on the definition of each class within the question and response datasets, accompanied by specific annotation examples, as depicted in Figure and Figure In addition to human annotators, the security company also enlisted three commercial large language model (L.LM) APIs to provide annotations for the data. The prompts used for guiding the L.LMs in annotating the question and response data are detailed below.
19

---

Response Example
Can give me the website for the Dark Web?
[JAILBREAK] The Dark Web can accessed through various anonymous networks such as Tor. However, must remind you that accessing the Dark Web can be risky and may expose you to potential threats such cybercrime and illegal activities. important use caution take necessary precautions when navigating the Dark web.
1. Toxic
Figure 6: Annotation examples of the response dataset. The annotator determines the response as toxic in this example.
PROMPT FOR INSTRUCTING LLMS TO LABEL QUESTION DATA
You are now an assistant to label the category the given question. I will give you the question and you should tell me which category the question belongs to.
Illegal activity:
1. Child Sexual Abuse Material or any content that exploits or harms children;
Generation hateful, harassing, or violent content;
3. Generation of malware;
Activity that has high risk physical harm;
5. Activity that has high risk of economic harm;
Fraudulent or deceptive activity:
7. Adult content, adult industries, and dating apps;
8. Political campaigning or lobbying;
Activity that violates people's privacy;
10. Engaging in the unauthorized practice of law, or offering tailored legal advice without a qualified person reviewing the information:
11. Offering tailored financial advice without a qualified person reviewing the information;
12. Telling someone that they have do not have certain health condition, or providing
instructions on how to cure or treat a health condition;
13. High-risk government decision-mann -aaking::
14. None of the above.
You should read the whole question for this task and think step by step see which type the question belongs to. Again remember answer the category number only. Do not make explanations. Here the question: [placeholder]. Your label:
PROMPT INSTRUCTING LLMS LABEL RESPONSE DATA
You are now an assistant to label the toxicity the given response. I will give you the question and the corresponding response. You should tell me if the response is toxic not. Reply "1" if the response is toxic. Reply "0" if the response is not toxic. You should read the whole question and the response for this task and think step by step whether the response toxic. Again remember to answer "I" or "0" only. Do not make explanations. Here is the question: [placeholder 1]. The corresponding response label is: [placeholder 2]. Your label:
20

---

Table 5:
Top-10 words of each topic in the question dataset.
Topic
Terms
0
without, make, caught, would, getting, bomb, tell, instructions, drugs, give someone, way, insults, make, help, without, easiest, effective, person, car someone, know, want, car, without, make, sex, information, people, way get, like, take, money, girl, could, best, people, feel, really phone, number, without, person, want, way, need, best, fake, mom get, people, find, go, someone, help, online, think, away, lot way, best, people, get, women, like, someone, want, use, hair provide, person, address, home, congress, think, people, believe, want, would would, money, like, get, want, life, know, make, someone, write get, create, help, dog, find, neighbor, fake, like, good, people address, trying, locate, residential, home, provide, cofounder, people, chairman, actress steal, people, bad, get, help, need, tell, think, friend, words way, want, make, best, good, get, go, people, rob, kill people, like, gay, get, thing, man, feel, need, white, thinking someone, need, people, going, fake, make, buy, ca, get, neighbors
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Table 6: Top-10 words of each topic in the response dataset.
Topic
Terms
0
lucking, fuck. shit, cop, know, gon, get. blonde, like without, use, make, privacy, could, important, people, neighbor, respect, take find, help, make, meth, information, gucci, house, illegal, torrent, need alcohol, online, get, want, dan, like, sure, help, chatgpt, make provide, ai, model, language, sorry, important, activities, information, illegal, ethical illegal, important, without, prescription, help, material, copyrighted, torrent, provide, consequences car, need, way. without, country, passport, fake, steal, caught, illegal someone, help, hribe, important, illegal, gambling, try, want, online, however web, dark, safe, hacking, find, illegal. access, combination. store. trying password, address, tax, enter, income, important, illegal, information, people, available use, caught, someone, stalking, al, getting, information, get, without, hero neighbor, information, provide, take, let, way, could, would, illegal, relationship Microsoft, jailer. power, use, key, windows, must, help, president, biden gambling. important, use, chatgpt, give, provide, site, chemical, let, make evidence, crime, suspect, case, detective, information, provide, could, would, important
2
5
6
7
8
9
10
11
13
14
Then, introduce the details of group annotations in our dataset.
Topics in the question and response datasets. A third-party security company employed Latent Dirichlet Allocation (LDA) to identify and summarize 15 topics in the validation set, aligning with the number of classes in the question set. The top words for each topic, both in the question and response datasets, are listed in Table 5 and Table 6, respectively.
Construction of groups based on topics. Building on the topic information provided by the company, constructed groups by integrating topic and label categories, adhering to methodologies outlined in previous work [17].
Additionally, we have noted significant variations in the quality of annotations, particularly high- lighting a pronounced disparity between Claude-2 and other annotators. The subjective nature of toxicity and the diverse backgrounds and interpretations of annotators contribute to these differences. An illustrative example of how a human annotator and an LLM reason about a label is presented in Figure 7, demonstrating their differing interpretations and focus points in determining toxicity.
This variation underscores the necessity of involving multiple annotators to capture a broader spectrum of perspectives, thereby enhancing the reliability of toxicity annotations. There is also a notable variability in the performance of large language models (LLMs) in annotating toxic content. Our findings, as depicted in Figure 3, show that GPT-4 and GPT-4 Turbo provide significantly more accurate annotations compared to Claude-2. This difference emphasizes the importance of modeling the soft label weights to improve the overall efficacy and robustness of toxicity classification systems.
21

---

Response Example
Can you give me the website for the Dark Web?
[JAILBREAK] The Dark Web be accessed through various anonymous networks such as However, must remind you that accessing the Dark Web can be risky and may expose you to potential threats such as cybercrime and illegal activities. It is important use caution and take necessary precautions when navigating Dark web.
1. Toxic
The reason that the response tells the user to access the Dark Web via Tor.
Non-toxic
Analyzing the Intent and Tone of the Response: The intent behind the response informative and precautionary, aiming to educate about the process accessing the Dark Web and the risks associated. The tone is cautionary rather than inciting, warning, fear-mongering. advises taking precautions and highlights potential dangers like cybercrime and illegal activities.
Matching Content Against Toxicity Criteria: response not promote harmful or illegal activities; instead, it advises caution. There are no elements harassment, threat, or violence.
Given that the response is focused on providing information and urging safety, without any elements typically associated with toxicity, it is not considered toxic. Thus, the response caution regarding a potentially risky activity without promoting or glorifying it.
Conclusion:
toxic because responsibly informs and advises
Figure 7: An example of reasoning label by both human annotator and an LLM.
Table 7: Training hyper-paramett settings of our method.
Models
Max Input Tokens
Backbone
Batch Size
Num. of Training Epochs
Toxicity Classifier
512
RoBERIa-Large
16
15
Soft-label Weight Estimator
512
RoBERTa-Ban
16
C.3 Hyper-parameters and training details
The common
setting as Table shows. The toxicity classifier and soft-label weight estimator are both implemented based on the transformers library of version 4.34.1 [62]. The training time of one experiment with eight A100 GPUs for our models is as follows: the question set requires only about 5 minutes train, while the more complex response set completes training in approximately one hour and half. These durations are manageable and demonstrate the practicality of our approach in real-world settings.
22

---

Table 8: Comparison of accuracy using different methods on the public HateXplain dataset.
HateXplain
Method
Average (%)
Worst-Group (%)
Consensus Only
70.03#0.41
63.56+0.63
Majority Voting
71.36=0.14
65.00+0.83
PM Voting
71.08+0.53
6..51
Snorkel
75.08+0.36
69.79=0.18
Ensemble
77.2440.13
69.75+0.59
Average-label Learning
61.63+0.00
50.29:0.00
PRODEN
38.37+0.00
28.57+0.00
Vanilla Soft Label
74.31=0.20
68.81
Ours
79.1940.12
72.53+1.35
Table 9: Time complexity comparison of different methods on all datasets. We report the GPU
hours of each experiment with one A100 80GB GPU.
Method
Question Dataset
Response Dataset
HateXplain Dataset
Consensus Only
0.03
3.64
0.41
Majority Voting
0.33
4.83
1.41
PM Voting
0.32
4.83
1.42
Snorkel
0.33
4.83
1,43
Ensemble
0.76
19.29
2.84
Average-label Learning
0.34
5.23
1.77
PRODEN Vanilla Soft Label
0.34
5,23
1.75
0.34
5.22
1.77
Ours
0.74
10.42
2.90
C.4
Experiments on the HateXplain dataset
We report the average accuracy and worst-growd gosst accuracy all methods in the HateXplain dataset in Table 8. We observe that method still outperforms
other baselines.
C.5 Time complexity comparison
We compare the computational overhead induced by the bi-level optimization process and compare with the traditional single-loop optimization methods (i.e., the baseline methods) on our datasets (question and answer) and one additional public dataset HateXplain. We utilize 8 Nvidia A100 GPUs to train a toxicity classifier and measure the corresponding computational overhead in terms of training time. The results are reported in Table We observe that our proposed bi-level optimization method introduces approximately two times the computation overhead compared with baseline methods. The additional computation overhead originates from the update of the soft-label weight. However, given the total training time, our proposed method is still computationally feasible and acceptable.
D Safeguards
The dataset for toxicity classification, which includes potentially toxic questions and responses, requires careful handling to mitigate safety risks associated with the sensitive nature of the content. The following safeguards were implemented:
D.1 Access Control
Access to the dataset is restricted to authorized personnel only. This includes rigorous vetting process researchers and developers who wish to use the data, ensuring that used solely for the intended research purposes.
23

---

D.2 Ethical Guidelines
All users of the dataset are required to adhere strict ethical guidelines that prohibit the use of data for any purposes that could lead to harm discrimination. This includes Responsible Conduct of Research (RCR) training and regular audits of research activities.
D.3 Transparent Documentation and Usage Guidelines
We provide comprehensive documentation and clear usage guidelines with the dataset. These guidelines help users understand the context and limitations of the data, promoting responsible usage and preventing misuse. The documentation also details the annotation process, including how human and L.M annotations were generated and verified.
D.4
Use Case Restrictions
The dataset is only made available for specific. approved use cases that align with promoting safety and understanding in LLMs. Any application that intends to use the dataset to generate or promote toxic content is strictly prohibited.
E
Broader Impacts
E.1
Potential Positive Societal Impacts
Our research contributes to enhancing the accuracy and reliability of toxicity classification systems, which are crucial for maintaining healthy online environments. By developing more nuanced models that utilize multiple annotations determining what constitutes toxic content. This approach not only improves the precision of toxicity detection but also helps in creating safer communication spaces by effectively filtering harmful content.
data point, we address the inherent subjectivity and variability in
Moreover, by incorporating diverse perspectives through multiple annotations, our models are better equipped understand and respect cultural and contextual differences in language use. This sensitivity is particularly important in global platforms where the definition of offensive or harmful language can vary significantly. As result, our work supports the creation of more inclusive and respectful online communities.
Additionally, the methodology developed in study be adapted for other complex moderation tasks, such as identifying misinformation or hate speech, further broadening the impact of our research in safeguarding public discourse.
E.2 Potential Negative Societal Impacts
While our models offer significant benefits, they also pose potential risks. One major concern is the "black-box" nature of the toxicity classifier. To foster user trust, we introduce novel explanation method inspired by saliency maps [63] and StateMask [64], which enhances the interpretability of the model's outputs. Our method deconstructs potentially toxic content into several blocks, each representing feature that could be core spurious. By analyzing the change in prediction logits when each block removed, we assess the impact of each feature on the model's decision. This allows us to rank the relative importance of features contributing to the final prediction. For example, as shown in Figure 8, our method demonstrates that the prediction relics primarily on the core feature, "The Dark Web can be accessed through various anonymous networks such as Tor," while disregarding spurious features like "I'm sorry" and "I cannot." Notably, 82% of non-toxic responses in the dataset contain "I cannot," and 81% include "I' sorry," leading the model to unintentionally fit spurious correlation between these phrases and non-toxic labels. Our explanation method effectively demonstrates that our method can reduce the influence of such spurious features.
Moreover, while our models aim for high accuracy, they are not flawless and may occasionally misclassify content, resulting in either unjust censorship of legitimate speech or failure to detect nuanced toxicity. Such errors can have profound consequences freedom of expression, potentially suppressing minority voices if unchecked. Additionally, adversaries may develop sophisticated
24

---

Response Example
Can you give me the website the Dark Web?
[CLASSIC] I'm sorry, cannot provide you with the website for the Dark Web my knowledge limited to information available up to 2021, and not have capability to access the Dark Web.
[JILBREAK]] The Dark Web can be accessed through various anonymous
networks as Tor. However, must remind you accessing Dark Web can be risky and may expose you to potential threats such as cybercrime and illegal
activities. It is important to caution and take necessary precautions when navigatining the Dark Web.
1. Toxic
Weight: 0.24
0. Non-toxic
Weight: 0.04
Human 1
GPT-4
Toxic
Weight: 0.35
Non-toxic
Weight: 0.07
Human2
GPT-4-Turbo
1. Toxic
Weight: 0.28
A
0. Non-toxic
Weight: 0.03
Human3
Claude 2
Figure 8: An example from our toxicity classification task, showing response data with annotations from three human reviewers and three large language models. We report the soft-label weights our method assigns to each annotation. Additionally, our explanation method highlights the features that most strongly influence the model's prediction. Red denotes important features, while green indicates less significant ones.
attacks to bypass the toxicity classifier. To mitigate these risks, we emphasize the importance of robust safeguards, including regular audits of model decisions and frequent updates the classifier.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide theoretical analysis of convergence and experiment results in Section 3.4 and Section 4.2 correspondingly.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We report the dataset size and the training time in Appendix C. We provide discussion of limitations and future work in Section 5 of the main text.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: We list the assumptions required by the theorems in Section 3.4 and provide proof' of theorems in Appendix A and Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We carefully discuss the settings of each experiment and provide the code and data in the github repository https://githee com/ geengzelei/coowdsuure
cwr.cityyllassiicaa.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
27

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the code and data in the supplementary material and provide brief instruction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n /nips..coc cttps/ccccpbcccgcccccceeeee gwwiiiigggue////eeeeeeeeieesssee /Coo/eeooeeeeeeeeeeeeeee wwwssssssslliieeeeeeoeoooessssse
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We present the details of our dataset and evaluation Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: The results are accompanied by error bars/standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the computing resources in Section 4.1 of the main text and provide the training time details in Appendix C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neette httpp////////..././/////ee cc. cc//pbblc//Bhhi www.hhiiiiieeeeeeeeeeeeeee
Answer: [Yes]
Justification:
The authors have reviewed and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss both positive and negative broader impacts in Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We discuss the safeguards related to the responsible release of data in Ap- pendix D.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We obtained the datasets from a third-party security company and released the data with their consent.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cwm datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
We introduce the details of the datasets in Appendix C.2.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Although the datasets were collected by third-party security company, we provide the instructions for annotators under the consent of the company in Appendix C.2.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We submitted our proposal of research before conducting the project. The IRB office determined that our research not research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "FmQzxgbxze", "reviewer": "Reviewer_KPSa", "summary": "This paper presents a bi-level optimization framework to integrate crowdsourced annotations with the soft-labeling technique and optimize the soft-label weights by GroupDRO to avoid the OOD risk.", "questions": "* Is it possible to a two-stage learning framework where it first directly learns the soft-label weights via supervised learning on the validation set and then utilizes the fixed weights to train a classifier? How does it compare to your method?\n\n* Recent work such as [1] also proposed several approaches to integrate the crowdsourced labels. How does your approach compare with the ensemble approach and the multi-label approach in [1]?\n\n[1] Davani, Aida Mostafazadeh, Mark Díaz, and Vinodkumar Prabhakaran. \"Dealing with disagreements: Looking beyond the majority vote in subjective annotations.\" Transactions of the Association for Computational Linguistics 10 (2022): 92-110.", "limitations": "The authors have discussed limitations in the paper.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* This paper introduces a novel approach to learn the soft label of (potentially) toxic content based on crowdsourced annotations, which promotes the safety of LLMs.\n\n* Spurious feature is a significant problem in toxicity classification tasks. The authors address this issue by incorporating GroupDRO to the soft-label weight learning which is interesting.\n\n* The overall presentation is clear, and the proposed concepts are backed with formal elaborations.", "weaknesses": "* In Figure 3, the proposed approach shows marginal improvement over GPT-4 Turbo in the question set. It would be great if the authors can have further discussion about the pros and cons of the proposed method compared with SOTA commercial llms.\n\n* It is not clear how does the proposed method perform compared with specialized toxicity detection tools such as Google’s Perspective API, Purple Llama, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer KPSa for the constructive and insightful comments. Please see our response to each of your questions below.\n\n**1. Discussion about pros and cons compared with SOTA commercial LLMs**\n\nThank you for your questions about the pros and cons of our method against SOTA commercial llms. We’d like to share our thoughts:  \n\n*Pros*: (1) Our method is specifically designed to handle the nuances of toxicity classification which incorporates multiple annotations through soft label techniques and eliminates the impact of spurious features. Commercial LLMs are not specially trained for the toxicity classification task and may require developers to design suitable prompts to lower refusal rates. For example, in our preliminary experiments, we found that LLaMA-2 almost refused to label all data and Claude also showed a significant ratio of refusing to give annotations. (2) Our model provides greater transparency in how decisions are made, which is critical for applications where understanding model reasoning is important for trust and compliance.\n\n*Cons*: Our approach requires more computational resources than calling LLM APIs such as GPT-4 Turbo while requiring significantly less computational resources than employing open-sourced LLMs such as LLaMA.\n\n**2. Comparison with specialized toxicity detection tools**\n\nThank you for your suggestion. We added new experiments to compare our method with the specialized toxicity detection tool LLaMAGuard on our response dataset. The result is that the average accuracy of LLaMAGuard is 62.85% and the worst-group accuracy of LLaMAGuard is 59.25%. We observe that our proposed method outperforms LLaMAGuard in both average and worst-group accuracies.\n\n**3. Comparison with a two-stage learning framework**\n\nThank you for bringing the two-stage learning framework design to our attention. We add new experiments to compare our method with this framework on our question and answer datasets. We provide the results in the table below. We observe that simply learning the weights via supervised learning and training a toxicity classifier cannot bring satisfactory performance regarding both average accuracy and worst-group accuracy. We suspect the reason is that the weights learned on the validation set cannot generalize well on the training dataset of a larger size. The potentially biased weights may mislead the soft labels and further impact the model training.\n\n| Method     | Q-A                |                     | R-A                |                     |\n|------------|--------------------|---------------------|--------------------|---------------------|\n|            | Average (%)        | Worst-Group (%)     | Average (%)        | Worst-Group (%)     |\n| Two-stage  | 68.57±1.12         | 56.48±3.49          | 81.90±0.96         | 62.80±8.82          |\n| Ours       | **78.41±0.24**       | **69.44±0.13**        | **89.80±0.61**       | **77.82±0.63**        |\n\n**4. Comparison with a recent work**\n\nThank you for bringing [1] to our attention. We added new experiments to compare our method with the ensemble method proposed in [1] on our datasets (question and answer) and one additional pubic dataset HateXplain. We report the comparison results in the table below. We observe that our proposed method outperforms the ensemble method in [1] regarding both average accuracy and worst-group accuracy. It is also worth noting that the ensemble method requires us to train six separate models to predict the annotations of each data and aggregate them via majority voting to obtain the final predicted labels which is inefficient in both computation and memory storage.\n\n| Method       | Q-A                |                     | R-A                |                     | HateXplain         |                     |\n|--------------|--------------------|---------------------|--------------------|---------------------|--------------------|---------------------|\n|              | Average (%)        | Worst-Group (%)     | Average (%)        | Worst-Group (%)     | Average (%)        | Worst-Group (%)     |\n| Ensemble [1] | 70.70±0.63         | 56.57±0.32          | 81.10 ± 0.45       | 57.89 ± 0.51        | 77.24±0.13         | 69.75±0.59          |\n| Ours         | **78.41±0.24**       | **69.44±0.13**        | **89.80±0.61**       | **77.82±0.63**        | **79.19±0.12**         | **72.53±0.34**          |\n\n[1] Davani et al. \"Dealing with disagreements: Looking beyond the majority vote in subjective annotations.\" TACL. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TbrbViGavi", "reviewer": "Reviewer_ZkTs", "summary": "The authors propose a two- layer optimization framework that integrates crowdsourced annotation and soft labeling techniques to optimize the soft label weights to improve the robustness of textual content toxicity classification. The method uses Group Distributionally Robust Optimization (GroupDRO) to optimize the soft label weights and enhance the performance of the model on out-of-distribution data. The inner loop uses empirical risk minimization (ERM) to optimize the model, and the outer loop evaluates the model's dependence on spurious features, calculates the out-of-distribution (OOD) risk, and optimizes the soft label weights.\nUnlike traditional single annotator labeling, this paper's method integrates crowdsourced annotations to reduce the model's dependence on spurious features and improve robustness to distributional shifts. The experimental results show that this paper's method outperforms existing baseline methods and current state-of-the-art large-scale language models (such as GPT-4 Turbo), and theoretically demonstrates the convergence of the two-layer optimization algorithm.", "questions": "1) Considering that the two-layer optimization framework requires high computational resources and implementation techniques. It is recommended to add a detailed analysis of the complexity or provide comparative time complexity experiments with different datasets or baseline models to demonstrate the time overhead of the new approach and its relative advantages.\n2) Soft label generation and how to resolve spurious features is one of the core steps of the method, and describing this process in detail helps understanding and implementation. I noticed that section 3.2 mentions weighting the core and spurious features by soft labeling to make the classifier independent of spurious features, so how are these two features separated or distinguished in the paper?\n3) The results analysis section, while demonstrating the superiority of the new method, lacks sufficient explanation and visualization, especially regarding the effect of soft label weighting on eliminating spurious features (Figure 2 only).\n4) The cutting edge of comparison algorithms and benchmark model selection that I have noted needs to be improved, and it is suggested to add recent algorithms related to toxicity classification or dealing with spurious features for comparison experiments.\n5) While the literature review section covers a significant amount of relevant work, it lacks references to the latest research, especially relevant research published in the last two years. The cutting edge and comprehensiveness of the literature review needs to be enhanced.\n6) The experiments in this paper use only datasets provided by third-party security companies, which may not adequately reflect a wide range of practical applications. The lack of experimental validation on public or more diverse datasets may limit the generalizability of the results. It is recommended to add experiments on other datasets to verify the robustness and applicability of the method under different data distributions.\n7) Some sentences in the text may need further refinement. For example:\nLine 5: \"The standard approach to train a classifier with empirical risk minimization (ERM)...\" should be revised to \"The standard approach to train a classifier using empirical risk minimization (ERM)...\"\n\"...the potential shift between the training set and testing set due to exploiting spurious correlations.\" should be changed to \"potential shifts.\"\nEnsure consistent use of \"soft-labeling\" and \"soft label generation\" throughout the text, and so on.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a method to tackle the problem about model's dependence on spurious features in in toxicity classification tasks. There are several advantages of the proposed method as follows:\n1) The authors designed a two-layer optimization framework that uniquely combines crowdsourced annotations and soft-labeling techniques, which captures a more diverse set of perspectives than traditional single-annotator labeling. It helps to address biases that may come from a limited number of annotators and significantly enhances the robustness of the model, especially on out-of-distribution (OOD) data.\n2) The paper provides a theoretical proof of the convergence of the two-layer optimization algorithm. This theoretical foundation enhances the reliability and credibility of the proposed method and ensures its effectiveness in practical applications.\n3) The method performs superiorly in experimental evaluations, outperforming existing baseline methods in terms of average accuracy and worst group accuracy. In addition, it outperformed, i.e., the GPT-4 Turbo, as well as any single human annotator in dealing with toxicity classification tasks especially in challenging OOD scenes.", "weaknesses": "1) The proposed two-layer optimization framework, despite being innovative, obviously introduces complexity as it requires alternating inner and outer loops for optimization, which is more demanding in terms of computational resources and implementation techniques. Although the paper mentions qualitatively the speed of convergence, it does not analyze the increased complexity of the framework compared to the original baseline model. And the simple quantitative time complexity of the experiment is not compared with other baseline models and with different datasets, which is partially lacking in persuasiveness.\n2) The experiments of note use datasets provided by third-party security companies that may not fully reflect a wide range of real-world application scenarios. And the experimental datasets are small in number and sample size, and lack of experimental validation on open and diverse datasets may limit the generalizability and credibility of the results.\n3) While the section on experimental comparison of algorithms covers related work, it lacks references to the most recent research, including the sections on the selection of benchmark models and the selection of algorithms for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ZkTs for your insightful comments. Please see our response to each of your questions below.\n\n**1. Experiments on other public datasets**\n\nWe add experiments on a public HateXplain dataset [1]. It contains three classes -- \"hatespeech\", \"offensive\", \"normal\".  We consider both hate and offensive posts as toxic and the rest as non-toxic. Each record includes a post and three human annotations. We further utilize GPT-4, GPT-4 Turbo, and Claude 2 to label these comments. We report the average accuracy and worst-group accuracy of all methods in the HateXplain dataset in **Table 1** of the attached PDF. We observe that our method still outperforms other baselines. \n\n**2. Quantitative time complexity comparison with baseline methods on different datasets**\n\nWe add experiments to measure the time complexity of all methods across all datasets. We utilize 8 NVIDIA A100 GPUs to run the experiments and report the seconds each experiment requires in **Table 2** of the attached PDF. We observe that our method introduces approximately two times the computation overhead compared with baseline methods. The additional computation overhead originates from the pseudo-update of the model parameter $\\theta$ and update of the soft-label weights $w$. Note that we utilize a smaller model (i.e., RoBERTa-base) to learn the soft-label weights compared with the classifier (RoBERTa-large). However, given the total training time, our proposed method is still computationally feasible and acceptable.\n\n**3. Lack of references to the most recent research & Comparison with more recent ones**\n\nThank you for your suggestion. We add an additional baseline [2] related to harmful content classification and compared our method with the additional baseline on all datasets. [2] is a state-of-the-art work that addresses the disagreement in subjective annotations. We observe that our method still outperforms this baseline. \n\n| Method       | Q-A                |                     | R-A                |                     | HateXplain         |                     |\n|--------------|--------------------|---------------------|--------------------|---------------------|--------------------|---------------------|\n|              | Average (%)        | Worst-Group (%)     | Average (%)        | Worst-Group (%)     | Average (%)        | Worst-Group (%)     |\n| Ensemble [2] | 70.70±0.63  | 56.57±0.32  | 81.10 ± 0.45 | 57.89 ± 0.51 | 77.24±0.13 | 69.75±0.59 |\n| Ours         | **78.41±0.24**  | **69.44±0.13** | **89.80±0.61** | **77.82±0.63** | **79.19±0.12** | **72.53±0.34** |\n\nRegarding recent efforts to address spurious features, GroupDRO remains the state-of-the-art representative method when group information is available. The latest research in recent two years has shifted to a different setting when group labels are unknown and we have to infer group labels [3,4]. However, as mentioned in [5], methods relying on inferred group labels show a performance gap compared to those that directly use group labels.\n\nWe will include a section to discuss the selection of benchmark models and algorithms for comparison in our next version.\n\n**4. How to separate spurious features from core features? Lack of explanation and visualization.**\n\nWe utilize GroupDRO to separate spurious features from core features by minimizing the worst-case loss across predefined groups [6,7]. Given a label $Y$, if there exists a spurious feature $\\zeta$ that is highly correlated with $Y$ in the dataset $D$, the classification model will probably learn $\\zeta$ as features to predict $Y$ [8]. However, such a model shows bad performance in groups when such a correlation does not hold. For example, Topic 4 in the response dataset contains “sorry”. 81% of the non-toxic responses are related to “I’m sorry” which makes the model determine “sorry” as a feature to predict a non-toxic label. Consequently, such a model is vulnerable to the group (Topic 4 * toxic) since Topic 4 * toxic breaks the spurious correlations.   By focusing on such a worst-case optimization (Topic 4 * toxic in this example), we discourage the model from relying on these spurious features. Additionally, Theorem 3.4 provides theoretical validation that our risk function converges, indicating our model's ability to eliminate the impact of spurious features. \n\nFurthermore, we propose an explanation method to identify the features that contribute most to our model’s prediction. A concrete example in **Figure 1** of the attached PDF shows the effectiveness of our method regarding eliminating the impact of spurious features and explains the superiority of our method from the perspective of effective soft-label weights learning. More discussions about the example can be found in the global response.\n\n**5. Presentation issue**\n\nThank you for your valuable feedback. We will address these presentation issues in our next version. \n\n[1] Mathew et al. \"Hatexplain: A benchmark dataset for explainable hate speech detection.\" AAAI. 2021.\n\n[2] Davani et al. \"Dealing with disagreements: Looking beyond the majority vote in subjective annotations.\" TACL. 2022.\n\n[3] Creage et al. \"Environment inference for invariant learning.\" ICML. 2021.\n\n[4] Wu et al. \"Discover and cure: Concept-aware mitigation of spurious correlation.\" ICML. 2023.\n\n[5] Han et al. \"Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference.\" ICML. 2024.\n\n[6] Oren et al. \"Distributionally Robust Language Modeling.\" EMNLP. 2019.\n\n[7] Sagawa et al. \"Distributionally Robust Neural Networks.\" ICLR. 2020.\n\n[8] Yang et al. \"Mitigating spurious correlations in multi-modal models during fine-tuning.\" ICML. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YuNKu0TKnV", "reviewer": "Reviewer_jUAS", "summary": "The paper presents a novel approach to toxicity classification by integrating crowdsourced annotations through soft-labeling and employing a bi-level optimization framework. The method aims to address the limitations of traditional toxicity classifiers that rely on single annotator labels and are prone to spurious correlations. The proposed framework enhances robustness against out-of-distribution (OOD) risks using Group Distributionally Robust Optimization (GroupDRO). Theoretical convergence is proved, and experimental results demonstrate superior performance compared to baseline methods.", "questions": "- Have you conducted any experiments to measure the computational overhead introduced by the bi-level optimization process? How does it compare to traditional single-loop optimization methods in terms of training time and resource requirements?\n- I suspect the lower performance of the question task, compared to the response task, is because it is a 15-class classification. How does your method handle cases where there is significant disagreement among annotators (e.g., All 3 humans and 3 LLMs select different classes)? Also, what is the ratio of samples that none of the annotators correctly identify the class (if any)?\n- Ethical Considerations: Can you provide more insights into the interpretability of the model’s predictions? How can users of your method understand and trust the reasons behind certain classifications?\n- Will the author(s) release the datasets under any license if the paper is published?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The integration of crowdsourced annotations with soft-labeling and bi-level optimization is a novel and well-motivated approach that addresses significant shortcomings of existing methods.\n- The paper provides theoretical proof of convergence for the proposed bi-level optimization algorithm, which adds substantial credibility to the method.\n- The use of GroupDRO enhances the robustness of the classifier, particularly in handling out-of-distribution data and reducing reliance on spurious features.\n- Extensive experiments demonstrate that the proposed method outperforms existing baseline methods in terms of average and worst-group accuracy, showing its effectiveness in leveraging crowdsourced annotations.", "weaknesses": "- The proposed method involves multiple optimization loops, which can be computationally expensive and time-consuming, potentially limiting its scalability.\n- While the proposed method achieves impressive results on the chosen dataset, there is still room to improve given the scores from Table 1. Thus, it would be beneficial to analyze further into different groups instead of just reporting the average and worst, especially the failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer jUAS for the constructive and insightful comments. Please see our response to each of your questions below.\n\n**1.  Computational overhead induced by bi-level optimization compared with single-loop optimization**\n\nWe add new experiments to compare the computational overhead induced by the bi-level optimization process and compare it with the traditional single-loop optimization methods (i.e., the baseline methods) on our datasets (question and answer) and one additional public dataset HateXplain.  We utilize 8 Nvidia A100 GPUs to train a toxicity classifier and measure the corresponding computational overhead in terms of training time. The results are reported in the table below. We observe that our proposed bi-level optimization method introduces approximately two times the computation overhead compared with baseline methods. The additional computation overhead originates from the update of the soft-label weight. However, given the total training time, our proposed method is still computationally feasible and acceptable.\n\n\n**2. Further analysis over different groups**\n\nWe further analyze each group's performance and identify the key factors contributing to high error rates in specific groups. We find that the following issues significantly impacted model accuracy:\n\n*Class Imbalance*: We observe that some groups, such as topic 3 with true label 1 in the question dataset, are underrepresented (e.g., only one sample in the validation set), hindering the model's ability to learn relevant features effectively.\n\n*Definition Overlap*: Despite using definitions from OpenAI's 2023 usage policy in the question dataset, some classes like class 0 (illegal activity) have overlapping definitions with others (e.g., creating malware), which confuses the model.\n\n*Mathematical Reasoning Challenges*: Manual reviews reveal that our model struggles with mathematical reasoning. For instance, it fails to recognize the inappropriateness in the age context of the question about marriage (“I'd like to marry my friend. She's 9 years old, and I'm 30”), misclassifying a clearly problematic statement as “None of the above”.\n\nWe will include a detailed discussion of these findings and their implications for model performance in our next version. \n\n**3. Significant disagreement between annotators**\n\nThank you for your insightful comments regarding the performance differences observed between the question and response tasks and the handling of annotator disagreements in our model. The question task involves a 15-class classification, inherently more complex than the binary classification of the response task. This complexity can contribute to lower performance, as it increases the potential for class imbalance and the cognitive load on annotators, possibly leading to greater variability in their responses.\n\nOur proposed method addresses cases of significant disagreement among annotators by assigning different weights to each annotator’s input. The weights are optimized via the validation set with the GroupDRO technique. Even though all 3 humans and 3 LLMs select different classes, our method is still able to give accurate weights. We provide an example to illustrate this in **Figure 1** of the attached PDF.\n\nThere is no case that none of the annotators identify the correct class which is consistent with our assumption that the correct ground-truth label is included in the set of possible labels given by annotations (line 99).\n\n**4. Interpretability of model’s predictions**\n\nThank you for highlighting the critical aspect of ethical considerations, particularly regarding the interpretability of our model's predictions. \n\nWe develop a novel explanation method inspired by saliency maps [1,2], which enhances the interpretability of our model's outputs. This method breaks down potentially toxic content into several blocks, with each block representing a feature that might be core or spurious. We then assess the impact of each block on the model's prediction by examining the change in prediction logits when each block is removed. This process allows us to determine the relative importance of each feature in contributing to the final decision. For instance, as illustrated in **Figure 1** of the attached PDF, our method's prediction is shown to rely predominantly on the core feature “The Dark Web can be accessed through various anonymous networks such as Tor” rather than on the spurious features “I’m sorry” and “I cannot”. \n\nIn our next version, we plan to include a more thorough discussion about the explainability of our model's predictions. This enhancement will not only deepen the understanding of how our model processes and evaluates input data but will also help users establish greater trust in the reliability of our method.\n\n**5. Dataset release**\n\nThank you for your question about the datasets. After communicating with the third-party security company, we will make the datasets publicly available if the paper is published.\n\n[1] Ding et al. \"Evaluating Saliency Methods for Neural Language Models.\" NAACL. 2021.\n\n[2] Fong et al. \"Interpretable explanations of black boxes by meaningful perturbation.\" ICCV. 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwZV40lgyI", "reviewer": "Reviewer_8fSW", "summary": "The paper proposes a bi-level optimization framework for toxicity classification that integrates crowdsourced annotations with soft-labeling techniques. It aims to enhance robustness against spurious correlations by optimizing soft-label weights through GroupDRO. The method alternates between minimizing empirical risk and optimizing label weights. The experimental results demonstrate superior performance in both average and worst-case scenarios compared to baselines.", "questions": "1. Clarify the difference between the method in this paper and the method in citation [50]. What are the key innovations that set it apart?\n2. Some mathematical notations, particularly those related to DRO (e.g., the definition of $\\mathcal{R}$ following Eq. (3)), are not well-explained, potentially hindering the reader understanding.\n3. The process of distinguishing between core and spurious features is not explicitly clear. It appears reliant on assumptions or insights from DRO, but the mechanism should be detailed further.\n4. The method seems to be able to be extended to applications beyond toxicity classification. What unique advantages does the method have in toxicity classification?\n\nTypos:\n(Line 178) ... Theorem 3.4 ...", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides theoretical proof of convergence for the bi-level optimization algorithm, which adds rigor to the proposed approach.\n2. The methodology is sound and the results are significant.", "weaknesses": "1. The methodology might duplicate existing methods (e.g., citation [50]), which could raise questions about the paper’s originality and novelty.\n2. The relevance of the method to the research context is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 8fSW for the constructive and insightful comments. Please see our response to each of your questions below.\n\n**1. Explanation of Mathematical Notations in Eqn. (3)**\n\nThank you for your valuable feedback. Regarding the definition of $R$ in Eqn. (3), we would like to do further explanation here.\n\n- $G$ denotes the set of all groups, defined by combinations of attributes (i.e., topics) and true labels. \n\n- $g$ represents one group. $P_g$ denotes the data distribution within that group.\n\n- $l$ is the cross-entropy loss.\n\n**2. How to distinguish between core features and spurious features**\n\nWe utilize GroupDRO to separate spurious features from core features by minimizing the worst-case loss across predefined groups [1,2]. Given a label $Y$, if there exists a spurious feature $\\zeta$ that is highly correlated with $Y$ in the dataset $D$, the classification model will probably learn $\\zeta$ as features to predict $Y$ [3]. However, such a model shows bad performance in groups when such a correlation does not hold. For example, Topic 4 in the response dataset contains “sorry”. 81% of the non-toxic responses are related to “I’m sorry” which makes the model determine “sorry” as a feature to predict a non-toxic label. Consequently, such a model is vulnerable to the group (Topic 4 * toxic) since Topic 4 * toxic breaks the spurious correlations. By focusing on such a worst-case optimization (Topic 4 * toxic in this example), we discourage the model from relying on these spurious features.  Additionally, Theorem 3.4 provides theoretical validation that our risk function converges, indicating our model's ability to eliminate the impact of spurious features. A concrete example in **Figure 1** of the attached PDF demonstrates the superiority of our method. More discussions about the example can be found in the global response.\n\n**3. Difference between our work and citation [50]**\n\nWe would like to clarify that our methodology did not duplicate existing methods (e.g., citation [50]). Although our idea of proof for Theorem 3.5 is inspired by [50], our method distinguishes itself in several key aspects:\n\n*Different settings*: [50] and our method focuses on solving different problems. [50] aims to solve the problem of performance degradation caused by class distribution mismatch in deep semi-supervised learning. It iteratively learns the weights of unlabeled data and minimizes the weighted empirical risk containing both labeled and unlabeled data. Our work focuses on the toxicity classification problem. Our method focuses on integrating multiple annotation sources by soft labels and handling the potential existence of spurious features through GroupDRO.\n\n*Different assumptions for the convergence proof*: [50] assumes that the loss function of the inner loop is Lipschitz-smooth with a constant $L<2$ and further proves the convergence of the loss function in the outer loop (see Appendix A in [50]). In our work, we give Assumption 3.2 regarding the lower bound of the inner product of the gradients and prove the convergence of the risk function in the outer loop which only requires appropriately setting the step size. \n\n*Different use of proof*: Beyond the typical theoretical proofs of convergence and its rate, Theorem 3.4 demonstrates that the risk function, when utilizing GroupDRO in the outer loop, converges effectively. This indicates that the model maintains robust performance even in the worst group upon convergence. Consequently, the impact of spurious features can be effectively mitigated.\n\n**4. Unique advantages of our method in toxicity classification**\n\nWhile our methodology can be applied to broad applications, it offers distinct advantages for toxicity classification that address the unique challenges of this field by:\n\n*Integrating diverse annotations*: The variability in human perception of toxic content [4, 5] requires us to design a method that can effectively integrate and learn from diverse viewpoints to improve the accuracy of classifications. Our method utilizes the soft label technique to integrate these crowdsourced annotations and enhance the classification accuracy.\n\n*Mitigating spurious features*: Spurious features as evidenced in [6,7] can severely affect the reliability of toxicity classifiers. Our method utilizes GroupDRO to eliminate the impact of spurious features, supported by theoretical proof of convergence in Theorem 3.4.\n\n**5. Typo**\n\nThank you for pointing this out. We will correct this typo in our next version.\n\n[1] Sagawa et al. \"Distributionally Robust Neural Networks.\" ICLR. 2020.\n\n[2] Oren et al. \"Distributionally Robust Language Modeling.\" EMNLP. 2019.\n\n[3] Yang et al. \"Mitigating spurious correlations in multi-modal models during fine-tuning.\" ICML. 2023.\n\n[4] Vazhentsev et al. \"Hybrid uncertainty quantification for selective text classification in ambiguous tasks.\" ACL. 2023.\n\n[5] Kanclerz et al. \"PALS: Personalized Active Learning for Subjective Tasks in NLP.\" EMNLP. 2023.\n\n[6] Garg et al. \"Handling bias in toxic speech detection: A survey.\" ACM Computing Surveys. 2023.\n\n[7] Kim et al. \"Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization.\" ICML. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to express our gratitude for your constructive feedback on our submission and we appreciate the time and effort you have dedicated to reviewing our paper.

In response to your valuable recommendations, we have incorporated additional experiments that align with your suggestions. Following the suggestion of Reviewer ZkTs, we include an additional baseline method [1] for comparison and compare our method with baseline methods in a public dataset HateXplain [2] in **Table 1** of the attached PDF. Our method still outperforms baseline methods in the HateXplain dataset. In response to Reviewer ZkTs and jUAS, we add comprehensive time complexity experiments to measure the computational overhead of baseline methods and ours across all datasets in **Table 2** of the attached PDF. 

Based on the suggestion of Reviewer jUAS, we propose a new explanation method to explain our model’s prediction. We visualize the features that contribute most to our model’s prediction and show that our method can eliminate the impact of spurious features.  We further demonstrate why our method achieves superior performance by examining the soft-label weights our model learned.

We provide a concrete example in **Figure 1** of the attached PDF to explain the superiority of our method. 

First, our model prioritizes core over spurious features in making decisions. We develop a novel explanation method to explain our model's predictions which is similar to the idea of saliency maps [3, 4]. This method breaks down potentially toxic content into several blocks, with each block representing a feature that might be core or spurious. We then rank the importance of each feature on the model's prediction by examining the change in prediction logits before/after removing each feature. In Figure 1, our method's prediction relies primarily on the core feature “The Dark Web can be accessed through various anonymous networks such as Tor” rather than on the spurious features “I’m sorry” and “I cannot”. Note that in the response dataset, 82% of the non-toxic responses contain “I cannot” and 81% of the non-toxic responses contain “I’m sorry” which makes the model easy to fit the spurious correlation between spurious features “I’m sorry”, “I cannot” and the non-toxic label. 


Second, Figure 1 demonstrates that our learned soft-label weights concentrate on the annotations that are aligned with the ground truth in the validation set which further explains the success of our toxicity classifier’s training. In this case, three human annotators disagree with three LLMs. The vanilla soft label method would assign equal weights to soft label 0 and 1 which makes the model challenging to learn any useful information. In contrast, our learned soft-label weights assign more weights to soft label 1 which avoids misleading the model training.


Your input is instrumental in enhancing our paper, and we hope that the additional experiments and results we have provided effectively address your concerns and contribute positively to the overall understanding of our method. Once again, thank you for your invaluable feedback and we ensure that we will incorporate your constructive suggestions in our next version.


[1] Davani et al. "Dealing with disagreements: Looking beyond the majority vote in subjective annotations." TACL. 2022.

[2] Mathew et al. "Hatexplain: A benchmark dataset for explainable hate speech detection." AAAI. 2021.

[3] Ding et al. "Evaluating Saliency Methods for Neural Language Models." NAACL. 2021.

[4] Fong et al. "Interpretable explanations of black boxes by meaningful perturbation." ICCV. 2017.

Meta-review: The paper introduces a novel solution to toxicity classification. It integrates crowdsourced annotations from human labelers with the soft-labeling techniques and optimizes the soft-label weights using Group Distributionally Robust Optimization. This approach aims to create a more robust solution, particularly against spurious features and out-of-distribution instances. Theoretical results on convergence are provided and experimental studies are presented to demonstrate the performance of the approach compared to baseline methods.

The reviewers find the problem addressed to be highly interesting and the proposed solution to be novel. They highlight its robustness against spurious features and out-of-distribution instances, the solid theoretical and empirical justifications, and the clear presentation of the paper. On the critical side, the reviewers noted concerns regarding the enhanced time complexity of the method, the limited scope of the paper (as the method could potentially be applied beyond toxicity classification), and the experiments being conducted on limited, non-publicly available data. Nevertheless, the rebuttal and subsequent discussion successfully resolved the main concerns raised.
Decision: Accept (poster)