Collaborative Refining for Learning from Inaccurate Labels
Authors: BIN HAN
Yi-Xuan Sun
Ya-Lin Zhang
Libang Zhang
Haoran Hu
Longfei Li
JUN ZHOU
Guo Ye
HUIMEI HE
Venue: NeurIPS 2024 poster (2024)
Keywords: Inaccurate Labels
Annotator Agreement
Data Refinement
Sample Selection
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: This paper considers the problem of learning from multiple sets of inaccurate labels, which can be easily obtained from low-cost annotators, such as rule-based annotators. Previous works typically concentrate on aggregating information from all the annotators, overlooking the significance of data refinement. This paper presents a collaborative refining approach for learning from inaccurate labels. To refine the data, we introduce the annotator agreement as an instrument, which refers to whether multiple annotators agree or disagree on the labels for a given sample. For samples where some annotators disagree, a comparative strategy is proposed to filter noise. Through theoretical analysis, the connections among multiple sets of labels, the respective models trained on them, and the true labels are uncovered to identify relatively reliable labels. For samples where all annotators agree, an aggregating strategy is designed to mitigate potential noise. Guided by theoretical bounds on loss values, a sample selection criterion is introduced and modified to be more robust against potentially problematic values. Through these two methods, all the samples are refined during training, and these refined samples are used to train a lightweight model simultaneously. Extensive experiments are conducted on benchmark and real-world datasets to demonstrate the superiority of our methods.

Collaborative Refining for Learning from Inaccurate Labels
Bin Han, Yi-Xuan Sun, Ya-Lin Zhang, Libang Zhang, Haoran Hu Longfei Li, Jun Zhou', Guo Ye, Huimei He
Ant Group
(binlin.hb hb, xuan.syx, lyn.zyl, libang. zlb, hhr327996, longyao. 11f, jun. zhoujun, yeguo. yg, huimei. wtw.//aattrouup.coo
Abstract
This paper considers the problem of learning from multiple sets of inaccurate labels, which can easily obtained from low-cost annotators, such as rule-based annotators. Previous works typically concentrate all the annotators, overlooking the significance of data refinement. This paper presents collaborative refining approach for learning from inaccurate labels. To refine the data, we introduce the annotator agreement as an instrument, which refers to whether multiple annotators agree or disagree the labels for given sample. For samples where some annotators disagree, comparative strategy is proposed to filter noise. Through theoretical analysis, the correlations among multiple sets of labels, the respective models trained on them, and the true labels uncovered, that relatively reliable labels can be identified. For samples where all annotators agree, an aggregating strategy is designed to mitigate potential noise. Guided by theoretical bounds on loss values, sample selection criterion is introduced and improved to be more robust against potentially problematic values. Through these two modules, all the samples are refined during training, and these refined samples are used to train a lightweight model simultancously. datasets, which
aggregating information from
Extensive experiments are conducted on benchmark and demonstrate the superiority of the proposed framework.
1 Introduction
Deep learning has recently demonstrated remarkable achievements across wide array of appli- cations (5 6. 15 33 The cornerstone of its success rests on the availability of high-quality datasets. However, in industrial environments, obtaining accurate labels can often be costly and time-consuming. For example, in financial scenarios, accurately assign labels to predict whether a user will default. Currently, many banks and companies utilize rules as low-cost autonomous annotators, especially in the early stages of some financial products. In healthcare scenarios, many rule- algorithms are also employed to identify patients risk of developing certain conditions, allowing for early intervention and potentially preventing more severe complications. Although these annotators can provide labels in time- and cost-efficient manner
challenging for human annotators to
44
45
single low-cost annotator usually produces biased or inaccurate labels in
practice 8
3
24
29 40).
On the other hand, multiple annotators can provide diverse perspectives with different insights and knowledge to mitigate errors and biases of individual annotators [30 (5). Learning from multiple sets of inaccurate labels provided by multiple annotators has garnered widespread attention and
"orresppndiing author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

relative works can be divided into two main categories. One stream of research focuses on advanced aggregation algorithms that infer the true labels before or during the training stage. The simplest aggregation algorithm is Majority Voting [38] which treats the labels equally by voting, while the approach known as Weighted Majority Voting (20] uses a weight vector to model annotators' expertise. Enhanced Bayesian Classifier Combination (EBCC) 22 tends to infer truth by modeling the correlation between annotators.
Others focus on training models under the supervision of all annotations. By viewing the ground-
labels as latent variables, some methods that consider the relationships among multiple sets of labels have been proposed to infer true labels based on the Expectation DnsectationMaximiinan Maximization (EM) algorithm [ 2 43]. Despite the effectiveness of these EM algorithms, they suffer from computational complexity during the training phase. In recent studies, an increasing number of studies focus on end-to-end can directly learn from multiple sets
learning. These end-to-end algorithms 33
27 45 30
of noisy labels and map these noisy labels to part of the model (e.g.. transition matrix), encouraging the model to learn knowledge from all noisy labels collectively
45
Previous works typically concentrate on aggregating information from all the annotators through label aggregation techniques (e.g., voting) or using labels from all annotators in an end-to-end manner (e.g., mapping multiple sets of noisy labels to part of the model). Most related works often neglect the significance of data refinement. Although some methods have recognized the need for data refinement (21 42), they merely adopt the small-loss criterion [[I] to filter noisy samples, and refinement is not their core contribution or their key point. On the contrary, we contend that refining a relatively clean dataset during training is the central point, which can alleviate the demands placed on model design and enhance the model's performance. To refine the dataset, an essential step is to assess the reliability of labels from multiple sets. We consider annotator agreement as an instrument, which refers to whether multiple annotators agree or disagree on the labels for a given sample. In this paper, we leverage the annotator agreement information and propose a novel framework named Collaborative Refining for Learning from inaccurate labels (CRL). For samples where some annotators disagree, labels are ambiguous, and learning from all sets of labels can degrade the model's performance, a comparative strategy is proposed to mitigate noise. For samples where all annotators agree, aggregating strategy is designed to filter out potential noise. The main contributions of this work can be described as follows:
For samples where some annotators disagree, we conduct theoretical analysis to uncover the correlations among multiple sets of labels, the respective models trained on them, and the true labels. Guided by theoretical insights, a method called Label Refining for samples with Disagreements (LRD) is proposed to identify the most reliable label by comparing loss values.
For samples where all annotators agree, we analyze the theoretical bounds on loss values. Based on these bounds, a method called Robust Union Selection (RUS) is proposed, in which we propose loss- based selection criterion to select trustworthy samples and improve it to be more robust against potentially problematic values.
Comprehensive experiments are conducted on benchmark and real-world datasets, which demon- strate the effectiveness of our framework. Moreover, our framework designed to be indepen- dent of any specific model architecture, making it compatible with most existing methods, which is confirmed by further experiments.
2 Preliminaries
This paper concentrates on binary classification problems with multiple sets of inaccurate labels. Let D (z,, J.N.N be dataset consisting of N instances labeled by R annotators. 2; X' C Rd is the feature vector of the i-th instance and ÿ. { JR2 is a R-dimensional vector representing the labels provided by A annotators, V. E {0, 13. Denote y as the unobserved ground-truth label for the i-th instance. y is considered as a latent variable decided by a latent function f* i.e., f*(x)) y/- fe denotes a classifier parameterized by θ, and fe (2:)) = f(x,) denotes the predicted probability after the activation function, i.e., sigmoid. The binary cross-entropy loss function of the pair (æ,, k( and model fe is
(f)( 7 5े log(fo( (xx)) + (1 - ; log(1 (1- fe(x.)). The goal of learning from multiple annotators is to get the optimal classifier so- that satisfies (x. (x.).
(I)

---

For the model architecture, several shared embedding layers are used to jointly extract information, followed by R + submodels [fe.)r=1:
R++1 R submodels ffe are responsible for making predictions based on Rt sets of labels, and the (R + 1)-th submodel feR.1 generates the final prediction. These R + submodels are simultaneously trained, with (/ ) learning from R sets of labels while fe concurrently training from the refined data.
3 Method
In this paper, we present a novel framework called collaborative refining for learning from inaccurate labels (CRL). Within the framework, annotator agreement is utilized to partition the samples into two categories, i.e., samples where annotators disagree or agree. This concept facilitates the development of targeted strategies for each category.
3.1 Collaborative Refining Framework
Based on the annotator agreement, the whole dataset can be partitioned into two parts: Da which contains samples where some annotators disagree, i.e., Aro 11 S (1, (1,..., R). 17 + ;f Da which contains samples where all annotators agree, i.c., Vro, T1 C {I,... Rt), 3/, = D. As mentioned in the introduction, for Da. the labels are ambiguous, and learning from all sets of labels may degrade the model's ' performance. To this end, we propose Label Refining for samples with Disagreements (LRD) to select relatively reliable labels. For Da, where labels are more reliable but not immune to errors. To address this, we present Robust Union Selection (RUS) to select trustworthy samples.
In the framework, through LRD and RUS, R submodels {fe, ).2_ collaboratively refine unreliable dataset Da and D., into relatively reliable dataset D2 and D Dž and D are utilized to train the final submodel feg:1 with binary cross-entropy loss function shown in Eq. ( For prediction, feR:1 is utilized. Note that RUS and LRD operate concurrently and share an identical set of R + 1 submodels. Our framework improves model performance exclusively by refining information, which is independent of any particular backbone, allowing for seamless substitution and flexibility in the model structure. The complete CRL framework is shown in Algorithm
3.2 Label Refining for Samples with Disagreements
When some annotators disagree, learning from all sets of labels may degrade the model's performance. rational idea is to figure which label is more reliable. Through theoretical analysis, we uncover the relationships among multiple sets of labels, the ground-truth label, and the submodels' predictions for given sample, which in turn guide the design of the algorithm.
We follow the widely used class-conditional noise assumption [00 26), i.c., p(j""yy", x) = ((7""y"") Vr E {l,..., R}.. Under this assumption, the noise transition matrix for the r-th an- notator can be formulated as T E 2 where Tü = p(ū" = jjy"" = i) denotes the probability of an i-th class sample flipped into the j-th class for the r-th annotator.
Inspired by Gui et al. [[]], we give the following theorem about the relationships among multiple sets of labels, the ground-truth label, and the submodels' predictions, proof can be found in Appendix A.I Assume neural networks feo Se, are used to minimize the expected loss using labels from two annotators respectively.
Theorem 1.
Let (x, v" v'.""""" J T²) ) be any sample with ground-truth label y and two conflicting labels go, i from two annotators, ie.. jj + ÿ². Assume TO and T satisfy T > 0.5 and TŞ > 0.5, di 10,1). {(fe) (") < E(fe (m), (2) if and only if j
Remark. Theorem I indicates that when the diagonal elements of two noise transition matrices greater than 0.5, if two models are trained with these two sets of labels, for a sample x on which these two annotators disagree, the more reliable label can be selected by comparing the loss values in their respective models.
Then, the above theorem from the scenario of two annotators can be expanded into multiple annotators, proof can be found in Appendix A.2 Assume a series of neural networks (fe, J are used to minimize the expected loss using labels from R annotators respectively.
3

---

Corollary 1. Let (z, ve, ....".''' };-)) ...) be any sample with ground-truth label y and R. conflicting labels { lr= from R annotators, i.e. Ero: "i S (L...,,yy go + gn Assume TV 0.5, vi {0, 13 (x).
and r {L,.., R). if (f(e (x) 5²)
y
Remark. Corollary D indicates that if R models are trained with R sets of labels, when the observed labels of sample are not the same, we can get the most reliable label for this sample by choosing the label which has the smallest loss value.
Corollary 1 encourages to infer the most reliable labels of the samples where some annotators disagree. Based on Theorem and Corollary [ we propose our LRD method to deal with Da. As mentioned in the preliminaries, R single-label submodels are trained. The training loss for these R submodels can be written as:
LACF NR ME w oj log(Je, + - o.) log(1 (2x4))
where w" is weighting factor introduced to address the predictive bias for positive and negative samples which may arise due to sample imbalance. Denote the number of positive labels in r-th set of labels as ni and that of negative labels as ngg common strategy for setting w" is w" "g
Through these submodels, we identify the most reliable labels by comparing their respective loss values over the same sample. Based on Corollary for sample (æ:, (/))))))) E Da. we get the refined label V by
where the most reliable index & for sample æ; is acquired through:
k argmin (z:), 00)
(4)
J In this way, for any instance (æ,, { Ir=1), 1) we can refine it into (æi, M. Then we can construct the refined dataset Dū and utilize it to train the submodel fēxı1 with the binary cross-entropy loss. This submodel can be deployed and utilized for final prediction. In practice, these refined labels are held constant after several training epochs mitigate the over-fitting issue. The procedure of LRD is Note that LRD and RUS operate concurrently, with LRD focusing on Da
included in Algorithm [ and RUS on D..
3.3 Robust Union Selection
For samples where all annotators agree, the labels are generally presumed to be reliable, however, they are not immune errors. For instance, if the annotators have similar defects, although the annotators agree, the labels may still be inaccurate. To this end, guided by theoretical bounds on loss values, we introduce loss-based selection criterion and modify it to be more robust against potentially problematic values.
Inspired by the small-loss selection criterion mentioned in many works under single noise label scenario 111 13 231, small-loss data tends to be more clean. However, relying solely on the prediction provided by of the submodels is unstable. Once the selection is wrong, the inferiority of accumulated errors will arise [41]. Since we have R predictions for each sample with R sets of labels from our model, naturally, average loss values can be utilized, which can be denoted as:
R Σ ((fe
1
(5)
which is more stable and serves an estimation of the mean μ. Thanks to the R sets of labels provided by R annotators, these submodels can be diverse, which can ease the inferiority of accumulated errors [13].
However, simply taking the average can also lose some correctly labeled samples. This issue originates from two aspects: annotator defects and hard samples. Firstly, different annotators may differ quality or be suitable for different kinds of samples, which leads to the situation where some submodels may struggle with prediction and incur a larger loss for some correctly labeled samples.

---

Secondly, some correctly labeled samples may be difficult to learn, as result, submodels may give unstable predictions them. Note that this issue is avoided in LRD since we use the minimum loss value of the same sample across submodels there. To address this issue, a robust criterion for sample selection is defined.
We begin by defining non-decreasing smooth function & for loss values, which can be written as:
<(2) log(1 - 2
where z is positive variable, this function is inspired by the Taylor expansion of the exponential function. Loss values can be easily substituted into this equation to serve an soft estimation of the underlying mean μ.. Eq. o can reduce the side effect of extremum, which can mitigate annotator defects and hard samples issues. Based on Eq. 66) the robust average loss ja can be written as:
n
Σ (z))
To enhance the robustness of p. the predictions from historical models are introduced. Denote the submodel's parameters O in the I-th epoch as Or. the chosen set of epochs as T, then Eq. 7 can be rewritten as follows:
R
Σ
Σ (llsee (c::), D)),
(8)
R|T
LET
where [T'] represents the total number of the selected epochs, for instance, we can choose the latest five epochs or some fixed epochs as T during the training process.
Based on the soft estimation n. of the underlying mean μ., we further introduce the lower bound of the underlying mean μ to give correctly labeled samples which discarded due annotator defects or because they are hard samples a chance to be selected. Inspired by Xia et al. [39], we give the following theorem about the lower bound. The proof is similar to Theorem in [39].
Theorem 2. Let { be an observation set with mean μ and variance o². By utilize a non-
decreasing function 6(2) log(1 + + ))), have
0²(n + log(2n)
1
o(zz)
n
n
with probability at least
Remark. This theorem defines the lower bound the mean of an observation set. The first term in the lower bound is robust average, which can reduce the side effects of extremum. The second term related to the number of observed values and the variance.
Based on Theorem
2 our selection criterion e for a; is defined as:
0² (1 + a? log(2n)
G p
n²
(10)
n o2
where n is the number of observed loss values for I;, i.e., n R as shown in Eq. 8 variance of observed loss values for x;, Since we don't know the distribution of the loss values for æ; in submodels, o7 serves an approximation of the latent true variance.
o² is the
Generally speaking, the second term in Eq. (10) gives samples with relatively large variance chance to be selected for training. If we use the normal average loss, as shown in Fq. (5). these samples may be discarded due to annotator defects or because they are hard samples. Thanks to Eq. (0). they can be included in the selection; if these samples consistently yield large loss values throughout the training process and across the majority of submodels, they will be discarded in subsequent epochs.
By using the selection criterion in Eq. 10), we can sort the samples in D2 and select the smallest p proportion to form D", which can be utilized to train fen.1- In practice, positive and negative samples are selected separately. The process of RUS is included in Algorithm [
Note that RUS and LRD operate concurrently, and they share an identical set of R + 1 submodels which are trained simultaneously. RUS, the R submodels [fe,) collaboratively refine Da, while in LRD, these submodels are used to refine Da. ferm learns from the refined samples.
5

---

Experiment
4.1 Dataset
Benchmark datasets. All the methods are evaluated on benchmark datasets with two kinds of noise, including five NLP datasets named Agnews, 20News, IMDb, Yelp and Amazon, four tabular datasets named Diabetes, Backdoor, Campaign and Waveform, and four image datasets named Celeba, SVHN, FaSHIOn-MNIST (denote F-MNIST) and CIFAR-10 10. Note that these datasets only have ground-truth labels, we simulate three annotators per dataset with label quality 0.3 through the following two methods:
srttaceedepeedentt noise. Inspired by Zhao et al. [45, k proportion of each dataset is used to train three sets of diverse tree- based models to act as rule-based annotators, i.e., Decision Tree, RandomForest, LightGBM. Their predictions are used as multiple sets of labels.
Class-dependent noise. The labels of the positive samples are randomly preserved at proportions of k, k + 0.1, and k + 0.2 respectively, while interchanging the labels for the remaining positive samples with those of negative samples, thus establishing three sets of labels. Note that these three sets of labels are generated independently.
Real-world datasets. Experiments also conducted on two real-world datasets: CIFAR-ION and Sentiment. Both datasets were published on Amazon Mechanical Turk for annotation.
Details of these datasets and labels can be found in Appendix B
4.2 Compared Methods and Implementation Details
We compare our framework with various methods: (1) Single, in which a three-layer MLP trained directly with one set of labels; (2) NN-Mjv (38) in which a three-layer MLP is trained using the labels aggregated through majority voting; (3) HE A and HE _ [37),
which train several individual models for each set of inaccurate labels and aggregate their outputs for predictions by averaging and maximizing; (4) CL (26]. which trains an end-to-end model with parametric source-specific transition matrices; (5) DN 110, which exploits information from multiple sets of labels with different softmax output layers; (6) Label aggregation methods called Enhanced Bayesian Classifier Combination (EBCC) and Independent Bayesian Classifier Combination (IBCC) [22] three-layer MLPs are trained using the labels inferred by EBCC or IBCC; (7) Weakly supervised end-to-ent learner WeaSEL [27]: (8) CoNAL 3, which assumes that the annotation noise attributed two sources (common noise and individual noise), and combines these two noise by a Bernoulli random variable; (9) SLF 7 which proposes to make the weight vectors and the confusion matrices data-dependent, and comes up with two regularization methods for the confusion matrix to guide the training process; (10) ADMoE [45), which leverages the Mixture of Experts (MoE) architecture to encourage specialized learning from multiple noisy sources. Note that for fair comparison, we do not use the noisy-label aware gating in ADMoE which inputs labels during both training and testing.
Most of the compared methods are adopted from their respective codebases. The hyperparameters set according to the recommendations in their papers. The only modification made to ensure fair comparison is to use three-layer MLP with hidden dimension of 128 as the backbone.
For our method, as mentioned in the preliminaries, we use three-layer MIP of hidden dimension 128 to act as shared layers, and R - three-layer MLPs of hidden dimension 128 to serve as R + 1 submodels. The (R 1) submodel fent: is utilized for final prediction. There is no hyperparameter in LRD. For RUS, we set the proportion of selected samples p = 0.8, and take the 5th epoch and the latest epoch during training the selected epochs in Eq. (). In practice, RRD-generated labels are held constant after 5 training epochs to mitigate the over-fitting issue.
For all of the methods, experiments are conducted with 0.001 learning rate, 100 training epochs, and 256 batch size on MLP with hidden dimension 128 for a fair comparison. For benchmark datasets, 70% of each dataset is utilized for training, 5% for validation, and 25% for testing. For world datasets, we use the original training and testing datasets. We report the average AUC of the last 10 epochs results.

---

4.3
Main Results
Main results. Comprehensive experiments are conducted on various datasets, including NLP, image, and tabular data, and we compare our framework against a variety of methods. The main results of our framework and all the compared methods class-dependent noise and instance-dependee summarized in Table T. As shown, our framework outperforms all the compared methods in most of the datasets, especially in the class-dependent noise setting. Our framework does not incorporate any special designs at the architectural level, it primarily acquires higher-quality data through collaborative refinement, followed by learning via MLP, which demonstrates the effectiveness of our framework.
noise are
Table 1: Main results with AUC as the evaluation metric. The best results are in bold.
Metbods
Noise
Dataset
Single
NN-Mjv
HE_A A
HE_M
M
CL
DN
NN-EBCC
NN-IBCC
WeaSEL
SLF
CoNAL
ADMoE
Ours
AgNews
0.660
0.634
0.723
0.724
0.626
0.757
0.703
0.746
0.833
0.791
0.769
0.762
0.855
20News
0.746
0.684
0.755
0.756
0.749
0.729
0.796
0.779
0.768
0.778
0.765
IMDb
0.666
0.602
0.667
0.670
0.614
0.689
0.673
0.699
0.709
0.700
0.702
0.707
0.766
Yelp
0.725
0.713
0.783
0.785
0.779
0.779
0.782
0.786
0.805
0.807
0.769
0.799
0.867
Amazon
0.586
0,567
0.681
0.687
0,631
0.661
0.635
0.657
0.718
0.679
0.672
0.652
0,775
Diabetes
0.648
0.610
0.576
0.592
0.633
0.686
0.657
0.663
0.680
0.577
0.696
0.646
0.728
Class
Backdour
0.530
0.535
0.681
0.687
0.651
0.765
0.668
0.771
0.640
0.816
0.716
0.814
0.937
Campaign
0.561
0.558
0.628
0.636
0.574
0.663
0.619
0.632
0.629
0.694
0.697
0.680
0.783
Waveform
0.772
0.660
0.663
0.792
0.770
0.788
0.807
0.818
0.823
Celeha
0.738
0.710
0.723
0.725
0.784
0.849
0.758
0.768
0.782
0.851
0.859
0.824
0.891
SVHN
0.637
0.639
0.671
0.671
0.692
0.701
0.676
0.679
0.671
0.730
0.726
0.718
0.761
F-MNIST
0.607
0.590
0.684
0.691
0,682
0.667
0,664
0.631
0.678
0.723
0.705
0.737
0,776
CIFAR-10
0.541
0.573
0.574
0.576
0.596
0.570
0.590
0.590
0.587
0.587
0.634
0.625
0.655
AgNews
0.842
0,819
0,830
0.829
0.848
0.867
0.817
0.786
0.836
0.768
0.882
0.842
0.916
20News
0.795
0.855
0.847
0.841
0.823
0.872
0.855
0.846
0.830
0.859
0.828
0,833
0.864
IMDb
0.668
0.729
0.710
0.708
0,722
0.739
0.635
0.621
0.644
0.686
0.717
0.713
0,749
Yelp
0.773
0.782
0,812
0.809
0.790
0.870
0.784
0.778
0.812
0,838
0.832
0.826
0.901
Amazon
0.745
0.690
0.774
0.772
0.777
0.767
0.696
0.673
0.715
0.660
0.790
0.772
0.804
Diahetes
0.747
0.731
0.749
0.748
0.719
0.778
0.728
0.722
0.747
0.772
0.767
0.765
0.818
Backdour
0.646
0.610
0.613
0.593
0.616
0,526
0,581
0.758
Instance
Campaign
0.776
0.726
0.892
0.890
0,811
0.887
0.740
0.733
0,737
0.761
0.883
0.707
0.914
Waveform
0.865
0.834
0.923
0.921
0,935
0,949
0.853
0.837
0.873
0.910
0.959
0.937
0.964
Celeba
0.854
0,918
0.885
0.883
0,909
0.825
0.920
0.922
0.936
0.933
0.914
0.916
0,944
SVHN
0.702
0.708
0.774
0.772
0,813
0,806
0.724
0.721
0.779
0.775
0.803
0.800
0.824
F.MNIST
0.814
0.801
0.888
0.887
0,856
0,840
0.792
0.799
0.819
0.801
0.870
0.888
0.912
CIFAR-10
0.730
0,702
0.778
0.777
0.772
0.764
0.696
0.688
0.747
0.632
0.777
0,799
0.790
Performance on real-world scenarios. Experiments are conducted on these two real-world noisy datasets, with both our methods and the compared methods. As summarized in Table 2 our method surpasses the compared methods on the real-world noisy datasets, these additional results further substantiate the reliability and applicability of our approach in handling real-world noisy labels.
Results under different label quality. Further experiments are conducted under different label quality (k- = 0.1, 0.15, 0.2, 0.25, 0.3) on AgNews, IMDb, Yelp, Diabetes, Celeba and F-MINST with dass-deeendent noise where NLP, image and tabular data are all involved. The results are shown in Figure 1 For clarity, we only include the four competitive compared methods and the Single baseline in the figure. With the increase label quality, all methods achieve better performance. Note that our framework consistently outperforms the compared methods in most of the experiments.
4.4 Discussion
Ablation study. Ablation experiments are conducted on datasets with class-dependent noise where NLP, image, and tabular data are all involved. The results are shown in Table 3 In the ablation experiments, NN-Mjy acts as the baseline. Labels are aggregated for D& / D., by majority voting instead of LRD/ / RUS, without changing any other parts of the framework. We also compare the RUS method with the naive selection method (denoted as N-RUS) by replacing our selection criterion in Eq. (10) with the naive one in Eq. (5). As shown in Table 3 with the assistance of LRD and RUS, an overall improvement of 0.171 171im in AUC is achieved over the baseline. The results indicate that while the naive selection does work, it's not as effective as RUS. Since LRD and RUS are designed for different parts of the dataset respectively, they can be effectively combined to achieve the best results.
7

---

Table 2: Real-world results with AUC the evaluation metric. The best results are in bold.
Methods
Datasel
Single
NN-Mjv
HE_A
HE M
CL
DN
NN EBCC
NN IBCC
WeaSEL
SLF
CONAL
ADMoE
Ours
Sentiment
0,712
0.727
0,744
0,730
0.724
0.732
0.728 0.850
0.736 0.849
0.730
0.686
0.741
0.722
0.753
CIFAR-I01 I0N
0.791
0.853
0.788
0.786
0.788
0.807
0.851
0.821
0.816
0.761
0.866
Single
SLF
DN
CoNAL
AD
Ours
Agnews
IMDb
Yelp
0.60
AUC
0.72
0.56
0.64
0.56
0.4
0.8
labol quality Diabates
labol quality Caleba
labal quality
F-MNIST
0.72
0.06
AJIC
0.72
AUC
0.04
0.50
0.48
0.1
0.15
0.2
025
0.95
0.2
0.25
0.3
label quality
quality
quality
Figure 1: AUC comparison under different label quality k.
Through these experiments, we demonstrate the effectiveness of LRD and RUS. These results also indicate that adding LRD to the baseline improves the performance more noticeably than when adding RUS. is reasonable because LRD is designed to correct labels in Da. while RUS is designed to select samples in Da- As is illustrated before, the samples in Da are more ambiguous and the samples in Da are more likely to be correctly labeled, thus greater improvement can be achieved by effectively dealing with Da-
Quality analysis of the refined data. Further experiments are conducted to explore how LRD and RUS work. Since the LRD method is designed to aggregate multiple sets of labels into a reliable one, we conduct experiments to show the label quality produced by LRD. We train our model for 5 epochs and collect the labels aggregated by the LRD algorithm on Da. Then we compare the quality of these labels with the three sets of original labels and the labels obtained through voting. The experiments are conducted on thirteen datasets mentioned above with class-dependent noise. The results are shown in Figure 2 the average AUC of these labels taken as the metric. LRD can consistently provide higher-quality labels. Naturally, our model trained with these labels yields better results than with the original labels.
Since RUS is designed to select samples, experiments are conducted to show the quality of the selected samples. The model is trained for epochs then the selected samples are fetched. RUS compared with the following two methods: (1) selecting the same proportion of samples randomly; (2) replacing the criterion in RUS with the mean of the loss values in Eq. (5) to seleet the same proportion of samples. The proportion of selection p is set as 0.8. Since the total number of ground-truth positive samples stays the same, we compare the number of true positive samples selected by the three methods mentioned above. These experiments are repeated for 100 times and the average improvements over random selection are presented in Figure] 3 As shown in Figure 3 on the four datasets including image, NLP, and tabular data, proposed method has better performance in selecting samples. Naturally, our model trained with these high-quality samples can get better results.
Further verification of LRD-generated labels. The quality of LRD-ggenerated labels is shown in the foregoing discussion in Figure [ whereby we demonstrate the effectiveness of L.D after training models for five epochs. We further examine LRD's performance in the early training stages. We train
8

---

Table 3: Ablation results with AUC as the evaluation metric.
Datasets
Methods
Avg.
AgNews
IMDb
Yelp
Diabetes
Celeba
F-MINST
Baseline
0.634
0,602
0.713
0.610
0.710
0.590
0.643
+RUS
0.786
0.708
0.846
0.613
0.838
0.703
0.749
+LRD
0.832
0,733
0.859
0.668
0.840
0.735
0.778
LLD and N-RUS
0.838
0.746
0.863
0.692
0.862
0.774
0.796
+ LLD and RUS
0.855
0.766
0.867
0.728
0.891
0.776
0.814
0.8
Mathed Annowwoort
Random
Mean
Oura
0.7
Annotatora M/y
Average AUG
0.6
0.4
188
MD6
Yolp
Diabates
F-MNIST
AI
NLP
Image
Tabular
Dataset
Dataset Type
Figure Average improvemen((%) in the quan- tity of true positive samples selected over ran- dom baseline.
Figure 2: The average AUC on Da subsets over different types of dataset.
model datasets with class class-dependent noise for 100 / 500 steps and examine the qualities of LRD-generated labels on Da. As shown in Table 4 in the early stages of LRD learning (e.g.. 100 steps), the LRD-generated labels already outperform the original labels. With ongoing training, label quality rises (e.g., 500 steps). When LRD-generated labels are frozen at step 100, the final average AUC on these four datasets 0.835, which quite close the average AUC of 0.845 obtained by freezing LRD at step 500. These results indicate that we have considerable flexibility in choosing when fix the refined labels. the main results, the LRD-ggneraan labels are frozen after training for five epochs.
Cooperating with other algorithms. As mentioned before, framework tends collaboratively refine the dataset into a higher-quality one, which is independent of the model architecture. We simply utilize a MLP learn from the refined dataset, which can be replaced by other algorithms. We train our model for epochs, extract the refined dataset, and complement it with the original three sets of labels to form new dataset. Then this new dataset is utilized train SLF, CoNAL, and ADMoE in the same manner as in the main experiments. As shown in Table 5 when cooperating with our framework, ADMoE, SLF, and CoNAL achieve improvement over their original versions, which demonstrates that our framework can cooperate well with the existing methods.
5 Conclusion and Future Work
In this paper, we present a framework called Collaborative Refining for Learning from inaccurate labels (CRL), which focuses on learning a model from multiple sets of inaccurate labels. In our framework, we utilize the annotator agreement to assess the reliability of labels from multiple sets with disparities and split the dataset into two parts: where some annotators disagree and where all annotators agree. For samples where some annotators disagree, Label Refining for samples with Disagreements (LRD) proposed select relatively reliable labels. Based on theoretical analysis of the relationships among multiple label sets, ground- ground-truth truth labels, and model predictions, LRD identifies most reliable label by comparing the loss values. For samples where all annotators agree, Robust Union Selection (RUS) is proposed to select trustworthy samples to form a higher- quality dataset. Guided by theoretical bounds on loss values, RUS introduces loss-based selection criterion and improves it to be more robust against potentially problematic values. Mcanwhile, the

---

Table 4: Label qualities on Da and the training results. AUC is the evaluation metric.
Label sources
AgNews
IMDb
Yelp
Celeba
Avrage
Best Annotator
0.647
0.630
0.671
0.656
0.651
Voting
0.340
0.348
0.418
0.366
0.368
LRD-1 100stepsss
0.775
0.665
0.870
0.736
0.762
LRD- L.DD-50ssteps
0.858
0.706
0.880
0.831
0.819
Table 5: Cooperation results with AUC as the evaluation metric.
Dataset
Methods
Average
AgNews
IMDb
Yelp
Diabetes
Celeba
F-MINST
ADMoR
0.762
0.707
0.799
0.646
0.824
0.737
0.746
+Ours
0.803
0.732
0.804
0.692
0.840
0.762
0.772
Improve(S6).
+5.47
+3.48
+0.70
+7.10
+1.99
+3.28
+3.54
SLF
0.791
0,700
0.807
0.577
0.851
0.723
0.741
+Ours
0.806
0.711
0.834
0.710
0.900
0.769
0.788
Improve(%%).
+1.91
+1,55
+3.37
+23.11
+5.85
+6.41
+6,35
CoNAL
0.769
0.702
0.769
0.696
0.859
0.705
0.750
+Ours 0.776
0.719
0.782
0.723
0.884
0.737
0.770
Improve(56).
+0.92
+2.40
+1.74
+3.86
+2.82
+4.64
+2.69
refined datasets are used to train a lightweight model that can be deployed and utilized for final prediction. Extensive experiments conducted to demonstrate the effectiveness of the framework. The framework designed to be independent of any specific model architecture, making it compatible with most existing methods, which is confirmed by further experiments.
This work currently focuses on the binary classification task with the existence of inaccurate labels, which is fundamental and commonly encountered in practice. This work also lays the foundation for subsequent research of multi-class classification tasks with inaccurate labels. However, in the multi-class scenario, the proposed method needs to be further adapted. For instance, issue arises with samples where annotators disagree, particularly when all the given labels for a particular sample incorrect in the multi-class setting. LRD tends to seleet one of these incorrect labels as the inferred label, negatively impacting the model's performance. In binary classification problems, this situation naturally avoided because when annotators disagree, solution could be to develop a more refined measure based on annotator agreement, which could be utilized to decide whether to discard these samples, using LRD or RUS.
of the labels must be correct. A possible
In the current approach, the samples are segmented based on the presence or absence of label disagreement, and further processes are performed on the correlated segments. We believe that the sample segmentation strategy can further improved. To achieve this, we can introduce new metric: the consistency rate, defined as the proportion of agreement among annotators for the same sample. By establishing a threshold for this consistency rate, we can effectively partition the dataset. Specifically, for samples with consistency rate exceeding the threshold, RUS is applied, whereas for samples with a consistency rate below the threshold, LRD is utilized. The determination threshold is related to the label qualities of the dataset and the number of label sets available, which suitable for practical applications. Since these ideas are not yet mature enough, we did not include these parts in the current manuscript. We leave these issues for future research.
this
Acknowledgments
This work is supported by Ant Group. We thank the anonymous referees for their constructive suggestions and thoughtful comments, which helped to improve the quality of the paper.
10

---

References
[1] Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirci, and Nassir Navab. 2016. Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Transactions on Medical Imaging (2016), 1313-1321.
[2] Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. 2019. Max-mig: an information theoretic approach for joint learning from crowds. arXiv preprint arXiv: 1ets: 13436 (2019).
[3] Zhendong Chu, Jing Ma, and Hongning Wang. 2021. Learning from crowds by modeling common confusions. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'21). 5832-5840.
[4] Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics) (1979), 20-28.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 09)). 248-255.
[6] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171-4186.
[7] Zhengqi Gao, Fan-Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Marc Engeler, Antonio Burazer, Linda Wildling, Luca Daniel, and Duane Boning. 2022. Learning from multiple annotator noisy labels via sample-wi samele-wise label fusion. In Proceedings of the European Conference on Computer Vision (ECCV'22). 407-422.
[8] Aritra Ghosh, Himanshu Kumar, and P. Sastry. 2022. Robust Loss Functions under La- bel Noise for Deep Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'22), 22). 1919-1925.
[9] Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. 2016. Deep Learning. MIT Press.
[10] Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. 2018. Who said what: Modeling individual labelers improves classification. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI' 18). 1108-3118.
[11) Xian-Jin Gui, Wei Wang, and Zhang- Hao Tian. 2021. Towards understanding deep learning from noisy labels with small-loss criterion. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI211.. 2469-2475.
[12] Bo Han, Jiangehao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. 2018 Masking: A new perspective of noisy supervision. In Advances in Neural Information Processing Systems (NIPS' 18). 5841-5851.
[13] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, IvorW. Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels. In Advances in Neural Information Processing Systems (8)) 6536-8546.
[14] Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. 2022. Adbench: Anomaly detection benchmark. In Advances in Neural Information Processing Systems (NeurIPS'22) 32142-32159.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16). 770-778.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR' 16). 770-778.
11

---

[17] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. 2018. Using trusted data train deep networks on labels corrupted by severe noise. In Advances in Neural Information Processing Systems (NIPS' 18). 10477-10486.
[18] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2018. MentorNet: Learning Dria-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. In Proceedings of the 35th International Conference Machine Learning (ICML' 18). 2309-2318.
[19] Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. 2019. Robust inference via generative classifiers for handling noisy labels. In Proceedings of the International Conference on Machine Learning (ICML'19) 3763-3772
[20] Qi Li, Yaliang Li, Jing Gao, Bo Zhao, Wei Fan, and Jiawei Han. 2014. Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation. Proceedings of the 2014 ACM SIGMOD International Conference on Management of data (SIGMOD' 14). 1187-1198.
[21] Shikun Li, Shiming Ge, Yingying Hua, Chunhui Zhang, Hao Wen, Tengfei Liu, and Weiqiang Wang. 2020. Coupled-view deep classifier learning from multiple noisy annotators. In Proceed- ings of the 34th AAAI Conference on Artificial Intelligence (AAAI 4667-4674.
[22] Yuan Li, Benjamin Rubinstein, and Trevor Cohn. 2019. Exploiting worker correlation for label aggregation in crowdsourcing. In Proceedings of the 36th International Conference on Machine Learning (ICML (9).. 3886-3895.
[23] Eran Malach and Shai Shalev Shalev-Shwartz. 2017. Decoupling "when to update" from "how to update". In Advances in Neural Information Processing Systems (7), 960-970.
[24] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. 2017. Making deep neural networks robust to label noise: a loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'17) 17). 2233-2241.
[25] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. 2020. Identifying mislabeled data using the area under the margin ranking. In Advances in Neural Information Processing Systems (NeurIPS 20). 17044-17056.
[26] Filipe Rodrigues and Francisco C Pereira. 2018. Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'18). 1611-1618.
[27] Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski. 2021. End-to-end weak supervision. In Advances in Neural Information Processing Systems (NeurIPS 21). 1845-1857.
[28] Jun Shu, Qian Zhao, Zongben Xu, and Deyu Meng. 2020. Meta transition adaptation for robust deep learning with noisy labels. CoRR (2020).
[29] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2022. Learning From Noisy Labels With Deep Neural Networks: A Survey. IEEE Transactions on Neural Networks and Learning Systems (2022), 1-19.
[30] Yi-Xuan Sun, Ya-Lin Zhang, Bin Han, Longfei Li, and Jun Zhou. 2024. Self-cognitive Denoising in the Presence of Multiple Noisy Label Sources. In Forty-first International Conference Machine Learning (ICML 24).
[31] Nianshang Tian, Ming Wu, Jiqiong Jiang. and Jing Zhang. 2022. Learning from crowds with mutual correction-basee based co-training. In Proceedings of the International Conference on Knowledge Graph (ICKG 22). 257-264.
[32] Tian Tian and Jun Zhu. 2015. Max-margin majority voting for learning from crowds. In Advances in Neural Information Processing Systems (NIPS' 15). 1621-1629.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems (NIPS' 177), 5998-6008
12

---

[34] Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. 2014. Community-bace based bayesian aggregation models for crowdsoureing. In Proceedings of the International Conference on World Wide Web (WWW /4). 155-164.
[35] Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Sau-Tao Xia. 2018. Iterative learning with open-set noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR' 18). 8688 86966.
[36] Hongxin Wei, Renchunzi Xie, Lei Feng, Bo Han, and Bo An. 2022. Deep learning from multiple noisy annotators as a union. IEEE Trunsactions on Neural Networks and Learning Dystems (2022).
[37] Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. 2020. Hyperparameter Ensembles for Robustness and Uncertainty Quantification. In Advances in Neural Information Processing Systems (NeurlPS'20. 20)).
[38] Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems (NIPS 2035-2043.
[39] Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. 2021. Sample Selection with Uncertainty of Losses for Learning with Noisy Labels. In Proceedings of the International Conference on Learning Representations (ICLR'21).
[40] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. 2015. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'IS) 15). 2691-2699.
[41] Xingrui Yu, Bo Han, Jiangchao Yao, Gang NIu, Ivor Tsang, and Masashi Sugiyama. 2019. How does disagreement help generalization against label corruption?. Proceedings of the 36th International Conference on Machine Learning (ICMI. 7164-7173.
[42] Hansong Zhang, Shikun Li, Dan Zeng, Chenggang Yan, and Shiming Ge. 2024. Coupled confusion correction: Learning from crowds with sparse annotations. In Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAl'244 (6732-16740
[43] Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael Jordan. 2014. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. In Advances in Neural Information Processing Systems (NIPS' 14). 1260-1268.
[44] Ya-Lin Zhang, Yi-Xuan Sun, Fangfang Fan, Meng Li, Yeyu Zhao, Wei Wang, Longfei Li, Jun Zhou, and Jinghua Feng. 2023. A Framework for Detecting Frauds from Extremely Few Labels. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining (WSDM'22 23). 1124-1127.
[45] Yue Zhao, Guoqing Zheng, Subhabrata Mukherjee, Robert McCann, and Ahmed Awadallah. 2023, ADMOB: Anomaly Detection with Mixture-of-Experts from Noisy Labels. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI'23). 4937-4945.
13

---

Appendix
In the Appendix, first give the proofs of Theorem Theorem 2, and Corollary 1 in Sections A. The details of datasets and labels are provided in Section B, and the implementation details of the algorithm are provided in Section C. The related works are summarized in Section D.
A Proof
A.1
Theorem
Proof. Let d [do. di] denote the one-hot version of y. i.c., d, = and d; = 0, J. For binary cross-entropy loss function ((fe(()))) 0) Ek=o de we consider the expected loss on noisy data:
(( = (m
Σ log(d,
=- =- | JeEX ==0
Σ d log(Ei(<)) p(=e
plae, v = j)dæ
1=0
(1I)
=- =-f JØEX
d.pl j|æo log(0((z))) p(z)dx
j=0
Bid, |a)] log()(
(() plardd..
xEX
Y=0
Therefore, minimizing the expected loss equals to minimizing - -' Eld, log().(z)) for each E X. For cross-entropy loss, there are constraints Ei (x( and 0 s p.(x) 1, e
{0, 1). So it can be formalized as the following optimization problem:
minimize
Eld [t] log(E; (x))
(12)
8.1. pa( = 1,0 1,0 (x) < 1i E (0, 1}
1=0
Using Lagrange multiplier method, we can derive that - - E (d x (og(p((=)) is minimized
when (xx) E[d; [=), Vi (0, 1}. Because Bid, [x] -o flp( j|==) p(ū" i|ız).
we have (xx = plū = i|æ)). Then we can obtain
(x(x) = p(ū = =| = Σ
p(ū = i, y ;,y=
1=0
Σ
ply = jlæ)p(ū ily x)
(13)
Σ ply = f1=) i1y =)
f=0
p(ū ily v²)
Tyr
where the fourth equation is due to the class class-ooniitional noise assumption and the fifth equation is due to that each has only true label f (x:) Therefore, the loss value of sample (xx ú) is
{(fer (x). 7) log(dg(x)) (T)
(14)
14

---

When facing with two conflicting observed labels j". 8' and two neural network feg.fe; the losses can be written as
P(Je; (x)),") = log R(for (a:), 0²) log (Ty y" go)
(15)
y
Then from the relationship between these two losses E(fe; (z)), 50) < (z),
we can get
- -log(T < log (
(16)
which means
T g0 > T grgi.
(17)
Because 7 T satisfy TA 0.5, T3
0.5, Vi {0, 1}. we can get T3f 0.5 and T 0.5.
Vi E {0,1), + j. Since 8 + "7 T 3o y'g² equals to T go T y" 37.
-- only when
T 4/" 17a
To y"
> 0.5 Ty-(1 (1- T
which satisfy Eq. 17 if y
y' go
T
y (1-yy) < 0.5 < Ty. 4" Ty 5%
which is in conflict with Eq. 17 Theorem is proved.
A.2 Corollary I
Proof. Let To, E (1,.., RJ. define er, e as
r = min((((ee(())))))) 8' = 8"" in.
( (()
can be written as Then {(for(x). min({{(Je; (z)), "0"00))) )0)
((fer / min(e" (rt).
(18)
If 1y = 779 from Theorem and Eq. 14) we can get
fo -l0g(0.5) tr
then {Je; (m). goo which means 8² 8 = y² Similarly, if y = 8², we can get 1/th = 8 y² Corollary is proved.
B Details of Datasets
As mentioned in Section 4.1 we use thirteen benchmark datasets including NLP, image and tabular datasets, and two real-world datasets.
Benchmark datasets. The information of benchmark datasets are summarized in Table Most of them come from [14).
use ResNet18[[6] which is pretrained the ImageNet to extract embedding after the last average pooling layer. We utilize the extracted 512-dimensional embeddings features. We define one of the multi-classes as negative and downsample the remaining classes to 5% of the total instances as positive.
For image datasets, following 14
For NLP datasets, we use BERT(6 pretrained on the BookCorpus and English Wikipedia to extract dosediienniiaall embeddings as features. As for labels, we define them in the following way: (1) Amazon and Imdb; regard the original negative class the anomaly class. (2) 20News and
15

---

Table 6: Data description of the thirteen benchmark datasets used in this paper.
Description
Dataset
# Samples
# Features
# Positives
Type
AgNews
10000
768
500
NLP
20News
3090
768
154
NLP
IMDb
10000
768
500
NLP
Yelp
10000
768
500
NLP
Amazon
10000
768
500
NLP
Diabetes
10927
21
676
Tabular
Backdoor
9555
196
224
Tabular
Campaign
9785
62
586
Tabular
Waveform
3443
21
100
Tabular
Celeba
25906
39
1118
Image
SVHN
5208
512
260
Image
F-MNIST
6315
512
315
Image
CIFAR-10
5263
512
263
Image
AgNews, set one of the classes normal and downsample the remaining classes to 5% of the total instances as anomalies. (3) Yelp: we regard the reviews and stars as the positive class, and the reviews of and stars as the negative class.
For tabular datasets, most of them comes from [4] we directly adopt these datasets. Diabetes dataset is sampled from a dataset on Kaggle', we increased the difficulty of learning model by randomly sampling positive and negative examples at different ratios.
Real-world datasets. The details of real-world datasets are as follows:
CIFAR-ION: is an image classification dataset, which consists of 50000 samples for training and 10000 for testing. We regard the 'Airplane' positive class and others negative class. Following [4], we use ResNet18 [6] which pretrained on the ImageNet(5] to extract embed- ding after the last average pooling layer. We utilize the extracted 51 512-dimensional embeddings as features.
Sentiment: It contains 5000 sentences from movie reviews extracted from the website Rotten- Tomatoes.com com and whose sentiment was classified as positive or negative. This dataset the original one in the website2
Table 7: AUC for different annotators with class-dependent noise and instance-dependen noise.
Type
Class-dependen Noise
Instance neseenee-dependent Noise
Annotator
Annotator2
Annotator3
Annotator
Annotator2
Annotator3
AgNews
0.632
0.684
0.738
0.692
0.679
0.642
20News
0.634
0.686
0.740
0.667
0.692
0.644
IMDb
0.632
0.684
0.738
0.624
0.607
0.605
Yelp
0.632
0.684
0.738
0.658
0.625
0.617
Amazon
0.632
0.684
0.738
0.623
0,597
0,566
Diabetes
0.627
0.681
0.735
0.672
0.641
0.639
Backdoor
0,643
0.694
0.746
0.683
0.761
0.746
Campaign
0.628
0.681
0.735
0.587
0.576
0.663
Waveform
0.640
0.691
0.748
0.672
0.593
0.635
Celeba
0.634
0.687
0.749
0.667
0.642
0.594
SVHN
0.632
0.684
0.739
0.693
0.679
0.642
F-MNIST
0.632
0.684
0.738
0.795
0.780
0.774
CIFAR-10 10
0.632
0.686
0.738
0.629
0.666
0.540
http://www..ggggeoooo//oosshhhhe healtehs-hhhhhhhheeeee sewalth-indicttos--aaaaae
http:///frooiiuuus.ccmmmuttttttt datasstkttttttttaaaaaa
16

---

Details of labels. As mentioned in Section 4.1 we use two methods to simulate three annotators. The AUC of these three labels is shown in Table
C Implementation Details of the Algorithm
The implementation procedure of the proposed method CRL is shown Algorithm
Algorithm 1 Collaborative Refining for Learning from inaccurate labels (CRL).
Input:
Dataset D = ( (7% 1 The chosen set of epochs T. The proportion of selected
samples p.
Output: Submodels M. Final submodel fen.1.
// Step Gather dataset Da and Da.
2: Obtain Da where annotators disagree, i.e. Fro, 11 S S (1,.... R. V. Di
3: Obtain Da where annotators agree, i.e., Vro, 11 S (1,..., RJ, 91 D.
4: Initialize R queues (0)) E (1,... RJ. each with length (7).
5: for = to max mnax_epoch do
6: // Label Refining for samples with Disagreements (LRD)
7: // Step 2: Generate the refined dataset D
8:
Obtain instance-wise loss values from {ʃe,) jRt by Eq.
9:
Obtain instance-wise refined label j7* by Eq. 3 and Eq.
10:
Construct the refined dataset D with refined label y".
11:
I/ Robust Union Selection (RUS)
12:
// Step Generate the refined dataset D
Obtain instance-witg loss values from submodels with every θ in (0; by Eq.
13:
14:
Calculate the instance- -wise selection criteria by Eq. (IO).
15:
Sort the samples in D, by the criteria and select the smallest p the refined dataset D
16:
I/ Step 4: Utilize D2 and D to train OR+1-
17:
Compute loss LDUUD= on D2 and D with fōx+1 by Eq. ①
18:
Update parameters 0 with loss LDjUDD-
19:
I/ Step 5: Utilize D to train (0) --
20:
Compute loss LD on with submodels (fe, ) by Eq.
21:
Update parameters {0,) with loss LD.
22:
// Step Update the queues with the latest parameters.
23:
if The queues have reached their capacity then
24:
Remove the first element of each queue, e.g., (0
25:
end if
26: Save submodels' latest parameters into their respective queues.
27:
end for
D Related Work
Learning from a single inaccurate label is well-established topic, and numerous approaches
have been proposed [11 17 19 25 28 R5) These works can divided into two main categories. Some works aim to estimate the noise transition matrix [12 17 24 28), which contains the probabilities of clean labels flipping into inaccurate labels. However, the noise transition matrix is hard to estimate accurately. The second approach is sample selection, employing sample selection methods to select possibly clean examples from a mini-batch and train the model with these examples
25 35 Many sample selection methods leverage observed patterns, such as the small-loss filtering criterion that considers samples with small losses as clean. The central challenge in these methods is to construct selection criteria.
[1I 13 19
23
Learning from multiple sets of inaccurate labels. The works can divided into two main categories. One stream of research focuses on advanced aggregation algorithms that infer the true labels before during the training stage, others focus on end-to-end learning, simultaneously learning from multiple sets of inaccurate labels.
17

---

In aggregation algorithms, the simplest method is Majority Voting [38.. where all the annotations are treated equally and labels are aggregated by voting. Some advanced methods go beyond ma- jority voting. Weighted Majority Voting 20) uses weight vector to model annotators' expertise, Max-margin Majority Voting 32] combines the concept of majority voting with the principles of margin maximization. Enhanced Bayesian Classifier Combination (EBCC) [22 utilizes the Bayesian classifier to infer truth by modeling the correlation between annotators. By viewing the true labels as latent variable, some methods considering relations among annotators are proposed to infer true labels based on the Dnpectation-Maximizaaii DS model) assume that every annotator has class dlas-dependeent confusion when providing annotations, which is modeled by an manotator-specifi noise confusion matrix. The DS model is the basis many works for aggregating labels from different annotators with the help of EM algorithm 2 34
(EM) algorithm. The seminal works 4 (known as the
In recent studies, an increasing number of studies focus on end-to-end learning. Crowdl Layer trains an end-to-end Deep Neural Network (DNN) with parametric source-specific transition matrices. DoctorNet 10) aims train DNNs that exploit multiple annotators' information with different softmax output layers. HyperEnsemble [37] trains several individual models for each set of inaccurate labels (i.e., models for sets labels) and combines their outputs by averaging and maximizing. referred to HE_A and HE M. respectively. UnionNet[36 36 learns transition matrix for all multiple sets of labels together. CoNAL assumes that the annotation noise attributed to two sources (common noise and individual noise), and combines these two noises by Bemoulli random variable. SLF[7] proposes to make the weight vectors and the confusion matrices data-dependent and comes up with two regularization methods for the confusion matrix to guide the training process. ADMoE 45 leverages the Mixture of Experts (MoE) architecture to encourage specialized and scalable learning from multiple inaccurate sources. SDM (3) employs self-cognition module to identify both instance- wise noise and manotator-wise quality and adopts a mutual-denoising module to aggregate these identifications and accordingly refine the model.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
As declared in the preliminaries, we focus on binary classification problems.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification: For Theorem 1 and Corollary 1, we provide the full set of assumptions and complete (and correct) proof. For Theorem 2, we provide the full set of assumptions and the proof is similar in the reference cited in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The implementation details are all provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: The data are processed from open-source benchmark datasets, which are readily accessible. We will consider open-sourcing the code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccccuclccuuueeeeeeeeeeeee Coaee//ddeeeSbmmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The full details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the space limitation and the extensive experiments, the averaged results are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: is not a critical factor for this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://uurr....ccppocac/caaaaaaa
Answer: [Yes]
Justification: Ethics.
The research conducted in the paper conform with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: As framework for learning from multiple sets of labels, this work does not suffer from particular ethical concerns or negative social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners of assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/codd/modell as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. The datasets are processed from open-source datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.


Reviews:
[{"review_id": "IYPFI5MGjS", "reviewer": "Reviewer_9hER", "summary": "This paper studied learning from multiple noisy labels via data refinement. It first uses the annotator agreement as an instrument to divide all samples into the samples where some annotators disagree and the samples where all annotators agree. Then, a comparative strategy is proposed to filter noise in the samples where some annotators disagree, and a robust union selection is used to select clean samples in the samples where all annotators agree. Experiments on multiple datasets show the effectiveness of the proposed method.", "questions": "1. How does the proposed method cooperate with other algorithms in Table 5?\n2. Why the robust union selection is called an aggregating strategy in Line 12 and Line 62?", "limitations": "1. The proposed methods need to train R + 1 submodels, which need a large amount of computing resources when R is large.\n2. The proposed methods may be not adapted to the annotation sparse case, which is common in learning from crowds.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The data refinement is important in this age of big data.\n2. The label refining for samples with disagreements is novel in learning with noisy labels with multiple annotators.\n3. The proposed method can cooperate with other learning-from-crowds algorithms.", "weaknesses": "1. The novelty of robust union selection is limited, since it is very similar to robust mean estimation in [1]. Could the authors clarify this?\n\n2. As mentioned in this work, some methods have adopted the small-loss criterion to refine data [2-4]. The comparison between the proposed method and one of them is necessary, which can show the necessity of designing the new data refinement way with multiple annotators. Besides, one latest work [5] in ICML 2024 also explored this direction, I think this work should at least discuss the differences with it.\n\n3. The proposed method seems also suitable for multi-class classification problems. Since many real-world cases are multi-class classification problems, why does this work only focus on binary classification problems?\n\n[1] Sample Selection with Uncertainty of Losses for Learning with Noisy Labels. ICLR 2022\n\n[2] Coupled-view deep classifier learning from multiple noisy annotators. AAAI 2020\n\n[3] Learning from crowds with mutual correction-based co-training. ICKG 2022\n\n[4] Coupled confusion correction: Learning from crowds with sparse annotations. AAAI 2024\n\n[5] Self-cognitive Denoising in the Presence of Multiple Noisy Label Sources. ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback, which is valuable for improving the quality of our research. We hope our response can satisfactorily address your questions.\n\n**Weakness 1: The novelty of robust union selection is limited, since it is very similar to robust mean estimation in [1]. Could the authors clarify this?**\n\nOur basic idea is consistent with [1], which is to use multiple sets of predictions to identify more reliable samples. [1] focuses on scenarios with single noisy labels, utilizing the robust mean of predictions made by the model at different training stages to filter samples. While in the field of learning from multiple sets of labels, to the best of our knowledge, there have been no relevant attempts. Aiming at the characteristics of this problem, our contribution is to collaboratively leverage multiple submodels' abilities to identify more reliable samples and consider the potential issue of abnormal predictions from some submodels.\n\nSpecifically, robust union selection not only incorporates robust mean estimation, using it as a very fundamental smoothing step, but also goes further by introducing the variance of losses from different submodels to form a selection criterion. This criterion allows clean samples that might have been overlooked due to higher mean losses resulting from inaccurate predictions of certain submodels to potentially be re-included in the training process, rather than being excluded. In contrast, [1] follows the smoothing step by introducing the number of times each sample has been used in training, ensuring that less utilized samples are given more opportunities to be included. These are different solutions aimed at different problems.\n\n**Weakness 2: As mentioned in this work, some methods have adopted the small-loss criterion to refine data [2-4]. The comparison between the proposed method and one of them is necessary, which can show the necessity of designing the new data refinement way with multiple annotators. Besides, one latest work [5] in ICML 2024 also explored this direction, I think this work should at least discuss the differences with it.**\n\nFollowing your suggestions, we have incorporated a comparison with [2], abbreviated as CVL. We present average AUC results under class-dependent noise here: \n|Methods|Agnews|20News|Yelp|IMDb|Amazon|Diabetes|Backdoor|Campaign|Waveform|Celeba|SVNH|FMNIST|CIFAR10|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|CVL|0.837|0.828|0.817|0.748|0.695|0.705|0.755|0.752|**0.868**|0.868|0.743|0.752|0.638|\n|Ours|**0.855**|**0.849**|**0.867**|**0.766**|**0.775**|**0.728**|**0.937**|**0.783**|0.840|**0.891**|**0.761**|**0.776**|**0.655**|   \n\n[5] is publicly released at ICML 2024 in July, which is later than the NIPS 2024 submission deadline. Following your suggestions, we try to discuss the differences with it. \n\n[5] treats all samples in the same way without distinction. In contrast, our work differentiates samples based on annotator agreement and designs relatively appropriate methods for each. Specifically, for samples where some annotators disagree, we directly determine the relatively accurate label based on theoretical considerations. For samples where all annotators agree, we robustly integrate multiple model predictions to select relatively reliable samples. These are two distinct perspectives. In the revised version of our paper, we will consider adding a comparison with [5].  \n\n**Weakness 3: The proposed method seems also suitable for multi-class classification problems. Since many real-world cases are multi-class classification problems, why does this work only focus on binary classification problems?**\n\nIn theory, our method is suitable for multi-class classification problems. However, we've encountered some difficulties during practical experiments. The primary issue arises with samples where annotators disagree, particularly when all the given labels for a particular sample are incorrect. Our current method, LRD (Label Refining for samples with Disagreements), struggles to address this situation. It tends to select one of these incorrect labels as the inferred label, negatively impacting the model's performance. In binary classification problems, this situation does not occur because when annotators disagree, one of the labels must be correct.\n\nWe are still working on modifying our approach to make it suitable for multi-class classification problems. Based on our framework, A possible approach could be to develop a more refined measure based on annotator agreement, which could be utilized to decide whether to discard these samples, use LRD, or use RUS (Robust Union Selection). Since these ideas are not yet mature enough, we did not include this part in the current work. This will be the focus of our future exploration.\n\n**Question 1: How does the proposed method cooperate with other algorithms in Table 5?**\n\nThank you for your attention to an important characteristic of our algorithm: its ability to cooperate with other algorithms. After training our model for several epochs, through LRD and RUS, we can produce a refined dataset. We extract the refined dataset and complement it with the original noisy labels. Then the refined dataset containing both the original noisy labels and the refined labels can be used to train other methods in the same manner as in the main experiments. This process is detailed in lines 309-310 of our paper.\n\n**Question 2: Why the robust union selection is called an aggregating strategy in Line 12 and Line 62?**\n\nThe \"aggregating strategy\" mentioned here refers to the concept of harnessing the collective intelligence of multiple submodels to select samples. Specifically, we aggregate the predictions of multiple submodels by calculating the robust mean and the variance, and use our selection criteria to filter samples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G09Ri3FB7C", "reviewer": "Reviewer_eKLA", "summary": "This paper introduces a framework for learning from inaccurate labels obtained from multiple annotators. It utilizes annotator agreement to assess label reliability and applies two strategies: one for samples with annotator disagreements (LRD) and another for samples where all annotators agree (RUS). In both cases, the framework uses a number of submodels equal to the number of annotators (with shared layers). The framework refines unreliable datasets into relatively reliable datasets that are used to train the final model. LRD selects reliable labels by comparing the losses of the submodels and choosing the label assigned by the submodel with the smallest loss. RUS enhances dataset quality using a different loss-based selection criterion. Experiments are conducted on class-dependent, instance-dependent, and real-world datasets.", "questions": "- What are the advantages of using $\\phi$ in equation 7 instead of directly using the loss?\n\n- Has a comparison regarding the running time of the proposed method and the baselines been conducted?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) The analysis conducted is both interesting and original.\n2) The paper is clearly written and is easy to follow.", "weaknesses": "1) The absence of multiple runs or statistical measures such as standard deviation limits the robustness and reliability of the reported results.\n\n2) Table 1 does not provide information on the quality or reliability of the annotations used in the presented setting.\n\n3) Including additional comparative methods, such as Dawid Skene [1], Iterative Weighted Majority Voting [2], and [3], would enrich the analysis and provide a broader evaluation of the proposed approach.\n\n[1] Dawid, A. P. and Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20– 28.\n[2] Li, H. and Yu, B. (2014). Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv preprint arXiv:1411.4086.\n[3]  Karger, D. R., Oh, S., and Shah, D. (2014). Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1–24.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to you for your insightful comments and questions.\n\n**Weakness 1: The absence of multiple runs or statistical measures such as standard deviation limits the robustness and reliability of the reported results.**\n\nThank you for pointing out the necessity of including standard deviation to reflect the statistical significance of our results. The results reported in our submission currently represent the average outcome of five results. Due to space constraints in the current formatting, we have only presented the mean values. We will include the variance in the final version of our paper.\n\n**Weakness 2: Table 1 does not provide information on the quality or reliability of the annotations used in the presented setting.**\n\nDue to page limits, this information can be found in Table 7 on page 16 in the Appendix. \n\n**Weakness 3: Including additional comparative methods, such as Dawid Skene [1], Iterative Weighted Majority Voting [2], and [3], would enrich the analysis and provide a broader evaluation of the proposed approach.**\n\nThank you for your suggestions. These methods [1][2][3] are algorithms that have provided some inspiration for the current methods. New approaches have made further advancements based on the ideas of these methods. For instance, the concept of weighting different label sources, as discussed in [2], has been significantly expanded upon in [4]. These newer approaches have been thoroughly compared and evaluated in the paper, therefore, we did not include these algorithms. We will consider adding a comparison to enrich the analysis in the revised version.\n\n[1] Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 1979.   \n[2] Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv 2014.   \n[3] Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 2014.  \n[4] Who said what: Modeling individual labelers improves classification. AAAI 2018.  \n\n**Question 1: What are the advantages of using $\\phi$ in equation 7 instead of directly using the loss?**\n\nEquation 7 utilizes a smooth function that narrows the range of the loss. Compared to directly using the loss, this approach can initially help mitigate the impact of outliers. Additionally, by introducing a lower bound for the losses, considering variances, we can retain clean samples that might otherwise be excluded due to abnormal predictions by certain submodels.\n\n**Question 2: Has a comparison regarding the running time of the proposed method and the baselines been conducted?**\n\nAs you suggested, we ran our method and some competitive baselines 10 times on 5 datasets and recorded the average running time. All the following experiments were conducted on an Apple computer with an M1 chip and 16GB of memory, maintaining the same parameters as in the main experiments, i.e., the same number of epochs, hidden size, and batch size. The results are as follows, the running time of our method does not show a significant difference compared to other competitive baselines.\n\n|Methods|CoNAL|ADMOE|SLF|Ours|\n|---|---|---|---|---|\n|Seconds per step|0.55|0.65|0.84|0.78|\n\n Moreover, our method offers room for further acceleration. One potential strategy could be to freeze the samples that have been refined through LRD and RUS after a few epochs. Then, these refined samples could be used only to train the final submodel while freezing the other submodels. Additionally, the computational load is indeed related to the number of label sets used. We can consider initially aggregating some labels using methods like majority voting to alleviate the computational burden.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2usRH6TGv", "reviewer": "Reviewer_hvQ6", "summary": "This paper considers binary classfication from multiple sets of noisy labels, focusing on data refinement to generate clean labels. At each step of training, It proposes to first separate the dataset by whether label disagreement exists, and then tackle each subset using different methods. For the subset with disagreed labels, authors propose to follow the label with lowest lost. For the subset with all same labels, authors propose a delicated designed term to filter out unreliable data points. Thorough empirical evaluation shows good performance under various settings.", "questions": "- Is it possible to lower the bar of annotator agreement? In other words, is it possible that labels with few disagreement being filtered out using RUS, so it is not neccessary to keep all labels the same before RUS?\n- LRD is designed under class-conditional noise assumption, but empirically performs better for instance-dependent noise. Is there any empirical observation deserves to discuss?\n- At the bottom of page 5, it is an interesting observation that relatively large variance are selected. How selection of $\\phi(x)$ would change this behavior? What leads to the adoption of this function based on Taylor expansion?", "limitations": "Limitations are properly addressed in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is well-motivated by theory and of high practical interest.\n- The overall presentation of the paper is well-written and clear enough to understand.\n- Comprehensive empirical evaluation has been carefully conducted and the performance of the proposed method has been clearly demonstrated.", "weaknesses": "- Limitations such as binary classification and class-conditional noise assumption for LRD can be addressed more clearly in introduction.\n- Discussions on model architecture selection can be improved. The design of using R + 1 heads with a shared backbone first appears in section 2 without any detailed introduction.\n- Algorightm 1 is mentioned several times in the paper but itself is located in Appendix. This draws concern on abusing the page limit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort you have dedicated to reviewing our work. We appreciate your insightful comments and questions, which are valuable for improving the quality of our research. \n\n**Weakness 1: Limitations such as binary classification and class-conditional noise assumption for LRD can be addressed more clearly in introduction.**\n\nThank you for your suggestion. We will discuss these limitations in the introduction in the revised version.\n\n**Weakness 2: Discussions on model architecture selection can be improved. The design of using R + 1 heads with a shared backbone first appears in section 2 without any detailed introduction.**\n\nDue to page limitations, we have provided a brief description of the model structure. We will provide a more detailed explanation in the revised versions. We design such a shared backbone to enhance information sharing and to reduce the overall computational cost of the method. With the shared backbone, each sub-model only requires a simple three-layer MLP.\n\n**Weakness 3: Algorightm 1 is mentioned several times in the paper but itself is located in Appendix. This draws concern on abusing the page limit.**\n\nThank you for your suggestion, we will improve it in the revised version.\n\n**Question 1: Is it possible to lower the bar of annotator agreement? In other words, is it possible that labels with few disagreement being filtered out using RUS, so it is not neccessary to keep all labels the same before RUS?**\n\nYes, it's an interesting and sensible idea. For example, we can introduce a new hyperparameter: the consistency rate. For samples where the consistency rate is higher than the threshold, the RUS method is used, and for samples where the consistency rate is lower than the threshold, the LRD method is employed. Since setting this threshold is related to the quality of the dataset itself and the number of label sets, it is more suitable for practical scenarios, especially when we have a good understanding of the label quality.\n\n**Question 2: LRD is designed under class-conditional noise assumption, but empirically performs better for instance-dependent noise. Is there any empirical observation deserves to discuss?**\n\nThis is a complex question, and we try to provide some tentative insights here.\n\nOn the one hand, class-dependent noise is random and unrelated to features, while instance-dependent labels are related to the features and inherently contain the annotator's intelligence, thus providing more information. This makes instance-level labels more conducive to learning when the proportion of correctly labeled samples is close. This phenomenon aligns with our experiments, where the compared algorithms also tend to perform better under instance-dependent noise conditions. This suggests that the additional information contained in instance-dependent noisy labels can indeed facilitate learning.\n\nOn the other hand, our algorithm conceptually integrates the intelligence of multiple models and leverages information from multiple sets of labels. This design enhances the robustness and flexibility of our algorithm, allowing it to adapt to different noise scenarios. Our experiments demonstrate that our algorithm not only performs well on benchmark datasets but also shows competitive performance on real-world datasets.\n\n\n\n**Question 3: At the bottom of page 5, it is an interesting observation that relatively large variance are selected. How selection of 𝜙(𝑥) would change this behavior? What leads to the adoption of this function based on Taylor expansion?**\n\n$\\phi(x)$is used for a basic smoothing. The significance of this smoothing is that it reduces the impact of outliers when calculating the mean. Therefore, the current method is a good choice. We speculate that other smoothing methods, such as truncating outliers, could also be effective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2mfodchPdo", "reviewer": "Reviewer_qnq5", "summary": "This paper proposes a collaborative refining approach for learning with inaccurate labels provided by low-cost annotators, such as rule-based systems. It introduces strategies based on annotator agreement to filter out noise and enhance data quality. The method includes comparative filtering for conflicting labels and aggregation for consistent annotations, all guided by theoretical bounds on loss values. Extensive experiments on various datasets demonstrate significant improvements in learning performance despite the presence of label inaccuracies.", "questions": "Please check the weaknesses, and some minor comments are presented below:\n\n- The reference format of the equation is inconsistent, sunch as some are written as Eq. 1, while others are written as Eq. (10).", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a collaborative refining method for handling inaccurate labels obtained from low-cost annotators, such as rule-based systems, in contrast to traditional label aggregation approaches.\n\n- This paper proposes strategies based on annotator agreement to filter out inaccuracies through comparative analysis and aggregation, thereby enhancing the quality of the training data.\n\n- Theoretical analysis uncovers relationships among multiple sets of labels, corresponding models, and true labels, providing a foundation for reliable label selection.\n\n- Extensive experiments conducted on various benchmark and real-world datasets demonstrate the effectiveness of the proposed methods in improving learning performance despite label inaccuracies.", "weaknesses": "- The proposed method of this paper seems similar to some research methods explored in unreliable partial label learning. Is there any relations between the two?\n\n- When the model is not the Bayesian optimal model, why does $\\ell(f_{\\theta_0^*}(x),\\tilde y^0)<\\ell(f_{\\theta_1^*}(x),\\tilde y^1)$ hold in Theorem 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and questions. We hope our response can satisfactorily address your questions.\n\n**Weakness 1: The proposed method of this paper seems similar to some research methods explored in unreliable partial label learning. Is there any relations between the two?**\n\nIn terms of the overall goal—extracting useful information from imperfect data—our proposed method is consistent with unreliable partial label learning, which is also a common goal for other works in our field. However, our method differs significantly from the methods explored in unreliable partial label learning. Guided by theoretical insights, we train multiple submodels and utilize the relationships among these submodels, multiple sets of noisy labels, and true labels to make label decisions and sample selections. To the best of our knowledge, there is no similar work in unreliable partial label learning that addresses the problem from this angle. This significant difference from previous methods is the key contribution of our method. \n\n\n**Weakness 2: When the model is not the Bayesian optimal model, why does $\\ell \\( {f_{\\Theta_0^\\star} }{(x)},\\tilde{y}^0 \\) < \\ell \\({f_{\\Theta_1^\\star} }{(x)},\\tilde{y}^1\\)$ hold in Theorem 1?**\n\nIt is challenging to theoretically prove the inequality holds when the model is not the optimal one. We provide an intuitive explanation and experimental verifications here. The training of the model to its optimal state is a gradual process. As the model progressively approaches optimal state during training, its parameters and structure stabilize, and its behavior and performance tend to converge towards that of the theoretical optimal model. At this point, the effects of Theorem 1 begin to manifest.\n\nWe've provided our experimental verifications in Table 4 (page 9). The results show that after training for 100 steps, the label quality produced by LRD (based on Theorem 1) is significantly higher than the original labels (average AUC 0.762 vs 0.368 on $D_d$). With ongoing training, label quality rises (e.g., average AUC 0.819 on $D_d$ at 500 steps). \nThis evidence suggests that: \n\n● Theorem 1 begins to be valid after training for some steps rather than being effective only in the optimal state.\n\n● The validity of Theorem 1 improves gradually during the convergence process, which is consistent with our intuitive explanation.\n\nThese verifications bridge theory and practical application, offering considerable flexibility in choosing when to use the refined labels.\n\n**Question 1: Please check the weaknesses, and some minor comments are presented below: The reference format of the equation is inconsistent, such as some are written as Eq. 1, while others are written as Eq. (10).**\n\nThanks for your suggestions, we will improve it in the revised version.\n\nShould there be any remaining questions or points of clarification required, we would be more than willing to provide further details or engage in additional discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zJmVToK37p", "reviewer": "Reviewer_qood", "summary": "In common practical scenarios, autonomous annotators are used to create labeled datasets, reducing the dependence on manual labeling, which can be costly and time-consuming. Learning methods leverage multiple weak labels to annotate large amounts of data, though these weak labels are often noisy and imperfect. The paper presents a collaborative refining approach for learning from these inaccurate labels. To refine the data, the authors differentiate between cases where multiple annotators agree or disagree on the labels for a given sample. For samples with disagreements among annotators, the authors propose a noise-filtering method, while for samples with unanimous agreement, they suggest an aggregating method.", "questions": "What happens if an annotator chooses to abstain from labeling a sample?\n\nIs it realistic for all annotators to provide the same label? In the weak supervision datasets such as WRENCH datasets, this rarely occurs.\n\nBuilding on the previous question, what if one of the datasets $D_d$ or $D_a$ is empty?\n\nWhat assumptions are made about the annotators?", "limitations": "See above", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "The paper is well written and easy to follow.\n\nThe authors compare their method with many others and use multiple datasets.", "weaknesses": "My main issue with this paper is that the problem is not well explained, there is a lack of literature, and most experiments are not conducted with real datasets. If I understand correctly, the problem addressed involves unlabeled samples and the output from multiple annotators. The goal is to obtain labels for these unlabeled samples. This problem is commonly referred to as programmatic weak supervision [1]. In programmatic weak supervision, the objective is to obtain probabilistic labels for the unlabeled samples, and models used to generate these label probabilities are commonly known as label models. Subsequently, these labels are utilized to train a classifier (end model).\n\nIn the problem formulation, it is initially stated that the aim is to obtain labels for unlabeled samples, and then (line 85) it is mentioned that the goal is to learn a classifier. What is the objective of the problem being addressed?\n\nIf the objective is to obtain labels for unlabeled samples, why aren't probabilistic labels obtained as in programmatic weak supervision?\n\nThe WRENCH library [2] collects multiple datasets with outputs from multiple annotators, along with state-of-the-art methods. Why weren't the annotators from that library used?\n\nIn the experiments, comparisons are made with methods such as EBCC, IBCC, and WeaSEL. As far as I know, EBCC and IBCC are label models, whereas WeaSEL is a joint model (label model + end model). Therefore, the objectives of these methods are not the same.\n\nReferences\n\n[1] Zhang, J., Hsieh, C. Y., Yu, Y., Zhang, C., & Ratner, A. (2022). A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433.\n\n[2] Zhang, J., Yu, Y., Li, Y., Wang, Y., Yang, Y., Yang, M., & Ratner, A. (2021). WRENCH: A comprehensive benchmark for weak supervision. arXiv preprint arXiv:2109.11377.\n\nRatner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017, November). Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB endowment. International conference on very large data bases (Vol. 11, No. 3, p. 269). NIH Public Access.\n\nRatner, A. J., De Sa, C. M., Wu, S., Selsam, D., & Ré, C. (2016). Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29.\n\nMazzetto, A., Cousins, C., Sam, D., Bach, S. H., & Upfal, E. (2021, July). Adversarial multi class learning under weak supervision with performance guarantees. In International Conference on Machine Learning (pp. 7534-7543). PMLR.\n\nBalsubramani, A., & Freund, Y. (2015, June). Optimally combining classifiers using unlabeled data. In Conference on Learning Theory (pp. 211-225). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We recognize that there may be some misunderstandings regarding our paper and would like to take this opportunity to clarify these points and address your concerns. \n\n**Main issue 1: \"My main issue with this paper is that the problem is not well explained, there is a lack of literature.\"**\n\nThe problem is explicitly stated in our manuscript, specifically in lines 1 and 31-32: \"This paper considers the problem of learning from multiple sets of inaccurate labels.\" Our paper primarily explores training a model using multiple sets of noisy labels to make predictions for unseen samples (the joint model you mentioned). The entire process of our method is described multiple times in the paper, e.g., lines 8-16, 106-108, 319-325.\n\nThis is a well-established and clearly-defined research topic with substantial contributions from various researchers [1-10]. A comprehensive summary of these works is provided in Lines 30-47. These works represent widely acknowledged methods of this topic, and we have provided an overview of them as well as conducted a comprehensive comparison in our experiments. The literature and compared algorithms discussed in the most recent works [10] (AAAI2024) and [12] (ICML2024) have been adequately covered in this paper as well.\n\n**Main issue 1: \"If I understand correctly, the problem addressed involves unlabeled samples and the output from multiple annotators. The goal is to obtain labels for these unlabeled samples.\"**\n\nIt seems that there might be a misunderstanding regarding our research focus. The problem we are addressing has significant differences from what you describe. While your description aligns with inferring labels for existing samples (label model), our paper primarily explores training a model using multiple sets of noisy labels to make predictions for unseen samples (joint model). This objective is highlighted in several places in our manuscript, including Lines 1, 14-16, 85, and 106-108. and this is also the common focus of many works [1-4, 6-10, 12]. \n\nThe methods you mentioned, such as EBCC and IBCC, are targeted as one important line of compared works. In our experiments, we utilize generated labels from these label models to train a Multi-Layer Perceptron (MLP) for a fair comparison, which follows previous works [4, 6, 7, 8, 9, 10, 12]. Moreover, our comparison includes a broader range of methods, ensuring a comprehensive evaluation of our proposed approach.\n\n**Main issue 3: \"Most experiments are not conducted with real datasets\" and about WRENCH.**\n\nOur experiments have included CIFAR10N and Sentiment Polarity, which were two real-world datasets published on Amazon Mechanical Turk for annotation and were utilized in the most recent works like [10] (AAAI 2024) and [12] (ICML 2024). The results are summarized in Table 2 (Page 7), substantiating the reliability and applicability of our approach in handling real-world noisy labels.\n\nWhen we face the problem of learning from noisy labels, due to the uncontrollable noise levels in real-world datasets, experiments are typically first validated under controlled noise conditions on benchmark datasets, before being further tested on real-world datasets. Many studies follow this approach [1-12], and we conducted our experiments in the same way. \n\nWe also use 13 benchmark datasets, some of which are featured in recent studies, such as [9] (AAAI 2023) and [12] (ICML 2024). Hence, we believe our experiments are relatively comprehensive. Since we followed recent works on this topic and our experiments were relatively thorough, we did not use the WRENCH library at that time. We will consider incorporating some WRENCH datasets and adding the results in the revised version.\n\n**Reference**\n\n[1] Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. TMI 2016. \n[2] Deep learning from crowds. AAAI2018.  \n[3] Who said what: Modeling individual labelers improves classification. AAAI2018.  \n[4] Max-mig: an information theoretic approach for joint learning from crowds. arxiv 2019.  \n[5] Exploiting worker correlation for label aggregation in crowdsourcing. ICML2019.  \n[6] Coupled-view deep classifier learning from multiple noisy annotators. AAAI2020.  \n[7] Learning from crowds by modeling common confusions. AAAI2021.  \n[8] Learning from crowds with mutual correction-based co-training. ICKG 2022.  \n[9] Admoe: Anomaly detection with mixture-of-experts from noisy labels. AAAI2023.  \n[10] Coupled confusion correction: Learning from crowds with sparse annotations. AAAI2024.  \n[11] End-to-end weak supervision. NIPS2021.  \n[12] Self-cognitive Denoising in the Presence of Multiple Noisy Label Sources. ICML 2024.  \n\n**Question 1: What happens if an annotator chooses to abstain from labeling a sample?**\n\nUnder the binary classification scenario, as considered in our paper, annotations can be swiftly generated with no sparsity, e.g., by multiple rules. Even when such sparsity is present, it can often be generally addressed by defaulting to the majority class.\n\n**Question 2: Is it realistic for all annotators to provide the same label? In the weak supervision datasets such as WRENCH datasets, this rarely occurs.**\n\nIt is realistic. For instance, CIFAR-10N and Sentiment Polarity datasets are both manually labeled via Amazon Mechanical Turk, in which 58.28% and 60.36% of the samples, respectively, are given the same labels by all the annotators. \n\n**Question 3: Building on the previous question, what if one of the datasets $D_d$ or $D_a$ is empty?**\n\nIn this case, we can only use LRD in our framework to deal with $D_d$ or only use RUS to deal with $D_a$.\n\n**Question 4: What assumptions are made about the annotators?**\n\nIn this paper, we focus on the labels themselves. The two most important assumptions are (1) class-conditional noise assumption and (2) the diagonal elements of noise transition matrices are greater than 0.5. These assumptions are widely used in many papers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for learning from multiple sets of inaccurate labels based on collaborative refinement approach. The proposed approach is interesting and novel in learning with inaccurate labels with multiple annotators. The effectiveness of the proposed method is well demonstrated in the experiments. This paper is well written. More discussion on the existing work is needed. I encourage the authors to improve their paper by reflecting the reviewers’ comments and the author response.
Decision: Accept (poster)