Collaborative Refining for Learning from Inaccurate Labels

Authors: bin  han
yi-xuan  sun
ya-lin  zhang
libang  zhang
haoran  hu
longfei  li
jun  zhou
guo  ye
huimei  he
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 13:40:52

Abstract: This paper considers the problem of learning from multiple sets of inaccurate labels, which can be easily obtained from low-cost annotators, such as rule-based annotators. Previous works typically concentrate on aggregating information from all the annotators, overlooking the significance of data refinement. This paper presents a collaborative refining approach for learning from inaccurate labels. To refine the data, we introduce the annotator agreement as an instrument, which refers to whether multiple annotators agree or disagree on the labels for a given sample. For samples where some annotators disagree, a comparative strategy is proposed to filter noise. Through theoretical analysis, the connections among multiple sets of labels, the respective models trained on them, and the true labels are uncovered to identify relatively reliable labels. For samples where all annotators agree, an aggregating strategy is designed to mitigate potential noise. Guided by theoretical bounds on loss values, a sample selection criterion is introduced and modified to be more robust against potentially problematic values. Through these two methods, all the samples are refined during training, and these refined samples are used to train a lightweight model simultaneously. Extensive experiments are conducted on benchmark and real-world datasets to demonstrate the superiority of our methods.

Main paper: 
 introduction: 
 Deep learning has recently demonstrated remarkable achievements across a wide array of applications [5, 6, 9, 15, 33] . The cornerstone of its success rests on the availability of high-quality datasets. However, in industrial environments, obtaining accurate labels can often be costly and time-consuming. For example, in financial scenarios, it is challenging for human annotators to accurately assign labels to predict whether a user will default. Currently, many banks and companies utilize rules as low-cost autonomous annotators, especially in the early stages of some financial products. In healthcare scenarios, many rule-based algorithms are also employed to identify patients at risk of developing certain conditions, allowing for early intervention and potentially preventing more severe complications. Although these annotators can provide labels in a time-and cost-efficient manner [7, 44, 45] , a single low-cost annotator usually produces biased or inaccurate labels in practice [8, 13, 18, 24, 29, 40] .On the other hand, multiple annotators can provide diverse perspectives with different insights and knowledge to mitigate errors and biases of individual annotators [30, 45] . Learning from multiple sets of inaccurate labels provided by multiple annotators has garnered widespread attention and relative works can be divided into two main categories. One stream of research focuses on advanced aggregation algorithms that infer the true labels before or during the training stage. The simplest aggregation algorithm is Majority Voting [38] , which treats the labels equally by voting, while the approach known as Weighted Majority Voting [20] uses a weight vector to model annotators' expertise. Enhanced Bayesian Classifier Combination (EBCC) [22] tends to infer truth by modeling the correlation between annotators.Others focus on training models under the supervision of all annotations. By viewing the ground-truth labels as latent variables, some methods that consider the relationships among multiple sets of labels have been proposed to infer true labels based on the Expectation-Maximization (EM) algorithm [1, 2, 34, 43] . Despite the effectiveness of these EM algorithms, they suffer from computational complexity during the training phase. In recent studies, an increasing number of studies focus on end-to-end learning. These end-to-end algorithms [3, 7, 10, 21, 27, 45, 30] can directly learn from multiple sets of noisy labels and map these noisy labels to part of the model (e.g., transition matrix) , encouraging the model to learn knowledge from all noisy labels collectively [45] .Previous works typically concentrate on aggregating information from all the annotators through label aggregation techniques (e.g., voting) or using labels from all annotators in an end-to-end manner (e.g., mapping multiple sets of noisy labels to part of the model). Most related works often neglect the significance of data refinement. Although some methods have recognized the need for data refinement [21, 31, 42] , they merely adopt the small-loss criterion [11] to filter noisy samples, and refinement is not their core contribution or their key point. On the contrary, we contend that refining a relatively clean dataset during training is the central point, which can alleviate the demands placed on model design and enhance the model's performance. To refine the dataset, an essential step is to assess the reliability of labels from multiple sets. We consider annotator agreement as an instrument, which refers to whether multiple annotators agree or disagree on the labels for a given sample. In this paper, we leverage the annotator agreement information and propose a novel framework named Collaborative Refining for Learning from inaccurate labels (CRL). For samples where some annotators disagree, labels are ambiguous, and learning from all sets of labels can degrade the model's performance, a comparative strategy is proposed to mitigate noise. For samples where all annotators agree, an aggregating strategy is designed to filter out potential noise. The main contributions of this work can be described as follows:• For samples where some annotators disagree, we conduct theoretical analysis to uncover the correlations among multiple sets of labels, the respective models trained on them, and the true labels. Guided by theoretical insights, a method called Label Refining for samples with Disagreements (LRD) is proposed to identify the most reliable label by comparing loss values. • For samples where all annotators agree, we analyze the theoretical bounds on loss values. Based on these bounds, a method called Robust Union Selection (RUS) is proposed, in which we propose a loss-based selection criterion to select trustworthy samples and improve it to be more robust against potentially problematic values. • Comprehensive experiments are conducted on benchmark and real-world datasets, which demonstrate the effectiveness of our framework. Moreover, our framework is designed to be independent of any specific model architecture, making it compatible with most existing methods, which is confirmed by further experiments. 
preliminaries: 
 This paper concentrates on binary classification problems with multiple sets of inaccurate labels. Let D = {x i , ỹi } N i=1 be a dataset consisting of N instances labeled by R annotators. x i ∈ X ⊆ R d is the feature vector of the i-th instance and ỹi = {ỹ r i } R r=1 is a R-dimensional vector representing the labels provided by R annotators, ỹr i ∈ {0, 1}. Denote y * i as the unobserved ground-truth label for the i-th instance. y * i is considered as a latent variable decided by a latent function f * , i.e., f * (x i ) = y * i . f Θ denotes a classifier parameterized by Θ, and f Θ (x i ) = p(x i ) denotes the predicted probability after the activation function, i.e., sigmoid. The binary cross-entropy loss function of the pair(x i , ỹr i ) and model f Θ is ℓ(f Θ (x i ), ỹr i ) = ỹr i log(f Θ (x i )) + (1 -ỹr i ) log(1 -f Θ (x i )).(1) The goal of learning from multiple annotators is to get the optimal classifier f Θ * that satisfiesf Θ * (x i ) = f * (x i ).For the model architecture, several shared embedding layers are used to jointly extract information, followed by R + 1 submodels {f Θr } R+1 r=1 : R submodels {f Θr } R r=1 are responsible for making predictions based on R sets of labels, and the (R + 1)-th submodel f Θ R+1 generates the final prediction. These R + 1 submodels are simultaneously trained, with {f Θr } R r=1 learning from R sets of labels while f Θ R+1 concurrently training from the refined data. 
method: 
 In this paper, we present a novel framework called collaborative refining for learning from inaccurate labels (CRL). Within the framework, annotator agreement is utilized to partition the samples into two categories, i.e., samples where annotators disagree or agree. This concept facilitates the development of targeted strategies for each category.Based on the annotator agreement, the whole dataset can be partitioned into two parts: D d which contains samples where some annotators disagree, i.e., ∃r 0 , r 1 ⊆ {1, ..., R}, ỹr0 i ̸ = ỹr1 i ; D a which contains samples where all annotators agree, i.e., ∀r 0 , r 1 ⊆ {1, ..., R}, ỹr0 i = ỹr1 i . As mentioned in the introduction, for D d , the labels are ambiguous, and learning from all sets of labels may degrade the model's performance. To this end, we propose Label Refining for samples with Disagreements (LRD) to select relatively reliable labels. For D a , where labels are more reliable but not immune to errors. To address this, we present Robust Union Selection (RUS) to select trustworthy samples.In the framework, through LRD and RUS, R submodels {f Θr } R r=1 collaboratively refine unreliable dataset D d and D a into relatively reliable dataset D * d and D * a . D * d and D * a are utilized to train the final submodel f Θ R+1 with binary cross-entropy loss function as shown in Eq. ( 1). For prediction, f Θ R+1 is utilized. Note that RUS and LRD operate concurrently and share an identical set of R + 1 submodels. Our framework improves model performance exclusively by refining information, which is independent of any particular backbone, allowing for seamless substitution and flexibility in the model structure. The complete CRL framework is shown in Algorithm 1.When some annotators disagree, learning from all sets of labels may degrade the model's performance. A rational idea is to figure out which label is more reliable. Through theoretical analysis, we uncover the relationships among multiple sets of labels, the ground-truth label, and the submodels' predictions for a given sample, which in turn guide the design of the algorithm.We follow the widely used class-conditional noise assumption [10, 26] , i.e., p(ỹ r |y * , x) = p(ỹ r |y * ), ∀r ∈ {1, . . . , R}. Under this assumption, the noise transition matrix for the r-th annotator can be formulated as T r ∈ R 2×2 , where T r ij = p(ỹ r = j|y * = i) denotes the probability of an i-th class sample flipped into the j-th class for the r-th annotator.Inspired by Gui et al. [11] , we give the following theorem about the relationships among multiple sets of labels, the ground-truth label, and the submodels' predictions, proof can be found in Appendix A.1. Assume neural networks f Θ0 , f Θ1 are used to minimize the expected loss using labels from two annotators respectively. Theorem 1. Let (x, y * , ỹ0 , ỹ1 ) be any sample with ground-truth label y * and two conflicting labels ỹ0 , ỹ1 from two annotators, i.e., ỹ0 ̸ = ỹ1 . Assume T 0 and T 1 satisfy T 0 ii > 0.5 andT 1 ii > 0.5, ∀i ∈ {0, 1} , ℓ(f Θ * 0 (x), ỹ0 ) < ℓ(f Θ * 1 (x), ỹ1) if and only if y * = ỹ0 . Remark. Theorem 1 indicates that when the diagonal elements of two noise transition matrices are greater than 0.5, if two models are trained with these two sets of labels, for a sample x on which these two annotators disagree, the more reliable label can be selected by comparing the loss values in their respective models. Then, the above theorem from the scenario of two annotators can be expanded into multiple annotators, proof can be found in Appendix A.2. Assume a series of neural networks {f Θr } R r=1 are used to minimize the expected loss using labels from R annotators respectively. Corollary 1. Let (x, y * , {ỹ r } R r=1 ) be any sample with ground-truth label y * and R conflicting labels {ỹ r } R r=1 from R annotators, i.e., ∃r 0 , r 1 ⊆ {1, ..., R}, ỹr0 ̸ = ỹr1 . Assume T r ii > 0.5 , ∀i ∈ {0, 1} and r ∈ {1, ..., R} , if ℓ(f Θ * k (x), ỹk ) = min({ℓ(f Θ * r (x), ỹr )} R r=1 ), y * = ỹk .Remark. Corollary 1 indicates that if R models are trained with R sets of labels, when the observed labels of sample x are not the same, we can get the most reliable label for this sample by choosing the label which has the smallest loss value.Corollary 1 encourages us to infer the most reliable labels of the samples where some annotators disagree. Based on Theorem 1 and Corollary 1, we propose our LRD method to deal with D d . As mentioned in the preliminaries, R single-label submodels are trained. The training loss for these R submodels can be written as:EQUATIONwhere w r is a weighting factor introduced to address the predictive bias for positive and negative samples which may arise due to sample imbalance. Denote the number of positive labels in r-th set of labels as n r 1 and that of negative labels as n r 0 , a common strategy for setting w r is wr = n r 0 n r 1 .Through these submodels, we identify the most reliable labels by comparing their respective loss values over the same sample. Based on Corollary 1, for sampleEQUATIONwhere the most reliable index k for sample x i is acquired through:EQUATIONIn this way, for any instance (x i , {ỹ r i } R r=1 ), we can refine it into (x i , ỹ * i ). Then we can construct the refined dataset D * d and utilize it to train the submodel f Θ R+1 with the binary cross-entropy loss. This submodel can be deployed and utilized for final prediction. In practice, these refined labels are held constant after several training epochs to mitigate the over-fitting issue. The procedure of LRD is included in Algorithm 1. Note that LRD and RUS operate concurrently, with LRD focusing on D d and RUS on D a .For samples where all annotators agree, the labels are generally presumed to be reliable, however, they are not immune to errors. For instance, if the annotators have similar defects, although the annotators agree, the labels may still be inaccurate. To this end, guided by theoretical bounds on loss values, we introduce a loss-based selection criterion and modify it to be more robust against potentially problematic values.Inspired by the small-loss selection criterion mentioned in many works under single noise label scenario [11, 13, 23] , small-loss data tends to be more clean. However, relying solely on the prediction provided by one of the submodels is unstable. Once the selection is wrong, the inferiority of accumulated errors will arise [41] . Since we have R predictions for each sample with R sets of labels from our model, naturally, average loss values can be utilized, which can be denoted as:EQUATIONwhich is more stable and serves as an estimation of the mean µ. Thanks to the R sets of labels provided by R annotators, these submodels can be diverse, which can ease the inferiority of accumulated errors [13] .However, simply taking the average can also lose some correctly labeled samples. This issue originates from two aspects: annotator defects and hard samples. Firstly, different annotators may differ in quality or be suitable for different kinds of samples, which leads to the situation where some submodels may struggle with prediction and incur a larger loss for some correctly labeled samples.Secondly, some correctly labeled samples may be difficult to learn, as a result, submodels may give unstable predictions to them. Note that this issue is avoided in LRD since we use the minimum loss value of the same sample across submodels there. To address this issue, a robust criterion for sample selection is defined.We begin by defining a non-decreasing smooth function ϕ for loss values, which can be written as:EQUATIONwhere z is a positive variable, this function is inspired by the Taylor expansion of the exponential function. Loss values can be easily substituted into this equation to serve as an soft estimation of the underlying mean µ. Eq. ( 6) can reduce the side effect of extremum, which can mitigate annotator defects and hard samples issues. Based on Eq. ( 6), the robust average loss μϕ i can be written as:EQUATIONTo enhance the robustness of μϕ i , the predictions from historical models are introduced. Denote the submodel's parameters Θ r in the t-th epoch as Θ t r , the chosen set of epochs as T , then Eq. ( 7) can be rewritten as follows:EQUATIONwhere |T | represents the total number of the selected epochs, for instance, we can choose the latest five epochs or some fixed epochs as T during the training process.Based on the soft estimation μϕ i of the underlying mean µ, we further introduce the lower bound of the underlying mean µ to give correctly labeled samples which are discarded due to annotator defects or because they are hard samples a chance to be selected. Inspired by Xia et al. [39] , we give the following theorem about the lower bound. The proof is similar to Theorem 1 in [39] . Theorem 2. Let {z j } n j=1 be an observation set with mean µ and variance σ 2 . By utilize a nondecreasing function ϕ(z) = log(1 + z + z 22 ), we haveEQUATION)with probability at least 1 -1 n . Remark. This theorem defines the lower bound of the mean of an observation set. The first term in the lower bound is a robust average, which can reduce the side effects of extremum. The second term is related to the number of observed values and the variance.Based on Theorem 2, our selection criterion c i for x i is defined as:EQUATION)where n is the number of observed loss values for x i , i.e., n = R|T | as shown in Eq. ( 8), σ2i is the variance of observed loss values for x i . Since we don't know the distribution of the loss values for x i in submodels, σ2i serves as an approximation of the latent true variance. Generally speaking, the second term in Eq. ( 10) gives samples with relatively large variance a chance to be selected for training. If we use the normal average loss, as shown in Eq. ( 5), these samples may be discarded due to annotator defects or because they are hard samples. Thanks to Eq. ( 10), they can be included in the selection; if these samples consistently yield large loss values throughout the training process and across the majority of submodels, they will be discarded in subsequent epochs.By using the selection criterion in Eq. ( 10), we can sort the samples in D a and select the smallest p proportion to form D * a , which can be utilized to train f Θ R+1 . In practice, positive and negative samples are selected separately. The process of RUS is included in Algorithm 1.Note that RUS and LRD operate concurrently, and they share an identical set of R + 1 submodels which are trained simultaneously. In RUS, the R submodels {f Θr } R r=1 collaboratively refine D a , while in LRD, these submodels are used to refine D d . f Θ R+1 learns from the refined samples. 
dataset: 
 Benchmark datasets. All the methods are evaluated on 13 benchmark datasets with two kinds of noise, including five NLP datasets named Agnews, 20News, IMDb, Yelp and Amazon, four tabular datasets named Diabetes, Backdoor, Campaign and Waveform, and four image datasets named Celeba, SVHN, Fashion-MNIST (denote as F-MNIST) and CIFAR-10. Note that these datasets only have ground-truth labels, we simulate three annotators per dataset with label quality k = 0.3 through the following two methods:• Instance-dependent noise. Inspired by Zhao et al. [45] , k proportion of each dataset is used to train three sets of diverse tree-based models to act as rule-based annotators, i.e., Decision Tree, RandomForest, LightGBM. Their predictions are used as multiple sets of labels.• Class-dependent noise. The labels of the positive samples are randomly preserved at proportions of k, k + 0.1, and k + 0.2 respectively, while interchanging the labels for the remaining positive samples with those of negative samples, thus establishing three sets of labels. Note that these three sets of labels are generated independently.Real-world datasets. Experiments are also conducted on two real-world datasets: CIFAR-10N and Sentiment. Both datasets were published on Amazon Mechanical Turk for annotation.Details of these datasets and labels can be found in Appendix B.We compare our framework with various methods: (1) Single, in which a three-layer MLP is trained directly with one set of labels; (2) NN-Mjv [38] , in which a three-layer MLP is trained using the labels aggregated through majority voting;(3) HE_A and HE_M [37] , which train several individual models for each set of inaccurate labels and aggregate their outputs for predictions by averaging and maximizing; (4) CL [26] , which trains an end-to-end model with parametric source-specific transition matrices; (5) DN [10] , which exploits information from multiple sets of labels with different softmax output layers; (6) Label aggregation methods called Enhanced Bayesian Classifier Combination (EBCC) and Independent Bayesian Classifier Combination (IBCC) [22] , three-layer MLPs are trained using the labels inferred by EBCC or IBCC; (7) Weakly supervised end-to-end learner WeaSEL [27] ; (8) CoNAL [3] , which assumes that the annotation noise is attributed to two sources (common noise and individual noise), and combines these two noise by a Bernoulli random variable; (9) SLF [7] , which proposes to make the weight vectors and the confusion matrices data-dependent, and comes up with two regularization methods for the confusion matrix to guide the training process; (10) ADMoE [45] , which leverages the Mixture of Experts (MoE) architecture to encourage specialized learning from multiple noisy sources. Note that for a fair comparison, we do not use the noisy-label aware gating in ADMoE which inputs labels during both training and testing.Most of the compared methods are adopted from their respective codebases. The hyperparameters are set according to the recommendations in their papers. The only modification we made to ensure a fair comparison is to use a three-layer MLP with a hidden dimension of 128 as the backbone.For our method, as mentioned in the preliminaries, we use a three-layer MLP of hidden dimension 128 to act as shared layers, and R + 1 three-layer MLPs of hidden dimension 128 to serve as R + 1 submodels. The (R + 1)-th submodel f Θ R+1 is utilized for final prediction. There is no hyperparameter in LRD. For RUS, we set the proportion of selected samples p = 0.8, and take the 5th epoch and the latest epoch during training as the selected epochs in Eq. ( 8) . In practice, LRD-generated labels are held constant after 5 training epochs to mitigate the over-fitting issue.For all of the methods, experiments are conducted with 0.001 learning rate, 100 training epochs, and 256 batch size on MLP with hidden dimension 128 for a fair comparison. For benchmark datasets, 70% of each dataset is utilized for training, 5% for validation, and 25% for testing. For real-world datasets, we use the original training and testing datasets. We report the average AUC of the last 10 epochs as results.Main results. Comprehensive experiments are conducted on various datasets, including NLP, image, and tabular data, and we compare our framework against a variety of methods. The main results of our framework and all the compared methods on class-dependent noise and instance-dependent noise are summarized in Table 1 . As shown, our framework outperforms all the compared methods in most of the datasets, especially in the class-dependent noise setting. Our framework does not incorporate any special designs at the architectural level, it primarily acquires higher-quality data through collaborative refinement, followed by learning via a MLP, which demonstrates the effectiveness of our framework. 1 . For clarity, we only include the four competitive compared methods and the Single baseline in the figure. With the increase in label quality, all methods achieve better performance. Note that our framework consistently outperforms the compared methods in most of the experiments.Ablation study. Ablation experiments are conducted on datasets with class-dependent noise where NLP, image, and tabular data are all involved. The results are shown in Table 3 . In the ablation experiments, NN-Mjv acts as the baseline. Labels are aggregated for D d / D a by majority voting instead of LRD / RUS, without changing any other parts of the framework. We also compare the RUS method with the naive selection method (denoted as N-RUS) by replacing our selection criterion in Eq. ( 10) with the naive one in Eq. (5) . As shown in Table 3 , with the assistance of LRD and RUS, an overall improvement of 0.171 in AUC is achieved over the baseline. The results indicate that while the naive selection does work, it's not as effective as RUS. Since LRD and RUS are designed for different parts of the dataset respectively, they can be effectively combined to achieve the best results. Through these experiments, we demonstrate the effectiveness of LRD and RUS. These results also indicate that adding LRD to the baseline improves the performance more noticeably than when adding RUS. It is reasonable because LRD is designed to correct labels in D d , while RUS is designed to select samples in D a . As is illustrated before, the samples in D d are more ambiguous and the samples in D a are more likely to be correctly labeled, thus greater improvement can be achieved by effectively dealing with D d .Quality analysis of the refined data. Further experiments are conducted to explore how LRD and RUS work. Since the LRD method is designed to aggregate multiple sets of labels into a reliable one, we conduct experiments to show the label quality produced by LRD. We train our model for 5 epochs and collect the labels aggregated by the LRD algorithm on D d . Then we compare the quality of these labels with the three sets of original labels and the labels obtained through voting.The experiments are conducted on thirteen datasets mentioned above with class-dependent noise.The results are shown in Figure 2 , the average AUC of these labels is taken as the metric. LRD can consistently provide higher-quality labels. Naturally, our model trained with these labels yields better results than with the original labels.Since RUS is designed to select samples, experiments are conducted to show the quality of the selected samples. The model is trained for 5 epochs then the selected samples are fetched. RUS is compared with the following two methods: (1) selecting the same proportion of samples randomly; (2) replacing the criterion in RUS with the mean of the loss values in Eq. ( 5) to select the same proportion of samples. The proportion of selection p is set as 0.8. Since the total number of ground-truth positive samples stays the same, we compare the number of true positive samples selected by the three methods mentioned above. These experiments are repeated for 100 times and the average improvements over random selection are presented in Figure 3 . As shown in Figure 3 , on the four datasets including image, NLP, and tabular data, our proposed method has a better performance in selecting samples. Naturally, our model trained with these high-quality samples can get better results.Further verification of LRD-generated labels. The quality of LRD-generated labels is shown in the foregoing discussion in Figure 2 , whereby we demonstrate the effectiveness of LRD after training models for five epochs. We further examine LRD's performance in the early training stages. We train our model on datasets with class-dependent noise for 100 / 500 steps and examine the qualities of LRD-generated labels on D d . As shown in Table 4 , in the early stages of LRD learning (e.g., 100 steps), the LRD-generated labels already outperform the original labels. With ongoing training, label quality rises (e.g., 500 steps). When LRD-generated labels are frozen at step 100, the final average AUC on these four datasets is 0.835, which is quite close to the average AUC of 0.845 obtained by freezing LRD at step 500. These results indicate that we have considerable flexibility in choosing when to fix the refined labels. In the main results, the LRD-generated labels are frozen after training for five epochs.Cooperating with other algorithms. As mentioned before, our framework tends to collaboratively refine the dataset into a higher-quality one, which is independent of the model architecture. We simply utilize a MLP to learn from the refined dataset, which can be replaced by other algorithms. We train our model for 5 epochs, extract the refined dataset, and complement it with the original three sets of labels to form a new dataset. Then this new dataset is utilized to train SLF, CoNAL, and ADMoE in the same manner as in the main experiments. As shown in Table 5 , when cooperating with our framework, ADMoE, SLF, and CoNAL achieve improvement over their original versions, which demonstrates that our framework can cooperate well with the existing methods. 
conclusion and future work: 
 In this paper, we present a framework called Collaborative Refining for Learning from inaccurate labels (CRL), which focuses on learning a model from multiple sets of inaccurate labels. In our framework, we utilize the annotator agreement to assess the reliability of labels from multiple sets with disparities and split the dataset into two parts: where some annotators disagree and where all annotators agree. For samples where some annotators disagree, Label Refining for samples with Disagreements (LRD) is proposed to select relatively reliable labels. Based on a theoretical analysis of the relationships among multiple label sets, ground-truth labels, and model predictions, LRD identifies the most reliable label by comparing the loss values. For samples where all annotators agree, Robust Union Selection (RUS) is proposed to select trustworthy samples to form a higherquality dataset. Guided by theoretical bounds on loss values, RUS introduces a loss-based selection criterion and improves it to be more robust against potentially problematic values. Meanwhile, the This work currently focuses on the binary classification task with the existence of inaccurate labels, which is fundamental and commonly encountered in practice. This work also lays the foundation for subsequent research of multi-class classification tasks with inaccurate labels. However, in the multi-class scenario, the proposed method needs to be further adapted. For instance, an issue arises with samples where annotators disagree, particularly when all the given labels for a particular sample are incorrect in the multi-class setting. LRD tends to select one of these incorrect labels as the inferred label, negatively impacting the model's performance. In binary classification problems, this situation is naturally avoided because when annotators disagree, one of the labels must be correct. A possible solution could be to develop a more refined measure based on annotator agreement, which could be utilized to decide whether to discard these samples, using LRD or RUS.In the current approach, the samples are segmented based on the presence or absence of label disagreement, and further processes are performed on the correlated segments. We believe that the sample segmentation strategy can be further improved. To achieve this, we can introduce a new metric: the consistency rate, defined as the proportion of agreement among annotators for the same sample. By establishing a threshold for this consistency rate, we can effectively partition the dataset. Specifically, for samples with a consistency rate exceeding the threshold, RUS is applied, whereas for samples with a consistency rate below the threshold, LRD is utilized. The determination of this threshold is related to the label qualities of the dataset and the number of label sets available, which is suitable for practical applications. Since these ideas are not yet mature enough, we did not include these parts in the current manuscript. We leave these issues for future research. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: As a framework for learning from multiple sets of labels, this work does not suffer from particular ethical concerns or negative social impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The creators or original owners of assets used in the paper are properly credited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 Theorem 1Proof. Let d = [ d0 , d1 ] denote the one-hot version of ỹ, i.e., dỹ = 1 and di = 0, ∀i ̸ = ỹ. For binary cross-entropy loss function ℓ(f Θ (x), ỹ) = -1 i=0 di log(p i (x)), we consider the expected loss on noisy data:E (x,ỹ) [ℓ(f Θ (x), ỹ)] = -E (x,ỹ) 1 i=0 di log(p i (x)) = - x∈X 1 j=0 1 i=0 di log(p i (x)) p(x, ỹ = j)dx = - x∈X 1 i=0 1 j=0 di p(ỹ = j|x) log(p i (x)) p(x)dx = - x∈X 1 i=0 E[ di |x] log(p i (x)) p(x)dx. (11)Therefore, minimizing the expected loss equals to minimizing -1 i=0 E[ di |x] log(p i (x)) for each x ∈ X .For cross-entropy loss, there are constraints 1 i=0 pi (x) = 1 and 0 ≤ pi (x) ≤ 1, ∀i ∈ {0, 1}. So it can be formalized as the following optimization problem: minimize -EQUATIONUsing Lagrange multiplier method, we can derive that -EQUATIONwhere the fourth equation is due to the class-conditional noise assumption and the fifth equation is due to that each x has only one true label f * (x). Therefore, the loss value of sample (x, ỹ) isEQUATIONWhen facing with two conflicting observed labels ỹ0 , ỹ1 and two neural network f Θ * 0 ,f Θ * 1 , the losses can be written asEQUATIONThen from the relationship between these two losses ℓ(fEQUATIONwhich meansEQUATIONBecause T 0 , T 1 satisfy T 0 ii > 0.5, T 1 ii > 0.5, ∀i ∈ {0, 1}, we can get T 0 ij < 0.5 andT 0 ij < 0.5, ∀i ∈ {0, 1}, i ̸ = j. Since ỹ0 ̸ = ỹ1 , T 0 y * ỹ0 > T 1 y * ỹ1 equals to T 0 y * ỹ0 > T 1 y * (1-ỹ 0 ) , only when y * = ỹ0 , T 0 y * ỹ0 = T 0 y * y * > 0.5 > T 1 y * (1-ỹ 0 ) = T 1 y * ỹ1, which satisfy Eq. ( 17). if y * = ỹ1 ,T 0 y * ỹ0 = T 0 y * (1-y * ) < 0.5 < T 1 y * y * = T 1 y * ỹ1, which is in conflict with Eq. ( 17). Theorem 1 is proved.Proof. Let r 0 , r 1 ∈ {1, ..., R}, define ℓ r0 , ℓ r1 asℓ r0 = min(ℓ(f Θ * i (x), ỹi )), ỹi = ỹr0 , ℓ r1 = min(ℓ(f Θ * i (x), ỹi )), ỹi = ỹr1 . Then ℓ(f Θ * k (x), ỹk ) = min({ℓ(f Θ * r (x), ỹr )} R r=1) can be written asEQUATIONIf y * = ỹr0 , from Theorem 1 and Eq. ( 14), we can getℓ r0 < -log(0.5) < ℓ r1 , then ℓ(f Θ * k (x), ỹk ) = ℓ r0 which means ỹk = ỹr0 = y * . Similarly, if y * = ỹr1 , we can get ỹk = ỹr1 = y * . Corollary 1 is proved.As mentioned in Section 4.1, we use thirteen benchmark datasets including NLP, image and tabular datasets, and two real-world datasets.Benchmark datasets. The information of benchmark datasets are summarized in Table 6 . Most of them come from [14] .For image datasets, following [14] , we use ResNet18 [16] which is pretrained on the ImageNet [5] to extract embedding after the last average pooling layer. We utilize the extracted 512-dimensional embeddings as features. We define one of the multi-classes as negative and downsample the remaining classes to 5% of the total instances as positive.For NLP datasets, we use BERT [6] pretrained on the BookCorpus and English Wikipedia to extract 768-dimensional embeddings as features. As for labels, we define them in the following way: (1) Amazon and Imdb: we regard the original negative class as the anomaly class. (2) 20News and AgNews, we set one of the classes as normal and downsample the remaining classes to 5% of the total instances as anomalies.(3) Yelp: we regard the reviews of 0 and 1 stars as the positive class, and the reviews of 3 and 4 stars as the negative class.For tabular datasets, most of them comes from [14] , we directly adopt these datasets. Diabetes dataset is sampled from a dataset on Kaggle1 , we increased the difficulty of learning a model by randomly sampling positive and negative examples at different ratios.Real-world datasets. The details of real-world datasets are as follows:• CIFAR-10N: It is an image classification dataset, which consists of 50000 samples for training and 10000 for testing. We regard the 'Airplane' as positive class and others as negative class.Following [14] , we use ResNet18 [16] which is pretrained on the ImageNet [5] to extract embedding after the last average pooling layer. We utilize the extracted 512-dimensional embeddings as features.• Sentiment: It contains 5000 sentences from movie reviews extracted from the website Rotten-Tomatoes.com and whose sentiment was classified as positive or negative. This dataset is the original one in the website2 . The AUC of these three set of labels is shown in Table 7 .The implementation procedure of the proposed method CRL is shown in Algorithm 1.Algorithm 1 Collaborative Refining for Learning from inaccurate labels (CRL). In aggregation algorithms, the simplest method is Majority Voting [38] , where all the annotations are treated equally and labels are aggregated by voting. Some advanced methods go beyond majority voting. Weighted Majority Voting [20] uses a weight vector to model annotators' expertise, Max-margin Majority Voting [32] combines the concept of majority voting with the principles of margin maximization. Enhanced Bayesian Classifier Combination (EBCC) [22] utilizes the Bayesian classifier to infer truth by modeling the correlation between annotators. By viewing the true labels as a latent variable, some methods considering relations among annotators are proposed to infer true labels based on the Expectation-Maximization (EM) algorithm. The seminal works [4] (known as the DS model) assume that every annotator has class-dependent confusion when providing annotations, which is modeled by an annotator-specific noise confusion matrix. The DS model is the basis of many works for aggregating labels from different annotators with the help of EM algorithm [1, 2, 34, 43] .Input: Dataset D = {x i , {ỹ r i } R r=1 } N i=1 .In recent studies, an increasing number of studies focus on end-to-end learning. CrowdLayer [26] trains an end-to-end Deep Neural Network (DNN) with parametric source-specific transition matrices. DoctorNet [10] aims to train DNNs that exploit multiple annotators' information with different softmax output layers. HyperEnsemble [37] trains several individual models for each set of inaccurate labels (i.e., k models for k sets of labels) and combines their outputs by averaging and maximizing, referred to as HE_A and HE_M, respectively. UnionNet [36] learns a transition matrix for all multiple sets of labels together. CoNAL [3] assumes that the annotation noise is attributed to two sources (common noise and individual noise), and combines these two noises by a Bernoulli random variable. SLF [7] proposes to make the weight vectors and the confusion matrices data-dependent and comes up with two regularization methods for the confusion matrix to guide the training process. ADMoE [45] leverages the Mixture of Experts (MoE) architecture to encourage specialized and scalable learning from multiple inaccurate sources. SDM [30] employs a self-cognition module to identify both instancewise noise and annotator-wise quality and adopts a mutual-denoising module to aggregate these identifications and accordingly refine the model. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The data are processed from open-source benchmark datasets, which are readily accessible. We will consider open-sourcing the code after the paper is accepted. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://www.kaggle.com/datasets/julnazz/diabetes-health-indicators-datasethttp://fprodrigues.com//mturk-datasets.tar.gz 


Decision: This paper proposes a method for learning from multiple sets of inaccurate labels based on collaborative refinement approach. The proposed approach is interesting and novel in learning with inaccurate labels with multiple annotators. The effectiveness of the proposed method is well demonstrated in the experiments. This paper is well written. More discussion on the existing work is needed. I encourage the authors to improve their paper by reflecting the reviewers’ comments and the author response.

Reviews: review: paper_summary: paper_summary: This paper studied learning from multiple noisy labels via data refinement. It first uses the annotator agreement as an instrument to divide all samples into the samples where some annotators disagree and the samples where all annotators agree. Then, a comparative strategy is proposed to filter noise in the samples where some annotators disagree, and a robust union selection is used to select clean samples in the samples where all annotators agree. Experiments on multiple datasets show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The novelty of robust union selection is limited, since it is very similar to robust mean estimation in [1]. Could the authors clarify this?

2. As mentioned in this work, some methods have adopted the small-loss criterion to refine data [2-4]. The comparison between the proposed method and one of them is necessary, which can show the necessity of designing the new data refinement way with multiple annotators. Besides, one latest work [5] in ICML 2024 also explored this direction, I think this work should at least discuss the differences with it.

3. The proposed method seems also suitable for multi-class classification problems. Since many real-world cases are multi-class classification problems, why does this work only focus on binary classification problems?

[1] Sample Selection with Uncertainty of Losses for Learning with Noisy Labels. ICLR 2022

[2] Coupled-view deep classifier learning from multiple noisy annotators. AAAI 2020

[3] Learning from crowds with mutual correction-based co-training. ICKG 2022

[4] Coupled confusion correction: Learning from crowds with sparse annotations. AAAI 2024

[5] Self-cognitive Denoising in the Presence of Multiple Noisy Label Sources. ICML 2024
questions: questions: 1. How does the proposed method cooperate with other algorithms in Table 5?
2. Why the robust union selection is called an aggregating strategy in Line 12 and Line 62?
limitations: limitations: 1. The proposed methods need to train R + 1 submodels, which need a large amount of computing resources when R is large.
2. The proposed methods may be not adapted to the annotation sparse case, which is common in learning from crowds.
paper_summary: paper_summary: This paper introduces a framework for learning from inaccurate labels obtained from multiple annotators. It utilizes annotator agreement to assess label reliability and applies two strategies: one for samples with annotator disagreements (LRD) and another for samples where all annotators agree (RUS). In both cases, the framework uses a number of submodels equal to the number of annotators (with shared layers). The framework refines unreliable datasets into relatively reliable datasets that are used to train the final model. LRD selects reliable labels by comparing the losses of the submodels and choosing the label assigned by the submodel with the smallest loss. RUS enhances dataset quality using a different loss-based selection criterion. Experiments are conducted on class-dependent, instance-dependent, and real-world datasets.
strength_weakness: strength_weakness: 1) The absence of multiple runs or statistical measures such as standard deviation limits the robustness and reliability of the reported results.

2) Table 1 does not provide information on the quality or reliability of the annotations used in the presented setting.

3) Including additional comparative methods, such as Dawid Skene [1], Iterative Weighted Majority Voting [2], and [3], would enrich the analysis and provide a broader evaluation of the proposed approach.

[1] Dawid, A. P. and Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20– 28.
[2] Li, H. and Yu, B. (2014). Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv preprint arXiv:1411.4086.
[3]  Karger, D. R., Oh, S., and Shah, D. (2014). Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1–24.
questions: questions: - What are the advantages of using $\phi$ in equation 7 instead of directly using the loss?

- Has a comparison regarding the running time of the proposed method and the baselines been conducted?
limitations: limitations: Yes
paper_summary: paper_summary: This paper considers binary classfication from multiple sets of noisy labels, focusing on data refinement to generate clean labels. At each step of training, It proposes to first separate the dataset by whether label disagreement exists, and then tackle each subset using different methods. For the subset with disagreed labels, authors propose to follow the label with lowest lost. For the subset with all same labels, authors propose a delicated designed term to filter out unreliable data points. Thorough empirical evaluation shows good performance under various settings.
strength_weakness: strength_weakness: - Limitations such as binary classification and class-conditional noise assumption for LRD can be addressed more clearly in introduction.
- Discussions on model architecture selection can be improved. The design of using R + 1 heads with a shared backbone first appears in section 2 without any detailed introduction.
- Algorightm 1 is mentioned several times in the paper but itself is located in Appendix. This draws concern on abusing the page limit.
questions: questions: - Is it possible to lower the bar of annotator agreement? In other words, is it possible that labels with few disagreement being filtered out using RUS, so it is not neccessary to keep all labels the same before RUS?
- LRD is designed under class-conditional noise assumption, but empirically performs better for instance-dependent noise. Is there any empirical observation deserves to discuss?
- At the bottom of page 5, it is an interesting observation that relatively large variance are selected. How selection of $\phi(x)$ would change this behavior? What leads to the adoption of this function based on Taylor expansion?
limitations: limitations: Limitations are properly addressed in the paper.
paper_summary: paper_summary: This paper proposes a collaborative refining approach for learning with inaccurate labels provided by low-cost annotators, such as rule-based systems. It introduces strategies based on annotator agreement to filter out noise and enhance data quality. The method includes comparative filtering for conflicting labels and aggregation for consistent annotations, all guided by theoretical bounds on loss values. Extensive experiments on various datasets demonstrate significant improvements in learning performance despite the presence of label inaccuracies.
strength_weakness: strength_weakness: - The proposed method of this paper seems similar to some research methods explored in unreliable partial label learning. Is there any relations between the two?

- When the model is not the Bayesian optimal model, why does $\ell(f_{\theta_0^*}(x),\tilde y^0)<\ell(f_{\theta_1^*}(x),\tilde y^1)$ hold in Theorem 1?
questions: questions: Please check the weaknesses, and some minor comments are presented below:

- The reference format of the equation is inconsistent, sunch as some are written as Eq. 1, while others are written as Eq. (10).
limitations: limitations: NA
paper_summary: paper_summary: In common practical scenarios, autonomous annotators are used to create labeled datasets, reducing the dependence on manual labeling, which can be costly and time-consuming. Learning methods leverage multiple weak labels to annotate large amounts of data, though these weak labels are often noisy and imperfect. The paper presents a collaborative refining approach for learning from these inaccurate labels. To refine the data, the authors differentiate between cases where multiple annotators agree or disagree on the labels for a given sample. For samples with disagreements among annotators, the authors propose a noise-filtering method, while for samples with unanimous agreement, they suggest an aggregating method.
strength_weakness: strength_weakness: My main issue with this paper is that the problem is not well explained, there is a lack of literature, and most experiments are not conducted with real datasets. If I understand correctly, the problem addressed involves unlabeled samples and the output from multiple annotators. The goal is to obtain labels for these unlabeled samples. This problem is commonly referred to as programmatic weak supervision [1]. In programmatic weak supervision, the objective is to obtain probabilistic labels for the unlabeled samples, and models used to generate these label probabilities are commonly known as label models. Subsequently, these labels are utilized to train a classifier (end model).

In the problem formulation, it is initially stated that the aim is to obtain labels for unlabeled samples, and then (line 85) it is mentioned that the goal is to learn a classifier. What is the objective of the problem being addressed?

If the objective is to obtain labels for unlabeled samples, why aren't probabilistic labels obtained as in programmatic weak supervision?

The WRENCH library [2] collects multiple datasets with outputs from multiple annotators, along with state-of-the-art methods. Why weren't the annotators from that library used?

In the experiments, comparisons are made with methods such as EBCC, IBCC, and WeaSEL. As far as I know, EBCC and IBCC are label models, whereas WeaSEL is a joint model (label model + end model). Therefore, the objectives of these methods are not the same.

References

[1] Zhang, J., Hsieh, C. Y., Yu, Y., Zhang, C., & Ratner, A. (2022). A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433.

[2] Zhang, J., Yu, Y., Li, Y., Wang, Y., Yang, Y., Yang, M., & Ratner, A. (2021). WRENCH: A comprehensive benchmark for weak supervision. arXiv preprint arXiv:2109.11377.

Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017, November). Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB endowment. International conference on very large data bases (Vol. 11, No. 3, p. 269). NIH Public Access.

Ratner, A. J., De Sa, C. M., Wu, S., Selsam, D., & Ré, C. (2016). Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29.

Mazzetto, A., Cousins, C., Sam, D., Bach, S. H., & Upfal, E. (2021, July). Adversarial multi class learning under weak supervision with performance guarantees. In International Conference on Machine Learning (pp. 7534-7543). PMLR.

Balsubramani, A., & Freund, Y. (2015, June). Optimally combining classifiers using unlabeled data. In Conference on Learning Theory (pp. 211-225). PMLR.
questions: questions: What happens if an annotator chooses to abstain from labeling a sample?

Is it realistic for all annotators to provide the same label? In the weak supervision datasets such as WRENCH datasets, this rarely occurs.

Building on the previous question, what if one of the datasets $D_d$ or $D_a$ is empty?

What assumptions are made about the annotators?
limitations: limitations: See above

Comments: title: keeping my score
comment: Thank you again for your insightful and constructive suggestions, especially regarding renaming the baselines and adding comparative baselines related to the small-loss criterion, which are of great significance for perfecting our work. 

**Q1: How to choose the hyperparameter p in robust union selection**

In our experiments, we set this parameter to 0.8 without adjusting for a fair comparison. If you want to choose an optimal hyperparameter p for a specific dataset, it may be necessary to use cross-validation methods to assess the impact of different p values on model performance. Typically, this optimal value varies across different datasets. 

Empirically speaking, the selection of p is related to the size of the dataset, and the quality of the labels. We recommend choosing p between 0.5 and 0.8. If the label quality of the dataset is high, it can be further increased to 0.9. 

__Q2: It seems that $f\_{\Theta\_k\^\*}$ has not been defined clearly.__

As is defined in the Preliminaries, $f_{\Theta^{\*}}$ is the optimal classifier and $f_{\Theta_k}$ is the model trained with $k$-th label set. $f_{\Theta_k^\*}$ is the optimal model trained with the $k$-th label set. 

**Q3: When cooperating with other algorithms, is the refined data are regards as a new annotators? Does the new formed dataset have the same size with the original dataset?**

Yes, we treat the refined labels as contributions from a new annotator. And the size of the newly formed dataset is a littile smaller than the original, as RUS has filtered out some samples. 

**It is better to unify the terms of the proposed strategies or make them clearer in the whole paper. In Abstract and Introduction, a comparative strategy and an aggregating strategy are mentioned, while they don't appear and are not explained in the Method.** 

Thank you for your detailed suggestion. We will add a description of these two concepts in the Method to make it clearer.

**To make the compared baselines clearer, I suggest to rename baselines that are training with the results of the label model (e.g. Majority Voting, EBCC), with the type of end model (classifier). For example, If Majority Voting, EBCC baselines are training deep neural networks with the results from Majority Voting, EBCC, it is better to name them as NN-MV, NN-EBCC [6], or DL-MV, DL-EBCC [7], these names can make the readers of the whole community easily understand the focused problem.**

Thank you for your thoughtful suggestion. We are indeed planning to make that change, and in the revised versions, we will ensure that the name of the baselines is updated to the clearer version you suggested.
Thank you for addressing my questions and doubts about the paper. I have reviewed the authors' rebuttal, along with the other reviewers' comments. After careful consideration, I have decided to keep my score.
Thanks for the response of the authors. It addressed my major concerns. I decide to increase my score to 5.
Besides, there are some minor questions and suggestions:
- How to choose the hyperparameter p in robust union selection?
- It seems that $f_{\Theta_k^*}$ has not been defined clearly.
- When cooperating with other algorithms, is the refined data are regards as a new annotators? Does the new formed dataset have the same size with the original dataset?
- It is better to unify the terms of the proposed strategies or make them clearer in the whole paper. In Abstract and Introduction, a comparative strategy and an aggregating strategy are mentioned, while they don't appear and are not explained in the Method
 section.
- To make the compared baselines clearer, I suggest to rename baselines that are training with the results of the label model (e.g.  Majority Voting, EBCC), with the type of end model (classifier). For example, If Majority Voting, EBCC baselines are training deep neural networks with the results from  Majority Voting, EBCC, it is better to name them by NN-MV, NN-EBCC as [6], or by DL-MV, DL-EBCC as [7],  these names can make the readers of the whole community easily understand the focused problem.

[6] Deep learning from crowdsourced labels: Coupled cross-entropy minimization, identifiability, and regularization. ICLR 2023

[7] Transferring annotator- and instance-dependent transition matrix for learning from crowds. TPAMI 2024
We appreciate your feedback, which is valuable for improving the quality of our research. Thank you for the time and effort you have dedicated to reviewing our work.
We are sincerely grateful for your reevaluation of our paper, thanks for your time.
Thank you for thoroughly addressing my questions and clarifying my doubts. After considering the feedback from other reviewers, I have decided to maintain my score.
Thank you for answering all my questions. My doubts have been solved. After reading the comments from other reviewers and the reviewers' responses I have decided to raise my score to a 5.
Dear Reviewers,

We extend our deepest appreciation for the valuable insights and constructive suggestions you've provided for our submission to NeurIPS 2024. We are glad that reviewers appreciate the novelty and high practical interest of our work, the interesting and original analysis we have done, and the importance of data refinement that we focus on. 

Following your suggestions, we have dedicated substantial effort to address each point raised during the review process, including additional experiments as suggested. As the author-reviewer discussion phase is crucial for refining submissions, we kindly encourage you to review our rebuttal responses at your earliest convenience. Any further discussions or suggestions would be greatly appreciated. 

We hope that our efforts in addressing the concerns have resonated well with the paper's objectives and have potentially influenced your evaluation. Thank you once again for your dedication and time. We look forward to your valuable feedback.

Warm regards,

Authors of submission 16683 at NeurIPS 2024
We would like to extend our heartfelt thanks for the time you spent reviewing our responses to the comments from all reviewers and for your continued support. We highly concur with and appreciate the ideas you suggested to lower the bar of annotator agreement. We will ensure that the final version of the manuscript reflects all the valuable feedback we have received, including yours, to further enhance its quality.
I have read through other reviewers comments and corresponding authors response. I thank authors for their patient and detailed response.
For my concerns, authors kindly considers adjusting the degree of annotator disagreement. For question2, authors disentangled my conception of class-conditional and instance-dependent assumptions.
I believe this is a manuscript worth to evaluate, thus I would like to keep my score.