IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing

Authors: wenjun  zhang
liangxiao  jiang
chaoqun  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 10:24:09

Abstract: In recent years, a large number of algorithms for label integration and noise correction have been proposed to infer the unknown true labels of instances in crowdsourcing. They have made great advances in improving the label quality of crowdsourced datasets. However, due to the presence of intractable instances, these algorithms are usually not as significant in improving the model quality as they are in improving the label quality. To improve the model quality, this paper proposes an instance weighting-based bias-variance trade-off (IWBVT) approach. IWBVT at first proposes a novel instance weighting method based on the complementary set and entropy, which mitigates the impact of intractable instances and thus makes the bias and variance of trained models closer to the unknown true results. Then, IWBVT performs probabilistic loss regressions based on the bias-variance decomposition, which achieves the bias-variance trade-off and thus reduces the generalization error of trained models. Experimental results indicate that IWBVT can serve as a universal post-processing approach to significantly improving the model quality of existing state-of-the-art label integration algorithms and noise correction algorithms.

Main paper: 
 introduction: 
 Crowdsourcing eases the difficulty of obtaining training datasets for supervised learning [8] . In crowdsourcing scenarios, instances are annotated not by domain experts but by crowd workers from crowdsourcing platforms [1, 12] . While crowd workers are more cost-effective compared to domain experts, they typically possess inferior expertise and are thus more prone to assigning noisy labels [2, 13] . To mitigate the impact of noisy labels, a common practice in crowdsourcing is repeated annotating, where each instance is annotated by multiple workers to obtain multiple noisy labels [21] . Subsequently, a multitude of algorithms have been proposed to infer the unknown true label of an instance from its multiple noisy labels [3, 11, 15] . Specifically, these proposed algorithms can be roughly classified into two categories, namely label integration algorithms and noise correction algorithms. Label integration algorithms aim to integrate multiple noisy labels of each instance to infer an integrated label that is as close as possible to its unknown true label [16, 30] . Noise correction algorithms focus on identifying and correcting noise in integrated labels obtained from label integration algorithms [19, 32] . Therefore, both label integration algorithms and noise correction algorithms inevitably pay more attention to the label quality of crowdsourced datasets, i.e., the proportion of instances in crowdsourced datasets whose integrated labels are equal to the unknown true labels. Indeed, these proposed algorithms have achieved empirical success in improving the label quality of crowdsourced datasets.However, due to the presence of intractable instances, these algorithms often fall short of achieving anticipated improvements in the model quality. Here, the model quality is the proportion of instances whose predicted labels are equal to the unknown true labels when models classify test instances. On the one hand, the proportion of intractable instances in datasets tends to be low, which makes them have less impact on the label quality. For this reason, the above algorithms pay little attention to intractable instances in improving the label quality. On the other hand, the reason why intractable instances are hard to label and infer is that their attributes are ambiguous. Ambiguous attributes affect the effectiveness of models in learning classification rules from crowdsourced datasets. Therefore, intractable instances have a greater impact on the model quality compared to the label quality. Considering the fact that collecting high-quality integrated labels is ultimately aimed at training highquality models, improving the model quality should be paid more attention compared to improving the label quality in crowdsourcing.To improve the model quality, this paper proposes an instance weighting-based bias-variance trade-off (IWBVT) approach for crowdsourcing. IWBVT at first proposes a novel instance weighting method based on the idea of complementary set and entropy, which mitigates the impact of intractable instances and thus makes the bias and variance of trained models closer to the unknown true results. Subsequently, IWBVT performs probabilistic loss regressions based on the bias-variance decomposition, which achieves the bias-variance trade-off and thus reduces the generalization error of trained models. In general, the contributions of this paper can be summarized as follows:• We focus on the performance of existing algorithms in terms of the model quality and reveal that existing algorithms are not as significant in improving the model quality as they are in improving the label quality. • We propose a novel instance weighting method based on the complementary set and entropy.This new instance weighting method is more robust and can be applied to more complex crowdsourced scenarios. • We propose IWBVT to improve the model quality. IWBVT mitigates the impact of intractable instances by instance weighting and achieves the bias-variance trade-off by probabilistic loss regressions. • We demonstrate that IWBVT can serve as a universal post-processing approach to significantly improving the model quality of existing state-of-the-art label integration algorithms and noise correction algorithms. 
related work: 
 With repeated annotating, crowdsourcing collects multiple noisy labels for each instance in datasets. Subsequently, label integration is usually used to integrate multiple noisy labels to infer the unknown true label for each instance. Initiating the area of label integration, [4] leveraged an expectationmaximization (EM) algorithm to estimate a confusion matrix, which models workers and class priors in clinical diagnostics. In contrast, [20] performed majority voting based on multiple noisy labels of instances, and the class receiving the highest number of votes was determined as the integrated label. Enhancing this concept by incorporating worker reliability, [10] performed weighted majority voting by iteratively estimating worker weights and integrated labels. [23] further proposed max-margin majority voting, which integrates labels by maximizing the margin between classes.Recently, [3] augmented the label space for an instance by considering the labels of its neighbors, which distinguishes the impact of different neighbors by instance weighting. Inspired by label distribution learning [6, 17, 26] , [8] proposed multiple noisy label distribution propagation, which absorbs label distributions of neighboring instances into the label distribution of the focal instance.No matter how powerful the label integration algorithms are, a certain degree of noise is always present in integrated labels. Subsequently, noise correction has been proposed to identify and correct these noises. Initiating the area of noise correction, [19] introduced three distinct algorithms: polishing labels (PL), self-training correction (STC), and cluster-based correction (CC). PL divides datasets into subsets to train multiple models and then performs majority voting based on the models' predictions to correct the original integrated labels. STC filters the dataset into a clean set and a noise set, iteratively training models on the clean set to predict and correct instances in the noise set. CC estimates the probability of each instance belonging to each class through repeated clustering and performs weighted majority voting to correct original integrated labels. Forgoing the above three algorithms, [32] adaptively estimated the proportion of noise in datasets based on multiple noisy labels to filter out a clean set and a noise set. Recently, drawing from multi-view learning [31] , [15] used multi-view learning for correcting noise instances. They trained dual models on the attribute and multiple noisy label views of the clean set to correct instances in the noise set. [11] focused on the effect of neighboring instances on noise filtering before noise correction, utilizing multiple noisy label distributions of neighbors to identify noise instances more accurately.In essence, both label integration algorithms and noise correction algorithms aim to improve the model quality by improving the label quality. Unfortunately, due to the presence of intractable instances, these algorithms are usually not as significant in improving the model quality as they are in improving the label quality. Currently, although there exist several supervised or semi-supervised approaches focused on improving the model quality from noisy training datasets [7, 14, 28] , they are not perfectly applicable to crowdsourcing. On the one hand, they cannot utilize the multiple noisy labels specific to crowdsourcing. On the other hand, semi-supervised approaches typically assume that true labels of a few instances are known, which cannot be satisfied in crowdsourcing. In this context, we propose IWBVT, as the first universal post-processing approach to improve the model quality of both label integration algorithms and noise correction algorithms in crowdsourcing. 
notations and preliminaries: 
 Let D denote a crowdsourced dataset {(x i , L i )} N i=1 , where N represents the number of instances, and x i is the i-th instance, represented as {x i1 , . . . , x im , . . . , x iM }. Here, M signifies the dimension of attributes and x im denotes the attribute value of x i on the m-th attribute A m . L i denotes multiple noisy labels of x i , which can be represented as {l ir } R r=1 . R denotes the number of workers, l ir denotes the label of x i annotated by the r-th worker u r . l ir takes a value from {-1, c 1 , . . . , c q , . . . , c Q }, where Q denotes the number of classes, c q denotes the q-th class and -1 denotes that u r does not annotate x i . The purpose of label integration and noise correction is to infer an integrated label ŷi for x i and to minimize the error between ŷi and the unknown true label y i .Given (x i , L i ), the weight of x i is denoted by w i . Intuitively, the smaller the value of w i , the more likely that x i is an intractable instance. To estimate w i , L i is first transformed into a multiple noisy label distributionP i = {P (c q |L i )} Q q=1, where the probability P (c q |L i ) reflects the proportion of labels in L i that take the value c q . Subsequently, several representative instance weighting methods have been proposed based on P i . First, [21] proposed estimating w i by P (ŷ i |L i ), i.e., w i ∝ P (ŷ i |L i ). Take MV as an example, the probability P (ŷ i |L i ) consistently equals the maximum value in P i . This method is usually effective when Q = 2. However, when Q > 2, P (ŷ i |L i ) is not sufficient to distinguish different distributions, such as {0.5, 0.3, 0.2} and {0.5, 0.4, 0.1}.Subsequently, [27] proposed estimating w i by the entropy of P i , i.e., w i ∝ 1 Ent(Pi) , whereEQUATIONBased on the maximum entropy principle, when P i conforms more closely to the uniform distribution, the entropy Ent(P i ) increases, leading to a decrease in the corresponding weight w i . Though entropybased methods can weight instances in multi-class datasets, they still fail to distinguish some complex distributions, such as {0.4, 0.3, 0.3} and {0.4, 0.4, 0.2}. Recently, [3] proposed estimating w i by the class margin as follows:EQUATIONwhere max(P i ) and sec(P i ) denote the largest and second largest values in P i , respectively. This method focuses on the confusing classes in crowdsourced datasets. Nevertheless, it still struggles to distinguish some complex distributions, such as {0.5, 0.3, 0.1, 0.1} and {0.4, 0.2, 0.2, 0.2}.In addition to these methods, there are also a few methods that estimate w i with classification models or evolutionary algorithms [22, 29] . However, these methods have not been discussed here as their performance is affected by the selected models, loss functions, parameter settings, etc.The bias-variance decomposition is an effective way to analyze the generalization error of models.Referring to [9] , given a model f , its generalization error E f can be denoted as follows:EQUATIONwhere P (x i ) denotes the probability of selecting x i from D. bias 2 i and var i denote the bias term and the variance term, respectively. They are estimated as follows:EQUATION)EQUATIONwhere P (c q |x i ) denotes the true probability that x i belongs to c q , P (c q |f, x i ) denotes the probability that f classifies x i into c q in multiple results generated by cross-validation. Therefore, P (c q |x i ) is independent of f and is only related to D. σ 2 i denotes the noise term, which is also only related to D.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We give proofs to theorems that appear in the paper. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
approach: 
 The primary objective of IWBVT is to improve the model quality through bias-variance trade-off. However, in crowdsourcing scenarios, P (c q |x i ) can only be roughly estimated because y i is unknown and ŷi is inaccurate. To estimate the bias and variance of f as accurately as possible, IWBVT first mitigates the impact of intractable instances by instance weighting. Subsequently, to achieve the bias-variance trade-off, IWBVT learns probabilistic loss by probabilistic loss regressions.As previously mentioned, existing instance weighting methods struggle to distinguish certain complex distributions effectively. In response, IWBVT introduces a novel instance weighting method that leverages the complementary set and entropy to overcome this limitation. The innovative aspects of this method are depicted in Figure 1 . As shown in the figure, P (ŷ i |L i ) reflects the proportion of the integrated label in L i . A higher P (ŷ i |L i ) suggests more workers reach a consensus on x i , thereby indicating a decreased likelihood of x i being an intractable instance. To extend our method to multi-class datasets, we also focus on the entropy of Pi , i.e., Ent( Pi ). Here, Pi is the complementary set of {P (ŷi |L i )} in P i ({P (c q |L i )} Q q=1 ). Intuitively, workers tend to reach a consensus on a special class on tractable instances, so they should be more randomized on other classes. The entropy of Pi reflects the degree of randomization. Accordingly, our instance weighting method considers four cases, shown to the right side of the arrow in Figure 1 . Among them, the case in the upper left corner indicates that when Ent( Pi ) is fixed, a lower P (ŷ i |L i ) results in a lower w i . Conversely, the case in the upper right corner indicates that a higher P (ŷ i |L i ) results in a higher w i . The case in the lower left corner indicates that when P (ŷ i |L i ) is fixed, a lower Ent( Pi ) results in a lower w i . The case in the lower right corner indicates that the higher Ent( Pi ) results in a higher w i . To cover these cases, specifically, we estimate w i as follows:EQUATIONwhere log(Q -1) is the normalization factor. When Q = 2, we set Ent( Pi) log(Q-1) to 1. To demonstrate the superiority of our weighting method over existing methods, we calculate instance weights with each method on all the complex distributions mentioned in Section 3.1. Table 1 reports the detailed comparison results. Here, "✓" and "×" indicate whether the weighting method is effective in distinguishing the corresponding complex distribution, respectively. Empirically, the weight corresponding to the front distribution in each example should be higher than the latter. The results show that only our method can distinguish all these types of complex distributions, while existing methods cannot. By Eq. ( 6), we calculate weights of all instances as W = {w i } N i=1 . Theorem 1. When Ent( Pi ) remains constant, Eq. (6) covers w i ∝ P (ŷ i |L i ). When Q > 2 and P (ŷ i |L i ) is the maximum value in P i , Eq. ( 6) covers w i ∝ max(P i ) -sec(P i ).Proof. When Q = 2, we set Ent( Pi )/log(Q -1) to 1, so Eq. ( 6) simplifies to w i ∝ P (ŷ i |L i ). w i ∝ P (ŷ i |L i ) still holds in Eq. ( 6) when Q > 2 and Ent( Pi ) remains constant. When Q > 2 and P (ŷ i |L i ) remains constant, w i ∝ Ent( Pi ) holds. According to the maximum entropy principle, Ent( Pi ) takes its maximum value when any element of Pi is equal to 1-P (ŷi|Li)Q-1. At this point, if P (ŷ i |L i ) is the maximum value in P i , max(P i ) -sec(P i ) takes its maximum value. Conversely, when Ent( Pi ) takes its minimum value, max(P i )-sec(P i ) also takes its minimum value. Therefore, w i ∝ max(P i ) -sec(P i ) holds when Q > 2 and P (ŷ i |L i ) is the maximum value in P i . Due to the limited pages, more detailed proof of Theorem 1 is provided in Appendix A.After instance weighting, the impact of intractable instances is mitigated. Therefore, P (c q |x i ) can be calculated more accurately, and then bias and variance can be estimated more accurately. Based on this result, the bias-variance trade-off will be more effective. After instance weighting, we train a classification model f on D with W . Let C denote the label space {c 1 , . . . , c q , . . . , c Q }, in this paper, f classifies the test instance x with the bias-variance trade-off as follows:EQUATIONwhere f (c q |x) denotes the probability that x belongs to c q predicted by f . h q (x) is the prediction of the regression model h q trained on the following probabilistic loss regression task:EQUATION)where X is the attribute matrix consisting of all training instances. L q is the probabilistic loss vector for c q , which can be represented as {L 1q , . . . , L iq , . . . , L N q } T . L iq is calculated as follows:EQUATION)Theorem 2. When the probabilistic loss is defined as in Eq. ( 9), performing probabilistic loss regressions constructed by Eq. ( 8) ensures that Eq. ( 7) asymptotically achieves the bias-variance trade-off.Proof. When f is adjusted, P (c q |f, x i ) changes with f , and this change is denoted as ∆ iq . Let bias2 iand ṽar i denote the changed bias term and variance term, they can be calculated as follows:biasEQUATION)EQUATIONDue to the limited pages, more detailed derivation of Eqs. ( 10) -( 11) is provided in Appendix B. Comparing Eq. ( 10) and Eq. (11) shows that the common termsQ q=1 ∆ iq P (c q |f, x i ) and 1 2 Q q=1 ∆ 2 iq in bias 2i and ṽ ar i have opposite signs. Therefore, when improving E f , the bias and variance tend to change in opposite trends, which is known as the bias-variance dilemma. Improving E f by synergistically considering changes in both bias and variance is known as the bias-variance trade-off. According to Eqs. ( 3), (10) , and ( 11), we can get the changed Ẽf as follows:EQUATIONIn general, when c q is the true label of x i , P (c q |x i ) tends to 1, otherwise it tends to 0. However, the true label y i is unknown in crowdsourcing scenarios. After instance weighting, the impact of intractable instances is mitigated, so we assume that ŷi is equal to y i . Therefore, when c q ̸ = ŷi , ∆ iq P (c q |x i ) tends to 0. When c q = ŷi , since the probability terms P (x i ) and P (c q |x i ) in Eq. ( 12) are non-negative, Ẽf is guaranteed to be less than E f as long as ∆ iq is greater than 0. In summary, the key factor of the bias-variance trade-off is ∆ iq (c q = ŷi ). To make ∆ iq (c q = ŷi ) greater than 0, the following optimization task can be constructed:EQUATIONHere, maximizing f (ŷ i |x i ) ensures that ∆ iq (c q = ŷi ) is greater than 0, while the constraint ensures that the prediction of f will be ŷi . Then, according to the Lagrange multiplier, the Lagrange function can be constructed as follows:EQUATIONwhere λ ≥ 0. For simplicity, L ′ (x i ) can be further constructed as follows:EQUATION)Since the probability max cq∈C∧cq̸ =ŷi f (c q |x i ) ≥ 0, so L(x i ) ≥ L ′ (x i ). Ultimately, Eq. ( 13) can be optimized to achieve a better result by maximizing L ′ (x i ). At the same time, since λ ≥ 0, the value of L ′ (x i ) is positively correlated with the following difference:EQUATIONAccording to Eq. ( 9), through probabilistic loss regressions, when c q = ŷi , f (c q |x) + h q (x) in Eq. ( 7) tends to 1. Conversely, when c q ̸ = ŷi , f (c q |x) + h q (x) tends to 0. Therefore, Eq. ( 7) is effective in maximizing the difference Eq. ( 16). Ultimately, Theorem 2 is proved.The whole learning process of IWBVT is shown in Algorithm 1. In Algorithm for i = 1 to N do 2:Calculate the weight w i of x i by Eq. ( 6); 3: end for 4: Train the classification model f on D with W = {w i } N i=1 ; 5: for q = 1 to Q do 6:for i = 1 to N do 7:Calculate the probabilistic loss L iq by Eq. ( 9); Construct the regression task T q by Eq. ( 8); 10:Learn the regression model h q on T q ; 11: end for12: return classification model f , H = {h 1 , h 2 , . . . , h Q }.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: This paper fully discloses all the information needed to reproduce the main experimental results of the paper. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 To validate the effectiveness of IWBVT, we conduct a series of experiments on the whole 34 simulated and 2 real-world crowdsourced datasets published on the Crowd Environment and its Knowledge Analysis (CEKA) [33] platform. First, we illustrate the setup of our experiments, including comparison algorithms and their parameter settings. Next, we describe the simulation process and present the simulated experimental results in terms of the model quality. Finally, to further validate the strength of IWBVT, we analyze the experimental results of comparative experiments and ablation experiments on real-world crowdsourced datasets.We select seven state-of-the-art algorithms for experiments, including: majority voting (MV) [20] , iterative weighted majority voting (IWMV) [10] , label augmented and weighted majority voting (LAWMV) [3] , multiple noisy label distribution propagationg (MNLDP) [8] , adaptive voting noise correction (AVNC) [32] , multi-view-based noise correction (MVNC) [15] , and neighborhood weighted voting-based noise correction (NWVNC) [11] . Among them, MV is the simplest label integration algorithm and is used as a baseline for all algorithms. IWMV, LAWMV and MNLDP are three state-of-the-art label integration algorithms. AVNC, MVNC and NWVNC are three state-ofthe-art noise correction algorithms. They are used to validate the effectiveness of IWBVT for label integration and noise correction. All these algorithms are implemented based on the CEKA platform and their parameter settings are consistent with the corresponding published papers. AVNC, MVNC and NWVNC are all performed based on integrated labels inferred by MV. Besides, we use linear regression as h q in IWBVT. All experiments are conducted on a Windows 10 machine with an AMD Athlon(tm) X4 860K Quad Core Processor @ 3.70 GHz and 16 GB of RAM.Datasets and simulation process. We conduct our simulated experiments on all simulated datasets published on the CEKA platform. These datasets come from a wide variety of application domains and represent plentiful crowdsourcing scenarios. Considering that the selected label integration [25] platform to replace all missing values. Specifically, ReplaceMissingValues uses the mean of numerical attribute values or the modes of the nominal attribute values from the available data to replace missing values. Subsequently, to generate multiple noisy labels for each instance, we simulate the crowdsourcing process for these datasets. First, we randomly generate five workers whose label quality follows a normal distribution with N(0.65, 0.05 2 ). The label quality of a worker reflects the probability that the noisy label annotated by this worker to an instance is the same as this instance's unknown true label. Then, we hide true labels and use these simulated workers to annotate datasets. Finally, we use the selected algorithms to infer integrated labels for these datasets. For each simulation, we evaluate the original model quality and the corresponding model quality improved using IWBVT through stratified 10-fold cross-validation. Here, we use Naive Bayes (NB) [5] as the target model. The above processes are repeated ten times independently for each algorithm on each dataset.Experimental results. Table 2 shows the detailed model quality (%) comparisons of each algorithm on each dataset, respectively. The columns ORI and IWBVT correspond to the original model quality and the model quality using IWBVT, respectively. The symbols • and • in the table denote the model quality has a statistically significant improvement or degradation using our proposed IWBVT with a corrected paired two-tailed t-test with the significance level α = 0.05 [18] , respectively. Besides, the averages and the Win/Tie/Lose (W/T /L) values are summarized at the bottom of Table 2 . The W/T /L implies that when improving the original model quality, IWBVT wins on W datasets, ties on T datasets, and loses on L datasets. These experimental results validate the effectiveness of IWBVT, and we can summarize the following highlights:• The average model quality of MV using IWBVT on 34 datasets is 79.03%, which is higher than the original model quality of all selected algorithms. This demonstrates both the limitations of label integration algorithms or noise correction algorithms and the effectiveness of IWBVT in improving the model quality. • The average model quality of IWMV (78.94%), LAWMV (80.22%), and MNLDP (80.24%) using IWBVT are higher than the original results of these state-of-the-art label integration algorithms. This demonstrates that IWBVT is still effective for more sophisticated label integration algorithms in improving the model quality. • The average model quality of AVNC (80.77%), MVNC (80.44%), and NWVNC (80.54%) using IWBVT are also higher than the original results of these state-of-the-art noise correction algorithms. This demonstrates that IWBVT can serve as a universal post-processing approach to significantly improving the model quality. • Based on the t-test results, the number of datasets in which IWBVT wins significantly (W ) is always much higher than the number of datasets in which it loses significantly (L) for all algorithms. This strongly demonstrates the effectiveness and robustness of IWBVT.Datasets. To demonstrate the robustness of IWBVT, the above simulation process pays more attention to common factors of crowdsourcing. However, training models on real-world datasets may also be affected by other factors, such as sparsity and annotating bias. To verify the effectiveness of IWBVT in real-world crowdsourced scenarios, we also construct our experiments on two widely used real-world crowdsourced datasets, Leaves and Income, published on the CEKA platform [34] .Here, Leaves and Income are selected through the online platform Amazon Mechanical Turk (AMT).Leaves is annotated by 83 workers and each instance is annotated by 10 workers. There are 6 classes, 384 instances, 3840 labels, 64 numeric attributes, and 0 missing values in Leaves. Income is annotated by 67 workers and each instance is also annotated by 10 workers. There are 2 classes, 600 instances, 6000 labels, 10 nominal attributes, and 0 missing values in Income. We only evaluate the original model quality and the corresponding model quality using IWBVT by stratified 10-fold cross-validation one time because real-world datasets do not have a random simulation process.Experimental results. Figure 2 shows the model quality (%) comparisons of MV, IWMV, LAWMV, MNLDP, AVNC, MVNC and NWVNC before and after using IWBVT on Leaves and Income. With Figures 2a and 2b , we can find that IWBVT can also serve as a universal post-processing approach to significantly improving the model quality in real-world crowdsourced scenarios. Besides, we can also find the original model quality of several state-of-the-art algorithms is even lower than the original model quality of MV. These results once again demonstrate the limitation of label integration and noise correction in improving the model quality.Ablation experiment. The above results only demonstrate the effectiveness of IWBVT as a whole, yet they do not delineate the contributions of its two key components: instance weighting and bias-variance trade-off. In IWBVT, instance weighting is used to mitigate the impact of intractable instances to make the bias and variance of trained models closer to the unknown true results. Therefore, to independently verify the effectiveness of instance weighting, we first observe the bias and variance of trained models before and after instance weighting. To estimate the bias and variance, referring to [24] , NB is tested on Leaves and Income by ten runs of three-fold cross-validation. Figure 3a shows the bias and variance comparisons before and after using instance weighting on Leaves and Income. In Figure 3a , object 1 denotes the estimation results of NB trained directly with true labels. object 2 denotes the estimation results of NB trained with integrated labels inferred by MV. The conditions for object 3 and object 2 are the same, besides considering instance weighting. As can be seen in Figure 3a , the bias and variance of models trained only with integrated labels are usually higher than unknown true results. When instance weighting is introduced, both the bias and variance of models tend to be closer to unknown true results. These results demonstrate that instance weighting successfully corrects the bias and variance of trained models by mitigating the impact of intractable instances. Therefore, IWBVT performs the instance weighting before the bias-variance trade-off, which is more effective in improving the generalization performance of trained models.Additionally, we also analyze the effectiveness of another component, the bias-variance trade-off, in improving model quality. Similarly, we still fix the label integration algorithm to be MV, and then introduce the instance weighting and bias-variance trade-off individually to observe their influence in improving model quality. Figure 3b shows the model quality (%) comparisons of MV using IWBVT or its components on Leaves and Income. In Figure 3b , object 1 denotes the model quality of MV, object 2 denotes the model quality of MV using the bias-variance trade-off, object 3 denotes the model quality of MV using the instance weighting, and object 4 denotes the model quality of MV using the whole IWBVT. As can be seen in Figure 3b , both the instance weighting and the bias-variance trade-off effectively improve the model quality of MV. This demonstrates that the two components of the IWBVT are both effective. Moreover, the model quality of MV using the whole IWBVT is the highest on both Leaves and Income, which suggests that it is reasonable for IWBVT to utilize both components at the same time. In addition, in Figure 3b , we can also find that the instance weighting is more effective on Leaves, while the bias-variance trade-off is more effective on Income. This is because the average label quality of Leaves is low. Instance weighting helps to identify rare instances that are inferred correctly, and therefore has a greater impact on Leaves. However, the average label quality of Income is high, and more instances can be correctly inferred than in Leaves. Therefore, the bias and variance are estimated closer to the unknown true values, so the bias-variance trade-off is more effective on Income.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification:This paper provides open access to the data and code, and provides a document to guide readers in reproducing all experimental results in supplemental material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion and future work: 
 To improve the model quality of models trained on crowdsourced datasets, we propose a universal post-processing approach called IWBVT. IWBVT first mitigates the impact of intractable instances by instance weighting to make the bias and variance of trained models closer to the unknown true results. Then, IWBVT reduces the generalization error of trained models by the bias-variance trade-off.Experimental results suggest that IWBVT can significantly improve the model quality of existing state-of-the-art label integration algorithms and noise correction algorithms.Though the above experimental results sufficiently demonstrate the effectiveness of IWBVT, some anomalies are found in experiments. Table 2 shows that IWBVT degrades the model quality on a few datasets such as labor and lymph. The datasets such as labor and lymph contain some numerical attributes that are significantly higher in magnitude than other attributes. However, the linear regression chosen for IWBVT in experiments is not robust to regression tasks constructed for these datasets. Therefore, in the future, we will further improve the robustness of IWBVT to make trained models insensitive to anomalous attributes.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: This paper specifies all the training and test details necessary to understand the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We explain in detail the experimental metrics in the paper and provide the results of the significance tests.Guidelines:• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The new approach proposed in this paper helps to train models more efficiently from crowdsourced datasets. There are no negative societal impacts. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We used the datasets and algorithmic implementations published by the CEKA platform, which is described in detail in the paper. The license and terms of use explicitly are properly respected in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification:This paper provides open access to the data and code, and provides a document to guide readers in reproducing all experimental results in supplemental material. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper did not select new crowdsourced datasets. The datasets used for experiments are publicly available and their information is described in detail in the paper.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the proof of Theorem 1, according to the maximum entropy principle, we get that Ent( Pi ) takes its maximum value when any element of Pi is equal to 1-P (ŷi|Li)Q-1. Here, we provide its detailed derivation. First, by the definition of Pi , we simplify it to Pi = {P q } Q-1 q=1 with Q-1 q=1 P q = 1 -P (ŷ i |L i ). Then, we can find the maximum value of Ent( Pi ) as follows:EQUATION)Then, according to the Lagrange multiplier, the Lagrange function L( Pi ) can be constructed as follows:EQUATION)Now, to obtain the maximum value of L( Pi ), we can take the partial derivative of L( Pi ) concerning P q , and set this derivative equal to zero as follows:EQUATIONAccording to Eq. ( 19), we can obtain P q = 2 λ-1 . Bringing this result into the constraints, we can obtainQ-1 q=1 P q = (Q-1)2 λ-1 = 1-P (ŷ i |L i ). Therefore, it is clear that P q = 2 λ-1 = 1-P (ŷi|Li) Q-1.Finally, bringing P q = 1-P (ŷi|Li)Q-1into Eq. ( 17), we can calculate the maximum value of Ent( Pi )as (1 -P (ŷ i |L i )) log Q-1 1-P (ŷi|Li) .Appendix B Derivation of Eqs. (10) - (11) Due to the limited pages, Eqs. ( 10) - (12) in the main text only give the derived results. Here, we provide their detailed derivation. First, when f is adjusted, P (c q |f, x i ) changes with f , and this change is denoted as ∆ iq . Then, we bring ∆ iq into Eq. ( 4) and Eq. ( 5), respectively. The following derivation can be obtained:EQUATIONused reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: This paper proposes a new instance weighting-based approach to improving the quality of models trained on datasets collected through crowdsourcing, and theoretically demonstrates that an appropriate bias-variance trade-off is achieved. 
The paper makes a significant technical contribution to a major problem in this topic, and the issues raised by the reviewers have been appropriately addressed, and further enhanced the completeness of this work.

Reviews: review: paper_summary: paper_summary: The paper studies the problem of improving quality of models trained on datasets collected through crowdsourcing. The authors propose an approach (IWBVT) that post-processes data after crowdsourcing with the goal to mitigate the impact of intractable instances by means of instance weighting. As a result, the bias and variance of trained models becomes closer to the unknown true labels. It is proven that the novel method reduces the generalization error of trained models by the bias-variance trade-off. The paper also contains extensive experimentation that demonstrates: IWBVT significantly improve the model quality of existing label integration algorithms and noise correction algorithms.
strength_weakness: strength_weakness: -	Formalization of results 

-	Clearness of explanation in some points behind the novel approach

(see Questions)
questions: questions: - I do not understand the message behind Figure 1. How should I tract the 4 distributions on the right side of the arrow?

- Lines 164 – 165: “it can be verified that Eq. (6) can distinguish all complex distributions we showcased in Section 3.1. By..” It is hard to assess the value of this claim, because it is not clear how broad complex distributions are covered. Is it just 2 among 100? Among 1000 possible? Any quantifiable / qualifiable ways to measure the distinguish power for complex distributions?

- Line 166 “Theorem 1. In some special cases,” Please, specify cases in the theorem. Otherwise, I would not agree that this might be a theorem statement. The definition of the cases must be contained outside of the proof. Right now, I can treat the theorem as always true, because “some special cases” might be = \emptyset .

- Similar, Line 185 in Theorem 2: “Eq. (8) helps Eq. (7)”… The word “help” does not have formal mathematical definition, while Theorem is a mathematical instrument.
limitations: limitations: the authors adequately addressed the limitations
paper_summary: paper_summary: This paper proposes a novel label integration method called IWBVT for crowdsourcing by using a weighting method and probabilistic loss regressions to improve the model quality. The main problem this paper solves is the model quality caused by the presence of the intractable instances. The paper is well organized and the writing is smooth. The method proposed in the paper has improved the model quality, thus boosting the performance on simulation and real-world datasets. The authors also provide theoretical proof to deepen the understanding of the proposed method.
strength_weakness: strength_weakness: 1. Compared with real-world dataset, experimental results on a few simulated datasets doesn’t improve (e.g., MV for breast-cancer/breast-w datasets). Furthermore, the number of loss for MV and IWMV is more than that for other methods in Table 1. I suggest adding some discussion about this phenomenon.
2. Main experiments adopt significance level to indicate the performance for each method. How to decide the value of alpha and what is the influence? More detail will be helpful.
questions: questions: Please refer to weakness.
Other questions: 
1. In the experimental section, why are methods like significance testing used instead of direct comparison?
limitations: limitations: yes.
paper_summary: paper_summary: This paper explores methods to enhance the training of machine learning models using crowdsourced datasets and proposes a novel post-processing approach called IWBVT. The proposed IWBVT first performs instance weighting based on the entropy of the complementary label distribution. Subsequently, it employs a bias-variance trade-off to minimize the generalization error of trained models. Extensive experiment results demonstrate that IWBVT significantly improves the model quality of existing state-of-the-art label integration and noise correction algorithms.
strength_weakness: strength_weakness: 1. More detailed explanations about experimental results are desired. For example, in Figure 3b, instance weighting significantly improves model quality on the Leaves dataset, whereas the bias-variance trade-off is more effective on the Income dataset. Understanding the properties of the datasets that lead to these differences would help clarify the conditions under which the proposed approach is most effective.
2. The experiments in this paper primarily focus on the robustness of IWBVT across various label integration and noise correction algorithms and different datasets. To further validate the effectiveness of IWBVT, it would be beneficial to observe its performance on a range of different models.
3. More limitations of IWBVT should be discussed. These include its robustness in the presence of extremely noisy labeling, the computational complexity when handling very large datasets, and potential challenges in real-world applications.
questions: questions: 1.	According to Figure 3b, when improving model quality, why is instance weighting more effective on the Leaves dataset while the bias-variance trade-off is more effective on the Income dataset?
2.	How does IWBVT perform on different machine learning models? Demonstrating that IWBVT can improve the quality of various machine learning models would further validate its effectiveness.
3.	How does IWBVT perform in more complex crowdsourcing scenarios, such as those with extremely noisy labeling or very large datasets? Discussing the potential limitations of IWBVT in these contexts can help enhance its overall effectiveness and applicability.
limitations: limitations: Please refer to the Weaknesses.

Comments: title: Response for Rebuttals.
comment: As the discussion period deadline nears, we would be deeply appreciative if you could kindly review our rebuttal and let us know if we have addressed your concerns. We’re more than happy to continue the conversation if you have any further questions. Thank you very much for your time and consideration.
Thank you for the helpful response that addressed my concern.
I am satisifed with the authors' rebuttal and keep on original scoring and confidence.