IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing
Authors: Wenjun Zhang
Liangxiao Jiang
Chaoqun Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Crowdsourcing
Instance weighting
Bias-variance decomposition
Bias-variance trade-off
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: This paper proposes an instance weighting-based bias-variance trade-off (IWBVT) approach to improving the model quality.

Abstract: In recent years, a large number of algorithms for label integration and noise correction have been proposed to infer the unknown true labels of instances in crowdsourcing. They have made great advances in improving the label quality of crowdsourced datasets. However, due to the presence of intractable instances, these algorithms are usually not as significant in improving the model quality as they are in improving the label quality. To improve the model quality, this paper proposes an instance weighting-based bias-variance trade-off (IWBVT) approach. IWBVT at first proposes a novel instance weighting method based on the complementary set and entropy, which mitigates the impact of intractable instances and thus makes the bias and variance of trained models closer to the unknown true results. Then, IWBVT performs probabilistic loss regressions based on the bias-variance decomposition, which achieves the bias-variance trade-off and thus reduces the generalization error of trained models. Experimental results indicate that IWBVT can serve as a universal post-processing approach to significantly improving the model quality of existing state-of-the-art label integration algorithms and noise correction algorithms.

IWBVT: Instance Weighting-based Bias- Variancr Trade-off for Crowdsourcing
Wenjun Zhang
Liangxiao Jiang"
School of Computer Science China University of Geosciences
School of Computer Science
China University of Geosciences
Wuhan 430074, China w wjzhang@eug.edi www.:euwu.eduu....m edu. cn
Wuhan 430074, China 1 wewgggduuguuuu......
Chaoqun Li
School of Mathematics and Physics China University of Geosciences Wuhan 430074, China chqli@cug edu. cn
Abstract
In recent years, large number of algorithms for label integration and noise correction have been proposed to infer the unknown true labels of instances in crowdsourcing. They have made great advances in improving the label quality of crowdsourced datasets. However, due to the presence of intractable instances, these algorithms are usually
as significant improving the model quality as they are in improving the label quality. To improve the model quality, this paper proposes an instance weighting based vias-variance trade-off (IWBVT) approach. IWBVT at first proposes novel instance weighting method based on the com- plementary set and entropy, which mitigates the impact of intractable instances and thus makes the bias and variance of trained models closer to the unknown true results. Then, IWBVT performs probabilistic loss regressions based on the bias-variance decomposition, which achieves the bias-variance trade-off and thus reduces the generalization error trained models. Experimental results indicate that IWBVT can serve universal processoc approach significantly improving the model quality of existing state-of-the-arr label integration algo- rithms and noise correction algorithms. Our codes and datasets are available at https//ggttuuuooom///m///ggiizzg
Introduction
Crowdsourcing eases the difficulty of obtaining training datasets for supervised learning (8) In crowdsourcing scenarios, instances are annotated not by domain experts but by crowd workers from crowdsoureing platforms [2].. While crowd workers are more cost-effective compared to domain experts, they typically possess inferior expertise and are thus more prone to assigning noisy labels 2 113. To mitigate the impact of noisy labels, a common practice in crowdsourcing repeated annotating, where each instance is annotated by multiple workers to obtain multiple noisy labels (2T). Subsequently, multitude of algorithms have been proposed infer the unknown true label of an instance from its multiple noisy labels [3] 15)
"Corresoondig author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Specifically, these proposed algorithms can be roughly classified into two categories, namely label integration algorithms and noise correction algorithms. Label integration algorithms aim to integrate multiple noisy labels of each instance to infer an integrated label that is as close as possible to its unknown true label [T6] Noise correction algorithms focus on identifying and correcting noise in integrated labels obtained from label integration algorithms (19 32]. Therefore, both label integration algorithms and noise correction algorithms inevitably pay more attention to the label quality of crowdsourced datasets, i.e., the proportion of instances in crowdsourced datasets whose integrated labels are equal to the unknown true labels. Indeed, these proposed algorithms have achieved empirical success in improving the label quality of crowdsourced datasets.
However, due to the presence of intractable instances. these algorithms often fall short of achieving anticipated improvements in the model quality. Here, the model quality is the proportion of instances whose predicted labels are equal to the unknown true labels when models classify test instances. On the one hand, the proportion of intractable instances in datasets tends to be low, which makes them have less impact on the label quality. For this reason, the above algorithms pay little attention to intractable instances in improving the label quality. On the other hand, the reason why intractable instances are hard to label and infer is that their attributes are ambiguous. Ambiguous attributes affect the effectiveness of models in learning classification rules from crowdsourced datasets. Therefore, intractable instances have a greater impact on the model quality compared to the label quality. Considering the fact that collecting high-quality integrated labels is ultimately aimed at training high- quality models, improving the model quality should be paid more attention compared to improving the label quality in crowdsoureing.
To improve the model quality, this paper proposes an instance weighting- bias-variance trade-off (TWBVT) approach for crowdsourcing. IWBVT at first proposes novel instance weighting method based on the idea of complementary set and entropy, which mitigates the impact of intractable instances and thus makes the bias and variance of trained models closer to the unknown true results. Subsequently, IWBVT performs probabilistic loss regressions based on the bias-variance decomposition, which achieves the bias-varianc trade-off and thus reduces the generalization error of trained models. In general, the contributions of this paper can be summarized as follows:
We focus on the performance of existing algorithms in terms of the model quality and reveal that existing algorithms are not as significant in improving the model quality as they are in improving the label quality.
We propose a novel instance weighting method based on the complementary set and entropy. This new instance weighting method is more robust and can be applied to more complex crowdsourced scenarios.
We propose IWBVT to improve the model quality. IWBVT mitigates the impact of in- tractable instances by instance weighting and achieves the bias-variance trade-off by proba- bilistic loss regressions.
We demonstrate that IWBVT can serve as universal post-processing approach to signifi- cantly improving the model quality of existing state-of-the-art label integration algorithms and noise correction algorithms.
2 Related work
With repeated annotating, crowdsourcing collects multiple noisy labels for each instance in datasets. Subsequently, label integration is usually used to integrate multiple noisy labels to infer the unknown true label for each instance. Initiating the area of label integration, (4) leveraged an expectation- maximization (EM) algorithm to estimate a confusion matrix, which models workers and class priors in clinical diagnostics. In contrast, (20) performed majority voting based on multiple noisy labels of instances, and the class receiving the highest number of votes was determined as the integrated label. Enhancing this concept by incorporating worker reliability, [10] performed weighted majority voting by iteratively estimating worker weights and integrated labels. (23) further proposed max-margin majority voting, which integrates labels by maximizing the margin between classes. Recently, 3 augmented the label space for an instance by considering the labels of its neighbors, which distinguishes the impact of different neighbors by instance weighting. Inspired by label distribution learning 66] 17 26). (8] proposed multiple noisy label distribution propagation, which absorbs label distributions of neighboring instances into the label distribution of the focal instance.

---

No matter how powerful the label integration algorithms are, certain degree of noise is always present integrated labels. Subsequently, noise correction has been proposed to identify and correct these noises. Initiating the area of noise correction, (19] introduced three distinct algorithms: polishing labels (PL.) self-training correction (STC), and cluster-based correction (CC). PI, divides datasets into subsets to train multiple models and then performs majority voting based on the models" predictions to correct the original integrated labels. STC filters the dataset into a clean set and a noise set, iteratively training models on the clean set to predict and correct instances in the noise set. CC estimates the probability of each instance belonging to each class through repeated clustering and performs weighted majority voting correct original integrated labels. Forgoing the above three algorithms, (32) adaptively estimated the proportion of noise in datasets based on multiple noisy labels to filter out clean set and noise set. Recently, drawing from multi-vies learning (3T), (15] used multi-view learning for correcting noise instances. They trained dual models on the attribute and multiple noisy label views of the clean set correct instances in the noise set. [11] focused on the effect of neighboring instances on noise filtering before noise correction, utilizing multiple noisy label distributions of neighbors identify noise instances more accurately.
In essence, both label integration algorithms and noise correction algorithms aim to improve the model quality by improving the label quality. Unfortunately, due to the presence of intractable instances, these algorithms are usually not as significant in improving the model quality as they are in improving the label quality. Currently, although there exist several supervised or semi-supervised approaches focused on improving the model quality from noisy training datasets [7] 28), they are not perfectly applicable to crowdsourcing. On the one hand, they cannot utilize the multiple noisy labels specific to crowdsourcing. On the other hand, semi-supervised approaches typically assume that true labels of few instances are known, which cannot be satisfied in crowdsourcing. In this context, we propose IWBVT, as the first universal post- poot-prcessing approach to improve the model quality of both label integration algorithms and noise correction algorithms in crowdsourcing.
3 Notations and preliminaries
Let D denote a crowdsourced dataset {(x,, L:)] where N represents the number of instances,
and æ; is the i-th instance, represented as {raI, ..... Fiims..... TTMM }. Here, M signifies the dimension of attributes and Zim denotes the attribute value of æ; on the m-th attribute Am- L, denotes multiple noisy labels of æ;, which can be represented as {4,,
R denotes the number of workers, 4,, denotes the label of x; annotated by the r-th worker uy. lir takes a value from (1,,c)..... Cg. eq }. where Q denotes the number of classes, Gy denotes the q-th class and - denotes that u, does not annotate ;; The purpose of label integration and noise correction is to infer an integrated label ÿt for 2; and to minimize the error between 9; and the unknown true label yt-
3.1 Instance weighting for crowdsoureing
Given (xr, L.), the weight of æ; is denoted by w, Intuitively, the smaller the value of w;, the more likely that æ; is an intractable instance. To estimate w;. L; is first transformed into multiple noisy where the probability P(c) reflects the proportion of labels in L; that take the value cy. Subsequently, several representative instance weighting methods have been proposed based on P. First, [21] proposed estimating w; by P(ù. L.), i.c., wi x P(ù L.). Take MV as an example, the probability P L.) consistently equals the maximum value P.. This method is usually effective when Q 2. However, when Q 2, P( L.) is not sufficient to distinguish different distributions, such as {0.5, 0.3, 0.2] and (0.5, 0.1.0.1).
label distribution P: (P(c, L.) y==1, {
Subsequently, (27) proposed estimating w; by the entropy of P., i.e., En((P,) where
Ent P.) = Σ P(cy L.) log La)
Based on the maximum entropy principle, when P. conforms more closely to the uniform distribution, the entropy (n.( increases, leading to decrease in the corresponding weight w; Though entropy- based methods can weight instances in multi-class datasets, they still fail to distinguish some complex distributions, such as (0.4, 0.3, 0.3) and (0.4, 0.4, 0.2}. Recently, 3 proposed estimating w; by the
3

---

La)
probalility
Ent(PP)
class
Figure 1: The illustration of our new instance weighting method.
class margin as follows:
Wy X max P.)) sec (.),,
(2)
where max(P:) and sce( denote the largest and second largest values in P., respectively. This method focuses on the confusing classes in crowdsourced datasets. Nevertheless, it still struggles to distinguish some complex distributions, such (0.5, 0.3, 0.1, 0.13 and (0.4, 0.2,
In addition to these methods, there are also a few methods that estimate wi with classification models evolutionary algorithms (9] However, these methods have not been discussed here their performance is affected by the selected models, loss functions, parameter settings, etc.
3.2 Bias-variance decomposition
The bias-variance decomposition is an effective way analyze the generalization error of models. Referring to 19, given a model f, its generalization error Ey can be denoted as follows:
E Σ P(x.) (bias var; of).
where P(x,) denotes the probability of selecting æ; from D. bias? and var, denote the bias term and the variance term, respectively. They are estimated follows:
Q
bias?
1
P(cylas)) P(clls, ax))
2
1=1
var;
1
- Σ 4=1
P(cyls, 22.)
(5)
2
where P(cylan (z;)) denotes the true probability that æ; belongs to Cg. P(cy f ax)) denotes the probability
that I classifies æ; into cg in multiple results generated by cross-validan tross-validation. Therefore, P(cy ((c.la:)) is independent of f and is only related to D. of denotes the noise term, which is also only related to D.
4 Approach
The primary objective of IWBVT is to improve the model quality through varianae trade-off. However, in crowdsourcing scenarios, P(cyl az;) can only be roughly estimated because y! is unknown and ÿ; is inaccurate. To estimate the bias and variance of f as accurately as possible, IWBV VT first mitigates the impact of intractable instances by instance weighting. Subsequently, to achieve the bias- virs-nariance trade-off, IWBVT learns probabilistic loss by probabilistic loss regressions.
4.1 Instance weighting
As previously mentioned, existing instance weighting methods struggle to distinguish certain complex distributions effectively. In response, IWBVT introduces novel instance weighting method that leverages the complementary set and entropy to overcome this limitation. The innovative aspects of this method are depicted in Figure As shown in the figure. P(ğ. (L)) reflects the proportion

---

Table 1: The comparison results of instance weighting methods on complex distributions.
Complex distributions
P(ù. L.)
max(P.) (P.)
P(). L,) (o 1)
P))
Bnt(PP))
[0.5, 0.3, 0.2) & 10.5, 0.10
0,50 & 0.50
0.67 & 0.73
0.20 & 0.10 /
0.70 & 0,52 /
[0.4. 0.3, 0.3) & (0.4, 0.4, 0.2)
0.40 & 0.40 X
0.64 & 0.66 x
0.10 & 0.00 v
0.58 & 0.53 v
[0.5. 0.3, 0.1. 0.1) (0.4, 0.2, 0.2, 0.2)
0.50 0.40 V
0.59 & 0.52
0.20 & 0.20
0.62 & 0.58 v
of the integrated label in L, A higher P(ù: L suggests more workers reach a consensus on Z;. thereby indicating a decreased likelihood of æt being an intractable instance. To extend our method to multi-class datasets, we also focus on the entropy of P. i.e. (.)(P.). Here, P. is the complementary set of (P(u: L:)} in P. ((p (=))==)) Intuitively, workers tend to reach consensus on special class on tractable instances, so they should be more randomized on other classes. The entropy of P. reflects the degree of randomization. Accordingly, our instance weighting method considers four
cases, shown to the right side of the arrow in Figure 1 Among them, the case in the upper left corner
indicates that when Enl( is fixed, lower P(ù; L results in lower w;. Conversely, the case in
the upper right corner indicates that a higher P( L results in a higher wis The case in the lower left corner indicates that when ( L;) is fixed, lower Ent results in a lower wy. The case in the lower right corner indicates that the higher Ent(P.) results in a higher wy To cover these cases, specifically, we estimate w; as follows:
(.))
W; = Li)
log(Q - -1
where log(Q - 1) is the normalization factor. When Q = 2, we set
Ent( (P.) [Qg(Q-1) to
To demonstrate the superiority of our weighting method over existing methods, we calculate instance weights with each method on all the complex distributions mentioned in Section 3.1 Table I reports the detailed comparison results. Here, "v" and "x" indicate whether the weighting method is effective in distinguishing the corresponding complex distribution, respectively. Empirically, the weight corresponding to the front distribution in each example should be higher than the latter. The results show that only our method can distinguish all these types of complex distributions, while existing methods cannot. By Eq. (6). we calculate weights of all instances as W = ]
Theorem 1. When (..) remains constant, Eq. 6 covers WA x P(ú L.). When Q and P(ù. L.) is the maximum value in P., Eq. (6) covers Wi X max P.) see P.).
6 simplifies to w, x P(i. wi x P(ii L:) still holds in Eq. (6) when Q > 2 and Ent(P.) remains constant. When Q > 2 and P(D:) L.) remains constant, w; x Ent(P) holds. According to the maximum entropy principle,
Proof. When Q = 2, we set Ent( P.) log 1) to I, so Eq.
Enn P.) takes its maximum value when any element of P. is equal to 1-P(g.)L.) At this point, if Q-1
P(ù: Li) is the maximum value in P., max (P.) sec( takes its maximum value. Conversely, when Ent(P, (P)) takes its minimum value, max(P) P.) sec P.) also takes its minimum value. Therefore, w; x (P.) sec( (P.) holds when Q 2 and P(ja L;) is the maximum value in P. Due the limited pages, more detailed proof of Theorem iss provided in Appendix A
4.2 Bias-variance trade-off
After instance weighting, the impact of intractable instances is mitigated. Therefore, P(cylax)) can be calculated more accurately, and then bias and variance can be estimated more accurately. Based on this result, the bias-variancs
trade-off will be more effective. After instance weighting, we train a classification model f on D with W. Let C' denote the label space {cI,.... .................. C....., CQ). in this paper, f classifies the test instance with the bias-variance trade-off as follows:
c(x)
arg max (f h) (x)).
3,EC
where f( (x) denotes the probability that x belongs to c, predicted by f. h, is the prediction of the regression model hn trained on the following probabilistic loss regression task:
Tg (X; W; W.C.). Lg),
(8)
5

---

where X' is the attribute matrix consisting of all training instances. Lg is the probabilistic loss vector for Co. which can be represented as [Lig.... Liq....,
Lig is calculated as follows:
f(cyle x.)
Cq = Ds Cq
Lig 0 [(o x.,)
+ in
Theorem 2. When the probabilistic loss is defined as in Eq. (. performing probabilistic loss regressions constructed by Eq. (8 ensures that Eq. asymptotically achieves the bias-variance trade-off.
When f is adjusted, P(c, [f; m:) changes with S. and this change is denoted Dig. Let bias, and vār; denote the changed bias term and variance term, they can be calculated as follows:
Proof.
1 Q
bias; 2 bias? +
Aig P(cglSi
Σ
ig P (cg
(10)
9=1
q=1
9=1
Q vār; = v0r; - P(egle -
Q
A
(11)
q=1
Due the limited pages, more detailed derivation of Eqs. [0]
is provided in Appendix
Comparing Eq. TO and Eq. [1] shows that the common terms
Σg Ai P(el/s ax) and
E9= A2 in bias; and vär, have opposite signs. Therefore, when improving E,. the bias and variance tend to change in opposite trends, which is known as the bias-variance dilemma. Improving Ey by synergistically considering changes in both bias and variance is known as the bias-variance trade-off. According to Eqs.
(T0). and (TT). we can get the changed Fy as follows:
E B==E E Σ
P(E.) Σ Dig ((c (cg)))))
(12)
1=1
q=1
In general, when Cg is the true label of x, P ((cylar)) tends to 1, otherwise it tends to 0. However, the true label y is unknown in crowdsourcing scenarios. After instance weighting, the impact of intractable instances is mitigated, so assume that it is equal to yı. Therefore, when Cg *f y.. Acyp((c)):;) tends to O. When G = ÿt, since the probability terms P(x;) and P(cy in Eq. ( are non-negative, E, is guaranteed to be less than E, as long as Aig is greater than 0. In summary, the key factor of the bias- varianar trade-off is Aig (Cg = ÿ:). To make Dig (Cg ge) greater than the following optimization task can be constructed:
mazimize f(b((z,))
(13)
s.t. (0i(zz;
max [c xx;) 2 0. c,EChe,#0:
Here, maximizing (ū) (e;) ensures that Aig (C, = 9:) is greater than O, while the constraint ensures that the prediction of f will be ÿi. Then, according to the Lagrange multiplier, the Lagrange function can be constructed as follows:
L(x;) f(A() {[5((i));)
(14)
where x 2 0. For simplicity,
['(x;) can be further constructed as follows:
L' L'(x;) = L(x;)
max C,ECAeAt
f(yy(x;)
(15)
max = =11 A)( C,EChe,
f(cglx:)]
(x;) 2 0. so L(x;) 2 L'(x;). Ultimately, Eq. (13) can be optimized achieve better result by maximizing L'(x,). At the same time, since 0. the value of L'(x.) is positively correlated with the following difference:
Since the probability maxx,ECAc, I (<,la;;)
f(jllz)))
max C,ECAne f(cglx,).
(16)
According to Eq. (.. through probabilistic loss regressions, when Co = -r. [(c ax:) + -h(()))))))))))())) in Eq. tends 1. Conversely, when ey & Gis [(yla=) + h (ar) tends to Therefore, Eq. (7) effective maximizing the difference Eq. (16). Ultimately, Theorem 2 proved.
D

---

The whole learning process of IWBVT is shown in Algorithm 1 In Algorithm 1 lines 1-3 learn a weight for each instance and their time complexity is O RQ).. Line trains classification model whose training time complexity is denoted as (. Lines 5-11 learn probabilistic loss regression model hg for each class Cy and their time complexity ((Q(Ni2 + fz)). Here, l2 denotes the prediction time complexity of f' on each class and f3 denotes the training time complexity of h In this paper, we select NB as the classification model and linear regression as the regression model. Therefore, 1,, 12. 13 are equal to M)) (M), and (N M + M respectively. If only the highest order terms are taken, the time complexity of IWBVT is ON RQ + NQM + QM*).
Algorithm The learning process of IWBVT
Require: δ = ((x,, L., )=) 1=1 a crowdsourced dataset with integrated labels. Ensure: classification model J, regression model set H.
1: for = 1 to N do
2: Calculate the weight wy of by Eq. 6);
3: end for
4: Train the classification model f on D with W {w,)ami
5: for
to Q do 6: fori to N do
7: Calculate the probabilistic loss Cig by Eq.
9
8: end for
9: Construct the regression task Tg by Eq.
8)
10: Learn the regression model h2 on Tgi
11:
end for
return classification model f, H {h1, h2. hv, ho).
Experiments
To validate the effectiveness of IWBVT, we conduct a series of experiments on the whole 34 simulated and 2 real-world crowdsourced datasets published on the Crowd Environment and its Knowledge Analysis (CEKA) (33) platform. First, we illustrate the setup of our experiments, including comparison algorithms and their parameter settings. Next, we describe the simulation process and present the simulated experimental results in terms of the model quality. Finally, to further validate the strength of IWBVT, we analyze the experimental results of comparative experiments and ablation experiments on real-world crowdsourced datasets.
5.1
Experimental setup
We select seven state-of-the-art algorithms for experiments, including: majority voling (MV) (20) it-
erative weighted majority voting (IWMV) (10), label augmented and weighted majority voting
(LAWMV) (3), multiple noisy label distribution propagationg (MNLDP) (8). adaptive voting noise correction (AVNC) [32].
viee-ieewwbbddd noise correction (MVNC) [15), and neighbor-
hood weighted voting-based noise correction (NWVNC) [1]. Among them, MV is the simplest label
integration algorithm and used as baseline for all algorithms. IWMV, LAWMV and MNLDP are three state-of- oftte-ofthheatt the-art art label integration algorithms. AVNC, MVNC and NW are three state-of- the-art noise correction algorithms. They are used to validate the effectiveness of IWBVT for label integration and noise correction. All these algorithms are implemented based on the CEKA platform and their parameter settings are consistent with the corresponding published papers. AVNC, MVNC and NWVNC are all performed based on integrated labels inferred by MV. Besides, we use linear regression as hy in IWBVT. All experiments are conducted on a Windows 10 machine with an AMD Athlon(tm) X4 860K Quad Core Processor @ 3.70 GHz and 16 GB of RAM.
5.2 Experiments on simulated datasets
Datasets and simulation process. We conduct our simulated experiments on all simulated datasets published on the CEKA platform. These datasets come from wide variety of application domains and represent plentiful crowdsourcing scenarios. Considering that the selected label integration
7

---

Table 2: The model quality (%) comparisons of MV, IWMV, LAWMV, MNLDP, AVNC, MVNC and NWVNC before and after using IWBVT on 34 simulated datasets.
Dataset
MV
IWMV
LAWMV
MNLDP
AVNC
MVNC
NWVNC
ORI
IWBVT
ORI
IWBVT
ORI
IWBVT
ORI
IWBVT
ORI
IWBVT
ORI
IWBVT
ORI
IWBVT
anneal
611
81) 42.
68.22
67.84
81.15
81.39
81.96
67.
79.49
audiology
59.93
61.39
60.08
61.15
63.44
60.50
68.93
65.49
64.77
64.09
59.10
61.50
autls
$4.31
59.98
52.13
58.82
56.20
60.733
56. 12
59.24
57.55
59.87
57.10
60.61
$6.51
60.01
Ialance-scale
N6.57
87.67
88 38
88.
88.62
89-022
89.95
89.24
84.
87.37
64.34
87.12 +
86, 58
88.
biodag
72.28
68
72.26
77.02
71.32
75.33
74.91
75.19
72.76
breasl-cancer
70.41
67.75 o
70.24
68. 13
72.50
71.56
70.55
69.90
72. 86
72,80
72.89
72.22
72. 72.62
72.83
bresst-w
18
94.999 o
96.
15.10
96.04
96.17
96. 11
96. 15
96.00
95.87
96.29
196.25
966. 16
96. 16
77.88
81.91
80,09
83.69
71.12
72.28
73.07
76.36
80,02
81.59
76.93
81.25
75.81
78.72
crcdit-a
77.55
77.91
81.12
80.07
credit-g
72.38
72.54
72.55
73.04
73.29
72.89
72.72
72.97
74.12
74.31
73.47
73.96
73.76
74.41
diabctics
73.85
71.13
74.24
74.27
72.43
73.01
71.65
75.24
74.75
75.33
71.19
75.20
75. 74.59
heartic
82.86
$1.45
62.65
$1.95
$4.05
84.09
83.72
83,08
83.10
82.52
83.34
83.23
83.67
83.37
beurt-h
$2.74
81.77
82.70
83.57
83,50
$2.58
82.33
$3.55
83.07
heart-stutlog
82.52
79.80
82.81
80.78
84.76 78
84.19
83.81
82. 56
83.52
82.56
83.33
$2.37
84, 26
84.00
bepatitis
78.98
76.97
79.22
76.50
85.78
B4.32
$4.05
83. 18
83.15
81.32
82.70
80.95
$3.45
83.05
bonse-colic
74.69
76.90
75.12
76.63
76.84
80,69
73.94
77.55
80.67
82.35
77.29
80.59
75.37
79,50
hypothyroid
93:50
93.76
93.99
Sonosphere
$1.23
82.22
81.08
81.57
77.90
81.39
81.03
86.24
$3.05
86.46 *
81.45
85.12 *
79.32
87.03
ins
90.67
93.077
91. 33
93.47
95.53
95. 60
95.13
95,40
95.53
98.47
94.73
95.73
95,
95.
kp-ys-kp
85.26
93.22
85.29
93.19
80.48
90.07
81.27
91.14
87.22
94.42
85:57
93:90
83.36
92.23
84.30
77. o
79 90
74.03
VIL20
86.57
38
86.87
85.53
87,15
82.35
79.88
86 82
84.73
eeter
63.48
64.55
62. 94
64.338
63.95
64.77
64.00
64.73
64.05
65.09
64.05
64.93 e
63.58
64.63
Smmph
79.53
74.68
80:45
75
80.82
79.63
80.32
79.00
79.17
76,94
80,18
77,82
80.64
81.34
mushroom
91.
93.42
segment
74.06
83.91
72.41
79.94
86.52
79.78
86.40
80.71
87.10
80.09
86.49 +
78
86.15
sick
46.48
93.88
46.61
93.93
89.10 10
92.83+
75.59
93.94
75.44
87.40
63.52
93.92 -
46.60)
93.52
sonar
66.18
63.33
66.23
63.49
66.33
68.D1
65.99
09.35
66.34
67.36
65.19
65.40
64.73
67.86
spambase
71.81
71.73
69.71
71.13
Ge-tac-toe
68,77
85
68.54
69.11
70.63
71.00
65.59
65.90
71.02
71.02
70.09
70.55
71
71.52
vehicle
43.54
67.98
41.47
68.28
43.25
66.02
42.85
66.95
43.72
65.85 +
43.99
68:64 a
4.5.11
66.42
volc
89.36
90.30
89.35
90--18
89.20
89.63
89,32
90.43
90.29
93.18
89:47
90.67
89.39
90.35
vowel
59.28
59.51
62.94
63.01
64.32
60.35
waveform
78:45
82.31
77.73
81.40
79.51
81.45 +
79.57
81.70
80.40
82.83
78.74
$2.43 =
79.12
81.76
200
89.73
90.48
89.30
90.38
90.25
91.98
90.90
91.77
88.27
88.81
87.63
87.66
89.92
91.70
Average
75.45
79.03
75.31
78.94
78.00
80.22
76.96
80.24
78.57
80.77
77.16
80.44
76.42
80.54
W/T/L
17/12/5
16/13/5
13/21/0
15/19/0
15/18/1
17/17/0
18/16/0
algorithms and noise correction algorithms handle the missing values of datasets differently, we use the unsupervised attribute filter ReplaceMissing Values in the Waikato Environment and Knowledge Analysis (WEKA) (25) platform to replace all missing values. Specifically, ReplaceMissing Values uses the mean of numerical attribute values or the modes of the nominal attribute values from the available data to replace missing values. Subsequently, to generate multiple noisy labels for each instance, we simulate the crowdsourcing process for these datasets. First, we randomly generate five workers whose label quality follows a normal distribution with N(0.65, 0.053). The label quality of a worker reflects the probability that the noisy label annotated by this worker to an instance is the same as this instance's 's unknown true label. Then, we hide true labels and use these simulated workers to annotate datasets. Finally, we use the selected algorithms to infer integrated labels for these datasets. For each simulation, we evaluate the original model quality and the corresponding model quality improved using IWBVT through stratified 10-fold cross- vooss-alidatiio. Here, use Naive Bayes (NB) (5) as the target model. The above processes are repeated ten times independently for each algorithm on each dataset.
Experimental results. T the detailed model quality (%) comparisons of each algorithm on each dataset, respectively. The columns ORI and /WBVT correspond the original model quality and the model quality using IWBVT, respectively. The symbols and o in the table denote the model quality has a statistically significant improvement or degradation using our proposed IWBVT with a corrected paired two-tailed t-test with the significance level a = 0.05 [18). respectively. Besides, the averages and the Win/Tie/Lose (W/T/L) L) values are summarized the bottom of Table The W W/T/L L implies that when improving the original model quality, IWBVT wins on W datasets, ties on T datasets, and loses on L datasets. These experimental results validate the effectiveness of IWBVT, and we can summarize the following highlights:
The average model quality of MV using IWBVT on 34 datasets is 79.03%, which is higher than the original model quality of all selected algorithms. This demonstrates both the limitations of label integration algorithms or noise correction algorithms and the effectiveness of ITBVT in improving the model quality.
8

---

34
6823
(a) Leaves
(b) Income
Figure 2: The model quality (%) comparisons MV, IWMV, LAWMV, MNLDP, AVNC, MVNC and NWVNC before and after using IWBVT on Leaves and Income.
The average model quality of IWMV (78.94%), LAWMV (80.22%) and MNLDP (80.24%) using IWBVT are higher than the original results of these state-of-the-art label integration algorithms. This demonstrates that IWBVT is still effective for more sophisticated label integration algorithms in improving the model quality.
The average model quality of AVNC (80.77%) MVNC (80.44%), and NWVNC (80.54%) using IWBVT are also higher than the original results of these state-of-ther thettrrr noise correc- tion algorithms. This demonstrates that IWBVT can serve as universal post post-processing approach to significantly improving the model quality.
Based the t-test results, the number of datasets in which IWBVT wins significantly (W) is always much higher than the number of datasets which it loses significantly (L) for all algorithms. This strongly demonstrates the effectiveness and robustness of IWBVT.
5.3 Experiments real-world datasets
Datasets. To demonstrate the robustness IWBVT, the above simulation process pays more attention common factors of crowdsourcing. However, training models real-world datasets may also be affected by other factors, such as sparsity and annotating bias. To verify the effectiveness of WTVT in real-world crowdsourced scenarios, used real-world crowdsourced datasets, Leaves and Income, published on the CEKA platform (34). Here, Leaves and Income selected through the online platform Amazon Mechanical Turk (AMT). Leaves is annotated by 83 workers and each instance annotated by 10 workers. There are classes, 384 instances, 3840 labels, 64 numeric attributes, and 0 missing values in Leaves. Income is annotated by 67 workers and each instance is also annotated by 10 workers. There are 2 classes, 600 instances, 6000 labels, 10 nominal attributes, and missing values in Income. We only evaluate the original model quality and the corresponding model quality using IWBVT by stratified 10-fold cross-validation one time because
also construct experiments two widely
datasets do not have random simulation process.
Experimental results. MNLDP, AVNC, MVNC and NWVNC before and after using IWBVT on Leaves and Income. With Figures 2a and 26 we can find that IWBVT can also serve as a universal post-processing approach to significantly improving the model quality real-world crowdsourced scenarios. Besides, we can also find the original model quality of several state-of-the-an algorithms is even lower than the original model quality of MV. These results once again demonstrate the limitation of label integration and noise correction in improving the model quality.
Figure showows the model quality (%) comparisons of MV, IWMV, LAWMV,
Ablation experiment. The above results only demonstrate the effectiveness of IWBVT as whole, yet they do not delineate the contributions of its two key components: instance weighting and bias-variance trade-off. In IWBVT, instance weighting is used mitigate the impact of intractable instances to make the bias and variance of trained models closer to the unknown true results. Therefore, independently verify the effectiveness of instance weighting, we first observe the bias and variance of trained models before and after instance weighting. To estimate the bias and variance, referring to [24]. NB tested on Leaves and Income by ten runs of three-fold cross-validation. Figure[3asshows the bias and variance comparisons before and after using instance weighting on Leaves and Income.

---

324
130
047
(a) bias and variance
(b) model quality
Figure The ablation experiment comparisons of IWBVT and its components on Leaves and Income.
In Figure 3a objectl denotes the estimation results of NB trained directly with true labels. objectz denotes the estimation results of NB trained with integrated labels inferred by MV. The conditions for objects and objectz are the same, besides considering instance weighting. As can be seen in Figure [a] the bias and variance of models trained only with integrated labels are usually higher than unknown true results. When instance weighting introduced, both the bias and variance of models tend to be closer to unknown true results. These results demonstrate that instance weighting successfully corrects the bias and variance of trained models by mitigating the impact of intractable instances. Therefore, IWBVT performs the instance weighting before the bias- trade-off, which is more effective in improving the generalization performance of trained models.
Additionally, we also analyze the effectiveness of another component, the bias-variance trade-off, in improving model quality. Similarly, we still fix the label integration algorithm to be MV, and then introduce the instance weighting and bias-variance trade-oft off individually to observe their influence in improving model quality. Figure 3b shows the model quality (%) comparisons of MV using IWBVT or its components on Leaves and Income. In Figure 3b) object, denotes the model quality of MV, objectz denotes the model quality of MV using the bias-variance trade-off, objects denotes the model quality of MV using the instance weighting, and object, denotes the model quality of MV using the whole IWBVT. As can be seen in Figure (36] both the instance weighting and the bias-variance trade-off effectively improve the model quality of MV. This demonstrates that the two components of the IWBVT are both effective. Moreover, the model quality of MV using the whole IWBVT the highest on both Leaves and Income, which suggests that it is reasonable for IWBVT to utilize both components at the same time. In addition, in Figure 3b] we can also find that the instance weighting is more effective on Leaves, while the variana trade-off is more effective because the average label quality of Leaves is low. Instance weighting helps to identify rare instances that are inferred correctly, and therefore has greater impact Leaves. However, the average label quality of Income is high, and more instances can be correctly inferred than in Leaves. Therefore, the bias and variance are estimated closer to the unknown true values, so the birs-variance trade-off is more effective on Income.
Income. This is
6 Conclusion and future work
To improve the model quality of models trained on crowdsourced datasets, propose a universal post-processing approach called IWBVT. IWBVT first mitigates the impact of intractable instances by instance weighting to make the bias and variance of trained models closer to the unknown true results. Then, IWBVT reduces the generalization error of trained models by the bias-variance trade-off. Experimental results suggest that IWBVT can significantly improve the model quality of existing state-of-the-art label integration algorithms and noise correction algorithms.
Though the above experimental results sufficiently demonstrate the effectiveness of IWBVT, some anomalies are found in experiments. Table [ shows that IWBVT degrades the model quality on a few datasets such as labor and lymph. The datasets such as labor and lymph contain some numerical attributes that are significantly higher in magnitude than other attributes. However, the linear regression chosen for IWBVT in experiments is not robust to regression tasks constructed for these datasets. Therefore, in the future, we will further improve the robustness of IWBVT to make trained models insensitive to anomalous attributes.
10

---

Acknowledgment
The work was partially supported by National Natural Science Foundation of China (62276241), Foun- dation of Key Laboratory of Artificial Intelligence, Ministry of Education, P.R. China (A12022004), and Science and Technology Project of Hubei Province-Unveiling System (2021BEC007).
References
[1] P. Chen, Y. Yang, D. Yang. H. Sun, Z. Chen, and P. Lin, "Black-box data poisoning attacks on crowdsoure- ing," in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China. ijeai.org, 2023, pp. 2975-2983.
[2] Z. Chen, H. Sun, H. He, and P. Chen, "Learning from noisy crowd labels with logics." in 39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, CA. USA. April 3-7, 2023. IEEE 2023, pp. 41-52.
[3] Z. Chen, L. Jiang, and C. Li, "Label augmented and weighted majority voting for crowdsourcing." Inf. Sci., vol. 606, pp. 397-409 2022.
[4] A. Dawid A. M. Skene, "Maximum likelihood estimation observer error-rates using the em algorithm," Journal of the Royal Statistical Society: Series C (Applied Statistics), vol. 28, no. 1. pp. 20-28, 1979.
[5] N. Friedman, D. Geiger, and M. Goldszmidt, "Bayesian network classifiers," Mach. Learn., vol. 29, no. 2-3, pp. 131-163, 1997.
[6] L. He, Y. Lu, W. Li, and X. Jia, "Generative calibration of inaccurate annotation for label distribution learning"" in Thirty- Eighth AAAI Conference Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Ad- vances in Artificial Intelligence, EAAl 2014, February 20-27, 2024, Vancouver, Canada, M. J. Wooldridge, J. G. Dy, and S. Natarajan, Eds. AAAT Press, 2024, pp. 12 12394-12401.
[7] Z. Huang, L. Shen, J. Yu, B. Han, and T. Liu, "Flatmatch: Bridging labeled data and unlabeled data with cross- coos--harppees for semi-supervised learning." in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA. USA, December 10 16, 2023, A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., 2023.
[8] L. Jiang, Н. Zhang, Tao, and Li, "Learning from crowds with multiple noisy label distribution propagation." IEEE Trans. Neural Networks Learn. Syst., vol. 33, no. 11, pp. 6558-6568, 2022.
[9] R. Kohavi and D. H. Wolpert, "Bias plus variance decomposition for zero-onc loss functions," in Machine Learning, Proceedings of the Thirteenth International Conference (ICML '96), Bari, Italy, July 3-6, 1996. L. Saitta, Ed. Morgan Kaufmann, 1996, pp. 275-283.
[10] H. Li and B. Yu, "Error rate bounds and iterative weighted majority voting for crowdsourcing." CoRR, vol. 0bs/1411.4086 2014.
[11] H. Li, L. Jiang, and S. Xue, "Neighborhood weighted voting-basee noise correction for crowdsourcing." ACM Trans. Knowl. Discov. Data, vol. 17, no. 7, pp. 96:1-96:18 2023.
[12] J. Li, H. Sun, and J. Li, "Beyond confusion matrix: learning from multiple annotators with awareness of instance features," " Mach. Learn., vol. 112, no. 3. pp. 1053-1075, 2023.
[13] Li, Kawase, Y. Baba, and Kashima, "Performance as constraint: An improved wisdom crowds using performance regularization," in Proceedings of the Twenry-Ninth International Joint Conference on Artificial Intellignce, IJCAI 2020, C. Bessiere, Ed. ijcai.org, 2020, pp. 1534-1541.
[14] M. Li, R. Wu, H. Liu, J. Yu, X. Yang, B. Han, and T. Liu, "Instant: Semi-supervised learning with instance- dependent thresholds." ," in Advances in Neural Information Processing Systems 36: Annuul Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., 2023.
[15] X. Li, Li, and Jiang, "A multi-view-Be view-based noise correction algorithm crowdsourcing learning." Inf. Fusion, vol. 91, pp. 529-541, 2023.
[16] Y. Li. B. I. P. Rubinstein, and T. Cohn, "Exploiting worker correlation for label aggregation in crowdsoure- ing." in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 2019, pp. -8865

---

[17] Y. Lu and X. Jia, "Predicting label distribution from multi-label ranking," in Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., 2022.
[18] C. Nadeau and Y. Bengio, "Inference for the generalization error"" Mach. Learn., vol. 52, no. 3. pp. 239-281, 2003,
[19] B. Nicholson, V. S. Sheng, and J. Zhang, "Label noise correction and application in crowdsoureing," Expert Syst. Appl., vol. 66, pp. 149-162, 2016.
[20] V. S. Sheng. F. J. Provost, and P. G. Ipcirotis, "Get another label? improving data quality and data mining using multiple, noisy labelers," in Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008, Y. Li, B. Liu, and S. Sarawagi, Eds. ACM, 2008, pp. 614-622.
[21] V. S. Sheng, J. Zhang, B. Gu, and X. Wu, "Majority voting and pairing with multiple noisy labeling," HEEE Trans. Knowl. Data Eng., vol. 31, no. pp. 1355-1368, 2019.
[22] F. Tao, Jiang, and C. Li, "Differential evolution-based weighted soft majority voting for crowdsoureing." Eng. Appl. Artif. Intell., vol. 106, p. 104474, 2021.
[23] T. Tian, J. Zhu, and Y. Qiaoben, (Max-margin majority voting for learning from crowds," IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 10. pp. 2480-2494 2019.
[24] G. I. Webb, "Multiboosting: A technique for combining boosting and wagging." Mach. Learn., vol. 40. no. 2, pp. 159-196, 2000.
[25] H. Witten, E. Frank, and M. A. Hall, Data mining: practical machine learning tools and techniques, 3rd
Edition. Morgan Kaufmann, Elsevier, 2011.
[26] C. Xu and X. Geng, "Hierarchical classification based on label distribution learning." " in The Thirty- Thind AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, Jamuary 27 - 1, 2019. AAAI Press, 2019. pp. 5533-5540.
[27] W. Xu, L. Jiang, and C. Li, "Improving data and model quality in crowdsourcing using cross-entropy-based noise correction." Inf. Sci., vol. 546, pp. 803-814, 2021.
[28] S. Yang, Wu, E. Yang. B. Han, Liu, M. Xu, Niu, and Liu. "A parametrical model for instance- dependent label noise," IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 12. pp. 14 14 2023.
[29] Yang, Li. and Jiang, "Learning from crowds with decision trees." Knowl. Inf. Syst., vol. 64, no. pp. 2123-21211 2243,210,2222 2022.
[30] L. Yin, J. Han, W. Zhang, and Y. Yu. "Aggregating crowd wisdoms with label-aware autoencoders." in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, C. Sierra, Fd. ijcai.org, 2017, pp. 1325-1331.
[31] C. Zhang, X. Jia, Z. Li, C. Chen, and H. Li, "Learning cluster-wise anchors for multi-view clustering," in Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, M. J. Wooldridge, J. G. Dy, and S. Natarajan, Eds. AAAI Press, 2024, pp. 16 696-16 704.
[32] Zhang, V. S. Sheng, Li, and X. Wu, "Improving crowdsourced label quality using noise correction." " IEEE Trans. Neural Networks Learn. Syst., vol. 29, no. 5, pp. 1675-1688, 2018.
[33] J. Zhang, V. Sheng, B. Nicholson, and X. Wu, "CEKA: tool for mining the wisdom of crowds." J. Mach. Learn. Res., vol. 16. pp. 2853-2858, 2015.
[34] J. Zhang, X. Wu, and V. S. Sheng, "Learning from crowdsourced labeled data: a survey," Artif. Intell, Rev. vol. 46, no. pp. 543-576 576, 2016.
12

---

Appendix A Additions to the proof of Theorem
In the proof of Theorem according the maximum entropy principle, we get that End(Pp takes its maximum value when any element of P; is equal to L_)( Here, we provide its
Q-1
detailed derivation. First, by the definition of P, we simplify it to P. = (Pa) with E Pg =
1 - P(il L.). Then, can find the maximum value of Ent( (P.) as follows:
Q-1
arg max Ent( = arg max
Σ
Ps log Pa|
Pg
Ps
(17)
Q-1 1
s.t.
P
9=1
Then, according to the Lagrange multiplier, the Lagrange function I(P.) can be constructed as follows:
Q-1 =(P.)= = Σ -- Pa log Pg] + + P. 1+ P(.
Q-1
(18)
Now, to obtain the maximum value L(P.), we can take the partial derivative of L(P.) concerning Pg. and set this derivative equal to ZeTo as follows:
(L(P.) = -log Pg +
(19)
OP,
According to Eq. 19. we can obtain P. = 22 Bringing this result into the constraints, we can
obtain - P. (Q-1)2 111-1122=---- P-P L.). Therefore, is clear that Pa = 2^ 1-P(0: L.)
Q-1
Finally, bringing Pg I-P( L.) into Eq. 17 can calculate the maximum value of Entt P.))
Q-1
as (1 P(ú: log I-P(O.LL.)
Q-1
Appendix B Derivation of Eqs.
10)
Due to the limited pages, Eqs. 10 provide their detailed derivation. First, when f is adjusted, P(cglf, changes with f. and this change is denoted as Aig. Then, we bring Aig into Eq. 4 and Eq. (5), respectively. The following derivation can be obtained:
in the main text only give the derived results. Here,
bias;
2
(())
+ Ann)
2
q=1
Q
+ (P(cylf,z;) + Aiy) (() + An))
2
Q
P(cyl12)²² P(cylf, + a + P(cl,(c)) 2(c)/): 2D P(ylaa))
2
P(cyl22,) + P(cylJ, 2P(cy
Σ
A + 22 20 P(cyl,,a,) 20 P(cg(2))))
2
9=1
)²+
P(cylz,)) P(cylf.2z))
Σ
A + P(cylfa.) P(cplz())
2
bias?
Σ
D in P(cgls,
+
A2
Aig P(cgl2:).
2 g=1
g=1
(20)
13

---

vār;
-
P(c)l)) + Ain)
(P(col,, + Diy + P(g)l,
= 2
1=1
(21)
Q
1 ②
P(cylf. x;,3
Ain 22ing P(cylf.z))
2
4=1
9=1
Dig P(gp,,0))
A 34
var,
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly claim the abstract and introduction section that the approach proposed in paper is mainly used to improve the model quality in crowdsourcing. The contributions of our paper are also highlighted in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mentioned the limitations of our IWBVT in the Conclusion and future work section. IWBVT robust on datasets containing anomalous attributes.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model speciffiiic,,,nn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be
14

---

used reliably to provide closed captions for online lectures because it fails to handle technical Jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknow judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
We give proofs to theorems that appear in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
15

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This paper provides open access to the data and code, and provides a document guide readers in reproducing all experimental results in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/niip.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccppppccceeeeeee.eeeeceelt wwwpc//ggiesssseee http::///giieee/eeeeuuiiiiiii ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification; This paper specifies all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
16

---

7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We explain in detail the experimental metrics in the paper and provide the results of the significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe in detail the experimental setting on the computer resources in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
The research conducted in this paper conforms with the NeurIPS Code of Answer: [Yes]
Justification:
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
17

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The new approach proposed in this paper helps to train models more efficiently from crowdsourced datasets. There are no negative societal impacts.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
18

---

Answer: [Yes]
Justification: We used the datasets and algorithmic implementations published by the CEKA platform, which is described in detail the paper. The license and terms of use explicitly are properly respected in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papeesswii cooo... com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper provides open access the data and code, and provides document to guide readers in reproducing all experimental results in supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper did not select new crowdsourced datasets. The datasets used for experiments publicly available and their information is described in detail in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve the research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "UTQVgY0nCG", "reviewer": "Reviewer_Td4g", "summary": "The paper studies the problem of improving quality of models trained on datasets collected through crowdsourcing. The authors propose an approach (IWBVT) that post-processes data after crowdsourcing with the goal to mitigate the impact of intractable instances by means of instance weighting. As a result, the bias and variance of trained models becomes closer to the unknown true labels. It is proven that the novel method reduces the generalization error of trained models by the bias-variance trade-off. The paper also contains extensive experimentation that demonstrates: IWBVT significantly improve the model quality of existing label integration algorithms and noise correction algorithms.", "questions": "- I do not understand the message behind Figure 1. How should I tract the 4 distributions on the right side of the arrow?\n\n- Lines 164 – 165: “it can be verified that Eq. (6) can distinguish all complex distributions we showcased in Section 3.1. By..” It is hard to assess the value of this claim, because it is not clear how broad complex distributions are covered. Is it just 2 among 100? Among 1000 possible? Any quantifiable / qualifiable ways to measure the distinguish power for complex distributions?\n\n- Line 166 “Theorem 1. In some special cases,” Please, specify cases in the theorem. Otherwise, I would not agree that this might be a theorem statement. The definition of the cases must be contained outside of the proof. Right now, I can treat the theorem as always true, because “some special cases” might be = \\emptyset .\n\n- Similar, Line 185 in Theorem 2: “Eq. (8) helps Eq. (7)”… The word “help” does not have formal mathematical definition, while Theorem is a mathematical instrument.", "limitations": "the authors adequately addressed the limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-\tNovel method\n\n-\tExtensive experimentation over 34 simulated datasets and 2 real ones", "weaknesses": "-\tFormalization of results \n\n-\tClearness of explanation in some points behind the novel approach\n\n(see Questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer Td4g：**\n\n**Q1:** I do not understand the message behind Figure 1. How should I tract the 4 distributions on the right side of the arrow?\n\n**Author Response:** Thanks for your valuable comments. The four distributions on the right side of the arrow in Figure 1 correspond to the four cases of our new instance weighting method. Among them, the distribution in the upper left corner indicates that when the entropy of the complement $Ent(\\bar{P}_i)$ is fixed, a lower $P(\\hat{y}_i|L_i)$ (the probability of the integrated label in multiple noisy label distribution) results in a lower weight $w_i$ assigned to the instance. Conversely, the distribution in the upper right corner indicates that a higher $P(\\hat{y}_i|L_i)$ results in a higher $w_i$. The distribution in the lower left corner indicates that when $P(\\hat{y}_i|L_i)$ is fixed, a lower $Ent(\\bar{P}_i)$ results in a lower $w_i$. Conversely, the distribution in the lower right corner indicates that the higher $Ent(\\bar{P}_i)$ results in a higher $w_i$. These explanations are provided in lines 154-162, and we will refine them further in the final version of the paper. Thanks again for your valuable comments.\n\n**Q2:** Lines 164 – 165: “it can be verified that Eq. (6) can distinguish all complex distributions we showcased in Section 3.1. By..” It is hard to assess the value of this claim, because it is not clear how broad complex distributions are covered. Is it just 2 among 100? Among 1000 possible? Any quantifiable / qualifiable ways to measure the distinguish power for complex distributions?\n\n**Author Response:** Thanks for your valuable comments. To further demonstrate the distinguish power of Eq. (6) for complex distributions, we employed a quantifiable way. Specifically, on all the complex distributions exemplified in Section 3.1, the mentioned instance weighting methods produced the following results:\n|Complex distributions|A|B|C|D|\n|--|--|--|--|--|\n|{0.5, 0.3, 0.2} and {0.5, 0.4, 0.1} (on line 122)|(0.50, 0.50) ✖|(0.67, 0.73) ✖|(0.20, 0.10) ✔|(0.70, 0.52) ✔|\n|{0.4, 0.3, 0.3} and {0.4, 0.4, 0.2} (on line 127)|(0.40, 0.40) ✖|(0.64, 0.66) ✖|(0.10, 0.00) ✔|(0.58, 0.53) ✔|\n|{0.5, 0.3, 0.1, 0.1} and {0.4, 0.2, 0.2, 0.2} (on line 131)|(0.50, 0.40) ✔|(0.59, 0.52) ✔|(0.20, 0.20) ✖|(0.62, 0.58) ✔|\n\nHere, A, B, C, and D denote $P(\\hat{y}_i|L_i)$, $\\frac{1}{Ent(P_i)}$, $max(P_i)-sec(P_i)$ and our new instance weighting method, respectively. \"✔\" and \"✖\" indicate whether the weighting method is effective in distinguishing the corresponding complex distribution, respectively. Empirically, the weight corresponding to the front distribution in these examples should be greater than the latter. The results show that our new instance weighting method can distinguish all types of complex distributions that existing methods cannot. In the final version of the paper, we will include these comparisons and analyses to demonstrate the distinguish power of our new method. Thanks again for your valuable comments.\n\n**Q3:** Line 166 “Theorem 1. In some special cases,” Please, specify cases in the theorem. Otherwise, I would not agree that this might be a theorem statement. The definition of the cases must be contained outside of the proof. Right now, I can treat the theorem as always true, because “some special cases” might be = \\emptyset .\n\n**Author Response:** Thanks for your valuable comments. Indeed, the special cases mentioned in Theorem 1 should be specified. In the final version of the paper, we will move the definition of these special cases from the proof of Theorem 1 to Theorem 1 itself. Specifically, the refined Theorem 1 will be \"When $Ent(\\bar{P}_i)$ remains constant, Eq. (6) covers $w_i \\propto P(\\hat{y}_i|L_i)$. When $Q>2$ and $P(\\hat{y}_i|L_i)$ is the maximum value in $P_i$, Eq. (6) covers $w_i \\propto \\max(P_i) - \\sec(P_i)$.\". Thanks again for your valuable comments.\n\n**Q4:** Similar, Line 185 in Theorem 2: “Eq. (8) helps Eq. (7)”… The word “help” does not have formal mathematical definition, while Theorem is a mathematical instrument.\n\n**Author Response:** Thanks for your valuable comments. Indeed, we should use a more formal mathematical definition in our Theorem 2. In the final version of the paper, Theorem 2 will be refined as follows: \"When the probabilistic loss is defined as in Eq. (9), performing probabilistic loss regressions constructed by Eq. (8) ensures that Eq. (7) asymptotically achieves the bias-variance trade-off.\". Thanks again for your valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1EVrEIyeSw", "reviewer": "Reviewer_fq5P", "summary": "This paper proposes a novel label integration method called IWBVT for crowdsourcing by using a weighting method and probabilistic loss regressions to improve the model quality. The main problem this paper solves is the model quality caused by the presence of the intractable instances. The paper is well organized and the writing is smooth. The method proposed in the paper has improved the model quality, thus boosting the performance on simulation and real-world datasets. The authors also provide theoretical proof to deepen the understanding of the proposed method.", "questions": "Please refer to weakness.\nOther questions: \n1. In the experimental section, why are methods like significance testing used instead of direct comparison?", "limitations": "yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well organized and the writing is smooth. The authors provide necessary symbol explanations, which is very helpful for reading.\n2. The idea proposed in this paper is technical soundness. The authors incorporate instance weighting and bias-variance trade-off components to complete the overall algorithm. Furthermore, the authors also provide the corresponding theory to prove the effectiveness and generalization of the designed components. Experiments on simulation and real-world datasets demonstrate the superiority of the proposed method.\n3. For instance weighting component, the authors prove that existing methods [1] and [2] are special cases of the method proposed by the authors. Hence, the method proposed by the author demonstrates strong generalization and can be applied to a wider range of situations.\n\n[1]. V.S. Sheng, F.J. Provost, and P.G. Ipeirotis. Get another label? Improving data quality and data mining using multiple, noise labels. SIGKDD, 2008.\n\n[2]. Z. Chen, L. Jiang, and C. Li. Label augmented and weighted majority voting for crowdsourcing. Inf. Sci, 2022.", "weaknesses": "1. Compared with real-world dataset, experimental results on a few simulated datasets doesn’t improve (e.g., MV for breast-cancer/breast-w datasets). Furthermore, the number of loss for MV and IWMV is more than that for other methods in Table 1. I suggest adding some discussion about this phenomenon.\n2. Main experiments adopt significance level to indicate the performance for each method. How to decide the value of alpha and what is the influence? More detail will be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer fq5P：**\n\n**Q1:** Compared with real-world dataset, experimental results on a few simulated datasets doesn’t improve (e.g., MV for breast-cancer/breast-w datasets). Furthermore, the number of loss for MV and IWMV is more than that for other methods in Table 1. I suggest adding some discussion about this phenomenon.\n\n**Author Response:** Thanks for your valuable comments. Our proposed IWBVT is not restricted to specific crowdsourcing scenarios, so we conducted experiments on the whole 34 simulated datasets published by the CEKA platform. However, no algorithm can achieve the best performance on all datasets. Indeed, IWBVT does not improve MV on a few datasets, such as breast-cancer and breast-w. This is because the performance of IWBVT is affected by the performance of the label integration algorithm it aims to improve. According to Eqs. (6) and (9), IWBVT depends on integrated labels inferred by label integration algorithms for its instance weighting and probabilistic loss regressions. Therefore, noise in these integrated labels affects the performance of IWBVT. MV and IWMV, as classical label integration algorithms, generally perform worse than other state-of-the-art algorithms, resulting in integrated labels inferred by MV and IWMV containing more noise. This ultimately leads to more losses for MV and IWMV compared to other state-of-the-art algorithms in Table 1. In the final version of the paper, we will include these discussions to explain this phenomenon. Thanks again for your valuable comments.\n\n**Q2:** Main experiments adopt significance level to indicate the performance for each method. How to decide the value of alpha and what is the influence? More detail will be helpful.\n\n**Author Response:** Thanks for your valuable comments. The significance level is a crucial concept in statistics that directly affects the results of the corrected paired two-tailed t-test used in our experiment. In a t-test, the significance level is typically set to a fixed value, such as 0.05 or 0.1. If the p-value is less than this significance level, we reject the null hypothesis, indicating a significant difference between the two comparison algorithms. Therefore, the significance level affects the stringency of the test: a lower significance level (e.g. 0.05) implies a more stringent test, as the null hypothesis is rejected only when the evidence is very strong. Conversely, a higher significance level (e.g. 0.1) is relatively relaxed and more likely to accept the null hypothesis. In the final version of the paper, we will include a more detailed explanation of the significance level. Thanks again for your valuable comments.\n\n**Q3:** In the experimental section, why are methods like significance testing used instead of direct comparison?\n\n**Author Response:** Thanks for your valuable comments. To exclude the effect of randomness, our simulated experiments were independently repeated ten times. In each experiment, we evaluated the model quality using 10-fold cross-validation. Therefore, each label integration algorithm produced 100 pairs of comparison results (original model quality vs. IWBVT improved model quality) on each dataset. In Table 1, we reported the averages (arithmetic mean) of these results to provide a general indication of relative performance. However, these averages do not indicate whether the comparison results are significantly different. Therefore, to more accurately evaluate the performance of IWBVT, we statistically compared these results by the corrected paired two-tailed t-test. The corrected paired two-tailed t-test determines whether IWBVT effectively improves the corresponding label integration algorithm by assessing whether the differences between the 100 pairs of comparison results satisfy the null hypothesis. Thanks again for your valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CLWmnk7dSk", "reviewer": "Reviewer_5XV6", "summary": "This paper explores methods to enhance the training of machine learning models using crowdsourced datasets and proposes a novel post-processing approach called IWBVT. The proposed IWBVT first performs instance weighting based on the entropy of the complementary label distribution. Subsequently, it employs a bias-variance trade-off to minimize the generalization error of trained models. Extensive experiment results demonstrate that IWBVT significantly improves the model quality of existing state-of-the-art label integration and noise correction algorithms.", "questions": "1.\tAccording to Figure 3b, when improving model quality, why is instance weighting more effective on the Leaves dataset while the bias-variance trade-off is more effective on the Income dataset?\n2.\tHow does IWBVT perform on different machine learning models? Demonstrating that IWBVT can improve the quality of various machine learning models would further validate its effectiveness.\n3.\tHow does IWBVT perform in more complex crowdsourcing scenarios, such as those with extremely noisy labeling or very large datasets? Discussing the potential limitations of IWBVT in these contexts can help enhance its overall effectiveness and applicability.", "limitations": "Please refer to the Weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThis paper proposes a novel post-processing approach that significantly improves the quality of machine learning models trained with crowdsourced datasets. The paper is technically sound, providing detailed derivations and justifications for the proposed approach. The proofs for theorems are clear and rigorous.\n2.\tThe paper is well-written, logical and easy to understand. It clearly describes the problem to be solved and provides a logically rigorous argumentation process. The figures and tables provided in the paper are helpful in illustrating the concept and effectiveness of IWBVT.\n3.\tThe experiments are well-designed, with appropriate datasets and evaluation metrics used to validate the proposed approach.", "weaknesses": "1. More detailed explanations about experimental results are desired. For example, in Figure 3b, instance weighting significantly improves model quality on the Leaves dataset, whereas the bias-variance trade-off is more effective on the Income dataset. Understanding the properties of the datasets that lead to these differences would help clarify the conditions under which the proposed approach is most effective.\n2. The experiments in this paper primarily focus on the robustness of IWBVT across various label integration and noise correction algorithms and different datasets. To further validate the effectiveness of IWBVT, it would be beneficial to observe its performance on a range of different models.\n3. More limitations of IWBVT should be discussed. These include its robustness in the presence of extremely noisy labeling, the computational complexity when handling very large datasets, and potential challenges in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer 5XV6：**\n\n**Q1:** According to Figure 3b, when improving model quality, why is instance weighting more effective on the Leaves dataset while the bias-variance trade-off is more effective on the Income dataset? \n\n**Author Response:** Thanks for your valuable comments. Indeed, in Figure Figure 3b, the instance weighting is more effective on the Leaves dataset, while the bias-variance trade-off is more effective on the Income dataset. This is because the average label quality of the Leaves dataset is low. The instance weighting helps to identify the small number of instances that are correctly inferred and therefore has a greater impact on the Leaves dataset. However, the average label quality of the Income dataset is high, and more instances can be correctly inferred than in the Leaves dataset. Therefore, the bias and variance are estimated closer to the unknown true values, so the bias-variance trade-off is more effective on the Income dataset. In the final version of the paper, we will provide a more detailed explanation of our experimental results. Thanks again for your valuable comments.\n\n**Q2:** How does IWBVT perform on different machine learning models? Demonstrating that IWBVT can improve the quality of various machine learning models would further validate its effectiveness.\n\n**Author Response:** Thanks for your valuable comments. Indeed, we currently used only a simple linear regression (LR) in the probabilistic loss regressions of IWBVT and validated its effectiveness only on Naive Bayes (NB). In fact, IWBVT is effective for other various machine learning models as well. We constructed experiments on the Leaves dataset to validate this conclusion. Specifically, we considered both LR and model tree (MT) as the regression models and used NB and C4.5 as the target models. the experimental results are as follows:\n| |MV| IWMV| LAWMV| MNLDP| AVNC| MVNC| NWVNC|\n|--|--|--|--|--|--|--|--|\n|LR+NB|(59.75, 62.28) ✔|(61.30, 62.85) ✔|(58.74, 60.09) ✔|(55.81, 61.54) ✔|(60.44, 63.02) ✔|(58.44, 60.55) ✔|(58.57, 59.49) ✔|\n|LR+C4.5|(52.10, 55.42) ✔|(52.29, 56.17) ✔|(56.15, 56.64) ✔|(52.69, 58.09) ✔|(56.89, 56.31) ✖|(54.92, 56.31) ✔|(56.12, 58.05) ✔|\n|MT+NB|(59.21, 62.05) ✔|(60.65, 61.45) ✔|(59.21, 62.26) ✔|(55.88, 61.36) ✔|(58.55, 62.15) ✔|(57.97, 59.45) ✔|(61.02, 63.01) ✔|\n|MT+C4.5|(51.96, 55.56) ✔|(51.41, 56.64) ✔|(55.11, 57.06) ✔|(52.09, 56.83) ✔|(57.29, 56.78) ✖|(55.05, 56.53) ✔|(56.74, 58.03) ✔|\n\nHere, \"✔\" indicates that IWBVT improves the model quality of the corresponding label integration algorithm, while \"✖\" indicates the opposite. From these results, it can be seen that IWBVT is effective for various machine learning models. In the final version of the paper, we will include a discussion on the effectiveness of IWBVT across different machine learning models. Thanks again for your valuable comments.\n\n**Q3:** How does IWBVT perform in more complex crowdsourcing scenarios, such as those with extremely noisy labeling or very large datasets? Discussing the potential limitations of IWBVT in these contexts can help enhance its overall effectiveness and applicability.\n\n**Author Response:** Thanks for your valuable comments. Our proposed IWBVT is not restricted to specific crowdsourcing scenarios, so we conducted experiments on the whole 34 simulated datasets published by the CEKA platform. These datasets contain some large datasets such as the letter dataset. The experimental results in Table 1 indicate that IWBVT is effective on these large datasets. Additionally, the real-world dataset Leaves, whose percentage of noisy labels exceeds 0.4, is an extremely noisy labeling dataset. The experimental results shown in Figure 2 indicate that IWBVT is effective on this extremely noisy labeling dataset as well. In Section 6, we have discussed some limitations of IWBVT. In the final version of the paper, we will further refine our explanation for these limitations. Thanks again for your valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new instance weighting-based approach to improving the quality of models trained on datasets collected through crowdsourcing, and theoretically demonstrates that an appropriate bias-variance trade-off is achieved. 
The paper makes a significant technical contribution to a major problem in this topic, and the issues raised by the reviewers have been appropriately addressed, and further enhanced the completeness of this work.
Decision: Accept (poster)