KFNN: K-Free Nearest Neighbor For Crowdsourcing
Authors: Wenjun Zhang
Liangxiao Jiang
Chaoqun Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Crowdsourcing learning
Label integration
K-free nearest neighbor
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We propose a novel label integration algorithm called K-free nearest neighbor (KFNN).

Abstract: To reduce annotation costs, it is common in crowdsourcing to collect only a few noisy labels from different crowd workers for each instance. However, the limited noisy labels restrict the performance of label integration algorithms in inferring the unknown true label for the instance. Recent works have shown that leveraging neighbor instances can help alleviate this problem. Yet, these works all assume that each instance has the same neighborhood size, which defies common sense. To address this gap, we propose a novel label integration algorithm called K-free nearest neighbor (KFNN). In KFNN, the neighborhood size of each instance is automatically determined based on its attributes and noisy labels. Specifically, KFNN initially estimates a Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution is then utilized to enhance the multiple noisy label distribution of each instance. Subsequently, a Kalman filter is designed to mitigate the impact of noise incurred by neighbor instances. Finally, KFNN determines the optimal neighborhood size by the max-margin learning. Extensive experimental results demonstrate that KFNN significantly outperforms all the other state-of-the-art algorithms and exhibits greater robustness in various crowdsourcing scenarios.

KFNN: K-Free Nearest Neighbor For Crowdsourcing
Wenjun Zhang
Liangxiao Jiang"
School of Computer Science China University of Geosciences Wuhan 430074, China wjzhang@cug. edu. cm
School of Computer Science China University of Geosciences
Wuhan 430074, China 1j iang@cug edu. cn
Chaoqun Li
School of Mathematics and Physics China University of Geosciences Wuhan 430074, China chqli@cug edu. en
Abstract
To reduce annotation costs, is common in crowdsoureing collect only few noisy labels from different crowd workers for each instance. However, the limited noisy labels restrict the performance of label integration algorithms in inferring the unknown true label for the instance. Recent works have shown that leveraging neighbor instances can help alleviate this problem. Yet, these works all assume that each instance has the same neighborhood size, which defies common sense. To address this gap. we propose a novel label integration algorithm called K-free nearest neighbor (KFNN). In KFNN, the neighborhood size of each instance is automatically determined based on its attributes and noisy labels. Specifically, KFNN initially estimates Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution is then utilized to enhance the multiple noisy label distribution of each instance. Subsequently, a Kalman filter is designed to mitigate the impact of noise incurred by neighbor instances. Finally, KFNN determines the optimal neighborhood size by the max-margin learning. Extensive experimental results demonstrate that KFNN significantly outperforms all the other state-of-the-art algorithms and exhibits greater robustness in various crowdsourcing scenarios. Our codes and datasets available at https://hhhbbbomm/aaaiiiiikaKaaa
Introduction
cost-effective way to obtain annotated instances than traditional expert annotation [I]. Through crowdsourcing platforms such as Figure Eight and Clickworker, instances be annotated by crowd workers a low cost 2 3. While more affordable, these workers possess less expertise than domain experts and are more prone to assigning noisy labels to instances 4 To address this issue, the concept of repeated annotation is introduced and becomes popular in crowdsourcing [5]. With repeated annotation, each instance is annotated by several workers, thereby obtaining multiple noisy labels. To train supervised models using multiple noisy labels, two main categories of methods have been developed: one-stage methods and two-stage methods. One-stage methods (6 7 g train models directly using multiple noisy labels. Two-stage methods 9 first infer the unknown true label for each instance from its multiple noisy labels via
Crowdsourcing provides
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

label integration (also known as answer aggregation or ground truth inference) [00] and then train models on integrated labels. One-stage methods, although end-to-end, can only be used to train specifically designed models. As result, label integration, which is required for the more common two-stage methods, has received a great deal of attention from researchers.
It has been theoretically demonstrated that, when worker annotation is more accurate than random annotation, the more noisy labels an instance receives, the easier becomes to infer its unknown true label [Τ.. However, to reduce annotation costs, only few noisy labels can be collected for each instance in crowdsoureing. The limited noisy labels restrict the performance of label integration algorithms in inferring the unknown true label for the instance. Furthermore, some common strategies in crowdsoureing, such as worker modelling, worker elimination and task assignment [12). fail to mitigate the effects of limited labels in label integration. To alleviate this problem, recent works have begun to focus on leveraging neighbor instances [13 14 I. These works successfully improve the performance of label integration by leveraging the information from neighbor instances obtained by the K-nearest neighbor (KNN) algorithm. However, due to the use of KNN, these algorithms all assume that each instance has the same neighborhood size. This assumption is difficult to hold because it defies common sense, e.g. instances close to the center of classes should have more neighbors than instances close to the boundary of classes.
To address this gap, we propose novel label integration algorithm called K-free nearest neighbor (KFNN). In KFNN, the optimal neighborhood size of each instance is automatically determined based on its attributes and noisy labels. Notably, KFNN is different from some supervised works [ [6] that determine the optimal K-value for KNN. Unlike in supervised learning, the true label of each instance in crowdsoureing unknown and only its multiple noisy labels can used, which makes difficult to model the relationship between the instance and all classes. To do this, KFNN initially estimates a Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution is then utilized to enhance the label distribution for each instance. Subsequently, a Kalman filter is designed to mitigate the impact of noise incurred by neighbor instances. Finally, KFNN determines the optimal neighborhood size by the max- max-mn learning. In general, the contributions of this paper can be summarized as follows:
We reveal the limitations caused by fixing the neighborhood size in existing label integration algorithms and propose a novel algorithm called KFNN. In KFNN, the neighborhood size of each instance is automatically determined based on its attributes and noisy labels.
We estimate a Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution enhances the multiple noisy label distribution of each instance.
We design Kalman filter mitigate the impact of noise incurred by neighbor instances and then determine the optimal neighborhood size by the max-margin learning, which provides strong theoretical support for our algorithm.
Extensive experimental results demonstrate that KFNN significantly outperforms all the other state-of-the-arr label integration algorithms and exhibits greater robustness than existing algorithms in various crowdsourcing scenarios.
2
Related work
Depending on whether neighbor instances are leveraged or not, existing label integration algorithms can be divided into two categories. The first category of algorithms does not leverage neighbor instances, which considers only the information of the instance itself or the information of all instances globally in label integration. For example, [17] models the ability of each worker with a confusion matrix. In this matrix, each element reflects the probability that this worker annotates an instance with the class corresponding to the row as the class corresponding to the column. [ [9] are Bayesian versions of [17,, which can be used for binary tasks and multi-class tasks, respectively. Further, 20 21 improve [99 by introducing the correlation between workers. [1] 22 are classical algorithms based on majority voting and they tend to use the label with the highest number synchronously model the ability of workers and the
of votes as the integrated label. (24 25 26
difficulty of tasks from different perspectives.
[27 28 use clustering algorithms to divide instances into different clusters from different views, and then map these clusters to different classes. Recently, 29 augments the multiple noisy label distributions of instances as new attributes to the original

---

attribute space and then learns a classifier on the augmented attribute space to predict the integrated labels of instances. multi-order information in label integration.
9 constructs graphs for workers and uses graph neural network to aggregate
The second category of algorithms performs label integration by leveraging the information from neighbor instances obtained by the KNN algorithm. For example, [13) proposes to use the labels assigned the neighbor instances of an instance to augment this instance's multiple noisy labels and use the augmented multiple noisy labels to infer the integrated label of this instance. [4] considers both nearest and farthest neighbors in weighted voting to address blass-imbalanced tasks. Further, inspired by label distribution learning [30 31], given an instance, [T] iteratively absorbs the label distributions of its neighbor instances into its label distribution through label distribution propagation.
While simpler and more efficient, the first category of algorithms are limited in effectiveness because each instance can only obtain few noisy labels. Both experimental results and theoretical analysis demonstrate the effectiveness of the second category of algorithms in leveraging the information from neighbor instances. However, these algorithms all assume fixed neighborhood size for each instance, which is often unrealistic and thus limits their performance. To further ensure that each instance has free neighborhood size, this paper proposes novel label integration algorithm called KFNN. KFNN automatically determines the optimal neighborhood size for each instance based on its attributes and noisy labels, which improves the performance and robustness of label integration.
3 Algorithm
In this section, we respond to how to automatically determine the optimal neighborhood size for each instance. First, we present some basic notations crowdsourcing and then define the problem settings. Subsequently, we introduce our KFNN for label integration.
3.1 Preliminary
Let D = ((a,, L.)) denote a crowdsourced dataset, where N is the number of instances, and æ; denotes the i-th instance in D. æ; can be represented as {rim DM mm=1- Here, M is the dimension of attributes, and Zim denotes the attribute value of x; on the m-th attribute Am- L; denotes multiple noisy labels of æ;, which can be expressed as {
R is the number of workers and 4,, denotes the label of x; annotated by the r-th worker u,. 4ir takes a value from a fixed set (1,, ......, [. where Q is the number of classes, Gg denotes the y-th class and -1 indicates that u, has not annotated æ,, Label integration aims to infer an integrated label ÿt for a;
and minimize the error between d and the unknown true label yı.
Recent works mitigate the restriction of limited noisy labels on the performance of label integration. Here, at denotes the k-th nearest neighbor of æ; and K is the neighborhood size. However, in these works, the value of K is fixed for each instance within the same dataset, which does not make sense. On the one hand, instances closer to the center of a class benefit from a larger K, as it enables them to collect more labels from similar instances. Conversely, for instances close to the boundary of classes, a larger K plays a negative role in label integration. On the other hand, using fixed K can bias algorithms towards the majority class llas--imbllanced datasets, as instances from the majority class are more likely to dominate the neighborhood of instances from minority classes. Therefore, we define the Problem to be addressed in this paper as follows:
have shown that leveraging neighbor instances N₂ { of x; can
Problem 1. Given a crowdsourced dataset D, how to automatically determine the optimal neighbor- hood size K for each instance T, with {1 J and [44r but without y:.
Problem I camnot be treated simply as learning an optimal neighborhood size for the KNN algorithm in supervised learning [15, [6]. This is because the true labels of instances in crowdsourcing are unknown. As a result, K can not evaluated accurately by supervised metrics such as classification accuracy. Moreover, label integration does not divide the crowdsourced dataset into training, vali- dation and test sets, which means that KFNN has to determine K immediately when inferring fi, rather than with validation phase.
3

---

3.2 K-free nearest neighbor algorithm
In this subsection, we propose our KFNN address Problem We argue that K should be related to the information from both the attribute space and the multiple noisy label space. Based on this, KFNN divides Problem| into two parts: 1) How fuse the information from the attribute space and the multiple noisy label space? 2) How to determine an optimal K for a.? Correspondingly, KFNN consists of two components, namely label distribution enhancement and K-free optimization, which used to address the two parts of Problem
3.2.1 Label distribution enhancement
For each instance T;, {Zim m= reflects all the information of the attribute space and (1rr ( reflects all the information of it in the multiple noisy label space. Inspired by label enhancement (LE)
32 33 we design a label distribution enhancement (L.DE) component for KFNN. LDE recovers a potential label distribution using (Fam m==1 and then enhances the multiple noisy label distribution
calculated from {4r by this potential label distribution. Specifically, KFNN first uses majority
voting to initialize the integrated label üt for æ; follows:
U arg max
p(cyl Lt);
(1)
cEfer.e......
where p(cg|Li) can be calculated follows:
ΣP= 8llir, Cg)
L.) L.)
(2)
Σ9 E 8(1or,Cg)
Here, p(c, LL) reflects the proportion of labels in L; that take the value ey. The function 8(-) outputs I if its two parameters are identical, and 0 otherwise. Subsequently, according to yi. the crowdsourced dataset D can be divided into Q subsets (Dy]g= The subset D, contains all instances with initial integrated labels of cg. i.e., D, = {æ; 10: = G Then, KFNN calculates a Mahalanobis distance
distribution {d(æ;, Dg) as follows:
d(x,, Dg) = V (ar, M4 C, (a., 14))
where u, denotes the centroid of Dy and C,1 denotes the inverse matrix of the covariance matrix of Dg- d(x,,, D2) is the Mahalanobis distance from æ, to D, calculated in the attribute space. A larger d(x:; D₂) means that æ; is less likely to belong to Cq. conversely a smaller d(x,, D2) means that an; tends to belong cq. Therefore, {d(#,, Dy) can be used model the relationship between each instance and all classes. Based on this, {d(x,, D can be transformed into a potential label
distribution {p(c, x; D as follows:
max(fd(æ,, D2) d(xx; D₂)
p(cylæ,, D2) mor((d(z;,DD))
(4)
mén({d(as D=)
where mar(.) and min(-) denote the maximum and minimum values of the set, respectively.
In addition to the potential label distribution, a multiple noisy label distribution {p(c, [L) can also be directly transformed from L; Different from {p(c,/æ, D2) which learns the potential relationship between instances and classes from the attribute space, {p(c,) 1 learns the label distribution reflected by noisy labels from the multiple noisy label space. Finally, KFNN fuses them
into an enhanced label distribution P: = {pig ( by averaging as follows:
p(cy x Dg) p(cy L.)
Pig pp D₂) p(cy (.)
(5)
In this way, the enhanced label distribution P. can fuse the information from the attribute space and the multiple noisy label space. Therefore, the first part of Problem Tas been addressed.

---

3.2.2 K-free optimization
After obtaining P. by label distribution enhancement, KFNN proceeds to determine the optimal neighborhood size Kद for æ;. First, KFNN calculates the distance between each pair instances æ and 22 by:
d( d( x22
Σ
d( x2 Dy)
(6)
9=1
where (2 can be calculated as follows:
d( (2) = ((n 22) C (a) 22).
Compared to the Euclidean distance, Eq. (6) introduces the label information by calculating the distance between #1 and 22 on each subset 7g According Eq. (6) can calculate distances between 2; and all instances in D. By sorting these distances we can obtain a neighbor sequence < wi..... a,.,.,,,...... ,....., x > for æ;, Here, at is the k-th neighbor instance æ; satisfying d(æ,, at) Z d( (e-)) when is greater than 1. Then, we calculate the weight wis for a as follows:
Σ 8(lir: likr)
Wik ΣK 8(1rr, 1)] 8(likr, -1]]
where lür denotes the label of œ annotated by the r-th worker u, Wik. reflects the proportion of workers assigned the same label for x, and æ². Subsequently, æ; is allowed to absorb the enhanced label distributions of neighbor instances in the neighbor sequence one by one. Let P. t denote the label distribution of æ; after absorbing at which can be updated follows:
Pig k-1 + Wik * Pikg
Pig
2 2,
k-1 Wik Pikg!
where Pikq denotes the probability value corresponding cq in the enhanced label distribution of æ Since the first neighbor instance of is itself, P P, when & is equal to 1.
According to P. KFNN calculates a class margin as follows:
M max (P:) sec(P.).
(10)
where sec(-) denotes the larnes- value of the set. Since the true labels are unknown, Eqs. are all designed based on multiple noisy labels, which lead to that Mr contains a degree of noise incurred by neighbor instances. Therefore, KFNN designs Kalman filter mitigate the impact noise in Mg as follows:
M₁ = =Ma-1
PF PP
Kr Pt
(11)
Pf +
Mr Mī + Kr * (Mr - M²)
Pa = - Kc) * Pr
where Me denotes the filtered margin, determined by both the estimated margin M and the calculated margin Mg. The designed Kalman filter can be divided into an estimation phase and an update phase. In the estimation phase, the filter estimates M, and the estimated error Pf based on the filtered margin M and error Pk-1 of the previous time index. In the update phase, the filter first updates the Kalman gain Kc of the k-th time index and then updates M₁ and error Pk of the k-th time index according to KR- and 3 are the process error and the measurement error in the Kalman filter. When & is equal to O, Mr takes the value of 0 and Ps takes the value of 1.
5

---

To address the second part of Problem by the max-man margin learning as follows:
1
KFNN determines the optimal neighborhood size K for æ;
Ki arg max Mr. (1 ,N)
(12)
Ultimately, according K?, KFNN updates the integrated label in for æy as follows:
Di arg max
(13)
oE (cr,,aa
The whole learning process of KFNN is shown in Algorithm 1 In Algorithm | lines 1-3 initialize integrated label and multiple noisy label distribution for each instance and their time complexity is O(NQR). Line 4 divides the crowdsourced dataset D into Q subsets and its time complexity is O(NQ). Lines 5-9 perform label distribution enhancement and their time complexity is O(N Line 11 calculates the distances from D; to other instances and sorts these distances, its time complexity is O + N log(N)). Lines 12- 16 calculate the margins {M& k-= and their time complexity ONN + NQ). Line filters the margins [M and its time complexity O(N). Line 18 determines the optimal neighborhood size and its time complexity is O(V). Line 19 infers the integrated label for each instance and its time complexity is O()). Therefore, the time complexity of lines 10-20 is O(N (M²Q + log (N) + R)). If only the highest order terms are taken, the time complexity of KFNN is Q + Wogg(N) + NR + QR)).
Algorithm 1 The learning process of KFNN
Require: D { (æ,, 1 crowdsourced dataset; a, the predefined parameters
Ensure: {0:),-1 the integrated labels
1: for 1 N do
2: Initialize 0J and (L)), for x, by Eqs.
3: end for
4: Divide D into {D₂) based on Ú:
5: for to N do
6: Calculate {d(x,, Dy) for Φ, by Eq. (3)
7: Transform (d(æ;; Dg) g=1 into (ple, ai. D by Eq.
8: Fuse {p(c, xi, D,) and (p(c,) L,)] into P. {pig) 1= by Eq. 5
9: end for
10: for to N do
Calculate xi.... ......,
T. ................................ a. for æ; by Eqs. 6
12: for k 1 to N do
Caleulate the weight wi for o. by Eq. Update the label distribution P. by Eq.
15:
Calculate the M2 by Eq. I0
16:
end for
17:
Filter {M&) k=1 using the designed Kalman filter by Eq. 11
18: Determine the optimal neighborhood size K for æ; by Eq. 12
19: Infer the integrated label ÿi for æ; by Eq. [3
20:
end for return (0:1111
21:
Theoretical analysis
In this section, provide some detailed theoretical analysis for KFNN. First, in Eq. 66) KFNN
defines the distance d(x xy2) between 21 and Vy based on the Mahalanobis distance di d( x2 Dg) rather than the traditional Euclidean distance d 22). According Eqs. 77. the Mahalanobis
distance works based on a basic assumption, which can be described as follows:
Assumption 1. Given the subser Dar its covariance matrix C2 is nonsingular matrix.
The Assumption I holds based on the condition that C, is non-zero, which is usually satisfied. Even this condition is not satisfied, we can ensure that the Assumption Caaiiiooooooeee Tholds by adding a small value to each element of the principal diagonal on C, until |C, is non-zero.

---

Theorem 1. If Assumption hollds there will be an orthogonal matrix P satisfying that P - Ppp P" PP A, where A is diagonal matrix with all M eigenvalues of Cy as its elements of the principal diagonal.
Due to the limited pages, the proof of Theorem is provided in Appendix A Based on Theorem we can obtain some interesting corollaries about Eqs. (
Corollary 1. Compared to d x2)), d( x2) in Eq. 66 does not suffer from the correlation and magniude of attributes.
Proof. According to Theorem ! d( D2) can transformed as follows:
D2
(a) - 22) ( - x22)
1P (ap-1)-1(aa) 1-11(11 -PP) =V(P" (P" x2)) (x) x22)
(x1 - x2) =22
x22)
(14)
When A is not considered, the derivation of Eq. (4 implies that d(æ1 22 (Dg) is the Euclidean distance of instances after orthogonal transformation using p' After orthogonal transformation, the attributes are independent of each other, so d( X2) does not suffer from the correlation of attributes. A- is equivalent diag( X Ag ..... Xxr ) where AM is the M-th eigenvalue of C, and is
equal to the variance on the direction of the corresponding eigenvectors. Briefly, A -1 ensures that the calculated result each dimension is normalized by the corresponding variance when calculating the distance by Eq. 77). Therefore, d(æ1,0)) 22) does not also suffer from the magnitude of attributes. O
Corollary 2. Compared to (æ( x2). d( x2) in Eq. (6) provides smaller distance for æ and 22 coming from the same class.
Proof. pd causes the original attribute space be rotated according to the direction of the eigenvec- tors of C. and A-1 causes the rotated attribute space to be scaled according to the eigenvalues C.. Referring to the principle of principal component analysis [34), the eigenvectors of C2 reflect the principal component directions of Dg- This means that d(x11 222) will provide smaller distance for instances coming from the same class compared to (x 22)).
Assumption 2. When we estimate Mī based on M the estimated error satisfies N(O, Pi ).
When measure My by Eq. 100, the measurement error satisfies N(O, (3).
The Kalman filter we designed as Eq. (11) works based on Assumption 2 which usually holds
because the noise in practice usually satisfies normal distribution. Since Ma-1 changes in each
time index, variance of the estimated error PE changes with the time index. Since Eq. (TO remains constant, the variance of the measurement error 3 is constant. According to Assumption 2 the following theorem can proved:
Ph_ the error between the filtered margin
Theorem 2. When the Kalman gain K₂ takes the value P, +B*
Mg and the true margin M2 minimized.
Proof. When Assumption 2 holds, due Mr = ME K₂ (M₂ -M5), it can be proved that minimizing the error between Mr and M2 is equivalent to minimizing the variance of Mg. Since MR and Mr are independent of each other, the following equation can be derived:
(ar(MR) = Var(M + Kr (M, - -M,)) = = - K2)2 Var(M + KG Var(Mr)
(15)
where Var(-) denotes the variance of the variable. According to Assumption 2 V (ar(M ( equals to P and (M)) equals to 3. To minimize the error between the filtered margin Mr and the true
7

---

margin Mag we can calculate the partial derivative @ @Ver(Mee) 8K* as follows:
a aVar(Mu) = -2 -2* * 11* K&)) * Pk + KR #B.
(16)
Ultimately, it can be proved that K& is equal to
Pe Pr +B
by setting 2v aVar(Me) to O.
8K*
Theorem 3. The larger Mr is, the better the corresponding neighborhood size k is.
Theorem 3 ensures the effectiveness of KFNN in determining the optimal neighborhood size by the max-margin learning, and its proof is provided in Appendix B due to the limited pages.
Experiments
5.1 Experimental setup
To evaluate the effectiveness of KFNN, we construct extensive experiments on the whole 34 simulated and two real-world crowdsourced datasets published on the Crowd Environment and its Knowledge Analysis (CEKA) 35 platform. For simulated datasets, filter ReplaceMissings Values in the Waikato Environment and Knowledge Analysis (WEKA) [36] platform replace all missing values. Subsequently, with the CEKA platform, we hide true labels of simulated datasets and simulate five workers whose label qualities are randomly generated from a normal distribution with N(0.65, 0.05²) annotate these datasets. The real-world datasets, Income and Leaves, which were both collected from the online platform Amazon Mechanical Turk (AMT), can be used directly without any processing since they do not contain missing values.
first use the unsupervised attribute
We compare our KFNN with six state-of-the art label integration algorithms. Among them, MV (majority voting) [11] is the simplest label integration algorithm and is used as baseline for all algorithms. IWM WVV (iterative weighted majority voting) [22]. AALI (attribute augmentation based label integration) 229]. and LAGNN (label aggregation with graph neural networks) ( are three state-of-the-art label integration algorithms that do not leverage neighbor instances. LAWMV (label augmented and weighted majority voting) [33 and MNLDP (multiple noisy label distribution propagation) 1 are two state-of-the-art label integration algorithms that leverage neighbor instances. For MV, we use the existing implementation of the CEKA platform. For IWMV, AALI, LAGNN, LAWMV, and MNLDP, we use the implementations provided by their authors. All parameters of the comparison algorithms are set to the recommended values the corresponding published papers. In addition, since true labels are unknown in our experiments, we use the lazy version of LAGNN. In our KFNN, a and are set 0.1 and by default.
The performance each algorithm is evaluated using the Macro-Fl score, which highlights the performance of algorithms different classes and better reveals algorithmic limitations compared to traditional integration accuracy. Due to the limited pages, more detailed descriptions of the experimental datasets and metrics provided in Appendix C All experiments are independently repeated ten times on Windows 10 machine with an AMD Athlon((m) X4 860K Quad Core Processor @ 3.70 GHz and 16 GB of RAM, and we report the average results of ten experiments.
5.2 Results and discussions
Simulation experiment results. Table shows the detailed Macro-F1 score (%) comparisons of each label integration algorithm on each simulated dataset, respectively. Based on these results, we perform the Wilcoxon signed-rank test (37) to further compare each pair of algorithms. Table 3 summarizes the Wilcoxon test results. In Table 3 the symbol indicates that the algorithm in the row significantly outperforms the algorithm in the corresponding column, the symbol indicates the exact opposite of that indicated by the symbol e, and the missing item indicates no significant difference between the algorithm in the row and the algorithm in the column. The significance levels of the lower and upper diagonals are a = 0.05 and a 0. respectively. Based on these experimental results, we can summarize the following highlights: 1) The average Macro-F1 score of KFNN on all datasets 99,,4, which is much higher than those of MV (72.46%), IWMV (72.71%) 1)), AALI (72.95%), LAGNN (73.71%), LAWMV (73. (73,44%) and MNLDP (76.68%). KFNN achieves the highest Macro-Fl
8

---

JWMV Table 1: The Macro-Fl score (%) comparisons for KFNN versus its comparison algorithms on 34 simulated datasets.
Table 2: The integration aceuracy (%) compar- isons for KFNN versus its comparison algorithms
on 34 simulated datasets.
Dansee
MV
IWMV
AALE
LAGNN
LAWMY
MNLDP
KINN
Danser
MBV
IWMV
AALE
LAINN
LAWMV
MNLDP
KFNN
smecl
77.51
79.30
53.69
smncal
64.06
63:11
91.38
ualislgy
45.17
aulislagy
75.45
79.30
78.10
R1:77
scalnce-saalz
80.55
$1.36
$1.87
76.41
belane-scalz
79.25
80.32
80.35
63.52
boding
66.20
06:20
70131
69:25
73.00
75.51
74:12
79
64.59
10.99
K1.91
lecas--eacer
655.73
65:73
66.92
66.79
55.29
$9.63
57.24
becal--cnner
76.08
76.08
75.94
77.24
50.10
77.17
89.23
beeast W
65.92
68.92
79.92
09.51
92.58
87.45
$4.50
beeat.
75.15
76.15
$7.00
77.14
95.71
92.54
$7.55
tar
80.22
81.42
81.83
62.56
$1.85
70.84
94.25
car
81.46
R2.31
82.93
63.26
63.25
85. M0
94.34
credit
72.88
72.88
79.19
74.32
$7.36
76.38
61.72
credit a
25.17
75.17
77.13
76.43
K8.91
11 23
$3.33
cralling
65:42
65.23
cralling
75.81
75.81
76.00
77.05
70.78
19.62
diabetes
dubetes
74.39
74.45
79.53
75.55
74.49
79.50
heart-h
78.50
78.50
78.58
72.07
82.08
79.56
84.25
50.17
73.61
71.51
75.43
14.0
76.25
78.00
75
75:00
76.61
64.41
81.26
79.30
hepailti
$4,34
54.38
$7.05
55.74
00.08
65.01
61.77
hepaitis
74.55
74.58
74.26
75.08
57.10
95.19
79.25
hunc-eatii
67.00 60
67.59
70.35
12.18
75.34
71.52
73.34
73.04
73.04
74.51
77.35
6A.51
77.47
79-02
typortsseid
58.04
58.49
42.40
00.70
24.00
62.16
63.56
hyportooood
90.52
81.53
79.25
K3.24
92.29
93.27
92.40
commpphrrd
70.61
70.71
76.38
63.69
75.91
62.31
77.00
77.15
79.86
73.06
90.74
85.00
ina
81:12
81.84
97.82
07.14
81.57
81.53
98.27
07.13
97.13
67.03
75.45
72.16
75.61
74.39
82.63
laterr
03.77
94.19
95.58
94.55
98.97
99.61
99.49
lnterr
93:70
94.17
95.64
94.75
96.57
99.52
99.49
kymph
69.49
08.69
$6.85
71.52
64.56
59.88
70662
byuph
77.97
77.70
78.24
78.31
54.56
82.97
50.41
mushroom
75.06
75.06
$1.97
76.19
92.09
95.03
97.98
mushromm
76.71
76.71
84.21
76.25
95.29
95.78
00.09
ecgmani
89.35
90:70
92.10
00.77
96.55
98.16
98.50
eogmani
89.35
00:50
02.10 10
90.77
96.55
16.16
98.80
SK3
25:60
29:40
31.23
24.77
4.85
46.92
45.46
SKS
20.87
20.87
78.34
76.28
91.98
89.81
89.78
74.75
74.75
77.94
77.49
82:04
75.91
73.91
75.19
50.00
83.41
81.78
spambase
71:11
sparnhase
67.57
42.70
74.43
71.99
vebicle
95.38
97.23
$7.29
$8.85
92.00
85.39
87.23
53.97
92.09
96.43
65.69
65.69
67.18
71 03
90.35
83.07
91.39
14:11
14.11
77.94
75.37
92.64
85.54
91.20
wwwl
92.48
03307
94.23
92.35
95.23
99.51
99.05
vowel
02.35
12.99
94.21
92.25
94.98
99.81
99.05
waseform
82.15
83:70
83.04
64.53
94.88
91.88
92.49
Mavefomm
82.13
83.50
82.96
64.53
94.59
91.89
92:49
200
75.50
70.34
76.02
08
$1.93
$2.64
$0.00
200
90.40
80.24
87.13
51.08
91.78
91.78
$6.63
Avcrage
72.46
72.71
72.95
73.71
73.44
76.08
Average
79.37
79.30
57.32
95.33
Table 3: The Macro-FI score (%) comparisons using Wilcoxon tests for KFNN versus its com- parison algorithms.
Table 4: The integration accuracy (%) compar- isons using Wilcoxon tests for KFNN versus its
comparison algorithms.
MV JWMV
AALI
LAGNN
LAWMY
MNLDP
KENN
MV IWMV
AALI
LAGNN
LAWMY
MNLDP KUNN
MV
MV
IWMV
AALI
AALI
LAGNN
LAGNN
LAWMV
LAWMV
MNLDPP
MNLDP
KPNN
KINN
score, which indicates that KFNN is more effective and robust than these comparison algorithms in various crowdsourcing scenarios. 2) Among all comparison algorithms of KFNN, MNLDP performs better than IWMV, AALI and LAGNN, which demonstrates the advantages of leveraging neighbor instances. 3) Based on the Wilcoxon test results, KFNN significantly outperforms all comparison algorithms, which strongly validates the effectiveness and robustness of KFNN. Besides, we also observe the experimental results in terms of the integration accuracy, which are shown in Tables 2 and According to Tables 2 integration accuracy compared with these sfate-of-the-at label integration algorithms. These results again validate the effectiveness and robustness of KFNN.
and 4 we can see that KFNN can also achieve better or comparable
Real-world experiment results. Compared to simulated crowdsourced datasets, real-world crowd- sourced datasets may include some special factors that influence label integration work effectively, such as sparsity and bias. Therefore, we further observe the performance of KFNN and its comparison algorithms on two real-world datasets, Income and Leaves. Figure shows the detailed Macro-F1 score (%) and integration accuracy (%) comparisons of each label integration algorithm on Income and Leaves, respectively. As can be seen from Figure [] compared these state-of-the-art label integration algorithms, our KFNN achieves the highest integration accuracies and Macro-Fl scores on both Income and Leaves. These results strongly support the effectiveness of our KFNN
Parameter sensitivity analysis. There are two parameters a and 33 that can be adjusted in the Kalman filter designed by KFNN. To observe the effect of these two parameters on the performance of KFNN, we perform the parameter sensitivity analysis for KFNN on Income and Leaves We change both and 3 from 0.1 to I and then observe the Macro-Fl score of KFNN on two datasets.

---

100
C2
(a) F1 (Income)
(b) Accuracy (Income)
(c) Fl (Leaves)
(d) Accuracy (Leaves)
Figure The Macro-Fl scores (%) and integration accuracies (%) of KFNN and its comparison algorithms on the Income and Leaves datasets.
Figure 2ajand and Figure sho the Macro-Fl score of KFNN on Income and Leaves when a and B vary. Based these results, we find that KENN is more sensitive compared a. As tends to I, KFNN tends to achieve optimal performance. Therefore, the default value of 3 this paper set to 1. hardly affects the performance of KFNN, which is 0.1 by default.
(a) Income
(b) Leaves
(a) Macro-FI FI score
(b) Class margin
Figure 2: The Macro-F1 score (%) of KFNN on the Income and Leaves datasets when a and B vary from 0.1 to 1.
Figure 3: The Macro-F1 score (%) and class
margin (%) of KFNN its components the Income dataset.
Ablation experiment. There two components in KFNN, namely label distribution enhancement (LDE) and K-free optimization (KF). To validate their effectiveness, we observe the Macro-Fi -1 score of KFNN after taking away each component the Income dataset. For simplicity, we use "KFNN- KF" to denote the variant of KFNN after taking away the component KF. Similarly, we create its another two variants "KENN-LDE" and "KFNN-KF-LDE" Based the results shown in Figure [3 can be seen that the performance becomes worse when any component is taken away. These results validate the effectiveness of LDE and KF. Figure 33 shows the change of the class margin before and after using our designed Kalman filter (observed on the first instance of Income). As can be seen from Figure 3b compared the margin before filter (M&), the filtered margin (M&) changes smoother. These results validate the effectiveness of our designed Kalman filter, which successfully mitigates the impact of noise incurred by neighbor instances.
6 Conclusion and future work
To ensure that each instance in crowdsourced datasets has free neighborhood size, we propose novel algorithm called KFNN. KFNN consists of two key components, namely label distribution enhancement and K-free optimization. Label distribution enhancement fuses the information from the attribute space and the multiple noisy label space. K-free optimization automatically determines the optimal neighborhood size for each instance by the max-margin learning. Both theoretical analysis and experimental results validate the effectiveness and robustness of KFNN.
Nevertheless, there are still some limitations in KFNN that can be improved in the future. For example, the parameters a and 8 in the Kalman filter designed by KFNN can automatically adapt to the dataset, which restricts the robustness of KFNN. In addition, in Eq. (4), transforming distance distribution into potential label distribution using max-min normalization rough. Considering that the distance metric is not effective across all datasets (e.g., autos and breast-cancer in Table [D. this transformation may also lead KFNN performing poorly. In the future, we will design more sophisticated parameters and transformations improve KFNN.
10

---

Acknowledgment
The work was partially supported by National Natural Science Foundation of China (62276241), Foun- dation of Key Laboratory of Artificial Intelligence, Ministry of Education, P.R. China (A (AI2022004), and Science and Technology Project of Hubei Province- Province-Unveiling System (2021BEC007).
References
[1] L. Jiang, H. Zhang, F. Tao, and Li, "Learning from crowds with multiple noisy label distribution propagation," " IEEE Trans. Neural Networks Learn. Syst., vol. 33, no. 11. pp. 6558-6568, 2022.
[2] P. Chen, Y. Yang. D. Yang. H. Sun, Z. Chen, and P. Lin, "Black-box data poisoning attacks crowdsoure- ing." in Proceedings of the Shirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China. ijeai.org, 2023, pp. 2975-2983.
[3] Z. Chen, H. Sun, H. Hc, and P. Chen, "Learning from noisy crowd labels with logics," in 39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, CA, USA, April 3-7, 2023. IEBE, 2023, pp. 41-52.
[4] J. Li, Y. Kawase, Y. Baba, and H. Kashima, "Performance as constraint: An improved wisdom of crowds using performance regularization," in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, C. Bessiere, Ed. ijeai.org, 2020, pp. 1534-1541.
[5] V. S. Sheng. J. Zhang, B. Gu, and X. Wu, "Majority voting and pairing with multiple noisy labeling," IEEE Trans. Knowl. Data Eng., vol. 31, no. 7, pp. 1355-1368, 2019.
[6] F. Rodrigues and F. C. Pereira, "Deep learning from crowds." in Proceedings of the Phirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (FAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, S. A. Mcllraith and K. Q. Weinberger, Eds. AAAT Press, 2018 pp. 1611-1618.
[7] Z. Chen, Wang. Sun, P. Chen, T. Han, X. Liu, and J. Yang, "Structured probabilistic end-to-end learning from crowds," " in Proceedings of the Twenty-Ninig International Joint Conference on Artificial Intelligence, IJCAI 2020, C. Bessiere, Ed. ijcai.org, 2020, pp. 1512-1518 1518.
[8] J. Li, H. Sun, and J. Li, "Beyond confusion matrix: learning from multiple annotators with awareness of instance features," " Mach. Learn., vol. 112, no. 3, pp. 1053-1075, 2023.
[9] Z. Ying, J. Zhang, Q Li, M. Wu, and V. S. Sheng, "A little truth injection but big reward: Label aggregation with graph neural networks," IEEE Trans. Pattern Anal. Mach. Intell. vol. 46, no. 5, pp. 3169 3182, 2024.
[10] J. Zhang. X. Wu, and V. Sheng, "Learming from crowdsourced laheled data: a survey," Artif. Intell. Rev. vol. 46, no. 4. pp. 543-576, 2016.
[11] V. S. Sheng, F. Provost, and P. G. Ipeirotis, "Get another label? improving data quality and data mining using multiple, noisy labelers," in Proceedings of the 14th ACM SICKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008, Y. Li, B. Liu, and
S. Sarawagi, Eds. ACM, 2008, pp. 614-622.
[12] G. J. Wang, Zheng, and M. J. Franklin, "Crowdsourced data management: survey," IEEE Trans. Knowl. Data Eng., vol. 28, no. 9, pp. 2296-2319, 2016.
[13] Z. Chen, L. Jiang, and C. Li, "Label augmented and weighted majority voting for crowdsoureing." Inf. Sei., vol. 606, pp. 397-409, 2022.
[14] W. Zhang, L. Jiang. Z. Chen, and C. Li, "Fnnwy: Farthest- nearest neighbor- based weighted voting for class- iass-imbalanced crowdsourcing," Sci. China Inf. Sci., pp. 10070/// 432-123-33322221 023-3834-1333 2023.
[15] J. Olsson, "An analysis of the coupling between training set and neighborhood sizes for the knn classifier," in SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, USA, August 6-11, 2006, E. N. Efthimiadis,
S. T. Dumais, D. Hawking, and K. Järvelin, Eds. ACM, 2006, pp. 685-686.
[16] S. Zhang, X. Li. M. Zong, Zhu, and D. Cheng, "Learning for knn classification." ACM Trans. Intell. Syst. Technol., vol. 8, no. 3. pp. 43:1-43:11 2017.
[17] A. P. Dawid and A. M. Skene, "Maximum likelihood estimation of observer error-rates using the em algorithm." Journal of the Royal Statistical Society: Series (Applied Statistics), vol. 28, no. 1. pp. 20-28, 1979.
[18] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy, "Learning from crowds."
J. Mach. Learn. Res., vol. 11, pp. 1297-1322, 2010.

---

[19] H. Kim and Z. Ghahramani, "Bayesian classifier combination." in Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012, La Palma, Canary Islands, Spain, April 21-23, 2012, ser. JMLR Proceedings, N. D. Lawrence and M. A. Girolami, Eds., vol. 22. JogR.org. 2012, pp. 619-627.
[20] M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi, "Community- -oomeunit--bsedd based bayesian aggregation models for crowdssurreing" in 23rd International World Wide Web Conference, WWW '14, Scoul, Republic of Korea, April 7-11, 2014, C. Chung, A. 7., Broder, K. Shim, and T. Suel, Eds. ACM, 2014, 4, pp. 155-164.
[21] Y. Li, B. P. Rubinstein, and T. Cohn, "Exploiting worker correlation for label aggregation in crowdsoure- ing. in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 2019, pp. 3886-3895.
[22] H. Li and B. Yu, "Error rate hounds and iterative weighted majority voting for crowdsourcing." CoRR, vol. 0bs/1411.4086 2014.
[23] T. Tian. J. Zhu, and Y. Qiaoben, MMax-margin majority voting for learning from crowds." JEEE Trans. Pattern Anal. Mach. Intell. vol. 41, no. 10. pp. 2480-2494, 2019.
[24] J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. R. Movellan, "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise," in Advances in Neural Information Processing Neural Information Processing Systems 2009. Proceedings of meeting held 7-10 December 2009, Vancouver, British Columbia. Canada, Y. Bengio, D. Schuurmans, Lafferty, C. K. I. Williams, and A. Culotta, Eds. Curran Associates, Inc., 2009, pp. 2035-2043.
Systems 22: 23rd Annual Conference
[25] P. Welinder, S. Branson, S. J. Belongie, and P. Perona, "The multidimensional wisdom of crowds," in Advances in Neural Information Processing Systems 23: 24th Annual Conference Neural Information Prucessing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia. Canada, J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, Eds. Curran Associates, Inc., 2010, pp. 2424-222444 2234-
[26] A. Kurve, D. J. Miller, and G. Kesidis, "Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention." IEEE Trans. Knowl. Data Eng.. vol. 27, no. 3. pp. 794 8044809,270 2015.
[27] J. Zhang, V. S. Sheng, J. Wu, and X. Wu, "Multi-class ground truth inference in crowdsourcing with clustering." IEEE Trans. Knowl. Data Eng. vol. 28, no. pp. 1080-1085, 2016.
[28] G. Wu, L. Zhou, J. Xia, L. Li, X. Bao, and X. Wu, "Crowdsourcing truth inference based on label confidence clustering," ACM Trans. Knowl. Discov. Data, vol. 17, no. 4. pp. 46:1-46:20, 2023.
[29] Y. Zhang, L. Jiang, and C. Li, "Attribute augmentation- based label integration for crowdsoureing." Frontiers Comput. Sci., vol. 17, no. 5, p. 175331,2023.
[30] C. Xu and X. Geng, "Hierarchical classification based on label distribution learning"" " in The Thirty- Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Fire First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, Janury 27 - -Februrby 1, 2019. AAAI Press. 2019, pp. 5533-5540.
[31] Y. Lu and X. Jia, "Predicting label distribution from multi-lahel ranking." in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA. November 28- December 9, 2022, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. 202..
[32] N. Xu, A. Tao, and X. Geng, "Label enhancement for label distribution lcarning," in Proceedings of the Twenry-Sevene International Joint Conference Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, J. Lang, Ed. ijcai.org, 2018. pp. 2926 2932.
[33] K. Wang, N. Xu, M. Ling, and X. Geng, "Fast label enhancement for label distrihution learning," IEEF Trans. Knowl. Data Eng., vol. 35, no. 2, pp. 1502-1514, 2023.
[34] Y. Gao, T. Lin, Y. Zhang, S. Luo, and Nic, "Robust principal component analysis based on discriminant information," IEEE Trans. Knowl. Data Eng., vol. 35, no. 2, pp. 1991-2003 2023.
[35] J. Zhang, S. Sheng, B. Nicholson, and X. Wu, "CEKA: tool for mining the wisdom of crowds." J. Mach. Learn. Res., vol. 16, pp. 2853-2858, 2015.
[36] I. H. Witten, F. Frank, and M. A. Hall, Data mining: practical machine learning tools and techniques, 3rd Edition. Morgan Kaufiaann Elsevier, 2011.
[37] J. Demsar, "Statistical comparisons of classifiers over multiple data sets," J. Mach. Learn, Res., vol. 7, pp.
1-30, 2006.
12

---

Appendix A The proof of Theorem
Proof. The covariance matrix Cg is a symmetric matrix. Therefore, if Assumption Tholds, i.e., the covariance matrix C2 is a nonsingular matrix, C2 must be also M-rrerr symmetric matrix. This means that we can obtain M different eigenvalues and M mutually orthogonal normed eigenvectors when C given. These orthogonal normed eigenvectors can form an orthogonal matrix P, and thus P satisfies P cpP = CPP = A. Here, is diagonal matrix with all M eigenvalues of C its elements of the principal diagonal. Moreover, the order of eigenvalues in A should correspond to the order eigenvectors in P.
Appendix B The proof of Theorem 3
Proof. Theorem 3 hoolds when the distance metric can work effectively given crowdsourced dataset. The distance metric works effectively, which means that the smaller the d(æ1, x2), the more similar 21 and 22 to each other and the more likely they the neighbor sequence < ....... at ..... x. > of æ;, ax and æ;; are more likely to belong to the same class when is small. At this point, when P is updated, the probability corresponding to the unknown true label y/ of æ; will increase. When A gradually increases and exceeds certain threshold, 2; and æ. begin to belong to different classes, at which point the probability corresponding to y; will decrease. In other words, & increases from O, the probability corresponding y; increases first. As i exceeds a certain threshold (the optimal neighborhood size), the probability corresponding to yi begins decrease. Therefore, max Pk) tends to be the probability corresponding to y/ when increases from 0, and tends to be K" when Mr achieves the highest value.
to belong to the same class. Therefore, in
Appendix C More descriptions of the experimental datasets and metrics
Simulated datasets. The descriptions of the whole 34 simulated datasets are listed in Table 5 Here, "#Instances" denotes the number of instances, "#Autributes" denotes the number of attributes. "#Classes" denotes the number of classes, "Missing" denotes whether the dataset contains missing values and "Attribute type" denotes the type of attributes the dataset contains. These datasets are collected from different application scenarios and represent different crowdsourcing requirements.
Real-world datasets. The Income dataset is annotated by 67 workers through the online platform Amazon Mechanical Turk (AMT), and each instance is annotated by 10 different workers. The Income dataset is a binary crowdsourced dataset, which contains 600 instances, 6000 labels, 10 attributes (nominal attributes) and 0 missing values. The Leaves dataset is annotated by 83 workers through AMT, and each instance is annotated by 10 different workers. The Leaves dataset is multi-class crowdsourced dataset, which contains 384 instances, 3840 labels, 64 attributes (numeric attributes) and O missing values.
Experimental metrics.
The integration accuracy is calculated as follows:
ΣN
11=1 8(fi, y:)
Accuracy
(17)
N
The Macro-F1 score is calculated follows:
2o Precision, Recally Precisiong + Recally
F1
(18)
Q
where Precision, and Recall, can be calculated follows:
Ei=1 8(ii, Cg) * 6(vi Cg).
Precisiong
(19)
8(Di, Cty)
Recall, Eisl 8(fi, Cg) 5(ys,, Cig)
N
(20)
8(yi, Co)
13

---

Table 5: The descriptions of 34 simulated datasets.
Dataset
#Instances
#Attrihures
#Classes
Missing
Attribute type
anneal
898
38
6
yes
hybrid
audiology
24
yes
nominal
autos
205
25
7
yes
hybrid
halance-scale
3
n0
numeric
biodeg
1055
41
2
numeric
breast-cancer
286
2
yes
nominal
breast-w
699
2
yes
numeric
1728
6
4
nO
nominal
credit-a
yes
hybrid
eredit-g
1000
20
2
no
hybrid
diabetes
768
no
numeric
heart-c
303
13
yes
hybrid
heart-h
294
13
5
yes
hybrid
acart-statiog
270
13
no
numeric
hepatitis
155
19
2
yes
by hybrid
horsc-colic
yes
hybrid
hypothyroid
3772
29
4
yes
hybrid
ionosphere
351
34
2
no
numeric
150
3
numeric
kr-vs-kp
3196
36
2
no
nominal
labor
57
16
yes
hybrid
letter
20000
16
26
no
numeric
lymph
no
hybrid
mushroom
8124
22
2
yes
nominal
segment
2310
19
по
numeric
sick
3772
29
yes
hybrid
sonar
208
60
2
no
numeric
spamhase
4601
no
numeric
tic-tactoe
958
9
2
n0
nominal
vehicle
n0
numeric
vote
435
16
2
yes
nominal
vowel
990
13
11
no
hybrid
waveform
5000
40
3
numeric
Zoo
101
17
7
no
hybrid
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly claim in the abstract and introduction section that the KFNN proposed in our paper mainly used improve the effectiveness and robustness of label integration in crowdsourcing. The contributions of our paper are also highlighted in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We mentioned the limitations of our KFNN in the Conclusion and future work section. KFNN has two empirical parameters and its distribution transformation process not refined enough.
14

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We give proofs to theorems that appear in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This paper provides open access the data and code, and provides document to guide readers in reproducing all experimental results in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: //apps.cc/public.. htcplli///giieeeee guides/ Cooeeuumeeeeeeee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper specifies all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We explain in detail the experimental metrics in the paper and provide the results of the significance tests.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe in detail the experimental setting on the computer resources in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ht htsps: http://////arrpp....////////////
ines?
Answer: [Yes]
Justification: The research conducted in this paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The new algorithm proposed in this paper helps improve the effectiveness
and robustness of label integration. There are no negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
18

---

Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We used the datasets and algorithmic implementations published by the CEKA platform, which is described in detail the paper. The license and terms of use explicitly are properly respected in this paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper choo . com/datasers has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper provides open access to the data and code, and provides document to guide readers in reproducing all experimental results in supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
19

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper did not select new crowdsourced datasets, The datasets used for experiments are publicly available and their information is described in detail in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve the research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
20


Reviews:
[{"review_id": "oXg2B0S9uw", "reviewer": "Reviewer_MQSy", "summary": "This paper introduces a new algorithm for label integration called KFNN. Existing methods related to KNN produce more noisy labels; however, they fix the neighborhood size, regardless of the fact that instances close to the center of classes should have more neighbors than instances close to the boundary of classes. To tackle this problem, KFNN estimates a Mahalanobis distance distribution between each instance and all classes to enhance the multiple noisy label distribution and utilizes a Kalman filter to mitigate the impact of noise. Finally, KFNN can automatically determine the optimal neighborhood size through max-margin learning.", "questions": "D1. The paper focuses on the KNN-related methods for label integration. However, the introduction didn’t justify the motivation of this concentration with convincing proofs. For instance, the motivation is basically explained with the sentence, “to alleviate this problem, recent works have begun to focus on leveraging neighbor instances [1, 11, 12] …”. However, there are also alternatives for label integration, so why considers KNN-related instead of the other types of solutions? Besides, there are much more studies (eg [R1]) that also target on this problem, which should be carefully discussed their pros and cons. Otherwise, the motivation looks weak.\n\nD2. From the perspective of crowdsourcing, the studied problem is closely related to “truth inference”. However, in the references, there are only two papers on this topic: [25] (published in 2016) and [26] (published in 2023). More studies, which can be easily found in Google Scholar or DBLP, should be reviewed and compared (if possible).\n\nD3. It is a little unclear how the principle of employing the same neighborhood size can impact performance. Please give more explanations.\n\nD4. In addressing the question of fusing information from the attribute space and the multiple noisy label space, this paper tends to take an average between the multiple noisy label distribution and the potential label distribution. However, it might be worth exploring the possibility of introducing a tunable parameter to achieve a more optimal balance between these two distributions, rather than relying solely on an equal (50%) average. \n\nD5. The application scope of the proposed method in crowdsourcing is limited. In my opinion, the proposed KFNN can be only used in simple and micro tasks in crowdsourcing, there are many other kinds of tasks in a real-world crowdsourcing platform, such as ranking [R2], which is not considered in the problem setting. Yet, the title, “KFNN: K-Free Nearest Neighbor For Crowdsourcing”, is a little over-claimed. At least, the paper should explicitly define the application scope. More types of crowdsourcing task can be found in existing surveys [R3, R4] on crowdsourcing.\n\nD6. The performance improvement of the proposed method is unsatisfactory.    \n(1) Although the average Macro-F1 score of KFNN is better than the compared baselines, it can be notably worse than some of the baselines in certain datasets (eg MNLDP on the anneal dataset). This pattern weakens the motivation, since it’s unclear whether the limitation of existing solutions has been well addressed or not.   \n(2) In Table 2, the integration accuracy of KFNN is lower than that of MNLDP. Besides, it can be also notably worse than some of the baselines in terms of the integration accuracy (eg LAGNN and LAWMV on the breast-cancer dataset).   \n(3) Based on the current experimental results, the effectiveness of the proposed solution KFNN is questionable.\n\nD7. Although several datasets are conducted in the experimental study, existing work on truth inference in crowdsourcing (eg [R2, R5]) usually deploys their solution in a real-world platform, such as AMT, to verify the performance. Therefore, the setup of the experimental study can be simplifier and less practical than the real-world scenario.\n\nReferences:  \n[R1] Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learning. AAAI 2024.   \n[R2] Xi Chen et al. Pairwise ranking aggregation in a crowdsourced setting. WSDM 2013.   \n[R3] Guoliang Li et al. Crowdsourced Data Management: A Survey. IEEE TKDE 2016.   \n[R4] Hector Garcia-Molina et al. Challenges in Data Crowdsourcing. IEEE TKDE 2016.   \n[R5] Yudian Zheng et al. Truth Inference in Crowdsourcing: Is the Problem Solved? VLDB 2017.", "limitations": "Please refer to the weaknesses and questions.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "S1. The paper studies an important problem.   \nS2. A new solution is proposed to tackle the problem.   \nS3. Experiments are conducted on several datasets.", "weaknesses": "W1. The motivations need more enhancements.   \nW2. Some technical details require more explanations.   \nW3. The application scope of the proposed method in crowdsourcing is limited.    \nW4. The performance improvement of the proposed method is unsatisfactory.    \nW5. Experiments are conducted in a simulation environment, which can be much simpler than a real-world crowdsourcing platform.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer MQSy：**\n\nThanks a lot for your comments. Please find our detailed responses to your seven questions as follows.\n\n**Q1:** First, our research focuses on label integration in crowdsourcing, which differs from other research domains such as noisy label learning (NLL). Crowdsourcing typically employs workers to assign multiple noisy labels to each instance (one instance corresponds to multiple noisy labels), and the label integration aims to infer the unknown true label of this instance from these multiple noisy labels. In contrast, NLL aims to train robust classifiers from datasets with a single noisy label per instance (one instance corresponds to one noisy label). Therefore, existing works outside of label integration, such as [R1], are not within the scope of this paper. Second, due to cost constraints, each instance in crowdsourcing typically obtains only few noisy labels, which restricts the performance of label integration. State-of-the-art KNN-related label integration algorithms address this problem directly and efficiently, forming the basis of our research. We have described this motivation in the introduction (lines 32-42) and will refine it further in the final version of this paper.\n\n**Q2:** In crowdsourcing scenarios, the technical term \"label integration\" and \"ground truth inference\" are all common and widely used. Reference [1] describes that ''Ground truth inference is defined as a process of estimating the true label of each example from its multiple noisy label set. If we only focus on the label itself, it is also called label integration.\". In the context of our paper, these terms are synonymous, all referring to the process of inferring the true label of each instance from multiple noisy labels. In our paper, we have chosen \"label integration\" to represent this process. Meanwhile, in our related work, we have comprehensively surveyed existing works, introducing a total of 18 representative works in label integration, with the latest published in 2024.\n\n[1] Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review, 2016, 46(4):  543-576.\n\n**Q3:** The goal of finding nearest neighbors for an instance is to find potential instances of the same class around this instance. However, the number of instances of the same class around each instance is naturally different. Instances near the class center are surrounded by more instances of the same class, so they need larger neighborhood sizes to collect sufficient labels from similar instances. In contrast, instances near the class boundary need smaller neighborhood sizes to avoid including too many instances from other classes. These reasons are mentioned in lines 111-115 of our paper, and we will provide a more detailed explanation in the final version.\n\n**Q4:** Indeed, introducing a tunable parameter can help achieve a more optimal balance between the multiple noisy label distribution and the potential label distribution. Originally, we devised a weighted version of Eq. (5) as follows:\n$$\n  p_{iq} = \\frac{\\lambda * p(c_q|x_i,D_q) + (1-\\lambda) * p(c_q|L_i)}{\\sum_{q=1}^{Q}[\\lambda * p(c_q|x_i,D_q) + (1-\\lambda)* p(c_q|L_i)]},\n$$\nwhere $\\lambda$ is a tunable parameter to balance these two distributions. However, we found experimentally that KFNN can still achieve good results with $\\lambda$ set to 0.5 (equal average). To keep KFNN simple and with fewer parameters, we currently used an equal average to eliminate $\\lambda$.\n\n**Q5:** Indeed, crowdsourcing is a broad research domain that includes various tasks such as label integration, noise correction, ranking, and so on. Our paper specifically focuses on label integration and presents a novel algorithm for label integration. The current paper title was proposed by referring to some classical works in label integration such as [2], hoping to inspire broader tasks in crowdsourcing beyond just label integration. In the final version of the paper, we will explicitly define our application scope by renaming our title to \"KFNN: K-Free Nearest Neighbor for Label Integration in Crowdsourcing\".\n\n[2] Community-based bayesian aggregation models for crowdsourcing. WWW `14, 2014: 155-164.\n\n**Q6:** Our proposed KFNN is not restricted to specific crowdsourcing scenarios, so we conducted experiments on the whole 34 datasets published by the CEKA platform. However, no label integration algorithm can achieve the best performance on all datasets. Therefore, we performed the Wilcoxon signed-rank test to further compare each pair of algorithms. The results in Table 3 show that KFNN significantly outperforms existing state-of-the-art label integration algorithms in terms of the Macro-F1 score. Additionally, as described in lines 254-258, we used the Macro-F1 score instead of integration accuracy as the main experimental metric. This is because the Macro-F1 score better reflects the performance of algorithms across different classes, while integration accuracy may not accurately reflect the performance on class-imbalanced datasets. Nevertheless, the results in Table 4 still show that KFNN achieves better or comparable integration accuracy compared with existing state-of-the-art algorithms. These results and analyses demonstrate the effectiveness and robustness of KFNN. \n\n**Q7:** Due to the double-blind policy, we did not deploy our solution in a real-world platform, such as AMT, and only submitted our code as a supplemental material. To verify the performance of KFNN in real-world crowdsourcing scenarios, as done in [R2, R5], we compared KFNN with existing state-of-the-art label integration algorithms using datasets Income and Leaves collected from the real-world crowdsourcing platform AMT. The results shown in Figure 1 are sufficient to demonstrate the effectiveness of the KFNN in real-world crowdsourcing scenarios. In the final version of the paper, we will provide a more detailed description of the collection of datasets Income and Leaves from the AMT platform.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LwE0tdFpTM", "reviewer": "Reviewer_qVhb", "summary": "This paper proposes a novel label integration approach KFNN by adaptively determining the optimal neighborhood size. KFNN utilizes a Mahalanobis distance distribution to model the relationship between each instance and all classes. The authors also provide adequate theoretical analysis to illustrate the effectiveness of the proposed method. Experiments demonstrate that the proposed method can achieve the state-of-the-art performance on simulation and real-world dataset. The paper is well-written and easy to follow. This idea is very intuitive and effective for crowdsourcing task. The paper proves the effectiveness of introducing Mahalanobis distance distribution for crowdsourcing from the perspective of methodology, theory and experiments.", "questions": "Please refer to weakness. My biggest concern is the experiments for the comparison between Mahalanobis distance and Euclidean distance.", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. The logic of the whole paper is clear.\n2. The paper’s idea is very intuitive and effective for crowdsourcing task. The authors introduce the Mahalanobis distance distribution to model the relationship between each instance and all classes. Experiments verify that the proposed method can achieve the best performance compared with SOTAs.\n3. The authors provide adequate evidences to verify the effectiveness of the proposed method from the perspective of methodology, theory and experiments on simulation and real-world datasets.", "weaknesses": "1. In section 2, the authors introduce two categories of label integration algorithms. And the proposed KFNN belongs to the algorithms which leverage neighbor instance. I suggest adding some discussion about the pros and cons of these two categories of approaches.\n2. In methodology part and theoretical analysis part, the authors discuss the superiority of Mahalanobis distance compared with Euclidean distance. Can the authors verify the difference between Mahalanobis distance and Euclidean distance on this task from an experimental perspective?\n3. In Table 3 and Table 4, why some results are missing? Appropriate explanation facilitates reading of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer qVhb：**\n\n**Q1:** In section 2, the authors introduce two categories of label integration algorithms. And the proposed KFNN belongs to the algorithms which leverage neighbor instance. I suggest adding some discussion about the pros and cons of these two categories of approaches.\n\n**Author Response:** Thanks for your valuable comments. The first category of algorithms does not leverage neighbor instances, considering only the information of the instance itself or the information of all instances globally in label integration. While simpler and more efficient, these algorithms are limited in effectiveness because each instance can only obtain few noisy labels. The second category of algorithms performs label integration by leveraging information from neighbor instances obtained by the KNN algorithm, which improves performance by using additional information from neighbor instances. However, these algorithms all assume a fixed neighborhood size for each instance, which is often unrealistic and thus limits their effectiveness. In the final version of the paper, we will add a paragraph to section 2 discussing the pros and cons of these two categories of algorithms. Thanks again for your valuable comments.\n\n**Q2:** In methodology part and theoretical analysis part, the authors discuss the superiority of Mahalanobis distance compared with Euclidean distance. Can the authors verify the difference between Mahalanobis distance and Euclidean distance on this task from an experimental perspective?\n\n**Author Response:** Thanks for your valuable comments. The main reason we chose the Mahalanobis distance in KFNN is that it can directly measure the distance from an instance to a dataset. Meanwhile, the Mahalanobis distance does not suffer from the correlation and magnitude of attributes. To respond to this comment, we defined the Euclidean distance of an instance to the centroid of a dataset as the distance of the instance to this dataset, replacing Equation 3 in the paper. Equation 6 was directly replaced by the Euclidean distance between the two instances. We denote this version of KFNN as KFNN-Euc. Subsequently, we compared KFNN and KFNN-Euc on the Income dataset. The Macro-F1 score and integration accuracy of KFNN-Euc on the Income dataset are 50.35% and 52.33%, respectively, significantly lower than those of KFNN (78.69% and 77.17%) shown in Figure 1. These experimental results validate the superiority of the Mahalanobis distance in KFNN. In the final version of the paper, we will include KFNN-Euc in the ablation experiment to demonstrate the superiority of the Mahalanobis distance in KFNN. Thanks again for your valuable comments.\n\n**Q3:** In Table 3 and Table 4, why some results are missing? Appropriate explanation facilitates reading of the paper.\n\n**Author Response:** Thanks for your valuable comments. Tables 3 and 4 show the results of the Wilcoxon signed-rank test. In Tables 3 and 4, the symbol • indicates that the algorithm in the row significantly outperforms the algorithm in the corresponding column, and the symbol ◦ indicates the exact opposite of that indicated by the symbol •. Missing items indicate no significant difference between the algorithm in the row and the algorithm in the column. The significance levels of the lower and upper diagonals in Tables 3 and 4 are 0.05 and 0.1, respectively. For example, the Wilcoxon test result between MV and LAWMV on the upper diagonal of Table 3 is missing, indicating no significant difference between MV and LAWMV in terms of the Macro-F1 score when the significance level is 0.1. In the final version of the paper, we will include a detailed explanation of the missing items. Thanks again for your valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RESF88VyTp", "reviewer": "Reviewer_B2kv", "summary": "The paper presents a novel label integration algorithm, KFNN (K-Free Nearest Neighbor), designed to enhance the performance of crowdsourcing platforms by intelligently determining the optimal neighborhood size for each instance based on its attributes and noisy labels. The authors propose a two-component solution involving label distribution enhancement and K-free optimization, which leverages the Mahalanobis distance and a Kalman filter to mitigate noise from neighbor instances. The paper's claims are well-aligned with the theoretical and experimental results, demonstrating the effectiveness and robustness of KFNN against existing state-of-the-art algorithms in various crowdsourcing scenarios.", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tNovel contribution to an important problem\nThe innovative approach of highlighting the limitations caused by fixed neighborhood sizes in existing label integration algorithms, and using attributes and noisy labels to determine the neighborhood size for each instance automatically, is a significant contribution to crowdsourcing. \n\n2.\tComplete and rigorous theoretical proof\nThe theoretical underpinnings are sound, with clear assumptions and proofs provided for the proposed methods. The use of the Mahalanobis distance and the Kalman filter is well-justified.\n\n3.\tGood writing quality and clarity\nThis paper is well-written and enjoyable to read. The challenges are clearly stated and the contributions are easy to capture. \n\n4.\tReproducibility\nThe paper's open data and code policy is highly appreciated, promoting research transparency. Enhancing reproducibility with clear versioning and setup instructions would be a valuable addition, showcasing a strong commitment to open scientific practices.", "weaknesses": "1.\tSimulation experiment results\nThe symbol • indicates that the algorithm in the row significantly outperforms the algorithm in the corresponding column. How is \"significantly outperforms\" defined for Macro-F1 score and integration accuracy?\n\n2.\tAblation experiment results\nSince this study focuses on automatically adjusting neighborhood sizes, how does the performance of this method compare with baselines that use fixed neighborhood sizes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer B2kv：**\n\n**Q1:** Simulation experiment results The symbol • indicates that the algorithm in the row significantly outperforms the algorithm in the corresponding column. How is \"significantly outperforms\" defined for Macro-F1 score and integration accuracy?\n\n**Author Response:** Thanks for your valuable comments. The Wilcoxon signed-ranks test evaluates the performance differences between two algorithms across multiple datasets by ranking the absolute values of their differences. Specifically, we first calculate the differences between the performance scores (Macro-F1 score or integration accuracy) of the two algorithms on the 34 datasets. Then, these differences are ranked according to their absolute values, with average ranks assigned in case of ties. Subsequently, Let $R^+$ and $R^-$ be the sum of ranks for positive and negative differences, respectively. They can be calculated as follows:\n$$\n  R^+ = \\sum_{d_i>0}rank(d_i) + \\frac{1}{2}\\sum_{d_i=0}rank(d_i),\n$$\n$$\n  R^- = \\sum_{d_i<0}rank(d_i) + \\frac{1}{2}\\sum_{d_i=0}rank(d_i),\n$$\nwhere $d_i$ denotes the difference between the performance scores of the two algorithms on $i$-th out of 34 datasets, $rank(d_i)$ denotes the rank of $d_i$. Next, let $T$ be the smaller of $R^+$ and $R^-$. From the table of exact critical values of the Wilcoxon test, it can be found that the exact critical values of $T$ at significance levels $\\alpha$ = 0.05 and $\\alpha$ = 0.1 are 182 and 200 when the number of datasets is 34, respectively. This means that the two algorithms are significantly different with $\\alpha$ = 0.05 when $T$ is less than or equal to 182. At this time, the second algorithm significantly outperforms the first if $T$ is equal to $R^-$. The first algorithm significantly outperforms the second if $T$ is equal to $R^+$. Thanks again for your valuable comments.\n\n**Q2:** Ablation experiment results Since this study focuses on automatically adjusting neighborhood sizes, how does the performance of this method compare with baselines that use fixed neighborhood sizes?\n\n**Author Response:** Thanks for your valuable comments. Among the existing label integration algorithms, both LAWMV and MNLDP are baselines using a fixed neighborhood size. Our simulated and real-world experimental results demonstrate that KFNN significantly outperforms both MNLDP and LAWMV. Additionally, in our ablation experiment, KFNN-KF is the version of KFNN employing a fixed neighborhood size. It can be found from Figure 3(a) that KFNN performs better than KFNN-KF. These findings highlight the superior performance of KFNN compared to baselines using a fixed neighborhood size. Thanks again for your valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gI7xweG3qs", "reviewer": "Reviewer_4Kyj", "summary": "This paper proposes a novel algorithm, KFNN (K-free Nearest Neighbor), which is specifically designed to enhance label integration for crowdsourcing. KFNN integrates two key components named label distribution enhancement and K-free optimization, which significantly contribute to improving the effectiveness and robustness of the label integration process. The idea of automatically determining the optimal neighborhood size for each instance is particularly innovative and well-executed. The experimental results further validate the effectiveness and robustness of the proposed algorithm.", "questions": "Please refer to the Weaknesses.", "limitations": "The authors openly discuss the limitations of their work, particularly the empirical parameters in the Kalman filter and the roughness of the distribution transformation process. Please refer to the Weaknesses for other limitations I have found.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe KFNN proposed in this paper is interesting and innovative. The authors reveal the limitations of fixed neighborhood sizes in existing label integration algorithms and propose an algorithm that automatically determines the optimal neighborhood size based on instance attributes and noisy labels. This algorithm significantly improves the robustness of label integration.\n2.\tThe paper provides a solid theoretical foundation for the proposed KFNN algorithm, followed by comprehensive experimental validation. The theoretical analysis is robust and convincingly demonstrates the expected performance improvements. The experiments are well-designed and cover a wide range of datasets, both simulated and real-world, to ensure the generalizability of the results. The experimental results, including comparisons with baseline algorithms, further validate the effectiveness and robustness of the proposed algorithm.\n3.\tThe paper is well-written and clearly presents the proposed methodology and findings. The structure of the paper is logical, making it easy to follow the complex concepts introduced. The use of figures and tables to illustrate key points is effective and aids in comprehension.", "weaknesses": "1. While the paper provides strong theoretical and experimental results, there is limited discussion on the computational efficiency and scalability of the proposed KFNN algorithm. I suggest moving the algorithmic flow and time complexity analysis from Appendix A to the main text.\n2. There are some repetitive sentences and structures in this paper that should be further condensed. For example, Sections 5.1 and 5.2 should be merged and the repetitive statements in them should be deleted.\n3. The experiments are already comprehensive, but analysis and discussion of the optimal neighborhood size determined by KFNN could still be added, which would help to understand how the neighborhood size should be set. Moreover, according to the results presented in Tables 1-4, KFNN is generally highly effective. However, on few datasets, KFNN does not perform as well as MV. These anomalies are valuable for identifying deficiencies in KFNN and should be further investigated and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer 4Kyj：**\n\n**Q1:** While the paper provides strong theoretical and experimental results, there is limited discussion on the computational efficiency and scalability of the proposed KFNN algorithm. I suggest moving the algorithmic flow and time complexity analysis from Appendix A to the main text. \n\n**Author Response:** Thanks for your valuable comments. Although our KFNN needs to determine an optimal neighborhood size for each instance, it is not a wrapper algorithm. Label integration does not divide the crowdsourced dataset into training, validation, and test sets. As a result, KFNN determines $K_i$ immediately when inferring $\\hat{y}_i$ without a validation phase. Therefore, the computational efficiency and scalability of KFNN are comparable to existing KNN-related label integration algorithms. In the final version of the paper, we will move the algorithmic flow and time complexity analysis from Appendix A to the main text. Thanks again for your valuable comments.\n\n**Q2:** There are some repetitive sentences and structures in this paper that should be further condensed. For example, Sections 5.1 and 5.2 should be merged and the repetitive statements in them should be deleted.\n\n**Author Response:** Thanks for your valuable comments. Indeed, there are some repetitive sentences and structures in our paper. In the final version of the paper, we will merge Sections 5.1 and 5.2 and delete these repetitive sentences. Thanks again.\n\n**Q3:** The experiments are already comprehensive, but analysis and discussion of the optimal neighborhood size determined by KFNN could still be added, which would help to understand how the neighborhood size should be set. Moreover, according to the results presented in Tables 1-4, KFNN is generally highly effective. However, on few datasets, KFNN does not perform as well as MV. These anomalies are valuable for identifying deficiencies in KFNN and should be further investigated and discussed.\n\n**Author Response:** Thanks for your valuable comments. In our KFNN, the optimal neighborhood size for each instance depends on both its attributes and multiple noisy labels. Based on its attributes, we can calculate the distance between this instance to the corresponding subset of each class. If the distance of this instance to one class is much smaller than the distance to other classes, it tends to be close to the center of this class. At this time, its optimal neighborhood size tends to be larger. In addition, if the multiple noisy labels of the instance are highly consistent, it means that this instance is easily distinguishable. At this time, its optimal neighborhood size tends to be smaller. Indeed, KFNN does not perform as well as MV on a small number of datasets, such as autos, breast-cancer, and diabetes. Considering that other KNN-related algorithms (LAWMV and MNLDP) also typically perform poorly on these datasets, we believe that the reason for the poor performance of KFNN is that these datasets are not well suited for distance measures. In the final version of the paper, we will include these discussions on optimal neighborhood size and anomalous experimental results. Thanks again for your valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel label integration approach KFNN by adaptively determining the optimal neighborhood size. KFNN utilizes a Mahalanobis distance distribution to model the relationship between each instance and all classes.  The reviewers have reached a consensus to accept this paper.  Please follow the reviewers' suggestions to polish the main paper.
Decision: Accept (poster)