Title: KFNN: K-Free Nearest Neighbor For Crowdsourcing
Venue: NeurIPS 2024
Accepted: yes
Keywords: Crowdsourcing learning

Label integration

K-free nearest neighbor
Abstract: To reduce annotation costs, it is common in crowdsourcing to collect only a few noisy labels from different crowd workers for each instance. However, the limited noisy labels restrict the performance of label integration algorithms in inferring the unknown true label for the instance. Recent works have shown that leveraging neighbor instances can help alleviate this problem. Yet, these works all assume that each instance has the same neighborhood size, which defies common sense. To address this gap, we propose a novel label integration algorithm called K-free nearest neighbor (KFNN). In KFNN, the neighborhood size of each instance is automatically determined based on its attributes and noisy labels. Specifically, KFNN initially estimates a Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution is then utilized to enhance the multiple noisy label distribution of each instance. Subsequently, a Kalman filter is designed to mitigate the impact of noise incurred by neighbor instances. Finally, KFNN determines the optimal neighborhood size by the max-margin learning. Extensive experimental results demonstrate that KFNN significantly outperforms all the other state-of-the-art algorithms and exhibits greater robustness in various crowdsourcing scenarios.

KFNN: K-Free Nearest Neighbor For Crowdsourcing

Abstract

To reduce annotation costs, it is common in crowdsourcing to collect only a few noisy labels from different crowd workers for each instance. However, the limited noisy labels restrict the performance of label integration algorithms in inferring the unknown true label for the instance. Recent works have shown that leveraging neighbor instances can help alleviate this problem. Yet, these works all assume that each instance has the same neighborhood size, which defies common sense. To address this gap, we propose a novel label integration algorithm called K-free nearest neighbor (KFNN). In KFNN, the neighborhood size of each instance is automatically determined based on its attributes and noisy labels. Specifically, KFNN initially estimates a Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution is then utilized to enhance the multiple noisy label distribution of each instance. Subsequently, a Kalman filter is designed to mitigate the impact of noise incurred by neighbor instances. Finally, KFNN determines the optimal neighborhood size by the max-margin learning. Extensive experimental results demonstrate that KFNN significantly outperforms all the other state-of-the-art algorithms and exhibits greater robustness in various crowdsourcing scenarios. Our codes and datasets are available at https://github.com/jiangliangxiao/KFNN.

1 Introduction

Crowdsourcing provides a more cost-effective way to obtain annotated instances than traditional expert annotation [1]. Through crowdsourcing platforms such as Figure Eight and Clickworker, instances can be annotated by crowd workers at a low cost [2, 3]. While more affordable, these workers possess less expertise than domain experts and are more prone to assigning noisy labels to instances [4]. To address this issue, the concept of repeated annotation is introduced and becomes popular in crowdsourcing [5]. With repeated annotation , each instance is annotated by several workers, thereby obtaining multiple noisy labels. To train supervised models using multiple noisy labels, two main categories of methods have been developed: one-stage methods and two-stage methods. One-stage methods [6, 7, 8] train models directly using multiple noisy labels. Two-stage methods [1, 9] first infer the unknown true label for each instance from its multiple noisy labels via

label integration (also known as answer aggregation or ground truth inference) [10] and then train models on integrated labels. One-stage methods, although end-to-end, can only be used to train specifically designed models. As a result, label integration, which is required for the more common two-stage methods, has received a great deal of attention from researchers.

It has been theoretically demonstrated that, when worker annotation is more accurate than random annotation, the more noisy labels an instance receives, the easier it becomes to infer its unknown true label [11]. However, to reduce annotation costs, only a few noisy labels can be collected for each instance in crowdsourcing. The limited noisy labels restrict the performance of label integration algorithms in inferring the unknown true label for the instance. Furthermore, some common strategies in crowdsourcing, such as worker modelling, worker elimination and task assignment [12], fail to mitigate the effects of limited labels in label integration. To alleviate this problem, recent works have begun to focus on leveraging neighbor instances [13, 14, 1]. These works successfully improve the performance of label integration by leveraging the information from neighbor instances obtained by the K-nearest neighbor (KNN) algorithm. However, due to the use of KNN, these algorithms all assume that each instance has the same neighborhood size. This assumption is difficult to hold because it defies common sense, e.g. instances close to the center of classes should have more neighbors than instances close to the boundary of classes.

To address this gap, we propose a novel label integration algorithm called K-free nearest neighbor (KFNN). In KFNN, the optimal neighborhood size of each instance is automatically determined based on its attributes and noisy labels. Notably, KFNN is different from some supervised works [15, 16] that determine the optimal K-value for KNN. Unlike in supervised learning, the true label of each instance in crowdsourcing is unknown and only its multiple noisy labels can be used, which makes it difficult to model the relationship between the instance and all classes. To do this, KFNN initially estimates a Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution is then utilized to enhance the label distribution for each instance. Subsequently, a Kalman filter is designed to mitigate the impact of noise incurred by neighbor instances. Finally, KFNN determines the optimal neighborhood size by the max-margin learning. In general, the contributions of this paper can be summarized as follows:

We reveal the limitations caused by fixing the neighborhood size in existing label integration algorithms and propose a novel algorithm called KFNN. In KFNN, the neighborhood size of each instance is automatically determined based on its attributes and noisy labels.

We estimate a Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution enhances the multiple noisy label distribution of each instance.

We design a Kalman filter to mitigate the impact of noise incurred by neighbor instances and then determine the optimal neighborhood size by the max-margin learning, which provides strong theoretical support for our algorithm.

Extensive experimental results demonstrate that KFNN significantly outperforms all the other state-of-the-art label integration algorithms and exhibits greater robustness than existing algorithms in various crowdsourcing scenarios.

2 Related work

Depending on whether neighbor instances are leveraged or not, existing label integration algorithms can be divided into two categories. The first category of algorithms does not leverage neighbor instances, which considers only the information of the instance itself or the information of all instances globally in label integration. For example, [17] models the ability of each worker with a confusion matrix. In this matrix, each element reflects the probability that this worker annotates an instance with the class corresponding to the row as the class corresponding to the column. [18, 19] are Bayesian versions of [17], which can be used for binary tasks and multi-class tasks, respectively. Further, [20, 21] improve [19] by introducing the correlation between workers. [11, 22, 23] are classical algorithms based on majority voting and they tend to use the label with the highest number of votes as the integrated label. [24, 25, 26] synchronously model the ability of workers and the difficulty of tasks from different perspectives. [27, 28] use clustering algorithms to divide instances into different clusters from different views, and then map these clusters to different classes. Recently, [29] augments the multiple noisy label distributions of instances as new attributes to the original

attribute space and then learns a classifier on the augmented attribute space to predict the integrated labels of instances. [9] constructs graphs for workers and uses a graph neural network to aggregate multi-order information in label integration.

The second category of algorithms performs label integration by leveraging the information from neighbor instances obtained by the KNN algorithm. For example, [13] proposes to use the labels assigned to the neighbor instances of an instance to augment this instance's multiple noisy labels and use the augmented multiple noisy labels to infer the integrated label of this instance. [14] considers both nearest and farthest neighbors in weighted voting to address class-imbalanced tasks. Further, inspired by label distribution learning [30, 31], given an instance, [1] iteratively absorbs the label distributions of its neighbor instances into its label distribution through label distribution propagation.

While simpler and more efficient, the first category of algorithms are limited in effectiveness because each instance can only obtain few noisy labels. Both experimental results and theoretical analysis demonstrate the effectiveness of the second category of algorithms in leveraging the information from neighbor instances. However, these algorithms all assume a fixed neighborhood size for each instance, which is often unrealistic and thus limits their performance. To further ensure that each instance has a free neighborhood size, this paper proposes a novel label integration algorithm called KFNN. KFNN automatically determines the optimal neighborhood size for each instance based on its attributes and noisy labels, which improves the performance and robustness of label integration.

3 Algorithm

In this section, we respond to how to automatically determine the optimal neighborhood size for each instance. First, we present some basic notations in crowdsourcing and then define the problem settings. Subsequently, we introduce our KFNN for label integration.

3.1 Preliminary

Let D = { ( x i , L i ) } N i =1 denote a crowdsourced dataset, where N is the number of instances, and x i denotes the i -th instance in D . x i can be represented as { x im } M m =1 . Here, M is the dimension of attributes, and x im denotes the attribute value of x i on the m -th attribute A m . L i denotes multiple noisy labels of x i , which can be expressed as { l ir } R r =1 . R is the number of workers and l ir denotes the label of x i annotated by the r -th worker u r . l ir takes a value from a fixed set {-1 , c 1 , . . . , c q , . . . , c Q } , where Q is the number of classes, c q denotes the q -th class and -1 indicates that u r has not annotated x i . Label integration aims to infer an integrated label ˆ y i for x i and minimize the error between ˆ y i and the unknown true label y i .

Recent works [1, 13] have shown that leveraging neighbor instances N i = { x k i } K k =1 of x i can mitigate the restriction of limited noisy labels on the performance of label integration. Here, x k i denotes the k -th nearest neighbor of x i and K is the neighborhood size. However, in these works, the value of K is fixed for each instance within the same dataset, which does not make sense. On the one hand, instances closer to the center of a class benefit from a larger K , as it enables them to collect more labels from similar instances. Conversely, for instances close to the boundary of classes, a larger K plays a negative role in label integration. On the other hand, using a fixed K can bias algorithms towards the majority class in class-imbalanced datasets, as instances from the majority class are more likely to dominate the neighborhood of instances from minority classes. Therefore, we define the Problem 1 to be addressed in this paper as follows:

Problem 1. Given a crowdsourced dataset D , how to automatically determine the optimal neighborhood size K ∗ i for each instance x i with { x im } M m =1 and { l ir } R r =1 but without y i .

Problem 1 cannot be treated simply as learning an optimal neighborhood size for the KNN algorithm in supervised learning [15, 16]. This is because the true labels of instances in crowdsourcing are unknown. As a result, K can not be evaluated accurately by supervised metrics such as classification accuracy. Moreover, label integration does not divide the crowdsourced dataset into training, validation and test sets, which means that KFNN has to determine K ∗ i immediately when inferring ˆ y i , rather than with a validation phase.

3.2 K-free nearest neighbor algorithm

In this subsection, we propose our KFNN to address Problem 1. We argue that K ∗ i should be related to the information from both the attribute space and the multiple noisy label space. Based on this, KFNN divides Problem 1 into two parts: 1) How to fuse the information from the attribute space and the multiple noisy label space? 2) How to determine an optimal K ∗ i for x i ? Correspondingly, KFNN consists of two components, namely label distribution enhancement and K-free optimization, which are used to address the two parts of Problem 1.

3.2.1 Label distribution enhancement

For each instance x i , { x im } M m =1 reflects all the information of it in the attribute space and { l ir } R r =1 reflects all the information of it in the multiple noisy label space. Inspired by label enhancement (LE) [32, 33], we design a label distribution enhancement (LDE) component for KFNN. LDE recovers a potential label distribution using { x im } M m =1 , and then enhances the multiple noisy label distribution calculated from { l ir } R r =1 by this potential label distribution. Specifically, KFNN first uses majority voting to initialize the integrated label ˆ y i for x i as follows:

where p ( c q | L i ) can be calculated as follows:

Here, p ( c q | L i ) reflects the proportion of labels in L i that take the value c q . The function δ ( · ) outputs 1 if its two parameters are identical, and 0 otherwise. Subsequently, according to ˆ y i , the crowdsourced dataset D can be divided into Q subsets { D q } Q q =1 . The subset D q contains all instances with initial integrated labels of c q , i.e., D q = { x i | ˆ y i = c q } N i =1 . Then, KFNN calculates a Mahalanobis distance distribution { d ( x i , D q ) } Q q =1 as follows:

where µ q denotes the centroid of D q and C -1 q denotes the inverse matrix of the covariance matrix of D q . d ( x i , D q ) is the Mahalanobis distance from x i to D q calculated in the attribute space. A larger d ( x i , D q ) means that x i is less likely to belong to c q , conversely a smaller d ( x i , D q ) means that x i tends to belong to c q . Therefore, { d ( x i , D q ) } Q q =1 can be used to model the relationship between each instance and all classes. Based on this, { d ( x i , D q ) } Q q =1 can be transformed into a potential label distribution { p ( c q | x i , D q ) } Q q =1 as follows:

where max ( · ) and min ( · ) denote the maximum and minimum values of the set, respectively.

In addition to the potential label distribution, a multiple noisy label distribution { p ( c q | L i ) } Q q =1 can also be directly transformed from L i . Different from { p ( c q | x i , D q ) } Q q =1 , which learns the potential relationship between instances and classes from the attribute space, { p ( c q | L i ) } Q q =1 learns the label distribution reflected by noisy labels from the multiple noisy label space. Finally, KFNN fuses them into an enhanced label distribution P i = { p iq } Q q =1 by averaging as follows:

In this way, the enhanced label distribution P i can fuse the information from the attribute space and the multiple noisy label space. Therefore, the first part of Problem 1 has been addressed.

3.2.2 K-free optimization

After obtaining P i by label distribution enhancement, KFNN proceeds to determine the optimal neighborhood size K ∗ i for x i . First, KFNN calculates the distance between each pair of instances x 1 and x 2 by:

where d ( x 1 , x 2 | D q ) can be calculated as follows:

Compared to the Euclidean distance, Eq. (6) introduces the label information by calculating the distance between x 1 and x 2 on each subset D q . According to Eq. (6), we can calculate distances between x i and all instances in D . By sorting these distances we can obtain a neighbor sequence < x 1 i , . . . , x k i , . . . , x N i > for x i . Here, x k i is the k -th neighbor instance of x i satisfying d ( x i , x k i ) ≥ d ( x i , x k -1 i ) when k greater than 1. Then, we calculate the weight w ik for x k i as follows:

where l ikr denotes the label of x k i annotated by the r -th worker u r . w ik reflects the proportion of workers assigned the same label for x i and x k i . Subsequently, x i is allowed to absorb the enhanced label distributions of neighbor instances in the neighbor sequence one by one. Let P k i = { p k iq } Q q =1 denote the label distribution of x i after absorbing x k i , which can be updated as follows:

where p ikq denotes the probability value corresponding to c q in the enhanced label distribution of x k i . Since the first neighbor instance of x i is itself, P k i = P i when k is equal to 1.

According to P k i , KFNN calculates a class margin as follows:

where sec ( · ) denotes the second-largest value of the set. Since the true labels are unknown, Eqs. (5) (7) (8) are all designed based on multiple noisy labels, which lead to that ˜ M k contains a degree of noise incurred by neighbor instances. Therefore, KFNN designs a Kalman filter to mitigate the impact of noise in ˜ M k as follows:

where ˆ M k denotes the filtered margin, determined by both the estimated margin ˆ M -k and the calculated margin ˜ M k . The designed Kalman filter can be divided into an estimation phase and an update phase. In the estimation phase, the filter estimates ˆ M -k and the estimated error P -k based on the filtered margin ˆ M k -1 and error P k -1 of the previous time index. In the update phase, the filter first updates the Kalman gain K k of the k -th time index and then updates ˆ M k and error P k of the k -th time index according to K k . α and β are the process error and the measurement error in the Kalman filter. When k is equal to 0, ˆ M k takes the value of 0 and P k takes the value of 1.

To address the second part of Problem 1, KFNN determines the optimal neighborhood size K ∗ i for x i by the max-margin learning as follows:

Ultimately, according to K ∗ i , KFNN updates the integrated label ˆ y i for x i as follows:

The whole learning process of KFNN is shown in Algorithm 1. In Algorithm 1, lines 1-3 initialize the integrated label and multiple noisy label distribution for each instance and their time complexity is O ( NQR ) . Line 4 divides the crowdsourced dataset D into Q subsets and its time complexity is O ( NQ ) . Lines 5-9 perform label distribution enhancement and their time complexity is O ( NM 2 Q ) . Line 11 calculates the distances from x i to other instances and sorts these distances, its time complexity is O ( NM 2 Q + N log( N )) . Lines 12-16 calculate the margins { ˜ M k } N k =1 and their time complexity is O ( NR + NQ ) . Line 17 filters the margins { ˜ M k } N k =1 and its time complexity is O ( N ) . Line 18 determines the optimal neighborhood size and its time complexity is O ( N ) . Line 19 infers the integrated label for each instance and its time complexity is O ( Q ) . Therefore, the time complexity of lines 10-20 is O ( N 2 ( M 2 Q +log( N ) + R )) . If only the highest order terms are taken, the time complexity of KFNN is O ( N ( NM 2 Q + N log( N ) + NR + QR )) .

Algorithm 1 The learning process of KFNN

Require: D = { ( x , L ) } N - a crowdsourced dataset; α , β - the predefined parameters

i i i =1 Ensure: { ˆ y i } N - the integrated labels

4 Theoretical analysis

In this section, we provide some detailed theoretical analysis for KFNN. First, in Eq. (6), KFNN defines the distance d ( x 1 , x 2 ) between x 1 and x 2 based on the Mahalanobis distance d ( x 1 , x 2 | D q ) rather than the traditional Euclidean distance d E ( x 1 , x 2 ) . According to Eqs. (3) (7), the Mahalanobis distance works based on a basic assumption, which can be described as follows:

Assumption 1. Given the subset D q , its covariance matrix C q is a nonsingular matrix.

The Assumption 1 holds based on the condition that | C q | is non-zero, which is usually satisfied. Even if this condition is not satisfied, we can ensure that the Assumption 1 holds by adding a small value to each element of the principal diagonal on C q until | C q | is non-zero.

Theorem 1. If Assumption 1 holds, there will be an orthogonal matrix P satisfying that P -1 C q P = P T C q P = Λ , where Λ is a diagonal matrix with all M eigenvalues of C q as its elements of the principal diagonal.

Due to the limited pages, the proof of Theorem 1 is provided in Appendix A. Based on Theorem 1, we can obtain some interesting corollaries about Eqs. (3) (6) (7).

Corollary 1. Compared to d E ( x 1 , x 2 ) , d ( x 1 , x 2 ) in Eq. (6) does not suffer from the correlation and magnitude of attributes.

Proof. According to Theorem 1, d ( x 1 , x 2 | D q ) can be transformed as follows:

When Λ -1 is not considered, the derivation of Eq. (14) implies that d ( x 1 , x 2 | D q ) is the Euclidean distance of instances after orthogonal transformation using P T . After orthogonal transformation, the attributes are independent of each other, so d ( x 1 , x 2 ) does not suffer from the correlation of attributes. Λ -1 is equivalent to diag ( 1 λ 1 , 1 λ 2 , . . . , 1 λ M ) , where λ M is the M -th eigenvalue of C q and is equal to the variance on the direction of the corresponding eigenvectors. Briefly, Λ -1 ensures that the calculated result on each dimension is normalized by the corresponding variance when calculating the distance by Eq. (7). Therefore, d ( x 1 , x 2 ) does not also suffer from the magnitude of attributes.

Corollary 2. Compared to d E ( x 1 , x 2 ) , d ( x 1 , x 2 ) in Eq. (6) provides a smaller distance for x 1 and x 2 coming from the same class.

Proof. P T causes the original attribute space to be rotated according to the direction of the eigenvectors of C q , and Λ -1 causes the rotated attribute space to be scaled according to the eigenvalues C q . Referring to the principle of principal component analysis [34], the eigenvectors of C q reflect the principal component directions of D q . This means that d ( x 1 , x 2 ) will provide a smaller distance for instances coming from the same class compared to d E ( x 1 , x 2 ) .

Assumption 2. When we estimate ˆ M -k based on ˆ M k -1 , the estimated error satisfies N(0, P -k ). When we measure ˜ M k by Eq. (10) , the measurement error satisfies N(0, β ).

The Kalman filter we designed as Eq. (11) works based on Assumption 2, which usually holds because the noise in practice usually satisfies a normal distribution. Since ˆ M k -1 changes in each time index, the variance of the estimated error P -k changes with the time index. Since Eq. (10) remains constant, the variance of the measurement error β is constant. According to Assumption 2, the following theorem can be proved:

Theorem 2. When the Kalman gain K k takes the value P -k P -k + β , the error between the filtered margin ˆ M k and the true margin M k is minimized.

Proof. When Assumption 2 holds, due to ˆ M k = ˆ M -k + K k ∗ ( ˜ M k -ˆ M -k ) , it can be proved that minimizing the error between ˆ M k and M k is equivalent to minimizing the variance of ˆ M k . Since ˆ M -k and ˜ M k are independent of each other, the following equation can be derived:

where V ar ( · ) denotes the variance of the variable. According to Assumption 2, V ar ( ˆ M -k ) equals to P -k and V ar ( ˜ M k ) equals to β . To minimize the error between the filtered margin ˆ M k and the true

margin M k , we can calculate the partial derivative ∂V ar ( ˆ M k ) ∂ K k as follows:

Ultimately, it can be proved that K k is equal to P -k P -k + β by setting ∂V ar ( ˆ M k ) ∂ K k to 0.

Theorem 3. The larger ˆ M k is, the better the corresponding neighborhood size k is.

Theorem 3 ensures the effectiveness of KFNN in determining the optimal neighborhood size by the max-margin learning, and its proof is provided in Appendix B due to the limited pages.

5 Experiments

5.1 Experimental setup

To evaluate the effectiveness of KFNN, we construct extensive experiments on the whole 34 simulated and two real-world crowdsourced datasets published on the Crowd Environment and its Knowledge Analysis (CEKA) [35] platform. For simulated datasets, we first use the unsupervised attribute filter ReplaceMissingValues in the Waikato Environment and Knowledge Analysis (WEKA) [36] platform to replace all missing values. Subsequently, with the CEKA platform, we hide true labels of simulated datasets and simulate five workers whose label qualities are randomly generated from a normal distribution with N(0.65, 0.05 2 ) to annotate these datasets. The real-world datasets, Income and Leaves , which were both collected from the online platform Amazon Mechanical Turk (AMT), can be used directly without any processing since they do not contain missing values.

We compare our KFNN with six state-of-the-art label integration algorithms. Among them, MV (majority voting) [11] is the simplest label integration algorithm and is used as a baseline for all algorithms. IWMV (iterative weighted majority voting) [22], AALI (attribute augmentation-based label integration) [29], and LAGNN (label aggregation with graph neural networks) [9] are three state-of-the-art label integration algorithms that do not leverage neighbor instances. LAWMV (label augmented and weighted majority voting) [13] and MNLDP (multiple noisy label distribution propagation) [1] are two state-of-the-art label integration algorithms that leverage neighbor instances. For MV, we use the existing implementation of the CEKA platform. For IWMV, AALI, LAGNN, LAWMV, and MNLDP, we use the implementations provided by their authors. All parameters of the comparison algorithms are set to the recommended values in the corresponding published papers. In addition, since true labels are unknown in our experiments, we use the lazy version of LAGNN. In our KFNN, α and β are set to 0.1 and 1 by default.

The performance of each algorithm is evaluated using the Macro-F1 score, which highlights the performance of algorithms on different classes and better reveals algorithmic limitations compared to traditional integration accuracy. Due to the limited pages, more detailed descriptions of the experimental datasets and metrics are provided in Appendix C. All experiments are independently repeated ten times on a Windows 10 machine with an AMD Athlon(tm) X4 860K Quad Core Processor @ 3.70 GHz and 16 GB of RAM, and we report the average results of ten experiments.

5.2 Results and discussions

Simulation experiment results. Table 1 shows the detailed Macro-F1 score (%) comparisons of each label integration algorithm on each simulated dataset, respectively. Based on these results, we perform the Wilcoxon signed-rank test [37] to further compare each pair of algorithms. Table 3 summarizes the Wilcoxon test results. In Table 3, the symbol · indicates that the algorithm in the row significantly outperforms the algorithm in the corresponding column, the symbol ◦ indicates the exact opposite of that indicated by the symbol · , and the missing item indicates no significant difference between the algorithm in the row and the algorithm in the column. The significance levels of the lower and upper diagonals are α = 0 . 05 and α = 0 . 1 , respectively. Based on these experimental results, we can summarize the following highlights: 1) The average Macro-F1 score of KFNN on all datasets is 79.64%, which is much higher than those of MV (72.46%), IWMV (72.71%), AALI (72.95%), LAGNN(73.71%), LAWMV (73.44%) and MNLDP (76.68%). KFNN achieves the highest Macro-F1

score, which indicates that KFNN is more effective and robust than these comparison algorithms in various crowdsourcing scenarios. 2) Among all comparison algorithms of KFNN, MNLDP performs better than IWMV, AALI and LAGNN, which demonstrates the advantages of leveraging neighbor instances. 3) Based on the Wilcoxon test results, KFNN significantly outperforms all comparison algorithms, which strongly validates the effectiveness and robustness of KFNN. Besides, we also observe the experimental results in terms of the integration accuracy, which are shown in Tables 2 and 4. According to Tables 2 and 4, we can see that KFNN can also achieve better or comparable integration accuracy compared with these state-of-the-art label integration algorithms. These results again validate the effectiveness and robustness of KFNN.

Real-world experiment results. Compared to simulated crowdsourced datasets, real-world crowdsourced datasets may include some special factors that influence label integration to work effectively, such as sparsity and bias. Therefore, we further observe the performance of KFNN and its comparison algorithms on two real-world datasets, Income and Leaves . Figure 1 shows the detailed Macro-F1 score (%) and integration accuracy (%) comparisons of each label integration algorithm on Income and Leaves , respectively. As can be seen from Figure 1, compared to these state-of-the-art label integration algorithms, our KFNN achieves the highest integration accuracies and Macro-F1 scores on both Income and Leaves . These results strongly support the effectiveness of our KFNN.

Parameter sensitivity analysis. There are two parameters α and β that can be adjusted in the Kalman filter designed by KFNN. To observe the effect of these two parameters on the performance of KFNN, we perform the parameter sensitivity analysis for KFNN on Income and Leaves . We change both α and β from 0.1 to 1 and then observe the Macro-F1 score of KFNN on two datasets.

75.22

75.61

75.27

76.42

77.13

75.46

78.69

73.00

74.00

75.00

76.00

77.00

78.00

79.00

80.00

MV

IWMV

AALI

LAGNN

LAWMV

MNLDP

KFNN

(a) F1 (

Income

)

72.33

71.83

73.17

73.17

74.00

71.17

77.17

65.00

67.00

69.00

71.00

73.00

75.00

77.00

79.00

MV

IWMV

AALI

LAGNN

LAWMV

MNLDP

KFNN

Income

)

55.79

55.96

60.20

54.83

57.52

56.09

65.40

52.50

54.50

56.50

58.50

60.50

62.50

64.50

66.50

MV

IWMV

AALI

LAGNN

LAWMV

MNLDP

KFNN

(c) F1 (

Leaves

)

64.32

64.58

67.19

64.06

66.15

65.10

72.14

60.00

62.00

64.00

66.00

68.00

70.00

72.00

74.00

MV

IWMV

AALI

LAGNN

LAWMV

MNLDP

KFNN

Leaves

)

Figure 2a and Figure 2b show the Macro-F1 score of KFNN on Income and Leaves when α and β vary. Based on these results, we can find that KFNN is more sensitive to β compared to α . As β tends to 1, KFNN tends to achieve optimal performance. Therefore, the default value of β in this paper is set to 1. α hardly affects the performance of KFNN, which is set to 0.1 by default.

































(a)

Income

































(b)

Leaves

78.69

77.95

76.13

75.22

72.50

73.50

74.50

75.50

76.50

77.50

78.50

79.50

KFNN

KFNN-KF

KFNN-LDE

KFNN-KF-LDE

(a) Macro-F1 score

20.00

30.00

40.00

50.00

60.00

70.00

k

Before filter

After filter

(b) Class margin

Ablation experiment. There are two components in KFNN, namely label distribution enhancement (LDE) and K-free optimization (KF). To validate their effectiveness, we observe the Macro-F1 score of KFNN after taking away each component on the Income dataset. For simplicity, we use "KFNNKF" to denote the variant of KFNN after taking away the component KF. Similarly, we create its another two variants "KFNN-LDE" and "KFNN-KF-LDE". Based on the results shown in Figure 3a, it can be seen that the performance becomes worse when any component is taken away. These results validate the effectiveness of LDE and KF. Figure 3b shows the change of the class margin before and after using our designed Kalman filter (observed on the first instance of Income ). As can be seen from Figure 3b, compared to the margin before filter ( ˜ M k ), the filtered margin ( ˆ M k ) changes smoother. These results validate the effectiveness of our designed Kalman filter, which successfully mitigates the impact of noise incurred by neighbor instances.

6 Conclusion and future work

To ensure that each instance in crowdsourced datasets has a free neighborhood size, we propose a novel algorithm called KFNN. KFNN consists of two key components, namely label distribution enhancement and K-free optimization. Label distribution enhancement fuses the information from the attribute space and the multiple noisy label space. K-free optimization automatically determines the optimal neighborhood size for each instance by the max-margin learning. Both theoretical analysis and experimental results validate the effectiveness and robustness of KFNN.

Nevertheless, there are still some limitations in KFNN that can be improved in the future. For example, the parameters α and β in the Kalman filter designed by KFNN can not automatically adapt to the dataset, which restricts the robustness of KFNN. In addition, in Eq. (4), transforming the distance distribution into the potential label distribution using max-min normalization is rough. Considering that the distance metric is not effective across all datasets (e.g., autos and breast-cancer in Table 1), this transformation may also lead to KFNN performing poorly. In the future, we will design more sophisticated parameters and transformations to improve KFNN.

Acknowledgment

The work was partially supported by National Natural Science Foundation of China (62276241), Foundation of Key Laboratory of Artificial Intelligence, Ministry of Education, P.R. China (AI2022004), and Science and Technology Project of Hubei Province-Unveiling System (2021BEC007).

References

L. Jiang, H. Zhang, F. Tao, and C. Li, 'Learning from crowds with multiple noisy label distribution propagation,' IEEE Trans. Neural Networks Learn. Syst. , vol. 33, no. 11, pp. 6558-6568, 2022.

P. Chen, Y. Yang, D. Yang, H. Sun, Z. Chen, and P. Lin, 'Black-box data poisoning attacks on crowdsourcing,' in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China . ijcai.org, 2023, pp. 2975-2983.

Z. Chen, H. Sun, H. He, and P. Chen, 'Learning from noisy crowd labels with logics,' in 39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, CA, USA, April 3-7, 2023 . IEEE, 2023, pp. 41-52.

J. Li, Y . Kawase, Y . Baba, and H. Kashima, 'Performance as a constraint: An improved wisdom of crowds using performance regularization,' in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020 , C. Bessiere, Ed. ijcai.org, 2020, pp. 1534-1541.

V. S. Sheng, J. Zhang, B. Gu, and X. Wu, 'Majority voting and pairing with multiple noisy labeling,' IEEE Trans. Knowl. Data Eng. , vol. 31, no. 7, pp. 1355-1368, 2019.

F. Rodrigues and F. C. Pereira, 'Deep learning from crowds,' in Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , S. A. McIlraith and K. Q. Weinberger, Eds. AAAI Press, 2018, pp. 1611-1618.

Z. Chen, H. Wang, H. Sun, P. Chen, T. Han, X. Liu, and J. Yang, 'Structured probabilistic end-to-end learning from crowds,' in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020 , C. Bessiere, Ed. ijcai.org, 2020, pp. 1512-1518.

J. Li, H. Sun, and J. Li, 'Beyond confusion matrix: learning from multiple annotators with awareness of instance features,' Mach. Learn. , vol. 112, no. 3, pp. 1053-1075, 2023.

Z. Ying, J. Zhang, Q. Li, M. Wu, and V. S. Sheng, 'A little truth injection but a big reward: Label aggregation with graph neural networks,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 46, no. 5, pp. 3169-3182, 2024.

J. Zhang, X. Wu, and V. S. Sheng, 'Learning from crowdsourced labeled data: a survey,' Artif. Intell. Rev. , vol. 46, no. 4, pp. 543-576, 2016.

V. S. Sheng, F. J. Provost, and P. G. Ipeirotis, 'Get another label? improving data quality and data mining using multiple, noisy labelers,' in Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008 , Y. Li, B. Liu, and S. Sarawagi, Eds. ACM, 2008, pp. 614-622.

G. Li, J. Wang, Y. Zheng, and M. J. Franklin, 'Crowdsourced data management: A survey,' IEEE Trans. Knowl. Data Eng. , vol. 28, no. 9, pp. 2296-2319, 2016.

Z. Chen, L. Jiang, and C. Li, 'Label augmented and weighted majority voting for crowdsourcing,' Inf. Sci. , vol. 606, pp. 397-409, 2022.

W. Zhang, L. Jiang, Z. Chen, and C. Li, 'Fnnwv: Farthest-nearest neighbor-based weighted voting for class-imbalanced crowdsourcing,' Sci. China Inf. Sci. , pp. 10.1007/s11 432-023-3854-7, 2023.

J. S. Olsson, 'An analysis of the coupling between training set and neighborhood sizes for the k nn classifier,' in SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, USA, August 6-11, 2006 , E. N. Efthimiadis, S. T. Dumais, D. Hawking, and K. Järvelin, Eds. ACM, 2006, pp. 685-686.

S. Zhang, X. Li, M. Zong, X. Zhu, and D. Cheng, 'Learning k for knn classification,' ACM Trans. Intell. Syst. Technol. , vol. 8, no. 3, pp. 43:1-43:19, 2017.

A. P. Dawid and A. M. Skene, 'Maximum likelihood estimation of observer error-rates using the em algorithm,' Journal of the Royal Statistical Society: Series C (Applied Statistics) , vol. 28, no. 1, pp. 20-28, 1979.

V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy, 'Learning from crowds,' J. Mach. Learn. Res. , vol. 11, pp. 1297-1322, 2010.

H. Kim and Z. Ghahramani, 'Bayesian classifier combination,' in Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012, La Palma, Canary Islands, Spain, April 21-23, 2012 , ser. JMLR Proceedings, N. D. Lawrence and M. A. Girolami, Eds., vol. 22. JMLR.org, 2012, pp. 619-627.

M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi, 'Community-based bayesian aggregation models for crowdsourcing,' in 23rd International World Wide Web Conference, WWW '14, Seoul, Republic of Korea, April 7-11, 2014 , C. Chung, A. Z. Broder, K. Shim, and T. Suel, Eds. ACM, 2014, pp. 155-164.

Y. Li, B. I. P. Rubinstein, and T. Cohn, 'Exploiting worker correlation for label aggregation in crowdsourcing,' in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 2019, pp. 3886-3895.

H. Li and B. Yu, 'Error rate bounds and iterative weighted majority voting for crowdsourcing,' CoRR , vol. abs/1411.4086, 2014.

T. Tian, J. Zhu, and Y. Qiaoben, 'Max-margin majority voting for learning from crowds,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 41, no. 10, pp. 2480-2494, 2019.

J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. R. Movellan, 'Whose vote should count more: Optimal integration of labels from labelers of unknown expertise,' in Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada , Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta, Eds. Curran Associates, Inc., 2009, pp. 2035-2043.

P. Welinder, S. Branson, S. J. Belongie, and P. Perona, 'The multidimensional wisdom of crowds,' in Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada , J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, Eds. Curran Associates, Inc., 2010, pp. 2424-2432.

A. Kurve, D. J. Miller, and G. Kesidis, 'Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention,' IEEE Trans. Knowl. Data Eng. , vol. 27, no. 3, pp. 794-809, 2015.

J. Zhang, V. S. Sheng, J. Wu, and X. Wu, 'Multi-class ground truth inference in crowdsourcing with clustering,' IEEE Trans. Knowl. Data Eng. , vol. 28, no. 4, pp. 1080-1085, 2016.

G. Wu, L. Zhou, J. Xia, L. Li, X. Bao, and X. Wu, 'Crowdsourcing truth inference based on label confidence clustering,' ACM Trans. Knowl. Discov. Data , vol. 17, no. 4, pp. 46:1-46:20, 2023.

Y. Zhang, L. Jiang, and C. Li, 'Attribute augmentation-based label integration for crowdsourcing,' Frontiers Comput. Sci. , vol. 17, no. 5, p. 175331, 2023.

C. Xu and X. Geng, 'Hierarchical classification based on label distribution learning,' in The ThirtyThird AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019 . AAAI Press, 2019, pp. 5533-5540.

Y. Lu and X. Jia, 'Predicting label distribution from multi-label ranking,' in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., 2022.

N. Xu, A. Tao, and X. Geng, 'Label enhancement for label distribution learning,' in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden , J. Lang, Ed. ijcai.org, 2018, pp. 2926-2932.

K. Wang, N. Xu, M. Ling, and X. Geng, 'Fast label enhancement for label distribution learning,' IEEE Trans. Knowl. Data Eng. , vol. 35, no. 2, pp. 1502-1514, 2023.

Y. Gao, T. Lin, Y. Zhang, S. Luo, and F. Nie, 'Robust principal component analysis based on discriminant information,' IEEE Trans. Knowl. Data Eng. , vol. 35, no. 2, pp. 1991-2003, 2023.

J. Zhang, V. S. Sheng, B. Nicholson, and X. Wu, 'CEKA: a tool for mining the wisdom of crowds,' J. Mach. Learn. Res. , vol. 16, pp. 2853-2858, 2015.

I. H. Witten, E. Frank, and M. A. Hall, Data mining: practical machine learning tools and techniques, 3rd Edition . Morgan Kaufmann, Elsevier, 2011.

J. Demsar, 'Statistical comparisons of classifiers over multiple data sets,' J. Mach. Learn. Res. , vol. 7, pp. 1-30, 2006.

Appendix A The proof of Theorem 1

Proof. The covariance matrix C q is a symmetric matrix. Therefore, if Assumption 1 holds, i.e., the covariance matrix C q is a nonsingular matrix, C q must be also a M -order symmetric matrix. This means that we can obtain M different eigenvalues and M mutually orthogonal normed eigenvectors when C q is given. These orthogonal normed eigenvectors can form an orthogonal matrix P , and thus P satisfies P -1 C q P = P T C q P = Λ . Here, Λ is a diagonal matrix with all M eigenvalues of C q as its elements of the principal diagonal. Moreover, the order of eigenvalues in Λ should correspond to the order of eigenvectors in P .

Appendix B The proof of Theorem 3

Proof. Theorem 3 holds when the distance metric can work effectively given a crowdsourced dataset. The distance metric works effectively, which means that the smaller the d ( x 1 , x 2 ) , the more similar x 1 and x 2 are to each other and the more likely they are to belong to the same class. Therefore, in the neighbor sequence < x 1 i , . . . , x k i , . . . , x N i > of x i , x k i and x i are more likely to belong to the same class when k is small. At this point, when P k i is updated, the probability corresponding to the unknown true label y i of x i will increase. When k gradually increases and exceeds a certain threshold, x i and x k i begin to belong to different classes, at which point the probability corresponding to y i will decrease. In other words, as k increases from 0, the probability corresponding to y i increases first. As k exceeds a certain threshold (the optimal neighborhood size), the probability corresponding to y i begins to decrease. Therefore, max ( P k i ) tends to be the probability corresponding to y i when k increases from 0, and k tends to be K ∗ i when ˆ M k achieves the highest value.

Appendix C More descriptions of the experimental datasets and metrics

Simulated datasets. The descriptions of the whole 34 simulated datasets are listed in Table 5. Here, '#Instances' denotes the number of instances, '#Attributes' denotes the number of attributes, '#Classes' denotes the number of classes, 'Missing' denotes whether the dataset contains missing values and 'Attribute type' denotes the type of attributes the dataset contains. These datasets are collected from different application scenarios and represent different crowdsourcing requirements.

Real-world datasets. The Income dataset is annotated by 67 workers through the online platform Amazon Mechanical Turk (AMT), and each instance is annotated by 10 different workers. The Income dataset is a binary crowdsourced dataset, which contains 600 instances, 6000 labels, 10 attributes (nominal attributes) and 0 missing values. The Leaves dataset is annotated by 83 workers through AMT, and each instance is annotated by 10 different workers. The Leaves dataset is a multi-class crowdsourced dataset, which contains 384 instances, 3840 labels, 64 attributes (numeric attributes) and 0 missing values.

Experimental metrics. The integration accuracy is calculated as follows:

The Macro-F1 score is calculated as follows:

where Precision q and Recall q can be calculated as follows:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly claim in the abstract and introduction section that the KFNN proposed in our paper is mainly used to improve the effectiveness and robustness of label integration in crowdsourcing. The contributions of our paper are also highlighted in the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We mentioned the limitations of our KFNN in the Conclusion and future work section. KFNN has two empirical parameters and its distribution transformation process is not refined enough.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We give proofs to theorems that appear in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper fully discloses all the information needed to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: This paper provides open access to the data and code, and provides a document to guide readers in reproducing all experimental results in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴ ♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿ ✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This paper specifies all the training and test details necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We explain in detail the experimental metrics in the paper and provide the results of the significance tests.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe in detail the experimental setting on the computer resources in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❊/a116❤✐❝/a115●✉✐❞❡❧✐♥❡/a115 ?

Answer: [Yes]

Justification: The research conducted in this paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The new algorithm proposed in this paper helps to improve the effectiveness and robustness of label integration. There are no negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We used the datasets and algorithmic implementations published by the CEKA platform, which is described in detail in the paper. The license and terms of use explicitly are properly respected in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ♣❛♣❡/a114/a115✇✐/a116❤❝♦❞❡✳❝♦♠✴❞❛/a116❛/a115❡/a116/a115 has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper provides open access to the data and code, and provides a document to guide readers in reproducing all experimental results in supplemental material.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper did not select new crowdsourced datasets. The datasets used for experiments are publicly available and their information is described in detail in the paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve the research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel algorithm, KFNN (K-free Nearest Neighbor), which is specifically designed to enhance label integration for crowdsourcing. KFNN integrates two key components named label distribution enhancement and K-free optimization, which significantly contribute to improving the effectiveness and robustness of the label integration process. The idea of automatically determining the optimal neighborhood size for each instance is particularly innovative and well-executed. The experimental results further validate the effectiveness and robustness of the proposed algorithm.\n\nSTRENGTHS:\n1.\tThe KFNN proposed in this paper is interesting and innovative. The authors reveal the limitations of fixed neighborhood sizes in existing label integration algorithms and propose an algorithm that automatically determines the optimal neighborhood size based on instance attributes and noisy labels. This algorithm significantly improves the robustness of label integration.\n2.\tThe paper provides a solid theoretical foundation for the proposed KFNN algorithm, followed by comprehensive experimental validation. The theoretical analysis is robust and convincingly demonstrates the expected performance improvements. The experiments are well-designed and cover a wide range of datasets, both simulated and real-world, to ensure the generalizability of the results. The experimental results, including comparisons with baseline algorithms, further validate the effectiveness and robustness of the proposed algorithm.\n3.\tThe paper is well-written and clearly presents the proposed methodology and findings. The structure of the paper is logical, making it easy to follow the complex concepts introduced. The use of figures and tables to illustrate key points is effective and aids in comprehension.\n\nWEAKNESSES:\n1. While the paper provides strong theoretical and experimental results, there is limited discussion on the computational efficiency and scalability of the proposed KFNN algorithm. I suggest moving the algorithmic flow and time complexity analysis from Appendix A to the main text.\n2. There are some repetitive sentences and structures in this paper that should be further condensed. For example, Sections 5.1 and 5.2 should be merged and the repetitive statements in them should be deleted.\n3. The experiments are already comprehensive, but analysis and discussion of the optimal neighborhood size determined by KFNN could still be added, which would help to understand how the neighborhood size should be set. Moreover, according to the results presented in Tables 1-4, KFNN is generally highly effective. However, on few datasets, KFNN does not perform as well as MV. These anomalies are valuable for identifying deficiencies in KFNN and should be further investigated and discussed.\n\nQUESTIONS:\nPlease refer to the Weaknesses.\n\nLIMITATIONS:\nThe authors openly discuss the limitations of their work, particularly the empirical parameters in the Kalman filter and the roughness of the distribution transformation process. Please refer to the Weaknesses for other limitations I have found.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel label integration algorithm, KFNN (K-Free Nearest Neighbor), designed to enhance the performance of crowdsourcing platforms by intelligently determining the optimal neighborhood size for each instance based on its attributes and noisy labels. The authors propose a two-component solution involving label distribution enhancement and K-free optimization, which leverages the Mahalanobis distance and a Kalman filter to mitigate noise from neighbor instances. The paper's claims are well-aligned with the theoretical and experimental results, demonstrating the effectiveness and robustness of KFNN against existing state-of-the-art algorithms in various crowdsourcing scenarios.\n\nSTRENGTHS:\n1.\tNovel contribution to an important problem\nThe innovative approach of highlighting the limitations caused by fixed neighborhood sizes in existing label integration algorithms, and using attributes and noisy labels to determine the neighborhood size for each instance automatically, is a significant contribution to crowdsourcing. \n\n2.\tComplete and rigorous theoretical proof\nThe theoretical underpinnings are sound, with clear assumptions and proofs provided for the proposed methods. The use of the Mahalanobis distance and the Kalman filter is well-justified.\n\n3.\tGood writing quality and clarity\nThis paper is well-written and enjoyable to read. The challenges are clearly stated and the contributions are easy to capture. \n\n4.\tReproducibility\nThe paper's open data and code policy is highly appreciated, promoting research transparency. Enhancing reproducibility with clear versioning and setup instructions would be a valuable addition, showcasing a strong commitment to open scientific practices.\n\nWEAKNESSES:\n1.\tSimulation experiment results\nThe symbol • indicates that the algorithm in the row significantly outperforms the algorithm in the corresponding column. How is \"significantly outperforms\" defined for Macro-F1 score and integration accuracy?\n\n2.\tAblation experiment results\nSince this study focuses on automatically adjusting neighborhood sizes, how does the performance of this method compare with baselines that use fixed neighborhood sizes?\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nsee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel label integration approach KFNN by adaptively determining the optimal neighborhood size. KFNN utilizes a Mahalanobis distance distribution to model the relationship between each instance and all classes. The authors also provide adequate theoretical analysis to illustrate the effectiveness of the proposed method. Experiments demonstrate that the proposed method can achieve the state-of-the-art performance on simulation and real-world dataset. The paper is well-written and easy to follow. This idea is very intuitive and effective for crowdsourcing task. The paper proves the effectiveness of introducing Mahalanobis distance distribution for crowdsourcing from the perspective of methodology, theory and experiments.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow. The logic of the whole paper is clear.\n2. The paper’s idea is very intuitive and effective for crowdsourcing task. The authors introduce the Mahalanobis distance distribution to model the relationship between each instance and all classes. Experiments verify that the proposed method can achieve the best performance compared with SOTAs.\n3. The authors provide adequate evidences to verify the effectiveness of the proposed method from the perspective of methodology, theory and experiments on simulation and real-world datasets.\n\nWEAKNESSES:\n1. In section 2, the authors introduce two categories of label integration algorithms. And the proposed KFNN belongs to the algorithms which leverage neighbor instance. I suggest adding some discussion about the pros and cons of these two categories of approaches.\n2. In methodology part and theoretical analysis part, the authors discuss the superiority of Mahalanobis distance compared with Euclidean distance. Can the authors verify the difference between Mahalanobis distance and Euclidean distance on this task from an experimental perspective?\n3. In Table 3 and Table 4, why some results are missing? Appropriate explanation facilitates reading of the paper.\n\nQUESTIONS:\nPlease refer to weakness. My biggest concern is the experiments for the comparison between Mahalanobis distance and Euclidean distance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new algorithm for label integration called KFNN. Existing methods related to KNN produce more noisy labels; however, they fix the neighborhood size, regardless of the fact that instances close to the center of classes should have more neighbors than instances close to the boundary of classes. To tackle this problem, KFNN estimates a Mahalanobis distance distribution between each instance and all classes to enhance the multiple noisy label distribution and utilizes a Kalman filter to mitigate the impact of noise. Finally, KFNN can automatically determine the optimal neighborhood size through max-margin learning.\n\nSTRENGTHS:\nS1. The paper studies an important problem.   \nS2. A new solution is proposed to tackle the problem.   \nS3. Experiments are conducted on several datasets.\n\nWEAKNESSES:\nW1. The motivations need more enhancements.   \nW2. Some technical details require more explanations.   \nW3. The application scope of the proposed method in crowdsourcing is limited.    \nW4. The performance improvement of the proposed method is unsatisfactory.    \nW5. Experiments are conducted in a simulation environment, which can be much simpler than a real-world crowdsourcing platform.\n\nQUESTIONS:\nD1. The paper focuses on the KNN-related methods for label integration. However, the introduction didn’t justify the motivation of this concentration with convincing proofs. For instance, the motivation is basically explained with the sentence, “to alleviate this problem, recent works have begun to focus on leveraging neighbor instances [1, 11, 12] …”. However, there are also alternatives for label integration, so why considers KNN-related instead of the other types of solutions? Besides, there are much more studies (eg [R1]) that also target on this problem, which should be carefully discussed their pros and cons. Otherwise, the motivation looks weak.\n\nD2. From the perspective of crowdsourcing, the studied problem is closely related to “truth inference”. However, in the references, there are only two papers on this topic: [25] (published in 2016) and [26] (published in 2023). More studies, which can be easily found in Google Scholar or DBLP, should be reviewed and compared (if possible).\n\nD3. It is a little unclear how the principle of employing the same neighborhood size can impact performance. Please give more explanations.\n\nD4. In addressing the question of fusing information from the attribute space and the multiple noisy label space, this paper tends to take an average between the multiple noisy label distribution and the potential label distribution. However, it might be worth exploring the possibility of introducing a tunable parameter to achieve a more optimal balance between these two distributions, rather than relying solely on an equal (50%) average. \n\nD5. The application scope of the proposed method in crowdsourcing is limited. In my opinion, the proposed KFNN can be only used in simple and micro tasks in crowdsourcing, there are many other kinds of tasks in a real-world crowdsourcing platform, such as ranking [R2], which is not considered in the problem setting. Yet, the title, “KFNN: K-Free Nearest Neighbor For Crowdsourcing”, is a little over-claimed. At least, the paper should explicitly define the application scope. More types of crowdsourcing task can be found in existing surveys [R3, R4] on crowdsourcing.\n\nD6. The performance improvement of the proposed method is unsatisfactory.    \n(1) Although the average Macro-F1 score of KFNN is better than the compared baselines, it can be notably worse than some of the baselines in certain datasets (eg MNLDP on the anneal dataset). This pattern weakens the motivation, since it’s unclear whether the limitation of existing solutions has been well addressed or not.   \n(2) In Table 2, the integration accuracy of KFNN is lower than that of MNLDP. Besides, it can be also notably worse than some of the baselines in terms of the integration accuracy (eg LAGNN and LAWMV on the breast-cancer dataset).   \n(3) Based on the current experimental results, the effectiveness of the proposed solution KFNN is questionable.\n\nD7. Although several datasets are conducted in the experimental study, existing work on truth inference in crowdsourcing (eg [R2, R5]) usually deploys their solution in a real-world platform, such as AMT, to verify the performance. Therefore, the setup of the experimental study can be simplifier and less practical than the real-world scenario.\n\nReferences:  \n[R1] Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learning. AAAI 2024.   \n[R2] Xi Chen et al. Pairwise ranking aggregation in a crowdsourced setting. WSDM 2013.   \n[R3] Guoliang Li et al. Crowdsourced Data Management: A Survey. IEEE TKDE 2016.   \n[R4] Hector Garcia-Molina et al. Challenges in Data Crowdsourcing. IEEE TKDE 2016.   \n[R5] Yudian Zheng et al. Truth Inference in Crowdsourcing: Is the Problem Solved? VLDB 2017.\n\nLIMITATIONS:\nPlease refer to the weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-12"}