Label Distribution Propagation-based Label Completion for Crowdsourcing
Authors: Tong Wu
Liangxiao Jiang
Wenjun Zhang
Chaoqun Li
Venue: ICML 2025 poster (2025)
Keywords: Crowdsourcing learning
Label completion
Worker similarity
Label distribution propagation
Primary area: general_machine_learning->supervised_learning
TL;DR: This paper proposes a novel label distribution propagation-based label completion (LDPLC) algorithm.

Abstract: In real-world crowdsourcing scenarios, most workers often annotate a few instances only, which results in a significantly sparse crowdsourced label matrix and subsequently harms the performance of label integration algorithms. Recent work called worker similarity-based label completion (WSLC) has been proven to be an effective algorithm to addressing this issue. However, WSLC considers solely the correlation of the labels annotated by different workers on per individual instance while totally ignoring the correlation of the labels annotated by different workers among similar instances. To fill this gap, we propose a novel label distribution propagation-based label completion (LDPLC) algorithm. At first, we use worker similarity weighted majority voting to initialize a label distribution for each missing label. Then, we design a label distribution propagation algorithm to enable each missing label of each instance to iteratively absorb its neighbors’ label distributions. Finally, we complete each missing label based on its converged label distribution. Experimental results on both real-world and simulated crowdsourced datasets show that LDPLC significantly outperforms WSLC in enhancing the performance of label integration algorithms. Our codes and datasets are available at https://github.com/jiangliangxiao/LDPLC.

Label Distribution Propagation-based Label Completion for Crowdsourcing
Tong Wu' Liangxiao Jiang'
Wenjun Zhang'
Chaoqun Li?
Abstract
sition often presents significant challenges (Tu et al., 2019; Wu al., 2023; Zhang et al., 2023). Traditionally, high- quality annotated data is generated through manual annota- tion by domain experts, but which is both conecconsuning and expensive. As a result, efficiently obtaining large-scale and high-quality annotated data has become a major bottle- neck in the practical application of supervised learning.
In real-world crowdsoureing scenarios, most workers often annotate a few instances only, which results in significantly sparse crowd- sourced label matrix and subsequently harms the performance of label integration algorithms. Re- cent work called worker similarity-bast -based label completion (WSLC) has been proven to be an effective algorithm to addressing this issue. How- ever, WSLC considers solely the correlation of the labels annotated by different workers on per indi- vidual instance while totally ignoring the correla- tion of the labels annotated by different workers among similar instances. To fill this gap, we pro- pose a novel label distribution propagation- based label completion (LDPLCC) algorithm. At first, we use worker similarity weighted majority voting to initialize label distribution for each missing label. Then, design label distribution prop- agation algorithm to enable each missing label of each instance to iteratively absorb its neigh- bors' label distributions. Finally, we complete each missing label based on its converged label distribution. Experimental results on both real- world and simulated crowdsourced datasets show that LDPLC significantly outperforms WSLC in enhancing the performance of label integration algorithmss. Our codes and datasets are available at http:///gthbb.com///aiiiinaaaa/p
To break through this bottleneck, crowdsourcing is intro- duced an efficient and cost-effective method to rapidly ob- tain large volumes of annotated data (Xu et al., 2022; Ji et al., 2023). Currently, mainstream crowdsourcing platforms in- clude Amazon Mechanical Turk (AMT), CloudCrowd and CrowdFlower. These platforms utilize the internet to out- source tasks traditionally performed by domain experts to a distributed group of non-professional crowd workers (Liang et al., 2023; Mak & Lam, 2023). However, due to variations in the knowledge and abilities of crowd workers, the anno- tated labels are often accompanied by noise and inconsistent quality. To address this issue, each instance is typically assigned to multiple crowd workers and thus obtains its mul- tiple noisy label set. Then, label integration used infer the integrated label of each instance from its multiple noisy label set (Sheng et al., 2008; Zhang et al., 2024a). In recent years, label integration has attracted widespread attention from researchers, and large number of algorithms have been proposed from different perspectives (Zheng et al., 2017; Zhang, 2022; Zhang et al., 2024c).
However, in real-world crowdsourcing scenarios, most workers often annotate few instances only, thus most in- stances do not receive sufficient labels, leading to large number of missing labels in the crowdsourced label matrix. The sparse label matrix significantly harms the performance of label integration algorithms. To mitigate the problem of sparsity, a small number of researchers have paid attention to the importance of the preprocessing of crowdsourced la- bel matrix and proposed few label completion algorithms. For example, to address binary-class crowdsourcing tasks, Jung & Lease (2012) propose probabilistic matrix factor- ization (PMF) label completion algorithm. To further improve its performance, Yang et al. (2024) propose a PMF- based three-stage label completion (PMF-TLC) algorithm. Recently, to address multi-class crowdsourcing tasks, Wu et al. (2024) propose worker similarity- based label comple- tion (WSLC), which significantly improves the performance
1. Introduction
Supervised learning is critical branch of machine learn- ing, with the primary goal of learning the mapping between inputs and outputs from annotated data (Li & Tang, 2017; Jiang et al., 2019; Li et al., 2021). High-quality annotated data is crucial for the success of model training, but its acqui-
'School of Computer Science, China University of Geosciences, Wuhan 430074, China "School of Mathematics and Physics, China University of Geosciences, Wuhan 430074, China. Correspon- dence to: Liangxiao Jiang <ljiang ccu..ccccc.cd.. edu.en>>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Label Distribution Propagation-basee Label Completion for Crowdsourcing
of label integration algorithms.
utilizes the expectation maximization framework to itera- tively estimate true labels and observer error rates. Demar- tini et al. (2012) propose ZenCrowd(ZC), which employs two-element probability parameter to iteratively estimate the reliability of each individual worker. Li & Yu (2014) proposes iterative weighted majority voting (IWMV), which optimizes error rate bounds to find reliable workers. Zhang et al. (2015b) propose positive label frequency threshold (PLAT), which is designed to address imbalanced scenarios and uses multiple noisy label sets to model the decision thresholds. Zhang et al. (2016) also propose ground truth inference using clustering (GTIC), which generates features from multiple noisy label sets and then clusters instances into different groups utilizing generated features. Tao et al. (2021) propose differential evolution- weighted soft majority voting (DEWSMV), which exploits differential evolution algorithm estimate the quality of crowd workers labeling different instances. Chen et al. (2022) propose label augmented and weighted majority voting (LAWMV), which merges the neighbors' multiple noisy label sets to obtain its augmented multiple noisy label set and weights cach neighbor. Jiang et al. (2022) propose multiple noisy label distribution propagation (MNLDP), which at first estimates the multiple noisy label distribution of each instance from its multiple noisy label set and then propagates its multiple noisy label distribution to its nearest neighbors.
To the best of our knowledge, WSLC considers solely the correlation of the labels annotated by different workers on per individual instance while totally igmoring the correlation of the labels annotated by different workers among similar instances. To fill this gap, we propose a novel label comple- tion algorithm called label distribution propagation- based label completion (LDPLC). At first, we use worker similar- ity weighted majority voting initialize a label distribution for each missing label. Then, we design label distribu- tion propagation algorithm to enable each missing label of each instance to iteratively absorb its neighbors' label dis- tributioos. Finally, we complete each missing label based on its converged label distribution. In general, the main contributions of this work can be summarized follows:
We find that WSLC considers solely the correlation of the labels annotated by different workers on per indi- vidual instance while totally ignoring the correlation of the labels annotated by different workers among similar instances.
We propose a label distribution propagation-basee based label completion (LDPLC) algorithm. In LDPLC, we use worker similarity weighted majority voting to initialize a label distribution for each missing label and then design a label distribution propagation algorithm to enable each missing label of each instance iteratively absorb its neighbors' label distributions.
To the best of our knowledge, all above works primarily aim to design more effective label integration algorithms while paying little attention to the preprocessing of crowd- sourced label matrix before label integration. In real-world crowdsourcing scenarios, most workers often annotate few instances only, which results in significantly sparse crowdsoureed label matrix and subsequently harms the per- formance of label integration algorithms. To mitigate the problem of sparsity, a small number of researchers have paid attention the importance of the preprocessing of crowd- sourced label matrix and proposed few label completion algorithms. For example, Jung & Lease (2012) propose an algorithm to utilize probabilistic matrix factorization (PMF) to address the challenge of predicting missing labels in binary-class crowdsourcing tasks. To further enhance its performance, Yang et al. (2024) propose a PMF--baedd three- stage label completion (PMF-TI.C) algorithm, which flips low-quality raw labels based on a confidence-based strategy, employs PMF to complete missing labels, and finally filters low-quality output labels using a between-class margin- based algorithm. Recently, to address multi-class crowd- sourcing tasks, Wu et al. (2024) propose worker similarity- based label completion (WSLC), which completes each missing label by worker similarity. Specifically, they esti- mate the similarity between each pair of workers and com- plete the missing labels based on the estimated worker simi- larity and the labels from similar workers. However, WSLC
We conduct a series of experiments on both real-world and simulated crowdsourced datasets. The experimen- tal results indicate that LDPLC further improves the performance of label integration algorithms compared to WSLC.
The rest of the paper is organized as follows. Section 2 briefly reviews related work on label integration and label completion. Section 3 proposes our LDPLC. Section 4 reports the experimental setup and results. Section 5 con- cludes the paper and outlines future work.
2. Related Work
Over the past few years, a large number of label integration algorithms have been proposed from different perspectives. Among them, majority voting (MV) is a simple yet effec- tive label integration algorithm, which selects the most fre- quently assigned label the integrated label. Since MV ignores the annotation quality of each crowd worker, it is sensitive to noise and unsuitable for complex tasks. To improve its performance, researchers propose many more sophisticated label integration algorithms. For example, Dawid & Skene (1979) propose Dawid-Skens (DS), which
2

---

Label Distribution Propagation-base based Label Completion for Crowdsourcing
considers solely the correlation of the labels annotated by different workers on per individual instance while totally ignoring the correlation of the labels annotated by differ- ent workers among similar instances. To fill this gap, we propose a novel label distribution propagation -aased label completion (LDPLC) algorithm in Section 3.
Feskure Learnnn
Leamed Fentium
Worker Siniinity Estimation
Warker
?
Label 1ill alluttonn
Oeggial Duteaet
Distitutionn
Completr Dutreel
Neilllor (herdda
Wegtt Cittmntation X
Noigloor
Notghoor Weight
3. The Proposed LDPLC
Figure I. Overall framework of LDPLC.
3.1. Motivation
each worker. Secondly, we use cosine similarity to esti- mate worker similarity for each pair of workers based on the learned feature vectors. Thirdly, we initialize label distribution for each missing label based on the original crowdsourced dataset and the estimated worker similarity of each pair of workers. Fourthly, we query neighbors for cach instance in the feature space of the original crowdsourced dataset. Fifthly, we optimize the neighbors' weights using local linear embedding. Finally, we propagate the initialized label distribution from weighted neighbors to each missing label of each instance and then complete each missing label based on its converged label distribution. In the following subsections, we will present the detailed process.
As discussed in Section 2, existing works have primarily focused on label integration while paying little attention to the preprocessing of crowdsourced datasets. In real-world crowdsoureing scenarios, most workers often annotate a few instances only, which results in a significantly sparse crowdsourced label matrix. WSLC has been validated as an effective label completion algorithm to address this is- sue. WSLC assumes that different workers with similar cognitive abilities will annotate similar labels on the same instance. Based on this assumption, WSLC completes the missing labels based on the worker similarity and the la- bels from similar workers. However, since each instance is annotated by only few workers, WSL.C relies on limited information from these similar workers to complete missing labels, which does not fully use the correlation of the labels in crowdsourced dataset. To overcome this limitation, we introduce the assumption that the same workers will also annotate similar labels on similar instances. This additional perspective allows each missing label to not only absorb the label distribution information from similar workers on the corresponding instance but also absorb the label distribution information from the corresponding worker across similar neighboring instances.
3.2. Label Distribution Initialization
In a crowdsourcing scenario, a crowdsourced dataset D can be denoted as { L,) where N denotes the num- ber of instances. a,, donates the i-th instance in D, which can be denoted as fa.1..... a.m...... a&M }. M denotes the number of features and aim denotes the m-th feature value of the i-th instance. L; denotes the multiple noisy label set of æ;, which is denoted as denotes the number of workers and 4rr denotes the label annotated by r-th worker u, on F; 4rr is drawn from a fixed set {-1, ci,..., Cqs. cQ). where Q denotes the number of classes, Cg denotes the q-th class and -1 means that u,, does not annotate .;;
.r). R
Specifically, we attempt to initialize a label distribution for each missing label and then design a label distribution prop- agation algorithm to enable each missing label to absorb information from its neighbors. Therefore, two core issues need be addressed. The first issue is how to initialize a la- bel distribution. WSLC assumes that different workers with similar cognitive abilities will annotate similar labels on the same instance. Based on this assumption, we use worker similarity weighted majority voting to initialize a label dis- tribution for each missing label. The second issue is how to propagate the initialized label distribution. We assume that the same worker will also annotate similar labels on similar instances. Therefore, we first query neighbors, then optimize their weights, and finally propagate the distribution from weighted neighbors to each missing label.
According to the framework of LDPLC, worker similarity estimation is crucial step in label distribution initialization. In worker similarity estimation, we first need to construct a dataset D, for uy where D, consists of all the instances annotated by u, in D. Next, we learn a feature vector V, for u, based on the label variable and the original feature variables of D,. We denote V, as (#r1...., Urm: ..., Uy ) where Vrm denotes the m-th feature value of u,, Here, we use Pearson correlation to learn the feature vector (Hall, 2000; Wu et al., 2024) by Eq. (1).
cor Cor, Arm)
To this end, we propose a novel label distribution propagation- label completion (LDPLC) algorithm. Its framework can he graphically shown in Figure Firstly, we use Pearson correlation learn feature vector for
> (brr Crs(amm - Apm)
(1)
Dr (lrr C, ElDr (airm - Arm)
where Cr and Arm denote the label variable and the m-th
3

---

Label Distribution Propagation-base based Label Completion for Crowdsourcing
feature variable of D,.. respectively. And dirm denotes the m-th feature value of the i-th instance in D,- C₁ and Arm denote the mean values of C, and Arm. respectively. Or and Arm can be calculated by Eqs. (2) and (3), respectively.
Vr, which is calculated by Eq. (7).
V V
COS
cos(V, Vir
V
V
Σ Em=1 Urm Up'm
M
C, =
1
Σ Kr-
De
(2)
Σ M
2m=1 Vpmm
M Em= vmmm
Drl
1=1
Since cos Vr, Vyr) takes values in the interval [-1,]], we take max-min normalization for the values of cos( Vr, V,r), which calculated by Eq. (8).
|0
1 |D
Arm-
Airm-
(3)
8(Ur, upp') cos Vr, --1-
(8)
1-(-1)
Since the label variable is discrete variable, we use weighted Pearson correlation. To be specific, if we want to calculate Pearson correlation between C, and Arm. we can binar decompose C, into multiple binary label variables, each binary label variable corresponding to a value of C,.. The value of each binary label variable takes value when the corresponding value of C, occurs and 0 for other values. Next, we calculate Pearson correlation between each binary label variable and Apm, and perform a weighted summa- tion by the prior probability of the corresponding value. If Arm is also discrete, we decompose it into multiple binary feature variables and then calculate Pearson correlation be- tween each binary label variable of C, and each binary feature variable of Arm: Based on the above analysis, the feature value frm of u, is calculated by Eq. (4).
With the above calculation of worker similarity, we can proceed initialize label distribution for each label. Let P.r. {Parl..... Pirg............................ Pirqs..... PirQQ } denote the label distribu- tion of lir, where Piry denotes the probability 4ü- belongs to cy. Therefore, for annotated labels and unannotated labels, we can directly initialize their label distributions by Eqs. (9) and (10), respectively.
1, is lir 1-
pirg
(9)
0, otherwise
R Σ 5llir', C.g Cg)o Urr)
Pirg =
(10)
Σ p(C, = (o)cor(CCr cor(Cor; Arm),
q= Ep= 8(1sr', (urr up)
if Arm is continuous variable,
3.3. Label Distribution Propagation
Trm
ΣΣ p(C, = Lr Arm arm) cor (Chr, Abrm)s
In the process of label distribution propagation, we design an algorithm to enable each label to iteratively absorb infor- mation from its neighbors. First, we use the HEOM metric to query K neighbors of ar, where neighbors N₂ {a:x ) k= and æu denotes k- neighbor of æ;; Then, we use the local linear embedding (LLE) to optimize the weights of K neighbors (Miao et al., 2022; Ghojogh et al., 2022), where the weights are denoted as w; = {was K Here, we construct a minimization objective function based on LLE as Eq. (11).
6pm
otherwise.
(4)
denoted as
where 4, and dym denotes the values that C, and Arm take, respectively. Cor and A are binary label variable of C, and binary feature variable of Arm. respectively. The prior probabilities p(C,- = 4,) and p(C, = Lr, Arm arm) can be calculated by Eqs. (5) and (6), respectively.
p(C, (,) E.Pil 8(lir: (n)
(5) L(w;)) =
Σ
Wisk (2. (kk, )T (xx: -Kky Juik.
[r1l
kj Rogita, Fag EN;
(11)
p(C,- = Arm = arm) Elerl (1 arm)
This objective can be solved using the least squares algo- rithm, which is defined as Eq. (12).
Drl
(6)
where 6(.) denotes an indicator function that outputs 1 if the two input values are equal and outputs O otherwise.
min L(w:) we
K =1
By Eqs. (1)-(6, (6), we calculate the learned feature vector V, for u, Finally, we use cosine similarity to estimate worker similarity s(U, ) between u, and uy based on V, and
Wik
(12)
s.t.
1=1
Vwia E wi Wik 2 0
4

---

Label Distribution Propagation-base Label Completion for Crowdsourcing
Algorithm L.PPLC
in Algorithm After the above process, the completed
D' can be fed into existing label integration algorithms enhance their performance.
1: Input: {(x,, - -a crowdsourced dataset
2: Output: D' ohe completed crowdsourced dataset
3: for rr- 1 to I do
4: Construct datasct D, for
3.4. Time Complexity Analysis
to do 6: Learn the feature value Drm of ur by Eqs. (1)(6)
5: for
In Algorithm 1, lines 3-8 learn feature vectors with time complexity of O(R(N + +M (numa (D)))), where ny and no are the average number of values for a label variable and an original feature variable, respectively. Lines 9-13 esti- mate worker similarity with time complexity of (RR² Lines 14-22 initialize label distributions with a time com- plexity of (NNR(Q + R)). Lines 23-26 identify neigh- bors and optimize their weights with time complexity of O(N N Lines 27-33 propagate the label distri- butions with a time complexity of O(TNRKQ) Finally, lines 34-40 complete missing labels with time complexity of O RQ). only the highest order terms are taken, the overall time complexity of LDPLC O D.JRM + R M N + N NK + TNRKK).
7: end for
8:
end for
9: for to R do
for
R
11: Estimate the worker similarity s(u,, up:) by Eqs. (7)
and (8)
12: end for
13:
end for
= to do
15: ror-= = to It do
16: !,-! - -1 then
17:
Initialize the label distribution P.. Eq. (9)
18: else
19:
Initialize the label distribution P.. by Eq. (10)
20: end if
21: end for
22:
end for
23: = to do
3.5. Convergence Analysis
24: Query the neighbors M, for æ;
25: Optimize the neighbors' weights w: by Eqs. (11) and (12)
In the part of the previous subsection, we describe the label distribution propagation for up on E₁. Now, we discuss the propagation process of u, across the whole dataset D. First, which has size of N x Q. The matrix W is defined as the weights of the nearest neighbors, with size of N x N, where each element Wiy W denoted by Eq. (15).
26: end for
27:
= 1
28:
for = 1 N do
we define P, [P] ir...... P! .r. ....... P .f.
29:
for :-r== =1 1 to do
30:
Propagate the neighbors' label distributions P.r
Eq. (13) end for 32: end for
31:
33: end for
Wike if = Tik
34: for = 1 to /V do
Way
(15)
35:
to R
0, otherwise
36:
if 1,r == then
37:
Complete the missing label 1úr by Eq. (14)
38:
end if
Then, we update Eq. (13) to Eq. (16).
39: end for
40: end for
D
W
P
+
Pr 2
(16)
41: return D'
2
Eq. (16) is a linear iterative formula, where W controls the weight of propagation, and P. preserves the influence of the original distribution. When is equal to 0, it is obtained that Ph is equal to P,. By expanding the recursion using mathematical induction, we derive the explicit expression of P as Eq. (17).
Therefore, we can obtain a weight was for each neighbor Fik of 2;.- Next, we propagate the label distributions from each neighbor Vi to x; by Eq. (13).
p1+1 ExSHEE Wik P + P
(13)
where p denotes the distribution of the label annotated by u, Jak after iterations of propagation. When the propagation converges, we obtain the converged distribution P of each missing label 4,, Finally, we complete 4, based on Pr. which is calculated by Eq. (14).
W P
W 2
P
P
(17)
Since the elements in W satisfy W3, E [0, 1]. the spectral radius of W is less than 1, ensuring that ( approaches 2 as increases, as shown in Eq. (18),
4p org max P e, C.E.(L.e2...og))
(14)
W y = 0.
To this end, the learning algorithm of LDPLC is described
lim 1-
(18)
2
5

---

Label Distribution Propagation-basee Label Completion for Crowdsourcing
worker
MSL
00-
$1.00
11.39
$144
ROME
Annotation quality
10
6.-
5.000
0923
0.050
0.250 0.375
notittion ratto
7.820
0.223
0.575
NV 05
TWNV
GTC
DEWSOV
WALLP
Figure 2. Worker distribution of the "LabelMe" dataset.
Figure 3. Integration accuracy (%) comparisons of six label inte- gration algorithms after label completion by WSLC and LDPLC on the "LabelMc" datasct, respectively.
According to the geometric series formula, EE can be calculated by Eq. (19).
4.2. Experiments on Real-world Datasets
W
(W] -
(2)
To evaluate the performance of LDPLC, we conduct experi- ments on the widely used real-word crowdsourced dataset "LabelMe". The dataset is collected from AMT platform, de- signed for image classification. The dataset "LabelMe" con- tains 1000 instances, and includes 8 classes: "highway", "in- side city", "tall building". "street", "forest", "coast", "moun- tain", and "open country". In total, there are 2547 multiple noisy labels from 59 workers, with a mean annotation qual- ity of 74.05% and mean annotation ratio of 4.32%. Each worker's 's annotation ratio and annotation quality denote the proportions of the instances he annotated in all instances and the true labels in all his annotated labels, respectively, shown in Figure 2. It can be seen that most workers' anno- tation qualities are greater than 0.5, enabling each label to absorb more accurate information during the propagation process. Although few workers' annotation qualities below 0.5, their annotation ratios very low, resulting in minimal impact on the propagation process. Therefore, we use this dataset in our experiments.
lim
lim
2
2
(19)
W
(1 2 )-
Therefore, Pt, converges to fixed matrix, which is calcu- lated by Eq. (20).
W
Pr.
lim P = 100
)
(20)
2
2
4. Experiments and Results
4.1. Experiments Setup
To validate the effectiveness of the LDPLC algorithm, we use WSLC (Wu et al., 2024) and LDPLC to complete miss- ing labels in real-world and simulated datasets, respectively, and compare their effectiveness in terms of the integration accuracy of each label integration algorithm. In our exper- iments, we implement LDPLC and WSLC on the Crowd Environment and its Knowledge Analysis (CEKA) (Zhang et al., 2015a) platform. The label integration algorithms used include MV (Sheng et al., 2008), DS (Dawid & Skene, 1979), IWMV (Li & Yu, 2014), GTIC (Zhang et al., 2016), DEWSMV (Tao et al., 2021) and MNLDP (Jiang et al., 2022). For MV and GTIC, we use the existing implementa- tions on the CEKA platform. For DEWSMV and MNLDP. we implement them on the CEKA platform. The parameter settings of all label integration algorithms are consistent with those specified in their original papers. For simplicity, we set both the number of neighbors K and the number of it- erations to 5. All experiments are independently repeated ten times, and the average results of the ten experiments are used the final results.
Figure shows the integration accuracies of six label in- tegration algorithms after label completion by WSLC and LDPLC on the "LabelMe" dataset, respectively. We can see that after label completion using LDPL.C, the integration ac- curacy of each integration algorithm improves significantly. Specifically, the integration accuracies of MV (81 18..0%)) DS (8 1.50%) IWMV (81.60%) and MNLDP (82.50%) after label completion by LDPLC are much higher than those of MV (76.40%), DS (76.70%), IWMV (76.30%) GTIC (76.40%). DEWSMV (76.50%) and MNLDP (80.50%) after label completion by conclude
GTIC (81.70%), DEWSMV
WSLC, respectively. From these results, that LDPL.C further improves the performance of all label integration algorithms compared to WSLC. Except for the dataset "LabelMe", we also conduct experiments on two other widely used real-world crowdsourced datasets "Ruters" and "Leaves" to validate the effectiveness of L.PLCC Due
6

---

Label Distribution Propagation-basee Label Completion for Crowdsourcing
the limited pages, the detailed description of these datasets and experimental results provided in Appendix A. These results also demonstrate the effectiveness of LDPLC.
Table Description of 34 simulated datasets.
Dataset
#Instances
#Features
#Classes
Missing
Feature type
anneal
898
yes
hybrid
4.3. Experiments on Simulated Datasets
audiology
226
69
yes
nominal
autos
205
25
yes
hybrid
In this subsection, conduct series of experiments the whole 34 simulated datasets published the CEKA platform. The detailed description of these datasets is pro- vided in Table 1. Here, "#Instances" denotes the number of instances, "#Features" denotes the number of attributes, "#Classes" denotes the number of classes, "Missing" denotes whether the dataset contains missing values and "Feature type" denotes the type of attributes the dataset contains. Since MNLDP cannot handle missing feature values, we need to complete missing feature values on the datasets. Specifically, we replace all missing feature values with the mean of numerical feature values or the modes of nominal feature values from the available data, using an unsuper- vised feature filter ReplaceMissing Values in the Waikato Environment and Knowledge Analysis (WEKA) (Witten al., 2011) platform. Subsequently, we simulate 40 crowd workers to replicate a real-world crowdsourcing process according to the characteristics of the "LabelMe" dataset. Observed in Figure 2, most workers' annotation ratios and annotation qualities are in the interval [o, 0.1] and [0.6, 0.9), respectively. Therefore, for each simulated crowd worker uy, we randomly sample its annotation ratio and annotation quality from the uniform distribution [0, 0.1] and [0.6, 0.9). respectively. As a result, approximately 95% of the labels in the label matrix are missing, reflecting the sparsity observed in real-world crowdsourced datasets.
balance-scale
625
4
80
numeric
biodeg
1055
41
no
numeric
breast cancer
286
yes
breastww
699
9
yes
numeric
car
1728
no
nominal
credit-a
690
15
yes
hybrid
credit-g
1000
no
hybrid
diabetes
768
no
numeric
heart-c
303
13
yes
hybrid
heart-h
294
13
yes
hybrid
heart statlo
270
13
no
numeric
hepatitis
155
yes
hybrid
horse-colic
368
22
yes
hybrid
hypothyroid
3772
29
yes
hybrid
ionosphere
351
34
Do
numeric
iris
150
4
no
numeric
kr-vs-kp
3196
36
no
nominal
labor
57
16
yes
hybrid
letter
20000
16
26
DO
numeric
lymph
148
18
no
hybrid
mushroom
8124
yes
nominal
segment
2310
19
numeric
sick
3772
29
yes hybrid
sonar
208
60
DO numeric
spambase
4601
57
DO
numeric
tic-tac-toe
958
DO
vehicle
846
18
B0
numeric
vote
435
16
yes
nominal
vowel
990
13
no hybrid
5000
no hybrid
numeric
101
DS (83.57%), IWMV (83.54%), GTIC (83.45%) DEWSMV (83.54%) and MNLDP (83. (3%99) after label completion by LDPLC are much higher than those of MV (74.56%), DS (74.08%), IWMV (74.57%) GTIC (74.43%), DEWSMV (74.55%) and MNLDP (76.80%) after label completion by WSLC. respectively. The result once again demonstrates that LDPLC further improves the performance of all label integration algo- rithms compared to WSLC.
Table presents the integration accuracy comparisons of six label integration algorithms after label completion by WSLC and LDPLC, respectively. Based on these results, compare each pair of label integration algorithms after label completion by WSLC and LDPLC using the corrected paired two-tailed t-test with the significance level a 0.05 (Nadeau & Bengio, 2003; Zhang et al., 2024b). The symbols and in the table indicate that the integration accuracy has statistically significant improvement or degradation using our proposed LDPLCC compared to WSLCC respec- tively. The bottom of Table 2 denotes the mean integration accuracy and the W Win/Tic Lose (W/T/D L) across the 34 datasets. The mean integration accuracy reflects the aver- age performance of different label integration algorithms after label completion by WSLC and LDPLC. The W/T/L indicates that in terms of improving the performance of label integration algorithms, LDPLC wins WSLC on W datasets, ties on T datasets, and loses on L datasets. These experimental results verify the effectiveness of our proposed LDPLC. Specific conclusions are summarized as follows:
LDPLC significaal wins WSLC on 30, 30, 30, 30, 30, and 27 datasets for MV, DS, IWMV, GTIC, DEWSMV and MNLDP, respectively, while significantly losing only on 1 dataset. In t-test results, the number of datasets in which LDPLC wins significantly (W) is always much higher than the number of datasets in which it loses significantly (L) for all label integra- tion algorithms. This further strongly demonstrates the effectiveness of LDPLC.
In addition, to validate the robustness of LDPLC under different annotation quality distributions, we conduct an- other set of experiments. In new experiments, we replace the distribution of the annotation quality with a Gaussian distribution NO.775, 0.152), while keeping other settings
The average integration accuracies of MV (83.51%)
7

---

Label Distribution Propagation-basee Label Completion for Crowdsourcing
Table Integration accuracy (%) comparisons of six label integration algorithms after label completion by WSL. C and I.PLCC C the uniform distribution, respectively.
MV
DS
IWMV
GTIC
DEWSMV
MNLDP
Dataset
WSLC
LDPLC
WSLC
LDPLC
WSLC
LDPLC
WSLC
LDPLC
WSLC
LDPLC
WSLC
LDPLC
anneal
70.45
86.31
68.21
86.24
70.41
86.29
69.82
86.26
70.35
86.31
74.70
86.38
audiology
69.25
81.19
69.20
81.24
69.11
81.33
68.98
80.44
69.12
81.19
71.20
80.62
autos
71.76
82.68
71.61
82.68
82.63
82.68
73.95
bulance-scale
75.15
83.46
72.34
83.01
74.83
83.34
73.46
83.34
74.99
83.44
79.49
84.16
biodeg
77.95
81.42
77.72
81.31
77.92
81.39
77.82
81.44
77.92
81.44
79.99
81.75
breast-cancer
78.39
78.15
77.10
78.14
78.04
78.14
77.80
78.11
78.50
78.32
79.27
77.72
breast-w
78.48
85.81
78.37
85.64
78.44
85.72
78.43
85.84
78.44
85.82
82.47
88.58
78.40
credit-a
74.59
81.06
74.75
81.03
74.65
81,05
74.64
81,06
74.59
81.06
76.42
81.32
credit-g
77.00
77.80
75.95
77.72
76.76
77.79
76.61
77.79
76.93
77.82
78.44
77.55
diabetes
76.63
78.31
76.05
78.09
76.56
78.15
76.39
78.27
76.69
78.27
78.16
78.18
heart-c
77.33
84.29
76.27
84.23
77.16
84.19
75.58
84,33
77.43
84,33
80.69
84.16
heart-h
77.18
83.74
76.67
83.67
77.07
83.70
76.33
77.14
83.70
80.54
83.23
heart-statlog
74.92
78.52
74.48
78.59
74.44
78.81
74.37
78.41
74.96
78.56
74.26
78.85
hepatitis
69.36
83.03
69.74
82.84
71.29
83.03
70.26
82.84
69.29
83.03
67.55
83.94
horse-colic
72.31
77.69
72.69
72.39
77.72
72.28
77.69
72.94
77.34
hypothyroid
83.18
88.79
79.58
88.72
83.12
88.79
82.33
88.79
83.18
88,80
88.05
89.02
ionosphere
72.59
83.90
73.85
83.88
73.62
83.82
73.59
83.96
72.56
83.96
75.13
85.02
iris
72.00
86.87
72.27
87.13
71.87
87.13
71.80
87.07
72.27
86.67
75.33
90.93
kr-vs-kp
76.04
85.33
76.08
85.32
76.06
85,32
76.02
85.32
75.98
85.32
78.10
85.61
labor
66.14
80.70
50.35
81.75
letter
71.18
90.58
72.75
91.20
73.32
91.19
71.20
90.57
71.24
90.57
78.99
91.32
lymph
69.26
84.80
69.19
84.80
69.53
84.80
69.66
84.80
69.19
84.80
70.14
mushroom
76.66
88.02
76.67
88.02
76.67
88.02
76.70
88.02
88.37
sogment
69.64
87.63
69.33
87.51
69.65
87.53
69.67
87.64
69.58
87.61
75.80
88.46
sick
81.26
77.50
84,30
81.20
84.54
80.50
81.27
84.58
84.93
85.12
sonar
71.54
78.56
72.40
79.04
72.07
78.94
72.36
78.80
71.54
78.85
69.76
79.42
spumbuse
78.36
83.40
78.28
83.39
78.36
83,39
78.33
83.40
78.36
83.40
80.12
83.64
tic-tac-toe
76.75
77.83
75.96
77.80
76.70
76.54
76.86
77.83
78.66
76.75
vehicle
72.67
82.45
72.71
82.37
72.71
82.38
72.72
82,46
72.73
82.42
76.76
82.35
vote
76.57
83.10
76.39
83.11
76.50
83.11
76.46
83,08
76.69
83.11
78.80
84.09
vowel
73.49
91.53
73.48
91.53
73.47
91.53
73.48
91.53
73.55
91.53
78.23
91.79
waveform
74.53
84.84
74.01
84.51
74.45
84.73
74.49
84.84
74.55
84,86
79.05
85.73
200
Mean
74.56
83.51
74.08
83.57
74.57
83.54
74.43
83.45
74.55
83.54
76.80
83.99
W/1M
30/4/0
30/4/0
30/4/D
30/4/0
30//00
27/6/1
unchanged, and the results are shown in Table 3. Specifi- cally, Table 3 shows the integration accuracies of six label integration algorithms after label completion by WSLC and LDPLCC C the Gaussian distribution and the t-test results. From these results, proves the performance of all label integration algorithms compared to WSLC. These results demonstrate that LDPLC not sensitive to annotation quality distributions.
81.5
still see that LDPLC further im-
80
78
79.0
Besides WSLC, PMF and PMF-TLC have also been pro- posed for label completion, but they can only handle binary- class crowdsourcing tasks. To further evaluate the effec- tiveness of LDPLC, we conduct additional experiments to compare the performance of PMF, PMT-TLC and LDPLC on all binary datasets in Table 1. Detailed experiments and results are provided in Appendix B. These results also demonstrate the effectiveness LDPLC.
78.5
78.0
12
12
10
Figure 4. Integration accuracy (%) comparisons MV after label completion by LDPLC on the "LabelMe" dataset when K and T vary from to 15.
4.4. Parameter Sensitivity Analysis of LDPLC
In LDPLC, we can adjust two parameters: K and T. and denote the number of neighbors and the number
of iterations in the label distribution propagation process, respectively. Now, we analyze the impact of these two
8

---

Label Distribution Propagation-basee Label Completion for Crowdsourcing
Table Integration accuracy (%) comparisons of six label integration algorithms after label completion by WSL. C and I.PLCC the Gaussian distribution, respectively.
MV
DS
IWMV
GTIC
DEWSMV
MNLDP
Dataset
WSLC
LDPLC
WSLC
LDPLC
WSLC
LDPLC
WSLC
LDPLC
WSLC
LDPLC
WSLC
LDPLC
anneal
68.30
84.37
65.69
84.16
68.35
84.32
67.59
84.21
68.23
84.36
71.35
84.26
audiology
68.54
81.33
68.41
81.33
68.54
81.33
68.54
79.42
68.63
81.28
69.69
80.44
autos
69.42
81.76
81.76
81.76
71.22
bulance-scale
73.81
83.02
71.31
82.24
73.58
82.82
73.01
82.90
73.73
82.99
78.64
83.74
biodeg
78.40
81.65
78.11
81.56
78.37
81.59
78.26
81.64
78.47
81.66
80.26
81.83
breast-cancer
77.45
77.55
76.61
77.52
77.20
77.52
77.03
77.55
77.31
77.52
79.37
77.59
breast-w
80.04
86.28
79.89
86.15
80.00
86.22
79.88
86.28
80.10
86.32
83.55
89.07
79.19
credit-a
74.81
80.19
75.41
80.16
75.03
80.22
75.04
80.23
74.83
80,20
76.29
80.67
credit-g
74.48
74.70
73.76
74.63
74.45
74.72
74.43
74.67
74.51
74.70
13
74.15
diabetes
76.63
77.88
75.37
77.83
76.06
77.75
76.58
77.79
78.11
77.92
heart-c
76.24
84.29
75.25
84.29
75.98
84.29
74.85
84.29
76.17
84,29
80.79
84.49
heart-h
76.12
82.99
74.76
82.99
75.95
82,99
74,59
82.99
76,05
83.02
80.61
83.30
heart-statlog
73.63
78.00
73.70
77.85
73.78
77.82
73.56
78.00
73.67
78.00
74.11
77.78
hepatitis
65.68
79.29
66.58
79.61
67.42
79.74
66.90
79.23
65.68
79.22
61.93
79.93
horse-colic
71.66
78.12
72.06
77.93
71.79
78.04
71.66
78.02
72.15
77.47
hypothyroid
83.16
88.49
79.87
88.36
83.13
88.48
82.48
88.47
83.15
88.48
87.74
88.86
ionosphere
68.20
77.95
70.09
78.52
70.20
78.32
70.17
77.89
68.32
77.92
69.54
79.03
iris
71.07
87.13
71.40
87.33
72.40
86.87
72.87
87.67
70.60
87.07
75.53
91.47
kr-vs-kp
75.78
84.25
75.77
84.24
75.80
84.25
75.75
84.23
75.81
84.23
77.50
84.62
labor
60.88
79.30
78.95
39.65
78.42
letter
71.44
90.74
67.67
88.55
71887
90.64
71.42
90.74
71.43
90.73
79.52
91.53
lymph
69.05
83.51
69.05
83.51
69.39
83,45
69.32
83.45
69.12
83.58
71.28
83.85
mushroom
76.96
87.85
76.97
87.85
87.85
87.85
87.85
79.04
88.20
sogment
70.75
87.53
70.75
87.50
70.75
87448
70.74
87.55
70.72
87.53
77.20
88.59
sick
79.42
82.33
75.47
81.96
82.36
78.50
82,26
79,36
82,32
82.93
82.99
sonar
69.81
78.99
70.15
79.62
69.62
79.52
70.14
79.18
69.66
79.04
67.89
78.94
spumbuse
76.87
81.63
76.83
81.63
76.87
81.62
76.85
81.64
76.88
81.64
78.43
82.01
tic-tac-toe
76.36
77.47
75.27
77.47
76.36
76.09
77.50
76.34
77.46
78.53
76.65
vehicle
72.78
81,97
72.86
81.97
72.86
81.93
72.94
81.98
72,80
82,08
76.50
82.17
vote
77.61
84.99
77.49
84.99
77.68
85.01
77.68
84.99
77.75
84.99
79,91
86.35
vowel
71.18
89.94
71.22
89.94
71.23
89.94
71.20
89.94
71.28
89.94
75.86
90.17
waveform
72.65
83.36
72.22
82.98
72.51
83.25
72.59
83.32
72.56
83.35
77.52
84.31
200
79.80
Mean
73.61
82.72
73.04
82.63
73.81
82,73
73.68
82.66
73.59
82.70
75.66
83.13
W/1M1
28/6/0
29/5/D
28/6/0
29/5/0)
28/6/0
25/8/1
parameters on MV after label completion by LDPLC on the "LabelMe" dataset. We change both K and T from 15 and set the step size 1. Figure shows integration accuracies of MV after label completion by LDPLCC CC when K and T' vary. We can that as K varies from 2 to 15, the integration accuracies of all label integration algorithms vary within one percentage point. This indicates that LDPLC is rarely sensitive to K. Considering that K takes values in the interval [3, 6] when LDPLC achieves the best performance, we set K to in this paper. As T varies, the integration accuracy increases rapidly at first in the interval [1, 4) and then stabilizes after 4, which is consistent with the analysis in subsection 3.5. Therefore, we set T to 5 in this paper.
among similar instances. To address this issue, we proposed novel label distribution propagation-bes based label comple- tion (LDPLC) algorithm. In LDPLC, we first use worker similarity weighted majority voting initialize label dis- tribution for each missing label. Then, design label distribution propagation algorithm to enable each missing la- bel of each instance to iteratively absorb its neighbors' label distributions. Finally, we complete each missing label based on converged label distribution. Extensive experiments demonstrate that LDPLC further improves the performance of label integration algorithms compared WSLC.
Nevertheless, there are still some limitations in LDPLC. For example, in crowdsoureing scenarios with high noise ratios, the quality of the labels tends to be low, which results in inaccurately initializing the label distributions. These inaccurately label distributions enlarge the propagation of erroneous information during the label distribution propa- gation process. Therefore, how further extend LDPLC to the crowdsourcing scenarios with high noise ratios is the main direction of our future work.
5. Conclusion
WSLC has been proven to be an effective algorithm for la- bel completion. this paper, find that WSLC considers solely the correlation of the labels annotated by different workers per individual instance while totally ignoring the correlation of the labels annotated by different workers

---

Label Distribution Propagation- Label Completion for Crowdsourcing
Acknowledgments
IEEE Trans. Neural Networks Learn. Syst., 33(11): 6568, 2022.
The work was partially supported by National Natural Sci- ence Foundation of China (62276241) and Hubei Provincial Collaborative Innovation Center for Basic Education Infor- mation Technology Services (OFHUE202312).
Jung, H. and Lease, M. Improving quality of crowd- sourced labels via probabilistic matrix factorization. In Chen, Y., Ipeirotis, P. G., Law, E.,, von Ahn, L.,, and Zhang, H. (eds.), The 4th Human Compulation Workshop, HCOMP@ AAAI 2012. Toronto, Ontario, Canada, July 23, 2012, volume WS-12-08 of AAAI Technical Report AAAI Press, 2012.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Li, H. and Yu, B. Error rate bounds and iterative weighted majority voting for crowdsourcing. CoRR, abs/1411 abs/1411.4086, 2014.
References
Li, S., Huang, S., and Chen, S. Crowdsourcing aggregation with deep bayesian learning. Sci. China Inf. Sci., 64(3), 2021.
Chen, Z., Jiang, L., and Li, C. Label augmented and weighted majority voting for crowdsourcing. Inf. Sci., 606:397-409, 2022.
Li, Z. and Tang, J. Weakly supervised deep matrix factoriza- tion for social image understanding. IEEE Trans. Image Prucess. 26( (1 278- 2017.
Dawid, A. and Skene, A. M. Maximum likelihood es- timation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20 28,2000211111 1979.
Liang, L., Fu, J., Zhu, H., and Liu, D. Solving the team allocation problem in crowdsourcing via group multirole assignment. IEEE Trans. Comput. Soc. Syst., 10(3):833- 854, 2023.
Demartini, G., Difallah, D. E.,, and Cudré- Mauroux, P. Zen- crowd: leveraging probabilistic reasoning and crowd- sourcing techniques for large-scale entity linking. In Mille, A., Gandon, F., Misselis, J., Rabinovich, M., and Staab, S. (eds.), Proceedings of the 21 World Wide Web Conference 2012, WWW 2012, Lyon, France, April 16-20, 2012. pp. 469-478. ACM, 2012.
Mak, S. H. and Lam, A. S. Two-stage auction mecha- nism for long-term participation in crowdsourcing. IEEE Trans. Compul. Soc. Syst., 10(3):855-866 2023.
Miao, J., Yang, T., Sun, L., Fei, X., Niu, L., and Shi, Y. Graph regularized locally linear embedding for unsuper- vised feature selection. Pattern Recognit., 122:108299, 2022.
Ghojogh, B., Ghodsi, A., Karray, F., and Crowley, M. The- oretical connection between locally linear embedding, factor analysis, and probabilistic PCA. In Kiringa, I. and Gambs, S. (eds.), 35th Canadian Conference on Artificial Intelligence, Toronto, Ontario, Canada, May 30 - June 3, 2022. Canadian Artificial Intelligence Association, 2022.
Nadeau, C. and Bengio, Y. Inference for the generalization error. Mach. Learn., 52(3):233--2) 281, 2003.
Sheng, V., Provost, J., and Ipeirotis, P. G. Get another label? improving data quality and data mining using mul- tiple. noisy labelers. In Li, Y., Liu, B., and Sarawagi, S. (eds.), Proceedings of the 14th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008 pp. 614-622, ACM, 2008
Hall, M. A. Correlation- feature selection for discrete and numeric class machine learning. In Langley, P. (ed.), Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000, pp. 359-366. Morgan Kaufmann, 2000.
Tao, F., Jiang, L., and Li, C. Differential evolution-based weighted soft majority voting for crowdsourcing. Eng. Appl. Artif. Intell., 106: 1044744, 2021.
Ji, Q. Jiang, L., and Zhang. W. Dual-view noise correc- tion for crowdsourcing. IEEE Internet Things J., 10(13): 11804-11812, 2023.
Tu, J., Yu, G., Domeniconi, C., Wang, J., and Zhang, X. Ac- tive multi-label crowd consensus. CoRR, abs/1911.02788 2019.
Jang L., Zhang, L., Yu, L., and Wang. D. Class-specific attribute weighted naive bayes. Pattern Recognit., 88: 321-330, 2019.
Witten, I. H., Frank, E., and Hall, M. A. Data mining: practical machine learning tools and techniques, 3rd Edition. Morgan Kaufmann, Elsevier, 2011.
Jiang, L.. Zhang, H., Tao, F., and Li, C. Learning from crowds with multiple noisy label distribution propagation.
10

---

Label Distribution Propagation-basee Label Completion for Crowdsourcing
Wu, X., Jiang, L.,, Zhang, W., and Li, C. whree--ay decision- based noise correction for crowdsoureing. Int. Approx. Reason., 160:108973, 2023.
Wu, X., Jiang, L.,, Zhang. W., and Li, C. Worker similarity- based label completion for crowdsoureing. IEEE Trans- actions on Big Data, 2024.
Xu, Y. Xiao, M., Liu, A., and Wu, J. Edge resource pre- diction and auction for distributed spatial crowdsourcing with differential privacy. IEEE Internet Things J., 9(17); 15554-15569, 2022.
Yang, B., Jiang, L., and Zhang, W. Probabilistic ma- trix factorization-based stage label completion for crowdsoureing. Proceedings of the 24th IEEE Interna- tional Conference on Data Mining, pp. 540-549, ICDM, 2024.
Zhang. J. Knowledge learning with crowdsourcing: A brief review and systematic perspective. IEEE CAA Autom. Sinica, 95)):49-772,, 2022.
Zhang. J., Sheng, V. S., Nicholson, B., and Wu, X. CEKA: tool for mining the wisdom of crowds. J. Mach. Learn. Res., 16:2853-2858 2015a.
Zhang, J., Wu, X., and Sheng, V. Imbalanced multiple noisy labeling. IEEE Trans. Knowl. Data Eng., 27(2): 489-503, 2015b.
Zhang, J., Sheng, S., Wu, J., and Wu, X. Multi-class ground truth inference in crowdsourcing with clustering. IEEE Trans. Knowl. Data Eng., 1884):0080-1085, 2016.
Zhang, W., Jiang, L., and Li, C. Eldp: Enhanced label distribution propagation for crowdsourcing. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024a.
Zhang, W., Jiang, L., and Li, C. IWBVT: Instance weighting- bias-variance trade-off for crowdsoure- ing. In The Thirtt-eighth Annual Conference Neural Information Processing Systems, 2024b.
Zhang, W., Jiang, I.., and Li, C. KFNN: K-free nearest neighbor for crowdsoureing, In The Tiirty-eighth Annual Conference on Neurul Information Processing Systems, 2024c.
Zhang, Y., Jiang, L.., and Li, C. Attribute augmentation- based label integration for crowdsourcing. Frontiers Com- put. Sci., 17(5): 2023.
Zheng, Y., Li, G., Li, Y., Shan, C., and Cheng, Truth inference in crowdsourcing: Is the problem solved? Proc. VLDB Endow, 10(5):541-552, 2017.

---

Label Distribution Propagation-basee Label Completion for Crowdsourcing
A. Experiments on the "Ruters" and "Leaves" datasets.
The dataset "Ruters" contains 1799 instances and classes. In total, there are 5410 multiple noisy labels from 38 workers, with a mean annotation quality of 59.59% and mean annotation ratio of 7.91%. The dataset "Leaves" contains 384 instances and classes. In total, there are 3840 multiple noisy labels from 83 workers, with a mean annotation quality of 55.03% and mean annotation ratio of 12.05%. The integration accuracies of six label integration algorithms after label completion by WSLC and LDPLC on the "Ruters" and "Leaves" datasets are shown in Figure 5. From these results, can conclude that LDPLC further improves the performance of all label integration algorithms compared to WSLC.
75-
WOLE
WSLU
LDPLC
LDPIC
74.02
29.54
66
05.30
65.36
75
64-
70
85
60-
55
56-
50
MNLDP
(a) Ruters
(b) Leaves
Figure Integration accuracy (%) comparisons of six label integration algorithms after label completion by WSLC and LDPLC, respectively.
B. Experiments on 18 simulated binary datasets.
We construct sets of experiments 18 simulated binary datasets. The first set of experiments compares the performance of PMF and LDPLC, and the second set of experiments compares the performance of PMF-TLC and LDPLC. In each set of experiments, we simulate 40 crowd workers. For each simulated crowd worker, we randomly sample its annotation ratio and annotation quality from the uniform distribution [0, 0.1] and [0.6, 0.9], respectively. The experimental results are shown in Table 4 and Table From the results, we can conclude that LDPLC further improves the performance of most label integration algorithms compared PMF and PMF-TLC on binary datasets.
Table Integration accuracy (%) comparisons of label integration algorithms after label completion by PMF and LDPLC on 18 simulated binary datasets, respectively.
MV
DS
IWMV
GTIC
DEWSMV
MNLDP
Datusct
PMF
LDPLC
PMF
LDPLC
PMF
LDPLC
PMF
LDPLC
PMF
LDPLC
PMF
LDPLC
biodeg
72.48
81.55
74.60
81.53
72.35
81.53
75.77
81.56
74.83
81.53
77.54
81.76
breasf-cancer
38.15
41.96
36.96
41.53
80.73
38.01
80.70
34.34
79.79
breast-w
49.40
84
48.27
85.04
43.62
84
62.43
85
49.03
85.21
50.66
87.39
credit-a
58.71
82.74
58.83
82.78
58.55
82.75
63.50
82.72
58.70
82.69
57.55
82.74
credil-g
70.62
77.28
72.77
77.26
69.92
77.31
75.96
77.24
73.21
77.27
71.75
76.99
diabetes
49.86
78.18
50.56
78.20
46.38
78.22
68.43
78.11
49.88
78.18
49.27
78.20
heart-statlog
48.34
49.89
47.85
80.
47.85
80.
48.26
46.67
79.18
hepatitis
77
82.52
76.84
82,45
77.35
82.32
76.12
82.32
77.48
82.71
78.84
83.68
horsc-colic
63.53
79
63.37
79.08
63.64
79.08
64.89
79.05
63.56
79.08
63.56
78.75
onosphere
65.98
83.13
65.33
82.79
65.73
82.96
64.84
83.13
65.93
82.96
65.30
84.33
kr-ys-kp
76.93
86.50
77.06
86.49
76.81
86.49
79.22
86.50
77.14
86.51
84.91
86.72
labor
66.
80.17
66.32
81.58
66.32
81 93
65.26
79 30
65.96
80.88
66.84
83.68
mushroom
76.46
87 90
76.68
87.91
76.37
87.91
18.92
87.91
76.71
87.11
88.44
88
sick
78.26
86.5
80.40
86.22
78.13
86.52
73.66
86.47
82.68
86.53
86.55
87.25
56.59
25
$6.73
81.73
56.40
81
58.65
81444
$6.64
81.30
55.91
81.54
spamhase
76.55
83.88
83.88
76.59
79.58
83.88
77.77
83,88
83.72
84.16
trc-tac-toe
75.48
78.55
77.37
78.57
75.19
78.56
79.47
78.57
77.43
78.53
77.91
77.77
vote
43.63
86.16
44,90
86.12
43.01
86.12
45.7
86.16
43.63
86.14
41.82
86.83
Mean
63.59
82,31
64.42
82,36
62.84
82,39
66.77
82.24
64.27
82.35
65.64
82.72
W/T/L
17/1/0
16/2/0
17/1/0
16/2/0
16/2/0
13/5/0
12

---

Label Distribution Propagation- Label Completion for Crowdsourcing
Table Integration accuracy (%) comparisons of six label integration algorithms after label completion by PMP-TIC and LDPI.C 18
simulated binary datasets, respectively.
MV
DS
IWMV
GTIC
DEWSMV
MNLDP
Dataset
PMF-TLC
LDPLC
PMF-TLC
LDPLC
PMF-TLC
LDPLC
PMF-TLC
LDPLC
PMF-TLC
LDPLC
PMF-TLC
LDPLC
biodeg
74.38
81.55
75.82
81.53
75.21
81.53
78.38
81.56
77.45
81.53
83.08
81.76
brcast-cancer
80.66
70.77
80.80
77.38
80.63
79.79
80.73
78.74
80.70
78.32
79.79
breast-w
87
credit-a
76.09
82.74
57.70
82.78
$2.75
79.91
82.72
75.35
82.69
$2.74
credit-g
73.19
77.28
75.56
77.26
73.85
77.31
79.00
77.24
76.46
77.27
76.72
76.99
diaheles
73.58
78. 18.18
69.19
78
74.43
78
80.33
71
75.44
78.18
78.67
78.20
beart-station
73.63
80.30
58.81
80.11
74.74
80.11
75.77
80.11
74.85
80.26
80.78
79.18
hepatitis
76.84
K2.52
80.71
82.45
75
$2.32
73.16
82.32
82.71
85
83.68
borse-colic
72.42
79.10
79.08
72.47
78.09
79.05
71.22
79.08
78.75
oonssphere
74.84
K3.13
66.38
82.79
82.96
76.23
83.13
73.65
82.96
84.62
kr-vs-kp
80.47
86.50
80.86
86.49
80.45
86.49
83.08
86.50
80.74
86.51
88.92
86,72
labor
77.37
80.17
55.09
81.58
76.14
8193
74.03
79.30
76.32
80.88
84.74
83.68
mushroom
81.27
87.90
81.54
87.91
81.33
87.91
83.89
81.48
87.91
92.08
88 22
83.24
81.99
91.45
87.23
sonar
73.41
81.25
78.7
spambase
78.19
83,88
79.48
83.88
78.00
83.88
80.65
83,88
79.29
83.88
85.57
84.16
ti-ttactoe
74.92
78.55
75.90
78.57
75.88
78.56
19.72
78.57
77.28
78.53
72.89
71.77
vote
78.23
86.16
63.61
86.12
77.81
86.12
84.18
86.16
79.22
86.14
91.40
86.83 c
Mcan
76.41
82.31
70.27
82.36
76.70
82.39
79.45
82.24
77.34
82.35
84.15
82.72
W/T/
17/1/0
14/3/
18/0/
13/4/1
13/5/0
1/11/6
13


Reviews:
[{"review_id": "Wi7q1rB9M6", "reviewer": "Reviewer_qna3", "summary": "This paper proposes a crowdsourcing label completion method to complement subsequent truth-inference/label-integration methods. \nThe proposed method primarily focuses on improving the existing method WSLC, which “considers solely the correlation of the labels annotated by different workers on per individual instance while totally ignoring the correlation of the labels annotated by different workers among similar instances”.\n\n## update after rebuttal\n\nThanks for the feedback from the authors. I have raised my scores accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your comments. Please find our detailed responses to your concerns as follows.\n\n**Author Response to Research Motivation and Research Problem:** Label integration (truth inference) is indeed a long-standing research problem. Over the past decades, numerous algorithms have been proposed to improve the performance of label integration. These algorithms have gradually reached a consensus: when worker annotation is more accurate than random annotation, the more noisy labels an instance receives, the easier it becomes to infer its unknown true label. However, in real-world scenarios, each worker typically annotates only a small number of instances, and few labels are typically collected per instance to reduce cost, resulting in a highly sparse crowdsourced label matrix. This fact leads to label integration failing to achieve the expected performance relying solely on the existing labels in the label matrix. To address this issue, label completion has been proposed to fill in missing labels in sparse label matrices and is gaining increasing attention. To the best of our knowledge, WSLC is currently the most advanced and convincing benchmark for label completion. However, WSLC solely considers the correlation of the labels annotated by different workers on per individual instance. Our proposed algorithm, LDPLC, is the first to jointly model both worker similarity and instance similarity. Specifically, LDPLC first estimates worker similarity by learning feature vectors, enabling a more accurate initialization of label distributions. Next, it finds neighbors and employs locally linear embedding (LLE) to optimize their weights, capturing the geometric structure among instances. Finally, LDPLC propagates label distributions across instances, allowing missing labels to absorb information from similar instances, thereby overcoming the limitations of WSLC.\n\n**Author Response to Experimental Designs:** We have carefully considered these issues and conducted additional experiments to support our work. Below are the updates we make: (1) In addition to “LabelMe”, we have now included two real-world crowdsourced datasets, “Ruters” and “Leaves”.“Ruters” and “Leaves” are also collected from the AMT platform. “Ruters” contains 1799 instances, 5410 labels, 8 classes, and 38 workers. “Leaves” contains 384 instances, 3840 labels, 6 classes, and 83 workers. (2) We have added the DS, KOS, and IWMV algorithms to the experiments to provide a more comprehensive comparative analysis. The experimental results are as follows:\n\n|Dataset|MV_WSLC|MV_LDPLC|GTIC_WSLC|GTIC_LDPLC|DEWSMV_WSLC|DEWSMV_LDPLC|MNLDP_WSLC|MNLDP_LDPLC|DS_WSLC|DS_LDPLC|KOS_WSLC|KOS_LDPLC|IWMV_WSLC|IWMV_LDPLC|\n|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|\n|LabelMe|76.40%|**81.70%**|76.40%|**81.70%**|76.60%|**81.60%**|80.50%|**82.50%**|76.70%|**81.50%**|76.30%|**81.70%**|76.30%|**81.70%**|\n|Ruters|70.37%|**79.32%**|70.48%|**79.32%**|70.37%|**79.32%**|73.49%|**79.54%**|70.43%|**79.32%**|70.37%|**79.32%**|70.48%|**79.32%**|\n|Leaves|63.80%|**65.10%**|63.80%|**65.10%**|63.80%|**65.10%**|64.58%|**65.36%**|63.80%|**65.10%**|64.06%|**65.10%**|63.80%|**65.10%**|\n||\n\nFrom these results, it can be found that the integration accuracies of both classic algorithms (MV, DS, KOS) and the lastest algorithms (GTIC, IWMV, DEWSMV, MNLDP) improve significantly after label completion using LDPLC compared to WSLC across all three datasets. These findings confirm that LDPLC approximates crowds more accurately than WSLC and further improves the performance of all label integration algorithms. These results once again verify the effectiveness and robustness of LDPLC. In the final version of the paper, we will further extend the experiment and analysis on real-world datasets. \n\n**Author Response to Essential References:** By studying this reference, we have gained a clearer understanding of the differences between various classic algorithms and the future directions of crowdsourcing research, which is highly beneficial for our future work. In the final version of the paper, we will cite [1] and discuss its relationship with our work.", "claims_and_evidence": "Regarding this point, my main concern lies in the experiments. Please refer to the reviews on “Experimental Designs Or Analyses”.", "methods_and_evaluation_criteria": "Regarding this point, my main concern lies in the experiments. Please refer to the reviews on “Experimental Designs Or Analyses”.", "theoretical_claims": "Yes, I have reviewed the methodology part of this paper.", "experimental_designs_or_analyses": "The experiment section has the following mian issues:\n1) Only one real-world dataset, LabelMe, is used. In fact, there are many real-world crowdsourcing annotation datasets available, e.g., [1, 2].\n2) The truth inference methods considered lack the classic DS method and other methods. It is recommended to include the DS method and the methods used in benchmark [1].\n3) The experimental analysis is insufficient. In summary, for experiments on real-world datasets, all results are only involes Figure 3.\n\n[1] Truth inference in crowdsourcing: Is the problem solved? VLDB 2017.\n[2] Deep learning from crowds. AAAI 2018.", "relation_to_broader_scientific_literature": "The key contributions of the paper are important for research on supervised learning.", "essential_references_not_discussed": "It is recommended to cite Reference [1].\n[1] Truth inference in crowdsourcing: Is the problem solved? VLDB 2017.", "other_strengths_and_weaknesses": "Strengths:\n1)\tFirst, the overall narrative of the paper is clear and easy to follow, with logical flow (especially in the introduction section). The paper has not excessive embellishment.\n2)\tIn the experiment section, particularly on the real-world dataset LabelMe, the proposed method outperforms the optimized method WSLC.\n\nWeaknesses\n1）\tRegarding the research motivation and the research problem.\nThe research motivation and the research problem are clearly introduced, but the significance of the addressed problem is not sufficiently compelling.\nThis is because the paper primarily focuses on improving the existing method WSLC to eliminate its limitation—it “considers solely the correlation of the labels annotated by different workers on per individual instance while totally ignoring the correlation of the labels annotated by different workers among similar instances”.\nFurthermore, the proposed method aims to enhance the accuracy of subsequent truth inference processes.\nHowever, the problem of truth inference for crowdsourced annotations is a long-standing issue, with mature benchmark datasets and review papers already available since 2017, e.g., [1].\nAs a long-standing research problem, publishing high-level papers on truth inference requires higher standards and the resolution of sufficiently significant pain points.\n\n2）\tRegarding the experiments.\n\n Please refer to the above reviews on “Experimental Designs Or Analyses”.\n\n\n[1] Truth inference in crowdsourcing: Is the problem solved? VLDB 2017.", "other_comments_or_suggestions": "none.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x1ba5aFI0v", "reviewer": "Reviewer_8VZy", "summary": "1. The paper primarily addresses the shortcomings in WSLC, which traditionally considers only the correlations among labels annotated by different workers for individual instances. The authors propose the LDPLC algorithm, which additionally accounts for correlations among labels annotated by different workers across similar instances.\n2. Overall, the reported results demonstrate superior performance, consistently outperforming most existing methods on both real-world and simulated datasets.\n3. The robustness of the proposed algorithm is well-supported by thorough theoretical analysis and comprehensive parameter sensitivity analysis.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your comments. Please find our detailed responses to your concerns as follows.\n\n**Author Response to Convergence Efficiency:** As shown in Figure 4, LDPLC converges after just 4 iterations on the “LabelMe” dataset. To address the reviewer's concerns, we further observe the convergence efficiency of LDPLC on the first five simulated datasets. We observe that integration accuracies (%) of MV after label completion by LDPLC when T varies as follows:\n\n|Dataset\\T|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|\n|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|\n|anneal|89.64|94.65|94.99|95.66|95.55|95.55|95.55|95.55|95.55|95.55|95.55|95.55|95.55|95.55|95.55|\n|audiology|89.38|93.36|94.69|94.25|94.69|94.25|94.69|94.25|94.25|94.25|94.25|94.25|94.25|94.25|94.25|\n|autos|88.29|84.88|90.73|90.73|90.73|91.22|91.22|91.22|91.22|91.22|91.22|91.22|91.22|91.22|91.22|\n|balance-scale|76.80|87.20|86.72|87.52|87.04|87.36|87.68|87.36|87.36|87.36|87.36|87.52|87.20|87.52|87.20|\n|biodeg|70.14|72.42|72.80|72.32|71.94|72.23|72.32|72.13|72.13|72.23|71.85|71.85|72.13|72.13|72.23|\n||\n\nThese results confirm that LDPLC can achieve convergence within 10 iterations. In the final version of the paper, we will add the convergence efficiency analysis of LDPLC.\n\n**Author Response to Extension:** To the best of our knowledge, local linear embedding (LLE) is a nonlinear spectral dimensionality reduction and manifold learning method [1]. Because for most datasets, the assumption of local linear correlation in LLE is satisfied. This is why LDPLC performs well on most datasets. However, if the local space is particularly complex, the assumption of LLE may not be satisfied. Therefore, to handle more complex nonlinear problems, we think LDPLC can be extended to more complex scenarios, such as kernel-based methods or deep neural networks. By leveraging kernel methods, such as Kernel LLE, the nonlinear feature information can be extracted when mapping input data into some high dimensional feature space. Moreover, deep neural networks can map a nonlinear feature space to a linear feature space. In the new feature space, we can better find neighbors and optimize their weights. These will become important directions for our future work. \n\n[1] Theoretical Connection between Locally Linear Embedding, Factor Analysis, and Probabilistic PCA;\n\n**Author Response to Computational Complexity:** Below, we provide a detailed analysis of LDPLC’s computational complexity. In Algorithm 1, lines 3-8 learn feature vectors with a time complexity of $O(R(N+M(n_ln_a|D_r|)))$, where $n_l$ and $n_a$ are the average number of values for a label variable and an original feature variable, respectively. Lines 9-13 estimate worker similarity with a time complexity of $O(R^2M)$. Lines 14-22 initialize label distributions with a time complexity of $O(NRQ)$. Lines 23-26 identify neighbors and optimize their weights with a time complexity of $O(N^2M+NK^3)$. Lines 27-29 propagate the label distributions with a time complexity of $O(TNKQ)$. Finally, lines 30-36 complete missing labels with a time complexity of $O(NRQ)$. If only the highest order terms are taken, the overall time complexity of LDPLC is $O(RM(n_ln_a|D_r|)+R^2M+NRQ+N^2M+NK^3+TNKQ)$. In addition, we compare the run time of the WSLC and LDPLC, as follows:\n|Dataset|WSLC|LDPLC|\n|--|--|--|\n|LabelMe|2.48s|12.40s|\n||\n\nThe experimental results show that LDPLC takes more time than WSLC. Although LDPLC slightly increases the run time, it has better performance. In the final version of the paper, we will add the computational complexity of LDPLC.\n\n**Author Response to A Large Number of Categories:** Among 34 simulated datasets, “audiology” and “letter” contain 24 and 26 categories, respectively. In addition, to further address the reviewer’s concerns, we simulate a dataset with 10000 instances and 100 categories, named “Simulation_100”, to conduct a new experiment. The results are as follows:\n|Dataset|MV_WSLC|MV_LDPLC|GTIC_WSLC|GTIC_LDPLC|DEWSMV_WSLC|DEWSMV_LDPLC|MNLDP_WSLC|MNLDP_LDPLC|\n|--|--|--|--|--|--|--|--|--|\n|Simulation_100|90.50%|**92.70%**|90.50%|**92.66%**|90.52%|**92.64%**|91.56%|**92.24%**|\n||\n\nFrom these results, it can be found that the integration accuracy of each integration algorithm improves significantly after label completion using LDPLC compared to WSLC. These results further verify that LDPLC can perform equally well in scenarios involving a large number of categories. In the final version of the paper, we will include more detailed descriptions of dataset characteristics. \n\n**Author Response to Essential References:** In the final version of the paper, we will add a discussion of the relevant literature related to manifold consistency. \n\n**Author Response to Figures 1 and 3:** Indeed, the text in Figures 1 and 3 is too small, reducing readability. In the final version of the paper, we will enlarge the font size and optimize the layout of the figures and tables.", "claims_and_evidence": "The method proposed in the paper was verified in the experimental stage.", "methods_and_evaluation_criteria": "The choice of comparison methods, evaluation criteria, and datasets is appropriate and well-justified.", "theoretical_claims": "I am not an expert in this field, but upon reviewing the theoretical claims briefly, I found no obvious errors. \n\nOne minor limitation is the absence of a consistency analysis for other architectures (e.g., kernel methods or deep neural networks).", "experimental_designs_or_analyses": "1. The experimental design is comprehensive, including extensive comparisons across multiple datasets and thorough parameter analysis.\n2. A potential limitation is the absence of detailed descriptions of dataset characteristics. Therefore, it remains unclear whether the proposed method can perform equally well in scenarios involving a large number of categories.", "relation_to_broader_scientific_literature": "This paper integrates manifold consistency with crowdsourcing learning.", "essential_references_not_discussed": "The core contribution of this paper is its consideration that different workers with similar cognitive abilities tend to annotate similar labels for the same instance, and that the same worker tends to annotate similar labels across similar instances. In other words, the authors address manifold consistency among workers and labels, as well as among instances and labels. Therefore, I suggest that the paper should include discussions on relevant literature related to manifold consistency.", "other_strengths_and_weaknesses": "Strengths:\n(1) The paper considers multiple manifold consistency issues within the crowdsourcing learning process and employs local linear embedding for Label Distribution Propagation.\n(2) Experiments conducted across various datasets demonstrate the feasibility and effectiveness of the proposed method.\n(3) The paper also provides a convergence analysis.\n\nWeaknesses:\n(1) The current work is limited to linear scenarios. It would be valuable to discuss whether this approach can be extended to kernel-based methods or deep neural networks.\n(2) The authors have not discussed the computational complexity of the algorithm. Specifically, computing neighbors in feature space can be time-consuming, which raises my concerns about practical applicability to real-world scenarios.", "other_comments_or_suggestions": "The text in Figures 1 and 3 is too small, reducing readability. I suggest enlarging the font size or redesigning these figures to enhance clarity.", "questions_for_authors": "1. In practical experiments (rather than theoretical analysis alone), what is the convergence efficiency of this algorithm?\n2. Can the proposed approach be extended to more complex scenarios, such as kernel-based methods or deep neural networks? This is particularly relevant because, in practice, feature data often do not satisfy linear assumptions.\n3. Could the authors discuss the computational complexity of the proposed algorithm?\n\nIf the authors address these questions clearly, I am willing to raise my score.\n\n\n## update after rebuttal\nThe author's response has partially addressed my concerns, and after reviewing the other reviewers' comments, I think the current article should be supplemented with a kernel-based extension method to enrich the completeness of the article, which is not difficult. Therefore, I maintain my original score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vQRoW5oWLI", "reviewer": "Reviewer_vQCd", "summary": "This paper proposes a novel label distribution propagation-based label completion (LDPLC) algorithm to address the sparsity issue in crowdsourced label matrices. Existing worker similarity-based label completion (WSLC) algorithm only considers the correlation of labels annotated by different workers on individual instances, ignoring the correlation of the labels annotated by different workers among similar instances. To fill this gap, LDPLC initializes label distributions using worker similarity weighted majority voting and then propagates these distributions iteratively to absorb information from neighboring instances. Finally, LDPLC completes each missing label based on converged label distributions. Experimental results on both real-world and simulated datasets validate the effectiveness of LDPLC.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The key motivation for proposing LDPLC in this paper lies in the limitation that WSLC does not take into account the correlation of the labels annotated by different workers among similar instances. However, why WSLC has this limitation and what consequences it may lead to are not discussed. To further clarify the motivation of this paper, these aspects are necessary to address.\n\n**Author Response to Q1:** Thanks for your valuable comments. The core assumption of WSLC is that workers with similar cognitive abilities will annotate similar labels on the same instances. Therefore, it completes each missing label solely based on the labels annotated by similar workers on this corresponding instance. However, in real-world crowdsourcing scenarios, each instance usually has few labels, thus WSLC struggles to complete its missing labels. By introducing the assumption that the same worker will also annotate similar labels on similar instances, the missing labels can not only be inferred from labels of similar workers on the same instance but also absorb the distribution information of labels from all workers across neighboring instances. In the final version of the paper, we will include these discussions on why WSLC has this limitation and what consequences it may lead to. Thanks again for your valuable comments.\n\n**Q2:** As shown in Tables 1 and 2, the proposed LDPLC performs significantly worse than WSLC only on the “tic-tac-toe” dataset when the label integration algorithm is MNLDP. Why does this happen? A detailed explanation is necessary.\n\n**Author Response to Q2:** Thanks for your valuable comments. Through an in-depth understanding of the “tic-tac-toe” dataset, we found that this dataset has a unique feature structure. This dataset encodes the complete set of possible board configurations at the end of tic-tac-toe games. Each instance contains 9 structured features, corresponding to the nine positions on a 3×3 game board, precisely recording the occupancy state of each cell: player X (marked as ‘x’), opponent O (‘o’), or unoccupied (‘b’). The different occupancy state of one cell on the board can directly affect the outcome of the game. However, when corresponding to instances, we consider them similar because only a small number of features differ, leading to incorrect information being propagated across these similar instance. Since MNLDP also has a propagation stage, it propagates more erroneous information, which leads to a decrease in integration accuracy. In the final version of the paper, we will include these discussions on why the proposed LDPLC performs significantly worse than WSLC only on the “tic-tac-toe” dataset when the label integration algorithm is MNLDP. Thanks again for your valuable comments.\n\n**Other Comments Or Suggestions:** On line 107 in page 2, the full name of the “EM” framework should be provided when it first appears.\n\n**Author Response:** Thanks for your valuable comments. In the final version of the paper, we will provide the full name of the EM, i.e., Expectation-Maximization. Thanks again.", "claims_and_evidence": "The most important claim made in the paper is that the proposed LDPLC can considers not only the correlation of the labels annotated by different workers on per individual instance, but also the correlation of the labels annotated by different workers among similar instances. By doing so, LDPLC can further improve the performance of label completion.\n\nTo support this claim, LDPLC initializes label distributions through worker similarity weighted majority voting, which utilizes the correlation of the labels annotated by different workers on per individual instance. Subsequently, LDPLC iteratively propagates these distributions to absorb information from neighboring instances, which utilizes the correlation of the labels annotated by different workers among similar instances. The results shown in Figure 3, Table 1, and Table 2 demonstrate that LDPLC can further improve the performance of label completion.", "methods_and_evaluation_criteria": "Yes. LDPLC utilizes the correlation of the labels annotated by different workers among similar instances through label distribution propagation, which effectively fills the gap left by WSLC. Regarding evaluation criteria, this paper adopts the integration accuracy, which is commonly used in other label completion studies.", "theoretical_claims": "Yes. The paper provides a detailed theoretical analysis of the convergence properties of LDPLC. I have verified the correctness of the theoretical analysis.", "experimental_designs_or_analyses": "Yes. The paper conduct extensive experiments on both real-world and simulated datasets, providing strong empirical evidence to support the effectiveness of LDPLC.", "relation_to_broader_scientific_literature": "This paper makes a contribution to the field of label completion for crowdsourcing by addressing a critical limitation of existing work. The proposed LDPLC utilizes the correlation of the labels annotated by different workers among similar instances through label distribution propagation, which effectively fills the gap left by existing work and thus further improve the performance of label completion.", "essential_references_not_discussed": "No. All essential references have been cited/discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n1) This paper reveals a critical limitation of existing work: WSLC considers solely the correlation of the labels annotated by different workers on per individual instance while totally ignoring the correlation of the labels annotated by different workers among similar instances.\n2) To fill this gap, this paper proposes a label distribution propagation-based label completion (LDPLC) algorithm. The proposed LDPLC utilizes the correlation of the labels annotated by different workers among similar instances through label distribution propagation. \n3) This paper provides a detailed theoretical analysis of the convergence properties of LDPLC (subsection 3.4), ensuring the algorithm's reliability.\n4) This paper provides extensive experiments on both real-world and simulated datasets, providing strong empirical evidence to support the effectiveness of LDPLC. The results show that LDPLC significantly outperforms WSLC in terms of label integration accuracy across multiple datasets and label integration algorithms.\n\nWeaknesses:\n1) The key motivation for proposing LDPLC in this paper lies in the limitation that WSLC does not take into account the correlation of the labels annotated by different workers among similar instances. However, why WSLC has this limitation and what consequences it may lead to are not discussed. To further clarify the motivation of this paper, these aspects are necessary to address.\n2) As shown in Tables 1 and 2, the proposed LDPLC performs significantly worse than WSLC only on the “tic-tac-toe” dataset when the label integration algorithm is MNLDP. Why does this happen? A detailed explanation is necessary.", "other_comments_or_suggestions": "I have found a small issue in writing as follows:\n1) On line 107 in page 2, the full name of the “EM” framework should be provided when it first appears.", "questions_for_authors": "See the Weaknesses parts:\n1) The key motivation for proposing LDPLC in this paper lies in the limitation that WSLC does not take into account the correlation of the labels annotated by different workers among similar instances. However, why WSLC has this limitation and what consequences it may lead to are not discussed. To further clarify the motivation of this paper, these aspects are necessary to address.\n2) As shown in Tables 1 and 2, the proposed LDPLC performs significantly worse than WSLC only on the “tic-tac-toe” dataset when the label integration algorithm is MNLDP. Why does this happen? A detailed explanation is necessary.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "62IPK6RLPy", "reviewer": "Reviewer_CJ1M", "summary": "To complete the missing labels, this paper proposes a novel label completion method for crowdsourcing by utilizing label distribution propagation. Both the worker similarity and the label correlation are considered to generate the label distribution for missing labels. Based on the worker similarity, the weighted majority voting is applied to obtain the initialized label distribution, and local linear embedding is adopted to finish the label distribution propagation. Experimental results and related analysis demonstrate the effectiveness of the proposal.\n\n## update after rebuttal\nThe author's response has addressed my concerns, and after reviewing the other reviewers' comments, I support the acceptance of this paper. Therefore, I maintain my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your comments. Please find our detailed responses to your concerns as follows.\n\n**Author Response to Q1:** We choose WSLC as the primary comparison method because WSLC is the most recent and relevant for our work. Moreover, both WSLC and our work are designed to address multi-class crowdsourcing tasks. In contrast, PMF and PMF-TLC are primarily designed for binary-class crowdsourcing tasks. To address the reviewer’s concerns, we conduct additional comparative experiments on four simulated datasets specifically designed for binary-class crowdsourcing tasks. The results are as follows:\n\n|Dataset|MV|GTIC|DEWSMV|MNLDP|\n|--|--|--|--|--|\n|biodeg (PMF)|73.18%|75.77%|76.30%|78.39%|  \n|biodeg (PMF-TLC)|74.22%|78.38%|77.73%|**80.56%**|  \n|biodeg (LDPLC)|**79.05%**|**81.36%**|**79.05%**|79.43%|\n|breast-w (PMF)|54.36%|62.43%|55.94%|51.36%|  \n|breast-w (PMF-TLC)|77.83%|83.03%|79.54%|85.99%|  \n|breast-w (LDPLC)|**83.12%**|**85.97%**|**83.12%**|**86.98%**|\n|credit-a (PMF)|59.57%|63.59%|59.28%|57.83%|  \n|credit-a (PMF-TLC)|73.62%|79.91%|74.78%|82.03%|\n|credit-a (LDPLC)|**82.90%**|**80.48%**|**82.90%**|**83.19%**|\n|diabetes (PMF)|41.93%|68.43%|43.88%|37.89%|  \n|diabetes (PMF-TLC)|74.87%|**80.43%**|77.08%|79.08%|\n|diabetes (LDPLC)|**79.04%**|79.18%|**79.04%**|**79.56%**|\n||\n\nFrom the experimental results, it can be observed that LDPLC outperforms PMF and PMF-TLC in most binary-class crowdsourcing tasks, further demonstrating its effectiveness and applicability in label completion tasks. In the final version of the paper, we will add the reasons why we only compare WSLC in the experiments.\n\n**Author Response to Q2:** To address the reviewer's concerns, we conduct two groups of experiments: (1) We compare the performance of Pearson correlation and mutual information for learning workers' feature vectors. We refer to the method with mutual information as LDPLC-MI. (2) We compare the performance of cosine similarity and Euclidean distance for estimating worker similarity using the learned feature vectors. We refer to the method with Euclidean distance as LDPLC-ED.\n\nWe conduct the first group of experiments on all seven simulated datasets with discrete variables. The results are as follows:\n|Dataset|MV|GTIC|DEWSMV|MNLDP|\n|--|--|--|--|--|\n|audiology (LDPLC-MI)|80.53%|79.65%|80.53%|78.32%|\n|audiology (LDPLC)|80.53%|79.65%|80.53%|78.32%|\n|breast-cancer (LDPLC-MI)|74.48%|74.48%|74.48%|71.68%|\n|breast-cancer (LDPLC)|**75.17%**|**75.17%**|**75.52%**|**75.52%**|\n|car (LDPLC-MI)|83.56%|83.56%|83.56%|**85.76%**|\n|car (LDPLC)|83.56%|83.56%|83.56%|85.65%|\n|kr-vs-kp (LDPLC-MI)|**85.54%**|**85.51%**|**85.51%**|**85.70%**|\n|kr-vs-kp (LDPLC)|85.45%|85.42%|85.45%|85.64%|\n|mushroom (LDPLC-MI)|**91.31%**|**91.30%**|91.30%|91.49%|\n|mushroom (LDPLC)|91.30%|91.29%|91.30%|91.49%|\n|tic-tac-toe (LDPLC-MI)|**81.63%**|**81.63%**|**81.63%**|80.48%|\n|tic-tac-toe (LDPLC)|81.52%|81.42%|81.42%|**80.69%**|\n|vote (LDPLC-MI)|82.53%|82.53%|82.53%|84.83%|\n|vote (LDPLC)|**82.99%**|**82.99%**|**82.99%**|**85.06%**|\n||\n\nFrom the results, it can be found that there is no significant difference in performance between Pearson correlation and mutual information. However, it's worth noting that mutual information can only be used to measure the correlation of discrete variables. With the aim of broadening the applicability of the proposed LDPLC, we choose Pearson correlation as the preferred method to learn workers' feature vectors.\n\nWe conduct the second group of comparative experiments on the \"LabelMe\" datasets. The results are as follows:\n\n|Dataset|MV|GTIC|DEWSMV|MNLDP|\n|--|--|--|--|--|\n|LabelMe (LDPLC-ED)|81.4%|81.4%|81.4%|81.7%|\n|LabelMe (LDPLC)|**81.7%**|**81.7%**|**81.6%**|**82.5%**|\n||\n\nThe experimental results clearly indicate that cosine similarity is better than Euclidean distance in estimating worker similarity. Therefore, we choose cosine similarity as the preferred method to estimate worker similarity. In the final version of the paper, we will give related analysis and discussion of the methods employed in this paper.\n\n**Author Response to WSLC's Weaknesses:** The core assumption of WSLC is that workers with similar cognitive abilities will annotate similar labels on the same instances. Therefore, it completes each missing label solely based on the labels annotated by similar workers on this corresponding instance. However, in real-world crowdsourcing scenarios, each instance usually has few labels, thus WSLC struggles to complete its missing labels. By introducing the assumption that the same worker will also annotate similar labels on similar instances, the missing labels can not only be inferred from labels of similar workers on the same instance but also absorb the distribution information of labels from all workers across neighboring instances. In the final version of the paper, we will further analyze the reasons why WSLC has these weaknesses in the introduction.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. The theoretical claim is about the convergence of the proposed method.", "experimental_designs_or_analyses": "Yes. This paper designs the related experiments to show the effectiveness of the proposal.", "relation_to_broader_scientific_literature": "This paper discusses the effectiveness of label completion for crowdsourcing problem, and proposes a label propagation method for label completion.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1 This paper studies the label completion problem in crowdsourcing area, which is a very important issue as usually the missing proportion is very high in real-world scenarios. \n2 This paper proposes a label completion method based on label distribution propagation. Compared to existing classical WSLC method, both worker similarity and label correlation are considered.\n3 This paper theoretically analysis the convergence of the proposed method.\n4 This paper is well-written and easy to read.\n\nWeaknesses:\n1 There are several label completion methods mentioned in related work, why only compare WSLC only in the experiments?\n2 For the methods employed in this paper, it is better to give the thorough analysis and discussion, such as explaining why Pearson correlation is selected to learn feature vectors rather than other correlation methods. Why is cosine similarity employed to obtain worker similarity? An introduction and discussion on the reasonableness of using these methods should be presented.", "other_comments_or_suggestions": "Regarding the weaknesses of WSLC, the paper only provides a conclusion without conducting a detailed analysis. It is necessary to further analyze the reasons why WSLC has these weaknesses in the introduction.", "questions_for_authors": "1 There are several label completion methods mentioned in related work, why only compare WSLC only in the experiments?\n2 For the methods employed in this paper, it is better to give the thorough analysis and discussion, such as explaining why Pearson correlation is selected to learn feature vectors rather than other correlation methods. Why is cosine similarity employed to obtain worker similarity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper tackles an important problem in crowdsourcing, that is, how to fill in missing labels when only a few workers label each item. 
It points out a clear weakness in a well-known method (WSLC) and proposes a new solution (LDPLC) that uses both worker similarity and instance similarity. The effectiveness of the proposed method was supported by experiments on a wide range of datasets.

The reviewers raised some concerns, but the authors addressed them carefully with new experiments and detailed explanations. Most reviewers were satisfied with the responses and supported acceptance.

Overall, this paper is well-structured, technically sound, and makes a clear contribution. It meets the standard for acceptance.
Decision: Accept (poster)