TLLC: Transfer Learning-based Label Completion for Crowdsourcing
Authors: Wenjun Zhang
Liangxiao Jiang
Chaoqun Li
Venue: ICML 2025 spotlightposter (2025)
Keywords: Crowdsourcing learning
Label Completion
Worker modeling
Transfer Learning
Primary area: general_machine_learning->supervised_learning
TL;DR: This paper proposes a novel transfer learning-based label completion (TLLC) algorithm.

Abstract: Label completion serves as a preprocessing approach to handling the sparse crowdsourced label matrix problem, significantly boosting the effectiveness of the downstream label aggregation. In recent advances, worker modeling has been proved to be a powerful strategy to further improve the performance of label completion. However, in real-world scenarios, workers typically annotate only a few instances, leading to insufficient worker modeling and thus limiting the improvement of label completion. To address this issue, we propose a novel transfer learning-based label completion (TLLC) method. Specifically, we first identify all high-confidence instances from the whole crowdsourced data as a source domain and use it to pretrain a Siamese network. The abundant annotated instances in the source domain provide essential knowledge for worker modeling. Then, we transfer the pretrained network to the target domain with the instances annotated by each worker separately, ensuring worker modeling captures unique characteristics of each worker. Finally, we leverage the new embeddings learned by the transferred network to complete each worker’s missing labels. Extensive experiments on several widely used real-world datasets demonstrate the effectiveness of TLLC. Our codes and datasets are available at https://github.com/jiangliangxiao/TLLC.

TLLC: Transfer Learning- based Label Completion for Crowdsourcing
Wenjun Zhang' Liangxiao Jiang' Chaoqun Li?
Abstract
domain experts is often expensive and time-consuming (Lu et al., 2023). Fortunately, crowdsourcing offers faster and more cost-effective alternative by employing crowd workers for annotation (Li et al., 2021). Due to varying expertise among workers, the labels collected from crowd workers contain a lot of noise (Xia et al., 2024). To address this, repeated annotation has been widely adopted, where each instance is annotated by multiple workers to obtain multi- ple noisy label set (Sheng al., 2008). Thus, simultaneously for multiple instances, a label matrix will be obtained. Sub- sequently, label aggregation is applied to infer the unknown true label of each instance based on this matrix.
Label completion serves as a preprocessing ap- proach to handling the sparse crowdsourced la- bel matrix problem, significantly boosting the effectiveness of the downstream label aggrega- tion. In recent advances, worker modeling has been proved to be a powerful strategy to fur- ther improve the performance label comple- tion. However, real-world scenarios, workers typically annotate only a few instances, leading to insufficient worker modeling and thus limit- ing the improvement of label completion. To address this issue, we propose novel transfer learning- based label completion (TLLC) method. Specifically, we first identify all high- cigh-coofieeece instances from the whole crowdsourced data as source domain and use pretrain Siamese network. The abundant annotated instances in the source domain provide essential knowledge for worker modeling. Then, we transfer the pretrained network to the target domain with the instances annotated by each worker separately, ensuring worker modeling captures unique characteristics of each worker. Finally, we leverage the new embeddings learned by the transferred network to complete each worker's missing labels. Ex- tensive experiments on several widely used real- world datasets demonstrate the effectiveness of TLLCC Our codes and datasets are available at htnps//ggthubbcoom//jjaaaaggnnnt
To improve the performance of label aggregation, numerous methods have been proposed over the past decades (Dawid & Skene, 1979; Sheng et al., 2008; Zhang et al., 2016; Ro- drigues & Pereira, 2018; Jiang al., 2022; Li et al., 2023; Ying et al., 2024). These methods have gradually reached a consensus: when worker annotation is more accurate than random annotation, the more noisy labels an instance re- ceives, the easier it becomes to infer its unknown true la- bel (Chen et al., 2022; Zhang et al., 2023b). However, in real-world scenarios, each worker typically annotates only small number of instances, and few labels are typically collected per instance to reduce cost, resulting in highly sparse crowdsourced label matrix (Jung & Lease, 2012). This fact leads to label aggregation failing to achieve the expected performance relying solely on the existing labels in label matrix. To address this issue, label completion has been proposed to fill in more missing labels for the sparse label matrix, which is gaining increasing attention.
Although only few methods have been proposed so far, they have already demonstrated that label completion can serve as preprocessing approach to boost the effectiveness of the downstream label aggregation. Among them, recent advances further highlight the strength of worker modeling in improving the performance of label completion. Specif- ically, Yang et al. (2024) filter out potential noisy labels through worker modeling, and Wu et al. (2024) estimate worker similarity through worker modeling, both achiev- ing notable improvements. However, to the best of our knowledge, despite its effectiveness, worker modeling is still constrained by the limited number of instances anno- tated by each worker. Insufficient annotated instances fail to accurately reflect the annotation ability of each worker,
1. Introduction
Supervised learning has achieved remarkable performance across diverse tasks, and its success relies on sarge-scale annotated data (Jiang et al., 2019; Zhang et al., 2023a). How- ever, acquiring large-scale accurately annotated data from
'School of Computer Science, China University of Geosciences, Wuhan 430074, China "School of Mathematics and Physics, China University of Geosciences, Wuhan 430074, China. Correspon- dence to: Liangxiao Jiang <ljiang@ecus edd..ccccu..d............ edu.en>.
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Transfer Learning-based Label Completion for Crowdsourcing
leading to insufficient worker modeling- Subsequently, in- sufficient worker modeling may misguide label completion, thereby limiting the improvement of label completion.
Table I. Differences among existing label completion methods,
Methol
Label maiss
Isotancc atribanes
Worker modking
Applicable sccnarios
Jus Lesae (2012)
Bisary
To address this issue, we propose a novel transfer learning- based label completion (TLLC) method. Specifically, we first identify all high-confidence instances from the whole crowdsourced data as a source domain and use it to pretrain Siamese network. The abundant annotated instances in the source domain provide essential knowledge for worker modeling. Then, we transfer the pretrained network to the target domain with the instances annotated by each worker separately, ensuring worker modeling captures unique char- acteristics of each worker. Finally, we leverage the new embeddings learned by the transferred network to complete each worker's 's missing labels. In general, the contributions of this paper can be summarized as follows:
Watasbee & Kadina (2014)
Bisary
Zioe & Be 12016)
Mulli class
You et (2024)
Baxayy
et (3024)
Mutii class
estimated worker similarities and leveraged these similar- ities to complete missing labels in binary crowdsourcing scenarios. Zhou & He (2016) used a three-dimensional ten- sor to represent the crowdsourced label matrix, and then performed tensor augmentation and completion to complete missing labels. Recently, worker modeling has been intro- duced into lahel completion, achieving significant progress. Yang et al. (2024) utilized worker modeling to improve the PMF-based label completion method (Jung & Lease, 2012). They modeled each worker by training a classifier and used the classifier to filter out potential noisy labels annotated by this worker before PMF. Wu et al. (2024) proposed a worker similarity-bassed label completion method called WSLC. WSLC first modeled each worker by learning a correlation vector between worker labels and instance attributes. Then, WSLC measured the cosine similarity between correlation vectors as worker similarity and used worker similarity to perform weighted voting for estimating missing labels. Ta- ble summarizes the differences among the above methods, including whether they utilize the label matrix, whether they utilize instance attributes, whether they utilize worker modeling, and their applicable scenarios.
We reveal the limitations of existing methods that lever- age worker modeling to improve label completion. The fact that each worker annotates only few instances leads to insufficient worker modeling and thus limits the improvement of label completion.
We construct source and target domains for worker modeling using crowdsourced data. The source do- main provides essential knowledge for worker model- ing and target domains ensure worker modeling cap- tures unique characteristics of cach worker.
We propose a novel transfer learning-hased label com- pletion (TLLC) method. TLLC introduces transfer learning to avoid insufficient worker modeling and leverages the new embeddings learned by the trans- ferred network to complete missing labels.
Transfer learning. To alleviate the insufficient worker modeling problem, introduce transfer learning into label completion. Based on whether the source and target do- mains share the same attribute space, transfer learning can be divided into homogeneous transfer learning and heteroge- neous transfer learning (Weiss et al., 2016). Homogeneous transfer learning ( (Yao & Doretto, 2010; Shi Sha, 2012: Moustakas & Kolomvatsos, 2024) is applicable when the source and target domains have identical attribute spaces, while heterogeneous transfer learning (Sukhija, 2018; Bica & van der Schaar, 2022: Syu et al., 2025) is applicable when the source and target domains have different attribute spaces. In crowdsourcing scenarios, both the source and target do- mains should be derived from the same crowdsourced data, ensuring identical attribute spaces. Therefore, this work draws inspiration from homogeneous transfer learning to improve label completion.
The rest of this paper is organized as follows: Section 2 briefly reviews closely related work. Section 3 provides detailed description of our proposed TLLC. Section 4 reports the experiments and results. Section 5 concludes this paper and outlines future research directions.
2. Related Work
In this section, we briefly review the related work on label completion and transfer learning.
Label completion. In crowdsoureing scenarios, label completion was initially proposed by Jung & Lease (2012). They applied probabilistic matrix factorization (PMF) to label completion, successfully completing missing labels in binary crowdsourcing scenarios. Inspired by collaborative filtering (Resnick et al., 1994), Watanabe & Kashima (2014) assumed that workers with similar annotation tendencies are more likely to assign the same labels. Subsequently, they
3. The Proposed TLLC
3.1. Preliminary
Before introducing our proposed method in detail, we first define the basic notations for label aggregation and label completion in crowdsourcing. Let D denote the crowd-
2

---

Transfer Learning-basee Label Completion for Crowdsourcing
Source domain data
Protrain
2
crowdsourced
Original
Contract
Pretrained metwork
Trandar
?
Transfereed networks
Learn
embeddings
Complete
?
crowdsourced
Completed
data
data
Target donsii data
Figure 1. Overall framework of TLLC.
sourced data ((x, L:) where at is the i-th instance in D. Lg is the multiple noisy label set of 2;, and N is the number of instances. 2; can be represented as Tim.....,Im........ F11h J. where M is the dimension of at- tributes, and Fam is the attribute value of æ; on the m-th attribute Am. L, can be represented as {4,-)/=1, where R the number of workers, and 1tr is the label of æ; anno- tated by the r-th worker uy 44r takes value from fixed set (-1,c1,--.. p...----Cpp Cg. CQ) ) where Q is the number of classes, Cq is the q-th class, and -1 means that up does not annotate Based on these notations, we define label aggregation and label completion by Definitions 3.1 and 3.2.
plete each worker's missing labels with new embeddings to obtain completed crowdsourced data.
To this end, TLLC needs to address three key issues: 1) How to construct the source and target domains from a given crowdsourced data? 2) How to perform worker modeling via transfer learning? 3) How to perform label completion? In the following subsections, we provide a detailed description of TLLC based on these three key issues.
3.2. Source and Target Domains Construction
First, we define the domain and task in transfer learning by Definitions 3.4 and 3.5, respectively.
Definition 3.1. Label aggregation infers the unknown true label yi of each instance x; based on {<x,, L,)) min- imizing the error between the aggregated label ÿ and the unknown true label ys-
Definition 3.4. A domain 'D consists of an attribute space X' and marginal probability distribution P(X), i.c. D (X,)
Definition 3.2. Label completion infers the missing label 4, 1 of each instance æ; based on f(x L,)) 3-m1 en- suring that the completed label 4.r is the most likely label annotated to æ; by worker Up..
Definition 3.5. task consists of label space y and an objective predictive function f. i.c., f).
Then, we can denote the source domain and the target do- main as Dy and Dr respectively. Since there is only D available in crowdsoureing scenarios, we let X's X' X' (A1, Am,-- ... Am and V's = Yr y (cI...., Cq............. C.....,cc.... {Q} in this paper. To learn fs, we need to construct the source domain data Ds from D. Inspired by confident learning (Northcutt et al., 2021), for each in- stance I; E D, we first obtain the initial aggregated label ÿ and the corresponding confidence P(ts) as follows:
In addition, we use X to represent all instances in D, X" to represent the instances annotated by u, L" to represent the labels u, annotated for X", and X to represent the instances not annotated by ut. Meanwhile, to simplify the complexity of label aggregation and label completion, D satisfies Assumption 3.3 in this paper. For clarity, all notations defined in this paper are summarized in a table, which is provided in Appendix A due to the limited pages.
Assumption 3.3. The annotation difficulty of instances in D is the same across all classes.
1 = arg max P(cg GEY)
(I)
ΣR E 5(lir, Cg)
As discussed above, to alleviate the insufficient worker mod- eling problem, we propose a novel transfer learning-basee label completion (TLLC) method. Its framework can be graphically shown in Figure/ 1. Firstly, we construct the source and target domain data from the original crowd- sourced data. Secondly, we pretrain a Siamese network with the source domain data. Thirdly, we transfer the pretrained network with each worker's corresponding target domain data, respectively. Fourthly, we use the transferred network to learn new embeddings for each worker. Fifthly, we com-
P(e L.)
(2)
> -r=1 ΣR 8(1ir, Ci)
where 8(-) an indicator function that returns if its two parameters identical, and 0 if its two parameters different. Subsequently, we calculate the average confidence Mag for eo as follows:
Σ 1-m 8(1i, C.yy P (i))
Me,
(3)
Σm= 8(1r, cr)
3

---

Transfer Learning-based Label Completion for Crowdsourcing
Algorithm 1 Source and Target Domains Construction
of Ds and Dr. À reflects the difference between fs and fr According to Assumption 3.3 and Equation (4), we can get P(Xs) = P(X). Therefore, Equation (5) will not change L' (Ds. Dr). Meanwhile, since À is only related to fs and fr. Equation (5) will not change X. Equation (4) filters out low-confidence instances for Xs. which reduces the noise in Ds. This means that es will be reduced, so the upper bound of eT will be reduced.
Require: crowdsourced data D.
Ensure: source and target domain data: Ds. (
1: for = 1 to N do
2: Calculate is and ((cy(L)) by Equations (1) and (2);
3: end for
4: forq qp=11 1 to Q do
5: Calculate May by Equation (3);
6: end for
7: Construct Xs by Equation (4);
3.3. Worker Modeling
8: Construct D3 by Equation (5);
) After constructing Ds and { we perform worker modeling via transfer learning. According to different trans- fer strategies, homogeneous transfer learning can be divided into four classes: instance-based, attribute-based, parameter- based, and relational based (Weiss et al., 2016). Inspired by fine-tuning (Guo et al., 2020), we leverage parameter-based transfer learning to perform worker modeling.
9: for 1 to R do
10: Construct D. by Equation (6);
11: end for
12:
return Ds. (D
Next, we can get Xs s as follows:
Specifically, we set up both Is and fr as Siamese networks with the same structure (Li al., 2022). Let Ì to denote Ds or Dr. f to denote fs or fr, we first generate the training data D' using D as follows:
X3 = L.)= 2 Mgg,, for i == 1,2,..., (4)
Finally, we construct the source domain data D3 as follows:
Ds = ((sss, Is:) for i = 1,2,..., |xs), (5)
D' = ((X,,,X,,b, | for i,j = 1,
X
(8)
where Xs1 is the number of instances in X s. Xss is the i-th instance in Xs, (ss equals to the initial aggregated label of Xs: For the target domain Dr, we construct a target domain data Di, for each worker up as follows:
where X contains all instances in D, Jhy is the supervisory information for training f. Let 1, denote the label corre- sponding to X, in D. We set yiy to 0 if 4 1j. otherwise to 1. f has two parts, Jo and Sa. Jg is used to learn new embedding = for an instance as follows:
Dj- {(X,, LJ) | for 1,2, ... X*
(6)
Ultimately, Ds contains abundant high-confidence anno- tated instances, which provide essential knowledge for worker modeling. Dr contains all instances annotated by worker uy. which reflect the unique characteristics of uy-
== [ (=1 ( ........ 2k =K),
where K is the dimension of the new embedding. Ta is used to calculate the Euclidean distance between new em- beddings z; and zj as follows:
The whole construction process of Ds and { in TLLC is shown in Algorithm 1. In Algorithm 1, lines 1-3 calculate the initial aggregated labels and their confidences with a time complexity of O(NQR). Lines 4-6 calculate the average confidences with a time complexity of O Q). Line 7 constructs Xs, and line 8 constructs Ds, both with a time complexity of (NN. Lines 9-11 construct the target domain data { with time complexity of O(N R). Considering only the highest-order terms, the overall time complexity of Algorithm 1 is O(NQR).
Ta(2,, 21) =
Σ
(Zsky zjx)
(10)
For each instance (a,, x².3/) ) in D', we optimize F using the Mean Squared Error (MSE) loss function as follows:
1x Σ (F)(T 11. -1? (1I)
Cmse
X
Theorem 3.6. Constructing Ds based on Equation (5) can reduce the generalization error in transfer learning.
According to Equations (8) and (11), we first pretrain fs using Ds. After pretraining, fs has learned the essential knowledge for worker modeling, with fr. Then, we create copy of fr as J for u, Simi- larly, we fine-tune fi using DI by Equations (8) and (11), After fine-tuning, fí is transferred from Ds to D fur- ther capturing the unique characteristics of up. Therefore, fine-tuning f is equivalent to modeling Uy.
Proof. According to Ben-David et al. (2010), the generaliza- tion error of transfer learning can be expressed as follows:
we share its parameters
eT S Es + L' (Ds, Dr) + X,
(7)
where eg and éT are the errors in the source domain and tar- get domain, respectively. T.' (Ds,, Dr) is the I.- divergence
4

---

Transfer Learning-based Label Completion for Crowdsourcing
Algorithm 2 Worker Modeling
where d' is the distance calculated by fa. Since is inde- pendent of v/. Ee[] = 0, and E E(e²] o², Equation (13) can finally be simplified to:
Require: source and target domain data: Ds, {Di 1r=1-
Ensure: transferred networks /r
1: Generate data D's using Dş by Equation (8);
Lise (16// - d' d')
(14)
2: Pretrain fs using D's by Equation (11);
3: Share the parameters of Js with fT;
Therefore, the effect of is a fixed constant, which means that worker modeling is robust to e.
4: for = to do
5: Copy fr f7:
6: Generate data Di using Dj by Equation (8);
3.4. Label Completion
7: Fine-tune f using Di by Equation (11);
8: end for
From the defined Equations (8) and (11), it can be found that a transferred f" should satisfy:
9: return (
(=)) < [2 25), if = 12r l + lar, (15)
The whole process of worker modeling in TLLC is shown in Algorithm 2. In Algorithm 2, line generates data D's with a time complexity of O(N2). Let Ogg) represent the time complexity of fg when training f (the time complexity of fa is O(K)). O(g) depends the scale of fg and is generally much larger than O(K). Line 2 pretrains Is with a time complexity of (N) Line shares the parameters of fs with fr and its time complexity of Og)). Lines 4-8 fine-tune { } with a time complexity of O(N2 Considering only the highest-order terms, the overall time complexity of Algorithm 2 is ONN Rg).
where zf is æ1 's new embedding learned by fg. 4 the label of æ; annotated by U4. According Equation (15), TLLC completes the missing labels of u,- by calculating the distance between new embeddings of unannotated instances and annotated instances.
Specifically, we first use fi to learn the new embedding for each instance in X by Equation (9). Then, we calculate the centroid zy of new embeddings for each class cy as follows:
2/ (2
qg kgk
(16)
Theorem 3.7. Parameter-baat based transfer learning can re- duce the generalization error in worker modeling.
where E/e can be calculated as follows:
E. S(L,,c)) E (g)
(17)
9k
Proof. Dg and Di. share the same attribute space. Mean- while, both Jr and fy. are trained to map the attribute space to the label space. These consistencies make transfer learn- ing feasible. Based on parameter- -based transfer learning, we perform pre- vre-trainin and tine-tuning with the same Siamese networks on Dg and Di., respectively. According to Equa- tion (7), we adopt the same networks to reduce the difference between fr and fT. thereby reducing X, which further re- duces the upper bound of ér. This is equivalent to reducing the generalization error in modeling worker Uh.
Subsequently, we calculate the averaged distance d; of new embeddings for each class Cg follows:
X S(L Cg) 27) T 11
(18)
E Cg)
Finally, for each unannotated instance X E X", we obtain its new embedding 2 by Equation (9) and complete 4r with Cg if the following condition is satisfied:
Ll
Theorem 3.8. When the noise in D' follows independent and identically distributed (i.i.d.) Gaussian distribution, worker modeling is robust to noise.
ftalz, - - o ^ X > 2Q.
(19)
Here, fTa(z1 xy) < d ensures that Xi is more similar
to the instances annotated as Gg. X > 2Q encourages scenarios where u, annotates at least two instances for each class Cg. although the two are not strictly equivalent.
Proof. We first use y and v/ to represent the supervisory information calculated using noisy labels and true labels, respectively. Then, y can be further expressed as follows:
The whole process of label completion in TLLC is shown in Algorithm 3. In Algorithm 3, line 2 constructs X" and X" for u, with time complexity of O(N). Lines 3-5 learn new embeddings for X" with time complexity of O Lines 6-9 calculate the centroid and the averaged distance for each class with a time complexity of O(NQK).
s - vll te, e~ o²(
(12)
Therefore, Equation (11) can be derived as follows:
Lisee = El(y' - d') = ((V + c 2')²]
(13) Lines 10-18 complete missing labels for X" with a time
= El(v/ - =d)) (1/ d''cc +2)c++B(())
5

---

Transfer Learning-based Label Completion for Crowdsourcing
Algorithm 3 Label Completion
Table 2. Detailed network structure and parameter settings of f.
Require: crowdsourced data D, networks ( 1/ Ensure: completed crowdsourced data D.
Layer type
Output dimension
Activation function
1: for 11 to R do
Input layer
128
ReLU ReLU
2: Construct X and X using D;
Fully connected layer
64
Output layer
2
3: for = to |XX] do
4: Learn 2 for X by Equation (9);
5: end for
The label aggregation methods used in our experiments in- clude majority voting (MV) (Sheng et al., 2008), ground truth inference using clustering (GTIC) (Zhang et al., 2016), differential evolution-ban weighted soft majority voting (DEWSMV) (Tao al.. 2021), multiple noisy label distri- bution propagation (MNLDP) (Jiang et al., 2022), attribute augmentation- based label integration (AALI) (Zhang et al., 2023c), and label aggregation with graph neural networks (LAGNN) (Ying et al., 2024). For MV and GTIC, we use the existing implementations on the Crowd Environment and its Knowledge Analysis (CEKA) (Zhang et al., 2015) platform. For WSLC, DEWSMV, MNLDP, and AALI, we implement them on the CEKA platform. For LAGNN and TLLC, we implement them in Python. The parameter settings of all existing methods are consistent with those specified in their original papers. For TLLC, we set K = 2, the number of epochs to Q, and the batch size to 32. Additionally, we set the Siamese network f in TLLC to small scale to en- sure convergence, and the detailed network structure and parameter settings of f are shown in Table 2.
6: for 1= do
7: Calculate Zy for cq by Equation (16);
8: Calculate dy for cq by Equation (17);
9:
end for
for 1 to |X| do
11: Learn 2. for X by Equation (9); for q = to Q do
12:
13:
if Equation (19) holds then
14:
Complete a label lip = Co for X";
15:
break;
end end for 18: end for
19: end for
20: Reconstruct with " and {X.)
21: return 1.
complexity of (g + QK)). Due to Ogg) >> OQK)) the time complexity of lines 1-19 should be Rg)) Line 20 reconstructs D with time complexity of O(NQR). Considering only the highest-order terms, the overall time complexity of Algorithm 3 is Rg)
To provide more comprehensive comparison, conduct experiments on three different real-world datasets: Income, Leaves, and Music genre. All three widely used datasets are collected through the online platform Amazon Mechanical Turk (AMT), and they represent different crowdsourcing re- quirements. Specifically, Income is collected from a binary scenario, while Leaves and Music-genre are collected from multi-class scenarios. In Income and Leaves, each instance is annotated by 10 workers, whereas in Music Music-genre, genre, each instance is annotated by 4.2 workers. The proportion of missing labels in Income, Leaves, and Music 0.88, and 0.90, respectively. Therefore, the label matrices of all three datasets are highly sparse, which aligns with the application scenarios of label completion. Due the limited pages, more detailed information about these three datasets is provided in Appendix B.
By combining Algorithms 1 to 3, the overall time complex- ity of TLLC is O(NQR + 2 2 + N Rg). Considering only the highest-order terms, the overall time complexity of TLLC is O(N Rg), which is caused by worker modeling.
4. Experiments and Results
are 0.85,
To validate the effectiveness of TLLC, we conduct extensive experiments. This section presents our experiments through three aspects: experimental setup, results, and analysis.
4.1. Experimental Setup
As shown in Table the state-of-the-an WSLC (Wu et al., 2024) employs worker modeling and supports multi-class crowdsourcing scenarios, making it key baseline for com- paring with our proposed TLLC. We evaluate WSLC and TLLCC by completing the same crowdsourced datasets and measuring the aggregation accuracy of label aggregation methods on their completed datasets, where aggregation accuracy represents the proportion of instances with aggre- gated labels matching true labels.
4.2. Experimental Results
Aggregation accuracy. To reduce the impact of random- on the experimental results, independently repeat the experiments on each dataset ten times. Figure 2 shows the averaged aggregation accuracy of each label aggregation method after performing label completion by WSLC and TLLC, respectively. Based on these experimental results, we can summarize the following highlights:
6

---

Transfer Learning-based Label Completion for Crowdsourcing
71.
WAC
125
200
(a) Income
Income
700
(b) Leaves
14:0
DEASWRY
LAGNN
(b) Leaves
(c) Music genre
Figure 3. Critical difference diagrams of significance tests.
On dataset Music gente, although TLLC does reach its outstanding performance levels as in datasets Income and Leaves, it still maintains relatively high upper bound of performance. Specifically, the aggre- gation accuracies of MNLDP (78.00%) and LAGNN (78.57%) after completion by TLLC are competitive with those of MNLDP (78.86%) and LAGNN (77.86%) after completion by WSLC, respectively.
M7
012
104.09
AX22
L4G8
(c) Music genre
Figure Averaged aggregation accuracy (%) of each label aggre- gation method completed by WSLC and TLLC.
On dataset Income, after completion by TLLC, the ag- gregation accuracy of each label aggregation method improves significantly. Specifically, the aggregation ac- curacies of MV (74.97%), GTIC (74.67%), DEWSMV (74.80%), MNLDP (72.67%), AALI (75.37%), and LAGNN (74.67%) after completion by TLLC are much higher than those of MV (71.17%), GTIC (71.83%), DEWSMV (71.15%), MNLDP (71.83%), AALI (64.87%), and LAGNN (71.00%) after comple- tion by WSLC, respectively.
addition to comparing the averaged aggregation accuracies of repetitions, we directly per- form Friedman test with corresponding post-hoc tests (e.g., Nemenyi test) (Demsar, 2006; Jansen al., 2023) each dataset using the results of ten repetitions. These significance tests allow us to compare the performance dif- ferences between the label aggregation methods completed by WSLC and TLLCC Based on test results, we present the critical difference (CD) diagrams Figure 3, As shown in Figure 3, on datasets Income and Leaves, the label ag- gregation methods completed by TLLC achieve superior average rankings. Moreover, while TLLC's performance on dataset Music is less pronounced compared to its performance on Income and Leaves, no statistically signifi- cant differences are observed between the label aggregation methods completed by WSLC and TLLC.
Significance tests.
On dataset Leaves, after completion by TLLC, the ag- gregation accuracy of each label aggregation method improves significantly. Specifically, the aggregation ac- curacies of MV (68), GTIC (67.19%), DEWSMV (69.01%) MNLDP (69.79 (99.79% AALI (72.40%), and LAGNN (68.75%) after completion by TLLC are much higher than those of MV (63.80%) GTIC (63.80%) DEWSMV (63.85%). MNLDP (64.58%), AALI (64.06%), and LAGNN (63.80%) after comple- tion by WSLC, respectively.
4.3. Discussion and Analysis
The experimental results above clearly validate the effec- tiveness of TLLC. Specifically, on datasets Income and

---

Transfer Learning-basee Label Completion for Crowdsourcing
that after filtering, the aggregation accuracies for almost all classes across all datasets are significantly improved, which strongly supports the rationality of Equation (4).
Subsequently, considering the impact of insufficient worker modeling on label completion, we introduce transfer learn- ing into worker modeling. To validate the rationality of this strategy, we focus on a worker with relatively few labels 2) from dataset Music-genre Figure 5 illustrates the new embeddings of X learned by the Siamese network f corre- sponding to this worker, obtained through two approaches: using transfer learning (pre-training få and then transferring to f and without transfer learning (directly training ft). The visualization in Figure demonstrates that the former approach better clusters instances with the same true labels, indicating its ability capture more essential knowledge for worker modeling effectively.
(a) Income
(b) Leaves
(c) Music.gena
Figure 4. Comparison aggregation accuracy in and Xg.
Finally, to complete the missing labels, we design Equa- tion (19), which determines whether or not to complete missing labels based on the distances between the new embeddings of annotated and unannotated instances. To validate the rationality of Equation (19), we analyze the changes in annotation quality of workers before and after label completion dataset Income, shown in Figure 6 (results of datasets Leaves and Music_genre are provided in Appendix C due to the limited pages). From Figure 6 we can see that after label completion using WSLC, workers' annotation quality tends to converge, indicating WSLC as- signs similar labels across workers. This erases workers' unique characteristics, violating Definition 3.2. In contrast, TLLC maintains smaller changes in workers' annotation quality, preserving their unique characteristics and better adhering to Definition 3.2. These results strongly validate the rationality of label completion in TL..C and confirm that ff effectively captures the unique attributes of u,..
(a) With transfer learning
(b) Without transfer learning
Figure 5. Visualization of new embeddings learned by the Siamese
network fr with and without transfer learning.
TLEC
08
WALC
06-
0.2-
Wirker index
Figure 6. Changes in annotation quality of workers before and after label completion on dataset Income.
Leaves, TLLC consistently outperforms WSLC in aggrega- tion accuracy across all label aggregation methods. Even on dataset Music genre, significance tests indicate that TLLCC still demonstrates strong potential. In this subsection, we provide deeper analysis of TLLC, validating its underlying rationality and exploring the reasons behind its suboptimal performance on dataset Music genre.
Ablation experiment. Through rationality analyses, we have preliminarily validated the effectiveness of each strat- egy in TLLC. To further investigate the impact of different strategies on TLLC's performance, we conduct an ablation experiment on dataset Income. Specifically, we fix MV as the label aggregation method to evaluate the aggregation accuracy achieved by TLLC and its variants. For clarity, we denote the variants of TLLC without instance filtering, pre- training, and transfer learning "TI.LC-IF". "TLLCCPTT" and "TLLC-TL", respectively. The detailed experimental results are shown in Figure 7. Based on these results, it can be found that the performance degrades when any of these strategies is removed from TLLCC These findings further highlight the critical role of instance filtering, pretraining, and transfer learning in enhancing TLLC's performance.
Rationality, To improve the performance of label comple- tion, we introduce several innovative strategies for TLLC. First of all, when constructing Xs, considering the noise in the label matrix, we draw inspiration from confident learn- ing and design Equation (4) to filter out high-confidence annotated instances based on the initial aggregated labels. To validate the effectiveness of this strategy, we compare the aggregation accuracies in X (before filtering) and X (after filtering) for each class across three datasets. The detailed re- sults are shown in Figure 4. Based on Figure 4, we can find
Sensitivity analysis. In addition to evaluating the effec- tiveness and rationality of TLLC, we also perform param-
8

---

Transfer Learning-baser Label Completion for Crowdsourcing
75.00
16
Antontiiin quality
14.00
0.4
72.30
38
130
2.0
200
TLLC
TLLC-IF
TL
TLLC
Figure 7. Aggregation accuracy (%) achieved by TLLC and its variants on dataset Income.
Figure 8. Relationship between the number of annotated instances and annotation quality for each worker in dataset Music genre.
Table Aggregation accuracy (%) achieved by TLLC on dataset Income as the parameters change.
pronounced compared to its performance on datasets Income and Leaves. To explore the underlying reasons, Figure illustrates the relationship between the number of annotated instances and annotation quality for each worker in dataset Music
New embedding dimension
Value
2
4
6
8
10
Music.genre genre (results of datasets Income and Leaves are pro-
Accurary (%)
74.94
71.83
71.66
73.33
72.66
vided in Appendix E due to the limited pages). Notably, Figure identifies three adversarial workers (bighlighted in red) who annotated large number of instances with exceptionally low quality. Since these adversarial work- ers annotate large number of instances, they significantly influence fi in TLLC during transfer learning, as f cap- tures their unique but erroneous characteristics. Further- more, since TLLC adheres to Definition 3.2 and seeks to complete the labels workers are most likely to annotate, it inadvertently completes incorrect labels for these work- ers, In contrast, as shown in Figure 6, WSLC reduces the impact of adversarial workers by changing the annotation quality of workers, though this comes at the cost of crasing their unique characteristics. These observations explain the anomaly in Figure 2(c) and reveal the lack of robustness in TLLC against adversarial workers with numerous labels.
Epochs
Value
2 74.94
4
8 72.16
10
Accurary (%)
72.33
73.00
72.83
Batch size
Value
8
16
64
128
Accurary (%)
71.83
72.50
74.94
73.33
73.16
eter sensitivity analysis for it. TLLC includes three key adjustable parameters: the new embedding dimension, the number of epochs, and the batch size. To observe the impact of these parameters TLLC's performance, we conduct dataset Income (using
sensitivity analysis experiments MV as the label aggregation method). In each experiment, two parameters are fixed, while the remaining one is var- ied. The detailed experimental results are shown in Table 3. From these results, it can be found that TLLC's effective- ness shows only slight variation with changes in parameter values. Given that the aggregation accuracy of MV after label completion using WSLC is 71.17% (as shown in Fig- ure 2(a)), it is clear that TLLC consistently achieves superior performance. Therefore, the effectiveness of TLLC is not highly sensitive to parameter settings.
5. Conclusion
This paper is the first reveal the limitations of insufficient worker modeling on label completion. To address this issue, we design a novel algorithm to construct the source and target domains from crowdsourced data, which makes it possible to introduce transfer learning into crowdsourcing. Subsequently, we train Siamese networks to model workers through transfer learning, which significantly mitigates the impact of insufficient worker modeling. Ultimately, both the theoretical analysis and experimental results validate the effectiveness and rationality of the TLLC we proposed.
Besides parameter settings, we conduct another set of ex- periments observe the impact of datasets' missing rate (proportion of missing labels) on TLLC. The results reveal that TLLC is more effective in scenarios with high missing rate. This finding aligns with our objective of addressing the challenges posed by insufficient worker modeling. Due to the limited pages, more detailed settings and results of these experiments are provided Appendix D.
However, the experimental results also highlight some limi- tations of TLLC, particularly the lack of robustness against adversarial workers who annotated a large number of in- stances. Refining the transfer learning process address this issue remains crucial direction for future research to improve the performance of TLLC.
Abnormality. According to Figure 2(c), we can observe that the performance of TLL.C on dataset Music genre less

---

Transfer Learning-based Label Completion for Crowdsourcing
Acknowledgments
Jiang, L.,, Zhang, L., Li, C., and Wu, J. A correlation-basel feature weighting filter for naive bayes. IEEE Trans.
The work was partially supported by National Natural Sci- ence Foundation of China (62276241) and Hubei Provincial Collaborative Innovation Center for Basic Education Infor- mation Technology Services (OFHUE202312).
Knowl. Data Eng., 31(2):201-213, 2019.
Jiang, L., Zhang, H.,, Tao, F., and Li, C. Learning from crowds with multiple noisy label distribution propagation. IEEE Trans. Neural Networks Learn. Syst. 33(1 1):588-- 6568, 2022.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Jung, H. J. and Lease, M. Improving quality of crowd- sourced labels via probabilistic matrix factorization. In Chen, Y., Ipeirotis, P. G., Law, F.,, von Ahn, I.,, and Zhang, H. (eds.), The 4th Human Computation Workshop, HCOMP@ AAAI 2012, Toronto, Ontario, Canada, July 23, 2012, volume WS-12-08 of AAAI Technical Report. AAAI Press, 2012.
References
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Mach. Learn., 79(1-2):151-175, 2010.
Li, J., Sun, H., and Li, J. Beyond confusion matrix: learn- ing from multiple annotators with awareness of instance features. Mach. Learn., 11((3):005--100 1075,, 2023.
Li, S., Huang, S., and Chen, S. Crowdsoureing aggregation with deep bayesian learning. Sci. China Inf. Sci., 64(3), 2021.
Bica, I. and van der Schaar, M. Transfer learning on hetero- geneous feature spaces for treatment effects estimation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Infor- mation Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Li, Y., Chen, C. L. P., and Zhang, T. A survey on siamese network: Methodologies, applications, and opportunities. IEEE Trans. Artif. Intell., ((6):994-1014, 2022.
Lu, Y. Li, W., Li, H., and Jia, X. Predicting label distribu- tion from tie-allowed multi-label ranking. IEEE Trans. Pattern Anal. Much. Intell., -5(12):15364-15377 2023,
Chen, Z., Jiang, L., and Li, C. Label augmented and weighted majority voting for crowdsourcing. Inf. Sci., 606:397-409 409, 2022.
Moustakas, T. and Kolomvatsos, K. Homogeneous transfer learning for supporting pervasive edge applications. Evol. Syst., 15(4):1179-1)
Dawid, and Skene, A. M. Maximum likelihood es- timation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20-28 1979.
2024.
Northeutt, C. G., Jiang, L., and Chuang, I. L. Confident learning: Estimating uncertainty in dataset labels. J. Artif. Intell. Res., 10:1373-1411, 2021.
Demsar, J. Statistical comparisons of classifiers over multi- ple data sets. J. Mach. Learn. Res., 7:1-30, 2006.
Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., and Riedl, J. Grouplens: An open architecture for collabo- rative filtering of netnews. In Smith, J. B., Smith, F. D., and Malone, T. W. (eds.), CSCW '94, Proceedings of the Conference on Computer Supported Cooperative Work, Chapel Hill, NC, USA, October 22-26, 1994, pp. 175-186. ACM, 1994.
Guo, Y., Li, Y., Wang, L., and Rosing, T. Adafilter: Adap- tive filter fine-tuning for deep transfer learning. In The Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2020, New York, NY, USA February 7-12, 2020, pp. 4060-4066 AAAI Press, 2020.
Rodrigues, and Pereira, C. Deep learning from crowds. In Mcllraith, S. A. and Weinberger, K. Q. (eds.), Proceed- ings of the Thirty-Second AAAI Conference Artificial Intelligence, (AAA1-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18). and the 8th AAAI Sym- posium on Educational Advances in Artificial Intelligence (EAAI-18) 18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 1611-1618. AAAI Press, 2018
Jansen, C., Nalenz, M., Schollmeyer, G., and Augustin, T. Statistical comparisons of classifiers by generalized stochastic dominance. J. Mach. Learn. Res., 24:231:1- 231:37, 2023.
10

---

Transfer Learning-based Label Completion for Crowdsourcing
Sheng, V. S., Provost, F. J., and Ipeirotis, G. Get another label? improving data quality and data mining using mul- tiple, noisy labelers. In Li, Y., Liu, B., and Sarawagi, S. (eds.), Proceedings of the 14th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008, pp. 614-622. ACM, 2008.
Data Mining, ICDM 2024, Abu Dhabi, UAE, December 9-12, 2024, pp. 540-549. IEEE, 2024.
Yao, Y. and Doretto, G. Boosting for transfer learning with multiple sources. The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pp- 1855- 1862. IEEE Computer Society, 2010.
Shi, Y. and Sha, F. cnpormation-thoortiiie learning of dis- criminative clusters for unsupervised domain adaptation. In Proceedings of the 29th International Conference on Machine Learning ICML 2012, Edinburgh, Scotland, UK, June 26- July 2012. ieml.ec/ Omnipress, 2012.
Ying, Z., Zhang. J., Li, Q., Wu, M., and Sheng, V. A little truth injection but a big reward: Label aggregation with graph neural networks. IEEE Trans. Pattern Anal. Mach. Intell., (696)))3199-11333 2024.
Zhang, H., Jiang, L.., Zhang, W., and Li, C. Multi-view attribute weighted naive bayes. IEEE Trans. Knowl, Data Eng. 35(7):2291-73000 2023a.
Sukhija, S. Label space driven heterogeneous transfer learn- ing with web induced alignment. In Mcllraith, S. A. and Weinberger, K. Q (eds.). Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intel- ligence (1AAI-18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp- 8165-8166. AAAI Press, 2018
Zhang, J., Sheng, V. S., Nicholson, B., and Wu, X. CEKA: a tool for mining the wisdom of crowds. J. Mach. Learn. Res., 16:2853 2858, 2015.
Zhang, J., Sheng. V. S., Wu, J., and Wu, X. Multi-class ground truth inference in crowdsourcing with clustering. IEEE Trans. Knowl. Data Eng.,
2016.
Syu, J., Fojeik, M., Cupek, R., and Lin, J. C. HTTPS: het- erogeneous transfer learning for split prediction system evaluated on healthcare data, Inf. Fusion, 113: 113:102617, 2025.
Zhang, W., Jiang, L., Chen, Z., and Li, C. Fnnwv: Sarthest-nearest neighbor- weishbor-based weighted voting for class-imbalanced crowdsourcing. Sei. China Inf. Sci., 2023b. doi: 10.1007/s1 11.1007/11132-002 143--033-884--
Tao, F., Jiang, L., and Li, C. Differential evolution-based weighted soft majority voting for crowdsourcing. Eng. Appl. Artif. Intell., 106: 104474, 2021.
Zhang, Y., Jiang, I.., and Li, C. Attribute augmentation- based label integration for crowdsourcing. Frontiers Com- put. Sci., 17(5)117533,, 2023c.
Watanabe, T. and Kashima, H. A label completion approach to crowd approximation. In Loo, C. K., Yap, K. S., Wong, K. W., Jin, A. T. B., and Huang, K. (eds.), Neural In- formation Processing- 21st International Conference, ICONIP 2014, Kuching, Malaysia, November 3-6, 2014. Proceedings, Part II, volume 8835 of Lecture Notes in Computer Science, pp. 377-385. Springer, 2014.
Zhou, Y. and He, J. Crowdsourcing via tensor augmentation and completion. In Kambhampati, S. (ed.), Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9- 15 July 2016, pp. 2435-2441 ICAUIAAII Press, 2016.
Weiss, K. R., Khoshgoftaar, T. M., and Wang, D. A survey of transfer learning. J. Big Data, 3:9, 2016.
Wu, X., Jiang, L., Zhang. W., and Li. C. Worker similarity- based label completion for crowdsourcing. IEEE Trans. Big Data, 2024. doi: 10. 109/1BDAAAA2222444225666 10.
Xia, M., Huang, Z., WU, R., Lyu, G., Zhao, J., Chen, G., and Wang, H. Unbiased multi-label learning from crowd- sourced annotations. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Yang, B., Jiang, L., and Zhang, W. Probabilistic ma- trix factorization-batt based three-stage label completion for crowdsoureing. IEEE International Conference on

---

Transfer Learning-based Label Completion for Crowdsourcing
A. Summary of commonly used notations.
As the method proposed this paper involves crowdsoureing, worker modeling, transfer learning, and Siamese networks, a large number of notations are introduced. Therefore, Table 4 provided to summarize the notations used in the paper to reduce reading difficulty.
Table 4. Summary commonly used notations.
Notation
Description
Notation
Description
D
data
instance
N
number of instances in D dimension of attributes number of workers
attribute value
M
new embedding
R
embedding value
Q
number of classes attribute
true label
aggregated label
source
D
supervisory information
T
target
noisy label missing label
K
dimension of the new embedding time complexity
= -1
O
completed label
D
completed data data corresponding ur training data L divergence all instances in D instances annotated by uy instances not annotated by ur multiple noisy label set labels uy annotated for X marginal probability distribution probability/ confidence
f
objective predictive function
Dr
fa
distance function
TY
Sg.
embedding function in
L
fr
function corresponding to ur
X
error
X"
X
difference functions
X
d
distance
L
class
D
worker
P(X)
2²
variance
P(e L)
average value
D
domain
z"
new embedding corresponding to
TT
task
2g
centroid corresponding to Cq
x
attribute space
d'
distance corresponding to f"
V
label space
dg
distance corresponding to Gq
N
Gaussian distribution
set size example object
C
loss function
E
expectation
indicator function
B. Detailed information of experimental datasets.
The descriptions of three real-world crowdsourced datasets are listed in Table 5. Here, "#lnstances" denotes the number of instances, "#Workers" denotes the number of workers, "#Labels" denotes the number of labels, "#Attributes" denotes the number of attributes, and "#Classes" denotes the number of classes. These datasets are collected from different application scenarios and represent different crowdsourcing requirements. We have uploaded these datasets and our codes, which are available at http://gghhubbbooo//ijaaaaaggnga
C. Changes in annotation quality of workers on datasets Leaves and Music Musicc genre.
The changes in annotation quality of workers before and after label completion on datasets Leaves and Music genre are shown in Figure 9. Similar to Figure 6, it can be found from Figure 9 that after label completion using WSLC, workers' annotation quality tends to converge, indicating WSLC assigns similar labels across workers. This erases workers' unique characteristics, violating Definition 3.2. In contrast, TLLC maintains smaller changes in workers' annotation quality, preserving their unique characteristics and better adhering to Definition 3.2. These results strongly validate the rationality
12

---

Transfer Learning-based Label Completion for Crowdsourcing
Table 5. Descriptions of the three real-world datasets used in experiments.
Dataset
Income
Leaves
Music genre
#Instances
600
384
700
#Workers
67
83
44
#Labels
6000
3840
2946
#Attributes
(nominal)
(numeric)
(numeric)
#Classes
10
Averaged #laaels per instance Proportion of missing labels
10
10
4.2
0.85
0.88
0.90
TLLC
0.2
WSEL!
0.5-
Anncrrion quala quality
0.4 Aunolation
0.6
0.6
0.4
Worker index
Werktr index
(a) Leaves
(b) Music_genre
Figure 9. Changes in annotation quality of workers before and after label completion on datasets Leaves and Music-genre.
Table 6. Aggregation accuracy (%) achieved by WSLC and TLL.C the missing rate changes.
Missing rate
0.9
0.7
0.5
0.3
0.1
WSLC
70.17
80.33
81.67
92.67
94.83
TLLC
71.16
81.16
82.33
92.33
94.00
of label completion in TLLC and confirm that f7. effectively captures the unique attributes of uy.
D. More detailed settings and results of sensitivity analysis experiments.
To analyze the impact of the missing rate on the performance of TLLC, we conduct simulated experiments on dataset Income. Specifically, we simulate 40 workers annotating the dataset, where each worker's annotation quality is randomly generated from uniform distribution of [0.55, 0.75). The missing rate is controlled by adjusting workers' annotation probabilities, ensuring it varies from 0.9 to 0.1 in intervals of 0.2. When the label aggregation method is fixed as MV, the aggregation accuracy achieved by WSLC and TLLC is shown in Table 6. It can be found that when the missing rate exceeds 0.5, TL.LC outperforms WSLC. However, as the missing rate decreases further, WSLC becomes more effective than TLLC. Therefore, TLLC is more effective scenarios with high missing rate. This finding aligns with our objective of addressing the challenges posed by insufficient worker modeling. TLLC improves label completion by addressing insufficient worker modeling. A higher missing rate increases the likelihood of insufficient modeling. making TLLC's advantages pronounced. As the missing rate decreases, TLLC's effectiveness relative to WSLC gradually diminishes.
E. Relationship distribution in datasets Income and Leaves.
The relationships between the number of annotated instances and annotation quality for each worker in datasets Income and Leaves are shown Figure 10. From Figure 10, it can found that datasets Income and Leaves also contain adversarial workers with low annotation quality. However, since they did not annotate large number of instances, TLLC's performance
13

---

Transfer Learning-based Label Completion for Crowdsourcing
insensitive them. This finding indicates that TLLC lacks robustness only against adversarial workers who annotated large number of instances.
Aunoation
50
100
1.50
200
250
300
150
50
100
150
200
250
300
390
400
Number of annossted instunces
Number of annetaned instances
(a) Income
(b) Leaves
Figure 10. Relationship between the number of annotated instances and annotation quality for each worker in datasets Income and Leaves.
14


Reviews:
[{"review_id": "3AAZLuN2i9", "reviewer": "Reviewer_x4PP", "summary": "The paper proposes a Transfer Learning-based Label Completion (TLLC) method for crowdsourcing scenarios. The authors address the issue of sparse label matrices, where individual workers annotate only a few instances, leading to insufficient worker modeling and poor label completion. The key idea of TLLC is to pre-train a Siamese network on high-confidence instances (source domain) and transfer it to model individual workers (target domain), thereby improving label completion. The method is evaluated against WSLC across three real-world datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your comments. Please find our detailed responses to your concerns as follows.\n\n**Author Response to Contributions:** This paper is the first to identify and address the limitation from insufficient worker modeling. Moreover, based on our review of related work, this paper is also the first to introduce transfer learning into label completion. While transfer learning is a well-established technique, its application in crowdsourcing—particularly when only a single crowdsourced dataset is available—poses a significant challenge in constructing source and target domains. Our paper proposes a novel algorithm to address this issue, achieving impressive results. In the final version of the paper, we will expand on these contributions in detail.\n\n**Author Response to Benchmarks:** Among existing label completion methods, apart from WSLC, only PG-TAC (Zhou & He, 2016) can handle both binary and multi-class classification problems (consistent with TLLC). Therefore, we compare TLLC with PG-TAC on the Income and Leaves datasets. The results are as follows:\n| |MV|GTIC|DEWSMV|MNLDP|AALI|LAGNN|\n|--|--|--|--|--|--|--|\n|PG-TAC (Income)|72.00%|71.67%|72.17%|**73.00%**|73.17%|71.83%|\n|TLLC (Income)|**74.97%**|**74.67%**|**74.80%**|72.67%|**75.37%**|**74.67%**|\n||\n|PG-TAC (Leaves)|63.54%|62.76%|63.54%|64.32%|64.58%|63.54%|\n|TLLC (Leaves)|**68.88%**|**67.19%**|**69.01%**|**69.79%**|**72.40%**|**68.75%**|\n||\n\nThese results further demonstrate the superior performance of TLLC.\n\n**Author Response to Experimental Analysis:** In our current paper, we analyze TLLC’s rationality and abnormality in the Discussion and Analysis subsection. For rationality, we explain why TLLC performs well by analyzing the effectiveness of each component in TLLC (see the Rationality paragraph on page 7). For abnormality, we reveal that TLLC is not robust to adversarial workers providing numerous labels, and we discuss the reasons behind this phenomenon (see the Abnormality paragraph on page 8). Additionally, to address the reviewer **ksp6’s Q2**, we conduct an ablation study on the Income dataset. The results of the ablation study further indicate the rationality of TLLC. In the final version of the paper, we will provide a more in-depth analysis based on the ablation study.\n\n**Author Response to Sensitivity Analysis:** The hyperparameters in TLLC include the new embedding dimension ($K$), the number of epochs, and the batch size. We conduct parameter sensitivity analysis experiments on the Income dataset (using MV as the label aggregation method) to observe TLLC’s performance. In each experiment, we fix two hyperparameters and vary the remaining one. The results are as follows:\n|$K$|2|4|6|8|10|\n|--|:--:|:--:|:--:|:--:|:--:|\n|Income|**74.94%**|71.83%|71.66%|73.33%|72.66%|\n||\n\n|Epochs|2($Q$)|4|6|8|10|\n|--|:--:|:--:|:--:|:--:|:--:|\n|Income|**74.94%**|72.33%|73.00%|72.16%|72.83%|\n||\n\n|Batch Size|8|16|32|64|128|\n|--|:--:|:--:|:--:|:--:|:--:|\n|Income|71.83%|72.50%|**74.94%**|73.33%|73.16%|\n||\n\nThese results show that TLLC’s performance varies slightly with changes in hyperparameter values. Considering that the aggregation accuracy of MV before label completion is 71.17%, it is evident that TLLC’s effectiveness is not highly sensitive to hyperparameter settings. In the final version of the paper, we will provide a detailed description and discussion of these parameter sensitivity experiments. \n\n**Author Response to Theoretical Claims:** As far as we know, adversarial labels are a type of noise but not the most dominant noise. In reality, workers hired from the general public usually lack expertise, and thus the noisy labels they provide are often random, satisfying the i.i.d. assumption. Moreover, Figure 6 does not indicate that TLLC fails to handle adversarial workers, but rather that it struggles with adversarial workers who annotate a large number of labels. In the final version of the paper, we will provide more explanations regarding Figure 6 and the characteristics of noisy labels to clarify these points.\n\n**Author Response to Complexity:** We conduct a new experiment on the Income dataset to compare the runtime of WSLC and TLLC. This experiment is conducted on a Windows 10 machine with an AMD Athlon(tm) X4 860K Quad Core Processor @ 3.70GHz and 16 GB of RAM. The runtime required to complete the Income dataset for WSLC and TLLC is as follows:\n|Dataset|WSLC|TLLC|\n|--|:--:|:--:|\n|Income|0.87s|150.33s|\n||\n\nThe results show that TLLC requires more runtime compared to WSLC. However, the primary computational cost of TLLC arises from transfer learning to train Siamese networks. Both transfer learning and Siamese networks are widely adopted techniques and are not computationally expensive to use in practical scenarios. Therefore, TLLC remains efficient and applicable in real-world scenarios. In the final version of the paper, we will include a detailed explanation of the computational cost of TLLC to further clarify this point.", "claims_and_evidence": "Most of the claims in this paper are well demonstrated. However, I still have some concerns: \n1) While Theorem 3.8 suggests that TLLC is resistant to i.i.d. Gaussian noise, real-world crowdsourcing noise is often adversarial (workers deliberately provide incorrect labels). Figure 6 shows that TLLC fails to handle adversarial workers, leading to poor performance on the Music Genre dataset.\n2) The paper claims that TLLC’s complexity is O(N²Rg), but lacks the theoretical and empirical comparison with WSLC.", "methods_and_evaluation_criteria": "The use of transfer learning to improve worker modeling makes sense given the sparse crowdsourced label matrix problem. However, there are some issues that should be further resolved:\n1) Only WSLC is used as a label completion baseline and more label completion baselines are needed to demonstrate to effectiveness of proposed TLLC.\n2) The paper does not provide a runtime comparison between TLLC and other methods. Given the O(N²Rg) complexity, TLLC may be computationally expensive, but this is not empirically analyzed.", "theoretical_claims": "The paper presents three key theoretical claims, and I have checked the logic and correctness of these proofs and identified the following issues:\nThe derivation of Theorem 3.8 is mathematically correct for i.i.d. Gaussian noise. However, real-world crowdsourcing noise is not i.i.d. Workers can introduce systematic bias rather than random Gaussian noise.", "experimental_designs_or_analyses": "I have checked the experimental designs and analyses of this paper. Experimental design is mostly valid, with realistic datasets and strong statistical tests. However, the evaluation has notable limitations:\n1) It lacks comparisons with other label completion baselines beyond WSLC, making it unclear how TLLC performs against broader alternatives.\n2) The experimental analysis lacks depth in several aspects. While the paper provides basic performance comparisons and statistical tests, it does not thoroughly investigate why TLLC performs well in some cases but struggles in others.", "relation_to_broader_scientific_literature": "TLLC builds on well-established ideas in transfer learning, worker modeling, and label completion. However, its novelty is limited because similar techniques exist in prior literature. Stronger comparisons with alternative label completion methods are needed to clarify its contribution.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1) The paper provides a well-structured introduction to the problem of sparse crowdsourced label matrices and explains why worker modeling is essential for label completion.\n2) The analysis of worker annotation quality before and after label completion (Figure 5) is an insightful addition.\n\nWeaknesses:\n1) The core techniques—transfer learning for label completion, worker modeling, and label filtering—are all adaptations of existing methods, rather than fundamentally new contributions.\n2) Lack of benchmarks against more label completion methods. It remains unclear if TLLC truly outperforms all alternatives.\n3) The paper does not analyze where TLLC makes mistakes, nor does it investigate failure cases in depth.\n4) No sensitivity analysis is conducted on hyperparameters, which could impact model stability.", "other_comments_or_suggestions": "Please see the weakness.", "questions_for_authors": "Please refer to the concerns above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RXCVhEHevD", "reviewer": "Reviewer_ksp6", "summary": "This paper at first reveals the limitations of existing methods that leverage worker modeling to improve label completion for Crowdsourcing and then proposes a novel transfer learning-based label completion (TLLC) method, which introduces transfer learning to avoid insufficient worker modeling and leverages the new embeddings learned by the transferred network to complete missing labels.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The proposed TLLC transfers the pretrained Siamese network to the target domain. In the paper, the authors just said: “Specifically, we set up both and as Siamese networks with the same structure (Li et al., 2022).” What are the detailed network structure and parameter settings?\n\n**Author Response to Q1:** Thanks for your valuable comments. In TLLC, the Siamese network is used to model each worker, and the new embeddings it learns are ultimately used to complete the worker’s missing labels. Since each worker annotates only a few instances, we set the network to a small scale to ensure convergence. The detailed network structure and parameter settings are as follows:\n|Layer Type|Output Dimension|Activation Function|\n|--|:--:|:--:|\n|Input Layer|128|ReLU|\n|Fully Connected Layer|64|ReLU|\n|Output Layer|2|-|\n||\n\nTo address the reviewer’s concerns, we will include the above information in the final version of the paper. Additionally, we have already submitted our code and datasets in Supplementary Material. At the same time, we will also open-source our code to facilitate the reproduction of our results once our paper is accepted. Thanks again for your valuable comments.\n\n**Q2:** Although the authors have already provided a deeper analysis of TLLC to validate its underlying rationality: 1) With and without instance filtering; 2) With and without transfer learning, a group of thorough ablation experiments are needed.\n\n**Author Response to Q2:** Thanks for your valuable comments. To address the reviewer’s concerns, we conduct an ablation study on the Income dataset for TLLC and its variants (using MV as the label aggregation method). The experimental results are as follows:\n| |TLLC|TLLC1|TLLC2|TLLC3|\n|--|--|--|--|--|\n|Aggregation Accuracy|**74.97%**|71.83%|74.16%|71.66%|\n||\n\nHere, TLLC1, TLLC2, and TLLC3 represent the variants of TLLC without instance filtering, pretraining, and transfer training, respectively. Considering that the aggregation accuracy of MV before completion is 71.17% (as shown in Figure 1), it can be observed that all TLLC variants outperform MV. Meanwhile, each variant performs worse than the complete TLLC, further indicating the superior performance and rationality of TLLC. In the final version of the paper, we will provide a detailed description and discussion of the setup and results of this ablation study. Thanks again for your valuable comments.\n\n**Q3:** On page 8, Figure 6 illustrates the relationship between the number of annotated instances and annotation quality for each worker in dataset Music genre. Why are the axes named Aggregation accuracy (%) and Number of aggregated instances?\n\n**Author Response to Q3:** Thanks for your valuable comments, and we apologize for our typos in Figure 6. In the final version of the paper, we will revise the titles of the horizontal and vertical axes in Figure 6 to \"Number of annotated instances\" and \"Annotation accuracy (%)\". Similarly, Figure 8 will be adjusted accordingly. Additionally, we will double-check and improve the writing of our paper. Thanks again for your valuable comments.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper proposes a novel transfer learning-based label completion (TLLC) method, which is the first work to introduce transfer learning to avoid insufficient worker modeling and then leverages the new embeddings learned by the transferred network to complete missing labels.", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "Strengths:\n1. The authors reveal the limitations of existing methods that leverage worker modeling to improve label completion for Crowdsourcing.\n2. To address this issue, the authors propose a novel transfer learning-based label completion (TLLC) method, which is the first work to introduce transfer learning to avoid insufficient worker modeling and then leverages the new embeddings learned by the transferred network to complete missing labels.\n3. The authors conduct extensive experiments to validate the effectiveness, rationality and abnormality of the proposed TLLC on the widely used real-world datasets.\n4. The organization of the paper is quite good and it is easy to follow the topic and the proposed method.   \n\nWeaknesses:\n1. The proposed TLLC transfers the pretrained Siamese network to the target domain. In the paper, the authors just said: “Specifically, we set up both $f_S$ and $f_T$ as Siamese networks with the same structure (Li et al., 2022).” What are the detailed network structure and parameter settings?\n2. Although the authors have already provided a deeper analysis of TLLC to validate its underlying rationality: 1) With and without instance filtering; 2) With and without transfer learning, a group of thorough ablation experiments are needed.\n3. On page 8, Figure 6 illustrates the relationship between the number of annotated instances and annotation quality for each worker in dataset Music genre. Why are the axes named Aggregation accuracy (%) and Number of aggregated instances?", "other_comments_or_suggestions": "NO", "questions_for_authors": "The proposed TLLC transfers the pretrained Siamese network to the target domain. In the paper, the authors just said: “Specifically, we set up both $f_S$ and $f_T$ as Siamese networks with the same structure (Li et al., 2022).” What are the detailed network structure and parameter settings?\n    Although the authors have already provided a deeper analysis of TLLC to validate its underlying rationality: 1) With and without instance filtering; 2) With and without transfer learning, a group of thorough ablation experiments are needed.\n    On page 8, Figure 6 illustrates the relationship between the number of annotated instances and annotation quality for each worker in dataset Music genre. Why are the axes named Aggregation accuracy (%) and Number of aggregated instances?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lHcj8vWJAz", "reviewer": "Reviewer_1QrC", "summary": "Existing worker modeling-based label completion methods have successfully improved the performance of label completion, but they remain constrained by the insufficient annotated instances per worker. To address this issue, this paper proposes a transfer learning-based label completion (TLLC) method. TLLC begins by identifying all high-confidence instances from the whole crowdsourced data as a source domain to pretrain a Siamese network. Next, TLLC transfers the pretrained network to target domains, where it is fine-tuned using the instances annotated by each worker individually. Finally, TLLC utilizes the new embeddings learned by the transferred network to complete the missing labels for each worker. Experimental results validate the effectiveness and rationality of TLLC.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** This paper uses three algorithms to describe the construction of source and target domains, worker modeling, and label completion, respectively. However, how these three algorithms are combined to form the complete TLLC remains unclear. A framework diagram is needed to provide a complete introduction of TLLC.\n\n**Author Response to Q1:** Thanks for your valuable comments. The complete process of TLLC is as follows: Given a crowdsourced dataset, we first construct the source and target domains. Next, we pretrain a Siamese network on the source domain. Subsequently, for each worker's corresponding target domain, we individually transfer the pretrained network. Finally, we use the transferred network to learn new embeddings for each worker and infer the missing labels based on these embeddings. In essence, this process corresponds to the sequential execution of Algorithm 1, Algorithm 2, and Algorithm 3. In the final version of the paper, according to the reviewer’s comments, we will incorporate a framework diagram to clearly present the complete process of TLLC. Thanks again for your valuable comments.\n\n**Q2:** The current ablation study is not comprehensive. Although the paper validates the rationality of each strategy in TLLC from multiple perspectives, this is not directly reflected in the evaluation criteria of label completion. In my opinion, it is necessary to construct a complete ablation study based on aggregation accuracy.\n\n**Author Response to Q2:** Thanks for your valuable comments. To address the reviewer’s concerns, we conduct an ablation study on the Income dataset for TLLC and its variants (using MV as the label aggregation method). The experimental results are as follows:\n| |TLLC|TLLC1|TLLC2|TLLC3|\n|--|--|--|--|--|\n|Aggregation Accuracy|**74.97%**|71.83%|74.16%|71.66%|\n||\n\nHere, TLLC1, TLLC2, and TLLC3 represent the variants of TLLC without instance filtering, pretraining, and transfer training, respectively. Considering that the aggregation accuracy of MV before completion is 71.17% (as shown in Figure 1), it can be observed that all TLLC variants outperform MV. Meanwhile, each variant performs worse than the complete TLLC, further indicating the superior performance and rationality of TLLC. In the final version of the paper, we will provide a detailed description and discussion of the setup and results of this ablation study. Thanks again for your valuable comments.\n\n**Q3:** In Equation 11, there should be a minus sign before $y_i’$ instead of a comma.\n\n**Author Response to Q3:** Thanks for your valuable comments. In the MSE loss, it should indeed be a minus sign instead of a comma. Meanwhile, to address the reviewer **3DPZ’s** concerns regarding notation, we will revise $x\\_{i1}$ and $x\\_{i2}$ to $x\\_{i}^1$ and $x\\_{i}^2$. Accordingly, in the final version of the paper, we will update Equation (11) as follows:\n$$\n\\mathcal{L}\\_{mse} = \\frac{1}{|\\widetilde{\\mathbf{X}}|^2}\\sum\\_{i=1}^{|\\widetilde{\\mathbf{X}}|^2}(\\widetilde{f}\\_{d}(\\widetilde{f}\\_{g}(\\mathbf{x}\\_{i}^1), \\widetilde{f}\\_{g}(\\mathbf{x}\\_{i}^2)) - y\\_i')^2.\n$$\nThanks again for your valuable comments.\n\n**Q4:** On line 366 in page 7, “dataset Music_genre dataset” should be “dataset Music_genre”.\n\n**Author Response to Q4:** Thanks for your valuable comments. In the final version of the paper, we will correct \"dataset Music_genre dataset\" to \"dataset Music_genre\" on line 366 in page 7. Additionally, we will double-check and improve the writing of our paper. Thanks again for your valuable comments.", "claims_and_evidence": "There are three important claims made in the paper:\n1) Worker modeling has been proved to be a powerful strategy to improve the performance of label completion.\n2) Workers typically annotate only a few instances, which leads to insufficient worker modeling and thus limiting the improvement of label completion.\n3) The proposed transfer learning-based label completion method helps alleviate the issue of insufficient worker modeling.\nIn response to these claims, this paper provides the corresponding evidence as follows:\n1) For Claim 1, this paper summarizes and discusses existing label completion methods in the introduction and related work sections. The latest label completion methods have indeed achieved impressive results by leveraging worker modeling.\n2) For Claim 2, the paper cites an existing work (Jung & Lease, 2012) to emphasize that, in real-world scenarios, each worker typically annotates only a few instances. Furthermore, the description of real-world datasets in Section 4.1 similarly supports this phenomenon. Based on this phenomenon, existing worker modeling-based label completion methods are indeed constrained by the insufficient annotated instances per worker.\n3) For Claim 3, the experimental results presented in Section 4 demonstrate the effectiveness of the proposed method. Specifically, Figures 1 and 2 validate the effectiveness of TLLC for improving the performance of label completion, while Figure 4 independently validates the effectiveness of transfer learning for insufficient worker modeling.", "methods_and_evaluation_criteria": "Yes. In the proposed method, using transfer learning to address the issue of insufficient worker modeling is reasonable. Regarding evaluation criteria, this paper adopts the aggregation accuracy, which is commonly used in other label completion studies.", "theoretical_claims": "Yes. The theories and corresponding proofs presented in the paper support the effectiveness and rationality of the proposed method.", "experimental_designs_or_analyses": "Yes. The experimental section first validates the effectiveness of TLLC through comparative experiments and significance tests. Then, the rationality of TLLC is verified through ablation studies on each strategy. Finally, potential limitations of TLLC are analyzed by discussing its abnormality.", "relation_to_broader_scientific_literature": "This paper is the first work to introduce transfer learning into label completion, addressing the impact of insufficient worker modeling on label completion, thereby further improving the performance of label completion.", "essential_references_not_discussed": "No. All essential references have been cited/discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n1) This paper reveals a critical limitation of existing label completion methods: insufficient worker modeling due to insufficient annotated instances per worker. By introducing transfer learning, the proposed method effectively addresses this issue.\n2) The use of a Siamese network for both pretraining and fine-tuning is innovative in worker modeling. The idea of constructing source and target domains from the same crowdsourced data and leveraging high-confidence instances for pretraining adds robustness to the method.\n3) The paper provides theoretical proofs to support the claims of the paper (Theorems 3.6, 3.7, and 3.8). These theorems and proofs strengthen the claims about reduced generalization error and robustness against noise.\n4) This paper provides extensive experiments to validate the effectiveness and rationality of TLLC. The paper first validates the effectiveness of TLLC through comparative experiments and significance tests. Then, the rationality of TLLC is validated through ablation studies on each strategy. Finally, potential limitations of TLLC are analyzed by discussing its abnormality.\n\nWeaknesses:\n1) This paper uses three algorithms to describe the construction of source and target domains, worker modeling, and label completion, respectively. However, how these three algorithms are combined to form the complete TLLC remains unclear. A framework diagram is needed to provide a complete introduction of TLLC.\n2) The current ablation study is not comprehensive. Although the paper validates the rationality of each strategy in TLLC from multiple perspectives, this is not directly reflected in the evaluation criteria of label completion. In my opinion, it is necessary to construct a complete ablation study based on aggregation accuracy.", "other_comments_or_suggestions": "I have found few typos as follows:\n1) In Equation 11, there should be a minus sign before ${y}_i’$ instead of a comma.\n2) On line 366 in page 7, “dataset Music_genre dataset” should be “dataset Music_genre”.", "questions_for_authors": "See the Weaknesses parts：\n1) This paper uses three algorithms to describe the construction of source and target domains, worker modeling, and label completion, respectively. However, how these three algorithms are combined to form the complete TLLC remains unclear. A framework diagram is needed to provide a complete introduction of TLLC.\n2) The current ablation study is not comprehensive. Although the paper validates the rationality of each strategy in TLLC from multiple perspectives, this is not directly reflected in the evaluation criteria of label completion. In my opinion, it is necessary to construct a complete ablation study based on aggregation accuracy.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WxELrxC42P", "reviewer": "Reviewer_3DPZ", "summary": "To complete the missing labels, this paper proposes a novel label completion method for crowdsourcing by utilizing transfer learning. All high-confidence instances from the original data are selected as the source domain, and a Siamese network is pretrained based on the instances coming from the source domain. After transferring the pretrained network to the target domain, some fine-tunings are applied to obtain the unique characteristics of each annotator, also called worker modeling. Corresponding theorems are provided to prove that the proposed transfer learning-based method can reduce the generalization error. Experimental results and related analysis also demonstrate the effectiveness of the proposal.\n\n## update after rebuttal\nThe author's response satisfactorily resolves my concerns, and upon considering the feedback from the other reviewers, I support the acceptance of this paper. Thus, I keep my initial rating unchanged.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** There appears to be an error in Equation (11). In the MSE loss, there should be a minus sign ‘-’ instead of a comma ‘,’. Additionally, the notation for $x_{i1}$ and $x_{i2}$ is somewhat confusing to me, as it could easily be interpreted as the values of two attributes for the same instance $x_{i}$.\n\n**Author Response to Q1:** Thanks for your valuable comments. In the MSE loss, it should indeed be a minus sign instead of a comma. Meanwhile, to address the reviewer’s concerns regarding notation, we will revise $x\\_{i1}$ and $x\\_{i2}$ to $x\\_{i}^1$ and $x\\_{i}^2$. Accordingly, in the final version of the paper, we will update Equation (11) as follows:\n$$\n\\mathcal{L}\\_{mse} = \\frac{1}{|\\widetilde{\\mathbf{X}}|^2}\\sum\\_{i=1}^{|\\widetilde{\\mathbf{X}}|^2}(\\widetilde{f}\\_{d}(\\widetilde{f}\\_{g}(\\mathbf{x}\\_{i}^1), \\widetilde{f}\\_{g}(\\mathbf{x}\\_{i}^2)) - y\\_i')^2.\n$$\nThanks again for your valuable comments.\n\n**Q2:** Some parts of the description are not clear enough. For example, what is the ‘g’ of the time complexity O(g) refer to? If g is not a kind of quantity, O(N^2g) is not a correct expression.\n\n**Author Response to Q2:** Thanks for your valuable comments. In our paper, $\\widetilde{f}\\_{g}$ is explicitly defined as the network structure used for learning new embeddings in a Siamese network. The time complexity of $\\widetilde{f}\\_{g}$ is not determined by Algorithm 2 but is instead related to the scale of $\\widetilde{f}\\_{g}$. Therefore, we denote its time complexity as $O(g)$. In the final version of the paper, we will explicitly clarify that $g$ represents the scale of $\\widetilde{f}\\_{g}$. Additionally, we will carefully review and refine other descriptions to improve clarity and precision. Thanks again for your valuable comments.\n\n**Q3:** In the appendix, it is shown that the missing rates for all three datasets are very high, exceeding 0.85. Do the different missing rates have an impact on the method proposed in this paper? What kind of impact do they have?\n\n**Author Response to Q3:** Thanks for your valuable comments. The missing rates significantly impact TLLC’s performance. Specifically, TLLC improves label completion by addressing insufficient worker modeling. A higher missing rate increases the likelihood of insufficient modeling, making TLLC’s advantages more pronounced. Conversely, as the missing rate decreases, TLLC’s effectiveness relative to WSLC gradually diminishes. To validate this analysis, we conduct simulated experiments on the Income dataset. We simulate 40 workers annotating the dataset, where each worker’s annotation quality is randomly generated from a uniform distribution of [0.55, 0.75]. The missing rate is controlled by adjusting workers’ annotation probabilities, ensuring it varies from 0.9 to 0.1 in intervals of 0.2. When the label aggregation algorithm is fixed as MV, the label completion performance of WSLC and TLLC is as follows:\n|Missing Rates|0.9|0.7|0.5|0.3|0.1|\n|--|--|--|--|--|--|\n|WSLC|70.17%|80.33%|81.67%|**92.67%**|**94.83%**|\n|TLLC|**71.16%**|**81.16%**|**82.33%**|92.33%|94.00%|\n||\n\nThese results confirm our analysis: when the missing rate exceeds 0.5, TLLC outperforms WSLC. However, as the missing rate decreases further, WSLC becomes more effective than TLLC. In the final version of the paper, we will thoroughly describe and discuss these experiments about missing rates. Thanks again for your valuable comments.\n\n**Q4:** In Line 211, ‘we’ should be ‘We’. In Equation (3), it would be better to change the representation of the average value P, since the overline has been used to denote the complement set.\n\n**Author Response to Q4:** Thanks for your valuable comments. In the final version of the paper, we will revise the statement in line 211 from \"we set $y_{ij}'$ to 0 if $l_{i} = l_{j}$\" to \"We set $y_{ij}'$ to 0 if $l_{i} = l_{j}$\". Additionally, to avoid ambiguity caused by using the overline to represent both the average value and the complement set, we will consistently use $\\mu$ to denote the average value. Accordingly, Equation (3) will be revised as follows:\n$$\n \\mu\\_{c\\_q} = \\frac{\\sum\\_{i=1}^{N}\\delta(\\hat{y}\\_i, c\\_q)P(\\hat{y}\\_i|\\mathbf{L}\\_i)}{\\sum\\_{i=1}^{N}\\delta(\\hat{y}\\_i, c\\_q)}.\n$$\nEquations (4) and Table 2 will also be updated accordingly. Thanks again for your valuable comments.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. The theoretical claims are all about reducing the generalization error, and the proofs are correct.", "experimental_designs_or_analyses": "Yes. This paper designs the related experiments to show the effectiveness of the proposal, and gives some analysis to explain the advantages or disadvantages on different situations.", "relation_to_broader_scientific_literature": "This paper discusses the effectiveness of transfer learning for crowdsourcing problem, and proposes a method of how to construct source domain data, which can theoretically guarantee the reduction of generalization error. The source domain construction method and theoretical analysis can be easily applied to other transfer learning-based application areas.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1 This paper studies the label completion problem in crowdsourcing area, which is a very important issue as usually the missing proportion is very high in real-world scenarios. \n2 This paper proposes a label completion method based on transfer learning. The comparative experiments demonstrate that the idea is simple yet efficient.\n3 This paper theoretically discusses the generalization error reduction by utilizing transfer learning into label completion problem.\n4 This paper constructs a novel source domain algorithm, which can be easily extended to various application scenarios. The theoretical analysis guarantees its effectiveness.\n\nWeaknesses:\n1 There appears to be an error in Equation (11). In the MSE loss, there should be a minus sign ‘-’ instead of a comma ‘,’. Additionally, the notation for $x_{i1}$ and $x_{i2}$ is somewhat confusing to me, as it could easily be interpreted as the values of two attributes for the same instance $x_i$.\n2 Some parts of the description are not clear enough. For example, what is the ‘g’ of the time complexity O(g) refer to? If g is not a kind of quantity, O(N^2g) is not a correct expression.\n3 In the appendix, it is shown that the missing rates for all three datasets are very high, exceeding 0.85. Do the different missing rates have an impact on the method proposed in this paper? What kind of impact do they have?", "other_comments_or_suggestions": "1 In Line 211, ‘we’ should be ‘We’.\n2 In Equation (3), it would be better to change the representation of the average value P, since the overline has been used to denote the complement set.", "questions_for_authors": "1 Some parts of the description are not clear enough. For example, what is the ‘g’ of the time complexity O(g) refer to? If g is not a kind of quantity, O(N^2g) is not a correct expression.\n2 In the appendix, it is shown that the missing rates for all three datasets are very high, exceeding 0.85. Do the different missing rates have an impact on the method proposed in this paper? What kind of impact do they have?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method to improve label completion in crowdsourcing tasks called TLLC. Its advantages include: (1) innovative use of transfer learning by repurposing high-confidence labels within the same dataset as the source domain; (2) siamese network architecture which allows for effective learning of similarities between instances; (3) theoretical backing with proofs for robustness and reduced generalization error; and (4) strong empirical performance across multiple datasets. There also exist some potential disadvantages, including high computational cost and performance degradation as the missing label rate decreases. Overall, this is a qualified paper and I recommend acceptance.
Decision: Accept (spotlight poster)