Towards a Foundation Model for Crowdsourced Label Aggregation
Authors: Hao Liu
Jiacheng Liu
Feilong Tang
Long Chen
Jiadi Yu
Yanmin Zhu
Qiwen Dong
Yichuan Yu
Xiaofeng Hou
Venue: ICLR 2026 Poster (2026)
Keywords: Label Aggregation
Primary area: foundation or frontier models, including LLMs
TL;DR: We created CrowdFM, a foundation model for label aggregation, which is pre-trained once on synthetic data to accurately combine noisy crowdsourced labels on any new dataset without retraining.

Abstract: Inferring ground truth from noisy, crowdsourced labels is a fundamental challenge in machine learning. For decades, the dominant paradigm has relied on dataset-specific parameter estimation, a non-scalable method that fails to transfer knowledge.  Recent efforts toward universal aggregation models do not account for the structural and behavioral complexities of human-annotated crowdsourcing, resulting in poor real-world performance. To address this gap, we introduce CrowdFM, a foundation model for crowdsourced label aggregation. At its core, CrowdFM is a bipartite graph neural network that is pre-trained on a vast, domain-randomized synthetic dataset to learn diverse behavioral patterns. By leveraging a size-invariant initialization and attention-based message passing, it learns universal principles of collective intelligence and generalizes to new, unseen datasets. Extensive experiments on 22 real-world benchmarks show that our single, fixed model consistently matches or surpasses bespoke, per-dataset methods in both accuracy and efficiency. Furthermore, the representations learned by CrowdFM readily support diverse downstream applications, such as worker assessment and task assignment. Codes are available at https://github.com/liiuhaao/CrowdFM.

Published as a conference paper at ICLR 2026
TOWARDS A FOUNDATION MODEL FOR CROWD- SOURCED LABEL AGGREGATION
Hao Liu', Jiacheng Liu Feilong Tang Long Chen', .iadi Yu', Yanmin Zhu',
Qiwen Dong" Yichuan Yu', Xiaofeng Hou
East China Normal University
The Hong Kong University of Science and Technology
Shanghai Jiao Tong University
*Simon Fraser University
'Huawei Technologies
ABSTRACT
Inferring ground truth from noisy, crowdsourced labels is a fundamental chal- lenge in machine learning. For decades, the dominant paradigm has relied on dataset-specific parameter estimation, non-scalable method that fails to trans- fer knowledge. Recent efforts toward universal aggregation models do not ac- count for the structural and behavioral complexities of human-annotated crowd- sourcing, resulting poor real-world performance. To address this gap, we introduce CrowdFM, foundation model for crowdsourced label aggregation. At its core, CrowdFM is a bipartite graph neural network that is pre-trained vast, dommiin-andomizzed synthetic dataset to learn diverse behavioral pat- terns. By leveraging a size-invariant initialization and attention- -hased message passing, learns universal principles of collective intelligence and generalizes to new, unseen datasets. Extensive experiments on 22 real-world benchmarks show that our single, fixed model consistently matches or surpasses bespoke, per-dataset methods in both accuracy and efficiency. Furthermore, the repre- sentations learned by CrowdFM readily support diverse downstream applica- tions, such as worker assessment and task assignment. Codes are available at https:/ wtt.uuubocoom//icaaa/aaaoooo CrrodddF
INTRODUCTION
Crowdsourcing enables efficient collection of large-scale labeled datasets by distributing annotation tasks to non-cxpert workers (Howe et al., (006). However, variations in worker expertise often pro- duce noisy and conflicting labels (Sheng & Zhang 2019 Label aggregation serves a fundamental step to infer reliable ground truth from such inputs (Zheng et al. 2017 Zhang et al. 2024b a).
For decades, two conflicting paradigms have dominated this field. On one hand, Majority Voting (MV) stands as the de facto industry standard for scalable deployment (Penrose) 1946). Its un- matched simplicity ensures high scalability and, crucially, retraining-free nature, allowing to be applied universally across any dataset. However, its assumption of uniform worker quality leads to suboptimal accuracy in heterogeneous environments [Tullock] 1959). On the other hand, in pursuit of higher accuracy, a vast body of advanced methods has emerged, from probabilistic models (Dawid Skene, 1979. Whitehil al. 2009) to deep learning approaches
ang Yin et al., 2017; Wu et al. [023a]] Liu et al.) (024). Yet, these methods sacrifice the very properties that make MV prac- tical. They are all confined to a dataset-specific paradigm: they require learning dataset-specific parameters from scratch, rendering them non-scalable, brittle, and unable to transfer.
Bridging this gap requires a model that combines the accuracy of advanced methods with the scal- ability and universality of MV. A promising solution is foundation model for crowdsourced label aggregation that generalizes across datasets without dataset-specific retraining. HyperLM (Wu et al.) 2023b) pioneered cross-dataset GNN aggregation but fails to address the structural heterogeneity in-
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
trinsic to crowdsourcing settings. Specifically, its architecture lacks explicit worker-task modeling and its training relies on uniform synthetic data that misaligns with real-world patterns.
To overcome these limitations, realizing truly transferable crowdsoureing foundation model faces two main challenges. First, universal crowdsourcing representation: datasets vary widely in the number of workers, tasks, and label options, and may exhibit diverse annotation patterns. A universal model must flexibly encode any such configuration into a meaningful representation that captures worker and task heterogeneity, enabling effective generalization downstream adaptation. Second, realistic synthetic data generation: foundation models require large-scale pre-training resources, but existing crowdsoureing datasets are limited. The training data must faithfully reflect real-world crowdsourcing patterns to support robust transfer.
In response to these challenges, we introduce CrowdFM, a foundation model for crowdsourced label aggregation. CrowdFM employs a graph neural network that explicitly models workers, tasks, and label options, naturally capturing structural heterogeneity across crowdsoureing datasets. To address realistic synthetic data generation, we design domain-randomized generator that creates diverse scenarios closely matching real crowdsourcing datasets. Pre-training on this data enables CrowdFM to learn robust, transferable aggregation principles. The main contributions of this work are summarized as follows:
We propose CrowdFM, a foundation model based on a bipartite graph neural network for crowdsourced label aggregation that explicitly represents workers, tasks, and options, achieving generalization across heterogeneous datasets without dataset-specific training.
We develop a synthetic crowdsourcing data generator that produces diverse datasets reflect- ing real-world crowdsourcing patterns, enabling the model to pretrain on varied scenarios and learn aggregation principles transferable to unseen datasets.
Extensive experiments show that CrowdFM achieves performance comparable to or su- perior to state-of-the-art aggregation methods, while remaining efficient, stable, and retraining-free. addition, can be readily applied to multiple downstream crowdsourcing applications, highlighting its versatility in practical workflows.
2 PRELIMINARY
Crowdsourced data. We consider crowdsourcing setting with N tasks T (4....., (N) and M workers W = ( w.r. where each task ly has a true label y/ E O = (1..... K}. Each annotation dij is an observed label provided by worker wy task ty. forming the annotation set A = [w., ij, am;) }. The goal of label aggregation is to infer the true labels y {u;} from A.
Dataset Sppeift modeling in conventional approach. Conventional approaches adopt a per- dataset modeling paradigm: for each dataset Dís) = (A((), ,(+)), they define a generative model p(A(.) (())) and estimate dptaset- latent variables (+)) via maximum likelihood:
e (+) arg max log
lanxlogp(AA' ((A(+) ()))), 5o = ((a)
((()))
(1)
where ())) denotes the learned latent variables such as worker abilities and task difficulties for dataset ((s), and (+) represents the inference procedure applied within the same dataset using the fitted model. Since no parameters are shared across datasets, these models must be re-estimated from scratch for every new deployment.
Cross-dataset generalization in Approach. In contrast, we follow cross-dataset gener- alization paradigm. We learn single, parameter-shared aggregation function Fe A y by minimizing the expected loss over a distribution of potential datasets PD:
θ arg mij DoymjnEELAyye Ep(Ay)<pp [(),,)))))), 5r Fo- (A')..
The model Fe- is fixed after pretraining and deployed zero-shot datasets without estimation. This design enables direct, retraining-fiee free inference across diverse domains, funda- mentally differing from per-dataset estimation.
2

---

Published as a conference paper at ICLR 2026
1. Crowdsourced Data Generation
2. Pretrain Synthetic Datasets
Sample Global Structure
Sample le Worker/Task Heterogeneity
W1
W2
W
W2
83
Y1. Fu.Bi,a,.C
Size- Invarinn Initialization
Task Assignment
Generate Answers
200 M
4u Rij
[t+3]
XL
(1)
Attention Network
Compute Pu
[++1)
Density
Zay
Label Aggregation
Worker/Tass Assessment
Task Assignment
Sanple Answer Ry
Loads
3. Multiple Downstream Adaptations
Figure 1: The overall framework of CrowdFM.
3 METHODOLOGGY
We propose CrowdFM, framework for learning generalizable crowdsourcing aggregation models through synthetic data pretraining. The overall pipeline, illustrated in Figure comprises three key components: generating diverse synthetic crowdsourced datasets, pretraining the model on these datasets to learn robust aggregation rules, and applying the pretrained model for label aggregation as well as other downstream applications.
3.1 SYNTHETIC CROWDSOURCED DATA GENERATOR
Public available real-world crowdsourcing datasets are highly scarce, making them insufficient for pretraining robust aggregation models. To address this, we design synthetic data generator that produces diverse crowdsourcing datasets by randomizing key aspects of the data, including global structure, worker and task characteristics, assignment patterns, and annotation generation.
Global structural randomization. To ensure that models can adapt to crowdsourcing scenarios of varying scales and sparsity levels, randomize the global structure of each synthetic dataset. This exposes models to environments with diverse combinations of size and labeling density, encourag- ing the development of robust aggregation behaviors. In practice, for each dataset independently sample the number of tasks N, workers M, answer options K, and the expected number of annota- tions per task A from broad, realistic ranges.
Behavioral heterogeneity. Real-world crowdsourcing datasets consistently show significant vari- ation in worker reliability and task difficulty, both of which strongly influence annotation quality. To model this heterogeneity, we sample worker ability as 0: ~ N(uo, az)). Task difficulty 83 is drawn from A ((p, o²)), task discrimination aj from Eniform(Aminn Ommxx). ), and task guessing rate C from Uniform( (1/K, Cuppa). Each task's 's true label y; is sampled from {l..... K}. Notably, the parameters governing these distributions are themselves randomly drawn for each dataset, producing heterogeneous worker and task profiles and more realistic annotations.
Task assignment mechanism. Crowdsourcing typically exhibits long-tailed participation, where a few workers contribute many labels while most contribute only few. To capture this, we draw each worker's labeling capacity L, from heavy-tailed distribution, ensuring heterogencous partici- pation levels across workers. For each task fj. the number of assigned annotators ni, drawn from Poisson distribution parameterized by A, capturing the variability in annotation density across tasks. Annotators for each task are selected from workers with remaining capacity. This mechanism pro- duces naturally uneven annotation coverage, exposing models to diverse participation patterns that miror real-worle data.
3

---

Published as a conference paper at ICLR 2026
Annotation generation via the 3PL response model. Observed annotation labels should reflect both worker skill and task properties, while still allowing for random errors. We adopt the three-
parameter logistic (3PL) model from Item Response Theory (DeMars 2010 to capture this. For
each worker-task pair (wi, 12), the probability of a correct annotation is
Poj 9j c,- (0, -0)-
(3)
where (2) (1 [1++ (1)= denotes the logistic function and D is scaling constant. The observed label aij is generated according to this probability, taking the true label yj with probability P.j. and with probability 1 Piy. incorrect option is randomly chosen from the remaining K labels.
Generated dataset output. The final synthetic dataset is D = (A,3), where A = {(w,, (,, aty)} contains all collected annotations and y {yy} contains the corresponding ground-trun labels. The full generation procedure is summarized in Algorithm[[ Each dataset instance is independently sampled from broad distributions, exposing models to varying scales, sparsity levels, and noise patterns. This encourages the learning of aggregation rules that generalize across datasets rather than overfitting to a fixed setting.
3.2 GNN STRUCTURE FOR CROSS-DATASET GENERALIZATION
Initialization of representations. In real-world crowdsourcing datasets, workers and tasks typi- cally have no additional features beyond their IDs. Traditional aggregation methods often rely on dataset-specific information, such annotation statistics one-hot identity features, which limits them to the given dataset and prevents generalization to datasets of different sizes. To overcome this, our model adopts size-invariant initialization strategy: all worker nodes share the same learnable vector zw Rd. all task nodes share another vector Rd,, and option nodes are independently initialized for each category from a dimensiimnsi Gaussian distribution.
(0) L:w = Fw (Vw; E W), tt Ie (VL, T),
=00 ~ NN0, Ia) (Vow E O).
(4)
This design reflects the idea that workers and tasks are indistinguishable before any annotations are observed, and their differences emerge once relational information is incorporated, without intro- ducing dataset-specific priors. Mcanwhile, The random initialization of option embeddings ensures sufficient diversity distinguish among candidate labels regardless of the number of options.
Attention-based encoder. Starting from the size-invariant initialization, workers and tasks are treated as nodes in a heterogeneous graph. Rather than learning dataset-specific emheddings, the model allows their representations evolve solely through relational evidence encoded in observed annotations. The encoder propagates information across the graph with L layers attention-baat aggregation, gradually differentiating otherwise identical worker and task nodes according to their annotation patterns.
At layer for each observed annotation (w,,t, azz) A, construct triple representation:
(8) Fw;
2gj Zo.z |e 3d
(5)
and compute queries, keys, and values via type-specific linear projections:
Lij
W. W. + by
kij
Whh (1) + bb, Mi W.h (1) + by.
(6)
Attention weights are obtained by scaled dot-product and normalized over all annotations incident to the same center node:
(gbg.kig) Ki))
= softmax
Vd
The embeddings of workers and tasks are then updated by aggregating values with these attention weights, followed by residual connection and layer normalization:
2 (++1) 1) = LayerNorm
1
+
Σ
ag
Dij
(8)
(UJ)EN

---

Published as a conference paper at ICLR 2026
where the summation is restricted to the neighborhood A of the corresponding worker task node.
Repeating this process for layers allows the model to encode progressively richer relational pat- terns, enabling inference over workers and tasks without introducing dataset-specifie priors.
Aggregated label prediction. After obtaining refined embeddings from the encoder, we compute aggregated label predictions by combining task and option embeddings. For each task J, its embed- ding 2,, is concatenated with all option embeddings 2o and passed through shared feed-forward network to produce logits for each task-option pair:
l,a a( Ætys Zoz
This design allows the model to handle tasks with any number of candidate options, while capturing task-option interactions to facilitate accurate label aggregation. The predicted probability distribu- tion over options for task iÚ is then obtained via softmax, and the aggregated label is selected the option with the highest probability:
exp(lxt) Pyk= EK=n exp((xx))
D3 arg max Pzk-
(10)
3.3 PRETRAINING AND INFERENCE
To enable cross-dataset generalization, the model pretrained on a collection of annotation datasets { (D(>) (A(*), ),)))))))))))))))))))))))) produced by the synthetic data gener-
Pretraining on synthetic data.
ator (Section 3.1. The encoder and the prediction network are optimized jointly by minimizing the average cross-entropy loss across all synthetic datasets:
K
(x) kooo log Pyk
C= S
(1I)
where N₂ and Kg are the number of tasks and options in dataset D(s), respectively. The function 1 denotes the indicator function, which equals the condition is true and 0 otherwise. The term Pjk denotes the predicted probability of option oę for task tú in dataset ((s)
Inference datasets. We denote the aggregation pipeline, including embedding initializa- tion, message passing, and prediction, as a function F(.). After pretraining, the parameters are fixed obtain 1" (-). Given a new annotation set A', the aggregated labels computed
y (AA',
(12)
where y = {6, 1 denotes the set of predicted labels. This procedure enables direct deployment unseen datasets without any further training or parameter updates.
EXPERIMENTS
Our experimental evaluation spans both the main paper and the appendices. Section 4.2 presents label aggregation accuracy results on 22 real-world crowdsourcing datasets. Section investi- gates CrowdFM's ability learn transferable representations of workers and tasks that generalize to diverse downstream adaptation scenarios. Section 4.4 provides ablation studies and sensitivity anal- yses of key model components and hyperparameters. Appendix SaaudixFFiia includes quantitative analysis comparing synthetic and real-world datasets. Appendix Gvallutes CrowdFM's performance under a range of synthetic dataset configurations.
4.1 EXPERIMENTAL SETUP
Our model is trained on dynamically generated synthetic crowdsourced datasets, where diverse figurations are sampled at each training step to expose the model to wide variety of scenarios. Implementation details of the synthetic dataset generation, along with the specifie parameter ranges
5

---

Published as a conference paper at ICLR 2026
100
MV
CrowdFM
95
+0.20
+1.48
+2.18
+0.04
+0.27
+1.30
90
-0.08
+0.20
+0.45
+0,60
+0.60
85
SP*
PosSent
Fact
Product
RTE
SP
Senti
CF
ZC_us
CF*
WS
90
+12.93
+0.27
+1.0B
+3.70
+9.43
+0.48
+0.18
+0004
+0.04
+0,24
ZC_ all
Dog
Aircrowd6 LabelMe
Adult
Bird
ZC_in
Web
MS
Trec
Face
Figure 2: Accuracy comparison between MV and CrowdFM across 22 datasets. The numbers above the bars indicate the performance improvement.
used, are provided in Appendix For evaluation, we consider 22 real-world datasets drawn from practical human-labeled scenarios across various domains, with full details of each dataset included Appendix [D Since all the considered datasets are intended for classification tasks, accuracy adopted as the primary evaluation metric, which is defined as the proportion of correctly predicted labels among all instances.
4.2
MAIN RESULTS
#Win
Acc.t
Runtime.
P-value
Comparison with majority voting. We conduct a comprehensive evaluation
MV
81.78
0.04
0.00003
of framework 22 real-world crowdsourcing datasets, comparing its
PM
13
80.27
0.47
0.00647
CATD
15
83.06
2.59
0.20700
performance against the majority voting
DS
83.02
5.24
0.31889
(MV) baseline. As shown in Figure 2 our method consistently outperforms MV
BWA
17
83.31
0.10
0.60871
IBCC
83.07
0.12
0.36658
across most datasets, yielding an average improvement of +1.64 percentage points in accuracy. This consistent superiority highlights the strong generalization capability of our approach, capturing the inherent heterogeneity of workers and tasks, while effectively transferring the aggregation strategy learned from synthetic data real-world scenarios.
EBCC
17
84.08
2.95
0.90089
GLAD
82.75
494.26
0. 0.19475
LAA
10
78.42
223.06
0.04935
TiReMGE
80.29
26.77
0.00230
GOVERN
13
82.61
95.43
0.28992
HyperLM
12
80.81
0.88
0.01793
CrowdFM
21
83.41
0.53
Table 1: Performance comparison across real-world crowdsourcing datasets. Win counts indicate the num- ber of datasets where each method outperforms MV. Ac- curacy and runtime averaged over all successfully completed runs (LAA and GOVERN failed on several large datasets due to extremely high memory require- ments). The P-value reports the one-sided p-value from the Wilcoxon signed-ranks test comparing each method against CrowdFM. with lower values indicating stronger significance in difference. Full per-dataset results provided in Appendix E
Notably, our model delivers particularly large gains several benchmarks. The most significant improvements are ob- served on the Web and MS datasets, where CrowdFM surpasses MV by +12.93% and +9.43%, respectively. Substantial enhancements are also achieved on Bird (+3.70%) and RTE (+2. 18%), indicating the effectiveness of our model in aggre- gating reliable labels. Across the remain- ing datasets, CrowdFM maintains consistently strong performance, with only marginal drop Senti of 0.08%, even though Senti deviates from our synthetic training data (Appendix (F). highlight- ing its robustness across diverse crowdsourcing scenarios.

---

Published as a conference paper at ICLR 2026
Importantly, several datasets exhibit domain shifts relative to the synthetic training data, such as differences in size and annotation density (see Appendix Despite these shifts, CrowdFM main- tains performance comparable to or better than MV, consistently demonstrating strong generalization across varying scales and noise patterns. In contrast, many dataset-specific methods often underper- form MV (Appendix [. they tend overfit partieular structures and fail to generalize to other datasets.
Comparison with dataset-specifie methods. We further compare CrowdFM against range of state-of-the-ar dataset- -ataset-specific aggregation methods. As shown in Table | our method achieves the highest number of wins over MV, outperforming MV on 21 out o2 benchmarks. This demonstrates its strong competitiveness and broad applicability in diverse crowdsourcing scenarios. Among the dataset-specific methods, BWA and EBCC achieve the next highest number of wins with 17 datasets. GOVERN, the top deep learning--er based method, wins on 13 datasets. Despite these performances, none match the consistent superiority of CrowdFM across the full set of datasets.
Specifically, CrowdFM achieves an average accuracy of 83.41 %. which is competitive with top- performing models such as EBCC (84.08%) and superior to others including BWA and DS. To assess the statistical significance of these improvements, we conduct the one-sided Wilcoxon signed- ranks test (Demšar (006) between CrowdFM and each baseline. The results show that CrowdFM significantly better than MV, PM, LAA, TiReMGE, and HyperLM. These significant gains confirm the effectiveness of our method. Notably, despite EBCC's marginally higher average accuracy, not statistically significant (p 0.90089), and EBCC incurs
the performance differences significantly higher computational cost (2.95 seconds per dataset), whereas CrowdFM maintains efficient inference only 0.53 seconds per dataset, comparable to lightweight methods.
HyperLM,, a training-free method designed for programmatic weak supervision, fails to adapt to crowdsourcing settings. outperforms MV on only 12 out of 22 datasets and achieves an average accuracy of 80,81, which is notably lower than that of our approach (83.41) and even underperform- ing MV (81.78). With an average runtime of 0.88 seconds, it is notably slower than CrowdFM, which runs in 0.53 seconds. HyperLM also exhibits poor scalability, taking 16.72 seconds on large- scale datasets such as Senti, compared to our method's 5.75 seconds (Appendix [..2). These results confirm that HyperL.M is neither as accurate nor as efficient as CrowdFM and is ill-suited for real- world crowdsourcing applications.
Despite being a deep learning- based method, CrowdFM demonstrates favorable runtime efficiency, running much faster than other deep learning approaches such as LAA (223.06 s), TiReMGE (26.77 s). and GOVERN (91.44 19..46.5 s). while remaining comparable in speed to simpler, lightweight methods such as PM (0.47 s). This efficiency and robustness arise from CrowdFM's ability to learn a gen- eralizable aggregation mechanism capturing real-world crowdsoureing patterns during pretraining, eliminating the need for dataset-specific training or iterative parameter estimation at inference time.
4.3 MULTIPLE DOWNSTREAM ADAPTATIONS
Having demonstrated strong performance in label aggregation, we now explore the broader utility of CrowdFM beyond its primary task. By design, our model learns to capture the heterogeneity inherent in crowdsoureing environments, learning rich representations worker behavior and task characteristics through training diverse synthetic data. This ability aligns with the foundation model paradigm, where a single pretrained network acquires transferable knowledge applicable to multiple downstream adaptations.
For adapting multiple downstream applications, we keep the encoder fixed and only train lightweight cownstream-speccfic heads. These heads are trained once and can be directly deployed new datasets without further adaptation. This design ensures that CrowdFM remains efficient and poployment-friendyy enabling rapid extension diverse downstream applications while preserv- ing the pretrained knowledge.
4.3.1 WORKER AND TASK ASSESSMENT
In crowdsourcing scenarios, identifying skilled workers and difficult tasks is essential for ensuring annotation quality (Daniel al. 2018). In this section, we assess the transferability of the learned
7

---

Published as a conference paper at ICLR 2026
Predicted Worker Ability
Pearson=0.721 Opearman=0.719
Predicted Task Difficuity
Pearson=0:752 Spearman=07786
-1.0
-0.5
0.0
0.5
1.0
True Worker Ability
True Difficulty
Figure 3: Assessment result on synthetic data.
Predicted Worker Ability
Pearson=0.449 050
Predicted Task Difficulty
Pearson=0.60 606 Spearman=0.584
-1
-2
-2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
True Worker Accuracy
True Task Error Rate
Figure 4: Assessment result on real-world data (Web).
representations from me-ttrinnng by applying them to quality assessment tasks: estimating global worker ability and inferring global task difficulty.
Adaptation setup. We extract worker and task embeddings produced by CrowdFM during infer- ence and use them features to train lightweight regression heads, supervised by the ground-truth from synthetic data generator introduced Section 3.1
à; (z(os) C- =
Σ (a, 0,)3, W.WW
-d(()) La T
1
Σ GET
W
(13)
where à; and dj denote the predicted ability of worker w; and difficulty of task lj. respectively; zw; and 24, are the corresponding learned embeddings; 0g and 8y are the ground-truth values from the synthetic generator. The losses La and La are mean squared errors (MSE) computed over all workers W and all tasks T, respectively.
MV (Predictor)
MV (Random)
Performance synthetic data. We evaluate worker and task assessment performance on syn- thetic data where ground truth is available. The Pear-
CrowdFM (Predictor)
CrowdFM (Random)
0.90
0.85
son and Spearman correlation scores show strong agreement between the predicted and true values (Figure 3). indicating that the model successfully captures latent heterogeneity and enables effective assessment, even without explicit supervision for these attributes during pretraining.
0.80
0.75
0.70
0.65
0.60
Generalization to real-world data. We further evaluate the model's ability to generalize to real- world settings using the Web dataset. Since ground- truth worker abilities and task difficulties are un- available, we use individual worker accuracy and task error rate as empirical proxies. As shown in Figure 4 the predictions from CrowdFM trained solely on synthetic data exhibit strong correlation with these observed metrics. This indicates that the model successfully generalizes to real-world data, capturing realistic patterns of worker behavior and task properties.
8000
10000
12000
14000
Figure 5: Comparison CrowdFM and MV under compatibility prediction-basee (solid) and random (dashed) assignment strategies over four rounds on the Web dataset. The x- axis shows the number of assigned worker- task pairs, and the y-axis is aggregation ac- curacy. The lefumost point is the initial 50% observed assignments, and rightmost all eligible pairs assigned, which are identi- cal across strategics and therefore overlap.
8

---

Published as a conference paper at ICLR 2026
84-
75.0
72.5
70.0
2
10
6
32
(a) Ablation on modules
(b) Ablation on layer
(c) Ablation on dimension
Figure 6: Ablation study key modules and hyperparameters of CrowdFM. (a) Effect of attention- based message passing (AT) and the synthetic data generator (SG) on performance. (b) Performance with varying numbers of GNN layers. (c) Performance across different embedding dimensions. The y-axis shows average accuracy over real-world datasets.
4.3.2 TASK ASSIGNMENT
A major challenge in crowdsourcing is to maximize annotation quality under limited budgets, ideally by assigning each task to workers most likely to provide correct responses (Ho & Vaughan 2012)) In this section, we demonstrate that CrowdFM, once pretrained, can be effectively repurposed to predict worker-task compatibility, enabling intelligent task assignment.
Adaptation setup. We apply a lightweight compatibility head Je to the model's output embed- dings. To enhance discrimination among workers for the same task and prevent bias, perform data filtering: for each task ij we sample an equal number of correct and incorrect responses based on agreement with the ground truth yj, ensuring balanced and informative training pairs. The com- patibility head is trained with binary cross-entropy (BCE) objective to predict whether worker will provide the correct label for given task:
Gg = 9o( 2t,), Le -
Σ
log (ei Ilacj W;)),
(14)
[C]
(LJ)E(C)
where [asj, Mj) = if d:j = yΙ, and -1 otherwise. is the set of filtered worker-task pairs used for training, and (-) denotes the sigmoid function.
Evaluation setup. We design controlled evaluation protocol to compare task assignment based on predicted compatibility against a random baseline. The evaluation begins with 50% of the initial worker-task assignments used as historical data. CrowdFM leverages these observed assignments to predict compatibilities for the remaining unassigned worker-task pairs, providing guidance for subsequent allocation decisions.
The allocation is carried out over four sequential rounds. each round, every task is assigned a fixed number of new labeling slots. Under the Predictor strategy, each task selects the top-ranked unassigned workers based on predicted compatibility scores, whereas under the Random strategy, workers chosen uniformly at random fill the slots.
Performance on real-world data. As shown in Figure using compatibility-bseed assignment strategy (Predictor) results in significantly higher accuracy for both MV and CrowdFM compared to random assignment (Random). This demonstrates that CrowdFM supports intelligent task assign- ment in a retraining-free manner, effectively leveraging its pretrained knowledge to guide worker- task allocation.
Notably, the accuracy of MV begins to decline in the third and fourth rounds under the compatibility predictor strategy. This is because higher-quality worker-task pairs are typically allocated in earlier rounds based on the model's predictions. As the process progresses, the remaining pool consists of increasingly noisy assignments, involving more ambiguous or less compatible interactions. While MV is sensitive to this degradation in annotation quality, CrowdFM maintains stable performance, demonstrating its resilience to such challenging cases and highlighting its advantage in handling noisy, real-world annotation patterns.

---

Published as a conference paper at ICLR 2026
4.4 ABLATION STUDY
Ablation on modules We create two variants of CrowdFM: one replaces the attention mechanism with a mean aggregator (w/o AT), and the other a uniformly random generator instead of our synthetic data generator (w/o SG), inspired by (Wu et al. 2023b) (Algorithm 2 As shown in Figure [aa] removing either component degrades performance significantly. The w/o SG variant causes the largest drop in accuracy, underscoring the importance of diverse synthetic data for sim- to-real transfer. The performance drop with w/o AT' confirms that attention is crucial for modeling annotation heterogeneity.
Ablation on hyperparameters We evaluate the impact of GNN depth and embedding dimen- sion d. As shown in Figure 6b performance improves steadily with deeper layers, suggesting that longer-range message passing helps capture annotation patterns. In Figure 66c| we observe that di- mension 2 is insufficient to represent annotator behaviors, dimension 4 provides basic capacity, and higher dimensions lead to consistent improvements. These trends highlight the scalability of our design and suggest further gains with larger configurations.
RELATED WORK
Label Aggregation. Existing label aggregation methods typically estimate dataset-speciftt param- eters to model annotation heterogeneous characteristics. PM (Aydin et al. 2014) estimates worker quality, while CATD (Li al. 2014) quantifies reliability with confidence intervals. GLAD mod- els worker expertise and task difficulty (Whitehill et al. 2009). DS (Dawid & Skene 1979) and IBCC model per-worker confusion matrices, and EBCC (Li al. 2019b) extends IBCC by captur- ing worker correlations. BWA (Li et al. 2019a) adopts Inverse-Gamma priors for label estimation. Deep learning approaches such LAA (L' Li'ang Yin et al.) 2017). TiReMGE (Wu al. 2023a), and GOVERN (Liu et al., 2024) learn worker and task embeddings through self-supervised learning. Despite their complexity, all these methods require training from scratch on each dataset and lack transferability. In contrast, the simplest method MV enables retraining-free inference across datasets but fails to capture annotation heterogeneity.
HyperI.M M (Wu et al. 2023b) shares the retraining- free objective for programmatic weak supervision aggregation but is not a foundation model for crowdsourcing with human annotations: HyperL.M adopts uniform random data generation process, which misaligns with real-worle annotation pat- terns; its wee---errbbaraannotttee to large-scale annotations categories; and it lacks explicit representations of workers or tasks, pre- venting direct application to downstream adaptation.
graph design incurs high computational cost and does not scale
Foundation Model. Recently, foundation models pretrained large-scale corpora have demon- strated remarkable generalization capabilities (Zhou et al., 2024). Notably, large language models (LLMs) (Achiam et al.
2023 Guo al., 2025) excel in natural language understanding and gen-
learning for zero-shot or few-shot adaptation. Concurrently, graph foundation models (Liu al. 2025 Yu et al., (2055) learn transferable structural representations, enabling effective generalization across graph-based tasks. However, existing foundation models are not designed for crowdsourced sceneries. LLMs focus on sequential text and cannot model worker-task relationships or annotation quality (Appendix [ Graph models rely on rich node fea- tures, which are often unavailable in crowd settings. Therefore, these models are fundamentally ill-suited for direct application in crowdsoureed label aggregation.
eration, leveraging in-context
6 CONCLUSION
In this work, we introduce CrowdFM, foundation model for crowdsourced label aggregation. By pretraining a GNN on a wide range of simulated labeling scenarios, learns to capture annotation heterogeneity and generalizes across diverse datasets without dataset-specific training. The model demonstrates strong cross-datases generalization, robustness, and adaptability to multiple down- stream tasks. We hope this work provides a useful foundation for future research, particularly in improving the realism of synthetic data generation and extending the framework to more complex annotation types such continuous or structured labels.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our method uses only synthetic data for training and publicly available datasets for evaluation. It does not involve human subjects or sensitive information.
REPRODUCIBILITY STATEMENT
Implementation details and dataset sources are provided the appendices. Our code and trained models have been open-sourced ensure reproducibility.
ACKNOWLEDGMENT
This work was supported in part by the National Natural Science Foundation of China under Grant 62472176, Grant 62172275, Grant 62472277, Grant 62572309 and Grant 62441225, in part by Shanghai East Talents Program under Grant 2023-177, and part by Huawei Technologies Com- pany, Ltd. under Grant (C20251104033
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0m: 087744 2023.
Bahadir Aydin, Yavuz Selim Yilmaz Yavuz Selim Yilmaz, Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas. Crowdsourcing for multiple-choice question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, pp. 2946-2953, 2014.
Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allah- bakhsh. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Computing Surveys (CSUR), 51(1): 1-::))1140,)) 2018
Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 1979.
DeepSeck-AL. Deepseek-v3 technical report, 2024. URL https::/aaxi...og/////aa 19438
Christine DeMars. Item response theory. Oxford University Press, 2010.
Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux. Zencrowd: leveraging probabilistic reasoning and crowdsoureing techniques for large-scale entity linking. Proceed- ings of the 21 international conference on World Wide Web, pp. 469-478, 2012.
Janez Demăar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan): 2006.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in mlms through reinforce- ment learning. Nature,
2025.
Chien-Ju Ho and Jennifer Vaughan. Online task assignment in crowdsourcing markets. In Proceed- ings of the AAAI conference on artificial intelligence, volume 26, pp. 45-51, 2012.
Jeff Howe et al. The rise crowdsourcing. Wired magazine, 14(6): 176-188, 2006.
Tatiana Josephy, Matt Lease, Praveen Paritosh, Markus Krause, Mihai Georgescu, Michael Tjalve, and Daniela Braga. Workshops held at the first aaai conference on human computation and crowd- sourcing: A report. AI Magazine, 35(2):75-78. 2014.

---

Published as a conference paper at ICLR 2026
Qi Li, Yaliang Li, Jing Gao, Lu Su, Bo Zhao, Murat Demirbas, Wei Fan, and Jiawei Han. A confidence-awand approach for truth discovery on long-tail data. Proceedings of the VLDB En- dowment (PVLDB), 2014.
Yuan Li, Benjamin IP Rubinstein, and Trevor Cohn. Truth inference scale: A bayesian model for adjudicating highly redundant crowd annotations. In The World Wide Web Conference (WWW), 2019a.
Yuan Li, Benjamin Rubinstein, and Trevor Cohn. Exploiting worker correlation for label aggregation in crowdsourcing. In International conference on machine learning (ICML), 2019b.
Hao Liu, Jiacheng Liu, Feilong Tang, Peng Li, Long Chen, Jiadi Yu, Yanmin Zhu, Min Gao, Yangin Yang, and Xiaofeng Hou. Graph contrastive learning for truth inference. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pp. 263-275. IEEE, 2024.
Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip S Yu, et al. Graph foundation models: Concepts, opportunities and challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence 2025.
Jianhua Han Li Li'ang Yin, Weinan Zhang, and Yong Yu. Aggregating crowd wisdoms with label- aware autoencoders. In Proceedings of the 26th International Joint Conference on Artificial In- telligence (ICCAI, 2017.
Winter Mason and Siddharth Suri. Conducting behavioral research on amazon's mechanical turk. Behavior research methods, 44(1):1-23, 2012.
Barzan Mozafari, Purna Sarkar, Michael Franklin, Michael Jordan, and Samuel Madden. Scaling up crowd-sourcing to very large datasets: a case for active learning. Proceedings of the VI.BB Endowment, 8(2):125-136 2014.
NVIDIA. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025. //tttts:///rx//// wwt.:oorgga//////eeee 14444
Lionel S Penrose. The elementary statistics of majority voting. Journal of the Royal Statistical Society, 109(1):53-57, 1946.
Filipe Rodrigues and Francisco Pereira. Deep learning from crowds. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. Learning from multiple annotators: distinguishing good from random labelers. Pattern Recognition Letters, 34(12): 1223-:422814444 1436,, 2013.
Victor S Sheng and Jing Zhang. Machine learning with crowdsourcing: A brief summary of the past research and future directions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 9837-9843, 2019.
Rion Snow, Brendan O' connor, Dan Jurafsky, and Andrew Y Ng. Cheap and fast-but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 confer- ence on empirical methods in natural language processing, pp. 254-263, 2008
Gordon Tullock. Problems of majority voting. Journal of political economy, 67(6):571-579, 1959.
Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. Community- based bayesian aggregation models for crowdsourcing. In Proceedings of the 23rd international conference on World wide web, pp. 155-164, 2014.
Jiannan Wang, Tim Kraska, Michael J Franklin, and Jianhua Feng. Crowder: Crowdsourcing entity resolution. Proceedings of the VLDB Endowment, 5(11), 2012.
Peter Welinder, Steve Branson, Pictro Perona, and Serge Belongie. The multidimennsinal wisdom of crowds. Advances in neural information processing systems, 23, 2010.
Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems (NIPS), 2009.
12

---

Published as a conference paper at ICLR 2026
Gongqing Wu, Xingrui Zhuo, Xianyu Bao, Xuegang Hu, Richang Hong, and Xindong Wu. Crowd- soureing truth inference via reliability-driee multi-view graph embedding. ACM Transactions on Knowledge Discovery from Data (TKDD), 2023a.
Renzhi Wu, Shen-En Chen, Jieyu Zhang, and Xu Chu. Learning hyper label model for programmatic weak supervision. International Conference on Learning Representations (ICLR), 2023b.
XAI
Grok 4 fast model card, 2025. pdf
URL https://data.x: a.ai/
-o----------roo----f------------
Jianxiang Yu, Jiapeng Zhu, Hao Qian, Ziqi Liu, Zhiqiang Zhang, and Xiang Li. Relation-aware graph foundation model. arXiv preprint arXiv:2505. S.ii:2505./2027,, 2025.
Jing Zhang, Victor Sheng, Jian Wu, and Xindong Wu. Multi-class ground truth inference in crowdsourcing with clustering. IEEE Transactions on Knowledge and Data Engineering, 28(4): 1080-1085, 2015.
Wenjun Zhang, Liangxiao Jiang, and Chaoqun Li. Iwbvt: Instance weighting-based bias-variance trade-off for crowdsoureing. Advances in Neural Information Processing Systems, 37:85722- 85741, 2024a.
Wenjun Zhang, Liangxiao Jiang, and Chaoqun Li. Kfnn: K-free nearest for crowdsourcing. Advances in Neural Information Processing Systems, 37:116493-1165111 2024b.
Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. Truth inference in crowdsourcing; Is the problem solved? Proceedings of the VLDB Endowment, 10(5):511-552,
2017.
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang. Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, pp. 1-65, 2024.
Dengyong Zhou, Sumit Basu, Mao, and John Platt. Learning from the wisdom crowds by minimax entropy. Advances in neural information processing systems, 25, 2012.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
The Use of Large Language Models (LLMs)
S2
Synthetic Data Generator Implementation
S2
B.1 Sampling Ranges
S2
B.2 Task Assignment
S2
Experimental Environment
S3
D Real-world Crowdsourced Datasets Information
S4
Per-dataset Experimental Result
S5
E. Accuracy Result
S5
Runtime Result
S6
F
Rim-to-Real Gap
S7
Performance on Synthetic Data
S10
Comparison with Uniformly Synthetic Generation
S11
LLMs-based Label Aggregation
S14
SI

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used GitHub Copilot in Visual Studio Code, powered by models GPT-S mini and Claude Sonnet to assist with code completion and implementation. The generated code was reviewed, modified, and approved by the authors.
We also used GPT-5 and Qwen3-223 235---22-----5555555 to improve the clarity and fluency of the manuscript during the writing process. All text was written and revised by the authors, with LLMs used only for language refinement.
B SYNTHETIC DATA GENERATOR IMPLEMENTATION
This section provides implementation details of the synthetic crowdsoureed data generator described in Section The generator implements the procedure outlined in Algorithm to simulate diverse and realistic crowdsourcing environments through domain randomization.
Algorithm Synthetic Crowdsourced Data Generator
1: Input: Sample ranges Raizes Rwoorerr Russk Rassignment-
2: Sample number of tasks N, workers M, options K, and expected answers per task A from Rsize-
3: Sample worker distribution parameters μıd, dg from Rweker.
4: for i ,..,, .M do
5: Sample ability 0, ~ ((po,
6: end for
7: Sample task distribution parameters Pa, G3, miin mmax: Cupper from Rynsk-
8: for 1- 1,.... N do
9: Sample truth label 1/ ~ Uniform[1....,K]
10: Sample difficulty 3j ~ Non~~((i ((13, o)))
11: Sample discrimination ar ~ Uniform(( Omin 0max).
Sample guessing Cj Uniform(1/K3 Cupper..
13: end for
Sample assignment parameter a from Rassignment-
15: Draw worker loads L; ~ Pareto(a) + 1, normalize so Σ, AA for each task j
17: Sample required answers nj Poisson( A), clip to [1, M).
18: Select ny available workers with L; > 0 uniformly without replacement, and decrement their
loads. end for
19:
20: for each assigned pair (i,j) j] do
21: Compute Pij using Eq. (3).
22: With probability Pij. aij yj; otherwise sample random incorrect option.
23:
end for
Return annotation dataset D (), where A {(w,, tj. an) and {v
B. SAMPLING RANGES
To promote structural and behavioral diversity across synthetic datasets, all key parameters are sam- pled from broad, realistic ranges. These ranges summarized in Table
B.2 TASK ASSIGNMENT
Each worker wi is assigned a capacity L; to reflect heterogencous participation. The capacities are generated
L Pareto(a) + 1,
(15)
where the shape parameter a Uniform (0.2, 1.2) is sampled per dataset. The resulting loads are normalized so that E, La N A, ensuring alignment between total labeling demand and supply.
S2

---

Published as a conference paper at ICLR 2026
Component
Parameter
Sampling Range
Dataset Size
Number of tasks N
[100, 300]
Number of workers M
20, 100]
Number of answer options K
2, 20
Expected annotations per task A
10]
Worker Ability
Mean μe of 0; ~ (N oō)
-3.0, 1.5]
Std 08
0.2, 1.2]
Task Difficulty
Mean Ms of B3 (N o3)
[-2.0,
Std 0g
005,, 1.5)
Task Discrimination
Lower bound O'in
0.55, 1.0)
Upper bound O'ax 0g Uniform((mmin @max
[1.0, 2.0]
Task Guessing Rate
Upper bound Cupper
[1/K, 0.1 + 1/K]
C Uniform( (aifform1//,, (1/K, Cupper)
Task Assignment
Pareto shape parameter a for worker loads
(0.2,1.2
Answer Generation
Scaling constant in 3PL model
1.7
Table 1: Parameter ranges used in the synthetic data generator.
For each task 1j, the number of assigned workers ny is drawn from a Poisson distribution with mean A, then clipped [1, M]: " clwppPoissoonnAA A), 1, M).
(16)
Workers are selected at random without replacement from those with remaining load (L, > 0), and their loads are decremented after assignment.
C EXPERIMENTAL ENVIRONMENT
Our model is trained on a machine equipped with Intel Xeon W-3175X CPU (56 cores, 4.3 GHz), an NVIDIA A100 40GB GPU, and 96 GB RAM. For evaluation, all experiments are conducted on the same machine equipped with an AMD Ryzen 9950X processor (32 threads, 4.291 GHz) and 32 GB RAM. Results averaged over five independent runs with random seeds 42-46 to ensure reproducibility and reliable comparison.
S3

---

Published as a conference paper at ICLR 2026
D REAL-WORLD CROWDSOURCED DATASETS INFORMATION
We evaluate our method on 22 real-world crowdsourced datasets collected from popular platforms such Amazon Mechanical Turk. These datasets span multiple domains, including text (e.g., tex- tual entailment), visual (e.g.. image classification of birds), and audio (e.g., music genre classifica- tion), covering a wide range of modalities and task types.
The datasets include: Adult (Mason & Suri 2012) Aircrowd6 (Zhang et al.
2015). Bind (Welinder
2010), Dog (Mozafani al. (2014). Face (Mozafari al., 2014), LabelMe (Rodrigues &
Pereira 2018), MS (Rodrigues et al., 2013). PosSent (Zheng et al., 2017). Product (Wang et al.
2012
RTE Snow et al. (2008), Trec (Li et al.
2019a), WSS (Wu al., 2023a), Web (Zhou et al,
2012
CF. CF*, SP, SP Venanzi et al. (2014). Fact, Senti (Josephy et al. (2014). ZC_all, jn and
ZC_us
Demartini et al. 2012). The detailed statistics these datasets are summarized Table 2
While the synthetic data generator samples key parameters from broad ranges (Table D. several real-world datasets fall outside these ranges terms size and annotation density. For example, Fact and Senti contain 42,624 and 98,980 tasks, respectively, far exceeding the synthetic range of [100, 300]. Datasets such as Bird, CF PosSent, SP*, and WS exhibit very high annotation densities per task, compared to the synthetic range of [2, 10). These extreme configurations were not included the synthetic generator to maintain computational efficiency and avoid overly sparse or dense settings that uncommon in typical crowdsourcing scenarios. Despite these differences, CrowdFM maintains strong performance, demonstrating robustness to variations in dataset scale and annotation density that differ substantially from the synthetic training distributions.
Dataset
#Worker
#Task
#Option
#Answer
#Answer/#
#Answer/# Task
Adult
825
11,040
92,721
112.39
8.40
Aircrowd6
51
593
1,588
31.14
2.68
Bird
39
108
4,212
108.00
39.00
CF
461
300
1,720
3.73
5.73
CF#
110
300
6,030
54.82
20.10
Dog
109
807
8,070
74.04
10.00
Face
27
584
5,242
194.15
8.98
Fact
57
42,624
216,725
3802.19
5.08
LabelMe
59
1,000
2,547
43.17
2.55
MS
44
700
10
2,945
66.93
4.21
PosSent
85
1,000
2
20,000
235.29
20.00
Product
176
8,315
24,945
141.73
3,00
RTE
164
800
8,000
48.78
10.00
SP
203
4,999
27,746
136.68
5.55
SP*
143
500
10,000
69.93
20.00
Senti
1,960
98,980
290.50
5.75
Trec
762
19,033
88,385
115.99
4.64
WS
111
300
6,000
54.05
20.00
Web
177
2,665
15,567
87.95
5.84
ZC all
78
2,040
21,855
280.19
10.71
7C_.in
25
2,040
11,205
448.20
5.49
ZC.us
74
2,040
12.190
164.73
5.98
Table Summary statistics of 22 real-world crowdsourcing datasets: number workers, number of tasks, number of answer options, total number of answers, average answers per worker, and average answers per task.
S4

---

Published as a conference paper at ICLR 2026
E PER-DATASET EXPERIMENTAL RESULT
E.1
ACCURACY RESULT
Adult
Aircrowd6
Bird
CF
CF
Dog
Face
Fact
MV
75.98
80.84
75.93
87.87
85.27
81.83
63.90
90.17
PM
76.58
76.12
78.70
88.53
85.67
81.91
59.76
89.58
CATD
74.83
80.37
77.78
88.13
86.00
82.01
61.64
88.89
DS
73.57
84.49
87.96
79.67
85.67
84.26
64.04
84.90
BWA
74.17
82,46
75.93
89.33
86.00
83.15
61.82
88.72
IBCC
74.47
77.07
88.89
88.33
85.67
83.89
64.04
87.67
EBCC
74.59
81,45
86.30
88.00
86.07
84.06
62.71
89.03
GLAD
76.58
80.61
76.85
88.00
85.67
83,52
62.84
90.17
LAA
94.28
77.34
92.22
83.87
85.60
84.34
66.06
61.44
TiReMGE
75.26
78.75
55.56
87.87
84.53
81.14
63.29
90.17
GOVERN
70.51
81.69
87.22
90.07
85,80
82.48
66.13
NaN
MyperI.M
78.68
82,29
80.56
86.67
83.00
83.27
66.78
89.58
CrowdFM
76.16
81.92
79.63
88.07
85.87
82.26
64.14
90.21
LabelMe
MS
PosSent
Product
RTE
SP
SPa
Senti
MV
76.64
70.37
93.60
89.66
89.35
88.62
94. 16
88.26
PM
73.76
79.51
95.08
89.81
92.12
89.65
94.20
83.30
CATD
76.60
79.11
95.50
92.66
92.45
91.45
94.36
88.32
DS
79.40
76.57
96.00
93.99
92.75
91.50
94.40
81.50
BWA
77.40
78.57
95.60
91.94
92.75
91.70
94.60
89.00
IBCC
76.40
79.00
96.00
93.83
92.75
91.50
94.40
83.10
EBCC
78.60
78.80
95.70
93.50
93.00
91.22
94.16
86.24
GLAD
76.70
78.43
95.20
92.15
92.62
91.78
94.80
89.40
LAA
62.08
68.03
95.84
75.27
91.75
88.51
94.56
NaN
TiReMGE
75.92
71.60
93.54
89.66
91.68
88.35
94.36
88.84
GOVERN
77.86
76.11
95.48
85.26
93.15
90.07
94.24
NaN
HyperLM
77.70
70.57
93.20
67.05
91.87
88.96
94.40
85.60
CrowdEM
77.12
79.80
95.08
89.93
91.53
89.92
94.36
88.18
Tree
WS
Web
ZC.all
ZC
ZC.us
MV
65.10
85.27
73.03
83.11
74.02
86.08
PM
69.37
85.73
39.40
81.75
73.55
81.81
CATD
60.79
86.00
80.35
82.22
76.91
90.88
DS
70.33
85.67
82.21
79.61
75.74
82.25
BWA
60.44
86.00
82.25
83.48
76.37
91.08
IBCC
70.55
85.67
75.08
79.51
76.96
82.70
EBCC
70.37
85.93
74.36
86.25
78.09
91.23
GLAD
57.49
85.67
79.68
81.52
77.01
83.82
LAA
18.99
85.60
87.45
79.39
72.07
82.12
TiReMGE
64.49
84.47
58.98
83.01
77.97
87.01
GOVERN
65.03
85.00
91.05
78.98
73.03
83.09
HyperLM
66.15
83.33
78.97
77.11
72.25
79.90
CrowdFM
65.14
85.87
85.96
83.38
74.06
86.53
Table 3: Per-method accuracy each dataset. Colors reflect performance relative to the MV base- line (green: superior, red: inferior).
S5

---

Published as a conference paper at ICLR 2026
E.2
RUNTIME RESULT
Adult
Aircrowd6
Bird
CF
CF
Dog
Face
Fact
MV
0.05
0.01
0.00
0.01
0.01
0.01
0.01
0.14
PM
0,80
0.01
0.01
0.01
0.02
0.04
0.02
2.11
CATD
4.11
0.06
0.06
0.05
0.11
0.16
0.09
12.77
DS
7.59
0.14
0.11
0.18
0.34
0.40
0.26
20.18
BWA
0.20
0.03
0.01
0.01
0.01
0.02
0.02
0.24
IBC
0.09
0.01
0.01
0.01
0.01
0.02
0.01
0.10
EBCC
1.20
0.07
0.01
0.11
0.02
0.04
0.06
5.27
GLAD
1276.54
2.39
9.77
10.89
19.21
41.55
42.90
2100.37
LAA
2782.28
13.18
6.40
72.38
17.35
20.80
12.41
237.67
TiReMGE
36.91
3.51
3.77
4.35
4.40
5.18
4.33
104.60
GOVERN
397.52
3.51
2.29
4.30
4.00
6.59
6.13
NaN
HyperLM
0.73
0.02
0.01
0.02
0.04
0.04
0.02
0.58
CrowdFM
0.99
0.03
0.04
0.03
0.06
0.07
0.10
2.25
LabelMe
MS
PosSent
Product
RTE
SP
SPa
Senti
MV
0.01
0.01
0.01
0.04
0.01
0.03
0.01
0.41
PM
0.02
0.01
0.09
0.20
0.04
0.19
0.04
5.48
CATD
0.08
0.07
0.33
0.75
0.15
0.66
0.17
31.55
DS
0.30
0.38
0.55
1.05
0.22
0.92
0.26
74.99
BWA
0.04
0.03
0.02
0.10
0.02
0.10
0.01
0.96
IBC
0.07
0.02
0.02
0.10
0.01
0.06
0.01
1.76
EBCC
0.12
0.19
0.08
0.17
0.04
0.26
0.02
56.10
GLAD
12.36
81.72
75.42
195.45
38.58
227.08
42.83
4729.94 .94
LAA
38.94
42.59
22.20
83.68
25.71
69.29
27.23
NaN
TiReMGE
4.20
4.15
10.84
16.06
5.20
15.00
5.57
280.19
GOVERN
5.00
2.37
12.30
194.46
6.81
66.77
8.59
NaN
HyperLM
0.04
0.06
0.03
0.08
0.02
0.07
0.02
16.72
CrowdFM
0.04
0.04
0.18
0.22
0.07
0.23
0.09
5.75
Trec
WS
Web
ZC.all
ZC.in
ZC.us
MV
0.06
0.01
0.02
0.02
0.02
0.02
PM
0.84
0.02
0.09
0.13
0.06
0.07
CATD
4.29
0.10
0.39
0.43
0.25
0.26
DS
4.65
0.33
1.10
0.64
0.35
0.39
BWA
0.20
0.01
0.05
0.04
0.03
0.05
IBCC
0.08
0.01
0.07
0.03
0.03
0.03
EBCC
0.57
0.02
0.29
0.10
0.08
0.09
GLAD
1475.34
32.67
105.35
140.51
117.52
95.33
LAA
883.01
37.25
169.53
41.55
38.69
42.20
TiReMGE
39.15
4.33
10.33
11.58
7.02
8.38
GOVERN
1121.03
5.03
19.24
16.70
12.88
12.98
HyperLM
0.56
0.04
0.10
0.03
0.02
0.03
CrowdFM
0.93
0.06
0.14
0.17
0.10
0.17
Table 4: Per-method runtime on each dataset, measured in seconds..
S6

---

Published as a conference paper at ICLR 2026
F SIM-TO-REAL GAP
Since the real-world datasets do not provide attributes such as worker quality, task difficulty, dis- crimination guessing parameters defined in our synthetic data generation procedure, sample 10,000 synthetic datasets and compute 16 metrics for comparison with real-world datasets. Specifi- cally. we compare accuracy and load ratios (actual annotation number available number) between synthetic and real data, reporting the mean, standard deviation, 25th percentile (Q25), and 75th percentile (Q75) for both workers and tasks.
The results are shown in Figure to F We visualize the distributions using boxplots overlaid with individual data points representing each dataset. As can be observed, the synthetic data covers the majority of real datasets and exhibits similar distribution pattern. We compute the Wasser- stein distance between synthetic and real distributions, all of which are below 0.15, indicating close alignment. Notably, some real datasets such Senti, Fact, Adult, and Trec are outliers that deviate from the overall pattern. Nevertheless, our method achieves performance on these datasets that is either superior to comparable with MV. In contrast, most other methods underperform MV on these datasets, as demonstrated Table 3
Task Accuracy Mean (Wasserstein: 0.1172)
Task Accuracy Std (Wasserstein: 0.0625)
Sp-
Syntheti Data Real world Data
Product
2C
0.40
Begg
Face
0.35
Labe te
0.6
PinSent
0.30
CP
Aurmowdt
ZC,
MS
Whel
0.25
Dog
0.15
Auu
Synihatic Data
Trac
Adult
0.10
Par
PosSent
Raukwordd Jalu
Pact
Senti
Synthetic Data
Real-world Data
Synthetic Data
Real-world Data
Task Accuracy Q25 (Wasserstein: 0.0938)
Task Accuracy Q75 (Wasserstein: 0.1272)
1.0
Synthetic Data Reak-sorts Data
1.0
Aircrewe
0.8
SP
0.8
Dogg
0.6
WS
0.5
CP
Abccomon
Labellte
0.4
Jo
0.4
0.2
0.2
Synthatic Dute
Adult
Sent
0.0
Addit Tee
0.0
Thec
Synthetic
Synthetic
Figure 1: Comparison of task accuracy distributions.
S7

---

Published as a conference paper at ICLR 2026
Worker Accuracy Mean (Wasserstein: 0.1258)
Worker Accuracy Std (Wasserstein: 0.0529)
Synthatic
RTE
ze_us
Duta
Product
0.6
Dog
Labelite
0.3
Product
AlrcroweG
Bind
Face
W75
Mil SC
0.2
".3 1300
PetSent
0.4
Witt
D60
RTE
0.1
Boc
Adut
byndistic Data Realworrd Data
Adult War2e
Face
Sent
0.0
Synthetic Data
Real world Data
Synthetic Data
Real world Data
Worker Accuracy Q25 (Wasserstein: 0.1480)
Worker Accuracy Q75 (Wasserstein: 0.1123)
1.0
1.0
Product
Spm
HTE
SP*
0.8
N5
Preduck
0.8
CooelMe
Dog
0.6
0.8
Dag.
0.4
0.4
0.2
0.2
Tigs
Synthedt
0.0
Rantt
0.0
Synthetic
Real-world Data
Synthetic Real-world Data
Figure 2: Comparison of worker accuracy distributions.
Task Ratio Mean (Wasserstein: 0. 0.0540)
Task Ratio Std (Wasserstein: 0, 0,0329)
Synuhtatic
Bh
Systhetic
Dato
0.150
0.125
28:31
PosSent
0.050
Auscocaddi
--- 2eg
CF4
po-óll
Roce
Tre:
0.025
Serii
Potéent
Fech
Prodi
4p
Broduct
Serii
Toats
0.000
RTE
Synthetic Data
Heal-world Data
Synthetic Data
Real world Data
Ratio Q25 (Wasserstein; 0.0609)
Ratio Q75 (Wasserstein: 0.0627)
1.0
Syndieeicc Data
1.0
Syndictic Data Real world Data
Real world Data
0.8
0.9
0.6
0.6
0.4
PosSent
Faco
0.4
Fect
0.2
Pact
2C
Wee
0.2
MS
Alroroid
CF
0.0
Adult
0.0
Product
Adul
Synthetic Data
Real-world
Synthatic
Real-dorld Data
Figure 3: Comparison of task load ratios.
S8

---

Published as a conference paper at ICLR 2026
Worker Ratio Mean (Wasserstein: 0.0540)
Worker Ratio Std (Wasserstein: 0.1110)
Symthttic Data
Symmhatic Data world Data
Data
0.35
0.25
ZCun
2C
6
PasSent
0.15
M5
Sp
0.10
Dog
Fasz
RTE
Aicroudt
0.05
Product
Webb
babda
Ck
Bel
Thee
Sant
0.00
Synthetic Data
Real-world Data
Synthetic Data
Real-world Data
Worker Ratio Q25 (Wasserstein: 0.1089)
Worker Ratio Q75 (Wasserstein: 0.0923)
1.0
Synchetic Data
1.0
Synthetic Data
Real world Data
Real world Dats
0.8
0.8
0.6
0.5
0.4
0.4
PosSent
RIE.
Pact
0.2
SPM
FacE
0.2
MSA
Labe
Dog-
Aiccowoo Kogenr Web
0.0
Sennt
0.0
Synthetic
Real-world
Synthetic
Real-world Data
Figure 4: Comparison of worker load ratios.
S9

---

Published as a conference paper at ICLR 2026
G PERFORMANCE ON SYNTHETIC DATA
To evaluate the robustness and generalization of CrowdFM, we conduct comprehensive experiments on synthetic datasets with systematically varied characteristics. Specifically, craft 1,000 syn- thetic datasets, and measure the accuracy improvement of CrowdFM over MV, across five key data properties: worker ability, task difficulty, answer density, category distribution entropy, and number of options. As shown in Figure CrowdFM consistently outperforms MV across most scenarios, demonstrating its effectiveness under diverse conditions.
15
15
10
10
5
0
-1.0
-0.5
0.0
0.5
1.0
1.5
-1.0
-0.5
0.0
0.5
1.0
Worker Ability
Task Difficulty
Accuracy Improvement (%)
15
10
10
0.0
0.1
0.2
0.3
0.4
0.5
0.2
0.4
0.6
0.8
1.0
Answer Density
Normalized Categories Distribution Entropy
10
-5
2.5
5.0
7.5
15.0 Number of Options
10.0
12.5
17.5
20.0
Figure 5: Accuracy improvement of CrowdFM MV across five synthetic data dimensions. The y-axis in all subplots represents the absolute improvement in classification accuracy, expressed in percentage points. The x-axis each subplot corresponds systematically varied property of the synthetic datasets.
S10

---

Published as a conference paper at ICLR 2026
H COMPARISON WITH UNIFORMLY SYNTHETIC GENERATION
To investigate the importance of realistic modeling in synthetic data generation, we implement a baseline approach inspired by the uniform generation strategy used in HyperLM Wu al. (2023b), which assumes that labeling functions start from random guessing and are then calibrated to be better than random. The full procedure detailed Algorithm
We generate 10,000 datasets using this uniform synthesis method and compare their statistical prop- erties with real world datasets across the same 16 metrics as Appendix [F) including accuracy and load ratios for workers and tasks (mean, std, Q25, Q75). However, shown in Figure 66 Figure the distributions exhibit significant discrepancies from real data. Most Wasserstein distances be- tween synthetic and real distributions range from 0.3 to 0.6 across metrics, substantially higher than those produced by our proposed simulator, indicating a large sim to real gap. Models trained on such synthetic data may learn unrealistic assumptions such as independent errors uniform reliability, leading to poor real world generalization.
Algorithm Uniformly Synthetic Data Generator
1: Input: Sample ranges Rizzzs Reoorker: Raskk-
2: Sample number of tasks N, workers M, options K.
3: Generate ground truth labels y ...,.... UN), where yy Uniform (L...., K}.
4: for i-ri=, l..... M do
5: Sample number of assigned tasks k; Uniform(1,A3.
6: Randomly select k; distinct tasks.
7: Set dij ~ Uniform(1. (1.... K}
for the selected k; tasks.
8: Count correct predictions ca-
9: end for
10: Compute minimum required correct count: Cuinn [N/K].
11: Identify well- well-performing workers: = {i G 2 Cmin ).
12: Determine how many underperforming workers need improvement:
A = max (0, M
[B)).
2
13: if A then
14:
From workers not in B, sort by deficit (Cmin -c, in descending order.
Select top A workers with largest deficits.
16: for each selected worker do
Correct up to (Cunn 4) errors by setting Oij V/ random wrong positions.
18: end for
19:
end if
20:
Return annotation dataset D (A, where A = [[(w (,, (ig)) and y = {vj}.
S11

---

Published as a conference paper at ICLR 2026
Task Accuracy Mean (Wasserstein: 0.4275)
Task Accuracy Std (Wasserstein: 0.1591)
Symihaii Data
Symrhatic Date
Rea-world Data
Rea-world Data
Labelhe
RTD WS
20,00 al
0.30
CF
NS
Dog
Product
0.6
Plosaar
0.25
KS
0.70
SP
0.4
0.15
Bird
RTE
Ag
PosSent
0.2
0.10
Adut:
0.05
0.0
Synthetic Data
Real world Data
Synthetic Data
Real world Data
Task Accuracy Q25 (Wasserstein: 0.3287)
Task Accuracy Q75 (Wasserstein: 0.5584)
Synitetic Data
1.0
Synthctic Data Acal no id Duta
Roal-wordd Data
25
Tabalga
0.8
oduct
Product
HTE
Dog
0.8
PosSenc
0.5
Aircrosd
Labebit
24
0.4
0.2
0.2
Fact
0.0
Senti
0.0
Synthetic
Real-world Data
Synthetic
Real-world Data
Figure 6: Comparison of task accuracy distributions. (Uniformly synthetic generation)
Worker Accuracy Mean (Wasserstein: 0 0.4336)
Worker Accuracy Std (Wasserstein: 0 0.0931)
Synihaatic Dato
Synrhbatic
world Data
0.8
ZC US
0.30
20g
Product
0.25
MS
W5
0.6
Archtedd CF-
We
Dog
0.20
0.4
0.15
SP N05Sent
In
RIE
0.10
Ther
0.05
Face
Senti
Synthetic Data
Real-wort world Data
Synthetic Data
Real-world Data
Worker Accuracy Q25 (Wasserstein: 0.4036)
Worker Accuracy Q75 (Wasserstein: 0.5061)
Syndieetcc Data Real world Data
1.0
ZC
Prndaat
PolSeot
Product
NS
ZC, all
Possient
WS
Dog
zC
0.8
Zt_in
CF
Labellfte
ZiC_nn
Pace
0.6
Synthetic
world
"Vee
0.4
vit
0.2
Adut
Fact
0.0
Senti
0.0
siliee
Acult
Synthetic Data
Real-wortd Data
Synthatic Data
Rea Real-world Data
Figure 7: Comparison of worker accuracy distributions. (Uniformly synthetic generation)
S12

---

Published as a conference paper at ICLR 2026
Task Ratio Mean (Wasserstein: 0.4005)
Task Ratio Std (Wasserstein: 0.0424)
Symihati
Synthatic
Rea-woold Data
0.10
Duta
2C_h
0.08
2C_a1
0.06
Face
20,
Afrcroode
Fact
20 Di
Faiter
Doc
0.02
Ligh kie
Labaae
Sernt
The
Mort
rokdo
Product
SP
0.00
WS
Protiic
Nicferl
Dog
Synthetic Data
Real world Data
Synthetic Data
Real world Data
Task Ratio Q25 (Wasserstein: 0.3726)
Task Ratio Q75 (Wasserstein: 0.4330)
1.0
Synihetic Data
1.0
Synthetic Data
Roal-world Data
Rcal id Duta
0.8
0.8
0.6
0.5
0.4
Tace
Pos
0.4
SP
w5
WS
RTE
Airgelidd
0.2
0.2
tabeik
Kety
0.0
Product
0.0
AEB
Synthetic
Real-world Data
Synthetic
Weal-world Data
Figure 8: Comparison of task load ratios. (Uniformly synthetic generation)
Worker Ratio Mean (Wasserstein: 0.4005)
Worker Ratio Std (Wasserstein: 0.1591)
Synuhaatic
dv3
Dato
0.35
20, in
0.25
0.20
M5
0.15
Dog
ZC SP
0.10
Aecrowde
Tree Senti
Artrowac
MS
0.05
Synchetic Data Real world Dats
Acult
Product
Adert Ti oount
LabelMo
0.00
Brft
Sorth
LabelMe
Synthetic Data
Real- Real-world Data
Synthetic Data
Real-worl world Data
Worker Ratio Q25 (Wasserstein: 0.2588)
Worker Ratio Q75 (Wasserstein: 0.5856)
1.0
Syndheticc Data
dira
1.0
Real world Data
0.8
0.9
0.6
0.6
0.4
0.4
ZC, in
PosSont
0.2
PosSent
RyR
Fact
0.2
Bod
LabelMe
Gooedd
0.0
Agelowds
Adult
0.0
Product
Adul
Synthetic Data
Real-world Data
Synthatic Data
Real-world Data
Figure 9: Comparison of worker load ratios. (Uniformly synthetic generation)
S13

---

Published as a conference paper at ICLR 2026
I LLMS-BASED LABEL AGGREGATION
Most Existing crowdsourcing aggregation methods typically rely on dataset-specific parameter esti- mation. this section, explore an alternative, parameter-estimation-- approach that leverages large language models (LLMs) to perform label aggregation without any dataset-specific training.
In this method, the LLM is provided with CSV file of raw annotations (label, csv) and prompts it to generate the predicted aggregated truth CSV (predict csv) in a specified format. The input prompt used guide the LLM is as follows:
LLM Label Aggregation Prompt
You are a precise label aggregation engine for crowdsourced classification tasks.
Input format:
task: integer ID of the task
worker: integer ID of the annotator
answer:
annotation label
Your job:
1. Analyze the annotations and infer the most likely true label for each task.
2. Consider factors such worker reliability, task difficulty, and potential biases when
estimating the truth.
3. Output the aggregated results in CSV format with two columns: 'task' and 'truth'. Only include tasks that appear in the input.
Do not include any explanations, headers beyond 'task,truth', or additional text. Only output the CSV data starting with: trut,
Input:
((1abel.csv)
Output:
We selected three representative LLMs for evaluation: GROK FAST (AA 2025). NEMOTRON NANo 9B V2 (NVIDIA 2025). and DEEPSEEK V3.1 DeeeSSeekAA 2024). with the results pre-
sented in Table
Overall, the performance of LLM-based aggregation is generally poor. Only in few datasets (e.g.. PosSent, SP*, RTE) do some models achieve accuracies close to MV, while on the majority of datasets, all LLMs perform substantially worse. In many cases, accuracy drops dramatically, and some models even fail to produce meaningful outputs.
More critically, LLMs consume very large number of tokens during inference. This excessive token usage not only leads to significant computational and monetary costs but also imposes strict scalability bottleneck, preventing the method from running on larger datasets.
In summary, while L.MSs are foundation models that demonstrate strong capabilities across wide range of scenarios, their application label aggregation still limited. They struggle both effec- tiveness and efficiency: they cannot reliably capture structured information, account for annotation heterogeneity, or infer aggregated labels without incurring excessive computational costs.
MV GROK 4 FAST NEMOTRON NoNo 9B V2 DEEPSERK V3.1
Adult
Accuracy Prompt Tokens
75.98
NaN
NaN
NaN NaN NaN
NaN
NaN NaN
Completion Tokens
NaN
Aircrowd6
Accuracy
80.84
29.51
24.62
29.51
S14

---

Published as a conference paper at ICLR 2026
MV
GROK 4 FAST
NEMOTRON NANO 9B V2
DREPSEEK V3.1
Prompt Tokens
9,797
12,945
9,797
Completion Tokens
15,111
6,151
15,111
Bird
Accuracy 75.93
56.48
44.44
75.00
Prompt Tokens
25,541
32,709
25,428
Completion Tokens
5,153
2,240
436
CF
Accuracy
87.87
38.33
32.00
38.00
Prompt Tokens
10,589
16,369
10,476
Completion Tokens
14,550
3,025
1,204
CF
Accuracy Prompt Tokens
85.27
37.67
6.00
37.33
36,449
51,737
36,336
Completion Tokens
3,230
8,348
1,204
Dog
Accuracy Prompt Tokens
81.83
36.18
30.24
35.81
48,689
70,288
48,576
Completion Tokens
10.478
7,836
3,232
Face
Accuracy
63.90
46.06
30.48
46.06
Prompt Tokens
31,721
45,325
31,608
Completion Tokens
10,930
4,322
2,340
Fact
Accuracy
90.17
NaN
NaN
NaN
Prompt Tokens
NaN
NaN
NaN
Completion Tokens
NaN
NaN
NaN
LabelMe
Accuracy
76.64
14.90
4.40
15.90
Prompt Tokens
15,551
22,210
15,438
Completion Tokens
9,431
7,457
4,004
MS
Accuracy
70.37
20.29
0.00
20.43
Prompt Tokens
17,939
25,403
17,826
Completion Tokens
6,693
1,802
2,804
PosSent
Accuracy
93.60
90.10
NaN
84.40
Prompt Tokens
120,269
NaN
120,156
Completion Tokens
6,553
NaN
4,004
Product
Accuracy
89.66
1.84
NaN NaN
NaN
Prompt Tokens
171,884
NaN
Completion Tokens
3,313
NaN
NaN
RTE
Accuracy
89.25
82.87
53.25
90.62
Prompt Tokens
48,269
68,891
48,156
Completion Tokens
8,872
5,378
3,204
SP
Accuracy Prompt Tokens Completion Tokens
88.62
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
SP*
S15

---

Published as a conference paper at ICLR 2026
MV
GROK 4 FAST
NEMOTRON NANO 9B V2
DREPSEEK V3.1
Accuracy
94.16
91.00
79.20
93.40
Prompt Tokens Completion Tokens
60,269
90,201
60,156
8,863
8,767
2,000
Senti
Accuracy
88.26
NaN
NaN
NaN
Prompt Tokens
NaN
NaN
NaN
Completion Tokens
NaN
NaN
NaN
Tree
Accuracy
65.10
NaN
NaN
NaN
Prompt Tokens
NaN
NaN
NaN
Completion Tokens
NaN
NaN
NaN
WS
Accuracy
85.27
38.00
28.33
38.00
Prompt Tokens
36,269
50,980
36,156
Completion Tokens
2,377
4,515
1,204
Web
Accuracy
73.03
1.93
NaN
20.47
Prompt Tokens
103.395
NaN
103,282
Completion Tokens
13,766
NaN
12.329
7.C_ll
Accuracy
83.11
6.67
NaN
76.86
Prompt Tokens
142,419
NaN
142,306
Completion Tokens
4,923
NaN
15,364
ZC_in
Accuracy Prompt Tokens
74.02
65.34
NaN
59.26
73,154
NaN
73,041
Completion Tokens
13.194
NaN
7,264
ZC.us
Prompt Tokens 86.08
Accuracy
77.45
7.06 113,421
72.99
79,584
79,471
Completion Tokens
12,236
6,978
10.414
Table 5: Accuracy and token usage of LLM-based label aggregation compared to MV across datasets (NaN indicates the model could not produce result).
S16


Reviews:
[{"review_id": "uqv9k29RNr", "reviewer": "Reviewer_mWdt", "summary": "The paper aims at building a foundation model for crowdsourced label aggregation that generalizes across heterogeneous datasets without dataset-specific training. It first generates a synthetic crowdsourced data generator to produce diverse synthetic datasets, and then trains a bipartite graph neural network which can be generalizable to other data and tasks. Experiments show the foundation model is efficient and achieve performance comparable to or superior to state-of-the-art aggregation methods.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Generating the synthetic data and then train a generalizable GNN model to approximate it is a smart idea to unify the data-specific crowdsourcing models. I think it is essentially a type of knowledge distillation, which distills the knowledge from the data generator to a GNN with less input features (e.g. task difficulty/worker ability, which are generally regarded as latent parameters in traditional probabilistic models). \n2. The method is sound, and the presentation is clear.\n3. Experiments show good accuracy with more efficiency.", "weaknesses": "1. The backbone model is relatively simple and not new, but the design is a good fit for crowdsourcing, e.g. the size-invariant initialization for worker/task emebddings. I do not see it as a weakness, but just not a novel contribution. I still like the whole framework of building synthetic data and train a simple domain-indepedent model as foundation models.\n\n2. The possible limitation of the model is it may lack the flexibility to adapt to the case when workers/tasks have their attributes, since these nodes are initialized with fixed pretrained embeddings.\n\n3. Another possible weakness is the limitation of the data generator. Although it is a quite general scheme, there might be other graphical models that it cannot cover; and also the prior distribution of the parameters might cause some bias for the pretrained foundation model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9KfRZDWWAo", "reviewer": "Reviewer_BaKD", "summary": "Traditional methods (e.g., Dawid–Skene, GLAD, EBCC) typically adopt a dataset-specific modeling paradigm: they require training model parameters from scratch for each new dataset, resulting in poor generalization. Meanwhile, Majority Voting (MV), widely used in industry, requires no training but ignores differences in annotators' capabilities, leading to limited accuracy.\nTo bridge this gap, the authors propose a universal aggregation paradigm: by pre-training a bipartite GNN on large-scale, domain-randomized synthetic datasets, the model learns transferable \"collective intelligence\" aggregation rules. During the inference phase, the model can be directly applied to any real crowdsourced dataset without fine-tuning or retraining.", "questions": "Same as weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tStrong paradigm innovation: For the first time, the foundation model concept was introduced into crowdsourced label aggregation, breaking the dataset-specific paradigm that has persisted for decades.\n2.\tThe problem definition has practical significance: Shifting the aggregation of crowdsourced labels from the \"dataset-by-dataset modeling\" paradigm to the \"unified fundamental model\" paradigm aligns with the urgent demand of the industrial sector for scalable and retraining-free systems.\n3.\tThe experiments were thorough: 22 real datasets covered text, images, and audio, and included extreme scale/density scenarios, verifying the generalization ability.\n4.\tThe accuracy rate and running time were reported, and it was pointed out that some methods failed due to insufficient memory", "weaknesses": "1.\tHow can we ensure that the distribution of synthetic data is sufficient to cover the feature space of real crowdsourcing tasks? Is there any performance degradation under certain types of tasks (such as extremely unbalanced ones)?\n2.\tCan CrowdFM be regarded as the specialization of GraphFM in the field of crowdsourcing? Has the pre-training objective of the existing GFM been borrowed?\n3.\tThe performance for extremely large-scale data (such as Senti and Fact) slightly decreases, but the reasons are not discussed.\n4.\tThe absence of ablation experiments makes the contributions of each module unclear\n5.\tAlthough it includes recent works such as EBCC, GOVERN, TiReMGE, etc., some new methods for 2024-2025 are missing (such as Zhang et al., KFNN and IWBVT of NeurIPS 2024 are cited but not used as baselines);\n6.\tThe superparameters such as the number of layers L and dimension d of GNN were not discussed in detail.\n7.\tThe \"Sim-to-Real Gap\" between synthetic data and real data has not been fully quantified\n8.\tHas the model truly learned the \"principle of collective intelligence\", or has it merely fitted the statistical patterns of synthetic data? How to prove.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "enZwvoVcya", "reviewer": "Reviewer_7XU2", "summary": "This paper proposes CrowdFM, the first foundation model for crowdsourced label aggregation. It replaces traditional per-dataset parameter estimation with a pretrained bipartite graph neural network trained on a vast, domain-randomized synthetic dataset. By leveraging a size-invariant initialization and attention-based message passing, it learns universal principles of collective intelligence and generalizes to new, unseen datasets. Experiments on 22 real-world datasets show that CrowdFM matches or outperforms state-of-the-art methods in both accuracy and efficiency, while its learned representations also support worker assessment and task assignment.", "questions": "The same as the weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.The proposed CrowdFM is the first foundation model for label aggregation, enabling cross-dataset generalization without retraining and marking a key step toward universal label aggregation.\n\n2.CrowdFM further exhibits strong versatility, as its pretrained representations can be directly adapted to multiple downstream tasks such as worker assessment and task assignment without additional training.\n\n3.Extensive experiments on 22 real-world benchmarks demonstrate that the proposed single, fixed model consistently matches or surpasses dataset-specific methods in both accuracy and efficiency.", "weaknesses": "1.The paper lacks a systematic quantitative analysis of synthetic and real-world crowdsourced data. Although the authors validate generalization on real datasets, they do not analyze the distributional differences of key parameters (e.g., worker ability θ, task difficulty β, task guessing rate c) between synthetic and real data, making it difficult to assess how well the learned patterns reflect real human annotation behaviors. Besides, how to obtain these sampling ranges?\n\n2.Although the parameters in the synthetic data generator are randomly sampled, the paper lacks a sensitivity analysis of key parameters. Random sampling alone cannot fully verify the model’s robustness across different crowdsourcing conditions. It is recommended to include such an analysis to strengthen the credibility of CrowdFM’s generalization claims.\n\n3.With regard to the comparison results, statistical tests are needed in the comparison results. The detailed description about statistical tests for comparisons of multiple algorithms on multiple datasets can be found from the papers such as Statistical comparisons of classifiers over multiple data sets.\n\n4.The paper lacks ablation studies to verify the individual contributions of key components (e.g., attention-based aggregation, size-invariant initialization, and synthetic data diversity). Without such analysis, it is difficult to determine which design choices are most critical to the model’s performance gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AFzpEQAD1T", "reviewer": "Reviewer_QfN2", "summary": "This paper introduces CrowdFM, a graph neural network-based \"foundation model\" for crowdsourced label aggregation, pretrained on synthetic data to enable retraining-free, cross-dataset generalization. The model is evaluated on 22 real-world datasets and shows strong empirical performance and efficiency. However, the main weakness of this paper is its failure to acknowledge and position itself with respect to highly relevant prior work—specifically, the hyper label model (Wu et al., ICLR 2023). The methodology, motivation, and even the technical design of CrowdFM are very similar to those of the hyper label model, with the only substantive difference being the application domain (crowdsourcing vs. programmatic weak supervision) and the technical details in training data generating and the graph neural network architecture design. The paper overstates its novelty and does not provide a meaningful discussion or comparison to this existing work. As a result, the contribution is incremental and the claims of being the \"first\" to propose such a foundation model are not justified.", "questions": "1. Are the authors aware of the hyper label model[1]? Can the authors clarify the technical and conceptual differences between CrowdFM and the hyper label model, beyond the application domain? Please explicitly discuss this prior work in the related work section, and provide a detailed comparison (both conceptual and empirical) to clarify what is novel in CrowdFM.\n2. How does CrowdFM perform compared to the hyper label model?\n3. What new insights does CrowdFM provide for the crowdsourcing setting on top of what is discussed in[1].\n4. Is CrowdFM able to provide any theoretical guarantees as in[1].\n\n[1] Wu et al., \"Learning Hyper Label Model for Programmatic Weak Supervision\" (ICLR 2023)", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The paper addresses a real and important challenge in crowdsourced label aggregation: the need for scalable, retraining-free aggregation methods that generalize across datasets.\n- The paper provides a synthetic data generation process for training such models.\n- CrowdFM achieves fast inference times, comparable to simple methods like Majority Voting, while outperforming more complex dataset-specific models.", "weaknesses": "1.  Lack of proper acknowledgement of closely related work. The most significant issue with this paper is its failure to properly acknowledge and position itself with respect to existing work that is nearly identical in methodology and motivation. In particular, the ICLR 2023 paper[1] that proposes a hyper label model for programmatic weak supervision that is, in essence, the same as the approach in this paper:\n  - Both solve the label aggregation problem. \n  - Both aim for cross-dataset generalization and retraining-free inference.\n  - Both papers propose a GNN-based model that is pretrained on synthetic data to learn a generalizable label aggregation function.\n  - Both use size-invariant initializations and GNN architectures to handle variable numbers of annotators (workers/LFs) and tasks.\n  - Both demonstrate that their model, once trained, can be applied to new datasets in a single forward pass, outperforming dataset-specific methods in both accuracy and efficiency.\n\nWhile the paper claims to be the \"first foundation model for label aggregation,\" this is not accurate. The hyper label model[1] is a direct precedent, and the technical similarities are substantial. The only notable difference is the application domain: The hyper label model[1] focuses on programmatic weak supervision (labeling functions), while this paper focuses on crowdsourcing (human workers). However, the mathematical setup is essentially the same (a bipartite label matrix with noisy annotators) in these two domains.\nThe lack of discussion, comparison, or even citation of this highly relevant prior work is a serious omission. This gives the misleading impression that the proposed approach is novel, when in fact it is a straightforward adaptation of an existing method to a closely related domain.\n\n2. The paper repeatedly claims to be the \"first\" to propose a foundation model for label aggregation, and to introduce a new paradigm. Given the existence of[1], these claims are overstated.\n\n3.  Given the existence of the hyper label model approach[1], the main contribution of the paper seems to be modifications to the synthetic data generator and GNN architecture.\n\n[1] Wu et al., \"Learning Hyper Label Model for Programmatic Weak Supervision\" (ICLR 2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)