AutoAnnotator: A Collaborative Annotation Framework for Large and Small Language Models
Authors: Yao Lu
Ji Zhaiyuan
Jiawei Du
Yu Shanqing
Qi Xuan
Joey Tianyi Zhou
Venue: Accepted by TMLR ()

Abstract: Although the annotation paradigm based on Large Language Models (LLMs) has made significant breakthroughs in recent years, its actual deployment still has two core bottlenecks: first, the cost of calling commercial APIs in large-scale annotation is very expensive; second, in scenarios that require fine-grained semantic understanding, such as sentiment classification and toxicity classification, the annotation accuracy of LLMs is even lower than that of Small Language Models (SLMs) dedicated to this field. To address these problems, we propose a new paradigm of multi-model cooperative annotation and design a fully automatic annotation framework AutoAnnotator based on this. Specifically, AutoAnnotator consists of two layers. The upper-level meta-controller layer uses the generation and reasoning capabilities of LLMs to select SLMs for annotation, automatically generate annotation code and verify difficult samples; the lower-level task‑specialist layer consists of multiple SLMs that perform annotation through multi-model voting. In addition, we use the difficult samples obtained by the secondary review of the meta-controller layer as the reinforcement learning set and fine-tune the SLMs in stages through a continual learning strategy, thereby improving the generalization of SLMs. Extensive experiments show that AutoAnnotator outperforms existing open-source/API LLMs in zero-shot, one-shot, CoT, and majority voting settings. Notably, AutoAnnotator reduces the annotation cost by 74.15% compared to directly annotating with GPT-3.5-turbo, while still improving the accuracy by 6.21%. The code is available in https://github.com/Zhaiyuan-Ji/AutoAnnotator.

Published in Transactions on Machine Learning Research (12/2025)
AutoAnnotator: A Collaborative Annotation Framework for Large and Small Language Models
Yao Lu
yaolu.ajut@gmaii.co com com
Institute of Cyberspace Security, Zhejiang University of Technology Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology Centre for Frontier Al Research, Agency for Science, Technology and Research
Zhaiyuan Ji
jicaiinaangmaiil.com com
Institute of Cyberspace Security, Zhejiang University of Technology Binjiang Institule of Artificial Intelligence, Zhejiang Universily of Technology
Jiawei Du
duju wattaaredd..... eddu.sg
Centre for Frontier Al Reseorch, Agency for Science, Technology and Research Institute of High Performance Computing, Agency for Science, Technology and Research
Shanqing Yu
yushanging@eejuu.eeee edu.ccm
Institute of Cyberspace Security, Zhejiang University of Technology Binjiang Institate of Artificial Intelligence, Zhejiang University of Technology
Qi Xuan
auongiizzjll.cd edw.cull..uuu.rr
Institute of Cyberspace Security, Zhejiang University of Technology Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology
Joey Tianyi Zhou
Centre for Frondser AI Reseorch, Agency for Science, Technology and Research Institute of High Performance Computing, Agency for Science, Technology and Research
Reviewed on OpenReview: https: oppenrevew,,,.. net/ net/forum? forum? id= LauojjjjAA
Abstract
Although the annotation paradigm based on Large Language Models (LLMs) has made sig- nificant breakthroughs in recent years, its actual deployment still has two core bottlenecks: first, the cost of calling commercial APIs in large-scale annotation is very expensive; second, in scenarios that require fine-gre seee-grained semantic understanding, such as sentiment classifica- tion and toxicity classification, the annotation accuracy of LLLSs is even lower than that of Small Language Models (SLMs) dedicated to this field. To address these problems, paradigm of multi-model cooperative annotation and design fully an- tomatic annotation framework Auto based on this. Specifically, Annotatoo consists of two layers. The upper-level meta-controller layer uses the generation and reason- ing capabilities of LLMs to select SLMs for annotation, automatically generate annolation code and verify difficult samples; the lower-level task-specialist layer consists of multiple SLMSs that perform annotation through multi-model voting. addition, we use the difficult samples obtained by the secondary review of the meta-controller layer as the reinforcement learning set and fine-tune the SLMs in stages through continual learning strategy, thereby
propose

---

Published in Transactions on Machine Learning Research (12/2025)
80
MTEB-Sentiment
RoBERTa
LYT-Sentiment
GPT-3,5-turbo
HG
JW
Deepseek-V3
60
Accuracy (%)
85
70
40
25
MTEE
KC
JPP
LYT
RoBERTs
BERTweet
Deepseek-V3
Minimax
1PT-3.5-turbio
QwQ-328
Uama3-708
a
(b)
Figure 1: (a) Comparison of classification performance between Large Language Models (LLMs) and Small Language Models (SLMs) two sentiment classification tasks, proving that SLMs outperform LLMs on domain-related tasks. (b) Classification performance of LLMs and SLMs on 3 sentiment classification datasets Senttnneentmment, JPP-Sentiment, LYT-Sentiment) and 3 toxicity classification datasets (KC-Toxicity, JW- Toxicity, Texiciity illustrating that SLMs exhibit weaker generalization than LLMs.
improving the generalization of SLMs. Extensive experiments show that Aunotaao outperforms existing open-source//AP1 LLMs in zero-shot, one-shot, CoT, and majority vot- ing settings. Notably, AutoAnnotator reduces the annotation cost by 74.15% compared to directly annotating with GPT-3.5-turbo while still improving the accuracy by 6.21%. The code is available in httht::////// wwwpth.w.c.acaaaaaaaaaaaaaaaaa wwt.:uiAnAAtotttttt..o.o
1 Introduction
High-quality annotated data is key to advancing deep learning (Emam el al. 2021 Rasmussen et al. 2022:
Taori et al. 2023: Ye et al.] 2025 Jiang et al. 2025), yet acquiring such data requires specialized domain expertise and is costly (Denton ct al. 2021), especially when manually annotating a large number of samples.
With the rapid development of LLMs (Achiam et al. 2023: Guo al.
2025 Team et al. 2025), their powerful
semanntc understanding (Wu et al. 2023 Yu et al. 2022)]
contextual reasoning (Sun et al. 2024 Zhou 2025
Lan et al. 2025) and generation capabilities (Mo al.
2024
Zhou al.
2025) has driven researchers
develop LLMeb annotation methods (Yadav et al.
2024
Chen et al.
2024
Wu et al.
2024a Tekumalla
& Banda 2023
Flamholz et. al. 2024 Laskar et al.
2023
Ba et al. 2024 to reduce the cost. of manual
annotation.
However, our priori experiments show that this fine-sizc-fits-all approach does not work in all areas.
tasks such as sentiment classification (Brauwers Frasincar 2022 Jiang et al., 2011) and toxicity classifica- tion (Van Aken et al. 2018 He al. [024]] Li et al. (0244a), LLMs without special training perform much worse than smaller models that have been specifically fine-tuned (see Figure [a)). Besides, the annotation cost is often prohibitively expensive, especially when scaling to large datasets. For example, annotating 100, 000 short soriews-cach averaging 1024 input tokens and 20 output tokens-usine GPT-ol (at $15 per 1M input tokens and $60 per 1M output tokens) will cost roughly $1, 656. In contrast, the annotation cost of SLMs is almost. negligible.
So can we do the opposite: let SLMs (o.g., BERT Devlln et al. 2019 and Roberta (Liu al. 2019) take on the "main force" of data annotation, and efficiently generate initial annotations with its low annotation cost and rich domain knowledge; and when SLMs have low confidence or the sample is more difficult, L.I.Ms will
2

---

Published in Transactions on Machine Learning Research (12/2025)
Label
SLMs
Label
Human Annolation
(b) SLM-Based Annotation
Easy Label
Prompl
Guide
SLMs
Label
Check
Hard Label
(c) L..e Annotation
(d) Multi-model Collaborative Annotation
Figure 2: Different data annotation paradigms. (a) represents the traditional manual annotation paradigm. (b) denotes SLM-based annotation paradigm. (c) is the most popular LLM-basod annotation paradigm. (d) denotes the multi-model collaborative annotation paradigm proposed by us. Our paradigm can not only improve the annotation accuracy, but also significantly reduce the annotation cost.
provide secondary review, so as to balance cost-effectiveness and annotation quality. The reason for using LLMs to re-verify difficult examples is that although SLMs outperform LLMs on their familiar domains, their limited generalization ability unreliable the case of diverse real data annotations (see Figure (b)). This multi-model collaborative annotation paradigm can not only reduce overall API overhead, but also leverage the powerful reasoning capabilities of LLMs improve the labeling accuracy of key samples. Table I and Figure 2 illustrate the difference between proposed annotation paradigm and other existing paradigms.
In this paper, we introduce a self-evolving Automated Data Annolation framework, dubbed as Auto notator, to improve the existing annotation paradigm. AutoAnnotator coordinates both the generative generation and reasoning capabilitics of LLMs and the fine-grained task determination capabilitics of SLMs, achieving adaptive model selection, automatic code generation, multi-moded consensus annotation, and model iterative evolution. By leveraging the low-cost efficiency of SLMSs and selectively invoking LLMs only when necessary, our framework significantly reduces annotation cost while achieving superior comparable an- notation quality. By leveraging the low-cost, -ppeiii nature of SLMs and selectively invoking LLMs only when necessary, AutoAnnot significantly reduces annotation costs while achieving superior annota- tion performance.
The entire AutoAnnoo framework can be viewed hybrid expert system and consists of two layers: a meta-controller layer and a task-specialist layer. Specif- ically, the meta-controller layer, powered by I.LMs, responsible for selecting ap- propriate SLMs from Hugging Face]] based
Table 1: Comparison of efficiency, accuracy and generalization ability of different annotation paradigms.
Human
SLMs
LLMs
Annotator
No manual labeling required
X
Low annotation cost
X
x
/
High labeling accuracy Good generalization
/
V
/
the given annotation task and automatically generating the code required for the entire annotation process. Since SLMs have limited generalization ability on out-of-domain (difficult) samples, and LLMs have stronger generalization due to pre-training on massive and diverse data, meta-controlles will call LLMs to perform a second review of these difficult samples, thereby significantly improving the generalization performance of the overall labeling system.
The task-specialist layer comprises the selected SLMs by the meta-controller layer. Specifically, each input is fed to all SLMs, and the predictions of these SLMs aggregated through majority voting consensus mechanism to generate high-confidence labels. Samples that do not reach the consensus threshold are
https://hugginnfaccccco
3

---

Published in Transactions on Machine Learning Research (12/2025)
automatically labeled and returned to the meta-controller layer for secondary verification using the LLM. Once the hard-sample pool reaches a predefined threshold, these expert-verifeed examples trigger an iterative fine-tuning cycle: each SLM is updated on the collected hard samples, and the refined models then rejoin the consensus pool for subsequent annotation. This contimual enhancement loop ensures that the specialists progressively improve their generalization. Overall, our contributions can be summarized as follows:
A new paradigm of data annotation. We propose the paradigm of LLMs guidance with SLMs execution, where LLM uses its powerful generation and reasoning capabilities to build the annotation environment and review the annolation results, while SLMs apply their domain-specific knowledge to carry out the actual labeling.
A fully automatic annotation framework. We introduce a two-layer annotation framework Au- to Annotaatr,, which fully automates the annotation model selection, code generation, annotation verification, and annotation model iterative update process.
Cost Reduction and Improved Performance. AutoAnmotator outperforms existing opened- source LLMs (7B-70B) and API models (including Minimax, Deepsock-V2 V3,, Deepsock-R1, GPT- 3.5-turbo and GPT-4o), and consistently maintains optimal performance under multiple labeling strategies such as zero-shot, one-shot, CoT, and majority-vote. Besides, AutoAnnotator reduces the annotation cost by 74.15% compared to directly annotating with GPT-3.5-turbo, while still improving the accuracy by 6.21%.
2 Related Work
LLM-Based Data Annotation. Thanks to the remarkable capabilities of LLMs across wide range of tasks (Zhao et al. 2025 Lu et al., 2024 Li et al. 2025b a, recent research has gained increased interest in using LLMs for data annotation. For instance, Jachav et al. (2024) utilize both closed-source and open-source LLMs to annotate low-resource language Marathi. CCen et al. (2024) utilize LLMs to generate samples that are consistent with the data distribution of the benchmark dataset for event extraction, thereby alleviating the challenges of data imbalance and scarcity. Similarly, Li et al. 2024b) use LLMs for high-quality code retrieval query annotation. Choi et al. (2024) extend cost-effective LLM- LLM-based annotation beyond traditional data annotation tasks filter out noisy documents from a multi-document summarization dataset. Liu al. (2025b) leverage LLMs in combination with historically annotated data and expert-constructeed codebooks to extrapolate and extend longitudinal network datasets into future periods. Besides, some studies use LLMs to improve the original annotations made by human annotators (Laskar et al. 2023 Flamholz et al.) 2024
Although LLM-based data annotation methods have made significant progress, their application still faces two major challenges: on the one hand, the high cost of API calls makes it difficult to achieve largo-scale economy; on the other hand, in lasks that. require fine-grained semantic understanding (such as sentiment classification (Brauwers & Frasincar 2022 Jiiag et al. 2011] and toxicity classification (Van Aken et al. 2018 He et al., 2024 Li et al. 2024a)) ) the annotation performance of LLMs is often inferior to that of specially fine-tuned SLMs.
Collaboration between LLMs and SLMs. Collaboration between LLMs and SLMs combines the for- mer's generalization and reasoning strengths with the latter's efficient, domain-specific expertise, yielding superior performance and cost-efficiency across various tasks, especially on resource- edge devices. For example, Xu al. (2023) use predictions from SLMs improve L.M in-context learning. CoGene- sis (Zhang al.] 2024) integrates LLMs (hosted on cloud infrastructure) and SLMs (deployed on local devices) to address privacy concerns logically. CITER (Zheng et al., 2025) adopts token-level routing strategy, routing non-critical tokens to the SLM to improve effciency, while routing critical tokens to the LLM to ensure generation quality. Collab-RAG (Xu et al. 2025) employs an SLM to dccomposc complex queries and improves the SLM's decomposition ability through leedback signals provided by a black-box LLM. Glocker et al. (2025) use a task-specific LLM as the "brain" to drive multiple seeiiappe SLMs to perform sub-tasks such as routing and task planning. Inspired by existing studies on LLMs and SLMs collaboration, we innovate the existing LLM- based annotation framework and propose a two-layer automated annotation system, with I.L.ss guidance and SLMSs execution.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Task-Specialist Layer
BBB sentiment
Unlabeled Data
Toxic
Type
Toxic classification
e voting Consensus Annotation
Have you ever considered starting child elouse ring with your fhiend?
Fieldh
loss Iterative Refinement
wish you great day, and your efforts be rewarded with
Deployment
Secondary Review
raain
Sentiment classfiiation
Early monday cramming... yay. Only few weeks left.
M
hate days
that. Cooking microwave pizzas, yummy
Model Selection
Code Generation
Meta-Controller Layer
Figure 3: Visualization of the pipeline of Aunooooo Autolnnot consists of two layers: a meta- controller layer and a task-specialist layer. The meta-controller layer, powered by LLMs, is responsible for selecting appropriate SLMs from Hugging Face, automatically generating the code required for the entire annotation process and performing secondary review on samples that are difficult for SLMs. The task- specialist layer comprises the selected SLMs by the meta-controller layer. SLMs use majority voting mechanism to annotate samples and periodically use difficult samples from LLMs for secondary review to continuously update themselves.
Multi-Agent LLM System. To enhance the problem-solving and decision-making capabilities of LLMs (Zeng et al. 2025a b). multi-agent LLM systems are introduced (Wu et al. 2024b Liu et al. 2025a)
For example, Liang al. (2023) and Du et al. (2023) use multi-agent debate as an effective method encourage divergent thinking and improve factuality and reasoning. MetaGPT {Hong et al. 2023 utilizes multi-agent conversation framework to help automatic software development. Besides, AutoDefense (Zeng
et al. 2024) assigns different roles to LLM agents and employs them to complete the defense task collabo- ratively. Lou et al. (2025) introduce a dynamic reputation filtering framework to quantify the honesty and capability of agents, thereby improving agent selection efficiency. Recently, Wang et al. (2024) construct a layered MoA architecture wherein each layer comprises multiple LLM agents to improve the reasoning and language generation capabilities. Each agent. takes all the outputs from agents the previous layer as auxiliary information in generating its response. However, these methods differ fundamentally from ours. These multi-agent frameworks are homogencous collaboration (LLM to LLM), while our Autotnno is heterogeneous framework that uses single LLM to coordinate and iteratively improve set of low-cost SLMs specifically for data annotation.
3 Method
In this section, delve into the Auto hierarchical system that synergizes LLMs with SLMs for automated data annotation. As illustrated in Figure 3 the system operates through two interdependent layers: the Meta-Controlles Layer and the Task-Specialist Layer This design complements the pow- erful generation and reasoning capabilities of the LLMs with the eflicient domain expertise of the SLMs, not only achieving better annotation performance, but also significantly reducing annotation costs.
3.1 Meta-Controller Layer
The Meta-Controller Layer serves the ddciisionmmkkig unit that orchostrates the entire annotation process. It mainly implements three core functions: adaptive model selection, automatic code generation and difficult sample verification. Next, will introduce these functions in detail.

---

Published in Transactions on Machine Learning Research (12/2025)
Adaptive Model Selection. Assuming there is dataset D = { 12,, .......,........................ In to be amnotated, AutoAnno- lalor first needs to determine SLMs for annotation. However, faced with millions of open-source model on platforms such as HuggingFaed non-professionals often find it difficult to filter out models that are suitable for the current task from the complex model descriptions. To address this challenge, we built an adaptive model selection engine using LLMs, eliminating human intervention in model selection. Specifically, given an annotation task T, we utilize the LLM give a list of task-related model recommendations and take the Top-k models for annotation. This process can be formulated as follows:
M, = Top-k (sim (fLLM (T), (1IM(d)))
(1)
where d denotes the description of the corresponding model. It is worth noting that when the annotation is so niche that no pre-trained or fine-tuned SLM is available, in such a "cold start" scenario, set of labeled data is needed first. In this case, we can first use LLMs or manually annotate to generate a seed dataset for training several imitial SLMs. Then we replace SLMs recommended by LLM with these self-trained SLMs for further experiments. We conduct an experiment in Section 4.2 to demonstrate the fcasibility of this strategy.
Automatic Code Generation. After obtaining the recommendation list, an intuitive method to down- load and deploy these models locally, Then, we can start the data annotation and subsequent processing steps. However, in this workflow, many processes usually require manual programming, such as SLMs deploy- ment, data annotation, and SLMs fine-tuning, which makes the entire process labor-intensive. To address this limitation and maximize automation, we equip the meta-controller layer with an automatic code gen- eration capability. Given the powerful code generation capabilities of LLMs (Wang & Chen, 2023 Roziere et al. 2023 Jiang et al. 2024h, we directly prompt it to generate all the scripts required for the annotation pipeline.
Difficult Sample Verification. While SLMs exhibit superior performance on domain-specifie annotation tasks, they often struggle with out-of-domain samples. In other words, SLMs have limited generalization ability, making them less reliable when performing complex annotation tasks. In contrast, LLMs, especially those like GPTs and DeepSeck, trained on diverse data, show stronger generalization capabilities (Li et al. 2025e) and can better handle more complex situations (see Figure a))- To this end, Aumolalll leverages LLMs in the mneaaconnroller layer to perform secondary validation on complex or uncertain samples, achieving both high accuracy and strong generalization across various data conditions.
3.2 Task-Specialist Layer
The Task-Specialist Layer is responsible for the actual annotation, using a set of lightweight, domain-specific pre-trained SLMs to efficiently label the data. It consists of two components: the Multi-Model Consensus Annotation module and the Expert-Guided Tterative Refinement module.
Multi-Model Consensus Annotation. As mentioned earlier, SLMs exhibit poor generalization ability, To address this, we aggregate predictions from a diverse pool of SLMs and only accept labels on which they reach high agreement. Formally, let D = {x, J.N, be the unlabeled dataset and M = {m1, the pool of SLMs recommendod from the Mcta-Controller Layer. For cach data sample I; E D, run all k models in parallel to obtain their annotation results:
..,... ... m } be
y = ut
}
(2)
The final label gi is determined by majority voting of these k models:
f Majority Voting (Dt).
(3)
In order to evaluate the degree of consensus among models, the uncertainty metric I is introduced, which is defined as:
u (z.) = max, # {vi = y
(4)
k
2 As of May 12, 2025, there 1,685, 478 open source models on HuggingFace. 3 Phttps//hhuggiggaaee
6

---

http://ttt-----ooooonnnaaniiiiia Published in Transactions on Machine Learning Research (12/2025)
Table 2: The appropriate Hugging Face models selected by LLMs based on the type of annotation task.
Task Type
Model ID
Parameters
HIF Downloads
Nick name
125M
2.43M
SLMI
Sentiment
125M
2.06M
SLM2
110M
1.06M
SLM3
110M
160K
SLM1
Toxicity
JungleLee/
110M
46.3K
SLM2
garak-llmm/txxiccooomeii /lmm///ooic--commm---mdoeed
67M
9.67K
SLM3
where # (1) = f) represents the number of models that predict y. If 4 (2x) is greater than the predefined value e, the sample is considered have a large disagreement and automatically stored in the secondary review pool Duard:
Route(x)) Direct Labeling
U(x) < Secondary Review U(x) 2
The Task-Spocialist Layer dynamically selects between two verification modes (automatic LLM annotation or manual human annotation) based on user needs and cost-accuracy trade-off analysis.
Expert-Guided Iterative Refinement. To explicitly enhance the generalization of specialist SLMs difficult (out-of-domain) samples, we introduce continual fine-tuning procedure guided by expert labels (from either LLMs or human annotators). Specifically, once the number of samples in the hard sample pool Dhard reaches a predefined size B, we pause the consensus annotation and start the continual fine-tuning cycle. For each SLM m, E M, we fine-tune it on Dhard for up to a budgeted number of epochs, producing updated specialists m4. Taking the model m, as an example, we define the loss function as follows:
[ = Σ (E(m;(z;(),)) (x,y) Dhard
(6)
where CE(-) represents the cross-entropy loss. Then gradient descent is used to update the parameters:
0 0 0 aVo(()
where a denotes the learning rate. Subsequently, the refined model m replaces m, for annotation. After that, we resume annotation remaining unlabeled data until Dardd exceeds B again, triggering further cycle of continuous fine-tuning.
4 Experiments
4.1 Experimental Settings
Datasets. To evaluate the effectiveness of AutoAnnotator, we conduct extensive experiments on two representative annotation tasks, sentiment classification and toxicity classification, using a total of six real-world datasets. Specifically, in this study, we select. mteb/tweet. sentiment cectimment_extraattinn wangls//wiiter--maniia--eewwwe
and LYTinm/cniimenttaallysss.ss http://wee_ssnttmmnttttttt cation task and Barlhikaruun//Cbebuulll Teweerbulllying-ooiiie-----e Toxieity-Tweels, Dooxin-wen/Impleii----e toxic conversations balanced for the toxicity classification task. We provide detailed introduction to these dataset in the Appendix A To simplify the description, use the following aliases for these datascts: MTEB-Sentiment, JPP-Sentiment, LY'T-Snntmmet,, KC-Toxicity, JW-Toxicity, and HG- Toxiciiy,,, respec- tively.
for the sentiment classifi-
and heegyu/-
Models. In AutoAnnotator, all SLMs involved annotation are automatically selected by the LLM in the meta-controller layer based on task characteristics. this paper, we select SLMs for each annotation task by default. We provide details of the model selected by the LLM in Table

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Comparison of the proposed AutoAnnotator with existing methods on different toxicity and sen- timent annotation tasks. It is worth noting that the SLMI for sentiment classification and the SLM1 for toxicity classification not the same model. The specific models of each model are shown in Table 2
Chectfication
Todatty Caasthottion
Scatiment
Toacetty
Tooiity
IIC: Toatety
Aug
SLMP
5595
59 9656
72.5256
67.08%
59-2755
80.87%
61.00%
SLAB
70.17%
69.22%
78.83F
72.7408
65.1356
39,3076
67.0775
58.175%
Opanesnuivee E&Mss, Zarovaloct
Mistral-TBB-V..
35 9196
25.94%
18-3776
78: 78.17%
383195
0355
50-3956
50 18.0995
68475
Batchuaz: TB: Dase
38 607
24.905
30.385
31,3225
38390
20,9956
61. 62.2255
67.0895
40. 43.9075
43475
Qwen2.5-7B. 5-B-ttttee
65 6255
71 71,025
67 8655
69.3095
38396
63,9255
76.2556
72.3395
72. 86.66%
48475
Uhmmak HBf- Enstrnct
31 M056
07.3
33.1553 15.1555
34.147%
380195
724856
63.1 13.1
001.0775
10. 10.0855
68470
F/amuzh 13Rt
18 8555
36 56
0256
40.2655
383995
53.9956
75.7156
67 67.0655
00.0055
48415
QuQ 2-10
54 2556
63 03.0155
53.72%
70.815
77.9776
70.7005
45475
LiunsS- 708
36.8255
46.3555
49.0250
70.6055
68.26%
48475
LLMS, sbot
Mistall 7B- VB-
24%24%5
76.73%
02.8056
55.7005
08.10%
48476
40.3195
77.0055
48475
Quen25-7B:
58.1356
79.186
71.0075
101 4076
48875
1kmmas Eratrrnt
39-31F5
03.3655
36-3176
60.345
3K395
55.0556
8358556
64.2595
[0. 0888
48575
Lomo2 13B
38. 5005
65.1776
63.0496
63.2425
38306
00.0855
88.79F6
74.1126
45870-
QwQ-328
35. 5955
68. 47%
37 1655
$5.7 7.99
3K396
02.9055
87. 8815
70.1999
73 907%
18475
Lamad T0B
30 00-3856
05.055
30.9006
322 2472
383M5
05.0256
71.0256
60.2895
10 50.588
08470
Open SOuEEc
LLM5,, CoT
Mittaal-7--00.2
10-4555
25.9155
34.215 28.2155
32 8496
38395
69.2456
74.9055
69.1995
61.61%
18475
Datchnan TD: Dase
36 5355
23.7456
29 41F%
20.322 0229227
383995
47.5356
80.3766
66,700
64.91.%
88475
Quen25-7B- -t-rttrrtttt
41.4
7256
0331155
05.1256
76.119
18. 16.195
18875
1luno3.1 HB: Lustroct
51.7496
59.7135
$4.5876
47.00%%
64.507
45475
E2mo22113 Qu1-320
51.2450
71.3150
06.35%
18375
4356
07.7956
75.047
T4.07%
Limnd:: 708
46.815
42.7155
46.9455
45.4995
50.0155
02.0855
71,0795
63.78%
48476
Deepeedss
81.3066
79.2775
74.44%
48475
Deepereis-Ii RR
06.37%
24.2556
07 9006
69.37%
38395
T33355
74 2056
27.000
70 76.00%
68475
Minimox-ahall whallo--aaaa..... de-dutt GPT-3. S-tarb
67.47%
77.225
68.5895
71.0655
38306
72,875
68.1858
76.1455
72.23%
48475
65.1455
72.915
69.
3K396
61.2055
740056
78:2196
71.55%
48475
L.,.,,, Majunnty Vote
7 Openesouzee LLbs Voting (acm-shot)
54.6055
65.86%
57 67.6
56.0735
268772
65.20%
76.0555
27.0005
72.97%
850325
1 Openecurce L&Me Voring (un--bee)
03 20F5
71 1355
5456
65.67%
205772
09.755
86 7356
72. 12.9495
70. 18.16%
3393355
4 API Model Voting
67.78%
77,855
70.505
72.143
153584
72.077
72.995
79.4775
74.54%
150500
Auto.Annssatorr(CC (Ouzs)
AutoAmnrator Malimux AntoAmecator Derppawi-V3 Autolmmstttor GPT tucbo
67. (7.7
81. 20.
74.805
74.2995
10643
73.73%6
73.7558
59,500
77.01%
182LD
7755
77.965
22.76%
10537
61.2996
85 25.2656
83-100 35-1599
70.61 56
17886
GT 8976
78.
73.1476
67.09%
82.0256
82.9975
17.
18942
Antokuantaloe
78.35%
82.855
81.709
83.005
91.7855
88
Implementation Details. By default, we use 3 (k: -6=33 SLMs for each annotation task. Once the hard- sample pool Dhard reaches a predefined threshold / = 2, 000, we will pause to finc-tune all 3 specialists on Dhard, ensuring they can continually learn new things throughout the annotation process.
As for fine-tuning the SLMs, we set the initial learning rate, weight decay and epoch to 2e 5, 5, 0.01 and 3, respectively. All experiments are conducted on 2 NVIDIA A100.
We provide detailed ablation experiments in Section 4.3
Methods for Comparison. To evaluate the effectiveness of AutoAnnotttor, we compare it with three types of baselines: (1) SLMs selected by the LLM in the meta-controller layer (2) Open-source LLMs, range from Mistral-7B, Baichuan2-78 (Yang et al., 2023), 2-wen2.5-BB-Inseee
(Yang et al. 2024a), Ilama3.1-8P Instruct (Grattafiori et al. 2024). Llama2-13B ( (Touvron al. 2023), QwQ-32B and Llama3 70B (Grattafiori 2024h. (3) API models, such as MiniMax, DeepSeek V3 (Liu et al., 2024), DeepSeek R1 (Guo et al.
et al.
2025 and GPT-3.5-Thrbo. We provide all prompts used by baselines and Autottnoo in Table and
Table A
4.2
Main Experiment Results
Comparison with SLMs. As shown in Table 3 under the "SLMs Only" setting, the strongest SLM achieves 72.74% average accuracy on sentiment tasks and 63.83% on toxicity. By integrating these SLMs into our Annotatoo framework, we boost sentiment accuracy to 74.59% (+1.85%) and toxicity to 77.56% (+13.73%). These experimental results demonstrate that AutoAnnotator can significantly improve the per- formance of SLMSs on sentiment classification and toxicity classification annotation tasks.
8

---

Published in Transactions on Machine Learning Research (12/2025)
[Sentiment Classification| Warren Buffett Just Became More Heavily Invested Rat Poison Cryptocurrency.
Negative
Negative
Negative
Auto/Anottator
Neutral
X
[ooxiiiyy Classification| Just keep your head up..you kick so much ass...
Non-toxic
Non-toxic
Non-toxic
AutoAnnotator
Toxic
Figure 4: Visualization of representative samples correctly annotated by AutoAnnotator but misclassified by APIs.
Table 4: Comparison of the proposed Auto tor with API models in few-shot mode.
Table 5: Comparison of the proposed AutoAnnota- tor with MoA.
Settings
Acc
Method
Acc
GP-T-35-ttrrb (zero-shot)
72.91%
MoA
72.18%
GPT-3.5-turbo (three-shot)
73.62%
A Auto Anottttt V3
81.20%
GPT-3.5-turbo (five-shot)
73.70%
77.96%
Auto -ttoomomtttto-----35---tt-t
78.56%
AutoAmnotator-MMinnn
81.20%
AuloAnmololor-HHanan
82.83%
aatt.AAnnoottoo--eesccck V3
77.96%
Aulo 4
78.56%
Auto Anmotatrt Human
82.83%
Comparison with Open-sourced LLMs. To benchmark AutoAnnotttor against open-source LLMs, we evaluate the latter in three widely used annotation settings, zero-shot, one-shot, and chain-of-thought (Wei ct al.) 2022; (CoT) prompting, on both sentiment and toxicity classification tasks. As for the one-shot setting, each model is given a single in-context example before annotation. As for the CoT setting, we add a CoT prompt "Let's think step by step like an operations research expert." behind the zero-shot prompt. Among these settings, we find that one-shot prompting consistently outperforms zero-shot, as the single in-context example helps the model calibrate its label distributions and reduces misunderstanding of the task. By contrast, chain-of-thought prompting hints only marginally improve annotation accuracy, which we believe is because the generated step-by-step reasoning shifts the model's focus away from the classification task. Overall, AutoAnnotator consistently outperforms zero-shot, one-shot, and chain-of ofain-offre thouhhttt prompting strategies, demonstrating its superior annotation accuracy and validating the effectiveness of our multi-model collaborative paradigm.
Comparison with API Models. We further benchmark against APT models, including Minimax, Deepseek-V3. Deepseek-R1 and GPTT-35-turbo. We report our main results in Table 3 The strongest Minimax achieves 71.06% average accuracy on sentiment tasks, while Deepseek-VS V3 leads toxicity at 74.44%. By integrating these API models into Annolaor,,, we boast sentiment accuracy to 74.59% (+3.53%) and toxicity to 76.61% (+2.17%). Similarly, when GPT-3.5-turbo is used alone, the sentiment accuracy reaches 69.10% and the toxicity accuracy reaches 71.35%; when integrated into Anmotatr, the sentiment accu- racy rises to 73.12% (+4.02%) and the toxicity accuracy rises to 77.56% (+6.21%). Besides, compared with direct LLM annotation, Anmotatoo significantly reduces the number of LLM calls (60%+ for sentiment tasks and 70%- for toxicity tasks). It is worth noting that AutoAnnotator not only outperforms existing API models in terms of performance, but also far exceeds them in terms of annotation cost and efficiency (see below). We provide some samples that AutoAnmotator can annotate correctly, but API models amno- tate incorrectly in Figure 4 In addition, we compare AutoAnnotator with API models of few-shot settings. Specifically, we use GPT-3.5-turbo to lest the performance of zero-shol, three-shot, and five-shol sellings on the JPP-Sentiment dataset. The results are shown in Table 4 As expected, increasing the sample size in prompts from zero to five does indeed improve the accuracy of GPT-3.5- However, while the five-shot bascline (73.70%) is more competitive, our AutoAnnotator still significantly outperforms this stonggr baseline, which further demonstratees the effectiveness of our method. Further-
from 72.91% to 73. 73.70%
9

---

Published in Transactions on Machine Learning Research (12/2025)
more, using a five-shot setting significantly increases API call costs. This contradicts AutoAnnotator's core objective of reducing costs.
Comparison with LLM Majority- Vote. We additionally evaluate majority voting across multiple LLMs. As shown in Table 3. AutoArmotttorr+Miiiiii needs only 10, 643 API calls to achieve 74.59% sentiment accuracy, outperforming both open-source and API-voting baselines while reducing LLM calls by over 93%. Regardless of whether we ensemble multiple open-source or API LLMs even with zero-shot or one-shot voting voting-Auto Anootooo consistently outperforms all voting schemes while using far fewer LLM calls.
Comparison with Mixture-of-Agemee (Wang et al., 2024). In this subsection, we compare AutoAn- notator with Mixture-of-Agent (MoA). MoA involves collaboration between multiple powerful LLMs, rather than dedicated SLMs, to solve complex tasks through multiple iterations. In contrast, Armolaao facil- itates collaboration between two different types of models: powerful LLMs and dedicated SLMs. To verify the effectivencss of our proposed method, we conduct experiments on JPP-Sentiment dataset and compare it with MoA. As shown in Table 5 MoA achieves an accuracy of 72.18%, which is far lower than our method. The reason for MoA's poor performance lies in the fact that it is composed of multiple LLMs. Figure aa) clearly shows that on fine-grained semantic understanding tasks such as sentiment classification, the accuracy of general-purpose purpose LLMs (such as Llama3-70B, QwQ-32B, etc.) is significantly lower than that of specially fine-tuned SLMs (such as RoBERTa, BERTweet). Therefore, the MoA framework essentially integrates multiple models that perform poorly on this specific task. While integration can bring some performance improvement, it does not outperform our method.
Extend Annotato to more tasks. To verify the ver- satility of AuloAnnolalor beyond sentiment and toxicity classifi- cation, we conduct additional experiments in the more chal- lenging domain of mathematical reasoning. Specifically, we select two widely used benchmarks, MathQA (Amini et al. AQUA-RAY (Ling et al. 2017 for
Table 6: Comparison of the proposed AutoAnnotator with statc-of- Che-arl. baselines on math tasks. Model
Ace on MathQA
Ace AQUA-RAT
Mathstral-7B. 7B-
39.60%
36.83%
Qwee2..5 5e..-----hh------turre
80.29%
79.87%
DeepSeek-matlee matt-----ettt-tt-ttttrtettt
66.43%
05.33%
Self-Cousisteney (CPT-3.5-turbo)
65.00%
64. 20%
DynsThink- Selinnk-SSff-CCaa Colf-Consistency (CPT-3.5-turoo
68. 68.00%
61.50%
2019) and
Autotnnottoor
85.00%
83.39%
the annotation task. Following the setup of our framework, we use three publicly available mathematical Srren2.--atth-7B-Insteee
O (Yang et al. We compare the performance of Anmotator with two strong baselines: (1) the performance of each specialist SLM individually, and (2) the advanced GPT-3.5-turbo plus advanced inference strategies, including self-consistency and ThnaThink + self-consistency. As shown in Table 6 AutoAnnoottto achieves the highest accuracy on both datasets, scoring 85.00% MathQA and 83.39% on AQUA-RAT. This performance significantly surpasses the strongest single SLM and far exceods the results achieved using only LLM. These experimental results further demonstrate that the proposed method a general approach that can improve annotation quality even in complex domains.
models as SLMs for the task-specialist layer: Mathstral-7B-vo.FF 2024b), and (aapsoee--math----Tnnttruu
Scenarios where no pre-trained SLM exists. In this paper, we assume that LLMs can download pre-trained SLMs from Hugging Faces. However, for some niche tasks, these high-quality, ex- isting SLMs may not exist. To address the feasibility of this 'cold start", we de- sign the following experiments to sim- ulate this scenario. We use the JPP- Sentiment dataset as an example to compare two cases. In this cold start scenario, we first manually label 2000 samples from the JPP-Sentiment dataset to form small seed dataset. Next, we use this seed dataset
Table 7: Performance comparison of AuloAnnoootor with and without pre-trained SLMs.
Model
Scene
Acc
Downloaded from huggingface
78.56%
GPT-3.5-turbo
Trained using self-labeled data 78.29%
hhtpps///uuggnggeeecoo///aaaaa// Matisrraalll/aaatsrraa---B--- 5 http:///wuggggieeeooo///eeeeea/o weeaaahh----sssnnne 7b-instruct
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 8: Comparison of annotation cost and efficiency between API models and AutoAnnotator
Model
Token (Iuput Outputt
CPU Memory
Time Cost
Time Reducction
Crstt
Cost Reduction
V3
86023
71.19 minutess
0.027202
Autodmmotator + V3
25620
4458 MB
26.06 minutes
63.40%
0.008085
70.28%
Deepseek-R1
356532
212.93 minutes
0.6508341
AutoAmnotator R Desperok-R
100668
4458 MB
39.40 minutes
81.50%
0.2115811
67.47%
Minimax
91721
93.7 70 minutes
0.012723
Axto Antaatttttorrr MetuuatttorMMiiimns Minimax
19357
4458 MB
30.67 minutes
07.27%
0.003112
75.54%
GPT-3.5-turbo
88514
31.17 miontes
0,048423
VatoAmmotttor+GPP-33335
22235
4458 MB
17.75 minutes
48.00%
0.012519
74.15%
GPT-1
88027
30.56
minutes minutes
2.751180
Auto Auto-hwooottrr-GPT
23843
4458 MB
18.34
40.00%
0.754470
72.68%
GPT-4o
87915
23.20
minntes
0:2465-00
AutoAnnotttor-GPT--
22087
4458 MB
15.21
minutes
31-44%
0.06-300
73.92%
Table 9: Detailed comparison of the time cost of annotation and fine-tuning with SLMs and the time cost of annotation with LLMs.
SLMs
Modell
Labeling
Fineturing
Total
Model
Labeling
cardiffnlp/twwii /twitter--oobeeteeeeee roberta- base setteeenntttte---eeeeee -etteeseee-ttttttt
0.32 minutes
0.54 mimates
0.86 minutess
Deepwook V3
71.19 mimatess
cacdiffalp//ww wwiliter-xln--obeee beseett-----e-eeee innett
0.38 minutee
0.68 minutes
1.04 minutes
Despseek-RR R1
212.93 mimutee
fnitesutomataa/ bertweet- base-seat iementeeettttt -aadyyes
0.30 minutes
0.58 minutes
0,88 minutes
Minimax--bbale abab66.5e-chat
83.70 minutes
s-ulp/roberts /roberta todicity_ classifler
0.29 mimutes
0.56 mimutes
0.85 minutes
GPT-3 ----bbb-
34.17 minutes
JungleLee weee-tooci--oomneee classfication
minutee
0.52
minutes
0.81 minutes
GPT-to
23.20 minutee
gozak- / toxte-oomment-moot
0.23 mimutes
0.35
minutes
0.58 minutes
GPT-4
30.56 minutes
to fine-tune 3 SLMs and treat them as experts in the task-specialist layer. Finally, we utilize GPT-3.5-turbo the rellerontroller to the complete Aunolaaot process. As shown in Table 7 the cold start strategy achioves similar annotation accuracy to the original strategy, further demonstrating the fcasibility and robustness of Aulo even for niche tasks without pre-trained SLMs.
The efficiency and cost of AutoAnnotator. To evaluate the annotation cost and efficiency of API models and AutoAnnotator, we conduct a quantitative analysis from three dimensions: computing resource consumption (the number tokens and GPU memory usage), annotation time cost, and economic cost. All experiments are performed on NVIDIA A100 GPUs, and the annotation task scale is uniformly set to 1000 samples. As for Ammotatt,,, we set the 3 = 200. We conduct experiments on Deepsock-V3, Deepscek- R1, Minimax, GPT-3.5-turbo, GPT-4 and GPT-4o, respectively. As shown in Table g AutoAnnotator reduces the annotation time by 34.44% (GPT-4o) to 81.50% (Deepseek-R1), with an average reduction of 55.85%. Besides, the annotation cost is reduced by 75.54% (Minimax) at the highest and 67.47% (Deepseek- R1) at the lowest, with an average saving of 72.32%. In Table g we provide the annotation time cost for the whole pipeline of AutoAnnotator. Herein, we further provide the annotation time cost for SLMs as well as their fine-tuning cost. For a fair comparison, we annotate and fine-tune on the same 1000 difficult samples. As shown in Table g compared to LLMs, SLMs require significantly less time for annotation and fine-tuning, typically under 1 minute in total per model. In contrast, LLM-based annotation is substantially more time-consuming, with labeling times ranging from 23 to over 200 minutes depending on the model. This highlights the practical advantage of using SLMs for efficient labeling and finc-tuning in real-world scenarios. In general, AuloAmmolaoor has achieved a significant improvement in annotation efficiency and a significant reduction in annotation costs while maintaining annotation quality through the LLMs and SLMs collaborative annotation paradigm.
4.3
Ablation Study
The number of SLMs used in the task-specialist layer. To explore the impact the number of SLMs on the annotation performance, we perform ablations on the Jen-Sentiment datasct using GPT-3.5-turbo as the meta-controller LLM. We vary the number of SLMs participating in the multi-model consensus annotation from 2 to and report the final annotation accuracy in Table 10 We find that the annotation
11

---

Published in Transactions on Machine Learning Research (12/2025)
Table 10: The impact of the number of SLMs used for annotation the annotation perfor- mance.
The impact of the number of hard samples used for continuous finc-tuning each stage on the final perfor- mance.
Table 11:
2
3
4
500
1000
2000
3000
Acc 73.32% 78.56%
76.26%
76.26%
Acc 76.17%
73.91%
78.56%
73. 73%12%
performance is best when h 3. We believe that the reason for the decline in annotation performance when k is greater than is that when there are too many SLMs, the performance differences between them will cause the poorly performing SLMs to affect the voting results, ultimately affecting the annotation effect. Nevertheless, Aulo Annotator is flexible enough that it can still operate if the mela-controller layer finds only one or two relevant SLMs after searching. Therefore, considering the computational cost and annotation accuracy, we use k = 3 as the default in this paper.
The number of samples in the hard sample pool hardd To explore the impact of the number of hard samples used for continuous fine-tuning at each stage on the annotation performance, we perform ablations the JPP-Sentiment dataset using GPT-3.5-turbo. We set the sample size 3 to {500, 1000, 2000, 3000). As shown in Table [1] find that the annotation performance peaks at B 2000, therefore, adopt B = 2000 as the default hard-sample batch size in this paper.
The annotation accuracy between SLMs and LLMs on the hard sample pool. To verify that LLMs in- deed improve annotation qual- ity, report the accuracy on Sen-ientmmnnt dataset of SLMs and LLMs for samples in the hard sample pool, us- ing gold labels for evaluation. As shown in Table 12. LLMs consistently outperform SLMs difficult samples, achiev- ing notably higher annotalion accuracy. This also explains why our framework can pro- vide high-quality responses for continual training without human intervention.
Table 12: The annotation accuracy between SLMs and LLMs on the hard sample pool.
Number of Difficult Samples
Model
Acc
cardiffulp/
58.83%
cardiffnip/
het//ttttrrrllme seatttrr--lm-eee roherrtaaaaa hase sentiment
42.97%
2881
finitesutomata,
57.20%
GPT-3.5 3-T--3.5-m 3.5-turbb
65.29%
cardiffulp/ reddifulp/twitter-reoe /tt//tter-oobbrtteeeee wesss-ssntimettlateee
58.39%
curdiffnlp/ taniffinn////// httwt//rw-xxmm-eeeeeeeee -ooore--------taa base-sentiment
42.77%
2836
finitesutomata, //baa//ebw/weee weet wee--base setteeeetttttttteeeeeeee
56.52%
Minimax abab6.5e 5e chatt
68.48%
cardiffulp/ hettllllpptttttteee wtwtttet--obbbreaaaaaaaaeaaeaaaa base-sentimmnt latess
61.73%
curdifinlp/ twitter--dmmrobertae buso-sentiment
47.11%
3148
finitesntomata,
haw.l//tttaaaabb/tebweeeee wertweet. base sentiment analysis
61.50%
Deepseek-V V3
69.85%
Voting using only SLMs. To verify that the performance gains of AutoAnnotator do not come solely from simple SLM ensembles, we introduced "SLMs Voting" baseline. Specifically, use the same 3 SLMs, employ simple majority voting mechanism for data labeling on the JPP-Sentiment dataset, and do not include secondary review by LLMs or expert-guided iterative optimization. As shown in Table [3 SLMs voting even performs worse than the best single SLM. This shows that simple integrated voting can be polluted by poor-performing SLMs (such as SLM2),, leading to a decline overall performance. Tn comparison, AutoAnnotator achieves better results, demonstrating the importance of secondary review and expert-guided iterative optimization.
How to select SLMs. In this paper, we use LLM as the meta-controller to adaptively select SLMs. Here, we conduct additional experiments using existing model selection methods. Specifically, we choose classic and effective model selection algorithm, LogME YYuu et al., (021). and use it for model recommendation on the JPP-Sentiment dataset. Since LogME requires some models to be given advance, we select 10 models for sentiment classification from Huggingface, inchuding the three SLMs used in our paper. We run the LogME algorithm evaluate and recommend the three best SLMs from a pool of 10 models. Then, replaced the models recommended by LLM with the SLMSs recommended by LogME for data labeling.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Table 13: Comparison of the proposed AutoAnnotator with
Table 14:
Ablation experiments on
SLMs Voting.
different SLM selection methods.
Method
Acc
Method
Acc
Method
Acc
SLM1
70.31%
Munok
81.20%
Recommended by LogME
74.57%
SLM2
59.76%
Auto
V3
77.96%
Recommended by LLM 78.56%
SLM3
69.22%
Auto mttoo///mooaltoo--77333...tr
78.56%
SLMs Voting
67.76%
AutoAmootooor+Humann
82.83%
As shown Table 14 compared to SLMs rocommended by LogME (74.57%), SLMs recommended by LLM achieves a higher annotation accuracy (78.56%) under the AuloAnnolator framework. This demonstrates that LLM's powerful reasoning and generalization capabilities enable to form a more comprehensive and profound understanding of annotation tasks, thus surpassing traditional, single-metric- based strategies in model selection.
5 Conclusion
In this paper, we propose new paradigm for multi-model collaborative annotation and designs fully automatic annotation framework AutoAnnotator based it. Specifically, AutoAnnotator consists of a meta-controller layer and task-spocialist layer. Specifically, the meta-controller layer is responsible for recommending appropriate annotation SLMs, generating the code required for ammoattion, and rechecking difficult samples that cannot determined by SLMs. while the task-specialist layer responsible for the actual annotation. To enhance the generalization of the SLMs, we use the difficuul samples obtained from the second verification of the LLM reinforcement learning set, and periodically send the SLMs for continuous fine-tuning. Extensive experiments demonstrate the effectiveness of A on six datasets.
Limitations
While promising, there are still some drawbacks of AutoAnnolaoor. The model selection, code generation, and difficult sample review of the entire framework are all driven by LLMs. Therefore, the performance of Ammotaor depends to certain extent the quality of LLMs.
Acknowledgments
This work was partially supported by the Key R&D Program Zhejiang under Grant 2022C01018 and by the National Natural Science Foundation of China under Grant U21B2001, 62301492 and 61973273, by the Baima Lake Laboratory Joint Fund of Zhejiang Provincial Natural Science Foundation of China under Grant LBMHH225F020002 and by Key Technology Research and Development Program Project of Hangzhou under Grant 2024SZD1A23 and 2025SZD1A41.
2024C01025, by the National Key R&D Program under Grant 2025YFA151090 and 2025YFA151090
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint "XXi:2333308774 2023.
Aida Amini, Saadía Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannanch Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint Xrx:::1905. 13319, 2019.
Yang Ba, Michelle Mancenido, and Rong Pan. Fill in the gaps: Model calibration and generalization with synthetic data. ar Xie preprint orXiv:2 10. 10864, 2024.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Gianni Brauwers and Flavius Frasincar. A survey aspect-based sentiment classification. ACM Computing Surveys, 55(4):1-37, 2022.
Ruirui Chen, Chengwei Qin, Weifeng Jiang, and Dongkyu Choi. Is large language model good annotator for event extraction? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17772 17780, 2024.
Juhwan Choi, Jungmin Yun, Kyohoon Jin, and YoungBin Kim. Multi-news+: Cost-efficient dataset cleansing via Ilm-based data annotation. arXiv preprint arXiu:2444.0963 2024.
Remi Denton, Mark Diaz, Ian Kivlichan, Vinodkumar Prabhakaran, and Rachel Rosen. Whose ground truth? accounting for individual and collective identities underlying dalaset annotation. ar preprinl arXiv:21 12.04554, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 conference of the North American chapter of the association for compulational linguislies: hamman language technologies, volume 1 (long and short papers), pp. 4171 4186, 2019.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference Machine Learning, 2023.
Zeyad Emam, Andrew Kondrich, Sasha Harrison, Felix Lau, Yushi Wang, Aerin Kim, and Elliot Branson. On the state of data in computer vision: Human annotations remain indispensable for developing deep learning models. orXiv preprint arXiv:2108. 000101, 2021.
Zachary Flamholz, Steven J Biller, and Libusha Kelly. Large language models improve annotation of prokaryotic viral proteins. Nature Microbiology, 9(2):537 549, 2024.
Mare Glocker, Peter Hönig, Matthias Hirschmanner,, and Markus Vineze. Llm-empowered embodied agent for memory-angmented task planning in household robotics. arXiv preprint orXiv:2504.21716 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 2rX:::440..21788 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.12948 2025.
Xinlei He, Savvas Zannettou, Yun Shen, and Yang Zhang. You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content. In 2024 IEEE Symposium on Security and Privacy (SP), pp. 770-787. IEEE, 2024.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2023.
Suramya Jadhav, Abhay Shanbhag, Amogh Thakurdesai, Ridhima Sinare, and Raviraj Joshi. On limitations of Il as annotator for low resource languages. arXiv preprint arXio:2411.17637 2024.
Feng Jiang, Zongfei Zhang, and Xin Xu. Cmfdnet: Cross-mamba and feature discovery network for polyp segmentation. In International Conference on Neural Information Processing, pp. 17 30. Springer, 2025.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. survey on large language models for code generation. Xie preprint or 2xXi:22/660055 2024.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. Target-dependent twitter sentiment classifi- cation. In Proceedings of the 49th annual meeting of the associalion for compulational linguislics: human language technologies, pp. 151 160, 2011.
Guangchen Lan, Huseyin A Inan, Sahar Abdelnabi, Janardhan Kulkarni, Lukas Wutschitz, Reza Shokri, Christopher G Brinton, and Robert Sim. Contextual integrity in LLMs via reasoning and reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025.
Md Tahmnid Rahman Laskar, Mizanur Rahman, Israt Jahan, Enamul Hoque, and Jimmy Huang, Can large language models fix data annotation errors? an empirical study using debatepedia for query-focused text summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.
10245 10255, 2023.
Lingyao Li, Lizhou Fan, Shubham Atreja, and Libby Hemphill. "hot" chatgpt: The promise of chatgpt in detecting and discriminating hateful, offensive, and toxic comments on social media. ACM Transactions on the Web, 18(2):1-36, 2024a.
Rui Li, Qi Liu, Liyang He, Zheng Zhang, Hao Zhang, Shengyu Ye, Junyu Lu, and Zhenya Huang. Optimizing code retrieval: High-quality and scalable dataset annotation through large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2053 2063, 2024b.
Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. Lion-fs: Fast & slow video-language thinker as online vidco assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3240 3251, 2025a.
Wei Li, Renshan Zhang, Rui Shao, Jie He, and Liqiang Nie. Cogyla: Cognition-aligned dision-language-eet model via instruction-driven routing sparsification. arXiv preprint arXiv:2508.2100 2025b.
Yunzhe Li, Junting Wang, Hari Sundaram, and Zhining Liu. A zero-shot gencralization framework for llm-driven cross-domain sequential recommendation. arXie preprint arXiv:2501.1922) 2025c.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Fncouraging divergent thinking in large language models through multi-agent debate. arXiv preprint Xiv: 19118, 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv: 1rXie:1705.00 2017.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXie preprint arxiv:2412. 19437, 2024.
Mingchao Liu, Yu Sun, Ruixiao Sun, Xin Dong, Xiang Shen, and Hongyu Xiong. Agentps: Agentic process supervision for content moderation with multimodal Ilms, 2025a. URL https://arxiv.. org/ 2b1/
15251
Xiao Liu, Zirui Wu, Jiayi Li, Zhicheng Shao, Xun Pang, and Yansong Feng. Automated annotation of evolving corpora for augmenting longitudinal network data: A framework integrating large language models and expert knowledge. ar preprint arXiv:2503.0111 01672, 2025b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xrv: 11692, 2019.
Yuwei Lou, Hao Hu, Shaocong Ma, Zongfei Zhang, Liang Wang, Jidong Ge, and Xianping Tao. Drf: Llm-agent dynamic reputation filtering framework. In International Conference on Neural Information Processing, pp. 127-141. Springer, 2025.
Yao Lu, Hao Cheng, Yujie Fang, Zeyu Wang, Jiaheng Wei, Dongwei Xu, Qi Xuan, Xiaoniu Yang, and Zhaowei Zhu. Reassessing layer pruning in 1ms: New insights and methods. ar Xio preprint or arXiv:2 11. 15558, 2024.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Yuhong Mo, Hao Qin, Yushan Dong. Ziyi Zhu, and Zhenglin Li. Large language model (Ilm) ai text generation detection based on transformer deep learning algorithm. arXie preprinl arXiv:2405.06662, 2024.
Christoffer Bøgelund Rasmussen, Kristian Kirk, and Thomas B Moeslund. The challenge of data annotation in deep learning case study on whole plant corn silage. Sensors, 22(4):1596, 2022.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint Xiv:2300 12950, 2023.
Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji-Rong Wen, and Rui Yan. Determlr: Augmenting Ilm-based logical reasoning from indeterminacy to determinacy. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9828 9868, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloinng Ilama model. https://githuu.... com/ tatau-lab/stanfors alpaca 2023.
Kimi Team, Angang Du, Bofci Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement. learning with 1lms. arXiv preprint 2501 12599, 2025.
Ramya Tekumalla and Juan M Banda. Leveraging large language models and weak supervision for social media data annotation: an evaluation using covid-19 self-reported vaccination tweets. In International Conference on Human-Compuler Interuction, pp. 356 366. Springer, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint orXiv:2307.0922 2023.
Betty Van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. Challenges for toxic comment classification: An in-depth error analysis. arXiv preprint (=X.::180.00777. 2018.
Jianxun Wang and Yixiang Chen. A review code generation with Ilms: Application and evaluation. In 2023 IEEE International Conference on Medical Artificial Intelligence (MedAl), pp. 284 289. IEEE, 2023.
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406. 04692, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain--ffthought prompting elicits reasoning in large language models. Advances in neural information processing systems, 95:24824-2488 2022.
Jianfei Wu, Xubin Wang, and Weijia Jia. Enhancing text annotation through rationale-driven collaborative few-shot prompting. arXiv preprint arXiv:2409. 09615, 2024a.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran WI, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen Ilm applications via multi-agent conversa- tions. In First Conference on Language Modeling, 2024b.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint ar arXi::2309.05: 055199 2023.
Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, and Julian McAuley. Small models are valuable plug-ins for large language models. arXiv preprint ar arXin:2305.0884 08844 2023.
Ran Xu, Wenqi Shi, Yuchen Zhuang, Yue Yu, Joyee C Ho, Haoyu Wang, and Carl Yang. Collab-rag: Boosting aangentl-nngeented generation for complex question answering via white-bot box and black-box llm collaboration. ar preprint arXi::254.00095, 2025.
16

---

Less is more for Published in Transactions on Machine Learning Research (12/2025)
Sachin Yadav, Tejaswi Choppa, and Dominik Schlechtweg. Towards automating text annotation: A case study semantic proximity annotation using gpt-4. arXiv preprint axXiv:447.041330 2024.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309. 10305, 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1511 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingleng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang, Owwen2.5-math technical report: Toward mathematical expert model via self- improvemeen.. arXiv preprint arXiv:2400.1 12122, 2024b.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. reasoning. arXio preprint arXiv:2502 2xx::2502.03387 2025.
Limo:
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical asscssment of pre-trained models for transfer learning. In International Conference on Machine Learning, pp. 12133 12143. PMLR, 2021.
Yong Yu, Sifan Zhou, Dawei Yang, Shuoyu Li, Shuo Wang, Xing Hu, Chen Xu, Zukang Xu, Changyong Shu, and Zhihang Yuan. Mquant: Unleashing the inference potential of multimodal large language models via static quantization. In Proceedings of the 33rd ACM International Conference Multimedio, pp. 1783-1792, 2025.
Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. Autodefense: Multi-agent Ilm defense against jailbreak attacks. arXiv preprint or arx::2403.00 04783, 2024.
Yiming Zeng, Jinghan Cao, Zexin Li, Wanhao Yu, Zhankai Ye, Dawei Xiang, Ting Hua, Xin Liu, Shangqian Gao, and Tingting Yu. Hyperedit: Unlocking instruction-based text editing 1ms via hypernetworks. arXiv preprint ar Xr11:2212.. 12544, 2025a.
Yiming Zeng, Wanhao Yu, Zexin Li, Tao Ren, Yu Ma, Jinghan Cao, Xiyan Chen, and Tingting Yu. Bridging the editing gap in 1lms: Fincodit for precisc and targeted text modifications. arXiv e-prints, pp. arXiv- 2502, 2025b.
Kaiyan Zhang, Jianyu Wang, Ermo Hua, Biqing Qi, Ning Ding, and Bowen Zhou. Cogenesis: A frame- work collaborating large and small language models for secure context-aware instruction following, arXiv preprint ar XxXi:2240.0031 2024.
Weitao Zhao, Zati Hakim Azizul, Chaw Seng Woo, Weijie Kuang, and Yafeng Li. Potential-driven multi- learning particle swarm optimisation. Swarm and Evolutionary Compulation, 96: 96:101993, 2025.
Wenhao Zheng, Yixiao Chen, Weitong Zhang, Souvik Kundu, Yun Li, Zhengzhong Liu, Eric P Xing, Hongyi Wang, and Huaxiu Yao. Citer: Collaborative inference for efficient large language model decoding with token-level routing. arXio preprint or arXiu:2502.019%% 2025.
Sifan Zhou, Shuo Wang, Zhihang Yuan, Mingjia Shi, Yuzhang Shang, and Dawei Yang. GSQ-tuning: Group- shared exponents integer in fully quantized training for LLMs on-device fine-tuning. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 22971 22988, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-891766-56-5.
Zhiyin Zhou. Beyond chat: a framework for 1mm as human-centered support systems. arXiv preprint vrXiv:2511.0372 2025.
17


Reviews:
[{"review_id": "yEDPtGRlQs", "reviewer": "Reviewer_6sy8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces AutoAnnotator, a fully automated annotation framework based on a novel multi-model collaborative annotation paradigm. The key contributions of the paper are to automatically choose k models among a large suite of SLM's (using a frontier LLM), then identify if a particular query is difficult and further finetune the SLM's in a continued fashion.", "claims_and_evidence_yn": "No", "claims_explanation": "There are several glaring gaps in the paper\n\n1. In most tasks, there will not exist a finetuned SLM in the first place which can cater to the task. Finetuning the SLM itself will require a large amount of data - where will that data come from if the task is quite niche?\n\n2. The cost of 1656$ is not prohibitive depending on the usecase. However, often there are glaring issues in the quality of annotations from frontier LLMs itself espescially for niche yet important tasks such as NL (natural language) to SQL conversion in large databases. The authors have discussed about data annotation but what is the usecase of such data. If the purpose of such data is finetuning then the system generating the data can be directly used for inference. \"High Quality\" is a subjective term - however the data collected in this fashion will not be enough to improve the frontier/teacher LLM itself.\n\n3. Experiments on more datasets is necessary - code generation and code verification in particular. The two tasks considered are very generic and does not cover the gamut of niche tasks for which data is a bottleneck in the first place. \n\n4. The adaptive model selection layer seems to require 1.69 M LLM queries per task. Why is k fixed? Again, for a niche task, I would imagine that only 1 or 2 SLM's might exist if we are lucky. \n\n5. How are you getting the data for the hard samples? There is a disagreement between the SLM's so they are not trustworthy and the LLM is worse than the SLM according to the paper? So where are the high quality responses for continual training coming from in the framework where there is no manual involvement?", "audience_yn": "No", "audience_explanation": "At this moment, the claims of this paper have glaring gaps.", "requested_changes": "Please provide detailed answers to my questions and also expand the coverage of experiments.", "broader_impact_concerns": "None", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}, {"review_id": "k6DH6syLbF", "reviewer": "Reviewer_TGDB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes AutoAnnotator, a two-layer automated annotation framework that combines LLMs (meta-controller) and SLMs (task specialists). The LLM layer handles model selection, automatic code generation, and secondary review of difficult samples, while the SLM layer conducts consensus-based annotation and iterative fine-tuning. Experiments on six sentiment and toxicity datasets demonstrate significant improvements in accuracy and cost efficiency.\n\nStrengths:\n1. Clear motivation: Tackles high costs and low domain-specific accuracy of LLM based annotation.The annotation is always expensive, which is an important topic.\n2. Strong experimental results: Outperforms both standalone SLMs and LLMs across zero-shot, one-shot, CoT, and majority voting baselines.\n3. Practical value: Demonstrates that lightweight models can anchor annotation while selectively leveraging LLMs.\n\nWeakness:\n1. The evaluation is restricted to sentiment and toxicity classification, which are relatively shallow NLP tasks. It remains unclear whether the framework generalizes to complex reasoning tasks (e.g., multi-label, causal inference) or non-text modalities (vision, speech).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments are conducted on several of the major LLMs, yielding consistent results.", "audience_yn": "Yes", "audience_explanation": "Yes. I believe this approach (based on the results of extensive experiments) is suitable for the NLP task at this point, and annotation is always very expensive for humans. This work reduces the cost by introducing the framework.", "requested_changes": "I wanted to check if the work can be extended to non-text data or a multi-label dataset. But, I understand that may be a different topic.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XKxRRjqeUb", "reviewer": "Reviewer_yg5N", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an auto labeling framework tailored to two downstream tasks, sentiment analysis and toxicity classification, which reduces inference costs by iteratively fine-tuning openly available SLMs for LLM-labelled data points with prediction disagreements.\n\nThe approach is evaluated for the two tasks on selected datasets and compared to individual SLM performance, standard majority voting, open source LLMs and APIs.\n\nThe results show that the LLM-guided approach is partially better compared to the best individual SLMs. It is better than general LLMs and APIs though.", "claims_and_evidence_yn": "No", "claims_explanation": "- The task layer problem is strongly related and could partially be reduced to classical expert advice / mixture of experts / boosting, but no baseline is used at all (except for standard majority voting). This literature is not discussed in the related works. It is also missing as baselines for the empirical evaluation.\n- Also, the approach should be compared to classical teacher student architecture / knowledge distillation.\n- There are also more general solutions for LLM available such as Mixture-of-Agents (https://openreview.net/pdf?id=h0ZfDIrj7T)\n- The paper title suggest a general annotation framework, but the evaluation only targets sentiment analysis and toxicity classification. This seems like an overstatement\n- Even though ablation study was done, the approach seems very dependent on hyperparameter k being small, which feels like the majority voting might not be enough\n- The Meta-Controller Layer is not well-evaluated. It seems the SLMs were chosen by the layer, but it is unclear how many other options would be available (i.e., how good this choice is globally) or how difficult this task even is. Here, it would be important to check if selection diversity wrt predicted classes is important.\n- The evaluation results for the LLM-guided SLM fine-tuning are not always better than using individual SLMs. This increases importance to test simple weighting of SLMs based on their performances on held out validation data. There are prominent model selection methods one could also compare to (based on a pool of \"general\" options for the task)", "audience_yn": "Yes", "audience_explanation": "- In general, the paper's goals and possible findings are of interest for sure, as they would support reducing API costs and improving predictive performance. I believe, however, that the evaluation and presentation of the paper needs to be improved for that.", "requested_changes": "Please change the manuscript and evaluation based on the mentioned shortcomings above plus by answering these questions:\n- How does the approach related to mentioned foundational works, e.g., expert advice / mixture of experts / teacher-student architectures?\n- How does the approach related to mixture of agents? \n- Do you have an idea why k=3 is performing the best? Why can the system not efficiently use more than 3 SLMs? Maybe the 5 chosen SLMs are not diverse enough, so individual \"brilliant\" SLMs are not strongly weighted enough?\n- Are the combinations of SLMs of size 3 which might not work together in your approach?\n- Why is not possible to compare against some of the mentioned LLM combination papers you mention in your related works?\n- What are the theoretical requirements for the SLM performances? If no disagreement occurs and the SLMs perform bad, you never improve.\n- What if one gives an LLM or API some more few-shot examples based on a held-out validation set? Could this be more competitive? \n- In the one-shot setting, is there really only one training example passed in-context?", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)