ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
Authors: Lequan Lin
Dai Shi
Andi Han
Feng Chen
Qiuzheng Chen
Jiawen Li
Zhaoyang Li
Jiyuan Zhang
Zhenbang Sun
Junbin Gao
Venue: NeurIPS 2025 poster ()
Keywords: data annotation
data pipeline
multimodal large language models
llm-as-a-judge
supervised learning
Primary area: deep_learning
TL;DR: This paper proposes the ACT data pipeline, which reduces human annotation costs by using MLLMs as annotators and error detectors, and provides a theoretical analysis to ensure effective downstream training.

Abstract: Supervised learning relies on high-quality labeled data, but obtaining such data through human annotation is both expensive and time-consuming. Recent work explores using large language models (LLMs) for annotation, but LLM-generated labels still fall short of human-level quality. To address this problem, we propose the Annotation with Critical Thinking (ACT) data pipeline, where LLMs serve not only as annotators but also as judges to critically identify potential errors.  Human effort is then directed towards reviewing only the most "suspicious" cases, significantly improving the human annotation efficiency. Our major contributions are as follows: (1) ACT is applicable to a wide range of domains, including natural language processing (NLP), computer vision (CV), and multimodal understanding, by leveraging multimodal-LLMs (MLLMs). (2) Through empirical studies, we derive 7 insights on how to enhance annotation quality while efficiently reducing the human cost, and then translate these findings into user-friendly guidelines. (3) We theoretically analyze how to modify the loss function so that models trained on ACT data achieve similar performance to those trained on fully human-annotated data. Our experiments show that the performance gap can be reduced to less than 2% on most benchmark datasets while saving up to 90% of human costs.

ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
Lequan Lin
Dai Shi 1,2 Andi Han 1,3
Feng Chen
Qiuzheng Chen $
Jiawen Li
Zhaoyang Li
Jiyuan Zhang
Zhenbang Sun Junbin Gao
University of Sydney, Australia
University of Cambridge, United Kingdom
Riken AIP, Japan
University of Adelaide, Australia
5 ByteDance, Australia
Abstract
Supervised learning relies on high-quality labeled data, but obtaining such data through human annotation is both expensive and time-consuming, Recent work explores using large language models (LLMs) for annotation, but LLM-generated labels still fall short of human-level quality. To address this problem, we propose the Annotation with Critical Thinking (ACT) data pipeline, where LLMs serve not only annotators but also as judges to critically identify potential errors. Human effort is then directed towards reviewing only the most "suspicious" cases, significantly improving the human annotation efficiency. Our major contributions are as follows: (1) ACT is applicable to wide range of domains, including natural language processing (NLP), computer vision (CV), and multimodal understanding, by leveraging multimdal--LMM (MLLMs). (2) Through empirical studies, we derive insights how enhance annotation quality while efficiently reducing the human cost, and then translate these findings into user-friendly guidelines. (3) We theoretically analyze how to modify the loss function so that models trained on ACT data achieve similar performance to those trained on fully human- annottted data. Our experiments show that the performance gap can be reduced less than 2% on most benchmark datasets while saving up to 90% of human costs.
Introduction
High-quality labeled data is essential for the success of supervised learning, but human annotation
remains costly and difficult scale 22 [5] While large language models (LLMs) have recently
7 g 9 I0], the labels they generate often lack the The trade-off between annotation cost and label quality leads to our research question: How can we incorporate LLMs into the data pipeline to efficiently reduce human cost without compromising downstream training performance? To address this challenge. first propose the Annotation with Critical Thinking (ACT) data pipeline. In this approach, an LLM handles the majority of annotation workloads, while a limited human budget is strategically allocated to reviewing samples flagged potentially erroneous by another LLM-based error detector (the criticizer). Then, we provide theoretical analysis of how to ensure that models trained on ACT data comparable those trained on fully illustration the ACT data pipeline and downstream training is provided in Figure [
emerged as alternative for data annotation 6
accuracy required training reliable models 11
12 13).
data. An
While the majority of M..M-based data pipelines solely focus natural language processing (NLP) 14 [5 16 17], our method further extends to visual scenarios by leveraging multimodal-LLMs
"The work done during Lequan Lin's internship ByteDance. lequan lindaydney. -eataeeyyyddd..d edu.au. Project lead.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ACT Data Pipeline- - An Image Classification Example
User Guidelines
Untabelled
MLLM-odiiatted Probabilit
Labels
ACT Data Annotation
(Vanill dererully goed without moskt selectiond Default settings
MLLM ceicter CheeghT Ao with and protush direcity gamerated the
(Pro opdivize annotaaioo efficinncy via model selectiont Steps modiee selection:
Prepare the cand dasa MLLK's and asmall set human unnceatee cate cesting:
Human
she anrossion uccur acy dase MLLM was the cuesut ss deteuttasttiga
scurssy
S
ACT Downstream Training
4GT Lom
Sampiing
140.20
"hoedhaldingg chhone cezoonn-ommppe
activaate protabi thondrric.
Traninnstrviiegg
inference otla w pie traited modes Transing
Act ve Instimation See losa functoo Scuation 18
Figure Illustration of the ACT data pipeline (left) and the user guidelines (right).
(MLLMs), thus supporting a wider range of practical domains, such as computer vision (CV) and multimodal understanding. In addition, unlike some existing methods that are limited to white-box models (i.e., those with accessible internal representations such as logit scores) [18] 19, our approach agnostic to model accessibility. This allows us effectively unify both white-box and black-box MLLMs, and enables the exploration into a broader spectrum of model families. Furthermore, in contrast to some related works [66 I7 19], our data pipeline is training-free, waiving the need for additional training costs on annotators or criticizers.
The primary goal of this paper is to introduce practically valuable and user-friendly data annotation pipeline. Specifically. we aim to enable future users-when faced with new data annotation task-to quickly determine how set up the entire pipeline and seamlessly deploy in production. To this end, we preface a summary of key takeaways in the form of user guidelines in Figure I which distill the core insights from subsequent explorations in our paper into actionable steps.
The main discussions in this paper are divided into two parts, each addressing an important aspect of the annotation and training process. The first part (Sections 2 to 4 focuses on the annotation stage, where we explore how to choose appropriate MLLMs and prompt strategies to enhance label accuracy while efficiently reducing human effort. The second part (Section[5) addresses the challenge of downstream training. We theoretically analyze how to optimally sample data for human review and appropriately modify the loss function, such that models trained on ACT data achieve similar performance to those trained on fully buman-annotated datasets. Empirically, we validate that, using the loss modified from active M-estimation (20), the performance gap can be narrowed within 2% while saving up to 90% of human costs.
2 ACT Data Pipeline: Formulation & Key Evaluation Metrics
In this section, we provide a formal mathematical formulation of the ACT data pipeline and introduce key metrics evaluate its performance.
Data Annotation. Let X' be a space of unlabeled data with a finite space of labels y. We then denote the dataset D {xx, (x) E X, ys Y, i I), where the ground truth label y, is assumed to be unknown, and I - .2..... N} is a set of indices. We formulate any data annotation approach as a function from the functional space F { X' + >)} that maps the unlabeled data labels. Ideally, given a dataset D, the optimal data annotation f E F satisfies (xx(x) = yr, Vi E I. We further denote the sets of all annotations conducted by humans and machines as (Fh) C F and Frm) F, respectively. Without loss of generality, we consider a representative human annotator f(h) E F(h) and an MLL.M machine annotator f(mm) E F[(m) in following discussions. Let Di (h) = (x,, and de (m) = f(m)) be the human- and machine-annotale labels, respectively.
Measure Annotation Quality. We define scalar function Q F x F + [Qmin, Quaxl as the quality measure. Formally, we evaluate the quality of an annotation method f F as:
1 Σ (xx) SEI
Qf f) = ExEx (x)) x
(I)
N
where s(.,)) is a similarity function defined on the label space, such as 0-1 loss when Q is accuracy, thus the higher value of Q indicates better quality. We generally observe that , f()) > (( indicating that human annotation is usually more accurate than the machine. Following the common practice of supervised learning, where
labels are used as the ground
2

---

truth labels when the latter is not available, we assume that D/()) ys for most if not all i E Z, thus Q(f", f(h) so + f*)
Qmax.
Review Machine Errors with Human. Assume that 6 {0, 11 is a machine error indicator,
where 6, 1 means 0/(h) # yi, and 0 otherwise. Then, the refinement of machine annotation
can be done via correction operator: Kss (xx,) = (1 6,) f f(m) (xx) 6ef (x;). Practically, regarding aanaaat-dntttedd labels as the ground truth, the correction operator can be approximated by
the human-correction operator: (s[f(m)[(x)) se (h) [x(m)((x)) Accordingly, our goal is find å for the following optimization problem:
(1 6;) flml ((h) (x:).
max Q (f(h) K8
(h) (f(m)))
(2)
Ideally, the optimal solution is the Kronecker delta 6 bgmm).ga such that (p(m)) gg 1 if p(m))
0
and 0 otherwise. This means we replace the erroneous machine-annotater labels with the human- annotated labels.
ACT Data Pipeline. Now the problem is, we do not know when the machine makes mistakes.
Previous works have validated the judgmental capabilities of LLMs, where LLM evaluates answers generated by another LLM or itself to enhance the generator's inference quality 21 [2] 23 24).
Enlightened by this idea, adopt pre-trained MILM g as the criticizer determine whether
the annotator (m) assigns the wrong label for each data pair (x: :/, ) Considering that human resources are limited, instead of generating binary decisions, we query the criticizer to estimate the probability of error. Then, we sample the data for human review based on this estimated probability, ensuring that the total sample size does not exceed a given human budget B. Mathematically, let (m))
|x;) denote the true error probability, which then estimated by & = g(x,, V/ m/ Recall that N is the dataset size. Given human budget B S N, we define budget-aware Here, B denotes the Bernoulli distribution, and TB(-) is transformation that adjusts the error probability é; based on the budget B, ensuring that Σx (B) S B. In this paper, we consider the following sampling rules:
& = P(y: * V.
sampling function 8(B) where (B)
Normalization 20 TB(E:) (B e,) Σx & such that d.),(B) S B.
Expomential Weighting: (&,) 1/(1 + e (()),, where a E [0. and 93 E IR, are hyperparameters that control the center and sharpness of the transformed error distribution. The hyperparameters are set to satisfy Σx (B) < B.
Thresholding: BE)) 1(4 2 r), where T is a sampling threshold chosen such that d:B))= < B. This is special case of exponential weighting when and / co.
With the human budget B, our ultimate goal is modified from Equation (2) by adding the budget constraint as
max Q (f(h), (h) 8(B)
( (5(m))) s.t.
Σ 5t(B) < B,
T
(b) where (8(B) (xx)) = (1 - (B) f( (xm) + (N) is the budget-constrained human
correction operator.
As summary, the ACT data pipeline consists of three steps: (1)Annotation: generating MILM- annotated labels for all unlabeled data in the dataset; (2) Error estimation: utilizing another MLLM as the criticizer to estimate the error probability for each annotation job; (3) Correction: sample data with high likelihood of machine annotation errors for human review and correction, with the sample size controlled by a human budget.
Metrics. In the following sections, we will evaluate the effectiveness of various MLLMs and prompt strategies as components of the ACT data pipeline. Here, we propose two key metries: (1) Annotation Quality Gain (AQG): measurement of the annotation quality improved by ACT from machine annotation given a fixed human budget B: (2) Area under Budget Sensitivity (ABS): measurement of the overall efficiency of human budget, for which higher value implies better budget utilization.
Definition 2.1 (Annotation Quality Gain). With a machine annotator f(m)), a machine criticizer (h) and correction operator ((B) under human budget B, where 8, (B) BB(T(())) with é =
3

---

g(X,, (x,)) we define the Annotation Quality Gain (AQG) of the data pipeline as
AQG (flm), Q(f(h),
(k) K KBBB (1 [f(m)] Q f(m))
(4)
Qmax Q (f(h) f(m))
where the denominator Quax Q(f()), f(m)) scales AQG into range [0, 1], which offsets the influence of room for improvement.
Definition 2.2 (Area under Budget Sensitivity). Let E [0, 1] be the proportion of human budget over the dataset, such that B = [, where [-] means round down to the largest integer. With machine annotator f(m) and machine criticizer 9. we define the Area under Budget Sensitivity (ABS) of the data pipeline as
ABS (f(m), = [ AQG (flm), (o.l db
Σ AQG (f(m), B=0
(5)
N
3 Experimental Details
Datasets.
Since MLLMs are capable of handling multiple data types, we consider comprehensive
variety of annotation tasks, including image classification: CIFARIO [25), Fashion-MNIIST (Fashion)
(26). and Stanford Cars (Cars) [27]: text classification: Emotion 28 and Irony 28 and vision
questio--answeringg (VQA): the close-ended VQA-RAD
29
MLLMs, We conduct experiments using models from six prominent MLLM families: Chat- GPT 4o 2024-08-06 (GPT4o) [30]. Gemini-1.5-Prou
(Gemini1.5P) [I], Claude 3.5 Sonnet (Claude3.55 32]. LLaVA One Vision 72B (LLaVA-OV) [3), Qwen 2.5 VL 72B (Qwen2.5VL) B4], and Intern VI. 2.5 78B (Intern
35 Among these models, LLAVA-OV, Qwen2.5VL, and V1.255 are white-box models that provide access to intermediate outputs such as logit scores, facilitating further fine-grained analyses. In contrast, GPT4o, Geminil.5p and Claude3.55 black-box models, offering only final responses. Larger models are generally associated with better criticism functionality (24), hence, select white-box models with large parameter sizes each ex- ceeding 70 billion, to evaluate the data pipeline. For all models, we adopt sampling hyperparameters p 0.9,t 0.7, k 50, and max new tokens 500 when applicable.
Downstream Models. We employ three pre-trained models for our downstream experiments. For image classification tasks,
ResNet18 (36). initialized with default weights pretrained on ImageNet-I1 (37]. For text classification, we utilize the RoBERTa-base model (RoBERTa) [3 For the VQA task, adopt the BLIP-VQA model (blip-vaa-base)
339
More experimental details of the datasets and downstream training can be found in Appendix
What Makes ACT Work Better?
In this section, we investigate how to boost ACT's performance by systematically exploring each component, namely, the machine annotator, the criticizer, and the associated prompt strategies. We choose accuracy as the annotation quality measure. In addition, we do not assume a fixed human annotation budget and adopt ABS as the evaluation metric to capture the pipeline's 's overall efficiency in leveraging human resources to boost the annotation quality. In consideration time and computational costs, results reported are from single run per experiment; but a stability analysis in Appendix demonstrates the robustness of the observations. Examples of prompt strategies explored provided in Appendix C
4.1 Exploration of Machine Annotators
We consider two types of prompt strategies for machine annotators: (1) Naive annotation (naive): directly output the label; (2) Annotation with Chain-of-Thougee (CoT) 40) output the step-by-step reasoning with the label. The results are shown in Figure 2
Insight 1: GPT4o is generally promising annotator. We observe that GPT4o achieves the highest annotation accuracy on most datasets. While LLaVA-OV performs best on VQA-RAD, it struggles with image classification tasks (CIFAR-10 and Fashion).
Insight 2: CoT is not consistently helpful with annotation. The results do not show clear benefit from using CoT overall. However, for GPT4o specifically, CoT improves performance across all

---

tasks. Therefore, in the following exploration of criticizers, we will adopt GPT4o with the CoT prompting strategy as the base annotator.
4.2 Exploration of Machine Criticizers
Recall that we categorize MLLMs into two types: black-box and white-box, based on whether they provide access to the underlying logit probabilities. Accordingly, we design several strategies for each type to measure the error probabilities of the annotator. For simplicity, we use the thresholding sampling rule in the criticizer experiments. In Appendix B we demonstrate that different sampling rules have minimal impact on the ABS results.
4.2.1 Black-box Criticism Strategies
Black-box criticism strategies refer to methods that query models for error elicitation the response token space. These strategies are broadly applicable to any model with chatbot functions, which means they can also be applied to white-box models. In our study, we evaluate four black-bor strategies. While they may not encompass every possible approach, they are representative and straightforward to implement. A detailed description of these strategies is provided in Table Since the strategy implicitly measures error probability through error levels, sampling with random selection when multiple options fit the budget. Other sampling rules may not directly apply to this strategy. Note that the mc and devil strategies are CoT variants differing mainly in their error measurement approach. The results of the black-box strategies are presented in Figure 3
preshold-basedd
Table 1: Details of black-box criticism strategies.
Strategy
Description
Input (Prompt) + Output (Response)
Naive Estimation
Directly estimate the error probability based on the given data and label.
(naïve)
data
label
error_pie _probp_pbbbb
Chain-of-Thought (CoT) [40)
Output step-by-step reasoning and error probability.
data
label
CoT
error _porrppobb
Select from predefined emror levels from to where higher level means higher error probability.
Multiple Choice
(mc) 41 42 [3)
data
label
CoT
error level
Devil's Advocate (devil)
Critically assess the CoT produced by the annotator and estimate error probability.
data
CoT_A
CoT
error- _prbb
* "CooT_ the CoT obtained from the machine annotator.
Insight 3: Cross-criticism outperforms self-criticism with black-box strategies. Recall that we GPT40 the machine annotator. When GPT4o also serves the criticizer, the process is referred to as self-criticism. In contrast, when the criticizer is another model, the process is known as cross- criticism. Figure 3 indicates that the best ABS scores are typically achieved through cross-criticism. However, the performance of self-criticism remains highly competitive in most datasets.
Insight 4: Black-box models are better criticizers with black-box strategies. We generally observe that black-box models (GPT4o, Geminil .5P, & Claude3.55 3,55) achieve higher ABS scores than white- box models VLLOV Qwen2.5VL, & InternVL2.55 when using black-box strategies. Specifically, GPT4o and Geminil .5P tend to perform better on image classification tasks, while Claude3.5$ demonstrates stronger performance language-focuset tasks, including text classification and VQA.
Insight 5: CoT is more helpful with criticism than annotation. Compared to our observations with the machine annotator, CoT consistently shows more benefits in the criticism process, leading to an ABS improvement of up to 22.46% over the naive strategy. We suppose that this is because
CIFAR10
Fasbion
Emotion
VOA-RAD
Dalon
datteer
pareal Avt
Cover 103 Meeeniza
Figure 2: Accuracy of various models as machine annotators with different prompt strategies.
5

---

detecting errors requires explicit reasoning and justification, whereas annotation often relies more on direct pattern recognition. CoT also demonstrates a clear advantage over other prompt strategies. achieving the best results on 4 out of the 6 datasets.
4.2.2 White-box Criticism Strategies
With white-box models, we propose two approaches to estimate the error probability: logit probability and perplexity (PPL). In the logit probability approach, we ask the criticizer to predict there an eπτor and answer "yes" or "no". The estimated error probability is then calculated as:
? = ("P"yes""((("ye"""""" + TP ))),
(6)
where P("yes") and ")"no") denote the logit probabilities assigned to tokens "yes" and "no". In the PPL-based approach, adopt the CoT prompt strategy and indirectly measure the error by computing the PPL of the criticizer's s COT. We note that only the threshold- based sampling is applicable for PPL, are prioritized based on the following selection order:
cannot directly estimate the error probability. During sampling, instances
PPLL( CoT
"yes") -1 ( ""ye"" >1 PPL CoT 'no""
-| PPL CoT "noo", (7)
where PPL( CoT
"yes") and PPL( CoT ""no" are corresponding to the PPL of the CoT when the final decision "yes" and "no":: and + means higher and lower PPL values; and > indicates higher priority in the sampling process. For further clarity, we summarize the white-box strategies in The comparison between the black- and white-box strategies are shown in Figure
Table 2
Table 2: Details of white-box criticism strategies.
Input (Prompt) - Output (Response)
Strategy
Description
+ Error Measurement
Naive Estimation
Output "yee" or "no" based on the given data and label. Error probability estimated by the logit probability.
data
label
"yea" """o"
with Logit Probability (naive-logit) [46]47] 48
Error estimation via logit probabilities
Chain-of-Thooe Thought with Logit Probability (Cogi)
Output step step-by-ste step
beesstteeeeaeeee reasoning before "yes" "ne". Frvor probability is estimated by
data
& [labe]
CCoT & "yes"/"no"
the logit probability.
+ Error estimation via logit probabilities
Chain-of- Thought with Perplexity (CoT-PPL.)
Output step-hy-ster reasoning hefore "yes" or "no". Errrr is indirectly measured by PPL of the CoT.
data & [labe]
CoT & "yas"/"no"
PPI. calculatiion
Insight 6: White-box strategies can occasionally enhance criticism performance. Compared to black-box strategies, models often achieve higher ABS scores with white-box methods, particularly with naive-logit and CoT-PPL. This improvement is more consistently observed with InternVL2.5, while LLaVA-OV and Qwen2.5VL show smaller gains.
CIFARIO
Fashion
Care
75.5
30
533
E7.
545
51.0
56.7
54.5
36,1
69
OCT
60.8
67.1
76.1
72.0
165
62.8
61.1
642
664
14.1
09.5
26.4
41
413
TLB
264
654
71.0
664
18.
64.0
65.4
67.4
424
363
515
Manokoo my
Chonde-F
davel.te
Emotion
Innny
VIA-HAA
66
72.8
72.3
:2.4
0..
537
56.3
55.5
513
632
53.4
546
53.
60
%44
20.8
647
632
565
60.0
54
odl
G
GL2
4%
666
514
46
G6.9
Domant
Demabe
Figure 3: Results of black-box strategies. The metric shown is ABS ()). where higher values indicate better annotation efficiency of the ACT data pipeline. The best results are highlighted with black frames.

---

CHARIO
Paskion
Care
68
20
LLAWW
Oven TML
Owert 9VL
LLKWWW
Ovent TVL
46
LLINAOV
OVeL25LL
LLAVWOV
OVM2
Pesl Black
Figure 4: Comparison of ABS (%) for the same model using black-box and white-box strategies. Green bars correspond to black-box strategies (B), while blue bars represent white-box strategies (W). The horizontal dashed lines indicate the best black-box results for reference.
Insight 7: White-box strategies do not consistently outperform black-box strategies. Although white-box methods occasionally outperform the best black-box ABS, this occurs in only 2 of the datasets tested. The inconsistent performance arises primarily because white-box strategies cannot leverage powerful black-box models, limiting their effectiveness on the same tasks. Therefore, conclude that black-box strategies remain more promising for practical use based on current results. However, we also conjecture that, as MLLMS continue advance, future white-box model with capabilities comparable to black-box models could make it possible for white-box strategies to consistently achieve superior performance.
4.3 Bridging Insights and Practical Usability of the ACT Data Pipeline
From the explorations above, have gained some insights that lead practical guidelines: (1) GPT4o with the CoT strategy can serve as a generally good annotator; (2) Black-box criticism strategies, especially the CoT strategy, provide more generalized practical values with current MIL.M;; (3) While GPT4o self-criticism is promising setting for the ACT data pipeline, the best ABS is typically achieved through cross-criticism. Building these, further examine model selection to offer guidance that makes ACT more user-friendly.
The key question of interest is: for given dataset, how do we choose the annotator and criticizer to maximize pipeline efficiency? Of course, the effortless default is GPT4o self-criticism with CoT, which often works well but may not be optimal. To To facilitate better selection, we propose a simple but effective approach based empirical observations (Figure 5) First, annotation and criticism abilities are positively correlated, implying that models with strong annotation ability are also likely to perform well as criticizers. Second, across datasets, the best criticism ABS consistently achieved by using the top-performing annotator and the model with second-best annotation ability (i.e., best among models other than the annotator) as criticizer. We therefore recommend: (1) obtaining a small test set of human-annolated samples; (2) evaluating annotation ability across available MLLMs; and (3) selecting the best model as the annotator and the second-best as the criticizer.
CIFAR10 CAngetator: (PT4e)
Pushion (Anostator:: GPE4e)
Emetiio (Ansotator: GFT4c)
VQA- RAD (Annoiltter: LLaVA
Cleck
76
70.0 125 113
(GTT Stratag))
Stratngg
Atsotataa Accutaay A00 Strungs)
Siage 12g32m A35
Sape ligher
Hkgher
Stope
Figure 5: Relationship between annotation and criticism abilities. The base annotators for criticism ability evaluation are shown in the titles (all with top I annotation ability from Figure Each point represents a model. The models with the highest ABS are highlighted with stars. The slopes of the fitted regression lines are indicated in the legends.

---

ACT as Human: Enhance Downstream Performance with Modified Loss
5.1 Theoretical Discussions of Downstream Training with ACT Data
Problem: The Downstream Challenge. With a dataset annotated by the ACT data pipeline (referred to as ACT data), the next challenge lies effectively leveraging it for downstream tasks, such as supervised fine-tuning (SFT). The ACT data consists of two components: the high-quality human- annotated data, and the comparatively lower-quality machine-annotater data, which may include erronoous labels. Relying solely when human budget is limited. On the other hand, incorporating both human- and machine-annotuan data risks introducing noisy labels, which may disturb the learning process and Iead to suboptimal performance in downstream tasks.
data results in significant data loss, particularly
Solution: Active M-estimation. A potential solution to the downstream challenge is active M- estimation, which proposed a modified loss to ensure effective statistical properties that lead to promising downstream training [T7 20]. In this work, we adapt the modified loss to ACT data and design the ACT loss as follows:
2 ((CC))
(lo)) + (fo. elm))
(B)
(8)
TB TB(())
where la and e(T)) are losses computed with ground truth and machine-amnotated data, respectively. (B) (+)) the eπror indicator as discussed in Section 2 In practical implementation, ground foound-truth loss is usually estimated with the gigh--uulity muman-annotatee data.
According to Proposition 5. (see proof in Appendix D). while the ACT loss is an unbiased estimate of the ground truth loss, note that the variance of ACT loss is minimized when the expectation term in Equation (9) equals to 0. This can be satisfied or mostly satisfied with: (1) perfect machine annotator: the machine annotator is perfectly accurate such that g(m) ((h) lg; (2) astute machine criticizer: when the machine annotation is not sufficiently accurate, the second term can be reduced by accurate error probability estimation, i.e., (é))) + 1 when machine makes mistakes. This further demonstrates the importance our exploration Section
Proposition 5.1 Statistical Properties of ACT Loss
Suppose La = E(Lo(x, y)) is a p-strongly convex loss. Let p(mm) = lo(x, jf(m)) ) be the loss computed with machine machine-annotated data. Then, the ACT loss defined in Equation (8) an unbiased loss estimate: E (c) -Lo [20]. The variance of the ACT loss is given by:
Var (cLACT)
Var (Lo) + E ( ()
TT(() (
1)
Important Factor: The Sampling Rule. In Section 2 we introduced three sampling rules to keep the volume of data for human review within a pre-specified human budget. While all rules adapt to the ACT loss and share similar statistical properties, they can yield very different training outcomes. We show via Theorem 5.2 (see proof in Appendix D) that when the machine annotator fixed (i.e., C is fixed), the upper bound of the gap between parameters learned with ACT loss and ground truth loss decided by q, the lower bound of the transformed errors of samples reviewed by human. A small g increases the likelihood of large parameter gap and poor model performance. Active M-estimation small.
17 20 uses the normalization rule, but this may perform poorly when the human budget This is because (&) (B x e Σx & becomes close to 0 (g + 0) when B is small compared to the accumulation of errors in a large dataset. To address this, we propose exponential weighting and thresholding rules, which map errors of selected samples close to 1 (g - 1), ensuring small parameter gap, and eventually, model performance similar the ground truth loss. Further details of the ACT losses with different sampling rules are presented in Appendix P

---

Theorem 5.2 Probabilistic Upper Bound of the Parameter Gap
Let N be the dataset size. Suppose &g is j-sttoonly convex, and there exist constants q, C. > 0 such that the transformed error ι B ((6)) 2 q for all i with 6,(B) 1, and the gradient gap
| -Va, I C. for all i E {1, ...,,.......... N.. Then, for an arbitrary E (0, 1), with
probability at least 1 - p, we can bound
8c1 (2pp)
[I0!ACT) - -0.11 5
(10)
H2N
where ACTT arg ming Co C(ACT), 0. = arg ming Lo, and e1 -1-
5.2 Experiments of Downstream Training with ACT Losses
To empirically support our theoretical analysis, we conducted experiments across various data and loss combinations, with results presented in Table Without affecting the theoretical validity, we adopt the Cross-entropy loss as la and apply the power-tuning hyperparameter for all ACT losses following [17]. We used GPT4o with the CoT strategy for both annotation and criticism as the default settings of ACT. The human budget is determined by GPT4o's annotation accuracy-for instance, with 88.48% accuracy on CIFAR10, the human budget is set 11. 11.52% of the dataset. We treat this as the "ideal" budget, as it is the least budget we shall assign to fix all machine errors. In practice, however, the human budget should be set based on available resources. To further enhance the completeness of our analyses, we provide the sensitivity analysis of human budget in Appendix F
We highlight four key observations from the results. First, training solely on machine-annotated data yields up to 10. 15% lower accuracy compared to human-annotated data, whereas ACT reduces the performance gap to less than 2% for most datasets while saving approximately 70% 90% human costs. Second, exponential weighting (exp.) and thresholding (thre.) outperform normalization (norm.), especially under limited human budgets. On Cars, with ideal budget of 9.56%, normaliza- tion leaves 76.34% gap from full supervision, while exponential weighting and thresholding reduce it to 1.69% and 1.88%. Third, ACT data with Cross-entropy loss also perform comparably well, as is a special case of thresholding (see Appendix[E] E). Lastly, exponential weighting and thresholding perform similarly overall. Although exponential weighting often yields slightly better results, we recommend thresholding for its simplicity, as the threshold is easier to decide with given human budget than the hyperparameters the exponential weighting transform (a and 33)
Table 3: Results of downstream training- test accuracy (%) in form of mean + std over 5 runs.
Training Dala Loss
CIFARIO (ResNet18)
Fashion (ResNet18)
Cars (ResNet18)
Emotion (RoBERTa)
Irony (RoBERTa)
VQA-RAD (BLIP-VQA)
Human only Cross-cntropy Loss
88.66 + 0.97
93.01 + 0.63
87,88 + 0.36
81.82 + 0.57
70.18 +0.18:30 3.23
67.81 + 1.47
Machine only - Cross-entropy Loss
81.55 + 1.93
82.86 + 0.84
83.68 + 0.17
78 96 + 2.40
60.71 + 5.43
61.03 + 2.05
ACT data- Cross entropy loss
85.59 + 0.52
87.50 + 0.86
85.88 + 0.26
80.82 + 1.08
67.83
2.82
61.83
3,27
ACT data - ACT norm, loss
64.70
+: 5.46
69.27
7.25
11.54 + 0.96
79.87 + 0.88
65.66
+: 2.00
62.55
3.01
ACT data ACT exp. loss (Ours)
87.73
0.36
89.73
0.35
86.19
+ 0.14
81.44
+ 0.51
68.49
3.20
67.73 1.33
ACT data - ACT thre. loss (Ours)
87.95 + 0.35
89.16 + 0.89
86.00 + 0.26
81.41 + 0.64
68.21 + 1.94
67.02 + 1.32
Mumann-Mchhiee performance gap (%k
7.11%
10.15%
4.20%
2.86%
9.47%
6.87%
Human ACT performance gop (%)
0.71%
3.28%
1.69%
0.38%
1.69%
0.08%
ACT human budget (%)
11.52%
21.81%
9.56%
17.98%
33.79%
30%
6 Related Works
Data Annotation with LLMs LLMs' annotation ability has been widely studied, but mainly in NLP
66.7 13) The most relevant to our work is CDI [17, which uses a trained XGBoost [49 to detect LLM errors for human review. However, CDI (1) employs an error detector that lacks flexibility, as often requires task-specific design and training with additional data; and (2) uses normalization-basel active M-estimation loss, which we find suboptimal in downstream experiments. We further discuss
CDI and other prior works [1] 16 I 19]. along with experimental comparisons, in Appendix G
LLM-as- M-M-assa-Juddee Judge L.M LLa--asa-- refers to using an LLM to evaluate the outputs generated by other
LLMSs 23 50 5] 52 Many prior works considered the case where the verifier and generator are the
same model, and focused on the phenomenon of L.LM self-improvement 21 22 24 53 55 (56..
which is the concept we refer to as self-criticisn in study. However, recent studies suggest that

---

self-criticism may introduce bias, as the verifier tends to favor its own outputs [57 58]. This may explain why we observe better performance with cross-criticism in our ACT data pipeline.
7 Conclusion
In this paper, we introduce the ACT data pipeline, which uses MLLM-generattdd annotations for most data while strategically allocating human effort to potentially erroneous cases, identified by another MLLM criticizer. We further analyze how to modify the training loss to align the performance of models trained ACT data with those trained fully theoretical analysis and empirical evidence. Although our study is based on current MLLMs, and the efficiency of the proposed pipeline is constrained by their capabilities, adapted to more advanced models in the future. The insights gained from our explorations are likely to generalize and provide valuable guidance for practical applications and future research.
data, supported by both
approach be readily
The main limitation of this work is that we focus primarily on classification tasks, without involving more complex tasks such as text summarization open-ended question answering. We provide discussion the Appendix G on how our approach can extended these more complex scenarios.
10

---

References
[1] Alexander Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in Neural Information Processing Systems. 29, 2016.
[2] Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-leaming from noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5138-5147, 2019.
[3] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5051-5059, 2019.
[4] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions Neural Networks and Learning Systems, 34(11):8135-811)) 2022.
[5] Curtis G Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors test sets destabilize machine learning benchmarks. arXiv preprint urXiv:2103. 14749, 2021.
[6] Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. In Empirical Methods in Natural Language Processing, pages 930-957, 2024.
[7] Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. LLMaAA: Making large language models as active annotators. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13088-13103, 2023.
[8] Ruirui Chen, Chengwei Qin, Weifeng Jiang, and Dongkyu Choi. Is a large language model good annotator for event extraction? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17772-17780, 2024.
[9] Margherita Martorana, Tobias Kuhn, Lise Stork, and Jacco van Ossenbruggen. Zero-shot topic classificattin of column headers: Leveraging LLMs for metadata enrichment. arXiv preprint arXiv:2403.0088. 2024.
[10) Xiang Cheng, Raveesh Mayya, and João Sedoc. To err is human; annotate, SILICON? Reducing measurement error in LLM annotation. arXiv preprint arXiv:2412.14461 2024.
[ay]Jay Mohta, Kenan AK, Yan Xu, and Mingwei Shen. Are large language models good annotators? In NeurIPS 2023 Workshop: Can't Believe It's Not Better; Failure Modes in the Age of Foundation Models, 2023.
[12] Nicholas Pangakis and Samuel Wolken. Knowledge distillation in automated annotation: Supervised text classification with L.L--genertedd training labels. Proceedings of the Sixth Workshop on Natural Language Processing and Computational Social Science (NLP+ CSS 2024), 2024.
[13] Yu-Min Tseng, Wei-Lin Chen, Chung-Chi Chen, and Hsin-Hsi Chen. Are expert-level language models expert-level annotators? arXiv preprint arXiv:2410.03254, 2024.
[14] Minzhi Li, Taiwei Shi, Caleb Ziems, Min- Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. Coannotating: Uncertainty-guides work allocation between human and large language models for data annotation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1487-1505, 2023.
[15) Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang. MEGAnno+:
A human-LLM collaborative annotation system. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. pages 168-176, 2024.
11

---

[16] Xinru Wang, Hannah Kim, Sujjadur Rahman, Kushan Mitra, and Zhengjie Miao. Human-LLan collaborative annotation through effective verification of L.M labels. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1 1-21, 2024.
[17] Kristina Gligorié, Tijana Zrnic, Cinoo Lec, Emmanuel Candes, and Dan Jurafsky. Can unconfi- dent LLM annotations be used for confident conclusions? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 3514-3533, 2025.
[18] Costas Mavromatis, Balasubramanian Srinivasan, Zhengyuan Shen, Jiani Zhang. Huzefa Rang- wala, Christos Faloutsos, and George Karypis. Which examples to annotate for in-context learning? Towards effective and efficient selection. arXiv preprint arXiv:2310.20046 2023.
[19] Yifan Wang, David Stevens, Pranay Shah, Wenwen Jiang, Miao Liu, Xu Chen, Robert Kuo, Na Li, Boying Gong, Daniel Lee, Jiabo Hu, Ning Zhang, and Bob Kamma. Accelerating multimodal AI data annotation with LLMs. arXiv preprint arXiv:2409. 10702, 2024.
(MILO)):
[20] Tijana Zmncc and Emmanuel J. Candès. Active statistical inference. In Proceedings of the 41st International Conference Machine Learning, pages 22993-63011 2024.
[21] Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051-1063 2023.
[22] Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can LLMS actually correct their own mistakes? A critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics, 12:1417-1440, 2024.
[23] Jiawei Gu, Xuhui Jiang. Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang. Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ni, Wen Gao, Yuanzhuo Wang, and Jian Guo. A survey on LLM--a--aJuudge The Innovation, 2024.
[24] Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self- improvement capabilities of large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[26) Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST. A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 17X8v:1708.007441 07747, 2017.
[27] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine- grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[28) Francesco Barbieri, Jose Camacho-Collader Luis Espinosa Anke, and Leonardo Neves. TweetE- val: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, 2020.
[29] Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific Data 5(1):1-10, 2018
[30] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-40 system card. arXiv preprint arXiv:2410.21276, 2024.
12

---

[31] Gemini Team, Petko Georgiev, Ving lan Lei. Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unloeking multimodal understanding across millions tokens of context. arXiv preprint arXiv:2403.055300 2024.
[32] Anthropic. Claude 3.5 Sonnet overview, 2024.
Claude-3-b-sonne
[33] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. LLaVA-One Vision: Easy visual task transfer. arXiv preprint arXiv:2408.0333 2024.
[34] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL. technical report.
ArXiv, abs/2502. 1 13923, 2025.
[35] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv preprint arXiv:2412.05277 2024.
[36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[37] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[38] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11699 2019.
[39] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified rision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, pages 12888-12900 2022.
[40] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, et al. Chain-of-Thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24833 2022.
[41] Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Machine Translation, pages 1066-1083, 2023.
[42] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. self-rewarding language models. In Forty-first International Conference Machine Learning, 2024.
[43] Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang. Jiaheng Liu, Chenghua Lin, Lei Ma, Wenhao Huang, and Jiajun Zhang. I-SHEEP: Self- alignment LLMs from scratch through an iterative self-enhancement paradigm. The Thirteenth International Conference on Learning Representations, 2025.
[44] Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevie, and Hao Su. Deductive verification of chain-of- thought reasoning. Advances in Neural Information Processing Systems, 36.36407-36433, 2023.
13

---

[45] Yongqi Tong, Dawei Li, Sizhe Wang. Yujia Wang, Fei Teng, and Jingbo Shang. Can LLMs learn from previous mistakes? Investigating L.LMs errors to boost for reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 3063-3080, 2024.
[46] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. SLiC- HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
[47] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. The Twelfth International Conference on Learning Representations, 2024.
[48] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF Transactions on Machine Learning Research, 2024.
[49] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
[50] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. LLMs--a-Judges: A comprehensive survey LLM-based evaluation methods. arXiv preprint arXiv:2412.05579, 2024.
[51] Cheng- Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties ( (Volume I: Long Papers), pages 15607-15631, 2023.
[52] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems, (6:46595-446223 2023.
[53] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jachoon Lee, Kelvin Xu, al. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2024.
[54] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, 2024.
[55] Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. Volcano: Mitigating multimodal hallucination through self-feedback guided revision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pages 391-404, 2024.
[56] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint 05X:v:2200.05882, 2022.
[57] Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang. Pin-Yu Yu Chen, et al. Justice prejudice? quantifying biases in aludae-a-judge. The Thirteenth International Conference on Learning Representations, 2024.
[58] Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. Preference leakage: A contamination problem in LLM-as-a-Judge arXiv preprint arXiv:2502.01533 2025.
14

---

[59] Jonas Moritz Kohler and Aurelien Lucchi. Sub-sampled cubic regularization for non-convex optimization. In International Conference on Machine Learning, pages 1895-1904, 2017.
[60] Anastasios N Angelopoulos, John C Duchi, and Tijana Zmnce. PPI++: Efficient prediction-
powered inference. arXiv preprint arXiv:231..0145. 2023.
[61] Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang. Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. LOOK-M: Look-once optimization in KV cache for efficient multimodal long-context inference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4065-4078, 2024.
[62] Xidong Wang, Dingjie Song, Shunian Chen, Junying Chen, Zhenyang Cai, Chen Zhang, Lichao Sun, and Benyou Wang. LongLLaVA: Scaling multi-modal LLMs to 1000 images efficiently via a hybrid architecture. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 21419-21436, 2025.
[63] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20:273-297, 1995.
[64] Leo Breiman. Random forests. Machine Learning, 45:5-32, 2001.
[65] Jacob Devlin, Ming- Beig-WWeiie Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short pupers), pages 4171-4186, 2019.
[66] Alicja Raczkowska, Alcksandra Osowska- Osowska-Kurezab Kurczab, Jacek Szczerbinski, Kalina Jasinska-Kobbs and Klaudia Nazarko. AlleNoise: Large-scale text classification benchmark dataset with real-world label noise. arXiv preprint arXiv:2407.10992 2024.
15

---

Appendices Contents
Supplementary Experimental Details
A.1. Datasets
A.2. Device & Random Seed
A.3. Downstream Training
Stability Analyses
B.1. Robustness of Annotation and Criticism Results
B.2. Impact of Different Sampling Rules Criticism
C Examples of Prompt Strategies
Theoretical Proofs
D.1. Proof Proposition 5.1 Statistical Properties of ACT Loss (Variance)
D.2. Proof of Theorem 5.2 Probabilistic Upper Bound of the Parameter Gap
E Further Details of ACT Losses
E.1. ACT Losses with Different Sampling Rules
E.2. Distributions of Transformed Errors with Different Sampling Rules
Sensitivity Analyses of Human Budget
F.1 AQG vs. Human Budget
F.2. Downstream Performance vs. Human Budget
Extended Discussions and Experiments
G.1. Additional Related Works
G.2. Potential Improvements of ACT
G.3. Supplementary Experiments
G.3.1. Comparison with CDI
G.3.2. Comparison with Active Learning and Pseudo Labels
G.3.3. More Challenging Tasks: Biased and Long-tail Data
G.3.4. Analyses on Critic True Positives and False Positives
G.3.5. Ablation of Annotator-Criticzzz Selection
G.3.6. Cost-Benefit Analysis
G.4. Extend ACT More Complex Tasks
G.5. Ethical Considerations
16

---

A Supplementary Experimental Details
A.1
Datasets
Table 4: Dataset details.
Datasel
Type
Description
#Classes
Size Train
Size
Test
CIFARTO
Image classification of basic image categories.
10
50,000
10,000
Fashion-MNIST
CV
Image classification of cloth items.
10
60,000
10,000
Stanford Cars
CV
Image classification of car models.
196
8,143
8,040
Tmmttival-Emotion
NLP
Text emotion classification.
4
3,257
1,421
TweetEval-Irony
NLP
Text ireny detection.
2
2,862
784
VQA-RAD (Close-end)
VQA
Question-answar pairs on radiology images.
2
940
Table 5: Dataset sources and licenses retrieved from httpss//paprrrsiiboodddoo//////
Dataset
Source
License
CIFARIO
N/A
Fashion-MNISS
MIT
Stanford Cars
Custom (hon-commeccial)
TweetEval-Enun Emottion
N/A
TweetEval-Irony
X/A
VQA-RAD
httpp///ppprruttttoooo..caaaaaas
CCO
1.0 Universal
A.2
Device & Random Seed
All experiments are conducted with 1 to 8 NVIDIA SXM5 H100 GPUs with 80GB memories. When applicable, we set the random seed to 42 for all controllable sources of randomness.
A.3
Downstream Training
Sampling For the thresholding sampling rule, the threshold is determined by the quartile corre- sponding to the human budget proportion. Specifically, we rank the errors descending order and set the quartile value that matches the given proportion of the human budget. For the exponential weighting sampling rule, we try 3 = 10 or 100 for all datasets. The mean transition parameter a is set in the same way r, based on the corresponding quartile.
R8 For datasets, we fine-tune the ResNet18 model initialized with ImageNet-preerainne weights for 10 epochs. The batch size is set to 4096 for CIFAR- 10 and the Stanford Cars dataset. We use the Adam optimizer for CIFAR-10 and Fashion-MNDI while the SGD optimizer is employed for Cars, following the implementation described in https:// www. www.kaggle.cmm com code/ www.cchaaiiieddaa..oaaa www.edd//eeeeett--rrrr.. hyperparameters include the learning rate, with a search space of [le-2, 1e-3, 5c-4, 1c-4], and the power-tuning parameter for ACT losses, with values selected from [0.6, 0.7, 0.8, 0.9, 1.0] (see Appendix B for more details about the power-tuning parameter).
and 32 for
tan ord car dane-datseett The key
RoBERTa For text classification tasks, we fine-tune the RoBERTa-base model for epochs. We set the batch size to 32 and use the AdamW optimizer. The key hyperparameters include the learning rate, with a search space of [le-4, 5e-5, 2e-5, 1e-5], and the power-tuning parameter for ACT losses, with values selected from [0.6, 0.7, 0.8, 0.9, 1.0].
fine-tune the BLIP-VOL VQA model initialized with the blip-vqa-bart base for 10 epochs. Since we adopt the close-ended version of VQA-RAD, where answers are limited to either "Yes" "No", we use these two tokens as ground truth. During both training and inference, the model is prompted to generate either "Yes" or "No" in the response token space. We evaluate trained model by checking if the first generated response token the correct answer. We use a batch size of 32 and optimize the model using the AdamW optimizer. The key hyperparameters include the learning rate, searched over [2e-4, 1e-4, Se-5, 2e-5], and the power- wwwer-tuning parameter for ACT losses, selected from [0.6, 0.7, 0.8, 0.9, 1.0].
BLIP-VQA For VQA-RAD,
'Yes' or -machhss
17

---

B Stability Analyses
B.1 Robustness of Annotation and Criticism Results
We assess the robustness of both annotation and criticism by repeating the process times for each MLLM involved in our explorations. We perform the stability test on a subset of all datasets (i.e., 100 random samples per class). The results are shown in Figure 6 Figure 7 and Figure & respectively. We observe that the standard deviations are generally low (most within 2%). In addition, the rank of abilities does not change after taking account of potential variations in metric values, indicating that the conclusions drawn from our explorations are robust to randomness.
CIFAR1O (Anootation)
Fashion (Annotation)
Emotion (Amnotation)
VQA-RAD (Anaotation)
CP14e
GP046
Gemint.ar S
64.15 1.05
Laminil 13
70.20 a 0:45
Gemiit 15
77.40 a 0.33
Gemint. 50"
65.80 0.57
Cluude3.55 58
55.42 a 0.73
Claude355
57.08 1.58
Claude2.55
73.03 1.39
Cloud:3.55
02:45 0.53
68.07 1.85
66.36 1.29
777.08 0.81
60.00 1.15
Qwwe2 SVI.
14 72.05 = 1.02
Dwend SVI.
+60.30 = 0.86
Qwen2 5VI
147.551 = 1.23
Quwn2
x71.95 = 0.46
InteraV12.5
InternVL2.55
InterVi22
lateraM12.
66 Ies Accuracy (9)
65
90 100 Accurssy (85)
60 65 100
06 380 Accuracy (P6)
65
Accuracy (50
Figure 6: Annotation accuracy with error bars (mean + std). Results are presented for the CoT prompt strategy across 6 MLLM annotators.
CIFARIO (Criticism)
Fashion (Criticism)
Emotion (Criticism)
VQA-RAD (Criticiams
GPTM>
228.88 1.42
GPT
#70.60
GPT
272.38 0.35
GPT
H65122 a 1.53
76.38
Geninil
(73.19 1.78
172.58 0.94
Geminil
59.53 1:35
Claadd x5
65.54 : 1.63
Cloude 55
60.01 = 1.50
Chode1 55
601.47 = 1.52
Cbude3.55 55
365.01 a 1.09
LL
12aW
LLaVA-OV
Qwne2 SVL
..17 10 2.23
Ownn2 SVT.
1/00166 242
Owen? 5V1
55.77 1.41
Owne? NN.
+51 55 0.49
InteraV22.5
108.83
Intemm13.5
146.32 a 126
Iater8N1..5
#95.23 a 1.22
IateraV122 5
+45.39 2 0.94
lie
25
180
ARS (%)
ABS (54)
(%0
ABS (W)
Figure 7: Criticism ABS with error bars (mean + std). Results are presented for the black-box CoT prompt strategy across 6 MLLM criticizers.
CIFAR1O (Critidis))
Fashion (Criticism)
Emotion (Criticism)
VQA-RAD (Criticism)
Llatwoo
64.22
LLaVV--V
v61.30
LLaVi-DV
168.
77.23 1.02
#55.51
IateraVi2.55
+80.15 a 1ntem
167.01 027
lateraVi22.
</2.42 175
InterWVL..5
69.35 0.22
CB)
ABS (s)
ARS
ABS 176)
Figure 8: Criticism ABS with error bars (mean + std). Results are presented for the white-box naive-logit prompt strategy across 3 MLLM criticizers.
B.2 Impact of Different Sampling Rules on Criticism
In Figure we present results computed on the full datasets using different sampling rules to evaluate the criticism ABS. The results indicate that the choice of sampling rule has minimal impact on the comparative outcomes. Therefore, the insights derived from our analyses using the thresholding rule remain consistent across other sampling methods as well.
CIFAR10
Rashion
Emotiin
VO4-RAD
Nera
Ei.
Nara
Ci.
12x
Norm
Ei.
Tlx
Nora
2q.
Rly..
28
SPC
Figure 9: Criticism ABS with various sampling rules. Results are presented for the black-box CoT strategy across 6 MLLM criticizers.
18

---

C Examples of Prompt Strategies
Here, we present examples of prompt strategics. For annotation prompt naive, we include examples for all three tasks (image classification, text classification, and VQA) provide comprehensive illustration. For other prompt strategies, we only present examples for image classification, but they be easily adapted to other tasks by following the same pattern the annotation prompts. In the following examples, (purple) denotes inputs, while [blue] denotes outputs. Note that we require MLLMs follow specific output format simplify the extraction of CoT and labels. In practice, we observe that MLLMs follow the formatting instructions well.
Illustrations of Inputs and Outputs
[image_ _dteeddaaaaaaaaaa [text ). and { qquestion:: Images, texts, and questions from the datasets.
(label list_with with index]: label list with indices of each label, for example, "0: airplane, automobile, 2: bird, 3: cat, 4: deer, 5: dog, 6: frog, 7: horse, 8: ship, 9; truck".
first_label]: The first label the label list, for example, "airplane".
[label _ece__inexx___ & jimbel]ideex:: The label index generated by the annotator.
[error probability] & [error level]: The error probability or level generated by the criticizer.
[CoT_At & [CoT__]] The CoT generated by the annotator.
[CoT]: The CoT generated by the criticizer.
[Image Classification]
Annotation Prompt naive
Prompt
[image_ ]
Determine the label of the image classification task. The list of labels is: [label_list_ye index The required output format is: jinde__,dex].. For example, if the label is (first]labei) you should output [0]. Do not return other texts.
Output
[label index]
[Text Classification] Annotation Prompt naïve
Prompt
Determine the label of the text classification task. The text (text data]. The list of labels is: (label list index). The required output format is: [label index], For example, if the label first label], you should output [0]. Do not return other texts.
Output
[label _ineel]__ex]]]_
[VQA-Binary] Annotation Prompt naive
Prompt
fimage_ data]
Answer the question based on the given image. The question is: (question) J. The required output format is [0] for No and [1] for Yes. Do not return other text.
Output
[answer_inds
[Image Classification) Annotation Prompt CoT
Prompt
(image_date
Determine the label of the image classification task. The list of labels is: (label list_withh The required output format is: [label index]. Think step-by-step and provide your reasoning. Example of required output format is: [reasoning][labll index]. The first brackets contain your step-by-step reasoning, and the second brackets contain the label index such [0] for (first_label). Do not return other texts.
Output
[CoT_A][label_indee)
19

---

[Image Classification]
Black-box Criticism Prompt naäve
Prompt
(image data]
Your task is to produce the probability that the label of the given image is wrong. The list of The label of the image is: [label_index). The required output format is: [error_ pitro__robbbillty). For example, [0.911]. The error probability should be reported in 3 decimals. Do not return other texts. Output
labels is: habel_list_with_aeee
[error pooo__poobbbit]]
[Image Classification] Black-box Criticism Prompt CoT
Prompt
fimage_ )
Your task is to produce the probability that the label given image wrong. The list of labels is:
The label of the image is: jibbl_iidex).. Think step-by-step and provide your reasoning. The required output format [reasoning][error ceceoopprobablltty__ The first brackets contain your step-by step reasoning, and the second brackets contain the error probability such as [0.911]. The error probability should reported in 3 decimals. Do not return other texts.
Output
[Image Classification] Black-box Criticism Prompt multiple choice
Prompt
(image_dan) __te__ddaa____
Your task is to analyze if label of the given image is wrong and select from []: correct, 2: correct but not sure, 3: not sure, incorrect but not sure, incorrect). The list of labels is: (label list with j.tex), The label of the image is: (label inddx).. Think step-by-step and provide your reasoning. The required output format is: [reasoning [(eror [[erro__evee]]. The first brackets contain your step-by-step reasoning, and the second brackets contain the error level such as [5] for incorrect. Do not return other texts.
Output
[CoT]]errro level]
[Image Classification] Black-box Criticism Prompt devil's advocate
Prompt
fimage ]
Your task is produce the probability that the statement related the label of the given image is wrong. The list of labels is: [label list with index). The statement of the image label is: [CoT_A]. Think step-by-step and provide your reasoning. The required output format is: ||/error_pppobbbllityll The first brackets contain your step-by-step reasoning, and the second brackets contain the error probability such as [0.911]. The error probability should be reported in 3 decimals. Do not return other texts. Output
[CoT][ennor_rrbbbbliy)
20

---

[Image Classification]
White-box Criticism Prompt naïve
Prompt
(image data]
Your task is to decide whether the label of the given image is wrong. The list of labels is: Laabel_lstt_w__ee with_index] ). The label the image is: [inbe_iindxx.. The required output either Yes or No, where Yes means mistake and No otherwise. Do not return other texts.
Output
Yes/No
[Image Classification] White-box Criticism Prompt CoT
Prompt
(image_ data)
Your task is decide whether the label of the given image wrong. The list of labels is: [label list_withh with j___inddex. The label of the image is: inneex_ Think step-by-step and provide your reasoning. The required output format [reasoning][answeee contain your step-by-step reasoning, and the second brackets contain either Yes or No with Yes meaning mistake and No otherwise. Do return other texts.
The first brackets
Output
[CoT][[Yes/No]
21

---

D Theoretical Proofs
D.1 Proof of Proposition 5.1 Statistical Properties of ACT Loss (Variance)
Proof. Recall that the ACT loss is defined as:
(LACT)
N
((m) 0,
(Co.:
(;(
(11)
N
TB(()
where (B) B(TB(())) Then, we define
(m) BB) (E))
Zi = f0 (Bo.
0,8
and Z =
2 Lg
((CTT)
Assuming that data are i.i.d, the variance of the ACT loss now equivalent to the variance of Z, which is
N
Var(2) Var
Z
Var(Z.).
(12)
N
Hence, suffices to only calculate the variance of Zí which can be decomposed into two parts as Var(Z]
B[Z?] (B(Z.))?. We first expand:
(B) 2
z?
((m)) 0,i
2 + (lo.s <3%)1)
TB( (e,)
0, 26/ (lo.; - pm)) 6,(
+ (Lo,s
plm)
(,
(13)
(C))))
TB (
where we have (B) (B) because &, (B) is either or 1. We assume that (B) and lg.s are
independent. In addition, it is easy to see that [[],(B)] = T (&,), and that E[Z,] = Ef So, we calculate the expectation of 2? as follows:
B|z,] = ((m) 2
20(m) E
Co.
dm)
+ E
(Co+
{(m)) 2
1
TB(
(lp.- - -
27
+
(l0. - ((11))
x (Ei)
(80,i - e(mm)) )²
2
(é( 1)
[6g +
(14)
Thus, we have
Var Var(Z,) = E [eg,]] +E (Lo.: (oo))) )
pmm)) 2
-1]] E
((:))
Var(Lo,) +
(Co.:
((m))
0,0
2
-1 1)]
(15)
T (E:)
Finally, we can show that
Var (c(ACT))
1
Var(Z)
Var(Z;)
N2
N
Var(lo.) + E (401- (lm"))² 2 ( (mntte
-1) 1))
N
Var(Co) + - 2 (13 TT(()) -1
(16)
N
This completes the proof.
22

---

D.2 Proof of Theorem 5.2 Probabilistic Upper Bound of the Parameter Gap show that
Proof. From the definition of ACT loss in Equation
N
V V
1
T V V
0,1
V
(m)
N
N
N
N
= VLa +
T
Vlosi
Voo.i)
+
Σ
(1 - T.
N 1=1
N
1=1
N
VLa + N i=1
(1 Ti)
(V 0,1
Vlos)
(17)
where we let म.
6.(B) ((((
and Le = N Lo.1-
is easy see i (1- mi) (velrr) 0,1 Vle.s) 0. Assume there exist constants q, C such that the transformed error (T((;)) 2 g for all i with 6,( (B(B) 1, and the gradient gap for all i (1,2:-.. N}. Next, bound for any that
V Vla.3 S
(|c1 (1l-m) mt) ( (lm)) ((...) S -111-=1 / V max
(1,
1 1-1 q
C Co
and E B1[(1- 11-mi n:) (vesm) Vlo:))
|| S E (1 -.mmmm=))p m
Velpma Vlo.s
1 - C2 c1.
9
Then we apply the vector Bernstein's 's inequality (e.g., Lemma 18 in [5 such that
1
mi) --))
Σ
N
n-
(vel Vlo.s) 2 S 2xpp
(18)
N
1=1
for 0 <
C1 cg
Then, with a probability of at least - p where p E (0, 1), we have
1
N Σ
(1-mi m.) (veg - vlo.) 5
Se, (2/p)
(19)
N
N
1=1
for any N 8cJ (2pp2//)/c
Finally, due to the p-strong convexity of d. and thus c/., with probability of at least p. can bound the parameter gap
1
V (ACT) - V.C. (ACT)
<
11
1
V VLo. vo(.CT)|
/
(1 mi) (vegm)) Vlo..))
"
8e1 log (2/p)
(20)
N
where (ACCT) arg ming Lg This completes the proof.
(ACT) and 0. arg ming La.
23

---

E Further Details of ACT Losses
E.1
ACT Losses with Different Sampling Rules
The ACT losses with different sampling rules listed follows:
Normalization
17 20
N
de + (lss 20 x 8((B3 En En
(ACT)
(21)
B x éi
1=1
Exponential Weighting
(ACTT)
1
(epoen + (lo. - (egg)) di((3) (1 e (1++-((())))
Cg
(22)
Thresholding
N N
(ACT)
1
(xelma) (Lo.. (lox-)))fr)) d((B)),
(23)
where À E [0, 1] is the power tuning parameter 17 60). which controls the extent to which machine annotations are utilized. Specifically, X = 0 corresponds to completely ignoring machine annotations, while À = 1 corresponds to the full usage. Notably, when employing the thresholding sampling rule with = 1, the ACT loss is equivalent to the standard Cross-entropy loss computed using human-annotated labels when available, and machine-annotate labels otherwise. Therefore, we stated in Section 5.2 that the Cross-entropy loss a special case of the ACT loss.
E.2 Distributions Transformed Errors with Different Sampling Rules
In Figure 10 we present the distributions of transformed errors for data samples reviewed by humans (8(B) 1) under different sampling strategies. We observe that, with normalization sampling, the lower bounds of the transformed errors are close to 0 across all presented datasets. In contrast, for exponential weighting, the lower bounds typically around 0.8, while thresholding yields a consistent lower bound of 1.0. Based on Theorem 5.2, these results provide an explanation for why exponential weighting and thresholding can lead to better downstream training performance.
Normalization
Exponential Weighting
Thresholding
min-0.01
min-0.82
10000
min
Pasbion
5000
1000
5000
0.00
0.25
0.50
0.75
0.85
0.90
0.95
1.00
0.50
0.75
1.00
1.25
1.50
200
im=0.01
min=0.79
min=1.00
200
Cars
0
0.00
0.25
0.50
0.75
1.00
0.80
0.85
0.90
0.95
1.00
0.50
0.75
1.00
1.25
1.50
min=0.05
min=0.87
mic=1.00
VQA-RAD
50
0.90
0.95
1.00
0.75 transformed error thre
transformed error
transformed error
Figure 10: Distributions of transformed errors with different sampling rules (5(B) =1). to the ideal human budget for each dataset.
24

---

Sensitivity Analyses of Human Budget
F1 AQG vs. Human Budget
In Figure 11 illustrate how Annotation Quality Gain (AQG) changes with the human budget proportion. We observe that as the human budget increases, AQG generally rises rapidly at first, then begins to plateau after certain point. This initial rapid increase suggests that the human-corrected samples tend to be more obvious and easily identifiable errors. On most datasets, AQG does not reach 100% before the human budget reaches maximum. In other words, it is usually difficult to achieve perfect annotation quality without reviewing all examples. This indicates that some subtle or hard-to-detect errors are unavoidable. However, we will show in Figure 12 that this does not undermine the effectiveness of using ACT to reduce human effort. With the ACT loss, promising downstream training does not rely on perfectly labeled data.
CIFAR10
Fashion
1.00
Ansutation Qualtt Cain (50)
Aurostatton Qualitty Crein (56)
20
60
103
20
60
103
Human Budget (5)
Human Budget (%)
Cars
Emo Emotion
Antortation Quaare Caln (98)
60
Armnttatten Quallty Coiin
60-
20-
ma/on
Col
mC
TBC
Human Buulett (%)
Human Budget (%)
Irony
VQA-RAD
100
1.00
Antoratton Qualty Cain (%)
60
Anmontation Dualitt Gain
60-
69
09-
60-
40
20
20
120
mB
davi
me/vv
mc
devi
Budget
Budget
Figure 11: How AQG changes with human budget (%). The results are presented for GPT4o self- criticism with 4 black-bor strategies.
25

---

F.2 Downstream Performance vs. Human Budget
conducted the human budget sensitivity experiments using GPT4o self-criticism and the threshold- ing sampling rule. The results are shown in Figure [2] We observe that both annotation accuracy and downstream accuracy generally increase with a larger human budget. When using the ideal budget, a performance gap be observed across all datasets. This is because the criticizer is not perfectly accurate, leading to some overlooked mislabeled data, which slightly degrades the final training performance. In Figure [2] also show the downstream performance gain achieved by adding 10% buffer budget on top of the ideal budget. In 4 out datasets, this buffer nearly eliminates the performance gap, while the gap significantly reduced in the other 2 datasets. Therefore, recommend first evaluating the annotator's s accuracy, and then adding a reasonable buffer to the ideal budget based the observed error rate.
100
100
Ideal 10% Butteer Budget
40.3
Ideal 10% Butter Budget
100
-0.3
Annoratio Accursey
64
Dowvtalrees Actrinary
Amnotation Avsurcy
Downs. www.erteoon rerocm Accuarsc
22
50.2
24
090
.0.33
20
00
-05
Assvelliis,
.22.5
Ansotatic Acouacy
Dowtsstrom Anouney
Drwnstrram Antuncy
60
Hurnnn end Downdroam Accurasy
[deal Ehrnan Burige
90.3
00-
Ideal Htrmae Bedgye
Huzza only Devmutreass Accurscy
10
50 Human Buddet 186
10
66 Humen Buskgst
69
50
190
Cars
Em otion
Ideal 10% Buffer Budgot
197.5
Ideal + 10% Butter Budget
3000
-0.3
Amcocttion Accuncc
30
80.3
Downotrwee Arounacy
Amnotationn Araunnyyy
57.3
57
15.0
31.3
66
60-
10.0
60
77.3
83
Asseritiaz Accurscy
Jenurey
-0.3
66
Asselation Accuracy
Acceneyy
750
77.5
725
Budget
Budget
Irony
VQA-RAD
100
100
Ideal 10% Buffee
Ideal 10% Buffer
66
77
Aanostattonn Accurscy
60-
Acrentaiio Accurarcy
05-
Annetatior, Dowtstrain Anounay
Ansotadiz Acourscy
Dewnstroam Ansunwy
Hn Drwnsdrom Anturacy
Humon only Drwnstream Accuriyy
Ideal thuman Budget
Mdaal Humes Budget
70
10
60
86
Human Budget 196)
Htuman Budget (55)
Figure 12: How annotation accuracy and downstream accuracy change with the human budget (%) The black vertical line shows the position of the ideal human budget (i.e., minus the initial annotator accuracy). The grey area shows the buffer budget over the ideal budget.
26

---

Extended Discussions and Experiments
G.1 Additional Related Works
Enhance Data Annotation with LLMs
To improve the quality LLM annotation, AdaICL adopts in-context learning (ICL) with examples annotated by human [8]. The ICL examples are actively selected based on LLM logit probabilities during annotation, which means their method only supports white-box LLMs. addition, AdaICL lacks mechanisms for long-context visual inference, making it difficult apply directly to vision- based tasks. This is because retrieving and encoding a large number of visual examples during inference would incur prohibitive computational costs [6] 62]. However, consider ICL potential direction for future work, particularly as component in our data pipeline for building an annotator.
Other related works typically follow a three-step IL-M-human collaborative framework: (1) LLMs generate initial labels, (2) verifier assesses the correctness of these labels and outputs verification scores, and (3) human annotators re-annotate a selected subset of labels based on these scores 17 19]. The primary distinction among these methods lies in the design of the verifier. For instance, Model-in-the-Logg (MILO) [99] utilizes the logit scores from another LLM- verifier (similar probabilities from the LLM annotator itself. Another framework proposed by [19 uses a verifier implemented as a Support Vector Machine (63), Random Forests (64), BERT 65), trained
11
16
white-box criticizer). In contrast, MEGAnno+ [13] directly employs the logit
additional human-annotated data. Unlike our approach, the aforementioned methods focus solely annotation accuracy without considering the utility of annotations for downstream training. This narrow focus limits their effectiveness, as high-accuracy labels do not necessarily translate into meaningful model improvements.
The most relevant work to our research CDI 17. which identifies LLM errors using a trained XGBoost model [49] and relies human annotators for correction. During the annotation process, CDI prompts annotators to provide both labels and corresponding verbalized confidence scores. These confidence scores are provided in black-box manner, where higher values indicate greater confidence. For example, an annotator might respond, "The label is cat, and my confidence is 0.999," " express high certainty. The XGBoost model then uses these confidence scores input learn and predict error probabilities. The ground truth is either 0 or 1 depending on the correctness of the annotation, and then the logit probabilities regarded error probabilities. However, CDI has two key limitations: (1) its eгror detection mechanism lacks flexibility, requiring task-specific design and additional training data, and it employs normalization-bal based active M-estimation loss, which find suboptimal in downstream tasks.
27

---

G.2 Potential Improvements of ACT
Here, we outline several potential improvements to the ACT data pipeline, as inspired by the related works. First, drawing from AdaICL [18], could enhance the prompts of the MLLM annotator particularly for NLP tasks-by by incorporating in-context examples. Second, following the approach of MEGAanno+ [T}] it may be beneficial combine the annotator's confidence scores with the criticizer's s error estimations to better capture the insights from both perspectives. Finally, while the current pipeline relies single MILM amnotator-criticize pair, could be extended to multi-model setup using techniques such as majority voting or peer discussion in 13 provided in Figure
An illustration
For decision of the human budget, one may consider dynamically estimating and adjusting the human budget during the review process. As more human-verified labels are accumulated over time, it becomes increasingly feasible to refine the error rate estimation and update the budget allocation accordingly.
In-context Learning
Human Annotator
ICL Examples
Confidence Scores
Data
1nnotator
Criticizer 1
Annotator
Training
Annotator
Criticizer
Majority Vote
Debate
Poer-Discussion
MLLMM
MLLM
Annotator
Figure 13: Illustration of potential improvements of the ACT data pipeline.
G.3
Supplementary Experiments
G.3.1
Comparison with CDI
In T we compare the downstream performance of models trained on ACT and CDI data using various loss functions. Note that CDI data with the ACT norm. loss is corresponding to the approach proposed in [17. We apply the same human budget (the ideal human budget) both ACT and CDI. For CDI, proportion of this budget is allocated to training the XGBoost error detector. The results demonstrate that ACT consistently outperforms CDI in reducing the downstream performance gap.
Table 6: Comparison between ACT and CDI in downstream tasks. The test accuracy (%) is reported form of mean +'2 std over 5 runs
CIFARIO (ReaNet18)
Fashion (ResNet18)
Cars (ResNet18)
Emotion (RoBERTa)
Irony (RoBERTa)
VQA-RAD (BLIP-VQA)
Training Data- Loss
Human only Cross-entropy Loss
88.66 0.97
93.01 0.63
87.88 0.36
81.82 + 0.57
70.18 +
67.81 1.47
data Cross-entropy
84.02 0.85
86.99
0.72
85.61
79,91
66.63 1 3.44
61.77 3.41
CDI data ACT norm. loss
72.22 1.71
83.38 1 1.79
10.76 1 1.12
79.05 1: 1.15
65.96 1 3.36
62.24 3.05
data exp.
0.31
0.54
67.33 1.66
CDI data ACT thre. loss
84,91 0.57
87.64 + 0.52
85.89
0.27
80.00 + 0.83
68.19 + 2.29
67.44 = 2.16
ACT data Cross-entropy loss
85.59 0.52
87.50
0.86
85.88 + 0.26
80.82 + 1.08
67.83 + 2.82
61.83 + 3.27
ACT ACT
64.70 5.46
69.27
7.25
11.54 0.96
79.87
65.66 +
62.55 3.01
ACT data ACT exp. loss (Ours)
87.73 0.36
89.73 173-444 0.35
86. 19
+ 0.14
81 + 0.51
68.49 + 3.20
67.73 1.33
ACT data ACT thre. loss (Ours)
87.95 0.35
89.16
0.89
0.26 86.00
81 = 0.64
68.21 + 1.94
67.02 = 1.32
Human-CDI perfonnance gap (%)
3.67%
5.29%
1.85%
1.31%
1.74%
0.37%
Human-ACT performance gap (%)
0.71%
3.28%
1.69%
0,38%
1.69%
0.08%
Human budget (%)
11.52%
21.81%
9.56%
17.98%
33.79%
28

---

G.3.2 Comparison with Active Learning and Pseudo Labels
We compare method with existing baselines from related fields CIFARIO (ResNet18):
Pseudo Labelling: We randomly label 11.52% of the data to fine-tume ResNet18, then iteratively add high-confidence predictions (e.g., logit probability > 0.9) as pseudo labels to the training set.
Active Pseudo Labelling:
We start by labelling 100 samples per class and progressively select uncertain samples (based on prediction entropy) for annotation until the human budget is exhausted.
The results shown in Table 7 indicate that our approach outperforms these two methods by 4.28% and 2.56% in terms of performance gap from 100% human annotation.
Table 7: Comparison with baselines from related fields.
Method
Accuracy (%)
Human Performance Gap (%
Human-only CE loss
88.66 = 0.97
ACT ACT thre. loss (Ours)
87.95 = 0.35
0.71
Pseudo laheling CE loss
83.67 + 0.32
4.99
Active Pseudo labeling CE loss
85.39 = 0.91
3.27
G.3.3 More Challenging Tasks:
Biased and Long-tail Data
To validate the effectiveness of our method in more challenging tasks, we conduct two experiments:
AlleNoise [66] is product classification dataset where similar category names often result in subjective and biased labelling. We focus 20 labels under the "Sports & Travel" category, which exhibit approximately 70% label similarity-makinne the task ambiguous and prone to bias. Additionally, the largest category contains 27 times more samples than the smallest, reflecting a significant long-tail distribution. The results are shown in Table g demonstrating that ACT yields only 1.4% performance gap in accuracy and 0.0041 difference in Fs-se compared to full human annotation.
Table 8: Experiment results on AlleNoise with RoBERTa.
Data Loss (AlleNoise RoBERTa)
Accuracy (%)
F1-score
Human annotation CE
91.54 + 1.84
0.8448
0.0165
ACT data ACT thre. loss (GPT4o self-criticism)
90.14 + 1.79
0.8397 0.0191
Human-ACT performance gap
1.40
0.0041
CIFARIO-LT is long-tailed variant of CIFAR1O with an imbalance ratio of 100 between the most and least frequent classes. As shown Table ACT achieves a performance gap of 1.97% in accuracy and 0.0148 in Fl-score compared to full human annotation.
Table 9: Experiment results on CIFARIO-LT with ResNet.
Data - Loss (CIFAR10-LTT ResNet)
Accuracy (%)
F1-score
Human annotation CE loss
70.29 = 2.15
0.6820 + 0.0261
ACT data ACT thre. (GPT4o self-criticism)
68.32 = 1.66
0.6672 0.0263
ACT Human performance gap
1.97
0.0148
29

---

G.3.4 Analyses on Critic True Positives and False Positives Analyze Criticism Reliability with False Positives
Table reports the number of false positives produced by different error detection methods across six datasets. The human budget is consistent with T We compare GPT4o self- celf-criticism and the pest-performing cross-criticizer for each task, with random error sampling included as baseline. observe the following:
MLLM-based error detection significantly outperforms random sampling, consistently yielding fewer false positives across tasks.
False positives account for approximately 5%-15% of the total dataset, depending on the complexity and ambiguity of the task.
These results suggest that MLLM- -bseedd criticism is an effective strategy for identifying annotation errors. In addition, there remains room for improvement, especially through incorporating sample- specific analysís and adaptive selection mechanisms in real-world applications.
Table 10: Number of false positives by different error detection methods across six datasets.
Error Detection Methods
CIFAR10 (data s0zk
Fashion (data size~-60k)
Cars (data size--8k)
Random Sampling
4866
9852
704
GPT4o Self-Criticizer
2805
6395
594
Optimal Cross-Criticizer
2692
6087
514
Error Detection Methods
Emotion (data size-.3k)
Irony (data size-.3k)
VQA-RAD (data size-~1k)
Random Sampling
480
640
198
GPT4o Self- Seff-Criticizer
302
591
123
Optimal Cross-Criticios
299
455
b. Compare Self- and Cross-Criticismn with True Positives
We use dataset VQA-RAD as example, where GPT4o only achieves 69.85% annotation accuracy. The human budget is set to 30.15%. In Table ID we present the number of true positives and annotation accuracy after human correction. Randomly sampled errors are included as baseline. It is clear that all MILM criticizers perform much better than random sampling, while self- and cross-criticism do show huge differences. This further strengthens the claim that self-criticism performs competitively with cross-criticism.
Table 1 Number of true positives and annotation accuracy after human correction (VQA-RAD).
Error Detection Methods
#True Positives
Final Annotation Accuracy
Random Sampling
85
78.89%
GPT4o (Self-critccism)
86.87%
Geminil .PP (Cross-crificism)
153
86.12%
Claude3.5S (Cross-criticism)
159
86.76%
LLaVA-OV (Cross-criticism)
158
86.66%
Qwen2.5VL (Cross-criticism)
170
87.94%
Inter IntemVL2.5 VL2.5 (Cross-criticism)
149
85.70%
30

---

G.3.5 Ablation of Annotator-Criticizz Selection
We conduct ablation experiments that examine how annotator-criticize choice affects final down- stream model performance. The experiments are conducted with Cars, because variations self- and cross-criticism performance large this dataset based Figure 3 So, would like know whether the downstream performance also varies a lot.
In Table T2] we find that the effectiveness of criticism is indeed related to downstream performance. The best-perforning criticizer (highest ABS) yields the smallest performance gap. However, the default GPT4o also performs reasonably well, keeping the gap within 2%.
Table 12: Ablation on how annotator-criticizer choice affects final downstream model performance. The results are reported in ABS (%) and accuracy (%).
Cars 4PT Annotation
GPT4o
Geminil.5P
Claude3.88
LLaVA-OV
Qwen2.5VL
InternVL2.55
Criticism ability ABS
60.6
70.9
53.2
55.9
54.4
57.5
ACT data ACT thre. loss
86.00 + 0.26
86.21 + 0.31
85.21 + 0.35
85.39 + 0.29
85.80
0.29
86.45 + 0.31
Human ACT performance
1.88
1.67
2.67
2.49
2.08
2.43
G.3.6 Cost-Benefit Analysis
We present a cost-benefit analysis using CIFARIO (50k samples) to compare three annotation strategies: (I) 100% human annotation, (2) ACT with GPT4o self-criticism, and (3) ACT with GPT4o & Qwen2.5VL. cross-criticism. Note that we exclude the cases where GPUs are not rented but owned, whose costs are difficult estimate.
As shown in Table API-based approaches are the most cost-efficient. We also highlight that mon- etary costs not the only concern. Human annotation also incurs significant time and educational costs, further highlighting the efficiency of our machine-based approach.
Table 13: Cost comparison between human-only annotation and ACT approaches.
Item
Human--nly
ACT (GPT44GGGP))
ACT (GPT4o-Qwen2.5VL)
GPT4o Annotation
~200 tokens x 10/ 1M 0.002/im-
Same as left - $100
costs (OpenAl)
age. Total 50,000 0.002 $100
4PT4o CoT Criticism costs (OpenAl)
~200 tokens x 10 1M = 0.002/i--
age. Total 50,000 0.002 $100
Qwen2.5VL CoT Criticism GPU rent costs (RunPod)
A100 GPUs x 144 hrs x $1.74/r =u $2000
Human Annotation / Correction
50,000 images $0.08 = $4000
10% 50,000 5,000 images
Same as left + $400
Human costs (AWS avg.)
++ 5,000 x S0.08 $400
Total Costs
$4000
$600
$2500
31

---

G.4 Extend ACT to More Complex Tasks
For tasks where the notion "correct" and "incorrect" becomes subjective, the criticizer can output continuous quality score relative ranking rather than a binary enror probability.
The criticizer can leverage strong evaluator, e.g., an MLLM-based evaluator or reward model, to detect semantic inconsistencies, factual mistakes, or language issues in each candidate output and return numerical score or rank.
For content-dense tasks such as multi-fact text summarisation, we can first segment the summary into individual factual statements and then evaluate each statement separately. This allows human to focus only on the segments flagged as uncertain, rather than re-checking the entire summary.
G.5 Ethical Considerations
potential ethical consideration ACT lies its influence on traditional annotation jobs, as the reduction of human effort could be perceived threat to existing human roles. However, the primary motivation of ACT is to alleviate the scalability limitations inherent fully manual annotation, rather than to replace human annotators. By design, ACT facilitates human-Al collaboration, assigning human oversight to high-risk or ambiguous cases where nuanced judgment is essential. Importantly, this approach may also help mitigate the psychological burden associated with labeling harmful or sensitive content (e.g., depictions of violence or nudity), since human intervention is required only for limited subset of critical instances.
32


Reviews:
[{"review_id": "pu90spPNLE", "reviewer": "Reviewer_MY7W", "summary": "This paper proposes ACT (Annotation with Critical Thinking), a general-purpose, LLM-powered data annotation framework that aims to reduce human annotation costs while maintaining high label quality across diverse domains. In ACT, a multimodal large language model (VLM) first serves as the annotator, generating initial labels. A second VLM acts as a critic, estimating the likelihood that each label is incorrect. Only those samples with high estimated error probability are then reviewed by human annotators, allowing strategic use of limited human resources.\n\nThe paper evaluates ACT across six datasets spanning image classification, text classification, and visual question answering, and demonstrates that ACT can reduce the human annotation effort by up to 90%, while achieving downstream model performance within 2% of fully human-annotated baselines.", "questions": "1. The paper mentions that self-criticism performs competitively with cross-criticism. It would be helpful if the authors could provide statistical evidence showing how often the critic, whether self or cross, correctly identified annotation errors. Reporting the frequency of true error detections by the critic across different settings would strengthen this claim.\n\n2. The paper would benefit from a deeper analysis of cases where the annotator provides a label, the critic flags it as incorrect, but a human evaluator determines that the annotator was actually correct. Including such examples and their frequency can offer valuable insight into the critic's reliability and limitations in ambiguous or edge cases.\n\n3. While the experiments are conducted on relatively simple datasets like Fashion-MNIST and CIFAR, it remains unclear how the approach would perform in more complex tasks where label ambiguity and bias are more prominent. Specifically, it would be useful to discuss scenarios where the annotator and critic share similar biases, or when their biases differ significantly, and how that affects overall performance. Can they identify tasks or setup, where maybe more human intervention is required and to what degree human budget will change.\n\n4. The current setup treats the annotator as fixed. Would the annotation quality improve if the annotator received feedback from the critic after each generation? It would be interesting to evaluate this in both self-critique and cross-critique conditions. This feedback loop could potentially enhance annotation quality over time and is worth exploring.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nPractical and Scalable Annotation Pipeline: The ACT framework offers a practical solution to reduce human annotation effort by combining LLM-generated labels with a critic model that flags potentially erroneous annotations. This pipeline is model-agnostic, training-free, and scalable across domains including vision, language, and multimodal tasks.\n\nStrong Empirical Results with High Cost Efficiency: The framework achieves near-parity with fully human-labeled data (within 2% accuracy gap) while saving up to 90% of human effort. The results are consistent across diverse benchmarks such as CIFAR-10, Fashion-MNIST, VQA, and text classification tasks, validating both the quality and efficiency of the method.\n\nInsightful Analysis: The paper provides detailed empirical insights (e.g., when CoT helps, self vs. cross-criticism).\n\nWeaknesses:\n\nLimited Evaluation on Complex or Biased Tasks: The experiments are limited to relatively simple datasets like Fashion-MNIST and CIFAR. It is unclear how well the framework generalizes to more complex tasks, particularly those with noisy or biased labels. The paper does not address how annotator–critic interactions behave when both agents share similar biases or when their biases are orthogonal, which could significantly impact performance in real-world settings.\n\nInsufficient Insight into Annotator–Critic Disagreements: There is limited discussion of cases where the critic flags an error but a human adjudicator ultimately agrees with the annotator. These disagreement scenarios are crucial for understanding the critic’s reliability and could reveal whether critics tend to overcorrect or introduce new types of errors.\n\nPotential Bias Propagation to Downstream Models: Since the ACT pipeline relies heavily on VLM-generated annotations and corrections, there is a risk that any systematic biases in these large models will propagate to the final annotated dataset and subsequently to smaller downstream models trained on it. The paper does not address how such biases are detected, mitigated, or evaluated.", "was_revised": true, "final_justification": "I am satisfied with the experiments provided. Especially that the approach scales to noisy and biased dataset, making the overall contribution much stronger.", "rebuttal": "Dear Reviewer MY7W,\n\nWe sincerely appreciate the time and effort you devoted to evaluating our work. We are especially inspired by your 4th question regarding the annotation feedback loop, which provides a valuable direction for our future work. In the following, we address your concerns by: (1) adding a new case study to evaluate ACT’s robustness to bias and noise, (2) providing deeper analyses of critic error capture and overcorrection, and (3) discussing potential bias propagation and directions for improvements.\n\n&nbsp;\n\n## **[W1 & Q3] Complex Task & Critic Bias**\nEvaluating robustness of ACT under noisy or biased scenarios is indeed a good direction. We address this concern in two parts: (1) a case study on noisy & biased dataset AlleNoise [https://github.com/allegro/AlleNoise] (2) an explanation of how bias was reflected in our original experiments.\n\n&nbsp;\n\n### **(1) Case Study on Noisy & Biased Data**\n**[Data & Task Description]** AlleNoise is a product classification dataset where similar category names often lead to subjective and biased labelling. Merchant-provided labels contain ~15% noise, while expert-annotated labels serve as unbiased ground truth. Due to time constraints, we test on 20 labels under \"Sports & Travel\" with ~70% label similarity, making it ambiguous and prone to bias. We try to explore if ACT downstream performance is close to the unbiased labels, which means ACT is robust to label ambiguity. We also compared the difference between self- and cross-criticism.\n\n**[Experiment Results: Can ACT be generalized to biased & noisy data?]** The short answer is **YES**. In the Table below, we can see that ACT data performance is very close to unbiased labels, and outperforms the biased labels by 11.94% in accuracy. Self- and cross-criticism show similar results.\n\n| **Data - Loss (AlleNoise - RoBERTa)**                        | **Accuracy (%)** | **F1-score**    |\n| ------------------------------------------------------------ | ---------------- | --------------- |\n| Biased merchant annotation - CE loss                         | 78.37 ± 1.18     | 0.6828 ± 0.0232 |\n| Unbiased expert annotation - CE loss                         | 91.54 ± 1.84     | 0.8448 ± 0.0165 |\n| GPT4o machine annotation - CE loss                           | 81.40 ± 2.09     | 0.7385 ± 0.0094 |\n| ACT data - ACT thre. loss (GPT4o self-criticism)             | 90.14 ± 1.79     | 0.8397 ± 0.0191 |\n| ACT data - ACT thre. loss (GPT4o-Gemini1.5P cross-criticism) | 90.31 ± 1.57     | 0.8407 ± 0.0133 |\n\n&nbsp;\n\n### **(2) Bias Reflected in Our Original Experiments**\n\n**[Insights on Agent Bias]** In our original setup, self-criticism can be seen as a same-bias setting, while cross-criticism reflects different biases (from different model families). Based on this, we find that bias difference does not significantly impact performance, as both shown in the paper and the case study. This somehow alleviate the burden of model selection. Nonetheless, we also include bias-reduction strategies later in response to [W3] as a precaution, in case bias does impact downstream performance in very complex real-world scenarios.\n\n**[Impact of Bias on Human Budget]** Since human budgets can be affected by both subjective differences (bias) and predictive uncertainty (variance), we sugget focusing more on final annotation quality, not specific bias types. To estimate human effort, we recommend labelling a small subset (100-300) manually, then evaluating annotator error rate. For example, in VQA-RAD, due to the complexity of medical data, GPT annotator reached only ~70% accuracy, so we used a 30% human budget, which closed the downstream gap with 100% human (see Table 3).\n\n&nbsp;\n\n## **[W2 & Q1 & Q2] Deeper Analyses on Critic True Positives & False Positives**\nHere we consider two situations and apply deeper analyses to the criticism results: (1) From W2 and Q2, the situation where critic flags an error but a human adjudicator ultimately agrees with the annotator (False Positives). (2) From Q1, the situation where the criticizer correctly identified annotation errors (True Positives). These results will be included in the revised paper to further enrich insights on the ACT data pipeline.\n\n&nbsp;\n\n### **(1) Analyze Criticism Reliability with False Positives**\nThe table below reports the number of false positives produced by different error detection methods across six datasets. The human budget is consistent with Table 3 in the main paper. We compare GPT4o self-criticism and the best-performing cross-criticizer for each task, with random error sampling included as a baseline. We observe the following:\n\n- MLLM-based error detection significantly outperforms random sampling, consistently yielding fewer false positives across tasks.\n- False positives account for approximately 5%–15% of the total dataset, depending on the complexity and ambiguity of the task.\n\nThese results suggest that MLLM-based criticism is an effective strategy for identifying annotation errors. In addition, there remains room for improvement, especially through incorporating sample-specific analysis and adaptive selection mechanisms in real-world applications.\n\n| #False Positives |  |  |  |  |  |  |\n|-------------------|--|--|--|--|--|--|\n| **Error Detection Methods**             | **CIFAR10 (data size~50k)** | **Fashion (data size~60k)** | **Cars (data size~8k)** | **Emotion (data size~3k)** | **Irony (data size~3k)** | **VQA-RAD (data size~1k)** |\n| Random Sampling            | 4866                          | 9852                           | 704                        | 480                           | 640                         | 198                           |\n| GPT4o Self-Criticizer      | 2805                          | 6395                           | 594                        | 302                           | 591                         | 123                           |\n| Optimal Cross-Criticizer   | 2692                          | 6087                           | 514                        | 299                           | 455                         | 113                           |\n\n&nbsp;\n\n### **(2) Compare Self- and Cross-Criticism with True Positives**\n\nWe use dataset VQA-RAD as an example, where GPT4o only achieves 69.85% annotation accuracy. The human budget is set to 30.15%. In the following table, we present the number of True Positives and annotation accuracy after human correction. Randomly sampled errors are included as a baseline. It is clear that all MLLM criticizers perform much better than random sampling, while self- and cross-criticism do not show huge differences. This further strengthens the claim that self-criticism performs competitively with cross-criticism.\n\n| # True Positives and Final Annotation Accuracy |  |  |\n|-----------------------------------------------|--|--|\n| **Error Detection Methods**           | **#True Positives** | **Final Annotation Accuracy** |\n| Random Sampling           | 85                  | 78.89%                        |\n| GPT4o (Self)              | 160                 | 86.87%                        |\n| Gemini1.5P (Cross)        | 153                 | 86.12%                        |\n| Claude3.5S (Cross)        | 159                 | 86.76%                        |\n| LLaVA-OV (Cross)          | 158                 | 86.66%                        |\n| Qwen2.5VL (Cross)         | 170                 | 87.94%                        |\n| InternVL2.5 (Cross)       | 149                 | 85.70%                        |\n\n&nbsp;\n\n## **[W3] Potential Bias Propagation to Downstream Mode**\n\nIndeed, our method can be viewed as a type of label distillation from large models. Thus, it is possible that systematic bias of VLM annotator and critic propagate to downstream tasks. In practice, such bias can be identified and mitigated via targeted cast analyses and prompt refinements accordingly. Besides, one can also leverage RAG to integrate a knowledge base that helps reduce bias. We will include more discussions in the revised paper.\n\n&nbsp;\n\n## **[Q4] Annotation Feedback Loop**\n\nThank you for this constructive suggestion. We believe this will be a valuable direction for our future work, which involves building an agentic system that inplements an annotation feedback loop. Such a system is expected to improve firt-run label quality over time and reducing reliance on human correction. We will include a discussion of this idea in Appendix G."}, {"review_id": "f49CGOET1d", "reviewer": "Reviewer_sK1A", "summary": "This paper presents a data annotation pipeline that combines Multimodal LLMs and human reviewers to reduce annotation cost while maintaining the quality of annotations. The core idea is to use an LLM to generate labels and another LLM to act as a criticizer, estimating the probability that a given label is incorrect. Human annotations are required only on the most suspicious cases, minimizing manual effort.\n\nThe authors show that it generalizes to NLP, vision, and multimodal tasks, and that with appropriate sampling strategies and a custom loss function derived from active M-estimation, downstream model performance can be brought within 2% of full human supervision while reducing human annotation load by up to 90%.", "questions": "- The paper sets the human review budget equal to the LLM’s error rate, but that assumes prior knowledge of how often the LLM is wrong. In a real-world scenario where no ground-truth labels exist, how would you recommend setting this budget? Could it be estimated dynamically?", "limitations": "No. While the authors mention the limitations in the paper checklist, they do not have a limitation section. It is strongly recommended to create an additional limitations sections and discuss the point that authors already mentioned in the checklist.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The annotation setup is very well described and is easy to follow. It’s practical and general-purpose, as it can be applied to many annotation tasks.\n- The authors test on image, text, and multimodal datasets, and show that the method works broadly.\n- The experimentation is done quite thoughtfully, comparing different models, prompting strategies, and error estimation techniques. The results support the main claims well. The insights from each experiment is also clearly stated. Also, the separate strategies for black-box and white-box models were also very intuitive and well thought. \n- The modified loss function smartly balances between human-verified and LLM-generated labels using estimated error probabilities, allowing the model to learn effectively even when most labels are noisy.\n\n\n### Weaknesses\n- The core idea, use LLMs to annotate and then review, is an incremental refinement of existing human-in-the-loop or LLM-as-judge pipelines. While the execution is polished, it is not conceptually novel.\n-  The effectiveness of the loss adjustment depends heavily on how well the LLM criticizer estimates label errors and on the choice of sampling parameters, which may not generalize easily across tasks or models.\n- As far as I understand, the paper assumes the ideal human review budget equals the LLM’s error rate, which means if GPT-4o is 88% accurate, then 12% of samples should be reviewed. But in real-world settings, we don’t know the true error rate without already having human labels which defeats the purpose. If the budget is set too low, important errors may be missed; if it’s too high, it wastes human effort. The framework doesn’t offer a way to estimate or adapt the budget dynamically, which limits its practicality.", "was_revised": true, "final_justification": "Thanks to authors and reviewers for the discussion. I would keep my current score.", "rebuttal": "Dear Reviewer sK1A,\n\nWe sincerely thank you for giving our paper a high score of 5. We’re glad to hear that you found our annotation setup clear and easy to follow. In particular, your comment on human budget estimation in practice was especially insightful and inspiring. In response, we clarified how annotator accuracy can be reliably estimated using only 100–300 labelled samples, which can also guide model selection. Additionally, we added a discussion on dynamic budget adjustment as a promising direction for future work.\n\n &nbsp;\n\n## **[W1] Concept Novelty**\nWe would like to respectfully clarify a potential misunderstanding. While our ACT data pipeline indeed builds upon concepts related to human-in-the-loop and LLM-as-judge frameworks, our primary contribution does not lie in proposing a new technical paradigm. Instead, our core novelty lies in providing a **systematic exploration and theoretical foundations** on how to effectively integrate LLMs into the data annotation process to efficiently reduce human cost without compromising downstream training performance. For example:\n\n- **[Guidelines on Choosing Annotator & Criterizer]** We show concrete guidelines on how to choose suitable annotators and criticizers for different tasks and datasets.\n- **[Novel Evaluation Metrics]** We propose two new metrics: AQG and ABS to quantify annotation quality improvements and human budget efficiency. These metrics offer a novel way to assess annotation pipelines beyond traditional accuracy.\n- **[Theoretical Foundations]** We provide a detailed analysis of the ACT loss, showing that models trained on ACT-annotated data can achieve remarkable performance compared to those trained on fully human-annotated data. Our formulation is supported by theoretical guarantees on loss unbiasedness and parameter convergence.\n\nThese contributions demonstrate that ACT is not merely a refinement of existing ideas, but rather a practical and theoretically grounded framework for scalable, high-quality data annotation. To better communicate this, we will revise the abstract to more clearly emphasise these innovations.\n\n\n\n &nbsp;\n\n\n## **[W2] Generalizability of Criticism & Sampling**\nWe appreciate the reviewer’s thoughtful concern about the generalizability of criticism and sampling. Please allow us to offer the following explanation, based on our empirical observations and design considerations.\n\n- **[Generalizability of LLM-based Criticism]** While our theoretical analysis does indicate that the performance of the criticizer plays an important role in downstream effectiveness, we find that current LLMs already provide reasonably strong judgment capabilities in practice. Across a range of datasets, the criticizer consistently achieves over 50% recall in identifying machine annotation errors, many of which are inherently ambiguous or challenging. This results in a 5%–20% improvement in annotation accuracy over purely machine-generated labels. We are optimistic that as LLMs continue to advance, their criticism abilities will generalize even more effectively across diverse domains.\n- **[Easy Selection of Sampling Parameters]** Our recommended threshold-based sampling strategy only has one parameter, the threshold. It can be directly determined by the human annotation budget. We further explain how this budget can be easily estimated in practical applications in the response to weakness 3 and question 1 [W3 & Q1].\n\n\n\n &nbsp;\n\n\n\n## **[W3 & Q1] Human Budget Estimation in Practice**\nWe appreciate the reviewer’s observation regarding the challenge of setting an appropriate human review budget in the absence of prior labels. Please allow us to clarify how this issue is addressed in our design and empirical workflow:\n\n- **[Annotator Accuracy Can be Estimated with Minimal Human Efforts]** We agree with the reviewer that the exact error rate of an LLM annotator cannot be determined without some human labels. However, this does not contradict our goal of reducing human effort. In practice, we find that annotator accuracy can be reliably estimated using a small number of labelled samples—typically between 100 and 300, which represents only a tiny fraction of the dataset. This lightweight supervision is sufficient to guide the estimation of an appropriate human budget. In addition, as we have shown in Appendix F.2, slightly increase this budget by about 10% can effectively safeguard downstream performance across most datasets.\n- **[Labelled Data for Budget Estimation Can Also be Used for Model Selection]** Furthermore, as discussed in Section 4.3, although GPT-4o already performs reasonably well under self-criticism, choosing the most suitable annotator–criticizer pair for a given task typically requires a small set of human-labelled samples. The labelled data are thus also used for the model selection process. This dual-purpose design avoids additional human annotation efforts while improving the overall performance of the ACT pipeline.\n- **[Dynamic Budget Adjustment is a Promising Direction for Future Improvement]** Finally, we agree that dynamically estimating and adjusting the human budget during the annotation process is a meaningful and practical strategy. As more human-verified labels are accumulated over time, it becomes increasingly feasible to refine the error rate estimation and update the budget allocation accordingly. We will incorporate this discussion into the paper to highlight the potential for further improving the adaptability of the ACT pipeline.\n\n &nbsp;\n\n\n## **[Limitation]**\nThank you very much for pointing this out. We will add a limitation section in our revised paper."}, {"review_id": "R5FehiiWHe", "reviewer": "Reviewer_zzKU", "summary": "This paper proposes the \"Annotation with Critical Thinking\" (ACT) pipeline to address the high cost of human data annotation. The method uses one Multimodal Large Language Model (MLLM) as an annotator and a second as a \"criticizer\" to identify potentially incorrect labels, strategically guiding limited human review to the most suspicious cases. Key contributions include validating the approach across diverse multimodal domains, offering practical implementation guidelines from empirical studies, and introducing a theoretically-grounded \"ACT loss\" function to ensure robust downstream training. Experiments demonstrate that this pipeline can achieve performance within 2% of models trained on fully human-annotated data, while reducing human annotation costs by up to 90%.", "questions": "1.What is the content of the Annotation Prompt? How does it guide the mllm to perform 1-0 label annotation on the input image? \n\n2.Can the proposed method be extended to the training process of multimodal large models, not just for downstream tasks?\n\n3.How would the ACT pipeline, particularly the criticizer's \"error probability\" estimation, be adapted for more subjective or generative tasks like rich image captioning or text summarization.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1.The paper introduces specific metrics (AQG and ABS) to quantitatively assess the pipeline's efficiency at different budget levels. This provides a more nuanced performance analysis than a single accuracy score.\n\n2.The proposed ACT incorporates the error probabilities of the critics, thereby providing a structured approach for training on mixed-quality data.\n\n3.The validation across six recent MLLM families, covering both black-box and white-box types, provides a strong empirical basis for the paper's findings. \n\n\nWeakness：\n(1)The article is a bit hard to read, includes too many notations, and lacks details about the Figure 1 pipeline. This prevents the reader from having a clearer understanding of the proposed method.\n\n(2)The evaluation is confined to tasks with simple categorical or short-text outputs. The framework's applicability to more complex, structured annotation tasks like object detection or semantic segmentation remains unexplored.\n\n(3)While the paper focuses on saving human effort, it omits an analysis of the computational and API costs required to run two large MLLMs over entire datasets. A full cost-benefit analysis is necessary to assess the pipeline's true economic viability.\n\n(4)The experiment was insufficient. The downstream validation is limited to a standard architecture (e.g., ResNet-18 for vision tasks ). The claims would be more compelling if the ACT-annotated data were tested on a broader range of models, including both modern Transformer-based architectures (e.g., ViT, CLIP) and models of a larger scale.\n\n(5)The paper lacks ablation experiments that will examine how annotator/critic choice affects final downstream model performance.\n\n(6)The experiments in Table 3 lack a comparison with existing baselines from related fields such as semi-supervised or traditional active learning, and further comparison would enhance the effectiveness of the method.", "was_revised": true, "final_justification": "I think the authors did a good job addressing my concerns. After reading other reviews‘ comments and the responses, I decide to raise my score.", "rebuttal": "Dear Reviewer zzKU,\n\nWe sincerely thank you for the valuable suggestions regarding improving our experiments and extending our method. In response, we have (1) included 4 additional experiments/analyses, such as ablation studies and cost analysis to further demonstrate the effectiveness of our approach, and (2) discussed the potential extension of our method to broader tasks, including its role in MLLM training and subjective tasks. We will incorporate these improvements into the revised paper. Please see below for detailed additions and modifications.\n\n&nbsp;\n\n## **[W1] Notations & Understandability**\n\nThank you for your comment on notations and clarity. We first want to highlight that the notations are necessary for defining the key metrics (i.e., AQG, ABS) as well as facilitating our theoretical analysis in Section 5. In our revised version, we will (1) include a notation table for better reference, and (2) refine the caption of Figure 1, including an explanation of our ACT pipeline.\n\n&nbsp;\n\n## **[W2] More Complex Structured Tasks**\n\nThank you for your question. In our paper, we consider any task with a finite space of labels (as defined in Section 2), which includes many common tasks such as image classification, text classification, and VQA. These tasks are of critical importance because they require substantial annotation costs.\n\nHowever, we believe our proposed framework can be naturally extended to deal with more complex structured tasks that also have finite label sets, such as object detection and semantic segmentation.  For semantic segmentation, each pixel’s label still comes from a finite set, and our method can flag potential error regions, allowing human to correct those areas and therefore lower the cost significantly. For object detection, confidence scoring of candidate bounding boxes lets human focus on boxes with low confidence or prediction inconsistencies.\n\nWe believe this strategy can be generalized to other tasks, and we will add more discussions in our revised version.\n\n&nbsp;\n\n## **[W3] Cost-Benefit Analysis**\nWe present a cost-benefit analysis using CIFAR10 (50k samples) to compare three annotation strategies: (1) 100% human annotation, (2) ACT with GPT4o self-criticism, and (3) ACT with GPT4o–Qwen2.5VL cross-criticism. As shown in the table below, API-based approaches are the most cost-efficient. Note that we exclude the cases where GPUs are not rented but owned, whose costs are difficult to estimate.\n\nWe would also like to highlight that monetary costs are not the only concern. Human annotation also incurs significant time and educational costs, further highlighting the efficiency of our machine-based approach.\n\n| **Item** | **Human-only** | **ACT (GPT4o–GPT4o)** | **ACT (GPT4o–Qwen2.5VL)** |\n|----------|----------------|------------------------|----------------------------|\n| **GPT4o Annotation**  • API costs (OpenAI) | – | ~200 tokens × $10/1M = $0.002/image. Total = 50,000 × $0.002 = **$100** | Same as left → **$100** |\n| **GPT4o CoT Criticism**  • API costs (OpenAI) | – | ~200 tokens × $10/1M = $0.002/image. Total = 50,000 × $0.002 = **$100** | – |\n| **Qwen2.5VL CoT Criticism**  • GPU rent costs (RunPod) | – | – | 8 A100 GPUs × 144 hrs × $1.74/hr ≈ **$2000** |\n| **Human Annotation / Correction**  • Human costs (AWS avg.) | 50,000 images × $0.08 = **$4000** | 10% × 50,000 = 5,000 images → 5,000 × $0.08 = **$400** | Same as left → **$400** |\n| **Total Costs** | **$4000** | **$600** | **$2500** |\n\n&nbsp;\n\n## **[W4] Broader Downstream Model Structures**\n\nWe have tested the ACT-annotated data on a broader range of models, including an example of modern Transformer-based architectures (ViT) and another example of models with a larger scale (Qwen2.5 7B). The results can be found below. The small performance gap of 0.73% and 1.79% shows the generalizability of our method to a wide range of downstream structures.\n\n| **Data – Loss**                       | **Accuracy (CIFAR10-ViT)** | **Accuracy (Irony-Qwen2.5 7B)** |\n|--------------------------------------|-----------------------------|----------------------------------|\n| Human only – CE loss                 | 88.76 ± 0.08                | 77.17 ± 1.69                     |\n| Machine only – CE loss               | 86.59 ± 0.19                | 70.53 ± 1.53                     |\n| ACT data – ACT thre. loss            | 88.03 ± 0.16                | 75.38 ± 1.77                     |\n| **Human–Machine performance gap**    | 2.17                        | 6.63                             |\n| **Human–ACT performance gap**        | 0.73                        | 1.79                             |\n| **Human budget**                     | 11.52%                      | 17.98%                           |\n\n\n\n&nbsp;\n\n## **[W5] Ablation of Annotator-Critisizer Selection**\nWe have added ablation experiments that examine how annotator/criticizer choice affects final downstream model performance. The experiments are conducted with Cars, because variations in self- and cross-criticism performance are large on this dataset, based on Figure 2 in the paper. So, we would like to know whether the downstream performance also varies a lot.\n\nWe find that the effectiveness of criticism is indeed related to downstream performance. The best-performing criticizer (highest ABS)  yields the smallest performance gap. However, the default GPT4o also performs reasonably well, keeping the gap within 2%.\n\n| **Cars – GPT4o Annotation**         | **GPT4o** | **Gemini1.5P** | **Claude3.5S** | **LLAVA-OV** | **Qwen2.5VL** | **InternVL2.5** |\n|:------------------------------------|:--------:|:--------------:|:--------------:|:------------:|:-------------:|:---------------:|\n| Criticism ability – ABS             |  60.6    |   **70.9**     |     53.2       |     55.9     |     54.4      |      57.5       |\n| ACT data – ACT thre. loss           | 86.00 ± 0.26 | **86.21 ± 0.31** | 85.21 ± 0.35 | 85.39 ± 0.29 | 85.80 ± 0.29 | 85.45 ± 0.31 |\n| Human–ACT performance gap           |   1.88   |    **1.67**    |      2.67      |     2.49     |     2.08      |      2.43       |\n\n\n&nbsp;\n\n\n## **[W6] Comparison with More Baselines**\nHere we present the results of comparison with more existing baselines from related fields on ResNet18-CIFAR10:\n\n- **[Pseudo Labelling]** We randomly label 11.52% of the data to fine-tune ResNet18, then iteratively add high-confidence predictions (e.g., logit prob. > 0.9) as pseudo labels to the training set.\n- **[Active Pseudo Labelling]** We start by labelling 100 samples per class and progressively select uncertain samples (based on prediction entropy) for annotation until the human budget is exhausted.\n\nThe results shown in the Table below indicate that our ACT approach outperforms these two methods by 4.28% and 2.56% in terms of performance gap from 100% human annotation.\n\n\n| **Method**                         | **Accuracy (%)**        | **Human Performance Gap (%)** |\n|:----------------------------------|:-------------------:|:-------------------------:|\n| Human-only – CE loss              | 88.66 ± 0.97        | –                         |\n| ACT – ACT thre. loss              | 87.95 ± 0.35        | 0.71                      |\n| Pseudo labeling – CE loss         | 83.67 ± 0.32        | 4.99                      |\n| Active Pseudo labeling – CE loss  | 85.39 ± 0.91        | 3.27                      |\n\n&nbsp;\n\n## **[Q1] Annotation Prompt**\nWe provide the detailed annotation prompt in Section C (page 4) of our Appendix. To guide the LLM in performing 0–1 label annotation, we directly prompt it using a structured format that includes the label list alongside the corresponding image data. This prompt design enables the model to associate relevant visual features with the appropriate binary labels effectively. For example, the naive annotation prompt of binary VQA is:\n#### Prompt:\n\"{image_data} Answer the question based on the given image. The question is: {question}. The required output format is [0] for No and [1] for Yes. Do not return other text.\"\n#### Output\n\"[answer_index]\"\n\n&nbsp;\n\n## **[Q2] Extension to MLLMs**\n\nAs discussed in Weakness 4, we extend our approach beyond downstream tasks by applying it to the supervised fine-tuning (SFT) stage of LLMs, demonstrating its potential as a generalizable open-world annotator. Specifically, we annotate the Irony dataset using our method and fine-tune Qwen2.5-7B on this annotated data. The results show that training with ACT-annotated data leads to only a 1.89% drop in accuracy compared to training with human annotations, while achieving a 4.85% improvement over training with machine-generated annotations.\n\n\n&nbsp;\n\n## **[Q3] Adaptation to Subjective or Generative Tasks**\n\n\nWe appreciate the question on subjective or generative tasks. For tasks where the notion of “correct/incorrect” becomes subjective, the criticizer can output a continuous quality score or a relative ranking rather than a binary error probability. \n* The criticizer can leverage a strong evaluator, e.g., an MLLM‑based evaluator or a reward model, to detect semantic inconsistencies, factual mistakes, or language issues in each candidate output and return a numerical score or rank.\n* For content‑dense tasks such as multi‑fact text summarisation, we can first segment the summary into individual factual statements and then evaluate each statement separately. This allows human to focus only on the segments flagged as uncertain, rather than re‑checking the entire summary.\n\n\nWe will incorporate a discussion of these extensions in the revised version."}, {"review_id": "rumiaJzlbt", "reviewer": "Reviewer_CvVf", "summary": "This paper proposes the **Annotation with Critical Thinking (ACT)** data annotation pipeline to address the high cost and inefficiency of manual data labeling. The authors utilize multimodal large language models (MLLMs) as both annotators and critics. By leveraging a criticism mechanism to identify potential errors, human efforts are focused on reviewing only the most \"suspicious\" samples, improving annotation efficiency.", "questions": "1. **Model Selection and Generalization**:\n   - Does ACT’s performance depend on specific MLLM models for different tasks? Could smaller-scale models achieve similar results?\n   - Has the ACT pipeline been tested on open-domain datasets or real-world long-tail tasks for better generalization?\n2. **Criticism Mechanism Improvements**:\n   - What are the performance differences between self-criticism and cross-criticism for more complex tasks? Are there potential ways to enhance the criticism mechanism further?\n3. **Trade-offs in Annotation Quality**:\n   - Since CoT (Chain-of-Thought) improves criticism but not annotation, could optimized CoT strategies enhance annotation quality?\n4. **Sampling Strategy Selection**:\n   - How should sampling strategies (e.g., thresholding or exponential weighting) be chosen in practice? Are there automated methods for parameter selection?\n5. **Social and Ethical Considerations**:\n   - What are the potential impacts of scaling up the ACT pipeline on human annotators? Have ethical concerns related to such impacts been considered?\n\n### **Suggestions for Improvement**:\n- Conduct experiments on more complex tasks and data distributions to better evaluate the ACT pipeline’s applicability.\n- Provide a more comprehensive discussion of societal and ethical impacts, such as the potential effects on the annotation industry.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n1. **Theoretical and Practical Contributions**:\n   - Provides a strong theoretical framework for improving annotation quality through criticism mechanisms.\n   - Empirical results demonstrate that models trained on ACT data achieve performance gaps of less than 2% compared to fully human-annotated datasets.\n2. **Resource Efficiency**: \n   - Focuses human review efforts on high-risk samples, reducing total human costs by 70%-90%.\n   - Sampling strategies (e.g., thresholding, exponential weighting) effectively improve resource utilization.\n3. **Practical Usability**:\n   - Offers detailed user guidelines and actionable recommendations for real-world deployment.\n\n### **Weaknesses**\n1. **Criticism Mechanism Limitations**:\n   - While the paper explores self-criticism and cross-criticism, it lacks in-depth discussion on their stability across tasks and model scales.\n2. **Limited Experiment Scope**:\n   - Experiments focus on standard datasets (e.g., CIFAR-10, Fashion-MNIST), leaving the applicability to more complex or long-tail datasets unverified.\n   - Benchmark experiments are based on single runs, which, despite stability analysis, could benefit from multiple independent runs for stronger statistical validity.\n3. **Theoretical Assumption Issues**:\n   - The method assumes human-annotated labels closely approximate ground truth, which may not hold in noisy or biased scenarios.\n4. **Lack of Societal Impact Discussion**:\n   - The paper lacks a thorough discussion of potential negative societal impacts, such as effects on human annotators or privacy concerns.", "was_revised": true, "final_justification": "The responses have clarified several key concerns I had about your work. The additional experiments on long-tail datasets, clarification about multiple independent runs, expanded ethical considerations, and comparison between self-criticism and cross-criticism mechanisms help strengthen your paper. While there remain opportunities to further explore the stability of criticism mechanisms across more diverse tasks and model scales, your current results showing reasonable performance gaps compared to human annotation while significantly reducing annotation effort are promising.", "rebuttal": "Dear Reviewer CvVf,\n\nWe sincerely thank you for the thoughtful feedback and for recognizing both the theoretical and practical significance of our method. We are especially grateful for the mention of ethical issues. This gives us a valuable opportunity to discuss the ethical implications of our approach in real-world applications. In addition, we appreciate the suggestion to consider long-tail data, which helps us to enrich the analyses of our method. Please see below for the modifications we have made in response to your feedback, as well as clarifications to address potential misunderstandings. \n\n&nbsp;\n\n## **[W1] Discuss Limitations of Criticism Mechanism**\nWe evaluate the effectiveness of our method across a diverse range of tasks, covering both vision and language modalities (e.g., CIFAR10, CAR, Irony), multiple disciplines (e.g., Fashion, VQA-RAD), and human emotion understanding (e.g., Emotion, Irony). To further address the stability of the criticism mechanisms, we provide a systematic analysis in Section 4.2.1, where we derive three key insights into optimal configurations across broader task and model settings.\n\nWe also discuss the influence of model scale in Section 4.2.1. Specifically, we treat the choice between black-box and white-box models as a practical consideration of model scale, since black-box models typically involve substantially larger parameter counts compared to their white-box counterparts. Given our motivation to leverage more capable large-scale LLMs to reduce human annotation workload, we intentionally select a substantially larger model as our annotator, rather than relying on smaller-scale models such as the 7B variant.\n\n&nbsp;\n\n\n## **[W2] Expand Limited Experiment Scope**\nHere we address the two points mentioned in [W2] separately: (1) applicability to more complex/long-tail datasets, (2) multiple independent runs for stronger statistical validity.\n\n### **(1) Applicability to More Complex/Long-tail Datasets**\nWe agree that it is important to evaluate on more complex or long-tailed datasets. Therefore, we have added two additional experiments to address this concern:\n\n**[Allenoise]** Allenoise [https://github.com/allegro/AlleNoise] is a product classification dataset where similar category names often result in subjective and biased labelling. Due to time constraints, we focus on 20 labels under the \"Sports & Travel\" category, which exhibit approximately 70% label similarity—making the task ambiguous and prone to bias. Additionally, the largest category contains 27 times more samples than the smallest, reflecting a significant long-tail distribution. The results are shown below, demonstrating that ACT yields only a 1.4% performance gap in accuracy and a 0.0041 difference in F1-score compared to full human annotation.\n\n| **Data – Loss (AlleNoise – RoBERTa)**                     | **Accuracy (%)**   | **F1-score**         |\n|:----------------------------------------------------------|:------------------:|:--------------------:|\n| Human annotation – CE loss                                | 91.54 ± 1.84       | 0.8448 ± 0.0165       |\n| ACT data – ACT thre. loss (GPT4o self-criticism)          | 90.14 ± 1.79       | 0.8397 ± 0.0191       |\n| **Human–ACT performance gap**                             | 1.40               | 0.0041               |\n|\n\n\n**[CIFAR10-LT]** CIFAR10-LT [https://huggingface.co/datasets/tomas-gajarsky/cifar10-lt] is a long-tailed variant of CIFAR10 with an imbalance ratio of 100 between the most and least frequent classes. As shown below, ACT achieves a performance gap of 1.97% in accuracy and 0.0148 in F1-score compared to full human annotation.\n\n| **Data – Loss (CIFAR10-LT – ResNet)**                    | **Accuracy (%)**   | **F1-score**         |\n|:---------------------------------------------------------|:------------------:|:--------------------:|\n| Human annotation – CE loss                               | 70.29 ± 2.15       | 0.6820 ± 0.0261       |\n| ACT data – ACT thre. loss (GPT4o self-criticism)         | 68.32 ± 1.66       | 0.6672 ± 0.0263       |\n| **ACT – Human performance gap**                          | 1.97               | 0.0148               |\n|\n\n&nbsp;\n\n### **(2)Multiple Independent Runs for Stronger Statistical Validity**\nWe would like to clarify that our stability analysis indeed refers to multiple independent runs of the same experiment. As detailed in Appendix B (page 3 of the supplementary material), each experiment was conducted with 5 independent runs, and we report both the mean and standard deviation across these runs. Similarly, all results presented in Table 3 in the original paper are also averaged over 5 independent training runs, with standard deviations provided accordingly.\n\n&nbsp;\n  \n## **[W3] Explain Theoretical Assumption Issue**\nWe thank the reviewer for pointing out the distinction between the theoretical assumption of perfect human labels and the realities of practical deployment. We agree that this issue can arise in real-world applications, especially for tasks involving ambiguous or subjective labels. In general, fully human-annotated data is considered the highest-quality supervision we can obtain, and it is standard to treat such labels as ground truth. However, this may not prevent the possibility of errors introduced by human annotators. Accordingly, we would like to emphasize that this is not a limitation specific to our method, but rather an inherent limitation of human annotation itself. \n\n&nbsp;\n\n## **[W4 & Q5] Discuss Social & Ethical Considerations**\nWe recognize a potential ethical concern that ACT, by reducing manual annotation workload, could impact traditional annotation jobs. However, ACT is designed to address the scalability challenges of human annotation, not to replace human roles. It enables human–AI collaboration by reserving human oversight for high-risk or ambiguous cases. Moreover, ACT may reduce the psychological burden of labelling harmful content (e.g., violence, nudity, etc.) by limiting human involvement to key instances. We will elaborate on these aspects in the revised paper.\n\n&nbsp;\n\n# **[Q1] Model Selection & Generalization**\n\nThank you for the question, we will answer the two questions separately: (1) ACT's dependence on model types and scales, (2) test ACT on open-domain or real-world long-tail datasets.\n\n### **(1) ACT's Dependence on Model Types and Scales**\n**[Specific Model Types]** Please refer to the user guideline provided in Figure 1. For most tasks, a general-purpose approach is to directly use GPT-4o self-criticism (vanilla version), which already reduces the downstream performance gap to within 2% while saving up to 90% of human annotation effort. This method does not rely on any specific model. For users seeking optimal performance, the Pro version allows for model selection to further improve results.\n\n**[Model Scales]** Please see response to [W1].\n\n\n### **(2) Test ACT on Open-domain or Real-world Long-tail Datasets**\n\nHas the ACT pipeline been tested on open-domain datasets or real-world long-tail tasks for better generalization? The short answer is **YES**. We have provided empirical results on public long-tailed datasets under \"[W2] Expand Limited Experiment Scope\".\n\n&nbsp;\n\n## **[Q2] Criticism Mechanism Improvements**\n\n**[Difference betweem Self- and Cross-Criticism in Complex Tasks]** Using AlleNoise as an example, we compare GPT-4o self-criticism with GPT-4o–Gemini1.5P cross-criticism. As shown in the table below, both approaches yield very similar downstream performance, with cross-criticism providing a slight improvement of 0.17% in accuracy and 0.001 in F1-score.\n\n| **Data – Loss (AlleNoise – RoBERTa)**                              | **Accuracy (%)**   | **F1-score**         |\n|:-------------------------------------------------------------------|:------------------:|:--------------------:|\n| ACT data – ACT thre. loss (GPT4o self-criticism)                   | 90.14 ± 1.79       | 0.8397 ± 0.0191       |\n| ACT data – ACT thre. loss (GPT4o–Gemini1.5P cross-criticism)       | 90.31 ± 1.57       | 0.8407 ± 0.0133       |\n\n&nbsp;\n\n**[Methods to Further Improve Criticism]** We see several promising directions for extension:\n- As discussed in Appendix G.2, we can build a multi-agent criticism system using strategies like majority voting, debate, or peer discussion among different models.\n- We can enhance the criticizer with ICL by providing examples of known annotation errors to guide its judgment.\n- For more complex tasks, RAG can be used to supply the model with relevant external knowledge for more informed criticism.\n\n&nbsp;\n\n## **[Q3] Trade-offs in Annotation Quality**\nTo clarify, as discussed in Section 4.1 (Insight 2), our findings do not suggest that Chain-of-Thought (CoT) reasoning is ineffective for annotation in general. Instead, we observed that its benefits may depend on the specific annotator model used. Specifically, for GPT-4o—the annotator we selected due to its consistently superior annotation performance—we observe that CoT reasoning does indeed provide measurable benefits. Thus, further optimization of CoT prompts may lead to additional improvements in annotation quality.\n\n&nbsp;\n\n## **[Q4] Sampling Strategy Selection**\nWe thank the reviewer for the comment and offer the following clarifications:\n\n**[Thresholding is Preferred]**: As noted in line 297, the threshold-based strategy is simple and involves only one parameter, which can be set according to the available human annotation budget. For example, with 10,000 samples and a 10% review budget, the threshold can be set at the 90th percentile of predicted error probabilities.\n\n**[Human Budget is Easy to Estimate]**: As discussed in Section 5.2, model accuracy—and thus human review needs—can be estimated from 100–300 labelled samples. For example, with 80% annotator accuracy, a 20% review budget suffices, assuming the criticizer catches all errors."}]

Author rebuttal:

Meta-review: This paper proposes a data annotation pipeline built on an Annotation-with-Critical-Thinking (ACT) design that uses multimodal large language models as both annotators and critics to identify potentially incorrect labels, directing human review effort to fewer samples. The work provides some insights into how to modify the loss function, and demonstrates up to 90% reduction in human annotation costs while maintaining performance within 2% of fully human-annotated baselines across a few tasks (mainly in classification).

While the overall rating of this paper is ok, there are some points to be addressed to further strengthen the contribution:

(1) First, the experimental evaluation needs substantial expansion beyond simple datasets like Fashion-MNIST and CIFAR to include more complex and (potentially) biased tasks, particularly for subjective tasks like image captioning and open questions (from Reviewer MY7W, Reviewer CvVf, Reviewer zzKU).

(2) The framework requires better handling of bias propagation from the annotator and critic to downstream tasks, including methods to detect and mitigate systematic biases that could affect the final annotated datasets (from Reviewer MY7W, Reviewer sK1A). 

(3) The analysis should include some in-depth discussion of annotator-critic disagreement cases and scenarios where both agents share similar biases, which is crucial for understanding critic reliability (from Reviewer MY7W, Reviewer sK1A, Reviewer CvVf).
Decision: Accept (poster)