Title: ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
Venue: NeurIPS 2025
Accepted: yes
Keywords: data annotation

data pipeline

multimodal large language models

llm-as-a-judge

supervised learning
Abstract: Supervised learning relies on high-quality labeled data, but obtaining such data through human annotation is both expensive and time-consuming. Recent work explores using large language models (LLMs) for annotation, but LLM-generated labels still fall short of human-level quality. To address this problem, we propose the Annotation with Critical Thinking (ACT) data pipeline, where LLMs serve not only as annotators but also as judges to critically identify potential errors.  Human effort is then directed towards reviewing only the most "suspicious" cases, significantly improving the human annotation efficiency. Our major contributions are as follows: (1) ACT is applicable to a wide range of domains, including natural language processing (NLP), computer vision (CV), and multimodal understanding, by leveraging multimodal-LLMs (MLLMs). (2) Through empirical studies, we derive 7 insights on how to enhance annotation quality while efficiently reducing the human cost, and then translate these findings into user-friendly guidelines. (3) We theoretically analyze how to modify the loss function so that models trained on ACT data achieve similar performance to those trained on fully human-annotated data. Our experiments show that the performance gap can be reduced to less than 2% on most benchmark datasets while saving up to 90% of human costs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the **Annotation with Critical Thinking (ACT)** data annotation pipeline to address the high cost and inefficiency of manual data labeling. The authors utilize multimodal large language models (MLLMs) as both annotators and critics. By leveraging a criticism mechanism to identify potential errors, human efforts are focused on reviewing only the most \"suspicious\" samples, improving annotation efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n1. **Theoretical and Practical Contributions**:\n   - Provides a strong theoretical framework for improving annotation quality through criticism mechanisms.\n   - Empirical results demonstrate that models trained on ACT data achieve performance gaps of less than 2% compared to fully human-annotated datasets.\n2. **Resource Efficiency**: \n   - Focuses human review efforts on high-risk samples, reducing total human costs by 70%-90%.\n   - Sampling strategies (e.g., thresholding, exponential weighting) effectively improve resource utilization.\n3. **Practical Usability**:\n   - Offers detailed user guidelines and actionable recommendations for real-world deployment.\n\n### **Weaknesses**\n1. **Criticism Mechanism Limitations**:\n   - While the paper explores self-criticism and cross-criticism, it lacks in-depth discussion on their stability across tasks and model scales.\n2. **Limited Experiment Scope**:\n   - Experiments focus on standard datasets (e.g., CIFAR-10, Fashion-MNIST), leaving the applicability to more complex or long-tail datasets unverified.\n   - Benchmark experiments are based on single runs, which, despite stability analysis, could benefit from multiple independent runs for stronger statistical validity.\n3. **Theoretical Assumption Issues**:\n   - The method assumes human-annotated labels closely approximate ground truth, which may not hold in noisy or biased scenarios.\n4. **Lack of Societal Impact Discussion**:\n   - The paper lacks a thorough discussion of potential negative societal impacts, such as effects on human annotators or privacy concerns.\n\nQUESTIONS:\n1. **Model Selection and Generalization**:\n   - Does ACT’s performance depend on specific MLLM models for different tasks? Could smaller-scale models achieve similar results?\n   - Has the ACT pipeline been tested on open-domain datasets or real-world long-tail tasks for better generalization?\n2. **Criticism Mechanism Improvements**:\n   - What are the performance differences between self-criticism and cross-criticism for more complex tasks? Are there potential ways to enhance the criticism mechanism further?\n3. **Trade-offs in Annotation Quality**:\n   - Since CoT (Chain-of-Thought) improves criticism but not annotation, could optimized CoT strategies enhance annotation quality?\n4. **Sampling Strategy Selection**:\n   - How should sampling strategies (e.g., thresholding or exponential weighting) be chosen in practice? Are there automated methods for parameter selection?\n5. **Social and Ethical Considerations**:\n   - What are the potential impacts of scaling up the ACT pipeline on human annotators? Have ethical concerns related to such impacts been considered?\n\n### **Suggestions for Improvement**:\n- Conduct experiments on more complex tasks and data distributions to better evaluate the ACT pipeline’s applicability.\n- Provide a more comprehensive discussion of societal and ethical impacts, such as the potential effects on the annotation industry.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the \"Annotation with Critical Thinking\" (ACT) pipeline to address the high cost of human data annotation. The method uses one Multimodal Large Language Model (MLLM) as an annotator and a second as a \"criticizer\" to identify potentially incorrect labels, strategically guiding limited human review to the most suspicious cases. Key contributions include validating the approach across diverse multimodal domains, offering practical implementation guidelines from empirical studies, and introducing a theoretically-grounded \"ACT loss\" function to ensure robust downstream training. Experiments demonstrate that this pipeline can achieve performance within 2% of models trained on fully human-annotated data, while reducing human annotation costs by up to 90%.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1.The paper introduces specific metrics (AQG and ABS) to quantitatively assess the pipeline's efficiency at different budget levels. This provides a more nuanced performance analysis than a single accuracy score.\n\n2.The proposed ACT incorporates the error probabilities of the critics, thereby providing a structured approach for training on mixed-quality data.\n\n3.The validation across six recent MLLM families, covering both black-box and white-box types, provides a strong empirical basis for the paper's findings. \n\n\nWeakness：\n(1)The article is a bit hard to read, includes too many notations, and lacks details about the Figure 1 pipeline. This prevents the reader from having a clearer understanding of the proposed method.\n\n(2)The evaluation is confined to tasks with simple categorical or short-text outputs. The framework's applicability to more complex, structured annotation tasks like object detection or semantic segmentation remains unexplored.\n\n(3)While the paper focuses on saving human effort, it omits an analysis of the computational and API costs required to run two large MLLMs over entire datasets. A full cost-benefit analysis is necessary to assess the pipeline's true economic viability.\n\n(4)The experiment was insufficient. The downstream validation is limited to a standard architecture (e.g., ResNet-18 for vision tasks ). The claims would be more compelling if the ACT-annotated data were tested on a broader range of models, including both modern Transformer-based architectures (e.g., ViT, CLIP) and models of a larger scale.\n\n(5)The paper lacks ablation experiments that will examine how annotator/critic choice affects final downstream model performance.\n\n(6)The experiments in Table 3 lack a comparison with existing baselines from related fields such as semi-supervised or traditional active learning, and further comparison would enhance the effectiveness of the method.\n\nQUESTIONS:\n1.What is the content of the Annotation Prompt? How does it guide the mllm to perform 1-0 label annotation on the input image? \n\n2.Can the proposed method be extended to the training process of multimodal large models, not just for downstream tasks?\n\n3.How would the ACT pipeline, particularly the criticizer's \"error probability\" estimation, be adapted for more subjective or generative tasks like rich image captioning or text summarization.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a data annotation pipeline that combines Multimodal LLMs and human reviewers to reduce annotation cost while maintaining the quality of annotations. The core idea is to use an LLM to generate labels and another LLM to act as a criticizer, estimating the probability that a given label is incorrect. Human annotations are required only on the most suspicious cases, minimizing manual effort.\n\nThe authors show that it generalizes to NLP, vision, and multimodal tasks, and that with appropriate sampling strategies and a custom loss function derived from active M-estimation, downstream model performance can be brought within 2% of full human supervision while reducing human annotation load by up to 90%.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The annotation setup is very well described and is easy to follow. It’s practical and general-purpose, as it can be applied to many annotation tasks.\n- The authors test on image, text, and multimodal datasets, and show that the method works broadly.\n- The experimentation is done quite thoughtfully, comparing different models, prompting strategies, and error estimation techniques. The results support the main claims well. The insights from each experiment is also clearly stated. Also, the separate strategies for black-box and white-box models were also very intuitive and well thought. \n- The modified loss function smartly balances between human-verified and LLM-generated labels using estimated error probabilities, allowing the model to learn effectively even when most labels are noisy.\n\n\n### Weaknesses\n- The core idea, use LLMs to annotate and then review, is an incremental refinement of existing human-in-the-loop or LLM-as-judge pipelines. While the execution is polished, it is not conceptually novel.\n-  The effectiveness of the loss adjustment depends heavily on how well the LLM criticizer estimates label errors and on the choice of sampling parameters, which may not generalize easily across tasks or models.\n- As far as I understand, the paper assumes the ideal human review budget equals the LLM’s error rate, which means if GPT-4o is 88% accurate, then 12% of samples should be reviewed. But in real-world settings, we don’t know the true error rate without already having human labels which defeats the purpose. If the budget is set too low, important errors may be missed; if it’s too high, it wastes human effort. The framework doesn’t offer a way to estimate or adapt the budget dynamically, which limits its practicality.\n\nQUESTIONS:\n- The paper sets the human review budget equal to the LLM’s error rate, but that assumes prior knowledge of how often the LLM is wrong. In a real-world scenario where no ground-truth labels exist, how would you recommend setting this budget? Could it be estimated dynamically?\n\nLIMITATIONS:\nNo. While the authors mention the limitations in the paper checklist, they do not have a limitation section. It is strongly recommended to create an additional limitations sections and discuss the point that authors already mentioned in the checklist.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ACT (Annotation with Critical Thinking), a general-purpose, LLM-powered data annotation framework that aims to reduce human annotation costs while maintaining high label quality across diverse domains. In ACT, a multimodal large language model (VLM) first serves as the annotator, generating initial labels. A second VLM acts as a critic, estimating the likelihood that each label is incorrect. Only those samples with high estimated error probability are then reviewed by human annotators, allowing strategic use of limited human resources.\n\nThe paper evaluates ACT across six datasets spanning image classification, text classification, and visual question answering, and demonstrates that ACT can reduce the human annotation effort by up to 90%, while achieving downstream model performance within 2% of fully human-annotated baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nPractical and Scalable Annotation Pipeline: The ACT framework offers a practical solution to reduce human annotation effort by combining LLM-generated labels with a critic model that flags potentially erroneous annotations. This pipeline is model-agnostic, training-free, and scalable across domains including vision, language, and multimodal tasks.\n\nStrong Empirical Results with High Cost Efficiency: The framework achieves near-parity with fully human-labeled data (within 2% accuracy gap) while saving up to 90% of human effort. The results are consistent across diverse benchmarks such as CIFAR-10, Fashion-MNIST, VQA, and text classification tasks, validating both the quality and efficiency of the method.\n\nInsightful Analysis: The paper provides detailed empirical insights (e.g., when CoT helps, self vs. cross-criticism).\n\nWeaknesses:\n\nLimited Evaluation on Complex or Biased Tasks: The experiments are limited to relatively simple datasets like Fashion-MNIST and CIFAR. It is unclear how well the framework generalizes to more complex tasks, particularly those with noisy or biased labels. The paper does not address how annotator–critic interactions behave when both agents share similar biases or when their biases are orthogonal, which could significantly impact performance in real-world settings.\n\nInsufficient Insight into Annotator–Critic Disagreements: There is limited discussion of cases where the critic flags an error but a human adjudicator ultimately agrees with the annotator. These disagreement scenarios are crucial for understanding the critic’s reliability and could reveal whether critics tend to overcorrect or introduce new types of errors.\n\nPotential Bias Propagation to Downstream Models: Since the ACT pipeline relies heavily on VLM-generated annotations and corrections, there is a risk that any systematic biases in these large models will propagate to the final annotated dataset and subsequently to smaller downstream models trained on it. The paper does not address how such biases are detected, mitigated, or evaluated.\n\nQUESTIONS:\n1. The paper mentions that self-criticism performs competitively with cross-criticism. It would be helpful if the authors could provide statistical evidence showing how often the critic, whether self or cross, correctly identified annotation errors. Reporting the frequency of true error detections by the critic across different settings would strengthen this claim.\n\n2. The paper would benefit from a deeper analysis of cases where the annotator provides a label, the critic flags it as incorrect, but a human evaluator determines that the annotator was actually correct. Including such examples and their frequency can offer valuable insight into the critic's reliability and limitations in ambiguous or edge cases.\n\n3. While the experiments are conducted on relatively simple datasets like Fashion-MNIST and CIFAR, it remains unclear how the approach would perform in more complex tasks where label ambiguity and bias are more prominent. Specifically, it would be useful to discuss scenarios where the annotator and critic share similar biases, or when their biases differ significantly, and how that affects overall performance. Can they identify tasks or setup, where maybe more human intervention is required and to what degree human budget will change.\n\n4. The current setup treats the annotator as fixed. Would the annotation quality improve if the annotator received feedback from the critic after each generation? It would be interesting to evaluate this in both self-critique and cross-critique conditions. This feedback loop could potentially enhance annotation quality over time and is worth exploring.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}