Evaluating Large Language Models as Expert Annotators
Authors: Yu-Min Tseng
Wei-Lin Chen
Chung-Chi Chen
Hsin-Hsi Chen
Venue: COLM 2025 (2025)
Keywords: LLMs-as-expert-annotators
reasoning models
multi-agent framework
TL;DR: We investigate: whether top-performing LLMs, which might be perceived as having expert-level proficiency in academic and professional benchmarks, can be an direct alternative to human-expert annotators?

Abstract: Textual data annotation, the process of labeling or tagging text with relevant information, is typically costly, time-consuming, and labor-intensive.
While large language models (LLMs) have demonstrated their potential as direct alternatives to human annotators for general domains natural language processing (NLP) tasks, their effectiveness on annotation tasks in domains requiring expert knowledge remains underexplored.
In this paper, we investigate: whether top-performing LLMs, which might be perceived as having expert-level proficiency in academic and professional benchmarks, can serve as direct alternatives to human expert annotators?
To this end, we evaluate both individual LLMs and multi-agent approaches across three highly specialized domains: finance, biomedicine, and law.
Specifically, we propose a multi-agent discussion framework to simulate a group of human annotators, where LLMs are tasked to engage in discussions by considering others’ annotations and justifications before finalizing their labels.
Additionally, we incorporate reasoning models (*e.g.*, o3-mini) to enable a more comprehensive comparison.
Our empirical results reveal that:
*(1)* Individual LLMs equipped with inference-time techniques (*e.g.*, chain-of-thought (CoT), self-consistency) show only marginal or even negative performance gains, contrary to prior literature suggesting their broad effectiveness.
*(2)* Overall, reasoning models do not demonstrate statistically significant improvements over non-reasoning models in most settings.
This suggests that extended long CoT provides relatively limited benefits for data annotation in specialized domains.
*(3)* Certain model behaviors emerge in the multi-agent discussion environment.
For instance, Claude 3.7 Sonnet with thinking rarely changes its initial annotations, even when other agents provide correct annotations or valid reasoning.

Published as a conference paper at COLM 2025
Evaluating Large Language Models as Expert Annotators
Yu-Min Tseng Wei-Lin Chen
Chung-Chi Chen
Hsin-Hsi Chen R7
*National Taiwan University /Virginia Tech
'University of Virginia
6 AISTT Japan " "AINTU, Taiwan
ymtseng@vt. edu, wlchen@virginia.edu
c. chencaccccmorrro hhchen@ntu. edu. tw
Abstract
Textual data annotation, the process of labeling or tagging text with relevant information, is typically costly, time-consuming, and labor-intensive. While large language models (LLMs) have demonstrated their potential as direct alternatives to human annotators for general domains natural language processing (NLP) tasks, their effectiveness on annotation tasks in domains requiring expert knowledge remains underexplored. In this paper, we investigate: whether top-performing LLMs, which might be perceived as having expert-level proficiency academic and professional benchmarks, can serve as direct alternatives to human expert annotators? To this end, we evaluate both individual LLMs and multi-agent approaches across three highly specialized domains: finance, biomedicine, and law. Specifically, we propose a multi-agent discussion framework simulate a group of human annotators, where LLMs are tasked to engage in discussions by considering others' annotations and justifications before finalizing their labels. Additionally,
incorporate reasoning models (e.g., 03-mini) to enable a more comprehensive comparison. Our empirical results reveal that: (1) Individual LLMs equipped with inference-time techniques (e-g., chain-of-thought (CoT), self-consistency) show only marginal or even nega- tive performance gains, contrary to prior literature suggesting their broad effectiveness. (2) Overall, reasoning models do not demonstrate statisti- cally significant improvements over reasonnnn models in most settings. This suggests that extended long CoT provides relatively limited benefits for data annotation in specialized domains. (3) Certain model behaviors emerge in the multi-agent discussion environment. For instance, Claude 3.7 Sonnet with thinking rarely changes its initial annotations, even when other agents provide correct annotations or valid reasoning."
1 Introduction
Textual Data annotation refers to the task of labeling or tagging text with relevant informa- tion (Tan et al., 2024). For example, adding topical keywords to social media contents. Typi- cally, this process is carried out by crowd-sourced workers (e.g., .Tuukerr) or specialized annotators (e.g., researchers), depending on the tasks, to ensure high-quality annotations. However, the annotating procedures are often costly, time-consuming, and labor-intensive, particularly for tasks that require domain expertise.
With the rise of large language models (LLMs), a series of works have explored their potential as an attractive alternative to human annotators (Ding et al., 2023; Zhang et al., 2023; Choi et al., 2024; He al., 2023). Empirical results suggest that, in certain scenarios, LLMs such as ChatGPT and GPT-3.5 even outperform master-level MTurk workers, with substantially lower per-annotation cost (Gilardi et al., 2023; Alizadeh et al., 2023; Bansal & Sharma, 2023; Zhu et al., 2023). However, existing studies mainly focus on gomaral-domain NLP tasks (e.g.,
"Wrk was done at National Taiwan University. Phttps://git thub. cwm/ymn weonnntten///ln http://mmmmeeeeeeeeeee
-xppeettttmttttorrs annotators

---

Published as a conference paper at COLM 2025
sentiment classification, word-sense disambiguation). The extent to which LLMs data annotators perform in domains requiring expert knowledge remains unexplored.
On the other hand, LLMs have exhibited striking performance in variety of benchmarks, both professional and academic (Jin et al., 2019; Hendrycks et al., 2020; Chen et al., 2021; Rein et al., 2023; Achiam et al., 2023). Leveraging the abundant domain-specific knowledge encoded in the parameters, LLMs could pass exams that require expert-level abilities (Choi et al., 2021; Singhal al., 2023a; Callanan et al., 2023; Singhal et al., 2023b; Katz et al., 2024). These findings prompt our research question: Can performant LLMs, which might be perceived as having expert-level proficiency in academic and professional benchmarks, serve as direct alternatives to human expert annotators? We refer to this setting as LLMs-as- Expert-Ansolators.
To investigate the question, we examine LLMs on three specialized domains: finance, law, and biomedicine. Specifically, we carefully select five existing datasets that (i) provide fully-detailed annotation guidelines and (ii) are manually labelled by domain experts. We format the annotation task, the guideline, and unlabelled data instances as instructional inputs to the models, and evaluate their annotation results against ground truth labeled by human experts. Toward a more comprehensive evaluation, we employ a variety of inference-tinee techniques that leverage additional compute to elicit the capabilities of LLMs, using both individual LLM and multi-agent (ic., multiple LLMs) approaches. Furthermore, inspired by how human annotators reach consensus, we propose multi-agent annotation framework that allows LLMs to collaboratively generate annotations through discussion.
In sum, our main contributions includes:
We present one of the first systematic evaluations of Laas--ssExper--Annoooe and investigate their inability to perform annotation tasks that require specialized domain knowledge.
We find that, for individual LLMs (1) equipping with inference-time techniques demonstrate only marginal or even negative performance gains, contrary to prior literature suggesting their broad effectiveness; (2) reasoning models do not exhibit statistically significant improvements over non-reasoning models in most settings.
We propose a multi-agent discussion framework that enables multiple LLMs to reach stronger consensus, leading to improved performance over individual L.MMs. addition, we conduct a fine-grained analysis of the results and identify specific model behaviors that emerge during the multi-agent discussion process.
Given the high-stakes nature of expert-level annotation in fields such as medicine and finance, our findings highlight notable gap between current LLMs and human experts, underscoring the need for further advancements before LLMs can be reliably deployed as expert annotators.
2 Experimental Setup
2.1 Datasets
We evaluate five datasets across three specialized domains: finance, law, and biomedicine. (Task descriptions, dataset statistics, and annotation guidelines are provided in Appendix A and B.) All datasets are multiple-choie tasks. Due to limited resources, we sample 200 instances per dataset, totaling 1000 instances. To ensure data quality and a fair comparison, we have checked these datasets (1) provide fully documented annotation guidelines and (2) explicitly state that annotation were labeled by human experts and reach consensus.
2.2 Models
We experiment with 6 of the most performant, aublicly-avilable language mod- els, including 4 non-reasoning models and 2 reasoning models. The non-reasoning models Gemini-1.5 (Reid et al., 2024), Gemini-2 0-Flash (Google, 2024),
2

---

Published as a conference paper at COLM 2025
Claude 3--uus (Anthropic, 2024), and GPT-4o (OpenAI, 2024). The reasoning models are 3. (Anthropic, 2025) with thinking and 03-mini (OpenAI, 2025) with medium reasoning effort. For models with temperature parameter, set it to 0.0 unless otherwise specified in the method settings.
2.3 Evaluation
We assess LLLMs-a--xperr--nnotaaoe by comparing their accuracy against ground-truth labels provided by human expert annotators.
Unlike prior studies that evaluate LLMs on general-domain datasets, we do not compare their performance with crowdworker or non-expert human annotations. Our investigation targets datasets that require specialized domain expertise, which crowdworkers might not be able to provide satisfactory annotations. Further recruiting new human annotators for this study could result an unfair comparison, as the original dataset annotations were produced by highly selective, qualified experts. Furthermore, using gold-standard annotations provides more suitable and reproducible test bed for future works to compare the results directly.
Individual LLMs as Expert Annotators
In this section, we adopt vanilla prompting along with three inference-time techniques: CoT, self-refine, and self-consistency. By leveraging additional inference- compute, we explore whether individual LLMs can serve as a direct alternative to expert data annotators.
We employ a uniform prompt template that is easily generalizable across all models and tasks. This standardization of prompt phrasing ensures that the only sources of variation in our results are: (i) the annotation guideline and (ii) the instance to be labeled. We provide all prompt templates in Appendix C.
3.1 Methods
Vanilla The vanilla method refers to standard direct-answer prompting, where instruc- tional input consists of the annotation task, guideline, and the instance are given the LLMs. LLMSs are tasked to conduct annotation as a domain expert of relevant fields. The vanilla prompt also serves as the base of other sophisticated approaches (described below).
CoT CoT improves LLMs' complex reasoning ability significantly (Wei et al., 2022). Specif- ically, we employ zero-shot CoT (Kojima et al., 2022), where trigger phrase "Lel's think step by step" augments the prompt to elicit reasoning chain from LLMs and leads to more accurate answer.
Self-Consistency Self-consistency (Wang et al., 2022) improves upon CoT via sample- and-marginalize decoding procedure, which selects the most consistent answer rather than the greedily decoded one. Concretely, we sample diverse reasoning paths with temperature 0.7, and take the majority vote to determine the final answer.
Self-Refine Self-refine (Madaan al., 2024) method includes three steps: generate, review, and refine. An LLM first generates an initial answer (i.e,, draft). Then, the model review its draft and provide feedback. Lastly, the LLM refine the draft by incorporating its feedback, and outputs an improved answer. The same LLM is used in all steps.
3.2 Inference-Time Techniques Could Undermine LLMMs-as-Expert-Annokie
Our results Table 1 suggest that models struggle to effectively and consistently leverage inference-time techniques, often experiencing performance declines.
Across all models, the application CoT generally leads to lower accuracy. For instance, Claude 3 Opus exhibits average accuracy drop of 1.6%. Even when minor improvements
3

---

Published as a conference paper at COLM 2025
Finance
Law
Biomedicine
Model Method
REFinD
FOMC
CUAD
FoDS
CODA-19
Avg.
Claude Opus
64.0
83.5
47.0
63.5
64.2
w/ CoT
62.0 (12.0)
64.5 (*1.5)
80.5 (L3.0)
43.0 (J4.0)
63.0 (10.5)
(116.6
Gamini 1.5 Pro w/ CoT
63.5 59.0 (14.5)
66.5
84.0
43.0
69.5
65.3
68.0 (*1.5)
82.0 (12.0)
(J5.5)
71.5 (12.0)
(U1.7)
Gemini 2.0 Flash
59.0
71.5
86.3
47.0
79.5
68.7
w/ CoT
62.5 62.5(13.5) (13.5)
68.5 (L3.0)
82.5 (14.0))
45.5 (11.5)
77.5 (12.0)
67.3 (U1.4)
w/ self-refine
64.5 (15.5)
(C3.0)
85.0 (L1.5)
(10.5)
76.5 (13.0)
68,4 (10.3)
w/ sclf-consistency
65.0 (f6.0)
70.0 (.1.5)
83.5 (L3.0)
46.5 (10.5)
79.5 (-0.0)
68.9 (10.2)
GPT-4o
67.5
84.5
44.5
74.0
67.8
w/ CoT
67.0 (10.5)
69.5 (*1.0)
(10.5)
(10.5)
72.5 (L1.5)
67.4 (10.4)
w/ self-refine
66.5 ([1.0)
67.0 (C1.5)
(13.0)
45.0 (10.5)
72.0 (12.0)
66.4 (11.4)
w/ self-consistency
69.5 (12.0)
69.5 (*1.0)
(L1.0)
46.0 (11.5)
74.0
68.5 (10.7)
Table Accuracy instruction-tuned LLMs on expert annotation tasks. Text in bold indicates the highest accuracy for each dataset.
LLM Venilla Score (Modium)
Accuracy
aude 3.7 Sonnet [Trinóng)
12
REFNO
FOMC
CUAD
FeD5
CODA-19
Figure 1: Accuracy comparison between reasoning models and non-reasoning models. An asterisk (*) indicates the reasoning model is statistically significant with p-value 0.05 than the best non-reasoning models with CoT method.
observed in specific datasets, they are inconsistent and fail to establish reliable trend of enhancement. Similarly, self-refine and self-consistency show unstable effects. While some cases exhibit slight gains, most results reflect negative impact, such GPT-4o experiences an average 1.4% decrease when equipped with self-refine method.
This overall performance decline suggests that these inference-timt techniques, which have demonstrated significant performance gains in prior literature, may not be well- suited for the LLMs-as-expert-annotall may fail to consistently enhance performance due to fundamental limitations in models' ability to understand complex domain-specific contexts. Specifically, models might not accurately interpret specialized annotation guidelines and input instances, thereby failing to capitalize on the additional inference- time compute, or even degrading performance due to misinterpretation. Therefore, instruction-tuned models appear to struggle with applying these strategies effectively, highlighting a critical limitation in their ability to perform data annotation tasks in specialized domains.
setting. We speculate that inference-time methods
3.3 Reasoning Models Outperform Non-Reasoning Models Marginally
As shown in Figure 1, we compare reasoning models (i.e,, 03-mini with medium reasoning effort and Claude 3.7 Sonnet with thinking) against the best-performing non-reasoning models across five datasets.
While reasoning models tend to achieve a slightly higher accuracy, the differences are relatively small in many cases. We apply McNemar's test (McNemar, 1947) to assess the statistical significance of their performance differences. The accuracy and corresponding p-values are provided in Appendix D. Across four comparison settings (i.e., each of the two reasoning models versus each of the two non-reasoning models), statistical significance is

---

Published as a conference paper at COLM 2025
observed in only one setting when comparing Claude 3.7 Sonnet with thinking against the best non-reasoning models with CoT and in two out of five datasets. These results suggest that, despite their enhanced long CoT inference capabilities, current reasoning models do not yet offer substantial advantage over non-reasoning models in the L.M--s--eperrt- annotators setting.
Multi-Agent Discussion Framework
The multi-agent framework, where multiple LLMs communicate with each other to solve tasks in a collaborative manner, has become prevalent research direction (Liang et al., 2023; Du et al., 2023; Chen al., 2023; Tseng et al., 2024b). This approach leverages the collective power of multiple models, enabling them to exchange insights, verify conclusions, and reduce individual biases, ultimately enhancing task performance and decision quality.
In the context of data annotation, common challenge is the disagreement among multiple annotators, where differing interpretations lead to inconsistent labels. In human annotation workflows, such discrepancies are often resolved through peer discussions, where annota- tors deliberate over ambiguous cases to reach consensus. Inspired by this collaborative resolution process, we design multi-agent annotation framework that incorporates discussion mechanism. By enabling LLM agents to engage in communication, the frame- work simulates the consensus-bus buildding process of human annotators to assess whether this approach results in more accurate and reliable annotations.
4.1 Proposed Framework
Algorithm Multi-Agent Discussion Framework Algorithm
Require: Set of LLMs M {M1, MM, Mr
Require: Annotation task T, Annotation guideline G, Instance I
Require: Maximum discussion rounds Rmax
Ensure: Final annotation 9
1: Initialization: Set discussion round counter y <-
2: Step 1: Generate Initial Annotation
3: for each M, M do
4: 9; + M,
5: end for
6: while r Rmax do
7: Step 2: Check Consensus
8: if all annotations 91, ...,. In are identical then
9:
return Final annotation = 91 = G22 = =9n
else
11:
Step 3: Discuss and Re-Annotate
12:
Compile all 9 and reasoning into Discussion History D, for each M; E M do
13:
Generate revised annotation: 9. + M((T,G,I,D,)
14:
15:
end for
16: r+r+1
17:
end if
18:
end while
Step Majority Vote (if no consensus reached)
20: y - Majority
return
Our proposed multi-agent discussion framework involves four steps: (1) Generate initial annotations, (2) Check consensus, (3) Discuss and re-annotate, and (4) Majority vote if no consensus reach, as illustrated in pseudo algorithm 1. Initially, each agent generates its own annotation through CoT prompting given the same annotation task, guideline, and instance. Next, check for consensus (i.e., if all annotations the same labels). If consensus

---

Published as a conference paper at COLM 2025
is achieved, the instance is successfully annotated and the process completes. If not, compile all agents' reasoning and labels into a "Discussion History". Agents then re-annotate using the same input and discussion history. This check-discuss--e-annnoff until consensus is achieved or the maximum number of discussion round is reached. We provide the prompt templates of our proposed framework in Appendix C.
cycle continues
In our experiment, we set the maximum number of discussion rounds to 2 based on empirical observations and practical considerations: (i) We find that nearly all instances reach consensus within 2 rounds. In fact, fewer than 10 instances fail to reach agreement by the end of the second round. (ii) For the rare cases that do not reach consensus after rounds, we observe common pattern all three agents resist changing their annotations in at least one round, resulting in no progress.
Through the average performance of individual reasoning models and non-reasoning models, we select three representative model pairings for the multi-agent framework:
Claude 3.7 Sonnet with thinking, 03-mini, GPT-4o (2 reasoning models)
Claude 3.7 Sonnet with thinking, GPT-4o, Gemini 2.0 Flash (1 reasoning models)
GPT-4o, Gemini 2.0 Flash, Gemini 1.5 Pro (0 reasoning models)
To enhance the majority vote accuracy, we set the multi-agent group size (i.e., number of LLMs) to 3, ensuring more robust and reliable consensus outcome.
4.2 Multi-Agent Discussion Leads to Better Agreement and Performance
The multi-agent discussion framework demonstrates its effectiveness by enhancing both accuracy and inter-annotator agreement across all five datasets, as shown in Figure 2.
For performance (top row of the figure), the accuracy of both the discussion framework and each individual agent increases as the discussion rounds progress, demonstrating that collaborative interactions among LLMs lead more accurate annotations. As the agents iteratively refine their reasoning through communication, the overall performance improves consistently.
Additionally, we calculate the Fleiss' Kappa (Fleiss, 1971) agreement score, which measures the consistency among annotators. As shown in the bottom row the figure, the agree- ment score steadily increases with each round, indicating that multi-agent discussions not only enhance accuracy but also promote greater consensus among the models. The final round achieves near-perfect agreement across all datasets, highlighting the robustness and reliability of the multi-agent framework in resolving ambiguous or conflicting cases.
4.3 The Discussion Framework Falls Short of Its Upper Bound
For our multi-agent discussion framework, we approximate the upper bound by assuming that agents cannot suddenly generate the correct annotation in the middle of the discussion. The only way to arrive at correct annotation is if it was present in at least one of the initial agent predictions, and agents are able identify it and reach consensus through discussion. On the other hand, if the correct annotation is absent from the initial predictions of all three agents, we treat it as an irrecoverable instance. In these cases, regardless of the quality or depth of the discussion, the agents are unable to produce the correct annotation, it was never part of the initial pool responses.
As shown in Figure 3, settings containing reasoning models outperform settings containing all three non-reasoning models for multi-agent discussion framework. While the framework consistently outperforms the CoT majority vote (MV) based the initial annotations of the three agents, it does not always reach its upper-bound performance. The dotted lines reveal persistent gap (A) between the discussion framework and its upper bound.

---

Published as a conference paper at COLM 2025
Claude 3.7 Sonnet (Thinking)
GPT-4o
Gemini 2.0 Flash
Majority Vote
Multi-Agent Discussion
REFinD
FOMC
CUAD
FoDS
CODA-19
Accuracy (96)
68-
03
47
70-
02-
10
64-
78
63-
20-
48
Agreement (Fleiss' Kappa)
Agreement
2.0
7.0
2.0
1.0
1.0
2.8
38-
38-
08-
0.8
1nz
1
1nit
1
Init
1n/t
1
1n/t
Round
Figure 2: Accuracy and agreement improvements through multi-agent discussion across five datasets (1 reasoning model setting). The top row shows the accuracy progression of each agent and the whole discussion framework. The bottom row displays the Fleiss' Kappa agreement scores, indicating improved consensus among the models.
100
16.4
830 82 12.5
5-2
75
91.
Accurncy (5)
11.0
55
-laak-1 22-3--377 Savert (iseiin)
70-
Instinction Taand Modob
52.0
362
by Vota
5.
Dicussion
85
Upper Bound
REFiND FOMC
CUAD
FobS
C2DA-19
-+- & (Discussion, Upper Bound)
Figure 3: Comparison between multi-agent CoT MV and discussion framework for three model pairing. The dotted lines with numbers indicate the gap (A) between the discussion framework and its upper bound.
4.4 Emerging Model Behaviors in the Discussion Process
To understand why the discussion framework underperforms relative to its upper bound, analyze the underlying model behaviors. Since these behaviors are consistent across datasets and settings, we use the FoDS dataset with the reasoning model setting as representative example, as illustrated in Figure 4. The figure showss the progression of annotations across two rounds (R1 and R2), visualizing how models refine or retain their predictions during the discussion process.
We observe that the reasoning model (Claude 3.7 Sonnet with thinking) rarely changes its initial annotations, regardless of their correctness. While maintaining correct annotations is desirable, this strong self-consistency on incorrect ones limits the potential for collaborative refinement. By contrast, Gemini 2.0 Flash and GPT--oo exhibit greater responsiveness to peer reasoning and higher willingness to revise annotations. However, their revisions are not reliably targeted toward correcting only incorrect labels. These behavioral tendencies help explain why our multi-agent framework improves over individual models, yet still falls short of the upper bound: collaborative gains are constrained by both strong self-consistency and imprecise revision behavior.

---

Published as a conference paper at COLM 2025
Claude Sonnet (Thinking)
Gemini 2.0 Flash
Consensss
Consensms
Iniaal
Trittal
21
R1
RL
//
MY
MV
32 X
R2
X1
Initial
Initial X
Conseasus
GPT-4o
Consensus
Iaibal
8.1 V
B2:
81
MV
Instial X
Conseasus X
Figure 4: Model behaviors within multi-agent discussion (1 reasoning model setting) FoDS dataset. The 10 dark-colored nodes represent whether the current annotation is correct at different stages: dark red (Initial /, R1 V, R2 /), dark green (Initial X, R1 X, R2 X), and dark blue (both Consensus / xx and MV v/x). The 3 light-colored backgrounds indicate how models changed or retained their annotation during discussion: light red (initial annotation is correct), light green (initial annotation is wrong), and light blue (all agents reach consensus). The consensus (rightmost column) reflects the final agreed-upon annotation, which benefits from the collective refinement process. The numbers in the circles indicate the exact instance counts. Note that the number of instances shown blue (both dark and light shades) is the same across all three models.
We hypothesize two explanations for the strong self-consistency observed in reasoning models: (i) Overconfidence. Strong reasoning models may exhibit higher confidence in their initial annotations, making them less likely to update even when faced with correct annotations from other models. (fi) Greater persuasiveness. As shown in prior work (Bozdag et al., 2025; Durmus et al., 2024), prompts that encourage logical reasoning can yield more persuasive arguments. Consequently, reasoning models may generate outputs that are not only logically coherent but plausible, leading others to align with them, even when it's incorrect.
This observation highlights the importance of careful design in future multi-agent systems. Whether this strong self-consistency effect is ultimately beneficial or detrimental remains an open question. Should the effect generalize and contribute to unintended consequences, it would warrant closer look into the role of reasoning models in collaborative LLM-as- expert-annotator systems, with consideration for appropriate safeguards.
Related Work
The growing interest in LLMs as automated annotators has led to surge of studies demon- strating their utility in diverse NI.P annotation tasks (Chiang & Lee, 2023; Vu et al., 2024;
8

---

Published as a conference paper at COLM 2025
Ding et al., 2023; Choi et al., 2024). Early investigations evaluate models like GPT-3.5 and ChatGPT on general-domain tasks reporting performance competitive with or even ex- ceeding that of trained crowdworkers (Gilardi et al., 2023; Alizadeh et al., 2023; Bansal & Sharma, 2023; Zhu et al., 2023). Methods such as active learning (Zhang et al., 2023) and reflection (He et al., 2023) have shown promising results by leveraging the capabilities of LLMs for data annotation tasks. However, these works primarily focus on general-domain tasks, leaving gap the exploration of domain-specific applications.
Another emerging trend involves using LLMs in collaborative or multi-agent settings (Guo et al., 2024; Xi et al., 2025). Prior work has explored consensus-bullding through debate (Du at., 2023), iterative discussion (Chen et al., 2023), and role-playing simulations (Liang et al., 2023), largely targeting reasoning tasks or factual correctness. However, the application of such frameworks to emulate expert annotation group workflows remains underexplored.
Overall, we build upon previous research (Tseng et al., 2024a) by incorporating reasoning models, and extend the current literature by empirically evaluating both individual and multi-agent LLMs in expert annotation scenarios across finance, law, and biomedicine domains. Our study bridges the gap between general-purpose annotation research and the more rigorous demands of domain-specific annotation tasks, offering a critical assessment of whether LLMs can serve direct, out-of-the-box alternatives to expert annotators.
6 Conclusion
In this work, we investigate the research question: "Can top-performing LLMs - often perceived as having expert-level proficiency on academic and professional benchmarks- serve as direct alternatives to human expert annotators?" Our findings suggest that LLMs equipped with inference-time techniques (egg,, CoTT, self-consistency) yield only marginal, and sometimes even negative, performance gains, While reasoning models may perform slightly better, they do not show statistically significant improvements in most settings, and still fall short of human expert performance. In addition, we introduce a multi-agent discussion framework that effectively enhances accuracy and consensus, outperforming individual LLMs. Finally, we conduct fine-grained analysis and identify specific model behaviors emerging during the multi-agent discussion process.
Our results spanning both single-L.LM and multi-agent methods across three domains and five datasets - indicates that performant LLMs may not serve as a direct alternative for annotation tasks requiring domain expertise. Relying solely on parametric knowledge to perform domain-specific, expert-level annotation remains a non-trivial challenge. Given that these specialized domains often involve high-risk sectors, ensuring the precision and accuracy of annotated data is critical. Overall, our findings highlight the gap between existing LLMs and human experts, underscoring the need for future efforts to develop systems.
reliable LLMs-as-Enn Maaa---peet-Annottee
Limitation and Future Works
As aim provide direct insight and observation on whether top-performing LLMs can perform as expert annotators out-of-the-bux, we minimize efforts in prompt engineer- ing. Some works have demonstrated that, for specific scenarios, one can achieve sizable improvement through carefully-crafted prompts. Consequently, our results may further benefit from a more exhaustive prompt optimization.
Another potential limitation is that we primarily focus on natural language understanding tasks with fixed label space. Towards a more comprehensive evaluation, natural language generation tasks could be further incorporated. Furthermore, all of our experimental settings involve zero-shot configurations using general-purpose chatbot LLMs. To unveil more of the capabilities of LLMs in annotation tasks, future directions could explore few-shot settings, domain-specific or fine-tuned LLMs tailored to the annotation tasks, retrieval- -lggmeetde generation methods, or a promising human-LLM hybrid annotation schema.

---

Published as a conference paper at COLM 2025
Acknowledgments
This work was supported by National Science and Technology Council, Taiwan, under grants NSTC 113-2634-F-002-003- and 114-2221-E-022-000MMYYY and Ministry of Education (MOE), Taiwan, under grant NTU-114L990000 The work of Chung-Chi Chen was supported in part by AIST policy-based budget project "R&D on Generative Al Foundation Models for the Physical Domain."
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. Open-source large language models out- perform crowd workers and approach chatgpt in text-annotation tasks. arXiv preprint ar drXiv:2307.02179, 2023.
AI Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
AI Anthropic. Claude 3.7 sonnet and claude code. 2025. URL hww.h:ttss////t news/claude-33 3-7-son-e-----t
com/
Parikshit Bansal and Amit Sharma. Large language models as annotators: Enhancing generalization of nlp models at minimal cost. arXiv preprint arXiv:2306.15766 2023.
Nimet Beyza Bozdag, Shuhaib Mehri, Xiaocheng Yang, Hyeonjeong Ha, Zirui Cheng, Esin Durmus, Jiaxuan You, Heng Ji, Gokhan Tur, and Dilek Hakkani- Must read: A systematic survey of computational persuasion. ar preprint ar arXi::2505.07775, 2025.
Ethan Callanan, Amarachi Mbakwe, Antony P'apadimitriou, Yulong Pei, Mathieu Sibue, Xiaodan Zhu, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. Can gpt models be finan- cial analysts? evaluation of chatgpt and gpt-4 mock cfa exams. arXiv preprint ar arXiv:2310.08678, 2023.
Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse 1lms. ar preprint arXiv:2309.13007, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. ar Xiv preprint ar arx:::2107.0333 2021.
Cheng-Han Chiang and Hung-Yo Yi Lee. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Ammual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607-15631, 2023.
Jonathan H Choi, Kristin E Hickman, Amy B Monahan, and Daniel Schwarcz. Chatgpt goes to law school. J. Legal Educ., 71:387, 2021.
Juhwan Choi, Eunju Lee, Kyohoon Jin, and YoungBin Kim. Gpts are multilingual annotators for sequence generation tasks. arXiv preprint ar 2001:2240,.5551, 2024.
Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. Is gpt-3 good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11173-11195, 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. Xiv preprint ar arXit:2305.14325 2023.
10

---

Published as a conference paper at COLM 2025
Esin Durmus, Liane Lovitt, Alex Tamkin, Stuart Ritchie, Jack Clark, and Deep Ganguli. Measuring the persuasiveness of language models, 2024. URL cottps://vee http:::://www..ttthhh http:/ww............../ homm//neww//maauuuinn-ooododd perpeeeoee--eesssssseeeeeeess
Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological
bulletin, 76(5):378, 1971.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd work- ers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30): e2305016120, 2023.
Google. Introducing gemini 2.0: our new ai model for the agentic era. 2024. URL https:// blog. sogglee/techhllogy/ sengle-deepmingg hoooogg cemini-ai-updoe -e--uuaaeeddeeebber #ceo-message.
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36, 2024.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. ar Xiv preprint ar arXi::2402.01 680, 2024.
Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint
Stt:23033.1854.. 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2020.
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. Cuad: An expert-annotated nlp dataset for legal contract review. ar arv preprint Xio:2103.00668,, 2021.
Ting-Hao Kenneth Huang, Chich-Yang Huang, Chien-Kuang Cornelia Ding, Yen-Chia Hsu, and Lee Giles. Coda-19: Using a non-expert crowd to annotate research aspects on 10,000+ abstracts in the covid-19 open research dataset. In ACL 2020 Workshop on Natural Language Processing for COVID-19 (NLP-COVID), 2020.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567-2577, 2019.
Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270):20230254, 2024.
Simerjot Kaur, Charese Smiley, Akshat Gupta, Joy Sain, Dongsheng Wang, Suchetha Sidda- gangappa, Toyin Aguda, and Sameena Shah. Refind: Relation extraction financial dataset. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3054-3063, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199 222119,2221122921 2022.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.191 19118, 2023.
11

---

Published as a conference paper at COLM 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Quinn McNemar. Note on the sampling error of the difference between correlated propor- tions or percentages. Psychometrika, 12(2):153-157, 1947.
OpenAI. Hello gpt4-o. 2024. URL
OpenAI. Openai 03-mini. 2025. URL
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint ar XrXo:2203.00550, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman, Gpqa: A graduate-level google- proof q&a benchmark. arv preprint "XXx:2311.12022, 2023.
Agam Shah, Suvan Paturi, and Sudheer Chava. Trillion dollar words: A new financial dataset, task & market analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6664-6679, 2023.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):122-180, 2023a.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. Towards expert-level medical question answering with large language models. ar Xrv preprint ar Xxv::2305.00611,, 2023b.
Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation: A survey. ar preprint ar arXio:2402. 13446, 2024.
Yu-Min Tseng, Wei-Lin Chen, Chung-Chi Chen, and Hsin-Hsi Chen. Are expert-level language models expert-level annotators? arx preprint ar XrXo:22400.32254, 2024a.
Yu-Min Iseng, Yu-Chao Huang, Teng-Yun Yun Hsiao, Yu-Ching Hsu, Jia-Yin Foo, Chao-Wer Huang, and Yun-Nung Chen. Two tales of persona in llms: A survey of role-playing and personalization. arXiv preprint ar arXiv:2406.01171, 2024b.
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuar Sung. Foundational autoraters: Taming large language models for better automatic evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17086-17105, 2024.
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Douglas Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, et al. Cord-19; The covid-19 open research dataset. ArXiv, 2020.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Repre- sentations, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. Adoances in neural information processing systemss, 35:24824-24833 2022.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121 101, 2025.
12

---

Published as a conference paper at COLM 2025
Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. Llmaaa: Making large language models as active annotators. arXiv preprint arXiv=2310.19596, 2023.
Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. Can chatgpt reproduce human-generatee labels? study of social computing tasks. arXiv preprint 1rX10:2304.10145, 2023.
A Datasets
A.1 Finance
REFinD (Kaur et al., 2023) is the largest relation extraction dataset over financial documents, comprising 8 entity pairs and 22 relations, with labels reviewed by financial experts. In this task, annotators are tasked to select the relation type between finance-specific entity pairs, such as pperson] is an employee of lorganization).
FOMC (Shah et al., 2023) is constructed for identifying sentiments about the future monetary policy stances, annotated by experts with correlated financial knowledge. The labels of this annotation task Dovish, Hawkish, and Neutral, where Dovish sentence indicates easing and Hawkish sentence indicates tightening.
A.2 Law
CUAD (Contract Understanding Atticus Dataset; Hendrycks et al., 2021) consists of legal contracts with extensive annotations from legal experts, created with year-long effort by dozens of law student annotators, lawyers, and machine learning researchers. The annota- tion task is to label 41 types out of legal clauses, classified into 5 answer categories, that are considered important in contract review related to corporate transactions. We manually use "Yes/ "Yes/No" answer category to construct our annotation task as the identification of 32 types of clauses.
FoDS (Function of Decision Section dataset; Guha al., 2024) comprises one-paragraph excerpts from legal decisions, annotated by legal professionals. In this task, annotators are tasked to review a legal decision and identify one out of seven function categories that each section (i.e., excerpt) of the decision serves.
A.3 Biomedicine
CODA-19 (COVID-19 Research Aspect Dataset; Huang et al., 2020) codes each segment aspect of English abstracts in the COVID-19 Open Research Dataset (Wang et al., 2020). In this task, annotators are tasked to label each segment as background, purpose, method, Enddnn/cnntribution, other sections. To ensure the quality of the labels, we only adopt instances annotated by biomedical experts. We provide data statistics of the five existing specialized datasets in Table 2.
B Annotation Guidelines
We provide annotation guidelines of each dataset from Figure 5 to Figure 11.
C Prompt Templates
We provide prompt templates of each methods from Figure 12 to Figure 18.
D Additional Results
We provide the accuracy and corresponding p-value of the comparison between reasoning models and non-reasoning models in Table 3 and Table 4.
13

---

Published as a conference paper at COLM 2025
Domain
Dataset
Instance Type
#Instances
#Labels
REFinD (Kaur al., 2023)
Sentence
22
Finance
FOMC (Shah et al., 2023)
Sentence
CUAD (Hendrycks et al., 2021)
Clause
200
32
Law
FoDS (Guha et al., 2024)
Excerpt
200
7
Biomedicine
CODA-19 (Huang al., 2020)
Paper Abstract
Table 2: The statistics of existing dopaif-specific datasets used in this paper.
Finance
Law
Biomedicine
Accuracy
REFinD
FOMC
CUAD
FoDS
CODA-19
Best Vanilla
67.5
71.5
86.5
47.0
79.5
Best CoT
67.0
69.5
84.0
45.5
77.5
03-mini (medium)
71.5
73.0
87.0
46.5
82.0
Claude 3.7 Sonnet (thinking)
73.0
74.0
86.5
84.5
Table 3: Accuracy comparison between reasoning models and non-reasoning models. Text in bold indicates the highest accuracy for each dataset.
Finance
Law
Biomedicine
P-value
REFinD
FOMC
CUAD
FoDS
CODA-19
03-mini vs. Best vanilla
0.230
0.728
1,000
1.000
0.533
03-mini vs. Best CoT
0.093
0.249
0.263
0.856
0.150
Claude 3.7 Sonnet vs. Best vanilla
0.135
0.500
1.000
1.000
0.110
Claude 3.7 Sonnet vs. Best CoT
0.043
0.163
0.332
0.839
0.013
Table P-values of the significance tests between reasoning models and non-reasoning mod- els. P-values rounded to three decimal places. Text in bold indicates the reasoning model is statistically significantly better than the best non-reasoning models, with p-value 0.05.
14

---

Published as a conference paper at COLM 2025
Relation Extraction (RE) is the task of extracting relationships between entities in sentence.
You be given sentence contains two entities: entity 1 and entity 2.
Entity is enclosed in double asterisks (i.e," "enttty"*) and entity is enclosed in double underscores (Le.,
_eenity
Each entity has own entity type specified in square brackets before the entity (eg., [PERSON]"entity"""
The definition of the entity types are as follows:
PERSON: People, including fictional.
ORGG Companies, agencies, institutions, etc.
UNIV: Universities, colleges, etc.
GOV_AGY: Covernment agencies and departments.
DATE: Absolute or relative dates or periods.
GPE: Countries, cities, states.
MONEY: Monetary values, including unit.
TITLL: Positions or titles, including military.
Please the relation between entity and entity a described the given sentence according the following label descriptions.
Note that the relation is directional, meaning that the order of entity and entity matters.
Note you only select the most appropriate label that is consist of the given type of entities. If you think there no relation or other relation between entity and entity 2, please select the label o.
""entity ** relation or relation entity 2
-1: [PERSON]"entity 1"* has/had the job title of [TITLE] entity 3
[PERSON]* |** is/was employee of [ORG]_enipg
3: [PERSON]"entitt>> is/was member of [ORG] [ORG]_entity> 2_
4: [PERSON/"entitt> 1** is (s/was founder of [ORG] .etitetttt............. __
5: [PERSON]]entitty 1** is/was employee of [UNIV]_entig
6: [PERSON]"entity is/was member of [UNIVI_,ntity 2_
7: [PERSON]"'entity"" 1* has/had attended [UNIV]_nniiyy
9: [ORG]* is/was formed on [GOV_AGY]
8: [PERSON]"'etitty"" is/was member of
2_
10: [ORG]' =* is/was acquired on [DATE]_entiyy 2
[ORG]* 1** is/was headquartered [GPE]_
12: [ORGi* 1* has/had operations in [GPEL_entity:
13: [ORG]* eettity 1* is/was formed in [GPE] _GPE\]_enitt
14: [ORG]' has/had shares of [ORG]_ entity 2__
15: [ORG]* [ORG)" is/was a subsidiary of c
16: (ORGI"'mttity" ** is/was acquired by [oOG]]_nnityy
177 [ORG]' cntityy"" has/had agreement with [ORG]_ entity
has/ha 18: [ORG]*entity: 1* has/had a revenue of Leniity__nttt>>>> 2
[ORG] e" 1** has/had profit of (MOOEY]__enttt
"eRiityntity has/had loss of [MONEY] entity 2
21: [ORG]*entin ** has/had a cost of [MONEY] eenttty 2_
Figure 5: The annotation guideline of REFinD dataset.
15

---

Published as a conference paper at COLM 2025
Hawkish-Dovish classification is to classify the sentiment about the future monetary policy stance into Dovish, Hawkish, Neutral.
In general:
Dovish sentences any sentence that indicates future monetary policy easing.
1:: Hawkish sentenees were any sentence that would indicate monetary policy tightening.
2: Neutral sentences were those with mixed sentiment, indicating no change in the monetary policy, or those that not directly related monetary policy stance.
You will be given sentence that falls into one of the following eight categories enclosed in square brackets. Please annotate the sentiment of the sentence according the following detailed label descriptions.
Note that you can only select one label is most appropriate.
Detailed label descriptions:
[Economic Status: A sentence pertaining the state the economy, relating to unemployment inflation.] or when inflation decreases, when unemployment increases, when economic growth projected as low. when inflation increases, when umemployment decreases when economic growth projected high when economic output is higher than potential supply/actual output when economic slack falls.
a: when unemployment rate or growth is unchanged, maintained, sustained.
[Dollar Value Change: sentence pertaining changes such appreciation depreciation value of the United States Dollar on the Foreign Exchange Market.]
or when the dollar appreciates.
when the dollar depreciates.
2: N/A
(Energy/House Prices: sentence pertaining to changes prices of real estate, energy commodities, energy sector a whole.]
oo: when
prices decrease, when house prices decrease. when oil/energy prices increase, when house prices increase.
2: N/A
[Foreign Nations: sentence pertaining to trade relations between the United States and a foreign country. If not discussing United States we label neutral.]
when the trade deficit decreases.
-: when the US trade deficit increases.
when relating foreign nation's economic trade policy.
[Fed sonicaatiossActiinns///aa other financial asset value.]
A sentence that diseusses changes the Fed yields, bond value, reserves, or any
Fed expects subpar inflation, Fed expecting disinflation, narrowing spreads of treasuryy bonds, decreases in treasury security yields, reduction of bank reserves.
:: Fed expects high inflation, widening spreads of treasury bonds, increase in treasury security yields, increase in TIPS value, increase bank reserves.
2: N/A
[Money Supply: A sentence that overtly discusses impact to the money supply or changes in demand.)
money supply low, increases, increased demand for loans.
t: money supply is high, increased demand for goods, low demand for loans.
2: N/A
[Key Words/Phrases: /Parrssrrsss sentence that contains key word phrase that would classify squarely into of the three label classes, based upon its frequent usage meaning among particular label classes.
or when the stance is "accommodaative" indicating focus on "maximum employment" and "price stability". indicating focus "price stability" and "sustained growth". 2: use of phrases "mixed", "moderate", "reaffirmed".
[Labor: sentence that relates to changes in labor productivity.] when productivity increases.
1: when productivity decreases.
2: N/A
Figure 6: The annotation guideline of FOMC dataset.
16

---

Published as a conference paper at COLM 2025
A/P Relation classification classify relation between Assessment and Plan Subsection in daily progress notes into DIRECT, INDIRECT, NETHERR or NOT RELEVANT:
You will be given pair passages, Assessment Assessment describes the patient and establishes the main symptoms or problems for their encounter. Plan Subsection addresses each differential slagnesis/problem with action plan treatment plan for the day.
Subsection, daily progress notes.
Please annotate the relation between Assessment and Plan Subsection in the given pair according to the following label descriptions.
that you can only select one label is most appropriate.
Label descriptions:
o: DIRECT. Assessment section includes a primary dingnosis/problem and mentioned in the Plan subsection, or Progress note includes primary diagnosis/ diagnosis/problem problem for hespitalization and it is mentioned in the Plan subsection, Plan subsection contains problem /diagnosis related the primary ggnnssymptoms the Assessment section. :: INDIRECT. Plan subsection contains pmmplicationssuuseqqee events or organ failure related to the primary diagnosis/pren problem from the Assessment section, or Plan subsection contains other listed diagnoses/problee the overall Progress Note in the Assessment section that subsection contains diagnosis/pronn to the primary diagnoses/ prgnnoses/probleee problems mentioned in the overall Progress Note or Assessment section.
from part of the primary haaggossss/prbleene
Plan that not previously mentioned but closely related (Le,, same organ system)
NEITHER. None of the criteria for Directly Related Indirectly Related met but diagnosis/problem other signs/sym syymmmmmsss are mentioned.
3: NOT RELEVANT. Plan subsection does not include dingmosis/problemt OR signs/symptoms.
Figure 7: The annotation guideline of AP-Relation dataset.
You will be given one paper abstract comprising several segments. Each segment short text describing a specifie aspect of the paper, including background, purpose, method, finding/contrbutionn or other.
Please annotate the aspects of each segment according to the following label descriptions. Note that you only select one label that is most appropriate for each segment. The total number of labels must be equal to the number segments the abstract.
Label descriptions:
Background. "Background" text segments answer one or more important?, What relevant works have been created before?, What is still missing in the previous works?, What are the high-level research questions?, How might this help other research researchers?
questions: Why this problem
u: Purpose. "Purpose" text segments answer one or more these questions: What specifie things do the researchers want to do?, What speciffi knowledge do the researchers want to gain?, What specific hypothesis do the researchers test?
2: Method. "Method" text segments answer one or more these questions: How researchers do the work or find what they sought?, What are the procedures and steps of the research? Trindin/Contrribtti
3: Sinding/Contributio
of these questions: What did researchers find out?, Did the proposed methods work? Did the thing behave as the researchers expected?
text segments
4: Other. Text segments that do not fit into any of the four categories above. Text segments that are not part of the article. Text segments that
English. Text segments that contain only reference marks (egg, "(.7...4.77) dates (eg,, "April 20, 2008"). Captions for figures tables (eg. "Figure :: Experimental Result of...." Formatting errors. Text segments the annotator does not know or not sure about.
Figure 8: The annotation guideline of CODA-19 dataset.
17

---

Published as a conference paper at COLM 2025
You will be given clause from a legal contract. Please annotate the category of the given clause according the following label descriptions. Note that you can only select one label for each segment that is most appropriate.
Label descriptions:
or Most Favored Nation. This clause provides that if third party gets better terms on the licensing sale of described the contract, the buyer of such haannologggood///neeeegg under the contract
shall entitled to those better terms.
a: Non Compete. This clause imposes restriction the ability of Party to compete with the other party operate in certain geography business technology sector.
2: Exclusivity. This clause provides for an exclusive dealing commitment between the parties of a contract. This clause also includes: commitment by party to procure all "requirements" from the other party of certain technology, goods, services; a prohibition against licensing selling technology, goods services to third parties, or prohibition on collaborating or working with other parties.
3: No-Solicit of Customers. This clause restricts a party from soliciting, contacting doing business with the other party's customers, vendors or partners.
4: Competitive Restriction Exception. This clause states the exception(s) to one of the following three labels: Exclusivity, Non-Compete, No-Solicit of Customers.
5: No-Solicit Employees. No-Solicit of Employee clause prohibits party from soliciting or hiring the other party's 'ss employees or consultants for itself or for a third party, during the contract or after the contract ends (or both),
6: Non-Disparagement This clause requires party not disparage or defame the other party's goodwill, reputation or image.
7: Termination for Convenience. This clause allows a party to terminate a contract without cause or penalty. It allows party unilaterally terminate contract by giving notice and oftentimes after waiting period expires.
8: Right of First Refusal, Offer or Negotiation Kaff/Roooo/Rofi This clause grants one party a right of first refusal, right of first offer or right of first negotiation to purchase, license, market, or distribute equity interest, technology. assets, products services.
9: Change of Control, This clause requires consent or notice other party if party undergoes change control, such as a merger, stock sale, transfer of all or substancially all of its assets or business (collectively, "CIC"). Anti- Assignment. This clause requires party to seek consent notice if the contruct is assigned, transferred sublicensed third party, in whole or part.
u: Levvenue/Profii Sharing, This clause requires one party to share revenue or profit with the other party for any
technology, goods, services.
12: Price Restriction. This clause restricts the ability of party to raise or reduce prices technology, goods, or services provided.
1:: Minimum Commitment. This clause requires minimum order size minimum amount units per-time period that one party must buy from the counterparty under the contract.
14: Volume Restriction. This clause charges fee requires consent if one party's use of the product/servee services exceeds certain threshold.
15: IP Ownership Assigument. This clause provides that intellectual property created by one party becomes the property of the other party, either per the terms of the contract or upon the occurrence of certain events.
Figure 9: The annotation guideline of CUAD dataset (1-1).
18

---

Published as a conference paper at COLM 2025
116: Joint IP Ownership. This clause provides joint shared ownership of intellectual property between the parties the contract.
17: License Grunt. This clause authorizes party to use intellectual property intangibles of the other party. It can be an authorization use reproduce, distribute, manufacture, etc. certain content, technology. or other items that are protected by intellectual property rights. This clause is very common, and is considered one the "factual" clauses. The purpose of this label is help human reviewers to understand what IP is licensed under contract and what restrictions imposed the license, including restrictions duration, territory and purpose of use.
propprrry
18: Non Transferable License. This clause prohibits one party to transfer, assign or sublicense in the contract.
19: Affiliate IP Sicense-Licensor This clause contains license grant by affiliates of the licensor that includes intellectual property of affiiiitttes of the licensor.
20: Affiliate IP License-Licensee Licensee. This clause contains a license grant to a licensee (incl. sublicensor) and the
affiliates of such license/ssublicensor.
Unlimited//Al /All--uu Can License. This clause contains provision granting one party an "enterprise, " "all you can eat" or unlimited usage license.
Irrevocable or Perpetual License. This clause contains irrevocable and/on perpetual license of IP. An irrevocable livense perpetual license that cannot cut short or terminated. perpetual license, on the other hand, may not be irrevocable. Namely, perpetual license can be terminated upon specified events such as material breach. Many license grant clauses use "irrevocable" and "perpetual" in the same sentence. The intent of some contracts may be to use the two terms interchangeably. a result, for purpose CUAD, you should label two types of licenses under the same label.
23: Source Code Escrow. This clause requires party deposit its source code into with third party into deposit account the other party, which can released to other party upon occurrence of certain events (bankruptcy, insolvency, etc.).
24: Past- Termination Services. This clause imposes obligutions party after the termination expiration of contract, including any post-termination transition, payment, transfer IP, wind-down, last-buy, or similar committments.
25: Audit Rights. This clause grants party the right audit the books, records, physical locations of the other party to ensure compliance with terms contract.
26: Uncapped Liability. This clause leaves party's lability uncapped upon the breach of its obligation in the contract. This also includes uneap liability for particular type of breach such infringement breach of confidentiality obligation.
Cap On Liability. This clause includes on liability upon the breach of a party's 's obligation. This includes time limitation for the counterparty bring claims or maxinum amount for recovery.
28: Liquidated Damages. This clause is an agreement to pay a party a pre-determined amount of damages the other party breaches the contract. For the purpose of CUAD, this clause also includes an early termination fee.
29: Insurance. This clause requires party to maintain insurance for the benefit of the other party.
3o: Covenant not to Sue. This clause restricts party from contesting the validity of the other party's 's ownership of
intellectual property or otherwise bringing claim against the other party that goes beyond the scope of standard Limitation on Liability clauses.
3r: Third Party Beneficiary. This clause provides that a non- contraaatg party a beneficiary to some or all the clauses in the contract and therefore can enforce its rights against contracting party.
Figure 10: The annotation guideline of CUAD dataset (1-2).
19

---

Published as a conference paper at COLM 2025
You will be given paragraph excerpt of legal decision. Please annotate the category of the given excerpt according the following label descriptions. Note that you can only select one label that is most appropriate for the excerpt.
Label descriptions:
o: Facts, A section of the decision that recounts the historical events and interactions between the parties that gave
rise the dispute.
i: Procedural History. section of the decision that describes the parties' prior legal filings and prior court decisions
that led the issue to be resolved by the decision.
Issue. section of the decision that describes legal factual issue considered the court.
3: Rule. A section the decision that states a legal rule relevant to resolution of case.
4: Analysis. A section of the decision that evaluates an issue before the court by applying governing legal principles
the facts of the
5: Conclusion. A section the decision that articulates the court's conclusion regarding question presented to it.
6: Decree. A section of the decision that announces and effectuates the court's resolution of the parties' dispute, for example, granting or denying party's motion affirming, vacating, reversing, remanding lower court's decision.
Figure 11: The annotation guideline of FoDS dataset.
You are [domain] expert tasked to annotate [domain] dataset. Please follow the annotation guideline below.
Annotation Guideline: [[guideline]i
[instanceettpe]]: [[instance]
Please strictly follow the guideline and output the label in the format of: 'The label is .... Do not include any reasoning explanation.
Figure 12: Vanilla prompt template.
[domain] expert tasked [domain] dataset. Please follow the annotation guideline below.
Annotation Guideline: [[guideline]i
[[instance_tye _typee]___pp [[instance]i
Please strictly fallow the guideline and output the reasoning and the label in the format of: 'Let's think step by step.... label
Figure 13: Chain-of-Thought prompt template.
20

---

Published as a conference paper at COLM 2025
You are [domain] expert tasked to annotate [domain] dataset. Please follow the annotation guideline below.
Annotation Guideline:
([guideline] ]]
[jintaanc__typee ([instance]
Please strictly follow the guideline output the reasoning label format of: think step by step... The label is....
Figure 14: Self-Refine prompt template. Step 1: Generate.
You are [domain] expert tasked to annotate [domain] dataset. Please follow the annotation guideline below.
Annotation Guideline: ([guideline]i
[finstance_typer tl [[instance]
Please strictly follow the guideline output the reasoning and label format of: step by step... The label is....
([model response from step 1.]]
Review your previous reasoning and annotation and find potential problems. For example, whether the annotation guideline violated, whether reasoning is not conclusive.
Figure 15: Self-Refine prompt template. Step 2: Review.
You are [domain] expert tasked to annotate [domain] dataset. Please follow the annotation guideline below.
Annotation Guideline:
([guideline]i
[jinstance_typel]]) ([instance]i
Please strictly follow the guideline and output the reasoning and the label in the format of: 'Let's think step by step... The label is ...!.
([model response from step 1.]
Review your previous reasoning and annotation find potential problems. example, whether annotation guideline is violated, whether the reasoning is not conclusive.
Review:
([model response from step 2.]}
Based the problems you found the above review, improve your annotation quality and reasoning and output the furmat of: 'Let's think step by step The label is...
Figure 16: Self-Refine prompt template. Step 3: Refine.
21

---

Published as a conference paper at COLM 2025
You [domain] expert tasked to annotate [domain] dataset. Please follow the annotation guideline below.
Annotation Guideline: ([guideline]i
(jinstance__yyr __tpe]]: [[instance]
Please strictly follow the guideline and output the reasoning and the label in the format of: 'Let's think step by step... The label is...
Figure 17: Multi-agent peer-discussion prompt template, Step 1: Generate initial annotation.
You are [domain] expert tasked to annotate [domain] dataset. Please follow the annotation guideline below.
Annotation Guideline: ([guideline]i
([instance_typee _type]k (iinstance]
Please strictly follow the guideline and output the reasoning and the label in the format ofi 'Let's think step by step The label is..
Discussion History: (Round
Annotator A thinks [model A's output, including reasoning path and the annotated label] Annotator B thinks [model B's output,... Annotator thinks [model output,-.]
Round ...]
You need to consider the above discussion history carefully. You maintain your point of view and annotation if others' reasons are not concrete or cannot convince you.
Please strictly follow the guideline and output the reasoning and the label in the format of: 'Let's think step by step....
The label
Figure 18: Multi-agent peer-discussion prompt template. Step 2: Discuss and re-annotate.
22


Reviews:
[{"review_id": "9sYJspHLMy", "reviewer": "Reviewer_jNni", "summary": "This paper explores the use of large language models (LLMs) as annotators in specialized domains such as finance, law, and biomedicine. The authors evaluate both instruction-tuned and reasoning-focused models on five datasets, using only annotation guidelines and unlabeled data. They also test inference-time techniques like chain-of-thought prompting, self-refinement, and self-consistency. Results show that inference-time techniques offer limited benefit for instruction-tuned models, while reasoning models perform slightly better overall. Multi-agent collaboration improves both accuracy and model agreement, but remains limited when initial predictions lack the correct answer. The analysis also finds that reasoning models are more rigid, while models like Gemini 2.0 Flash and GPT-4o revise outputs more flexibly.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper offers a comprehensive evaluation of LLMs as annotators across specialized domains (finance, law, and biomedicine) and addresses an underexplored but practically important area.\n- It compares a diverse set of models (i.e., instruction-tuned and reasoning models: Clause-3-Ops, Gemini 1.5 Pro, Gemini 2.0 Flash, GPT-4o, Clause-3.7-Sonnet, o3-mini) and inference-time techniques (i.e., CoT, self-consistency, self-refine, and multi-agent discussion), providing insights into their relative strengths and weaknesses.\n- The analysis of multi-agent discussion is detailed and insightful, showing both performance improvements and inherent limitations.\n- The paper presents results clearly through informative figures and tables, making the findings easy to interpret.", "reasons_to_reject": "Overall, this paper is highly interesting. My main concern lies in the limited discussion surrounding the key empirical observations reported in the experiments, specifically, why inference-time techniques underperform while multi-agent collaboration proves effective. These contrasting results are intriguing, yet the paper does not offer sufficient explanation or analysis to help the reader understand the underlying causes.\n\n1. The paper reports that inference-time techniques (e.g., chain-of-thought, self-refine, self-consistency) tend to degrade performance when applied to instruction-tuned models (while multi-agent collaboration proves effective). However, it does not provide a clear explanation for this phenomenon. This is especially notable given that prior work has shown these inference-time techniques to improve performance across a variety of tasks. Since expert-annotation tasks share a similar input-output formalization with those in existing studies, the lack of improvement raises important questions. For example, could the domain-specific nature of the required knowledge limit the effectiveness of these techniques? (If so, why?) Addressing such questions would enhance the paper’s depth and its potential to inspire further discussion within the community.\n2. The paper does not discuss whether the findings (such as the limited effectiveness of inference-time techniques and the benefits of multi-agent collaboration) are specific to expert domains (e.g., finance, law, biomedicine) or generalizable to other domains. Clarifying this point would help position the work more clearly within the broader literature.\n3. Important practical aspects such as annotation cost, inference latency, and scalability are not discussed, which limits the study’s implications for real-world deployment.", "questions_to_authors": "1. Why do you think inference-time techniques such as chain-of-thought prompting, self-consistency, and self-refinement underperform in your expert-annotation tasks, despite prior work showing their effectiveness in other NLP tasks? Does the domain-specific nature of the required knowledge plays a role? Could you elaborate on possible reasons?\n2. Do you expect the observed behaviors (particularly the failure of inference-time techniques and the success of multi-agent collaboration) to generalize beyond expert domains? Or do you think these are specific to the complexity and implicit knowledge requirements of domains like finance, law, and biomedicine?", "extra_scores": {}}, {"review_id": "rQXiJHtMQi", "reviewer": "Reviewer_CsN1", "summary": "This work focused on whether large language models (LLMs) can replace human expert annotators in specialized domains. The authors evaluated individual LLMs and multi-agent approaches across finance, biomedicine, and law fields. They empirically observed that (1) chain-of-thought techniques yielded minimal/negative gains, (2) reasoning models excelled in domain-specific annotation, and (3) some models remained stubborn about initial annotations despite contradicting evidence.RetryClaude can make mistakes. Please double-check responses.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The work identifies the gap of evaluating LLM’s annotation capability, benchmarked against expert annotation when most prior efforts use crowdworker annotation.  \n\n2) Model and benchmark choices are comprehensive and well justified.  \n\n3) Propose an effective multi-agent framework and explore its effectiveness.", "reasons_to_reject": "1) It will be good to discuss the significance of findings on top of describing the results.   \n\n2) Lack discussion of robustness: The authors can vary the prompt and average the performance to increase the robustness of the finding.   \n\n3) Suggest to add qualitative analysis on the challenging tasks/instances would be an important component for us to understand LLMs’ capability as expert annotators.", "questions_to_authors": "1) Lack in-depth discussion: the study shows inference methods may not consistently help. What are the possible reasons and what is the significance of this observation? Also, what are some possible reasons for reasoning models not changing their annotations? How is this observation helpful for future tasks?  \n\n2) Significance testing: Since the performance gain by the reasoning model is marginal, how significant is the performance improvement?  \n\n3) Max discussion round is set to 2: The choice of 2 is a bit arbitrary: if model choice is changed in future, this number of rounds may not be optimal. Why not letting models keep discussing until consensus is reached?  \n\n4) Possibility of improving current prompts: The authors mentioned the limitation of missing correct annotation in the initial round be alleviated. I wonder if this is the limitation of current prompt. If you design the peer-discussion prompt to inform that the model’s annotation and other models’ discussion could be both incorrect, can the model take inspiration from other models’ thinking process (not final output) and change annotation to correct one in following rounds?  \n\n5) Related work covers limited work. There are works studying LLMs’ capability as expert annotators (e.g. Are Expert-Level Language Models Expert-Level Annotators? (Tseng et al., 2024)) Please cite them and discuss your work’s difference from prior works.", "extra_scores": {}}, {"review_id": "DDYvdm7v16", "reviewer": "Reviewer_XTfE", "summary": "The paper proposes a general method for solving different classification tasks, showing that a simulation of inter-annotator discussion at inference timeleads to better matching the gold standard labels across tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed method is straightforwards and applicable to any classification task. \n\n* The experimental results show the positive impact of implementing a strategy that emulates a discussion between agents, both in terms of inter-agent agreement and accuracy.", "reasons_to_reject": "The selection of specific tasks on which the approach is tested may be critical, and it is not discussed. In fact, the definition of the tasks themselves is only found in the appendix. For one thing, different tasks call for different evaluation metrics. While accuracy is technically fit for any classification task, it may not be the best way to describe the performance, e.g. in the case of unbalanced label distribution. Generative models are also known to be sensitive to how they are prompted. Given the vast difference from one task to another, there need to be a strategy to minimize the dependance of the results from the variations in prompting.", "questions_to_authors": "Why is the approach cast as \"annotation\" and not just as an ensemble-based classification approach? Is there a fundamental difference?", "extra_scores": {}}, {"review_id": "besZfsBMQu", "reviewer": "Reviewer_6CbB", "summary": "**Summary**: This paper presents an extensive evaluation for using LLMs in expert annotation tasks. The evaluation is conducted on five datasets covering three domains that require expert knowledge (biomedicine, law, and finance). The authors first evaluate four state-of-the-art LLMs with various recent prompting techniques and find that none of them lead to a clear improvement across all domains. They then evaluate different combinations of models in a multi-agent discussion framework and find that this does improve the overall performance. The analysis reveals some interesting model behaviors and establishes the correctness of the initial predictions as the upper bound for the multi-agent discussion framework.\n\n* *Quality*: This paper presents a good analysis on using LLMs as expert annotators across different domains, models, and methods. The experiments are well motivated and conducted in a systematic manner (e.g., the authors explicitly focus on datasets that include annotation guidelines). \n\n* *Clarity*: The paper is well written and easy to follow. Some parts could be explained in more detail (see reasons to reject/ questions).\n\n* *Originality*: The paper is more a systematic study of existing methods and models; which somewhat impacts the originality. However, the selection of domains and datasets is somewhat creative and original.\n\n* *Significance*: Considering the recent trend of utilizing LLMs for annotation tasks and for synthetic data generation, the results of this work provide some interesting evidence towards how questionable such usage is. Moreover, this work makes a good case towards a more thorough evaluation of many frequently used approaches.\n\n----\n\nRebuttal update: raised score to 7", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a systematic study on the use of LLMs for expert annotation tasks. The paper is well written and clearly motivated; and the conducted experiments cover recent models as well as methods. The results clearly show the limitations of existing methods as well as models and raise some interesting questions for future research.", "reasons_to_reject": "While it nice that this work presents extensive results on many state-of-the-art models, this is at the same time one of the weaknesses of this work. The use of proprietary models makes it impossible to conduct in-depth analysis which could have been also very insightful (e.g., some attribution analysis to see which parts of the discussion contributed to the final vote). Finally, some details about the conducted experiments is missing which impacts reproducibility (see questions).", "questions_to_authors": "Thank you for this interesting work! Although the presented results were quite insightful, there are some open questions where responses would really help to fine-tune the rating:  \n\n1. How was the discussion history compiled? Is this just a concatenation of the individual model outputs? Or was there some additional summarization step? \n2. Figure 4: Looks quite nice, but could you please elaborate and provide some explicit numbers? What can be observed there? \n3. Are you planning to release the full responses of the models for further analysis?", "extra_scores": {}}]

Author rebuttal:

Meta-review: I recommend accepting this paper. It provides a thorough evaluation of LLMs as expert annotators across specialized domains, with novel insights about their collaborative behaviors in a multi-agent discussion framework. Three of four reviewers supported acceptance, and the authors addressed key concerns in their rebuttal. The findings about reasoning models being less willing to change their initial annotations, while instruction-tuned models show more flexibility, offer valuable insights for deploying LLMs in expert annotation tasks. This work makes a meaningful contribution to an important practical problem.
Decision: Accept