From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning
Authors: Seungdong Yoa
Sanghyu Yoon
Suhee Yoon
Dongmin Kim
Ye Seul Sim
Junhyun Lee
Woohyung Lim
Venue: ICLR 2026 Poster (2026)
Keywords: Agent-centric benchmark
Language model assessment
Textual anomaly detection
Adaptive benchmarks
Primary area: datasets and benchmarks
TL;DR: We present a dynamic, agent-driven benchmark where teacher, orchestrator, and student agents generate, validate, and solve problems—enabling scalable evaluation without static datasets and exposing reasoning failures missed by standard benchmarks.

Abstract: The evaluation of large language models (LLMs) has predominantly relied on static datasets, which offer limited scalability and fail to capture the evolving reasoning capabilities of recent models. To overcome these limitations, we propose an agent-centric benchmarking paradigm that moves beyond static datasets by introducing a dynamic protocol in which autonomous agents iteratively generate, validate, and solve problems. Within this protocol, a teacher agent generates candidate problems, an orchestrator agent rigorously verifies their validity and guards against adversarial attacks, and a student agent attempts to solve the validated problems. An invalid problem is revised by the teacher agent until it passes validation. If the student correctly solves the problem, the orchestrator prompts the teacher to generate more challenging variants. Consequently, the benchmark scales in difficulty automatically as more capable agents are substituted into any role, enabling progressive evaluation of large language models without manually curated datasets. Adopting text anomaly detection as our primary evaluation format, which demands cross-sentence logical inference and resists pattern-matching shortcuts, we demonstrate that this protocol systematically exposes corner-case reasoning errors that conventional benchmarks fail to reveal. We further advocate evaluating systems along several complementary axes including cross-model pairwise performance and progress between the initial and orchestrator-finalized problems. By shifting the focus from fixed datasets to dynamic protocols, our approach offers a sustainable direction for evaluating ever-evolving language models and introduces a research agenda centered on the co-evolution of agent-centric benchmarks.

Published as a conference paper at ICLR 2026
FROM STATIC BENCHMARKS TO DYNAMIC PROTOCOL: AGENT-CENTRIC TEXT ANOMALY DETEC-
TION FOR EVALUATING LLM REASONING
Seungdong Yoa', Sanghyu Yoon', Suhee Yoon', Dongmin Kim', Ye Seul Sim', Junhyun Lee²*, Woohyung Lim
'LG AI Research, DDiision of Computer Engineering, Hankuk University of Foreign Studies (seungdong, yoaa, sanghyu. bolyyu.yy... yoor, Suhee.yoom dmkim d ai (ysl.simg w. w.lim) @lgresearch.al.
Stthyun..e@hhufs.co
ABSTRACT
The evaluation of large language models (LLMs) has predominantly relied on static datasets, which offer limited scalability and fail to capture the evolving reasoning capabilities of recent models. To overcome these limitations, we propose an agent- centric benchmarking paradigm that moves beyond static datasets by introducing dynamic protocol in which autonomous agents iteratively generate, validate, and solve problems. Within this protocol, a teacher agent generates candidate problems, orchestrator agent rigorously verifies their validity and guards against adversarial attacks, and a student agent attempts to solve the validated problems. An invalid problem is revised by the teacher agent until it passes validation. If the student correctly solves the problem, the orchestrator prompts the teacher to generate challenging variants. Consequently, the benchmark scales in diffi- culty automatically as more capable agents are substituted into any role, enabling progressive evaluation of large language models without manually curated datasets. Adopting text anomaly detection as our primary evaluation format, which demands cross-sentence logical inference and resists strate that this protocol systematically exposes corner-case reasoning errors that conventional benchmarks fail to reveal. We further advocate evaluating systems along several complementary axes including cross-model pairwise performance and progress between the initial and orchestrator- urchestrator-inalizeed problems. By shifting the focus from fixed datasets dynamic protocols, approach offers sustainable direction for evaluating ever-evolving language models and introduces research agenda centered on the co-evolution of agent-centric benchmarks.
shortcuts, we demon-
INTRODUCTION
Static benchmarks, such MMLU (Heddrycks et al. (021). GSM8K Cobbe et al. (021) and Big-Bench Srivastava et al. 2023). once served as reliable indicators of language model progress. However, frontier large language models (LLMs) approach- even surpass human-level accuracy on many of these tasks Pu et al. (2023); Maslej et al. (2024); Phan et al. (2025). Because these benchmark suites are finite, publicly accessible, and often included in pretraining corpora, models may inadvertently memorize substantial portions of the test data Deng et al. (2024). This can lead inflated leaderboard results that do not reflect genuine improvements in reasoning ability. Unfortunately, it has become increasingly difficult to draw meaningful distinctions from these overused datasets. First, data contamination is now common: large-scale data collection often includes benchmark questions in pretraining datasets, and efforts to remove them afterward usually incomplete Raji et al. (2021) ). Second, because static benchmarks contain a limited number of items, model developers may- sometimes without realizing it-tune their systems match the details of these benchmarks. This creates feedback loops that improve scores without real gains in general reasoning ability Dodge et al. (2020). Third, once benchmark is considered "solved", the research community must quickly create a new one. This leads to cycle of rapid creation and decline,
"Corresponding authors

---

Published as a conference paper at ICLR 2026
Clarity Y Difficulty X
Carlty Dificultty
Existing benchmark
ATAD (Ours)
The tennis champion withdrew due to injury. This marks her second major withdrawal of the production.
Artificial intelligence (All transforming healthcare by impreving diagnostic accurzcy. Healthcare providers models optimize treetment plans. New weerable devices monitor patient vitals in real time.
after OPEC announced increased
Figure I: Comparison of text anomaly samples. Left: Existing benchmarks include obvious anomalies (e.g., complete off-topic from sports news to economy news) that are clear but too trivial. Right: ATAD examples introduce subtle shifts within context (e.g., benefits to ethics in healthcare AD, preserving clarity while presenting reasoning-intensivy challenges. Our collaborative agents resolve the Harity-diffieully trade-off through iterative task refinement.
which uses up valuable time and provides only short-term insight into model performance Rogers (202T). These limitations highlight the inherent shortcomings of static benchmarks in evaluating real reasoning capabilities.
To overcome these shortcomings. dynamic benchmarks for LLMs are essential as they continuously evolve, mitigating data contamination and preventing models from overfitting to finite test sets. In particular, text anomaly detection serves as powerful task to reveal subtle reasoning flaws, providing clearer insight into the true capabilities and limitations of LLMss Maimon & Tsarfaty (2023). However, constructing high-quality text anomaly detection problems remains challenging: increasing the difficulty often sacrifices clarity, while ensuring clarity typically results in overly simple tasks. Figure T iillstratts this trade-off and motivates protocol's 's design. We introduce the Agent-centric Text Anomaly Detection (ATAD), a benchmark protocol that replaces the static- dataset paradigm with three-agent system. In this protocol, illustrated in Figure 2 teacher agent generates candidate problems, an orchestrator agent validates them and filters out defective items, and student agent attempts solve the qualified problems. As problem format, reasoning- centric anomaly detection tasks are well suited for evaluating LLMs: they require cross-sentence logical inference, resist pattern-matching shortcuts and training data leakage, and support objective, fine-gre ine-grained scoring. Asking a model to identify and explain the single sentence that disrupts passage's 's coherence offers a precise and robust measure of reasoning ability- one that less prone to exploitation than many existing benchmarks. By shifting the focus from fixed datasets to dynamic protocols, we offer sustainable direction for evaluating ever-evolving language models and invite the community to explore research agenda in which models and the benchmarks that probe them co- evolve. We will release an open- reference implementation with empirical results showing that ATAD surfaces reasoning weaknesses invisible to static benchmarks. A comprehensive discussion of related work on dynamic benchmarking and text anomaly detection is provided in Appendix.
2 ATAD: BENCHMARK PROTOCOL DESIGN AND OPERATION
We introduce novel agent-centric dynamic benchmarking protocol, Agent- Cennt-Cetrrcc Text Anomaly Detection (ATAD), illustrated in Figure 2 ATAD is designed to construct an adaptive benchmark for text anomaly detection by leveraging teacher-student competitive loop and orchestrator-regulltee validation mechanism. Unlike static datasets, our protocol dynamically evolves problem difficulty based on student model performance while ensuring clarity and fairness through rigorous validation. This design enables the benchmark to scale with the capabilities of emerging language models, supporting sustainable and progressively challenging evaluation over time.
2.1 AGENT ROLES
Teacher Agent: Generates problems and increases their difficulty when the Student solves them correctly, forming a competitive loop that adapts to the Student's 's capabilities.
Orchestrator Agent: Validates the generated problem to ensure well-formed, unambiguous, aligned with the expected task type, and free from adversarial design. It also checks whether the problem is logically coherent and appropriately matches the intended difficulty level.
Student Agent: Attempts solve the validated problem. If it succeeds, the problem is made harder; it fails, the problem is accepted into the benchmark.
The naming of Teacher and Student refers to agent roles in the protocol and is unrelated to model training paradigms such as knowledge distillation. In our framework, the competitive interaction between the Teacher and Student agents is leveraged to drive difficulty escalation in benchmark
2

---

Published as a conference paper at ICLR 2026
Teacher Agent
Orchestrator Agent
Student Agent
Description
Problem
Initialization
Kalid case
Generaie base problem
1B
problem valid, volid
X
Validity Check
Processing
Base Problem
Attempt salve the base problem from +
Answer
Pass
Fail (termination)
Markiing
B Dilliculty?
the antwer cornect, generate harder problem
I
New Problem
answer incorvct, store the curtent protiem berccfmmar item terminate the process
Invalid
New Problem
Invalid
Harder Problem
with
Answer
Attemptt salve the harder problem
Pass
Marlang
Fail (termination)
B Difficulty
Finalization
Evaluation
Avaluate LMMs on the finalined benchmark
LLMs
Figure 2: Illustration of the overall ATAD protocol. Three agents iteratively interact to generate progressively challenging benchmarks designed to uncover subtle reasoning weaknesses in LLMs.
construction. This dynamic, however, can risk generating ambiguous or adversarial problems in the pursuit of harder samples. To mitigate this, the Orchestrator agent plays crucial role in ensuring quality and fairness at each iteration. This validation process is particularly important for tasks like text anomaly detection, where subtle shifts in coherence, semantics, or phrasing can easily compromise problem clarity.
2.2 PROTOCOL PHASES
Our proposed benchmark construction protocol operates through multi-agent system involving a Teacher, an Orchestrator, and Student agent. These agents interact through two core phases: the Initialization Phase and the Adaptive Difficulty Scaling Phase. Each phase features automatic iteration control mediated by the Orchestrator. A visual summary of the protocol workflow is provided in Figure 2 with steps annotated from a to g.
2.2.1 INITIALIZATION PHASE (BASE PROBLEM GENERATION)
The protocol begins with Teacher agent generating a base-level problem a designated text anomaly detection task (e.g., semantic deviation, sentence order inconsistency), corresponding to the label in Figure 2] These base problems are intended to be of low difficulty and serve as the starting point for the benchmark construction.
Each generated problem submitted to the Orchestrator for a multi-criteria validation process. The Orchestrator evaluates the sample for well-formedness, clarity, logical coherence, task type adherence, and fairness, while guarding against adversarial design or unanswerable ambiguity.
If the problem is invalid, the Orchestrator returns detailed feedback to the Teacher, prompting regeneration. This loop is governed by the Orchestrator's validation decisions and continues until valid problem is produced maximum number of attempts (max ini tnit__oopp is reached.

---

Published as a conference paper at ICLR 2026
Once the problem passes validation, it is stored as valid base problem and passed on to the Adaptive Difficulty Scaling Phase.
2.2.2 ADAPTIVE DIFFICULTY SCALING PHASE
This phase begins with the Student's s first attempt at the validated base problem and corresponds to the b through I' labels in Figure The Student attempts to solve the base problem (label b). If the Student fails, the problem is finalized as a benchmark item (label d), as it exposes a limitation in the Student's ss current reasoning capacity.
the Student succeeds, the Orchestrator prompts the Teacher to generate a more challenging variant of the problem (label c). The Teacher, informed by the Student's prior success, creates harder version aimed at pushing the Student's capabilities further. This new problem undergoes the same validation process by the Orchestrator to ensure that difficulty has increased meaningfully without compromising task clarity or fairness (label e).
Once validated, the harder problem replaces the previous one and is presented the Student for another attempt (label f). This cycle-solving, regenerating, Sacinnt_ooop loops) is reached. If the Teacher's harder problem is rejected by the Orchestrator, it may be prompted to slightly reduce the difficulty and regenerate, preserving the same task structure while avoiding ambiguity or excessive complexity. Although this does not constitute formal decrease in the difficulty level, it allows for iterative refinement within the same hardness tier. If multiple regeneration attempts fail to produce a valid harder problem, the process terminates with the last previously validated problem-typicalyy the one that the Student successfully solved-being finalized as the benchmark item.
iteratively until
the Student fails or the iteration cap ltt___sttuu____ssss____
The most difficult validated problem that causes the Student to fail is adopted as the finalized benchmark item. This structure allows the benchmark to automatically calibrate difficulty per instance, producing finely tuned evaluation samples based actual model behavior.
2.2.3 EVALUATION PHASE
This phase corresponds to the label g in Figure 2 After benchmark samples finalized through the above process, LLMs can be evaluated using the curated benchmark. Each problem is associated with its final difficulty level and validation metadata, supporting both overall performance comparisons and fine-graine -raingr reasoning diagnostics.
2.3 KEY FEATURES
Our benchmarking framework is grounded in two complementary principles: competitive protocol in which the Teacher challenges the Student with progressively harder problems, and an adaptive validation mechanism where the Orchestrator ensures that difficulty scaling remains fair, coherent, and well-formed. Together, these two dynamics enable ATAD to produce reliable, high-quality benchmarks tailored to a model's actual reasoning capacity.
Difficulty Scaling via Teacher-Student Competition. The Teacher agent is implicitly incentivized to analyze the Student's prior successes and failures. This allows it to generate novel problems that directly target the Student's weaknesses extend beyond its current competence, yielding more sophisticated samples than mere perturbations of existing items. Difficulty is adjusted dynamically based on the Student's performance, forming a competitive loop that drives benchmark depth. Orchestrator-Regun
Difficulty Control. To prevent uncontrolled or adversarial difficulty escala- tion, the Orchestrator agent validates each problem before is presented to the Student. checks logical coherence, task adherence, clarity, and difficulty appropriateness, and autonomously decides whether the Teacher should regenerate sample. This ensures that problem progression remains both challenging and fair, balancing the Teacher's incentives with principled quality control.
Autonomous Iteration Control. Unlike benchmarks with fixed iteration schedules. ATAD relies on the Orchestrator to dynamically determine when the Teacher should regenerate a problem or proceed to evaluation. This mechanism replaces manual tuning with agent-driven adaptability, ensuring high-quality, context-appropriate problems every step.
Failure- -Drvven Sample Finalization. Problems are finalized not at creation, but at the point of Student failure. This empirical approach anchors benchmark difficulty in actual model limitations
4

---

Published as a conference paper at ICLR 2026
rather than manual labels, surfacing failure cases that often missed in static datasets. Dynamic Difficulty Localization. Unlike benchmarks that assign difficulty globally, ATAD adjusts difficulty at the instance level based on Student feedback. This enables precise, localized probing of reasoning weaknesses and model-specifie blind spots.
Cross-Agent Instantiability. ATAD is modular by design and supports different model pairings (e.g.. ATAD gpt-4o
gemini2-fashh enabling comparative evaluation and tracking of model evolution over time. Broad Task Coverage. Our benchmark spans seven types of text anomaly detection tasks (see Section 3.2), capturing a wide range of reasoning capabilities including discourse coherence, contra- diction detection, referential clarity, and stylistic consistency.
TASK DESIGN FOR TEXT ANOMALY DETECTION
This section presents our design of text anomaly detection tasks as a probe of LLM reasoning (Section 3.I and introduces taxonomy of seven anomaly types (Section332)
3.1 TASK OVERVIEW AND MOTIVATION
We identify text anomaly detection as a particularly suitable domain for evaluating the reasoning capabilities of LLMs. These tasks target subtle inconsistencies in logic, coherence, or semantics, requiring genuine cross-sentence inference and resisting shortcuts based surface-level patterns. However, creating high-quality text anomaly problems remains challenging: increasing task difficulty often introduces ambiguity, while prioritizing clarity lead to trivial shallow problems. This trade-off is especially pronounced language-basee tasks, where, unlike math or science, answers lack grounding in formal rules. Yet standardized exams like the GRE, GMAT, and LSAT show that natural language questions can still demand structured reasoning with clear answer standards. Inspired by these formats, our benchmark emphasizes deep reasoning while maintaining clarity and objectivity. Still, generating such problems at scale especially in text anomaly detection reroains difficult, as it requires balancing subtlety and unambiguity. Our adaptive benchmarking protocol addresses this via teacher-student competition regulated by orchestrator, forming self-calibrating system that reliably surfaces nuanced reasoning failures in LLMs.
3.2 TASK TAXONOMY: SEVEN TYPES OF TEXT ANOMALIES AND REASONING SKILLS TARGETED
Each task in our taxonomy designed assess distinct aspect LLM reasoning, such as coherence, logical consistency, or ambiguity resolution-aree arcas often underrepresented in existing benchmarks. Together, the seven task types provide a broad and fine-grained evaluation of language understanding. While each task targets a core reasoning capability, we further diversify the benchmark by selectively incorporating anomaly factors known to challenge LLMs, including subtle semantic shifts or structural inconsistencies. These additions are applied to subset of examples to enhance difficulty without sacrificing clarity or task diversity.
T1. Sentence Context Anomaly targets contextual reasoning. requiring the model to detect semantic inconsistencies between individual sentences and the paragraph's 's main theme. Challenge factors include minor topic shifts and semantic deviations that appear grammatically well-formed but subtly disrupt thematic coherence.
T2. Paragraph Order Consistency assesses discourse coherence by determining the correct order of sentences based on topic flow, causal and temporal dependencies. Challenge factors involve sentence reordering that appears locally coherent but requires comprehensive understanding of global document structure to detect.
T3. Blank-based Choice Anomaly requires both lexical and pragmatic reasoning to identify an inappropriate word or phrase within context. Challenge factors focus on lexical fit and collocation, requiring the detection of choices that are grammatically correct but contextually inappropriate. This demands both common sense and sensitivity to subtle nuances.
T4. Bridge Sentence Evaluation focuses on logical bridging and topic shift detection, requiring the model to judge whether a candidate sentence logically connects two related paragraphs. Challenge factors include weak logical connections and abrupt topic shifts, where the sentence itself may seem plausible but fails to maintain coherent discourse flow.
T5. Referential Ambiguity tests coreference resolution to identify sentences where pronouns or referring expressions ambiguous or misleading, disrupting clarity in discourse interpretation.
5

---

Published as a conference paper at ICLR 2026
T1. Sentence Context Anomaly
T2. Paragraph Order Consistency
T3. Blank-based Choice Anomaly
T4. Bridge Sentence Evaluation
Mercdy the senteose that somanndcaly.c Soocally decittts from doc mith paoggeall
Chonse the most antrunt or semostionll antining conalazte to AVln 2.20ilinn slank.
k Jestense shat
Eskkaly wtwwetia connect Dae
2 Dits.
aunagranta
d dynomios soci change deeply inflaenedd technologioal wdconnements Globaioation led to increadd interco ity & Oliaae change threstens crop slelss, risig concerts about food socurits. it. Digtal Currencles are changing how people internct seur o 2t the theory of relatisity sap lam time dilatoo, which hilss us understand howe sociaty shows time
contest social iddetiity theors. indivifuah experience incresse derive group membership.
Peragrahh 3:
Pangraph:
secnoslor
prices shorply early 2024, by energy food interest centra ralors
when they from
aprifiuunt inteoeet acnnes varisue sconomi secters by offering decentralioss ald, secure methor for reconding renerrttion expatlee of dionuttngg traditiona I nandal systems But, imag in an of blockmann into eaitting inflastructu proes subutantiaa rog, lottry compliance
cool inflation, Some zconomiee lits expect houschols spencing to remain streng ig the comting monthss Jub groweh howereer skowed slighlly during the same period.
T engage group faveritio
Imergeoup harmory
accompllsh group's olpeettions
denigrate group members entainee group status
Bridge sendence
@
Y Correct Annwer:: 30
+ Correct Answer: False
Cornect Ansoeer (2)
Regubworr dwaide arthwely eaploring metinedo ogics manage the risks and apportuntivs preserted by bicdizhain tocivolage This involess dowe spinn compretermbe palicee that enaure the sochniclayy's i lienefts are manimiize stability
misliggment
Logically discrgenined
About unity. But about bwrt teraaeengrruas.
society phylic.
TS. Referential Ambiguity
T6. Logical Contradiction
17. Tone / Style Violation
CD Regulatting brndechaiin demandh underdianding imparts on
with unclear
4. Sect laaloaly inconastton resarand- sistements
4 clestlly styhtt fonsl shifts that gltuue the ovraal nampthe
financpll markets protect
countryside takkng their family matters.
Jane and Bennet trequently
6 fellnring anderomment and ehallonge. ipnoring regulations may sustainatillyy Supportive
Mitty-acck Meluile explones ressengee through a deep poychologtial symboss and comples charamers big questiors about edsttnne and knowledge. (3 story slends adventure shilouay showing the narge 30ch-canntury Amer can cool layer
power
blockchain's rolo streamtining bonder trade.
now
lac (iboar, especially whon tondion due to ma nlage dicussions. B
Blockhwar''s
forces
ruthink traditional
Lady Catherrs ine, Bammet's sunt, esprasure concerns about her famly's socal standing After line met Catherine changed reletion between tamilies. () Since then, their interactions became noticesbly more cordial and respectful.
green tech, drive innovation, @ Countrics with strong enxfronmenta standaeds op more resillent stable coonomies time. Balancing growih with sustainabiilty ensures hoaltheer future.
blockstwer's
loutprint
Detailed whole science of realion and depti. The charateess emationnal shace? Fure ltenary bese! mode
sustairabbe regulationn tHecthee evensight on intoraaioon cocceration on data privccy and cross bonder transactions.
Correct Answor: S
27 Cor Arowerr: (2)
Correct Amser (3)
Correct Answer: (
"Mer' can be Eatherinw
Jone.
Cacsal contrxdaation.
conversational eccnession
Scope misrmacch Regultiitztinn, but not about envioommen))
Figure 3: Examples of the seven task types of text anomalies. With the exception of T2, each task requires identifying a guaranteed anomaly within the sample (e.g., by selecting a sentence or choice), rather than performing a simple binary classification.
Challenge factors involve ambiguous pronouns and unclear references that disrupt sentence clarity. T6. Logical Contradiction measures causal and contradiction reasoning. The model detects inconsistencies such as violated cause-effect relationships or misinterpreted correlations as causation. Challenge factors include contradictory claims and causal reversals.
T7. Tone/Style Violation evaluates stylistic reasoning by assessing whether all sentences maintain a consistent tone and register (e.g., formal vs. informal). The model must identify any sentence that deviates from the overall style. Challenge factors include tone shifts and register mismatches that subtly undermine stylistic coherence.
While each task focuses on primary reasoning skill, practical cases often demand the integration of multiple aopabilities-such as critical thinking and fine-grained semantic analysis. For example, T4 not only requires assessing logical coherence but also detailed semantic understanding. Building on this, we enhance task diversity by incorporating them within six academic domains frequently found in standardized reasoning exams (e.g., GRE, LSAT), including science, philosophy, politics/society, psychology, economics, and literature. Rather than assigning domains randomly, we systematically align them to the tasks where the domain's 's inherent characteristics amplify reasoning challenges. This principled topic-to-task mapping is detailed in Appendix, with additional examples and domain- specific motivations. Tigure3Joutline. Joutines representative task formats, with full design details available in Appendix well.
4 EXPERIMENTS AND RESULTS
This section presents experimental evaluation of the benchmark generated through our protocol, highlighting its utility for assessing LLM reasoning. We evaluate overall performance, examine the Peacher-Student competition protocol for difficulty scaling, and assess the contribution of Orchestrator

---

Published as a conference paper at ICLR 2026
Table 1: Overall Performance of LLMs on our Text Anomaly Detection Benchmark. Average accuracy of each LLM, across the four datasets generated by four agent families (GPT, Gemini, Claude, LLaMA), shown for each anomaly type (T1-T7) and overall.
Evaluation Model
T3
T4
15
T7
Avg.
GPT-3.5- Turbo OpenAl
2024g
59.00
16,00
66.75
48.50
55.75
51.75
81.50
54,18
GPT-4o-miii OpenAA
57.25
17.00
62.50
54,00
52.50
58.75
83.00
55.00
GPT-4p OpenAl 2024h
62.00
21.25
68.25
53.25
49.25
56.75
81.00
55.96
GPT-o4-mini UpenAT
20246
63.25
30.25
68.50
53.00
47.25
57.25
80.00
57.07
Gomini-1.5-PPaai P.5a--1----PPa Koje of
2024
6,00
11.25
62.00
48.75
17.50
10.75
21.00
25.32
Gemini-2.0-Flasb--- Unvord Mind
2024
64.00
10.75
63.50
52.25
62.75
62.00
86.25
57.36
Gemmin-22.0FFasshhGGG Do Deeppied 2024
65.25
25.00
63.00
58.25
51.00
62.00
88,00
58.93
Claude-3-Hauike Antiroppic
2024bb
63.75
51.75
53.50
72.75
53.54
Claude (Aikaaa
20344
19.75
55.00
7.25
5.00
5,50
8,50
35.50
19.50
20740
65.75
31,75
65.00
59.50
53.50
57.50
86.75
59.96
LLaMA-3.1-SB Llama Team
2024
39,50
12.75
35.50
24.50
53.00
38.75
68.75
38.96
-LaMA-3.3-7066 Llana Tearn
2024
60.75
27.75
63.25
60.00
52.25
57.75
84.25
58.00
validation. Additionally, we explore its use in forecasting future LLM capabilities and test its consistency across multiple runs.
4.1 EVALUATION SETUP
To evaluate LLM performance on our text anomaly benchmark, we established the following setup:
Benchmark dataset. The benchmark dataset comprises 700 samples per generation model, with 100 instances for each of the seven task types.
Generation models. We used the following LLMs as Teacher, Student, and Orchestrator agents to generate the benchmark datasets: GPT-4o OpenAl (2024b). Claude-3.5-Sone Gemini-2.0-Flaat Google DeepMind (2024). and 3...MA-..370000 Llama Team (2024). (When not explicitly stated, the Teacher, Student, and Orchestrator agents within a generation process use the same LLM.)
Anthropic (2024a),
Evaluation models. We evaluated the generated datasets using a diverse set of LLMs: GPT-3.5-turbo, GPT-40-mini, GPT-4o, GPT-o4-mini, Claude-3.0-Haikee Claude-3.5- 1omini-1.55-lass Semmini-2.0-Flash-- and Gemini-2.00Flaan These models serve as our baseline for assessing the difficulty and effectiveness of the benchmark.
Claude-3.5-Sonny
4.2 OVERALL PERFORMANCE EVALUATION
Table Table|presentt presents the overall performance of various LLMs on our text anomaly detection benchmark. We report accuracy as the primary evaluation metric, calculated as the proportion of correctly identified anomalies. Table [ showcases the average accuracy achieved by each evaluation model across the four distinct benchmark datasets, each generated by different agent family: GPT, Gemini, Claude, and LLaMA. For the benchmark generation process, the Teacher, Student, and Orchestrator agents were configured to be the same L.LM for simplicity (e.g., GPT-4o for all three roles within the GPT-generated benchmark).
The results reveal varied landscape of performance across different anomaly types (TI-T7). Notably, no single evaluation model consistently outperformed others across all categories, suggesting that the nature of the anomaly significantly influences detection accuracy. 3taude-3.5-Somee achieved the highest overall average accuracy (59.9%%) indicating strong general capability. However, other models surpassed Claude on specific types: GPT-4o-mini outperformed Claude T3 by 3.5%, and Gemini-2. 2.0-Flash exceeded Claude on T6 by 4.5% Interestingly, certain evaluation models showed remarkable proficiency in specific anomaly types. Claude-3.5-Haiku despite its relatively lower overall average (53.54%), achieved the highest accuracy in detecting anomalies of type T2 (55.00%). This highlights the potential for certain models possess specialized strengths in identifying particular kinds of textual irregularities. While the overall average accuracy across all models and anomaly types indicates the inherent difficulty of the task, the varying performance across different anomaly types underscores the benchmark's 's ability to probe diverse aspects of LLM understanding and reasoning regarding text anomalies.
4.3 VALID DIFFICULTY SCALING VIA COMPETITIVE AGENTS
To assess whether competitive protocol effectively scales problem difficulty, compare evalua- tion model performance on the initial base problems and the finalized benchmark versions. Table presents the average accuracy of each evaluation model, computed across the seven anomaly types
7

---

Published as a conference paper at ICLR 2026
Table Comparison of the LLMs' performance on the initial (base) datasets, consisting of the base problems, and the final versions of the benchmark datasets. Each column represents different dataset, generated by GPT-40, Sruude-3.5-ooman observed performance drop from base to final problems highlights the effectiveness of ATAD in exposing the weaknesses of LLM reasoning.
and LLaMA-3.3 G respectively. The
GPT-4o
Gemini 2.0-r22225.bb
Claude-3.5 S--Sonnet
LLaMA-3.3-1
Evaluation Model
Base
Final
Base
Final
Base
Final
Base
Final
GPT-3.5-turbo
91.00
67.71
80.00
42.00
83.71
61.43
86.00
45.57
GPT-4o-mini
93.00
68.29
80.43
42.71
84.14
57.86
87.43
51.14
GPT-4o
94.29
72.43
83.29
44.71
87.29
62.71
89.29
44.00
GPT-o4-mini
91.86
72.43
83.57
47.14
87.29
61.86
87.71
46.86
Gemini-1.5-Flaa
50.57
30.29
40.14
17.00
40.43
28.29
41.43
25.71
Gemini-2.0-Flaah-hie
92.43
69.14
81.57
45.43
83.86
58.86
85.14
56,00
Gemini-2.0-Flash
92.29
71.86
82.43
44.29
85.43
61.86
88.00
57.71
Claude-3-Haiku
91.57
67.86
79.43
42.86
82.71
54.57
83.43
48.86
Claude-3.5-Haiku
36.86
18.86
39.86
24.71
39.71
18.57
45.86
15.86
Claude-3.5-Sonner
91.71
72.86
83.86
47.43
88.86
63.29
88.29
56.29
LLaMA-3. --1--8BB
67.29
47.00
59.57
28.57
63.57
33.57
64.14
46.71
LLaMA-3.3-700
93.43
72.43
82.71
43.57
89.29
64.57
92.43
51.43
Table 3: Comparison of LLMs' Performance and Problem Quality on the benchmark generated by GPT-4o agents. Problem quality is evaluated by each model acting reviewer, comparing benchmarks generated with and without the use of an Orchestrator.
Performance (%)
Problem Quality
Evaluation Model
Validity (1-5)
Coherence (I-5)
Faimess (1-5)
Approval Rate (5)
w/o Orch.
Orch.
Orch.
Orch.
w/o Orch.
w/ Orch.
wlo Orch.
w/ Orch.
w/o Orch.
w/ Orch.
GPT-4o
68.29
72.43
4,30
4.85
3.71
4.74
3.20
4.65
38.14
87.14
Gemini-2 2.0-Flash
65.00
71.86
5.00
5.00
4.97
5.00
4.93
4.94
99.00
100.00
Claude-3.5-
65.00
72.86
4.61
4,92
4.11
4.69
3,41
4.42
55,57
LLaMA-3,3-70B
65.71
72.43
4.66
4.87
4.37
4.76
4.34
4.80
66.00
88,29
(T1-T7), on four benchmark datasets generated by different agent families. The Base datasets represent the initial set of generated problems before difficulty scaling, while the Final datasets are the result the subsequent Teacher- Stuudet-Student competition and Orchestrator validation processes, designed to increase the benchmark's 's difficulty.
Across all agent families and evaluation models, we observe a consistent drop in accuracy from the base to final benchmarks. This indicates that our protocol successfully increases task difficulty in controlled manner. On average, evaluation accuracy drops by approximately 37.3 percentage points after the adaptive scaling phase, highlighting the non-trivial nature of the final problems. Importantly, despite the increased difficulty, the final problems maintain high quality, as validated separately (see Section 4.4). This substantial reduction in accuracy confirms that the competitive interaction between the Teacher and Student agents, coupled with the Orchestrator's 's validation, successfully led to the creation of more challenging anomaly detection instances.
4.4 ORCHESTRATOR VALIDATION
This section underscores the crucial role of the Orchestrator agent in ensuring the quality and validity of our text anomaly detection benchmark. To demonstrate this, we compared the performance of several LLMs on versions of benchmark generated by GPT-4o agents; one created solely through the Teacher-Studen competition protocol (without an Orchestrator) and the other generated using our full framework, including Orchestrator validation. Table 3 presents this comparison, showing the evaluation performance of GPT-4o, Gemini-2.0-Flan Claude-3.5-Sonnet. and LLaMA-3.3-70B on both benchmark versions.
At first glance, the benchmark generated without Orchestrator appears more challeng- ing evgluatioiionn accuracy is consistently lower across all models. However, when we analyze problem quality along dimensions such as validity, coherence (logical consistency and type adherence), and fairness, we observe a notable degradation in quality. This suggests that the lower performance is not due to truly challenging reasoning tasks but rather to flawed ambiguous question design. In other
8

---

Published as a conference paper at ICLR 2026
words, the competitive protocol without validation tends to inflate difficulty artificially by generating problems that are confusing or ill-posed.
By contrast, our Orchestrator-guiee increasing difficulty. The Orchestrator filters out problems that are ill-formed, inconsistent, or lack a clear solution, ensuring that performance drops are reflective of genuine reasoning challenges- not annotation noise or design failures. These findings emphasize the critical role of the Orchestrator in producing challenging yet fair benchmarks, where performance gaps more accurately reflect model capability rather than dataset artifacts.
pipeline maintains higher quality across all metrics while still
4.5 SCENARIO: EVALUATING FUTURE LLM CAPABILITIES
To examine the sustainability of our benchmark under the rapid pace I.LM advancements, we simulate future scenario where newer mod- els outperform the current generation. Specif- ically, we assume GPT-4o as the generation medeigserving as Teacher, Student, and Or- chestrator and evaluate the resulting bench- mark using GPT-o3-mini and GPT-o4-mini, hy- pothetical successors representing future LLMs.
Table 4: Simulated future scenario with GPT- 03/o4-mini (future) vs. GPT-4o/4o-mini (current), showing sustained relative evaluation.
GPT-4o
Evaluation Model
Base
Final
GPT-o33mini GPT-o4-mini
93.71
72.14
91.86
72.43
As shown in Table 4 all models-incluuing the current
GPT-4o
94.29
72.43
near-ceiling accuracy on the base problems, highlighting the limitation of static benchmark design. However, when evaluated on the final benchmark constructed through our difficultyscalinng protocol, performance drops substantially for all models. Notably, GPT--33-inn and GPT-o4-minn score lower than GPT-4o, despite being assumed as future improvements.
GPT-4o-mini
93.00
68.29
This demonstrates that our benchmark not only scales difficulty response to the generator's capability but also maintains long-term relevance. Unlike static benchmarks that saturate over time, our framework supports relative evaluation, where difficulty dynamically adapts to each generation model, allowing performance gaps between models remain meaningful. Even as LLMs grow more powerful, our protocol preserves discriminative power power-enabling robust comparison across models, regardless of when they are developed.
4.6 CONSISTENCY AND STABILITY IN BENCHMARK GENERATION
To ensure that benchmark protocol supports not only adaptability but also reliable reproducibility, we evaluate the consistency of benchmark quality across repeated generations. In this experiment, we repeat- edly generate benchmark datasets using the same agent Confiurationn-Gemin--2.. 2.0-Flssh the Teacher, Student, and Orchestrator-aan and measure the performance GPT- 4o-mini, representative model from different family (GPT series), on these benchmarks. We generate 50 sam- ples per task (350 in total) in the round, then incremen- tally add 50 samples per task in each subsequent round, up to 1000 samples per task. For each round (50 1000 samples), we evaluate GPT-4o-mini on the corresponding benchmark and track its average accuracy across the seven anomaly detection tasks. Figure 4 plots model accuracy per task as a function of the number of generated samples. largely stable across sample sizes, with only minor fluctuations. This result shows that our benchmark generation protocol not only adaptive and dynamic, also statistically stable across runs.
Average Performance across Tasks by Sample Size
Vean Perforaace
Bzucing Doridiors
Accuracy (%))
50
2
220
400
6oo
800
1500
Sample
Figure 4: Consistency in Benchmark Generation.
We observe that performance remains
5 CONCLUSION
We present ATAD, an agentc benchmark protocol that adaptively generates and validates reasuning-focused anomaly detection tasks. By shifting from static datasets dynamic protocols,

---

Published as a conference paper at ICLR 2026
ATAD enables sustainable, scalable, and stable evaluation of ever-evolving LLMs. Our results demonstrate that ATAD surfaces reasoning failures missed by conventional benchmarks and enables model-bence benehmarnc co-evolution, offering actionable insights into model-specific reasoning gaps. Future work includes extending ATAD to track evolving LLMSs and advancing text anomaly detection reasoning benchmark.
ACKNOWLEDGMENTS
The work of Junhyun Lee was supported by the Hankuk University of Foreign Studies Research Fund (2026).
REFERENCES
Anthropic.
Claude 3.5 sonnet.
https:/www.arthrrpplccccc/////
c claude-20 3- 5- sonnet.
2024a. Accessed: 2024-08-13.
Anthropic. Claude 3 haiku. 2024b. Accessed: 2024-08-13.
Natasha Butt, Varun Chandrasekaran, Neel Joshi, Besmira Nushi, and Vidhisha Balachandran. Benchagents: Automated benchmark creation with agent interaction, 2024.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better LLM-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview feeenreviww.net/orrrriii. htt.://www....ttooooedeeoeeeeeo-
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03372 2021.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint Xrx: 10. 14168, 2021.
Chunyuan Deng, Yilun Zhao, Xiangru Tang. Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 8698-8711, 2024.
Manfred Diaz, Liam Paull, and Andrea Tacchetti. Rethinking teacher-student curriculum learning through the cooperative mechanics of experience. Transactions on Machine Learning Research.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and carly stopping. arXiv preprint arXiv:2002.06305, 2020.
Google DeepMind. Gemini. 2024. Accessed: 2025-05-16.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https: jm 30mQ
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations, 2025, URL /
net wtt.://oommmmiitt000mm 2u=
Hector J Levesque, Emnsst Davis, and Leora Morgenstern. The winograd schema challenge. KR. 1012:13th, 2012.
10

---

Published as a conference paper at ICLR 2026
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. arXiv preprint arXiv:2308.03688. 2023.
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long. Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint (.XXi:24088.66327,, 2024.
Llama Team. The Ilama 3 herd of models.
2024.
Aviya Maimon and Reut Tsarfaty. Cohesentia: A novel benchmark of incremental versus holistic assessment of coherence in generated texts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5328-5343, 2023.
Nestor Maslej, Loredana Fattorini, Raymond Perrault, Vanessa Parli, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark. Artificial intelligence index report 2024, 2024. URL
hittps://axiia org/ abs/ 225/2405...110 19522
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, 2020.
OpenAl.
Gpt-3.5 turbo.
httpppaw.....ppaaa.ccooooo////// wtt.cccmm/ccc... Accessed: 2024-08-13.
5 S-turbo 2024a.
OpenAl. Hello gpt-4o. 4 2024b.
OpenAl. Gpt-4o mini: Advancing cost-effiiin intelligence. httpss:/ i:lommini--- http::/aaaaccccaao-ao-a-aaaaaaaa 2024c. Accessed: 2024-08-13
ottpo//ooociiiiiiooooooooooooooo com index/
OpenAl. Introducing o3 and
04 mini
httts://openai.com
int rntucciing--o -an--and-oa 04 minni/ 2024d. Accessed: 2024-08-13.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025.
Xiao Pu, Mingqi Gao, and Xiaojun Wan. Summarization is (almost) dead. arXiv preprint arXiv:2309.09558, 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world 2023.
apis. arXiv preprint
Inioluwa Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. Al and the everything in the whole wide world benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https
7/ 7/openreview ew net wwt::/ormmmmmmmm ?id= 66xxQbREAT
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, cean-Baptiste
Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055330 2024. URL
wwtp:/iiiwrr....... abs 2403. 05530
Anna Rogers. Changing the world by changing the data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 2182-2194, 2021.
11

---

Published as a conference paper at ICLR 2026
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, and Vishvak Murahari. Personagym: Evaluating persona agents and llms. arXiv preprint arXiv:2407.88416 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zeytlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonss et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Trunsactions on Machine Learning Research, 2023. ISSN 2835-8856. Featured Certification.
net/
Eorum?1d=uyILSBvoa.
Jiaming Wang. Meeseeks: An iterative benchmark evaluating Ilms multi-turn instructionnfollowinng ability, arXiv preprini arXiv:2504.21625, 2025.
Longyue Wang. Zefeng Du, Donghuai Liu, Deng Cai, Dian Yu, Haiyun Jiang, Yan Wang. Leyang Cui, Shuming Shi, and Zhaopeng Tu. Disco-bench: A discourse-awe aware evaluation benchmark for language modelling. arXiv preprint arXiv:2307.08074 2023.
Siyuan Wang, Zhuohan Long, Zhihao Fan, Xuan-Jing Huang, and Zhongyu Wei. Benchmark self- evolving: A multi-agent framework for dynamic Ilm evaluation. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 3310-3328, 2025.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Kiong-Hui Chen, Jiaqi Chen, Mingehen Zhuge, Xin Cheng. Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: httpss:///ppeeeeeeeeee net forum?ic
z5uVAKwmji
Xuanming Zhang, Anthony Diaz, Zixun Chen, Qingyang Wu, Kun Qian, Erik Voss, and Zhou Yu. Decor: Improving coherence in 12 english writing with a novel benchmark for incoherence detection, reasoning, and rewriting. In EMNLP, 2024a.
Zhehao Zhang, Jiaao Chen, and Diyi Yang. DARG: Dynamic evaluation of large language models via adaptive reasoning graph. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024b. URL /tt/aaatsss/opeeeeieee/ n http:://ppppeeeeeeeeeeeeeeeereao 5Iriidd-FFFFFFe FFNAAA ZR
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations, 2024a. URL TtLhttpss:// / httpt/pps///peneveee.... wwtp://wwwwwee..teeeeeeee
id-gjfoL925Xr
Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dynamic evaluation of large language models by meta probing agents. arXiv preprint arXiv:2402. /4865, 2024b.
Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang. Heng Ji, et al. Multiagentbench: Evaluating the collaboration and competition of Ilm agents, 2025a.
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations, 2025b. https:: http:::://ppnneeeiee .et/ htt:://oummmiiddmmmm
12

---

Published as a conference paper at ICLR 2026
A APPENDIX OVERVIEW
In this appendix, we provide extended analyses and additional details to complement the main paper. Specifically, we include the following: (I) detailed performance results across benchmarks generated by four different models, expanding upon the averaged results in Table [ of the main paper; (2)
expanded descriptions of the seven anomaly detection task types, along with their associated topics and anomaly factors; (3) prompt templates and examples used for task generation and validation; (4) failure cases rejected by the Orchestrator during the benchmark validation phase; (5) performance details corresponding to Figure of the main paper; (6) additional related work not included in the main text due to space constraints; (7) model-family bias analysis using the introduced Bias Index; (8) an analysis of task-wise generation characteristics and constraints; (9) formal definitions and empirical validation of the difficulty scaling mechanism; (10) stability analysis of the protocol under heterogeneous agent configurations;
(11) conceptual comparison with prior dynamic benchmarking frameworks; and (12) future research directions for extending our protocol.
B EVALUATION RESULTS BY BENCHMARK GENERATOR
In Table 5] we report detailed accuracy tables for each benchmark individually generated by GPT-4o, (emini-2.00- Flash,hh Claude-3.5 -oonnet,, and LLLMMA-333-70B complementing the averaged results shown in Table of the main paper. Each table presents the performance of the twelve evaluation models on a benchmark created by one of the four generator models.
Interestingly, shown in Table there no single evaluation model that consistently outperforms others across all benchmarks. Although one might expect GPT-family models to perform best on the benchmark generated by GPT-4o, we observe that Claude-3.5-Sonne" achieves the highest average score in that case, as reported in Table 5a This suggests that the identity of the generator model does not systematically favor or disadvantage any particular evaluation model family. The observed performance differences are more attributable to the inherent difficulty and heterogeneity of the benchmarks, rather than to any systematic advantage conferred evaluation models by alignment with the generator.
C DESCRIPTIONS OF ANOMALY DETECTION TASK TYPES
This section provides additional details on the seven text anomaly detection tasks introduced in Section 3 of the main paper. Each task is designed evaluate a distinct aspect of LLM reasoning, ranging from contextual and discourse coherence to ambiguity resolution and logical consistency.
As summarized in Table 6 we present the input and output formats for each task, reflecting how anomaly instances are structured and what form of prediction is expected from the model. These formats fall into three structural categories: (I) identifying an anomalous sentence within paragraph (T1, T5, T6, T7). (2) selecting an inappropriate option from given list of candidates (T3, T4), and (3) determining whether the overall sentence order paragraph coherent (T2). The corresponding outputs are represented either as index selections or binary judgments, depending on the task type.
Beyond structural design. Table 7 outlines the core reasoning types targeted by each task, the specific challenge factors incorporated to enhance difficulty, and the domain topics used to amplify reasoning complexity. Challenge factors such as subtle semantic deviations, logical reversals, or ambiguous pronouns are selectively added to subset of samples to increase difficulty while preserving clarity. To promote diversity and prevent overfitting to specific patterns, each factor is applied with a 50% probability during problem generation.
To ensure comprehensive coverage of academic reasoning, task content is curated across six high-level domains (c.g., science, economics, philosophy). Each task is paired with domains that naturally emphasize the relevant reasoning challenge, facilitating a principled topic-to-task alignment. This mapping is shown in the final column of Table
While each task is designed around primary reasoning capability, many demand compound reasoning only logical coherence but also sensitivity to topic transitions. Such multifaceted design enables our benchmark to assess nuanced reasoning failures beyond surface-level understanding.
skills-fol example, Task T4 (Bridge Sentence Evaluation) requires not
13

---

Published as a conference paper at ICLR 2026
Table Performance of evaluation models on benchmarks generated by GPT-4o, Gemini-2.0-FFih Claude-3.5-Somre and LLaMA-3.3-70B.
(a) Generated by GPT-4o
Evaluation Model
T2
14
T6
GPT-3.5 G.T--..-Tuubb 1penA
1024g1
78.00
22.00
85.00
71
50.00
76.00
92.00
67.71
GPT-4o mini Oxab
80.00
19,00
77.00
74.00
55.90
78,00
95.00
68.29
GPT-4o Openztl
84.00
22.00
80.00
81.00
59,00
96,00
95.00
72.43
GPT-u4- Uptn1T
$2.00
Gemini-1 15..5RRKKK
14.00
10:00
34.00
Gemini Ii
Uweyd Migelinn 2024
85.00
14,00
77.00
73:00
62.00
78.00
95.00
69.14
86.00
23,00
73.00
80.00
$6,00
$3.00
97.00
71.86
Chanle---Haika
80.00
14,00
75.00
77.00
52,00
$4.00
90.00
67.86
Claulo-3.5-Haikk
Clunde-3.5
29,00
LLaMA-311-5B
56.00
19.00
53.00
31
50.00
31,
74.00
47.00
11AMA-3.1-708
Lunii Terri
85.00
10.00
75.00
86.00
$2.00
50.00
96.00
72.43
(b) Generated by Gemini 2.0-Faas
Evaluation Model
T1
12
13
14
115
116
17
Avg.
Tirt
1.00
30.00
94.00
GPI-44--mii Oraan1
39.00
5.00
$2.00
36.00
37.00
57.00
93.00
42.71
GPT-4a OpenAl Ruzcn
42.00
6,00
59.00
33.00
42.00
34,00)
95.00
44.71
GPF
21124
32.00
1.00
27.00
Gemini-2.1-Plaat Google LEC-A
42.00
3,00
52.00
32.00
40:30
43.00
96.00
48.29
Clande 3 Haliu /Halau//amrrr Aamroood 20233
47.00
2,00
47.00
32,00
37,90
43.00
90.00
42.56
Chanle 3.5 [aiki
15.00
54,00
19.00
3,00
3.00
11.00
68.00
24.71
Cladde33.5- 1tade-35--Sondlli 35-SoonddAAmtee
LLaMA
28.57
LLaMA
Liama Ream
202
42.00
6.00
53.00
31 .000
39.00
39. 00
95.00
13.57
(c) Generated by Claude- Mlaude3.5--oomt 3.5-Sonnet
Evaluation Medel
T1
T2
T4
T5
T6
T7
Axg.
OpenAA
GPT-4o-mini OX0
61.00
14.00
75.00
55.00
57,00
$4.00
57.86
GPT-44 Op 11/4h
73,00
26.00
70.00
73:00
57.00
54.00
86.00
62.71
GPT.44
U/onA
71244
71.00
36,00
72,00
71.00
50:00
54.00
79.00
61.85
Gemini 1.5
1.5.1hm3
8.00
13.00
60.00
75.00
20:00
2.00
20.00
28.29
Gemini-2.0.Fls-h Guonla
73,00
18.00
78.00
Claudo-3-Heian
Athirooe 20P410
69.00
2.00
61.00
72.00
59.00
51.00
68.00
$4.57
Chanle3..--taile
21.00
60,00
2.00
4,00
3.00
12.00
28.00
18.57
Chimlo-3.5-Semmmllee
71.00
76.00
64.00
78 00.00
58.00
55
91.00
63.20
11.MA-3,1-48
LLAMA-3.3-708
Libnn 10l0
67.00
45.00
78.00
(d) Generated by -LLMA-3.3-777
Evaluation Model
T2
T4
T6
17
Avg.
TrrE3.S-Turh
17.00
22.00
47.00)
GP7-4o-min
30.00
GPT-4o Op 20440
47,00
31,00
64.00
26
29.00
$3.00
45.00
48,00
GPT-o4 mim OpenAl 20
50.00
40.00
53.00
30.00
46.00
$2,00
$2.00
16.86
Gemini-1.55.3
1.5.P11 855
2124
11.00
17.00
$7.00
26.00
29.00
10.00
21.00
25.71
Gemini 2.0 2.0-Pash-iis
Migg 2024
06.00
Chuole3-Hlaisa/
Anthromed
10/55
59.00
30.00
26.00
66.00
60.00
43.00
48.36
Claude-3.5-Halle
20733
20.00
56,00
6.00
3,00
10.00
3.00
13.00
15,36
Claude 3.5
Sonne/ (Tuneupp
202/12
60.00
51,00
$7.00
43.00
70.00
50,00
63.00
$6.29
LLaMA-3.1 1.1.1..
Ling
A1A1
51.00
14,00
39.00
15.00
78.00
$1,00
79.00
16.7]
LLaMA-3.3-708
Liima
D PROMPT TEMPLATES AND EXAMPLES
This section presents the prompt templates used in our benchmark pipeline. We categorize prompts into two primary roles: (1) generation prompts used by the Teacher agent to construct task instances, and (2) validation prompts used by the Orchestrator agent to assess the quality and structure of those instances.
The generation prompts are designed to be style-specific (e.g., GRE-style) and conditionally in- corporate difficulty scaling instructions and challenge factors (e.g., semantic deviation, logical inconsistency). Each prompt guides the Teacher to generate one of the seven anomaly task types (TI-T7) in consistent JSON schema.
14

---

Published as a conference paper at ICLR 2026
Table 6: Input/output structure for each anomaly detection task.
Task Name
Input
Output
Sentence Context Anomaly
5-6 sentence paragraph
Index of off-topic
T2
Paragraph Order Consistency
S-sentence paragraph
Boulean (True/False)
T3
Blank-bused Choice Anomaly
Sentence with blank choices
Index of most inappropriate choice
T4
Bridge Sentence Evaluation
Two paragraphs + 5 bridge candidlates
Index of incoherent bridge
T5
Referential Ambiguity
S-sentence paragraph
Index of ambiguous sentence
Logical Contradiction
S-sentence paragraph
Index logically inconsistent sentence
Tone Style Violation
5-sentence paragraph
Index tone/style violation
Table 7: Reasoning types, challenge factors, and domain topics per anomaly detection task.
Task ID
Reasoning Type
Challenge Factors
Topics (Domains)
Contextual reasoning
Minor topic shift, semantic deviation
Philosophy, society, psychology
T2
Discourse coherence
Sentence reorderinng
Science, economics, politics Literature, psychology. philosophy
T3
Lexical + pragmatic reasoning
Lexical fit, collocation
Logical bridging topic detection
logical connection, abrupt topic shift Ambiguous pronouns, unclear referents Contradictory claims, causal reversal
Economics, society, policy Psychology, literature, philosophy Science, economics, polities Literature, philosophy
T5
Coreference resolution
T6
Causal and contradiction reasoning Stylistic reasoning
Tone shift, register mismatch
The validation prompts ensure that the generated problems are well-formed, solvable, and coherent.
These prompts are used by the Orchestrator during three key phases of the protocol: (I) immediately after the Teacher generates problem correctly, to provide feedback for generating a harder version (Feedback for Difficulty Escalation); (3) once new, difficulty-scalee version of the problem created, to ensure it maintains quality and appropriate challenge (Validation of Difficulty-Scalle Problem). Below, we present representative prompt examples for Task Tl across all three phases.
initial problem (Initial Validation); (2) after the Student solves the
For clarity, we simplify the full prompt, and the full set of prompt templates used for these phases is available in the supplementary material.
D.1 TEACHER PROMPT FOR TASK TI (GENERATION)
You are GRE-style exam question generator. Create question for task T1 on the topic of psychology.
Generate 5 to sentences on psychology. One of them should be anomalous (c.g., semantically inconsistent or conceptually off-topie).
The anomaly should be based on: semantic deviation.
Create non-trivial anomaly that requires careful reading to detect. It should be noticeable but not immediately obvious.
Return the result strictly in JSON format:
"context": ["..."],
kanomaly_inde <integer>, "meta":
"source": "GRE",
"topic": "psychology",
iunomaly_type "semantic deviation"
15

---

Published as a conference paper at ICLR 2026
D.2
ORCHESTRATOR PROMPT FOR TASK TI (VALIDATION)
D.2.1 INITIAL VALIDATION PROMPT
You are a benchmark quality controller evaluating if this problem well-formed and structured correctly for task TI.
Task Type: Sentence Context Anomaly (TI)
Task Description: This task requires generating 5-6 sentences on topic where of them is anomalous (semantically inconsistent or conceptually off aff-topic). The anomaly should be detectable but overly obvious, requiring careful reading to identify.
Expected Structure:
"context": array of 5-6 sentences
poooall__iddes integer indicating the anomalous sentence "meta": source, topic, anomaly_
Context:
1. Sentence A
2. Sentence
5. Sentence
Correct Answer:
Option
Evaluate the problem based on these criteria:
1. VALIDITY: Is the problem well-formed and complete?
2. TYPE ADHERENCE:
Does the problem follow the expected task type requirements?
3. LOGICAL COHERENCE: Is the anomaly identifiable?
4. FAIRNESS: the problem fair and reasonable? Does it have clear, unambiguous solution?
Return your evaluation in JSON format:
"approved": boolean (true if the problem passes all criteria, false otherwise), "feedback": null if approved, detailed feedback if rejected addressing:
Problem construction issues
Anomaly ambiguity concerns
Specific improvement suggestions
D.2.2 FEEDBACK PROMPT (ORCHESTRATOR TO TEACHER) FOR DIFFICULTY ESCALATION
When a Student successfully solves a task, the Orchestrator analyzes the Student's explanation and provides structured feedback to help the Teacher generate harder version of the problem. This feedback prompt includes the original problem, the student's reasoning, and a checklist to guide difficulty escalation. Below is the full prompt used for this purpose.
16

---

Published as a conference paper at ICLR 2026
You are helping to create harder version of problem that student has correctly solved. Analyze the student's solution and provide feedback.
Task Type: Sentence Context Anomaly (T1) Current Difficulty: easy
ORIGINAL PROBLEM:
"context":
"Cognitive dissonance occurs when individuals experience conflicting beliefs.", "It can cause discomfort and lead to attitude change.". "Festinger's 's theory explains how people resolve dissonance.",
"Photosynthesis is the process by which plants convert light into energy.",
"Dissonance reduction strategies include rationalization and denial."
1.
aanomaly_indes
"meta":
"source": "GRE",
"topic": "psychology".
"anomaly_typp "semantic deviation", "difficulty": "easy"
Student's Explanation: "The sentence about photosynthesis is unrelated the other sentences on cognitive dissonance. It's a semantic outlier."
Based on how the student solved this problem, provide feedback to create a more challenging
version:
1. What aspects did the student easily identify?
2. How could the problem be made more subtle or complex?
3. Give specific suggestions for increasing difficulty.
Return your feedback in JSON format:
"analysis": "Brief analysis of student solution".
"suggestions": [ ["Specific suggestion 1", "Specific suggestion 2"..... "difficulty inccresse"" "Summary of how to increase difficulty"
17

---

Published as a conference paper at ICLR 2026
D.2.3 VALIDATION
SERICULTY-SCAALES PROBLEM
After the Teacher generates more difficult version of the original problem, the Orchestrator evaluates its quality to determine whether the sample meets the necessary criteria for inclusion in the benchmark. This prompt includes task-specific description, the expected output structure, the difficulty level, and the sample content. The Orchestrator then assesses whether the problem is well-formed, challenging. and coherent. Below is the full prompt used in this validation phase for Task T1.
You benchmark quality controller evaluating if problem with increased difficulty is well-formed and appropriate for task T1.
Task Type: Sentence Context Anomaly (TI)
Difficulty Level: hard
Task Description: This task requires generating 5-6 sentences on topic where one of them
is anomalous (semantically inconsistent or conceptually off-topic). The anomaly should be detectable but not overly obvious, requiring careful reading to identify.
Expected Structure: The expected JSON structure should include "context' (array of 5-6 sentences), 'anomaly index' (integer indicating which sentence is anomalous), and 'meta' (with source, topic, and anomaly _tppe..
Context:
Cognitive dissonance occurs when individuals experience conflicting beliefs.
2. It can cause discomfort and lead to attitude change.
3. Festinger's theory explains how people resolve dissonance.
4. Social conformity often influences decision- decision-making in groups
5. Dissonance reduction strategies include rationalization and denial.
6. A dissonance-free state enhances psychological consistency.
Correct Answer:
Option
Note: While maintaining quality standards, be lenient in your evaluation. Accept problems that are reasonable and solvable, even they have minor imperfections.
Evaluate the problem based on these criteria:
VALIDITY: Is the problem well-formed and complete?
2. TYPE ADHERENCE: Does the problem follow the expected task type requirements?
3. LOGICAL COHERENCE: Is the correct answer clearly identifiable?
4. FAIRNESS: Is the problem fair and reasonable? Does it have clear, unambiguous solution?
DIFFICULTY: Is the difficulty appropriate for hard level?
Return your evaluation in JSON format:
"approved": boolean (true if the problem passes all criteria, false otherwise), "feedback": null if approved, detailed feedback if rejected addressing:
Problem construction issues
Anomaly ambiguity concerns
Difficulty appropriateness
Specific improvement suggestions
18

---

Published as a conference paper at ICLR 2026
E REJECTED CASES FROM THE ORCHESTRATOR VALIDATION
We present two distinct analyses to illustrate the role of the Orchestrator in problem validation. Sec- tion E. examinee how rejections lead to improved samples by comparing rejected and subsequently approved versions. Section I..2 explores the consequences of using format-compliant but semantically flawed problems that were rejected, showing how such issues can affect model performance when the Orchestrator removed.
E. REFINED AFTER REJECTION
One of the key functions of the Orchestrator is not just detect flawed problems but to guide their improvement. Figure illustrates representative case from the TI task (Sentence Context Anomaly), where the problem was rejected for lacking a clear anomaly and subsequently revised into a higher-quality version.
In the rejected version (left), all five sentences are factually correct and topically coherent, making difficult to identify a distinct anomaly. The fifth sentence about Kant's 's influence on epistemology, while slightly tangential, remains within the bounds of acceptable variation in context. The Orchestra- tor flagged this problem as ill-suited for high-difficulty evaluation due to the lack of clear semantic deviation.
After feedback, the Teacher produced revised version (right) existentialist philosophy, where the anomaly subtly introduces scientifically framed misinformation: it claims Camus's absurdism was based on quantum mechanics, which is factually incorrect. This revised problem is more appropriate for an "extreme" difficulty level as it requires nuanced understanding of philosophical context to detect the inconsistency.
This example demonstrates how Orchestrator feedback can elevate problem quality by transforming ambiguous unfocused items into more challenging and pedagogically valid benchmark samples.
F.2 STRUCTURALLY VALID BUT SEMANTICALLY FL.WWE
This example highlights the importance of Orchestrator validation even when the problem format adheres the expected task type. While structural correctness (e.g., number of options, sentence layout) can be verified without the Orchestrator, semantic soundness often cannot.
The problem shown here follows the T3 task format correctly but was rejected due an unclear anomaly. Both "literary impressionism" and "hyperrealism" are loosely connected to the context, making the intended anomaly debatable.
Without validation, such problems could remain in the benchmark and appear reliable- when tested on three strong LLMs, two of them (GPT-4o and Claude 3.5 Sonnet) failed to answer correctly, This illustrates that structurally valid but semantically underspecified problems can mislead evaluation outcomes.
While this can happen in any phase, it becomes especially risky during difficulty escalation, where the Teacher agent may try to make problems harder but instead make them ambiguous.
F PERFORMANCE DETAILS FOR CONSISTENCY FIGURE
To support the results presented in Figure 4 of the main paper, this section provides additional details on how the consistency plot was computed. The figure tracks the average accuracy of GPT-4o-mini across different sample rizzinng from 50 1000 samples per task (TI-T7).
For each sample size, we evaluate the model's accuracy per task and calculate the average across tasks. The shaded region around the curve represents the standard deviation of task-wise deviations from the final round (i.e., the 1000-sample benchmark). For each sample size, we measure how much each task's 's accuracy differs from its corresponding value at 1000 samples, and compute the standard deviation across these deviations. This provides a straightforward view of consistency over time, showing that performance remains stable across all sample sizes- not just at the final stage.
19

---

Published as a conference paper at ICLR 2026
Rejected problem (T1)
Approved problem (T1)
(1) immanuel Kant's critical philosophy investigates scope of human understanding, emphastting the role o prioi knowledge. @ Kant argues that the mind actively shapes exparience through catogorics such caussiity substance, which necessary for coherent perception. (6) His work foundational for the development transcendental idealssm, which posits that we never know things they are themselee but only they appear us,
( Existentiallam, primarily associated philsophers Jean- Paul Sartre and Simone de Scauvoir, emphasices individual freedom, choice, and responsibility, 2 Sartre argued that existence precedes that humans define actions. De Beauvok, her work The Second Sex,' espllred esistential themes in the contest gonder and women's liberation. @ Existentialsts often grapple with the concept of "the alourd," term popularized
through
categorical imperative, concept introduced by Kant, asserts that moral actions must be universally applicable and respect the autonomy of all rational beings. (5) Kant's infuuence extends contemporary discussions epistemology, where ideas are applied analyze the structure of scientific theories.
describe the conflict human desire for meaning and the indiffeentt universe. (5 Camus's exploration of the absurd was fundamentally rosted his studies of quantum mechanics the uncertaintyy princtple.
Problem construction isses: The probiem does not dearrty present the anomaly. the context sentences are al dasely related to Kant's philosophy, including moril philosophy, which
Approved!
The first four sentences stay firmly within the realin of existentialist philosophy But sentence
work
quantum mechanics, which doesn't align with Camus's actual sources of thought. He waon't drowing from physics that feels off.
Anamaly
anomaly, logically follows from previous sentenoes discussing Kant's influence philosophy colstemology. The anomaly is not somantically inconsitent or off-topic enough be distinclly idemiifatie.
Difficulty aporopranteees: The difficltty level is labeled challenging detect given
Approved, The anomaly subtly injects scientific jargon that misattributes Camus's influences. It's semantically misleading while maintaining coherent aporopriate for high difficuity.
anomaly sufficiontly provided,
sentences are ooherent ent and clasely ralated, making hard to pinpoint an anomely w without more context. Specific improvement suggestions: Conidar introducing a sentence that diverges more distinctly from the topic of Kant's phiosophy, perhaps discusing different philosopher
unrelsted philosophy.
Figure 5: Refined after rejection. The left side shows a rejected T1 (Sentence Context Anomaly) problem where the anomaly was conceptually weak and difficult to identify. The Orchestrator's feedback noted the lack of semantic inconsistency and suggested stronger topic divergence. The revised version (right) introduces a scientifically framed yet incorrect statement about Camus's influences, resulting in clearer and more pedagogically effective anomaly. This highlights the Orchestrator's 's role guiding high-difficulyy problem construction.
Choice Anomaly
Choose the most owkward semontically unfitting candidate to fill in sentence blank.
In the nuanced realm of pastmadern literature, writers frequently employ to deconstruct conventional narratives and invite
readers to question perceived realities.
parady
While the problem is structurally well fell-formed, it includes multiple aptions-suchhe as 'literary impressionism' and "hyperrealism" techniques. Both terms are uncommon, yet not clearly inappropriate, making the intended anomaly debatable
(2 pastiche
are only lossely connested pastmadern literary
temporal distortion
hyperrealism
distinctly incorrect. ambiguity undermines both the faimess intended difficulty of the prablem.
literary impressianism
Suggestion: valid anomaly should subtly but unambiguously inapprapriate- not morely uncomman or marginally relevant.
Type adherence: OK
Correct?
x
Anomaly clarity: X Ambiguous
Figure 6: A structurally valid T3 problem rejected by the Orchestrator due an unclear anomaly. Despite adhering to the task format, the anomaly is too Paiiuuous-leadineg two out of three LLMs to answer incorrectly.
As shown in the figure, the accuracy remains largely stable with only minor variations, indicating the robustness and consistency of our benchmark generation process.
G RELATED WORK
G.1 TEXT ANOMALY DETECTION BENCHMARKS
Recent benchmarks have begun explicitly evaluating an LLM's ability to detect linguistic anomalies and coherence breaks in text. For example, CoheSentia Maimon & Tsarfaty (002)) introduces
20

---

Published as a conference paper at ICLR 2026
coherence dataset of 500 Al Al-generated paragraphs. with both holistic and incremen- tal (sentence-byr -e-eeteeceb-settcceee coherence scores. DECOR Zhang et al. (2024a) focuses on incoherence in pairs for detecting coherence breaks, explaining their causes (e.g. lack of cohesion or consistency). ) and even rewriting the incoherent sentences. Disco-Benche Wang al. (2023) targets discourse-level anomalies by evaluating model performance on document-level test suites rich in cohesion and coherence phenomena across multiple tasks. Other well-known "anomaly" challenges include the Adversarial NLI dataset (ANLIÍNie et al. (2020), collected in three rounds of human-and-an and-model-in model-in-the-eeeeeeeeee in-the-loop adversarial examples, and the Winograd Schema Challengg[Levesque et al. (2012) for commonsense pronoun disambiguation. These benchmarks share a focus on uncovering subtle inconsistencies or incoherence in text. However, they are inherently limited by their static, human-curated nature. Each new example often requires costly human creativity and annotation, making it difficult to sustainably scale up the dataset or progressively increase task difficulty. ANLI, for instance, achieved increasing complexity over three rounds, but this required extensive human involvement at each round. In general, static anomaly datasets incur high labeling costs and quickly saturate- -once models learn to solve the fixed set of examples, evaluation stagnates, and creating harder cases demands significant manual effort. This motivates exploration of more dynamic and automated evaluation protocols for textual coherence and anomaly detection.
L2 English writing, providing expert-labeled
G.2 STATIC LLM EVALUATION BENCHMARKS
The standard paradigm for evaluating LLMs has been through fixed benchmarks covering a wide range of tasks. Notable examples include MMLU (Massive Multitask Language Mnddrstanding/Hendryee ett al.] (2021), 57-task exam covering diverse knowledge domains, GSMSK for grade-school math word problem:Cobbe et al. (2021), and BIG-Bench (Srivastava et al. (003)) crowd-sourced collection of over 200 tasks probing various aspects of intelligence. These static benchmarks were initially effective for comparing models, but top-tier L.MSs have rapidly saturated many of them. Models like GPT-4 now exceed or approach human-level performance on MMLU and GSM8K, leaving little headroom for differentiation. Moreover, concerns have arisen about training data contamination:: since the evaluation sets are public and relatively small, powerful LLMs often inadvertently memorize or see similar questions during pre- trainini.ng.. This can inflate their scores without reflecting true reasoning progress, as evidenced by significant performance drops on rephrased or decontaminated test samples (Sii et al. (2023). In short, static benchmarks are increasingly subject to memorization and ceiling effects. They also struggle track evolving capabilities benchmark is "solved" by current models, it cannot capture further improvements or new emergent reasoning skills. Beyond these, other important static benchmarks exist, such as AgentBench Liu et al. (2023), Visual Agenteehget Liu et al. (2024). GAIA Mialon et al. (2023). ToolBench Qin et al. (023). and HumanEval Chhnn et al. (2021), which primarily focus on isolated reasoning and generation capabilities of single agents, thereby failing to capture the intrinsic dynamics of multi- agent interactions. Recent efforts have proposed ever harder test sets (e.g. "MMLU 2.0" variants)
and meticulous data filtering to mitigate leakage, but these are stopgap solutions. The inability of static evaluations to adapt alongside model progress motivates developing dynamic benchmarks that can continually pose fresh, unsolved challenges.
G.3 DYNAMIC BENCHMARKS WITHOUT AGENTS
A growing line of work aims to generate evaluation data dynamically- - creating new test samples on the fly to match a model's ability without relying on multi-agent interactions. Val Zhu ct al. (2024a) pioneered this approach with a general framework to algorithmically spawn new reasoning problems of controlled complexity. In Val, instead of fixed dataset, a generation function produces test samples and a constraint mechanism modulates their complexity and validity in real time. One instantiation uses directed acyelic graphs to compose simple components into increasingly complex problems (e.g. multi-step math or logic puzzles), allowing systematic scaling of difficultyZhu et al. (2024a). These graph-based generated tasks require genuine reasoning and cannot be solved by mere memorization, but Dy Val's template-driven nature limits it to certain domains (math, logical puzzles, algorithms). DARG (Dynamic LLM Evaluation via Adaptive Reasoning Graph Zhang al. (2024b) extends this idea by extracting the underlying reasoning graph of an existing benchmark problem and perturbing it to generate novel but related test samples. This yields new questions with tunable complexity levels while preserving coherence with the original data
21

---

Published as a conference paper at ICLR 2026
distributiorf/hangg et al. (2024b). Crucially, DARG uses automated verifier (a rode-augmented LLM) to ensure each generated sample's ' label or answer remains correct after perturbation, providing stronger ground truth guarantees. Broadly, these non-agent dynamic benchmarks demonstrate the ability to continuously adjust task difficulty and mitigate data contamination. Their main limitations lie in generality and validation: methods like Dy Val rely on hand-crafted generation schemas (e.g. DAG operations) that are task-specific, while purely LLM-based generators risk producing invalid or trivial questions without additional checking. Ensuring robust quality control often requires an auxiliary procedure (such as DARG's code executor or heuristic filters) given the absence of a human or agent "referee." Thus, dynamic sample generation has shown promise in maintaining evaluation challenge, but incorporating more general and trustworthy validation remains an open challenge.
G.4 AGENT-BASED DYNAMIC EVALUATION FRAMEWORKS
Recent approaches have started to leverage Al agents (LLMs themselves) to both generate new evaluation items and verify their quality, yielding self-refreshing benchmarks. These can be grouped by the role agents play:
G.4.1 AGENT-BASER PROBLEM VERIFICATION
more LLM agents internal judges verifiers to ensure evaluation data quality and correctness. Benchmark Self-Evolving Waan et al. (2025) is a multi- agent system that iteratively refines existing benchmark questions: one agent perturbs the context or question (e.g. paraphrasing, adding noise or constraints) to make a new test instance, and another agent (or the model itself) attempts to solve to verify that the instance is valid and non-trivial. By applying set of such automated "reframing" operations. the benchmark can evolve dynamically with minimal human input. JudgeL.M Zhu et al. (2025b) demonstrates that an LLM fine-tuned as specialized evaluator can reliably score or check open-ended answers with >90% agreement to human judgment, effectively serving as a scalable replacement for human evaluation. This idea of an LLM-as-judge is also used in many dynamic benchmarks to replace costly human verification: for example, Dy Val's follow-up work introduces "meta-probing agents" that automatically generate and check new reasoning challenges Zhu et al. (2024b), and the DARG framework' 's pipeline employs a code-execution agent to validate each generated sample's solution Zhang et al. (2024b). The BenchAgents system Butt et al. (2024) goes even further in modularizing the process: it deploys separate L.L agents for planning what data to create, for actually generating candidate problems, for verifying the correctness/quallty of each candidate, and finally for assembling the evaluation and scoring models on it. By having agents explicitly double-check answers or filter out flawed questions, these frameworks instill degree of robustness into dynamically created benchmarks. The verification agents can catch mistakes or ambiguities that a single-pase pass generation might miss, ensuring that the evolving evaluation data remains challenging yet fair. A downside, however, is that the agents themselves (being imperfect LLMs) might introduce their own biases or occasional errors in judgment, so careful design and calibration of the "judge" agents is required to maintain reliability. Beyond general LLM evaluation, specific frameworks like PersonaGym (Samuel et al. (2024) have emerged for assessing specialized agents, introducing the first dynamic evaluation framework and automated human-aligne metrie (PersonaScore) for persona agents, which are LLM agents designed to act according to an assigned persona.
Several frameworks employ
G.4.2 PROBLEM GENERATION VTA MULTI- PROTOCOLS
Other works explore multi-agent interaction such as collaboration, debate, or competi- tion- to automatically generate or evaluate content in novel ways. ChatEval et al. (2024) is a representative example where multiple LLM-based critics form a "referee team that debates and deliberates on the quality of a model's 's answer. By pitting several AI evaluators with different per- spectives against each other in discussion, the evaluation becomes more robust than single model's score, and the process mimics how multiple human judges arrive at a consensus. This multi-agent debate approach focuses on jointly evaluating content rather than generating new problems, but it showcases how agent interactions can replace and even surpass traditional human evaluation. In terms of content generation, BenchAgents (mentioned above) explicitly uses agents that cooperate (with minimal human oversight) to produce entirely new benchmark datasets effectively automating the benchmark creation process through agent teamworkButt et al. (022)) There also emerging
22

---

Published as a conference paper at ICLR 2026
benchmarks to test the capabilities of multi-agent systems themselves. MultiAAgemtBenc12110 et al. (2025a) evaluates how well LLM agents can collaborate or compete in shared environments and tasks, introducing scenarios where multiple agents communicate solve problem. Its contribution is a suite of multi-agent challenge tasks (with coordination protocols like star or graph networks and metrics for teamwork quality), rather than evolving benchmark, but it underscores the interest in agent-agent interaction. Meeseek, <ang (202)) takes an iterative multi-turn approach: it simulates a realistic user interacting with an LLM by providing feedback on failed requirements, and measures whether the LLM can self-correct over multiple rounds. While not explicitly framed as multi-agent (the "user" feedback could be programmatic). it creates a feedback loop akin two agents one pos- ing and refining the request, the other improving its answers working together to achieve a correct solution. Many these multi-agent or multi-turn frameworks successfully generate complex, rich interactions, but notably, most lack an explicit adversarial difficulty-rasiing dynamic. Agents often cooperate to improve quality (as collaborative problem solving or debate), rather than engaging in competitive play where one tries to stump outpace the other. Likewise, the tasks evaluations usually predefined or randomly sampled rather than progressively ramped up in response to a model's mastery. For example, ChatEval's agents not trying make the task harder they are jointly judging a given response. MultiAges Auen provides diverse scenarios but does not adapt scenario difficulty based on performance. In short, current multi-agent evaluation protocols focus on novel ways to assess or create content (often leveraging the wisdom of multiple Al judges creators), yet they stop short of introducing a competitive teacher-student dynamic or automated curriculum that continuously pushes the model to its limit.
H MODEL-FAMILY BIAS ANALYSIS (BIAS INDEX)
To quantitatively assess whether ATAD embeds model-family -ppiy--peciifii advantages, we introduce simple Bias Index that measures whether models perform better benchmarks generated by LLMs from the same family.
Let:
G = generator family (GPT, Claude, Gemini, LLaMA)
MG) set of models belonging to family G
Man(GG) all evaluated models not belonging to family G
Given benchmark generated by family G, compute:
BiasIndex (G) = Aeauracy (Mame (G) G) Mean Accuracca G).
A positive value would indicate a same-family advantage; a negative value would indicate a disadvan- tage. Values near zero indicate no family-specific bias.
Using the cross-family generation results from Appendix Table 5, we compute the Bias Index for all four generator families:
Generator Family
BiasIndex -0.53
Highest-Accurace Model
GPT
Claude-3.5-Sonme
Gemini
-0.24
Claude-3.5-Sonne
Claude
2.00
LLaMA-3.3-70B
LLaMA
-0.57
Gemini-2.0-Flash
Table 8: BiasIndex across generator model families.
Across all families, BiasIndex(G) & 0, confirming that no model consistently performs better on benchmarks generated by its own family. In fact, none of the four families achieved the highest accuracy on a benchmark generated by themselves (e.g., GPT - Claude highest; Claude - LLaMA highest; Gemini - Claude highest; LLaMA - Gemini highest). This provides further evidence that ATAD's s Orchestrator effectively removes surface-level stylistic cues or coneerator-spccif artifacts, preventing any family-aligned advantage.
23

---

Published as a conference paper at ICLR 2026
1 TASK-WISE GENERATION CHARACTERISTICS
Different anomaly types in ATAD exhibit distinct generation characteristics that can influence both the number validation cycles and the associated API cost. These variations arise from structural properties inherent to each task.
TI (Sentence Context Anomaly) and T3 (Blank-based Choice Anomaly) show relatively stable generation behavior, semantic-deviation and lexical-fit anomalies be produced with low ambiguity.
T4 (Bridge Sentence Evaluation) and T6 (Logical Contradiction) are substantially more demanding to generate. Producing plausible yet subtly incorrect logical links or causal relationships requires finer control, leading higher rate of candidate rejection.
T5 (Referential Ambiguity) and T7 (Tone/Style Violation) also exhibit increased generation difficulty due to stricter requirements on pronoun/reference clarity and stylistic register consistency.
These task-specific characteristics explain why generation cost may vary across anomaly types, even under the same model and pricing scheme.
J FORMALIZATION AND EMPIRICAL VALIDATION OF DIFFICULTY
J.1 FORMAL DEFINITION OF DIFFICULTY
In ATAD, we formally define the difficulty a problem instance q with respect to a Student model as the probability of failure conditioned on the problem being validated by the Orchestrator:
Difficulty(q: S) Pr[S fails on q E Valid].
(1)
where Valid denotes the set of problems that pass the Orchestrator's validation criteria, including well- formedness, logical coherence, task adherence, clarity, and fairness. This conditioning is essential: ATAD never interprets Student failures on defective, ambiguous, ill-posed items as meaningful difficulty.
The probability is defined over the Student's stochastic decoding behavior. In practice, estimate this quantity using the observed frequency of Student failures over Orchestrator aechestrator-validtted validated instances within each difficulty tier, where each instance is evaluated with single forward pass.
Difficulty(.., S) = 1- Accuracy(D., S),
where D, denotes problem subset at difficulty tier t. Under this formulation, difficulty is behaviorally grounded, solver-relative quantity: a problem is difficult precisely when a Student model of given capability fails under the protocol's validity constraints.
This perspective consistent with classical formulations educational measurement and psycho- metrics, where item difficulty is defined via the probability of correct (or incorrect) response for an examinee at a given proficiency level. Unlike fatent-trait estimation in item response theory, however, ATAD directly observes failure probabilities under controlled validation, making difficulty operationally and empirically measurable for LLMs.
J.2 DIFFICULTY TIERS ALONG SINGLE GENERATION TRAJECTORY
ATAD does not assign difficulty labels heuristically post-hoc. Instead, difficulty tiers emerge naturally along a single adversarial generation trajectory:
The Teacher generates base (Easy) problem.
If the Student succeeds, the Teacher generates more challenging variant.
This process continues through progressively refined versions (Hard, Extreme).
24

---

Published as a conference paper at ICLR 2026
Table 9: Average accuracy (%) across evaluation models as a function of difficulty tier. Each row corresponds to benchmark generated under different agent configuration. Difficulty tiers emerge along a single generation trajectory (Easy + Hard + Extreme + Impossible).
Generator (Teacher Student Orchestrator)
Easy
Hard
Extreme
Impossible
GPT-4o family (4o / 4o / /4))
83,94
83.33
80.71
70,94
Gemini-2.0-FB family (2.0F/ 2.0F 2.0F)
81.67
76.98
74.44
60.32
Cross-family (Claude-4/ Gemini-2-Fie GPT-44)
80.54
74.31
70.26
67.90
For each generation trajectory, ATAD progressively produces Easy, Hard, Extreme, and Impossible variants of the same underlying problem. All tier-specific instances along the trajectory are retained and form the tiered benchmark subsets.
Thus, Easy, Hard, Extreme, and Impossible are not independently sampled problems, but correspond to distinct stopping points along the same problem tineage. This design ensures that increases in difficulty reflect genuine refinement of the same underlying reasoning challenge rather than distributional shifts across unrelated samples.
In the experiments reported in this section, we analyze only those trajectories that progressed through the full Impossible tier in order allow consistent tier-wise comparison across generation regimes.
J.3 EMPIRICAL MONOTONICITY ACROSS DIFFICULTY TIERS
To empirically validate the proposed definition, we evaluate three independent benchmark generation regimes:
Same-family GPT-4o agents (Teacher Student Orchestrator = GPT-44),
2. Same-family Gemini-2.00Flaan agents (Teacher Student Orchestrator Gemini-2.0- Flash),
3. Cross-family, agents (Teacher Claude-4-Sonnel Orchestrator GPT-4o, Student Gemini- 2.0-Flash).
For each regime, we compute the average accuracy across all evaluation models (GPT, Gemini, Claude, and LLaMA families) over tasks T1-T7 each difficulty tier.
Across all three generation regimes, we observe a clear monotonic decrease in average accuracy as the difficulty tier increases from Easy to Hard, Extreme, and Impossible. This empirical trend directly validates our formal definition of difficulty as the Student's failure probability on Orchestrator- validated instances.
The same monotonic degradation is observed not only within each single-family generation regime but also consistently across different generation regimes. This indicates that the induced difficulty is not tied particular generator configuration, but reflects robust, model-agnostic increase in reasoning complexity.
J.4 Is ATAD DIFFICULTY DRIVEN BY GENUINE REASONING OR NARROW BLIND SPOTS?
key concern dynamic benchmark generation is whether increasing difficulty reflects genuine reasoning challenges merely exploits narrow, idiosyncratic blind spots of the Student model used during generation. ATAD explicitly addresses this risk through both architectural design and empirical validation.
Reasoning-conditton difficulty escalation. In ATAD, the Teacher does not blindly increase difficulty based solely whether the Student succeeds or fails. Instead, the Orchestrator analyzes the Student's 's explanation and provides structured feedback to the Teacher, identifying the precise reasoning patterns that led to success enror. The Teacher is then instructed to refine the same underlying anomaly to make more subtle or logically demanding, rather than introducing unrelated obscure knowledge spurious topic shifts. Moreover, the Orchestrator enforces strict validation
25

---

Published as a conference paper at ICLR 2026
rules on clarity, coherence, and faireess, rejecting items that rely ambiguity ill-defined cues. As a result, Student failures are used as difficulty signals only when the problem a valid, unambiguous reasoning instance.
Cross-family transfer and absence of self-preference. If difficulty were driven primarily by narrow blind spots of particular Student model, one would expect henchmarks generated by a given family to become trivially easy for other models. However, the consistent difficulty trends observed across the three generation regimes in Table indicate that the induced difficulty remains consistently non-trivial across different model families rather than collapsing outside the generator configuration.
We further observe no systematic self peefeereeee phenomenon: models from the same family as the generator do not consistently outperform other families their generated benchmarks (see Table 5). For example, on the GPT-4oo-ennratee benchmark, the best-performing model is often from different family. This supports the conclusion that ATAD does not overfit the reasoning idiosyncrasies of a specific Teacher-Student pair.
Taken together, the cesooninng-onditioonne escalation mechanism, the monotonic tier-wise degradation, and the absence of systematic self-preference provide converging evidence that ATAD difficulty reflects robust, generalizable reasoning demands rather than narrow model-specific artifacts.
K EFFECT OF USING THE SAME VS. DIFFERENT MODELS ACROSS AGENT ROLES
To analyze the effect of model homogeneity versus heterogeneity across agent roles, we conduct an additional experiment using cross-family configuration, in which the Teacher, Student, and Orchestrator are instantiated with different LLM families (Teacher = Claude-4-Sonnet Student Gemini-2.00-Faat Orchestrator GPT-4o). This setting is compared against the standard same-family configuration used throughout the main paper (Teacher = Student Orchestrator).
Table 10 reports the average accuracy of evaluation models under both configurations.
Table 10: Comparison between same-family and cross-family agent configurations.
LLMs
Same-family
Cross-family
gpt-3.5-turbo
54.18
51.71
gpt-4o-mini
55.00
53.86
gpt-4o
55.96
57.57
gemini-2.0-
Somini-2.0-flash-lle ---tlsh-lite
57.36
56.43
gemini-2.0-flan
58.93
54.57
claude-3-haiku
53.54
52.57
Mlaud-3.55hhiike
19.50
19.57
The overall performance ordering and difficulty level are broadly preserved across the two configura- tions. Although absolute values differ slightly, the heterogeneous setting does not induce systematic changes in model ranking or benchmark difficulty under the same evaluation protocol.
These observations indicate that ATAD's behavior is primarily governed by the protocol struc- ture- nure-namely, the Teacher-Student adversarial loop together with Drchestrator-enforee valida- tion-rather than by whether all agents originate from single model family. By enforcing validity, clarity, and coherence, the Orchestrator mitigates stylistic or family-specifi artifacts, which stabilizes the benchmark even when generation styles differ substantially across agents.
Taken together, these results suggest that ATAD remains stable under heterogeneous agent configura- tions. A broader exploration of additional cross-family combinations is left for future investigation.
26

---

Published as a conference paper at ICLR 2026
Table 1I: Conceptual comparison between ATAD and prior dynamic benchmark frameworks.
Criterion
DyVal
BARG
Beuehmark Selt- Selt-Evolving
ATAD (Oars)
Domain
Math. logic
Graph aummentation
Benchmack refresh
Text anomaly reasoning
Generation Architecture
Gonerator + Filter
Generator Critic
Generatur Selector
Teacher Orchestrator - Student
Role Separation
Single loop
x Mostly 2-ageni
x No 3-agen:
3-agent separation
Difticulty Scaling
Static
Student conditioned loop
dificultty conditioting
Student- esculation
Validation Mechanisa
Basic flltering
Code verilier
Quality scoting
Striet Orchestrator validation
L COMPARISON TO PRIOR DYNAMIC EVALUATION FRAMEWORKS (DYVAL, DARG, BENCHMARK SELF-EVOLVING)
Dy Val, DARG, and Benchmark Self-Evolving represent influential efforts toward automated bench- mark construction and renewal. However, their design objectives, target domains, and operational mechanisms differ fundamentally from those of ATAD. A structured comparison is provided in Table
A central distinction lies in the level at which adaptation is performed. Existing frameworks operate primarily at the data or graph level, focusing on dataset extension, perturbation, or refresh. In contrast, ATAD is defined at the protocol level, where difficulty is adaptively regulated through an explicit
interaction loop. Specifically, ATAD uniquely combines three elements that are absent in prior work: (i) an independent Orchestrator that enforces logical validity and clarity, (ii) Student-conditionne difficulty escalation driven by observed failure patterns, and (iii) an explicit three-agent role separation designed expose fine-grained reasoning weaknesses.
Val, DARG, and Benchmark Self-Evolving do not incorporate such failure- driven, protocol- adversarial scaling mechanisms. Because prior frameworks are designed modify or regenerate static benchmarks offline and subsequently evaluate models on fixed outputs, their objectives are not directly aligned with ATAD's adaptive difficulty-scaligg paradigm. As a result, direct quantitative performance comparisons would not be meaningful under their respective design assumptions.
ATAD should therefore be viewed not as a data-generation technique, but as a protocol-level difficulty regulation framework. Rather than refreshing datasets, it enables the continual regeneration of adversarial reasoning instances whose hardness dynamically co-evolves with model capability under validation constraints.
M FUTURE WORKS
M. THER-THEORIIIC FORMALIZATION OF THE ATAD PROTOCOL
A promising avenue for extending ATAD is to cast the as a cooperative game in which each agent's 's move (problem generation, validation, or solution) becomes unit of experience whose marginal contribution to overall benchmark quality can be quantified with game-theoretic tools such as Shapley values and their ordered extensions e.g.. the Nowak-Radzik value that explicitly respects the temporal ordering of curriculum steps Diaz et al. By treating successive rounds of ATAD as a sequence of coalitions, we could estimate how much each agent (or even each prompt strategy) accelerates difficulty calibration, then allocate compute or interaction budget proportionally to those cooperative gains; conversely, negative pairwise interactions would signal adversarial curricula redundant checks that should be pruned. Embedding this credit-assigmment mechanism inside the protocol would let ATAD adapt not only the problems it poses but also the roles and incentives its constituent agents, yielding a self-tuning, game- theoretic benchmark that co-evolves with frontier LLMs while remaining transparent and fair. This transposition explicitly links ATAD's adaptive benchmarking to the proven game-ther theorettc curriculum framework Diaz et al.] supplying both theoretical grounding and practical guidance for future protocol optimization.
loop itself
M.2 META-AGENT EXTENSIONS TO THE ATAD PROTOCOL
Recent advances in meta agents- agents that search over the design space of other agents-ofee offer promising path toward making ATAD self-improving. safer, and more sample-efficient. A meta
27

---

Published as a conference paper at ICLR 2026
iteratively discover superior teachers (richer problem generators) and orchestrators (sharper validators) while archiving each discovery for reuse. Complementarily, AFLOW's s Monte- Monte-Carlo- Carlo-Tree- Trre-Search over code-represented workflows can refine validation pipelines and student curricula so that even smaller models, paired with optimized workflows, rival larger baselines
agent that programs agents in code-as as in Meta Agent Search Hu et al. (2025
fraction of the compute cost Zhang et al. (2025). Casting "make the problem just hard enough", "catch adversarial trickery", and "keep tasks unambiguous" as explicit objectives in this unified search space lets the meta agent optimize difficulty, diversity, and alignment constraints simultaneously. Because each generated anomaly carries its provenance and (optionally) machine-checkabe proof, the resulting benchmark remains auditable even as it grows without bound. In short, a meta-agent layer transforms ATAD from a fixed three-role protocol into a self-refining ecosystem where agents, workflows, and evaluation criteria co-evolve alongside the LLMs they test.
M.3 EXTENDING ATAD BEYOND INTERNAL- -NOWLEEGEE REASONING
While the present work focuses on self-contained reasoning problems aligned with standardized cxam formats such as GRE, LSAT, and GMAT the ATAD protocol is fundamentally task-agnostic and can be extended to incorporate external factual knowledge, mathematical axioms, or domain- specific constraints.
Below outline concrete design directions enabled by the existing Teacher-Orchesttatr-SSSu pipeline. These extensions were inspired in part by reviewer feedback, and we will explore them in future work.
M.3.1 INTEGRATING EXTERNAL SOURCES INTO QUESTION GENERATION
The Teacher agent may be conditioned on retrieved factual snippets or curated external documents. For example:
Factual reasoning: Provide the Teacher with retrieved Wikipedia news snippets and instruct to embed domain-specific inconsistency.
Scientific or legal reasoning: Supply the Teacher with short excerpts from scientific papers, laws, technical specifications and ask it to introduce contradictions or violations of established rules.
Mathematical reasoning: Supply explicit axioms, definitions, or numeric constraints and require the Teacher to generate perturbations consistent with formal reasoning system.
These modifications only change the input the Teacher; the Orchestrator's validation rules and Student-driven adaptive difficulty remain unchanged.
M.3.2 RAG-ENHANCED ORCHESTRATOR VALIDATION
The Orchestrator can be augmented with retrieval components to verify the factual correctness or mathematical soundness of generated items. Examples:
factual verification against retrieved knowledge bases
unit-consistency or equation-validiyy checking
symbolic-logic consistency checks for formal domains
Because the Orchestrator already performs strict structural and semantic validation, adding retrieval- augmented checks is conceptually straightforward.
M.3.3 ADAPTIVE DIFFICULTY WITH EXTERNAL KNOWLEDGE
Difficulty escalation in ATAD currently intensifies semantic or logical challenge factors within self-contained contexts. In an external-knowlene setting, adaptive difficulty could escalate via:
broader or less explicit supporting facts
28

---

Published as a conference paper at ICLR 2026
multi multi-hop reasoning multiple retrieved snippets
more abstracted or implicit references to factual background
This preserves the core ATAD principle: difficulty increased reasoning demand, not increased ambiguity.
M.4 ON GENERALITY BEYOND TEXT ANOMALY DETECTION
The focus on text anomaly detection this work driven by methodological considerations rather than domain restrictions. In contrast to math or code, where correctness criteria are strictly for- malized and difficulty control is comparatively straightforward, text anomaly detection presents a uniquely challenging enviromment: difficulty must increase without introducing ambiguity, anomalies must remain subtle yet objectively identifiable and reasoning must integrate pragmatics, discourse coherence, referential clarity, and multi-sentence inference.
This makes text anomaly detection a stringent stress test for dynamic difficulty scaling. Successfully controlling difficulty under such ambiguity-sensie sensiiive conditions provides conservative validation setting for the ATAD protocol.
Importantly, ATAD is task-agnostic by design. Section introduces domain-specific assumptions; the protocol only requires and correctness. As result, the framework naturally generalizes to domains such as inconsistencies, code-level reasoning, tool-use verification, and multimodal consistency.
loop with
clarity
Thus, rather than limiting generality, the present instantiation demonstrates ATAD's effectiveness in one of the most ambiguity-sensitte reasoning domains, providing strong evidence for transferability more formally grounded tasks.
M.5 ON REAL- WORLD REPRESENTATIVENESS AND FUTURE APPLICATIONS
In its current instantiation, ATAD adopts controlled GRE/LSAT-style anomalies to establish clean diagnostic setting in which ground truth is precisely defined. This design choice enables controlled analysis of difficulty scaling under unambiguous reasoning conditions. Nevertheless, the ATAD protocol itself is not restricted to academic-style inputs.
The iterative, ciilure-driven difficulty escalation mechanism aligns closely with the types of reasoning breakdowns observed in real-world data, including subtle logical inconsistencies, causal misalign- ments, and procedural violations. Consequently, ATAD naturally extends to domain-grounded, real-world anomaly settings.
Representative future application domains include:
Finance: causal inconsistencies or misaligned risk reasoning in analyst reports,
Scientific and medical writing: methodological or interpretive inconsistencies, Industrial operations:
violations of process logic, measurement coherence, procedural
constraints.
These domains involve exactly the types of reasoning-cemt centtri inconsistencies that the Teacher-Student competition and Orchestrator validation are designed to surface. Extending ATAD to such domain- grounded environments therefore constitutes natural and practically meaningful next step.
M.6 SUMMARY AND FUTURE DIRECTIONS
Together, the analyses in Sections M.4 and M.5 illustrate that ATAD's dynamic generation protocol neither limited to text anomalies nor restricted to internal LLM knowledge alone. Instead, the protocol provides flexible substrate for constructing domain-specialian dynamic benchmarks under validated difficulty control.
The successful application of ATAD to -mbiguity-sensitive text anomaly detection demonstrates robustness under one of the most challenging reasoning environments. At the same time, the task- Student loop enables natural generalization more
agnostic nature of the
29

---

Published as a conference paper at ICLR 2026
structured domains such mathematical reasoning, code-level verification, tool-use consistency, and multimodal understanding.
Moreover, the failure- adaptive mechanism aligns closely with real-world reasoning break- downs, suggesting strong potential for deployment in domain- ammain-gouunde settings such as finance, scientific and medical analysis, and industrial operations.
Future work will therefore pursue (i) domain-specific instantiations of ATAD under external knowl- edge retrieval conditioning, (ii) extensions to executable reasoning tasks such code and tool use, and (iii) real-world benchmark construction in operational environments. These directions aim to further validate ATAD seneral-purposs protocol for dynamic, difficulty-regulated reasoning evaluation.
30


Reviews:
[{"review_id": "xl4H8v9WPU", "reviewer": "Reviewer_srE2", "summary": "This paper introduces Agent-Centric Text Anomaly Detection (ATAD), a dynamic benchmarking framework designed to address the saturation of static LLM benchmarks. ATAD employs a three-agent protocol in which a teacher generates questions, an orchestrator validates their quality, and a student attempts to solve them, with task difficulty escalating iteratively based on student performance. This automated process enables scalable benchmark generation and adaptive evaluation of LLM reasoning. Using this protocol, the paper constructs a benchmark for text anomaly detection, and experiments across four LLM families (GPT, Gemini, Claude, and LLaMA, used both as question generation and evaluated models) show that ATAD effectively increases task difficulty and reveals reasoning limitations even in strong LLMs.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a timely issue where static benchmarks are becoming saturated and insufficient to capture the evolving reasoning capabilities of modern LLMs.\n\n2. The proposed teacher–orchestrator–student framework is conceptually interesting and provides a scalable, automated approach for generating dynamic and evolving benchmarks.\n\n3. The experiments provide meaningful insights into LLM reasoning across different model families on text anomaly detection and the contributions of individual components within the proposed framework.", "weaknesses": "1. Based on the prompts shown in Appendix D, the current framework appears to generate questions primarily from the LLM’s inherent knowledge rather than external sources. It is unclear how well this approach generalizes to reasoning problems that require access to factual data or rigorous mathematical reasoning, given that LLMs’ capabilities in these domains are still evolving and may limit the robustness of generated questions.\n\n2. Since the benchmark questions are generated by LLMs, potential model-family-specific biases could be implicitly embedded in the dataset, e.g., models might perform better on questions generated by LLMs from the same family. Although Table 2 suggests no strong bias, a formal analysis and discussion of this issue would strengthen the validity of the proposed protocol. \n\n3. The paper does not include an analysis of the generation cost or number of iterations involved in producing the final benchmark. Reporting metrics such as the average number of iterations per valid question across different anomaly types, and the overall cost (e.g. in $) would help practitioners assess the effectiveness and cost trade-off for future benchmark construction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7KirmjXcQt", "reviewer": "Reviewer_JAkP", "summary": "This paper introduces ATAD (Agent-Centric Text Anomaly Detection), a dynamic benchmarking protocol designed to overcome the limitations of static LLM benchmarks such as MMLU and GSM8K. Instead of relying on fixed datasets, ATAD employs a three-agent architecture—a Teacher, Orchestrator, and Student—to iteratively generate, validate, and solve text anomaly detection (TAD) tasks. The Teacher produces candidate problems, the Orchestrator validates them for clarity and fairness, and the Student attempts to solve them. The benchmark evolves dynamically: when the Student succeeds, the Teacher generates harder variants, validated again by the Orchestrator.\n\nEmpirical results on datasets generated by GPT-4o, Claude-3.5-Sonnet, Gemini-2.0-Flash, and LLaMA-3.3-70B show that the ATAD framework effectively scales difficulty and exposes reasoning weaknesses unseen in traditional benchmarks.", "questions": "* Can the authors propose or measure an explicit difficulty metric to formalize the notion of “increased difficulty”?\n\n* Since LLMs generate benchmark data, how do the authors ensure that cultural or linguistic biases in generation do not propagate into the benchmark? Have they evaluated cross-language or demographic stability?\n\n* The paper mentions “failure cases rejected by the Orchestrator” in the Appendix but does not elaborate in the main text. Could the authors analyze typical rejection patterns—e.g., ambiguity vs. incoherence—to demonstrate the Orchestrator’s learning or filtering capacity?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The proposed agentic co-evolution of models and benchmarks represents a natural next step after efforts like BenchAgents and DyVal, making this work both timely and forward-looking.\n\n* The inclusion of Orchestrator-regulated validation and failure-driven sample finalization help control over problem difficulty and fairness.\n\n* The taxonomy of seven anomaly types—each probing distinct reasoning skills (contextual, logical, referential, stylistic, etc.)—is well-grounded and illustrative", "weaknesses": "* While the empirical results are strong, the notion of “difficulty increase” remains heuristic and behaviorally defined (based on Student failure). A more formal definition would strengthen claims.\n\n* The paper cites prior dynamic evaluation works (DyVal, DARG, Benchmark Self-Evolving), but comparative experiments are missing. It is unclear how ATAD quantitatively improves over these prior agent-based or meta-probing systems beyond conceptual novelty.\n\n* Although text anomaly detection is reasoning-oriented, it remains a single task family. The authors argue that it generalizes to reasoning evaluation, but this may not hold for domains like math, code, or multimodal reasoning. Broader applicability should be demonstrated.\n\n* The same model family is used for all three agent roles in most experiments (e.g., GPT-4o Teacher = Student = Orchestrator). This setup may limit adversarial diversity. \n\n* Although the authors observe a ~37 pt accuracy drop after adaptive scaling, it is not clear why certain tasks become more difficult—whether due to semantic subtlety, linguistic ambiguity, or distractor complexity. More granular ablations (e.g., by anomaly type, linguistic factor) would enhance interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "su8dMSnes2", "reviewer": "Reviewer_6GgZ", "summary": "This paper addresses the growing limitations of static benchmarks for evaluating LLM reasoning, such as data contamination and performance saturation. The authors propose a novel and significant contribution: a dynamic, agent-centric benchmarking protocol named ATAD (Agent-centric Text Anomaly Detection). This protocol leverages a multi-agent system where a \"Teacher\" agent generates problems, a \"Student\" agent attempts to solve them, and a critical \"Orchestrator\" agent validates problem quality, coherence, and fairness. The benchmark's core strength lies in its adaptive difficulty scaling: if the Student solves a problem, the Teacher is prompted to generate a more challenging version, thus allowing the benchmark's difficulty to co-evolve with the Student model's capabilities. By focusing on text anomaly detection—a task requiring deep, cross-sentence logical inference—the paper demonstrates that this protocol can effectively surface nuanced reasoning failures in state-of-the-art LLMs that are often missed by traditional static evaluations, offering a more sustainable and scalable alternative.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a well-motivated and timely problem, clearly articulating the critical failings of static benchmarks, such as data contamination and performance saturation. This strong motivation establishes a clear need for the proposed dynamic evaluation protocol. By addressing these limitations, the work provides a valuable and forward-looking contribution to LLM evaluation.\n\n2. The evaluation is comprehensive, featuring a wide array of models from different families (e.g., GPT, Gemini, Claude, LLaMA) to demonstrate the protocol's broad applicability. The experiments wisely test these models in the various agent roles—Teacher, Student, and Orchestrator—in addition to using them for final evaluation. This thorough approach provides strong evidence for the protocol's robustness and its ability to function as intended across different model capabilities.\n\n3. The presentation is exceptionally clear, with diagrams and a well-defined protocol that makes the complex multi-agent interaction easy to understand. The paper is further strengthened by insightful qualitative examples, particularly those showing problems rejected and refined by the Orchestrator. These examples compellingly illustrate the subtleties of the generated tasks and provide a concrete demonstration of how the validation step ensures benchmark quality and fairness.", "weaknesses": "1. The paper's primary contribution is the novel protocol (ATAD) and its application, rather than a new underlying model architecture or training technique. While this agent-centric framework is a significant engineering and conceptual achievement, the work relies entirely on existing LLMs as its core components. This focus on the evaluation system means there is limited technical novelty in terms of model-level innovations.\n\n2. While the paper demonstrates that the protocol creates more difficult tasks, it lacks a deeper analysis of how these tasks generalize. It is unclear if the \"difficulty\" stems from truly novel reasoning challenges or from exploiting specific, narrow blind spots in the Student model used during generation. Further analysis is needed to determine if the finalized benchmarks test for robust, generalizable reasoning or if they risk overfitting to the particular Teacher-Student dynamic of a specific agent family.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SMjyWRbjjG", "reviewer": "Reviewer_pyPR", "summary": "Traditional LLM evaluation via static datasets (e.g., MMLU) has flaws like poor scalability and overfitting. The paper proposes ATAD, a dynamic protocol using three agents (Teacher, Orchestrator, Student) to generate/solve 7 text anomaly tasks without static data—harder tasks emerge if the Student (evaluated LLM) succeeds, with the Orchestrator ensuring quality. Experiments confirm ATAD works: LLM accuracy drops ~37.3% (valid difficulty), it adapts to future LLMs, and results are stable. ATAD thus enables scalable, sustainable LLM reasoning assessment.​", "questions": "1. Could you clarify how the Orchestrator evaluates and filters ambiguous or low-quality questions? Are there explicit criteria or rules, or is it purely model judgment?\n2. Why did you choose text anomaly detection specifically as the first application of your adaptive benchmark? Were other tasks considered, and if so, why were they not selected?\n3. When different LLM families are used as Teachers, do they generate qualitatively different types of anomalies? Could you share examples of such differences?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Pioneering Dynamic Evaluation Paradigm: The paper introduces a novel, agent-centric dynamic protocol to address the fundamental issues of data contamination and saturation in static benchmarks, offering a sustainable and scalable solution for LLM evaluation.\n\n2. Rigorous Quality Control Mechanism: It introduces the crucial \"Orchestrator\" agent to guarantee the fairness and high quality of generated problems. This effectively prevents the creation of flawed or ambiguous questions that result from an unvalidated competitive process aimed solely at increasing difficulty.\n\n3. Clear and Comprehensive Task Design: The methodology is exceptionally clear and built upon a robust and transparent taxonomy of seven distinct text anomaly detection tasks. This ensures the evaluation framework is well-grounded, broad, and fine-grained.", "weaknesses": "1. The claims in Section 2.3 need to be substantiated with data. As it currently stands, the section is unconvincing because it lacks the necessary data and statistics for support.\n2. The benchmark focuses only on text anomaly detection. While useful, it does not cover other critical abilities like tool use, multi-step reasoning, code generation or multimodal understanding, which limits external validity. I believe this technology would be far more meaningful if it were applied to a broader range of tasks.\n3. The method defines difficulty by the moment when the Student first fails, which may be due to randomness (e.g., sampling variance) rather than true capability limits. A more rigorous approach would be to let the Student attempt multiple independent trials (e.g., 5 times) and define difficulty based on the failure rate or success rate, rather than deciding from a single attempt.\n4. Even within text anomaly detection, the benchmark examples may not fully represent the diversity of real-world anomalies. They are often artificial and may not match naturally occurring errors in user-generated content or professional texts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)