TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture
Authors: Yongchao Chen
Jiefeng Chen
Rui Meng
Ji Yin
Na Li
Chuchu Fan
Chi Wang
Tomas Pfister
Jinsung Yoon
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Tool-Augmented Reasoning
Test-Time Scaling
Multi-Agent Systems
Code Interpreter
Search
Primary area: foundation or frontier models, including LLMs
TL;DR: TUMIX combines diverse tool-use agents (text, code, search) into a dynamic mixture, outperforming state-of-the-art test-time scaling with higher accuracy at half the cost.

Abstract: While integrating tools like Code Interpreter and Search has significantly enhanced Large Language Model (LLM) reasoning in models like ChatGPT Agent and Gemini-Pro, practical guidance on optimal tool use is lacking. The core challenge is effectively combining textual reasoning, coding, and search for diverse questions. In this paper, we propose Tool-Use Mixture (TUMIX), an ensemble framework that runs multiple agents in parallel, each employing distinct tool-use strategies and answer paths. Agents in TUMIX iteratively share and refine responses based on the question and previous answers. In experiments, TUMIX achieves significant gains over state-of-the-art tool-augmented and test-time scaling methods, delivering an average accuracy improvement of up to 3.55\% over the best baseline on Gemini-2.5-Pro and Gemini-2.5-Flash across key reasoning benchmarks, with near-equal inference costs. We find that agent diversity and quality are crucial and can be enhanced by using LLMs to auto-optimize agent designs. Furthermore, TUMIX can halt refinement upon reaching sufficient confidence, preserving performance at only 49\% of the inference cost. Further scaling can achieve higher performance, albeit at a greater cost.

Published as a conference paper at ICLR 2026
TUMIX: MULTI-AGENT TEST-TIME SCALING WITH TOOL-USE MIXTURE
Yongchao Chen 2+
Jiefeng Chen
Rui Meng'
Ji Yin Jinsung Yoon
Na
Chuchu Fan!
Chi Wang'
Tomas Pfister
MIT
Harvard
Google Cloud Al Research
*Google DeepMind
yongchaocheneeas fas harvard.
jinsungyoon@googee.ccmm
ABSTRACT
While integrating tools like Code Interpreter and Search has significantly enhanced Large Language Model (LLM) reasoning in models like ChatGPT Agent and Gemini-Pro, practical guidance optimal tool use is lacking. The core challenge is effectively combining textual reasoning, coding, and search for diverse questions. In this paper, we propose Tool-Use Mixture (TUMIX), an ensemble framework that runs multiple agents in parallel, each employing distinct tool-use strategies and answer paths. Agents in TUMIX iteratively share and refine responses based on the question and previous answers. In experiments, TUMIX achieves significant gains over state-of-the-an tool- cool-augmented and test-time scaling methods, delivering an average accuracy improvement of up to 3.55% over the best baseline on Gemini-2.5-Pry and Demini-225--Flas across key reasoning benchmarks, with near-equal inference costs. We find that agent diversity and quality are crucial and can be enhanced by using LLMs to auto-optimize agent designs. Furthermore, TUMIX can halt refinement upon reaching sufficient confidence, preserving performance at only 49% the inference cost. Further scaling can achieve higher performance, albeit at greater cost.
H Jumanity's Last Exam (HLE) Pro
GPQA, Diamond - Pro
AIME 24&25 Pro
100
87,9
88.1
96.7
Score
35
Score
85.5
86.0
B8.9
Score
93.7
30
29.3
29.5
28,1
85
26.9
90
25
wio ITs
20
Self-Mo4 Symhollc-Mok
DEI SeiMaster
TUMIX (Oure)
GBA
TUMIX+ (Ours)
wi TTS
80
SelfMos Symbolkc MoE
DEI SuMaster
TUMI (Curs)
GS4
TUMIK+ (Ours)
wie T7S
85
Self-Mo
Symb Mob
SalMaster
DEL
TUMIY (Ours)
GSA
TUMEX (Ours)
Humanity's 's Last Exam (HLE)
Flash
GPQA, Diamond
Flash
AIME 24625
Flash
30
90
90
06.7
25
77.3
85
80.7
Score
20
10.5
1933
17.4
-70
67.9
Score
79.7
64.1
62.6
60
70.0
10
70
50
Wio T7S
SelEMod Symbell--Mol
StiMaater
DEL
TUMIX (Oure)
CSA
TUMIX+ (Ours)
TTS
SelEMad
Symbollk MoE
DEI
BuMaster
TUMIX (Durs)
GSA
TUMIK+ (Dars)
win TTS
65
MalEMoA
Symbelk-MoB
DEI SaMatter
TUMIY (Oars)
OSA
TUMEX (Gurs)
Figure 1: Comparison of tool- -uggmeteed tist-time scaling methods on Gemini-2.5. 2.5-Pro (first row) and Gemini-2.5-Flan (second row) across HLE, GPQA, and AIME 24&25. Except for methods without test-time scaling (w/o TTS) or additional scaling (TUMIX+), all methods in the same subplot use nearly the same number of inferences and tokens. For fair comparison, methods that originally lacked tool use are run with strong tool-augmented agents instead text-only agents. Each score is the average of three repetitive runs.
"This work was done while Yongchao research intern Google Cloud Al Research.

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
While reinforcement learning-based fine-tuning has greatly improved L.M reasoning (Guo et al., 2025), models still struggle with seemingly simple tasks (Chen et al., 2024b). Such tasks are often better handled with code (Madaan et al., 2022; Chen et al., 2022) search (Jin et al., 2025; Li et al., 2025b). Textual reasoning is strong in semantics and commonsense, but weak in precise computation and in accessing or updating the latest knowledge.
A key challenge is fully utilizing the potential capabilities of textual reasoning, coding, and searching when facing distinctive questions with varied characteristics. Most input questions lack explicit cues for the best approach, and the combined text/code/search solution space is large. Frontier LLM-poweree products such as ChatGPT, Claude, Gemini, and Grok report using code and search at test time to augment reasoning, but without publishing detailed methods. Recent work (Chen et al., 2024b) shows that current Code Interpreter implementations in OpenAl models often fail to balance text and code, leaving coding capabilities underused, as shown in Appendix Fig. 11. Moreover, public research still lacks a clear understanding of how to integrate Code Interpreter and Search for improved LLM reasoning.
To better leverage both tool use and LLM self-reasoning, we propose Tool-Use Mixture (TUMIX), framework that integrates Code Interpreter and Search into LLMs via test-time scaling. TUMIX runs multiple diverse agents in parallel, each with different tool-use strategies. Their outputs are iteratively aggregated and refined across multiple rounds. In each round, every agent generates new solution by considering both the original question and the previous round's reasoning and answers from all agents. TUMI X uses diverse agents and tool- augmeneedd reasoning strategies to explore a wide range of possible solutions. The following iterative process encourages diverse reasoning paths and deeper integration. This design is inspired by prior test-time scaling methods such as Mixture-of-Agent (MoA) (Wang et al., 2024), which rely on multiple LLMs within a single framework and do not incorporate external tools. In contrast, TUMIX employs single LLM with both text-only and tool-augmented agent frameworks, making it more generalizable for practical applications. Furthermore, in cool-augeetted multi-agent test-time scaling. we find a diverse group of agents outperforms repeated use of the single best agent, conclusion that differs from MoA (Li et al., 2025a). We later reveal that human pre-designed agent group can be further optimized by querying L.LMs to self-design more diverse high-quality agents based on current ones, adding an average 1.2% improvement without cost increase.
Since questions vary in difficulty, they require different amounts of iterative refinement. We query the LLMS to decide whether to terminate refinement early, while still enforeing a minimum number of rounds to maintain answer quality. This adaptive carly-termination strategy reduces inference costs to 49% of the original two settings (termination in a fixed round number or by majority-vote consistency across rounds), while preserving or even improving performance. The improvement arises because over-refinement rarely changes the final result and can even degrade performance, as correct answers may be mistakenly discarded.
Compared to the model without test-time scaling, TUMIX delivers an average +7.8% and + +17.4% accuracy gains in benchmarks Humanity's Last Exam (HLE) (Phan et al., 2025), Graduate-Level Google- Proof Q&A (GPQA, Diamond) (Rein et al., 2024), and American Invitational Mathematics Examination (AIME 24&25) with base models Gemini-2.5-Pru and Gemini-2.5- F.saFa,,, respectively. Under the same inference costs, TUMIX also outperforms existing representative test-time scaling methods such as Self-MoA, Symbolic-MoE, DEI, SciMaster, and GSA, with an average +3.55% lifting compared to the best performing baselines. Notably, with further scaling, TUMIX raises Gemini-2.5-Pa accuracy on HLE from 21.6% to 34.1%, surpassing Gemini-22 2..--Pro Deep Research at 26.9% (32.4% with higher compute) (Comanici et al., 2025). Test-time scaling hinges on two stages (Brown et al., 2024): (1) generating diverse candidate solutions and (2) selecting the correct one. For questions with both small answer spaces (e.g., multiple-choice) and large ones, diverse sampling greatly improves coverage. While it achieves high coverage on HIE (among generated answers in the whole round, least one is correct on > 65% of questions), accuracy plateaus at about 34% because LLMs struggle to identify the correct answer among noisy candidates. We identify and explore four key factors: agent quality, agent diversity, refinement termination, and answer selection. Our work makes the following contributions:
1. TUMIX: A competitive tool-augmented test-time scaling method. We propose TUMIX, a novel framework for test-time scaling that integrates tool augmentation. Extensive experiments demonstrate that TUMIX consistently outperforms strong baselines, achieving an average improvement of +3.55% over the best best-performing prior methods.
2. Key factors and mechanisms in tool-augmented scaling. We provide a systematic analysis that distinguishes tool -ug-enegeet scaling from traditional test-time scaling:
Agent diversity and quality outweigh scale alone. High-temperature sampling increases coverage, but heterogeneous agent strategies yield higher accuracy and lower cost than repeatedly sampling from a single hest-performing agent.
Tool augmentation boosts performance. Agent groups equipped with tools such as Code Interpreter and Search achieve superior coverage and accuracy compared to text-only agent groups
2

---

Published as a conference paper at ICLR 2026
3. LLMs as agent designers. We show that prompting LLMs to automatically generate diverse, high-quality agents based on existing ones further improves TUMIX. This yields an additional average aceuracy lift of +1.2%.
4. LLM-as-Judge for refinement termination. We introduce an LLM--base judge to adaptively determine the optimal stopping round in iterative refinement. This prevents excessive refinement, which reduces diversity and can mistakenly discard correct answers. By enforcing minimum refinement depth and querying the judge for termination, we achieve near-optimal accuracy while reducing inference cost to ~49% of the original.
2 RELATED WORK
Code Interpreter and Search Many benchmark tasks can in fact be better solved through code (Gao et al., 2023) and search (Li et al., 2025b), and recent work extends coding to reasoning and semantic analysis (Li et al., 2023a; Weir et al., 2024). Most prior approaches use either text (Yao et al., 2024) or code (Bairi et al., 2024; Zhou et al., 2023) exclusively as output. Recent work (Chen al., 2024b) emphasizes the need to dynamically switch between modalities, proposing CodeSteer (Chen et al.) as guidance model. Extensions with retrieval (Jin et al., 2025; Li et al., 2025b) and tool use (Qian et al., 2025) further improve reasoning. but lack the thorough exploitation of Code Interpreter and Search tools. Leading models such as OpenAl's ChatGPT Agent, Google's ' Gemini-Pro (Comanici et al., 2025), and XAI's Grok4 report using code and search at test time to augment reasoning, but without publishing detailed methods. Open work such as ToRL (Li et al., 2025c) and ReTool (Feng et al., 2025) investigates training reasoning models to integrate with Code Interpreters. However, their training and evaluation are limited to math problems. leaving a significant gap from real-world applications that demand effectiveness across broader benchmarks. TooIRI (Qian et al., 2025) instead focuses on teaching models to select among multiple tools, where the generated codes and search queries are relative simple and the evaluation tasks require less reasoning capabilities. SciMaster (Chai et al., 2025) samples the same pre-designed tool-use agent five times, then uses other pre-designed agents to critique, refine, and aggregate the answers. This approach shows clear improvement over single-inference text-only baselines, but the extent and manner of tool exploitation remain underexplored. In summary. integrating Code Interpreter and Search into LLM reasoning is essential and challenging. The academic community currently lacks methods and studies that fully exploit the benefits of I.LM selffreasoning, code execution, and search, which is the focus of our work.
Test-time scaling LLM self-exploration, reflection, and evaluation can enhance task performance across do- mains (Yang et al., 2022; Welleck et al., 2022; Madaan et al., 2023). Models like OpenAl ol (Jaech et al., 2024) and DeepSeek RI (Guo et al., 2025) showcase agentic behavior via Chain-of-Thoughh (CoT) reasoning and self-reflection, which is learned by RL-based training with rule-based outcome rewards (Shao et al., 2024; Wei et al., 2025). Apart scaling, many research also explore scaling during LLM inference time by pre-designing
from the training-based prompt and agent frameworks. In these works, multi-agent reasoning has emerged as promising paradigm for enhancing complex problem-solving and pecision-makinge in AI systems (Wu ct al., 2023; Li et al., 2023b; Topsakal & Akinei, 2023). Prior work finds gathering the answers from different LLMSs improves LLM performance (Du al.). Mixture-of-Agemt (MoA) ( (Wang et al., 2024) further extends this idea by sharing and gathering answer among LLMs. However, Self-MoA (Li et al., 2025a) argues that LLM diversity may not be critical since replacing different types of LLMs with the best one achieves better performance. Symbolic-Mot (Chen et al., 2025b) further assigns different questions with different specialized LLMs. Instead of using different types of LLMs, many works such as DEI (Zhang et al., 2024), GSA (Li et al., 2025d), and SETS (Chen et al., 2025a) employ different agents from the same LLM for extensive test-time scaling, in which the agent types and frameworks are explored (Chen al., 2024a). Similar to our work, previous work in test-time scaling also finds the correct answer selection (Brown et al., 2024) is the main bottleneck. While previous work in test-time scaling do not incorporate tool-use of Code Interpreter and Search, we study how to utilize test-time scaling methods to better exploit the benefits of each reasoning mode.
3 TSOL-USE MIXTURE
Appendix B presents the full TUMIX algorithm, and Appendix C lists all agent prompts.
3.1 PRE-DESIGNED DIVERSE AGENTS
As shown in Fig. 2, we regard TUMIX as sequential decision-making under a compute budget with diverse and correlated experts (agents). Each round selects which agents to run, what they may read (communication policy), when to stop (optimal stopping), and how to aggregate (decision rule), trading off accuracy and cost. Let q be a task with unknown correct answer a' in answer space A. There is a pool of agents S = {s)..... .8...... [8) Agent 8; outputs answer Y; E A at cost e and has competence p: p:(g) = IP = a* q). Let Z, = 1[Y, = a²) } denote correctness indicators. Their dependencies (and hence ensemble diversity) are captured by a correlation or mutual-ine intuuaaaiioooniiinnn structure over {2,).
3

---

Published as a conference paper at ICLR 2026
Refine answers based on (all wwwwwerr_preeioss_oooeooo
CoT
Code
Code+
Code
Code+
Search
Selector
Question
+
Answer
Search
Dual-Tool
Search+
Human and LLM designed agents
Termination and selection methods
O
Guided Agent
Guided
Guided Agent
Agent
Round 1
Round 2
Termination Round T
Figure 2: Overview of TUMIX framework. At each iteration, the responses from all agents in the previous round are concatenated with the original question, forming a joint prompt for the next round. This prompt is then provided to all agents (either the same new agent groups) to produce refined answers. The subsequent prompts follow the structure illustrated in the above. The design of the agents, their number and specialization, the refinement termination criteria, and the selection strategies are the key factors that determine the effectiveness of the framework.
Table 15 dre-dee agents used in TUMIX.
Full Name
Short Name
Description (15 agents)
w/o TTS
Base
Direct prompt.
CoT Agent
CoT
Chain-of-Thouse prompt (Wei al., 2022).
CoT-Code Agent
CoToode
CoT prompt to output code.
Search Agent
Uses WebSearch (LLM inherent tool only).
Code Agent
Uses Code Interpreter (hase version).
Code Agent
c+
Uses Code Interpreter (hinted version with extra human pre-designed priors).
Dual-Tool Agent
Uses Code Interpreter + WebSearch (with 3 search variants).
Guided Agent
CSG
Dual-Tool agent (CS) guided by a steering module (Chen et al.) (with 3 search variants).
Guided Agent+ csG
Guided agent (CSG) with enhanced/hinted prompts (with 3 search variants).
A policy (our focus) chooses in each round: (i) which agents to run, (ii) the communication graph (what each agent may read from prior rounds), (iii) the stopping rule, and (iv) the aggregation rule producing åg. A canonical objective is
max ( a²} A Costx,
(1)
where trades off compute and accuracy. In our work, the Cost, the total number inference times and input and output tokens to generate the final answer. In the default TUMIX setting, we utilize the same 15 pre-designed agents in all answer refinement rounds. These 15 agents have distinct reasoning and tool-use strategies, as summarized in Table 1. Agents with search access have three search methods (Google Search API (gs), inherent LLM search function (11m), or their combination (com)), yielding three variants per agent. For agents employing multi-round interactions with Search or Code Interpreter, the maximum tool interaction round number is set to 5. In Section 5.3, we discuss how to further query LLMs to automatically optimize and design more diverse agents to achieve better performance. We also compare with dynamic setting where agent types vary across rounds.
3.2 REFINEMENT AS MESSAGE PASSING (ACCURACY RISES AND DIVERSITY SHRINKS)
In each round, every agent independently generates a new solution by considering both the original question and the solutions provided by all agents the previous round, as shown Fig. 2. We evaluate the refinement process using two metrics: average accuracy and coverage (the probability of at least one correct) across agents in each round, which capture the quality and diversity of group answers (Brown al., 2024). For set C S, the coverage is
Coverage(S)) IP Ut+ a')).

---

Published as conference paper at ICLR 2026
Coverage
Average Score
Single Agent
(a)
HLE
(b)
GPQA (Diamond)
(c)
AIME 2025 & 2024
60
100
100
Accuracy (%)
Accuracy (%)
Accuracy (%)
45
30
90
90
15
85
85
2
3 Round
5
2
3 Round
4
5
1
2
3 Round
4
5
Figure 3: Evolution of coverage, individual agent scores, and average scores across refinement rounds. Coverage decreases monotonically across all benchmarks. For HLE and AIME, the average score rises over the initial rounds before platcauing, For GPQA, the average score improves early on but subsequently declines with further refinement.
R3: 0
R5:
RD: 0
R-:1-2
R5-4-3
R0--1-3- I-
R3:
R4
RSs 4-13
12:18
R3:
84. 15
15
RM 12-14
R55-12-14
R6:12-114m
Figure 4: Sankey diagram the evolution of correctly answering agents across 2,500 HLE questions over refinement rounds. Based on the distribution dynamics, we define five categories: all wrong (0), few correct (1-3), moderate correct (41 high correct (1), and all correct (15).
Under independence, Coverage(S) = 1- (1es(1 - p:). With positive correlations, Coverage(SS) shrinks. Fig. 3 shows the typical evolution dynamies of coverage, individual agent accuracy, and average scores over refinement rounds. Across all three benchmarks, coverage steadily declines, indicating that some correct answers mistakenly discarded during iterative refinement. Note that from round 4 to round in GPQA, coverage slightly lifts. When we extend the refinement to rounds 6 and 7. the coverage decreases again expected, confirming that the observed anomaly is transient. For HLE and AIME, the average score rises in the early rounds and then plateaus, while for GPQA it improves from round to 2 but later declines. Fig. 4 visualizes the dynamics over 2.500 HLE questions. From round to 2, the number of partially correct cases (few/moderatte/hghh correct) increases, while both all wrong and all correct cases decrease. This suggests that initial thought-sharing broadens exploration and promotes diversity. After round 2, however, partially correct cases diminish toward near zero, while all wrong and all correct cases grow. This indicates that agents gradually converge to single shared answer across rounds, either correct incorrect.
3.3 TERMINATION OPTIMAL ROUNDS AND FINAL ANSWER SELECTION
The observed evolution indicates that mound-by-roumd refinement not only improves answers in the initial rounds but also drives convergence, as each agent selects based on prior responses. However, due to the limited reasoning ability of LLMs, many correct answers are prematurely discarded. Beyond the early rounds, refinement rarely yields further accuracy gains and, in some cases, even degrades performance. Thus, identifying an effective termination strategy is essential for both robust performance and cost efficiency. Let A, denote acquired accuracy after round Define the expected marginal value of another round
A
E Ar41 A, signals up to round r]
(3)
5

---

Published as a conference paper at ICLR 2026
Stop at the first round r where A, I X marginal cost (here is increased inference costs). A practical termination strategy decides whether to stop based on the estimated future gain A,, which relies on round-r statisties such as (i) diversity collapse (coverage drop; rising agreement), (ii) vote margin between top answers, and (iii) answer entropy.
In TUMIX, our termination determination strategy is to query the LLM to decide whether to stop refinement and finalize answers based on the current round, with a minimum round number of 2. We find this termination strategy achieves nearly the same performance with only 49% of the inference cost. In Section 5.2, we explore other termination methods such as stopping once the majority answer stabilizes across two consecutive rounds or termination based on LLM confidence scores (Fu et al., 2025). but find only worse performance. After termination, we obtain the final answer through majority voting over the agents' responses, with Gemini- 2.5-Pro selecting the most consistent output.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Benchmarks For a comprehensive evaluation and comparison across methods, we conduct experiments on three representative benchmarks that demand extensive reasoning and planning, particularly the ability to effectively leverage Code Interpreter and Search. HLE (Phan et al., 2025) consists of 2,500 highly challenging questions spanning diverse subject areas, including mathematics, biology, engineering, computer science, and the social sciences. It designed as final, closed-ended benchmark of broad academic capability. We evaluate both its text-only and multimodal subsets. In the following sections, we primarily use HLE to study different mechanisms, as it contains large number of questions spanning diverse domains and is the most challenging benchmark. GPQA (Rein et al., 2024) is multiple-choice dataset authored by domain experts in biology, physics, and chemistry. We focus on its most widely used subset, GPQA Diamond, which contains 198 of the most challenging and carefully curated questions. Finally, AIME 2024& comprises 60 problems from the 2024 and 2025 AIME exams, a notoriously difficult high school mathematics competition. All reported results are averaged over three independent runs.
Baselines/TuMIX ablations and extensions models As shown in Appendix Table 12, we compare against the following methods: (1) Ma worityyVoooo e (Brown et al., (0) GSA (Li et al., 2025d); (3) Se 1f wwwfeeefeeiiiii (Ji et al., 2023); (4) SETS (Chen et al., 2025a); (5) Self-MoA (Li et al.. 2025a); (6) Symbolic-Mon (Chen et al., 2025b); (7) DET (Zhang et al., 2024); (8) SciMaster (Chai al., 2025). For baselines (1)-(4), we use the CS agent, which has full access to both Code Interpreter and Search and achieves relatively high first-round accuracy (Appendix Table 20). For baselines (5)-(7, we select agents following their original methods. For SciMaster: er, we retain the original prompts and agents to ensure consistency with published results. We match the total inference counts of all baselines to TUMIX by adjusting agent numbers and sampling repetitions for fair comparison. All the baselines have full access to Code Interpreter and Search. We evaluate TUMIX variants with different design choices, either to ablate framework components or to introduce improvements over existing TUMIX, as shown in Appendix Table 21. TUMIX+ uses higher inference costs to test scaling effects, while other variants consume nearly the same inference and token counts. We evaluate our methods on the reasoning LLM Gemini-2.5-Pro and Gemini-2.5-Flash.
Evaluation protocol Answers are evaluated against ground-truth solutions, with Gemini-2.5-Pro assisting in normal- izing answer formats when necessary or serving directly as the judge for answer comparison. In cases where the model outputs code as the final answer, we extract the code using predefined algorithms and execute it to produce the final result. To avoid infinite loops, all code execution whether during intermediate or final rounds is limited to 60 seconds. If execution exceeds this limit, "code runtime error" is returned to the model for regeneration in intermediate rounds; in the final round, the task is marked as a failure. We report success rate as the primary evaluation metric. In addition to task performance, we also analyze token usage and inference time for each method in later sections.
4.2 OVERALL BETTER PERFORMANCE
Table 2 shows that TUMIX outperforms all baselines, with average accuracy improvements of 2.0% and 5.9% over the best methods using Gemini-2.5-Pro and Semini-2.5-Flashh respectively. Its superior performance over methods without answer sharing (Se1 E-Reflecct in,, SETS) highlights the importance of answer sharing in multi-round test-time scaling. Comparisons with methods lacking multi-round refinement (Ma jority-Vote, Symbol c-MME, DEI, GSA) demonstrate the benefits of refinement, while comparisons with methods lacking agent diversity (Self f-MoA, SciMaster) confirm the value of diverse agents. The accuracy improvement of SciMaster on HLE is smaller than reported by the authors. We suspect this discrepancy arises from differences in tools, as their Search and Code Interpreter modules are not open-sourced. In Appendix Table 13, we report both mean and standard deviation values. The performance gains of TUMIX over the strongest baselines exceed the reported deviations, indicating stable and consistent improvements. Additional experiments with diverse L.LM types (Appendix Table 18) further confirm the robustness and generality of these results. To validate the statistical reliability of these improvements, we also perform
6

---

Published as a conference paper at ICLR 2026
Table 2: Experimental results baseline and proposed methods on HLE, GPQA, and AIME 24&25, Except for the singl-iinferenee w/o TTS and the scaled-up TUMIX+, all methods use comparable inference costs for scaling. For some methods, Gemini-2.5-Pro" HLE results are used to select agents within their agentic framework. In these cases, the method has prior knowledge of HLE and the results cannot be strictly regarded as test performance. Such cases are marked with in the HLE results. All the values are average of three repetitive runs.
METHODS
BASELINE METHODS
PROPOSED METHODS
SELF- RE-EEEETTTOON
MAJORITY VOTE
SYMBOLIC-MON MOE
TUMIX TUMIX-FIXEDI FIXEDR
TUMIX TUMIX-EVOLWY EVVOVE
ACCURACY
SELF-MOA
SCIMASTER
Wo ITS
TUMIX
TUMIX
SETS
DEI
GSA
GEMINI-2.5-PRO
HLE
21.6
28.4
29.3"
29.5'
29.1"
23.5
27.9
26.9
28.7
32.3
32.4
32.7
34.1
GPQA
84.6
84.9
85.5
86.7
86.0
84.9
85.3
86.9
85.8
87.9
86.8
88.1
88.3
AIME 24&25
87.3
94.3
94.7
94.7
95.0
88.3
94.7
94.1
93.7
96.7
95.6
96.7
96.7
AVE. NORM.
64.5
69.2
69.8
70.3
70.0
65.6
69.3
69.3
69.4
72.3
71.6
72.5
73.0
GEMINI-2.S-FLASN
HLE
9.7
17.9
18.2
18.5
19.3
10.4
18.5
18.0
17.4
21.2
20.9
21.9
23.1
GPQA
50.0
63.1
65.4
64.1
64.9
53.2
63.2
67.9
62.6
77.3
76.8
79.8
82.1
AIME 24&25
70.0
80.0
80.3
80.7
82.3
72.3
74.0
79.1
79.7
83.3
83.3
86.7
86.7
AVE. NORM.
43.2
53.7
54.7
54.4
55.5
45.3
51.9
55.0
53.2
60.6
60.3
62.8
64.0
two-tailed paired t-tests using repeated run scores, as shown in Appendix Table 14. Across nearly all benchmarks and models, the resulting p-values below 0.05, confirming that the improvements of TUMIX statistically significant.
5 DISCUSSION
5.1 AGENT DIVERSITY AND QUALITY ARE CRITICAL
To investigate the role of agent diversity and quality in TUMIX performance, compare groups of agents with varying levels of diversity and capability, as shown in Fig. 5 and Appendix Table 22. Under the same amount of refinement rounds and inferences, increasing the number of agents from 1 to 3 to 15 leads substantial improvements in both coverage and average score across rounds on HLE and GPQA, indicating that diversity significantly benefits performance. Moreover, comparing single strong agent with single weak agent (see Appendix Table 20, where CS achieves higher first-round scores than w/o TTS), we observe that higher-quality agents consistently yield better coverage and higher average scores.
Code Interpreter and Search increase answer diversity. In Fig. 6, we evaluate three settings where each agent group consists of three agents, each sampling five times per round. The groups differ their tool access: in Code Text, agents cannot access Search; in Bearch_Text they cannot access the Code Interpreter; and in Code Code-Search. Search Text, agents have full access to both. While the average agent quality (as measured by first-round scores in Appendix Table 20) is comparable across groups, the group with access to both Code Interpreter and Search achieves notably higher coverage and average scores. This result demonstrates that integrating complementary tools within agents enhances both reasoning and answer diversity, thereby facilitating more effective problem solving.
5.2 TERMINATION AND SELECTION METHODS
Achieving optimal performance at 49% cost. Tasks of varying difficulty require different numbers of refinement rounds, and excessive refinement can degrade accuracy (see Fig. 3b). Thus, effective termination strategy is essential to balance performance and cost. We evaluate two termination strategies (Sec. 3.3): 1) Term LLM, which queries the LLM decide when stop refinement, subject to minimum round constraint; and Term_ Rule, which stops once the majority answer stabilizes across two consecutive rounds, also with minimum round constraint. We vary the minimum number of rounds examine how performance evolves as the number of rounds increases in Fig. 7, and we compare their peak performance in Appendix Table 22. Term LLM achieves nearly the same peak accuracy unlimited refinement, but with substantially fewer rounds. On average, Term LLM retains optimal performance while requiring only 49% of the LLM inferences needed to obtain the final answer (LLM judging cost counted). The token costs are even less (approximately 46%), as the number inference tokens used in later rounds exceeds that of the first two rounds. This demonstrates the effectiveness of using -uudeee to determine when refinement is sufficient
7

---

Published as a conference paper at ICLR 2026
Coverage
Average
(Diamond) Coverage GPQA (Diamond) Average
(a)
(b)
(C)
{d)
50-
30.0
Average Score (%)
87
45
Coverage (%)
40
91
25.0
35
22.5
30
90
84
20.0
Round
3 3_Agents
15_Agents
1_Agent_ Strong
1_Agent Weak
Figure 5: Coverage and average score vs. rounds under varying agent diversity and quality. In 15 Agenttss,, all 15 varied pre-designed agents generate one answer each round. In 3 3_Agents, three strong agents (C+, CS and CSCges top-3 performed agents in round as demonstrated in Appendix Table 20), each samples 5 times per round. In I LAgeet.Stroog and l_Agent_ Weak, CS and w/o TTS sample 15 times, respectively.
Code Text
Search Text
-
Code Search Text
Random Selector
Term LLM
Majority Vote
Term Rule
(a)
HLE- Coverage
(b)
HLE Average Score
LLM Selector
Average Score (%)
30
32
Coverage (%)
Accuracy (%)
27
42
24
36
34
21
2
Round
Round
20-
Average Round
Figure 6: Comparison of groups all with three agents but either partial or full accesses to textual reasoning, coding, and search: Code Text (CoT, C, C+), Search_ TTxt (CoT, S, CSgz), and Code_Search,Texx (CSgs+ C+, and CSGga).
Figure 7: Comparison of refinement termina- tion and answer selection strategies for higher accuracy with lower costs.
and answers can be finalized. However, Term.LLM still requires minimum number of refinement rounds (set to two across all benchmarks). This is because we observe that LLMs tend to be overconfident and may terminate refinement early, even when additional refinement could improve performance. Appendix Section F specifically discusses the impacts of minimum refinement round on TUMIX performance and efficiency.
For answer selection, we compare three strategies: (I) randomly choosing one agent's 's answer, (2) majority voting, and (3) M.LM-based selection with L.LM-as-Selector. Fig. 7 shows that majority voting and L.L.-based selection consistently outperform random choice, especially in early rounds when agent answers diverge. However, once answers converge in later rounds, all selection methods yield similar results, and their impact becomes negligible. The multi-round refinement process is also a selection process. We also explore improved selection based on LLM token confidence (Fu et al., 2025), but observe no significant differences (Appendix Fig. 13).
5.3
HUMAN PRE- DESIGNED AGENTS VS. LLM GENERATED AGENTS
The
agents and their tool-ut use strategies are built on existing frameworks or intuition. To explore whether stronger agents can be discovered automatically, we query Gemini-2.5-PPr with current agent code examples (total input prompt in Appendix Table 6) and ask to generate full implementations of more diverse and high-quality ones, where the agent prompts and frameworks are all determined by LLMs. This yield 25 diverse agents beyond the 15 ousannde ones. From these, we retain the 15 that perform best in HLE with first-round answer generation. We then combine the 15 human- human-designed and 15 LLM-generated agents into a pool of 30, randomly sample groups of 15, and evaluate their average score and coverage (Fig. Sa). Compared to the baseline of 15 human-designed agents (gray triangle), many mixed groups achieve both higher average score and coverage. We select the top-3 groups based on the combined metric
Coverage
Average Score;
Combined Score, E[Covragee]
(4)
[Average Score]
8

---

Published as a conference paper at ICLR 2026
(a)
25,000
(b)
Coverage (Baseline)
Coverage (Top-3)
Average Score (Baseline)
Average Score (Top-3)
55%
HLE
GPQA (Diamond)
60
100
54%
Pert dondy tigiaai)
Coverage
Accuracy (%)
Accuracy (%)
85-
Top 3
20
80
20% 20%
21%
2
Average Score (9)
Round
Round
Figure 8: We evaluate the coverage and average of 25,000 15-agent combinations sampled from 30 agents (15 pre-designed and 15 LLM- generattedd). Baseline refers to the original 15 pre-designed agents. Top-3 refers to the three sampled combinations with the highest joint performance in coverage and average score.
As shown in Fig. 8b, these groups outperform the original TOMIX in both HLE and GPQA. This demonstrates that increasing agent diversity and quality improves effectiveness, and that LLM-generated agents hold strong potential for further enhancing TUMIX. Appendix Table 23 describes each generated agent, whose strategies differ substantially from the original ones beyond prompt variations. Appendix Table 24 presents the agents in each top-3 group. with roughly half overlapping with the original group.
Evolve agents in each round to enhance the diversity In all previous experiments, the agent set remained fixed across refinement rounds. We now investigate whether dynamically varying agent types per round can improve performance. As shown in Appendix Table 22, the variant TUMIX-Evol top-3 sets each round, performs slightly worse than the fixed variant TUMIX-EvOR ve across all three benchmarks. Agent quality may decline because useful specialized agents get replaced, reducing their ability interpret reflect others' answers. However, the effect is minimal, so we conclude that agent evolution has no meaningful impact.
which randomly selects agents from the
Impact of number of agent types We next examine the marginal benefit of increasing the number of agents in TUMIX. Agents are randomly sampled from the pool of 30, with each contributing one inference per round. To isolate this effect, we exclude termination and selection and report the evolution of peak average accuracy across rounds. As shown in Fig. 10, accuracy rises quickly when the number of agents is below 12, but gains become negligible thereafter. This indicates that beyond certain point, increasing agent types and inference budget yields little benefit. Under constrained set of tools, the marginal gains from additional agent types diminish to nearly zero because agent diversity and viable tool-use strategies are inherently limited, while the growing number of candidate answers makes round-by-roum. selection increasingly challenging. Based these results, decide to only include 15 agents in TUMIX to balance performance and cost.
5.4
SCALING CURVES: PERFORMANCE VS. COSTS
Gemini-2.5-P Inference
Gemini-2.5- Token
Peak Round Acc (%)
33
33
30
30
28
Score
Score
26
24
24
Self-MoA
22
TUMIX
DEI
Symbolic-MoF
21
10
15
20
25
150
Number of Agent Types
Inference Num.
Token Num.
Figure 10: Peak round accuracy versus number of agent types. Agent types randomly sampled from the 30 well-performing agents with three repetition. Fach agent infers once per round.
Figure 9: Scaling behavior of HLE scores relative to inference cost and total token count across different tool-augmented test-time scaling methods, where the token count includes both input and output tokens.
We compare the scaling behavior different tool-augmented test-time scaling methods in terms of inference and token costs. In TUMIX and Self Self-MoA, scaling comes from adding more refinement rounds; in GSA, DEI, and
9

---

Published as a conference paper at ICLR 2026
Symbolic-Woe from repeating inference; and in TOMIX-, from both. As shown in Fig. 9 and Appendix Fig. 12, TUMIX consistently outperforms other methods, achieving the highest scores with fewer inference steps and tokens. TUMIX+ pushes peak performance further by repeating inference four times across the first two refinement rounds, but at substantially higher cost and lower efficiency. Appendix Section G, we report tool latency, code execution time, and API cost, showing that all tool operations are relatively negligible compared to LLM inference time and that TUMIX maintains lower or comparable financial costs than baselines. Overall, test-time scaling demands far more inferences and roughly two orders of magnitude more tokens, seemingly unavoidable trade-off.
5.5 DIFFERENT TYPES OF LLMS AND MIXTURE OF LLMS TO ENHANCE PERFORMANCE
In Appendix Table 18, we extend our evaluation to four new base models: Claude-somnet--22225000 DeepSeek- RI, GPT-oss-120B, and Qwen3-32B. We compare TUMIX with both the base method (w/o TTS) and the strongest baseline (DEI). Across all three benchmarks and four base models, TUMIX consistently yields higher performance under comparable token costs, demonstrating its robustness and broad applicability across heterogeneous LLMs. We also investigate heterogeneous mixtures of models, where agents are powered by different LLMs. Specifically, we evenly split agents between GPT-oss- 20B and Qwen3-32B, which show similar capabilities in Appendix Table 18. As reported Table 3, mixed-agent configurations outperform their single-model counterparts across rounds, confirming that heterogeneity enhances reasoning diversity and collaboration. However, when models differ substantially in capability (e.g.. DeepSeek-RP combined with weaker models), performance degrades, suggesting that mixtures are most effective when participating models have comparable strength. Overall, these findings reinforce that diversity is critical, and TUMIX generalizes well across both model families and heterogeneous agent settings.
Table 3: Mixed-LLM vs. Single-LLM agents. (Top:) mixtures of comparable LLMs (GPT-oss-1208) and Qwen3-32B). (Bottom:) mixtures with large capability gap (DeepSeck-R) with GPT+Qwen).
Base
Setting
HLE
GPQA
AIME 2024&
Comparable-strengghi models (GPT-oss-1208 & Qwen3-32B)
GPT-oss-120B
Single
17.8
72.3
89.1
GPT-oss-
Mixed (GPT+Qwen)
19.8
75.5
90.6
Owen3-32B
Single
18.0
68.3
88.1
Qwen3-32B
Mixed (GeT++wen)
19.0
70.0
91.4
Capability- gap mixture (DeepSeek-R) with GPT+Qwen)
DeepSeek-RI
Single
29.0
81.2
88.3
DeepSeek-RI
Mixed (DeepSee
22.6
73.4
91.8
5.6 TUMIX APPLICATION TO OPEN-ENDED TASKS: SUMMARIZATION
To further evaluate the generality of our approach beyond structured reasoning benchmarks, we test TUMIX on open-ended, real-world tasks where correctness may be ambiguous. We conduct experiments on long- document summarization, a representative open-ended task requiring both comprehension and abstraction. We adopt two benchmarks from the SCROLLS suite (Shaham et al., 2022): (1) GovReporl (Huang et al., 2021), consisting of multi-page U.S. government reports (CRS/GAO), and (2) SummScreen-FDD (Glaser et al., 2022), comprising TV-episode transcripts summarized into human-written recaps. These datasets feature realistic, lengthy documents with human references, enabling reproducible and objective evaluation. We report standard ROUGE- 1/2/L F1 metrics (Lin, 2004), which measure unigram, bigram, and sequence-leve overlap. As shown in Appendix Table 19, TUMIX consistently improves ROUGE-1/2/L. scores compared to w/c TTS and DEI across both datasets using Geminii2255FFlan and GPT-oss-1200 120B as base models, demonstrating the robust adaptability of TUMIX open-ended, real-world tasks.
6 CONCLUSION
We introduce Tool-use Mixture (TUMIX), a framework that leverages diverse tool-use strategies to improve reasoning in LLMs. By coordinating multiple agents with complementary approaches to textual reasoning, coding, and search, TUMIX substantially improves performance across challenging benchmarks, including HLE, GPQA, and AIME. Our findings highlight that diversity and quality of agents, rather than scale alone, drive these gains. Furthermore. automatic generation of agents and principled termination strategies enable both higher accuracy and significant efficiency improvements, reducing inference cost by nearly half without sacrificing performance. This work demonstrates that structured diversity and selective refinement are key to maximizing the potential of tool-augmented LLMs.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This paper contributes to advancing Foundation Models by augmenting language models with Code Interpreter and Search tools via test-time scaling, which has strong potential improve performance and alignment with human preferences. However, such capabilities are inherently dual-use, the same techniques that augment models toward harmless outputs can, with minor changes, be misused to generate harmful content. While misuse is concern, we believe the broader societal benefits outweigh the risks.
REPRODUCIBILITY STATEMENT
For better reproducibility, include detailed descriptions of 15 pre-designed agents and 15 LLM-generated agents in Table and Table 23. The prompts of all agents are in Appendix Sec. C. The complete algorithm of TUMIX is illustrated in Appendix Sec. B. Our code and dataset will be made publicly available under open-source license following the acceptance of the paper.
LARGE LANGUAGE MODEL USAGE FOR WRITING
In this paper, provided to these models for grammatical correction and structural refinement, after which the output was verified and further edited when necessary. Their use was strictly limited to refinement; they were not employed to generate new content or references.
L.Mss-pecificallly Gemin n and ChatGPT-as general-purpose writing aids. Draft text
ACKNOWLEDGMEENS
The authors would like to thank Zifeng Wang for the helpful discussion and comments the work. The work also partly supported by ONR under Award N00014-22-1-2478 and MIT-IBM Watson AI Lab. This article solely reflects the conclusions of its authors and not the sponsors.
REFERENCES
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Arun lyer, Suresh Parthasarathy, Sriram Rajamani, B Ashok, and Shashank Shet. Codeplan: Repository-lvvel coding using 1lms and planning. Proceedings of the ACM on Software Engineering, ((FSE))675699, 2024.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, arXiv preprint arXiv:2407.11887, 2024.
Jingyi Chai, Shuo Tang, Rui Ye, Yuwen Du, Xinyu Zhu, Mengeheng Zhou, Yanfeng Wang, Yuzhi Zhang, Linfeng Zhang. Siheng Chen, et al. Scimaster: Towards general-purpose scientific agents, part i. x-master as foundation: Can we lead on humanity's 's last exam? arXiv preprint arXiv:2207.052-- 2025.
Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, Jinsung Yoon, and Sercan Ô Ank. Sets: Leveraging self-verificatio and self-correction for improved test-time scaling. arXiv preprint arXi::2501.19330 2025a.
Justin Chih- Yao Chen, Sukwon Yun, Elias Stengel- Tianlong Chen, and Mohit Bansal. Symbolic mixture-of. experts: Adaptive skill-based routing for heterogeneous reasoning. arXiv preprini arXiv:2503.0564/ 2025b.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12558 2022.
Yongchao Chen, Yilun Hao, Yueying Liu, Yang Zhang, and Chuchu Fan. Codesteer: Symbolic-augmented language models via code/text guidance. In Forty-second International Conference on Machine Learning.
Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. Scalable multi-robot collaboration with large language models: Centralized or decentralized systems? In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4311-4311 IEEE, 2024a.

---

Published as a conference paper at ICLR 2026
Yongchao Chen, Harsh Jhamtani, Srinagesh Sharma, Chuchu Fan, and Chi Wang, Steering large language models between code execution and textual reasoning. arXiv preprint arXiv:2410.03524, 2024b.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint vrXiv:2507.06261 2025.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms. arXiv preprint arXiv:2504.11550 2025.
Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. Deep think with confidence. arXiv preprint arXiv:2508. /52i:, 2025.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uni Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764-10799. PMIR, 2023.
Nicholas Glaser, Maxime Peyrard, and Robert West. Summscreen: A dataset for abstractive screenplay summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 8565-8580 Dublin, Ireland, 2022. Association for Computational Linguisties. URL acl-long 589.
www.oooy..ooooogo..o. org//022 2022..
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi. et al. Deepseck-r1: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 124448, 2025.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Jonathan May. Efficient attentions for long document summarization. arXiv preprint arXiv:2104.02112, 2021. URL
abss/ 22144.104.0 02112.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiiv2442.16772 2024.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating Ilm hallucination via self reflection. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 1827-1843, 2023.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.
Chengshu Li. Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator. arXiv preprint 2.Xiv:23/2.04473 2023a.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36: 51991-52008, 2023b.
Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin. Rethinking mixture-of-agents: Is mixing different large language models beneficial? arXiv preprint arXiv:2502.00674, 2025a.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprini vrXiv:2501.05366, 2025b.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383 2025c.
Zichong Li, Xinyu Feng, Yuheng Cai, Zixuan Zhang, Tianyi Liu, Chen Liang, Weizhu Chen, Haoyu Wang, and Tuo Zhao. Lims can generate a better answer by aggregating their own responses. arXiv preprint 25X0:22003.0404,, 2025d.
Chin-Yew Lin. Rouge: package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pp. 74-81, Barcelona, Spain, 2004. Association for Computational Linguistics.
12

---

Published as a conference paper at ICLR 2026
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang. and Graham Neubig. Language models of code are few-shot commonsense learners. arXiv preprint 1rX.v:2210.07128, 2022.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprini urXiv:2303. 17651, 2023.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's 's last exam. arXiv preprint arXiv:2501.14249, 2025.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang. Xiusi Chen, Dilek Hakkani- -ürr, Gokhan Tur, and Heng Ji. Toolri: Reward is all tool learning needs. arXiv preprint arXiv:2504. 13958, 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Uri Shaham, Yonatan Belinkov, Jonas Pfeiffer, Emanuele Bugliarello, Pasquale Minervini, Ryan Cotterell, Anders Søgaard, Felix Reutner, Andreas Rücklé, Ivan Vulic, Hinrich Schütze, Iryna Gurevych, Yulia Tsvetkov, Tom McCoy, Yonatan Bisk, and Gabriel Stanovsky. Scrolls: Standardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 11892-11912, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https: //aclanthology org/ /2///2222 2022. emn p-ma------- i n. 8233.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingehuan Zhang. YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Oguzhan Topsakal and Tahir Cetin Akinci. Creating large language model applications utilizing langchain: A primer on developing IIm apps fast. In International Conference on Applied Engineering and Natural Sciences, volume 1, pp. 1050-1056, 2023.
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.00 04692, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, al. Chain- of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
Yuxiang Wei. Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing Ilm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502. 18449, 2025.
Nathaniel Weir, Muhammad Khalifa, Linlu Qiu, Orion Weller, and Peter Clark. Learning to reason via program generation, emulation, and search. arXiv preprint arXiv:2405.163377 2024.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint arXiv:230s.08155, 2023.
Kevin Yang. Yuandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4393-4479, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, et al. Diversity empowers intelligence: Integrating expertise of software engineering agents. arXiv preprint arXiv:2408.. 240800000,,,, 2024.
13

---

Published as a conference paper at ICLR 2026
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.0793 2023.
14

---

Published as conference paper at ICLR 2026
APPENDIX-TUMIX: MULTI-AGENT TEST-TIME SCALING WITH TOS MIXTURE
A Example of GPT-5 failure in code/text decision
B Algorithm of TUMIX
17
C Prompts of TUMIX
18
D Baseline methods
19
E Statistical significance of tested results on TUMIX and baseline methods.
20
F Impacts of minimum refinement round on TUMIX performance and efficiency.
22
G Analysis of runtime stability and financial efficiency.
22
H Experimental results w/o TTS, DEI, TUMIX with four different base models.
23
Performance of TUMIX open-ended summarization tasks.
24
Agent accuracy and coverage over multi-round answer refinement on HLE
25
K Illustration and experimental results of TUMIX variants
26
L New agents in TUMIX completely designed by Gemini-2.5-Pro automatically
27
M Scaling behavior of Gemini-2.5-flas
28
LLM token confidence of generated responses
15

---

Published as conference paper at ICLR 2026
A
EXAMPLE OF GPT-5 FAILURE IN CODE/TEXT DECISION
(a)
(b)
(c)
ChatGPT 5>
ChatGPT
ChatGPT
?
Solve:
Solve:
Solve: with code generation. No Textual reasoning
5.9 x 5.11
5.9
5.11
generation
We want to solve:
Thought seconds
X
X
5.9
5.11
0.23 (+)
= -0.21
Subtract 5.11 from both sides:
also generated code solve programmatically (using Python's subtraction) the calculation
Analysic
5.11
Pythor
confimms the result. (+)
Now compute carefully:
berline
x 12.aypoeeaa
Analysis
Since 5.90 5.11, the result negative:
Tooe decimal Legaet becinal, geteonts
scbuncer
Tecrwase prantictse juan to cas
Decamall (5.77 Decomal15, 15..111 5.1212
Paesf
De 79000000000000001
Resuit
Figure 11: Example of GPT-5 failure in code/text decision. In this case, the question is incorrectly solved with textual reasoning (a) but can be easily addressed through code generation (c). However, GPT-5 remains overconfident in textual reasoning, relying it even when prompted to use code, despite the generated code already yielding the correct solution (b).
16

---

Published as conference paper at ICLR 2026
B ALGORITHM OF TUMIX
Algorithm TUMIX: Multi-Agent Test-Time Scaling (answers only)
Require: question qi agent pool S (8,....,KK) 15 pre-designed agents (Code Search / Dual-Tool variants)
Require: minimum rounds Tin = 2: maximum rounds Tmaxi tool-interaction budget Booo 5; code time limit
T 60s
1: Au 4 2
answers from prior rounds
for 1,2,.... Tmax
// Round-r message passing: each agent reads y + prior answers
parallel for each & S
P. BUILDPROMPT(Q.-. Ap-1))
concatenate q with all answers from prior round b tool-augmented reasoning (Alg. 2)
V. + AGENTCALL(s: pÝ,, Reool, T)
A, {U, E S}
7: if 2 Tmin and LLMTERMINATE(Q,.CC A,) = STOP then break
9: end if
10: end for
11: at < MAJORITYVOTE(A.)
12: return a*
Algorithm 2 AGENTCALL(S.pp Root T): tool-augmented reasoning for agent s (returns answer only)
1: h b- while b < Rwol do
h is the running context; counts tool interactions
+- LLMGENERATE(s// h)
agent with its strategy/prompt
4: if contains final answer y then return
else if o proposes code and allows Code Interpreter then
< EXECUTECODE(O.7 if RURTIMEERROOR()) then + "Runtime error:" r; b f- b+ 1; continue else
hard limit T 60s; capture stdout/plots/tiles
8:
10:
11:
+ "Code result:" r; + 1; continue
12:
end if
13:
else if issues search query and s allows Search then
14:
E + WEBSEARCH(()
supports gs/1im/com variants
15:
h + "Retrieved evidence:" E; f- + 1; continue
16:
else
17: h + "Continue reasoning with current context."
encourage self-reflection
18: end if
19: end while
20: LLMGENERATE(S,, h))
budget exhausted; force decision
21: return
17

---

Published as conference paper at ICLR 2026
C
PROMPTS OF TUMIX
Table Prompt for answer refinement based on all agent answers the previous round.
Task:
Decide the final answer based on the following answers from other agents.
Question:
(question)
Candidate answers from several methods:
(joined answers)
Based on the candidates above, analyze question step by step and try list all the careful points. end of your response, directly output the answer to the question with the format <<<answer content>>>,
Table 5: Prompt for LLM-as-Judge of refinement termination.
Task: Carefully assess whether the answers below (enclosed by <<< >>>) show clear and strong consensus, or if another round of reasoning is needed improve alignment.
IMPORTANT: If there are any differences in reasoning, phrasing, emphasis, conclusions, or interpreiation of key details, you should conservatively decide to continue refinement.
The current round number is (round.nun). Note: Finalizing before round 3 is uncommon and discouraged unless answers fully aligned in both logic and language.
Question:
(question)
Candidate answers from different methods:
(joinedLanswers)
Instructions:
1. Identify any differences in wording, structure, or logic.
2. Be especially cautious about subtle variations in conclusion emphasis.
3. Err on the side caution: there's any ambiguity or divergence, recommend another round.
Output your reasoning first, then conclude clearly with <<<YES>>> if the answers are highly consistent and finalization safe, or <<<N0>>> if further refinement is needed.
Table 6: Prompt for Gemini-2.5-Pro synthesize diverse and high-quality agents based the code of human pre-designed agents.
Task:
Generate new. diverse, and high- high-quality agents based on the full code of existing agents. You have full access to tools Code Interpreter and Search. Output the complete implementation code for each new agent. Separate the agent framework code and the prompt code into files.
Instruction: Generate agents that totally different in reasoning and tool-use strategy, not just limited to prompt optinization. Existing agent codes:
(agent code}
Output:
18

---

Published as a conference paper at ICLR 2026
Table 7: Head prompt for CoT Agent.
Analyze the question step by step and try to list all the carcful points.
Then try to acquire the final answer with step by step analysis.
In the end of your response, directly output the answer the question.
not output the code execution.
Table 8: Head prompt for CoT-Code Agent.
You are a helpful Al assistant. Solve tasks using your coding skills.
In the following cases, suggest python code (in python coding block) for the user exccute.
Don't 't' include multiple code blocks in one response, only include one the response.
Do not ask users copy and paste the result. Instead, use the ' 'print' function for the output when relevant.
Think the task step by step if you need to. If a plan not provided, explain your plan first. can first output your thinking steps with texts and then the final python code.
Remember the final code you still need output each number or choice the final print!
Start the python block with python
Table Head prompt for Code Agent.
The User asks a question, and you solve You first generate the reasoning and thinking process and then provide the User with the final answer. During the thinking process, "*you can generate python code** for efficient searching. optimization, and computing with the format of starting the python block with python. * + code query must involve only a single e scrípt that uses 'print' funct on for the output. the generation. Then, the code interpreter platform will execute the code and return the execution output and error. Once you feel you are ready for the final answer, directly return the answer with the format <<<answee content.> >>>> at end of your response. Otherwise, you can continue your reasoning process and possibly generate more code query to solve the problem.
Once the code script complete, stop
Table 10: Head prompt for Dual Tool Agent.
The User asks question, and you solve it. You first generate the reasoning and thinking process and then provide the User with final answer.
During the thinking process, you can generate python code for efficient searching, optimization, and computing with the format of starting the python block with python. + code query must involve only a single script that uses 'print' function on for the output **.,, Once the code script is complete, stop the generation. Then, the code interpreter platform will execute the code and return the execution output and
If you lack the related knowledge, you can the Google Search Tool search the web and get the information. You can call a scarch query with the format <search>your search query</ searchh> c.g. <search>Who is the current pres iden of ssachhearch> The searched results will be returned between <infornation ton> and s/information> Once the search query is complete, stop the generation. Then, the search platform will return the searched results.
you need search the web, not generate code in the same response. Vice versa. You can also solve question without code and searching, just by your textual reasoning.
Once you feel you are ready for the final answer, directly return the answer with the format <<<answer content>>> at the end of your response. Otherwise, you can continue your reasoning process and possibly generate more code or search queries to solve the problem.
BASELINE METHODS
19

---

Published as a conference paper at ICLR 2026
Table 11: Head prompt for Guided Agent.
You are guiding another M to solve task. You will be presented with task that can solved using textual reasoning. coding, and web searching. Sometimes the L may need extra help to solve the task, such as generating code or searching the web. Then must follow the rules below for both query and return answer:
During the thinking process, you generate python code for efficient searching, optimization, and computing with the format of starting the python block with python. code query must involve only single script that uses 'print' function for the output... Once code script complete, stop the generation. Then, the code interpreter platform will execute the code and return the execution oulput and error.
If you lack the related knowledge, you can use the Google Search Tool to search the weh and get the information. You can call search query with the format of <search>your search query</ search>, e.g., <search>Who is the current presi dent of uS?</searcns </ s/information. Once the search query complete, stop the generation. Then, the search platform will return the searched results.
The searched results will be returned between <i foratt on> and
If you need to scarch the web, do not generate code in the same response. Vice versa. You can also solve the question without code and searching, just by your textual reasoning.
Once you feel you ready for the final answer, directly return the answer with the format << <<<answer content>>> at the end of your response. Otherwise, you can continue your reasoning process and possibly generate more code or search queries to solve the problem.
Your goal is to determine which method will be most effective for solving the task. Then you generate the guidance prompt the T to follow the next round. The final returned guidance prompt should be included between <<< and >>>, such as <<<You need to generate more complex code to solve...>2
Now, here is the task:
Table 12: Baseline methods compared against TUMIX.
Method Handle
Type
Description
Ma Majority-Vote
Voting
single agent runs multiple parallel inferences, with the final answer decided by majority voting, without sharing intermediate results. Uses CS agent.
GSA
Aggregation
Similar to Ma Majori ty-Vote, hut the same I.I.M generates a new response condi- tioned on multiple samples. Uses CS agent.
Self-RefLection Iterative Refinement
single agent iteratively refines its answer by reflecting on past responses (up to 10 accessible per round: varied or 15 in experiments, with no performance difference). Uses CS agent.
SKTS
Multi-Trial Voting
The same L.M performs multiple self-refiection trials, and the final answer is chosen by majority vote. Uses CS agent.
Best-Agent Selection
Selects the best-perforning agent among 15 candidates for parallel sampling, answer sharing, and multi-round refinement. Adapted to select the best agent within the same LLM instead of the original setting, selecting the best LLM across different LLMs. Categorizes questions (e.g., algebra, probability, coding, biology), pre-tests the top agents per category, and LLM judges the question category and assigns test questions to these top agents for sampling and aggregation.
Symbolic-MoE
Expert Selection
DEI
Committee Heuristic
Selects the top agents, generates multiple answers via repetitive sampling, and then uses a predefined agent committee with heuristics to select the best answer. Samples the same pre-designed tool-use agent five times, then employs other agents to critique, refine, and aggregate the answers. Original prompts/agents retained.
SciKaster
Critic and Refine
E STATISTICAL SIGNIFICANCE OF TESTED RESULTS ON TUMIX AND BASELINE METHODS.
Paired significance testing. As for the method to calculate p-value, we perform two-tailed paired 1-test their run-wise score differences d, x4--- yt. The test statistic is
d
(d, d)2
S4
Sd/ VT
n -
with df degrees of freedom. The corresponding p-value is p 2( F((ld] n 1)),
20

---

Published as conference paper at ICLR 2026
Table 13: Experimental results of baseline and proposed methods HLE, GPQA, and AIME 24&25. Except for the single-inference w/o TTS and the scaled-up TUMIX+, all methods use comparable inference costs for scaling. For some methods, Gemini-2.5-Pro" HLE results are used to select agents within their agentic framework. In these cases, the method has prior knowledge of HLE and the results cannot be strictly regarded as test performance. Such cases are marked with in the HLE results. All the values are the average of three repetitive runs. Here we report both the mean values and standard deviations, covering around 68% confidence intervals.
BASELINE METHODS
PROPOSED METHODS
SYMBOLIC-MOL
ACCURACY %
SCIMASTER
W/o W/OTTS ITS
SELF-MOK
TUMIX
TUMIX+
DEI
GSA
CEMINI-2.5-PRO
HLE
0.3
29.3* 0.5
29.5' 0.4
29.1
0.6
26.9
0.7
0.4
34.1 1
GPQA
84.6
8 0.4
86.7
0.5
0.6
8.6
85.8 0.6
87.9
0..90.44
88.3
1
0.6
AIME
$7.3.0.4
94.7 0.5
94.7
9..0
0.6
93.7 1.04
96.7 1.5
96.7
0.4
AVE.
64.5 0.4
69.8
0.5
70.3
70.0 0.5
69.3 0.5
69.4 0.6
72.3
0.4
73.0
0.5
GEMINI-2.5-FLASH
HLE
9.7
18.2 0.6
+.7
0.8
18.0 0.6
1.5
21.2 1 0.5
23.1 0.7
GPQA
50.0
0.6
0.7
6.7
67.9 0.7
6.7
77.3 0.7
82.
AIME
0.7
8.7
80.7
0.6
0.6
0.6
83.3 = 0.7
86.7 0.7
AVE. 4 0.6
54.7 = 0.6
54.4
55.5
55.0 15.0=00 0.6
53.2 + 0.6
60.6 = 0.6
64.0 + 0.7
where Fú is the cumulative distribution function of the distribution. We consider differences statistically significant when p 0.05.
Table 14: Statistical significance of TUMIX vs. baselines (DEL, SciMaster, Symbolic-MoE, GSA, Self-MoA) on each dataset. P-values come from two-tailed paired f-tests computed over run-wise score differences (d: 2, y:) across 3 runs. A indicates p<0..5 (statistically significant), while denotes non-significant differences.
Task
TUMIX
DEI
vs. SciMaster
vs. Symbolic-MoE
vs. GSA
vs. Self-MoA
p-value
Sig.
p-value
Sig.
p-value
Sig.
p-value
Sig.
p-vuuue
Sig.
Gemini-
2.5-PPr
HLE
32.3 + 0.4
< 0.01
10
0.01
0.01
0.01
GPQA
87.9 + 0.4
0.014
/
0.084
0,034
0.010
0.002
AIME
0.011
0.01
0.012
0.002
0.008
Semmini-25--Flae
HLE
21.2 +: 0.5
0.033
< 0.01
< 0.01
< 0.01
0.01
GPQA
77.3 + 0.7
< 10-1
10-1
10-1
< 10-1
< 10-1
AIME
83.3 + 0.7
0.135
< 0.01
0.014
0.003
0.006
21

---

Published as a conference paper at ICLR 2026
F IMPACTS OF MINIMUM REFINEMENT ROUND ON TUMIX PERFORMANCE AND EFFICIENCY.
To mitigate premature termination our semi-adaptive refinement process, we enforce a minimum number refinement rounds. Table 15 presents an ablation study evaluating this hyperparameter. We observe that enforcing at least two refinement rounds substantially improves performance across both Gemini-2.5-P and Gemini-2.55Flaas models. Increasing the minimum round number beyond two provides negligible gains in HLE scores but incurs a steep rise in token consumption. Hence, we adopt a minimum two rounds as the default setting, achieving the best trade-off between performance and computational efficiency.
Table 15:
Ablation study on the minimum refinement round requirement in our semi-adaptive termination mechanism. Setting the minimum round number to 2 consistently improves HLE scores across models while maintaining a favorable token cost. Increasing the round number beyond yields marginal gains substantially higher computational costs.
Minimum Round Number
2
4
HLE Score (Gemini-2.5-Pro)
31.3
32.2
32.3
32.1
Token Cost (Gemini-2.5-Prro, x1k tokens)
29.6
285
350
570
HLLE Score (Gemini-2.5-Flan
21.7
23.0
23.1
23.0
Token Cost (Gemini-2.5-FFaash xlk tokens)
22.7
230
300
522
G ANALYSIS OF RUNTIME STABILITY AND FINANCIAL EFFICIENCY.
Regarding practical runtime, opted for more stable and reproducible metric. Wall-clock time can be highly volatile and difficult to compare fairly, as is affected by factors such as network latency, server load, and specific hardware conditions. To ensure standardized and hardware-agnoste comparison, the original paper reports API token counts and LLM inference numbers as direct proxies for computational cost. This approach allows our evaluation to reflect the intrinsic efficiency of the methods themselves, independent of experimental environments.
Although overall wall-clock time is not a reliable performance metric, in Table 16 we additionally report the average code exccution and search latency. While these times are influenced by the content of generated code and search queries, we find that the average latencies are very similar across TUMIX and baseline methods. The average runtime per query for all tools remains under seconds, which is negligible compared LLM inference time (typically over 50 seconds for Gemini-2.5-Pro). Since each tool use corresponds to one LLM inference, this confirms that tool-use latency is not computational bottleneck.
Table 16: Average tool latency per query. While tool latency varies slightly by task, all remain well below the typical LLM inference time, indicating that tool invocation overhead is negligible.
Tools
Average Runtime (seconds)
Code Interpreter Execution Google Search API
2.3
1.2
Geminni-255PPro Search Gemini-2.5-Fiie Search
6.9
4.5
We also report the average financial API cost per sample (including input/output tokens, Google Search API, and Gemini Search API) for TUMIX and strong baseline methods in Table 17. As shown, TUMIX incurs lower or comparable costs relative most baseline methods, despite achieving higher performance and efficiency.
These analyses demonstrate that TUMIX achieves strong computational and financial efficiency while maintaining superior performance compared baseline methods. The inclusion of latency and cost analyses further supports the practicality of our approach.
22

---

Published as conference paper at ICLR 2026
Table 17: Average financial API cost per sample in HLE//PQA/AIME (in USD). TUMIX achieves competitive
lower costs than baseline methods while maintaining superior performance and computational efficiency.
Method
Gemini-2.5-Pro
Gemini-2.5-Flash
TUMIX
1.70
0.82
DEI
1.59
0,84
SciMaster
1.93
0,92
Symbolic-Man
2.19
0,99
GSA
2.25
0,99
Self-MoA
2.23
0.96
H EXPERIMENTAL RESULTS OF w/o TTS, DEI, TUMIX WITH FOUR DIFFERENT BASE MODELS.
Table 18: Experimental results of w/o TTS, DEI, TUMIX with four different base models.
Model
HILE
GPQA
AIME 2024&&2025
Cladee-somee-42222555444
8.2, 15.8, 21.8
44.6, 61.4, 72.3
34.4, 55.0, 70.8
DeepSeek-RI
15.2, 23.7, 29.0
74.7, 78.0, 81.2
69.3, 85.0, 88.3
GPT-oss-120B
13.6, 15.4, 17.8
66.3, 70.1, 72.3
60.2, 82.7, 89.1
Qwen3-32B
13.1, 16.0, 18.0
54.6, 64.1, 68.3
59.6, 84.4, 88.1
23

---

Published as conference paper at ICLR 2026
PERFORMANCE OF TUMIX ON OPEN-ENDED SUMMARIZATION TASKS.
Table 19: Performance TUMIX on open-ended summarization tasks from SCROLLS. TUMIX consistently out- performs both the baseline method without test-time scaling w/a TTS and the best baseline DEI across datasets and models, achieving the best ROUGE- 1/2/L Fl scores.
Metric
Model
w/o TTS
DEI (Zhang et al., 2024)
TUMIX
ROUGE-1 (GovReport)
Gemini-2.5-Flash
0.440
0,458
0.466
GPT-oss-120B
0.447
0,462
0.468
ROUGE-2 (GovReport)
Gemini-2.5-Flash
0.162
0.176
0.182
GPT-oss-120B
0.152
0.169
0.175
ROUGE-L (GovReport)
Gemini-2.5-Flash
0.204
0,216
0.222
GPT-oss-120B
0.203
0,214
0.220
ROUGE-1 (SummScreen-FDD
Gemini-2.5-Flash
0.282
0.315
0.323
GPT-oss-1200
0.273
0.304
0.310
ROUGE-2 (SummScreen-FI)
Gemini-2.5-Flash
0.056
0.064
0.063
GPT-oss-120E
0.051
0.058
0.060
ROUGE-L (SummSereen-FFD
Gemini-2.5-Flash
0.137
0.149
0.153
GPT-oss- 120B
0.134
0.146
0.149
24

---

Published as conference paper at ICLR 2026
AGENT ACCURACY AND COVERAGE OVER MULTI-ROUND ANSWER REFINEMENT ON HLE
Table 20: Accuracy each agent, average accuracy, and coverage across rounds for HLE. Dual-Tool Agent, Gu iedd Agent, and Gui ded Agent+ have three variants with different search strategies: Google Search API (gs), inherent LLM search function (11m), or their combination (com).
Humanity's Last Exam (HLE)
RD 1
RD 2
RD
RD 4
RD 5
RD 6
Coverage
51.92
44.20
43.48
37.04
34.85
33.87
Average
21.13
28.72
30.37
31.70
32.16
32.37
w/o TTS
20.32
28.08
30.88
31.60
32.18
32.36
COT Agent (CoT)
20.84
28.16
30.40
31.12
32.30
32.48
CoI-Code Agent (CoTcode)
18.36
28.28
31,40
31.52
31.96
32,40
Search Agent (S)
21.72
29.04
28.84
32.08
32.08
32.20
Code Agent (C)
21.16
29.68
31.40
31.96
32.12
32.36
Code Agent+ (C+)
22.96
29.00
31,44
31.88
32.20
32.40
Dual-Tool Agent (CSgs)
22.96
28.60
30.84
31.72
32.24
32.36
Dual-Tool Agent (CS)
21.36
28.36
31.28
31.20
32.48
32.32
Dual-Tool Agent (CSecm)
20.76
28.56
30.36
31.44
32.32
32.48
Guided Agent (CSGgs)
22.04
28.72
29.96
32.24
32.00
31.96
Guided Agent (CSG11m)
21.20
28.64
29.20
31.52
32.16
32.32
Guided Agent (CSGcam)
20.76
28.92
29.88
31.88
32.20
32.40
Guided Agent+ (CSG+
20.56
29.32
30.36
31.92
31.84
32.52
Guided Agent+ (CSG+1am)
21.56
28.80
29.20
31.64
32.16
32.52
Guided Agent+ (CSG+ com)
20.44
28.68
30.08
31.72
32.28
32.48
25

---

Published as conference paper at ICLR 2026
K
ILLUSTRATION AND EXPERIMENTAL RESULTS OF TUMIX VARIANTS
Table 21: TOMIX framework and variants, designed to ablate core components.
Method Handle
Component Ablated
Description
TUMIX
Main Method
(Default) Uses LLM query determine the optimal termination round (min. 2) and majority vote for final selection.
TUMEX-Rule
Termination
Replaces the LLM-quea termination with a rule: stops when the majority answer stabilizes across consecutive rounds.
TUMEX-Fixed
Termination
Replaces smart termination with fixed 5-round limit, followed by ma- jority voting for selection.
TUMEX-FixedR
Termination & Selection
Uses a fixed 5-round limit, followed by random selection.
X-Rvolve
Agent Composition
Replaces the 15 human-designed agents with static group of top- performing, LLM-generated agents for each refinement round.
TUMCX-EVOlveD
Agent Composition
Extends the above by dynamically sampling top-3 Evolved Agent groups for each refinement round.
agent group from the
TUMCX-Single
Agent Diversity
Ablates diversity by replacing the 15 distinct agents with single agent type from the CS family.
TUMEX-Three
Agent Diversity
Reduces diversity by using only three agent types (CS, C+ and CSO). each sampled 5 times per round.
TUMIX+
Inference Scaling
Extends 'TUMIX' with test-time scaling, running four inference passes per agent at different temperatures for the initial two rounds.
Table 22: Experimental results of TUMIX variants. All the values are the average of three repetitive runs.
METHODS
TUMIX VARIANTS
TUMIX- EUOLVEE EVOLVED
SUCCESS RATE %
TUMIX SINGLE
TUMIX- THREE
TUMIX- FUMIX-FIXEDR FIXEDR
FUMIX-EVOLVE
TUMIX-FIXED
TUMIX-RUIE RULE
TUMIX+
TUMIX
GEMINI-2.5-PRO
HLE
32.3
29.0
30.2
32.4
32.4
32.4
32.7
32.1
34.1
GPQA
87.9
86.1
86.6
86.8
86.7
87.7
88.1
87.4
88.3
AIME 24&25
96.7
95.0
95.3
95.6
96.7
96.7
96.7
96.1
96.7
AVE. NORM.
72.3
70.0
70.7
71.6
71.9
72.3
72.5
71.9
73.0
SEMINI-2.5-FLANE
HLE
21.2
18.2
18.6
20.9
20.8
21.3
21.9
21.3
23.1
GPQA
77.3
65.8
67.1
76.8
77.1
77.4
79.8
78.3
82.1
AIME 24&25
83.3
80.6
81.2
83.3
83.3
83.3
86.7
86.7
86.7
AVE. NORM.
60.6
54.9
55.6
60.3
60.4
60.7
62.8
62.1
64.0
26

---

Published as conference paper at ICLR 2026
L NEW AGENTS IN TUMIX COMPLETELY DESIGNED BY GEMINI-2.5. 2.5-N- AUTOMATICALLY
Table 23: Summary of 15 LLLM-generated agents, categorized by their framework characteristics.
Full Name
Short Name Description
Iterative Agents (Multi-turn conversational frameworks)
-Lan-Verity-Reiine
PVR
Iterativlyy plans, executes action (code search), and refines based checker feedback.
Sear rechhhenCCode
S-+C
Enforces a search-first, then code execution sequence an iterative loop.
CodeThenSearch
C-+S
Enforces pode-first. then search execution sequence an iterative loop.
Constrraintrrrue--ooleer
CPeetv
Iteratively prunes the solution space using constraints, guided by checker and tools (code/search).
MonteCarlo-verify
MCV
Uses Monte Carlo sampling via code to find likely answer and then deterministically verifies it.
Debate-CrossExem
DCE
Simulates Proposer/Skeptic debate guide tool use, with a checker cross-examination.
Mulld:llp-SSerrbbhkkggeettt
Sa +C
Enforces at least two sequential search actions before allow- ing any codc execution.
ODD-Code-Solver
TDDsoiv
TDD agent that lists tests, writes code to pass them, and uses a checker for iterative refinement.
Sequential Agents (Few-shof, non-conversational frameworks)
SearchThenAnswer
S-+A
A two-step agent that mandates a single search before formulating the final answer.
PlanThenCode
P-+C
A two-step agent that first generates plan, then single code block execute it. A three-step agent that generates a text answer, validates it with checker. and then refines it. Explicitly selects one tool (Search, Code, or Text) in the step. then finalizes.
VerifierRefine
VR
Toolšelecter
lypothesiserruer-code
EP code
Generates code to enumerate and prune solution hypotheses based on problem constraints.
DualBearch-Consensuss
Issues two distinct search queries and then synthesizes the results into consensus answer.
TDD-CodeThenFixx
TDDrix
A Test-Driven Development approach that writes tests and code, then generates a fix if tests fail.
27

---

Published as conference paper at ICLR 2026
Table 24: Comparison of original agent group and top-3 agent group used in TUMIX, each with 15 agents, either pre-designed or LLM-generated.
Original
Top-3-1
Top-3-2
Top-3-3
w/o TTS
typothesisfrunez-code
coe-Code-Solver
w/o TTS
Agent
CoT Agent Plan-verify-Refine
CoT Agent
CoT Agent
CoT-Code Agent
CoT-Code Agent
CoT-Code Agent
Search Agent
Search Agent Code Agent SearchThenCode
Search Agent
SearchThenCode
Code Agent
Code Agent Code Agent+ SearchThenCode
TDD-Code-Solver
Code +gent+
Ripotiaesistrruer--odde
Dual-Tool Agentę
Dual-Tool Agentas
bualsearcch-coneensuu
Dual-Tool Agentiin
ConstraintPrune-Sun
Plan-Verify- Refine
MonteCarlo-Veriffy
Dual-Tool Agent.on
Montecarlo-Veriiy
Dual-Tool Agenteon
ConstrrainGPruue--Solleee
Guided Agent,s
Guided Agentss Guided Agentiin
Guided Agent
Debate-CrossExam
Guided Agentiin
Guided Agentiı
Guided Agent1in
Guided Agent.m
Debate-CrossExam
Guided Agentem
Guided Agent.m
Guided Agent
Guided Agent SearchThenAnswer
MonteCarlo-Verit fy Guided Agent*ia
Guided Agentita
Guided Agenthin
Plan-Verišy-Refirr
Guided Agenttcon
Dualseaarch--Cnneesuu
Dual Saahseaacc--oeeeeuuu
Guided Agent
M SCALING BEHAVIOR OF GEMINI-2.5-FLASS
Gemini-2.5-Flan Inference
Gemini-2.5-Flash Token
21
21
Score
18
Score
18
15
15
12
TUMIX
Self-MoA
GSA
TUMIX +
DEI
Symbolic-MoF
50
100
150
10a
10a
10
Inference Num.
Token Num.
Figure 12: Scaling behavior of HLE scores in Gemini-2.5- relative to inference cost and total token count across different augeeneed test- time scaling methods, where the token count includes both input and output tokens.
28

---

Published as conference paper at ICLR 2026
N
LLM TOKEN CONFIDENCE OF GENERATED RESPONSES
Confidence Distributions (Correct vs Wrong)
Correct answers
0.30
Wrong answers
0.25
0.20
Ratio
0.15
0.10
0.05
0.00
0.75
0.80
0.85
0.90
0.95
1.00
Answer Token Confidence
Distribution of LLM response confidence for correct and wrong answers. The response confidence is calculated based on the average token probability of the whole generated response. Here we use the responses of agent CoT as representative, as we find the distribution characteristics are very close among different agents.
Figure 13:
29


Reviews:
[{"review_id": "dew99ReYtY", "reviewer": "Reviewer_FcrL", "summary": "This paper presents TUMIX (Tool-Use Mixture), a framework for test time scaling that integrates multiple LLM agents equipped with distinct tool use strategies such as textual reasoning, code execution, and search, to enhance reasoning capabilities. The framework operates by running 15 diverse agents in parallel, iteratively refining answers across multiple rounds and adaptively halting based on confidence estimation. The authors report consistent accuracy gains of up to 3.55% over strong baselines such as Self-MoA, Symbolic-MoE, and DEI on three reasoning benchmarks (HLE, GPQA, and AIME) using Gemini-2.5-Pro and Gemini-2.5-Flash. The work’s practical contribution lies in the combination of agentic diversity, tool integration, and cost effective refinement termination.\n\nThis work represents a thorough and thoughtful empirical study that advances understanding of tool augmented, multi-agent reasoning. It delivers consistent performance improvements and introduces practically relevant techniques such as adaptive termination and LLM driven agent generation. However, the methodological simplicity of the refinement mechanism, lack of theoretical or architectural novelty, and uncertain generalizability prevent it from reaching the standard expected at a premier venue. The contribution is valuable as a rigorous empirical report and could serve as a foundation for more principled frameworks combining structured agent communication with tool integration. Strengthening the theoretical underpinnings, expanding evaluation to multiple model families, and providing formal statistical validation would substantially improve the paper’s readiness for acceptance.", "questions": "Could the authors clarify why concatenating all agent outputs yields such robust gains? Have they explored more structured synthesis mechanisms (e.g., critique based debate or weighted aggregation)?\n\nHow sensitive is TUMIX to the underlying LLM’s capacity? Would smaller or open source models (Llama, Mistral) still benefit meaningfully from this framework?\n\nCan the authors provide confidence intervals or significance tests to confirm that improvements on smaller benchmarks (GPQA, AIME) are statistically valid?\n\nHow was the two round minimum for the LLM termination judge determined? Could a dynamic rule adaptively estimate the optimal refinement depth?\n\nRegarding the LLM generated agents, what specific prompting or selection process was used? Do these agents exhibit measurable differences in reasoning strategy or simply better prompt quality?\n\nHave the authors evaluated the method on open ended or real world tasks beyond academic benchmarks where consensus or correctness may be ambiguous?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is empirically strong and demonstrates commendable experimental rigor. It includes comprehensive ablations exploring agent diversity, tool combinations, and termination strategies, supported by clear figures and well documented appendices. The adaptive early termination strategy is an especially practical feature, halving inference costs with negligible performance loss. The empirical results establish that combining code and search tools within diverse agents produces measurable improvements across benchmarks. The use of LLMs to auto generate new agent types adds creative value, revealing potential for self improving agent systems. Clarity of presentation, transparency in reporting results, and a focus on practical reproducibility further enhance the paper’s quality.", "weaknesses": "The core methodological contribution is limited. The refinement mechanism relies on concatenating all prior responses into the prompt, a brute force aggregation that lacks principled structure or communication modeling. While empirically effective, it feels more like an engineering heuristic than a conceptual advance. The paper also depends heavily on proprietary, high capacity models (Gemini-2.5), raising concerns about reproducibility and generalization to open models. Claims about agent diversity remain loosely defined and most agents differ only in prompt phrasing or API configurations rather than fundamentally distinct reasoning styles. Furthermore, the evaluation lacks formal statistical significance tests and theoretical grounding for why diversity improves performance. The formulation in Section 3.1 sets up an optimization problem but doesn’t solve or analyze it, leaving the work largely descriptive. Finally, the reported improvements, though consistent, are modest relative to the significant computational and implementation cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yri4tHmTDL", "reviewer": "Reviewer_ZSeK", "summary": "This paper proposes TUMIX, a method run multiple tool strategies at test time (text, code, search, and hybrids) in parallel, share and revise answers across rounds, finish with majority voting; use an LLM-as-Judge for early stopping. Reports modest gains on HLE, GPQA-Diamond, and AIME with reduced cost under similar call budgets.", "questions": "1. Novelty vs. SciMaster: Could you clarify differences in architecture, judge/selector design, diversity control with the baseline SciMaster since I am still confused about the novelty?\n\n2. Early-stopping robustness: The judge-score heuristic can prematurely stop on small benchmarks. Do you have a concrete method to prevent this? For example, risk-limiting stopping with error control, SPRT-style sequential testing, minimum rounds per task type, judge–solver model decoupling, and sensitivity curves for thresholds.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Clear problem setup: practical test-time tool mixing and scheduling.\n\n2. Reusable engineering: parallel strategies + cross-round sharing + majority vote; pseudocode and prompt lists provided.\n\n3. Useful mechanism analysis: coverage drops over rounds; early stopping cuts cost; includes cost–performance curves and several ablations.", "weaknesses": "1. Loss of ensemble independence: cross-round sharing homogenizes answers; later selection policies become near-equivalent; majority vote is no longer independent experts and is prone to correlated failure.\n\n2. Tool–cost mismatch: cost is approximated by tokens or call counts only; wall-clock time, currency cost, search and code-execution delays and retries are not accounted for, so “equal-cost” comparisons are not valid and efficiency claims are unverifiable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jQ30IN73O8", "reviewer": "Reviewer_Dy1M", "summary": "This paper proposes TUMIX, a multi-agent framework for test-time scaling of tool-augmented LLMs. By orchestrating diverse specialized agents and enabling iterative response refinement, the system achieves an average gain of +3.55% on benchmarks including HLE and GPQA, using constrained inference budgets.\n\nA key contribution of this work is the insight that agent diversity yields better results than homogeneous replication. Furthermore, the proposed adaptive termination mechanism reduces inference costs by approximately 49% while maintaining model accuracy. The experimental analysis on performance improvement in multi-agent systems also provides valuable reference for the research community.", "questions": "- Based on the experimental results in Figure 5, it appears that increasing the diversity of agent answers and mitigating the decline in coverage rate contribute to performance improvement. However, this observation seems to differ from the trends shown in Section 5.3 (Table 13). Some clarification regarding these differing outcomes would be helpful for a more comprehensive understanding.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The TUMIX framework utilizes a coordinated multi-agent system in which specialized, tool-augmented agents perform parallel iterative refinement. Experiments report an average performance gain of 3.55% across several challenging benchmarks (HLE, GPQA, AIME 2024/2025), indicating the potential of this approach for test-time scaling.\n\n- Through the introduction of a coverage rate metric and systematic ablation studies, the authors provide empirical support for the relationship between agent complementarity and overall task performance. This analytical methodology offers useful insights into the functioning of multi-agent ensembles.\n\n- Empirical results consistently show that strategically diversifying agent composition leads to greater performance improvements than merely increasing the number of agents. This trend holds across various benchmark settings and model architectures.\n\n- The proposed adaptive termination mechanism reduces computational costs by approximately 49%, while largely preserving model performance.", "weaknesses": "- Based on the experimental analysis presented in the paper, increasing the number of rounds generally leads to a decrease in agent coverage rate, suggesting reduced answer diversity and eventual performance convergence. However, in the GPQA experiment (Figure 5, Section 5.1), an increase in the number of rounds conversely results in a higher coverage rate. It would be interesting to understand the factors behind the contrasting behavior observed in the GPQA benchmark (Figure 5, Section 5.1).\n- In Section 3.3, the authors note that many correct answers were incorrectly discarded by the LLM. Does this imply that the performance improvement is primarily influenced by the diversity of answers, as reflected by the coverage rate, while the use of the LLM to select responses may not contribute significantly to the performance gain?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UIAoWgKCoC", "reviewer": "Reviewer_tuT3", "summary": "This paper introduces TUMIX, an ensemble framework designed to solve the challenge of effectively combining LLM textual reasoning with Code Interpreter and Search tools. The framework operates by running multiple agents with diverse tool-use strategies in parallel, where agents iteratively refine their solutions by considering all outputs from the previous round. Experiments on Gemini-2.5-Pro and Flash show TUMIX significantly outperforms existing test-time scaling methods, achieving an average accuracy improvement of 3.55% over the best baseline.", "questions": "1. Regarding Agent Comparison (Figure 5): In the 3-agent vs. 15-agent comparison, the 3-agent group used three \"strong agents\" ($C^{+}$, $CS_{gs}$, and $CSG_{gs}$), each sampling 5 times. Were these three agents empirically determined to be the top-3 best-performing agents from the pool of 15? \n2. The process for generating new agents (Section 5.3) is described as \"query[ing] Gemini-2.5-Pro with current agent code examples\". Could you provide more detail on this methodology? Specifically, what prompts were used to guide the LLM to generate strategies that \"differ substantially from the original ones\" (as seen in Table 14 ) rather than just minor prompt variations?\n3. The framework was evaluated exclusively on Gemini-2.5-Pro and Gemini-2.5-Flash. Have you explored TUMIX's applicability to other models (e.g., GPT-5, Claude 4, or open-source models like DeepSeek and Qwen)? Furthermore, given the paper's emphasis on agent diversity, do you believe a heterogeneous mixture of models (i.e., agents powered by different LLMs) could be a viable strategy to further scale agent diversity and performance?\n4. You correctly note that in later rounds, as answers converge, the choice of selection method (Majority Vote, LLM Selector, Random) becomes negligible. Does this imply that the primary benefit of the framework lies in the iterative refinement process itself rather than the final selection, and that for a highly-converged set of answers, a simple majority vote is sufficient?\n5. You plot both inference number and token number against score in Figure 9. Given that token consumption is a more accurate measure of computational cost, and that different agents (e.g., CoT vs. Dual-Tool) have vastly different token profiles per \"inference,\" would it be more transparent to frame the primary cost analysis around token count rather than inference count?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experiments show that TUMIX can beat several baselines across several highly challenging reasoning benchmarks (HLE, GPQA, AIME). \n2. By using an LLM-as-Judge to determine the optimal stopping round, the framework achieves nearly identical performance to a fixed-round policy but reduces the inference cost.\n3. The paper provides a key insight that, unlike in text-only scaling, agent diversity is crucial for tool-augmented reasoning and outperforms repeated sampling from a single strong agent.", "weaknesses": "1. The main results are averaged over only three repetitive runs. On complex benchmarks like GPQA and AIME, the variance in LLM reasoning can be high. The reported gains, while positive, are sometimes small (e.g., 96.7% vs. 94.7% on AIME-Pro; 87.9% vs. 86.9% on GPQA-Pro). With only three runs, it is difficult to ascertain if these differences are statistically significant or fall within the margin of error, which undermines the robustness of the SOTA claims. Additionally, the results omit the actual inference count and token metrics\n2. The paper's reliance on \"inference counts\"  as a primary cost proxy is problematic; a single inference from a simple CoT agent is not \"comparable\" in cost to a single inference from a complex, multi-step dual-tool agent that consumes far more tokens.\n3. The paper notes that performance gains plateau after ~12 agent types (Figure 10) and attributes this to the increasing difficulty of answer selection. However, this analysis fails to consider that all 30 agents (human- and LLM-designed) are fundamentally limited to the same two tools (Code and Search). The performance plateau may not be due to the number of agents, but rather the lack of tool diversity. See more details in Question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)