An Architecture Search Framework for Inference-Time Techniques
Authors: Jon Saad-Falcon
Adrian Gamarra Lafuente
Shlok Natarajan
Nahum Maru
Hristo Todorov
Etash Kumar Guha
E. Kelly Buchanan
Mayee F Chen
Neel Guha
Christopher Re
Azalia Mirhoseini
Venue: ICML 2025 poster (2025)
Keywords: inference-time techniques
test-time scaling
machine learning
natural language processing
Primary area: deep_learning->large_language_models
TL;DR: We present Archon, a modular framework for designing inference-time architectures that outperforms top language models like OpenAI's o1, GPT-4o, and Claude 3.5 Sonnet on various benchmarks by optimally combining LLMs and inference techniques.

Abstract: Inference-time techniques, such as repeated sampling or iterative revisions, are emerging as powerful ways to enhance large-language models (LLMs) at test time. However, best practices for developing systems that combine these techniques remain underdeveloped due to our limited understanding of the utility of each technique across models and tasks, the interactions between them, and the massive search space for combining them. To address these challenges, we introduce Archon, a modular and automated framework for optimizing the process of selecting and combining inference-time techniques and LLMs. Given a compute budget and a set of available LLMs,
Archon explores a large design space to discover optimized configurations tailored to target benchmarks. It can design custom or general-purpose architectures that advance the Pareto frontier of accuracy vs. maximum token budget compared to top-performing baselines. Across instruction-following, reasoning, and coding tasks, we show that Archon can leverage additional inference compute budget to design systems that outperform frontier models such as OpenAI’s o1, GPT-4o, and Claude 3.5 Sonnet by an average of 15.1%.

An Architecture Search Framework for Inference-Time Techniques
Jon Saad-Falcon' Adrian Gamarra Lafuente'
Shlok Natarajan
Nahum Maru
Hristo Todorov
Etash Guha 2 E. Kelly Buchanan Mayee Chen
Neel Guha
Christopher Ré
Azalia Mirhoseini
Abstract
Averaged Performance: ArenaHardAuto, MATH, CodeContests
Inference-time techniques, such as repeated sampling or iterative revisions, are emerging as powerful ways to enhance large-language models (LLMs) at test time. However, best practices for developing systems that combine these techniques remain underdeveloped due to our limited understanding of the utility of each technique across models and tasks, the interactions between them, and the massive search space for combining them. To address these challenges, we introduce ARCHON, a modular and automated framework for optimizing the process of selecting and combining inference-time techniques and LLMs. Given a compute budget and set of available LLMs. ARCHON explores large design space to discover optimized configurations tailored to target bench- marks. It can design custom or general-purpost architectures that advance the Pareto fronticr of accuracy vs. maximum token budget compared to top-performing baselines. Across instruction- following, reasoning, and coding tasks, show that ARCIION can leverage additional inference compute budget to design systems that outperform frontier models such as ApenAl's ol, GPT-40, and Claude 3.5 Sonnet by an average of 15.1 15.1%.
75
70-
Archan
66
AFlow
60
ADAS
MoA
50
45
40
10K
25K
30K
73K
100K
Max. Input/Output Token Budget
Figure 1: ARCHON's Performance Effectively Scales with Increasing Inference Budget. Individual dataset analysis included in Figure
ensemble are queried in parallel, their responses are ranked, and the best ones are fused into a single, higher quality output, respectively (Jiang et al. 2023b] Wang et al. (2024a). Other types of inference- timee techniques are based querying single L.MM successively (via repeated sampling) and using voting strategy unit tests to select the top generation
Brown et al. 2024 Chen et al. 2024 Li et al. 2024a).
Recent work has made progress towards building robust inference-time architectures: systems composed of one or more large language models (LLMs) leveraging inference- time techniques. Examples include Mixture-of-Agee Agents (MoA) (Wang et al. 2024a) and L.M-BBender (Jiagg et al.) 2023b), well as single-model systems like ADAS (Hu et al. 2024) and AFlow (Zhang et al. (024). However, our exper- iments show that these top-performing baselines have limi- tations in compute utilization and task generalization. (see Section 4.2). We argue that designing effective and general- izable inference-time architectures requires the following:
1. Introduction
Inference-tine techniques Strrteggs that use additional compute during model inference-aa are gaining traction as effective methods for improving model capabilities. LLMs,
such as OpenAl's (OpenAl 2024), QwQ (Team 2024), and Sky-TI Team 2025). utilize such techniques translate additional inference compute into better performance across broad set of tasks. Example techniques include generation ensembling, ranking, and fusion, where models in the
Understanding the Utilities of Inference-Tinee Tech- niques: Inference-time architectures typically delegate their additional inference budget towards more model sampling calls (Chen et al.) 2024 Brown et al. 2024). which can be effective for math and coding tasks. Other tasks, such as following instructions and reasoning, have been shown benefit from additional techniques,
'Stanford University. Stanford, CA, USA Uniiversity of Wash- ington, Scattle, WA. USA. Correspondence to: Saad-Faleon cjonsaadfalcon @ stanford..duu. eduz.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Combining multiple An Architecture Search Framework for Inference-Time Techniques
Inputs
Optimizer
Outputs
Target Benchmark(s)
Inference Time Techniques
Optimized Archon Architecture
Architecture Optimizer (Hyperparameter Selection)
Proposed Archon Architecture
Generator
G
Fuser
Inference Call Budget
C
Available LLMs
Verifier
Sample Benchmark
R
Unit Test Generator
Results
Unit Test Eveluator
Output
Figure 2: Overview of ARCHON Framework: ARCHION's search algorithm requires the following inputs: target benchmarks, inference call budget, available LLMs, and available inference-time techniques (left). The search algorithm uses Bayesian optimization (Snock et al.) 2012) to construct and evaluate different ARCHON configurations (middle) before returning the optimized ARCHON architecture (right) for the target benchmarks (Section 3.3).
including ranking and fusion (Wang et al. 2024a [Jian jt al. 2023b). While all of these methods are valuable, it is essential to identify which inference-tine techniques are most effective for different task categories.
time techniques and explore the benefits of adding new models and new techniques individually. We find that generation ensembling combined with critique, verification, and fusion improves the final response quality beyond the oracle best candidate from individual (non-fused) responses, particularly for instruction-followingg and reasoning tasks Fiiuure 4 Figure 7] Table 5). We also demonstrate increased performance as we scale up the layers of inference-time tech- niques and combine multiple approaches together, allowing us discover effective new combinations of inference-time techniques (Sections 3.2
Understanding the Interactions Between Inference-
Time Techniques: While previous studies analyzed these techniques individually (e.g.. generation sampling in Chen et al. (2024). we need a more comprehensive understand- ing of the relationships between different inference- techniques across different tasks (e.g., is it better use more models or generate more samples per model?). Efficiently and Automatically Searching the Large Design Space of Inference-Tinee Architectures: Given a set of available LLMs and target tasks, there is currently no single prevailing inference- time architecture for maximiz- ing downstream accuracy across all tasks (Table I}. The search space for inference-time architectures expansive, requiring practitioners to make several key configuration decisions, such which LLMs to use, how many times to sample them, and how to combine and filter the candidate generations. These motivate the need for automated and adaptive architecture search approaches.
A.3
strategies significantly improves task performance, but deter- mining the specific combination remains challenging. This requires manually testing models, inference-time techniques, architecture designs, inference budgets, and more.
Third, drawing upon our analysis of inference- time techniques, we present A ARCOON, an open-source modular framework for automatically designing LLM systems composed of existing inference-tine techniques (or new ones). allowing practitioners to optimize for their desired objective functions: accuracy, latency, and cost (Sections 3.1 3.3p. Unlike alternative LM systems that perform prompt en- gineering and tool use over a single LM (Khattab
2023
In our work, we address each of these challenges. First, we evaluate the utilities of a comprehensive set of existing and proposed inference-time techniques across instruction-followii Using both open-source and closed-source models, we examine range of techniques such as ensembling, fusion, ranking, critiquing, verification, and model-based unit test generation/evallution (Sections 3.1 and 3.2) We find that no single technique completely dominates across all tasks, with different approaches being more effective for different tasks.
Yuksekgonul et al. 2024 Hu al.) 2024 Zhhng al. 2024).
our approach integrates multiple LMs in single architecture and reduces prompt selection to set of core components.
reasoning, and coding tasks.
The ARCHON framework utilizes automatic architecture search algorithms to maximize generation quality for the given tasks(s), leveraging Bayesian optimization (Snoek et al. 2012 Nardi et 2019) techniques inspired by (NAS) Zoph & Le 2017 Ren et al. 2021) rapidly traverse the space of potential inference architectures (Section 3.3)
We evaluate ARCHON architectures across diverse set
Second, we analyze the interactions between inference-
2

---

An Architecture Search Framework for Inference-Time Techniques
of inntrruction reasoning, and coding benchmarks
constructing inference-time architectures (Section 3.3
(Table T): M'T-Bench, Arena-Hard-Auto, Alpaca-2.0 2.0 Eval,
3.1. LLM Components of ARCHON
MixEval, MATH, and CodeContests (Zheng et al., 2023 Li et al. 2024b 2023 Ni et al., [024] Hendryykk et al. 2022). Our best ARCHON architectures surpass both frontier models (c.g. OpenAI's O1, GPT-4o and Claude-3.5 Sonnet) and prior top-performing inference-time architectures (c.g. ADAS, AFlow, and MoA), boosting state-of-the-art (SOTA) performance by 15. average. Furthermore, ARCHION achieves SOTA performances while using 20.0% less inference calls, 15. 55.1% less inpul tokens, and 13.5% less output tokens than alternative inference-time architectures (Figure } Table 1} Figurre 8). Even when solely using open-source LLMss, ARCHON architectures, on average, surpass SOTA LLMs by 11.2%.
In this section, we discuss the LLM components of ARCHON, which are LLMs that perform a specifie inference- time technique. We test an array of different components inspired by recent work, incorporating approaches for generating, ranking, and fusing candidates (Wang all, 2024a; Jiang et al.) 2023b) as well as approaches for improving candidate response quality through critiquing, verifying, and unit testing (Bai et al. 2022] Zheng et al., 2023). The components and their prompts are summarized in Table and [ppendix(A.2] We also perform an extensive ablation study of the given ARCHON components across instruction-following reasoning, coding benchmarks to better understand their individual utilities and their optimal combinations for different tasks (Appendix A.3)
2021; Li et al.
Overall, we present ARCHON an open-source inference- time framework, readily extensible to new inference-time techniques, models, and tasks via user-friendly interfaces.
Generator is an L.LM that takes in the instruction prompt and outputs candidate responses. Generators can be called in parallel to perform generation ensembling (i.e. calling multiple LLMs in parallel) (Wang al. 2024a) or sampled multiple times (Brown al. 2024). The number of models, samples, and generation temperature can be adjusted.
2. Related Work
Despite advancements in inference-tine architectures, many architectures focus on additional generations (Jiang et a 2023b Chen al. 2024 Davis et al., (024)) which is effective for reasoning tasks (Brown et al. 2024). However, for tasks like instruction- fostuuction-ollooiingg and reasoning, technique such as fusion and ranking are effective for bolstering task performances (Wang et al. [024a Jiang et al. 2023b). Prior studies have explored limited aspects of confi srfffiuuratioss,, often focusing on specific benchmarks (Jiang et al.
We find additional model sampling to significantly boost performance (Figure 6). particularly for coding tasks Table Tp. We see a similar pattern for model ensembling, where sampling from additional models leads to continual performance increases (assuming the models are ordered from best to worst for the given task) (Figure 7p.
2023b] Wang
al. 2024a Chen et al. |024 [i at. 2024a). It's crucial to efficiently develop inference-time architectures, as optimal configurations vary based on benchmarks, available models, and inference compute limits (Section (.2).. Furthermore, LM orchestration frameworks, such as DSPy (Khattab) et al. 2023), only optimize a single prompt for a single LM, better equipping it for tool use by utilizing supervised data but still unable to leverage multiple inference- techniques in parallel or sequentially. While each of these approaches manually selects a subset of existing techniques, ARCHON unifies available inference-time techniques and automates architecture construction with search algorithms, simplifying the model and component selection process for each set of tasks (Sections 3.1 and 3.3
Fuser is an LLM that, given an instruction prompt and set of proposed responses as input, combines these responses to generate one or more higher-quality fused responses.
For every benchmark explored, we found that the Fuser module substantially improved performance (8.9% on average) (Figure 6 Figure 7: Figure 4). Additionally, we observed similar benefits in the ARCHON framework when adding multiple layers of Fusers (Figure 4). The number of Fuser layers needed to improve performance varied by task Figure 12). with some tasks receiving limited benefits from added layers (1-2 point increase in accuracy for MixEval) while others experienced significant benefits with 3-4 fusion layers and more (10 to 15 point increase in win rate for MT Bench and Alpaca Eval 2.0).
3. Inference- Techniques for ARCHON
Ranker is an LLM that, given an instruction prompt and a set of proposed responses as input, ranks the candidate generations based on their quality, producing a ranked list of responses as output. This ranking is then used to filter the set of responses to the top-K, as specified.
With the proliferation of inference-time techniques, ARCHON introduces a systematic framework for under- standing and unifying these methods into inference-time architectures. Below, we elaborate on the structure, inputs, and outputs of each of the inference-tima techniques (Table 3). Then, we discuss how to combine the different techniques into an inference-time time architecture (Section 3.2p before finally exploring automatic approaches for
From our results in Table 5
Figure 6 and Figure our results
show the Ranker was most effective for instruction-folllving and reasoning tasks by using pair-wise comparisons that
3

---

An Architecture Search Framework for Inference-Time Techniques
focus on style and prompt adherence. We found that on MT Bench and Arena- Hard-Auto benchmarks, the Ranker improved output quality by 10.8% over random selection while performing within 2.7% of oracle selection.
3.1%, However, when we increased generation sampling and added unit test generation/evaluation for CodeContests, we observed a 56% boost in Pass @ I performance (Table I). increasing from 17.9 to 29.3% @
Critic an L.M that, given an instruction prompt and a of proposed responses as input, produces a list of strengths and weaknesses for each response, which is then used to improve the quality of the final response (Section 3.2 Figure 4).
3.2. Combining the LLM Components
Performance Gains from Scaling Inference- Time Techniques: We explore the utilities of individual AR- CHON components and evaluate whether combinations of inference-time techniques enable us to build LM systems greater than the sum of their parts. For our analysis, we look cnteructtoo-folllvviii
The Critic module proved effective for every task we explored in infFiuure Table With our 10-model +0B+ Generator ensemble and Fuser configuration of ARCHON, the added Critic improved performance on average by 11.5 percentage points across the benchmarks explored.
at seven datasets spanning mathematics, and coding: MT-Bench (Zheng et al.. 2023)
reasoning,
AlpacaEval 2.0 (Li et al. 2023). ) Arena Hard Auto (Li et.
Verifier is an LLM that verifies whether a provided candidate response has appropriate reasoning for given instruction prompt. It proceeds in two stages: Stage #1 takes in the instruction prompt and a candidate response as input and outputs reasoning for why the candidate response is correct; Stage #2 takes in the instruction prompt, candidate response, and produced reasoning before outputting reasoning and a verdict (i.e., binary [Correct] or [Incorrect]) for whether or not the candidate response is correct according the provided instruction prompt and reasoning. Only verified responses are passed to the next ARCHON layer.
2024b). MixEval (Ni et al. 2024), MixEval- Hard, MATH
(Hendrycks et al. 2021). 1. and CodeContests at. 2022)
We also test across the current SOTA open-source and closed-source LMs (Table 28 Table 29). For the analysis of each inference-tine technique, we focus on 1) testing across different benchmarks, 2) scaling its usage individually, 3) scaling it while randomly choosing another technique and holding that technique constant, and 4) varying its position among different components. We include these ablation experiments in Section [A.3] where we include the ARCHON component combinations Table al the model type used in the combinations in Table 6nd Table 9
The Verifier was most effective for the reasoning benchmarks explored in Table 5) improving performance by 8.4% for MixEval, MixEval Hard, and MATH. When just using a 70B+ Generator ensemble with Verifier module after generation, the ARCHON configuration lagged behind the ARCHON semble and fuser configuration by 1.5%, on average, across all benchmarks explored, suggesting verification is most ef- fective when combined with other inference-time techniques.
From our analysis, we find several trends (designated with Ts) across the combinations of inference-tine architectures:
T1: Repeated model sampling and additional ensemble models leads to substantial gains, leading to 9.3% and 18.5% increases, respectively (Figure 11 Figure 7).
T2: Scaling the layers of inference- time techniques signifi- cantly improves performance across instructionnfollooiing, reasoning, and coding tasks, such as always adding a single fuser as the last layer (Figure 4).
Unit Test Generator and Unit Test Evaluator are comple- mentary LLM components in our system: the Unit Test Gen- erator takes an instruction prompt and produces 5-10 concise test statements (Section 4.2 examples in Table 16) for as- sessing response accuracy and relevance, while the Evaluator takes the instruction prompt, candidate response(s), and these tests as input to rank responses by test passage. The Evalu- ator justifies and aggregates test verdicts across candidates, scoring each response for reasoning and coding tasks, and
T3: Scaling the diversity of inference-time techniques included also bolsters task performance across the explored tasks, with critics and rankers before fusers being particularly effective (Figure 4 Figure II).
T4: In reasoning tasks, incorporating the Verifier and Unit Test Generator/Evaluator modules alongside the Fuser improves performance by filtering out flawed responses, contributing to significant performance gains in tasks like Section
only responses passing all tests proceed to the next ARCHON layer. This approach extends evaluation beyond coding to various task types through configurable test quantities.
MixEval and CodeContests (Table 5
Drawing upon our analysis of the inference-time components, we propose ARCHON, a frame- work for automatically designing LLM systems composed of existing inference-time techniques (or new ones). In- spired by the structure of neural networks (Hinton et al. 1992). ARCHON consists of layers of LLM components (Fig ure 2 Section Each layer is composed of sets of LLM components called in parallel. These components perform a
Framework Overview:
The Unit Test Generator and Evaluator were most effective on reasoning and coding tasks, improving performance on benchmarks that required more verification steps (7.4% boost) (Table When the 70B+ ensemble of Generators was only combined with unit tests, it was less effective for reasoning tasks like Arena-Hard-Al and MixEval, lagging behind the ensemble and fuser configuration by

---

An Architecture Search Framework for Inference-Time Techniques
Prompt
1. Only one type of component is allowed in any given layer.
2. Generator components can only be placed in the first layer of ARCHION; you can place one or more Generators.
Generator 40
Generator Clasce Sornet
Generator
3. The Critic must come before a Ranker or Fuser. Other- wise, the generated strengths and weaknesses cannot be incorporated into generation ranking or fusion.
Critic Qwen2-728
4. Ranker, Critic, Verifier, and Unit Test Genera- tor/Evaluator layers can go anywhere ARCHON except the first layer. For each of these components, it must be the only module in its layer.
Ranker Closde Sernet
Fuser Qwest.5-1000
Fuser Liana3
Fuser Halky
5. Fuser components can go anywhere in A ARCHON except the first layer. Multiple Fusers can be used in a layer.
6. Unit Test Generators and Evaluators are placed in consec- utive layers, with the Unit Test Generator always first.
Verifier GPT 4e
Fuser Uama 4058
3.3. Architecture Search Algorithms
Search Hyperparameters: In this section, we explore how to automatically design inference-time architectures for target tasks via ARCHON's 's architecture search algorithms. Guided by the trends found in our analysis in Section we establish six axes of hyperparameters for the search space:
Output
Figure 3: Example A ARCHON Architecture: This architec- ture starts with ten generator models (cach sampled once), fol- lowed by a critic model, a ranker model, one layer of six fuser models, a verifier model, and finishes with a fuser model.
1. Top-K Generators for Ensemble:
The top-K models for the initial Generator ensemble, ranging from to 10 (T1). The top-K models are selected greedily based on their individual performances on target task(s) (Tble 29p.
text-to-text operation on the initial instruction prompt and the candidate responses from the previous layer. Furthermore, like neural network, some layers perform transformations of the provided list of strings (e.g., Generator and Fuser), converting a list of strings into a different list of strings (the numbers of candidates can vary from the original number of candidates). Other components introduce non-linearities into the ARCHON structure, performing filtering of the list of strings (c.g., Ranker and Verifier). Ultimately, the inputs and outputs for each layer is always a list of strings, whether that is the instruction prompt (i.e., a single string) or a list of candi- date responses. If a list of strings is outputted at the last layer of the ARCIION structure, the first string in the list is returned.
2. Top-K Generator Samples: The number of samples gathered from each ensemble generator (same for all the models), ranging from to 5 (T1). For CodeContests, we explore high-sample settings: [1, 10, 100, 500, 1000].
3. Number of Fusion Layers: Ranges from to 4. The last fusion layer will always have single Fuser (T2).
4. Top-K Fusers: Number of models used for each fusion layer, ranges from 2 to 10 in increments of 2 (T233).
5. Critic and Ranker Layers: We add critic and ranker layers before each fuser layer since we find they provide added benefits across the benchmarks explored (T3)
(Section 3.2 Figure 4 Figure 7).
6. Evaluation Layer: Option to add Verifier, Unit Test
Unlike a classical neural network, no weights are learned between the LLM components and the layers; in turn, the ARCHION architecture can be deployed off-the-shelff without any tuning. Additionally, a single state is transformed sequentially from the input layer to the final output; this single state is the initial instruction prompt and the current candidate responses (example architecture in Figure 3).
Gen./Eval,, or neither before the last Fuser layer (T4).
While it is possible to further expand the search space of potential ARCHON architectures (e.g.. different temperatures for generative LLM components, alternative prompts for each LLM component, additional LLM components for ARCHON, etc.), the trends we identify from Section 33.2] reasonably constrain the search space of configurations to focus on the most influential hyperparameters. In total, our search space contains 9,576 configurations, which we obtain by combining all possible hyperparameters and removing invalid configurations (for example, we discard configurations where the number of initial generations exceeds the context window of the fusers).
The LLM components in Section can only be placed in specific orders 4 While alterna- tive combinations and orderings of ARCHON components are technically viable, we found these orderings to be optimal after conducting an ablation study of ARCHON components across seven benchmarks and two model classes (open-source and -osere) (Appendix/A... A.3).
Rules for Construction:
5

---

An Architecture Search Framework for Inference-Tinge Techniques
1.0
0.9
Best Open-Source 70B+ Model, Sampled Once
Win Rate Accuracy
Best Open-Source 708+ Model, 8- 8-Samples it Fuser
0.7
0.6
Ensemble (Top-8 Best Models) Fuser
0.5
Ensemble Critic Fuser
Ensemble Critic
0.4
Layers
Ensemble Critic
0.3
Layers Fusers
0.2
Ensemble Layers
0.1
0.0
MT Bench
AlpacaEval 2.0
Arena Hard
MixEval
MixEval
MATH
Code
Auto
Hard
Contests
Benchmarks
When controlling for inference budget, generation ensembling and fusion across different 70B LLMs is generally more effective than repeated sampling with only the top performing model. Furthermore, adding layers of critique and fusion led to 18.8% boost in task performance, on average. However, the best inference-time architecture differed by task, such as MixEval and CodeContests (Section which inspired us to develop architecture search techniques for ARCIION (Section 3.3
Figure Performance Improves by Scaling Layers of Inference- Techniques:
The ARCHON search method takes in four inputs: the target benchmark(s), the inference call budget, the set of available LLMs, and the inference-time techniques for construction (Figure 2). As output, the search method outputs single optimized ARCHON architecture. We use 20% of each target dataset as a development set for guiding architecture search. We explore three approaches for ARCHON's s architecture search: random search (randomly test potential architectures in the search space), space).gredd search (greedily optimize individual hyperparameters one at time, starting from random initial architecture), and Bayesian Optimization (Snoek et al. 2012) (global hyperparameter optimization with Gaussian processes). As inputs, Bayesian optimization takes in a vector specifying the configuration choices for the generators (i.e., number of models and samples), layers fusers, numbers of fusers per layer, and final verifier/ unit tester (Section(.22) Bayesian optimization begins by sampling specified number random ARCHON architectures to calibrate its surrogate model. The task performance of these sampled architectures is used to guide more informed architecture suggestions during the configuration search. The algorithm repeats the following ayole-evaluatting each suggested architecture and using its performance refine future suggestions until discovers the optimal ARCIION configuration, or until the inference call budget is exhausted. For more details our open- source Bayesian optimization approach, please Appendix A.4 where we further discuss implementation and how to utilize alternative optimization functions, such as latency.
Search Method:
dom search (Figure 10). The effectiveness of Bayesian op- timization increases with the number of initial randomly sampled architectures, up around 230-240 samples, after which further testing is better focused on configuration search Table 20). For limited inference call budgets (<20 calls), Bayesian optimization is less effective, and traditional meth- ods like greedy search may perform comparably Tabll 21)
Adding Search Restrictions: To impose compute constraints during architecture search, we exclude any ARCHON architecture that would exceed the inference call, input token, output token budgets from the search space. Multiple restrictions can be added. For example, you can filter out architectures with more than 20 inference calls or more than 20,000 input tokens. This prevents our Bayesian optimization algorithm from even considering these invalid architectures our architecture search, allowing us to compute-mateds ARCHON against alternate inference-time frameworks such ADAS and AFlow (Figure 8 Figure 5).
4. Experiments
Our experiments focus answering the following questions: (1) how does ARCHON compare to existing SOTA L.LMs and inference- architectures terms of accuracy and compute efficiency (Section ? (2) how does ARCHON performance compare across the tasks explored (Section 4.3) (3) what are the considerations for model size, latency, and cost surrounding ARCHON (Section[4.4)) We outline the benchmarks, models, and techniques for constructing ARCHON architectures in Section
Bayesian optimization found the best architectures in 66.0% of searches and required 88.5% fewer architecture evaluations than greedy search and 90.4% fewer than ran-
6

---

An Architecture Search Framework for Inference-Time Techniques
Alpaca Eval 2.0
Arena Hard Auto
MixEval Hard
MixEval
MATH
Code Contests
Bench
Approaches
Average
Infer.
Calls
Average Average Input
Tokens
Output Tookens
Avg. PFLOPS
per Query
Dollars per Query
W.R.
L.C. W.R.
W.R
Acc.
Acc.
@1
Pass @
GPT-4o
95
549
0.1
0.01
44.2%
+0.5
57.8%
#0.6
80.6%
+0.6
63.4%
#0.2
87.5%
+0.3
83.5%
#0.4
18.1%
=0.2
LM
Claude 3.5 Sonnet Llama 3.1 405B
105
601
1,410,2
00001
N/A
52.7% 20,4
81.4%
10.4
68..%
20.2
89.1%
10.2
82.5%
10.7
12.3%
20.4
Baselines
118
631
0.5
0.01 0.01
44.1%
+0.3
40.7%
#0.5
64.5%
+0.7
66.0%
20.3
88.2%
+0.2
85.0%
+0.5
20.4%
+0.5
MoA ADAS AFlow ol
19
25,109
17,422
0.3
0.06
51.6%
+0.6
65.0% +0.3 85
3%
+0.3
62.3%%
+0.4
86.9%
+0.2
82.9%
+0.6
15.1%
LM
52
72,804
44,872
0.3
0.63 0.04
3%
+0.7 60.
=0.5
85.4% +0.4
64.2%
=0.2
87
06
+0.2
86.0%
+0.8
23.7%
=0.3
Systems
48
68,596
41,748
0.4 Unk.
0.05
62.4% 10.2 57.8%
20.6
83.2%
10.6 63.5%%
=0.3
87.2%
10.4
84.5%
20.2
21.1% 20.6
Unk.
112
Unk.
0.52 20.05 56.3% 10.5 59.3% 20.5 81.7%
40.3
72.0%
20.4
87.5%
#0.2
92.7%
#0.5
31.5%
Open
General Purpose
35
51,113
31,508
3.1 + 0.3
0.12 67.2%
+0.4 63.3%
+0.6
85.6%
+0.5
65.3%
#0.3
86.2%
+0.2
87.5%
+0.6
18.2% +0.4
Sourcee
Task Specific
44
63,157
39,949
3.7 4
0.15 & 0.02 71.1% 10.6 68.1% 20.4 89.6%
40.4
67.5%
20.2
88.8%
10.3 89.51%
20.3
28.9% 40.9
Archon
Closed
General Purpose
32
52,747
27,894
0.5
0.04 72.7% +0.3 63.9% #0.7 86.2%
#0.7
67.5%
#0.4
87.2%
#0.2
87.9%
#0.7
20.2% #0.6
Source
Task Specilic
59.085
37,271
48.2 0.4
0.49 77.0% +0.5 68, +0.5 90.5% +0.3
72.39
+0.3 89.55
+0.3
92, 1% +0.4
25. 35.1% +0.6
All
General Purpuse
35
50.427
30,461
27.8
0.32 + 0.04 76.2% +0.7 66.4% #0.3 89.8%
#0.6
69.8%%
#0.2 87.3% #0.4 89.30%
#0.5 23.4% +0.9
Source
Task Specific
58,250
36,114
0.37 0.6
79.5% 20.4 69.0% 10.6 92.5% 72.7% 20.3 89.7% 93.5% 20.6 41.4% 10.7
Table I: ARCHON's Strong Performance with Open Source, Closed Source, and All Source Models: Consistent outperformance over SOTA LLMs and LM Systems across explored benchmarks. The standard error numbers were calculated from 10 independent evaluation runs. *MATH and CodeContests use subset of their test sets for evaluation (Section 4.1
4.1. Benchmarks and Models
and Table 29 component, we set the generation temperature to 0.7. As baselines, we compare ARCHON against both SOTA single- call LLMs (GPT-40 (OpenAll et al. 2024), Claude 3.5 Sonnet (Anthropic 2024), and Llama 3.1 405B Instruct (AI@ Metal 2024)) as well as SOTA inference-time approaches (Ope- nAT's ol OpenAA 2024), MoA (Wang et 2024a), ADAS Hu et al.
For all the LLMs utilized and every ARCHON
We evaluate our models with several benchmarks for instruction-followingg reasoning, and
Benchmarks:
coding: MT-Bench (Zheng et al.) 2023), AlpacaEval 2.0 (Li
al. 2023), Arena Hard Auto (Liet al. 2024b), MixEval (Ni
jtt al. 2024), Hard, MATH (Hendrycks et al. (021).
and CodeContests (Li et al.
2022). We provide an overview
2024). and AFlow (Zhang et al. 2024)).
of each dataset in Table 22 Since we perform automatic architecture search a randomly sampled 20% subset of each benchmark, we evaluate on the remaining held-out 80% subset of the benchmark (Table 1) (for ARCIION performances on the entire benchmarks, please see Table 27). The delta between the ARCHON performance on the entire benchmark vs. 80% held-out subset is relatively small: only 0.44%, on average, across these datasets with an S.D. of 0.20%. For MATH, we evaluate a random sample of 200 problems from the dataset's test set. For CodeContests, we evaluate on the 140 test set questions that do not include image tags in the problem description.
Task-Specific and General-Purpose ARCHON Architec- tures: We compare custom ARCIION architectures, specifi- cally configured to a single evaluation dataset ( ("Task-specific ARCHON Architectures") "), and a generalized ARCHON ar- chitecture configured to handle all the evaluation datasets ("General-purpose ARCHON Architectures") (Table I}. For our three model selection settings for ARCHON (i.c. open- source, closed-source, and all-source), we utilize automatic architecture search to find targeted ARCHON architectures for each task (7 architectures total) and find a single general- ized ARCHON architecture for maximizing performance over all the tasks (Table I). The benchmarks are concatenated together and shuffled for generalized ARCHON architecture search. Importantly, all the ARCHON architectures utilized in Section urchitecture search technique, which searches over the hyper- parameter search space for ARCHON as covered in Section 3.3 For examples of targeted and generalized ARCHON archi- tectures, please see Figure Appendix/A... A.9 For our archi- tectures, we outline the average number of input tokens (i.e. combined total of tokens inputted over the entire architecture) and output tokens (i.e. combined total of tokens outputted over the entire architecture) for each category in Table [
Models: We test the efficacy of the ARCHON framework by creating different ARCHON architectures across three model categories: 8B less parameter models, 70B or more param- eter models, and closed-source model APIs. For our 8B and 70B+ models, we selected the top- 10 performing chat models for each parameter range on the Chatbot Arena Leaderboard (Chiang et al., 2024) as of July 2024. For our ARCHON architectures, we explore multiple model types: open-source, closed-source, and all-source (i.e. both open-source and closed-source available). For our closed-source model APIs, we include GPT-4o, GPT-4-Turbo, Claude Opus 3.0, Claude Haiku 3.0, and Claude Sonnet 3.5. We list and compare all of the models tested in the ARCHON framework in Table 28
4 Jre automatically generated by our Bayesian
7

---

An Architecture Search Framework for Inference-Time Techniques
4.2. A ARCHON vs. Closed-Source
GPQA
MMLU
MMLU
LLMs and Other Inference- Time Architectures
Diamond
Pro
All-Source Generalized AFlow
37. 37.1%++0.
53 53,0%6+0.4 43,4%=004
Task Performances: We start by comparing ARCHON architectures to existing SOTA closed-source LLMs and inference-time architectures across a set of instruction- following, reasoning, and coding tasks. Based on our results in Table we find that ARCHON architectures consistently match or surpass existing approaches across all the bench- marks explored. ARCHON architectures with open-source models demonstrate a 11.2% average improvement over SOTA open-source approaches; for its worst performance, our open-source ARCHON architectures are still 3.1% % above SOTA open-source approaches on AlpacaEval 2.0. ARCIION architectures with closed-source models achieve SOTA performance across MT Bench, Arena-Hard Auto, MixEval, and MixEval-Hard, leading to a 15.1% average improvement over flosed-source LMS and a 8.4% average improvement over open-source inference-tine time frameworks (i.e. MoA, ADAS, and AFlow). Compared to ol and ol-mini, ARCHON's is best targeted architectures beat them by 8.1 and 9.7%, on average, on MT Bench, AlpacaEval 2.0, Arena Hard Auto, MixEval, MixEval Hard, MATH, and CodeContests. For approaches that use all models available, both open and -losed-source, ARCHON achieves an average 10.9% improve- ment over existing SOTA single-call LLMS and an average 8.6% improvement over existing inference-time frameworks.
Task-Spocific AFlow
52.4%
718%:40.5
62.9%:20.5
AHlow Performance Preservation
70.8%
73.8%
67.0%
All Source Generalized ADAS
39.8%
53.5%440.
44, 14.17
Task-Speeific ADAS
54.4%
73.0%
66.0%2044
ADAS Performance Preservation
73.2%
73.3%
66.8%
All-Souree Geveralized ARCHON
56. 16.1%%00.4
76.5%:10.
71.0%=0.11
Task-Specific ARCHON
61.2%
81 1.5%++0.3
75.4%=004
Archon Porformance Preservation
91.7%
93.9%
94.2%
Table 2:
Generalized ARCHON Architecture Strong Performance on Out-of-Domain Tasks: The generalized ARCHINN architectures achieved 91 to 95% the performance of the specialized ARCHON architectures on GPQA, MMLU, and MMLU Pro, despite not being trained for these tasks. Standard error calculated from 10 independent evaluation runs. "For MMLU and MMLU Pro, we use randomly selected 500 query sample of the test set for evaluation.
Each subsequent layer has fewer fuser models (i.e. 8, 6, and 4), leading to a "funneling" effect on the generations before the final output. The best targeted architectures can vary by task. For instruction -ottructioo-folowiing and reasoning tasks, the targeted architectures tend to be multiple layers of critiquing and fusing with a diverse mix of LMs (Figure 14). For math tasks, the targeted architectures tend to consist of an initial broad set of generations before being reduced quickly to a chosen answer (Figure 1.5). For coding tasks, the targeted ar- chitectures tend to focus on multiple iterations of generation, critique, and fusion over a single response before outputting an answer (Figure 16). Besides the ARCHON architectures included in Appendix A.9 we include all the generalized and targeted ARCHIN architectures in our supplementary files.
Compute Efficiency: Compared to open-source inference- time frameworks (i.e. AFlow, ADAS, MoA), AR ARCHON is 20.0% more inference call efficient while having higher per- formances on all benchmarks tested (Table Tj. We also find that our best ARCHON architectures use 15.1% less input to- kens and 13.5% less output tokens compared to the best alter- native open-source inference-time frameworks. When we uti- lize ARCHON's architecture search technique with different token budgets (Figure 8). we find that the generated ARCHON architectures achieve 12.4% higher performance than alter- nate baselines when given the same budget. Overall, the gen- eralized all-source ARCHON architecture achieves 6.4% bet- ter performance across all the tasks while being 31% % more to- ken efficient than the best LM system baselines Table Ip. Fur- thermore, compared to the generalized all-source ARCHON architecture, the targeted all-source ARCHIN architectures use 15.5% and 18.6% more input tokens and output tokens, respectively, but they achieve 8.4% higher accuracies, on av- erage. The targeted architectures are more compute intensive since they can further leverage additional LM operations to- wards single set of specific task constraints (Appendix A.9)
To explore the efficacy of our general purpose ARCHON architectures, we evaluate them on three previously unseen tasks: GPQA (Rein et al. 2024). MMLU (Hendrycks et al., 2021 and MMLU Pro Wang et al. 2024b). We find that our all-source general purpose ARCHON architecture cap- tures 91 to 94% of the task-specific ARCHON architectures performances on these benchmarks, suggesting that our archi- tectures are more broadly applicable to out-of-doman tasks (Talle 2). The generalized ADAS and AFlow architectures only achieve 66% and 74% of their specialized architecture performance, respectively.
4.3. ARCHON by Task
Citruction-Followinng and Reasoning: On MT Bench, AlpacaEval 2.0, and Arena-Hard-Auto, open-source ARCHON architectures outperform current open-source baselines by 10.5%, on average, while closed-source ARCHON outperforms current closed-source baselines by 14.6% (Table T). With ARCHON, multiple models used for Generators and the depth of fusion layers lead
Discovered Architectures: We include the targeted and gen- eralized ARCHON architectures in Appendix A.9 Figure 13).
The best performing all-source, general- purpose ARCHON architecture starts with a broad initial layer of our 10 best generators before four successive layers of critique and fusion with Qwen2 72B and Claude 3.5 Sonnet, respectively.
8

---

An Architecture Search Framework for Inference-Time Techniques
Arena-Hard-Aur
MATH
CodeContests
100
90
90
10
Win Rate (%)
Pass@1 (%)
Passgj1 (%)
Archon
Aflow
ADAS
70-
70
20-
Moh
19
10
20
50
10
20
40 50
20
30 40
PFLOPs per Quory
PFLOPs per Query
PFLOPs per Query
Figure 5: ARCHON's Performance Exceeds Baselines across FLOP Budgets: Across different FLOP budgets (Section
3.3) we compare ARCHON architectures against top-performing inference-time system baselines. The MoA architecture and OpenAl's ol are static so they use the same number of tokens across budgets. The results were averaged over 10 independent evaluation runs. "MATH and CodeContests use subset of their test sets for evaluation (Section 4.1
to performance boosts on instruction-following tasks, increasing the richness of responses and allowing multiple iterations for step-by-step instruction-followinn (Table 30). For reasoning, while the performance boost from ARCHON is smaller when we consider the aggregate scores for MixEval and MixEval-Hard, we do see meaningful increases in performance when we create inference- time architectures for each individual task under MixEval and MixEval -ard Table 25). When we create individual ARCHON architectures for each subtask, we see 3.7 and 8.9 percentage point increases in accuracy, on average, for MixEval and MixEval-Hard, respectively. This finding suggests that reasoning tasks (e.g. math, sciences, logic) require more individualized inference-time architectures.
els, we can boost task performance over the hest individual 7B LM by 7.5%, on average, compared to the best individual 7B model Taale 32). Across tasks, 7B models work well for ranking but are less effective for critique and fusion.
Latency and Costs: Since ARCHON architectures make mul- tiple LLM API calls successively for different operations it can take 5x more time and money than single LLM API call Table 33 Table 34). Note that these increases in compute costs and latency translate to higher quality responses, and can be justified in many application domains, such as science, programming and complex agentic tasks (Rein et al. 2023] Mialon et al. 2023). Furthermore, LLM vendors are rapidly decreasing their inference costs TTable 33). For tasks in which speed is most preferred, future work should explore how distillation strategies (Sreenivas et al. 2024 DeepSeek] AI et al., 2025) could be used to pack the aggregate knowledge of ARCHON architectures into a smaller L.M.
(Table 24
We have observed that ensembling, fusion, and ranking techniques have limited impact on CodeContests (Figure 4). For example, when we apply the general all- source architecture from Table to CodeContests problems, we achieve small gains from ARCHON TTale T}. One con- tributing factor is that, unlike the distribution of instruction- following/reasoning tasks, coding tasks tend to have one or two LLMs that perform substantially better than the rest of models (Table 29). However, when we add unit test genera- tion/evaluation, and scale the number of samples, ARCHON's performance on CodeContests improves significantly (Ta ble T), allowing us boost GPT-4o Pass @1 performance by 44.3% for Pass @1 (from 40 to 58 out of 140 questions). For model-based unit test generation/evallatin
Coding:
Trome
Impact Statement
ARCHON''s methodology for optimizing multi-model LM architectures could transform how organizations deploy Al systems, enabling smaller companies and research labs to achieve state-of-the-art performance without access to the largest models or extensive computing resources. The framework's ability to boost performance while reducing compute costs results in energy savings and lower environ- mental footprint. The technique of effectively combining multiple specialized models through systematic architecture search could also find applications beyond language models, potentially generalizing to other AI models and tools. As ARCHON demonstrates how to extract significantly better performance from existing models rather than relying on scaling one model alone, it may influence the trajectory of Al development toward more efficient, specialized inference-time architectures rather than ever-larger models.
we generate 5 unit tests and use the LM to evaluate each candidate response against the generated unit tests, allowing us to rank the differ- ent candidate responses (details are provided Section A.2)
4.4. Discussion
Impact of Model Size: The ARCHON framework is most effective when utilizing LLMs with 70B+ parameters. When we build ARCHON architectures with 7B open-source mod-
9

---

An Architecture Search Framework for Inference-Time Techniques
Acknowledgments
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion,
J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A.,
We thank Simran Arora, Daniel Biderman, Bradley Brown, Ryan Ehrlich, Sabri Eyuboglu. Jordan Juravsky, Jerry Liu, Avanika Narayan, Benjamin Spector, Alyssa Unell, Benjamin Viggiano, and Michael Zhang for their constructive feedback during the composition of the paper. We would also like to thank our collaborators at the Stanford Artificial Intelligence Laboratory (SAIL) and TogetherAI.
McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L.. Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R.,, Ringer. S., Johnston, S., Kravec, S., Showk, S. E.,, Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Haffield-Dodds, 7., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J. Constitutional ai: Harmlessness from feedback, 2022. 2212 08073
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF2247015 (Hardware-Awann CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under Nos. W911NF-23-2-018
(Long-context) and INF-21122-0 (Interactive Human-Al Teaming); ONR under Nos. N000142312633
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL /ttpss//arrxvv.org/aab
(Deep Signal Processing); Stanford HAI under No. 247183; NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Meta, Google, and VM The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
217887
Chen, L.,, Davis, J. Q., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J. Are more Ilm calls all you need? towards scaling laws of compound inference systems, 2024. URL
Chiang, W.-L.,, Zheng, L., Sheng. Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating ilms by human preference, 2024.
Databricks. Dbrx technical report. 2024.
Davis, J. Q., Hanin, B., Chen, L.,, Bailis, P., Stoica, I,, and Zaharia, M. Networks of networks: Complexity class principles applied to compound ai systems design, 2024. URLhtttss://axxzz org abs 2407 2407.168311 16831
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja. J.. Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.1 14219, 2024.
DeepSeek-AI, Guo, D., Yang, D., Zhang. H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang. P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Z., Gao, Z. Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen,
AI@Meta: Llama 3 model card. ArXiv, 2024. URL
https:///i com/
com/meta- mwta//etaaaa///aaa lama /1 l L/ma3/
D., Ji, D., Li, E.,, Lin, F., Dai, F., Luo, F., Hao, G., Chen, Li, G., Zhang, H., Bao, H., Xu, H., Wang. H., Ding, H., Xin, H., Gao, H., Qu, H., Li, H., Guo, J., Li, J., Wang. J., Chen, J., Yuan, J., Qiu, J., Li, J.. Cai, J. L., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Zhao, L., Wang, L., Zhang, L., Xu, L.. Xia. L., Zhang, M., Zhang, M., Tang. M., Li. M., Wang. M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen, Q.. Du, Q., Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R. J., Jin, R. L., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S., Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S., WU, S., Ye, S., Yun, T., Pei, T., Sun, T., Wang, T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W. L.,, An, W., Liu, X.,
blob/ http//ainn/ie http///aanna//....a/a// MODEL CARD. md
Anthropic. The claude 3 model family: Opus, sonnet, haiku. ArXiv, 2024.
Bai, J.. Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y. Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J. Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R. Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., WU, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang. J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J.. Zhou, X., and Zhu, T. Qwen technical report. arXiv preprint arXiv:2309. / 16609, 2023.
10

---

An Architecture Search Framework for Inference-Tine Techniques
Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang. X., Shan, X., Li, Y. K., Wang, Y. Q. Wei, X.X., Zhang, Xu, Y. Li, Y., Zhao, Y. Sun, Y. Wang, Y. Yu, Y., Zhang, Y., Shi, Y., Xiong, Y. He, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y. Ou, Y., Wang, Y., Gong, Y., Zou, Y., He. Y., Xiong, Y., Luo, Y., You, Y. Liu, Yhouh Y. Zhu, Y. X., Xu, Y. Huang, Y., Li, Y. Zheng. Zhu, Ma, Y., Tang. Y., Zha, Y., Yan, Y., Ren, Z. Z., Ren, Z., Sha, 7., Fu, 7.,, Xu, Z.,, Xie, 7., Zhang, Z.,, Hao, Z. Ma. Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z.,, Zhang, Z. and Zhang. Z. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https:
s. Zhang, S., Vandenhende, S., Batra, S., Whitman,
S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V. Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang. X., Tan, X. E., Xie, X., Jia, X., Wang. X., Goldschlag, Y., Gaur, Y.. Babaei, Y., Wen, Song. Y., Zhang. Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z.. Chen, Z., Papakipos, Z., Singh, A., Grattafiori, A., Jain, A., Kelsey, A., Shajnfell, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples. A., Gu, A.. Ho, A.. Poulton, A., Ryan, A.. Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A.. Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti. B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Wyatt, D., Adkins. D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang. D., Le, D., Holland, D., Dowling, E., Jamil, E.,, Montgomery, E., Presani, E., Hahn, E., Wood, F.,, Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Ozgenel, F., Caggioni, F., Guzmán, F., Kanayet, F., Seide, F.. Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halper,, G., Thattai, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Molybog, I., Tufanov, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Asher, J., Gaya, J.. Marcus, J., Tang. J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong. J., Jin, J.. Yang. J.. Cummings, Carvill, J., Shepard, J., McPhie, Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Prasad, K. Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakhotia, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L.,, Bell, L.., Zhang, L.., Guo, L.., Yu, L., Moshkovich, L.. Wehrstedt, L., Khabsa, M., Avalani, M.. Bhatt, M., Tsimpoukelli, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Seltzer, M. I.., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng. N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli. O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux,
wrxiv.xiiivv.... /arps://arxiv.coo www./og/g/bbaaaooeoe 2501 12948
Dubey, A., Jauhri, A.. Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A.,
Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, McConell,,,, C., Keller, C., Touret, C., Wu, C.,, Wong. C., Ferrer, C. C. Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, F.., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Zhang, F., Synnaeve, G., Lee. G., Anderson, G. L., Nail, G., Mialon, G. Pang. G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H. Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J. Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong. J., Lee, J., Fu, J.. Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Cuntala-Yeary L., van der Maaten, L., Chen, L., Tan, L., Jenkins, I.,, Martin, I.., Madaan, I.., Malo, I., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Oldham, M.. Rita, Pavlova, M.. Kambadur, M., Lewis. M.. Si, M.. Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li. P., Vasic, P., Weng, P., Bhargava, P.. Dubal, P., Krishnan, P., Koura, S., Xu, P., He, Q., Dong, Q. Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa. S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale,
Choudhary, D., Mahajan, D., Garcia

---

An Architecture Search Framework for Inference-Time Techniques
P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj. P., Liang, Q. Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang, R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Shankar, S., Zhang. S., Zhang, S.. Wang. S., Agarwal, S.. Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfiel,, S., Govindaprasad, S., Gupta, S., Cho, S., Virk. S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li. T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang, X., Xia, X., Wu, X., Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Hao, Y., Qian, Y., He, Y. Rait, Z., De Vito. Z., Rosnbrick, Z., Wen, Z., Yang. Z., and Zhao, Z. The llama herd of models, 2024. URL https: :/ts:://arrra //wrxiv..vvvv//////// org/ abs/ 2407 21783
Jiang, D., Ren, X., and Lin, B. Y. LIm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting the iisociation Computational Linguistics (ACI. 2023b.
Khattab, O., Singhvi. A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M.. and Potts, C. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.33714, 2023.
Li, J., Zhang, Q Yu, Y, Fu, Q. and Ye, D. URL
More agents is all you need, 2024a.
Li, T., Chiang, W.-L.,, Frick, E., Dunlap, L., Banghua Zhu, J. E. G. and Stoica, I. From live data to high- quality benchmarks: The arena-hard pipeline, April URL
http:s////mmss.rr////// 2024- 04 19-4-19-22 1119--rena--ae -araanaaaaaa
2024b.
X., Zhang. T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, B. Alpacaeval: An auto- matic evaluator of instruction-following models. hhtpps wtt.huu..co//ttttteeette haaaaaaaaaaaaaaeee eval 2023.
Guo, D., Zhu, Q. Yang, D., Xie, Z. Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y. K., Luo, F., Xiong, Y., and Liang, W. Deepseek-coder: When the large language model meets program- ming - the rise of code intelligence, 2024. URL
Li Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J.. Gimeno, F., Dal Lago, A., et al. Competition-levell code generation with alphacode. Science, 37866664)))1019291999 2022.
2401.
Hartford, E. dolphin- 2.2. -miisrllss January 2024.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Meng. Y., Xia, M., and Chen, D. SimPO: Simple preference optimization with a reference-free reward. ArXiv, 2024.
Mialon, G., Fourrier, C., Swift, C., Wolf, T., Le- bench-
Cun, Y., and Scialom, T. Gaia: mark for general ai assistants, 2023,
Hinton, G. E. et al. How neural networks learn from experience. na, 1992.
URL
https://arxiv /ttp::///axxi.oorr//bb//////
12983
Hu, S., Lu, C., and Clune, J. Automated design of agentic systems. arXiv preprint arXi::2408.0844: 2024.
Nardi, L., Souza, A., Koeplinger, D., and Olukotun, K. Hypermapper: practical design space exploration framework. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pp. 425-426, 2019. doi: 10.1109/MASCOTS 201.AASSTTTSSSSSSS....M 2019.00053.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023a.
Ni, J., Xue, F., Yue, X., Deng, Y., Shah, M., Jain, K., Neubig, G., and You, Y. Mixeval: Deriving wisdom of the crowd from Ilm benchmark mixtures, 2024. URL
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S.. de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. F. Mixtral of experts, 2024.
06565
OpenAL.
Learning to reason with LLMs.
Septem-
ber 2024. Accessed November 13, 2024.
12

---

An Architecture Search Framework for Inference-Time Techniques
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I,, Balaji, S., Balcom, V. Baltescu, P. Bao, H., Bavarian, M., Belgum, Bello, I., Berdine, J., Bernadett-Shapiro. G., Berner, C., Bogdonoff,, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brookss T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C. Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B. Cho. C.,, Chu, C., Chung, H. W., Cummings, D., Currier, J. Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dumning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V. Gogineni, T., Goh, G. Gontijo-Lopss, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C.,, Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu. K. Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., B., Jun, H., Kaftan, T., Lukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L.,
C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L.,, Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q. Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T.,, Zhuang, J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2024, UtLhttps:///axxt http:////aaxxiiooorr/////323333
wttpg//bb////3/3 08774
Qwen. Qwen2 technical report. 2024.
Rein, D., Hou, B. L., Stickland, A. C., Petty. Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google qa benchmark, 2023. URL
http:////wwwoooo/////////////// htss://aazav..orr/aaa///1111
Rein, D., Hou, B. L.. Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL //openreviens eww net for.m/ormmmm//oo 357.7r72222..0777777558833
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Chen, X., and Wang, X. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1-34, 2021.
Kim, W. Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Lukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lec, T., Leike, J., LewnLeung.... J., Levy, D., Li, C. M., Lim, Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makamju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R. Neelakantan, A., Ngo. R. Noh, H., Ouyang, L., O' Keefe, C., Pachocki, J. Paino, A., Palermo, J., Pantuliano, A., Parascandolo,
Snoek, Larocchll, H., and Adams, R. P. Practical bayesian optimization of machine learning algorithms, 2012. URL https://arxiv..rq/ abs 1206 2944
Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Llm pruning and distillation in practice: The minitron approach. 2024. URL https: ftttss:/aaxiv.orgge wwtporggg//b.ooooeee 11796
Team, N. Sky-tl: Train your own ol preview model within 450. hilps : //novasky ai.github.io posts sky
t1,2025, Accessed:: 2025-0222202222 01 09.
G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F. Petrov, M.,
Team, Q. Qwq: Reflect deeply the boundaries of the
de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnum,, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P.. Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J.. Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. C., Vallone, A. Vijayvergyya, A., Voss, C., Wainwright, C., Wang, J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann,
unknown, November 2024. URL https://qwen.nm
github.io/blen wtt.:oooolllggo qwa- 32 32wp::p-iie-pppieeeeem preview/
Tran, H., Glaze, C., and Hancock, B. Iterative dpo alignment. Technical report, Snorkel AI, 2023.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. (et::2340.66944,.. 2023.
Wang, J. Wang. J., Athiwaratkun, B., Zhang, C., and Zou, J. Mixture-of-agent enhances large language model capabilities, 2024a.
URL
http://xxxvv.org/a////2222 04692
13

---

An Architecture Search Framework for Inference-Time Techniques
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z.,, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024b.
Xu, C.,, Sun, Q.. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. WizardI.M: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Reprexentations, 2024. URL https: //openreview. net/ forum? Ctw.uummdddCCC-C-C- CFXh.
Yuksckgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z. Guestrin, C., and Zou, J. Texigrad: Automatic "differentiation" via text. 2024.
Zhang, Xiang, J., Yu, Z., Teng, F., Chen, X., Chen,
J., Zhuge, M., Cheng, X., Hong, S., Wang. J., Zheng, B., Liu, B., Luo, Y., and Wu, C. Aflow: Au- lomating agentic workflow generation, 2024. URL
https://arxia: wtt.s//.org/bb/221...... 10762
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging Ilm-as-a-judge with mt-beenh and chatbot arena, 2023.
Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning, 2017. URL /ttps://arxiv wttp://wwwwrr//b////oo.o/bs http//aab/1111111.....
14

---

An Architecture Search Framework for Inference-Time Techniques
A. Appendix
A.1. Table of Contents
ARCHON LLM Components: Outline LM components used in ARCHON and the rules for combining them
2. Utilities and Interactions of LLM Components: Analysis of the effectiveness of individual components and their synergistic effects when combined
3. Bayesian Optimization for ARCHON: Description of search space and optimization methodology for architecture discovery
Bayes Optimization vs. Alternative Approaches: Comparative analysis of search techniques for finding optimal configurations
ARCHON Architecture Algorithms Comparisons: Evaluation of different optimization strategies across various inference
budgets
6. ARCHON Benchmarks and Results: Comprehensive evaluation results across followiig,,....oiiggg reasoning, and
coding tasks
7. ARCHON LLM Analysis: Details of models tested with parameter counts and sequence lengths
8. ARCHON Architectures: Diagrams and descriptions of optimized architectures for different task categories
ARCHON by Inference Compute Budget, Model Size, and Cost: Analysis of performance across different compute constraints and model sizes
A.2. ARCHON LLM Components
Inference-
Definition
Input
Output
Domains
Technique
Cost
Generates candidate response
call percand.
Generatoe
Instruction Prompt
Candidate Response(s)
from an instruction promup
Domains
Merges multiple candidate responses into a single response
Instruction Prompt
Fused Candidate
call All Domains
Candidate Response(S)
Response(s)
per percand.
Generates strengthsweaknnese
Instruction Prompt
Candidate Responso(s)
call All
Critic
for each candidate response
Candidate Response(s)
Strengtha/ Weaeth//eeeeeeeesess
Domains
Returns top-K
Instruction Prompt+ +
Ranked Candidate
call All
Ranker
candidate responses
Candidate Response(s)
Responso(s)
Domains
Returns the candidate responses with verified reasoning
Instruction Prompt + Candidate Response(S)
Verified Candidate
calls Reasoning Tasks
Verifier
Response(s)
percand.
Test
Generates
to evaluate
Instruction Prompt
Reasoning Tasks
Instruction Prompt
call
Generator
the candidate responses
Unit Tests
Instruction Prompt
Unit Test Evaluator
Uses generated unit tests evaluate candidate respoosse
Scored Candidate Responso(s)
call percand.
Reasoning
Unit Tests
Tasks
Candidate Responso(s)
Table 3: Overview of ARCHON's Inference-time Techniques: Definitions, Inputs, Outputs, Costs, and Application Domains.
Initial Layer Placement
Placement after Initial Layer
>1 Module
Increase Candidate Responses
Decrease Candidate
Module
in Layer
Responses
Generator
Yes
Yes
Yes
No
Fuser
No
Yes
Yes
Yes
Ranker
No
Yes
No
No
Yes
Critie
No
Yes
No
No
No
Verilier
No
No
No
Yes
Test
Generator
Na
Yes
No
No
No
Evaluator
Na
Yes
No
No
No
Table 4: Rules of ARCHON Construction; Allowed combinations of each LLM component from Section 3.1
15

---

An Architecture Search Framework for Inference-Time Techniques
Oracle Selection
Random Selection
Rankee Selection
Fuser Response
Ranked Fuser Response
MT Bench
Arena-Hard-Auto
Win Rate Against Sonnet 3.5
0.55
0.65
0.60
0.60
0.55
0.50
Sonnet 3,500
GPT-40
0.45
GP 0.45
8
10
5
10
Number of Repeated Samples
Number of Repeated Samples
Figure 6: Performance Gains from Applying Inference Time Techniques on Single Model:
We repeatedly sample more responses for each individual query. For each sample count, we choose the best response in different ways: (1) using an oracle (to get the upper bound for performance of best sample), (2) randomly, (3) using a ranker model, (4) by fusion, in which model synthesizes response based on all the samples, and (5) by ranking the top-5 best answers and then fusing them. For both MT Bench and Arena-Hard-An Hard-Auto, we find that fusion is an effective technique. In particular, ranking the candidates first, and then selecting the top-5 and fusing them scores the highest. The best open-source model for these tasks across all the 70B+ models we are considering is M- (Xu al. 2024) (see Table fo details). For both ranking and fusion, we use Qwen2 72B Instruct (Qwen 2024).
A.3. Utilities and Interactions LLM Components
In this subsection, we present our analysis the effectiveness of each LLM component (i.e. the Utility) and the relationships between each component (i.e. the Component Interactions) by evaluating on instruction-follow folloowin tasks (MT Bench, AlpacaEval 2.0, Prena-Hard-Auss reasoning tasks (MixEval, MixEval-Han Hard, MATH) and coding tasks (CodeContests) (Section[4.1 For our ARCHON models, we utilize a host of 70B+ open-source models (Section 4.1 Table 28).
A.3.1. GENERATOR
Utility: For our Generator module, we find additional model sampling to significantly boost performance FFigure 6 particularly for coding tasks (Table Ip. In settings with a limited inference call budget, additional model samples lead to the largest marginal benefit. We see a similar pattern for model ensembling, where sampling from additional models leads to continual performance increases (assuming the models are ordered from best to worst for the given task) (Figure"
1.3.2.FUSEE.
Utility: For every benchmark explored, found that the Fuser module substantially improved performance (Figure 6 Figure 4). For the single-generation 10-model ensemble of 70B+ models, the Fuser module improved downstream best model (Figure 7). When combined with the Ranker module for ranking the top-5 candidate responses, the Fuser improved downstream accuracy by 7.3 points and 3.6 points, on average, compared to the single-sampl best model and the oracle best candidate response, respectively (Figure 7}. Overall, we found that Fuser efficacy increased as more candidate responses were provided, demonstrating that additional candidate generations can continue to bolster inference-time architecture performance when combined with Fuser.
Figure7
accuracy by 5.2 points, average, compared to the
In previous work like ceature-of-Agemee (MoA) (Wang et al.) 2024a), multiple layers of Fusers was found to boost performance on some citrruction-oollwwing tasks (i.e. MT Bench and Alpaca Eval 2.0). Across all the benchmarks explored, we observed similar benefits in the ARCHON framework when adding multiple layers of Fusers (Figure 4p. However, based on our results Figure 12 the number of Fuser layers needed to improve performance varied by task, with some tasks receiving limited benefits from added layers (1-2 point increase in accuracy for MixEval) while others experienced significant benefits with 3-4 fusion layers and more (2 to 5 point increase in win rate for MT Bench and Alpaca Eval 2.0). We attribute
16

---

An Architecture Search Framework for Inference-Time Techniques
Oracle Selection
Random Selection
Raniee Selection
Fuser Response
Ranked + Fuser Response
MT Bench
Arena-Hard-Auto
Win Rate Against Sonnet 3.5
0.70
0.70
0.85
0.65
0.50
060
0.35
0.55
0.50
Sonnet 0,50
3.5
6PI-40
0.45
4c
0.45
0.40
0,40
0.35
0.35
4
10
5
8
10
Number of Models in Ensemble
Number of Models Ensemble
Figure 7: Performance Gains from Applying Inference-Time Techniques on an Ensemble of Models: We incrementally add more models to the ensemble, which consists of open-souree 70B+ models. The models are added the pool based on their performance for each task, from best worse (see Table for details). For each ensemble size, we choose the best response in 5 different modes: (1) using an oracle (to get the upper bound for performance of best individual response in the ensemble), (2) randomly, (3) using ranker model, (4) by fusion, in which one model synthesizes response based on all the responses of the ensemble models, and (5) ranking the top-5 best responses and then fusing them. For MT Bench and trena-Hard-Aut we find consistent performance improvements as we add more models to the ensemble. We find that fusion beneficial across various ensemble sizes and in particular& a fused candidate based on the top-5 ranked responses scores highest. The ensemble approach scores higher than applying the same techniques on repeated samples from single best-performing model (see Figure 6). For both ranking and fusion, we use Qwen2 72B Instruct (Qwen 2024).
this distinction to the difference in task requirements, with chat and instruction following tasks benefiting more from multiple iterations revisions through the multiple Fuser layers, leading to greater diversity in the final generation (Table 30).
Component Interactions: better understand how the Fuser module works with the other L.M components, we took the single-sample 10-model ensemble of Generators with Fuser and tried adding each of these components individually: Critic, Ranker, Verifier, and Unit Test Generator/Evallutoor Across all of the benchmarks, the added candidate response analyses from the Critic improved the Fuser's ability to effectively merge the different candidate responses, increasing performance by an average of 3.1 percentage points FFiuure 4). With the added Ranker, the ARCHON architecture improved the combined Ensemble + Critic + Fuser performance across all the benchmarks by 4.8 percentage points, on average (Figure 4). The Ranker proved most effective for style-oriemtn tasks (e.g. MT Bench and AlpacaEval 2.0) since the examples mostly focus on improving the instruction -uidaacce towards the provided prompt. With the added Verifier module (Figure 4), the performance of the Ensemble Critic Fuser configuration improved marginally for the instruction-following tasks (1.2 percentage points, on average, for MT Bench, AlpacaEval 2.0, and Arena- Hard-Harr---rr -utr). However, this configuration improved performance more reasoning tasks (3.2 percentage points for MixEval and MixEval-Hard, on average), assisting generation by filtering out irrelevant or flawed answers before the final fusion step (Figure 4). The added Unit Test Generator and Evaluator was less effective for the instruction-folloiing and reasoning tasks, only providing a 1.5 percentage points increase, on average, when added to the Ensemble Critic + Fuser configuration (Table 3). However, for coding tasks, we found unit test generation and evaluation significantly improved performance, leading to a 10.7 percentage point increase (56% performance increase comparatively) as we scale model sampling (Tblle I).
A.3.3. CRITIC
With our 10-model +0B+ Generator ensemble and Fuser configuration of ARCHON, the added Critic improved performance on average by 3.1 percentage points across the benchmarks explored.
Utility: The Critic module proved effective for every task we explored in Figure and Table 5
Component Interactions: While useful for most ARCHON architectures, the added strengths and weaknesses from the Critic module are particularly useful when combined with the Fuser module, helping guide generation fusion for a single layer
17

---

An Architecture Search Framework for Inference-Time Techniques
MT Bench
AlpacaEvvl
Arena Hard Auto
Miccrall Hard
MixEval
MATH
Code Contests
of Infer. Calls
/Ld System
W.R.
L.C. W.R.
Raw W.R.
WIR
Acc.
Acc.
Ave
Auc.
Contrel
Best Opess-Suree 70B+ Model, Samplad Onee
55. (0% +10.4
44.7%
+0.5
57.158
#0.6
45. 6%
+0.5
58. 7%8
#022
86.5%
+0.3
84.5%
+0.6
22,
+0.3
Ensemble 1 Flsser
58.4% #0.6
57.5%
+11.4
51.3%% ai..
54.3%
#0.7
00.15%
#0,5
157.3%
#0.2
85.5%
#0.3
23.1%
#10.
Ensemble Critic+ Fooer
65.8%
58.85
67.4%
+0.5
+0.4
Ensemble Ranker
52.5%
10.7
54.7%
10.5
47.6%
20.4
50.5%
20.6
58.7%
20.3
86.8% 20.4
80.4%
10.4
24.1%
40.4
Ensemblle Veitter
53.2% +0.5
56.2%
#0.3
50.256
52.4%
95.21%
#0.7
25.31
+1.5
Athations
Ensemble Uitr Test Gen./Evul.
18
51.5%
20.4
54.4%
20.5
49.45
20.5
40.1%
20.8
55.25%
20.4
56.0%
20.3
85.2%
20.5
24.6%
40.5
Enseuble Ranker Fiser
10
62.1% +0.8
60.35%
+1.4
63.658
m0.6
57.2% +0.5
99.7% +02
87
6.
+0.3
85.3%
+0.6
24.0%
+00.2
Ensemble Vorifer Fuser
#0.3
+0.7
58.7%
59.2%
68.3%
86.7%
10.4
#0.6
Ensemble Test Gen./Eval. Fuser
10.6
58.5%
56.4%
63.9
9% 20.3
56.4%
20.8
20.6
Ensemble Crdtie Verifor & Fises
25
61.3%
+0.5
60.09
+0.3
61
0%6
=0.7
59.5%
+0.3
(5.88
+0.4
57.8%
898
+0.4
16.1%
+0.3
26.8%
+01.3
Ensemble Crilic Rinker + Fuser
11
64.7% 20..
62.6%
=0.6
72.4%
+0.5
60.9%
20.6
06.855
204
N8.3% +0.2
87.3%
#0.5
25.5% #10.3
Table 5: Impact of Different Compositions of ARCHON's Inference-Time Techniques: We see increased task performances from adding new LLM components to ARCHON. For CodeContests, we find that there is single model (Llama 3. 405B Instruct) that performs considerably better than the rest of the LLMs studied, making it more effective leverage additional model sampling (Talle T). For our ensemble, we use the best 8 open-source 70B+ models for the task TTall 29). For our fuser, critic, ranker, and verifier components, we use the best fuser model found for the task (Table 29). For each evaluation benchmark, we explain its configuration in Tabll 22 and Section 4.1] The standard error numbers were calculated from 10 independent evaluation runs.
and even useful when placed between multiple fusion layers (on average 3.2 percentage point boost across benchmarks in Figure 4p. The Critic module was also effective with the Ranker module, providing additional information for comparing candidate responses (Figure 6) and leading to 5.9 percentage point increase, on average (Table 5)
A.3.4. RANKER
Utility: From our results in Table 5 Figure and Figure 7. we found the Ranker to be most effective for instruction-
following tasks, where pair-wise comparisons of answers focus on style and adherence to the prompt. To examine the candidate selection improvement provided by candidate ranking, we compare three approaches to the Ranker: (1) random selection of candidate generation, (2) oracle selection of candidate generation, and (3) the top-ranked candidate selected by our Ranker. For MT Bench and Arena-Hard- Hato-Auto, we find that the ranker improves generation output quality by 3.8% compared to random candidate selection and performs within 2.7% of oracle selection (Figure 6).
Component Interactions: Based on benehmark results in Table 5, the Ranker pairs well with the Critic module; the provided strengths and weaknesses helps guide ranking, particularly for instruction-folllwwe by 5.9 percentage points, on average. Furthermore, the Ranker was also effective when paired with the Fuser; the filtered list of candidate responses helped improve the final condensed response produced by the Fuser by 3.8 percentage points, on average (Figure 7). When paired with the Verifier and Unit Test Generator, the Ranker had neutral effects; performances changed marginally, either positively or negatively by 1-2 percentage points (Tbll 5p.
tasks, improving performance
Overall, our findings demonstrate the value of added Rankers for instruction instructton-followinng and reasoning tasks when paired with Fusers. We find that when Rankers used alone with ensemble of Generators, their performance lags behind the 1 10-sample best single model configuration by 3.0 percentage points, on average (Table 5). Additionally, our findings show the importance of building better rankers for more complex reasoning tasks, such as math and coding, which is a challenge also raised by (Brown et al. 2024).
A.3.5. VERIFIER
Utility: The Verifier was most effective for the reasoning benchmarks explored in Table When just using a 70B+ Generator ensemble with Verifier module after generation, the ARCHON configuration lagged behind the ARCHON ensemble and fuser configuration by 1.5 percentage points, on average, across all benchmarks explored. This suggests that the Verifier is most effective when combined with other inference-time techniques.
Component Interactions: As noted in Section (..tionn....2 A.3.2 the Verifier augmented the performance of the Critic and Fuser on reasoning tasks (c.g. Arena-Hard-Ae Altro-Auto, MixEval, MixEval- Hardd, boosting performance by 3.7 percentage points, on average, when combined together with these modules. Overall, the Verifier is most powerful when augmenting additional components
18

---

An Architecture Search Framework for Inference-Time Techniques
MT Bench
AlpacaEval
Arena
MixEval Hard
MixEval
MATH
Code Contests
L.C.
Model /...el/L.LMSS System
Infer. Calls
W.R.
W.R.
Acc.
Acc.
Acc.
Acc.
Control
Single Generation
44.2% +0.6
57.8%
+0.5
48.1% 1% #0.7
63.4%
+0.3
87.5%
+0.2
82.
12.11%
+0.4
17.9%
+0.3
Ensemble Fuser
53.7% =0.3
59.5%
+0.6
49.7%
65.5%
=0.2
82.0%
+0.3
+0.6
16.0% +0.4
Ensemble Critic Fuser
56.1% 20.7
59.7%
10.4
53.9%
20.6
67.4%
=0.4
82.0%
20.2
82.3%
10.5
18.9%
10.6
Ensemble Ranker
47.6%
49.7%
45.5%
Ensemble + Verifier
11
48.4% =0.5
51.2% +0.7
47.7%
+0.8
61.4%
+0.2
80.5%
+0.3
75.5%
+0.3
21.0% +0.4
Ablation
Ensemble+ + Unit Test Gen./Eval.
46.8%
49.3
19.3%
+0.3
41.2%
60.2%
=0.4
80.7%
+0.2
78,
+0.8
24.0%
+0.7
Ensemble Ranker Fuser
12
58.0% =0.2
60.1% 10.6
52.2%
40.3
65.0%
=0.3
#0.4 82.0%
82.1%
+0.4
18.0%
40.3
Ensemblee + Verijier + Fuser
55.8% 20.6
54.2% 10.4
60.3%
20.7
67.0%
20.2
40.3 82.5%
83.1%
10.6
22.4% 10.5
Ensemble Unit Gen./Evol, Fuser
56.5%
51.6%
67.7%
#0.2
84.3%
25.4% #0.6
Ensemble + Critic+ + Verifier + Fineer
13
56.6% 20.7
62.0%
10.3
55.0%
40.6
68.5%
40.3
82.7%
10.4
85. 75.
10.3
22.2 10.4
Ensemble
Ranker
62.8%
56.2%
87.0%
18.5%
Table 6: ARCHON Component Compositions with GPT-4o: The ensemble uses generates 10 samples for the given query. The standard error numbers were calculated from independent evaluation runs.
AlpucaEval 2.0
MixEval Hard
Code
Bench
Hard Auto
MixEval
MATH
Contests
#of Infer. Calls
L.C. W.R.
Model LLM System
W.R.
W.R.
Acc.
Acc.
Acc.
Acc.
Contrd
Single Generation
32.1% =0.7
38.5%
10.5
30.4%
#0.6
45.2%
=0.3
69.5%
#0.2
72.3%
+0.5
10.5%
+0.6
Ensemble Fiser
44.2%
20.3
43.0%
10.6
40.2%
20.4
46.0%
20.4
73.0%
20.3
70.5%
10.7
6.0%
20.4
Ensemble Critic + Fuser
12
46.6%
#0.5
44.2%
+0.4
44.4%
=0.7
47.9%
=0.2
73.0%
#0.4
+0.3 72.5%
8.4% =0.5
Ensemble Ranker
11
38.1% +0.6
40.2%
+0.7
36.0%
#0.5
43.8%
=0.3
72.1
+0.2
66.2%
+0.6
7.5% +0.4
Ensemble+ + Verifier
38.9% =0.4
41.7%
+0.3
38.
38.2%
41.9%
=0.4
71.0%
#0.3
68.5%
+0.4
19.0% +0.7
Ablatbans
Ensemble Unit Test Gen./Eval.
37.3% 20.8
39,8%
10.6
31.7%
20.3
40.7%
20.2
71.2%
20.4
69.8%
10.8
22.0%
10.3
Ensemble + Ranker Fuser
12
=0.2 48.0%
45
6%
+0.5
42.7%
20.6
45.0%
=0.3
#0.2 73.0%
70.1%
+0.5
8.0% #0.6
Ensemble+ + Verifler + Fuser
46.3%
20.5
Ensemble + Unit Test /Eval, Fuzer
22
47.0%
20.3
43.9% 10.7
42.1%
20.7
48.2%
20.2
72.2%
20.4
73.1%
10.6
23.5% #0.4
+
Verifier
52.4%
+0.5
+0.4
+0.7
Ensemble + Critie Ranker Fuser
50.5% +0.4
48.3% +0.6
46.7
% +0.3
55.1%
#0.4
73.7% +0.3
76.4% #0.5
8.1% +0.5
Table ARCHON Component Compositions with GPT-4o-mini: The ensemble uses generates 10 samples for the given query. The standard error numbers were calculated from 10 independent evaluation runs.
MT Bench
AlpacuEval 2.0
Arena Hard Auto
MixEval Hard
MixEval
MATH
Code
Contests
#f Infer.
L.C.
Model System
Calls
W.R.
W.R.
Acc.
Conttol
Single Generation
NIA
52.7%
#0.4
81.4%
+0.6
79.%%
+0.3
19
#0.2
83.5%
#0.5
12.5%
=0.3
Ensemble Fuser
11
NIA
53.0%
20.6
83.2%
10.4
69.3%
10.2
89.0%
20.3
81.8%
10.6
17.0%
20.4
Emsemble
Fuver
12
+0.7
Ensemble Ranker
11
NIA
50.2%
+0.5
85.7%
+0.5
63.8%
40.3
1%
+0.4
80.2%
+0.7
18.5% +0.5
Ensemble Verifier
11
NIA
51.7%
#0.7
78.2%
+0.3
60.9%
+0.2
80,
#0.3
21.0%
=0.4
Alladions
Ensemble Unit Test Gen//Evll,
21
NIA
49.8%
20.4
+0.8 71.7%
$9.0%
10.2
81.2%
+0.2
80.9%
20.8
22.0% =0.7
Ensemble+ Ranker Fuser
a0.5
Ensemble Verifier Fuser
12
NIA
54.7%
+0.3
85.0%
+0.6
70.5%
#0.2
89
39.
+0.3
84.1%
40.6
21.6% +0.5
Ensemble Test /eval. + Fuser
22
53.9%
82.
+0.5
+0.4
+0.2
82.0%
+0.6
Ensemble Critic + Verifler Fuser
13
N/A
56.0%
=0.4
85
0%
+0.3
71
11.0%
#0.3
89.4%
#0.4
83.1%
#0.3
21.4% =0.4
Fasemble Critic + Ronker Fuser
13
N/A
58.3%
+0.5
86.7% +0.7
73.0%
#0.2
89.7
#0.3
85.3%
+0.7
19.1% +0.5
Table 8: ARCHON Component Compasitions with Claude 3.5 Sonnet: The ensemble uses generates 10 samples for the given query. The standard error numbers were calculated from 10 independent evaluation runs.
19

---

An Architecture Search Framework for Inference-Time Techniques
AlpucaEsvl 2.0
MixEval Hard
Code Contests
MixEval
MATH
Bench
Hard Auto
#of Infer. Calls
L.C.
Model/ L.L.M M System
W.R.
W.R.
W.R.
Acc.
Acc.
Acc.
Acc.
Control
Single Generation
35.0% =0.5
10.6
36.8%
#0.7
64.6%
=0.2
73.2%
#0.3
74.3%
+0.4
10.0%
+0.5
Ensemble Finee
48.2% 20.3
47.0%
10.4
44.2%
20.5
66.5%
=0.3
77.0%
20.2
75.1%
10.7
10.8%
10.3
Ensemble + Critic Fuser
50.6% =0.7
48.2
48.2%
+0.5
48.4%
40.3
68.1%
=0.4
77.0%
40.4
76.3%
40.5
11.5%
40.6
Ensemble + Ranker
11
42.1% +0.4
44.2%
+0.7
40.0%
#0.6
58.8%
+0.3
76.1%
+0.2
71
8%
+0.6
11.9%
+0.4
Ensemble+ + Verifier
42.9% #0.6
45.7%
+0.3
42.2%
=0.8
57.9%
=0.2
75.0%
=0.3
70.5%
+0.4
12.0%
+0.7
Ablatios
Ensemble + Unit Test Gen./Eval.
41.3% =0.8
43.8%
10.6
35.7%
20.4
55.7%
=0.4
75.2%
20.2
74.1%
10.8
13.0% 10.3
Ensemblee+ + Ramker Fuser
52.0% #0.2
49.6%
10.5
46.7%
40.7
60.0%
#0.3
77.0%
40.4
75.0%
10.5
12.0% 40.6
Ensemble Verifier Faser
50.3% =0.5
48.7%
+0.4
48.7%
40.5
67.5%
+0.2
77.0%
+0.3
77.4%
+0.3
10.5% 40.5
Ensemble- + Unit Test Gen./Evol. Fuser
22
51.0% #0.3
97.9%
+0.7
46.
1%
40.6
64.2%
#0.4
76.2%
#0.2
78,
35
+0.6
14.3% #0.4
Ensemble+ + Critic Verifler Fuser
68.09
#0.4
10.0%
Ensemble + Critic Ranker Faser
13
54.5%
20.4
52.3% 20.6
50.7
%
20.3
70.4%
20.2
77.7%
20.3
80.5%
20.5
11.5% 40.5
Table 9: ARCHON Component Compositions with Claude-3-Haiku: The ensemble uses generates 10 samples for the
given query. The standard error numbers were calculated from 10 independent evaluation runs.
Arena-Hard-Aur
MATH
CodeContests
100
90
90
10
Win Rate (%)
Pass@1 (%)
Passgj (%)
AFlow
ADAS
70
20
Moh
19
10K
Z5K
50K
TGk
100K
10K
25K
50K
75K
100k
10K
25k
50K
75K
100K
Max. Input/Output Token Budget
Max. Input/Output Token Budget
Max. Input/Output Token Budget
Figure 8: ARCHON's Performance Exceeds Baselines across Token Budgets: Across different token budgets (Section 3.3) we compare ARCHON architectures against top-performing inference-time system baselines. The MoA architecture and OpenAl's ol are static so they use the same number of tokens across budgets. The results were averaged over 10 independent evaluation runs. "MATH and CodeContests use subset of their test sets for evaluation (Section 4.1
for tasks requiring verification of intermediate steps and the final response TTbll 5). for instruction-foll followiin tasks (e.g. .T Bench and AlpacaEval) but more effective for reasoning tasks (e.g. Arena-Hard Hard-Auto and MixEval).
Therefore, the Verifier was less helpful
A.3.6. UNIT TEST GENERATOR AND EVALUATOR
Utility: The Unit Test Generator and Evaluator most effective on reasoning and coding tasks, improving performance on benchmarks that required more verification steps, such as Arena-Hard-Auto. MixEval, MixEval-Hard Hard, MATH, and CodeContests Table 5). For the reasoning tasks, we found the unit test generator and evaluator to be most effective when combined with other components. When the 70B+ ensemble Generators was only combined with unit tests, it was less effective for reasoning tasks like Arena- Haiaa-Harr--uuuss and MixEval, lagging behind the ensemble and fuser configuration by 3.1 percentage points. This inspired us to look into other inference-time techniques combinations for unit test generation, such increased sampling and fusion. When we increased generation sampling and added unit test generation/ for CodeContests,
a 56% boost in Pass @ performance TTale 1}. increasing from 17.9 to 29.3 Pass @1.
Component Interactions: When combined with the Fuser module, the Unit Test Generator and Evaluator improved performance by 2.1 percentage points across the benchmarks explored Table 5). The combined ensemble, Unit Test Generator/Evaluator,, and Fuser ARCHON configuration was most effective on the reasoning benchmarks, leading to a 2.5 percentage point boost, on average. For coding, the unit test generator and evaluator was most effective when combined with the best performing Generator (using large sample counts) and final Fuser (subsection 4.2)
20

---

An Architecture Search Framework for Inference-Time Techniques
Arena-Hard-Aur
MATH*
CodeContests*
90
90
10
Win Rate (%)
Pass@1 (%)
Pass (%)
Aflow
20
Moh
0.2
0.4
0.8
10
0.2
0.6
0.8
1.0
0.4
0.6 0.8
Dollars per Query
Dollars per Query
Dollars per Query
Figure ARCHON's Performance Exceeds Baselines across Dollar per Query Budgets: Across different dollar per query budgets (Section 3.3). we compare ARCHON architectures against top-performing inference-time system baselines. The MoA architecture and OpenAl's ol are static they the same number of tokens across budgets. The results averaged over 10 independent evaluation runs. "MATH and CodeContests use subset of their test sets for evaluation (Section
<instruct
Table 10: Generator Prompt
rresponses their individual critiques strengths/weaknesess Your task synthesize these responses into a single, high-quality response. It crncial to oritically evaluate the information provided in these resporses and their provided critiques of trengths/weakknesees, recognizing that some of it may be hiased or incorrect. Your response should not simply replicate the but should refined, accurate, comprehensive reply instruction. Ensure your response is well-struttueed coherent, and adheres to the highest standards of accuracy and reliability. Responses from models:
been provided
various open source models
user query.
<resporse $1>
Critique: ceritique 2. <response #2> Criti <oritique
N. <response 4N
Critique: <oritique
<instruction here>
((a) With Critiques
You have heen provided with a set of responses from varinas open- source models the latest user query. Your task is to synthesize these responses into single, responses, recognizing some of may be biased incorrect.
critically evaluate the information provided Your response should not simply replicate the given answers but should olfer a relined, accurate, and comprehensive reply to instruction. Eisure your response is welll coherent, and adheres to the highest standards of accuracy and reliability. <resporse <response #2>
high-quality response.
creeponse IN> <instractien: here>
(bb) Without Critiques
Table 11: Fuser Prompt: Without and With Critiques
21

---

An Architecture Search Framework for Inference-Time Techniques
Twill provide you with NI responses, each indicated by mumerical identifier []. Rank the responses based on their relevance to the instruction: < cinstruction here>.
[1] <response +1>
cresponse
[N] <reaponse #1
Instruction; <insi:uctis ucttion
Rank the N responses above based on their relevance to the instruction. All the responses should be included and listed using identitiers, in descending order of relevance to the instruction. The output format should [) > [1,eg. [4] > [2). Only respond with the ranking results, do not say word or explain.
Table 12: Decoder- Based Ranking Prompt
You are helpful assistant. 1 will provide you with & responses, each indicated by a numetical identifler (o.g.. 1t..g..11.111....1.. 121, etc.). Rank the responses based on their relevance to the instruction:
rinstrrct heree.
[1] <reaponse 41>
[2] <response 42>
[N] <reaponse
Instruction: instructiio heze>,
Evaluate the responses above based their relevance to the instraction. All the responses should be included and listed using identifiers, Fur each resporse. slart the critique numerical identifier (e-g. g. [1]) followed hy strengths weaknesses. You the end of cach response's is analysis, include two now lines to separate the critiques. Do not include any preface or aftr the critiques. Do not include any reforences to previous critiques within critique. Start with the anilysis for the first response and end with the analysis for the last response. All of N responses should be included and evaluated using identiflers Structure each response' 's analysis follows:
strengths weaknesses, even there are of one than the
Strengths:
<strength #1>
<atzength
<strength ta>
Weaknesses:
sweakness 42>
sweakness 4n>
Table 13: Critic Prompt
will provide you response indicated by identifier 'Response.. <instruct ruction fon hezez.
reasoning for why response accurately completely addresses instruction:
Response: <sespense>
cinstructs on Provide the reasoning for the response above based on its relevance, ceseenns,,,mmmpttensss,, and accuracy when compared to the instruction. Do not include any preface or text after the reasoning.
Table 14: Verifier Prompt
Instruction Prompt: Given the following query, generate set of Rt unit tests that would evaluate the correctness of responses this query.
The unit tests should clearly stated
various aspects the query and comprehensive evaluation.
expected outcome..
The unit lests should be the form ol assertions that can be used to validate the correctness responses to the query.
-The unit test should be formatted like The answer mentions... "The answer states... ". "The answer uses... etc. followed by the outcome.
Solely provide the unit lests for the question helow. Do not provide text before or after the list. Only ourput the unil tests as a list of strings (e.g.. ('unit
lest H1". "unit lest #2", "unit #3'
Query: sinstruct
((a) With Unit Test Cap
Instruction Prompt: Given the following query, generate of unit tests that would evaluate the correctness of responses this query.
The unit tests should clearly stated
various aspects the query and ensure comprehensive evaluation.
unit
expected outcome.
The unit lests should be the form ol assertions that can be used to validate the correctness responses to the query.
'The unit test should be formatted like The answer mentions.." "The answer states... " 'The answer uses.... etc. followed by the expected outcome.
Solelyy provide the unit tests for the question helow. Do not provide text hefore or after the list. Only ourput the unil tests as list of strings (e-gr, ['unit
lest H1". "unit test 92', unit #3'
Query: cinstruction heze>
(b)) Without Unit Test Cap
Table 15: Unit Test Generator Prompt: With and Without Unit Test Cap
22

---

how An Architecture Search Framework for Inference-Time Techniques
Instruction Prompt: Compose un engaging travel blog posi about a recent trip to Hawaii, highlighling cultural experiences and musi see attractions.
Unit Test el: The blog post menions at least cultural experiences specific Hawaii.
2. Unit Tost #2: The blog post highlights at least three must-see attractions Hawaii
descriptive language would appeal to readers interested travel. 4. Unit Test 64; The blog post includes Lactual information about Hawaii's is culture, such as local customs, lestivals, or historical facts. structure, including an introduction, body, conclusion.
Unit Test
tone of the blog post is engaging
Test blog post
((a) mnstruction-Followvii Query
Alice and Bob have two dice. They roll the dice together, note the sum of the two vallaes shown, and repeat. For Alice to win, two consecutive tims (meaning, consecurive sums) to resalt 7. For to win, needs
Instruction Prompt:
eight followed seven. Who expect this game? 1. Unit Test #1: The response correctly identifies the winning condition for Alice (two consecutive sums of 7).
2. Unit Test #2: The response correclly identifies winning condition for Bab (a sum of followed by sum 7).
Unit Test
response explains probability achieving
7s when rolling dice.
4. Unit Test 64: The response explains the probabiliy of achaeving an 8 tollowed by a when rolling two dice.
Test responsee provides conclusion
likely based probubility analysis.
(b)) Reasoning Query
Table 16: Unit Test Examples
Given the following query. lest.
candidate
response,
and
unit
whether or not the response passes each unit response with the unit tests, retrieved documents, and aligns
tests,
evaluate
In your evaluation, you
should
consider
query.
Provide reasoning before you return your evaluation.
the end of evaluation, must finish with of verdicts corresponding each unit
You must include verdict with one these formatted options: [[assed]]' [Failed]'.
Here is an example of the output lormatl:
Unit Teat #1: [Passed]
Test [Failed]
Unit Test #3: [Passe]]
line and correspond to the unit test the position. Here query, response, and tests your evaluation:
Each verdict should he
Query: cinstruction hore>.
Candidate Response:
<reaponse>
Unit Teat #1: <tnit Teat #1>
Unit Test #2: <Unlt Test 42>
Unit Test #MI <Unit Test ANS
Table 17: Unit Test Evaluator Prompt
A.4. Bayesian Optimization for ARCHON
1. ARCHON SEARCH SPACE AND OBJECTIVE
The ARCHON configuration space can defined as = (Tn ffy Re, where:
Fg E [1,10] Number of generator models
E [1,5] Samples per generator (extends to [1,1000] for CodeContests)
Iy E [1,4] Number fusion layers, including the final fusion layer at the end
Iy E 2,10] Number of models per fusion layer, ranging from 2 to increments o
F2 E (0.1) Whether to use critic and ranker layers before each fuser
T. E (0,1) Whether
verification layer before final fusion
The total search space initially contains 18,750 configurations (00---544-).33- (0144-)))3117000) reduced to 9,576 after removing invalid configurations where: 1) initial generations exceed fuser context window (24 candidates); and 2) single fuser layer contains multiple fusers (zy= while x, 22)
23

---

An Architecture Search Framework for Inference-Time Techniques
Let (x) be the objective function evaluating an ARCHON configuration E X', defined as:
f(x) Performannccc( - >-Cost((z)
where Performance(z) is the accuracy on 20% sample of target tasks and Cost(z)) represents inference compute usage.
A.4.2. OPTIMIZATION PROCESS
We describe the components of the Bayesian optimization search approach for ARCHON.
First, define H. history of architectures and their resulting objective values, which accumulate throughout the optimization process. We use Expected Improvement (EI) the acquisition function for determining how select the next architecture configuration to search:
(2)
where f(*+) is the best-observed value of our objective so far for (x+ ff(x=(()))))))))))))) H. We use a Gaussian Process model as surrogate model for approximating f (x:).
We now describe the Bayesian optimization process. We first initialize the observation history H = 0. For timestep
t I,.....I, where T is the maximum number of iterations parameter, we do the following:
1. An architecture v, selected using the acquisition function, x; = -argmax DEX (I(xHH)
2. This architecture 2; is evaluated, and we obtain fzz).
3. We use fx;; to update our acquisition function and the surrogate model using H t (
The process continues until either:
A maximum number of iterations is reached,
Performance convergence: [[(*n+1)~ (xx
Budget exhaustion: Cost(x1,--.,.a)>>
For ARcHON''s implementation, we initialize with 230-240 random configurations, this was found to be optimal through empirical testing, Additional samples beyond this point provide diminishing returns and are better allocated configuration search. For our implementation, we utilize the Bayesian Optimization python package for global optimization with Gaussian processes.
This formulation allows ARCHON to efficiently explore the configuration space, requiring 88.5% fewer evaluations than greedy search and 90.4% fewer than random search, with Bayesian optimization finding the best architectures in 96.0% of iterations. Traditional greedy search methods may perform comparably for limited inference budgets (<20 calls), but Bayesian optimization becomes increasingly effective the search space and compute budget grow.
A.5. Bayes Optimization vs. Alternative Approaches
Search Techniques: Within the hyperparameter space, we explored three search algorithms for automating the development of inference-time architectures:
1. Random Search: Randomly selects combination of hyperparameters for our ARCHON architecture.
2. Greedy Search: Starting with a base ARCHON configuration, marginally changes each hyperparameter and test if it improves performance or not. it does, incorporate the change. If not, move on to the next hyperparameter.
3. Bayesian Optimization: Efficiently selects the most promising hyperparameter configurations for ARCHON by building a probabilistic surrogate model and leveraging an acquisition function for hyperparameter selection (Snoek et al. 2012 (Section A.4).
Nardi al. 2019)
To get our model ranking for the benchmark, we calculate the model ranking by testing each model individually on 20% sample of each dataset benchmark in the first stage of the search. To get fusion model ranking for the benchmark, use the same approach, testing each model's fusion performance with an ensemble of 10 randomly selected models from the available set. From our experiments, we found that the best generator and fusion models could vary widely dataset to dataset, making it beneficial to perform these rankings for new datasets (Table 29). For search, we use the same 20% sample of each dataset that was used for evaluating generation and fusion, allowing us to guide architecture search with improved evaluation speed while getting meaningful development signal.
24

---

An Architecture Search Framework for Inference-Time Techniques
Random Search
Greedy Search
Bayes Optimization
MT Bench
Arena Lama--ard-Auus
to Optimal Architecture (%)
100
100
Performance Relative
60-
60
40
40
20
20
1000
2000 3000
400C
5000
5000
7000
1000 2000 3000
4030
5000
6000
7000
Number of Configurations Tested
Number of Configurations Tested
Figure 10: Impact of Different Optimization Algorithms on ARCHON's Architecture Search: On the benchmarks MT Bench and Arena-Hard-Auue Autro we compare four approaches for finding the optimal inference- timee architecture: random search, greedy search, and Bayes Optimization. Bayes Optimization finds the optimal architecture in 88.5% less iterations compared to greedy search and 90.4% less iterations compared to random search.
we compare the effectiveness of each search algorithm on our explored benchmarks. While random search guarantees the optimal ARCHON configuration, we found Bayesian optimization be most effective in terms of tradeoff between finding the optimal configurations and minimizing the number of configurations tested. For 96.0% percent of the search iterations tested in (Fiuur 10, we found that Bayesian optimization had the optimal configuration amongst the four explored search algorithms. We use 230 initial samples for our Bayes Optimization architecture search (Section A. Bayesian optimization also found the best architecture configuration in 88.5% less evaluations than greedy search and %% less evaluations than random search.
Comparing Search Algorithms: In Figure 10
we explore how the number of initial testing points, the number of exploration iterations, and the ARCHON inference call budget impacts the effectiveness of Bayesian optimization. Additional initial testing points continue improving search efficacy up until 230-240 samples, where testing would be better delegated towards configuration search. For lower inference call budgets with ARCHON (e.g. <20 inference calls), Bayesian optimization proved less effective, performing more similarly to greedy search or random search given the limited search space (Table 21). Therefore, Bayesian optimization is more effective for more open-ended ARCHION architecture search with larger inference call budgets (e.g. >0 inference calls) whereas traditional component engineering might be better for more limited inference call budgets.
Bayesian Optimization Analysis: In Table 20
A.6. A ARCHON Architecture Algorithms Comparisons
of Init. Total Points
Iter. till Max. Config.
Comb. Iter.
Init. Points
of Total Configs
Iter. till Max. Contig,
Comh. Iter.
Configs
200
2.18%
353
553
200
2.18%
478
678
210
2 1.5%
324
534
210
2.29%
431
641
2.40%
301
220
2.40%
635
230
2.51%
284
514
230
2.51%
382
612
240
2.61%
261
501
240
2.61%
389
629
250
2.72%
265
515
250
2.72%
385
635
260
2.83%
256
516
260
2.83%
372
632
270
2.94%
638
Table 18: MT Bench
Table 19: Arena-Hard-Aurt
Table 20: Bayesian Optimization Hyperparameter Comparisons: On MT Bench and Arena-Harr Hard-Auto-, we compare Bayesian optimization configurations for the number of initial sample points. We find that 230 to 240 initial sample points minimizes the combined number of iterations (both initial sampling and exploring) to find the optimal configuration. For the configurations explored, the total number of hyperparameter choices is 9,576.
25

---

An Architecture Search Framework for Inference-Time Techniques
Iterations Convergence
Inference Budget
10
20
50
Random Selection
387
1152
2731
4359
5843
Greedy Search
Bayes Optimization
254
386
452
515
589
Table 21: ARCHON Architecture Search Algorithms Comparison by Inference Call Budget: For our comparison, we evaluate MT Bench.
A.7. ARCHON Benchmarks and Results
Example Count
Reference
Judge
Benchmark
Model
Model
Scoring Type
Metric
Pairwise
LCC. & Raw
AlpucaEival 2.0
805
Gur-o-Trbb
GPT-4-Ture
Comparison
Win Rates
Clande-3.5-Sonne
Pairwise Comparison
Arena-Hard-Aute
500
GPT-4-Turbo
Win Rate
GPT-4-0314
Pairwise
Adjusted
MT--encch
Claude-3.5 S-Sonnet
GPT--40314
Comparison
Win Rate
MixEval
2000
N/
N/A
Ground Truth
Accuracy
MixEival-Hard
NIA
Ground
Accuracy
200
MATH
(sampled from 5000)
N/A
N/A
Groun Truth
Pass@
CodeContests
(non-visual queries)
N/A
N/A
Goand Truth
Pass@l
Table 22: Benchmark Overview:
Evaluation configurations for AlpacaEval 2.0 et al. 2023). Arena-Hard-Auto (Li et al.
2024b), MT-Bench (Zheng et al. 2023), MixEval (Ni et al. 2024). MixEval Hard, MATH (Hendrycks al. 2021 and CodeContests [Li et al. 2022)
Osacle Selectioo
Randon Selectton
Herrieed Seleccion
Ramieed Top-5 Puber Resgonse
Win Rate Against Sonnet 3.5
0.65
0.65
0.50
0.60
Sonnet 0.5 GPT-40
Sonnet 3.5 GPT-00
0.45
0.45
0.10
6
10
10
Number of Repeated Samples
Number of Models In Ensemble
Figure 11: Performance Gains from Repeated Sampling, Ensembling, Ranking, and Fusing on Arena-Hard-Auto The ARCHON win-rate continues to grow significantly as we scale model sampling (left) or add additional models to the generator ensemble (right), increasing by 9.3% and 18.5%, respectively. These best results are achieved by selecting the top-5 responses and fusing them. The ensemble models are added based on their individual performance on this task, from best worse (Table 29). The oracle selection is the performance of picking the best answer generation out of all the generated samples from the ensemble. The results were averaged over 10 independent evaluation runs.
26

---

An Architecture Search Framework for Inference-Time Techniques
Arena-Hard-Auto
Model LLM System
Score
C.I.
Claude 3.5 Sonnet
N/A
N/A
GPT-4o
48.1%
(-2.3, 1.8)
Llama 3.1 405B Instruct
28.4%
(2.7,2.5
General-purpose
Open Sourcc
66.2%
(-2.4, 2.2)
ARCHON Architecture
Task-specific
69.0%
(-2.8, 2.5)
ARCHON Architectures
Generaa-purpose
Closed Source
70.5%
(-2.5, 2.0)
ARCHON Architecture
Task-specific
74.4%
(-2.3, 1.6)
ARCHON Architectures
General-purpose
AlI Source
72.5%
(-2.5, 1.8)
ARCHON Architecture
Task-specific
76.1%
(-1.8, 2.2)
ARCHON Architectures
Table 23; ARCHON Results Arena-Hard-Auto Results with Claude-3.--oonnet as Baseline Model: The baseline
model is Soaude3..5-Sonett (default baseline model: GPT-4-0314) while the judge model is GPT-4-Turbo.
MixEval Sub-Datasets
Model/ /LLM System
Calls
GSMRK
TriviaQA
DROP
MATH
BBH
AGlEval
Average
94.9
89.1
88.2
98.5
71.5
90.3
Claude
98.0
92.0
78.0
Llama 3.1 405B Instruct
98.2
87.9
89.6
91.5
95.8
73.2
89.6
General purpose
82.1
ARCHON Architecture
Task-spocific
ARCHON Architectures
98.2
96.7
95.6
98.5
98.8
84.2
95.7
Table 24: MixEval Results by Sub-Dataset: For the average computed, we do not introduce any weighting for each dataset.
MixHEvll Sub-Datasets
Infer: Calls
Model/1.LM System
GSM8K
TriviaQA
AGIFval
Average
GPT 4o 25-4-05-13
72.3
70.5
70.2
94.4
80.0
53.5
73.5
Claude 3.5 Sonnet
87.3
75.5
79.3
82.5
80.0
74.6
79.9
405B
General- General-purpoce
ARCHON Architecture
33
96.7
82.7
83.2
93.4
82.0
76.7
85.8
Task-spocific
ARCHON Architectures
37
98.9
86.2
85.2
96.2
86.0
80.1
88.8
Table 25: MixEval-Hard Results by Sub-Dataset: For the average computed, we do introduce any weighting for each dataset.
27

---

An Architecture Search Framework for Inference-Time Techniques
MILUU Math
HumanEval Python
GSM8K
MBPP
Model
Pass 1
Pass/
Pass 68
Pass
GPT-4o
87.5%
Clande 3.5 Sonnet
96.8%
90. 90.9%
90.2%
88.9%
Llama 3.1 405B Instruct
95.9%
85.4%
90.2%
88.6%
Table 26: Additional Math and Code Benchmarks Explored
Datasets
MT
Alpaca
Arena
Arena
MixEiall Mixtival MATII
Hard
Hard
GPT-4
GPT-4 Turbo
GPT-4
GPT-4
Judge Model
N/A
NIA
NIA
0314
Turbo
Turbo
Claude 3.5
GPT-4
Claude 3.5
GPT-4
Reference Model
Sonnet
Turbo
Sonnet
Turbo
NAA
NIA
NIA
Infer. Calls
Model LLM System
W.R.
W.R.
W.
Acc.
Acc.
W.R.
W.R.
6
GPT-4o QPT-40-2024-05-12 2024-05-13
44.7%
57.5%
51.3%
48.1%
80.3%
63.6%
88.0%
84.5%
Claude 3.5 Sonnei
N/A
52.4%
40.6%
NIA
80.9%
68.9%
89.7%
85.0%
Llama 4058 Instruct
28.4%
64.1%
88.9%
83.5%
6.
65.1%
59 8%
52.2%
84.2%
62.5%
87.3%
82.0%
MoA Lite
45.6%
59.3%
57.0%
40.6%
87.8%
61.1%
87.1%
83.0%
Open Source
ARCHON Architecture
General- purpesse
35
67.5%
63.0%
68.3%
66.2%
85.1%
65.5%
86.9%
86.5%
Task-specific
ARCHIN Architectures
44
71.6%
66.7%
70.7%
69.0%
89.5%
67.5%
89.6%
90.5%
Closed Source
ARCHON Architecture
General- pprppaeeppppsee
32
73.1%
63.5%
69.1%
70.5%
85.8%
67.7%
88.2%
88.0%
Task-specific
ARCHIO Architectures
40
77.5%
68.4%
72.1%
74.4%
90.2%
72.9%
90.4%
89.5%
Geseral-purpose
All Source
ABCHON Architecture
35
76.8%
65.8%
70.2%
72.5%
89.3%
70.1%
88.1%
90.0%
Task Saekiifecific
ARCHIN Architectures
39
80.4%
67.6%
73.3%
76.1%
92.1 16
72.9%
90.6%
93.5%
Table 27: ARCHON's Strong Performance on the Complete Evaluation Datasets after ARCHON Architecture Optimiza- tion: We find that ARCHON's inference-time architectures consistently outperform single-call state-of-the-ant LLMs, both open-source and closed source baselines, when evaluating the complete benchmarks TTable 22). We explore two configura- tions: architecture search for building custom ARCHON configurations for each individual benchmark and architecture search for building a single general-purpost ARCIION configuration for all the benchmarks (Section 4.1). We find that general ARCHON configuration lags behind the custom ones by only 3.2 percentage points, on average, across our all-source settings, which suggests the efficacy of general-purposs, inference-time architectures created with our framework. For Arena-Hard-Aun we also include configuration with Claude 3.5 Sonnet as stronger reference model for comparison against ARCHON inference-time architectures and to mitigate bias from GPT judges towards GPT generations. For MT Bench, we use GPT-4-0314 judge model instead of newer LLM judges to be consistent with previous results this benchmark. For our task-specific ARCHON architectures, we also provide the average inference calls across the given benchmarks. For our full-list of models explored, please see Table 28 For MATH, we use randomly sampled subset of size 200 for evaluation (Section
Table 22). We include our A ARCHON architecture results on the held-out 80% subset of each evaluation benchmark in Table
28

---

An Architecture Search Framework for Inference-Tine Techniques
MTT
Alpaca E.0
Auto
MixEval
MixEval Hard
MATH
CodeContests
Models
Gen
Fusion
Gen
Fusion
Gen
Fusion
Gen
Fusion
Gen
Fusion
Gen
Fusion
Fusion
GPT-4o
44.7%
61.9%
57.5%
64.5%
48.1%
69.2%
88.0%
89.4%
63 6%
65.4%
82.0%
81.0%
17.9%
19.4%
GPr-4-Turbo
42.2%
63.1%
55.0%
65.8%
48.1%
61.9%
88.9%
89.0%
64.1%
64.4%
79.5%
73.5%
9.3%
14.2%
Claude 3 Opus
30.9%
57.2%
40.5%
N/A
27.0%
47.9%
88.3%
88.2%
63.6%
64.0%
74.5%
74.0%:
10.0%
12.5%
Claude 3.5 Sonnet
71.9%
52.37%
63.6%
N/A
68.9%
83.5%
15.5%
Qwen 2 72B Instruct
35.0%
59.7%
37.48%
56.0%
14.5%
49.5%
86.5%
87.5%
58.7%
61.1%
81.0%
78.5%
3.6%
5.2%
DeepSeck LLM
67B Instruct
18.4%
20.0%
17.8%
17.1%
N/A
N/A
79.2%
N/A
42.5%
N/A
57.0%
N/A
5.7%
N/A
Qwen 72B Chat
24.7%
46.3%
36.6%
55.7%
14.4%
36.4%
84.5%
82.5%
50.3%
52.2%
71.5%
67.5%
15.0%
13.9%
Owen 110B Chat
34.4%
50.3%
43.6%
55.9%
21.9%
39.7%
85.3%
86.5%
51.8%
55.6%
67.0%
75.5%
3.6%
7.8%
Wizard 8x22B
53.8%
57.2%
44.7%
50.6%
45.6%
51.2%
83%
78.1%
54.3%
50.4%
76.0%
60.5%
7.1%
10.4%
Ilaaa 8B Instruct
33.1%
45.9%
25.6%
34.9%
11.9%
28.6%%
75.0%
57.5%
41.3%
46.5%
65.5%
60.5%
8.6%
7.8%
Llama 70B Instruct
45.0%
51.9%
35.6%
40.2%
23.8%
37.2%
85.7%
83.5%
61.1%
65.5%
74.0%
73.5%
20.7%
23.4%
IJama 3.1 405B Instruct
44.7%
N/A
N/A
N/A
Table 29: ARCHON Generation and Fusion Performances for Single Models: For Alpaca Eval 2.0, use the length-controlled win rate (LC WR). For fusion, we gather one candidate from each of the top-10 generator models.
A.8. ARCHON LLM Analysis
Parameter Count
Max Max Sequence Length
Model
Source Code
GPT-4o et
2024
Closed
GPT-4-Turbo (UpenA crai
2124
Closed-Soure
128K
Claude-3-Opus (Anthropic
2024
Closed-Souree
200K
Selude-3--Soomss Aaliropic
2024
Closed-Souree Source
200K
Claude Haiik Anthropic
Closed Source
200K
1lama-311-70B. Instiii
Duueey al
1024
Open Soun-S
70R
8k
Llama-3.1 10mm-331-405S-Inntmae
(Dibey at
2024
Open-Source
70B
DeepSeck LLM 67B Chat ef.
Open-Source Source
67B
32k
Qwen2 72B Instruct
Owen 2124
Open Source
72B
321.
Qwen1.5 110B Chat
Open-Source
L10B
Qwenl. 5 72B Chat
[Bar cl
20221
Open Source
72B
32k
(Jian
Open Source
Wizard 8x22R
eF
2024
Open-Source
176B
32k
dbrx-instruct (Dafabricks
2024
Open-Source
132B
32k
princeton-alp// prinecton-alp/Llame--33 hall:arallppaaaamm---nsttttt--- -imm
Meng et al
2024
Open-Source
8B
8%
princetom nip/Llamm Instruct
Sten el
Open Source
princeton-nliyll.
nlp/ Jama- 3-Instmuct-BB-PP RDP
Mong et
M1A
Open-Source
8B
8k
[Meng ct
Open-Source
Llama-3. 1-81--8--8 1-8B-Instruct Dubey et 2024
Open-Souree
8B
8%
Qwen2-7B-Bll
Oween
Open SourcS
Qwen/Qwen 1.5.7B.
[Far ei 2021
Open-Source Source
7B
32k
mistralai/Misttrll http:/iisrrrl------sttt------
(Ting al 2023g
Open-Source
2024
Open-Source Source
7B
32k
Ahdin et A17A
Open-Source Source
4B
4k
HugginngFaoH4//gggyyyytbbb
Open-Source
32k
microsolt/Phi-33 3-smalll-
(Abuia el
2024
Open Source
7B
8%.
snorkelai/ lai/S/ok////l Mistral PairRM.
Open Source
mistralaiMMistral- Instnnttuct-
(Tinng of al
Open-Source
7B
32k
Table 28: Models Tested with ARCHON.
29

---

An Architecture Search Framework for Inference-Time Techniques
Similarity (%)
Inference-Time Architecture
Arcua-Hard Auto
MixEval Hard
Code Contests
MI Bench AlpacaEival 2.0
Mixbvaa
MATH
Best Open-Source 70B+ Model, Sampled 8 Times Foser
45.3%
52.1%
48.4%
55.2%
58.9%
65.2%
63.7%
Ensemble (8 Top Models), Sampled Once Each + Fuser
31.6%
34.1%
28.9%
38.6%
40.9%
57.1%
53.4%
Table 30: Jaccard Similarities between Candidates Responses and Fused Response by Benchmark: For the fuser, use the best best-performing 70B+ model for each benchmark.
MT-Bench
Arena-Hard-Auto
MixEval
AlpacaEval 2.0
MixEval-Hard
Performance Plateau from Solely Increasing Fusion Layers
1.0
Win-Rate / Accuracy
0.8
0.7
0.5
0.4
Number of Fusion Layers
Figure 12: Fusion Layer Efficacy by Benchmark: From solely scaling the fusion layers, we see limited benefits across the benchmarks explored but when we add other inference-tine techniques, such as Critic and Ranker, we see increased downstream performance as we continue scaling inference-time compute (Figure 4). use an 8-model ensemble the top Generator models for each benchmark Table 29). For our Fuser layers, we use the best Fuser model for the final fuser layer (Table 29). For the intermediate layers, the top-8 Fuser models for each benchmark.
30

---

An Architecture Search Framework for Inference-Time Techniques
A.9. ARCHON Architectures
Prompt
Coverttrr
Outpet
Figure 13: All-Source Generalizable ARCHON Architecture: Using ARCHON's architecture search, we found this all-source ARCHON configuration to be effective across the henchmarks explored (except for CodeContests). In the diagram above, we use 10 SOTA all- all-source LLMs to create multiple successive layers of critic, ranker, and fusers, with each successive fuser layer having less fusers to produce "funneling" effect as the candidate generations are processed. The layers of critic, ranker, and fuser led to better candidate generations through iterative critique and rewriting. Each of the initial Generator models were sampled once.
Ceverator Hee
Puwer
Promot
Jander
Puter
Figure 14: All-Source ARCHON Architecture for Instruction Folllowig and Reasoning: Using ARCHON'S architecture search, we found this all-source ARCHON configuration to be effective across the instruction-folloiing benchmarks explored (MT Bench, AlpacaEval 2.0, ArenaHardAuto).
EPhie
CFfee
Generator
Faseer
Prompt
duvei La
Qude Mode
> Output
Generator
Faser
Wusid
Whane
B: samples each
Figure 15: All-Source ARCHON Architecture for Math: Using ARCHON's architecture search, we found this all-source ARCHIN configuration be effective across the math benchmarks explored (MATH).
31

---

An Architecture Search Framework for Inference-Time Techniques
4 Rounds
67748
Fusee
Prompt
Generator
Critic
Critic
Unit Teater Generator
Unit Test Explestor
Output
67148
Used Per Round
Claude Sonnet
2 Uamm 4058
3. CPT-to
Figure 16: All-Source ARCHON Architecture for Coding: Using ARCHON's architecture search, we found this all- source
ARCHON configuration be effective across the coding benchmarks explored (CodeContests)..
A.10. ARCHON by Inference Compute Budget, Model Size, and Cost
Datasets
Number of Inference Calls
Alpaca Eval 2.0
MixEval
MixEval
Bench
Hard Auta
Hard
55.0%
44.7%
45.6%
86.5%
61.1%
Models
10
52.5%
50.6%
45. 45.6%
86.5%
63.9%
70
60.4%
30
69.2%
64.5%
69.0%
89.5%
67.5%
50
71.6%
66.7%
69.0%
89.5%
67.5 %
45.0%
57.5%
48.1%
88.9%
68.9%
Closed Models
10
57.1%
63.2%
68.4%
90.00%
70.1%
30
70.2%
68.8%
77.4%
90.6%
72.9%
75.5%
68.8%
77.4%
72.9%
80.4%
68.8%
77.4%
90.6%
72.9%
Table 31: ARCHON with Different Inference Budgets: For AlpacaEval 2.0, we use the length-controlle win rate (LC WR).
Datasets
Models / L.LM Systems
MT
Alpuca Eval
Arena Hard
MixEval
MixEval Hard
Bench
Best 7B Model, I-Sample
15.7%
41. 11.0%
18.3%
76.2%
46.1%
Best 7B Model 10 Sample Ranking
16.5%
43.2%
18.9%
78.4%
48.5%
10-Model, I-Sample Ensemble Ranking
22.4%
48.2%
25.6%
81.5%
52.9%
Model, Sample
Fusion
73.2%
M0.Madel, J-Smplle Frsembble
Top-5 Ranking Fusion
15.9%
41. 41.2%
18.0%
75.1%
46.9%
10-Model, 1-Sample Eusemble Critic Fusion
10.5%
38.4%
16.5%
71.4%
42.5%
Table 32: ARCHON with 7B Open-Source Models: For AlpacaEval 2.0, we use the length-controlled win rate (L.C WR). We open-source 7B models for testing from Table 28
32

---

An Architecture Search Framework for Inference-Time Techniques
Models
Cost per Million Input Tokens
per
Million Output Tokens
Claude 3.5 Somnet
$3
$15
Claude 3.0 Opus
$15
GPT-4o
$5
$15
$10
$30
TogetherAT Ilama 3.1 405B Instruct
$5
$5
TogetherAI
TagetherAI - Other Models
$0.90
$0.90
Table 33: Model API Costs of November 2024
Cost (S) per Query
Model/ I.L.
MT Rench Alpucalival 2.0
Arena Hard
MixEival
MixEval Hard
MATH
Code Contests
Auto
0.0171
0,0212
0.0226
0.0325
GPT-4o
0.0481
0.0236
0.0324
0.0357
0.0361
0.514
0.562
Llama 3.1 405B Instruct
0.0281
0.0174
0.0185
0.0212
0.0205
0.305
0,372
General Purpose
ARCHON Architscture
0.364
0.189
0.195
0.284
0.252
0.375
0.461
Task Specific ARCHON Architecture
0.210
0.265
Table 34: ARCHON Costs per Query by Benchmark
33


Reviews:
[{"review_id": "frB2kDc7c6", "reviewer": "Reviewer_Bbe6", "summary": "This paper proposes a framework called ARCHON for optimizing combinations of inference time techniques to improve the performance of large language models. Their approach combines multiple inference techniques such as ensembles, rankings, etc. and uses automatic architecture fusion search to find the optimal combination of different LLMs.\n\n\n## update after rebuttal\nI appreciate the authors' detailed ablation results and clarifications in the revised version. They have presented more discussion regarding the placement of generation, critique, ranking, fusion, and verification modules, including their \"Rules of Construction\". This addresses some of my earlier concerns about whether shifting the position of modules can degrade performance. The additional experiments on GPQA, MMLU, and MMLU Pro also help illustrate the robustness of the framework.\n\nHowever, I still find the overall novelty to be somewhat incremental since ARCHON is largely an engineered combination of known inference techniques. Although the empirical gains are encouraging, the paper would benefit from clearer theoretical or conceptual insight into why specific module orderings are consistently beneficial and how these search strategies might extend to more complex tasks (e.g., advanced math or Olympiad-level problems). \n\nOn balance, while the rebuttal clarifies many issues, I remain slightly unconvinced that this submission offers sufficient conceptual novelty for a strong accept. I maintain my recommendation as a weak reject, though I acknowledge the paper’s potential and encourage the authors to strengthen the theoretical framing and expand the demonstration of generality in future submissions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper! We appreciate your feedback and comments.\n\n*The claim that Utilities of Inference-Time Techniques is not well studied:*\n- We agree that understanding the utilities of inference-time techniques is central to our contributions. We've enhanced the explanation of ablation findings in our revised draft (highlighted green in Sections 3.1, 3.2): https://storage.googleapis.com/anonymous-files/archon.pdf\n- Our key findings (Table 5, Figures 4, 7, 8) show the following performance increases:\n   - Generation: 10.5% across all tasks when increasing from 1 to 10 generators\n   - Ranking: 5.7% for instruction-following, reasoning, and math tasks\n   - Fusion: 13.2% across all tasks\n   - Critiquing: 8.1% when added before fusion for all tasks\n   - Verification / Unit Testing: 5.4% improvement across all tasks\n- Combining components leads to compounded gains—ranker+critic+fusion yields 15.1% average improvement across all benchmarks and models, while 10+ generators with verification/unit testing boosts math/coding performance by 12.1% (Tables 5-9).\n- These results come from extensive ablation studies across 7 benchmarks and 3 model classes detailed in Appendix A.3 (Tables 18-22).\n\n*Will shifting position for different modules cause performance degradation?:*\n- Our findings in Appendix A.3 (Tables 5-9) show:\n   - Generator components are most effective at architecture start\n   - Critic components work best before ranker/fusion components\n   - Critic, ranker, and fuser components can be stacked sequentially for improvement\n   - Verifier and unit testing components are most effective at architecture end\n- These findings informed our \"Rules of Construction\" (Section 3.2) to prevent performance-degrading placements. We've highlighted these findings in green in the revised paper and added pointers to the ablation study.\n\n*Lack of explanation about searched framework. How architecture differs across tasks is not well demonstrated:* \n- The all-source generalized Archon architecture is included in Figure 3 (Section 3.2). All generalized and specialized architectures from    \n- Table 1 are described in detail in Appendix A.9 (Figures 12-15) and our supplementary code.\n- In Section 4.3 (\"Archon by Task\"), we discuss task-specific architecture distinctions with these key trends: \n   - Instruction-following tasks benefit from additional generator models and deeper fusion layers (17.8% improvement in win-rate from 1 to 10 generators). \n   - For reasoning, task-specific architectures show meaningful improvements (10.1% improvement for specialized vs. generalized architectures). \n   - For coding, unit testing and increased sample scaling significantly improve performance (44.3% boost in Pass@1). \n   - Instruction-following and reasoning architectures use multiple critique-rank-fuse layers with diverse LMs, while math/coding architectures use repeated samples from a single LM before applying unit-testing or verification.\n\n*Incremental novelty. This paper is mainly combining different strategies without strong motivation or insights:*\n- The three intellectual contributions underlying Archon are:\n   - Observing that existing inference techniques can be naturally combined, studying their interactions, and optimizing model selection to maximize technique efficacy\n   - Providing an automatic approach for optimal inference technique combinations, yielding significant improvements over SOTA baselines over restricted and unrestricted compute budgets.\n   - Archon architectures generalize to unseen benchmarks, outperforming alternative frameworks like ADAS and AFlow\n- Point (1) motivated Archon's design through ablation studies across 7 benchmarks and 5 model classes (Sections 3.1, 3.2; Appendix A.3). No existing frameworks effectively search this large inference architecture space while balancing cost-performance tradeoffs—please inform us if we've overlooked comparable baselines.\n- Point (2) had never been pursued by previous inference-time papers, allowing us to exceed SOTA LMs and emerging frameworks by +15% accuracy and +30% FLOP efficiency (Table 1; Figures 5-6). While ADAS and AFlow plateaued at 30 PFLOPS per query, Archon continues improving beyond 50 PFLOPs per query.\n- For Point (3), we evaluated generalized Archon architectures on three new out-of-domain benchmarks: GPQA, MMLU, and MMLU Pro. Without using any ground truth labels for architecture adaptation, our generalized architectures preserve 91-95% of the performance gains from specialized architectures trained exclusively for these benchmarks (Table 2). In contrast, generalized ADAS and AFlow architectures only achieve 66% and 73% of their specialized performance.\n\nWe welcome any further questions! Thank you again for your feedback and comments! In light of the additional experiments and clarifications to address your comments, we would really appreciate it if you would re-examine our paper and consider raising your score.", "claims_and_evidence": "- The claim that to understand the Utilities of Inference-Time Techniques is not well studied. Lack of the evidence about the improvement percentage from ranking/fusion/verification/etc. with different position. Will shifting the position for different modules causes several performance degradation? Also lack of explanation about the searched framework. How specific the architecture differs from tasks to tasks is not well demonstrated.", "methods_and_evaluation_criteria": "Yes, make sense", "theoretical_claims": "N.A. in this paper.", "experimental_designs_or_analyses": "See concerns under `Claims And Evidence` regarding comprehension score and in-context evaluation.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**weakness**\nIncremental novelty. This paper is mainly a combination of different strategies. I don't see a strong motivation or insights with explanation or evidence.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I'd like to see the framework performance on frontier reasoning datasets such as GPQA, AIME25 and OlympiadBench.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RUtZIlgLOB", "reviewer": "Reviewer_TLQP", "summary": "The paper introduces a framework for optimizing inference-time techniques in large language models (LLMs). The contributions, as stated in the paper, are an algorithm that identifies optimal ways to combine inference-time techniques (such as ensembling, ranking, fusion, verification, and critique), as well as understanding the interactions between inference-\ntime techniques and their utilities. \n\nThey evaluate the method on multiple instruction-following, reasoning, and coding benchmarks, demonstrating improvements over state-of-the-art closed-source models.\n\nARCHON relies on Bayesian optimization to search over a large space of possible architectures, selecting the best-performing configuration for a given task. The results indicate that combining multiple inference-time techniques in a structured manner provides better performance than single techniques alone. The authors release an open-source framework to facilitate further research in this area.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper! We appreciate your feedback and comments.\n\n*However, compute budgets are not always controlled for... controlling only for token budget as in Table 1 and Figure 5 doesn't give a clear picture:*\n- We appreciate your concerns regarding compute matching experiments. For our comparisons, we originally utilized input/output tokens and inference calls because:\n   - We can only speculate on the true FLOPs of the closed-source models used by baseline approaches (i.e. o1 and GPT-4o for ADAS and AFlow)\n   - Comparing FLOPs between larger and smaller models can be misleading when evaluating efficiency (e.g. 1 PFLOPs with open-source 70B LMs vs. 1 PFLOPs with closed-source 200B LMs)\n   - The API providers, such as OpenAI, Anthropic, Google, Together, etc., price their APIs by input and output tokens per query, correlating directly with their FLOP costs at inference.\n- With our measurements, Archon is 20.0% more inference call efficient, using 15.1% less input tokens and 13.5% less output tokens compared to alternative frameworks (ADAS and AFlow), while delivering 15.1% better performance across all benchmarks (Table #1; Figure #5).\n- Importantly, Archon achieves these gains using open-source LMs of 72B parameters or less, while ADAS and AFlow use GPT-4o with an estimated 200B active parameters per forward pass (https://epoch.ai/gradient-updates/frontier-language-models-have-become-much-smaller).\n- To further address your comment, we've added a column in Table 1 showing PFLOPs per query using estimated active parameters from Epoch AI: https://storage.googleapis.com/anonymous-files/archon.pdf. When benchmarked against ADAS and AFlow in an unrestricted budget setting, Archon architectures use 32.1% less FLOPs while using models that are, on average, 53% smaller than GPT-4o.\n- In a restricted budget setting, this gap increases as we scale the allowed budget, allowing Archon to achieve the same performance as ADAS and AFlow with 38.9% less budget across various settings (Figure 6).\n\n*The improvements shows in Table 1 are quite predictable and not surprising:*\n- We'd like to respectfully push back on this claim. Additional inference compute does not always translate to better task performance (Figure 5). Unlike ADAS and AFlow, which plateau at 30 PFLOPs per query using GPT-4o, Archon continues to improve beyond 50 PFLOPs across all task types (Figure 6): https://storage.googleapis.com/anonymous-files/archon.pdf\n- Even with unlimited budget for inference-time architectures (Table 1, Figures 5-6), Archon architectures outperform alternatives by +15% accuracy while using 32.1% less FLOPs, 20.0% less inference calls, 15.1% less input tokens, and 13.5% less output tokens.\n\n*The contributions are closely related to recent trends in scaling test time compute, although they don't provide significant novelty:*\n- The three intellectual contributions underlying Archon are:\n   - Observing that existing inference techniques can be naturally combined, studying their interactions, and optimizing model selection to maximize technique efficacy\n   - Providing an automatic approach for optimal inference technique combinations, yielding significant improvements over SOTA baselines over restricted and unrestricted compute budgets.\n   - Archon architectures generalize to unseen benchmarks, outperforming alternative frameworks like ADAS and AFlow\n- Point (1) motivated Archon's design through ablation studies across 7 benchmarks and 5 model classes (Sections 3.1, 3.2; Appendix A.3). No existing frameworks effectively search this large inference architecture space while balancing cost-performance tradeoffs—please inform us if we've overlooked comparable baselines.\n- Point (2) had never been pursued by previous inference-time papers, allowing us to exceed SOTA LMs and emerging frameworks by +15% accuracy and +30% FLOP efficiency (Table 1; Figures 5-6). While ADAS and AFlow plateaued at 30 PFLOPS per query, Archon continues improving beyond 50 PFLOPs per query.\n- For Point (3), we evaluated generalized Archon architectures on three new out-of-domain benchmarks: GPQA, MMLU, and MMLU Pro. Without using any ground truth labels for architecture adaptation, our generalized architectures preserve 91-95% of the performance gains from specialized architectures trained exclusively for these benchmarks (Table 2). In contrast, generalized ADAS and AFlow architectures only achieve 66% and 73% of their specialized performance.\n\nWe welcome any further questions. Thank you again for your feedback and comments! In light of these additional experiments and clarifications to address your comments, we would really appreciate it if you would re-examine our paper and consider raising your score.", "claims_and_evidence": "The claim that ARCHON produces architectures that outperform existing state-of-the-art models is well-documented.\nThe authors report consistent improvements across MT-Bench, AlpacaEval, MixEval, MATH, and CodeContests, often surpassing top closed-source models. However, compute budgets are not always controlled for, making it difficult to isolate whether the improvements come from ARCHON’s methodology or simply from increased inference cost. The improvements shows in Table 1 are quite predictable and not surprising.\n\nAdditionally, controlling only for token budget as in Table 1 and Figure 5 doesn't give a clear picture. I understand that, for several models tested, the flops and the throughput are not available, but this causes a lot of issues when comparing the models from an inference budget perspective.", "methods_and_evaluation_criteria": "The evaluation uses multiple diverse benchmarks, covering instruction-following, reasoning, and coding tasks, which strengthens the paper’s claims.\nA significant concern as I said is the absence of compute-normalized comparisons. Many baseline models receive far fewer inference calls, and no information is available in terms of flops and generation time (I understand that this is an intrinsic limitation of dealing with closed source models). This does create confounding effects in the experimental results.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "-", "relation_to_broader_scientific_literature": "The contributions are closely related to recent trends in scaling test time compute, although they don't provide significant novelty.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "The open source library can be a very useful contribution for the community and speed up research on test-time compute.", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6s7NVsq8DF", "reviewer": "Reviewer_Uk8K", "summary": "This work focuses on how to combine inference-time techniques of LLMs to achieve better performance. It first proposes a framework termed Archon, which is able to incorporate different inference-time techniques rather flexibly. Then, a search method based on Bayesian optimization is designed, which takes as input the target benchmark, the budget for inference, the set of available LLMs, and the available inference-time techniques, and takes as output the layered architecture of different techniques defined in Archon.\n\n\n\n### update after rebuttal\n\nI agree with Reviewer mgTm's opinion that \"an optimized way to use them is very beneficial\", and my concern on the budget alignment is addressed. My remaining concern is not from the technical perspective but lies on the paper organization. IMHO, given that the paper does not introduce new techniques, the motivations and insights that drive this research become rather important, since they may motivate followers and differentiate this manuscript from technical reports. Overall, I am on the boundary of acceptance and rejection, and I would defer to AC's recommendation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper! We appreciate your feedback and comments.\n\n*However, I find that the major analyses/conclusions are not presented in Section 3.:*\n- We agree that understanding the utilities of inference-time techniques is central to our contributions. We've enhanced the explanation of ablation findings in our revised draft (highlighted green in Sections 3.1, 3.2): https://storage.googleapis.com/anonymous-files/archon.pdf\n- Our key findings (Table 5, Figures 4, 7, 8) show the following performance increases:\n   - Generation: 10.5% across all tasks when increasing from 1 to 10 generators\n   - Ranking: 5.7% for instruction-following, reasoning, and math tasks\n   - Fusion: 13.2% across all tasks\n   - Critiquing: 8.1% when added before fusion for instruction-following, reasoning, and math\n   - Verification / Unit Testing: 5.4% improvement across all tasks\n- Combining components leads to compounded gains—ranker+critic+fusion yields 15.1% average improvement across all benchmarks and models, while 10+ generators with verification/unit testing boosts math/coding performance by 12.1% (Tables 5-9).\n- These results come from extensive ablation studies across 7 benchmarks and 3 model classes detailed in Appendix A.3 (Tables 18-22).\n\n*For efficiency, it seems inappropriate to use the token budgets:* \n- We used input/output tokens and inference calls for compute-matching Archon because:\n   - True FLOPs of closed-source models used by ADAS, AFlow, and frontier LMs can only be estimated\n   - FLOPs comparisons between different-sized models can be misleading\n   - API providers price by tokens, correlating with their inference FLOP costs\n- Overall, Archon is 20.0% more inference-call efficient, using 15.1% less input tokens and 13.5% less output tokens than ADAS/AFlow while delivering 15.1% better performance (Table #1; Figure #5).\n- Importantly, Archon uses open-source LMs ≤72B parameters, while competitors use GPT-4o with estimated 200B active parameters: https://epoch.ai/gradient-updates/frontier-language-models-have-become-much-smaller\n- To further address your comment, we've added PFLOPs estimates in Table 1 using the estimated active parameters from EpochAI. In unrestricted settings, Archon uses 32.1% fewer FLOPs with models 53% smaller than GPT-4o. In restricted budget settings, Archon achieves equal performance with 38.9% less budget (Figure 6).\n\n*My major concern is that the paper heavily relies on the appendix. If accepted, how can you reorganize the paper for better readability while fitting page limits?:*\n- We appreciate your concerns about appendix reliance. To address your concern, we've:\n   - Added more ablation highlights in the main paper to better explain Archon's design decisions (highlighted in green, Sections 3.2, 3.3)\n   - Added table of contents for easier navigation of Appendix (A.1)\n   - Fixed misordering of Appendix references to corresponding tables/figures\n- These changes make the paper more self-contained while still adhering to final conference page limits.\n\n*How does such a design achieve optimality? How does it address challenges/limitations of previous works?:*\n- The three intellectual contributions underlying Archon are:\n   - Observing that existing inference techniques can be naturally combined, studying their interactions, and optimizing model selection to maximize technique efficacy\n   - Providing an automatic approach for optimal inference technique combinations, yielding significant improvements over SOTA baselines over restricted and unrestricted compute budgets.\n   - Archon architectures generalize to unseen benchmarks, outperforming alternative frameworks like ADAS and AFlow\n- Point (1) motivated Archon's design through ablation studies across 7 benchmarks and 5 model classes (Sections 3.1, 3.2; Appendix A.3). Existing frameworks can't effectively search the inference design space while balancing cost-performance tradeoffs—please note any comparable baselines we missed.\n- Point (2) had never been pursued by previous inference-time papers, allowing us to exceed SOTA LMs and emerging frameworks by +15% accuracy and +30% FLOP efficiency (Table 1; Figures 5-6). While ADAS and AFlow plateaued at 30 PFLOPS per query, Archon continues improving beyond 50 PFLOPs per query.\n- For Point (3), we evaluated generalized Archon architectures on three new out-of-domain benchmarks: GPQA, MMLU, and MMLU Pro. Without using any ground truth labels for architecture adaptation, our generalized architectures preserve 91-95% of the performance gains from specialized architectures trained exclusively for these benchmarks (Table 2). In contrast, generalized ADAS and AFlow architectures only achieve 66% and 73% of their specialized performance.\n\nThank you again for your feedback and comments! In light of these additional experiments and clarifications to address your comments, we would really appreciate it if you would re-examine our paper and consider raising your score.", "claims_and_evidence": "Partially. In the introduction, the authors claimed \n> we evaluate the utilities of a comprehensive set of existing and proposed inference-time techniques\n\n> we analyze the interactions between inference-time techniques and explore the benefits of adding new models and new techniques individually.\n\nHowever, I find that the major analyses/conclusions are not presented in Section 3. In my humble opinion, these are rather important, as they could bring new insights and guide readers to the proposed methods.", "methods_and_evaluation_criteria": "Reasonable.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Overall reasonable. However, since different LLMs take different costs, it seems inappropriate to unified the budgets as input/output tokens. Subsequently, it is unclear how can we align the budgets of different counterparts for a fair comparison.", "relation_to_broader_scientific_literature": "Inference-time techniques are important for empowering LLMs.", "essential_references_not_discussed": "Sufficient.", "other_strengths_and_weaknesses": "Overall, this work studies a timely and important topic, the framework is reasonably designed, and the experimental results are positive. \n\nHowever, my major concern is that the paper heavily relies on the appendix (e.g., Section 3.1 relies on Table 8, Section 3.2 relies on Table 9). This makes the paper hard to follow.\n\nSubsequently, since the analyses are placed in the appendix, there lacks motivation for your architecture design. For example, how such a design achieves optimality? And how does it address the challenges/limitations of previous works?", "other_comments_or_suggestions": "The tables and figures are not referred to in order.", "questions_for_authors": "Please elaborate the analyses, which are important for readers to understand the value and motivation of your work. \n\nIf this paper is accepted, how can you reorganize the paper so that it is much more readable while fitting the page limit?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bdzvyNqNWr", "reviewer": "Reviewer_mgTm", "summary": "The paper introduces an inference-time \"architecture search\" framework (not to be confused with NAS) designed to optimize language model performance on specific benchmarks. The framework systematically selects and integrates multiple inference-time techniques (ensembling, fusion, ranking, critiquing, verification) using Bayesian optimization to find optimal architectures for a given task and compute budget. The framework supports publicly available and closed-source models, showing performance gains over SOTA LLMs and inference-time methods, particularly in instruction-following, reasoning, and coding tasks​.\n\n## update after rebuttal\n\nThe authors have answered my concerns with every experiment I found lacking, and even more. Even after reading other reviews, I still believe this paper strongly merits acceptance, as I found most criticisms unconvincing, except for a single one of Uk8K’s, which the authors have responded to in our discussion. \nI think the paper has novelty and value. Even if some of the primitives from the paper existed prior to it, an optimized way to use them is very beneficial to practitioners utilizing agents.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper! We appreciate your feedback and comments.\n\nWe’d like to address each of your concerns individually:\n- *Weaknesses: -The biggest weakness of the method, in my opinion, is its requirement to get specific benchmarks to optimize the architecture for. While the setting is realistic for certain scenarios where we know exactly what's the type of data we'll encounter, a lot of scenarios require a less specialized approach. With that being said, I'm fine with the paper limiting its scope to the aforementioned subset of scenarios.*:  \n    - With regards to architecture optimization, we do currently use a dev set of at least 20 queries for optimizing the Archon architecture learned (Section 3.3). \n    - However, the learned architecture can be quite generalizable. For example, by the development sets of the seven benchmarks explored, we were able to create generalized Archon architectures that preserved 90-95% the performance of specialized Archon architectures (Table 1).\n    - This suggests that the learned Archon architectures generalize beyond any individual set of queries and perform well on unseen domains. \n    - To test this hypothesis, we ran an additional set of experiments and evaluated the generalized Archon architectures on three new benchmarks: GPQA, MMLU, and MMLU Pro. These benchmarks cover a variety of topics ranging from science to business to the humanities. \n    - **Note:** We did not use any ground truth labels from these datasets for developing Archon architectures.\n    - We find that our generalized Archon architectures are capable of preserving 91 to 95% the performance gains of specialized Archon architectures trained exclusively for these benchmarks individually.\n    - The generalized ADAS and AFlow architectures only achieve 66% and 73% of their specialized architecture performance, respectively.\n    - We highlight these added results in Table 2 of our revised paper: https://storage.googleapis.com/anonymous-files/archon.pdf.\n\n- *It's unclear to me how the computation budget constraints are enforced during inference and optimized for during the search*:  \n    - We impose the constraints by excluding any architecture that would exceed the inference call, input token, or output token budgets from the search space:\n    - Multiple restrictions could be added. For example, you can filter out architectures with more than 20 inference calls or more than 20,000 input tokens.\n    - This prevents our Bayesian optimization algorithm from even considering these invalid architectures in our architecture search (Appendix A.7).\n    - To address your concern, we added a new section to discuss this in Section 3.3 called “Adding Search Restrictions”.\n\n- We agree that the name Archon might elicit concepts surrounding model architectures. However, in our application, we use the term “archon” to highlight that we are using an *architecture of LMs*, rather than an architecture of neurons as in NAS.\n- Also, thank you for highlighting potential avenues for improving the Archon construction rules! \n    - Given an infinite inference budget, future work can further leverage existing techniques, such as generation and fusion, while even developing new components to better translate additional inference compute to improved performance, such as expanded verification systems.\n\nWe welcome any further questions. Thank you again for your feedback and comments!  In light of these additional experiments and clarifications to address your comments, we would really appreciate it if you would re-examine our paper and consider raising your score.", "claims_and_evidence": "All claims are supported as far as I can tell.", "methods_and_evaluation_criteria": "Evaluation setup: The paper benchmarks Archon against closed-source and publicly available models and existing inference-time techniques (MoA, ADAS, AFlow).\nPerformance is measured via accuracy and compute efficiency.", "theoretical_claims": "The paper does not introduce new mathematical proofs or theoretical claims.", "experimental_designs_or_analyses": "-The experiments use a diverse set of benchmarks (MT-Bench, Arena-Hard-Auto, MixEval, etc.), which adequately cover LLM performance.\nModel selection: The comparison against state-of-the-art LLMs (GPT-4o, Claude 3.5 Sonnet, Llama 3.1 405B) ensures a fair baseline.\n-The Bayesian optimization search space is well-justified, though further ablation studies could clarify interactions between components​.", "relation_to_broader_scientific_literature": "Inference-time architectures are an active area of research, and the paper builds on techniques similar to MoA, RouteLM etc.\nUnlike papers that optimize a single model, Archon uses multiple inference-time techniques simultaneously​.", "essential_references_not_discussed": "Not as far as I'm aware.", "other_strengths_and_weaknesses": "Strengths:\n-Automated inference-time architecture search framework integrating multiple techniques provides a way to utilize these various techniques in practice. It is a considerable strength.\n-Well written.\n-Outperforms other SOTA models and techniques.\n-Being able to use both publicly available and closed-source models in the architecture is a plus.\n\nWeaknesses:\n-The biggest weakness of the method, in my opinion, is its requirement to get specific benchmarks to optimize the architecture for. While the setting is realistic for certain scenarios where we know exactly what's the type of data we'll encounter, a lot of scenarios require a less specialized approach. With that being said, I'm fine with the paper limiting its scope to the aforementioned subset of scenarios.\n\n-It's unclear to me how the computation budget constraints are enforced during inference and optimized for during the search. From various places across the main body, including A.7.2. it seems the budget consists of model calls. But from various figures (such as Figure 1) it seems the budget is defined as the input/output token ratio (which seems more practical to me, but more challenging to measure). I'd love an explanation, but nonetheless, I think it should be clarified in the paper itself too.\n\n-Minor naming: I'm not sure I'd call Archon an \"architecture\" as it's a highly overloaded with the model's architecture. Especially since it's not learned. As a reader, I was expecting a NAS paper.\n\n-Minor (not a weakness): there are several \"construction rules\" that could be changed for cases with higher budgets. e.g., hypothetically speaking, in resource-rich tasks, generators could appear again in later layers to generate later parts in the response after the best candidates were to be chosen from earlier layers (e.g., writing a book chapter by chapter). This comes with a trade-off of expanding the search space.", "other_comments_or_suggestions": "-I appreciate the comprehensive explanation of the Bayesian Optimization (Snoek et al., 2012) algorithm at the appendix.", "questions_for_authors": "How the computation budget constraints work (see above).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Archon, a modular framework for inference-time architecture search that composes multiple LLMs and inference-time techniques (e.g., ranking, critique, fusion) to optimize downstream performance under a compute budget. The authors present a thorough experimental evaluation, demonstrating consistent improvements over strong baselines, including GPT-4o, Claude 3.5 Sonnet, and other SOTA systems. While some reviewers expressed reservations regarding the degree of novelty and clarity in the main text, the authors responded comprehensively with expanded ablations, generalization experiments, and additional cost-efficiency analyses. Inference-time scaling techniques abound but application builders are flummoxed as to how to use them and when. Archon is timely and opens up further research along this direction.
Decision: Accept (poster)