Archon: An Architecture Search Framework for Inference-Time Techniques

## Abstract

Abstract Inference-time techniques are emerging as highly effective tools to enhance large language model (LLM) capabilities. However, best practices for developing systems that combine these techniques remain underdeveloped due to our limited understanding of the utility of individual inference-time techniques and the interactions between them. Additionally, efficiently and automatically searching the space of model choices, inference-time techniques, and their compositions is challenging due to the large design space.
To address these challenges, we introduce \archon , a modular framework for selecting, combining, and stacking layers of inference-time techniques to construct optimized LLM systems for target benchmarks.
Rather than relying on a single LLM called once, we leverage a diverse set of LLMs and inference-time techniques, creating LLM systems greater than the sum of their parts . \archon defines an extensible design space, encompassing techniques such as generation ensembling, repeated sampling, ranking, fusion, critiquing, verification, and unit testing. It transforms the problem of building LLM systems into a hyperparameter optimization objective.
Given the available LLMs, inference-time techniques, and compute budget, \archon utilizes hyperparameter search techniques to discover optimized architectures for target benchmark(s).
We evaluate \archon architectures across a range of instruction-following, reasoning, and coding benchmarks, including MT-Bench, Arena-Hard-Auto, AlpacaEval 2.0, MixEval, MixEval Hard, MATH, and CodeContests. \archon architectures outperform frontier models, such as GPT-4o and Claude 3.5 Sonnet, on these benchmarks, achieving an average accuracy increase of 15.1 percentage points by using all available LLMs.
We make our code and datasets available publicly on Github: https://github.com/ScalingIntelligence/Archon .

## 1 Introduction

Inference-time techniques are gaining traction as effective methods for improving model capabilities.
Examples include generation ensembling, ranking, and fusion, where models in the ensemble are queried in parallel, their responses are ranked, and the best ones are fused into a single, higher quality output, respectively [, ].
Other types of inference-time techniques are based on querying a single LLM successively (via repeated sampling) and using a voting strategy or unit tests to select the top generation [, , ].
We divide these existing inference-time techniques into three categories: generative, meaning that new candidate responses are drawn from the models (e.g. generation ensembling and repeated sampling), reductive, meaning that the existing responses are aggregated or filtered to keep the top responses (e.g. fusion and ranking), or comparative, meaning they provide analysis of candidate responses (e.g. critiquing and unit testing),
as shown in Table 1.

Recent work has made progress towards building robust inference-time architectures, which are systems composed of one or more large language models (LLMs) and inference-time techniques.
Examples include Mixture-of-Agents (MoA) [] and LLM-Blender [], as well as single-model systems like LeanStar [] and rStar [].
However, our experiments show that existing architectures, such as MoA, still suffer from lack of generalization and become significantly less effective beyond the task(s) they were developed on (see Section 4.2).
We argue that designing effective and generalizable inference-time architectures requires:

Figure: Figure 1: Overview of \archon Framework: Inference-Time Architecture Search (ITAS) requires the following inputs: target benchmarks, inference call budget, list of available LLMs, and available inference-time techniques (left). The ITAS algorithm uses Bayesian optimization [] (Section A.5) to select and test different configurations of \archon (middle) before returning the optimized \archon architecture (right) for the target benchmarks (Section 3.4).
Refer to caption: extracted/5897895/figures/Budget-Loop-Diagram.png

- •
Understanding the Utilities of Inference-Time Techniques:
Inference-time architectures typically delegate their additional inference budget towards more model sampling calls [, ], which can be effective for math and coding tasks.
Other tasks such as instruction-following and reasoning are shown to benefit from additional techniques, including ranking and fusion [, ].
While all of these methods are valuable, it is essential to identify which inference-time techniques are most effective for different task categories.
- •
Understanding the Interactions Between Inference-Time Techniques:
While previous studies analyzed these techniques individually (e.g. generation sampling in []), we need a more comprehensive understanding of the relationships between different inference-time techniques across different tasks (e.g. is it better to use more models or generate more samples per model?).
- •
Efficiently and Automatically Searching the Large Design Space of Inference-Time Architectures: Given a set of available LLMs and target tasks, there is currently no single prevailing inference-time architecture for maximizing downstream accuracy across all tasks (Table 3). The search space of inference-time architectures is expansive, requiring practitioners to make several key configuration decisions: which LLMs to use, how many times to sample them, how to combine the candidate generations, what inference-time techniques to perform on the candidates, and more. These motivate the need for adaptive and automated architecture search approaches.

In our work, we address each of these challenges.
Firstly, we evaluate the utilities of
a comprehensive set of existing and proposed inference-time techniques across instruction-following, reasoning, and coding tasks.
Using both open-source and closed-source models, we examine a range of techniques such as ensembling, fusion, ranking, critiquing, and verification and introduce new methods such as model-based unit test generation and evaluation (Sections 3.1 and 3.2).

Secondly, we analyze the interactions between inference-time techniques, and explore the benefits of adding new models and new techniques individually.
We find that candidate fusion substantially improves the quality of the final response generation, and when combined with additional techniques like critiquing, verifying, and ranking, can improve generation quality beyond the oracle best candidate from individual (non-fused) responses (Figure 5; Figure 4).
Additionally, we find that candidate verification, unit test generation, and unit test evaluation are most effective for reasoning tasks, whereas critiquing and ranking are effective across instruction-following and reasoning tasks (Section 3.1; Table 2).

Thirdly, drawing upon our analysis of inference-time techniques, we present \archon, a framework for building inference-time architectures.
\archon utilizes automatic inference-time architecture search (ITAS) algorithms to maximize generation quality for a wide range of tasks, including instruction-following, reasoning, and coding.
Our \archon framework and ITAS algorithms draw inspiration from neural architectures and neural architecture search (NAS) [, , , ], respectively.
\archon is constructed of layers of LLMs, in which LLMs within the same layer run in parallel, but each layer runs sequentially.
The layers perform different inference-time techniques, either transforming the number of candidate responses through generation and fusion (analogous to linear transformations) or reducing the number of candidate responses to improve quality (akin to non-linearities) (Section 3.1).
The number of generators, samples per model, fusion layers, fusion models per layer, and more, are all treated as hyperparameters for optimization in our ITAS algorithms (Section 3.4).

Overall, our work makes the following contributions:
(1) We develop \archon, an open-source modular framework for designing LLM systems that combine inference-time techniques (Section 3.1).
We utilize ITAS as the optimizer engine for \archon, which enables automated inference-time architecture search for target benchmarks, leveraging Bayesian optimization [, ] (Section 3.4).
\archon is plug-and-play, allowing users to select from existing inference-time techniques (or add new ones) and specify their desired objective functions to optimize for accuracy, latency, and cost.
(2) We demonstrate increased performance as we scale up the layers of inference-time techniques and combine multiple approaches together, allowing us to discover effective new combinations of inference-time techniques (Sections 3.2, 4.2, 3.3).
We find that sequentially applying critique, ranking, top-k selection, and then fusion is a highly effective composition (Figure 5; Table 2), and we demonstrate the effectiveness of model-based unit test generation and evaluation for improving coding capability (Table 3).
(3) Our best \archon architectures surpass both single-call LLMs
(e.g. GPT-4o and Claude-3.5 Sonnet)
and prior top-performing inference-time architectures
(e.g. Mixture-of-Agents []),
boosting state-of-the-art performance by 15.1 percentage points, on average, across a diverse set of instruction-following, reasoning, and coding benchmarks (Table 3): MT-Bench, Arena-Hard-Auto, Alpaca-2.0 Eval, MixEval, MixEval Hard, MATH, and CodeContests [, , , , , ]. Even when just using open-source LLMs, \archon architectures on average surpass single-call state-of-the-art (SOTA) LLMs by 11.2 percentage points.
We make our code and datasets available publicly on Github:
[https://github.com/ScalingIntelligence/Archon](https://github.com/ScalingIntelligence/Archon).

## 2 Related Work

### 2.1 Scaling Laws of Language Models

Language models [, , , ] have transformed the field of artificial intelligence across a vast number of domains and tasks.
LLMs are pretrained on substantial amounts of textual data before being further aligned with human preferences through
instruction fine-tuning [, ],
direct policy optimization (DPO) [],
Kahneman-Tversky optimization (KTO) [],
reinforcement learning from AI feedback (RLAIF) [],
and other techniques.
As language models continue to gain improved abilities with further scaling of data, parameters, and compute [, ], the cost of developing new LLMs is ever increasing, requiring the curation of trillions of new tokens as well as substantial GPU-hours for pretraining.
Furthermore, as the current state-of-the-art in LLMs are primarily closed-source APIs, such as OpenAI’s GPT-4o [], Google’s Gemini [] and Anthropic’s Claude [], it is difficult to effectively explore and push the frontier of existing LLMs without being able to manipulate the parameters of these closed-source models and employing techniques such as continual pretraining [], instruction fine-tuning [], data mixing [], chain-of-thought [], among others.
A number of recent works show the effectiveness of scaling compute at test time [, , , ].
In particular, Large Language Monkeys [] characterizes inference-time scaling laws, showing a log-linear relationship between coverage–the fraction of problems solved by at least one attempt–and the number of samples drawn from the model across a broad range of reasoning tasks and LLMs.
We build on the inference-time compute scaling approaches and propose \archon, a formal framework for applying inference-time techniques, combining the strengths of multiple pretrained LLMs (Section 3.2), and exploring different inference-time architectures with ITAS (Section 3.4).

### 2.2 Inference-Time Techniques

By utilizing a single LLM or multiple LLMs, inference-time architectures allow us to combine multiple inference-time techniques (e.g. generation ensembling, sampling, ranking, and fusion), achieving superior performance compared to individual models.
Notable works in this domain include the Mixture-of-Agents (MoA) [], LLM Blender [], and RouteLM [, ], which have demonstrated the efficacy of these techniques in improving generation quality, limiting API costs, and reducing query latency.
LM frameworks, such as DSPy [] and TextGrad [], have even emerged for orchestrating LMs and other tool components (e.g. retrievers, search engines, calculators, compilers, etc.), optimizing prompt engineering for integrating these components.
Even with a single LLM, various inference-time techniques can bolster downstream performance by building better reasoning strategies.
These techniques include OpenAI’s o1, Chain of Thought, Branch-Solve-Merge, Rephrase and Respond, Lean-STaR, rStar, REBASE, and more [, , , , , , ].

Despite these advancements, several challenges remain for the development of inference-time architectures.
Firstly, many inference-time architectures today delegate the vast majority of their inference calls towards additional generations [, , ].
For reasoning domains like coding and mathematics, additional repeated inference calls are shown to be effective in improving benchmark performance [].
However, for other tasks such as chat and instruction-following, additional inference-time techniques such as generation fusion and ranking are shown to be useful [, ].
Additionally, for tasks without built-in verification (e.g. unit tests), it can be important to delegate additional compute towards reasoning generation and verification to improve downstream accuracy [].
Within the set of inference-time architectures, we still do not understand the trade-offs between different inference-time techniques in these systems.
Prior studies have only explored limited aspects of inference-time architecture configurations, often focusing on specific benchmarks without generalizing the findings to broader datasets [, ].
For example, both [] and [] explored the impact of LLM calls on downstream performance but they did not examine how other inference-time techniques, such as generation ensembling and fusion, might impact the trends found for LLM sampling.
Beyond analysis of inference-time techniques, it is also crucial to thoroughly and efficiently develop inference-time architectures since the optimal configuration can differ widely based on the benchmark, the models available, and the maximum number of inference calls allowed (Section 4.2).
To address these challenges, we analyzed multiple inference-time techniques (Section 3.1) and developed the \archon framework for automating the development of inference-time architectures with ITAS (Section 3.4).

## 3 Inference-Time Techniques for \archon

To better understand what inference-time techniques could be most effective for \archon,
we test an array of different techniques, incorporating existing approaches for generating, ranking, and fusing candidates [, ] as well as constructing new approaches for critiquing, verifying, and unit testing candidates, inspired by a host of recent work [, ].
Below, we elaborate on the structure, inputs, and outputs of each of the inference-time techniques, which we also include in Table 1.
Then, we discuss how to combine the different techniques into an inference-time architecture (Section 3.2) and the relationships between the different inference-time techniques (Section 3.3) before finally exploring automatic approaches for constructing inference-time architectures (Section 3.4).

### 3.1 LLM Components of \archon

In this section, we discuss the LLM components of \archon, which are LLMs that perform a specific inference-time technique. The components are summarized in Table 1.

**Table 1: Overview of \archon’s Inference-time Techniques: Definitions, Inputs, Outputs, Costs, and Application Domains.**
| Inference-time<br>Technique | Definition | Input | Output | Inference<br>Cost | Domains |
| --- | --- | --- | --- | --- | --- |
| Generator | Generates a candidate response<br>from an instruction prompt | Instruction Prompt | Candidate Response(s) | 1 call<br>per cand. | All<br>Domains |
| Fuser | Merges multiple candidate<br>responses into a single response | Instruction Prompt +<br>Candidate Response(s) | Fused Candidate<br>Response(s) | 1 call<br>per cand. | All<br>Domains |
| Critic | Generates strengths/weaknesses<br>for each candidate response | Instruction Prompt +<br>Candidate Response(s) | Candidate Response(s)<br>Strengths/Weaknesses | 1 call | All<br>Domains |
| Ranker | Returns top-K<br>candidate responses | Instruction Prompt +<br>Candidate Response(s) | Ranked Candidate<br>Response(s) | 1 call | All<br>Domains |
| Verifier | Returns the candidate responses<br>with verified reasoning | Instruction Prompt +<br>Candidate Response(s) | Verified Candidate<br>Response(s) | 2 calls<br>per cand. | Reasoning<br>Tasks |
| Unit Test<br>Generator | Generates unit tests to evaluate<br>the candidate responses | Instruction Prompt | Instruction Prompt<br>+ Unit Tests | 1 call | Reasoning<br>Tasks |
| Unit Test<br>Evaluator | Uses generated unit tests to<br>evaluate candidate response | Instruction Prompt +<br>Unit Tests +<br>Candidate Response(s) | Scored Candidate<br>Response(s) | 1 call<br>per cand. | Reasoning<br>Tasks |

Generator: The Generator module of \archon is a LLM that creates candidate responses
As input, the generator takes in the instruction prompt and outputs candidate responses. Generators can be called in parallel to perform generation ensembling [, ], or sampled multiple times [, ].
When calling the Generators in parallel, you can sample one or more LLMs one or more times.
The exact number of models, samples, and temperature for generation can be varied based on model configuration.
We provide the exact prompt used in the \archon generator in Table 5.

Fuser: The Fuser module of \archon is a LLM that combines multiple candidate responses to create one or more higher-quality responses. As input, the fuser takes in the instruction prompt and the set of proposed responses.
As output, the fuser generates a fused response that combines the proposed responses into a higher-quality generation for addressing the instruction prompt.
We provide the exact prompt used in the \archon fuser in Table 6.

Ranker: The Ranker module of \archon is a language model that ranks the current list of candidate generations based on their quality and the instruction prompt.
As input, the ranker takes in the instruction prompt and the set of proposed responses.
As output, the ranker produces a ranked list of the proposed responses.
We provide the exact prompt used in the \archon ranker in Table 7.

Critic: The Critic module of \archon is a LLM that produces a list of strengths and weaknesses for each candidate response in a provided set.
As input, the critic takes in the instruction prompt and the set of proposed responses.
As output, the critic produces a list of strengths and weaknesses for each respective candidate response.
We use the strengths and weaknesses to improve the quality of the final response (Section 3.2; Figure 5).
We provide the exact prompt used in the \archon critic in Table 8.

Verifier: The Verifier module of \archon is a LLM that verifies whether a provided candidate response has appropriate reasoning for a given instruction prompt.
It proceeds in two stages: Stage #1 takes in the instruction prompt and a candidate response as input and outputs reasoning for why the candidate response is correct; Stage #2 takes in the instruction prompt, candidate response, and produced reasoning before outputting reasoning and a verdict (i.e. binary [Correct] or [Incorrect]) for whether or not the candidate response is correct according to the provided instruction prompt and reasoning.
We provide the exact prompt used in the \archon verifier in Table 9.

Unit Test Generator: The Unit Test Generator module of \archon is a LLM that generates a set of unit tests for a given instruction prompt.
As input, the unit test generator solely takes in an instruction prompt.
As output, the unit test generator produces a list of unit tests that are consequential for the accuracy and relevance of a candidate response.
These generated unit tests are verified by the Unit Test Evaluator, allowing us to rank different candidate responses.
Each unit test is formatted as a concise declarative statement that can either be passed or failed.
We make the number of unit tests generated a configurable choice for the unit test generator but we find 5-10 generated unit tests to be most effective with our set of LM prompts (Section 4.2; Figure 7).
We include examples of unit tests for an instruction-following query and a reasoning query in Table 11.
We provide the exact prompt used in the \archon unit test generator in Table 10.

Unit Test Evaluator: The Unit Test Evaluator module of \archon is a language model that evaluates each candidate generation against a generated set of unit tests.
As input, the unit test evaluator takes in the instruction prompt, candidate response(s), and set of unit tests.
As output, the unit test evaluator outputs the candidate response(s), ranked in descending order by how many unit tests they pass.
We use model-based unit test evaluation by prompting the LLM to provide reasoning and verdicts for each unit test across each of the candidate responses.
By aggregating the unit test verdicts for each candidate response, the unit test evaluator ranks the candidate responses.
For reasoning tasks, particularly coding tasks, it can be useful to compare different candidate responses by the number of unit tests they pass to gauge for quality.
Additionally, by generating many candidate responses and evaluating each one against the unit tests, we can better leverage increased inference time compute budgets while improving the quality of the final returned response (Figure 7).
We provide the exact prompt used in the \archon unit test evaluator in Table 12.

### 3.2 Combining the LLM Components

Overview: Inspired by the structure of neural networks [], \archon is constructed of layers of LLM components (Figure 1; Section 3.1).
Each layer is composed of sets of these LLM components that are called in parallel, performing a text-to-text operation to the instruction prompt and the subsequently generated candidate responses.
Furthermore, like a neural network, some layers perform transformations of the provided list of strings (e.g. the Generator and Fuser components), converting a list of strings into a different list of strings (the numbers of candidates can vary from the original number of candidates).
Other components introduce non-linearities into the \archon structure, performing filtering of the list of strings (e.g. Ranker and Verifier).
Ultimately, the inputs and outputs for each layer is always a list of strings, whether that is the instruction prompt (e.g. a list with a single string) or a list of candidate responses (e.g. a list of many strings).
If a list of strings are outputted at the last layer of the \archon structure, the first string in the list is returned.

Unlike a classical neural network, no weights are learned between the LLM components and the layers; in turn the \archon architecture can be deployed off-the-shelf without any tuning.
This distinction makes architecture search much cheaper and more efficient since a new configuration is tested without an inner optimization of the architecture’s weights, which we discuss in Section 3.4.
Additionally, a single state is transformed sequentially from the input layer to the final output; this single state is the initial instruction prompt and the current candidate responses.
In Figure 2, we provide an example \archon architecture composed of six layers: an ensemble layer of generators, an intermediate critic layer, an intermediate ranker layer, an intermediate layer of fusers, an intermediate verifier layer, and a final fuser layer..

Figure: Figure 2: Example \archon Architecture: The architecture starts with ten generator models, followed by a critic model, a ranker model, one layer of six fuser models, a verifier model, and finishing with a fuser model.
Refer to caption: extracted/5897895/figures/example_archon_architecture_fig.png

Rules for Construction: While \archon is a modular framework, the LLM components in Section 3.1 can only be placed in specific orders.

- 1.
Only one type of module can be present in any given layer.
- 2.
Generator components must and can only be placed in the first layer of \archon; you can put multiple Generators or a single Generator in the layer.
- 3.
The Critic component must come before a Ranker or a Fuser, otherwise the generated strengths and weaknesses cannot be incorporated into generation ranking or fusion, respectively.
- 4.
Ranker, Critic, Verifier, and Unit Test Generator/Evaluator layers can go anywhere in the \archon structure (besides the first layer); for each of these components (as well as the Unit Test Generator), it must be the one and only module in its layer.
- 5.
Fuser components can also be placed anywhere in the \archon structure (besides the first layer); you can put multiple Fusers or a single Fuser in the layer.
- 6.
Unit Test Generators and Evaluators are placed in layers next to each other: generator first, then evaluator.

We provide an overview of the available placements and configurations for each LLM module in Table 4.

### 3.3 Utilities and Interactions of LLM Components

Figure: Figure 3: Performance Gains from Applying Inference Time Techniques on a Single Model: We repeatedly sample more responses for each individual query. For each sample count, we choose the best response in 5 different ways: (1) using an oracle (to get the upper bound for performance of best sample), (2) randomly, (3) using a ranker model, (4) by fusion, in which a model synthesizes a response based on all the samples, and (5) by ranking the top-5 best answers and then fusing them. For both MT Bench and Arena-Hard-Auto, we find that fusion is an effective technique. In particular, ranking the candidates first, and then selecting the top-5 and fusing them scores the highest. The best open-source model for these tasks across all the 70B+ models we are considering is WizardLM-2-8x22B [] (see Table 14 for details). For both ranking and fusion, we use Qwen2 72B Instruct [].
Refer to caption: extracted/5897895/figures/sampling_graph.png

In this subsection, we present our analysis of the effectiveness of each LLM component (i.e. the Utility) and the relationships between each component (i.e. the Component Interactions) by evaluating on instruction-following tasks (MT Bench, AlpacaEval 2.0, Arena-Hard-Auto), reasoning tasks (MixEval, MixEval-Hard, MATH) and coding tasks (CodeContests) (Section 4.1).
For our \archon models, we utilize a host of 70B+ open-source models (Section 4.1; Table 13).

#### 3.3.1 Generator

Utility:
For our Generator module, we find additional model sampling to significantly boost performance (Figure 3), particularly for coding tasks (Figure 7).
In settings with a limited inference call budget, additional model samples lead to the largest marginal benefit.
We see a similar pattern for model ensembling, where sampling from additional models leads to continual performance increases (assuming the models are ordered from best to worst for the given task) (Figure 4).

#### 3.3.2 Fuser

Utility: For every benchmark explored, we found that the Fuser module substantially improved performance (Figure 3; Figure 4; Figure 5).
For the single-generation 10-model ensemble of 70B+ models, the Fuser module improved downstream accuracy by 5.2 points, on average, compared to the single-generation best model (Figure 4).
When combined with the Ranker module for ranking the top-5 candidate responses, the Fuser improved downstream accuracy by 7.3 points and 3.6 points, on average, compared to the single-sample best model and the oracle best candidate response, respectively (Figure 4).
Overall, we found that Fuser efficacy increased as more candidate responses were provided, demonstrating that additional candidate generations can continue to bolster inference-time architecture performance when combined with a Fuser.

In previous work like Mixture-of-Agents (MoA) [], multiple layers of Fusers was found to boost performance on some instruction-following tasks (i.e. MT Bench and Alpaca Eval 2.0).
Across all the benchmarks explored, we observed similar benefits in the \archon framework when adding multiple layers of Fusers (Figure 5).
However, based on our results in Figure 8, the number of Fuser layers needed to improve performance varied by task, with some tasks receiving limited benefits from added layers (1-2 point increase in accuracy for MixEval) while others experienced significant benefits with 3-4 fusion layers and more (2 to 5 point increase in win rate for MT Bench and Alpaca Eval 2.0).
We attribute this distinction to the difference in task requirements, with chat and instruction following tasks benefiting more from multiple iterations of revisions through the multiple Fuser layers, leading to greater diversity in the final generation (Table 15).

Figure: Figure 4: Performance Gains from Applying Inference-Time Techniques on an Ensemble of Models: We incrementally add more models to the ensemble, which consists of open-source 70B+ models. The models are added to the pool based on their performance for each task, from best to worse (see Table 14 for details). For each ensemble size, we choose the best response in 5 different modes: (1) using an oracle (to get the upper bound for performance of best individual response in the ensemble), (2) randomly, (3) using a ranker model, (4) by fusion, in which one model synthesizes a response based on all the responses of the ensemble models, and (5) ranking the top-5 best responses and then fusing them. For MT Bench and Arena-Hard-Auto, we find consistent performance improvements as we add more models to the ensemble. We find that fusion is beneficial across various ensemble sizes and in particular a fused candidate based on the top-5 ranked responses scores highest. The ensemble approach scores higher than applying the same techniques on repeated samples from a single best-performing model (see Figure 3). For both ranking and fusion, we use Qwen2 72B Instruct [].
Refer to caption: extracted/5897895/figures/Ensembling_Graph.png

Component Interactions: To better understand how the Fuser module works with the other LLM components, we took the single-sample 10-model ensemble of Generators with a Fuser and tried adding each of these components individually: a Critic, a Ranker, a Verifier, and a Unit Test Generator/Evaluator.
Across all of the benchmarks, the added candidate response analyses from the Critic improved the Fuser’s ability to effectively merge the different candidate responses, increasing performance by an average of 3.1 percentage points (Figure 5).
With the added Ranker, the \archon architecture improved the combined Ensemble + Critic + Fuser performance across all the benchmarks by 4.8 percentage points, on average (Figure 5).
The Ranker proved most effective for style-oriented tasks (e.g. MT Bench and AlpacaEval 2.0) since the examples mostly focus on improving the instruction-guidance towards the provided prompt.
With the added Verifier module (Figure 5), the performance of the Ensemble + Critic + Fuser configuration improved marginally for the instruction-following tasks (1.2 percentage points, on average, for MT Bench, AlpacaEval 2.0, and Arena-Hard-Auto).
However, this configuration improved performance more on reasoning tasks (3.2 percentage points for MixEval and MixEval-Hard, on average), assisting generation by filtering out irrelevant or flawed answers before the final fusion step (Figure 5).
The added Unit Test Generator and Evaluator was less effective for the instruction-following and reasoning tasks, only providing a 1.5 percentage points increase, on average, when added to the Ensemble + Critic + Fuser configuration (Table 2).
However, for coding tasks, we found unit test generation and evaluation significantly improved performance, leading to a 10.7 percentage point increase (56% performance increase comparatively) as we scale model sampling (Figure 7).

Figure: Figure 5: Performance Gains from Scaling "Layers" of Inference-Time Techniques: To better understand the impact of scaling inference-time techniques, we evaluate different \archon architectures on AlpacaEval 2.0 [], Arena-Hard-Auto [], MT-Bench [], MixEval [], MixEval Hard, MATH [], and CodeContests []. For the ensemble, we use the top-8 70B+ open-source models (Table 14). We generally observe performance improvements as we scale the critic and fusion layers. Compared to sampling the best open-source model once, our inference-time architecture with an 8-model ensemble, 3 layers of critic and fusion (8 models in each layer), and a final fusion performs 17.3% higher, on average. For MixEval and CodeContests, we find that alternative inference-time architectures are more effective than generator ensembles and fusion layers. We break-down our results for MixEval and MixEval-Hard by subdataset in Section 4.2 (Table 26; Table 27). For CodeContests, we show the effectiveness of increased generator sampling combined with model-based unit test generation/evaluation in Figure 7.
Refer to caption: extracted/5897895/figures/Comparing_Compound_LMs_By_Benchmark_Expanded.png

#### 3.3.3 Critic

Utility: The Critic module proved effective for every task we explored in Figure 5 and Table 2.
With our 10-model 70B+ Generator ensemble and Fuser configuration of \archon, the added Critic improved performance on average by 3.1 percentage points across the benchmarks explored.

Component Interactions: While useful for most \archon architectures, the added strengths and weaknesses from the Critic module are particularly useful when combined with the Fuser module, helping guide generation fusion for a single layer and even useful when placed between multiple fusion layers (on average 3.2 percentage point boost across benchmarks in Figure 5).
The Critic module was also effective with the Ranker module, providing additional information for comparing candidate responses (Figure 3) and leading to a 5.9 percentage point increase, on average (Table 2).

#### 3.3.4 Ranker

Utility: From our results in Table 2, Figure 3, and Figure 4, we found the Ranker to be most effective for instruction-following tasks, where pair-wise comparisons of answers focus on style and adherence to the prompt.
To examine the candidate selection improvement provided by candidate ranking, we compare three approaches to the Ranker: (1) random selection of candidate generation, (2) oracle selection of candidate generation, and (3) the top-ranked candidate selected by our Ranker.
For MT Bench and Arena-Hard-Auto, we find that the ranker improves generation output quality by 3.8% compared to random candidate selection and performs within 2.7% of oracle selection (Figure 3).

Component Interactions: Based on our benchmark results in Table 2, the Ranker pairs well with the Critic module; the provided strengths and weaknesses helps guide ranking, particularly for instruction-following tasks, improving performance by 5.9 percentage points, on average.
Furthermore, the Ranker was also effective when paired with the Fuser; the filtered list of candidate responses helped improve the final condensed response produced by the Fuser by 3.8 percentage points, on average (Figure 4).
When paired with the Verifier and Unit Test Generator, the Ranker had neutral effects; performances changed marginally, either positively or negatively by 1-2 percentage points (Table 2).

Overall, our findings demonstrate the value of added Rankers for instruction-following and reasoning tasks when paired with Fusers.
We find that when Rankers are used alone with an ensemble of Generators, their performance lags behind the 10-sample best single model configuration by 3.0 percentage points, on average (Table 2).
Additionally, our findings show the importance of building better rankers for more complex reasoning tasks, such as math and coding, which is a challenge also raised by [].

**Table 2: Impact of Different Compositions of \archon’s Inference-Time Techniques: For the ensemble configuration of \archon, we see added benefits from ranker, critic, verifier, fuser, and unit test generator/evaluator. In particular, a composition of ensemble generator, critic, ranker, and fuser components improves performance across the explored benchmarks. For CodeContests, we find that there is a single model (Llama 3.1 405B Instruct) that performs considerably better than the rest of the LLMs studied, making it more effective to build architectures that use multiple inference-time techniques on a single model and leverage additional model sampling (Figure 7; Table 14). For our ensemble, we use the best 8 open-source 70B+ models for the task. For our fuser, critic, ranker, and verifier components, we use the best fuser model found for the task (Table 14). ^∗For MATH and CodeContests, we use sampled subsets of their test sets (Section 4.1; Table 25).**
|  |  |  | Datasets |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | MT<br>Bench | AlpacaEval<br>2.0 | Arena<br>Hard Auto | MixEval<br>Hard | MixEval | MATH^∗ | Code<br>Contests^∗ |  |
|  | Judge Model |  | GPT-4<br>0314 | GPT-4<br>Turbo | GPT-4<br>Turbo | GPT-4<br>Turbo | N/A | N/A | N/A | N/A |
|  | Reference Model |  | Claude-3.5<br>Sonnet | GPT-4<br>Turbo | GPT-4<br>Turbo | Claude-3.5<br>Sonnet | N/A | N/A | N/A | N/A |
|  | Model / LLM System | # of<br>Infer.<br>Calls | W.R. | L.C.<br>W.R. | Raw<br>W.R. | W.R. | Acc. | Acc. | Acc. | Acc. |
| Control | Best Open-Source 70B+ Model, Sampled Once | 1 | 55.0% | 44.7% | 37.1% | 45.6% | 58.7% | 86.5% | 73.5% | 27.1% |
| Ensemble + Fuser | 9 | 58.4% | 57.5% | 51.3% | 54.3% | 60.5% | 87.3% | 75.5% | 22.0% |  |
| Ensemble + Critic + Fuser | 10 | 60.9% | 58.7% | 65.8% | 58.8% | 62.4% | 87.4% | 77.0% | 24.5% |  |
| Ablations | Ensemble + Ranker | 9 | 52.5% | 54.7% | 47.6% | 50.5% | 58.2% | 86.8% | 71.5% | 23.5% |
| Ensemble + Verifier | 24 | 53.2% | 56.2% | 50.2% | 52.4% | 56.5% | 85.6% | 76.0% | 24.9% |  |
| Ensemble + Unit Test Gen./Eval. | 17 | 51.5% | 54.4% | 49.4% | 46.1% | 55.2% | 86.0% | 75.0% | 25.1% |  |
| Ensemble + Ranker + Fuser | 10 | 62.5% | 60.3% | 63.6% | 57.2% | 60.1% | 87.6% | 76.0% | 23.6% |  |
| Ensemble + Verifier + Fuser | 25 | 60.5% | 59.4% | 58.7% | 59.2% | 65.1% | 87.5% | 78.0% | 24.5% |  |
| Ensemble + Unit Test Gen./Eval. + Fuser | 17 | 61.4% | 58.5% | 55.1% | 56.4% | 62.8% | 86.9% | 77.0% | 26.3% |  |
| Ensemble + Critic + Verifier + Fuser | 25 | 61.3% | 60.0% | 61.0% | 59.5% | 65.5% | 87.8% | 78.0% | 24.8% |  |
| Ensemble + Critic + Ranker + Fuser | 11 | 64.7% | 62.6% | 72.4% | 60.9% | 67.0% | 88.3% | 79.5% | 24.1% |  |

#### 3.3.5 Verifier

Utility: The Verifier was most effective for the reasoning benchmarks explored in Table 2.
When just using a 70B+ Generator ensemble with Verifier module after generation, the \archon configuration lagged behind the \archon ensemble and fuser configuration by 1.5 percentage points, on average, across all benchmarks explored.
This suggests that the Verifier is most effective when combined with other inference-time techniques.

Component Interactions:
As noted in Section 3.3.2, the Verifier augmented the performance of the Critic and Fuser on reasoning tasks (e.g. Arena-Hard-Auto, MixEval, MixEval-Hard), boosting performance by 3.7 percentage points, on average, when combined together with these modules.
Overall, the Verifier is most powerful when augmenting additional components for tasks requiring verification of intermediate steps and the final response (Table 2).
Therefore, the Verifier was less helpful for instruction-following tasks (e.g. MT Bench and AlpacaEval) but more effective for reasoning tasks (e.g. Arena-Hard-Auto and MixEval).

#### 3.3.6 Unit Test Generator and Evaluator

Utility:
The Unit Test Generator and Evaluator were most effective on reasoning and coding tasks, improving performance on benchmarks that required more verification steps, such as Arena-Hard-Auto, MixEval, MixEval-Hard, MATH, and CodeContests (Table 2).
For the reasoning tasks, we found the unit test generator and evaluator to be most effective when combined with other components.
When the 70B+ ensemble of Generators was only combined with unit tests, it was less effective for reasoning tasks like Arena-Hard-Auto and MixEval, lagging behind the ensemble and fuser configuration by 3.1 percentage points.
This inspired us to look into other inference-time techniques combinations for unit test generation, such as increased sampling and fusion.
When we increased generation sampling and added unit test generation/evaluation for CodeContests, we see a 56% boost in Pass@1 performance (Figure 7), increasing from 17.9 to 29.3 Pass@1.

Component Interactions:
When combined with the Fuser module, the Unit Test Generator and Evaluator improved performance by 2.1 percentage points across the benchmarks explored (Table 2).
The combined ensemble, Unit Test Generator/Evaluator, and Fuser \archon configuration was most effective on the reasoning benchmarks, leading to a 2.5 percentage point boost, on average.
For coding, the unit test generator and evaluator was most effective when combined with the best performing Generator (using large sample counts) and a final Fuser (subsection 4.2).

### 3.4 Inference-Time Architecture Search (ITAS)

In this section, we explore different approaches for finding the best inference-time architecture (for a given task) through inference-time architecture search (ITAS).
Due to compute resources, we pre-filtered certain ways of combining LLM components to reduce the search space while still building effective inference-time architectures.
While it is possible to expand the search space of potential \archon architectures (e.g. different temperatures for generative LLM components, alternative prompts for each LLM component, multiple layers of Generator modules, additional LLM components for \archon, etc.),
we use our analysis from Section 3.2 to selectively limit our search space to configurations that fit our rules for \archon: starts with a layer of Generator modules, followed by layers performing fusing, ranking, critiquing, verifying, and unit testing.

Search Hyperparameters: With the studied LLM modules and their relationships within the \archon architecture, we selected five main axes for the hyperparameters in our search:

- 1.
Top-$K$ Generators for Ensemble: The top-$K$ models to be used for the initial Generator ensemble, ranges from 1 to 10.
The top-$K$ models are the best-$K$ LLMs for the given task, based on their individual performances (Table 14).
- 2.
Top-$K$ Generator Samples: The number of samples gathered from each Generator in the ensemble (it is the same for all the models), ranges from 1 to 5.
For MATH and Code-Contests, we also explore high sample settings over the following set of samples: [1, 10, 100, 500, 1000].
- 3.
Number of Fusion Layers: The number of layers of Fusers, ranges from 1 to 4.
The last fusion layer will always have a single Fuser.
- 4.
Top-$K$ Fusers for Fusion Layers: The top-K models used for each fusion layer, ranges from 2 to 10 and increases by 2 each time.

By combining all the hyperparameters, we create a search space of 6,250 configurations by multiplying each of the configuration option counts together ($10*5*5^{(4-1)}=6250$).
However, we remove configurations that are not viable: configurations in which the number of initial generations exceeds the context window of the fusers (i.e. 24 candidate generations) and configurations with only one fuser layer but multiple fusers declared.
This reduces our search space to 3192 configurations.
For these configurations, we add critic and ranker layers before each fuser layer since they’ve been shown to have added benefits across the benchmarks explored (Figure 4; Figure 5).
The ranker selects the top-5 candidate generations to send to the next layer.
Additionally, for our coding tasks (i.e. CodeContests), we use unit test generators and evaluators after our initial generation layer, with a default setting of 10 unit tests generated.
On our instruction-following and reasoning tasks (i.e. MT-Bench, AlpacaEval 2.0, Arena-Hard-Auto, MixEval, MixEval-Hard, and MATH), we also ablate adding a verifier before the final fuser layer (Table 2).
Ultimately, we could increase the search space substantially more along various other axes, such as additional combinations of verifier, unit test generation, and fuser layers, but given our compute resource limitations, we did not scale further.

Search Techniques: Within the hyperparameter space, we explored four search algorithms for automating the development of inference-time architectures:

- 1.
Random Search: Randomly selects a combination of hyperparameters for our \archon architecture.
- 2.
Greedy Search: Starting with a base \archon configuration, marginally changes each hyperparameter and test if it improves performance or not. If it does, incorporate the change. If not, move on to the next hyperparameter.
- 3.
Bayesian Optimization: Efficiently selects the most promising hyperparameter configurations for \archon by building a probabilistic surrogate model and leveraging an acquisition function for hyperparameter selection [, ] (Section A.5).

To get our model ranking for the benchmark, we calculate the model ranking by testing each model individually on the entire benchmark ($K$ = 1) in the first stage of the search.
To get our fusion model ranking for the benchmark, we use the same approach, testing each model’s fusion performance with an ensemble of 10 randomly selected models from the available set.
From our experiments, we found that the best generator and fusion models could vary widely dataset to dataset, making it beneficial to perform these rankings for new datasets (Table 14).
For search, we use a 20% sample of each dataset for guiding architecture search to improve the evaluation speed while getting meaningful development signal.

Figure: Figure 6: Impact of Different Optimization Algorithms on Inference-Time Architecture Search (ITAS): On the benchmarks MT Bench and Arena-Hard-Auto, we compare four approaches for finding the optimal inference-time architecture: random search, greedy search, and Bayes optimization. We find Bayes optimization to be the most computationally efficient, finding the optimal architecture in 80.4% less iterations compared to greedy search and 87.1% less iterations compared to random search.
Refer to caption: extracted/5897895/figures/Search_Algorithms.png

Comparing Search Algorithms: In Figure 6, we compare the effectiveness of each search algorithm on our explored benchmarks.
While random search guarantees the optimal \archon configuration, we found Bayesian optimization to be most effective in terms of tradeoff between finding the optimal configurations and minimizing the number of configurations tested.
For 95.2% percent of the search iterations tested in Figure 6, we found that Bayesian optimization had the optimal configuration amongst the four explored search algorithms.
We use 80 initial samples for our Bayes optimization architecture search (Section A.5).
Bayesian optimization also found the best architecture configuration in 80.4% less evaluations than greedy search and 87.1% less evaluations than random search.

Bayesian Optimization Analysis:
In Table 22, we explore how the number of initial testing points, the number of exploration iterations, and the \archon inference call budget impacts the effectiveness of Bayesian optimization.
Additional initial testing points continue improving search efficacy up until 80-90 samples, where testing would be better delegated towards configuration search.
For lower inference call budgets with \archon (e.g. <20 inference calls), Bayesian optimization proved less effective, performing more similarly to greedy search or random search given the limited search space (Table 23.
Therefore, Bayesian optimization is more effective for more open-ended ITAS with larger inference call budgets (e.g. >20 inference calls) whereas traditional component engineering might be better for more limited inference call budgets.

## 4 Experiments

In our experimental evaluations, we focus on four questions:
(1) how do \archon architectures compare to existing SOTA closed-source LLMs and other inference-time architectures?
(2) how does \archon performance compare across tasks?
(3) how does \archon performance compare when optimized for a set of tasks vs. an individual task?
(4) what are the current limitations of \archon and plans for future work?

### 4.1 Benchmarks and Models

In this section, we discuss the benchmarks and models used in our LLM component analysis and development of \archon.

Benchmarks: We evaluate our models with several benchmarks for instruction-following, reasoning, and coding: MT-Bench [], AlpacaEval 2.0 [], Arena Hard Auto [], MixEval [], MixEval-Hard, MATH [], and CodeContests [].
We provide an overview of each dataset in Table 25, where we compare their query counts, scoring type, evaluation metrics, baseline models, and judge models.
Since we perform ITAS on a randomly sampled 20% subset of each benchmark, we evaluate on the remaining held-out 80% subset of the benchmark (Table 3) (for \archon performances on the entire benchmarks, please see Table 24).
For MixEval and MixEval Hard, we use the 2024-06-01 dataset release.
For MT Bench and Arena-Hard-Auto, we also include a configuration with Claude-3.5-Sonnet as the baseline model (in addition to the original setting with GPT-4-0314) to have a stronger model for comparing against \archon architecture performances (Table 3) and mitigate the GPT-4-Turbo judge bias towards GPT responses.
Additionally, we chose not to use the single-scoring configuration for MT Bench due to the inconsistencies in LLM judge scoring on 1-10 scales [, ].
For MATH, we evaluate a random sample of 200 problems from the dataset’s test set.
For CodeContests, we evaluate on the 140 test set questions that do not include image tags in the problem description.

Models: For \archon, we test across three model categories: 8B or less parameter models, 70B or more parameter models, and closed-source model APIs.
For our 8B and 70B+ models, we selected the top-10 performing chat models for each parameter range on the Chatbot Arena Leaderboard [] as of July 2024.
For our closed-source model APIs, we include GPT-4o, GPT-4-Turbo, Claude Opus 3.0, Claude Haiku 3.0, and Claude Sonnet 3.5.
We list and compare all of the models tested in the \archon framework in Table 13 and Table 14.

### 4.2 \archon vs. Closed-Source LLMs and Other Inference-Time Architectures

**Table 3: \archon’s Strong Performance with ITAS Optimization on Open-Source, Closed-Source, and All-Source Models: We find that our targeted and generalized \archon inference-time architectures (Section 4.4) consistently outperform single-call state-of-the-art LLMs, both open-source and closed-source baselines, across the explored benchmarks (Section 4.1; Table 25). Since we perform ITAS optimization on a randomly sampled 20% subset of each benchmark, we evaluate on the remaining held-out 80% subset of the benchmark in Table 3 (for \archon performances on the entire benchmarks, please see Table 24). The delta between \archon’s performance on the entire benchmark vs. 80% held-out subset is relatively small: only 0.44 percentage points, on average, across these datasets with an S.D. of 0.20 percentage points). For our task-specific \archon architectures, we also provide the average inference calls across the given benchmarks. ^∗For MATH, we use a randomly sampled subset of size 200 for evaluation (Section 4.1; Table 25).**
|  |  |  | Datasets |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | MT<br>Bench | Alpaca<br>Eval 2.0 | Arena<br>Hard Auto | MixEval<br>Hard | MixEval | MATH^∗ |  |  |
|  | Judge Model |  | GPT-4<br>0314 | GPT-4<br>Turbo | GPT-4<br>Turbo | GPT-4<br>Turbo | N/A | N/A | N/A |  |
|  | Reference Model |  | Claude 3.5<br>Sonnet | GPT-4<br>Turbo | Claude 3.5<br>Sonnet | GPT-4<br>Turbo | N/A | N/A | N/A |  |
|  | Model / LLM System | Infer.<br>Calls | W.R. | L.C.<br>W.R. | Raw<br>W.R. | W.R | W.R | Acc. | Acc. | Pass<br>@1 |
|  | GPT-4o - 2024-05-13 | 1 | 44.2% | 57.8% | 52.1% | 47.5% | 80.6% | 63.4% | 87.5% | 73.2% |
|  | Claude 3.5 Sonnet | 1 | N/A | 52.7% | 41.2% | N/A | 81.4% | 68.7% | 89.1% | 73.1% |
|  | Llama 3.1 405B Instruct | 1 | 44.1% | 40.7% | 38.4% | 27.8% | 64.5% | 66.0% | 88.2% | 75.2% |
|  | MoA | 19 | 51.6% | 65.4% | 60.5% | 51.7% | 84.5% | 62.3% | 86.9% | 73.9% |
|  | MoA Lite | 7 | 45.6% | 59.6% | 57.7% | 39.8% | 88.3% | 60.9% | 86.4% | 71.8% |
| Open<br>Source | General-purpose<br>\archon Architecture | 35 | 67.2% | 63.3% | 69.0% | 65.5% | 85.6% | 65.3% | 86.2% | 76.6% |
| Task-specific<br>\archon Architectures | 44 | 71.1% | 67.1% | 71.3% | 68.5% | 89.6% | 67.5% | 88.8% | 81.9% |  |
| Closed<br>Source | General-purpose<br>\archon Architecture | 32 | 72.7% | 63.9% | 69.8% | 69.8% | 86.2% | 67.5% | 87.2% | 77.9% |
| Task-specific<br>\archon Architectures | 40 | 77.0% | 68.9% | 73.0% | 73.9% | 90.5% | 72.6 | 89.5% | 81.6% |  |
| All<br>Source | General-purpose<br>\archon Architecture | 35 | 76.2% | 66.4% | 71.0% | 71.9% | 89.8% | 69.8% | 87.3% | 79.3% |
| Task-specific<br>\archon Architectures | 39 | 79.5% | 69.0% | 74.1% | 75.6% | 92.5% | 72.7% | 89.7% | 82.1% |  |

We start by comparing \archon architectures to existing SOTA closed-source LLMs and inference-time architectures across a set of instruction-following, reasoning, and coding tasks with either pairwise ranking or accuracy metrics, as described in Section 4.1.
Since we perform ITAS on a randomly sampled 20% subset of each benchmark, we evaluate on the remaining held-out 80% subset of the benchmark (Table 3) (for \archon performances on the entire benchmarks, please see Table 24).
Based on our results in Table 3, we find that \archon architectures consistently match or surpass existing SOTA approaches across all the benchmarks explored.
On the evaluation suite, our \archon architectures with open-source models experience a 11.2 point increase, on average, above SOTA open-source approaches;
for its worst performance, our open-source \archon architectures are only 3.6% above SOTA open-source approaches on AlpacaEval 2.0.
For our \archon architectures with closed-source models, we set SOTA performance across MT Bench, Arena-Hard-Auto, MixEval, and MixEval-Hard, leading to a 15.8 percentage point increase, on average, compared to closed-source approaches.
Lastly, for approaches that use all-source models available, both open and closed-source, \archon achieves an increase of 10.9 points, on average, over existing SOTA single-call LLMs.

### 4.3 \archon by Task

We analyze \archon performance by task style: instruction-following tasks that use pairwise ranking for scoring, reasoning tasks that use accuracy-based metrics for scoring, and coding tasks that use Pass@1.
On instruction-following tasks like MT Bench, AlpacaEval 2.0, and Arena-Hard-Auto, open-source \archon architectures outperform current open-source baselines by 10.0 percentage points, on average, while closed-source \archon outperforms current closed-source baselines by 20.1 percentage points (Table 3).
On reasoning tasks like MixEval, MixEval-Hard, and MATH, open-source \archon architectures outperform existing open-source baselines by 2.9 percentage points while closed-source \archon architectures outperform current closed-baselines by 4.2 percentage points (Table 3).
On coding tasks (i.e. CodeContests), open-source \archon architectures match existing open-source baselines (0.2 percentage points difference) and all-source \archon architectures outperform all-source baselines by 2.5 percentage points (Figure 7).
All-source architectures of \archon outperforms existing all-source baselines by 16.1 and 3.8 percentage points, on average, for instruction-following tasks and for reasoning tasks, respectively (Table 3).

Instruction-Following and Reasoning:
With \archon, multiple models used for Generators and the depth of fusion layers lead to performance boosts on instruction-following tasks, increasing the richness of responses and allowing multiple iterations for step-by-step instruction-following (Table 15).
For reasoning, while the performance boost from \archon is smaller when we consider the aggregate scores for MixEval and MixEval-Hard, we do see meaningful increases in performance when we create inference-time architectures for each individual task under MixEval and MixEval-Hard (Table 26; Table 27).
When we create individual \archon architectures for each subtask, we see 3.7 and 8.9 percentage point increases in accuracy, on average, for MixEval and MixEval-Hard, respectively.
This finding suggests that reasoning tasks (e.g. mathematics, sciences, logic) require more individualized inference-time architectures for their particular queries.

Coding:
We have observed that ensembling, fusion, and ranking techniques have limited impact on CodeContests (Table 2).
For example, when we apply the general all-source architecture from Table 25 to CodeContests problems, we achieve small gains from \archon (see Figure 7).
One contributing factor is that, unlike the distribution of instruction-following/reasoning tasks, coding tasks tend to have one or two LLMs that perform substantially better than the rest of models (Table 14).
However, when we add unit test generation/evaluation, and increase the number of samples,
\archon’s performance on CodeContests improves significantly (Figure 7), allowing us to boost GPT-4o Pass@1 performance by 56% for Pass@1 (from 25 to 41 out of 140 questions).
Future work should focus on developing better \archon modules for handling the multi-step reasoning that is implemented in various code agents [, ].
For model-based unit test generation/evaluation, we generate 5 unit tests and use the LM to evaluate each candidate response against the generated unit tests, allowing us to rank the different candidate responses.
Lastly, we explored several additional benchmarks for math and code (GSM8K [], MMLU Math [], HumanEval [], and MBPP []) but existing approaches already reach fairly high performances (>90% Pass@1)(Table 28).

Figure: Figure 7: Performance Gains from Combining Multi-Sampling with LLM-based Unit-test Generation/Evaluation: In this architecture, each sample is evaluated against (the same) 5 model-generated unit tests and the first sample that passes all the unit tests is considered as the model’s final response. We use model-based evaluation for the unit tests, passing the prompt, query, and unit tests to the LLM directly for evaluation. For unit test generation and evaluation, we use the same LLM as we do for sampling. We observe strong performance improvements in Pass@1 performance as we scale the number of samples for GPT-4o and Claude 3.5 Sonnet. Our \archon all-source targeted architecture is included in Figure 12, which utilizes Llama 3.1 405B Instruct for generation (1000 samples) and uses GPT-4o for model-based unit test generation/evaluation. ^∗We use the subset of 140 CodeContests problems that do not require image handling. ^∗∗Our \archon all-source generalized architecture baseline (also found in Table 3) was optimized for instruction-following and reasoning tasks, not coding tasks (Section 4.4).
Refer to caption: extracted/5897895/figures/Sampling_for_Code_Contests.png

### 4.4 Task-Specific and General-Purpose \archon Architectures

Task-Specific vs. General-Purpose: We also compare custom \archon architectures, specifically configured to a single evaluation dataset ("Task-specific \archon Architectures"), and a generalized \archon architecture configured to handle all the evaluation datasets ("General-purpose \archon Architectures") (Table 3).
We utilize ITAS to find the generalized \archon architectures in Table 3 (subsection 3.4), maximizing performance over all of the benchmarks explored except CodeContests.
We exclude CodeContests from the generalized \archon architecture search since we found that \archon architectures for coding tasks are most effective with a different set of inference-time techniques compared to instruction-following and reasoning tasks (i.e. increased model sampling combined with model-based unit test generation/evaluation) (Section 3.2; Table 2).
For open-source models, we find that our generalized \archon architecture only lags behind the specialized \archon architectures by 3.4 percentage points, on average, across all the benchmarks, demonstrating the robustness of the \archon architecture found by the ITAS algorithms (Table 3).
We see similar gaps between the generalized and specialized \archon architectures for closed-source models (4.0 percentage points) as well as the all-source models (3.3 percentage points) (Table 3).

Insights from Architecture Construction: We include examples of our learned effective generalized \archon architectures constructed by ITAS in Section A.3, where we breakdown the exact LM components used for constructing each architecture.
For instruction-following and reasoning tasks, we found a generalizable \archon architecture to be most effective with multiple layers of critic-ranker-fuser, chained sequentially to improve candidate generation (Figure 9).
However, the specific models chosen for these LLM components could change task by task, with some tasks benefiting from using a single SOTA closed-source LLM for all the components (e.g. Arena-Hard-Auto and MixEval) (Figure 11) whereas others benefited from a diversity of LLMs in their ensemble (e.g. MT Bench and MixEval-Hard) (Figure 9; Figure 10).
Regardless of models used, we found that scaling inference layers including critics, rankers, and fusers improved performance on instruction-following and reasoning tasks (Figure 5; Section A.3).
For instruction-following and reasoning tasks, the verifier module is more effective than the unit test generation/evaluation module for task-specific \archon architectures (Section 3.2; Table 2).
For coding tasks, we found a high-sample setting to be the most effective, with added layers of unit test generation and evaluation to boost the quality of the final candidate generation (Figure 12; Figure 7).
Overall, our findings demonstrate the benefits of scaling inference-time compute through layering of techniques, showing the importance of effectively and efficiently constructing inference-time architectures.

### 4.5 \archon by Inference Budget

Finally, we compare different \archon architectures across inference budgets for both open-source models and closed-source models (Table 16).
For instruction-following and reasoning tasks, we find consistent improvements in downstream performance as we scale from 1 to 50 inference calls, increasing by 14.3 percentage points, on average, across our selected evaluation benchmarks (Table 16).
However, after roughly 50 inference calls, performance gains plateau. The results suggest that the early addition of LLM components in \archon (e.g. critic, ranker, layers of fusers) led to the most substantial gains in performance and after that, additional LLM components did not ultimately enhance the final generated response.
We see the trend most apparent for the MixEval and MixEval-Hard benchmarks, where additional layers of Critic, Rankers, and Fusers do not benefit performance beyond a 30 inference call budget (Table 16).
Notably, for math and coding tasks, we see continued improvements with additional inference calls by using generated unit tests to evaluate candidate responses, leading to a 56% increase in Pass@1 (Figure 7).

### 4.6 Limitations and Future Work of \archon

Parameter Count: The \archon framework is most effective with LLM with about 70B parameters or more.
When we utilize the \archon architecture with only 7B open-source models, we get a notable decrease in performance (Table 17).
The best 7B \archon configurations lag behind single SOTA (and much larger) models by 15.7% on across all the benchmarks, on average;
7B models work well for ranking but are less effective for critic and fusion. While this is expected, as small models generally exhibit lower performance, their weaker instruction following ability is a compounding factor.

Latency and Costs: \archon is not ideal for tasks that prefer the latency of a single LLM call, such as certain consumer chatbots.
Since \archon architectures often make multiple LLM calls successively for different operations (e.g. ensembling, critiquing, ranking, fusion, etc.), it can often take 5+ more time than a single LLM call (Section A.3).
Furthermore, it can require calling multiple API endpoints for a single query, leading to increased expenditures compared to single-call LLMs (Table 18; Table 19).
Note that these increases in compute costs and latency translate to higher quality responses, and can be justified in many application domains, such as science, math, and programming, or for addressing complex customer service issues.

Components: While \archon is a modular framework, allowing the easy incorporation of new LLMs, new inference-time techniques, and even tool use, we only explore seven LLM inference time techniques in our work (Section 3.1).
The addition of new techniques is a promising avenue for future research.
Furthermore, while different queries can be best suited by different \archon architectures (Table 26; Table 27), the ITAS algorithm selects the best single architecture for the evaluation set queries combined.
Future architecture search could focus on dynamic selection of components on a query-by-query basis.

## 5 Conclusion

This paper presents \archon, a modular framework for optimizing inference-time architectures by integrating multiple inference-time techniques, such as ensembling, ranking, fusion, critique, verification, and unit test generation.
Extensive experiments demonstrate that \archon consistently matches or exceeds the performance of leading closed-source LLMs, such as GPT-4 and Claude-3.5-Sonnet, while only using open-source models across diverse benchmarks, including MT-Bench, AlpacaEval 2.0, Arena-Hard-Auto, MixEval, MixEval-Hard, MATH, and CodeContests.
We attribute \archon’s boost in benchmark performance to two main factors.
The first factor is the ability to leverage inference-time compute towards utilizing multiple LLMs and additional operations (e.g. fusing, ranking, critiquing, verification, unit testing), leading to amplified benefits that scale with additional inference calls (Sections 3.1 and 3.3).
The second factor is the automatic approach for iteratively testing different \archon architectures with ITAS, guaranteeing the optimal configuration given enough exploration steps (Section 3.4).
These results underscore the potential of \archon and ITAS algorithms in advancing the development of high-performing and generally capable inference-time architectures.
The framework and datasets are publicly available on Github:
[https://github.com/ScalingIntelligence/Archon](https://github.com/ScalingIntelligence/Archon).

## 6 Acknowledgements

We thank Simran Arora, Bradley Brown, Ryan Ehrlich, Sabri Eyuboglu, Jordan Juravsky, Jerry Liu, Benjamin Spector, Alyssa Unell, Benjamin Viggiano, and Michael Zhang for their constructive feedback during the composition of the paper.
We would also like to thank our collaborators at the Stanford Artificial Intelligence Laboratory
(SAIL) and TogetherAI.

We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under
Nos. CCF2247015 (Hardware-Aware), CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to
Velocity), and 1937301 (RTML); US DEVCOM ARL under Nos. W911NF-23-2-0184 (Long-context)
and W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under Nos. N000142312633 (Deep
Signal Processing); Stanford HAI under No. 247183; NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft,
NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices,
Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford
Data Science Initiative (SDSI), and members of the Stanford DAWN project: Meta, Google, and
VMWare. The U.S. Government is authorized to reproduce and distribute reprints for Governmental
purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions
or recommendations expressed in this material are those of the authors and do not necessarily
reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S.
Government.

## Appendix A Appendix

### A.1 \archon LLM Components

**Table 4: Rules of \archon Construction: Allowed combinations of each LLM component from Section 3.1.**
| Module | Initial Layer<br>Placement | Placement after<br>Initial Layer | >1 Module<br>in Layer | Increase<br>Candidate<br>Responses | Decrease<br>Candidate<br>Responses |
| --- | --- | --- | --- | --- | --- |
| Generator | Yes | No | Yes | Yes | No |
| Fuser | No | Yes | Yes | Yes | Yes |
| Ranker | No | Yes | No | No | Yes |
| Critic | No | Yes | No | No | No |
| Verifier | No | Yes | No | No | Yes |
| Unit Test<br>Generator | No | Yes | No | No | No |
| Unit Test<br>Evaluator | No | Yes | No | No | No |

**Table 5: Generator Prompt**
| <instruction here>. |
| --- |

**(a) With Critiques**
| You have been provided with a set of responses with their individual critiques of strengths/weaknesses from various open-source models to the latest user query. Your task is to synthesize these responses into a single, high-quality response. It is crucial to critically evaluate the information provided in these responses and their provided critiques of strengths/weaknesses, recognizing that some of it may be biased or incorrect. Your response should not simply replicate the given answers but should offer a refined, accurate, and comprehensive reply to the instruction. Ensure your response is well-structured, coherent, and adheres to the highest standards of accuracy and reliability.<br>Responses from models:<br>1. <response #1><br>Critique: <critique #1><br>2. <response #2><br>Critique: <critique #2><br>...<br>N. <response #N><br>Critique: <critique #N><br><instruction here> |
| --- |

**Table 7: Decoder-Based Ranking Prompt**
| I will provide you with N responses, each indicated by a numerical identifier []. Rank the responses based on their relevance to the instruction: <instruction here>.<br>[1] <response #1><br>[2] <response #2><br>...<br>[N] <response #N><br>Instruction: <instruction here>.<br>Rank the N responses above based on their relevance to the instruction. All the responses should be included and listed using identifiers, in descending order of relevance to the instruction. The output format should be [] > [], e.g., [4] > [2]. Only respond with the ranking results, do not say any word or explain. |
| --- |

**Table 8: Critic Prompt**
| You are a helpful assistant. I will provide you with N responses, each indicated by a numerical identifier (e.g., [1], [2], etc.). Rank the responses based on their relevance to the instruction: <instruction here>.<br>[1] <response #1><br>[2] <response #2><br>...<br>[N] <response #N><br>Instruction: <instruction here>.<br>Evaluate the N responses above based on their relevance to the instruction. All the responses should be included and listed using identifiers. For each response, start the critique with the numerical identifier (e.g., [1]) followed by the strengths and weaknesses. You must include both strengths and weaknesses, even if there are more of one than the other. At the end of each response’s analysis, include two new lines to separate the critiques. Do not include any preface or text after the critiques. Do not include any references to previous critiques within a critique. Start with the analysis for the first response and end with the analysis for the last response. All of the N responses should be included and evaluated using identifiers. Structure each response’s analysis as follows:<br>Strengths:<br>- <strength #1><br>- <strength #2><br>- <strength #n><br>Weaknesses:<br>- <weakness #1><br>- <weakness #2><br>- <weakness #n> |
| --- |

**Table 9: Verifier Prompt**
| I will provide you with a response indicated by the identifier ’Response’. Provide reasoning for why the response accurately and completely addresses the instruction: <instruction here>.<br>Response: <response><br>Instruction: <instruction here>.<br>Provide the reasoning for the response above based on its relevance, completeness, and accuracy when compared to the instruction. Do not include any preface or text after the reasoning. |
| --- |

**(a) With Unit Test Cap**
| Instruction Prompt: Given the following query, generate a set of N unit tests that would evaluate the correctness of responses to this query.<br>- The unit tests should cover various aspects of the query and ensure comprehensive evaluation.<br>- Each unit test should be clearly stated and should include the expected outcome.<br>- The unit tests should be in the form of assertions that can be used to validate the correctness of responses to the query.<br>- The unit test should be formatted like ’The answer mentions…’, ’The answer states…’, ’The answer uses…’, etc. followed by the expected outcome.<br>- Solely provide the unit tests for the question below. Do not provide any text before or after the list. Only output the unit tests as a list of strings (e.g., [’unit test #1’, ’unit test #2’, ’unit test #3’]).<br>Query: <instruction here> |
| --- |

**(a) Instruction-Following Query**
| Instruction Prompt: Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions.<br>1.<br>Unit Test #1: The blog post mentions at least two cultural experiences specific to Hawaii.<br>2.<br>Unit Test #2: The blog post highlights at least three must-see attractions in Hawaii.<br>3.<br>Unit Test #3: The tone of the blog post is engaging and uses descriptive language that would appeal to readers interested in travel.<br>4.<br>Unit Test #4: The blog post includes factual information about Hawaii’s culture, such as local customs, festivals, or historical facts.<br>5.<br>Unit Test #5: The blog post contains a clear narrative structure, including an introduction, main body, and a conclusion. |
| --- |

**Table 12: Unit Test Evaluator Prompt**
| Given the following query, candidate response, and unit tests, evaluate whether or not the response passes each unit test.<br>- In your evaluation, you should consider how the response aligns with the unit tests, retrieved documents, and query.<br>- Provide reasoning before you return your evaluation.<br>- At the end of your evaluation, you must finish with a list of verdicts corresponding to each unit test.<br>- You must include a verdict with one of these formatted options: ’[Passed]’ or ’[Failed]’.<br>- Here is an example of the output format:<br>Unit Test #1: [Passed]<br>Unit Test #2: [Failed]<br>Unit Test #3: [Passed]<br>- Each verdict should be on a new line and correspond to the unit test in the same position.<br>- Here is the query, response, and unit tests for your evaluation:<br>Query: <instruction here>.<br>Candidate Response:<br><response><br>Unit Tests:<br>Unit Test #1: <Unit Test #1><br>Unit Test #2: <Unit Test #2><br>...<br>Unit Test #N: <Unit Test #N> |
| --- |

### A.2 \archon LLM Analysis

**Table 13: Models Tested with \archon.**
| Model | Source Code | Parameter<br>Count | Max Sequence<br>Length |
| --- | --- | --- | --- |
| GPT-4o [] | Closed-Source | — | 128K |
| GPT-4-Turbo [] | Closed-Source | — | 128K |
| Claude-3-Opus [] | Closed-Source | — | 200K |
| Claude-3.5-Sonnet [] | Closed-Source | — | 200K |
| Claude-3-Haiku [] | Closed-Source | — | 200K |
| Llama-3.1-70B-Instruct [] | Open-Source | 70B | 8k |
| Llama-3.1-405B-Instruct [] | Open-Source | 70B | 8k |
| DeepSeek LLM 67B Chat [] | Open-Source | 67B | 32k |
| Qwen2 72B Instruct [] | Open-Source | 72B | 32k |
| Qwen1.5 110B Chat [] | Open-Source | 110B | 32k |
| Qwen1.5 72B Chat [] | Open-Source | 72B | 32k |
| Mixtral 8x22B v0.1 [] | Open-Source | 176B | 32k |
| WizardLM 8x22B [] | Open-Source | 176B | 32k |
| dbrx-instruct [] | Open-Source | 132B | 32k |
| princeton-nlp/Llama-3-Instruct-8B-SimPO [] | Open-Source | 8B | 8k |
| princeton-nlp/Llama-3-Instruct-8B-DPO [] | Open-Source | 8B | 8k |
| princeton-nlp/Llama-3-Instruct-8B-RDPO [] | Open-Source | 8B | 8k |
| princeton-nlp/Llama-3-Instruct-8B-IPO [] | Open-Source | 8B | 8k |
| Llama-3.1-8B-Instruct [] | Open-Source | 8B | 8k |
| Qwen2-7B-Instruct [] | Open-Source | 7B | 32k |
| Qwen/Qwen1.5-7B-Chat [] | Open-Source | 7B | 32k |
| mistralai/Mistral-7B-Instruct-v0.2 [] | Open-Source | 7B | 32k |
| cognitivecomputations/dolphin-2.2.1-mistral-7b [] | Open-Source | 7B | 32k |
| microsoft/Phi-3-mini-4k-instruct [] | Open-Source | 4B | 4k |
| HuggingFaceH4/zephyr-7b-beta [] | Open-Source | 7B | 32k |
| microsoft/Phi-3-small-8k-instruct [] | Open-Source | 7B | 8k |
| snorkelai/Snorkel-Mistral-PairRM-DPO [] | Open-Source | 7B | 32k |
| mistralai/Mistral-7B-Instruct-v0.3 [] | Open-Source | 7B | 32k |

**Table 14: \archon Generation and Fusion Performances for Single Models: For Alpaca Eval 2.0, we use the length-controlled win rate (LC WR). For fusion, we gather one candidate from each of the top-10 generator models.**
|  | MT Bench | Alpaca Eval 2.0 | Arena Hard Auto | MixEval | MixEval Hard | MATH | CodeContests |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models | Gen | Fusion | Gen | Fusion | Gen | Fusion | Gen | Fusion | Gen | Fusion | Gen | Fusion | Gen | Fusion |
| GPT-4o | 44.7% | 61.9% | 57.5% | 64.5% | 48.1% | 69.2% | 88.0% | 89.4% | 63.6% | 65.4% | 72.0% | 75.5% | 17.9% | 19.4% |
| GPT-4-Turbo | 42.2% | 63.1% | 55.0% | 65.8% | 48.1% | 61.9% | 88.9% | 89.0% | 64.1% | 64.4% | 74.5% | 76.5% | 9.3% | 14.2% |
| Claude 3<br>Opus | 30.9% | 57.2% | 40.5% | N/A | 27.0% | 47.9% | 88.3% | 88.2% | 63.6% | 64.0% | 72.5% | 71.0% | 10.0% | 12.5% |
| Claude 3.5<br>Sonnet | N/A | 71.9% | 52.37% | 63.6% | N/A | 73.2% | 89.7% | 89.3% | 68.9% | 69.5% | 72.0% | 74.5% | 12.1% | 15.5% |
| Qwen 2<br>72B Instruct | 35.0% | 59.7% | 37.48% | 56.0% | 14.5% | 49.5% | 86.5% | 87.5% | 58.7% | 61.1% | 76.0% | 78.5% | 3.6% | 5.2% |
| DeepSeek LLM<br>67B Instruct | 18.4% | 20.0% | 17.8% | 17.1% | N/A | N/A | 79.2% | N/A | 42.5% | N/A | 45.0% | N/A | 5.7% | N/A |
| Qwen 1.5<br>72B Chat | 24.7% | 46.3% | 36.6% | 55.7% | 14.4% | 36.4% | 84.5% | 82.1% | 50.3% | 52.2% | 62.5% | 65.5% | 15.0% | 13.9% |
| Qwen 1.5<br>110B Chat | 34.4% | 50.3% | 43.6% | 55.9% | 21.9% | 39.7% | 85.3% | 86.5% | 51.8% | 55.6% | 67.0% | 72.5% | 3.6% | 7.8% |
| Wizard 8x22B | 53.8% | 57.2% | 44.7% | 50.6% | 45.6% | 51.2% | 83% | 78.1% | 54.3% | 50.4% | 69.0% | 58.5% | 7.1% | 10.4% |
| Llama 3.1<br>8B Instruct | 33.1% | 45.9% | 25.6% | 34.9% | 11.9% | 28.6% | 75.0% | 57.5% | 41.3% | 46.5% | 59.0% | 60.5% | 8.6% | 7.8% |
| Llama 3.1<br>70B Instruct | 45.0% | 51.9% | 35.6% | 40.2% | 23.8% | 37.2% | 85.7% | 83.5% | 61.1% | 65.5% | 69.0% | 71.5% | 20.7% | 23.4% |
| Llama 3.1<br>405B Instruct | 44.7% | N/A | 40.3% | N/A | 28.4% | N/A | 88.9% | N/A | 66.2% | N/A | 74.5% | N/A | 27.1% | N/A |

**Table 15: Jaccard Similarities between Candidates Responses and Fused Response by Benchmark: For the fuser, we use the best-performing 70B+ model for benchmark.**
|  | Jaccard Similarity (%) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Inference-Time<br>Architecture | MT Bench | AlpacaEval 2.0 | Arena-Hard<br>Auto | MixEval | MixEval<br>Hard | MATH | Code<br>Contests |
| Best Open-Source 70B+ Model,<br>Sampled 8 Times + Fuser | 45.3% | 52.1% | 48.4% | 55.2% | 58.9% | 65.2% | 63.7% |
| Ensemble (8 Top Models),<br>Sampled Once Each + Fuser | 31.6% | 34.1% | 28.9% | 38.6% | 40.9% | 57.1% | 53.4% |

Figure: Figure 8: Fusion Layer Efficacy by Benchmark: From solely scaling the fusion layers, we see limited benefits across the benchmarks explored but when we add other inference-time techniques, such as Critic and Ranker, we see increased downstream performance as we continue scaling inference-time compute (Figure 5). We use an 8-model ensemble of the top Generator models for each benchmark (Table 14). For our Fuser layers, we use the best Fuser model for the final fuser layer (Table 14). For the intermediate layers, we use the top-8 Fuser models for each benchmark.
Refer to caption: extracted/5897895/figures/Fusion_Layers_Analysis.png

### A.3 \archon Architectures

Figure: Figure 9: All-Source Generalizable \archon Architecture: Using ITAS, we found this all-source \archon configuration to be effective across the benchmarks explored (except for CodeContests). In the diagram above, we use 10 SOTA all-source LLMs to create multiple successive layers of critic, ranker, and fusers, with each successive fuser layer having less fusers to produce a "funneling" effect as the candidate generations are processed. The layers of critic, ranker, and fuser led to better candidate generations through iterative critique and rewriting. Each of the initial Generator models were sampled once.
Refer to caption: x1.png

Figure: Figure 10: Open-Source Generalizable \archon Architecture: Using ITAS, we found this open-source \archon configuration to be effective across the benchmarks explored (except for CodeContests). In the diagram above, we use 10 SOTA open-source LLMs to create multiple successive layers of critic, ranker, and fusers, with each successive fuser layer having less fusers to produce a "funneling" effect as the candidate generations are processed. The layers of critic, ranker, and fuser led to better candidate generations through iterative critique and rewriting. Each of the initial Generator models were sampled once.
Refer to caption: x2.png

Figure: Figure 11: All-Source \archon Architecture for Instruction-Following: Using ITAS, we found Claude-3.5-Sonnet as a generator, critic, ranker, and fuser to be an effective targeted architecture for instruction-following tasks, such as MT Bench and AlpacaEval 2.0. The ranker picks the top-5 candidate responses to send to the fuser. Each of the initial Generator models were sampled once.
Refer to caption: x3.png

Figure: Figure 12: All-Source \archon Architecture for CodeContests: By using Llama 3.1 405B for generation and GPT-4o for unit testing, we were able to get improved code generation on CodeContests (Figure 7). The unit test generator produces 10 unit tests for evaluation. Each of the initial Generator models were sampled once.
Refer to caption: x4.png

### A.4 \archon by Inference Compute Budget, Model Size, and Cost

**Table 16: \archon with Different Inference Budgets: For AlpacaEval 2.0, we use the length-controlled win rate (LC WR).**
|  |  | Datasets |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Number of<br>Inference Calls | MT<br>Bench | Alpaca<br>Eval 2.0 | Arena<br>Hard Auto | MixEval | MixEval<br>Hard |
| 70B+<br>Models | 1 | 55.0% | 44.7% | 45.6% | 86.5% | 61.1% |
| 10 | 52.5% | 50.6% | 45.6% | 86.5% | 63.9% |  |
| 20 | 65.3% | 60.4% | 59.4% | 89.0% | 65.0% |  |
| 30 | 69.2% | 64.5% | 69.0% | 89.5% | 67.5% |  |
| 40 | 69.5% | 66.7% | 69.0% | 89.5% | 67.5% |  |
| 50 | 71.6% | 66.7% | 69.0% | 89.5% | 67.5% |  |
| Closed<br>Models | 1 | 45.0% | 57.5% | 48.1% | 88.9% | 68.9% |
| 10 | 57.1% | 63.2% | 68.4% | 90.0% | 70.1% |  |
| 20 | 59.4% | 66.5% | 75.5% | 90.6% | 70.5% |  |
| 30 | 70.2% | 68.8% | 77.4% | 90.6% | 72.9% |  |
| 40 | 75.5% | 68.8% | 77.4% | 90.6% | 72.9% |  |
| 50 | 80.4% | 68.8% | 77.4% | 90.6% | 72.9% |  |

**Table 17: \archon with 7B Open-Source Models: For AlpacaEval 2.0, we use the length-controlled win rate (LC WR). We use open-source 7B models for testing from Table 13.**
|  | Datasets |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Models / LLM Systems | MT<br>Bench | Alpaca<br>Eval 2.0 | Arena<br>Hard Auto | MixEval | MixEval<br>Hard |
| SOTA Single-Model | 44.7% | 57.5% | 48.1% | 68.9% | 89.7% |
| Best Model, 1-Sample | 15.7% | 41.0% | 18.3% | 76.2% | 46.1% |
| Best Model - 10-Sample + Ranking | 16.5% | 43.2% | 18.9% | 78.4% | 48.5% |
| 10-Model, 1-Sample Ensemble + Ranking | 22.4% | 48.2% | 25.6% | 81.5% | 52.9% |
| 10-Model, 1-Sample Ensemble + Fusion | 14.3% | 39.4% | 17.5% | 73.2% | 45.2% |
| 10-Model, 1-Sample Ensemble<br>+ Top-5 Ranking + Fusion | 15.9% | 41.2% | 18.0% | 75.1% | 46.9% |
| 10-Model, 1-Sample Ensemble<br>+ Critic + Fusion | 10.5% | 38.4% | 16.5% | 71.4% | 42.5% |

**Table 18: Model API Costs as of August 2024**
| Models | Cost ($) per<br>Million Input Tokens | Cost ($) per<br>Million Output Tokens |
| --- | --- | --- |
| Claude 3.5 Sonnet | $3 | $15 |
| Claude 3.0 Opus | $15 | $75 |
| GPT-4o | $5 | $15 |
| GPT-4-Turbo | $10 | $30 |
| TogetherAI - Llama 3.1 405B Instruct | $5 | $5 |
| TogetherAI - Llama 3.1 70B Instruct | $0.88 | $0.88 |
| TogetherAI - Other Models | $0.90 | $0.90 |

**Table 19: \archon Costs per Query by Benchmark**
|  | Cost ($) per Query for Benchmark |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model /<br>LLM System | MT Bench | AlpacaEval 2.0 | Arena-Hard<br>Auto | MixEval | MixEval<br>Hard | MATH | Code<br>Contests |
| Claude 3.5 Sonnet | 0.0305 | 0.0171 | 0.0212 | 0.0231 | 0.0226 | 0.0325 | 0.384 |
| GPT-4o | 0.0481 | 0.0236 | 0.0324 | 0.0357 | 0.0361 | 0.514 | 0.562 |
| Llama 3.1 405B Instruct | 0.0281 | 0.0174 | 0.0185 | 0.0212 | 0.0205 | 0.305 | 0.372 |
| General Purpose<br>\archon Architecture | 0.364 | 0.189 | 0.195 | 0.284 | 0.252 | 0.375 | 0.461 |
| Task Specific<br>\archon Architecture | 0.401 | 0.210 | 0.221 | 0.295 | 0.265 | 0.425 | 0.448 |

### A.5 Bayesian Optimization

Bayesian Optimization is a sequential design strategy for global optimization of black-box functions that are expensive to evaluate []. It is particularly useful when dealing with functions that have unknown forms and are costly to evaluate, such as hyperparameter tuning in machine learning.

#### A.5.1 Basic Idea of Bayesian Optimization

The core idea behind Bayesian Optimization is to build a probabilistic model of the objective function and use it to select the most promising points to evaluate next.
This process involves two main components:

- 1.
Surrogate Model: A probabilistic model (often a Gaussian Process) that approximates the unknown objective function.
- 2.
Acquisition Function: A function that guides the search for the optimum by suggesting the next point to evaluate, based on the surrogate model.

#### A.5.2 Steps in Bayesian Optimization

- 1.
Initialization: Begin with a set of initial points $\mathcal{D}=\{(\mathbf{x}_{1},y_{1}),(\mathbf{x}_{2},y_{2}),\dots,(\mathbf{x}_
{n},y_{n})\}$, where $\mathbf{x}_{i}$ is the input, and $y_{i}=f(\mathbf{x}_{i})$ is the objective function value at $\mathbf{x}_{i}$.
- 2.
Model Building: Fit a surrogate model (e.g., Gaussian Process) to the observed data $\mathcal{D}$.
- 3.
Acquisition: Use the acquisition function to select the next point $\mathbf{x}_{n+1}$ to evaluate:
$\mathbf{x}_{n+1}=\arg\max_{\mathbf{x}}a(\mathbf{x}\mid\mathcal{D})$
where $a(\mathbf{x}\mid\mathcal{D})$ is the acquisition function.
- 4.
Evaluation: Evaluate the objective function at $\mathbf{x}_{n+1}$ to get $y_{n+1}=f(\mathbf{x}_{n+1})$.
- 5.
Update: Add the new data point $(\mathbf{x}_{n+1},y_{n+1})$ to the dataset $\mathcal{D}$.
- 6.
Repeat: Repeat steps 2-5 until convergence or a stopping criterion is met (e.g., budget exhausted, no significant improvement).

#### A.5.3 Gaussian Process as a Surrogate Model

A Gaussian Process (GP) is commonly used as a surrogate model in Bayesian Optimization. It is defined by a mean function $\mu(\mathbf{x})$ and a covariance function (kernel) $k(\mathbf{x},\mathbf{x}^{\prime})$:

$$ $f(\mathbf{x})\sim\mathcal{GP}(\mu(\mathbf{x}),k(\mathbf{x},\mathbf{x}^{\prime}))$ $$

Given a set of observations $\mathcal{D}$, the GP provides a predictive distribution for the objective function at a new point $\mathbf{x}$:

- •
Predictive Mean: The expected value of the function at $\mathbf{x}$:
$\mu(\mathbf{x}\mid\mathcal{D})=\mathbf{k}_{n}(\mathbf{x})^{T}\mathbf{K}_{n}^{-
1}\mathbf{y}$
where $\mathbf{k}_{n}(\mathbf{x})$ is the covariance vector between $\mathbf{x}$ and the training points, and $\mathbf{K}_{n}$ is the covariance matrix of the training points.
- •
Predictive Variance: The uncertainty in the function value at $\mathbf{x}$:
$\sigma^{2}(\mathbf{x}\mid\mathcal{D})=k(\mathbf{x},\mathbf{x})-\mathbf{k}_{n}(
\mathbf{x})^{T}\mathbf{K}_{n}^{-1}\mathbf{k}_{n}(\mathbf{x})$

#### A.5.4 Acquisition Functions

Acquisition functions guide the search for the optimum by balancing exploration (trying out areas with high uncertainty) and exploitation (focusing on areas with high predicted values). Common acquisition functions include:

- 1.
Expected Improvement (EI):
$\text{EI}(\mathbf{x})=\mathbb{E}[\max(0,f(\mathbf{x})-f(\mathbf{x}^{+}))]$
where $f(\mathbf{x}^{+})$ is the best observed value so far.
- 2.
Probability of Improvement (PI):
$\text{PI}(\mathbf{x})=\mathbb{P}(f(\mathbf{x})>f(\mathbf{x}^{+})+\xi)$
where $\xi$ is a small positive number.
- 3.
Upper Confidence Bound (UCB):
$\text{UCB}(\mathbf{x})=\mu(\mathbf{x}\mid\mathcal{D})+\kappa\sigma(\mathbf{x}
\mid\mathcal{D})$
where $\kappa$ controls the trade-off between exploration and exploitation.

#### A.5.5 Summary of Bayesian Optimization

Bayesian Optimization iteratively uses a surrogate model to approximate the objective function and an acquisition function to decide where to sample next. By focusing on promising areas of the search space and systematically exploring uncertain regions, it efficiently optimizes complex, expensive-to-evaluate functions.

### A.6 ITAS Algorithms Comparisons

**Table 20: MT Bench**
| # of Init.<br>Points<br>% of Total<br>Configs<br>Iter. till<br>Max. Config.<br>Comb.<br>Iter.<br>50<br>1.57%<br>389<br>439<br>60<br>1.88%<br>321<br>381<br>70<br>2.19%<br>287<br>357<br>80<br>2.51%<br>268<br>348<br>90<br>2.82%<br>265<br>355<br>100<br>3.13%<br>258<br>358<br>110<br>3.45%<br>262<br>372<br>120<br>3.76%<br>253<br>373<br>Table 20: MT Bench<br># of Init.<br>Points<br>% of Total<br>Configs<br>Iter. till<br>Max. Config.<br>Comb.<br>Iter.<br>50<br>1.57%<br>658<br>708<br>60<br>1.88%<br>575<br>635<br>70<br>2.19%<br>502<br>572<br>80<br>2.51%<br>453<br>533<br>90<br>2.82%<br>451<br>541<br>100<br>3.13%<br>455<br>555<br>110<br>3.45%<br>442<br>552<br>120<br>3.76%<br>437<br>557<br>Table 21: Arena-Hard-Auto |
| --- |

**Table 23: ITAS Algorithms Comparison by Inference Call Budget: For our comparison, we evaluate on MT Bench.**
|  | Iterations to Convergence |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Inference Budget | 10 | 20 | 30 | 40 | 50 |
| Random Selection | 310 | 643 | 1153 | 2062 | 2695 |
| Greedy Search | 222 | 576 | 787 | 1607 | 1685 |
| Bayes Optimization | 166 | 256 | 340 | 387 | 415 |

### A.7 \archon Benchmarks and Results

**Table 24: \archon’s Strong Performance on the Complete Evaluation Datasets after ITAS Optimization: We find that \archon’s inference-time architectures consistently outperform single-call state-of-the-art LLMs, both open-source and closed-source baselines, when evaluating on the complete benchmarks (Table 25). We explore two configurations: ITAS for building custom \archon configurations for each individual benchmark and ITAS for building a single general-purpose \archon configuration for all the benchmarks (Section 4.4). We find that a general \archon configuration lags behind the custom ones by only 3.2 percentage points, on average, across our all-source settings, which suggests the efficacy of general-purpose inference-time architectures created with our framework. For Arena-Hard-Auto, we also include a configuration with Claude 3.5 Sonnet as a stronger reference model for comparison against \archon inference-time architectures and to mitigate bias from GPT judges towards GPT generations. For MT Bench, we use a GPT-4-0314 judge model instead of newer LLM judges to be consistent with previous results on this benchmark. For our task-specific \archon architectures, we also provide the average inference calls across the given benchmarks. For our full-list of models explored, please see Table 13. For MATH, we use a randomly sampled subset of size 200 for evaluation (Section 4.1; Table 25). We include our \archon architecture results on the held-out 80% subset of each evaluation benchmark in Table 3.**
|  |  |  | Datasets |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | MT<br>Bench | Alpaca<br>Eval 2.0 | Arena<br>Hard Auto | Arena<br>Hard Auto | MixEval<br>Hard | MixEval | MATH^∗ |  |
|  | Judge Model |  | GPT-4<br>0314 | GPT-4<br>Turbo | GPT-4<br>Turbo | GPT-4<br>Turbo | N/A | N/A | N/A |  |
|  | Reference Model |  | Claude 3.5<br>Sonnet | GPT-4<br>Turbo | Claude 3.5<br>Sonnet | GPT-4<br>Turbo | N/A | N/A | N/A |  |
|  | Model / LLM System | Infer.<br>Calls | W.R. | L.C.<br>W.R. | Raw<br>W.R. | W.R. | W.R | Acc. | Acc. | Pass<br>@1 |
|  | GPT-4o - 2024-05-13 | 1 | 44.7% | 57.5% | 51.3% | 48.1% | 80.3% | 63.6% | 88.0% | 72.0% |
|  | Claude 3.5 Sonnet | 1 | N/A | 52.4% | 40.6% | N/A | 80.9% | 68.9% | 89.7% | 72.0% |
|  | Llama 3.1 405B Instruct | 1 | 44.7% | 40.3% | 37.7% | 28.4% | 64.1% | 66.2% | 88.9% | 74.0% |
|  | MoA | 19 | 51.6% | 65.1% | 59.8% | 52.2% | 84.2% | 62.5% | 87.3% | 72.5% |
|  | MoA Lite | 7 | 45.6% | 59.3% | 57.0% | 40.6% | 87.8% | 61.1% | 87.1% | 70.5% |
| Open<br>Source | General-purpose<br>\archon Architecture | 35 | 67.5% | 63.0% | 68.3% | 66.2% | 85.1% | 65.5% | 86.9% | 75.5% |
| Task-specific<br>\archon Architectures | 44 | 71.6% | 66.7% | 70.7% | 69.0% | 89.5% | 67.5% | 89.6% | 80.5% |  |
| Closed<br>Source | General-purpose<br>\archon Architecture | 32 | 73.1% | 63.5% | 69.1% | 70.5% | 85.8% | 67.7% | 88.2% | 77.0% |
| Task-specific<br>\archon Architectures | 40 | 77.5% | 68.4% | 72.1% | 74.4% | 90.2% | 72.9% | 90.4% | 79.0% |  |
| All<br>Source | General-purpose<br>\archon Architecture | 35 | 76.8% | 65.8% | 70.2% | 72.5% | 89.3% | 70.1% | 88.1% | 78.0% |
| Task-specific<br>\archon Architectures | 39 | 80.4% | 67.6% | 73.3% | 76.1% | 92.1% | 72.9% | 90.6% | 80.5% |  |

**Table 25: Benchmark Overview: Evaluation configurations for AlpacaEval 2.0 [], Arena-Hard-Auto [], MT-Bench [], MixEval [], MixEval Hard, MATH [], and CodeContests []**
| Benchmark | Example<br>Count | Baseline<br>Model | Judge<br>Model | Scoring Type | Metric |
| --- | --- | --- | --- | --- | --- |
| AlpacaEval 2.0 | 805 | GPT-4-Turbo | GPT-4-Turbo | Pairwise<br>Comparison | L.C. & Raw<br>Win Rates |
| Arena-Hard-Auto | 500 | Claude-3.5-Sonnet<br>GPT-4-0314 | GPT-4-Turbo | Pairwise<br>Comparison | Win Rate |
| MT-Bench | 80 | Claude-3.5-Sonnet | GPT-4-0314 | Pairwise<br>Comparison | Adjusted<br>Win Rate |
| MixEval | 2000 | N/A | N/A | Ground Truth | Accuracy |
| MixEval-Hard | 500 | N/A | N/A | Ground Truth | Accuracy |
| MATH | 200<br>(sampled from 5000) | N/A | N/A | Ground Truth | Pass@1 |
| CodeContests | 140<br>(non-visual queries) | N/A | N/A | Ground Truth | Pass@1 |

**Table 26: MixEval Results by Sub-Dataset: For the average computed, we do not introduce any weighting for each dataset.**
|  |  | MixEval - Sub-Datasets |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model / LLM System | Infer.<br>Calls | GSM8K | TriviaQA | DROP | MATH | BBH | AGIEval | Average |
| GPT-4o - 2024-05-13 | 1 | 94.9 | 89.1 | 88.2 | 98.5 | 98.3 | 71.5 | 90.3 |
| Claude 3.5 Sonnet | 1 | 98.0 | 92.0 | 92.6 | 96 | 95.6 | 78.0 | 92.0 |
| Llama 3.1 405B Instruct | 1 | 98.2 | 87.9 | 89.6 | 91.5 | 95.8 | 73.2 | 89.6 |
| General-purpose<br>\archon Architecture | 29 | 98.3 | 94.8 | 94.6 | 98.1 | 97.3 | 82.1 | 94.2 |
| Task-specific<br>\archon Architectures | 34 | 98.2 | 96.7 | 95.6 | 98.5 | 98.8 | 84.2 | 95.7 |

**Table 27: MixEval-Hard Results by Sub-Dataset: For the average computed, we do not introduce any weighting for each dataset.**
|  |  | MixEval - Sub-Datasets |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model / LLM System | Infer.<br>Calls | GSM8K | TriviaQA | DROP | MATH | BBH | AGIEval | Average |
| GPT-4o - 2024-05-13 | 1 | 72.3 | 70.5 | 70.2 | 94.4 | 80.0 | 53.5 | 73.5 |
| Claude 3.5 Sonnet | 1 | 87.3 | 75.5 | 79.3 | 82.5 | 80.0 | 74.6 | 79.9 |
| Llama 3.1 405B Instruct | 1 | 98.7 | 71.2 | 70.7 | 86.9 | 78.8 | 62.0 | 78.1 |
| General-purpose<br>\archon Architecture | 33 | 96.7 | 82.7 | 83.2 | 93.4 | 82.0 | 76.7 | 85.8 |
| Task-specific<br>\archon Architectures | 37 | 98.9 | 86.2 | 85.2 | 96.2 | 86.0 | 80.1 | 88.8 |

**Table 28: Additional Math and Code Benchmarks Explored**
|  | GSM8K | MMLU<br>Math | HumanEval<br>Python | MBPP |
| --- | --- | --- | --- | --- |
| Model | Pass@1 | Pass@1 | Pass@1 | Pass@1 |
| GPT-4o | 97.1% | 84.8% | 89.0% | 87.5% |
| Claude 3.5 Sonnet | 96.8% | 90.9% | 90.2% | 88.9% |
| Llama 3.1 405B Instruct | 95.9% | 85.4% | 90.2% | 88.6% |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: 1. The results seem to be generally strong. Even if it requires many calls to the models to yield the final result, the reported performance numbers on a variety of benchmarks are impressive, often matching or exceeding state of the art performance. 

2. The authors produce an open sourced framework so that others can reproduce and build on the results.
summary: This paper presents ARCHON, a framework for stacking different LLM components to leverage test-time compute to improve performance. While a more general architecture is introduced, the actual pipeline first generates K candidate solutions and then passes through the test through a number of fusion layers before a final verifier/unit test layer. Many hyperparameters are introduced to control the architecture and then they are optimized either with bayesian optimization or greedy search on a subset (20%) of the test set. This yields a pipeline that substantially improves performance on a variety of benchmarks like MT-bench and MATH.
weaknesses: 1. There are some weird formatting issues with the paper. All the characters seem to be compressed compared to other submissions I am reviewing. But also, the bottom margins are substantially enlarged.

2. The comparisons often don't seem to be very fair and baselines are lacking. We should match the FLOP/token/dollar budgets between different methods to see which methods are most effective at turning additional compute into performance. The results are often confounded by comparing across substantially different budgets. For example, comparing elaborate inference techniques with single model calls. This is interesting to show that open source models can be augmented to out-perform a closed source model, but for good science we would want to see an apples-to-apples comparison of different inference techniques on a fixed model or models. Fixing the budget in this comparison is important. For example, in the paper in Figure 3 we see a comparison of different ablations, but some of them require much more compute than others since they add in extra model components. It would be useful to compare with a fixed budget where e.g. if we add fusers, then we need to reduce K to keep the total FLOPs of the system constant.

3. It is not clear what is the real novelty of the method or what the underlying principles of the approach are. Each component of the model alone is a standard component from the literature, and there already exist methods for combining these sorts of things and optimizing them like DsPy and mixture of agents. The main contribution seems to be setting up a particular way to parameterize a hyperparameter optimization problem, but it is not clear why this particular version was chosen. There are a variety of ablations showing that adding various components can improve performance, but there does not seem to be much insight into what is actually going on here or how the specific hyperparameter problem was chosen.

4. The presentation does not track the actual method in the experiments very well, making it somewhat confusing. The presentation in 3.2 is very broad encompassing many architectures. Then 3.3 seems to choose one very specific architecture, which all the experiments use, but it is not clear exactly what this architecture is. Perhaps it would be more clear to just present the actual method used in the experiments directly, and then discuss generalizations as future work rather than flag-planting something that is not actually attempted.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: (1) In Figure 2, the example system architecture includes two GPT-4o generators in the first layer, yet Section 3.3 states that generators are selected from the top-K best models. Is there an inconsistency in the figure, or is there something I may have misunderstood?

(2) It is puzzling that the paper does not include a lightweight version in the experiments, i.e., exclusively using models with fewer than 8B parameters. On one hand, in practical applications, it is more common to deploy a mixture of smaller models due to resource constraints, making this approach more valuable. On the other hand, the coexistence of models of different sizes might cause the ITAS algorithm to disproportionately favor larger models, potentially disregarding the smaller ones. This tendency is also evident in Appendix A.4, where almost all architectures rely predominantly on large models. Why is there no consideration for the exclusive use of small-sized models?

(3) In Section 4.4, the construction strategy for the General-purpose ARCHON Architecture is unclear. What is meant by ``maximizing performance over all benchmarks''? Is ITAS performed sequentially on sampled subsets from each dataset, or only once on the concatenated sampled subsets?
soundness: 2
strengths: The paper provides an insightful summary of existing inference-time techniques in the LLM field, distilling their concepts into well-structured building blocks that establish a robust paradigm for constructing LLM systems. The proposed framework, ARCHON, demonstrates significant performance improvements on common benchmarks, highlighting its effectiveness. The results from the LLM component interaction experiments are also valuable in real-world practice. Additionally, the presentation is generally clear and fluid, facilitating a coherent understanding of the paper's work.
summary: This paper explores the composition and interactions of popular inference-time techniques in the field of large language models (LLMs) to better understand their relationships and performance across various tasks. It introduces a framework, ARCHON, designed to combine mainstream inference-time methods and select the optimal set of models for specific benchmarks or tasks. The proposed LLM system demonstrates significant improvements across a variety of datasets, outperforming any single LLM, including the most competitive models. The paper also provides a thorough empirical analysis of how various factors influence the outcomes.
weaknesses: While this work has involved substantial effort, I recommend rejecting the paper for the following reasons:
(1) The paper offers few novel insights into inference-time techniques, and the main conclusions drawn from the experiments are rather trivial, relying primarily on parameter tuning without theoretical justification; (2) The experimental setup is neither practical nor comprehensive enough to fully demonstrate the interaction mechanisms of mentioned methods; (3) The proposed framework is of limited practical value due to the high computational costs associated with parameter searching and inference.

The paper contributes little to the understanding of the utility or interactions of inference-time techniques. The improvements in downstream tasks through the combination of techniques, as discussed in Section 4, are predictable and have been partly addressed in prior work. Simply combining these techniques and presenting performance results does not constitute a substantial contribution. Additionally, the analysis of different utilities lacks theoretical grounding. The claimed insights, such as those in Sections 4.4 and A.2, are simply descriptive observations from comparative experiments. As seen in A.2, under the specific settings (using only 70B+ open-source models), the results are unlikely to provide generalizable rules.

Regarding the experimental setup, several points require clarification. See the question part for details.

In addition to these points, the proposed framework is significantly limited by its high computational cost. Even with parallel execution of LLMs within the same layer, the system takes five times longer to respond, not to mention the increased demand for computational resources and associated costs, as highlighted in Section A.5. Handling such a system for models of different architectures and sizes would be a great challenge to both hardware and operation-maintenance service in real-world applications. Given the flexibility of the proposed framework, it would be beneficial to consider budget and cost constraints when searching for the optimal architecture in future work. Providing a more cost-effective version of the general-purpose architecture would also enhance the framework's practicality.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: + Figure 5's table is somewhat confusing. While ARCHON results appear to have two sets of results (general and task specific), the other LLM systems are not categorized at all. Also, the inference calls are not close to each other---is there a reason why one cannot give the baselines a similar level of compute? 
+ Almost the entirety of the search algorithm is pushed to the appendix A6, which reads like a textbook overview of classic Bayesian optimization with a Gaussian process. This section should really be edited to state what the authors do, not generically about what Bayesian optimization is. For example, Gaussian processes are described as an example of a surrogate, and examples of acquisition functions are given, but the specific one used in the paper for the given approach is not described in A6. What is the specific algorithm being used? 
+ The set of rules may make architecture building with these LLM components as building blocks a bit more complex than traditional architecture search. In fact, with all of these rules, the search may end up being mostly parameter search instead of true architecture search, if the overall architecture is already mostly defined by the rules. Are there examples of different "paradigms" of combinations that arise from ARCHON? or are they all mostly the same?
soundness: 3
strengths: + The approach performs well, both in the closed/open model settings, and can produce combinations of open models that reach close to closed model performance
+ The approach is relatively inexpensive, using approximately 40x inference calls on a single query (though some opinions may differ on whether this is inexpensive)
+  The resulting architectures found in the appendix are relatively simple, and scaling is easy for certain components
summary: This paper proposes a NAS-like approach to assembling a hodge-podge of LLM inference-time techniques for a target test benchmark. The resulting architecture, called ARCHON, is able to improve by scaling the number of LLM samples/models used as a subunit. Performance is evaluated in a number of benchmarks. The paper overall studies "meta architectures" where specialized LLMs are the unit of computing. Search is done with a classic Bayes optimization algorithm with a Gaussian process.
weaknesses: + The proposed approach is not quite comparable to the baselines in terms of compute. While ARCHON uses 30+ inference calls, other LLM systems are given just one. It would be fairer to uses any single building block but scaled up (if possible) to having approximately the same inference cost, to demonstrate that the combination is truly driving the performance gains as opposed to the # of inference calls. If performance was better even when giving the baselines similar compute, this could help dampen the briefly discussed limitation of additional inference calls in the conclusion. 
+ Most of the LLM components in section 3.1 do not cite previous work (only the generator does). It would be good to describe what these modules are based on, especially since this paper's contribution is centered around the combination and not the proposal of new parts. 
+ No standard errors reported, and other details are light/missing (i.e. the specific search algorithm)

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: Latency vs. Performance Trade-Off: Given the confirmed high latency and compute cost, can ARCHON be adapted for real-time applications without sacrificing significant performance? If not, how do you foresee its practical deployment in latency-sensitive environments?
Task Generalization: While ARCHON performs well on a variety of benchmarks, there seems to be variability in how effective certain components are across different tasks. Have you considered any techniques for improving generalization, such as meta-learning approaches or task embeddings, to make component selection more robust across diverse problem sets?
Dynamic Adaptation: Your methodology for component selection heavily relies on empirical testing and pre-filtering configurations. Have you considered implementing a more adaptive or dynamic strategy that could optimize component selection in real-time based on task-specific features? If not, what are the main challenges or limitations you foresee in developing such a strategy?
soundness: 3
strengths: ## Comprehensive Framework Design: 
The ARCHON framework introduces a detailed and modular architecture that provides a systematic way to apply and optimize inference-time techniques for large language models. This innovation addresses a critical gap in current methodologies, which often focus on singular techniques. ARCHON's approach is especially compelling in its structured, neural-network-inspired layering of methods, allowing for both parallel and sequential inference-time operations. The paper meticulously defines each component, such as Generators, Fusers, Rankers, Critics, and Unit Test Generators, and explains how they interact within the framework to optimize results.
## Substantial Performance Improvements: 
The quantitative results are striking. ARCHON achieves an average increase of 15.1 percentage points in accuracy across benchmarks, outperforming models like GPT-4 and Claude 3.5 Sonnet. This improvement is not superficial but demonstrated over a diverse set of benchmarks, including instruction-following (e.g., MT-Bench, AlpacaEval 2.0), reasoning (e.g., MixEval, MixEval-Hard), and coding tasks (e.g., CodeContests). The use of detailed performance metrics, including Pass@1 for coding and win rates for instruction-following, adds credibility to these claims.
## Detailed Analysis of Technique Interactions: 
The paper rigorously examines how different inference-time techniques complement each other. For example, the Ranker is shown to improve response quality by 5.9 percentage points when paired with the Critic, and adding multiple layers of Fusers increases the richness of responses in instruction-following tasks. These insights demonstrate a nuanced understanding of the interplay between techniques, which is often lacking in similar studies.
## Effective Use of Hyperparameter Search: 
The use of Bayesian Optimization for Inference-Time Architecture Search (ITAS) is a strong point, enabling efficient exploration of a vast design space. The paper details how ITAS outperforms random and greedy search, finding optimal configurations in 96.0% of iterations while reducing the number of evaluations by over 88%. This shows a well-optimized and strategic approach to architecture search.
## Extensibility and Practical Utility: 
ARCHON is designed as a plug-and-play framework, allowing practitioners to easily incorporate new LLMs and inference-time techniques. This adaptability makes it highly valuable for real-world applications, from academic research to industry use cases. The authors emphasize its potential for wide adoption by providing detailed guidelines on how to customize the framework for different tasks and compute budgets.
summary: ARCHON is a modular framework that addresses the challenge of designing efficient and effective inference-time architectures for large language models (LLMs). The framework focuses on optimizing various inference-time techniques, such as ensembling, ranking, fusion, critiquing, verification, and unit testing, to improve LLM performance on tasks like instruction-following, reasoning, and coding. ARCHON employs a hyperparameter optimization approach called Inference-Time Architecture Search (ITAS) using Bayesian optimization to explore the vast design space and select the most effective configuration. The paper demonstrates that ARCHON outperforms existing state-of-the-art models and inference-time architectures, achieving significant performance gains across multiple benchmarks.
weaknesses: The authors themselves acknowledge these components-Verifier, for example-operate underperforming on simpler tasks, there is no clear mechanism or threshold available to disable or skip them. In their failure to do so, a weakness in the design of the framework is exposed: it could result in unnecessary computational overhead and inefficiency. Without a systematic way of assessing when the contribution of a particular component is negligible, the work in ARCHON far from completes. Again, the work has not suggested mechanisms of adaptive disabling or performance-based activation of components that would go a long way in enhancing the efficiency of the framework.

The high variability that ARCHON exhibits for different tasks suggests a major shortcoming in the way it was designed. Despite such broad generalization setup, the proposed framework generalizes poorly without significant adjustments, and therefore it is not robust. The authors lost the opportunity to embed, at least discuss how methods might enhance generalization, like meta-learning approaches or task embeddings. Because of its heavy reliance on manual tuning, the framework is far from real-world applications where task conditions cannot be predicted. The fact that this is a limitation greatly reduces the appeal and practicality of ARCHON, especially in service to those users who need to find a universally applicable solution.

This dependence of the ARCHON framework on empirical testing and the configuration of pre-filtering points to one very critical limitation-it cannot adaptively change the selection of components based on real-time performance feedback or task-specific features. This rigidity suggests inefficiency because, at any given time, ARCHON may never be optimal on every task without extensive trial-and-error tuning. For instance, any more adaptive strategy, such as reinforcement learning or meta-learning, could be investigated in order to let the system automatically adapt its configuration. Why such a strategy has not been considered is not discussed by the authors, together with the motivation of the mentioned feasibility challenges, thus leaving a wide gap in the discussion of scalability and adaptability of the paper.

Meta-review: This paper combines different LLMs and inference-time techniques to obtain better performance than with one of them alone. The problem of selecting the right combination of LLM and inference-time techniques is viewed as a hyperparameter optimization problem and addresses it with an off-the-shelf Bayesian optimization method. Reviewers highlighted various strengths, including strong empirical results, often exceeding SOTA methods, and a detailed analysis of the interplay between different methods. Criticisms included the low novelty and high compute time, combined with the lack of meta-learning. The most serious concern for me for the decision to reject was that of overselling and overblown contributions by Reviewer YLS1. While I decided for rejection following most reviewers, I do see a lot of potential in this work and encourage the authors to continue this line of work and submit to the next venue.

Decision: Reject