Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling

## Abstract

Abstract Large language models (LLMs) exhibit varying strengths and weaknesses across different tasks, prompting recent studies to explore the benefits of ensembling models to leverage their complementary advantages. However, existing LLM ensembling methods often overlook model compatibility and struggle with inefficient alignment of probabilities across the entire vocabulary. In this study, we empirically investigate the factors influencing ensemble performance, identifying model performance, vocabulary size, and response style as key determinants, revealing that compatibility among models is essential for effective ensembling. This analysis leads to the development of a simple yet effective model selection strategy that identifies compatible models. Additionally, we introduce the Uni on T op- k 𝑘 k italic_k E nsembling ( UniTE ), a novel approach that efficiently combines models by focusing on the union of the top-k tokens from each model, thereby avoiding the need for full vocabulary alignment and reducing computational overhead. Extensive evaluations across multiple benchmarks demonstrate that UniTE significantly enhances performance compared to existing methods, offering a more efficient framework for LLM ensembling.

## 1 Introduction

Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and have shown promising results in real-world applications (OpenAI, ; Yang et al., ; Dubey et al., ). Given the diversity in data sources, model architectures, and training methods, LLMs exhibit varying strengths and weaknesses depending on the task at hand. Consequently, rather than relying solely on training an LLM from scratch, an alternative approach is to create an ensemble of LLMs. This method allows for leveraging the complementary advantages of different LLMs (Jiang et al., ; Lu et al., ; Yu et al., ).

Existing model ensembling methods can be broadly categorized into three types: output-level, probability-level, and training-level approaches. Output-level methods (Jiang et al., ; Lu et al., ; Shnitzer et al., ) aggregate the complete outputs of multiple candidate models. Probability-level methods (Huang et al., ; Yu et al., ), integrate outputs based on probability distributions at each generation step through the intersection or union of the vocabulary. Training-level methods (Wan et al., ; Xu et al., ) utilize output probability vectors as labels for richer information extraction during training. While output-level methods are constrained by the limitations of existing outputs, and training-level methods introduce additional computational overhead, probability-level methods have garnered particular attention.

Existing methods for ensembling LLMs, such as DeePen (Huang et al., ) and GaC (Yu et al., ), grapple with two significant challenges. First, these approaches concentrate solely on the ensembling technique, sidestepping the crucial discussion of which types of models can be effectively combined. This oversight is critical because LLMs with substantial differences in architecture, size, and tokenizer may be inherently incompatible, leading to potential incompatibilities that undermine the benefits of ensembling(Dong et al., ; Lee et al., ). Second, these methods tend to align the probabilities across the entire vocabulary at each generation step. Such a strategy introduces substantial computational overhead during inference, which hinders performance and efficiency.

To address these challenges, we conduct a thorough investigation into the factors that truly influence ensembling performance. Our empirical analysis identifies three key factors: model performance, vocabulary size, and response process. We evaluate multiple widely recognized LLMs across various benchmarks to explore ensembling from these perspectives. Our findings reveal several important insights: 1) Variations in performance levels among base LLMs significantly affect their compatibility for ensembling; 2) The impact of vocabulary size is marginal; and 3) Even when performance and vocabulary size are aligned across LLMs, substantial differences in the reasoning process in the response can hinder successful ensembling.

Building on this analysis, we adopt a determine-then-ensemble strategy by starting with the best-performing LLM for target tasks, then iteratively selecting the next best-performing LLMs that meets ensembling criteria until the maximum number of models is reached or no further suitable candidates exist. Additionally, we seek to improve the efficiency and performance of ensembling. Instead of aligning the entire vocabulary of different LLMs, we propose the Union Top-$k$ Ensembling (UniTE). UniTE constructs a union of the top-k tokens from each model and expands this set using each model’s tokenizer. This is followed by probability aggregation to determine the next token. UniTE avoids the need for auxiliary mapping matrices and full vocabulary alignment, respecting the unique tokenization of each base LLM. Extensive experiments demonstrate the effectiveness of UniTE across various tasks, consistently outperforming state-of-the-art LLM ensemble methods.

Overall, our key contributions are: 1) We conduct an extensive analysis of existing model ensembling approaches and derive three significant insights. Building on these findings, we introduce a general model selection strategy for model ensembling;
2) We propose a well-motivated model ensembling method that efficiently operates on the top-k candidate tokens rather than the entire vocabulary.
3) Experiments demonstrate significant performance improvements, reduced operational tokens to less than 0.04% of current methods, and minimized latency to only 10 ms longer than the individual model, validating the effectiveness of our proposed approach.

## 2 Related Works

Based on the sequence of inference and fusion processes, multiple model collaboration methods can be broadly classified into two categories: model ensembling (Jiang et al., ; Yu et al., ) and model merging (Yu et al., ; Akiba et al., ). Model ensembling follows an inference-then-fusion approach, aiming to integrate the outputs of various models to achieve a more refined response. Conversely, model merging adopts a fusion-then-inference strategy, wherein different models are combined into a single model before inference. While model merging is typically applicable only to homologous models, our focus in this study is on the more general approach, namely model ensembling. We discuss different model ensembling methods as follows:

Output-Level Model Ensembling methods involve selecting multiple candidate models and utilizing their complete outputs for aggregation. For instance, Jiang et al. () developed PairRanker, an additional ranking model, to evaluate and select the best candidate output. Similarly, Lu et al. () and Shnitzer et al. () designed a router that determines the most appropriate candidate model based on the given question. However, these approaches are restricted by the existing outputs and become ineffective if all options are incorrect. Some studies have addressed this by training fusion models to combine outputs (Jiang et al., ; Wang et al., ), which alleviates the limitation of relying solely on available candidates and often yields improved results. Nevertheless, achieving generalization with fusion models remains a significant challenge, as they may not fully utilize the probability information generated at each step.

Probability-Level Model Ensembling methods focus on integrating outputs from different models by utilizing the probability distribution at each generation step. Cool-Fusion (Liu et al., ) let each source LLM generate tokens until reaching common word boundaries, then jointly reranks the segments.
However, this method relies on common word boundaries, limiting generation flexibility and diversity. Additionally, managing multiple source models can increase complexity. DeePen (Huang et al., ), based on the principles of relative representation theory, transforms each model’s probability distribution from its individual space into a common relative space for aggregation, which is determined through the intersection of the models’ vocabularies. On the other hand, GaC (Yu et al., ) constructs a mapping matrix that projects the probability vectors of multiple LLMs into the dimensional space of their union vocabulary, aggregating outputs to select the next token at each generation step. It is worth noting that all of these existing approaches operate on the entire vocabulary at every single step, resulting in significant computational overhead. In this work, we focus on the probability-level model ensembling without the need for additional training.

Training-Level Model Ensembling methods exemplified by FuseLLM (Wan et al., ), leverage output probability vectors from various models during the training process, using these vectors as labels instead of one-hot representations. This technique serves as a unique form of distillation, enabling the trained model to extract richer information from the probability outputs of the ensemble of teacher models. FuseCHAT (Wan et al., ) extends the framework of FuseLLM to fuse multiple chat LLMs with diverse architectures and scales. However, distillation is predominantly aimed at enhancing smaller models, which limits its effectiveness in further advancing larger models. On the other hand, EVA (Xu et al., ) addresses vocabulary discrepancies by learning token alignment across different vocabularies, utilizing overlapping tokens for assistance. However, vocabulary alignment should involve both output embedding alignment and weight alignment, EVA achieves vocabulary alignment solely by solving the mapping matrix of output embeddings.

Figure: Figure 1: The impact of performance disparity among models on ensemble effectiveness across different datasets and methods is examined. We compare these methods to the individual performances of LLaMA2 and Mistral, indicated by dashed lines.
Refer to caption: extracted/5895286/Figs/performance_datasets.png

## 3 Understanding Model Ensembling from Model Capacity, Vocabulary Size and Task

Existing model ensembling approaches (Huang et al., ; Yu et al., ) only focus on designing the model ensembling method, offering limited discussion on the selection of base models for ensembling.
However, insights from prior research (Dong et al., ; Lee et al., ), indicate that not all model pairs are compatible to be combined, particularly when there are significant differences in size, vocabulary, and performance.
In our preliminary experiments, we explored various factors that might affect the performance of model ensembling, including model size (e.g., 3B/7B/8B/13B/70B), model architecture (dense/sparse), performance discrepancies, tokenization strategies (BPE/WordPiece), vocabulary size (e.g., 102K/64K/32K) and tasks variations (e.g., text generation/QA/multiple choices).
Finally, we identified three representative factors for further analyses, including performance discrepancy, vocabulary size, and tasks variations.

### 3.1 Impact of Model Performance Discrepancy

To investigate the impact of performance disparity on model ensembling, we select LLaMA2-13B-Chat and Mistral-7B-Instruct-v0.3 as base models, which show a significant performance gap across various tasks. We evaluated the GSM8K (Cobbe et al., ), PIQA (Bisk et al., ), ARC-C (Clark et al., ), and NQ (Kwiatkowski et al., ) datasets with three comparative methods.

As shown in Fig. 1, we found that when there is a significant performance disparity between models, LLM-Blender, which selects the optimal response from model candidates, is not suitable for model ensembling.
Similarly, for methods like DeePen and GaC, which are achieved through token probability averaging, a substantial performance gap also leads to certain performance drops. This observation is straightforward and easily understood, as lower-performing models may introduce noise and bias, adversely affecting overall effectiveness.
Nonetheless, it has been observed that DeePen and GaC consistently enhance the performance of less effective models when combined with a superior model. Although the resultant performance does not surpass that of the superior model, these methods can also serve as an alternative way for training-free knowledge distillation.

To further figure out how performance discrepancy impacts the ensembling, we then identify model pairs based on the GSM8K dataset with performance gaps of approximately 40% (LLaMA2-7B-Chat and Mistral-7B-Instruct-v0.3), 25% (LLaMA2-13b-Chat and Mistral-7B-Instruct-v0.3), 15% (OpenChat-3.5 and Mistral-7B-Instruct-v0.3), and less than 10% (LLaMA3-8B-Instruct and Qwen2-7B-Instruct).
Fig. 2 shows that as the performance gap increases, the improvement from ensembling inferior models becomes more pronounced. In contrast, for superior models, the ensembling effect consistently falls below the baseline performance. When the performance difference is within 10%, ensembling may yield better results.

Figure: Figure 2: The impact of performance differences on model ensembling effectiveness on GSM8K dataset. OOM represents out of memory issue.
Refer to caption: extracted/5895286/Figs/performance_disparity.png

Moreover, it was observed that with the increase in vocabulary size, as exemplified by models such as LLaMA3 and Qwen2, whose vocabulary size exceeds 120K, approaches like DeePen, which depend on high-dimensional matrix operations, encounter heavy memory limitations that diminish their effectiveness.

Takeaway I: Smaller performance gaps lead to greater gains from model ensembling.

**Table 1: The influence of vocabulary size on model ensembling effectiveness**
| Methods | Dataset |  |
| --- | --- | --- |
| GSM | PIQA |  |
| DeepSeek-7B(102K) | 59.67 | 72.66 |
| Mistral-7B(32K) | 56.48 | 80.63 |
| LLM-Blender | 61.16 | 76.54 |
| (+1.49) | (-4.09) |  |
| DeePen(DeepSeek-7B) | 55.00 | 77.65 |
| (-4.67) | (+4.99) |  |
| DeePen(Mistral-7B) | 61.28 | 76.32 |
| (+4.80) | (-4.31) |  |
| \cdashline1-3<br>Ours(DeepSeek-7B) | 62.77 | 81.81 |
| (+3.10) | (+9.15) |  |
| Ours(Mistral-7B) | 58.88 | 81.81 |
| (+2.40) | (+1.18) |  |

### 3.2 Influence of Vocabulary Size

Yu et al. () found that large models employing different tokenization methods, such as BPE and BBPE, exhibit over 90% overlap in tokenizing Oxford 5000 common words. This indicates that tokenization methods have a marginal impact on model ensembling, leading us to focus on the effect of vocabulary size on ensemble performance.
To this end, we selected four models with varying vocabulary sizes that exhibit similar performance on certain datasets, namely LLaMA2-13B-Chat (vocabulary size: 32,000), Mistral-7B-Instruct-v0.3 (vocabulary size: 32,768), Yi-6B (vocabulary size: 64,000), and DeepSeek-LLM-7B-Chat (vocabulary size: 102,400).

Figure: Figure 3: Token distribution of different models. We extract the top 15 words from the vocabulary of each model and apply a softmax processing to their corresponding logits.
Refer to caption: extracted/5895286/Figs/token_distribution.png

As presented in Table 1, existing methods demonstrate consistent performance irrespective of the gap in vocabulary size, thereby indicating that vocabulary size does not significantly influence the efficacy of model ensembling.
To elucidate the underlying causes, we conducted a manual analysis of 100 responses generated by various LLMs. Our investigation revealed that approximately 80% of the words in the responses were common words while the tokenization of these common words exhibited no discernible differences across the different LLMs. Therefore, the vocabulary size, as well as the tokenization strategy, exerts minimal influence on the performance of model ensembling.
Notably, despite significant variations in vocabulary size, the ensembling results of models with similar performance suggest that when the performance gap between base models is within 10%, model ensembling may achieve better results than the baseline model, which is aligned with findings in Section 3.1.
More interestingly, results of evaluating DeepPen on the ARC-C dataset indicate that when model performances are comparable, ensembling inferior models can even outperform superior ones. This might imply that when performance differences are minimal, the selection of main base models does not necessarily need to prioritize those with superior performance.
Besides, it is important to note that these findings related to the vocabulary and tokenization consistently apply to LLM-Blender, DeePen and our method, with the exception of GAC.
We will detail the reason in Section 4.2.

Takeaway II: The influence of vocabulary size for model ensembling is marginal.

**Table 2: Responses of different methods on NQ dataset.**
| Question | who wrote he ain’t heavy he’s my brother lyrics |
| --- | --- |
| who was the ruler of england in 1616 |  |
| Gold answer | Bob Russell |
| James I |  |
| LLaMA3 response | Bob Russell \usym2713 |
| James I \usym2713 |  |
| Qwen2 response | He Ain’t Heavy, He’s My Brother was written by Mike D’Abo. \usym2717<br>The song was originally recorded by the British singer-songwriter and released as a single in 1969.<br>It became a hit and has since been covered by numerous artists.<br>$\cdots$ |
| \cdashline2-2 | The ruler of England in 1616 was King James I. \usym2713<br>He was the first monarch of the Stuart dynasty and reigned from March 24, 1603, until his death on July 19, 1625.<br>$\cdots$ |
| LLM-Blender response | He Ain’t Heavy, He’s My Brother was written by Mike D’Abo. \usym2717<br>The song was originally recorded by the British singer-songwriter and released as a single in 1969.<br>It became a hit and has since been covered by numerous artists.<br>$\cdots$ |
| \cdashline2-2 | The ruler of England in 1616 was King James I. \usym2713<br>He was the first monarch of the Stuart dynasty and reigned from March 24, 1603, until his death on July 19, 1625.<br>$\cdots$ |

### 3.3 Task-Specific Challenges in Model Ensembling

In the aforementioned experiments, we identified an interesting phenomenon:
Even when performance and vocabulary size are aligned across models, substantial differences in the response style can also hinder successful ensembling on specific tasks.
For example, LLaMA3-8B and Qwen2-7B exhibit comparable performance across various tasks. Existing ensembling approaches such as LLM-Blender have been shown to enhance outcomes on GSM8K and PIQA tasks when utilizing these two base models. However, ensembling these models to address the questions in TriviaQA and NQ datasets proves to be impractical due to the significant discrepancies in their response styles.

As shown in Table 2, under identical prompting conditions, LLaMA3 tends to directly return the answers, whereas Qwen2 is inclined to conduct analysis. In voting-based LLM-Blender, response length significantly influences judgments, with PairRanker consistently favoring longer responses. For instance, in our analysis of 100 randomly selected NQ and TriviaQA responses, the PairRanker favored the longer response as the answer in approximately 80% and 70% of cases accordingly, possibly due to its training data characteristics. More detailed analysis of TriviaQA and more experimental results are in appendix A. Similarly, probability-based ensembling methods like DeePen and GaC are also affected by the differences in the reasoning process, leading to significant deviations in vocabulary probability distributions. Figure 3 reveals a significant disparity in top token distributions between LLaMA3 and Qwen2 for the first two tokens. Averaging the probabilities directly could result in an excessively smoothed distribution, which is inappropriate.
The underlying reason for this phenomenon may be the differences in LLMs’ training data. Therefore, caution is warranted in specific tasks to avoid substantial disparities, despite effective ensembling in most other tasks.

Takeaway III: Even though the performance and vocabulary size are aligned across models, substantial differences in response style could also hinder successful ensembling.

## 4 Methodology

### 4.1 Model selection strategy

Given the aforementioned insights, we present a strategy to assess the compatibility of two models for ensembling and offer guidance for selecting base models from the candidate model pool.

When ensembling two models, we begin by comparing their performance on the target tasks. An overall performance gap within 10% is desirable, as it indicates the potential compatibility for successful ensembling.
If the performance gap falls within this range, it is crucial to compare the response styles of different models. This comparison is relatively subjective, with differences often identifiable through superficial features such as text length and BLEU score.

In this work, we use text length as a guiding metric by constraining the length of longer responses to not exceed twice the length of the shorter ones. When the response manner is also consistent across models, it is highly probable that these two models can be successfully ensembled to produce a superior response.
We exclude discrepancies in vocabulary size from consideration, as prior analyses have demonstrated its insignificance in affecting ensembling outcomes.

To select base models from the pool, we recommend initially choosing the best-performing model for the target tasks. Subsequently, select the next best-performing model that satisfies the criteria for successful ensembling with the first chosen model, continuing this process iteratively until the maximum number of base models is reached or no further suitable models can be found.

### 4.2 Union top- k 𝑘 k italic_k ensembling

Apart from the model selection strategy, we also endeavor to improve the efficiency and performance of model ensembling. Existing probability-level model ensembling methods, e.g. DeePen and GaC, attempt to combine the models through full vocabulary alignment. We argue this approach is sub-optimal since the candidate next token typically resides within the top-k tokens. Conversely, incorporating tokens of lower probability may introduce unnecessary noise, thereby diminishing the overall accuracy and effectiveness of the ensemble. Motivated by this, we propose the Union Top-k Ensembling (UniTE), a novel and efficient ensembling approach that only aligns the top-k tokens in each decoding step.

Given the base models for ensembling, denoted as LLMi with their respective tokenizer $T^{i}$ and vocabulary $V^{i}$, we feed the input text prompt into LLMi and obtain the probability distribution.
Instead of aligning the probability distributions across different models, we only focus on the subset of top-$k$ tokens since the candidate next token is highly likely to be within the subset. The selected top-$k$ tokens of LLMi is presented as TopKi = ($t^{i}_{1}$, $t^{i}_{2}$, …, $t^{i}_{k}$) with corresponding probability distribution $P^{i}$ = ($p^{i}_{1}$, $p^{i}_{2}$, …, $p^{i}_{k}$).

Figure: Algorithm 1 Union top-$k$ ensembling

Next, we aim to construct the union set $V^{u}$ of TopKi across the base models. The most straightforward solution to obtain the union set is to directly remove the duplicate tokens and retain all other distinct tokens. For the token in $V^{u}$ but not in $V^{i}$, the probability of this token in LLMi is assigned to 0. This strategy is adopted by GAC (Yu et al., ). However, we contend that this strategy is not entirely reasonable since it is heavily influenced by the tokenizer and vocabulary.
For example, the candidate token “James” is within the top-$k$ subset of LLM1 but not in $V^{2}$ while the token “Jam” appears in TopK2. The word “James” can be tokenized into “Jam” and “es” by LLM2. If the token probabilities $p^{1}_{\{Jam\}}$ and $p^{2}_{\{James\}}$ are set to 0, the overall probability of generating the word “James” should be inherently reduced, even if both base models exhibit a preference for this word. Therefore, we present a new token probability alignment strategy on the union set. Firstly, we obtain the top-$k$ union vocabulary $V^{u}$ by directly merging the TopKi subsets. Then, the token distribution $P^{i}$ and TopKi are updated according to following criteria:

- 1.
If the token $w$ appears in both $V^{u}$ and TopKi, $P^{i}_{\{w\}}$ and TopKi remains unchanged.
- 2.
If the token $w$ in $V^{u}$ is absent in TopKi but present in $V^{i}$, it will be appended to TopKi. $P^{i}$ also updates accordingly.
- 3.
If the token $w$ in $V^{u}$ does not exist in $V^{i}$, it should be tokenized by $T^{i}$. The first token of the result along with its token probability is then updated to TopKi and $P^{i}$.

Up to now, we can obtain the aligned top tokens Top$\hat{K}$i and the token distributions $\hat{P}^{i}$ of based models. Consequently, we normalize $\hat{P}^{i}$ as: $\hat{P}^{i}_{norm}=\operatorname{softmax}(\hat{P}^{i})$.

Since our method eliminates the need for full vocabulary alignment, it is essential to designate one model as the primary base model. As discussed in Section 3.2, the selection of primary base model is flexible when the candidate models exhibit comparable performance. To simplify the process, we directly employ the best-performing model as the primary base model. Then, the primary base model employs the average token probability $\hat{P}_{avg}$ to predict the next token, where $\hat{P}_{avg}=\frac{1}{n}\sum_{i=1}^{n}\hat{P}^{i}_{norm}$.

The next token is determined using the maximization-based greedy strategy (Li et al., ). The chosen token will be appended to the input text. This process is iteratively repeated until the predetermined stopping criterion is met, such as generating an end-of-sentence token or reaching the maximum length.

## 5 Experiments

### 5.1 Setup

#### Models

All of our experiments are conducted with the following commonly used models, including LLaMA2-7B-Chat (Touvron et al., ), LLaMA2-13B-Chat (Touvron et al., ), LLaMA3-8B-Instruct (Dubey et al., ), LLaMA3.1-8B-Instruct (Dubey et al., ), Mistral-7B-Instruct-v0.3 (Jiang et al., ), DeepSeek-LLM-7B-Chat (Bi et al., ), Yi-6B (Young et al., ), OpenChat-3.5 (Wang et al., ), Qwen2-7B-Instruct (Yang et al., ), Mixtral-8×7B (Jiang et al., ) and Qwen1.5-72B (Bai et al., ).

#### Baselines

We selected three typical model ensembling methods for further analyses.
1) LLM-Blender (Jiang et al., ) includes a reward model, PairRanker, to evaluate LLM responses, and a fusion model, GenFuser, to combine them. We only use PairRanker due to significant over-generation issues with GenFuser.
2) DeePen (Huang et al., ) employs relative representation theory to map each model’s probability distribution to a universal space for aggregation, computed through the intersection of model vocabularies.
3) GaC (Yu et al., ) projects multiple LLMs’ probability vectors into a unified vocabulary dimension using a mapping matrix and aggregates outputs at each generation step to select the next token. Although GaC suggests the importance of identifying keywords to improve latency, we still exclude it owing to its hindrance on the ensembling performance.

**Table 3: Results of ensembling on Mistral, DeepSeek, OpenChat. OpenChat is chosen as primary model for these experiments.**
| Method | Dataset | Avg. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GSM8K | PIQA | MMLU | ARC-C | TriviaQA | NQ |  |  |
| Mistral | 56.48 | 80.63 | 59.28 | 74.49 | 64.30 | 24.25 | 59.91 |
| DeepSeek | 59.67 | 72.66 | 46.97 | 58.73 | 47.63 | 11.37 | 49.51 |
| OpenChat | 73.46 | 87.10 | 60.80 | 78.05 | 61.77 | 31.08 | 65.38 |
| LLM-Blender | 70.79(-2.67) | 83.28(-3.82) | 60.10(-0.70) | 76.29(-1.76) | 56.35(-5.42) | 25.57(-5.51) | 62.06(-3.32) |
| DeePen | 73.06(-0.40) | 76.04(-11.06) | 61.91(+1.11) | 72.14(-5.91) | 67.24(+5.47) | 28.26(-2.82) | 63.11(-2.27) |
| GaC | 62.85(-10.61) | 67.85(-19.25) | 55.15(-5.65) | 73.04(-5.01) | 62.22(+0.45) | 20.72(-10.36) | 56.97(-8.41) |
| UniTE | 73.31(-0.15) | 87.50(+0.40) | 62.13(+1.33) | 78.70(+0.65) | 65.80(+4.03) | 31.78(+0.70) | 66.54(+1.16) |

**Table 4: Results of ensembling on LLaMA3, LLaMA3.1, Qwen2. Qwen2 is chosen as primary model for these experiments.**
| Method | Dataset | Avg. |  |  |  |
| --- | --- | --- | --- | --- | --- |
| GSM8K | PIQA | ARC-C | MMLU |  |  |
| LLaMA3 | 78.77 | 79.08 | 79.01 | 64.58 | 75.36 |
| LLaMA3.1 | 80.83 | 82.86 | 79.49 | 66.69 | 77.47 |
| Qwen2 | 80.78 | 84.57 | 84.92 | 64.96 | 78.81 |
| Two-model ensembling (LLaMA3+Qwen2) |  |  |  |  |  |
| LLM-Blender | 82.69 (+1.91) | 82.53 (-2.04) | 82.98 (-1.94) | 62.07 (-2.89) | 77.57 (-1.24) |
| DeePen | - OOM - |  |  |  |  |
| GaC | 80.67 (-0.11) | 80.96 (-3.61) | 84.93 (+0.01) | 67.05 (+2.09) | 78.40 (-0.41) |
| UniTE | 84.17(+3.39) | 85.53(+0.96) | 85.07(+0.15) | 69.78(+4.82) | 81.14(+2.33) |
| Three-model ensembling |  |  |  |  |  |
| LLM-Blender | 83.30(+2.52) | 83.47(-1.10) | 83.48(-1.44) | 62.55(-2.41) | 78.20(-0.61) |
| DeePen | - OOM - |  |  |  |  |
| UniTE | 84.99(+4.21) | 84.98(+0.41) | 85.39(+0.47) | 69.12(+4.16) | 81.12(+2.31) |

#### Benchmarks

We evaluate six benchmarks, which can be categorized into three main groups. 1) Comprehensive Examination: MMLU (5-shot) (Hendrycks et al., ), covering 57 subjects that humans typically learn; ARC-C (0-shot) (Clark et al., ), collected from standardized natural science tests. 2) Reasoning Capabilities: GSM8K (Cobbe et al., ) (4-shot), a dataset of high-quality problems at the grade school math level; PIQA (Bisk et al., ) (0-shot), a commonsense reasoning dataset. 3) Knowledge Capacities: TriviaQA (5-shot) (Joshi et al., ), compiled by Trivia enthusiasts; NaturalQuestion (NQ) (5-shot) (Kwiatkowski et al., ), a question-answering corpus consisting of queries issued to the Google search engine.

For a comprehensive evaluation, we conduct two- and multi-model ensembling experiments on the aforementioned benchmarks. We select base models following the strategy presented in Section 4.1. We choose two pairs from Mistral, Deepseek, and OpenChat for the two-model experiments. For the closely performing LLaMA3, LLaMA3.1, and Qwen2, we select LLaMA3 and Qwen2 for the two-model ensemble based on their release dates and subsequently employ all three models for the multi-model ensembling evaluations. The hyper-parameter $k$ is set to 10 in this work. All experiments are conducted on 46G NVIDIA L40 GPUs.

### 5.2 Main Results

As shown in Table 3 and Table 4, we have following observations:

(1) UniTE enhances individual model performance when the base models exhibit similar performance levels. As demonstrated, the ensemble of models such as OpenChat achieved an average improvement of approximately 1.2% across five benchmark tasks, including ARC-C and MMLU. However, in the GSM8K task, a 15% performance gap between DeepSeek and OpenChat led to a slight decline in overall performance, reinforcing our observation in Section 3.1 that ensemble methods are most effective when model performances are closely aligned. Furthermore, experiments with LLaMA3 and Qwen2 validated our approach, resulting in performance increases of 3.39% and 4.82% on the GSM8K and MMLU tasks, respectively.

(2) UniTE demonstrates greater robustness and generality. Ensembling experiments with LLaMA3 and Qwen2 indicate that while LLM-Blender improves performance on the GSM8K dataset, it significantly underperforms compared to baseline models in the PIQA, ARC-C, and MMLU benchmarks. Specifically, Blender demonstrates a decline of 2.89% in the MMLU task, and this issue is exacerbated in ensembles with the baseline models OpenChat and Mistral, which exhibit relatively low baseline performance, leading to an average performance drop of 3.32%. Additionally, GaC fails to improve the performance of individual models across most tasks and exhibits a significant decline of over 10% when combined with Mistral and OpenChat. Furthermore, due to its intersection limitations, DeePen is ill-suited for ensembling with the LLaMA3 model, which requires handling a large vocabulary. In contrast, across tasks with comparable performance levels, UniTE achieves the highest improvements and overall performance across the board, with the exception of a slight under-performance relative to DeePen on TriviaQA. This outcome underscores the effectiveness and robustness of UniTE.

(3) Collaborating with comparable LLMs does not necessarily yield better results. The ensembling experiments conducted with three models, following the integration of LLaMA3.1, demonstrate improved performance on the GSM8K and ARC-C benchmarks compared to the ensemble of LLaMA3 and Qwen2. However, this approach results in suboptimal outcomes on PIQA and MMLU. This observation is justified, as the analysis in Section 3.2 indicates that while combining models with similar performance may enhance overall efficacy, such improvement is not guaranteed.

Figure: Figure 4: Impact of the choice of hyperparameter $k$.
Refer to caption: extracted/5895286/Figs/impact_of_k.png

### 5.3 Ablation study

#### Effect of hyperparameter k 𝑘 k italic_k selection

To investigate the impact of the hyperparameter $k$, we conducted experiments using the Mistral and OpenChat models on the TriviaQA and ARC-C datasets. As illustrated in Fig. 5, increasing $k$ from 5 to 10 enhances ensemble performance. However, further increasing $k$ beyond 10 leads to either a slight decline or no change in performance. This finding supports our assertion that, in probability-level ensembling, aligning the entire vocabulary is unnecessary for predicting the next token.

#### Effect of the next token selection strategy

We also explored the impact of deterministic decoding and top-$k$ sampling (Fan et al., ) on the next token emission. It is important to clarify that our UniTE focuses on the fact that, when predicting the next token, we only need to ensemble a subset of tokens rather than the entire vocabulary. In contrast, greedy decoding and top-$k$ sampling emphasize how to determine the next token’s ID after the ensemble process is complete. To investigate this, we conducted experiments using LLaMA3.1 and Mistral on the PIQA and ARC-C datasets correspondingly. As shown in Figure 6, for these deterministic tasks, the maximization-based greedy method outperforms the random sampling approach, which aligns with intuition.

Figure: Figure 6: Comparison among different decoding methods.
Refer to caption: extracted/5895286/Figs/ablation.png

### 5.4 Further Analysis

#### Latency analysis

Following the settings detailed in GaC (Yu et al., ), we recorded the latency (ms/token) of different methods. The data presented in Fig. 5 reveals significant differences in latency among the various methods. Notably, the latency of UniTE is 87.78 ms/token, which is substantially lower than that of DeePen and GaC, and only about 10 ms longer than that of the individual models.

#### Tokens manipulated each step

Table 6 presents tokens manipulated at each step. Compared to DeePen and GaC, UniTE significantly reduces the number of tokens manipulated at each step, indicating its efficiency. As the vocabulary of the base model expands, the number of manipulated tokens of DeePen and GaC increases significantly, which not only increases the computational burden but may also lead to performance bottlenecks. Contrastly, UniTE is minimally affected, maintaining its effectiveness.

**Table 5: Tokens manipulated at each step. It’s noteworthy that DeePen encountered out-of-memory issues when conducting LLaMA3 and Qwen2 ensembling.**
| Base Model | Method | Tokens Manipulated Each Step |
| --- | --- | --- |
| Mistral(32678)<br>+<br>OpenChat(32002) | DeePen | 32000 |
| GaC | 32770 |  |
| UniTE | 14.46 |  |
| LLaMA3(128256)<br>+<br>Qwen2(151646) | DeePen | 109566 |
| GaC | 170336 |  |
| UniTE | 14.43 |  |

#### Ensemble of the dense model and the sparse model

We evaluate our method on the ensemble learning of the dense model and the sparse MoE model for reasoning and comprehensive examination tasks. Specifically, we utilize the widely-used large-scale dense model Qwen1.5-72B (Bai et al., ) alongside the popular sparse MoE model Mixtral-8×7B (Jiang et al., ) as our base models. As shown in Tabel 6, our UniTE achieves +4.86% and +7.29% performance on the ARC-C and PIQA datasets respectively, despite the base models already exhibiting high levels of performance.

## 6 Conclusion

In conclusion, our research highlights the effectiveness of ensemble methods in enhancing the performance of LLMs. By examining existing techniques, we identified key factors influencing ensembling success, such as model performance and response processes, while finding that vocabulary size has a minimal impact. In addition, the issue of vocabulary redundancy exposed by existing methods during ensembling lead us to propose UniTE, which efficiently aggregates a bunch of tokens from multiple LLMs without the computational overhead. Through extensive experimentation, UniTE consistently outperformed state-of-the-art ensemble methods, demonstrating its effectiveness in leveraging the strengths of diverse LLMs. Our contributions not only advance the understanding of model ensembling but also provide a practical framework for selecting and integrating LLMs to achieve superior performance.

## Appendix A Appendix

### A.1 Task challenges analysis on TriviaQA

Table 7 provides a detailed comparison of the differences in response styles between LLaMA3 and Qwen2 on the TriviaQA dataset. It is easily observable that, regardless of correctness, LLM-Blender consistently tends to choose longer responses as answers.

### A.2 Case Study of UniTE

Table 8 presents the specific cases of UniTE and other ensembling approaches, showcasing the validity and stability of our method.

**Table 7: Responses of different methods on TriviaQA dataset**
| Question | Who had a 70s No 1 hit with Kiss You All Over? |
| --- | --- |
| The flag of Libya is a plain rectangle of which color? |  |
| Gold answer | Sir Henry Campbell Bannerman |
| Green |  |
| LLaMA3 response | Henry Campbell-Bannerman \usym2713 |
| Green \usym2713 |  |
| Qwen2 Response | The hit song K̈iss You All Overẅas performed by the band Arrows, featuring vocalist Jim McCarty. \usym2717<br>The song reached number one on the UK Singles Chart in 1976.<br>The song was written by Arrows’ lead singer Jim McCarty, along with the band’s guitarist, Steve P̈igḦarrison.<br>It was produced by Tony Visconti, who had previously worked with David Bowie and T. Rex.<br>K̈iss You All Overẅas released as a single from the band’s debut album, Ärrows,ẅhich was also released in 1976.<br>The song’s catchy chorus and upbeat tempo made it a popular hit of the 1970s,<br>and it remains a well-known and beloved track from that era.<br>The song’s success helped to establish Arrows as a popular band of the time,<br>and it continues to be played on classic rock and oldies radio stations today. |
| \cdashline2-2 | The flag of Libya is a plain rectangle of green color. \usym2713<br>It was adopted in 1977 and features a green color field with no other symbols or designs.<br>The color green is significant in the country’s history and culture,<br>representing hope, fertility, and the country’s rich agricultural heritage.<br>The flag is rectangular in shape and is often flown vertically.<br>It is an important symbol of the Libyan state and is used in various official and ceremonial contexts. |
| LLM-Blender response | The hit song K̈iss You All Overẅas performed by the band Arrows, featuring vocalist Jim McCarty. \usym2717<br>The song reached number one on the UK Singles Chart in 1976.<br>The song was written by Arrows’ lead singer Jim McCarty, along with the band’s guitarist, Steve P̈igḦarrison.<br>It was produced by Tony Visconti, who had previously worked with David Bowie and T. Rex.<br>K̈iss You All Overẅas released as a single from the band’s debut album, Ärrows,ẅhich was also released in 1976.<br>The song’s catchy chorus and upbeat tempo made it a popular hit of the 1970s,<br>and it remains a well-known and beloved track from that era.<br>The song’s success helped to establish Arrows as a popular band of the time,<br>and it continues to be played on classic rock and oldies radio stations today. |
| \cdashline2-2 | The flag of Libya is a plain rectangle of green color. \usym2713<br>It was adopted in 1977 and features a green color field with no other symbols or designs.<br>The color green is significant in the country’s history and culture,<br>representing hope, fertility, and the country’s rich agricultural heritage.<br>The flag is rectangular in shape and is often flown vertically.<br>It is an important symbol of the Libyan state and is used in various official and ceremonial contexts. |

**Table 8: Case study of UniTE and other ensembling methods on GSM8K**
| Question | Tom plants 10 trees a year. Every year he also chops down 2 trees a year.<br>He starts with 50 trees. After 10 years 30% of the trees die.<br>How many trees does he have left? |  |
| --- | --- | --- |
| Gold Answer | He gets $10-2=<<10-2=8>>8$ new trees a year.<br>After 10 years he has $8*10=<<8*10=80>>80$ new trees<br>so he had $80+50=<<80+50=130>>130$ trees<br>He lost $130*.3=<<130*.3=39>>39$ trees<br>That means he had $130-39=<<130-39=91>>\hbox{\pagecolor{lightgreen}91}$ trees |  |
| LLaMA3 Response | Tom starts with 50 trees.<br>He plants 10 trees a year, so after 10 years he has 50+10*10= 150 trees.<br>However, he chops down 2 trees a year, so after 10 years he has 150-2*10= 140 trees.<br>30% of the trees die, which means 30/100*140= 42 trees die.<br>Tom is left with 140-42= 98 trees.<br>The answer is boxed{98} |  |
| LLaMA3.1 Response | Tom starts with 50 trees and plants 10 trees a year, so after 10 years he will have 50+10*10= 150 trees.<br>However, he also chops down 2 trees a year, so after 10 years he will have 150-2*10= 140 trees.<br>After 10 years, 30% of the trees die, so 30% of 140 is 0.3*140= 42 trees die.<br>The number of trees Tom has left is 140-42= 98 trees.<br>The answer is boxed{98} |  |
| Qwen2 Response | Tom starts with 50 trees and plants 10 trees a year, so after 10 years he will have 50+10*10= 150 trees.<br>However, he also chops down 2 trees a year, so after 10 years he will have 150-2*10= 140 trees.<br>After 10 years, 30% of the trees die, so 30% of 140 is 0.3*140= 42 trees die.<br>The number of trees Tom has left is 140-42= 98 trees.<br>The answer is boxed{98} |  |
| 2-models | LLM-Blender | Tom starts with 50 trees and plants 10 trees a year, so after 10 years he will have 50+10*10= 150 trees.<br>However, he also chops down 2 trees a year, so after 10 years he will have 150-2*10= 140 trees.<br>After 10 years, 30% of the trees die, so 30% of 140 is 0.3*140= 42 trees die.<br>The number of trees Tom has left is 140-42= 98 trees.<br>The answer is boxed{98} |
| GaC | Tom plants 10 trees a year and chops down 2 trees a year,<br>so the net increase in trees each year is 10 - 2 = 8 trees.<br>After 10 years, the total number of trees planted is 10 * 10 = 100 trees.<br>The total number of trees chopped down is 2 * 10 = 20 trees.<br>So the total number of trees after 10 years is 50 + 100 - 20 = 130 trees.<br>If 30% of the trees die, then the number of trees that die is 30/100 * 130 = 39 trees.<br>The number of trees left is 130 - 39 = 91 trees.<br>The answer is boxed{91} |  |
| UniTE | Tom plants 10 trees a year, so in 10 years he plants 10*10 = 100 trees.<br>He starts with 50 trees, so after planting 100 trees, he has 50+100 = 150 trees.<br>Every year he chops down 2 trees,<br>so in 10 years he chops down 2*10 = 20 trees.<br>So, after 10 years, he has 150-20 = 130 trees.<br>30% of the trees die, so 30% of 130 trees is 0.3*130 = 39 trees.<br>Therefore, after 30% of the trees die, he has 130-39 = 91 trees left.<br>The answer is boxed{91} |  |
| 3-models | LLM-Blender | Tom starts with 50 trees and plants 10 trees a year, so after 10 years he will have 50+10*10= 150 trees.<br>However, he also chops down 2 trees a year, so after 10 years he will have 150-2*10= 140 trees.<br>After 10 years, 30% of the trees die, so 30% of 140 is 0.3*140= 42 trees die.<br>The number of trees Tom has left is 140-42= 98 trees.<br>The answer is boxed{98} |
| UniTE | Tom plants 10 trees a year and chops down 2 trees a year,<br>so he has 10-2=8 more trees each year.<br>After 10 years, he will have 8*10=80 more trees.<br>He starts with 50 trees, so he will have 50+80=130 trees after 10 years.<br>After 10 years, 30% of the trees die, so 30% of 130 trees die, which is 0.3*130=39 trees.<br>Therefore, he will have 130-39=91 trees left.<br>The answer is boxed{91} |  |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could you provide more theoretical support or intuition for why limiting alignment to the top-k tokens effectively enhances ensemble performance? How does this approach balance between accuracy and computational efficiency at a probabilistic level?
- How does UNITE handle models with markedly different response styles in practice? Would introducing a preprocessing step to standardize response formats (e.g., for tasks like QA or summarization) enhance compatibility?
- Since UNITE is partially motivated by efficiency, adding a comparative breakdown of memory and latency for each method would clarify the computational trade-offs involved. Including charts or tables that detail average and worst-case latency per token would help underscore UNITE’s operational benefits.
soundness: 2
strengths: - The paper identifies compatibility challenges in LLM ensembling and focuses on top-k tokens, aligning this strategy with empirical evidence that vocabulary alignment often introduces computational inefficiencies. 
- The authors conduct extensive experiments on multiple models and benchmarks, analyzing factors like vocabulary size, response style, and latency. The results support UNITE’s superiority in maintaining high performance while minimizing latency and token manipulation.
-  The proposed determine-then-ensemble strategy offers a generalizable framework for selecting compatible models, making the findings applicable to real-world LLM applications that require efficient model collaboration.
summary: This paper introduces a novel ensembling approach, UNITE (Union Top-k Ensembling), that efficiently integrates large language models (LLMs) by focusing on the union of top-k tokens from each model rather than aligning the full vocabulary. It seeks to improve the computational efficiency and effectiveness of LLM ensembles by addressing key issues of compatibility, vocabulary size, and response styles. The authors propose a model selection strategy to identify compatible models, limiting the influence of incompatible LLMs on the ensemble's performance. Experimental results across multiple benchmarks validate the benefits of UNITE in enhancing performance, reducing latency, and decreasing the computational burden.
weaknesses: - While the paper addresses task-specific challenges, it could benefit from deeper exploration into why certain tasks (like MMLU) see greater performance improvements than others. Further insight into how task characteristics impact ensembling effectiveness would add depth to the analysis.
- The model selection process relies on response style consistency and performance alignment within a 10% range, which may limit scalability when dealing with a large pool of candidate models. The method would benefit from a more automated or quantitative metric for determining compatibility.
- While UNITE is evaluated across standard datasets, some benchmarks like GSM8K and TriviaQA may not fully capture the diverse range of LLM applications. Including more varied tasks could strengthen the argument for UNITE’s general applicability.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Does the optimal value of k for the top-k tokens vary across tasks and domains? Is there a principled way to determine k?
soundness: 3
strengths: - The paper presents concrete empirical conclusions about what factors are actually important for ensembling (performance gap, response style) or not (vocabulary size difference.)
- It presents UNITE, a novel top-k union ensembling approach that drops the computational overhead tremendously by using the top-k tokens of each model instead of the full vocabulary, and that obviates the need for full vocabulary alignment.
- UNITE significantly outperforms the baselines (LLM-Blender, DeePen, GAC) in average accuracy across six popular benchmarks for question-answering, reasoning, and knowledge.  
- The paper is clear and well written.
summary: The paper identifies and tests three hypotheses about importance factors that influence the performance of logit-based LLM ensembles: performance discrepancy, vocabulary size differences, and stylistic response differences, and provides guidelines for choosing models to ensemble with. The authors propose UNITE, an ensembling method that uses the top-k logits from each model, and show that it outperforms numerous ensembling baselines while being significantly cheaper.
weaknesses: - The analysis in Conclusion II on vocabulary size differences is restricted to English language tasks. It may benefit from a discussion of multilinguality, where tokenization might be less consistent between models.
- In Conclusion III the paper identifies differences in response style as a major problem for ensembling models. The proposed solution of limiting longer responses to 2x the length of shorter ones would benefit from theoretical justification. A robust solution to this problem which enables ensembling of models with different response styles would be ideal, as the current approach limits the practicality of the method.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Like Weaknesses#2 above, can you provide reasons or analysis why the top-k approach is better in performance than the whole vocabulary on performance?
soundness: 3
strengths: 1. It includes good analysis and preliminary experiments for LLM ensembling.
2. Based on the preliminary analysis, they reduce the vocabulary for the previous method into the top-k selection for the model ensembling.
summary: This paper proposed an LLM ensemble method, which is to ensemble among only top-k tokens probability.
weaknesses: 1. I believe the experiments should include the efficiency compared to the previous approach.
2. It seems, in theory, that including the whole vocabulary should work better, although maybe marginally or at least the same compared to top-k, since it includes the whole picture of the token distribution. I am questioning your experimental results because top-k is better in every dataset, which logically cannot be the case.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Do you have an insight about why increasing k beyond 10 does not improve perf? I think this result is counter-intuitive, and needs more analysis.
2. What is the impact of base model perf difference when using UNITE?
soundness: 3
strengths: 1. UNITE’s token-level aggregation without full vocabulary alignment is an innovative method which reduced computational needs and making ensembling more efficient.
2. Empirical results are shown over a concrete set of benchmarks.
3. Their focus on model selection strategy brings up insightful practical guidelines which are very useful in practice.
4. They present clear results comparing latency against existing methods.
summary: Paper has two main contributions:

1. Presents a thorough analysis of features affecting ensemble performance: bade model performance gap, vocabulary size, and response style consistency.
2. A new ensembling method, UNITE, that focuses on combining only the top-k tokens instead of the entire vocabulary. The method is efficient in runtime and shows a high performance across multiple benchmarks.
weaknesses: 1. Lack of analysis on possible limitations or settings or benchmarks where topk-k token alignment fails to improve base models perf.
2. Figures and tables could have more detailed captions with information to be self-contained. (ex figures 4,5 and 6)
3. Figures and plots' font size are very small. Consider increasing the font size to assists readers.

Meta-review: Many LLMs have been released recently, each of which is trained on different data and has its strengths and weaknesses when performing downstream tasks. This paper proposes a new method for ensembling LLMs, UNITE (UNIon Top-k Ensembling), to achieve better performance than any individual language model in the ensemble. UNITE ensembles LLMs by taking the union of the top-k tokens predicted by each model rather than considering the full vocabulary of the models; thus, the approach is more efficient than prior LLM ensembling methods and handles the compatibility challenges that come with ensembling diverse LLMs with different vocabularies. Furthermore, the experiments show that UNITE outperforms these prior approaches on multiple downstream tasks, despite the reduced computational overhead of the method.

Strengths:
- UNITE is an effective, general method for ensembling LLMs that improves over prior methods in this space while also solving an existing issue (compatibility across vocabularies)and reducing the computational costs of ensembling (nt9F, qMEZ, 22sv).
- The paper also extensively analyzes the factors that make ensembing LLMs effective (nt9F, qMEZ, QXTA).
- During revisions, the authors incorporated multiple experiments to address the reviewers' concerns, including a new LLM benchmark and an additional evaluation in a new language (Chinese) with CMMLU.

Weaknesses:
While the paper contains evaluations on many different benchmarks, there is limited analysis as to why some benchmarks benefit more from the UNITE method than others (nt9F) or in cases when the method fails (22sv).

Most of the other weaknesses raised by the reviewers, such as the effect of response style, were fully addressed in the author response. I recommend that the authors increase the size of the font in the figures, as they are currently difficult to read.

Decision: Accept (Spotlight)