Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling
Authors: Yuxuan Yao
Han Wu
Mingyang LIU
Sichun Luo
Xiongwei Han
Jie Liu
Zhijiang Guo
Linqi Song
Venue: ICLR 2025 Spotlight (2025)
Keywords: Model ensembling
LLM
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: This study introduces a general model selection strategy for ensembling and proposes an efficient ensemble method that operates on the top-k candidate tokens.

Abstract: Large language models (LLMs) exhibit varying strengths and weaknesses across different tasks, prompting recent studies to explore the benefits of ensembling models to leverage their complementary advantages. However, existing LLM ensembling methods often overlook model compatibility and struggle with inefficient alignment of probabilities across the entire vocabulary. In this study, we empirically investigate the factors influencing ensemble performance, identifying model performance, vocabulary size, and response style as key determinants, revealing that compatibility among models is essential for effective ensembling. This analysis leads to the development of a simple yet effective model selection strategy that identifies compatible models. Additionally, we introduce the \textsc{Uni}on \textsc{T}op-$k$ \textsc{E}nsembling (\textsc{UniTE}), a novel approach that efficiently combines models by focusing on the union of the top-k tokens from each model, thereby avoiding the need for full vocabulary alignment and reducing computational overhead. Extensive evaluations across multiple benchmarks demonstrate that \textsc{UniTE} significantly enhances performance compared to existing methods, offering a more efficient framework for LLM ensembling.

Published as a conference paper at ICLR 2025
DETERMINE-THEN--ENEEBBEI NECESSITY OF TOP-K UNION FOR LARGE LANGUAGE MODEL ENSEMBLING
Yuxuan Yao Han Wu Mingyang Liu Sichun Luo Xiongwei Han Jie Liu Zhijiang Guo Lingi Song 1.2
1 "Department of Computer Science, City University of Hong Kong
City University of Hong Kong Shenzhen Research Institute
3 Huawei Noah's Ark Lab
"North China University of Technology
'Hong Kong University of Science and Technology (Guangzhou)
yuxuanyao3-c8nm wtt.:cittyuc.....mme..e.mmmm. edu. hk wu hanl @huawei. com
linqi. songecityu .k
ABSTRACT
Large language models (LLMs) exhibit varying strengths and weaknesses across different tasks, prompting recent studies to explore the benefits of ensembling models to leverage their complementary advantages. However, existing LLM ensembling methods often overlook model compatibility and struggle with in- efficient alignment of probabilities across the entire vocabulary. In this study, we empirically investigate the factors influencing ensemble performance, iden- tifying model performance, vocabulary size, and response style as key determi- nants, revealing that compatibility among models essential for effective ensem- bling. This analysis leads to the development of simple yet effective model selection strategy that identifies compatible models. Additionally, we introduce the UNIon Top-k Ensembling (UNITE), novel approach that efficiently com- bines models by focusing on the union the top-k tokens from each model, thereby avoiding the need for full vocabulary alignment and reducing computa- tional overhead. Extensive evaluations across multiple benchmarks demonstrate that UNITE significantly enhances performance compared existing methods, offering a more efficient framework for LLM ensembling. The code is available at httttps://githhii com/ http///arrYyuuunn.ommmmm UniTE
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and have shown promising results in real-world upplications (OpenAl 2023 Yang al., 2024 Dubey et al. 2024). Given the diversity in data sources, model architectures, and training methods, T.LMs exhibit varying strengths and weaknesses depending the task at hand. Consequently, rather than relying solely on training an LLM from scratch, an alternative approach is to create an ensemble of LLMs. This method allows for leveraging the complementary advantages of different
LLMs (Jiagg et al. 2023b Lu et al. 2024 Yu et al. 2024b).
Existing model ensembling methods can be broadly categorized into three types: output-level, probability-level, and training-level approaches. Output-level methods (Jiang et al., 2023b; Lu al., 2024, Shnitzer al., 2023) aggregate the complete outputs of multiple candidate models. Probability-Ieeel methods (Huang et al., 2024 Yu et al.) 2024b). integrate outputs based on prob- ability distributions at each generation step through the intersection or union of the vocabulary. Training-level methods (Wan et al. 2024a Xu et al., (2024) utilize output probability vectors la- bels for richer information extraction during training. While output-level methods are constrained by the limitations of existing outputs, and training- methods introduce additional computational overhead, probability- methods have garnered particular attention.
Corresponding authors.

---

Published as a conference paper at ICLR 2025
Existing methods for ensembling LLMs, such DEEPEN (Huang et al. 2024) and GAC (Yu et al. 2024b). grapple with two significant challenges. First, these approaches concentrate solely on the en- sembling technique, sidestepping the crucial discussion of which types of models can be effectively combined. This oversight is critical because L.LMs with substantial differences in architecture, size, and tokenizer may be inherently incompatible, leading potential incompatibilities that undermine the benefits of ensembling(Dong et al.) 2023 Lee et al. 2023). Second, these methods tend to align the probabilities across the entire vocabulary at each generation step. Such a strategy introduces substantial computational overhead during inference, which hinders performance and efficiency.
To address these challenges, we conduct thorough investigation into the factors that truly influence ensembling performance. Our empirical analysis identifies three key factors: model performance, vocabulary size, and response process. We evaluate multiple widely recognized L.LMs across various benchmarks to explore ensembling from these perspectives. Our findings reveal several important insights: 1) Variations in performance levels among base I.LMs significantly affect their compati- bility for ensembling; 2) The impact of vocabulary size is marginal; and 3) Even when performance and vocabulary size are aligned across LLMs, substantial differences in the reasoning process in the response can hinder successful ensembling.
Building on this analysis, we adopt a eetemmnne-then-eseebblee strategy by starting with the best- performing LLM for target tasks, then iteratively selecting the next best-performing LLMs that meets ensembling criteria until the maximum number of models is reached or no further suitable can- didates exist. Additionally, stead of aligning the entire vocabulary of different LLMs, we propose the Union Top-k Ensembling (UNITE). UNITE constructs a union of the top-k tokens from each model and expands this set us- ing each model's tokenizer. This is followed by probability aggregation to determine the next token. UNITE avoids the need for auxiliary mapping matrices and full vocabulary aligmment, respecting the unique tokenization of each base LLM. Extensive experiments demonstrate the effectiveness of UNITE across various tasks, consistently outperforming state-of-the-art L.M ensemble methods.
seek to improve the efficiency and performance of ensembling. In-
Overall, our key contributions are: 1) We conduct extensive analysis of existing model ensem- bling approaches and derive three significant insights. Building on these findings, we introduce general model selection strategy for model ensembling; 2) We propose well-motivated model ensembling method that efficiently operates on the top-k candidate tokens rather than the entire vo- cabulary. 3) Experiments demonstrate significant performance improvements, reduced operational tokens to less than 0.04% % of current methods, and minimized latency to only 10 ms longer than the individual model, validating the effectiveness of our proposed approach.
2 RELATED WORKS
Based on the sequence of inference and fusion processes, multiple model collaboration methods can be broadly classified into two categories: model ensembling (Jiang et al.] (2023b] Yu et al.] 2024b) and model merging YYu et al.) 2024a| Akiba al. 2024). Model ensembling follows an inference- then-fusion approach, aiming to integrate the outputs of various models to achieve a more refined response. Conversely, model merging adopts a teiii--teen- strategy, wherein different models are combined into a single model before inference. While model merging is typically appli- cable only to homologous models, our focus in this study is on the more general approach, namely model ensembling. We discuss different model ensembling methods as follows:
Output-Level Model Ensembling methods involve selecting multiple candidate models and uti- lizing their complete outputs for aggregation. For instance, Jiang et al. (2023b) developed PAIR- RANKER, an additional ranking model, to evaluate and select the best candidate output. Similarly, Lu et al. (2024) and [Shnitzer et al. (2023) designed a router that determines the most appropriate candidate model based on the given question. However, these approaches are restricted by the exist- ing outputs and become ineffective if all options are incorrect. Some studies have addressed this by training fusion models to combine outputs (Jiang et al. 2023b; Wang et al. 2024b), which alleviates the limitation of relying solely on available candidates and often yields improved results. Neverthe- less, achieving generalization with fusion models remains a significant challenge, as they may not fully utilize the probability information generated each step.
2

---

Published as a conference paper at ICLR 2025
COMCK
POA
ABCC
M
Figure I: The impact of performance disparity among models on ensemble effectiveness across dif- ferent datasets and methods examined. We compare these methods the individual performances of LLaMA2 and Mistral, indicated by dashed lines.
Probability-Level Model Ensembling methods focus integrating outputs from different mod- els by utilizing the probability distribution at each generation step. COOL-FFUSION (Liu et al,, 2024 let each source LLM generate tokens until reaching common word boundaries, then jointly reranks the segments. However, this method relies on common word boundaries, limiting generation flexibility and diversity. Additionally, managing multiple source models can increase complexity. DEEPEN (Huang et al. (2024), based on the principles of relative representation theory, transforms each model's probability distribution from its individual space into a common relative space for ag- gregation, which is determined through the intersection of the models' vocabularies. On the other hand, GAC (Yu et al., (024b) constructs a mapping matrix that projects the probability vectors of multiple LLMs into the dimensional space of their union vocabulary, aggregating outputs to select the next token at each generation step. It is worth noting that all these existing approaches operate on the entire vocabulary at every single step, resulting significant computational overhead. In this work, we focus on the probability-level model ensembling without the need for additional training.
Training-Level Model Ensembling methods exemplified by FUSELLM (Wan et al. 2024a), lever- age output probability vectors from various models during the training process, using these vectors as lahels instead of one-hot representations. This technique serves
unique form of distillation, enabling the trained model to extract richer information from the probability outputs of the ensem- ble of teacher models. FUSECHAT (Wan et al.) 2024b) extends the framework of FUSELLM to fuse multiple chat LLMs with diverse architectures and scales. However, distillation is predomi- nantly aimed at enhancing smaller models, which limits its effectiveness in further advancing larger models. On the other hand, EVA (Xu et al.) (2024) addresses vocabulary discrepancies by learning token aligmment across different vocabularies, utilizing overlapping tokens for assistance. However, vocabulary alignment should involve both output embedding alignment and weight alignment, EVA achieves vocabulary alignment solely by solving the mapping matrix of output embeddings.
UNDERSTANDING MODEL ENSEMBLING FROM MODEL CAPACITY, VOCABULARY SIZE AND TASK
Existing model ensembling approaches (Huang al. 2024 Yu et al. 2024b) only focus de-
signing the model ensembling method, offering limited discussion on the selection of base models for ensembling. However, insights from prior research (Dong al. 2023 Lee et 2023). indi- cate that not all model pairs are compatible to be combined, particularly when there are significant differences in size, vocabulary, and performance. In preliminary experiments, explored var- ious factors that might affect the performance of model ensembling, including model size (c.g., BB/7B/8B/13B/000 model architecture (dense/sparse), performance discrepancies, tokenization strategies (BPE/WordPiece), vocabulary size (e.g., 102K/64K/32K) and tasks variations (e.g., text generation/QA//mltijp choices). Finally, we identified three representative factors for further anal- yses, including performance discrepancy, vocabulary size, and tasks variations.
3.1 IMPACT OF MODEL PERFORMANCE DISCREPANCY
To investigate the impact of performance disparity on model ensembling, we select LLaMA2- 13B- Chat and Bistral-B-Instrrut-- base models, which show significant performance gap across
3

---

Published as a conference paper at ICLR 2025
denner destto clanns Derten reacal
Adena
Oeitee trusuu 180 Deeter peseot
Lte clenge
Lone desnon Clerko rtnccll
Lnn dinser
Ghend
Figure 2: The impact of performance differences on model ensembling effectiveness on GSM8K dataset. OOM represents out of memory issue.
Datavel
Datnsct
Dataset
Methods
Methods
Methods
PIQA
MMLU
NQ
ARC-C
DeepScek-7B/ (02K)
59.67
72.66
DeepSeek-7B( (02K)
46.97
Mistal-7B(32K)
74.49
Mistal- 7B(32K
56.18
80.63
LLaMA2-13B
49.61
Yi-6B(64K)
22.55
73.21
61.16
76.54
48.86
22.97
LLE-BBENDER
L.M M.M-BEENDER
LLM-BLENDER
(+1.49)
(-4.09)
(-0.55)
(-0.89)
(-1.2%)
(-2.01)
55,00
77.65
52.81
60.00
25.26
76.50
DeEPENDeepSekk-7
DesPENDDeppeckkTe
DEEPEN(Misurl-P
(-1,77)
(-1,99)
(+5.8%)
(+1.27)
(-1,01)
(+1.0)
Drs
61,28
76.32
DEEPENILLaMA21383
54,09
62.39
DEEPEN(Yi-6B)
22,80
77,86
(+4.80)
(-4,31)
(+4,48)
(+10.86)
(+0.25)
(64.65
62,77
81.81
Ous<DeepSekk78)
48.90
60.16
OursMisstrll 7B)
24.76
76.54
(+3.10)
(-9.15)
(+1.933
(+1.43)
(-0.51)
(62.05)
Ous//Mistral-1B)
58.88
81.81
Ours/Mistral-BB)
48.90
60.16
23.28
76.54
(+2.40)
(+1.18)
(-0.71)
(+8.63)
(+0.73)
(43.33)
Table 1: The influence of vocabulary size model ensembling effectiveness
various tasks. We evaluated the GSM8K (Cobbe al. 2021), PIQA (Bisk al., 2020). ARC- C (Clark et al. 2018), and NQ (wiatkooski et al. 2019) datasets with three comparative methods.
As shown in Fig. I we found that when there is a significant performance disparity between mod- els. LLM- BLENDER, which selects the optimal response from model candidates, is not suitable for model ensembling. Similarly, for methods like DEEPEN and GAC, which are achieved through to- ken probability averaging, substantial performance gap also leads to certain performance drops. This observation is straightforward and easily understood, as pwwerpperforming models may intro- duce noise and bias, adversely affecting overall effectiveness. Nonetheless, it has been observed that DEEPEN and GAC consistently enhance the performance of less effective models when combined with a superior model. Although the resultant performance does not surpass that of the superior model, these methods can also serve as an alternative way for training free knowledge distillation.
To further figure out how performance discrepancy impacts the ensembling, then identify model pairs based on the GSM8K dataset with performance gaps of approximately 40% (LLaMA2-7B- Chat and
25% BLLMMA2-13b-Char and Ca-taal-7-Innstucttt (OpenChat-3.55 and -i-sral-t-tnntttttoo...... and less than 10% ALLaMA3-8BB-Instee and Qwen2- 7B-Instruct). Fig. 2 shows that as the performance gap increases, the improvement from ensembling inferior models becomes more pronounced. In contrast, for superior models, the ensembling effect consistently falls below the baseline performance. When the performance difference is within 10%, ensembling may yield better results.
15%
Moreover, was observed that with the increase in vocabulary size, as exemplified by models such as LLaMA3 and Qwen2, whose vocabulary size exceeds 120K, approaches like DEEPEN, which depend on nigh-dimensional matrix operations, encounter heavy memory limitations that diminish their effectiveness.
TAKEAWAY Smaller performance gaps lead to greater gains from model ensembling.
3.2
INFLUENCE OF VOCABULARY SIZE
Yu et al. 2024b) found that large models employing different tokenization methods, such as BPE and BBPE, exhibit over 90% overlap in tokenizing Oxford 5000 common words. This indicates that tokenization methods have a marginal impact model ensembling, leading to focus on the effect
4

---

Published as a conference paper at ICLR 2025
ain't heavy my brother lyrics
Question
was the riler of england in 1616
Bob Russell
Gold
James I
Bob Russell of
aLM response
James V
He Heavy, My
writien by Mike singer-songwriter and released as single in 1969.
song was originally recorded by
response
became and since been covered numerous artists.
The ruler of England in 1616 was King James I.
He was the first monarch of the Stuart dynasty and reigned from March 24, 1603, until his death on July 19, 1625.
He Ain't Heavy. He's My Brother writien Mike D' Abo. x
The song originally recorded by the Brirish songeriin and released as single in 1969.
L.M-Blnnder response
became hit and has since been covered numerous artists.
The ruler England was King James T. 7
He was the first monarch the Stuart dynasty and reigned from March 24, 1603, until death on July 19, 1625.
Table 2: Responses of different methods on NQ dataset.
of vocabulary size on ensemble performance. To this end, we selected four models with varying vocabulary sizes that exhibit similar performance on certain datasets, namely LLaMA2-138 (vocabulary size: 32,000), Mistral-7B ---sttttttttt (vocabulary size: 32,768), Yi-6B (vocabulary size: 64,000), and DeepSeek-LLLM-7B-CCTH (vocabulary size: 102,400).
distrteation Owen2
token distributionn Owwer2
Figure 3: Token distribution of different models. We extract the top 15 words from the vocabulary of each model and apply softmax processing to their corresponding logits.
As presented in Table [ existing methods demonstrate consistent performance irrespective of the gap in vocabulary size, thereby indicating that vocabulary size does not significantly influence the efficacy of model ensembling. To elucidate the underlying causes, conducted manual analysis of 100 responses generated by various LLMs. Our investigation revealed that approximately 80% of the words in the responses were common words while the tokenization of these common words exhibited no discernible differences across the different LLMs. Therefore, the vocabulary size, as well as the tokenization strategy, exerts minimal influence the performance of model ensembling. Notably, despite significant variations in vocabulary size. the ensembling results of models with similar performance suggest that when the performance gap between base models is within 10%, model ensembling may achieve better results than the baseline model, which is aligned with findings in Section 3.T. More interestingly, results of evaluating DEEPPEN on the ARC-C dataset indicate that when model performances are comparable, ensembling inferior models superior ones. This might imply that when performance differences are minimal, the selection of main base models does not necessarily need prioritize those with superior performance. Besides, it is important to note that these findings related to the vocabulary and tokenization consistently apply to LEMM-LENDER, DEEPEN and our method, with the exception of GAC. We will detail the reason in Section 42
outperform
TAKEAWAY II: The influence of vocabulary size for model ensembling is marginal.
5

---

Published as a conference paper at ICLR 2025
3.3 TASK-SPECIFE CHALLENGES IN MODEL ENSEMBLING
In the aforementioned experiments, we identified an interesting phenomenon: Even when perfor- mance and vocabulary size are aligned across models, substantial differences in the response style can also hinder successful ensembling specific tasks. For example, LLaMA3-8B and Qwen2- 7B exhibit comparable performance across various tasks. Existing ensembling approaches such as A.LM-BLENDDR have been shown to enhance outcomes on GSM8K and PIQA tasks when utilizing these two base models. However, ensembling these models to address the questions in TriviaQA and NQ datasets proves be impractical due the significant discrepancies in their response styles.
As shown in Table 2 under identical prompting conditions, LLaMA3 tends to directly return the answers, whereas Qwen2 is inclined conduct analysis. In voting-based sponse length significantly influences judgments, with PAIRRANKER consistently favoring longer responses. For instance, in our analysis of 100 randomly selected NQ and TriviaQA responses, the PAIRR ANKER favored the longer response as the answer in approximately 80% and 70% of cases accordingly, possibly due to training data characteristics. More detailed analysis of TriviaQA and more experimental results are in appendix [A] Similarly, probability- based ensembling methods like DEEPEN and GAC are also affected by the differences in the reasoning process, leading to signif- icant deviations in vocabulary probability distributions. Figure reveals significant disparity in top token distributions between LLaMA3 and Qwen2 for the first two tokens. Averaging the prob- abilities directly could result in excessively smoothed distribution, which is inappropriate. The underlying reason for this phenomenon may be the differences in L.MMs training data. Therefore, caution is warranted in specific tasks avoid substantial disparities, despite effective ensembling in most other tasks.
TAKEAWAY III: Even though the performance and vocabulary size are aligned across models, substantial differences in response style could also hinder successful ensembling.
4
METHODOLOGY
4.1
MODEL SELECTION STRATEGY
Given the aforementioned insights, we present a strategy to assess the compatibility of two models for ensembling and offer guidance for selecting base models from the candidate model pool.
When ensembling two models, we begin by comparing their performance on the target tasks. An overall performance gap within 10% desirable, as indicates the potential compatibility for suc- cessful ensembling. If the performance gap falls within this range, it is crucial to compare the response styles of different models. This comparison is relatively subjective, with differences often identifiable through superficial features such as text length and BLEU score.
In this work, we use text length as a guiding metric by constraining the length of longer responses to not exceed twice the length of the shorter ones. When the response manner is also consistent across models, it is highly probable that these two models can be successfully ensembled to produce a superior response. We exclude discrepancies in vocabulary size from consideration, prior analyses have demonstrated its insignificance in affecting ensembling outcomes.
To select base models from the pool, recommend initially choosing the best persorrerommng model for the target tasks. Subsequently, select the next best-performing model that satisfies the criteria for successful ensembling with the first chosen model, continuing this process iteratively until the maximum number of base models is reached or no further suitable models can be found.
4.2 UNION T A ENSEMBLING
Apart from the model selection strategy, we also endeavor to improve the efficiency and performance of model ensembling. Existing probability-Ievel model ensembling methods, e.g. DEEPEN and GAC, attempt to combine the models through full vocabulary alignment. We argue this approach sub-optimal since the candidate next token typically resides within the top-k tokens. Conversely, incorporating tokens of lower probability may introduce unnecessary noise, thereby diminishing the overall accuracy and effectiveness of the ensemble. Motivated by this, propose the UNTON Top-k

---

Published as a conference paper at ICLR 2025
Ensembling (UNITE), novel and efficient ensembling approach that only aligns the top-k tokens in each decoding step.
Given the base models for ensembling, denoted as LLM; with their respective tokenizer T and vo- cabulary V*, we feed the input text prompt into LLM, and obtain the probability distribution. Instead of aligning the probability distributions across different models, we only focus the subset of top- k tokens since the candidate next token highly likely to be within the subset. The selected top-k tokens of LLM, is presented as TopK; (1i, f2 .... b with corresponding probability distribution p = (p), P2 ..... p.))
Next, we aim to construct the union set V of TopK, across the base models. The most straightfor- ward solution obtain the union set is directly remove the duplicate tokens and retain all other distinct tokens. For the token in V but not in V the probability of this token in LLM; is assigned to O. This strategy is adopted by GAC (Yu et al. 2024b). However, we contend that this strategy is not entirely reasonable since it is heavily influenced by the tokenizer and vocabulary. For exam- ple, the candidate token "James" is within the top-k subset of LLM, but not in V while the token "Jam" appears in TopK2. The word "James" can be tokenized into "Jam" and "es" by LLM2. If the token probabilities P PiJam) and pî PiJames] are set O, the overall probability of generating the word "James" should be inherently reduced, even if both base models exhibit a preference for this word. Therefore, we present new token probability alignment strategy on the union set. Firstly, we obtain the top-k union vocabulary V by directly merging the TopK, subsets. Then, the token distribution p and TopK, are updated according to following criteria:
1. If the token w appears in both V and TopK;, Pt and TopK; remains unchanged.
2. If the token w in / is absent in TopK; but present in V it will be appended to TopK; pi also updates accordingly.
3. If the token w in V does not exist in V it should be tokenized by T". The first token of the result along with its token probability is then updated TopK; and Pt.
Up to now, we can obtain the aligned top tokens Topk; and the token distributions po of based models. Consequently, we normalize pi as: Piorm softmax(Pi).
Since our method eliminates the need for full vocabulary alignment, it is essential to designate one model as the primary base model. As discussed in Section (322, the selection of primary base model is flexible when the candidate models exhibit comparable performance. To simplify the process, we directly employ the best-performing model as the primary base model. Then, the primary base model employs the average token probability Pang to predict the next token, where Pang = n Ei pi norm
The next token is determined using the maximization-basel greedy strategy (Li et al. 2016)_ The chosen token will be appended to the input text. This process iteratively repeated until the pre- determined stopping criterion is met, such as generating end-of-sentence token or reaching the maximum length. The algorithm process can be referenced in the Appendix, Algorithm
EXPERIMENTS
5.1
SETUP
Models All of our experiments are conducted with the following commonly used models, includ-
ing LLaMA2-7B-CO
(Touvron et al., (003)), LLaMA2-13BB Chat (Touvron et al.) [023), LLaMA3-
8B-Instruct (Dubey et al. 2024), LLaMA3.1 1-BB- 1--8----B-Bstrt (Dubey et al. (024). Mistral-7B- Diirall-B-Instruct-
v0.3 (Jiang et al. (023a)), DeepSeek-LLMM-7B-CII OpenChat-3.. (Wang et al. 2024a), Qwen2-7B-Instrne (Yang et al. 2024), Mixtral-8x7B Jiang
(Bi al. (024). Yi-6B (Young et al. 2024).
ct al. 2024) and Qwenl.5 1----..-BBBBB (Bai et al. 2023).
Baselines We selected three typical model ensembling methods for further analyses. 1) LLM- BLENDER (Jiang et al. 2023b) includes a reward model, PAIRRANKER to evaluate LLM responses, and fusion model, GENFUSER, to combine them. We only use PAIRRANKER due to significant over- wver-generation issues with GENFUSER. 2) DEEPEN (Huang et al. 2024) employs relative repre- sentation theory map each model's probability distribution universal space for aggregation,
7

---

Published as a conference paper at ICLR 2025
Avg.
GSMBK
PIQA
MMLU
ARC
TrivaQA
NQ
Mistral
$6.48
80.63
59.28
74.49
64.30
24.25
99.91
DeepSeek
49.51
OpenChai
73:46
$7.10
60.80
28:08
61.27
31,08
63.38
LLM BLENDER
70.79(-2.67)
83.28((3.82)
60, (0(-0.70)
76.29X-1.76
56.35(-5.42)
25.57(-5:51)
62.06(-3.32)
DEEPEN
73.06/-0.0
76,04(11106)
61.9((+1.11)
52.1
67.21 (+5.7))
28.26(-2.82)
63.11(-2.27)
GAC
12.85(11061))
67.8%(-19.25)
55. 1565.655
73.04-5.001)
62:23(40.45)
30.72(-10.56)
(6.97(-88411)
UNITE
73.31(-0.15)
87.50 (+0.40)
62:13 (+1.33)
78.70 (+0.68)
65.80044.33)
31.78 (+0.70)
66.54 (+1.16)
Table 3: Results of ensembling on Mistral, DeepSeck, OpenChat. OpenChat chosen as primary model for these experiments.
Dataset
Method
Avg.
GSMSK
PIQA
ARC-C
MMLU
LLaMA3
78.77
79.08
79.01
64,58
75.36
LLaMA3.1
79.49
77.47
Qwen2
80.78
84.92
78.81
Two model ensembiing (LLaMA3+ Qwee2)
M-BE-NDDDRR
82.69 (+1.91)
82,53 (-2.04)
82.98 (-1.94)
62.07 (-2.99)
77.57 (-1.24)
DEEPEN
OOM
GAC
80.67 (-0.11)
80.96 (-3.61)
84,93 (+0.01)
67.05 (+2.09)
78.40 (-0.41)
(+3.39)
85.53 (+0.9))
(+0.15)
69.78 (+4.82)
81.14 (+2.33)
Three-model ensembling
M-BLENDER
83.30(+2.52)
83.47(-1.10)
83.48(-1.444
62.55(-2.41)
78.20X-0.61)
DEEPEN
0OM
UNITE
84.99 (+4.21)
84.98 (+0.4)
(+0.47)
69.12 (+4.16)
81.12 (+2.31)
Table 4: Results of ensembling on LLaMA3, LLaMA3.1, Qwen2. Qwen2 is chosen as primary model for these experiments.
computed through the intersection of model vocabularies. 3) GAC (Yu et al. (024b) projects mul- tiple LLMs' probability vectors into a unified vocabulary dimension using a mapping matrix and aggregates outputs at each generation step select the next token. Although GAC suggests the importance of identifying keywords to improve latency, we still exclude it owing to its hindrance on the ensembling performance.
Benchmarks We evaluate six benchmarks, which can be categorized into three main groups. 1) Comprehensive Examination: MMLU (5-shot) (Heudrycks et al. [021]. covering 57 subjects that humans typically learn: ARC-C (0-shot) (Clark et 2018)) collected from standardized natural science tests. 2) Reasoning Capabiliitis: GSM8K (Cobbe et al.
2021 (4-shot), dataset high- quality problems at the grade school math level; PIQA (Bisk et al. 2020) (0-shot), commonsense reasoning dataset. 3) Knowledge Capacities: TriviaQA (5-shot) (Joshi et al. 2017), compiled by
Trivia enthusiasts; NaturalQuestion (NQ) (5-shot) (K (Kwiatkosski al. 2019), question-answeringg
corpus consisting of queries issued to the Google search engine.
For comprehensive evaluation, we conduct two- and multi-model ensembling experiments on the aforementioned benchmarks. We select base models following the strategy presented in Section 4.1 We choose two pairs from Mistral, Deepseek, and OpenChat for the two-model experiments. For the closely performing LLaMA3, LLaMA3.1, and Qwen2, we select LLaMA3 and Qwen2 for the two-model ensemble based their release dates and subsequently employ all three models for the multi-model ensembling evaluations. The hyper-parameter & is set to 10 in this work. All experiments are conducted on 46G NVIDIA L40 GPUs.
5.2 MAIN RESULTS
As shown in Table 3 and Table 4
we have following observations:
8

---

Published as a conference paper at ICLR 2025
Impact the chorce typerparamennrt
Latercy Comparison Muthods
02
CMC
POU (MGC
Wethols
00.0
Trrabe bisk
Latency (marlusen)
Figure 4: Impact of the choice of hyperparameter k on the ARC and TriviaQA datasets. Increasing h beyond a certain point leads to slight decline no improvement performance.
Figure 5: Latency comparison of different methods. The darker the color, the greater the latency.
(1) UNITE enhances individual model performance when the base models exhibit similar per- formance levels. As demonstrated, the ensemble of models such OpenChat achieved an average improvement of approximately 1.2% across five benchmark tasks, including ARC-C and MMLJU However, in the GSM8K task, 15% performance gap between DeepSeek and OpenChat led to slight decline overall performance, reinforeing our observation Section3.I ensemble meth- closely aligned. Furthermore, experiments
ods are most effective when model performances with LLaMA3 and Qwen2 validated our approach, resulting performance increases of 3.39% and 4.82% on the GSM8K and MMLU tasks, respectively.
(2) UNTTE demonstrates greater robustness and generality. Ensembling experiments with LLaMA3 and Qwen2 indicate that while LLM-BLENDER improves performance the GSM8K dataset, it significantly underperforms compared to baseline models in the PIQA, ARC-C, and MMLU benchmarks. Specifically, BLENDER demonstrates decline of 2.89% the MMLU task, and this issue is exacerbated in ensembles with the baseline models OpenChat and Mistral, which exhibit relatively low baseline performance, leading to an average performance drop of 3.32%. Ad- ditionally, GAC fails to improve the performance of individual models across most tasks and exhibits a significant decline over 10% when combined with Mistral and OpenChat. Furthermore, due to its intersection limitations, DEEPEN is ill-suited for ensembling with the LLaMA3 model, which requires handling large vocabulary. contrast, across tasks with comparable performance levels, UNITE achieves the highest improvements and overall performance across the board, with the ception of a slight under-pedt performance relative to DEEPEN on TriviaQA. This outcome underscores the effectiveness and robustness of UNITE.
(3) Collaborating with comparable LLMs does not necessarily yield better results. The ensem- bling experiments conducted with three models, following the integration of LLaMA3.1, demon- strate improved performance on the GSM8K and ARC-C benchmarks compared to the ensemble of LLaMA3 and Qwen2. However, this approach results in suboptimal outcomes on PIQA and MMLU. This observation is justified, as the analysis in Section 3.2 indicates that while combining models with similar performance may enhance overall efficacy, such improvement is not guaranteed.
5.3 ABLATION STUDY
Effeet of hyperparameter l: selection To investigate the impact of the hyperparameter k. we con- ducted experiments using the Mistral and OpenChat models on the TriviaQA and ARC-C datasets. As illustrated Fig. increasing from 5 to 10 enhances ensemble performance. However, further increasing k beyond 10 leads to either slight decline no change in performance. This finding supports our assertion that, in probability ensembling, aligning the entire vocabulary is unnecessary for predicting the next token.
Effeet of the next token selection strategy We also explored the impact of deterministic decoding and top-k sampling (Fan et al. 2018) on the next token emission. is important to clarify that our UNITE focuses on the fact that, when predicting the next token, only need to ensemble subset of tokens rather than the entire vocabulary. In contrast, greedy decoding and top-k sampling em- phasize how determine the next token's IDD after the ensemble process is complete. To investigate

---

Published as a conference paper at ICLR 2025
Basc Model
Method
Tokens Manipulated Each Step
M165rak36678)
DiKPPiN
320001
Model
PIQA
Qwen1.5-72 1.55722
09.03
73.83
OpanChan(52002)
UNITE
14.46
Mikstal-Sx77(Spunnn
73.98
74.59
LLaMA3(28226)
T09566
78.84
81.88
+
170336
UNITE
Qwca2(151616)
UNITE
14.43
(+7.29)
Table 5: Tokens manipulated at each step. It's noteworthy that DEEPEN encoun- tered out-of-memory issues when conducting LLaMA3 and Qwen2 ensembling.
Table 6: Ensemble learning of the dense LLM Qwenl.5-72B and the sparse MOE model Mixtral-8x7B.
this, conducted experiments using LLaMA3.1 and Mistral on the PIQA and ARC-C datasets cor- respondingly. As shown in Figure for these deterministic tasks, the maximization-usn greedy method outperforms the random sampling approach, which aligns with intuition.
5.4 FURTHER ANALYSIS
Latency analysis Following the settings de- tailed in GAC (Yu et al. 2024b), we recorded the latency (ms/token) of different methods. The data presented in Fig. reveals significant differences in latency among the various meth- ods. Notably, the latency of UNITE is 87.78 ms/token, which is substantially lower than that of DEEPEN and GAC, and only about 10 ms longer than that of the individual models.
Comparison among Diffrrnt Decoding Methods
20.1
Tokens manipulated each step Table 5 presents tokens manipulated each step. Com- pared to DEEPEN and GAC, UNITE signifi- cantly reduces the number of tokens manipu- lated at each step, indicating its efficiency. As the vocabulary of the base model expands, the number of manipulated tokens of DEEPEN and GAC increases significantly, which not only increases the computational burden but may also lead to performance bottlenecks. Contrastly, UNITE is minimally affected, maintaining its effectiveness.
Decoding Nethed
Figure 6: Comparison among different decoding meth- ods. The greedy decoding strategy is more effective for eliciting the next token in deterministic tasks.
Ensemble of the dense model and the sparse model We evaluate our method on the ensemble learning of the dense model and the sparse MoE model for reasoning and comprehensive examina- tion tasks. Specifically, we utilize the widely-used large-scale dense model Qwen 1. .5-72B (Bai et al. 2023) alongside the popular sparse MoE model Mixtral-8x7B (Jiang et al. (024) our base mod- els. As shown in 6 our UNITE achieves +4.86% and +7 +7.29% performance on the ARC-C and PIQA datasets respectively, despite the base models already exhibiting high levels of performance.
6 CONCLUSION
In conclusion, our research highlights the effectiveness of ensemble methods in enhancing the per- formance of LLMs. By examining existing techniques, we identified key factors influencing ensem- bling success, such as model performance and response processes, while finding that vocabulary size has minimal impact. In addition, the issue of vocabulary redundancy exposed by existing methods during ensembling lead us to propose UNITE, which efficiently aggregates a bunch to- kens from multiple LLMs without the computational overhead. Through extensive experimentation, UNITE consistently outperformed ofate-of-the-as art ensemble methods, demonstrating its effective- ness in leveraging the strengths of diverse LLMs. Our contributions not only advance the under- standing of model ensembling but also provide a practical framework for selecting and integrating L.MMs achieve superior performance.
10

---

Published as a conference paper at ICLR 2025
ACKONWLEDGEMENTS
This work was supported in part by the Research Grants Council of the Hong Kong SAR under Grant GRF 11217823 and Collaborative Research Fund C1042-23GF, the National Natural Science Foundation of China under Grant 62371411, InnoHK initiative, the Government of the HKSAR, Laboratory for Al-Powered Financial Technologies.
REFERENCES
Takuya Akiba, Makoto Shing. Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. CoRR, abs/2403. 131177, 2024. doi: 10:4550/ARXX222403.111777 URL
www..o.rg/11... 48550/ w1t.:/a.........1.....
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men. Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. /6609, 2023.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Alex Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang. Bingxuan Wang. Peiyi Wang. Shiyu Wang, Yaohui Wang, Yongji Wang. Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong. Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang. Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek LLM: scaling open-source language models with longtermism. CoRR, abs/2401 022544 2024. doi: 10. RB550/ARXIV.240000222
URL htthttps: http:://ooo..gg..o 48540/555555 www.a.axi.v.... 02954
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao. and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Innovative Applications of Artificial Intel- ligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artifi- cial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439, AAAI Press, 2020. doi: v34105. 6239.
URL /oii btattts://doi.org///11
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs//803305s 803.05457, 2018. URI. htt.ntttp///rrxx.oor///ass httpa/////1800.0...
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman, Training verifiers to solve math word problems. CORR,
2021. URL
https: pattps//aase /atp:i//aaivv.vaa org/ abs / / 2200011111... 14168
Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distilla- tion without training. In JEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 11898-11908 IEEE, 2023. URL
doi: 10.1099CVPPR277922022
2023.0112
11

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Amirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits. David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadcokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024. doi: 1044550ARXX.2240727775 URL https: /ww. wrg/oo../o.o. 4850..455553300 2tt.:Xi..247..2177.
Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbouune, Australia, July 15-20, 2018, Volume 1; Association for Computational Linguistics, 2018. doi: 10.18653/V1/ Long Papers. pp. http:://acaaaatahhaa org 1082/ P18- 1082. URL
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- ew net. forfforuuui Porommmm um? d=d 77B jm 3Gm@
view.net, 2021. Utt.htttss:///peee
Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang. Bing Qin, and Ting Liu. Enabling ensemble learning for heterogeneous large language models with deep paral- lel collaboration. CORR, abs/2404.12215, 2024. doi: 10.4850/ARXXV.2444.222 2404. 2404.12715)
URL.
/tttps://dol..or/1....44
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.008255 2023a. doi: 10.4855//RXI..331000055 06825
URL htttps://doit wtt.:///0..0000...........
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary. Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. CoRR, abs/2401. 2024. doi: 10.48555/ARXIVV24001 0.40888 URL
https: doi org/l0.......oo..... 100.0000000 10.111.888500 48550/855440 arxiv 2401. 04088
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 14165- doi: 653666361113 / 22023 ac oong. 792
14178. Association for Computational Linguistics, 2023b. 792. htt.hhttp::///oo o rrg
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,
12

---

Published as a conference paper at ICLR 2025
ACL 2017, Vancouver. Canada, July 30 August 4, Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguistics, 2017. doi: 10. 10.18663/1//11-11447, URL https:
//doi.org/10. 10 18653/v1/P17-122
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Trans. Assoc Comput. Linguistics, 7:452 466, 2019. doi: 10.1162/TACI, A) 00276. URL. https: /do. .org/
10. 21l16//aacccc 0_0026666111
Hayeon Lee, Rui Hou, Jongpil Kim, Davis Liang, Sung Ju Hwang, and Alexander Min. A study on knowledge distillation from weak teacher for scaling up pre-trained language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, Toronto, Canada, July 9- 14, 2023, pp. 11239-11246. Association for Computational Linguistics, 2023. doi: wtw.inng-iiiinnn1111100 714
URL wttttt::s//o..oooeee/o 1ttp:/1...11611111111
Haonan Li, Yixuan Zhang. Fajri Koto, Yifei Yang. Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: measuring massive multitask language understanding chi- nese. In Lun-War Wei Ku, Andre Martins. and Vivek Srikumar (eds.), Findings of the Associa- tion for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 11260-11285, Association for Computational Linguistics, 2024. doi: 10. 18653/W1/022FFFIIIIGGSSS ACL.671. URL Findings-acl.67 671
10. 1et...16633v20200
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep inforcement learning for dialogue generation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1192-1202. The Asso- ciation for Computational Linguistics, 2016. doi: 10.18653/V1/DI6-11227 URL https:
1127
Cong Liu, Xiaojun Quan, Yan Pan, Liang Li, Weigang Wu, and Xu Chen. Cool-fusion: Fuse large language models without training. CoRR, abs/2407. 2024. doi: 10.48555//RXIV.2407. 19807. URL https::/ //co:::///o.o// www.oo.g.oo....oo. 48550/ wrwia.aiivv.... 2407 19807
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mex- ico, June 16-21, 2024, pp. 1964-1974. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2224.NAA hww.:/224..AAAALLLLLNGNGG. / 10. 1tt.ooooo0001111155//1//00000000 naacl-long 109
OpenAl GPT-44tchhiin report. ab02233.2//33 2023. doi: 1048550/ARXIV22300.0777. URL
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thomp- son, and Mikhail Yurochkin. Large language model routing with benchmark datasets. CoRR, Abs/2309.15788) 2023. doi: 10.48550/ARXXV.230991 15789. URL htt.hhttss:///t //doi. .tt.//org.....
48550/arXiv. 2309.15789
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint 2rXiv:2210.09266 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
13

---

Published as a conference paper at ICLR 2025
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a. URL
htttps://openreet ew. net www.ne/ooor.tttttteee
forum? iid= jiDkk 12qcz
Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, and Wei Bi. Fusechat: Knowledge fusion of chat models. CoRR, abs/2402.16107, 2024b. doi: 10.48550/RXXIV24022 16107. URL https::/dod wttp:ol..././///....../
Guan Wang. Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Ad- vancing open-source language models with mixed-quality data. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024a. /URpassss////peeereeeeeee
Hongyi Wang. Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric P. Xing, and Mikhail Yurochkin. Fusing models with complementary expertise. In The Twelfth International Conference on Learn- ing Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL /ettps://openrettt
net Eorum2idd-PMMIGMMRRR
Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. Bridging the gap between different vocabularies for LLM ensemble. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 7140-7152. Association for Computational I Linguistics, 2024. doi: URLhttps:: http://dfooogg...1// 1ttp///1..1663/111/11
395
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407.10671/ 2024. doi: (0.48550/ARXIV.2407.. 10671. URI. htttt:tt////ett 10671
405.000044555/axxxxx.. 2407.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue. Senbin Yang, Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by Ol.ai. CoRR, 0bb/24003004657 2024. doi: 10. htthttts:/// //do: 10 wtw.1:...550//01......
AS550/ARXIV.2403.044
wwwpv.iiiivv....om http:///wwww.......... 240.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models super mario: Ab- sorbing abilities from homologous models as a free lunch. In Forty-firs International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a. net forum? forrummid--f
htLphttss wtw.eeeeiwwinee//oumuuummmm
Yao-Ching Yu, Chun-Chih Kuo, Ziqi Ye, Yu-Cheng Chang, and Yueh-Se Li. Breaking the ceiling of the LLM community by treating token generation as a classification for ensembling. CoRR, abs/2406. 12585, 2024b. doi: 1044850/AXXV.2406.2222
htttp:://doi.ooo/
11550/arXiv.2400
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ALGORITHM OF UNITE
As presented in Algorithm I UNITE integrates the Top-k tokens generated by multiple base models to create joint set, updating the tokens based on probability rules, and ultimately uses a greedy strategy select the next token from this set, repeating the process until stopping criterion is met.
Algorithm Union top-k ensembling
Require: LLM,, Vocabulary V., Tokenizer T,, Demonstration prompt, Stop condition stop(*)
I: while not stop(()
Stopping criteria
2: TopK, Pi <- ..I.M.proopt)
b Generate top-k tokens and their prohabilities
3: for each model do
4:
if token w E V and w E TopK; then
5:
['w and TopK; remains unchanged.
6:
else if token w E V and w E V* and w w TopK; then
TopK; 4- uc f <- Pu
8:
else if token w E V" and w & then
9:
wi, ...., w + T"t (w)
10:
Topk. + wt, p Pin
11:
end
12:
end for
13:
p Pform softmax( Pr), Paog 1 Ei=1 Pr Prorm
14:
w argmax (Paug)
Predict the next token
=ETopR,
15:
prompt prompt
Update input sequence
end while
A.2 TASK CHALLENGES ANALYSIS ON TRIVIAQA
Table 7 provides detailed comparison of the differences in response styles between LLaMA3 and Qwen2 on the TriviaQA dataset. It is easily observable that, regardless of correctness, LLM- BLENDER consistently tends to choose longer responses answers.
We try to address the response style issue via preprocessing steps. Specifically, we provide an alternative simple solution by using the few-shot examples to standardize the response format. We employed a new 5-shot prompt designed to elicit answers the format: The answer is xxx. The responses are presented in the Table
As shown in Table 9 the original prompts elicit accuracy in the brackets. Since the tedious response style illustrated in Table Qwen incorporates answers into the analysis, randomly sampled 100 instances from the 1500-test set to manually extract the predictions. After adjusting the prompt, we can easily analyze the results for the entire test set. Consistent with our findings presented in the main text, when the base model's performance gap exceeds 10%, ensemble learning may yield little no improvement. Additionally, is important note that our UNITE approach still outperforms its competitors.
A.3 GENERALIZATION EVALUATION ON BIG-BENCH HARD BENCHMARK
We conduct additional experiments using the BBH (BIG-Bench Hard) benchmark (Suzgun et al., 2022), diverse evaluation suite of 23 challenging tasks such symbolic reasoning, to further validate UNITE's generalizability. Due to computational constraints and limited rebuttal time, we randomly selected 10 subsets, and the results are presented below. Consistent with main findings, UNITE demonstrates superior performance compared to other methods, highlighting the effective- ness and generalizability of our approach.
15

---

Published as a conference paper at ICLR 2025
had
Kiss Over?
Question
The flag Libya plain rectangle color?
Sir Henry Campbell Bannerman
Gold
Green
Henry Camphell-Bannerman /
LLMM response
Green /
The song reached number one performed by Arros,, featuring vocalist Jim McCarty.
song Kiss
Singles Chart 1976.
The song was written Arrows' lead singer Jim McCarty, along with the band's guitarist, Steve FigHorrison.
It was produced by Tony Viscanti, who had previously worked with David Bowie and T. Rex. Riss You All Overwas released as a single from the band's debut album, Aroos, Which was also releaved 1976.
Qwen2 Response
The song's catchy chorus and upbeat tempo made it popular bit of the 1970s,
and it remains well known and beloved track from that era.
The song's success helped to establish Arrows popular hand of the time, and it continues
played on classic rock and oldies radio stations today. The flag Libyu plain rectangle of green color. / It was adopted in 1977 and features green color field with other symbols designs. The color green is significant the country's history and culture. representing hope, fertility, and country's The flag is rectangular in shape and is often flown vertically. It is an important symbol of the Libyan state and used various official and ceremonial contexts.
agricultural heritage.
The hit song Kiss You All Overwas performed by the band Arrows, Leaturing vocalist Jim McCarty. X The song reached number one on the UK Singles Chart in 1976.
The song was written by Arrows' lead singer Jim McCarty, along with the band's guitarist, Steve FigHarrison. It was produced Tony Visconti, who had proviously worked with David Bewic and T. Rex.
Riss You All Overwas released as single from the band's debut album, Arows, also released 1976.
LLM-Blender response
The song's catchy chorus upbeat tempo made popular hit and remains well-knooe koolo- and beloved The song's 's success helped establish Arrows popular
the time,
and it continues to be played on classic rock oldies radio stations today. The flag Libya is plain rectangle of green color. / It was adopted in 1977 and features green color field with no other symhols or designs.
The color green is significant in the country's 's history and culture,
representing hope, ferrility, and the country's 's rich agricultural heritage.
The flag is rectangular in shape and is often flown vertically. is important symbol of Libyan and used various official and ceremonial contexts.
Table 7: Responses of different methods on TriviaQA dataset
TriviaQA Question
Which Lloyd Webher musical premieened in the US on 10th December 1993?
Question: In the 1971 Number One hit Emii by Benny Hill, what the name Emiies 's horse who kicked his rival, Ted from Teddington?
Original prompt (Response style referring Table 2)
Triggers.
Question: In the 1971 Number One hit Ernie by Benny Hill, what was the name of Emic's 's horse who was kicked by his rival, Two ton Ted from Teddington?
New Prompt
Answer: The answer is Triggers.
LLaMA3 response
answer Sunset Boulevard.
Qwen2.5 response
answer Boulevard.
Table 8: Case study of preprocessing on TriviaQA.
A.4 MULTILINGUAT TOKENIZATION CONSIDERATIONS
We also conducted experiments the CMMLU (Li et al. 2024h. Chinese Multitask Language Understanding Evaluation benchmark to validate our findings in section 3.2 We used Yi-6B (vo- cabulary size: 64,000) and Diven--bb-instruce (vocabulary size: 152,064) our base models, both supporting the Chinese language. Qwen2-70 Swen2-7b-instruch is the primary model. the results of various en- semble approaches presented in Table Similar to the results listed in Section 3.2 irrespective
16

---

Published as a conference paper at ICLR 2025
Method
TriviaQA
LLaMA3-8b-instruut
70.68(67)
2ween.5--bbinnstuul
57.85(52)
-LE-BBLENDER
64.77
UNITE
67.45
Table 9: Results of ensembling LLaMA3 and Qwen2.5 on TriviaQA after preprocessing. LLaMA3 chosen as primary model for these experiments.
Method
BBH
LLaMA3-8b-instre
73.00
Swen2-7b-insten
68.60
LLM BLENDER
68.79
DEEPEN
-0OM-
GAC
69.86
UNITE
73.52
Table 10: Results ensembling I.aMA33 and Qwen2 on BBH. LLaMA3 is chosen as primary model for these experiments.
Method
CMMLU
Yi-6B
75.21
Qwen2-7h-instruct
83.22
LLM-BLENDER
79.08
DEEPEN
-0OM-
GAC
75.88
UNITE
83.89
Table 11: Results of ensembling Yi and Qwen2 on CMMLU. Qwen2 is chosen as primary model for these experiments.
of the gap in vocabulary size, existing methods still demonstrate improvements, thereby indicating that vocabulary size for model ensembling is marginal.
A.5 FURTHER ANALYSIS OF HYPERPARAMETER A:
We clarify that the enhanced efficiency and effectiveness arise from reduced token options and our specialized union mapping method, as outlined in Section 4.2 UNITE constructs union of the top- tokens from each model and expands this set using each model's tokenizer, followed by probability aggregation determine the next token. UNITE eliminates the need for auxiliary mapping matrices and full vocabulary alignment, respecting the unique tokenization of each base LLM.
We further test with extremely large values to validate our effectiveness. As shown in Table 12 further increasing k leads to either a slight decline or no change in performance. This finding reinforces assertion that, in probability-level ensembling, it is unnecessary to align the entire vocabulary to predict the next token.
A.6
CASE STUDY OF UNITE
Table 13 presents the specific cases of UNITEE and other ensembling approaches, showcasing the validity and stability of our method.
17

---

Published as a conference paper at ICLR 2025
Setting Mistral
TriviaQA
64.30
OpenChat
61.77
64.52
I 10
65.80
i 20
65.77
& 100
65.65
& 1000
65.74
È 10000
65.72
Table 12: Impact of the large hyperparameter k TriviaQA dataset.
18

---

Published as a conference paper at ICLR 2025
Tom plants trees year, Every year chups a year. years
Question
starts
How many trees does have loft?
Ho gets 10 2 << 10
2 = >>>> new trees a year.
After 10 years he has 8 10
=< 8 10 80 >>> 80 new trees
Gold Answer
so he had 80 50
<<< 80 50
130 >> 130 trees
He lost 130 3
130
33
39 >> 39 trees
means
<<< 130
frees
Tom start with 50 trecs.
He plants 10 trees a year, so after 10 years he has 3++10-10= 150 trees. However, be chops down trees year, after 10 years he has 150-2*100 2010= 140 trees.
LLaMA3 Respomse
30% of the trees dic, which means 30/100"1 111100114000 140= 42 trees dic.
Tom is left with 140 42- 98 The answer boxed
Tom sturs with 50 trees and plants 10 trees
after 10 years he will have 50+10* I0= 150 trees.
However, he al chops down trees a year, so after 10 years he will have 150-2* 10= 140 trees. After 10 30% of the trees die, 30% of 40 is 0.3 0.3*140= 42 trees die.
ILMMA3.1 Response
The number of trees Tom has left is 140-42= trees. The answer boxed
Tom starts with S0 trees and plants 10 trees year, after 10 years he will have 50+10*I0s Id= 150 trees. However, he sl chops down trees year, so after 10 years he will have 150-2* 10m 140 trees. After 10 years, 30% of the trees dic, 30% of 140 is 0.3 140m trees dic.
Qwen2 Response
The number of trees Tom has left is 140 42=
The answer boxed
Tom sturts with 50 trees and plants 10 trees year, after 10 years he will have 50+10* 10= 150 trees. However, he also chops down trees a year, so after 10 years he will have 150.2 10= 140 trees. After 10 years. 30% of the trees die, 30% of 40 is 0.3 143=140== 42 trees die.
LLM.
L.LM-Blender Blender
2-models
The number of trees Tom has left 140-42= trees.
The answer boxed
Tom plants 10 trees year and chops down trees a year, the net increase in trees each year is 10 trees. After 10 years, the total number of trees planted is 10 10 100 trees.
The total number of trees chopped down * 10 20 trees.
GaC
So the total number of after 10 years 50 100 20 130 trees.
30% the trees then the number of that die 30/100 e 130 39 trees. number of left
91
The answer boxed 91
Too plants 10 trees year, so in 10 years he plants 10*10= 100 trees. He starts with 50 trees, so after planting 100 trees, he has 50+100 150 trees.
Every be chops down trees. in 10 years chops down 2410=
UNITE
So, after 10 years, 30% of 130 0.3* 30% the
150-20
Therefore,
trees die, 130-39
The answer is boxed 91
Tom with 50 trees and plants 10 year, after 10 years he will have 50+ 10 IQ= 150 trees.
However, he al chops down a year, after years will have 150-2* 10w 140
I.MM-Blender
Alter years. number
of 0.34140=
3-models
left
The answer is baxed 98
Tom plants 10 trees and chops down trees a year,
be has 10 2=8
each year.
After 10 years, will have 8* 1010880 more
He with trees, will have 50+80=130 after years. After years,
die, which 0.3 10-09 trees. Therefore, will 130-39w91 left.
trees
The answer boxed ( 91
Table 13: Case study of UNITE and other ensembling methods GSM8K
19


Reviews:
[{"review_id": "i4j0DvO3Qv", "reviewer": "Reviewer_nt9F", "summary": "This paper introduces a novel ensembling approach, UNITE (Union Top-k Ensembling), that efficiently integrates large language models (LLMs) by focusing on the union of top-k tokens from each model rather than aligning the full vocabulary. It seeks to improve the computational efficiency and effectiveness of LLM ensembles by addressing key issues of compatibility, vocabulary size, and response styles. The authors propose a model selection strategy to identify compatible models, limiting the influence of incompatible LLMs on the ensemble's performance. Experimental results across multiple benchmarks validate the benefits of UNITE in enhancing performance, reducing latency, and decreasing the computational burden.", "questions": "- Could you provide more theoretical support or intuition for why limiting alignment to the top-k tokens effectively enhances ensemble performance? How does this approach balance between accuracy and computational efficiency at a probabilistic level?\n- How does UNITE handle models with markedly different response styles in practice? Would introducing a preprocessing step to standardize response formats (e.g., for tasks like QA or summarization) enhance compatibility?\n- Since UNITE is partially motivated by efficiency, adding a comparative breakdown of memory and latency for each method would clarify the computational trade-offs involved. Including charts or tables that detail average and worst-case latency per token would help underscore UNITE’s operational benefits.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies compatibility challenges in LLM ensembling and focuses on top-k tokens, aligning this strategy with empirical evidence that vocabulary alignment often introduces computational inefficiencies. \n- The authors conduct extensive experiments on multiple models and benchmarks, analyzing factors like vocabulary size, response style, and latency. The results support UNITE’s superiority in maintaining high performance while minimizing latency and token manipulation.\n-  The proposed determine-then-ensemble strategy offers a generalizable framework for selecting compatible models, making the findings applicable to real-world LLM applications that require efficient model collaboration.", "weaknesses": "- While the paper addresses task-specific challenges, it could benefit from deeper exploration into why certain tasks (like MMLU) see greater performance improvements than others. Further insight into how task characteristics impact ensembling effectiveness would add depth to the analysis.\n- The model selection process relies on response style consistency and performance alignment within a 10% range, which may limit scalability when dealing with a large pool of candidate models. The method would benefit from a more automated or quantitative metric for determining compatibility.\n- While UNITE is evaluated across standard datasets, some benchmarks like GSM8K and TriviaQA may not fully capture the diverse range of LLM applications. Including more varied tasks could strengthen the argument for UNITE’s general applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3OODPkwCOH", "reviewer": "Reviewer_qMEZ", "summary": "The paper identifies and tests three hypotheses about importance factors that influence the performance of logit-based LLM ensembles: performance discrepancy, vocabulary size differences, and stylistic response differences, and provides guidelines for choosing models to ensemble with. The authors propose UNITE, an ensembling method that uses the top-k logits from each model, and show that it outperforms numerous ensembling baselines while being significantly cheaper.", "questions": "Does the optimal value of k for the top-k tokens vary across tasks and domains? Is there a principled way to determine k?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents concrete empirical conclusions about what factors are actually important for ensembling (performance gap, response style) or not (vocabulary size difference.)\n- It presents UNITE, a novel top-k union ensembling approach that drops the computational overhead tremendously by using the top-k tokens of each model instead of the full vocabulary, and that obviates the need for full vocabulary alignment.\n- UNITE significantly outperforms the baselines (LLM-Blender, DeePen, GAC) in average accuracy across six popular benchmarks for question-answering, reasoning, and knowledge.  \n- The paper is clear and well written.", "weaknesses": "- The analysis in Conclusion II on vocabulary size differences is restricted to English language tasks. It may benefit from a discussion of multilinguality, where tokenization might be less consistent between models.\n- In Conclusion III the paper identifies differences in response style as a major problem for ensembling models. The proposed solution of limiting longer responses to 2x the length of shorter ones would benefit from theoretical justification. A robust solution to this problem which enables ensembling of models with different response styles would be ideal, as the current approach limits the practicality of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FYuTekGVqm", "reviewer": "Reviewer_QXTA", "summary": "This paper proposed an LLM ensemble method, which is to ensemble among only top-k tokens probability.", "questions": "Like Weaknesses#2 above, can you provide reasons or analysis why the top-k approach is better in performance than the whole vocabulary on performance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It includes good analysis and preliminary experiments for LLM ensembling.\n2. Based on the preliminary analysis, they reduce the vocabulary for the previous method into the top-k selection for the model ensembling.", "weaknesses": "1. I believe the experiments should include the efficiency compared to the previous approach.\n2. It seems, in theory, that including the whole vocabulary should work better, although maybe marginally or at least the same compared to top-k, since it includes the whole picture of the token distribution. I am questioning your experimental results because top-k is better in every dataset, which logically cannot be the case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LEUtv7azha", "reviewer": "Reviewer_22sv", "summary": "Paper has two main contributions:\n\n1. Presents a thorough analysis of features affecting ensemble performance: bade model performance gap, vocabulary size, and response style consistency.\n2. A new ensembling method, UNITE, that focuses on combining only the top-k tokens instead of the entire vocabulary. The method is efficient in runtime and shows a high performance across multiple benchmarks.", "questions": "1. Do you have an insight about why increasing k beyond 10 does not improve perf? I think this result is counter-intuitive, and needs more analysis.\n2. What is the impact of base model perf difference when using UNITE?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. UNITE’s token-level aggregation without full vocabulary alignment is an innovative method which reduced computational needs and making ensembling more efficient.\n2. Empirical results are shown over a concrete set of benchmarks.\n3. Their focus on model selection strategy brings up insightful practical guidelines which are very useful in practice.\n4. They present clear results comparing latency against existing methods.", "weaknesses": "1. Lack of analysis on possible limitations or settings or benchmarks where topk-k token alignment fails to improve base models perf.\n2. Figures and tables could have more detailed captions with information to be self-contained. (ex figures 4,5 and 6)\n3. Figures and plots' font size are very small. Consider increasing the font size to assists readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)