Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration
Authors: Yichong Huang
Xiaocheng Feng
Baohang Li
Yang Xiang
Hui Wang
Ting Liu
Bing Qin
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Ensemble Learning
Large Language Model
Relative Representation
Primary area: natural_language_processing
TL;DR: Enable the probability distribution averaging between heterogeneous large language models

Abstract: Large language models (LLMs) exhibit complementary strengths in various tasks, motivating the research of LLM ensembling.
However, existing work focuses on training an extra reward model or fusion model to select or combine all candidate answers, posing a great challenge to the generalization on unseen data distributions.
Besides, prior methods use textual responses as communication media, ignoring the valuable information in the internal representations.
In this work, we propose a training-free ensemble framework \textsc{DeePEn}, fusing the informative probability distributions yielded by different LLMs at each decoding step.
Unfortunately, the vocabulary discrepancy between heterogeneous LLMs directly makes averaging the distributions unfeasible due to the token misalignment.
To address this challenge, \textsc{DeePEn} maps the probability distribution of each model from its own probability space to a universal \textit{relative space} based on the relative representation theory, and performs aggregation.
Next, we devise a search-based inverse transformation to transform the aggregated result back to the probability space of one of the ensembling LLMs (main model), in order to determine the next token.
We conduct extensive experiments on ensembles of different number of LLMs, ensembles of LLMs with different architectures, and ensembles between the LLM and the specialist model.
Experimental results show that (i) \textsc{DeePEn} achieves consistent improvements across six benchmarks covering subject examination, reasoning, and knowledge, (ii) a well-performing specialist model can benefit from a less effective LLM through distribution fusion, and (iii) \textsc{DeePEn} has complementary strengths with other ensemble methods such as voting.

Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration
Yichong Huang', Xiaocheng Feng Baohang Lit, Yang Xiang', Hui Wang
Ting Liu Bing Qin
THarbin Institute of Technology $ Peng Cheng Laboratory
(ychuang, xcfeng, baohangli, tliu, ginb]@ir... .itt edu. (xiangy, Jeir hit. edu.cn
Abstract
Large language models (LLMs) exhibit complementary strengths in various tasks, motivating the research LLM ensembling. However, existing work focuses on training extra reward model fusion model to select combine all candidate answers, posing great challenge to the generalization on unseen data distributions. Besides, prior methods textual responses communication media, ignoring the valuable information in the internal representations. In this work, we propose training-free ensemble framework DEEPEN, fusing the informative probability distributions yielded by different LLMs at each decoding step. Unfortunately, the vocabulary discrepancy between heterogeneous LLMs directly makes averaging the distributions unfeasible due to the token misalignment. To address this challenge, DEEPEN maps the probability distribution of each model from its own probability space to a universal relative space based on the relative representation theory, and performs aggregation. Next, we devise a search-based inverse transformation to transform the aggregated result back to the probability space of one of the ensembling LLMs (main model), in order to determine the next token. We conduct extensive experiments on ensembles of different number of LLMs, ensembles LLMs with different architectures, and ensembles between the LLM and the specialist model. Experimental results show that (i) DEEPEN achieves consistent improvements across six benchmarks covering subject examination, reasoning, and knowledge, (ii) well-pet well-performing specialist model can benefit from a less effective LLM through distribution fusion, and (iii) DEEPEN has complementary strengths with other ensemble methods such voting"
Introduction
With the scaling of model capacities and data volumes, generative large language models (LLMs) have shown impressive language understanding and generation abilities, shedding light for artificial general intelligence 35 p2 13 28). Due to diversities of data sources, model architectures, and training recipes, LLMs have different strengths and weaknesses in various tasks and cases. Therefore, recent research has explored the ensemble of LLMs to exploit the complementary potential [15] 19).
Existing methods can be categorized into selection-based and fusion-based ensembling. Selection- based ensembling selects the best candidate answer from all individual L.LMs' answers using an additionally trained reward model [15 31 25 19]. Fusion-basse ensembling combines all candidate answers using trained fusion model
15 However, these approaches inevitably face significant
Our code is available at: means corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

challenges in generalizing unseen data distributions and base models. Besides, prior methods enable collaboration via conveying the textual responses between LLMs while ignoring the rich information (e.g., confidence and alternative answers) in the internal representations.
An ideal solution to this issue is to apply the well-established technology of prediction fusion. 36 24 [[] For LLM ensemble, prediction fusion works at each decoding step, averaging the probability distributions from different LLMs to determine the next token. It could not only directly apply to the ensemble of any LLMs without extra parameter training, making it more general, but leverages the informative internal representations (i.e., probability distributions) as communication media. Unfortunately, the vocabulary discrepancy between different L.MSs makes it unfeasible average the distributions due to token misalignment.
In this work, we tackle this key challenge by drawing upon the cross-model invariance of relative representation, which represents each token using the embedding similarities of this token to a set of anchor tokens [21]. Specifically, Ensemble), enabling distribution fusion for heterogeneous LLMs. DEEPEN transforms the probability distribution from the heterogeneous probability space matrix formed by the relative representation of tokens. Next, DEEPEN aggregates the relative representations of all probability distributions in the relative space, coordinating the decision on the next token. Finally, the result of aggregation is transformed back to the probability space of the main model using search-based inverse transformation determine the next token.
propose an ensemble framework DEEPEN (Deep Parallel
homogeneous relative space, using
We conduct extensive experiments ranging from 2-model to 9-model ensembles, covering ensembles of models with parameters ranging from 6B to 70B, ensembles of dense and sparse models, and the ensemble of LLMs with specialist models. Experimental results on six widely-used benchmarks demonstrate that compared to baselines, DEEPEN achieves consistent improvements across all benchmarks. It is also discovered that DEEPEN has complementary strengths when combined with other ensemble methods.
2 Theoretical Analysis
We first introduce relative representation and then illustrate the theoretical support for our method.
2.1 Relative Representation
Previous study discovers that despite the misalignment between latent spaces of different neural networks, the embedding similarity between samples do not change across models [21
11 23
Specifically, Moschella et al. (21) propose relative representation, which represents each sample (+) by the embedding similarities to set of anchor samples A (x and A are identically distributed);
1g = (cos(egro, e,()), (.)1)))))))))))
where E()) denotes the embedding of samples, also is absolute representation.
It is empirically evidenced that relative representations possess cross-model invariance, i.e., the relative representation of the same sample keeps invariant across different models, which lays the theoretical foundation for our work to fuse heterogeneous probability distributions.
2.2 Theoretical Support for DEEPEN
Average probability distribution has been widely evidenced to effectively improve the predictive performance in the filed of image and text [0]. For generative language models, we understand. the underlying mechanism is to interpolate different output semantics represented by the probability distributions. However, for LLM ensemble, vocabulary discrepancy isolates these output semantics in semantic spaces with different basis vectors, making the interpolation infeasible. To tackle this challenge, we aim to enable the cross-model alignment for output semantics, i.e., find transformation to map the output semantics into universal space. To this effect, we propose to represent the output semantics with the convex combination of relative representations of all tokens where the weight the probability assigned to the token.

---

LLama2-13B
LLama2-7B
Mistral-7B
The same vocabulary
( Different vocabulary
Relative Representation Consistency - 75.4
Relative Representation Consistency 75.2
Figure 1: Visualizations for relative representations between models with the same vocabulary and between models with different vocabularies. PCA and K-means clustering are applied only for visualization. Different colors indicate different clusters of samples (word embeddings). The red block indicates the representation of tokens that only appear in Mistral's vocabulary. Relative representation consistency is obtained by calculating the cosine similarity between the relative representations of the same token different models.
Definition of output semantics in relative space. Formally, given the absolute representation of the output semantics p and the relative representation matrix [||] where V is the vocabulary and A V is the anchor token set. The i-th row of the relative representation of word w(();
R[i]
(cos(e (..,,,.. cos(ew eg(AAD) )),
and the relative representation of the output semantics p is defined
p R.
Model-invariance of relative representation of output semantic. Next, we illustrate why this rep- resentation scheme could align the output semantics isolated in heterogeneous absolute spaces. First, considering two LLMs A and fg with the same vocabulary (e.g., LLaMA2-7B and LLMMA2-13B) When expressing the same output semantic, these models output the same probability distribution (i.e., absolute representation) РΛ and pn- Besides, they have the same (highly similar in practice) relative representation matrix due the vocabulary consistency and cross-model invariance of relative representation. Therefore, the relative representations output semantics are also identical:
rA = PA RA
PB
RB = TB-
33)
Then, let's consider language model 0c with a different vocabulary (e.g., Mistral). Based on the fact that different LLMs typically share mass tokens in their vocabularies ($A). the vocabulary of model 0 is identical adding and removing partial tokens the vocabulary of fgg which leads to Pы 2 Pe and Rg 전 Rg. However, in our study, we discover that this change to the vocabulary has not incurred significant influence on the relative representation of the unchanged tokens (i.e., the common tokens between On and 0c), as shown in Fig. [ Therefore, we make the reasonable assumption that the local change the vocabulary could hardly influence the relative space.
Methodology
In this section, we first introduce the overall process of our ensemble framework DEEPEN and then describe the three parts DEFPEN detail.
3.1 Overview
We illustrate the process of DEEPEN in Fig. Given N models ensemble, DEEPEN first constructs their transformation matrices (i.e,, relative representation matrices) mapping the probability distributions from the heterogeneous absolute spaces into the relative space (3.2).. At each decoding step, all models perform prediction and output N probability distributions. These distributions are mapped into the relative space and aggregated ($3.3). Finally, the aggregation result is transformed back into the absolute space of the main model, in order to determine the next token ($3.4).

---

Absolute Space (LLaMA)
Relative Space
Absolute Space (Mistral)
Search-bun barchb-aseed Transformation
Emitted Taken
Relative Representation Matris a.l.M.M
Aggregationn
Relative Represennation Matris Mistral
Relarive Transformation
Transformation
Mistral (Main Model)
LLaMA
Figure 2: Overview of DEEPEN. The relative representation matrix of each LLM is directly derived by calculating the embedding similarities between each token with the anchor tokens.
3.2
Construction of Relative Transformation
Given N models to ensemble, DEEPEN first finds the intersection of vocabularies of all models, i.e, common token set C, and samples a subset or uses the full set of common tokens as the anchor token A C. Next, for each model, DEEPEN calculates embedding similarities of each token to the anchor words, obtaining the relative representation matrix R (as shown in Eq Finally, to overcome the relative representation degeneration of outlier words, which will be introduced later, we perform normalization on the relative representation of all tokens by softmax operation so that it becomes probability distribution. We denote the normalized representation matrix ft:
R so R(())-
(4)
Anchor Selection. The choice of anchor tokens is erucial for the relative representation capability. Previous research discovers that the capability improves as the number of anchor words increases (21. Therefore, we employ the full set of common words between LLMs as the anchor words. is also empirically proved that this method performs more stablely downstream tasks ($5.2).
Normalization of relative representation matrix. In DEEPEN, the relative representation each token is normalized by the softmax operation to avoid the relative representation degeneration of outlier words, which are referred to as words that are far away from other words (including the anchors) and become distinguishable in relative space since for being zero vectors. The softmax operation effectively resolves this problem by making each relative representation a probabilistic distribution instead of zero vector.
3.3 Aggregation in Relative Space
At each decoding step, once each model 0g outputs the probability distribution p,, DEEPEN transforms p; into the relative representation r; using the normalized relative representation matrix: r; P; R,
and aggregate all relative representations to obtain the aggregated relative representation:
ay Fo
(5)
where ag is the collaboration weight model 0g-
Collaboration Weight. As our work focuses on enabling the distribution fusion of heterogeneous LLMs instead of finding the optimal collaboration weights, we follow the most common prac- tice to uniformly aggregate the distributions (a 1/N, N is the number of models), which is named DEEPEN-Avg. Besides, we also adopt a simple and effective method of deducing weights, DEEPEN-Adape which heuristically sets larger value to the model with better performance on the development set: a; = 5i/E, Σ, 5j, where 8; = Acc(O, Ddce) e, Acc(:,) indicates the average accuracy of model 0, on the development set, and indicates the chance level on the evaluation task. Specifically, on the free-form generation tasks and 1/ K on the K -chhiiee tasks.

---

3.4 Inverse Transformation of Relative Representations
To decide the next token according to the aggregated relative representation, DEEPEN aims to transform it from the relative space back to the absolute space of the main model, which is empirically selected with the best-performing model on the development set. set. To enable this inverse transformation, adopt search-based strategy, finding the absolute representation whose relative representation is identical to the aggregated relative representation. This search problem is formulated as:
P: arg min (p. x R, F),
(6)
P.E.P. P.
where P. denotes the absolute space of model 0,, and {(-) is the loss function to measure the distance between relative representations. In this work, we adopt the KL-divergence due its convergence.
This search iteratively conducted under the guidance of the gradient of the loss in Eq with respect to the absolute representation P,- Specifically, we initialize the start point of searching P; main model's original absolute representation, and update as:
(0) with the
(++1)
(8) P 11
de
(8) l [0,
3p
where is an important hyperparameter named the relative ensemble learning rate, and T is the iterations number named relative ensemble learning steps. Finally, we use the updated absolute representation P (I') to determine the emitted token.
Experiments
4.1 Experimental Setup
Benchmarks. We mainly conduct experiments on six benchmarks, which can be categorized into:
Comprehensive Examination: (1) MMLU (5-shot) [2], which covers 57 subjects that humans learn, and (2) ARC-C (0-shot) [5] collected from standardized natural science tests.
Reasoning Capabilities: (1) GSM8K 6 (4-shot), which is a dataset of high quality problems at the grade school math level, and (2) PIQA (0-shot), which a commonsense reasoning dataset.
Knowledge Capacities: (1) TriviaQA (5-shot) [16], collected by Trivia enthusiast authored, and (2) NQ (5-shot) [18] which a QA corpus consists of queries issued to the Google search engine.
Evaluation. For all benchmarks, we follow the scripts of OpenCompass leaderboard. Specifi- cally, on the multiple-choice tasks (MMLU, ARC-C, and PIQA), the option with the highest likelihood selected calculate the accuracy. On the free-form generation tasks (GSM8K, TriviaQA and NQ), we calculate the exact match (EM) accuracy.
Individual models. As ensemble learning typically works models with comparable perfor- mance [24] 34], we select six well-performing LLMs whose performance are closely matched: LLaMA-2-13B 13B (29]. Mistral-7B-vO.1 [13. InternLM-20B (26). Yi-6B (1] Brwwork-13B-bbae (32).
and Tigerbot- - 13b-base-v2 44. To achieve better ensemble performance, we conduct experiments on the ensemble of the top-2 models and the top-4 models for each benchmark. Besides, we also consider ensembling various number of models ($4.) and ensembling more diverse models ($5.1).
Hyperparameters. In this work, we select all of the common tokens between LLMs as the anchor tokens to build the relative spaces, i.e., A = C' (..).. For the inverse transformation of relative representations, we search the optimal relative learning rate (1) in Eq. 77) from 0.05 to 0.30 with an interval of 0.05. We empirically set the number of relative ensemble learning steps T = 5 5 ($5.3)
Comparative methods. We We compare DEEPEN with (1) MINED 30 which maps the probability distributions of heterogeneous LLMs to the distribution of the main model via aligning tokens in different vocabularies with edit distance, and (2) L.M-BBLENDER (15), which comprises a reward model PAIRRANKER to score each response of LLMs and fusion model GENFUSER to fuse
5

---

Examination
Reasoning
Knowledge
Models
MMLU
ARC-C
GSM8K
PIQA
TriviaQA
NQ
Individual Models
LLaMA2-1B -3B
55.07
59.32
29.80
59.68
74.32
28.67
InternLM-2200
59.94
75.81
53.83
64.78
66,88
26.09
Skywork-13B
61.16
66.50
53.90
74.04
58.65
19.75
Tigerbot-138 133
51,95
57.44
48.82
68.28
66.22
22.71
Mistral-7B
62.13
73.33
47.50
65.61
73.18
27.62
Yi-6B
63.25
73.33
37.91
76.15
59.02
18.98
Top-2 Ensemble
63.85 (+0.60)
75.73 (- 0.08)
54.89 (+0.99)
78.31 (+2.16) 74.10 (- 0.22) 28.61 (-0.66
MINED
(41.79)
77.35 77.35 (+1.54)
18.50 (-35.40)
78.98 (42.83) 72.30 (- 2.02) 28.45 (-0.22)
DREPEN-AVg
64.68 (+1.43)
77.52 (+1.71)
55.42 (+1.52)
78.87 (42.72) 75.90 (+1.58) 30.17 (+1.50)
DEEPEN-Adapt
65.01 (+1.76)
77.52 (+1.7)
55.65 (+1.75)
79.37 (43.22)
76.08 (+1.76) 30.69 (42.02)
Top-4 Ensemble
-LEEDDENDER
61.44 (- 1.81)
71.03 4.78)
43.37(-10.53)
71.16 4.99)
67.87 (- 6.45) 24.18 (-4.49)
VOTING
64.88 (+1.63)
78.41 (+2.60)
63.15 (+9.25)
76.82 (+0.67)
MBR
62.09 (+8.26)
74.32 (+0.00) 30.28 (+1.61)
MINED
65.61 (42.36)
78.68 (42.87)
56,56 (42.66)
77.87 (+1.72) 71.62 2.70) 29.50 (+0.83)
DEEPEN-ANg 65.09 (+1.84)
78.70 (42.89)
56.18 (42.28)
77.15 (+1.00) 75.74 (+1.42) 31.55 (+2.88)
DEEPEN-Adapt 65.25 (+2.00)
79.15 (43.34)
56.25 (42.35)
78.59 (+2.44)
75.76 (+144) 31.77 (43.10)
+VOTING/MBR 65.40 (42.15) 79.44 (43.63) 65.25 (+11.35)
77.37 (+1)22) 75.65 (+1.3)) 32.11 (+3.44)
Table I: Main results. The best individual model is highlighted in red , and the best ensemble method highlighted in green, except for the results of the combined method (i.e., the last row). The top-4 models on each benchmark are underlined.
indicates that the method does not apply to the task.
candidate responses. In this work, we we only adopt the PAIRRANKER since GENFUSER suffers from serious over-generation under our training-free setting. In the ensemble of more than two models, we introduce two additional ensemble methods: (3) VOTING, which selects the choice favored by most models on the tasks with outputs limited to fixed set, and (4) MBR B 17). which selects the answer with the highest textual similarity to other candidate answers. The implementation details of baselines are illustrated in B
4.2
Main Results
The main results are shown in Tab. from which we have drawn the following observations:
(1) DEEPEN achieves consistent improvements over the individual models. These results prove that our DEEPEN successfully enables collaboration between heterogeneous LLMs via aggregating their probability distributions in the relative space. Specifically. DEEPEN-Avg achieves improvements of +144(MMMLU)
on the ensemble of top-2 models, and (0M on the ensemble of top-4 model. DEEPEN-Adapt gains improvements of +1 33..44TiiiiiAA))433..........
+3.30)4)44333444/0 34(RC--) on the ensemble of top-4 models.
(2) DEEPEN shows better stability than baselines. As shown, LLM-BLENDE struggles achieve improvements under the training-free setting. MINED shows unstable performance across different benchmarks. For example, MINED leads performance drops of -35.40 on the GSM8K benchmark under the top-2 models ensemble setting and -2.70 on the TriviaQA, indicating the limitation using textual similarity to align tokens in heterogeneous vocabularies. Through case studies, it is revealed that this method of aligning tokens with edit distance disturbs the decoding and produces incomplete words (demonstrated in $7). Instead, DEEPEN-Avg achieves consistent improvements and surpasses all baselines in 7/12 settings.
(3) DEEPEN has complementary strengths with other ensemble methods. VOTING achieves significant improvement the mathematical reasoning GSM8K, showing the effectiveness of

---

MMLU (Å- +2.48)
PIQA +4.11
NQ +3.44 )
65.42
66.33
65.14
79.37
79
53
79.61
79.87
79.76
30,65
31.77
31.16
30, 50
63,25
28.67
61
63.25
Ml-i 0012 S094
16.15
76.95
14.34
21
88
59.6
15.67
2500
18.00
1348
-laumd/79 -g
*k 13
-Reelgee 168
Interal M Jon
Tiger tot-
-Nabbigee 160
YuaB
-k Mkywork.11
- Mitraks:
Interal 17.00
+LAMAA-118
LlaM12110
-fpural-7
- M-300
Naubogge TTprbb
118 118
168
Individual
PEn Adppt
Indi iduall
Dre Adass
Individual
Dee Adap
Figure 3: Test set results of ensemble learning on various number models. Individual models arranged in descending order of their performance the development set, and sequentially incorporated into the ensemble. Å indicates the largest improvement achieved by DREPEN.
Model
GSM8K
PIQA
Model
En-De
De
En-Ro Ro
Ro-Enn
LLaMA2-70B
63.84
71.27
LLaMA2-13B
30.60
42.27
30.83
39.99
Mixtral-8x7B
65.73
71.88
NLLB B-B0-600MM
32.30
41.49
31.91
42.39
LLM-BLUNDER
64.22 (-1.21)
74.54 (42.66)
LLM-BLENDER 33.26 (40.6)) 43.28 (+1.01)
33.17
41.99 ((-0.40)
MINED
75.65 (+1.37)
(43.77)
MINED 25.12 36..4
29.91 (-2.00) 34.39 ((8800))
DEEPEN
(+1.60)
(43.22)
DREPEN 33.34 (3.70 32,95 (+1.04) 42.84 0.45)
Table 2: Ensemble learning of the dense large language model LLaMA2-70B and the sparse MoE model Mixtral-8x7B
Table 3: Ensemble learning of the generalist model
LLaMA2 and the specialist translator model NLLBB on the translation benchmark Flores-200.
reasoning with multiple paths. To evidence the complementary strength of DEEPEN with VOTING, we combine both methods. On the TriviaQA and NQ, VOTING is replaced with MBR. As shown that the combination both methods gains a further improvement over VOTING (63. 155.255..
(4) Collaboration with more worse-performing LLMs is double-edged sword. The ensemble performance of DEEPEN-Avg with top-4 models surpasses that with top-2 models on 4 benchmarks, but falls short on 2 benchmarks. This is reasonable because incorporating the 3rd and 4th ranked LLMs enhances complementary strengths but also causes the interference with the top-2 models.
4.3 Results on Different Numbers of Models
Next, we illustrate the effectiveness of DEEPEN on the ensemble of more models on the MMLU, PIQA, and NQ. We add Nanbeige- 16B into the ensemble on all three benchmarks, and add LLaMA2- 70B and Mixtral-8 - 7B on the PIQA due to their comparable performance. As illustrated in Fig. 3 the ensemble performance increases first and then decreases with the joining of more models in descending order of performance. And the ensemble performance peaks in the top-4 or top-5 models across three benchmarks.
Analysis
To deeply understand DEEPEN, we first evaluate its performance on the ensemble learning of model sets with diverse architectures, abilities, and performance gaps. Next, we conduct a series of analyses on the reverse transformation process of relative representations.
5.1 Results of Ensembling Diverse Models
Ensemble of the dense model and the sparse model. We first evaluate our method on the ensemble learning of the dense model and the sparse MoE model on the challenge reasoning tasks. Specifically, we use the widely-used large-scale dense model ILLMA2-70B (2] and the popular sparse MoE model Mixtral-8 7B [14 as the base models. As the results shown in Tab. 2 our DEEPEN achieves improvements of +1.60 and +3.22 on the GSM8K and PIQA datasets, even though the base models have achieved a high level of performance.
7

---

78.00
64.00
76.00
63.00
74.00
62.00
72.00
8k
Full
61.00
2k
16k
Set
ARC-C-Dee
(b) MMLU-Dev
Figure 4: Effect of the number of anchor words. The x-axis indicates the number of anchor words randomly sampled from the common words for 4 times.
30.65
30.89
29.81
31
29.06
29 29.22
28.92
MML.-Dev
TriviaQA-Der
28.01
Methods
28
ACC
ACC
A
DEEPEN
Baseline
63.61
61.19
+2,42
74.79
72.74
+2.05
22
29.06
27.95
26.51
22.99
22.77
19,97
19:25
Rel-Norm
60.73
-0.46
72.95
+0.21
LJAMA2-138
Mistral-77
Unteral M-20B
Naaheeire- Tigerbar- 13b
eill workk 1-8
13B
+1--8
Table 4: Ablation study of normalization on the relative representation matrix the ensembling performance on the development sets. Base- line refers to the best single model on each benchmark. DEEPEN refers the performance of ensembling top-2 models in the benchmark.
Individual
Figure 5: 2-model ensemble of the top-1 model (LLaMA2-13B) with different models on the NQ benchmark, respectively.
Ensemble of the generalist model and the specialist model. To investigate the effectiveness of DEEPEN on the ensemble of the generalist model and the specialist model for the specific task, we conduct experiments on the machine translation task using the ensemble of the large language model LLaMA2 and the machine translation model NLLB [27], which well-known open-source multilingual translator. We adopt the widely-used machine translation benchmark Flores-20 As the results in Tab. Billustrated, DEEPEN achieves better translation performance leveraging diverse translation knowledge in the generalist LLM and the specialist translator.
Ensemble of models with different performance gaps. To assess the stability of DEEPEN regarding to the performance gap of base models, we conduct an experiment on the ensemble of model pairs with increasing performance gaps. As the result demonstrated in Tab, S the performance of ensemble learning between a well-performin model (the rank-first model)with a worse-pent poorse-performing model could achieve improvements slightly lag behind the well cell-performing model.
5.2 Analysis on Relative Transformation
Effect of anchor selection. We demonstrate the impact of different numbers of anchor words through experiments with the top-2 ensemble models on the MMLU and ARC-C datasets. As shown in Fig. 4 increased number of anchor words improve performance for LLMS in downstream tasks, and selecting the full set of common words as anchors provides better performance.
Effect of normalization on relative representation matrix. To demonstrate the importance of normalization on the relative representation matrix the ensemble performance ($3.2), we conduct an ablation analysis. The result is shown in Tab, 4 the ensemble struggles to achieve improvements due the ineffective representation outlier words can be derived from the distribution of distance to nearest neighbor words, which is illustrated in Fig. 8 As illustrated, a remarkable proportion (> 30%) of words are distant from other words, i.e., cosine similarity to its nearest neighbor word is less than 0.3. Through the normalization operation, the output semantics that intend to emit outlier words could be prevented from becoming zero vectors by relative transformation.
outlier words, i.e., words distant to other words. The proportion of

---

5.3 Analysis of Reverse Transformation
To better understand the reverse transformation process ($3.4) transforming the relative representation back the absolute space of the main model, we further analyze each component of this process.
Analysis of relative ensemble learning rates. As shown in Tab. 5 the performance of DEEPEN sensitive the value of relative ensemble learning rate (n), which abbreviated by RELR. This observation motivates us to measure the generality of this hyperparameter. Specifically, we illustrate the cross-distribution performance of the searched optimal value of v Tab. 9 As observed, the optimal value of RELR varies across different datasets, which suggests that the inverse transformation from relative space absolute space requires adaptive mapping schemes.
Effect of iteration steps in relative ensemble learning. To give deep view of the dynamics of the inverse transformation in DEEPEN, we re- port the performance change along with different numbers of relative ensemble learning steps (7). Besides, the dynamics of loss of relative ensemble learning (1) in Eq. G also reported. As shown in Fig. 9 on the one hand, more steps of relative ensemble learning significantly lead to lower losses. However, the loss is hard to reach zero, i.e., under- fitting. On the other hand, increasing the number of steps of relative ensemble learning will cause the performance to increase first and then decrease. The reason behind the performance drop could be that in the early stage of optimization, the focus of optimization is on updating the high-probability tokens. In the later stage of optimization, since the probabilities all words will be adjusted equally, the high-probability tokens will be interfered with the high-probability ones, thus affecting the performance. Therefore, it recommended to set a modest value of step number (e.g., 5).
RELR (7)
0.05
0.10
0.15
0.20
0.25 0.30
MMLU
+2.42
+1.57
+1.77 +1.96
+1.31 +1.31
TriviaQA +1.31
+2.05
+1.63 +1.94 +1.82 +1.26
Table 5: Sensitivity analysis of relative ensem- ble learning rate (RELR). We report the im- provements of ensembling top-2 models over the best individual models.
Related Work
Selection- ensemble. Rerank is an intuitive solution utilize multi-model strengths. Jiang et al. [T5] take the first step towards LLM ensemble, training a reward model PAIRRANKER for pairwise comparison on candidate outputs. To overcome the huge computation costs of multi-LLM inference, several works have explored to train a ronter to predict the best pest-performing model out of a fixed set LLMS for the given input
25
Fusion-based ensemble. Towards a synergy between LLMs, Jiang et al. [15] propose GENFUSER. trained to combine multiple candidate answers. Different from these deening-deepedett ensemble methods which pose great challenge to the generalizability of the reward model fusion model, our DEEPEN is completely training-free, making more general. Similar to our method. MINED also aims tackle the vocabulary discrepancy via aligning the tokens in different vocabularies based on edit distance 30 D. Unfortunately, this textual similarity-based method exhibits unstable performance and produces abnormal text for L.LM ensemble (Tab. 7
There are several contemporaneous works related our work. Xu al. 333 propose EVA to tackle vocabulary discrepancy by learning token alignment between different vocabularies with the assistance of overlapping tokens. Our DEEPEN eliminates this training process via directly aligning tokens with the relative representation (more discussion is illustrated in B Mavromatis et al. [20] explore adaptive collaboration weights at test time by harnessing the perplexity on the input prompt. We emphasize that this work is complementary to work.
7 Conclusion
In this work, we propose a training-free LLM ensembling framework DEEPEN, which addresses the vocabulary discrepancy when fusing the probability distributions of heterogeneous L.MMs. Ex- perimental results on six widely-used benchmarks demonstrate that DEEPEN exhibits more stable

---

performance than baseline methods and has complementary strengths with other ensemble methods such as VOTING. We believe our work can inspire further research on the LLMss collaboration, model reuse, and knowledge distillation. In the future, we aim to explore more effective adaptive collaboration schemes to leverage the complementary strengths between different LLMs.
Acknowledgements
Xiaocheng Feng is the corresponding author of this work. We thank the anonymous reviewers for their insightful comments. This work was supported by the National Natural Science Foundation of China (NSFC) (U22B2059, grant62276078), the Key R&D Program of Heilongjiang via grant 2022ZX01A32, the International Cooperation Project PCL. PCL2022DO1 and the Fundamental Research Funds for the Central Universities (Grant No. NO.HHI..0CE..2223011)
References
[1] AI. Yi: Open foundation models by 01.ai, 2024.
[2] Z. Allen-Zhu and Y. Li. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning, 2020. URL.Ihtps://arxii.orr//// 2212 09816 2.2
[3] Y. Bisk, R. Zellers, R. Le bras, J. Gao, and Choi. Piqa: Reasoning about physi- cal commonsense in natural language. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 34(05):7432-7449, Apr. 2020. doi: 10. 10.1699aaai,.. www..aiiii..005333.. URL
https://ojs aaai wiwpr/gg/iii.nn/nn hhtp//AA//article/vieee 6239
[4] Y. Chen, W. Cai, L. Wu, X. Li, Z. Xin, and C. Fu. Tigerbot: An open multilingual multitask 11m, 2023. 4.1
[5] P. Clark, Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. 4.1
[6] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. 4.1
[7] X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma. A survey ensemble learning. Frontiers of Computer Science, 14:241-258. 2020.
[8] M. Freitag, B. Ghorbani, and P. Fernandes. Epsilon sampling rocks: Investigating sam- pling strategies minimum Bayes risk decoding for machine translation. H. Bouamor,
J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9198-9209 Singapore, Dec. 2023, Association for Computational Linguis- tics. doi: 10. 186518663v 1/2023 findings-emmlp.611.. URL findings-emnlp 617
[9] Y. Fu, H. Peng, L. Ou, A. Sabharwal, and T. Khot. Specializing smaller language models towards multi-step reasoning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10421-10430. PMLR, 23-29 Jul 2023. URL. http::ttt:/://oceeeiii
www.presssvveee wtt.:prrss////ff2e2ee2 html 4.1
[10] E. Garmash and C. Monz. Ensemble learning for multi-source neural machine translation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguisties: Technical Papers, pages 1409-1418, Osaka, Japan, Dec. 2016. The COLING 2016 Organizing Committee. URL https://ccaaatthooooyy wttpr//hooogyyooo////C-/-
2.2
[11] B. He and M. Ozay. Feature kernel distillation. In International Conference on Learning Representations, 2022. hoBllattteeeeoeeeeeeee//eeeee/ee
2.1
10

---

[12] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL http:://pwnnneeeeee..ooooorrrrno
4.1
[13] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D.S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, I.. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril,
T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023. [4.
[14] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, and et al. Mixtral of experts, 2024. 5.1
[15] D. Jiang, X. Ren, and B. Y. Lin. ILM--biendr: Ensembling large language models with pairwise ranking and generative fusion. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165-14178, Toronto, Canada, July 2023. Association for Computational Linguistics. doi:
1/6653///2023.ac// 4.1
URL
https:///ccaantholpgs org. 2023 acl-long. 792
1
6
6
[16] M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/
P17-1147. URL htttp:://aclannholpgg... htt.:/aaanthoooooooor//11111111 0r17
[17] S. Kumar and W. Byrne Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chupter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169-176, Boston, Massachusetts, USA, May 2 May 7 2004. Association for Computational Linguistics. URL https:///cclanthol..gg org orggeerr///e/////// NN04-
[18) T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, Polosukhin,
J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162/tacl_ a 200207060__000066....222 00276. URL htLhhtts::///cee /ttlttss////aaanoooloyy/ wtw.////////0// 1026
[19] K. Lu, H. Yuan, R. Lin, Lin, Z. Yuan, C. Zhou, and Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models, 2023. 6
[20] C. Mavromatis, P. Karypis, and G. Karypis. Pack of llms: Model fusion at test-time via perplexity optimization, 2024. [
[21] L. Moschella, V. Maiorca, M. Fumerr, A. Norelli, F. Locatello, and E. Rodolà. Relative representations enable zero-shot latent space communication. In The Eleventh International Conference on Learning Representations, 2023. URL
SrC-nwieG] 2.1 3.2
[22] OpenAI. Gpt-4 technical report, 2023.
[23] W. Park, D. Kim, Y. Lu, and M. Cho. Relational knowledge distillation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3967-3976, 2019. 2.1
[24] O. Sagi and L. Rokach. Ensemble learning: survey. WIREs Data Mining and Knowledge Discovery, 8(4)x 2018. doi: https://doi.or//// http://ddooorr//////// 1002/www 1249. URL https://wires onlinelibrary..wlley..u http://www..co//oo//////t//tette 1002/widm. 1249
[25] T. Shnitzer, A. Ou, M. Silva, K. Soule, Y. Sun, J. Solomon, N. Thompson. and M. Yurochkin. Large language model routing with benchmark datasets, 2024. URL httpp:://opereviiee net net/ForuumIdd-LyNsMWWLLL.
6
[26] Team. Internlm: A multilingual language model with progressively enhanced capabilities.
http://tttto..ommmeeemm/t/ttttet techheeett,tttteeeeeprrrrrro 2023, 4.1

---

[27] N. Team. No language left behind: Scaling human-centered machine translation, 2022. URL 04672 5.1
[28] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal,
H. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and
efficient foundation language models, 2023. [
[29] H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, and et al. Llama Open foundation and fine-tuned chat models, 2023.
5.1
[30] F. Wan, X. Huang, D. Cai, X. Quan, W. Bi. and S. Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
2qcz 4.1
[3]] H. Wang. F. M. Polo, Y.
Sun, S. Kundu, E. Xing, and M. Yurochkin. Fusing models with complementary expertise. In The Twelfth International Conference on Learning Representations. 2024. URL http::://ppeeeeeviiii
net weet/forummid--hrrrGG
6
[32] T. Wei, L.. Zhao, I.. Zhang, Zhu, Wang, and el. Skywork: more open bilingual foundation model, 2023. 4.1
[33] Y. Xu, Lu, and J. Zhang. Bridging the gap between different vocabularies for Ilm ensemble, 2024.
B
[34] C. Zhang and Ma. Ensemble Machine Learning: Methods and Applications. Springer Publishing Company, Incorporated, 2012. ISBN 1441993258. 4.
[35] W. X. Zhao, K. Zhou, J. Li, Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang. Z. Dong.
Y. Du, C. Yang. Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.- Y. Nic, and
J.RR. Wen. survey of large language models, 2023.
[36] Z.-H. Zhou. Ensemble methods: foundations and algorithms. CRC press, 2012.
12

---

Mistral-7B
InternLM--20B
Skywork-13B
LLaMA2-13B
Yi-6B
Tigerbot-138
Mistral-7B
32,000
26,759
24,983
24,184
24,360
25,121
InternLM-220B
26,759
103,168
41,204
22,566
50,362
44,885
Skywork-13B
24,983
41,204
63,519
32,000
33,646
49,693
LLaMA2-13B
24,184
22,566
32,000
32,000
20,301
32,000
Yi-6B
24,360
50,362
33,646
20,301
64,000
39,360
Tigerbot-13B
25,121
44,885
49,693
32,000
39,360
60,515
Figure Statistics of common words across different vocabularies.
A Statistics of Common Tokens across different LLMs
We count the number of common tokens shared among different LLM vocabularies and present the results in Fig. It is observed that large number of common words (>20k) exist across the different vocabularies. We also count the number of common tokens in all six L.MS and find that there are total of 18k common tokens, enabling DEEPEN to be applied to the ensemble learning of large number models.
B Details of Baselines
LLM-BLENDER. (1) the selection-bar based ensemble method PAIRRANKER Jiang et al. [5], which reward model to score each response of LLMs and (2) the fusion-based ensemble method GENFUSER Jiang et al. [15), which is a generative model to fuse multiple candidate responses. Both models trained on the constructed instruction tuning dataset MixInstruct. In experi- ments, as GENFUSER struggles to generate responses following the expected format, we only adopt PAIRRANKER.
For tasks with outputs limited fixed set (i.e. MMLU, ARC-C, PIQA, GSM8K benchmarks), we adopt the VOTING method on the ensemble learning of more than models. Concretely, we count each candidate answer's occurrences and select the most frequent the final output. In the event of tie, the main model's answer is used as the final output.
VOTING.
MBR. For generation tasks, we implement the MBR 8 method, which selects the answer with the highest lexical similarity other candidate answers. To measure this similarity, we experimented with the edit distance and chr metrics, ultimately choosing chrF due to its superior performance.
MINED. To bridge the gap between different vocabularies in LLM ensemble, MINED apply the Minimum Edit Distance (MinED) approach align tokens across different vocabularies, e.g., "get" to "gget"". However, this textual similarity-base -baed mapping method could disturb the text generation process and produce incomplete words.
EVA. Recently, Xu et al. 33 propose EVA tackle the vocabulary discrepancy by learning mappings between the vocabularies of different LLMs with the assistance of overlapping tokens. We have tried to re-implement their method with the released code. However, we encounter a technical problem in that EVA only supports the ensemble learning between LLMs with the same embedding dimension. This caused by the limitation of tool of vecmar which is used learn the token alignment.
13

---

Produce Incomplete Words Error Analysis Generation Process of MinED
Which Lloyd Webber musical premiered in the US 10th December 1993?
which American state is the Isabella. Stewart Gardner Muscum?
Question
Golden Answer
Sunset Boulevard
Mussachusetts
MinED Answer
unmasked
assachusetts
DoePEo Answer
Sunset Boulevard
Massachusstts
Top-10 tokens output by Main model
PMC 'B; 'The, 'Is,, 'MA, 'Inn, 'New,, Massachuserts' "Conncer]]
[S: Phh, "The', 'Wh,, IC, TT 'Asl, MJose, 'B", 'Stee]
Top-10 tokens output by Assistant Model
('Mass,, MM, 'New'. Massaahussttss" 'Inl CCnnne' (c0x0A-]]
['Sunt, 'Wh,, "The', 'Ph!, IC) 'S, 'Sch'' VT 'Ev, [As]
Mapped Top-10 10p-10 tokers of Assistant Model
l'usl, 'Whl 'The', 'C. 's., 'Sah', VV. As]]
Taass 'B., 'M. 'MA', 'New'' MMssaachssstts 'hhe' 'In", CCnnne' (00x0A0]]
Averaged Top-10 Tokens
[unl 1S', Jase')
V. Josepht,
'C.
Iass 'M', 'B', 'MA! 'New.. _Massaduusttss' 1s' In]]
Disturb Decoding
Figure 7: Analysis of the generation process of MINED. To illustrate the problematic generation process MINED, we list the top-10 10 high-probabillty tokens in the probability distribution of the assistant model and their aligned token.
MMLU-Dev
ARC-C-Dev
Models
INDIV
DEEPEN
INDIV
DEEPEN
Yi-6B
61.19
63.61 (+2.42)
72.72
77.55 (+4.83)
Mistral-7B
60.80
64.46 (+3.66)
73.88
77.73 (+3.85)
Table 6: Performance of DREPEN with choosing different main models on the development sets. INDIV refers to individual models. The result of DeePEn indicates the performance of using the model of this row the main model.
C
Additional Experiments
C.1 Choice of main model.
In the process of inverse transformation, DEEPEN maps the relative aggregated representation to the absolute space of the main model. Idcally, we expected the results of inverse transformation to keep invariant with the choice the main model. However, this objective is hard to achieve due to the underfitting observed in the search process. Therefore, illustrate the performance variance of choosing different main models Tab. 6 As the results shown on ARC-C, changing the main model from the first-ranked Mistral-7B the second-rank Yi-6B, the ensemble performance is decreased slightly from 77.73 to 77.55. Interestingly, changing the main model from the rank-1 Yi-6B to the rank-2 Mistral-7B on MMLU, the performance actually improved from 63.63 to 64.46, which indicates that Mistral-7B benefits more than Yi-6B from collaboration. Even so, choosing different main models does significantly affects the ensemble performance.
C.2 Comparison Vanilla Prediction Average
To compare our DEEPEN with vanilla prediction average, we conduct an experiment for ensembling two LLMs with the same vocabulary and comparable performance on MMLU, i.e., LLaMA2-7B and LLaMA1- 1AMMA113BB 13B. As shown in Tab. [7] the performance of DEEPEN is comparable, even better than, that of the vanilla prediction average. Theoretically, the performance the vanilla prediction average is the performance upper-bound of DEEPEN. The reason that DEEPEN could excel over the vanilla one on MMLU the anter-fitting in the inverse transformation process, which leads to the weights to aggregate the output semantics of different models not being a uniform distribution (i.e., 0..5...5)).
14

---

MMLU-Dev
MMLU--Tst
Models
INDIV VANIL DEEPEN
INDIV VANIL DEEPEN
LLaMAI-13B 43.26
44.37
43.70
44.22
45.48
45.01
MaMM22-BB 42.28
45.94
42.99
45.31
Table Comparison to vanilla prediction average (VANIL) on the ensemble of LLMs with the same vocabulary.
Yi-6B
Mistrul-7B3
0.3
0.31
0.31
0.28 0.29
Percentage
0.21
0.22
0.2
0.13
0.14
0.1
0.07
0.01
0.0
0.1
0.2
0.3
0.4
0.5
1.0
0.0
0.1
0.2
0.3
0.4
0.5
1.0
Cosine Similarity
Figure 8: Distance distribution nearest neighbor words. The distance is measured by calculating the cosine similarity between words.
For example, in Tab. 7 the weights for LLaMA1 and LLaMA2 could (0.6, 0.4), where the weight of the main model is larger than the other model.
C.3 Latency Analysis
To accomplish the fusion of heterogeneous distributions, DEEPEN first maps the distributions into the relative space and adopts the search-base based inverse transformation to map the aggregated relative representation back the main model's probability distribution, which incurs an extra latency. This latency is mainly caused by the inverse transformation process, which requires T-round search. To demonstrate this latency, we report the token-level inference latency of ensembling two LLMs (Mixtral-8>x > 7b and LLaMA2-70B), This experiment is conducted on 8 A100 GPUs. All of experiments can be re-implemented on A100 GPUs. As shown in Tab. g DEEPEN causes +1 +17% token-level inference latency. However, in practice, this latency could be greatly decreased since all individual models intend to emit the same token in 90% decoding steps. In these steps, we could skip the fusion process and the consistently agreed token the next token. In total, DEEPEN actually incurs less than 2% sentence-level inference latency.
Bascline T =
T
3
T
=
5
T 10
Inference Latency
0.19s
0.20s
0.21s
0.22s
0.24s
Relative Change
+7%
+11%
+17%
+29%
Table 8: Inference Latency of DEEPEN with different search steps T.
D Limitations
As illustrated in Tab. I collaboration with more LLMs can sometimes lead to performance drop caused by interference from lower-performing models. This issue limits the ensemble performance of our current method, even though we have explored setting different collaboration weights for each model each benchmark (DRePEN-Adap), weights at the sample level, or the token level, for each LLM, which remains a significant
ideal solution would be set adaptive collaboration
15

---

Baseline
TrivaQA
NQ
ARC-C MMLU
TriviaQA
73.42
75.9
75.41
75.56
75.44
NQ
29.11
30.55
30.65
30.42
30.69
ARC-C
60.29
69.32
72.31
74.19
73.76
MMLU
54.06
59.97
61.04
61.94
61.42
Table Cross-distributioo validation of relative ensemble learning rate (17). We report the perfor- mance of ensembling LLAMA2-13B and Mistral-7B. Each row indicates the test set used to evaluate performance. Each column indicates the development set used to search the optimal value r).
2.3e-6
Ensemble Learning
63.61
Loss of Relative
1.8e-6
Loss
63.25
Accuracy
ACC
1.3e-6
62.50
8.7e-7
61.75
4.0e-7
61.19
0
10
Relative Ensemble Learning Steps (
(T)
Figure Effect of different number of relative ensemble learning steps.
challenge. Despite this, our work represents an important step towards the distribution fusion of LLMs.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper proposes DEEPEN, a training-free ensemble framework that en- ables collaboration between large language models with heterogeneous vocabularies by averaging their probability distributions. The key innovation transforming the probability distributions from each model's vocabulary space into shared "relative representation" space based on embedding similarities to anchor tokens. The aggregated relative representa- tion is then mapped back to the vocabulary space of one LLM to determine the generated token.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
17

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The method proposed in this work is based on the cross-mode invariance of relative representation, which is described in the 'Introduction'.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
All the information of our experiments is illustrated in $4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
18

---

some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code will be submitted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce public/guiddss//ddeeuubbe ssiololiccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (ht //niss.ccppbbl. wwtpcccccccccpeeececeecccc Seggidde//ooSSummi wwt.suumisssiiiiiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the test details are described in 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since our work does not involve with model training, we have not conducted the error statistics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
As mentioned in CC3 all of our experiments conducted on 8 A100 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httts:://eerrrssscccoooo////tttt
Answer: [Yes]
Justification:
Our work conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our research motivates the research of prediction fusion and model fusion via tackling the vocabulary discrepancy. There is negative societal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer:[Yes
Justification:
All resource used in our work are from open source community.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paper thccode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
thcode.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not involved
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "edFT4xjpSS", "reviewer": "Reviewer_c5c3", "summary": "The paper introduces a method that maps output distributions of different LLMs to and from a universal relative space to aggregate them, based on which the next token is determined.", "questions": "- Figure 1: what do the three colors (blue, yellow, purple) indicate?\n- Table 1: Why did the authors not report DEEPEN-Adapt with Top-2 Ensembles? (DEEPEN-Adapt outperforms DEEPEN-Avg on all benchmark tasks with Top-4 Ensembles, so why is this not the default setting?)\n- Figure 3: How do the baseline methods perform with an increasing number of ensemble members considered?\n- Table 2 & 3: How do the baseline methods perform with the dense/sparse models and the generalist/specialist models?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is structured well and written clearly.\n- It proposes a novel method of ensembling the heterogeneous output distributions of LLMs.\n- The method is mathematically well-motivated.\n- It shows good empirical performance and ablation studies give valuable insights.", "weaknesses": "- **Presentation**: Although the paper is well-structured, some passages could be condensed. For instance, the paragraphs \"Anchor Selection\" and \"Normalization of relative representation matrix\" simply repeat what has already been stated in the general paragraph above. Also, sections 3.3 and 3.5 both consider the aggregation of relative representations, currently separated by section 3.4 which considers the inverse transformation. Combining those conceptually the same topics would improve the reading flow.\n- **Evaluation**: Although the paper provides numerous ablation studies, the method is only compared against the baselines in the main experiments (Tab. 1). Here, the method outperforms the baseline in (only) 7/12 settings. Also comparing to the baselines in the other experiments (see questions to Fig. 3. and Tab. 2-3 below) would provide further insights into the performance of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your constructive feedback. We appreciate the opportunity to address your comments.\n\n&nbsp;\n\n**Question-1: Comparison between DeePEn with other ensemble methods in experiments beyond the main experiment.**\n\nThanks for your insightful suggestion! We have followed your advice to supplement the results of the baseline ensemble methods (MinED, LLM-Blender, and Voting/MBR) in Fig.3  and Tab.2&3:\n\n**Figure-3** (Ensemble learning on various number of models):\n\n| Number of Models | Model Set     | Individual | LLM-Blender | MinED | Voting/MBR | DeePEn-Adapt (Ours) |\n| :--------------: | ------------- | :----------: | :-----------: | :-----: | :----------: | :-------------: |\n|1| LLaMA2-13B    |28.67|28.67|28.67|28.67|28.67|\n|2| +Mistral-7B |27.62|28.61| 28.45 |28.67| **30.65**|\n|3| +InternLM-20B |26.09|26.62|27.20| 30.06| **31.36**|\n|4| +Tigerbot-13B |22.71| 24.24| 29.50  | 30.28| **31.77**|\n|5| +Nanbeige-16B | 22.77| 22.63| 30.22 | 30.94| **31.02**|\n|6| +Skywork-13B  | 19.97| 22.71| 30.44 | 30.47| **31.16**|\n|7| +Yi-6B| 18.98| 21.25| 29.97 | 30.33| **30.50**|\n\nPlease note that due to time limitations, we have only supplemented the results of baselines on NQ. The results on the other benchmarks (MMLU and PIQA) will be included in the next version.\n\n**Table-2** (Ensemble learning between the dense and sparse models):\n\n| Model                   | GSM8K             | PIQA              |\n| ----------------------- | ----------------- | ----------------- |\n| LLaMA2-70B (*Dense*)    | 63.84             | 71.27             |\n| Mixtral-8×7B (*Sparse*) | 65.73             | 71.88             |\n| LLM-Blender             | 64.52 (-1.21)     | 74.54 (+2.66)     |\n| MinED                   | 67.10 (+1.37)     | **75.65 (+3.77)** |\n| DeePEn (Ours)           | **67.33 (+1.60)** | 75.10 (+3.22)     |\n\n**Table-3** (Ensemble learning between the generalist/specialist models):\n\n| Model                     | En→De             | De→En             | En→Ro             | Ro→En             |\n| ------------------------- | ----------------- | ----------------- | ----------------- | ----------------- |\n| LLaMA2-13B (*Generalist*) | 30.60             | 42.27             | 30.83             | 39.99             |\n| NLLB-600M (*Specialist*)  | 32.30             | 41.49             | 31.91             | 42.39             |\n| LLM-Blender               | 33.26 (+0.96)     | 43.28 (+1.01)     | **33.17 (+1.26)** | 41.99 (-0.40)     |\n| MinED                     | 27.12 (-5.18)     | 36.83 (-5.44)     | 29.91 (-2.00)     | 34.39 (-8.00)     |\n| DeePEn (Ours)             | **33.34 (+1.04)** | **43.70 (+1.43)** | 32.95 (+1.04)     | **42.84 (+0.45)** |\n\nAs the results show, unlike DeePEn, the **baseline ensemble methods perform quite unstably across various settings**. Specifically, MinED results in dramatic performance drops when ensembling the generalist LLaMA2-13B and the specialist NLLB-600B due to their large vocabulary divergence. LLM-Blender also leads to significant performance drops on NQ, GSM8K, and Ro→En translation due to its limited generalizability.\n\nWe will add these results in our paper.\n\n&nbsp;\n\n**Question-2: Results of DeePEn-Adapt in Top-2 Ensembles?**\n\nThanks for your important feedback! We have followed your advice to supplement the results of DeePEn-Adapt in the Top-2 ensembles of Tab.1:\n\n|| MMLU| ARC-C| GSM8K| PIQA| TriviaQA| NQ|\n| ------------ | ----------------- | ----------------- | ----------------- | ----------------- | ----------------- | ----------------- |\n| LLM-Blender   | 63.85 (+0.60)     | 75.73 (-0.08)     | 54.89 (+0.99)     | 78.31 (+2.16)     | 74.10 (-0.22)     | 28.61 (-0.06)     |\n| MinED        | **65.04 (+1.79)** | 77.35 (+1.54)     | 18.50 (-35.40)    | 78.98 (+2.83)     | 72.30 (-2.02)     | 28.45 (-0.22)     |\n| DeePEn-Avg   | 64.68 (+1.43)     | **77.52 (+1.71)** | 55.42 (+1.52)     | 78.87 (+2.72)     | 75.90 (+1.58)     | 30.17 (+1.50)     |\n| DeePEn-Adapt | 64.41 (+1.16)     | **77.52 (+1.71)** | **55.65 (+1.75)** | **79.37 (+3.22)** | **76.08 (+1.76)** | **30.65 (+1.98)** |\n\nWe will add these results to our manuscript.\n\n&nbsp;\n\n**Question-3: Why was the DeePEn-Adapt not the default setting in the top-2 ensembles?**\n\nOur original intention of devising DeePEn-Adapt was to alleviate the interference of lower-ranked models to the higher-ranked models. Therefore, we only tested DeePEn-Adapt in the top-4 model ensemble, where the 4th model could cause serious interference. \n\nThanks for your valuable comment. In the future, we will set the DeePEn-Adapt as the default setting of DeePEn.\n\n&nbsp;\n\n**Question-4: What do the three colors (blue, yellow, purple) indicate in Fig.1?**\n\nSorry for the unclarity. **Different colors indicate different clusters of samples**. \n\nTo clearly illustrate that relative representations remain consistent across different models, we performed K-means to cluster the sampled representations and set different colors for different clusters. \n\n&nbsp;\n\n**Suggestion-1: Some passages could be condensed.**\n\nThanks for your valuable feedback. **We will condense our manuscript in the future version.**\n\nIn the original version of the paper,  the general paragraph of section 3.2 primarily describes the **implementation** of constructing relative transformation, including the \"Anchor Selection\" and \"Normalization of relative representation matrix\".\n\nIn contrast, the specific paragraphs of  \"Anchor Selection\" and \"Normalization of relative representation matrix\" explain the **motivations** behind our implementations.\n\n&nbsp;\n\n**Suggestion-2:  Sections 3.3 and 3.5 both consider the aggregation of relative representations, currently separated by section 3.4.**\n\nThanks for your valuable feedback. We will merge section 3.5 into 3.3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M1lVTznHbL", "reviewer": "Reviewer_Q9GV", "summary": "The paper introduces DEEPEN, a novel training-free ensemble framework designed to leverage the complementary strengths of various large language models (LLMs). The key innovation of DEEPEN lies in its ability to fuse informative probability distributions from different LLMs at each decoding step, addressing the challenge of vocabulary discrepancies between heterogeneous LLMs.\n\nDEEPEN operates by transforming the probability distributions from each model's vocabulary space into a shared \"relative representation\" space based on the relative representation theory, which uses embedding similarities to a set of anchor tokens. The aggregated relative representations are then mapped back to the vocabulary space of one LLM (the main model) to determine the generated token.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper introduces DEEPEN, a novel ensemble learning framework that enables collaboration among heterogeneous large language models without the need for additional training. A significant contribution is the solution to the vocabulary mismatch problem between different LLMs, allowing for more effective fusion of their outputs.\n2) The paper includes extensive experiments across six benchmarks, demonstrating DEEPEN's consistent improvements in performance and its complementary strengths with other ensemble methods.\n3) DEEPEN demonstrates better stability compared to baseline methods, which struggle with generalization to unseen data distributions.\n4) The framework is open to further extensions and improvements, such as the development of more sophisticated methods for determining collaboration weights.", "weaknesses": "1) The paper does not provide a detailed analysis of how well DEEPEN generalizes to unseen data or across different types of tasks beyond the evaluated benchmarks.\n2) The method's performance is sensitive to the choice of hyperparameters, such as the relative ensemble learning rate. Finding optimal hyperparameters may require additional tuning and may not be straightforward.\n3) The performance of DEEPEN relies heavily on the selection of anchor words. The paper does not thoroughly explore the impact of different anchor word selection strategies on the ensemble performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments, which have greatly helped us improve our paper. We appreciate the opportunity to discuss and address your concerns.\n\n&nbsp;\n\n**Question-1: How well does DeePEn generalize to unseen data or across different types of tasks beyond the evaluated benchmarks?**\n\nWe understand your concern regarding the generalizability of DeePEn, which relies on a development set for each task to find the optimal relative ensemble learning rate. \n\nActually, **we analyzed the generalizability of DeePEn through *cross-distribution validation* and found that DeePEn generalizes well, which is shown in Tab.9 of the paper.**\n\nConcretely, we tested the performance of DeePEn on task $A$ with the hyperparameter found with the development set of task $B$. We totally consider 4 tasks: 2 generative knowledge QA tasks (NQ and TriviaQA), 2 multi-choice human examination tasks (MMLU and ARC-C). \n\nThe results show that the performance improvement using a different-task development set ($A \\neq B$) is **88%** of the improvement when using a same-task development set  ($A = B$). Moreover, using a similar-task development set, the performance improvement is **95%** of that achieved using a same-task development set.\n\nThanks for your important feedback. We will demonstrate this important analysis more clearly by modifying Tab.9 and placing it in the main document instead of the appendix.\n\n&nbsp;\n\n**Question-2: Finding optimal hyperparameters may require additional tuning.**\n\nWe acknowledge that our current method requires tuning to find the optimal relative ensemble learning rate for effective inverse mapping from the relative space to the absolute space. As the first attempt to address this challenge, we have achieved significant success. In the future, we plan to explore methods to achieve this inverse mapping without the need for extensive tuning.\n\n&nbsp;\n\n**Question-3: What impact do different anchor word selection strategies have on the ensemble performance?**\n\nThanks for your constructive suggestion. We had ever devised an anchor selection algorithm **AS-MRRC**  (Anchor Selection with Maximum Relative Representation Consistency). AS-MRRC aims to **infer the optimal anchor words** via maximizing the relative representation similarity of common words across different models:\n\n$A^* = \\mathop{argmax}\\limits_{A \\in C} \\ \\mathop{\\mathbb{E}}\\limits_{i\\in C} \\ cos(\\hat{R}_1(i|A), \\hat{R}_2(i|A)),$\n\nwhere $C$ is the common word set between different LLMs, $\\hat{R}_1(i|A)$ and $\\hat{R}_2(i|A)$ are the relative representations of word $i$ in different models using $A$ as anchor words, and $cos(,)$ refers to the cosine similarity function.\n\nWe compared AS-MRRC with the random selection of anchor words and the method of using all common words as anchors:\n\n| Method                   | Number of Anchors | Performance (ARC-C-Dev) |\n| ------------------------ | :---------------: | :---------------------: |\n| Random Selection         |        16k        |          76.88          |\n| Full Set of Common Words |        24k        |        **77.64**        |\n| AS-MRRC                  |        13k        |          77.43          |\n\nAs the result shows, our AS-MRRC outperforms random selection using fewer anchors while underperforming the method of using the full set of common words. **Therefore, we decided not to report this thankless method of AS-MRRC**.\n\nIn the future version, we will follow your advice and include this trial to help readers better understand the impact of different anchor word selection strategies. We are also going to explore anchor word selection strategies more thoroughly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHbyQiUgLQ", "reviewer": "Reviewer_JJTA", "summary": "The authors propose to use relative representations to unify the tokenization across different models for an effective ensemble. They first transform the prediction of each model to a relative space. Then, after averaging the relative predictions, they perform a gradient-based optimization to find the averaged prediction in the original space.", "questions": "1. Have you tried other normalization methods other than softmax?", "limitations": "Discussed in Appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose an interesting idea for dealing with different tokenization.\n2. The authors provide detailed analysis such as learning rate and the number of steps for the search step.", "weaknesses": "1. The proposed method requires gradient updates to project back to the original space at every generation step. This can cause extra overhead.\n2. The performance largely depend on the number of common anchor words between different LLMs (Figure 4). As a result, the method may be not effective when the domain of pretrained LLMs are very different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments, which greatly help us improve our paper. We are glad to have this discussion to address your concerns.\n\n&nbsp;\n\n**Concern-1: The proposed method requires gradient updates to project back to the original space at every generation step. This can cause extra overhead.**\n\nThanks for your insightful comment. We understand your concern considering the overhead of the reverse transformation process from the relative space to the absolute one. To address this concern, we quantitatively analyzed the overhead of this process in Appendix C.3.\n\n**Overall, DeePEn actually incurs $\\leq $2% inference latency in practice. This conclusion also applies to the overhead.**\n\n&nbsp;\n\n**Concern-2: As a result, the method may not be effective when the domain of pretrained LLMs are very different.**\n\nThanks for your insightful comment. We understand your concern regarding the number of common words between LLMs with quite different vocabularies. To address this concern, we analyzed the number of common words across different LLMs in Appendix. A (Fig. 6). Through case studies, we found that a large number of common words (>20k) exist across the different vocabularies since **LLMs (including LLMs tailored for specific domains) usually contain a lot of common English basic tokens.** This fact provides a solid guarantee for our DeePEn to work effectively.\n\n&nbsp;\n\n**Question-1: other normalization methods other than softmax?**\n\nThanks for your constructive suggestion. We follow your advice to experiment with an alternative normalization method of Softmax. Concretely, we replace softmax with Min-Max Scaling:\n\n$ x{\\prime} = \\frac{x - x_{min}}{x_{max} - x_{min}} $\n\nThe result shows that the normalization with softmax is better than the one with Min-Max Scaling:\n\n| Methods                                   | MMLU-Dev  | TriviaQA-Dev |\n| ----------------------------------------- | --------- | ------------ |\n| Baseline                                  | 61.19     | 72.74        |\n| DeePEn w/o. Normalization                 | 60.73     | 72.95        |\n| DeePEn w. Normalization (Softmax)         | **63.61** | **74.79**    |\n| DeePEn w. Normalization (Min-Max Scaling) | 60.67     | 73.55        |\n\nWe argue that Softmax pays more attention to the nearest neighbor words when modeling the structural relation (relative representation) than Min-Max Scaling, which could be more beneficial.\n\nWe will add this analysis to our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents DEEPEN, an ensemble learning framework for heterogeneous LLMs that uses a novel training-free method to address vocabulary discrepancies across different models. This method translates probability distributions from individual LLMs into a shared “relative representation” space, facilitating aggregation and synthesis back into coherent output. 

In addition to the innovative treatment of the token alignment problem inherent in multi-LLM integration, the proposed method performs well across multiple benchmarks. I agree with the reviewers that the strong theoretical grounding and the broad range of empirical validation are the strong points of this paper. The paper is also well-written and easy to understand without sacrificing details. The reviewers and I also appreciate the authors’ strong response in addressing the concerns around benchmark comparisons and methodological clarification during the rebuttal.

Even though I recommend acceptance for this paper, I recommend the authors pay attention to the weaknesses pointed out by the reviewers, especially regarding the depth of analysis concerning the generalization of DEEPEN across varied tasks and its sensitivity to hyperparameter settings.
Decision: Accept (spotlight)