Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration

Authors: yichong  huang
xiaocheng  feng
baohang  li
yang  xiang
hui  wang
ting  liu
bing  qin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:35:24

Abstract: Large language models (LLMs) exhibit complementary strengths in various tasks, motivating the research of LLM ensembling.
However, existing work focuses on training an extra reward model or fusion model to select or combine all candidate answers, posing a great challenge to the generalization on unseen data distributions.
Besides, prior methods use textual responses as communication media, ignoring the valuable information in the internal representations.
In this work, we propose a training-free ensemble framework \textsc{DeePEn}, fusing the informative probability distributions yielded by different LLMs at each decoding step.
Unfortunately, the vocabulary discrepancy between heterogeneous LLMs directly makes averaging the distributions unfeasible due to the token misalignment.
To address this challenge, \textsc{DeePEn} maps the probability distribution of each model from its own probability space to a universal \textit{relative space} based on the relative representation theory, and performs aggregation.
Next, we devise a search-based inverse transformation to transform the aggregated result back to the probability space of one of the ensembling LLMs (main model), in order to determine the next token.
We conduct extensive experiments on ensembles of different number of LLMs, ensembles of LLMs with different architectures, and ensembles between the LLM and the specialist model.
Experimental results show that (i) \textsc{DeePEn} achieves consistent improvements across six benchmarks covering subject examination, reasoning, and knowledge, (ii) a well-performing specialist model can benefit from a less effective LLM through distribution fusion, and (iii) \textsc{DeePEn} has complementary strengths with other ensemble methods such as voting.

Main paper: 
 introduction: 
 With the scaling of model capacities and data volumes, generative large language models (LLMs) have shown impressive language understanding and generation abilities, shedding light for artificial general intelligence [35, 22, 13, 28] . Due to diversities of data sources, model architectures, and training recipes, LLMs have different strengths and weaknesses in various tasks and cases. Therefore, recent research has explored the ensemble of LLMs to exploit the complementary potential [15, 19] .Existing methods can be categorized into selection-based and fusion-based ensembling. Selectionbased ensembling selects the best candidate answer from all individual LLMs' answers using an additionally trained reward model [15, 31, 25, 19] . Fusion-based ensembling combines all candidate answers using a trained fusion model [15] . However, these approaches inevitably face significant challenges in generalizing to unseen data distributions and base models. Besides, prior methods enable collaboration via conveying the textual responses between LLMs while ignoring the rich information (e.g., confidence and alternative answers) in the internal representations.An ideal solution to this issue is to apply the well-established technology of prediction fusion. [36, 24, 7, 10] . For LLM ensemble, prediction fusion works at each decoding step, averaging the probability distributions from different LLMs to determine the next token. It could not only directly apply to the ensemble of any LLMs without extra parameter training, making it more general, but leverages the informative internal representations (i.e., probability distributions) as communication media. Unfortunately, the vocabulary discrepancy between different LLMs makes it unfeasible to average the distributions due to token misalignment.In this work, we tackle this key challenge by drawing upon the cross-model invariance of relative representation, which represents each token using the embedding similarities of this token to a set of anchor tokens [21] . Specifically, we propose an ensemble framework DEEPEN (Deep Parallel Ensemble), enabling distribution fusion for heterogeneous LLMs. DEEPEN transforms the probability distribution from the heterogeneous probability space to a homogeneous relative space, using a matrix formed by the relative representation of all tokens. Next, DEEPEN aggregates the relative representations of all probability distributions in the relative space, coordinating the decision on the next token. Finally, the result of aggregation is transformed back to the probability space of the main model using a search-based inverse transformation to determine the next token.We conduct extensive experiments ranging from 2-model to 9-model ensembles, covering ensembles of models with parameters ranging from 6B to 70B, ensembles of dense and sparse models, and the ensemble of LLMs with specialist models. Experimental results on six widely-used benchmarks demonstrate that compared to baselines, DEEPEN achieves consistent improvements across all benchmarks. It is also discovered that DEEPEN has complementary strengths when combined with other ensemble methods.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: This paper proposes DEEPEN, a training-free ensemble framework that enables collaboration between large language models with heterogeneous vocabularies by averaging their probability distributions. The key innovation is transforming the probability distributions from each model's vocabulary space into a shared "relative representation" space based on embedding similarities to anchor tokens. The aggregated relative representation is then mapped back to the vocabulary space of one LLM to determine the generated token.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
theoretical analysis: 
 We first introduce relative representation and then illustrate the theoretical support for our method.Previous study discovers that despite the misalignment between latent spaces of different neural networks, the embedding similarity between samples do not change across models [21, 11, 23] . Specifically, Moschella et al. [21] propose relative representation, which represents each sample x (i) by the embedding similarities to a set of anchor samples A (x (i) and A are identically distributed): r x (i) = (cos(e x (i) , e a (1) ), ..., cos(e x (i) , e a (|A|) )),where e ( * ) denotes the embedding of samples, also is absolute representation.It is empirically evidenced that relative representations possess cross-model invariance, i.e., the relative representation of the same sample keeps invariant across different models, which lays the theoretical foundation for our work to fuse heterogeneous probability distributions.Average probability distribution has been widely evidenced to effectively improve the predictive performance in the filed of image and text [2, 10] . For generative language models, as we understand, the underlying mechanism is to interpolate different output semantics represented by the probability distributions. However, for LLM ensemble, vocabulary discrepancy isolates these output semantics in semantic spaces with different basis vectors, making the interpolation infeasible. To tackle this challenge, we aim to enable the cross-model alignment for output semantics, i.e., find a transformation to map the output semantics into a universal space. To this effect, we propose to represent the output semantics with the convex combination of relative representations of all tokens where the weight is the probability assigned to the token. and the relative representation of the output semantics p is defined as:r = p • R.Model-invariance of relative representation of output semantic. Next, we illustrate why this representation scheme could align the output semantics isolated in heterogeneous absolute spaces. First, considering two LLMs θ A and θ B with the same vocabulary (e.g., LLaMA2-7B and LLaMA2-13B).When expressing the same output semantic, these models output the same probability distribution (i.e., absolute representation) p A and p B . Besides, they have the same (highly similar in practice) relative representation matrix due the vocabulary consistency and cross-model invariance of relative representation. Therefore, the relative representations of output semantics are also identical:EQUATIONThen, let's consider a language model θ C with a different vocabulary (e.g., Mistral). Based on the fact that different LLMs typically share mass tokens in their vocabularies ( §A), the vocabulary of model θ C is identical to adding and removing partial tokens to the vocabulary of θ B , which leads to p B ≇ p C and R B ≇ R C . However, in our study, we discover that this change to the vocabulary has not incurred significant influence on the relative representation of the unchanged tokens (i.e., the common tokens between θ B and θ C ), as shown in Fig. 1 . Therefore, we make the reasonable assumption that the local change in the vocabulary could hardly influence the relative space.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have discussed the limitations of our work in §D. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
methodology: 
 In this section, we first introduce the overall process of our ensemble framework DEEPEN and then describe the three parts of DEEPEN in detail.We illustrate the process of DEEPEN in Fig. 2 . Given N models to ensemble, DEEPEN first constructs their transformation matrices (i.e., relative representation matrices) mapping the probability distributions from the heterogeneous absolute spaces into the relative space ( §3.2). At each decoding step, all models perform prediction and output N probability distributions. These distributions are mapped into the relative space and aggregated ( §3.3). Finally, the aggregation result is transformed back into the absolute space of the main model, in order to determine the next token ( §3.4).Relative Space Absolute Space (LLaMA)Emitted Token Given N models to ensemble, DEEPEN first finds out the intersection of vocabularies of all models, i.e., common token set C, and samples a subset or uses the full set of common tokens as the anchor token set A ⊆ C. Next, for each model, DEEPEN calculates embedding similarities of each token to the anchor words, obtaining the relative representation matrix R (as shown in Eq.2). Finally, to overcome the relative representation degeneration of outlier words, which will be introduced later, we perform normalization on the relative representation of all tokens by a softmax operation so that it becomes a probability distribution. We denote the normalized representation matrix R:EQUATIONAnchor Selection. The choice of anchor tokens is crucial for the relative representation capability. Previous research discovers that the capability improves as the number of anchor words increases [21] . Therefore, we employ the full set of common words between LLMs as the anchor words. It is also empirically proved that this method performs more stablely on downstream tasks ( §5.2).Normalization of relative representation matrix. In DEEPEN, the relative representation of each token is normalized by the softmax operation to avoid the relative representation degeneration of outlier words, which are referred to as words that are far away from other words (including the anchors) and become distinguishable in relative space since for being zero vectors. The softmax operation effectively resolves this problem by making each relative representation a probabilistic distribution instead of a zero vector.At each decoding step, once each model θ i outputs the probability distribution p i , DEEPEN transforms p i into the relative representation r i using the normalized relative representation matrix:r i = p i • Ri ,and aggregate all relative representations to obtain the aggregated relative representation:EQUATIONwhere α i is the collaboration weight of model θ i .Collaboration Weight. As our work focuses on enabling the distribution fusion of heterogeneous LLMs instead of finding the optimal collaboration weights, we follow the most common practice to uniformly aggregate the distributions (α = 1/N , N is the number of models), which is named DEEPEN-Avg. Besides, we also adopt a simple and effective method of deducing weights, DEEPEN-Adapt, which heuristically sets a larger value to the model with a better performance on the development set: α i = s i / j s j , where s i = Acc(θ i , D dev ) -ϵ, Acc(•, •) indicates the average accuracy of model θ i on the development set, and ϵ indicates the chance level on the evaluation task. Specifically, ϵ = 0 on the free-form generation tasks and ϵ = 1/K on the K-choice tasks.To decide the next token according to the aggregated relative representation, DEEPEN aims to transform it from the relative space back to the absolute space of the main model, which is empirically selected with the best-performing model on the development set. To enable this inverse transformation, we adopt a search-based strategy, finding out the absolute representation whose relative representation is identical to the aggregated relative representation. This search problem is formulated as:EQUATIONwhere P i denotes the absolute space of model θ i , and ℓ(•) is the loss function to measure the distance between relative representations. In this work, we adopt the KL-divergence due to its convergence.This search is iteratively conducted under the guidance of the gradient of the loss in Eq.6 with respect to the absolute representation p i . Specifically, we initialize the start point of searching pi with the main model's original absolute representation, and update it as:EQUATIONwhere η is an important hyperparameter named the relative ensemble learning rate, and T is the iterations number named relative ensemble learning steps. Finally, we use the updated absolute representation p (T ) i to determine the emitted token. 
experimental setup: 
 Benchmarks. We mainly conduct experiments on six benchmarks, which can be categorized into:• Comprehensive Examination: (1) MMLU (5-shot) [12] , which covers 57 subjects that humans learn, and (2) ARC-C (0-shot) [5] , collected from standardized natural science tests.• Reasoning Capabilities: (1) GSM8K [6] (4-shot), which is a dataset of high quality problems at the grade school math level, and (2) PIQA [3] (0-shot), which is a commonsense reasoning dataset.• Knowledge Capacities: (1) TriviaQA (5-shot) [16] , collected by Trivia enthusiast authored, and(2) NQ (5-shot) [18] , which is a QA corpus consists of queries issued to the Google search engine.Evaluation. For all benchmarks, we follow the test scripts of OpenCompass leaderboard. Specifically, on the multiple-choice tasks (MMLU, ARC-C, and PIQA), the option with the highest likelihood is selected to calculate the accuracy. On the free-form generation tasks (GSM8K, TriviaQA and NQ), we calculate the exact match (EM) accuracy.Individual models. As ensemble learning typically works on models with comparable performance [24, 34] , we select six well-performing LLMs whose performance are closely matched: LLaMA-2-13B [29] , Mistral-7B-v0.1 [13] , InternLM-20B [26] , Yi-6B [1] , Skywork-13B-base [32] , and Tigerbot-13b-base-v2 [4] . To achieve better ensemble performance, we conduct experiments on the ensemble of the top-2 models and the top-4 models for each benchmark. Besides, we also consider ensembling various number of models ( §4.3) and ensembling more diverse models ( §5.1).Hyperparameters. In this work, we select all of the common tokens between LLMs as the anchor tokens to build the relative spaces, i.e., A = C ( §5.2). For the inverse transformation of relative representations, we search the optimal relative learning rate (η in Eq. 7) from 0.05 to 0.30 with an interval of 0.05. We empirically set the number of relative ensemble learning steps T = 5 ( §5.3).Comparative methods. We compare DEEPEN with (1) MINED [30, 9] , which maps the probability distributions of heterogeneous LLMs to the distribution of the main model via aligning tokens in different vocabularies with edit distance, and (2) LLM-BLENDER [15] , which comprises a reward model PAIRRANKER to score each response of LLMs and a fusion model GENFUSER to fuse 1 : Main results. The best individual model is highlighted in red , and the best ensemble method is highlighted in green, except for the results of the combined method (i.e., the last row). The top-4 models on each benchmark are underlined. '-' indicates that the method does not apply to the task. candidate responses. In this work, we we only adopt the PAIRRANKER since GENFUSER suffers from serious over-generation under our training-free setting. In the ensemble of more than two models, we introduce two additional ensemble methods: (3) VOTING, which selects the choice favored by most models on the tasks with outputs limited to a fixed set, and (4) MBR [8, 17] , which selects the answer with the highest textual similarity to other candidate answers. The implementation details of baselines are illustrated in §B.The main results are shown in Tab. 1, from which we have drawn the following observations:(1) DEEPEN achieves consistent improvements over the individual models. These results prove that our DEEPEN successfully enables collaboration between heterogeneous LLMs via aggregating their probability distributions in the relative space. Specifically, DEEPEN-Avg achieves improvements of +1.43(MMLU)∼+2.72(PIQA) on the ensemble of top-2 models, and +1.00(PIQA)∼+2.89(ARC-C) on the ensemble of top-4 model. DEEPEN-Adapt gains improvements of +1.44(TriviaQA)∼+3.34(ARC-C) on the ensemble of top-4 models.(2) DEEPEN shows better stability than baselines. As shown, LLM-BLENDER struggles to achieve improvements under the training-free setting. MINED shows unstable performance across different benchmarks. For example, MINED leads to performance drops of -35.40 on the GSM8K benchmark under the top-2 models ensemble setting and -2.70 on the TriviaQA, indicating the limitation of using textual similarity to align tokens in heterogeneous vocabularies. Through case studies, it is revealed that this method of aligning tokens with edit distance disturbs the decoding and produces incomplete words (demonstrated in §7). Instead, DEEPEN-Avg achieves consistent improvements and surpasses all baselines in 7/12 settings.(3) DEEPEN has complementary strengths with other ensemble methods. VOTING achieves a significant improvement on the mathematical reasoning GSM8K, showing the effectiveness of reasoning with multiple paths. To evidence the complementary strength of DEEPEN with VOTING, we combine both methods. On the TriviaQA and NQ, VOTING is replaced with MBR. As shown that the combination of both methods gains a further improvement over VOTING (63.15→65.25).(4) Collaboration with more worse-performing LLMs is a double-edged sword. The ensemble performance of DEEPEN-Avg with top-4 models surpasses that with top-2 models on 4 benchmarks, but falls short on 2 benchmarks. This is reasonable because incorporating the 3rd and 4th ranked LLMs enhances complementary strengths but also causes the interference with the top-2 models.Next, we illustrate the effectiveness of DEEPEN on the ensemble of more models on the MMLU, PIQA, and NQ. We add Nanbeige-16B into the ensemble on all three benchmarks, and add LLaMA2-70B and Mixtral-8×7B on the PIQA due to their comparable performance. As illustrated in Fig. 3 , the ensemble performance increases first and then decreases with the joining of more models in descending order of performance. And the ensemble performance peaks in the top-4 or top-5 models across three benchmarks. 
analysis: 
 To deeply understand DEEPEN, we first evaluate its performance on the ensemble learning of model sets with diverse architectures, abilities, and performance gaps. Next, we conduct a series of analyses on the reverse transformation process of relative representations.Ensemble of the dense model and the sparse model. We first evaluate our method on the ensemble learning of the dense model and the sparse MoE model on the challenge reasoning tasks. Specifically, we use the widely-used large-scale dense model LLaMA2-70B [29] and the popular sparse MoE model Mixtral-8×7B [14] as the base models. As the results shown in Tab. 2, our DEEPEN achieves improvements of +1.60 and +3.22 on the GSM8K and PIQA datasets, even though the base models have achieved a high level of performance. Ensemble of the generalist model and the specialist model. To investigate the effectiveness of DEEPEN on the ensemble of the generalist model and the specialist model for the specific task, we conduct experiments on the machine translation task using the ensemble of the large language model LLaMA2 and the machine translation model NLLB [27] , which is a well-known open-source multilingual translator. We adopt the widely-used machine translation benchmark Flores-200 2 . As the results in Tab. 3 illustrated, DEEPEN achieves better translation performance leveraging the diverse translation knowledge in the generalist LLM and the specialist translator.Ensemble of models with different performance gaps. To assess the stability of DEEPEN regarding to the performance gap of base models, we conduct an experiment on the ensemble of model pairs with increasing performance gaps. As the result demonstrated in Tab. 5, the performance of ensemble learning between a well-performing model (the rank-first model)with a worse-performing model could achieve improvements or slightly lag behind the well-performing model.Effect of anchor selection. We demonstrate the impact of different numbers of anchor words through experiments with the top-2 ensemble models on the MMLU and ARC-C datasets. As shown in Fig. 4 , an increased number of anchor words can improve performance for LLMs in downstream tasks, and selecting the full set of common words as anchors provides better performance.Effect of normalization on relative representation matrix. To demonstrate the importance of normalization on the relative representation matrix to the ensemble performance ( §3.2), we conduct an ablation analysis. The result is shown in Tab. 4, the ensemble struggles to achieve improvements due to the ineffective representation of outlier words, i.e., words distant to other words. The proportion of outlier words can be derived from the distribution of distance to nearest neighbor words, which is illustrated in Fig. 8 . As illustrated, a remarkable proportion (> 30%) of words are distant from other words, i.e., cosine similarity to its nearest neighbor word is less than 0.3. Through the normalization operation, the output semantics that intend to emit outlier words could be prevented from becoming zero vectors by relative transformation.To better understand the reverse transformation process ( §3.4) transforming the relative representation back to the absolute space of the main model, we further analyze each component of this process.Analysis of relative ensemble learning rates. As shown in Tab. 5, the performance of DEEPEN is sensitive to the value of relative ensemble learning rate (η), which is abbreviated by RELR. This observation motivates us to measure the generality of this hyperparameter. Specifically, we illustrate the cross-distribution performance of the searched optimal value of η in Tab. 9. As observed, the optimal value of RELR varies across different datasets, which suggests that the inverse transformation from relative space to absolute space requires adaptive mapping schemes. Effect of iteration steps in relative ensemble learning. To give a deep view of the dynamics of the inverse transformation in DEEPEN, we report the performance change along with different numbers of relative ensemble learning steps (T ).Besides, the dynamics of loss of relative ensemble learning (η in Eq. 6)is also reported. As shown in Fig. 9 , on the one hand, more steps of relative ensemble learning significantly lead to lower losses. However, the loss is hard to reach zero, i.e., underfitting. On the other hand, increasing the number of steps of relative ensemble learning will cause the performance to increase first and then decrease. The reason behind the performance drop could be that in the early stage of optimization, the focus of optimization is on updating the high-probability tokens. In the later stage of optimization, since the probabilities of all words will be adjusted equally, the high-probability tokens will be interfered with the high-probability ones, thus affecting the performance. Therefore, it is recommended to set a modest value of step number (e.g., T = 5).Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code will be submitted. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
related work: 
 Selection-based ensemble. Rerank is an intuitive solution to utilize multi-model strengths. Jiang et al. [15] take the first step towards LLM ensemble, training a reward model PAIRRANKER for pairwise comparison on candidate outputs. To overcome the huge computation costs of multi-LLM inference, several works have explored to train a router to predict the best-performing model out of a fixed set of LLMs for the given input [31, 25, 19] .Fusion-based ensemble. Towards a synergy between LLMs, Jiang et al. [15] propose GENFUSER, trained to combine multiple candidate answers. Different from these training-dependent ensemble methods which pose a great challenge to the generalizability of the reward model or fusion model, our DEEPEN is completely training-free, making it more general. Similar to our method, MINED also aims to tackle the vocabulary discrepancy via aligning the tokens in different vocabularies based on edit distance [30, 9] . Unfortunately, this textual similarity-based method exhibits unstable performance and produces abnormal text for LLM ensemble (Tab. 7).There are several contemporaneous works related to our work. Xu et al. [33] propose EVA to tackle vocabulary discrepancy by learning token alignment between different vocabularies with the assistance of overlapping tokens. Our DEEPEN eliminates this training process via directly aligning tokens with the relative representation (more discussion is illustrated in §B). Mavromatis et al. [20] explore adaptive collaboration weights at test time by harnessing the perplexity on the input prompt. We emphasize that this work is complementary to our work. 
conclusion: 
 In this work, we propose a training-free LLM ensembling framework DEEPEN, which addresses the vocabulary discrepancy when fusing the probability distributions of heterogeneous LLMs. Experimental results on six widely-used benchmarks demonstrate that DEEPEN exhibits more stable performance than baseline methods and has complementary strengths with other ensemble methods such as VOTING. We believe our work can inspire further research on the LLMs collaboration, model reuse, and knowledge distillation. In the future, we aim to explore more effective adaptive collaboration schemes to leverage the complementary strengths between different LLMs. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Our research motivates the research of prediction fusion and model fusion via tackling the vocabulary discrepancy. There is no negative societal impacts of the work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Not involved Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Mistral-7B We count the number of common tokens shared among different LLM vocabularies and present the results in Fig. 6 . It is observed that a large number of common words (>20k) exist across the different vocabularies. We also count the number of common tokens in all six LLMs and find that there are a total of 18k common tokens, enabling DEEPEN to be applied to the ensemble learning of a large number of models.LLM-BLENDER. (1) the selection-based ensemble method PAIRRANKER Jiang et al. [15] , which is a reward model to score each response of LLMs and (2) the fusion-based ensemble method GENFUSER Jiang et al. [15] , which is a generative model to fuse multiple candidate responses. Both models are trained on the constructed instruction tuning dataset MixInstruct. In our experiments, as GENFUSER struggles to generate responses following the expected format, we only adopt PAIRRANKER.VOTING. For tasks with outputs limited to a fixed set (i.e., MMLU, ARC-C, PIQA, GSM8K benchmarks), we adopt the VOTING method on the ensemble learning of more than 2 models. Concretely, we count each candidate answer's occurrences and select the most frequent as the final output. In the event of a tie, the main model's answer is used as the final output.For generation tasks, we implement the MBR [8, 17] method, which selects the answer with the highest lexical similarity to other candidate answers. To measure this similarity, we experimented with the edit distance and chrF 3 metrics, ultimately choosing chrF due to its superior performance.To bridge the gap between different vocabularies in LLM ensemble, MINED apply the Minimum Edit Distance (MinED) approach to align tokens across different vocabularies, e.g., "get" to "gets". However, this textual similarity-based mapping method could disturb the text generation process and produce incomplete words.EVA. Recently, Xu et al. [33] propose EVA to tackle the vocabulary discrepancy by learning mappings between the vocabularies of different LLMs with the assistance of overlapping tokens. We have tried to re-implement their method with the released code. However, we encounter a technical problem in that EVA only supports the ensemble learning between LLMs with the same embedding dimension. This is caused by the limitation of tool of vecmap 4 , which is used to learn the token alignment. For example, in Tab. 7, the weights for LLaMA1 and LLaMA2 could be (0.6, 0.4), where the weight of the main model is larger than the other model.To accomplish the fusion of heterogeneous distributions, DEEPEN first maps the distributions into the relative space and adopts the search-based inverse transformation to map the aggregated relative representation back to the main model's probability distribution, which incurs an extra latency. This latency is mainly caused by the inverse transformation process, which requires T -round search. To demonstrate this latency, we report the token-level inference latency of ensembling two LLMs (Mixtral-8×7b and LLaMA2-70B). This experiment is conducted on 8 A100 GPUs. All of our experiments can be re-implemented on 8 A100 GPUs. As shown in Tab. 8, DEEPEN causes +17% token-level inference latency. However, in practice, this latency could be greatly decreased since all individual models intend to emit the same token in 90% decoding steps. In these steps, we could skip the fusion process and use the consistently agreed token as the next token. In total, DEEPEN actually incurs less than 2% sentence-level inference latency. As illustrated in Tab. 1, collaboration with more LLMs can sometimes lead to a performance drop caused by interference from lower-performing models. This issue limits the ensemble performance of our current method, even though we have explored setting different collaboration weights for each model on each benchmark (DEEPEN-Adapt). An ideal solution would be to set adaptive collaboration weights at the sample level, or even the token level, for each LLM, which remains a significant Table 9 : Cross-distribution validation of relative ensemble learning rate (η). We report the performance of ensembling LLaMA2-13B and Mistral-7B. Each row indicates the test set used to evaluate performance. Each column indicates the development set used to search the optimal value of η. NeurIPS Paper ChecklistOur code is available at: https://github.com/OrangeInSouth/DeePEn means corresponding author.https://github.com/facebookresearch/floreshttps://github.com/mjpost/sacrebleuhttps://github.com/artetxem/vecmap 


Decision: The paper presents DEEPEN, an ensemble learning framework for heterogeneous LLMs that uses a novel training-free method to address vocabulary discrepancies across different models. This method translates probability distributions from individual LLMs into a shared “relative representation” space, facilitating aggregation and synthesis back into coherent output. 

In addition to the innovative treatment of the token alignment problem inherent in multi-LLM integration, the proposed method performs well across multiple benchmarks. I agree with the reviewers that the strong theoretical grounding and the broad range of empirical validation are the strong points of this paper. The paper is also well-written and easy to understand without sacrificing details. The reviewers and I also appreciate the authors’ strong response in addressing the concerns around benchmark comparisons and methodological clarification during the rebuttal.

Even though I recommend acceptance for this paper, I recommend the authors pay attention to the weaknesses pointed out by the reviewers, especially regarding the depth of analysis concerning the generalization of DEEPEN across varied tasks and its sensitivity to hyperparameter settings.

Reviews: review: paper_summary: paper_summary: The paper introduces a method that maps output distributions of different LLMs to and from a universal relative space to aggregate them, based on which the next token is determined.
strength_weakness: strength_weakness: - **Presentation**: Although the paper is well-structured, some passages could be condensed. For instance, the paragraphs "Anchor Selection" and "Normalization of relative representation matrix" simply repeat what has already been stated in the general paragraph above. Also, sections 3.3 and 3.5 both consider the aggregation of relative representations, currently separated by section 3.4 which considers the inverse transformation. Combining those conceptually the same topics would improve the reading flow.
- **Evaluation**: Although the paper provides numerous ablation studies, the method is only compared against the baselines in the main experiments (Tab. 1). Here, the method outperforms the baseline in (only) 7/12 settings. Also comparing to the baselines in the other experiments (see questions to Fig. 3. and Tab. 2-3 below) would provide further insights into the performance of the method.
questions: questions: - Figure 1: what do the three colors (blue, yellow, purple) indicate?
- Table 1: Why did the authors not report DEEPEN-Adapt with Top-2 Ensembles? (DEEPEN-Adapt outperforms DEEPEN-Avg on all benchmark tasks with Top-4 Ensembles, so why is this not the default setting?)
- Figure 3: How do the baseline methods perform with an increasing number of ensemble members considered?
- Table 2 & 3: How do the baseline methods perform with the dense/sparse models and the generalist/specialist models?
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: The paper introduces DEEPEN, a novel training-free ensemble framework designed to leverage the complementary strengths of various large language models (LLMs). The key innovation of DEEPEN lies in its ability to fuse informative probability distributions from different LLMs at each decoding step, addressing the challenge of vocabulary discrepancies between heterogeneous LLMs.

DEEPEN operates by transforming the probability distributions from each model's vocabulary space into a shared "relative representation" space based on the relative representation theory, which uses embedding similarities to a set of anchor tokens. The aggregated relative representations are then mapped back to the vocabulary space of one LLM (the main model) to determine the generated token.
strength_weakness: strength_weakness: 1) The paper does not provide a detailed analysis of how well DEEPEN generalizes to unseen data or across different types of tasks beyond the evaluated benchmarks.
2) The method's performance is sensitive to the choice of hyperparameters, such as the relative ensemble learning rate. Finding optimal hyperparameters may require additional tuning and may not be straightforward.
3) The performance of DEEPEN relies heavily on the selection of anchor words. The paper does not thoroughly explore the impact of different anchor word selection strategies on the ensemble performance.
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The authors propose to use relative representations to unify the tokenization across different models for an effective ensemble. They first transform the prediction of each model to a relative space. Then, after averaging the relative predictions, they perform a gradient-based optimization to find the averaged prediction in the original space.
strength_weakness: strength_weakness: 1. The proposed method requires gradient updates to project back to the original space at every generation step. This can cause extra overhead.
2. The performance largely depend on the number of common anchor words between different LLMs (Figure 4). As a result, the method may be not effective when the domain of pretrained LLMs are very different.
questions: questions: 1. Have you tried other normalization methods other than softmax?
limitations: limitations: Discussed in Appendix.

Comments: comment: Thanks for the detailed response! I will maintain my positive stance towards the paper.
Thank you for the rebuttal. Your responses have adequately addressed my questions and concerns.