Knowledge Fusion of Large Language Models
Authors: Fanqi Wan
Xinting Huang
Deng Cai
Xiaojun Quan
Wei Bi
Shuming Shi
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Model Fusion
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: In this study, we have explored the realm of knowledge fusion for LLMs to create a unified model that combines the capabilities and distinctive strengths of multiple structurally diverse LLMs.

Abstract: While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling approach is to merge existing pre-trained LLMs into a more potent model. However, due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities of the target model beyond those of any individual source LLM. We validate our approach using three popular LLMs with different architectures—Llama-2, MPT, and OpenLLaMA—across various benchmarks and tasks. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation. Our code, model weights, and data are public at \url{https://github.com/fanqiwan/FuseLLM}.

Published as a conference paper at ICLR 2024
KNOWLEDGE FUSION OF LARGE LANGUAGE MODELS
Fanqi Wan'; Xinting Huang"! Deng Cai', Xiaojun Quan "f Wei Bi, Shuming Shi?
SSchoo of Computer Science and Engineering, Sun Yat-sen University, China
Tencent Lab
wanfq@maill sysu. edu. en, quanx (etnnxxjeemm1l..yuuuu. (t ttimxthuang, jeykcai, victorias abi, shumi shumingshijetenneene
ABSTRACT
While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling ap- proach merge existing pre-trained LLMs into more potent model. However. due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities the target model beyond those of any indi- vidual source LLM. We validate our approach using three popular LLMs with dif- ferent architectures -laattrrrs---aaa-22-11 MPT, and OpenLLaMA across various bench- marks and tasks. Our findings confirm that the fusion of L.MSs can improve the performance of the target model across a range of capabilities such as reasoning. commonsense, and code generation. Our code, model weights, and data are public at ht
com wttm://aagaa///use///o/ooooe
INTRODUCTION
With the continuous success of large language models (LLMs) such as GPT (Brown et al. 2020) and LLaMA (Touvron et al. 2023) series across wide range of natural language processing (NLP) tasks, it has become a strategic imperative for corporations to create their own LLMs. However, the costs associated with LLM development are astronomical. In addition to requiring vast amounts of training data, advanced techniques, substantial computational resources, and skilled labor, the development process also exerts significant pressure on energy consumption and the environment (Rillgg et al. 2023). While these LLMs exhibit structural and functional differences, they share similar capabilities across spectrum of NL.P tasks. Consequently, beyond the traditional approach of training an LLM from scratch, an alternative option is to combine existing LLMs into new, more powerful one, which is termed knowledge fusion of LLMs in this paper. successful, this fusion not only cuts the cost of initial training but also allows the integrated model to benefit from the strengths of all the LLMs. This new model can also be fine-tuned and adapted for various downstream tasks. Moreover, the fusion can also happen among fine-tuned LLMs that specialize in specific task.
The endeavor to integrate the capabilities of multiple models has been a long- standingg pursuit. For example, ensemble methods (Littlestone & Warmuth| 1994 Tiang et al., 2023) directly aggregate the outputs of different models to enhance prediction performance and robustness. However, this approach requires maintaining multiple trained models and executing each during inference, which is impractical for LLMs due to their substantial memory and inference time requirements. Likewise, this approach doesn't 't facilitate fine-tuning, which essential for many LLMs. Another approach is to directly merge several neural networks into a single network through parameter-wise arith- metic operations (Wortsman et al. 2022 Jin al. 2022). This approach typically assumes uniform network architectures and attempts to establish mappings between the weights of distinct neural net-
Work was done during the internship at Tencent Al lab.
Corresponding authors.

---

Published as a conference paper at ICLR 2024
Ensemble
Weight Merging
LUSELLM
Model
Model
Model
Model
Model
C
Protubilietic
Marrie
Fusel
Tampei
LLM
PUxcLLM
Fuse
Train
Updaoe
Aggregation
Math Operation
Syeae Ostpul
Model
Illustration of conventional model fusion techniques (ensemble and weight merging) and our knowledge fusion approach for LLMs (FusELLM). Different animal icons represent different LLMs, with various species denoting LLMs possessing differing architectures. FUSELLM exter- nalizes the knowledge from multiple L.Mss and transfers their capabilities to a target LLM.
Figure I:
works, which is often unattainable in the context of LLMs. Moreover, weight merging may lead to suboptimal results when substantial differences exist in the parameter space (Li et al. 2022).
In this paper, explore the fusion of LLMs from a probabilistic distribution perspective. For an input text, we argue that the probabilistic distributions generated by different source LLMs can reflect their inherent knowledge in understanding this text. Therefore, the proposed FUSELLM leverages the generative distributions of source LLMs to externalize both their collective knowledge and individual strengths and transfer them the target LLM through lightweight continual training. To achieve this, we develop new strategy for aligning tokenizations originating from different LLMs and explore two methods for fusing the probability distributions generated by these diverse L.MSs. During the continual training, FUSELLM places significant emphasis on minimizing the divergence between the target LLM'' probabilistic distributions and those of the source LLMs.
To empirically demonstrate the effectiveness of FUSELLM, we examine challenging yet general scenario of LLMs fusion, where the source models share minimal commonalities. Specifically, we focus on three popular open-source LLMs that possess distinct architectures and functionali- ties: Llama-2 (Tovrron et al., 2023), OpenLLaMA (Geng & Liu 2023). and MPT (Team] 2023). Evaluations across three benchmarks, which consist of total of 42 tasks spanning reasoning, com- monsense, and code generation, confirm that the target model trained by our method outperforms each source L.MM and the baseline in most tasks. Moreover, we simulate the existence of function- ally distinct L.MSs with identical architecture by continually training a single base model on several domain-specific corpora. When evaluated based on perplexity, our method demonstrates superior potential in combining the capabilities of these structurally identical LLMs compared to traditional ensemble and weight merging methods.
To sum up, this paper explores novel challenge called LLMs fusion, with the goal of creating a unified model that effectively utilizes the collective capabilities and unique strengths of diverse LLLMs. Illustrated in Figure 1 our proposed approach distinguishes itself from traditional ensemble and weight merging techniques by prioritizing the fusion of multiple LLMs through knowledge ex- ternalization and transfer. This study yields several findings that may spark future research. Firstly, while we demonstrate the effectiveness of our method through lightweight continual training on compact, high-quality corpus, the thoughtful selection of the training corpus can be crucial con- sideration, particularly with regard to its relevance to downstream tasks. Secondly, in scenarios where the capabilities of source LLMs vary significantly, the fusion function appears to be crucial in effectively combining their respective strengths. Lastly, when compared to traditional model en- semble and merging techniques, the field of LLMS fusion appears to be a more promising avenue for exploration, especially in light of the diverse structures and substantial model sizes of LLMs.
2 RELATED WORK
Model Fusion The integration of capabilities from diverse models has been long-standing ob- jective, with existing approaches mainly falling into two categories. Firstly, the traditional tech- nique of model ensemble combines the outputs of multiple models to enhance overall system perfor- mance (Littlestone & Warmuth| 1994 Sagi & Rokach| 2018). Note that this technique doesn't 't in- volve the explicit merging of multiple models into new one. Common methods for model ensemble
2

---

Published as a conference paper at ICLR 2024
typically employ weighted averaging (Littlestone & Warmuth 1994) majority voting (Monteith et al. 2011) to consolidate predictions from various models. Recently, Jiang et al. (2023) introduced an ensemble framework designed to leverage the diverse strengths of multiple open-source LLMs. This framework first employs a pairwise comparison method to detect subtle distinctions among candidate outputs. Then, it combines the top-ranked candidates to produce enhanced output, capitalizing on their strengths while mitigating their weaknesses.
Secondly, weight merging presents another approach that facilitates model fusion the parameter level. Cupta et al. (2020) and Wortsman et al. (2022) merged weights from models with identi- cal structures, obtained through different strategies or configurations, to achieve improved overall performance. Similarly, Cha et al.] (2021), |ame et al. (2022), and (rpit et al. (2022) explored weighted averaging of models derived from different configurations to enhance out-of-distribution generalization. Furthermore, Jin et al. (2022) merged models designed for specific domains or tasks to create a generalist capable of addressing all domains or tasks. Going beyond parameter merging of entire models, Waang et al. (2022b). Huang et al. (2023), and Zhang et al. (2023) applied linear mathematical operations to adapter parameters to achieve superior generalization performance.
In a nutshell, while model ensemble requires the parallel deployment of multiple models, weight merging is generally limited to models with identical architectures. In contrast, the approach pro- posed in this paper supports the fusion of multiple LLMs with diverse architectures by explicitly transferring their knowledge and capabilities to a target I.I.M.
Knowledge Distillation Knowledge distillation (Hinton et al. 2015), initially proposed for model compression, involves training a student model under the guidance of one or more teacher mod- els. In the NLP community, knowledge distillation has been widely applied to text classification tasks. These applications include training the student model to replicate the teacher's 's output distri- bution (Sanh et al.] 2019 Ture al., 2019), as well as features (Sun al. 2019 Jiao et al. 2020) and relations (Wang et al. 2020) derived from intermediate layers of the teacher model. In the realm of text generation, the conventional approach focuses on minimizing the KL divergence between the student and teacher generation distributions. This is achieved by using the teacher's probability distributions at each time step as supervision (Khanuja et al. 2021 Gu et al. 2023 garwaal al.,
2023) or by directly training on the teacher's generated texts (Peng et al. 2023 Xu et al. 2023).
While our method shares a framework similar multi-teacher knowledge distillation, there are two significant distinctions. First, in traditional knowledge distillation, the student models are typically constrained to be smaller size than the teachers. In our scenario, however, there are no limitations on the size of the target model. Second, traditional knowledge distillation often results in the student models lagging behind the teachers in performance after distillation. In contrast, we anticipate that after the fusion, the target model will surpass any of the source models in performance.
3 KNOWLEDGE FUSION OF LLMS
The primary objective of LLMs fusion to externalize the collective knowledge embedded within multiple source LLMs and integrate their capabilities into a target LLM. Given K source LLMs {Me} with varying architectures, each having undergone individual pre-training or fine-tuning on distinct datasets, the key idea behind our approach is to initially stimulate LLMs to manifest their inherent knowledge by challenging them to predict the next token. The probabilistic distributions of these predictions are thoroughly assessed, and the most accurate predictions are utilized con- tinually train the target I.LM M' on a corpus C using the causal language modeling objective. In the following sections, we start with brief introduction the preliminaries, followed by a detailed explanation of our LLMs fusion framework. Finally, we delve into the implementation details.
3.1 PRELIMINARIES
Let denote text sequence of length N sampled from the corpus C and lci (41, .2....;,4-1 denote the sequence preceding the ith token. The causal language modeling (CLM) objective for training a language model parameterized by is defined as minimizing the negative log- likelioohooo::
LaM = -EEc
Σ log log
(1)
3

---

Published as a conference paper at ICLR 2024
where (t, is the model's predicted probability for token i, given the preceding tokens.
The above objective decomposes sequence likelihood into token-level cross-entropy losses, compar- ing each token's 's predicted distribution to its one-hot representation. To provide more generalized perspective, we reframe this token-level view into a sequential distribution format. Specifically, for the text sequence trix, P E N where the i-th row represents the distribution predicted by the model for the ith token over the vocabulary of size V. The CLM objective can then be interpreted reducing the discrepancy between P and the one-hot label matrix, O E {0, 1 where each row is a one-hot representation of the corresponding gold token. Formally, the CLM objective is transformed into the following representation:
aggregate token-level predictions and create probabilistic distribution ma-
LLLM -Evc [D (P) Oz)]
where ()) represents the discrepancy function between two matrices, and is equivalent to Eq. when implemented as the KL divergence.
1
3.2 LLMS FUSION
Taking this perspective on a language model, we argue that the probabilistic distribution matrix can reflect its certain inherent knowledge understanding the text. Consequently, different probabilistic distribution matrices for the same text, originating from various LLMs, can be used to represent the diverse knowledge embedded within these models. Acknowledging this, the proposed FUSELLM approach tackles LLMs fusion through probabilistic modeling, aiming to create a unified LLM by merging the prohabilistic distributions of the source I.MMs. To achieve this, when starting with set of LLMs to fuse, FUSELLMM undergoes lightweight continual training of the target L.M.M on raw text corpus that mirrors the pre-training dataset. Instead of relying solely on the CLM objective, FUSELLM places significant emphasis on minimizing the divergence between the target LLM's probabilistic distributions and those of the source LLMs.
For each text in the corpus C. we apply the provided K source LLMs and obtain set of proba- bilistic distribution matrices, denoted as (P") ) where f, represents the parameters of the jth LLM. Utilizing these matrices, we externalize the knowledge from individual models into a unified
space, essentially creating unified probabilistic representations over the text. We acknowledge that
variances in vocabulary among the source LLMs can lead to misaligned matrices (P. 1K
{ fz=1- To ad-
dress this, we employ token alignment strategy, which is explained in Section 3.3 foster more coherent probabilistic interpretations across models.
Having aligned the probabilistic matrices, we proceed fuse them into single compact represen- tation. Various fusion strategies can be applied for this purpose, as detailed in Section 3.3 We use P, to represent the fused representation matrix as follows:
P Fusion(P?: ,P. .P?...., P!*),
(3)
where Fusion(-) denotes the function that combines multiple matrices, and the resulting matrix P. seen as a representation of the collective knowledge and distinctive strengths of the source LLMs.
To transfer the capabilities of source LLMs to the target LLM, we enforce alignment between the target LLM's predictions and the fused representation matrix Pt. We use Q, to represent the output distribution matrix of the target LLM for text I, and then define our fusion objective as follows:
Lpusion = E,c (QQ.PP))..
The overall objective for our continual training consists of weighted combination of the causal language modeling objective LCLM and the fusion objective L follows:
L = ALEM + (1 A)
(5)
3.3 IMPLEMENTATION OF FUSELLM
In this section, we present the implementation details of token alignment and the fusion function for fusing different LLMs in our FUSELLM method.
4

---

Published as a conference paper at ICLR 2024
Token Alignment Ensuring token alignment across multiple LLMs is crucial for effective knowl- edge fusion, as guarantees proper mapping of probabilistic distribution matrices. Fu et al. (2023) employed dynamic programming to recursively minimize the total cost of editing one sequence of tokens to match the other. If a one-to-one mapping exists between two tokens, the corresponding distributions perfectly mapped. Otherwise, the mapped distribution degenerates into one-hot vector. Since tokens generated by different tokenizers for the same sequence typically exhibit lim- ited differences, we propose enhance the success rate of token alignment by replacing the exact match (EM) constraint in Fu et al. (2023) with minimum edit distance (MinED) strategy, which maps tokens from different tokenizers based on MinED. This relaxation of token alignment helps preserve substantial information in the distribution matrices while introducing minor errors. For more details of the token alignment, please refer to Appendix A
Fusion Strategies To combine the collective knowledge of source LLMs while preserving their unique strengths, it is essential to evaluate the quality of different LLMs and assign varying levels of importance to their respective distribution matrices. For this purpose, when dealing with text t, utilize the cross-entropy loss between the distribution matrices and the gold labels as an indicator of the prediction quality of the LLMs (Marion et al.) 2023). A lower cross-entropy score for source LLM signifies a more accurate understanding of the text, and its prediction should be accorded greater significance. Based on this criterion, we introduce two fusion functions: (1) MinCE: This function outputs the distribution matrix with the minimum cross-entropy score; (2) AvgCE: This function produces a weighted average of the distribution matrices based on cross-entropy scores.
The complete process of the FUSELLM method is described in Algorithm 1
Algorithm FUSELLM for LLMs Fusion
Require: Source I.LMs {Mi)
K J1=1- training corpus C.
1: Initialize the target LLM M' with one of the source LLMs.
2: for text t C do
3: Apply the K source LLMs to compute probabilistic distribution matrices (P) 1イ
1-=1-
4: Align {P.² 1K
J==1 using the MinED alignment method.
5: Fuse (
Jy=1 to obtain P. with the MinCE or AvgCE fusion function.
6: Update parameters of M by minimizing the overall loss fuction in Eq.
7: end for
8: return M
EXPERIMENTS
In our experiments, we consider general but challenging scenario of L.MMs fusion where the source models share minimal commonalities in architectures or functionalities. Specifically, we conduct experiments on the 7B scale and select three representative open-source models: Llama-2, Open- LLaMA, and MPT as the source LLMs for fusion. Regarding the target LLM, we opt for another Llama-2 7B, which is generally the most robust one among the three source LLMs. The target LLM starts with the same pre- pre-trained weights as its source counterpart but differs in that it updates pa- rameters during training. To evaluate the performance of FUSELLM. we conduct experiments on benchmarks assessing the capabilities of L.Mss in reasoning, commonsense, and code generation.
4.1 EXPERIMENTAL. SETUP
Dataset for continual training To continullly train the target LLM for LLMs fusion, it essential have compact yet diverse training dataset. We have chosen MiniPile, meticulously curated dataset resulting from a thorough clustering and filtering process. MiniPile comprises approximately
1 million documents across 22 domains and 1.8 billion tokens, constituting less than 0.1 % of the trillion training tokens of Llama-2. More dataset details can be found in Appendix B
Fusion function For the fusion function, we use the minimum cross-entropy (MinCE). However, the impact of employing alternative fusion functions will be examined in Section 4.4
Training details We train the target LLM of Llama-2 7B using a batch size 128 and a maximum length of 2048 single node equipped with 8 NVIDIA 1000 GPUs, each with 40GB of memory.
5

---

Published as a conference paper at ICLR 2024
Our training framework is implemented based on the Huggingface Transformers (Wolf et al. 2020) and accelerated with FlashAttention (Dao et al. 2022). We empirically set the combination weight À in Eq. to 0.9. The training consists of only single epoch, which takes approximately 33 hours. For further hyper-parameter details, please refer to Appendix
Evaluation We evaluate FUSELLM on three benchmarks that represent different core capabiliitess of LLMs, spanning reasoning, commonsense, and code generation.
Bíg-Bench Hard (BBH) Suzgun et al. 2022 is a benchmark to evaluate the general reasoning ability of LLMs. It contains 23 multiple-choice tasks and 4 free-form generation tasks from the Big-Bench (Srivastava et al. 2022). which can be classified into four categories: algorithmic and arithmetic reasoning, natural language understanding, world knowledge, and multilingual knowl- edge and reasoning. We follow previous work (Wang et al. (023b) to generate the predictions based on few-shot chain-of-thought (CoT) prompts and then calculate the exact match (EM) accuracy.
Common Sense (CS) is a benchmark to evaluate the commonsense capability of LLMs. We con- sider standard multiple-choice tasks; ARC easy and challenge (Clark et al. 20118) BoolQ (Clark) et al. 2019a), HellaSwag (Zellers et al. 2019), and OpenBookQA (Mihaylov et al. 2018). We em- (Gao et al. 2021) to conduct a likelihood- zero-shot evaluation. Specif-
ploy
ically, we select the option with the highest likelihood given the context and report the accuracy.
MultiLLEE (ME) (Cassano et al. 2022) is multilingual programming benchmark to assess the coding ability LLMs. is translated from the Python benchmark (Chen et al. (02T) into parallel datasets in 18 programming languages. We use the bigcode waaee-evvvaaati--arareeeee
(Ben Allal et al.)
2022] to perform zero-shot code generation 10 popular programming languages in the HumanEval category and report the pass @1 I (Chen et al. 2021) based on 20 generated samples for each question.
Baselines In our experiments, we compare our FUSELLM with two sets of baselines: (1) original LLMs, including Llama-2 7B, OpenLLaMA 7B, and MPT 7B; and (2) Llama-2 CLM: continually trained Llama-2 7B on MiniPile using only the casual language modeling objective.
4.2 OVERALL RESULTS
Table | presents the overall results of FUSELLM in comparison to the baseline methods on BBH. We can observe that the three source LLMs exhibit varying performance across the 27 BBH tasks, with Llama-2 generally outperforming the others. After continual training with compact and di- verse corpus. 2 CLM shows relative improvement of 1.86% compared to Llama-2, although this improvement is relatively modest and inconsistent across tasks. On average, FUSELLM demon- strates an average relative performance gain of 5.16% over the original Llama-2 across all 27 tasks. In specific tasks, the enhancements achieved by FUSELLM are substantial (e.g., from 54.40 to 65.20 in the Hyperbaton task). In tasks such as Dick Languages where simple continual pre-training leads to a decline in performance, FUSELLM leverages the combined strengths of individual source LLMs to recover performance improvements. Note that FUSELLM occasionally exhibits degraded performance on tasks such Geometric Shapes and Word Sorting, which could be attributed to two reasons. First, the other source LLMs, apart from Llama-2, perform poorly on these tasks, affecting the fusion results. Second, the relevance between the continual training dataset and downstream tasks also contributes to the performance degradation.
Table shows the zero-shot performance of FUSELLM and the baseline methods on the Common Sense (CS) benchmark. The results demonstrate that FUSELLM consistently surpasses the base- lines across all five tasks, achieving a relative performance improvement of 1.25% over Llama-2. In contrast, Llama-2 CLM exhibits a marginal improvement, with only a 0.16% relative enhance- ment compared to Llama-2. Notably, substantial improvements from Llama-2 to FUSELLM are observed in the challenging ARC-challenge (2.40%) and OpenBookQA (2.71%) tasks, highlighting the effectiveness of FUSEL.LM in leveraging collective knowledge to address intricate problems.
For the code generation evaluation, the zero-shot performance of FUSELLM on the MultiPL-E (ME) benchmark is reported in Table 3 We observe that FUSELLM outperforms Llama-2 in 9 out of the 10 tasks, with notable enhancement in the pass @1 score for specific programming languages such as R, increasing from 4.97 to 5.84. Given that both OaM and MPT demonstrate remark- able performances in code generation tasks compared to Llama-2, the fusion result via FUSELLM achieves an average performance gain of 6.36%, which is considerably higher than the 1.37% im- provement observed in 1lama-2 CL... However, it's important to note that FUSELLM still exhibits

---

Published as a conference paper at ICLR 2024
Task
MPT
Llama-2
Llama-2 CLM
FUSELLM
Boolcan Expressions
74.40
66.00
68.80
76.00 (+10.47%)
71.60 (+4.07%)
Causal Judgement
45,45
50.80
50.80
46.52 (-8.43%)
46.52 (-8.43%)
Date Understanding
43.60
43,60
59.60
59.20 (-0.67%)
62.40 (+4.70%)
Disambiguation QA
36.00
47.60
46.80
48.00 (+2.56%)
50,00 (+6.
Dyck Languages
5.20
5.20
7.20
6.40 (-11.11%)
8.80 (+22.22%)
Formal Fallacies
50.80
52.80
49.20
48.80 (-0.81%)
49.20 (+0.00%)
Geometric Shapes
0.00
0.00
34.40
19.20 (-44.17%)
22.80 (-33.72%)
Hyperhaton
62.80
53.60
54,40
56.40 (+3.68%)
65.20 (+19.85%
Logical Deduction (3 objects)
43.60
40.80
54,00
57. (+5.93%)
60.40 (+11.85%)
Logical Deduction (5 objects)
31.60
31.20
35,60
(+14.108)
33.20 (+6.41%)
Logical Deduction (7 objects)
16.80
18.40
24,80
29.60 (+19.359))
25.60 (+3.23%%
Movie Recormenndation
39.60
52.00
72.80
71.60 (-1 11.6%
73.60 (+1.10%))
Multistep Arithmetic Two
0.80
0,40
0.80
4.40 (+550.00%)
4,80 (+500.008)
Navigate
54.00
48.80
56,00
61.20 (99.)%%)
64,40 (+15.00%)
Object Counting
49.60
40.40
49.60
51.60 (+4.03%)
55.20 (+11.29%)
Penguins in Table
28.08
28.08
32.19
31,51 (-2.11%)
32.88 (4%%)
Reasoning about Colored Objects
28.00
31.60
46.40
47.20 (+1.72%)
48,40 (+4.31%)
Ruin Names
31,20
23.20
34,00
30.80 (-9.41%)
32.40 (-4.71%)
Salient Translation Error Detection
14,80
0.00
27.60 (+11.29%)
29.20 (+17.74%)
Snarks
44.94
45.51
47.75
49.44 (+3.54%)
49.44 (+3.54%%
Sports Understanding
64.40
82.40
90.00
90.00 (+0.00%)
91.20 (+ (+333%%
Temporal Sequences
32,00
21.20
12.80
16.40 (328.13%)
16.40 (+28.13%% 13%)
Tracking Shuffled Objects (3 objects)
36.40
30.40
(+3.61%)
34.40 (+3.61%)
Tracking Shuflle Objects (5 objects)
19.20
14.40
15.60
15.20 (-2.56%))
55.60 (+0.005))
Tracking Shuffled Objects (7 objects)
10.80
2,00
11,20
9.60 (-14.29%)
10.40 (-7.14%)
Weh of Lies
51.60
63.60
50.80
61.60 (+21.26%))
65.60 (+29.13%%
Word Surting
5.60
6.80
12.80
7.60 (-40.63%)
7.60 (-40.63%)
Avg. Tasks
33.87
33.38
39.70
40.44 (+1.86%)
41.75 (+3.169)
Table I: Overall results of FUSELLM and baselines in reasoning evaluations on Big- Big-Bench Hard (BBH), where percentages indicate the rate of improvement/decrease compared to Llama-2.
Task
OpenI.LaMA
MPT
Llama-2
Llama-2 CLM
FUSELLM
ARC-easy
69.70
70.12
74.58
74.54 (=0.05%)
75.04 (+0.62%)
ARC-challenge
41.38
42.15
46.33
46.50 (40.37%)
47.44 (42.40%)
BuolQ
72.29
74.74
77.71
76.88 (-1.07%)
78.13 (+0.54%)
HellaSwag
74.53
76.25
76.00
76.57
(40.75%)
76,78 (+1.03%)
OpenBookQA
41.00
42.40
44.20
44.80 (+1.36%)
45.40 (+2.71%)
Avg. 5 Tasks
59.78
61.13
63.76
63.86 (40.165))
64.56 (+1.25%)
Table 2: Overall results of FUSELLM and baselines in commonsense evaluations on CommenSense (CS), where percentages indicate the rate improvement/decrease compared to Llama-2.
Task
OpenLLaMA
MPT
Llama-2
Llama-2 CLM
FUSELLM
C++
14.47
13.11
7.45
9.88 (432.62%)
9.25 9+24. 16%%
Go
68.20
66.96
57.02
54.44 (-4.52%)
59.78 (+4.84%)
Java
14,28
13.42
10.31
10.50 (+1.84%)
10.34 (+0.29%)
JavaScript
17.61
13.01
13.17
14.25 (48.20%)
14.32 (+8.73%)
PHIP
11.24
9.53
9.75
9,04 (-7.28%)
9,41 (-3.49%))
Python
15.96
17.24
13.85
13.07 (-5.63%%)
13,91 (+0.43%)
R
7.52
4.53
4.97
5.25 (+5.639))
5.84 (+17.51%)
Ruby
10.34
12.33
10.37
10.68 (+2.999%
11.24 (+8.399%
Rust
6.18
8.29
6.77
6.96 (+2.81%))
7.05 (+4.14%)
TypeScript
15.31
14.13
12.61
14.19
(+12.53%))
14.50 (+14.99%)
Avg. 10 Tasks
18.11
17.26
14.63
14.83 (+1.37%)
15,56 (+6.36%)
Table 3: Overall results of FUSELLM and baselines in code generation evaluations on MultiPL-E (ME), where percentages indicate the rate of improvement/deccrase compared to Llama-2.
a performance gap compared OpeAL.LMA or MPT in this evaluation. This discrepancy can be attributed to two primary reasons: the inferior performances of Llama-2 as the target model com- pared to other source LLMs and an insufficient proportion of coding-related texts in the continual training corpus, estimated at approximately 7.59%
Since MiniPile lacks specific data percentages for individual domains, we approximate this by considering the percentage of the Github domain in The Pile.
7

---

Published as a conference paper at ICLR 2024
4.3 THE FUSED PROBABILISTIC DISTRIBUTIONS
We investigate the efffcctiveness of the fused proba- bilistic distributions obtained from multiple LLMs and track the trend of performance improvement during the training process. Figure 2 illustrates the comparison of few-shot CoT performance between Llama-2 CLM and FUSELLM with varying scales of training data on BBH. Our observations reveal that FUSELLM en- hances the exact match (EM) accuracy by 2.5% com- pared to Llama-2 CLM and achieves the best perfor- mance of Llama-2 CLM within 0.52 billion tokens. No- tably, this represents 3.9x reduction in token require- ments compared to the 1.57 billion tokens needed by Llama-2 CLM. These results suggest that the proba- bilistic distributions derived from LLMs contain knowl- edge that is more readily learnable than the original text sequences, which accelerates the optimization process. This finding aligns with the observations in Hsieh et al. 2023). We further conduct an experiment to show that our performance improvement stems from the integration of knowledge from multiple LLMs rather than solely from continual training. The results and analysis are shown in Appendix
42.2
01.50
4122
Larad CLM
Lamus
3975
eTokinn Takox (Rillac)
Figure 2: Effect of the fused distribu- tions in accelerating the optimization pro- BBH, where the x-axis denotes the number of training tokens and the y-axis denotes the exact match accuracy.
4.4 ANALYSIS OF IMPLEMENTATION PROCESS
In this section, we delve into the crucial elements of FUSELLM's implementation, including the number of source LLMs, the criteria for token alignment, and the choice of the fusion function.
Number of source LLMs. In Table 4 we
Medel
BBH
CS
present the results of fusing different num-
33.1
$0776
18.11
bers of LLMs. We note that the performance of FUSELLM demonstrates apparent improve- ment as the number of models increases from I 3. Nevertheless, the benefits of integrat- ing additional models exhibit variations across benchmarks. Remarkably, a consistent per- formance improvement is observed in BBH. Whereas in CS or ME, the advantages are more prominent when fusing two models. This phe- nomenon may be attributed to the considerable performance differences among the three models on various tasks in BBH, while the performance differences in tasks of CS or ME are relatively smaller.
MPT
33.38
61.13
17.26
Llmm-2
70
63.76
14.83
Hama-2 CLM
40 (+1 (+1.869)
63 85 (+0.166))
14.83 (-1.3611)
Liama 2+ OpenLLaMM
41.00 (+3.27%)
(+1.061
15.51
(+6.02%)
Llamn2 MPT
41.16 (+3.688)
64.51 (+1.18%)
15.47 (45. 7496
FUSELLM
41.75 41.75(45.168))
64.56 (41.255)
15.56 (46.3693)
Table 4: Results of FUSELLM by incorporating varying numbers of models.
Criteria for token alignment. During the fusion of LLMs, ensuring the proper alignment of to- kens and vocabularies from multiple models is of paramount importance. In Table 5 (upper), we present comparison of two alignment criteria. It is evident that the proposed MinED method, which is based on minimum edit distance, consistently outperforms the EM method introduced by Fu et al. (003)). which relies on exact matching. We suggest that this performance enhancement results from MinED's s ability to relax the constraints of EM, as tokens separated by distinct tokeniz- ers within the same sequence often exhibit minor discrepancies. Consequently, MinED effectively supplements a considerable amount of useful token information while introducing negligible errors.
we intro- duce two variations of the fusion function for FUSELLM: one utilizing a distribution matrix with minimum cross entropy (MinCE) and the other adopting a weighted average of distribu- tion matrices based on cross entropy (AvgCE). A comparison of the two functions is presented in Table 5 (down). The findings demonstrate that FUSELLM with MinCE consistently out- performs AvgCE across all benchmarks. This can be attributed to the distortions introduced by the straightforward weighted summation used in AvgCE, which may diminish the distinct advantages of individual L.LMs.
Fusion function. In Section 3.3
Choice
Alignment Criteria
EM
41.37
13,49
64.24
MinED
41.75 (+0.43%)
15.56 (+0.45%)
64.56 (+0.50%)
Finsion Function
AvgCE
41.04
15.39
63.98
MinCE 41.75 (+1.73%) 15.56 (+1.10%) 64.56 (+0.91%)
Table 5: Comparison of different token alignment criteria (upper) and fusion functions (down).
8

---

Published as a conference paper at ICLR 2024
4.5 FUSELLM vs. KNOWLEDGE DISTILLATION
While knowledge distillation techniques can also be utilized to enhance a LLM's capabili- ties, FUSELLM stands out due to two distinct aspects, as previously outlined. In this section, we compare FUSELLM with traditional knowl- edge distillation. Specifically, we extract prob- abilistic distributions from Llama-2 13B and apply the conventional knowledge distillation method to transfer its abilities into Llama-2 7B. As illustrated in Table G the distilled model (Llama-2 KD) outperforms the original Llama- 2 7B across all benchmarks, demonstrating the effectiveness of knowledge distillation. How- ever, when compared to FUSELLM, the improvement achieved by Llama-2 KD is relatively modest, especially in the case of BBH (2.97% vs. 5.16%). This suggests that the superior results achieved by FUSELLM through the integration of three 7B models with diverse architectures via continual train- ing outweigh the benefits of simply distilling knowledge from a single 13B model. This observation highlights the idea that "More is different, but different can also be more" (Tay et al. 2022)
BRH
Llama 130
47.92
06.33
18.76
Opeal.LaMA
33,87
59.78
18.11
MPT
33.38
61.13
17.26
Llama-2
39.70
63.76
14.63
LJama-2 CLM
40.44
86%)
63.86 (+0.166))
14.83 (31)37))
Llama-2 KD FUSELLM
40.88
(+2. 97583
64.41 (+1.02%)
15.45 (45 60%)
41.75 (+5.166%)
64.56 (+1.25%)
(+6.36%)
Table 6: Comparison of FUSELLM and knowl- edge distillation. Llama-2 KD denotes the en- hanced Llama-2 7B achieved via knowledge dis- tillation from Llama-2 13B. Percentages indicate the rate of improvement compared to Llama-2.
4.6 FUSELLM VS. ENSEMBLE/MERGING
As previously mentioned, conventional tech- niques such as model ensemble and weight merging are commonly employed to fuse mul- tiple LLMs. To compare the efficacy of our FUSELLM with these existing fusion methods, we conduct experiments simulating scenarios where multiple LLMs originated from the same base model but were trained on distinct corpora. We first select three relevant domains (PhilPa- pers, NIH ExPorter, and USPTO Backgrounds) from The Pile and use 1 billion tokens from each domain to continually train Pythia 1B (Biderman et al. 2023), resulting in three distinct LLMS with identical structures. Then, we apply different fusion techniques to these LLMs: (I) The ensem- ble method calculates weighted average of the probabilities generated by all LLMs, considering the performance of each model; (2) The weight merging method merges multiple LLMs into single one within the parameter space, with the merging weights determined by model performance; (3) FUSELLM undergoes continual training on 0.1 billion tokens sampled from the three domains. The results of perplexity for FUSELLM and the other fusion methods on the test sets are presented in Ta- 7 We measure perplexity in bits per UTF-8 encoded byte (BPB) following the implementation in The Pile. We observe that after training with I billion tokens, the capabilities of the original LLM are transferred to each domain- -sseciifs LLM, resulting in decreased performance in other domains. While all fusion techniques achieves the lowest average perplexity across the three domains. This underscores its potential for harnessing collective knowledge more effectively than ensemble and weight merging methods.
Model
Phil
NTH
USPTO
Average
Pythi
0,9008
0.6740
0.6077
0.7275
Phil
0,8397
0.6861
0.6228
0.7162
NIH
0.9248
0.6215
0.6278
0.7247
USPTO
0.9296
0.6872
0.6017
0.7395
Ensemble
0.8960
0.6647
0.6180
0.7262
Weight Merging
0.8786
0.6496
0.6054
0.7112
LUSELLM
0.8463
0.6569
0.6068
0.7034
Table 7: Comparison of perplexity between FUSELLM and ensemble&wwight merging.
integrate the strengths of diverse models, FUSELLM consistently
CONCLUSION
In this study, we have explored the realm of knowledge fusion for LLMs create a unified model that combines the capabilities and distinctive strengths of multiple structurally diverse LLMs. We introduced a novel method, FUSELLM, which leverages the generative distributions of these source LLMSs to externalize their knowledge and employs them in the continual training of the target LLM. Through a series of experiments, we have demonstrated the superiority of FUSELLM over individ- ual source LLMs and established baselines. Notably, in simulated experiment featuring multiple structurally identical LLMs, FUSELLM has showcased its competitive effectiveness compared to ensemble and weight merging methods. Hence, the domain of LLMs fusion emerges more promising avenue for exploration, particularly given the diverse structures and substantial model sizes of L.MMs. We believe that these findings will inspire future research endeavors.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (No. 62176270), the Guangdong Basic and Applied Basic Research Foundation (No. 2883)))1501233)), and the Tencent Al Lab Rhino-Bird Focused Research Program.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306. 13649, 2023.
Devansh Arpit, Huan Wang. Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improv- ing model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 35:8265-8277, 2022.
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl- c: scalable and extensible approach benchmarking neural code generation. arXiv preprint arXiv:2208. 08222,, 2022.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Kat- suitho, Yoshino Koichiro, and Federmann Christian. Overview of the iwslt 2017 evaluation cam- paign. In Proceedings of the 14th International Workshop on Spoken Language Translation, pp-
2-14, 2017.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Infor- mation Processing Systems, 34:22405-22415 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
Christopher Clark, Kenton Lee, Ming-
Ming-Wei Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924-2936, 2019a.
Toutanova. Boolq:
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc Le. Bam! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5931-5937, 2019b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803 05457, 2018
10

---

Published as a conference paper at ICLR 2024
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, April 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359 2022.
Dheeru Dua, Yizhong Wang. Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 2368 2378, 2019.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step step reasoning. arXiv preprint arXiv:2301.12726. 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00033 2020.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, sep 2021.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Knowledge distillation of large language models. arXiv preprint arXiv:2306.0883 08543, 2023.
Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. International Conference on Learning Representa- tions, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv1150.00253,, 2015.
Cheng-Yu Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Rat- ner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-sten outperform- ing larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02:3 02301, 2023.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du and Min Lin. Lorahub: Effi- cient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13266 2023.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02556) 2023.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. Findings of the Association for 2020.
Computational Linguistics: EMNLP 2020, pp-
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietre and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2022.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
11

---

Published as a conference paper at ICLR 2024
Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt Kusner. No train gain: Revisiting efficient training algorithms for transformer-base language models. arXiv preprint arXiv:2307.06440, 2023.
Simran Khanuja, Melvin Johnson, and Partha Talukdar. Mergedistill: Merging language models using pre-trained distillation. In Findings of the Association for Compulational Linguisties: ACL- IJCNLP 2021, pp. 2874-2887, 2021.
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of Ilms. arXiv preprint WrXiv:3308.07317. 2023.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah Smith, and Luke Zettlemoye. Branch-train- Embarrassingly parallel training of expert language models. In First Workshop on Interpolation Regularizers and Beyond at NeurlPS 2022, 2022.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint N.Ki::2300006661 2023.
Nick Littlestone and Manfred K Warmuth. The weighted majority algorithm. Information and Computation, 108(2):212-2611 1994.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining 1lms at scale. arXiv preprint 04440423, 2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381-2391 2018
Kristine Monteith, James L Carroll, Kevin Seppi, and Tony Martinez. Turning bayesian model av- eraging into bayesian model combination. In The 2011 International Joint Conference on Neural Nenworks, pp. 2657-2663. IEEE, 2011.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 1525-1534, 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Baolin Peng, Chunyuan Li. Pengeheng Hc, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint 2rXhv:2304.03277 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2111)54885-551) 2020.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthicu Cord. Diverse weight averaging for out-of-distributiin generalization. Advances in Neural Information Processing Systems, 35: 15:10021-1083 2022.
Matthias C Rillig, Marlene Agerstrand, Mohan Bi, Kenneth A Gould, and Uli Sauerland. Risks and benefits of large language models for the environment. Environmental Science & Technology, 57 (9):3464 3466, 2023.
Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4)ee129, 2018.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 10.0/108. 2019.
12

---

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- Published as a conference paper at ICLR 2024
Sunny Sanyal, Jean Kaddour, Abhishek Kumar, and Sujay Sanghavi. Understanding the ef- fectiveness of early weight averaging for training large language models. arXiv preprint S3f1:22306003241, 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.046135 04661,,, 2022.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4323 4332, 2019.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks 2022.
and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.099
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0.1% extra compute. arXiv preprint arXiv:2210.11399, 2022.
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source commercially usable 11ms, 2023. Accessed: 2023-05-05.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307..00 092888 2023.
Iulia Ture, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: 1908. 08962, 2019.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint 2rXiv:2212.035333 2022a.
attention distillation for agnkstgi compression of pre-trained transformers. Advances in Neu- ral Information Processing Systems, 33:5776-5788, 2020.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jicyu Zhang, Satyen Subramaniam, Arjun Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scien- tific mrobeemssolvig abilities of large language models. In The 3rd Workshop on Mathematical Reasoming and Al at NeurlPS'23, 2023a.
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jian- feng Gao. Adamix: Mfxture-of-adapter for parameter-efficient tuning of large language models. arXiv preprint arXiv:2205./2444 2022b.
Yizhong Wang. Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning open resources. arXiv preprint 23XX::33060075.. 2023b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: fatte-of-the-and art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing:
System Dyiemonstrations, pp. 38-45, 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lpees, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Komblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 33965-23995 PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXhv:2304.1222222 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficien modules with arithmetic operations. arXiv preprint arXiv:2306. 14870, 2023.
A DETAILS OF TOKEN ALIGNMENT
For an input text, token alignment involves aligning two distribution matrices from two source LLMs. Therefore, the alignment comprises two dimensions: token-wisc with respect to the text and distribution-war with respect to the vocabulary. To provide a clear explanation, we show an example of different methods for token alignment in Figure
In the token dimension, we utilize the dynamic programming approach to recursively minimize the total cost of editing one sequence of tokens align with another. When the mapped tokens are iden- tical, such as the token "now" in the given example, these tokens are successfully aligned, allowing for the corresponding distributions align subsequently. However, when the mapped tokens exhibit differences, such as the "get" and "gets" tokens in the example, the previous EM method proposed by Fu et al. (2023) does not align these tokens, resulting the distributions degenerating into one- hot vectors. In contrast, our proposed MinED) method successfully aligns the "gets" token with the "get" token, as they exhibit the minimal edit distance in the vocabularies from the two source LLMs.
Concerning the distribution dimension, the alignment is performed between two vocabularies from different tokenizers of two source LLMs. Therefore, for distribution values with identical tokens, such as "current 0.05" and "current 0.04", they will be aligned effectively. For distribution values involving different tokens, such as "immediate 0.04" and "immediately 0.03", the EM method dis- regards this value. However, the proposed MinED method maps "immediately" to "immediate" due to their minimal edit distance, resulting in the successful alignment of these distribution values.
LLM Tokenization
LLM Tokenization
LLM I Tokenization
LLM 2 Tokenization
taken aligned
token aligned
BOW
noW
now
now
0.0
dists Rostion alignal
n91
now
0.50
distriaation aliguad
0.91
now
0.05
dierrivation aligned
0.04
0.55
distribation aligned
0.04
currast
immoduaee
0.04
distrbuttoon not aligned
insoediattnly
immedfite
0.04
mappping to "immedione""
0.08
immedianely
toben not aligzed
napppngg to "guct"
get
gets
get
gets
pet
0.50
gec
0.50
mapping
0.60
gels
gnins
0.38
degenerution to one bot
1.00
gel
0.38
distribttion aligned
0.22
gains
obcatts
0.10
BM
oboziss
0.10
diceriaation aligned
0.13
obtains
MnCD
Figure 3: An example of different methods for token alignment.
B DETAILS OF MINIPILE
MiniPile is curated from The Pile (Gao et al. 2020) through a three-stage pruning process: (1)
extracting embeddings for all documents with E5- (Wang et al., (022a), which is a sentence embedding model, (2) clustering the embeddings using K-means, and (3) filtering out low-quality clusters. Therefore, MiniPile retains a compact scale while exhibiting extensive diversity, making prevalent choice for efficient training of L.MMs (Kaddour et al. 2023] (Sanyal al. 2023
14

---

Published as a conference paper at ICLR 2024
C TRAINING DETAILS
Our model is optimized using the AdamW optimizer with 31 0.9 and 32 = 0.95, with gradient clipping set to 1.0 and weight decay to 0.1. A cosine learning rate schedule employed, with maximum learning rate of le-5 and warmup ratio of 0.008. To accelerate the training, we employ packing (Raffel et al. 2020), where multiple training instances are grouped into single sequence separated by end-of end-of-sequence tokens, allowing for training more tokens in each batch.
D ADDITIONAL EVALUATION RESULTS
To further illustrate the effectiveness of FUSELLM, we incorporate additional generative bench- marks related to knowledge-buse machine translation, and theorem application. The results presented in Table 8ilee8ghiggi hiihhlight Luse superiority over all source LLMs across all tasks.
qnextiionanswerrinn reading comprehension, content analysis,
TriviaQA (Joshi et al. 2017) benchmark to evaluate the knowledge-base
ability. We conduct a shot evaluation and report the EM accuracy.
DROP (Dua et al. 2019) is a benchmark to evaluate the reading comprehension ability. We
conduct few-shot evaluation with CoT prompts and report the EM accuracy.
LAMBADA (Paperno et al. 2016) is benchmark to evaluate the content analysis ability. We
conduct a zero-shot evaluation and report the EM accuracy.
IWSLT2017 (Cettolo al. 2017) benchmark evaluate the machine translation ability. We conduct a zero-shot evaluation and report the BLEU (Papineni et al. 20022) score.
SciBench (Wang et al. 2023a) is a benchmark to evaluate the theorem application ability. We conduct few-shot evaluation with CoT prompts and report the EM accuracy.
Task
OpenI.LaMA
MPT
Llama-2
Llama-2 CLM
FUSELLM
TrivialQA
39.96
28.89
52.46
53.14 (+1.30%
54,49 (+3.87%)
DROP
22.31
23.54
27.25
28.51 (44.62%)
28.97 (+6.31
LAMBADA
70.31
70.08
73.28
73.45 (40.23%)
73.72 (+0.60%)
IWSLT2017
5.51
5.49
6.48
6.91 (+6.64%)
6.75 (+4.17%)
SeiBench 0.68
0.88
0.14
0.94 (+571433%)
1.65 (+1778.57%)
Table 8: Overall results of FUSELLM and baselines in additional generative benchmarks, where percentages indicate the rate of improvement/decreese compared to Llama-2.
E FUSELLM vs. PREVIOUS MODEL FUSION METHODS
The motivation behind FuseLLMM is integrate the col- lective knowledge of multiple L.MMs with diverse archi- tectures and pre-training corpora. Consequently, the tra- ditional fusion method of model merging, which demands identical model architectures, is not directly applicable in this context. While the model ensemble technique aggre- gates predictions from multiple LLMs, the brawback lies the substantial memory and time costs when maintain- ing multiple source LLMs during inference. We further compare FUSELLM with an ensemble method for LLMs, LLM-BBender (Jiang et al. 2023). which ranks and com- bines the output texts from multiple LLMS with ranker and fuser models. Specifically, we conduct experiments on the Big- BighB Hard and MultiPL-E benchmarks using the open-source ranker and fuser models. Notably, the CommonSense benchmark, which utilizes perplexity-based evaluation, cannot adapt the LLM-Blender method. The experimental results are shown in Table 9 where LLL--Blender (Rank&Fuse) refers to using the ranker to obtain the top three results and then using the fuser to combine them, and LLM-Blender (Rank) represents simply using the ranker to obtain the top one result. We observed a notable performance deterioration after fusion when employ- ing both the ranker and fuser. This could be attributed to the fuser model's ss training within the
Model
BBI
ME
OpenLLaMA
33.87
18.11
MPT
33.38
17.26
Llama-2
39.70
14.63
LLM-Blender (Rank&Fuse)
24.48
0.06
LLMM-Blender (Rank)
37.17
17.85
Llama-2 CLM
40.44
14.83
FUSELLM
41.75
15.56
Table 9: Comparison of FUSELLM and -LM-BBlnder.
15

---

Published as a conference paper at ICLR 2024
instruction-tuning context, potentially leading to inadequate generalization to the test tasks. Further- more, while LLM-Blender (Rank) outperforms the LL -leeddrder (Rank&Fuse), it remains inferior to the best pest-performing source LLM. This suggests the ranker model's inability discriminate the optimal responses when combining different LLMs efficiently.
F INCORPORATING INSTRUCTION-
MODELS WITH FUSELLM
Recall that the proposed FUSELLM involves extracting distribution matrices from multiple distinct source LLMs and continually training the target LLM. Therefore, FUSELLM also applicable to instruction-tuning models, provided that all corresponding continual-triining samples adhere to the instruction-tuning format and mask the instruction part when calculating the training loss.
To confirm this, we conduct new experiments on the fu- sion of instruction-tuning LLMs. Specifically, we ini- tially fine-tune Llama-2, OpenLLaMA, and MPT us- ing 20k samples from Evol-Instruct (Xu et al.] 2023). ShareGPT (Chiang et al., 2023), and Open-Platypus (Lee jet al.] 2023) datasets, respectively. Consequently, the three source LLMs transitioned into instruction-tuning L.Mss. Then, we sample another 5k samples from each of the aforementioned datasets to create a corpus for con- tinual training, specifying Llama-2 Evol-Instruct as the target LLM for knowledge fusion. We assess the instruction-following performance the Vicuna Benchmark using GPT-4 an evaluator fol- lowing Chiang et al. (2023), which gives score from I to 10 for each answer. The results shown in Table [0] demonstrate that FuseLLM surpasses each individual source instruction tuning LLM, achieving the best performance with GPT-4 judgment.
Model
Vicuna Benchmark
Opent weenTTaae aMA ShareGPT
7.23
MPT Open- Peet-playyp
Llama-2 Evel-Instruct
7,88
Llama-2
(+1
FUSELLM
8,16 (-3.55%)
Table 10: Results of fusing instruction- tuning models with FUSELLM.
G CAUSE OF PERFORMANCE IMPROVEMENT
To further demonstrate that performance improvement stems from the integration of knowledge from multiple LLMs rather than solely from continual training, we conduct an evaluation on alternative corpus, RedPa- jama (Computer) (003).. To mitigate the impact of different tokenizers, we employ the bits per UTF-8 encoded byte (BPB) metric proposed by Gao et al.] (2020), where a smaller value indi- cates lower perplexity. Then, we compute the percentage of test samples in each domain ex- hibiting decreased BPB from Llama-2 CLM to FUSELLM and from Llama-2 to OpenLLaMA MPT. The results in Figure 4 suggest that when FUSELLM outperforms Llama-2 CLM, the performance of LaM or MPT typ- ically surpasses that of Llama-2, as evidenced by the slashed bars in each domain. This phenomenon is particularly pronounced domains such as Arxiv, StackExchange, Wikipedia, and Github, where it exceeds 95%. This compelling evidence suggests that the performance enhancements achieved by FUSELLM are indeed attributed to the integration of knowledge from multiple LLMs.
Aai
Wälpedie
Gabub
EM <l FLM
Beoka
(rpest LaMA xP -1bmas
Peneetag indecco with decrease IPD
Figure 4: Perplexity comparison on RedPajama. The bars denote the percentage of examples with reduced perplexity when transitioning from CLM to FUSELLM (solid) and from Llama-2 to Open- LLaMA or MPT (slashed).
H WEIGHT FOR LOSS COMBINATION
As alternative to the static weight for À in Eq we consider dynamic design, which is referred to as teacher annealing (Clark et al. 2019b). This technique gradually increases X during the training process, giving preference
Design
BBH
ME
CS
Dynamic Statie
41.75
15.52
64.50
41.75 (+0.00%)
15.56 (40.25%)
64.56 (+0.09%)
Table 11: Comparison of different designs for X.
16

---

Published as a conference paper at ICLR 2024
to Lrssion initially and subsequently shifting the focus to LCLM- In this experiment, set the À to increase linearly from 0.7 to 1.0. Table [ presents comparison between the static and dynamic designs of X, demonstrating that the two approaches yield comparable performance. Therefore, opt for the static design for simplicity.
I USING STRONGEST LLM AS TARGET
In the experiments, we consistently employ Llama-2 as the target LLM to maintain fixed setup. We further conduct supplementary experiments code generation tasks using the strongest LLM, OpenLLaMA, as the target LLM. The results are shown in Table 12 We observe decrease in the performance of OpenLLaMA CLM compared to the source OpenLLaMa and MPT. This decline is attributed to the inclusion of the StarCoder (Li et al., 2023) data in OpenI
pre-training corpus, whereas there is limited amount of code data our continual training corpus. For the same reason, even though FuseLLM showcases improved performance compared to OpenLLaMA CLM, still lags behind the original LaeALLaMA.
Task
OpenI.LaMA
MPT
IJama-2
OpenI.LaMA CLM
FUSELLM
C++
14.47
13.11
7.45
12.89
14.01
Go
68.2U
66.96
57.02
68.48
69.32
Java
14.28
13.42
10.31
12.80
13.26
JavaScript
17.61
13.01
13.17
15.50
17.02
PHP
11.24
9.53
9.75
10.06
10.09
Python
15.96
17.24
13.85
15.16
16.06
7.52
4.53
4.97
6.89
Ruby
10.37
9.44
10.03
Rust
6.18
8.29
6.77
5.99
5.65
TypeScript
15.31
14.13
12.61
14.10
15.37
Avg. 10 Tasks
18.11
17.26
14.63
17.13
17.75
Table 12: Overall results of FUSELLM using OpenI.LaMA as the target LLM code generation evaluations on MultiPL-EE (ME).
J CASE STUDIES
In Table 13 Table 14 and Table [5] present case studies to demonstrate how FUSELLM com-
bines the strengths of multiple source LLMs to produce accurate results in different tasks.
17

---

Published as a conference paper at ICLR 2024
BBH; Logical Deduction objects);
Question: The following paragraphs each describe a set of three objects arranged in a fixed order. The statements are logically consistent within each paragraph. A fruit stand sells three fruits: oranges, wa- termelons, and pears. The oranges are the second-most expensive. The pears are more expensive than the watermelons. Options: (A) The oranges are the second-most expensive (B) The watermelons are the second-most expensive (C) The pears are the second-most expensive
Target: (A)
OaeALL Let's think step by step. (1) The oranges are the second-moo most expensive: "(expensive) ? or- anges watermelons ? (expensive)". (2) The pears are more expensive than the watermelons: "(expensive) oranges pears watermelons (expensive)". (3) Combining (1) and (2) we get the following ordering: "(ex- pensive) oranges pears watermelons (expensive)". According to this ordering, the second-most expensive fruit is the oranges. The oranges are the second-mos most expensive. So the answer is (A).
"(most expensive)? or- anges? (least expensive)". (2) The pears more expensive than the watermelons: "(most expensive)? oranges? pears? (least expensive)". (3) Combining (1) and (2) we get the following ordering: "(most expensive) oranges pears (least expensive)". According this ordering, the second-most expensive fruit is the oranges. The oranges are the second-most expensive. So the answer is (A).
MPT: Let's think step by step. (I) The oranges the second-most expensive:
Llama-2: Let's think step by step. (1) The oranges are the second-most expensive: "(second-most expensive) oranges (most-expensive)". (2) The pears are more expensive than the watermelons: oranges pears mnost--xpensive"". (3) Combining (1) and (2) we get the follow- ing ordering: (xecomddmmoxt-expessivvv oranges pears (most-expensive)". According to this ordering, the second-most most expensive fruit the pears. The pears are the second most expensive. So the answer (C).
1lama-2 CLM:: Let's think step by step. (1) The oranges are the second-most expensive: "(sccond-most) oranges ? (most)". (2) The pears are more expensive than the watermelons: "(more) ? pears watermelons
(less)". (3) Combining (I) and (2) get the following ordering: "(second-most) oranges pears (most)
watermelons". According to this ordering, the second-most expensive fruit is the pears. The pears are the
second-most expensive, So the answer is (C).
FUSELLM: Let's think step by step. (1) The oranges are the second-most expensive: "(second-mos) oranges ? (most) ? (least) ? (least) ? (most) ? (second-most).. (2) The pears are more expensive than the watermelons: "(more) pears (less) ? watermelons ? (most) ? (least) (least) (most)". (3) Combining (1) and (2) we get the following ordering: "(second- -most) oranges pears watermelons (most)". According to this ordering. second-most expensive fruit is the oranges. The oranges are the second-man most expensive. the answer is (A).
Table 13: Case studies on the Logical Deduction (3 objects) task.
18

---

Published as a conference paper at ICLR 2024
BBH; Hyperbaton;
Question: Which sentence has the correct adjective order: Options: (A) repulsive prismlike glass shoe (B)
glass repulsive prismlike shoe
Target: (A)
OpenLLaMA: Let's think step by step. When there is more than one adjective before a noun, the adjectives
need to respect the following order before noun: "[I. opinion] [2. size| [3. age] [4. shape] [S. color] [6.
origin] [7. material] [8. purpose] noun". Option (A): "repulsive prismlike glass shoe". (1) "repulkive" falls into the opinion category. (2) "prismlike" falls into the shape category. (3) "glass" falls into the material category. Option (A) has the following udjective order: [1. opinion] [3. shape] [6. material] (or, in numeric ; 6 correct, (A) has the correct ordering. Option (B): "glass repulsive prismlike shoc". Option (B) has the following adjective order: [6. material] [1. opinion] [3. shape] (or, in numeric terms, 6 3). Because 6 is (A),
terms. 1 3 6). Because
not correct, (B) does not have the correct ordering. So the answer
MPT: Let's think step by step. When there is more than one adjective before noun, the adjectives need to respect the following order before noun: "[I. opinion] [2. size] [3. age] [4. shupe] [5. color] [6. origin] [7. material] [8. purpose] noun". Option (A): "repulsive prismlike glass shoe". (1) "repulsive" falls into the opinion category. (2) "prismlike" falls into the shupe category. (3) "glass" falls into the material category. Option (A) has the following adjective order: [I. opinion] [3. material] 12. shape] (or, numeric terms, I 32) Because 3 2 is not correct, (A) does not have the correct ordering. Option (B): "glass repulsive prismlike shoe", Option (B) has the following adjective order: [1. opinion] [3. materiall 12. shape] (or, in numeric terms, 2 ). Because 3 is correct, (B) has the correct ordering. So the answer is (B).
Llama-2: Let's think step by step. When there is more than one udjective before noun. the adjectives need to respect the following order before a noun: "[1. opinion] [2. size] [3. age] [4. shape] [5. color] [6. origin] [7. material] [8. purpose] noun". Option (A): "repulsive prismlike glass shoe". (1) "repulsive" falls into the opinion category. (2) "prismlike" falls into the shape category. (3) "glass" falls into the material category. Option (A) has the following adjective order: [1, opinion| 12. sizel [3. agel [6. origin| 17. material] (or, in numeric terms, 1 2 7). Because 1 i 22 6 is not correct, (A) does not have the correct ordering. Option (B): "glass repulsive prismlike shoe". Option (B) has the following adjective order: [1. opinion] [2. size] [3. age] [4. shape] [7. material] (or, in numeric terms, 34 7). Because ;; 2 2 3 4; is correct, (B) has the correct ordering. So the answer is (B).
1lama-2 CLM: Let's think step step. When there more than one adjective before noun, the adjectives need to respect the following order before noun: "[1. opinion] [2. size| [3. agel |4. shape] [5. color] [6. origin] 17. material] [8. purpose] noun". Option (A): "repulsive prismlike glass shoe". (1) "repulsive" falls into the opinion category. (2) "prismlike" falls into the shape category. (3) "glass" falls into the material category. Option (A) has the following adjective order: [1. opinion] [2. size] [3. age] [4. shape] [5. color] correct, (A) has the correct ordering. Option (B): "glass repulsive prismlike shoe". Option (B) has the following adjective order: 17. material] 11. opinion] [2. sizel [3. agel [4. shape] (or, in numeric terms, 7 1 2 3 4). Because 7 12;
17. material] (or, in numeric terms, 1 2 3 4 7). Because i ; 3 4
is correct, (B) has the correct ordering. So the answer is (B).
FUSELLM: Let's think step by step. When there more than one adjective before a noun, the adjectives need to respect the following order before noun: "[I. opinion]
[2. size] [3. age] [4. shape] [5. color] [6. origin] [7. material| [8. purpose] noun". Option (A): "repulsive prismlike glass shoe". (1) "repulsive" falls into the opinion category. (2) "prismlike" falls into the shape category. (3) "glass" falls into the material category. Option (A) has the following adjective order: [1. opinion| [2. sizel [3. age] [4, shape] [5. color] I6. origin] 17. material] (or, in numeric terms, 3 4 5 7). Because 1 2 4 5 6 1 is correct, (A) has the correct ordering. Option (B): "glass repulsive prismlike shoe". Option (B) has the following adjective order: [1. opinion] [7. material] [2. size] [3. age] [4. shape] [S. color] (or, in numeric terms, 1
2 3 4 5). Because ; 7 i 2; 3; 4; 5 is not correct, (B) does not have the correct ordering. So the answer is (A).
Table 14: Case studies on the Hyperbaton task.
19

---

Published as a conference paper at ICLR 2024
BBH; Disambiguation QA;
Question: In the following sentences, explain the antecedent of the pronoun (which thing the pronoun refers to), or state that it is ambiguous. Sentence: The technician told the customer that he could pay with cash. Options: (A) The technician could pay (B) The customer could pay (C) Ambiguous
Target: (R)
OpenLLaMA: Let's think step by step. Here we need to determine who the pronoun "he" might be refer- ring to. There are two possible referents for "he", namely the technician and the customer. The verb "told" might be able to help us determine which one is more likely (if either). Let X be the technician and Y the customer. The sentence is then of the form "X told Y that (X or Y) could pay with cash." Let's consider Y first: "X told that Y could pay with cash." This case makes sense, because X might want to inform Y that Y could pay with cash. Now, consider X: "X told that X could pay with cash." This case also makes sense, because X might want to inform Y that could pay with cash. Because both X and Y are possible at the same time, we conclude that the antecedent of the pronoun is ambiguous. So the answer is (C).
MPT: Let's think step by step. Here we need to determine who the pronoun "he" might be referring to. There are two possible referents for "he", namely the technician and the customer. The verb "told" might be able to help us determine which one is more likely (if either). Let X be the technician and Y the customer. The sentence is then of the form "X told Y that (X or Y) could pay." Let's consider Y first: "X told Y that Y could pay." This case makes sense, because Y might want to pay with cash. Now, consider X: "X told Y that X could pay." This case doesn't make sense, because X would already have the information that X could pay. because is information about themself. Because in this context, X the technician and Y is the customer, the answer should he the customer. So the answer is (B).
1lama-2: Let's think step by step. Here we need to determine who the pronoun "he" might be referring to. There are two possible referents for "he", namely the technician and the customer. The verb "told" might be able to help us determine which one is more likely (if either). Let X he the technician and Y the customer. The sentence is then of the form "X told Y that (X or Y) could do something." Let's consider Y first: "X told Y that Y could do something." This case makes sense, because Y is the one who is being told that they can do something. Now, consider X: "X told Y that could do something." This case doesn't It make sense, hecause X is the one who is telling that they can do something. Because in this context, X is the technician and Y is the customer, the answer should he the technician. So the answer is (A).
Ilama-2 CL.M: Let's think step hy step. Here need determine who the pronoun "he" might he referring to. There are two possible referents for "he", namely the technician and the customer. The verb "told" might be able to help us determine which one is more likely (if either). Let X be the technician and Y the customer. The sentence is then of the form "X told Y that (X or Y) could pay." Let's consider Y first: "X told Y that Y could pay." This case doesn't make sense, because Y is the receiver of the information, it is now X' turn pay. Now, consider X: "X told Y that X could pay." This case makes sense, hecause X is the one who is telling Y that X can pay. Because this context, X the technician and Y is the customer, the answer should be the technician. So the answer (A).
FUSELLM: Let's think step by step. Here we need to determine who the pronoun "he" might be referring to. There are two possible referents for "he", namely the technician and the customer. The verb "told" might be able to help us determine which one is more likely (if either). Let X he the technician and Y the customer. The sentence is then of the form "X told that (X or Y) could pay." Let's consider Y first: "X told Y that Y could pay." This case makes sense, because Y is the customer, and the customer is the one who is paying. Now, consider X: "X told Y that X could pay." This case doesn't ' make sense, because X is the technician, and the technician is not the one who is paying. Because in this context, X is the technician and Y is the customer, the answer should be the customer. So the answer is (B).
Table 15: Case studies on the Disambiguation QA task.
20


Reviews:
[{"review_id": "ALdDl1wprn", "reviewer": "Reviewer_fXfT", "summary": "This paper proposes to fuse the power of multiple pretrained LLMs (LLaMA2-7B, MPT-7B, and OpenLLaMA-7B), via a continual pretraining on top of LLaMA2 with the 1 Million-document MiniPile corpus, where the training objective consists of a weighted combination of a normal causal LM loss and a multi-teacher knowledge distillation loss. The result model FuseLLM performs better than LLaMA2 on reasoning (BBH), commonsense, and code generation (MultiPL-E) benchmarks.", "questions": "See \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper is well-written and easy-to-understand, and the details are clearly presented.", "weaknesses": "* The proposed approach relying on continual pretraining with a small pretraining corpus (MiniPile) looks limited to fusing the knowledge of the raw pretrained version of LLMs, but not applicable to \"-Chat\" or \"-Instruct\" models (e.g., Vicuna, Alpaca, MPT-instruct, ChatGPT, etc.)\n\n* All the benchmarks in this paper are classification tasks. It is unclear if the FuseLLM would practically produce better text outputs on instructions, such as real-world human instructions, e.g., Alpaca (https://raw.githubusercontent.com/tatsu-lab/stanford_alpaca/main/alpaca_data.json) and some generative tasks in Big-Bench, e.g., word_unscrambling.\n\n*  The performance improvement from the proposed fusion approach is not significant enough. Specifically,\\\n      on reasoning tasks: 39.70 (LLaMA2) -> 40.44 (LLaMA2 + MiniPile) -> 41.78 (LLaMA2 + MiniPile  + model fusion)\\\n      on commonsense tasks: 63.76 (LLaMA2) -> 63.86 (LLaMA2 + MiniPile) -> 64.56 (LLaMA2 + MiniPile + model fusion)\\\n      on code generation tasks: 14.63 (LLaMA2) -> 14.83 (LLaMA2 + MiniPile) -> 15.56 (LLaMA2 +MiniPile + model fusion)\\\nGiven the MiniPile continual pretraining is not a key contribution of this paper, the actual accuracy improvement brought by the proposed model fusion on commonsense and code generation is actually smaller than 1 percent. \n\n* The evaluation of this paper doesn't include the comparison of FuseLLM with any of the previous model fusion techniques, e.g., LLM-Blender (Jiang et al.) mentioned in the \"related work\" section. Notably, the LLM-blender also doesn't require heavy continual training on a 7B large model, which is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hf3cIMLXrK", "reviewer": "Reviewer_CN5G", "summary": "This work introduces a novel training method that utilizes prediction probabilities from LLMs as a teaching signal to tuning an LLM. Unlike conventional teacher-student distillation approaches, this work proposes token alignment to handle vocabulary differences and fusion methods to combine the predictions of multiple LLMs, which means the teaching signal is a mixed prediction distribution.", "questions": "To my understanding, teacher models are frozen during the training process, allowing us to infer and store their prediction distributions before training. This enables the utilization of larger models, such as employing 70+B LLMs as teachers. Since teacher inference occurs only once, the additional cost should not be significant.\n\nRegarding the Token alignment, I am still unsure the method after reading Appendix A. Can you provide an example or some explanations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Training LLMs is expensive, making the motivation to merge existing LLMs valuable.\n\nThe paper is well-written and easy to follow, and the experiment setup and ablation studies are solid and insightful.", "weaknesses": "I think the proposed method presents a promising training algorithm for leveraging multiple LLMs as teachers. However, it's better to have insights about what the student LLM has learned. The best empirical fusion method, Min Cross-entropy, utilizes token-level probability maximization across multiple teachers as the teaching signal. This teaching signal is not intuitive. Considering Hinton's dark knowledge theory, why does token-level max pooling outperforms average pooling?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0TyPWbz99y", "reviewer": "Reviewer_M2ne", "summary": "The paper discusses the challenges and drawbacks of training large language models (LLMs) from scratch and proposes a cost-effective approach called knowledge fusion (FuseLLM). Knowledge fusion aims to merge existing pre-trained LLMs by leveraging their collective knowledge and strengths to create a more powerful single LLM. The study validates this approach using three different LLM architectures and demonstrates improved performance in areas like reasoning, commonsense, and code generation.", "questions": "See in the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposed a novel fusion approach for LLMs from a probabilistic distribution perspective.\n- The experiments are relatively sufficient.\n- Then writing and presentation are clear and easy to read.", "weaknesses": "- The presented results could not support that the FuseLLM cuts the cost of initial training and has superiority to other fusion strategies. Lack of quantitative comparison (both results and the cost) between other strategies (continual training, ensemble, merging ...) and the proposed FuseLLM. I only find Table 7 that presents the perplexity.\n- The results (Table 1, 2 and 3) show that FuseLLM could not outperform original LLMs in some tasks (some in BBH and all the ME). Does it mean that the FuseLLM only work in some domains? Or in another word, does the FuseLLM could only inherit specific capabilities of LLMs? Have the authors studied this phenomenon? Overall, the improvements are not remarkable, will such fusion effort necessary (with a maybe large training cost)?\n- The experiments are only based on Llama-2, OpenLLama and MPT and on 7B scale. Why choosing these three types of models? Can the authors give some explanations? We know that the model will emerge powerful capabilities via scaling up. Maybe the fusion effectiveness will drop with the parameter scale grows. Authors may provide a discussion about this argue.\n- How do the Avg. values be calculated in the last lines of Table 1,2 and 3? Directly averaging all the values does not make sense due to the disparate data scales of each task.\n- Are the Q_t in Eq. 4 and the Q_t in Eq. 2 the same? I guess the Q_t in Eq. 4 should be used another notation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UlbMRW0ApA", "reviewer": "Reviewer_jpJv", "summary": "The paper states that current method for training a LLM from scratch requires a significant cost and may result in redundant capabilities. While current model ensembling method and direct parameters pooling are either computational expensive or impractical due to the difference of model structure. Based on this the paper treats the probabilistic matrix generated by a LLM given a text as the knowledge of LLM, and proposed a framework named FuseLLM to minimize the discrepancy between the probabilistic matrix of the target LLM and the source LLMs. The empirical study of the proposed approach show the strong performance of the method. Besides, the paper also compare FuseLLM to knowledge distillation and the traditional model ensembling / merging methods to further justify the performance of the FuseLLM.", "questions": "1. For the detailed implementation part, I am a little bit confused. Can you explain in details how MinED works for token alignment, and how MinCE/AvgCE merge different matrices into P_t?\n\n2. As a combined objective is used in FuseLLM (5), what is the choice of $\\lambda$ in your experiment? I saw from Appendix E that for dynamic design $\\lambda$ is set from 0.7 - 1.0, while 1.0 reduces the (5) to the regular CLM objective. Could you provide a more detailed elaboration on this hyperparameter?\n\n3. In Table 1, interestingly FuseLLM receive significant improvement in several tasks (e.g., Hyperbaton, Web of Lies) from MPT and OpenLLaMA, which is actually expected. While there are still a lot of the tasks like Formal Fallacies, Causal Judgement, and most of the code generation tasks, where significant improvement are not observed. What if the strongest source model is used for continuing training, instead of the Llama-2? For example, OpenLLaMA for code generation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The propose approach which allows merging knowledge from multiple sources of LLMs is novel and straightforward.\n2. The empirical study shows strong performance of the proposed strategy.", "weaknesses": "The implementation part is not very well written, I cannot completely understand the specific details toward the mechanism of the alignment and fusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)