Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization

Authors: costas  mavromatis
petros  karypis
george  karypis
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 03:30:48

Abstract: Fusing knowledge from multiple Large Language Models (LLMs) can combine their diverse strengths to achieve improved performance on a given task. However, current fusion approaches either rely on learning-based fusers that do not generalize to new LLMs, or do not take into account how well each LLM understands the input. In this work, we study LLM fusion at test-time, which enables leveraging knowledge from arbitrary user-specified LLMs during inference. We introduce Pack of LLMs (PackLLM), an effective method for test-time fusion that leverages each LLM’s expertise, given an input prompt. PackLLM performs model fusion by solving an optimization problem for determining each LLM’s importance, so that perplexity over the input prompt is minimized. First, our simple PackLLM-sim variant validates that perplexity is a good indicator for measuring each LLM’s expertise. Second, our PackLLM-opt variant approximately solves the perplexity minimization problem via a greedy algorithm. The derived importance weights are used to combine the LLMs during inference. We conduct experiments with over 100 total LLMs on a diverse set of tasks. Experimental results show that (i) perplexity is a reliable measure for LLM fusion, (ii) PackLLM outperforms test-time fusion baselines by 1.89% accuracy points, (iii) PackLLM can leverage new LLMs to improve performance over learning-based fusion approaches by 3.92–11.94% accuracy points, and (iv) PackLLM benefits over selecting the best or largest model and model merging in certain cases. Our code is provided at [https://github.com/cmavro/PackLLM](https://github.com/cmavro/PackLLM).

Main paper: 
 introduction: 
 Large Language Models (LLMs) (Brown et al., 2020; Touvron et al., 2023) have achieved state-of-the-art performance on a variety of natural language processing (NLP) tasks and are beginning to be applied as Foundation Models (Bommasani et al., 2021) for a gamut of applications. As a result, we observe a sharp increase of newly released LLMs, that are pretrained on diverse corpora or fine-tuned to solve specific tasks. Moreover, it has become a strategic interest that corporations train and release their own LLMs, amplifying the number of available LLMs. LLMs usually have architectural differences as well as they use different pretraining data, and as a result, their knowledge expertise is diversified. Therefore, an emergent question is "How can we effectively combine knowledge of the available LLMs to improve downstream performance?".Fusing knowledge from different models is a long-standing problem in machine learning (Ho, 1995; Sollich & Krogh, 1995; Kuncheva & Whitaker, 2003; Schapire, 2003) . Regarding LLMs, current approaches train dedicated fusion modules, such as neural networks (Wang et al., 2023) or additional LLMs (Jiang et al., 2023b; Wan et al., 2024; Ding et al., 2024) , that learn to combine LLMs from a seed set. However, these approaches are not modular, meaning they need to undergo expensive and time-consuming retraining to encompass newly released LLMs or remove specific LLMs, e.g., due to licensing issues.We present Pack of LLMs (PackLLM), a test-time fusion method. As shown in Figure 1 , PackLLM does not require any training of fusion modules, while it can combine arbitrary user-specified LLMs during inference. In order to fuse knowledge from the seed LLMs at test-time, PackLLM performs a weighted ensemble of the output logits by posing an optimization problem at test-time. PackLLM minimizes the perplexity over the input prompt, so that the fused LLM understands the task better. First, our simple PackLLM sim variant validates that perplexity is a good indicator for measuring each LLM's expertise. Second, our PackLLM opt variant approximately solves the perplexity minimization problem via a greedy algorithm. The derived importance weights are used for combining the LLMs during inference.We conduct experiments with over 100 total LLMs and evaluate performance on language modeling tasks (Section 4) as well as on downstream tasks (Section 5). Experiments show that PackLLM's perplexity-based framework is a reliable indicator for determining fusion, outperforming approaches such as cBTM (Gururangan et al., 2023) and DExperts (Liu et al., 2021) that do not take into account how well the LLMs understand the input, while its overall performance scales better with respect to these baselines as we increase the number of expert LLMs. In addition, PackLLM achieves a significant improvement of 1.72-1.89% accuracy points (averaged over 25 tasks) over existing test-time fusion approaches, such as top expert selection and uniform ensemble. Furthermore, by employing newly released LLMs, PackLLM outperforms competing learning-based fusion approaches, such as FuseLLM (Wan et al., 2024) and FoE (Wang et al., 2023) , by 3.92-11.94% accuracy points, using as little as 3.92× fewer parameters. Our contributions are summarized below:• Problem Formulation: We study fusion of LLMs at test-time as a weighted ensemble and pose a perplexity minimization problem for determining the LLM importance weights. Experimental results show that PackLLM benefits over selecting the best or largest model and model merging in certain cases.• Algorithm: We present PackLLM opt that solves the perplexity minimization problem via a greedy algorithm. We also present PackLLM sim , a simple, but effective, perplexity-based ensemble.  
related work: 
 Regarding fusion of LLMs, weight merging (Yadav et al., 2023) and mixture-of-experts (Komatsuzaki et al., 2022) are techniques that fuse knowledge stored in LLMs by combining their model parameters. However, such approaches require that the given LLMs share the same architecture, which is limited with the current availability of powerful LLMs of varying sizes. An alternative technique is majority voting (Li et al., 2024a) , but can only be applied to certain downstream tasks, such as classification tasks.Recent successful fusion approaches often follow a learn-to-fuse framework (see Figure 2 , left), where specialized learnable modules (Wang et al., 2023; Sukhbaatar et al., 2024) or even LLMs (Jiang et al., 2023b; Wan et al., 2024; Ding et al., 2024) , are trained to combine knowledge from a set of seed LLMs. However, these approaches are not modular, meaning they cannot generalize to arbitrary user-specified LLMs and require additional retraining when models need to be added in or removed from the seed set. Approaches, such as (Feng et al., 2024) , fuse the generations of different LLMs, but require a strong (large-scale) LLM as a fuser, which is computationally expensive.On the other hand, model ensemble approaches (Liu et al., 2021; Gururangan et al., 2023; Liu et al., 2024; Li et al., 2024b ) perform fusion at the output-level, i.e., output token logits or probabilities. However, effectively combining the LLMs during test-time, i.e., accurately determining each LLM's importance, has not been extensively studied as these approaches do not take into account how well each LLM understands the input. For instance, uniform ensemble may degrade performance if the set includes weak LLMs (Figure 2 , middle). 
pack of llms (packllm): 
 In this work, we study the following problem:Given a set of user-specified LLMs {M 1 , . . . , M K } at test-time, how can we effectively combine the knowledge of each LLMs to achieve improved performance?We present Pack of LLMs (PackLLM), a weighted ensemble of LLMs that fuses knowledge of arbitrary LLMs at test-time. As shown in Figure 2 (right), given a set of seed LLMs {M 1 , . . . , M K } at test-time, PackLLM fuses their knowledge with corresponding importance weights {λ 1 , . . . , λ K }. The importance weights λ k are calculated based on how well each LLM understands the prompt P.In more detail, at each step t, each model M k is conditioned on the prompt x <t , to obtain the logit score s k (x t |x <t ) for the next token x t . Logit scores s k are unnormalized scores over the LLM's vocabulary, before applying the softmax(•) operation over all vocabulary tokens which transforms them into probabilities p M k . Here, we assume the models share the same vocabulary, but we provide a generalized version in Section 3.4. PackLLM combines output logits to obtain the final probabilities asEQUATIONwhere we assume that the weightsλ k ∈ [0, 1] are normalized, i.e., ∑ K k=0 λ k = 1.In what follows, we answer the question "How can we accurately determine weights {λ 1 , . . . , λ K } at test-time?"PackLLM relies on the notion that perplexity is a suitable measure for a LLM's knowledge on a certain input (where the lower the better). Perplexity is a metric that aligns with the causal language modeling objective of LLMs, enabling to measure whether a test input falls into the LLM's expertise and how the test input relates to the LLM's pretraining data (Marion et al., 2023; Gonen et al., 2022) .Given a tokenized input prompt or sequence P = (x 0 , x 1 , . . . , x t ), perplexity (PPL) is defined as the exponentiated average negative log-likelihood of P asEQUATIONwhere log p k (x i |x <t ) is the log-likelihood of the i-th token conditioned on the preceding tokens x <t according to model M k . Note that PPL depends solely on the models and the input prompt and can be evaluated at test-time.Leveraging perplexity as a measurement for a LLM's expertise on the test prompt, we formulate the assignments of importance weights as an optimization problem that does not require any training or annotated data and can be solved at test-time. Formally, our optimization framework is given asEQUATIONwhere {λ * 1 , . . . , λ * K } are the optimal importance weights and s k are the output logits of each LLM. During inference, the computed weights {λ * 1 , . . . , λ * K } are used in Equation 1.An important question is whether perplexity is a reliable metric for determining fusion. As the first step, we present PackLLM sim , a simple perplexity-based approach for determining the importance weights {λ 1 , . . . , λ K }. Instead of solving the optimizing problem presented in Equation 3, PackLLM sim computes the weights λ k directly by using the perplexity scores over the prompt viaEQUATION)where τ is the temperature hyperparameter with default values τ ∈ {0.1, 1}. Weights {λ 1 , • • • , λ k } are inversely proportional to the models' normalized perplexity scores, i.e., lower perplexity yields a higher weight. PackLLM sim serves as a validation of whether the models' perplexity scores provide useful information for determining fusion.Unlike the simple heuristic that PackLLM sim uses (Equation 4), PackLLM opt approximately solves the perplexity minimization problem presented in Equation 3. The problem in Equation 3 is overdetermined, where the number of equations is t (t prompt tokens) and the number of variables is K (K seed LLMs), having many solutions. Nevertheless, it can be solved at test-time via grid search (LaValle et al., 2004) by evaluating different values λ k ∈ [0, 1] with the constraint ∑ k λ k = 1. However, exhaustive grid search is time-consuming during inference due to its combinatorial complexity on the number of LLMs.We propose an efficient greedy algorithm, which ensembles the LLMs sequentially. The sequential nature of the algorithm reduces the optimization problem to searching between linear combinations of two models rather than the full K models.First, we compute the perplexity PPL k (P) of each LLM k via Equation 2 and sort the seed LLMs {M 1 , . . . , M K } based on PPL k (P) asEQUATIONwhere [M * 1 , . . . , M * K ] are ordered by the lowest to highest perplexity scores. By using the ordered set, we can omit using irrelevant LLMs via early stopping of the sequential ensemble.At the first step of the greedy optimization, we determine the relative weights between the top-1 and the top-2 LLMs, M * 1 and M * 2 . We solveEQUATIONwhere s (1) and s (2) are the output logits by M * 1 and M * 2 , respectively, and λ ∈ [0, 1]. The optimization in Equation 6 is solved via grid search (greedy grid search), where different values of λ ∈ [0, 1] are evaluated (the default step is 0.05). The value λ (1) that results to the lowest perplexity is used to updates (2) = λ (1) s (1) + (1 -λ (1) )s (2) (7)for the next step.The same procedure is repeated to iterate through all seed LLMs [M * 1 , . . . , M * K ]. We can perform early stopping when we find λ (k) = 1, i.e., the effect of the current LLM is zero. Our overall greedy optimization is summarized in Appendix A.Equation 1 assumes that the logit vectors s k can be added together, which requires that the seed LLMs share the same vocabulary. However, this is not always the case, as different LLMs are trained based on different tokenizers. To address this, we employ a tokenizer selection and alignment strategy for combining LLMs with different tokenizers.First, we determine the reference tokenizer as the tokenizer of the top-1 LLM (Equation 5). Then, we follow the Minimum Edit Distance (MinED) approach (Fu et al., 2023; Wan et al., 2024) , where each token of the reference tokenizer is mapped to its textually closest token of another tokenizer, e.g., "get" to "gets". We tokenize the sequence with the reference tokenizer and use the MinED token mappings to obtain a valid input for another LLM. With the new input, we obtain the output logits of the LLM over its vocabulary. Then, we map the output logits back to the reference tokenizer's vocabulary via MinED in order to perform logit fusion via Equation 1. MinED mappings between tokenizers are precomputed before inference. We note that aligning different tokenizers is an open problem in NLP, and more effective alignment techniques may further improve PackLLM. 
language modeling experiments: 
 The first research question that we answer is: RQ1. Is perplexity a good measure for determining the LLM importance weights at test-time?We compare with expertise-based model ensemble approaches cBTM (Gururangan et al., 2023) and DExperts (Liu et al., 2021) , which, unlike PackLLM, do not rely on perplexity. cBTM performs model fusion at the output probabilities, where the weight of each expert is determined based on the tf-idf similarity between the given prompt and the expert's pretraining data. In particular, cBTM computes weights {λ 1 , . . . , λ K } for a test prompt P for test-time fusion asEQUATIONwhere d(•, •) is a Euclidean distance, h P is the tf-idf embedding of the prompt, and h k is a representation of the pretraining data of model k. Note that Equation 8does not use perplexity, but relies on data embedding similarity.DExperts and its follow-up works (Liu et al., 2024; Li et al., 2024b) employ an additional base model M and an anti-expert model M -to fuse knowledge from an expert model. In particular, the fusion is performed asEQUATIONwhere M * 1 is the top-1 expert from the seed LLMs, and λ is a hyperparameter (default value is λ = 1). DExperts uses fixed LLM weights.We follow the cBTM experimental setting, where C4 (Roberts et al., 2019) and S2ORC (Lo et al., 2019) datasets are clustered into K ∈ {1, 2, 4, 8, 16} clusters, and K OPT-1.3B models (Zhang et al., 2022) are trained on each cluster separately. Therefore, each OPT model is an expert on a specific semantic cluster of the training documents. The clusters are computed based on the tf-idf embeddings of the train data and the embedding of the cluster centroid is used as h k in Equation 8 of cBTM. Overall, we experiment with a total of 65 LLMs. To compare with DExperts, we employ the OPT-6.7B, OPT-13B and OPT-30B models as additional base models M. Following DExperts, the anti-expert is the original OPT-1.3B model. For each dataset, we report language modeling perplexity on 200 randomly-sampled held out documents. For each test document, the prompt P consists of the 32 first tokens or the first 20% of the sequence's tokens, whichever is shorter; perplexity is evaluated on the rest of tokens.Figure 3a shows perplexity results when we compare PackLLM with the weighted ensemble approach of cBTM. PackLLM outperforms cBTM under all values of the number of expert LLMs K. The performance differences grow as the number of LLMs increases and the largest improvement is observed when we have K = 16 experts. PackLLM takes into account the perplexity of the underlying experts to estimate their expertise, while cBTM relies on off-the-shelf similarity embeddings and may not successfully capture the LLMs' knowledge. PackLLM sim and PackLLM opt perform similarly on the C4 dataset, while PackLLM sim scales better on S2ORC dataset, which has shorter prompts. The results show that our perplexity-guided fusion is effective and scales better as we increase the number of expert models. Figure 3b shows averaged perplexity results when we add additional large models (OPT-6.7B OPT-13B, OPT-30B) to the set of seed expert LLMs (C4/S2ORC-tuned OPT1.3B models), evaluating whether the fusion approaches can effectively incorporate larger general-purpose LLMs. PackLLM outperforms cBTM and DExperts, and the improvement becomes larger as we increase the number of expert LLMs (value K). Moreover, PackLLM opt outperforms PackLLM sim in both C4 and S2ORC; its benefit lies on its perplexity optimization, which effectively combines knowledge from expert models with larger models to boost performance during inference. The result show that PackLLM is effective with LLMs of varying sizes and expertise. 
downstream tasks experiments: 
 The following research questions that we answer are: RQ2. How does PackLLM compare against other test-time fusion approaches in downstream tasks? RQ3. What is the benefit of PackLLM over learning-based fusion approaches?Datasets. We experiment with a diverse set of tasks. We employ the MMLU-STEM (Hendrycks et al., 2021) dataset to evaluate performance in knowledge intensive tasks, such college STEM exams. We employ ARC (Clark et al., 2018) , BoolQ (Clark et al., 2019) , HellaSwag (Zellers et al., 2019) , and OpenBookQA (Mihaylov et al., 2018) datasets to evaluate performance on Commonsense reasoning. Furthermore, we employ MMLU-Health, PubMedQA (Jin et al., 2019) , USMLE (Jin et al., 2021) datasets to evaluate performance on domain-specific knowledge (Medicine). In addition, we evaluate on miscellaneous (Misc.) classification tasks (Mavromatis et al., 2023) , such as topic classification, hatespeech detection, and sentiment analysis. Further details can be found in Appendix C. We report accuracy on question answering and classification as the evaluation metric.Seed LLMs. For MMLU-STEM and Commonsense tasks, we employ widely used LLMs, such as Mistral-7B (Jiang et al., 2023a) and LLaMa2-7B (Touvron et al., 2023) . For Medicine tasks, we employ specialized LLMs for biomedical data, such as AdaptLLM (Cheng et al., 2023) and BioMistral (Labrak et al., 2024) . For Misc. tasks, we use models with complementary expertise (Gururangan et al., 2023) . Overall, we experiment with a total of 52 LLMs (see Appendix D).Baselines. We compare with the following test-time and learning-based fusion approaches.(i) Top1-PPL selects the most relevant LLM for a test prompt P. We use prompt perplexity (Equation 5) to determine the top-1 LLM. (ii) Ensemble performs uniform aggregation of the LLM's logits where every weight λ k has value λ k = 1 K . (iii) FuseLLM (Wan et al., 2024 ) uses the output logits of the seed LLMs to train a fusion LLM via knowledge distillation. However, it requires an external corpus for causal language modeling pretraining and cannot encompass new LLMs at test-time. (iv) FoE (Wang et al., 2023) trains a neural network classifier to select the most relevant expert from the seed LLMs for a test prompt. However, it requires annotated data, e.g., validation data for the downstream tasks, which limits its applicability to new tasks and LLMs. We also compare with (v) cBTM and (vi) DExperts when possible, as they cannot handle arbitrary seed LLMs (see Appendix B).Table 1 compares PackLLM with competing test-time approaches, such as top-1 expert selection based on perplexity (Top1-PPL) and uniform ensemble (Ensemble). PackLLM outperforms these approaches by 1.72-1.89% accuracy points, averaged over different seed LLMs in 25 tasks. The improvement becomes larger when using a greater number of LLMs, e.g., in commonsense (+3.19% accuracy points) and medicine (+2.29% accuracy points) tasks. Uniform ensemble may be negatively impacted when we increase the number of LLMs (not all LLMs are equally strong), while Top1-PPL improves with more LLMs. PackLLM opt outperforms PackLLM sim in all tasks, except for Commonsense, where the input prompts are shorter (0-shot input). Moreover, PackLLM outperforms cBTM and DExperts in the cases they can compare against. The results show that PackLLM is an effective fusion approach, outperforming competing test-time approaches by up to 1.89% accuracy points, on average.In Table 2 , we compare PackLLM with learning-based fusion approaches. FuseLLM trains a fusion LLM that learns to combine knowledge from the seed LLMs over diverse pretraining data, while FoE trains a classifier to select the best expert LLM based on validation data. As a result, both FuseLLM and FoE have an advantage over our learning-free method when using the same seed LLMs, leading to 0.39-3.71% more accuracy points. However, PackLLM does not require any training or validation data and can quickly adapt as new LLMs are released. For example, PackLLM improves over FuseLLM and FoE by 4.37-11.94% accuracy points when using four recently released 7B LLMs. In addition, by using strong small LMs (Li et al., 2023) , PackLLM outperforms FuseLLM by 3.92-10.66% accuracy points without incurring additional computational costs (equal number of total LLM parameters). Full results of this section can be found in Appendix E. To further verify our conclusions, we experiment on the C4 and S2ORC language modeling tasks, where we have a large number of expert OPT models sharing the same vocabulary. In this case, PackLLM also outperforms the Best LLM (selected as the top1 expert).PackLLM vs. Largest Model. Selecting a larger LLM may yield better generalization than selecting smaller models with lower perplexity. However, smaller LMs are easy to fine-tune on in-domain data which can provide complementary knowledge (e.g., medicine data). In such cases, model fusion yields better results. To demonstrate this, we compare PackLLM with selecting the largest model from the set of models on three different tasks; Medicine, MMLU-STEM, and language modeling on C4 in Table 3B . The results show that PackLLM yields better performance than selecting the largest model in all settings. We provide example cases, where PackLLM improves a large model in Appendix G.PackLLM vs. Model Merging. Although it is not always possible to merge models as a fusion approach, model merging has lower inference costs compared to ensemble approaches. We provide results by comparing PackLLM with model merging (average weighting) on commonsense reasoning and language modeling tasks in Table 3C . The results that PackLLM outperforms model merging across these two tasks. 
ablation studies: 
 In this section, we ablate PackLLM sim and PackLLM opt with respect to their derived LLM importance weights, their sensitivity on the input prompt, and their time cost.Importance Weights. Figure 4a shows the derived importance weights λ k by the two approaches for four different tasks. PackLLM sim calculates the weights based on perplexity scores (Equation 4) and thus, fluctuations in perplexity scores among datasets and LLMs result in fluctuations in the induced weights. For example, the top-1 weights range within [0.54, 0.89] and top-2 weights range within [0.08, 0.37]. On the other hand, PackLLM opt solves an optimization of minimizing perplexity to determine weights λ k , which results in weights of smaller range. For example, the top-1 weights range within [0.56, 0.76] and top-2 weights range within [0.16, 0.31]. understand the input and shows minor improvements with longer prompts. On the other hand, both PackLLM sim and PackLLM opt significantly improve as we increase the length of the input. Moreover, PackLLM opt outperforms PackLLM sim due to its optimization framework, which specializes into understanding the input better. 4 ). However, it is not ensured that it can achieve the best performance (PPL).On the other hand, PackLLM opt employs a greedy solution to its optimization problem (greedy grid search), improving downstream performance (perplexity), while maintaining a linear time complexity with respect to the number of seed LLMs (#LLMs), i.e., 2-4 secs when using 2-5 LLMs. Although exhaustive grid search can slightly improve performance compared to greedy grid search, its time cost can be prohibitive. For example, when using 5 LLMs, exhaustive grid search results in a explosive time cost of 902 secs.Table 5 : Inference Time Cost.Single forward pass 0.06 Ensemble (K = 5 forward passes) 0.30 PackLLM (K = 5 forward passes) 0.32 Furthermore, we compare the inference time cost of PackLLM with the simple ensembe approach. PackLLM computes the output logits by each LLM once (as in the Ensemble baseline), which are then used for determining the importance weights as shown in Algorithm 1 of Appendix A (no additional LLM forward passes are required). We provide the time measurements for K = 5 LLMs per input on a A100 GPU in Table 5 . As evaluated, PackLLM does not incur significant time costs.PackLLM's limitations are discussed in Appendix H. 
conclusion: 
 We introduce PackLLM, an effective method for test-time fusion that leverages each LLM's expertise, given an input prompt. PackLLM performs model fusion by solving an optimization problem for determining each LLM's importance, so that perplexity over the input prompt is minimized. We conduct experiments with over 100 total LLMs on a diverse set of tasks. Experimental results show that (i) perplexity is a reliable measure for LLM fusion, (ii) PackLLM outperforms test-time fusion baselines by 1.89% accuracy points, (iii) PackLLM can leverage new LLMs to improve performance over learning-based fusion approaches by 3.92-11.94% accuracy points, and (iv) PackLLM benefits over selecting the best or largest model and model merging in certain cases.Following, we highlight the key difference between cBTM, DExperts, and PackLLM (ours) regarding how they compute the importance weights:• cBTM: Importance weights are data-dependent (Equation 8; using tf-idf embeddings); the weights would be similar for LLaMA-7B and LLaMA-70B as the two models have similar pretraining data. • DExperts: Importance weights are fixed (Equation 9). • PackLLM: Importance weights are model-dependent (perplexity-based), capturing how well each model understands the prompt; LLaMA-70B would be assigned a higher weight than LLaMA-7B.Figure 3 verifies the benefit of model-dependent weights (PackLLM vs. cBTM & DExperts). 
 
 Appendix: All datasets can be found in HuggingFace Hub.C4 (Roberts et al., 2019) is s a publicly available distribution of a Common Crawl snapshot.Following cBTM, we use 168B tokens of the no blocklist version (en.noblocklist) that is out of distribution to the OPT model's pretraining corpus. S2ORC (Lo et al., 2019) is a publicly available corpus of full-text academic papers from the Semantic Scholar. The corpus spans 20 fields of study (e.g., Biology, Computer Science, Art). Following cBTM, the training data consists of 168B tokens over multiple epochs. For each dataset, we report language modeling perplexity on 200 randomly-sampled held out documents. For each test document, the prompt P consists of the 32 first tokens or the first 20% of the sequence's tokens, whichever is shorter; perplexity is evaluated on the rest of tokens.MMLU (Hendrycks et al., 2021) consists of knowledge intensive tasks, such as STEM exams, that have a question and four multiple choice questions. The prompt template is "Question: {QUESTION} Choices: {Choices} Answer: ". We use the default 5-shot examples and evaluate on the test data.We employ ARC (Clark et al., 2018) , BoolQ (Clark et al., 2019) , HellaSwag (Zellers et al., 2019) , and OpenBookQA (Mihaylov et al., 2018) datasets to evaluate performance on commonsense reasoning. We use 0-shot prompts with the template "Question: {QUESTION} Choices: {CHOICES} Answer: " for ARC, BoolQ and OpenBookQA, and continuation "The topic is about {TOPC}. {CTX-A} {CTX-B}" for HellaSwag. We randomly sample 256 test data for evaluation. BoolQA is a binary classification, while ARC, HellaSwag, and OpenBookQA usually have four answer choices.For PubMedQA (Jin et al., 2019) and USMLE (Jin et al., 2021) , we follow (Cheng et al., 2023) . The template is "Question: {QUESTION} Choices: {CHOICES} Answer: " for PubMedQA and "{CONTEXT} {QUESTION} {ANSWER}" for USMLE. These datasets have four-way multiple choice answers. We randomly sample 256 test data for evaluation and use 0-shot prompts.Regarding miscellaneous classification tasks, we experiment with AGNews (Zhang et al., 2015) (topic classification), SST2 (Socher et al., 2013) In this section, we provide the full list of the LLMs we use.We use the following LLMs:• 2 Seed LLMs:• meta-llama/Llama-2-7b-hf FoE LLMs. FuseLLM uses the following 15 LLMs:• Aspik101/trurl-2-7b-pl-instruct unload• Charlie911/vicuna-7b-v1.5-lora-mctaco• Fredithefish/RedPajama-INCITE-Chat-3B-Instruction-Tuning-with-GPT-4• GOAT-AI/GOAT-7B-Community• TheTravellingEngineer/bloom-1b1-RLHF• ashercn97/manatee-7b• garage-bAInd/Platypus2-7B• golaxy/gogpt-7b-bloom• julianweng/Llama-2-7b-chat-orcah• lmsys/vicuna-7b-v1.3• lmsys/vicuna-7b-v1.5-16k• medalpaca/medalpaca-7b• rombodawg/LosslessMegaCoder-llama2-7b-mini• togethercomputer/GPT-JT-6B-v0• togethercomputer/GPT-JT-6B-v1Recent LLMs. As recently released LLMs, we use the following:• meta-llama/Llama-2-7b-hf• mistralai/Mistral-7B-v0.1• microsoft/phi-2• Deci/DeciLM-7BRecent SLMs. As the recently released SLMs, we use the following:• google/gemma-2b• microsoft/phi-2• Qwen/Qwen1.5-1.8BTable 6 provides full results on MMLU-STEM tasks.Table 7 provides full results on commonsense tasks.Table 8 provided full results on medicine tasks. Further MMLU-Health results are provided in Table 9 .Table 10 provides full results on miscellaneous classification tasks.We provide the experimental setup of the results in Table 3: • -15 seed LLMs of same vocab (FoE LLMs).-Best LLM: Platypus-7B.• (A) C4 and S2ORC:-9 OPT models (30B OPT + 8* 1.3 OPT experts) on C4 / S2ORC (cBTM models).-Best LLM: Top1 model selection.• (B) Medicine:-3 LLMs (LLaMA2-13B, MedLLaMA2-7B, LLaMA2-7B) on 8 Medicine tasks.-Largest LLM: LLaMA2-13B.• (B) STEM:-4 LLMs (FuseLLM-7B, Gemma-2B, phi2-2.7B, Qwen-1.8B) on 5 MMLU-STEM tasks. -Largest LLM: FuseLLM-7B.• (B) C4:-5 LLMs (OPT-30B, 4 domain experts OPT-1.3B) on C4 (cBTM models).-Largest LLM: OPT-30B.• (C) Com.:-2 LLMs (LLaMA2 and FuseLLM 7B models) on 4 Commonsense tasks.• (C) C4 / S2ORC: -4 LLMs (4 domain experts OPT-1.3B) on C4 / S2ORC (cBTM models). • OPT-6.7B: "retirement plan today will get you a return of 12-14%" (hallucination: incorrect return percentage) • OPT-6.7B + PackLLM (OPT-1.3B tuned model on C4): "retirement plan offers some tax advantages, but it's important to take action as early as possible." (helpful advice)We summarize two limitations of our approach below.1. One challenge is discussed in Section 3.4: Equation 1 assumes that the logit vectors can be added together, which requires that the seed LLMs share the same vocabulary. However, this is not always the case, as different LLMs are trained with different vocabularies. While we follow the MinED approach for tokenizer alignment, aligning different tokenizers is an open problem in NLP, and more effective alignment techniques may further improve PackLLM.2. PackLLM (similar to other fusion approaches) incurs additional inference costs compared to model merging approaches, such as weight averaging. Accelerating the inference time of fusion approaches, such as PackLLM and model ensemble, is a promising future research area. 


Decision: This work introduces a method named PackLLM to better perform test-time fusion of LLMs as a weighted ensemble as compared to a simple ensemble. This assumes that minimising the perplexity of the combination of LLMs with respect to the input prompt will produce the best model.

Pros:
The work proposes a simple and effective test-time approach for this. The work is also clearly written and clear with results on multiple datasets. The work also has a relatively low test-time computational cost.

Cons:
It would be good to more clearly clarify the difference between alternative baselines including cBTM and DExperts as well as perform thorough comparisons against them where possible.

Despite this, the low test-time computational cost, the improvements in downstream tasks and good performance compared to other baselines such as a simple ensemble and selecting the largest model point to this work likely having a significant impact on the field.

Reviews: review: paper_summary: paper_summary: This paper introduces a new method, PackLLM, to fuse LLMs at test-time as weighted ensemble, where the importance weights for seed LLMs are determined by minimizing the perplexity of the input prompt.

There are two variants of PackLLM, one that directly uses normalized perplexity scores of individual LLMs as weights, and the other that performs an efficient greedy algorithm to search the optimal scores that minimize the prompt perplexity of the ensemble.

The paper conducts extensive experiments to validate the performance of PackLLM. On pretraining perplexity, both variants of PackLLM outperform existing test-time model ensemble baselines like cBTM and DExperts, especially when the number of seed LLMs increases. On downstream tasks, PackLLM also outperforms simple baselines like uniform ensemble. While it trails other learning-based fusion methods like FuseLLM and FoE with the same set of seed LLMs, the learning-free nature allows PackLLM to achieve better performance by swapping in stronger seed LLMs.

The paper also includes some analysis of the weight values, the impact of prompt length, and the time complexity of the two variants.
strength_weakness: strength_weakness: I don’t have major concerns. I do have some questions listed below.
questions: questions: - I’m curious to know the impact of aligning different tokenizers. Have you done any analysis comparing the performance when fusing similar-sized models with the same vs. different vocabularies?
- In experimental results, I’d like to also see a comparison with the best seed LLM performance on each task, especially for Table 4, where most of the baseline results are not available.
- While not an exhaustive search, the greedy search algorithm still seems a very expensive operation that needs to be done for every test-time input. Can we situate the time complexity analysis in Table 3 by comparing the additional time cost from greedy search with the actual inference time of LLMs? This would be important to help understand the real-world applicability of the proposed method.
paper_summary: paper_summary: This paper proposes PackLLMs, which utilizes the findings that perplexity of input tokens is a good indicator of LLM's performance in domains. The authors propose PackLLM-simp that utilize the heuristics above and PackLLM-Opt, which approximates an optimization objective with a greedy search algorithm. Experiments show improvement over baselines such as cBTM, DExperts. The approach is further  modular and easy to use compared to other fusion methods that requires training LLMs.
strength_weakness: strength_weakness: - The authors should include discussions about the computational overhead of the approach. How much overhead is introduced by the grid search part, given the high dimensionality of logits (sequence length * vocab size for a single instance)? Is this overhead significant compared to a forward pass of the LM? I also encourage the authors to add a limitations sections to discuss potential limitations of the approach.
questions: questions: I think the paper is quite solid and has addressed my initial questions in their later sections.

A minor comment is that the second paragraph of the related works is repetitive given the second paragraph in introduction.
paper_summary: paper_summary: This paper presents a test-time LLM fusion method to leverage knowledge from the user-specified LLMs during inference. Specifically, the authors optimize the perplexity over the input prompt to perform model fusion.  The experimental results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: In the main experimental results reported in Table 1, the proposed method outperforms the baseline models. However, it is important to note that the experiments involving some test-time fusion methods, such as cBTM and DExperts, are not fully reported. This lack of complete data undermines the reliability of these results. This issue is similar to the one observed in Table 2, where the comparison with learning-based fusion models becomes less convincing due to incomplete experimental data.
questions: questions: - Is it possible to complete the results of the test-time baseline experiment? 

- It is advised to provide some case studies to show that the proposed ppl-based fusion method is superior to other methods.
paper_summary: paper_summary: This paper provides an alternative method for combining the responses from multiple LLMs. Weights were introduced to be used to re-weight the output logits. To determine the criterion for obtaining the optimal weights, the authors first discovered that the perplexity on the given prompts can be already served as a good approach to fuse the outputs of models. Next, the authors propose an greedy optimization-based algorithm that effectively fuses the outputs from different models. Experimental results show that the proposed algorithm can merge the outputs from models better than baseline methods such as ensemble and using only the model with lowest perplexity.
strength_weakness: strength_weakness: - I think the authors may also want to compare with always picking the responses from the largest model, because selecting the model with the lowest perplexity does not mean that model can generalize well. Selecting always the largest model may yield better performance. 
- I would also like to see how this model compared with model merging. I understand it is not always possible to merge models, especially when the number of parameters and the structures are diverse across models. However, I think the authors can at least find several publicly available models fine-tuned from llama-7b on different datasets and see if the proposed algorithm can outperform model merging.

Comments: comment: Thank you for the detailed response. It addressed most of my concerns.
I have read the authors' responses and my concern has been addressed.
**Comparison with cBTM & DExperts**. Thank you for your follow-up questions. cBTM’s LLMs are fine-tuned on top of the OPT model family, which are not relatively powerful models. For example, if we use cBTM’s LLMs, they perform ~25% at accuracy on MMLU, which is close to random answer selection. Generalizing cBTM to other model families for producing full results in Table 1 requires (i) proper data selection, and (ii) further model fine-tuning, which is limiting regarding compute resources. If all experts are known, PackLLM outperforms cBTM (Misc in Table 1)  and DExperts (Medicine in Table 1).

Following, we highlight the key difference between cBTM, DExperts, and PackLLM (ours) regarding how they compute the importance weights:
- *cBTM*: Importance weights are **data-dependent** (Eq. 8; using tf-idf embeddings); the weights would be similar for LLaMA-7B and LLaMA-70B as the two models have similar pretraining data.
- *DExperts*: Importance weights are **fixed** (Eq. 9).
- *PackLLM*: Importance weights are **model-dependent** (perplexity-based), capturing how well each model understands the prompt; LLaMA-70B would be assigned a higher weight than LLaMA-7B.

Figure 3 verifies the benefit of model-dependent weights (PackLLM vs. cBTM / DExperts).

**Table 2**. Thank you for pointing this out. We will remove the “Overall” column to improve the readability of Table 2.
Thanks for the authors' responses. However, it is still unconvincing to me. For example, what's the point of including cBTM and DExperts in Table 1 without reproducing the full experiments on all compared domain datasets? If all the experts and LLMs are known to cBTM and DExperts methods, do the proposed methods still outperform them? For Table 2, the last "Best Overall" column is misleading; the results are not comparable among the methods. I am still concerned about this.
Dear Reviewer Gnd4, thank you for **updating your review** in a timely manner. Please discard our previous message (we are unable to delete it). We reply to your questions below. 

**Q1) Impact of aligning different tokenizers**. Thank you for your insightful question. We do not have any experimental analysis evaluating the impact of the vocabulary size, but we agree that it would be easier to fuse knowledge from models with similar-sized vocabularies (e.g., LLama + Mistral) than models with different vocabularies (e.g., LLama + Gemma). Disentangling the effects of combining models with different vocabularies, e.g. LLama + Mistral vs. LLama + Gemma, is challenging as different models (Mistral vs. Gemma) perform differently on the underlying tasks.

**Q2) Evaluating best LLM**.
Thank you for your suggestion. We experiment with the best LLM (as evaluated on test data) on STEM and Commonsense tasks. We find that the best LLM works better when having few LLMs with different vocabularies, but when we have a large number of LLMs with the same vocabulary, PackLLM outperforms the best LLM. We hope with better tokenization mapping methods, PackLLM becomes more effective in the first scenario.

To further verify our conclusions, we experiment on the C4 and S2ORC language modeling tasks, where we have a large number of expert OPT models sharing the same vocabulary. In this case, PackLLM also outperforms the Best LLM (selected as the top1 expert). 

We provide the results in the following tables.

Setup: 5 MMLU-STEM tasks and 4 Commonsense tasks.
Best LLM: LLaMA2-7B (Fuse LLMs), Platypus-7B (FoE LLMs)

| |Fuse LLMs (3 seed LLMs, different vocab) | FoE LLMs (15 seed LLMs, same vocab) |
|-|-|-|
| | STEM / Com. | STEM / Com. |
PackLLM | 31.2 / 54.9 | **40.3** / **67.1** |
Best LLM (using test data) | **33.0** / **56.3** | 34.8 / 59.1 |

Setup: 9 OPT models (30B OPT + 8* 1.3 OPT experts) on C4 and S2ORC language modeling tasks.
Best LLM: Top1 model selection.

| Fusion | Perplexity (the lower the better) |
|-|-|
| | C4 / S2ORC |
| Largest (30B) | 12.71 / 11.46 |
| Select Best (Top1 model) | 12.53 / 11.46 |
|PackLLM | **11.03** / **11.05** |

**Q3) Timing of PackLLM**. Thank you for your clarifying question. PackLLM can be applied in real-world scenarios as it does not incur significant additional costs. PackLLM computes the output logits by each LLM once (as in the Ensemble baseline), which are then used for determining the importance weights as shown in Lines 10-15 of Algorithm 1 in Appendix A. No additional LLM forward passes are required. 

We also provide the following time measurements for K=5 LLMs per input on a A100 GPU:

| | Time|
|-|-|
|Single forward pass | 0.06 secs |
| K=5 forward passes (Ensemble) | 0.30 secs |
| PackLLM (K=5) | 0.32 secs |
As evaluated, PackLLM does not incur significant time costs.