On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion

Authors: chenghao  fan
zhenyi  lu
wei  wei
jie  tian
xiaoye  qu
dangyang  chen
yu  cheng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:00:52

Abstract: Efficient fine-tuning of large language models for task-specific applications is imperative, yet the vast number of parameters in these models makes their training increasingly challenging.
Despite numerous proposals for effective methods, a substantial memory overhead remains for gradient computations during updates. \thm{Can we fine-tune a series of task-specific small models and transfer their knowledge directly to a much larger model without additional training?} 
In this paper, we explore weak-to-strong specialization using logit arithmetic, facilitating a direct answer to this question.
Existing weak-to-strong methods often employ a static knowledge transfer ratio and a single small model for transferring complex knowledge, which leads to suboptimal performance. 
To surmount these limitations,
we propose a dynamic logit fusion approach that works with a series of task-specific small models, each specialized in a different task. 
This method adaptively allocates weights among these models at each decoding step,
learning the weights through Kullback-Leibler divergence constrained optimization problems. 
We conduct extensive experiments across various benchmarks in both single-task and multi-task settings, achieving leading results.
By transferring expertise from the 7B model to the 13B model, our method closes the performance gap by 96.4\% in single-task scenarios and by 86.3\% in multi-task scenarios compared to full fine-tuning of the 13B model. Notably, we achieve surpassing performance on unseen tasks. Moreover, we further demonstrate that our method can effortlessly integrate in-context learning for single tasks and task arithmetic for multi-task scenarios.

Main paper: 
 introduction: 
 In recent years, Large Language Models (LLMs) have shown impressive performance in a wide range of tasks [5, 36, 50-52, 54, 61, 62] , including code generation [11, 45] , mathematical reasoning [2, 37] , tool-use abilities [46, 53] , etc. However, training such LLMs requires substantial computational resources, often involving thousands of GPUs and processing trillions of tokens [30, 42] , making the adaptation of the base model for new knowledge inefficient. To address these challenges, parameterefficient tuning methods [12, 16, 23] have emerged, aiming to achieve comparable performance to full fine-tuning while reducing GPU memory requirements. However, challenges persist in tuning and deploying large-scale models on common hardware, as they still involve computation-intensive processes like gradient calculation and back-propagation. Furthermore, these methods may not be feasible when training data are private. This inspires us to ask: Can we fine-tune only small models and then transfer their knowledge to a much larger model without requiring additional gradient updates? If we could fuse the strong capabilities of a scaled LLM with the specialized knowledge acquired by a small model during fine-tuning, it would yield the practical benefit of approximating the results achieved by fine-tuning a large model, but without the associated computational costs. However, it is non-trivial due to the differences in representation width and layer numbers between the small and large models.Peter is 76 years old this year. How old will he be in 500 years? Recently, Mitchell et al. [41] and Liu et al. [33] attempt to address this challenge by transferring knowledge from a Small Language Model (SLM) to its larger counterpart through simple logit arithmetic operations during decoding. For instance, using models from the Llama-2 family, they can transfer the fine-tuned knowledge from a 7B-scale model to a 13B-scale model by performing log probability algebra: Llama-2-base 13B + (Llama-2-chat 7B -Llama-2-base 7B), where the first term represents the base log probabilities and the term in parentheses denotes the behavioral delta [41] . This behavioral delta can be weighted to adjust the balance between the pretrained knowledge and the transferred fine-tuned knowledge.Despite showing promise, logit arithmetic still exhibits a noticeable performance gap compared to directly fine-tuning large models, primarily due to two reasons: Firstly, they statically prespecifies the weight of behavioral delta at each decoding step identically. However, the importance of fine-tuned knowledge varies significantly across different tasks, inputs, and even different decoding steps. For instance, in a domain-specific question-answering process, we need more knowledge from fine-tuned small language models. Conversely, when decoding factual topics, we may need more knowledge from pretrained general models. Secondly, for unseen tasks, the lack of pre-adjusted weights for the behavioral delta prevents logit arithmetic from executing effectively, making it challenging to transfer tuned knowledge, especially for complex tasks. As shown in Figure 1 , when answering "Peter is 76 years old this year. How old will he be in 500 years?", using only the math expert does not ensure the result aligns with factual accuracy. Additionally, these techniques often assume that experts are trained and tested on the same data distribution, ignoring the heterogeneity of data that may be encountered at test time, rendering any single expert insufficient.In addressing these challenges, we reconsider the practice of logit arithmetic within a new framework. Specifically, we work with a set of finely tuned SLM experts, each specializing in different tasks. At each decoding step, we dynamically allocate weights among these task-specific SLMs. However, it is non-trivial to effectively determine suitable weights for diverse tasks and SLM experts, as tranditional approaches such as grid search [29] or combinatorial search [34] suffer from costly search processes. To practically automate weight allocation, we reframe the problem of weight search as a constrained distribution optimization problem. We tackle the fusion of multi-task knowledge by treating it as a centroid problem in Euclidean space using Kullback-Leibler divergence [24] , which offers interpretability and explicit guidance.We conduct thorough experiments on the LLaMA series to evaluate the effectiveness of our approach, applying our adaptive logit arithmetic method to task-specific fine-tuning in math, question-answering, summarization, and multi-domain tasks. We also analyze the performance of multi-model fusion across seen and unseen tasks. Additionally, we discussed the feasibility of combining our method with in-context learning for single tasks and with task arithmetic for multi-task scenarios.In our work, we make several key contributions. Firstly, we reassess existing logit arithmetic methods, highlighting the significant impact of fusion weights and the limitations imposed by a single small model on test performance. Secondly, we introduce a novel approach that autonomously learns fusion weights through constraint optimization, approximating the compute-intensive results of finetuning a large base model. Lastly, we conduct comprehensive experiments to validate our method, demonstrating substantial improvements in performance, generalization capabilities, and robustness.2 Related Work 
efficient specialization: 
 Specializing a pretrained model by fine-tuning for downstream tasks has become a primary paradigm [7, 9, 10, 44] . However, with the increasing scale of models, full fine-tuning has become impractical due to the large number of parameters, requiring costly training resources [5, 54] .To tackle this challenge, researchers have developed parameter-efficient tuning methods [28] . These methods aim to achieve performance similar to full fine-tuning while reducing the GPU memory required. They typically involve freezing the original weights and adding task-specific trainable modules [15, 22, 26] , low-rank matrices [12, 16, 23] , or bias [3] . Despite their benefits, these approaches still demand significant memory for updating gradients and might not be suitable for scenarios where training data are private or inaccessible. Another direction is model merging [17, 35, 38, 56, 58] , which trains task-specific models on different domains, and then combines them into a single model at deployment with weight average [57] , neuron permutation [1] , interpolation [20, 39] or task vectors [17, 35, 59 ]. However, they typically suffer from parameter interference between different tasks and static optimal solutions may struggle with multiple task domains.There is a growing research interest in specialized large models by eliciting existing knowledge, such as utilizing curated prompts via in-context learning [31] or employing small models to offer weakto-strong supervision [6, 33, 41] , generating weak labels or alignment signals. These methods are highly practical, requiring only the generated output (or logits). Our approach shares a similar weakto-strong intuition and proposes adaptive strategies to achieve a better balance between leveraging the pretrained general knowledge and acquiring task-specific knowledge. Furthermore, our method can be extended to multi-task scenarios.Different from knowledge distillation [4, 14, 49] , where a more capable teacher guides a student model, we explore using weaker models to teach stronger ones. Burns et al. [6] empirically demonstrate that weak supervisors can reliably elicit knowledge from much stronger models (e.g., supervising GPT-4 with a GPT-2). Such approaches represent a promising direction, as it is more practical and affordable to specialize in a much smaller model. Contrastive Decoding [27] enhances outputs by leveraging differences between large and small LMs, as patterns like repetition and incoherence are often more pronounced in smaller ones. Speculative sampling [25, 48] speeds up inference using a lower-parameter version of the LLM as a draft model, exploiting the fact that draft models can approximate easier subtasks well. SpecInfer [40] goes further by employing a set of small models to generate drafts in parallel. Jin et al. [19] train a projector to map the parameter of the weak expert to a larger version, but the projector needs to be trained and suffers from poor generation. Ji et al. [18] focuses on utilizing small models to rephrase unaligned answers from LLMs into more aligned ones. Mitchell et al. [41] and Liu et al. [33] leverage logits from small, fine-tuned models to inject specific knowledge into the pretrained LLM with the same vocabulary. Our approach differs from methods that require pretrained small model adapters [48] or pre-adjusted parameters [33, 41] . Instead, we dynamically transfer the capabilities of small models to the large model at each decoding step, without needing access to the large model's parameters or training data. 
problem background: 
 Autoregressive Language Models Modern autoregressive transformers generate continuations for input prompts token by token. Given a prompt x 1:k-1 (denoted as x <k ), the model computes the logits for the k-th token, represented asM (x k | x <k ) ∈ R |V |, where V denotes the size of the vocabulary. A probability distribution P (x k | x <k ) is then obtained through softmax normalization:P (x k | x <k ) = softmax(M (x k | x <k )). The next token x k is subsequently sampled from this distribution, i.e., x k ∼ P (x k | x <k ).Distance Between Language Model Outputs Distribution We can utilize the Kullback-Leibler(KL) divergence to measure the similarity of two distributions P and Q generated from two language models (with the same vocabulary), which can be viewed as the distance of the two language models:EQUATIONIf this is implied by the context, we will omit the conditioning on |x <k and simply use D KL (P ||Q).Logit Arithmetic Suppose we have two pretrained auto-regressive models with homogeneous architecture and the same vocabulary: a small model with parameter set θ S and a large model with parameter set θ L . We aim to fine-tune the small model to obtain θ S f t and transfer this fine-tuning knowledge to the large models. Previous work [33, 41] transferred fine-tuned knowledge to a large model by designing arithmetic between logits, resulting in the output distribution P for the large model as follows:EQUATIONwhere M L , M S , and M S f t represent the logits of the large model, small model, and fine-tuned small model, respectively. Their corresponding normalized distributions are denoted by P , Q, and Q f t . The detailed theoretical proof supporting this logit arithmetic formula is provided in Appendix B. Here, α is a pre-adjusted hyperparameter that controls the extent of knowledge transfer from the small model. Our analysis from Appendix C demonstrates that logit arithmetic attempts to approximate the shift (Q f t (x k |x <k ) Q(x k |x <k )) α between the fine-tuned distribution and the pretrained distribution by controlling the parameter α before inference.However, using a pre-defined α presents the issue that the resulting LLM's distribution P has a fixed trajectory, which may not be suitable for every step in decoding. So we need a dynamic α in the decoding steps. Nonetheless, solving this problem is non-trivial, so we attempt to transform it into an equivalent, optimizable objective. Here, we use the distance between language model outputs as defined in Equation (1) to represent the shift between distributions. We assume that, for different model sizes, at each decoding step, the distance between the fine-tuned model outputs and pretrained model output is the same:D KL ( P ||P | x <k ) ≈ D KL (Q f t ||Q | x <k ), D KL (P || P | x <k ) ≈ D KL (Q||Q f t | x <k ) (3)Following the assumption in Equation (3), we optimize the following expression at each decoding step:arg min P D KL ( P ||P ) -D KL (Q f t ||Q) 2 + D KL (P || P ) -D KL (Q||Q f t ) 2 (4)Based on this optimization problem, we can adaptive transfer the knowledge of the corresponding SLM expert to the large model at each decoding step in a single-task setting. Our method has been proven to be effective in experiments. To guarantee symmetry, a symmetrical term is incorporated:When dealing with complex tasks or new domains, a general LLM may lack the necessary expertise, and a single SLM might not provide sufficient specialized knowledge due to the capacity gap. To migrate this challenge, our method can be extended to fuse multiple SLMs and leverage their knowledge to compensate for the shortcomings in individual domain-specific knowledge. We finetune the SLM θ S on each domain to obtain multiple task-specific experts {θ S t } T t=1 , making it easier to acquire knowledge and dynamic fused to the LLM. During decoding, knowledge from these domain-specific SLMs is transferred simultaneously to the LLM. We modify the Equation (3) as follows:D KL ( P ∥ P ) ≈ D KL (Joint({Q 1...T }) ∥ Q) , D KL (P ∥ P ) ≈ D KL (Q ∥ Joint({Q 1...T })) (5)where Q t represents the distribution of θ S t from the t-th domain, Joint({Q 1...T }) represents the distribution of the combined knowledge of Q 1 , Q 2 , ..., Q T . When we impose constraints like Equation (5) , it attempts to align the joint distributions of the logits from the domain-specific small models. However, the distributions of the logits from the domain-specific small models are usually not independent of each other, so their joint distribution cannot be estimated through simple multiplication.Due to the difficulty in obtaining a joint distribution for multiple expert models, we decompose the joint distribution constraint problem into a multi-object marginal distribution optimization. The transformation process we prove in detail in Appendix E. By aligning the distributions of each domain-specific small model, we can infer an approximate optimal solution within the extension of Equation ( 2), as shown by:EQUATIONHere we use M S t to represent the logit of t-th expert. Our algorithm is outlined in pseudo-code in Algorithm 1 in Appendix F.Intuitively, this projects the KL divergences between the logits into Euclidean space and finds a central point with the minimum distance sum as the best KL value. This optimal KL value corresponds to the output distribution of the large model with multi-domain knowledge. In our experiments, we optimize α ∈ R T to obtain the optimal KL value. Generally, α is between 0 and 2. In a multi-task setting, we can accelerate the optimization process by optimizing the boundaries for only one expert, restricting only one SLM expert to be applied at the current decoding step. We will provide a more detailed explanation of this in our experiments. 
experiments: 
 In this paper, we use the LLaMA2 [55] family of models to test our method, which contains the same vocabulary, enabling us to employ logits arithmetic easily. Here, we use TinyLLaMA-1.1B [60] and LLaMA2-7B as our pretrained small models, and LLaMA2-13B as the large model for transfer. We conduct tests in single-domain and multi-domain settings to demonstrate that our method can effectively transfer knowledge to the large model in both scenarios.Following the setting in Liu et al. [33] , we evaluate on the following datasets: mathmetical reasoning (GSM8K [8] ); factual accuracy (TruthfulQA [32] ); realistic knowledge (TriviaQA [21] ); multi-domain general knowledge (MMLU benchmark [13] ); summarization (CNN-DailyMail (CNN/DM) [47] ). All datasets are tested using a 0-shot setting. Detailed information is provided in Appendix G.For all tasks except TruthfulQA, we construct prompts based on the task type and employ supervised instruction tuning to train each task expert. Following the previous work [33] , we use "Llama-2-7b-chat-hf" as the 7B expert for TurhfulQA, and TinyLLaMA-chat-version as the 1.1B expert for TurhfulQA. For full fine-tuning, we set the batch size to 128, learning rate to 2e-5, optimizer to Adam.For LoRA tuning, we set the rank to 64, learning rate to 1e-4, optimizer to Adam. We train for 3 epochs. For multi-task tuning, we perform full fine-tuning using a mixed-seed training dataset.During inference, we use greed decoding and set batch size to 256, top_p to 1.0 and temperature to 0.05. To accelerate inference, we use VLLM 3 , synchronizing the signals of logits during logit arithmetic to achieve efficient inference. All experiments are performed on H100 GPUs. To demonstrate the effectiveness of our method in efficiently transferring knowledge from a small model to a large model, we compare it against a small model fine-tuned in a single domain and a large model without fine-tuning. We also report the performance of a large model fine-tuned on a single domain or fine-tuned using LoRA to demonstrate the feasibility of our method as a replacement for fine-tuning approaches. Additionally, we compare our method with Proxy Tuning (setting α = 1.0 as in the original work) to highlight the superiority of our method in the transfer process. We use the LLaMA2-chat model as the expert for TruthfulQA (therefore, there is no corresponding LoRA model). For other experts, we use models fine-tuned on the respective training sets. In multi-task scenarios, we follow the common training practice of mixing the training data, with the results serving as our multi-task baseline. More details can be found in the Appendix G.As shown in Table 1 , our method improves upon the original experts when transferring knowledge from the 1.1B and 7B models to the 13B model. Across all tasks, our method transfers knowledge from 1.1B and 7B experts to the 13B model, achieving performance improvements of 37.6% and 4.4% over the original experts, respectively. This demonstrates that our method can leverage the existing knowledge in larger models to enhance the knowledge learned by the expert small models. Notably, we observe significant improvements on GSM8K and TriviaQA compared to the experts alone, indicating that our method effectively transfers expert knowledge while leveraging the inherent capabilities of the large model, such as stronger reasoning abilities and a richer knowledge base.Our method, which transfers knowledge from 1.1B and 7B experts, can close the performance gap by 72.9% and 96.4%, respectively, compared to 13B Full Fine-Tuning, achieving improvements of 6.5% and 3.8% over Proxy Tuning. Compared to 13B Full LoRA Tuning, our method can close the gap by 71.7% and 90.7%, respectively, when transferring knowledge from 1.1B and 7B experts. Additionally, it is worth noting that our method outperforms the 13B Full Fine-Tuning results on TriviaQA. Since our approach only requires fine-tuning a smaller model, it demands less computational and memory resources compared to fine-tuning a large model, making our method highly promising.In the multi-task scenario, we categorize the multi-domain MMLU task as an unseen task and the other four tasks as seen tasks. We then calculate the average performance on seen and unseen tasks to evaluate the overall generalization capability of our model. As shown in Table 2 , our method achieves a 36.35% and 13.37% improvement over directly fine-tuning on 1.1B and 7B, respectively, using multi-domain fine-tuning. Furthermore, our results show that transferring knowledge from 7B outperforms 13B overall by 3.93%. Specifically, the performance improvement is 11.9% for seen tasks and 0.1% for unseen tasks. This indicates that our approach can alleviate conflicts between multiple tasks to a certain extent and improve out-of-domain generalization by leveraging the capabilities of the large model itself.In multi-task settings, our method achieves a performance improvement of 71.3% and 86.3% over 13B multi-task tuning when transferring knowledge from 1.1B and 7B, respectively. It can be observed that our method performs worse in domain-specific tasks compared to multi-task tuning, e.g., TriviaQA. This is primarily because we cannot access the data needed by the experts, resulting in a bottleneck in handling task conflicts. In the future, we will explore more effective ways to effortlessly transfer multiple small expert models to a large model. Notably, Proxy Tuning is designed for single-domain scenarios with a known test distribution. It struggles at test time with inputs from unknown distributions, making it difficult to use in multi-task scenarios. However, our method achieves the best results on unseen tasks, demonstrating its effectiveness in enhancing the generalization ability of large models on new tasks. Moreover, our method can effortlessly transfer knowledge from pretrained experts across multiple domains to the large model without requiring access to their training data. 
analysis: 
 5.1 How α control the knowledge transfer from the small models?We aim to understand how α dynamically changes in single-task and multi-task scenarios, and what patterns it exhibits across different tasks. As shown in Figure 2 , we compare our method with predefined α values of 0.5, 1.0, and 1.5. It can be observed that our method consistently outperforms the predefined α settings, demonstrating the effectiveness of adaptive adjustment at each decoding step. Additionally, the predefined method exhibits significant variation across different tasks, whereas our method addresses this issue and avoids the extensive effort required for parameter tuning.As shown in Figures 3 and 4 , we illustrate the variation of α for a single expert and multiple experts, respectively. In Figure 3 , it can be observed that α fluctuates between the lower bound and upper bound. At the lower bound, less expert knowledge is transferred, and it is more important to retain the large model's inherent capabilities. Conversely, at the upper bound, more expert knowledge is transferred, requiring more GSM8K-related capabilities for those steps. In Figure 4 , when solving the "abstract algebra" problem in MMLU using four experts, it is evident that GSM8K transfers more knowledge overall, especially in the initial steps. Since "abstract algebra" is a mathematically oriented problem, these results demonstrate that our method can effectively select and transfer expert knowledge during the decoding steps.In a batch inference with a size of B, Proxy Tuning requires a complexity of approximately O(BV ) for each decoding step. In contrast, our method requires O(nBV ) complexity, where n (≤ 20) is the number of parameter searches. Since n ≪ V , our complexity is comparable to Proxy Tuning. Figure 5 : Enhance in-context learning and task arithmetic using our method.In multi-task settings, performing parameter searches for each expert and then merging them would result in exponential growth, with a complexity of O(n T BV ). To optimize this process, we avoid merging parameters between experts at each step and instead select a single expert for knowledge transfer. This reduces the algorithm complexity to O(nT BV ). Additionally, we can constrain the parameter search range for each expert to achieve better efficiency and performance.Here, we complete the training and inference on an H100 GPU for 1000 data, while recording the time taken. As shown in Table 3 , our method achieves similar efficiency to Proxy Tuning during inference. Due to the need for inference and communication between multiple models, our approach is approximately 2.5 times slower than direct inference with a 13B model. However, our method only requires training a small model, which provides a significant advantage in training efficiency.Comparison against In-Context Learning Enhancing in-context learning with logits arithmetic can improve large language models. Since both methods work at the token level and need only black-box access, they can be combined to boost model performance. We categorize tasks as either Single-Domain (excluding MMLU) or Multi-Domain (MMLU). The "All" category averages the results of both. Notably, we use 5-shot information as in-context examples, and we apply different few-shot samples and prompts for various MMLU tasks. In our experiments, we enhance a 13B model's performance using a 7B expert and compare it to the 13B model using in-context learning. We present the results in Figure 5a . We observed that our method outperforms 5-shot in-context learning on both Single-Domain and Multi-Domain tasks. Furthermore, when combined with incontext learning, our method shows a significant improvement on Multi-Domain tasks, resulting in an overall (All) increase of 18.3%. Specifically, combining our method with in-context learning does not significantly impact performance on Single-Domain tasks, indicating that our method alone has already achieved capabilities similar to in-context learning for individual tasks.Comparison against Task Arithmetic Task Arithmetic and logits arithmetic are similar in principle, both adjusting the shift between the expert and base model to control knowledge transfer. However, logits arithmetic isn't constrained by parameter size and can merge across model levels without needing access to specific parameters. Specifically, our method can be applied to combine experts from task arithmetic, integrating both approaches for multi-task learning. We use the same setup as our multi-task scenarios, treating "All" as the average of Seen and Unseen Tasks. In figure 5b , our method performs well on Unseen Tasks, resulting in an overall improvement of 5.7% compared to task arithmetic. When we combine our method with task arithmetic, we see improvements over task arithmetic alone, achieving the highest overall performance, with increases of 7.2% and 1.5% over task arithmetic and our method alone, respectively. 
conclusions: 
 In this paper, we introduce a dynamic logit fusion approach in weak-to-strong specialization, which utilizes a series of task-specific small models and allows for adaptive weight allocation among them.Through extensive experiments, we have demonstrated the effectiveness of our approach in both single-task and multi-task settings across various benchmarks. By transferring expertise from the 7B model to the 13B model, we have achieved significant performance improvements, closing the performance gap by 96.4% in single-task scenarios and by 86.3% in multi-task scenarios compared to full fine-tuning of the larger model. Our method also shows promising results on unseen tasks and can integrate in-context learning for single tasks and task arithmetic for multi-task scenarios. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification : The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: In Appendix G, all the creators or original owners of assets used in the paper are properly cited and and provides licensing information for datasets used in our work.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We submit the code in the supplementary material with detail scripts and guidance.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As illustrated in Figure 6 , our approach dynamically calculates the fusion weights of the logits of SLMs at each decoding step, transferring the knowledge from multiple experts to LLM. The architecture of our method. The small llama represents the small model, while the large llama represents the large model. M S /M L denotes the logits of the small/large language model. M S t represents the logits of a small expert language model for task t. The lower part of the figure illustrates our optimization in the decoding process, where each circle represents a decoding step. The upper part of the figure shows how our method transfers the knowledge of experts in the kth step. At each decoding step, our method dynamically adjusts the {α t } T t=1 value for each expert, transferring knowledge from the small models to the larger model.Reinforcement Learning from Human Feedback (RLHF) is commonly employed in optimizing language models, where the learning process involves maximizing a reward objective while penalizing the KL divergence between the learned policy and a reference policy with a coefficient β:EQUATIONFollowing the DPO [43] framework, we can reformulate the above objective as:EQUATIONwhereZ(x) = y π ref (y | x) exp 1 β r ϕ (x, y)is the partition function. Note that the partition function is a function of only x and the reference policy π ref , but does not depend on the policy π θ . We can now defineEQUATIONThen the equation is convert toEQUATIONHence we have the optimal solution:EQUATIONfor all x ∈ D.According to Rafailov et al. [43] , reward models can be reparameterized as:r(x, y) = β log π(y|x) πref(y|x)for some model π(y | x) and a given reference model π ref (y | x). Additionally, given a pretrained language model π and its fine-tuned counterpart π ft , the following relationship holds,EQUATIONThis indicates that the fine-tuned model π ft can be interpreted as the solution to constrained reinforcement learning problem with a constraint on the pre-trained model. Consequently, the theoretical framework is applicable to any fine-tuned model, providing an RL-based interpretation of fine-tuning.We can further introduce an S-size implicit reward for L-size finetuned models:π L ft (y | x) = 1 Z(x) π L (y | x) exp 1 β r S ft (x, y) ∝ π L (y | x) exp log π S ft (y | x) π S (y | x) ∝ π L (y | x) π S ft (y | x) π S (y | x) (13)By taking the logarithms, we derive the following logit arithmetic formula:EQUATIONwhere M L , M S , and M S f t represent the logits of the large model, small model, and fine-tuned small model, respectively. This completes the derivation of the Equation ( 2).Based on the definitions of the probability distributions Q, Q f t , P , and P , the model logits outputs M S , M S f t , and M L satisfy the following relationships:EQUATIONSuppose there are vectors x and y of dimension n. The relationship between them is as follows:EQUATION)For each value x i , as x i increases, x j -x i decreases, and 1 e x j -x i increases. Therefore, the corresponding y i is positively correlated with x i .So we can know that ln y ∝ y ∝ x. Therefore, based on Equation (15) , we can derive the following relationship:EQUATIONThus, logit arithmetic can be viewed as a product-of-experts ensemble as follows:EQUATIONEssentially, it assumes a predefined distribution shift (Q f t (x k |x <k ) Q(x k |x <k )) α between the fine-tuned and pretrained distributions of a small model, which can construct the trajectory of P . It then attempts to approximate the shift for the fine-tuned and pretrained distributions of the large model P (x k |x <k ) P (x k |x <k ) .In section 5.2, we conducted an Efficiency Analysis of logit Arithmetic. To better illustrate our efficiency, we further analyze the overall efficiency of our method here.Overall, during a single forward pass, our method has a similar time complexity to the static method. Given: current sequence length s, large model dimension h L , small model dimension h S , number of layers in the large model L 1 , number of layers in the small model L 2 , batch size B, vocabulary size V , number of searches per decoding step n. Assume the FLOPs for a single forward pass of the large model and the small model are F LOP s L and F LOP s S , respectively. The FLOPs can be calculated as:F LOP s L = L 1 * (12Bsh 2 L + 2Bs 2 h L ) + Bsh L V ,and F LOP s S = L 2 * (12Bsh 2S + 2Bs 2 h S ) + Bsh S V (here we ignore the kv cache). Therefore, the FLOPs for a single forward pass of our method on a single task is: F LOP s L + 2 * F LOP s S + nBV . Among these, only the nBV term (n ≤ 20) corresponds to the additional computational cost of our method, which is much smaller compared to the previous term and can be considered negligible in the overall time. Additionally, in our efficiency analysis, as shown in Table 3 , our method is only 0.008 seconds slower per sample compared to the static method, which is negligible.This section mainly explains how we extend the transfer problem to multiple small models. When transferring the knowledge of multiple expert SLMs to a LLM, we consider the following two aspects: 1. The fusion of knowledge from different domain experts. 2. The transfer of knowledge from SLM to LLM, i.e., the transfer of knowledge from a single expert, which was discussed in Section 3.2. Intuitively, we first focus on the fusion of different domain experts' knowledge before performing the transfer. Here, we define the distribution of the combined knowledge of these small models as J. Therefore, we aim to achieve D KL (P || P ) = D KL (Q||J).Since solving for J is difficult, we propose constraining it based on the relationship between J and {Q i } to approximate it. Here, we can transformD KL (Q||J) into D KL (Q||Q i ) + C J (Q i ), where C J (Q i )is the bias function from Q i to J. When we approximate J as the centroid of {Q i } on the KL-constrained plane, we can implicitly solve these bias functions. According to the definition of the centroid, J can be solved by minimizing the sum of the squared distances to each point, as shown below:EQUATIONSince our goal is D KL (P ∥ P ) = D KL (Q||J), substituting this into our equation gives us our final optimization objective:EQUATIONTo prove the reasonableness of our approximation, we provide a more rigorous proof below. Our initial objective is as follows:EQUATIONBy assuming D KL (Q||J) = D KL (Q||Q i ) + C J (Q i ), we can transform the original problemEQUATIONinto T constrained optimization problems:EQUATIONAfter jointly optimizing them, we have:EQUATIONSince C J-Q is a constant term independent of P , we can ignore it. Finally, we solve the original problem by optimizing this upper bound. When we symmetrize the terms in the KL divergence, we can obtain a similar conclusion. Therefore, in the multi-task setting, we can solve it using the following formula: (As shown in Equation ( 6)):EQUATIONRequire: generation prompt X, number of tokens to generate N , Large model θ L , domain number T , expert small models {θ S t } T t=1 , M L , M S , and M S t represent the logits outputs of the large model, small model, and t-th domain-specific small models. P , Q, and Qt represent the outputs distribution of the large model, small model, and t-th domain-specific small models.1: k ← len(X) -1 2: m ← ∞ 3: α ← [] 4: while len(X) < N and x k-1 ̸ = [EOS] do 5:for each domain t in T do 6:Get domain expert logits for x k from θ S t as M S t (x k |x <k ) 7:end for 8: # For multitask scenario, perform α search only for one task, for a total of T times. 9: for search α ′ ∈ R T do ▷ such as [0.0,2.0], step is 0.1 10: P (x k | x <k ) ← softmax M L (x k |x <k ) + T t=1 α ′ t • (M S t (x k |x <k ) -M S (x k |x <k )) 11: L ← T t=1 KL( P ||P ) -KL(Qt||Q) 2 + KL(P || P ) -KL(Q||Qt) 2 12: if L < mP ← softmax M L (x k |x <k ) + T t=1 αt • (M S t (x k |x <k ) -M S (x k |x <k )) 18:Sample the next tokenx k ∼ P (x k |x <k ) 19: X ← {X; x k } 20:k ← k + 1 21: end while 22: return generated text X Our algorithm is outlined in pseudo-code in Algorithm 1, we search for each task's α with a step of 0.1.• GSM8K [8] , a dataset of 8.5K high-quality linguistically diverse grade school math word problems created by human problem writers. We evaluate using exact match (EM.). • TruthfulQA [32] , a benchmark to measure whether a language model is truthful in generating answers to questions. There is no training set for this dataset. We evaluate using multiple-choice accuracy (Acc.). • TriviaQA [21] , a reading comprehension dataset containing over 650K question-answerevidence triples. We evaluate using exact match (EM.). • MMLU [13] , a benchmark designed to evaluate the capabilities of language models. It comprises approximately 16,000 multiple-choice questions across 57 tasks. We evaluate using multiple-choice accuracy (Acc.). • CNN-DailyMail (CNN/DM) [47] , a dataset for text summarization. We evaluate using Rouge2.The GSM8K, MMLU datasets are licensed under the MIT License. The TruthfulQA, TriviaQA, CNN-Dailymail datasets are licensed under the Apache-2.0 License. Thus, these datasets are available for non-commercial research purposes.As shown in Table 4 , our method achieves the correct output of 18 when targeting a single task, such as the GSM8K problem. Additionally, it can be observed that the model's output contains formatted information, e.g., "####" which matches the format learned in in-context learning. This indicates that our method achieves the ability of in-context learning at the logit level.For unseen tasks like MMLU's Abstract Algebra and Clinical Knowledge, our model generates correct answers by leveraging multi-expert knowledge transfer.Our approach shares the general limitations of existing logit-arithmetic-based methods: (1) Since our method operates at the decoding step, this architecture mainly tests its ability in generative models, without exploring its effectiveness in discriminative models; (2) Involving multiple models in the decoding step can slow down inference compared to inference with models of similar magnitude.In the future, we will explore more efficient methods of logit arithmetic and investigate their potential application in other architectural models.Our approach enables weak-to-strong knowledge transfer without accessing the model's parameters or expert-learning data. This is advantageous for protecting the privacy of expert data. Additionally, this data-free fine-tuning paradigm significantly reduces the cost of private deployment while enhancing data security, thus positively impacting societal data security. Our method does not exhibit any significant negative societal impacts.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Our paper uses publicly available datasets and provides the complete code and execution scripts in the supplementary material. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://github.com/vllm-project/vllm 


Decision: This paper proposes a dynamic logit fusion method for transferring knowledge from smaller, task-specific language models to a larger language model. The method adaptively learns the mixing weights between the models' logits at each decoding step by optimizing an objective based on KL divergence.

Reviewers found the method to be simple yet effective, achieving good results across various tasks. (YRcj: "The method is very simple... leading to better results."; frHX: "Paper well written and easy to follow... Works well in single task scenarios."). The experimental evaluation was considered comprehensive, covering multiple tasks, model sizes, and both single-task and multi-task scenarios. (YRcj: "The experiments are comprehensive... The authors also consider both single-task and multi-task scenarios."). The proposed method consistently outperformed baseline methods, including proxy tuning and full fine-tuning on smaller models, and approached the performance of directly fine-tuning the large model. (YRcj: "The results are good across all tasks... recovers a large fraction of the ceiling performance achieved by directly finetuning the large model on ground truth.")

However, the reviewers identified a few potential weakness in the initial reviews:
 - Limited theoretical justification: Initially, reviewers raised concerns about the theoretical motivation behind matching KL divergence as the objective function. (YRcj: "It is not intuitively obvious to me why matching the KL divergence is the right objective."; frHX: "Using the squared error between two KL’s is not theoretically motivated."). However, the authors addressed this concern in their rebuttal by providing a theoretical derivation based on KL-constrained reinforcement learning. (XMhg: "I thank the reviewers for their rebuttal and I appreciated their theoretical derivation. I believe that this important piece to be added to the paper.")
- Clarity of presentation: Some reviewers found certain aspects of the presentation to be unclear, particularly regarding the optimization method and the multi-task setup. (frHX: "More description of the optimization method in main text since it is a big part of the method... Missing baseline in multi-task tuning setup.") The authors addressed these concerns in their rebuttal by providing further details and clarifications.
 - Computational complexity: The method was noted to be slower at inference time compared to standard sampling. (YRcj: "As the authors mention, the proposed method is 2.5 times slower at inference time compared to standard sampling from the same model.") While the authors clarified that the increased cost is minimal compared to fine-tuning the large model, the efficiency aspect remains a potential limitation.

The paper presents a novel and effective method for weak-to-strong generalization in language models. The comprehensive experimental evaluation demonstrates the method's strong empirical performance. While some initial concerns regarding theoretical justification and computational cost were raised, the authors adequately addressed these concerns in their rebuttal.

Reviews: review: paper_summary: paper_summary: This paper focuses on the weak-to-strong generalization paradigm where the goal is to transfer knowledge from a small language model to larger one. The method they study is the one proposed by Mitchell et al. [1]: they use log probability algebra to combine the logits of the large model, the ones of a small model and the ones of a small model that has been finetuned. This combination involves a parameter $\alpha$ that controls the contribution of the small model. The main contributions of this paper is to point to the limitations of using a static $\alpha$ and to propose a method to adaptively learn such $alpha$. Their method consists in optimizing an objective based on the KL divergence and they show that their approach is consistently better than using a static alpha across a wide range of downstream tasks.




[1] Mitchell, Eric, et al. "An emulator for fine-tuning large language models using small language models." arXiv preprint arXiv:2310.12962 (2023).
strength_weakness: strength_weakness: Overall, my main concern is that I find the contribution limited and I have some doubts about the method. Here is a detailed list of my concerns:

- **Computational feasability**: I think that the authors should be more transparent in the computational cost of their procedure. Solving the optimizaiton problem at every decoding step may be very expensive and it is not clear to me that when one needs to do many generations, a finetuned model with LoRA on the large model is cheaper than the procedure proposed by the authors. Also, is it important to update alpha at each decoding step? Can't one get a more efficient procedure by updating it only every 100 tokens or so? 

- **Not enough clear to me that the method does much better than the static $\alpha$**: when I see the barplot of figure 2, it seems like $\alpha=1$ is a bit below than the learnt $\alpha$ but the gap is not huge. 

- **Theoretical justification for the optimization problem?**: So if I understand correctly, the authors objective function is to say "I want the distance between the predictions of $\tilde{P}$ and the large model is the same as the distance between the predictions of the finetuned small model and the small model".   This looks like a reasonable belief. However, it would have been nice to have a theoretical justification. For instance, when you do RL finetuning, if you solve the problem exactly, the distribution you end up generating from ends up being p_bayes(generation) * exp(reward model you are training on) (usually done with ppo, dpo). When you take the logs, you get log(p) = log(p_bayes) + reward model.  Then you can estimate the reward model by taking the difference of logits for any model scale and in this case the intuition of the authors make sense to me and it is principled. However, in the standard finetuning case, when the authors apply this intuition at the level of tokens, it is not clear to me why it should work.

- **Scaling experiments for studying the approach**: I know that the Llama suite starts at 7B but it would have been nice to study the behavior of the method with smaller models than 7b.  Understanding how robust the method is by varying the gap between the weak and strong models is important. There are maybe chances that the learnt $\alpha$ approach shows bigger  gaps with respect to the static $\alpha$ when the gap between the weak and strong models is large.
questions: questions: I would appreciate that the authors address the concerns I have regarding the theoretical justification of their optimization problem, the computational cost of their approach.
limitations: limitations: I think that the authors didn't clearly state the limitiations of their approach, which is regretful.
paper_summary: paper_summary: The paper studies the problem of adapting large general language models via smaller expert language models fine-tuned on specific tasks. Prior work proposed the idea of mixing logits between a large model and the differencei in logits pre- and post- finetuning of a small model. The authors take this idea a step further and compute the mixing weights adaptively per-token, leading to better results.
strength_weakness: strength_weakness: 1. It is not intuitively obvious to me why matching the KL divergence is the right objective. Could the authors please provide some intuition? I imagine it is something like this: when the small model updates significantly for some token, we want the large model to also udpate significantly. That seems reasonable, but probably doesn't always work well: if the small model is unaware of some fact or makes an arithmetic mistake, it may need to update significantly on the corresponding tokens, while a large model does not need to update.

2. The presentation is not always very clear. For example, in Eq. (5) it is not clear to me what the authors mean by the joint distribution of $Q_1, Q_2, \ldots, Q_T$. How can we compute KL between this joint and $Q$?

3. As the authors mention, the proposed method is 2.5 times slower at inference time compared to standard sampling from the same model.
questions: questions: 1.1 and 1.2: See weaknesses 1 and 2.

2. What exactly do you mean by supervised instruction tuning: what are the smaller models fine-tuned on? Are these chain-of-thoughts for solving the task, e.g. GSM8k? Where do they come from? 

3. In Figure 3, qualitatively, what do the tokens (decoding steps) where we set the weight to the lower bound correspond to, and same for the upper bound? Are they qualitatively different?

4. In Figure 3, why is the lower bound 0.8 and the upper bound 1.5? Are these tunable parameters? 0.8 seems quite high, shouldn't we want to set the lower bound to 0?
limitations: limitations: Limitations are adequately discussed.
paper_summary: paper_summary: They tackle the problem of merging the logits from multiple models. To do so, they propose an objective that minimizes the squared loss of the KL between the two pairs of (student, teacher) models. This is solved via a random search.
strength_weakness: strength_weakness: - Using the squared error between two KL’s is not theoretically motivated (at least that I am aware of) 
- More description of the optimization method in main text since it is a big part of the method 
- Missing baseline in multi-task tuning setup
questions: questions: - Where is the baseline proxy tuning for multi-task tuning in Table 2? 
- From algorithm 1 in Appendix C, the optimization is done just via random search? Guessing values and storing the best? I couldn’t find the method to optimize the objective mentioned in the main paper. It should be more clearly stated in the main paper, and how it handles multitask setup. 
- For the efficiency analysis, the BV can be done in 1 forward pass all in parallel, but the n parameter searches require 20 forward passes. It really is n times more compute, which is not minimal. For example, decoding 100 tokens vs decoding 1 token would be a constant in the Big O, but they are different efficiency wise. 
- In eq (2), is the second term (multiplied by alpha) not normalized but in logit space? 
- In fig 4, are there some special about the tokens being generated at the timesteps where $alpha$ is high?
limitations: limitations: NA
paper_summary: paper_summary: The article discusses existing weak-to-strong methods, noting that current approaches typically use a static knowledge transfer ratio and a single small model to convey complex knowledge, which results in suboptimal performance. Consequently, the article proposes a dynamic logit fusion method that employs a series of task-specific small models and adaptively allocates weights among these models at each decoding step. The weights are learned by optimizing a problem constrained by Kullback-Leibler divergence. The article conducts experiments on various benchmarks, including both multi-task and single-task scenarios.
strength_weakness: strength_weakness: 1. In Section 3.2, the proposed method is based on an assumption that lacks clear supporting evidence. Does this somewhat undermine the theoretical foundation of the algorithm?

2. In the multi-task experiments, the results for CNN/DM do not demonstrate the method's superiority. Additionally, in the experiments on unseen tasks, the method does not show significant improvement.

3. In the experiments, all evaluations were conducted in a 0-shot setting. How would the evaluation results change if a 5-shot experiment were conducted?
questions: questions: 1. In the multi-task setting, which 7B model is used as the expert to implement your algorithm on the 13B model? I am very confused. If you are using different sets of experts to operate on the 13B model, isn't this weak-to-strong? Because the parameters of multiple 7B models exceed 13B.
2. There seems to be a typo in formula (9) in Appendix B, it seems to be $\propto$ rather than $=$.
limitations: limitations: The authors have adequately addressed the limitations.

Comments: title: Post-rebuttal response
comment: We thank the reviewer for raising the score, and we will add this piece in our future versions.
Dear Reviewer Ttwe:

We thank the reviewer for their feedback and for raising the score!

If you still have any unresolved concerns about our paper, we are open to further discussion. If you feel that the previously mentioned weaknesses (or questions) have been addressed and have no remaining concerns, would it be possible for you to reconsider our paper?
I thank the reviewers for their rebuttal and I appreciated their theoretical derivation. I believe that this important piece to be added to the paper. I hope they will do it. i increase my score by one point.
Thanks for your response, I have raised the point.
We thank the reviewer for the feedback. We will address the concerns below.

1、It should be noted that our method does not require fine-tuning the 13B model. Instead, our approach involves fine-tuning multiple 7B models and then transferring the knowledge to the 13B model without the need for gradients.

2、Although we have multiple 7B models, their capability is still far from reaching the level of a 28B model. Therefore, we cannot view the entire process as a migration from 28B to 13B. As shown in the table below, the 7B-Expert Best (the best result from each dataset within the 7B-expert models) still struggles to outperform the 13B Multi-Task Tuning results, especially on unseen tasks. Other gradient-free methods (e.g., average[1], task arithmetic[2]) that do not involve training also find it difficult to combine four 7B models into a strong model equivalent to 28B, and they even fall short of surpassing the 13B model. Therefore, without training, the capability of four 7B models is weaker than that of the 13B model.

|                       | seen task | unseen task |
| --------------------- | --------- | ----------- |
| 13B Multi-Task Tuning | **40.78** | **50.58**   |
| 7B-Expert Best        | 40.02     | 46.61       |

3、In a multi-task setting, the logit arithmetic in formula (6) can be expressed as $13B+\sum_i^m  \alpha_i diff^{7B}_i=\sum_i^m (\frac{1}{m}13B+\alpha_i diff^{7B}_i)$ ，where $diff^{7B}_i$ represents the shift of the i-th expert on the 7B-base in logits. Compared to the 28B MoE, our method actually performs as a combination of multiple weak-to-strong approaches (marked by parentheses in the above formula). Our method adjusts $\alpha$ to control their combination ratio, thereby facilitating the transfer of multiple small experts, as described in the Supplementary Proof for the Fusion of Multiple SLMs Scenario in the Global Rebuttal.



- [1] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. (ICML2022)

- [2] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models (NIPS2023)
Thank you for answering my question, but I'm afraid I disagree with your response to the first question. As I understand it, if you use four 7B experts to fine-tune a 13B model, this doesn't qualify as **weak to strong**. In fact, you're using a 28B MoE model to fine-tune the 13B model. If you could clarify this point better, I would consider raising the score.
Dear Reviewer Ttwe:

We wish to thank you again for your constructive feedback which has helped us to improve the clarity and contribution of our work. As the discussion period draws to a close, we hope our response has effectively addressed all your concerns. Your insights are invaluable to us, and we remain open to further discussion if you have any questions regarding our response.
Dear Reviewer YRcj:

We wish to thank you again for your constructive feedback which has helped us to improve the clarity and contribution of our work. As the discussion period draws to a close, we hope our response has effectively addressed all your concerns. Your insights are invaluable to us, and we remain open to further discussion if you have any questions regarding our response.
Dear Reviewer XMhg:

We wish to thank you again for your constructive feedback which has helped us to improve the clarity and contribution of our work. As the discussion period draws to a close, we hope our response has effectively addressed all your concerns. Your insights are invaluable to us, and we remain open to further discussion if you have any questions regarding our response.
Thanks for the response. I keep my score as is.