On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion
Authors: Chenghao Fan
Zhenyi Lu
Wei Wei
Jie Tian
Xiaoye Qu
Dangyang Chen
Yu Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: weak to strong
natural language generation
Generative AI
large language model
gradient-free approach
Primary area: natural_language_processing
TL;DR: a dynamic logit fusion approach for transferring knowledge from a series of task-specific small models to a larger model

Abstract: Efficient fine-tuning of large language models for task-specific applications is imperative, yet the vast number of parameters in these models makes their training increasingly challenging.
Despite numerous proposals for effective methods, a substantial memory overhead remains for gradient computations during updates. \thm{Can we fine-tune a series of task-specific small models and transfer their knowledge directly to a much larger model without additional training?} 
In this paper, we explore weak-to-strong specialization using logit arithmetic, facilitating a direct answer to this question.
Existing weak-to-strong methods often employ a static knowledge transfer ratio and a single small model for transferring complex knowledge, which leads to suboptimal performance. 
To surmount these limitations,
we propose a dynamic logit fusion approach that works with a series of task-specific small models, each specialized in a different task. 
This method adaptively allocates weights among these models at each decoding step,
learning the weights through Kullback-Leibler divergence constrained optimization problems. 
We conduct extensive experiments across various benchmarks in both single-task and multi-task settings, achieving leading results.
By transferring expertise from the 7B model to the 13B model, our method closes the performance gap by 96.4\% in single-task scenarios and by 86.3\% in multi-task scenarios compared to full fine-tuning of the 13B model. Notably, we achieve surpassing performance on unseen tasks. Moreover, we further demonstrate that our method can effortlessly integrate in-context learning for single tasks and task arithmetic for multi-task scenarios.

On Giant's Shoulders:
Effortless Weak to Strong by Dynamic Logits Fusion
Chenghao Fan!.: Zhenyi Lu 12- Wei Wei!.2f
Jie Tian 122 Xiaoye Qu',
Dangyang Chen', Yu Cheng School of Computer Science & Technology, Huazhong University of Science and Technology
2 Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL.
Ping Property Casualty Insurance Company China, Ltd.
The Chinese University of Hong Kong
{facicofan, 1uuciicoann,uhheyiii229 1ww..nysshhenn92ggee11 J@gmall. com, (weiw, xiaoye]@hust. .ed.. cn en, chengyuucse. cuhk, edu. hk
chendanngyang227-92nngnaaa
"If I have seen further than others, it is by standing on the shoulders of giants."
Isaac Newton 1675
Abstract
Efficient fine-tuning of large language models for task-specific applications is imperative, yet the vast number of parameters in these models makes their training increasingly challenging. Despite numerous proposals for effective methods, substantial memory overhead remains for gradient computations during updates. Can fine-tune series of task-specific small models and transfer their knowledge directly to a much larger model without additional training? In this paper, we explore weak-to weak-to-stron, to--otooogg specialization using logit arithmetic, facilitating direct answer to this question. Existing weak-to-stee to-s-ongg methods often employ a static knowledge transfer ratio and a single small model for transferring complex knowl- edge, which leads to suboptimal performance. To surmount these limitations, we propose dynamic logit fusion approach that works with series of task-specific small models, each specialized in different task. This method adaptively allocates weights among these models each decoding step. learning the weights through Kullback-Lein divergence constrained optimization problems. We conduct ex- tensive experiments across various benchmarks in both single-task and multi-task settings, achieving leading results. By transferring expertise from the 7B model to the 13B model, our method closes the performance gap by 96.4% single-tase scenarios and by 86.3% in multi-task scenarios compared to full fine-tuning of the 13B model. Notably, we achieve surpassing performance on unseen tasks. More- over, we further demonstrate that our method can effortlessly integrate in-context learning for single tasks and task arithmetic for multi-task scenarios.
Introduction
In recent years, Large Language Models (LLLMs) have shown impressive performance a wide range of tasks [5, 36, 50-52, 54, 61, 62], including code generation [11, 45), mathematical reasoning [2, tool-use abilities [46, 53], etc. However, training such LLMs requires substantial computational resources, often involving thousands of GPUs and processing trillions of tokens [30, 42], making the adaptation of the base model for new knowledge inefficient. To address these challenges, parameter- efficient tuning methods [12, 16, 23] have emerged, aiming achieve comparable performance full fine-tuning while reducing GPU memory requirements. However, challenges persist in tuning
Equal contribution. Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and deploying large-scale models on common hardware, as they still involve computation intensive processes like gradient calculation and back-propagation. Furthermore, these methods may not be feasible when training data are private.
This inspires a much larger model without requiring additional gradient updates? If we could fuse the strong capabilities of a scaled LLM with the specialized knowledge acquired by small model during fine-tuning, it would yield the practical benefit of approximating the results achieved by fine-tuning a large model, but without the associated computational costs. However, is non-trivial due to the differences in representation width and layer numbers between the small and large models.
ask: Can we fine-tume only small models and then transfer their knowledge to
Recently, Mitchell et al. [41] and Liu et al. [33] attempt to address this challenge by transferring knowledge from Small Lan- guage Model (SLM) to its larger counterpart through simple logit arithmetic operations during decoding. For instance, using models from the Llama-2 family, they can transfer the fine-tuned knowledge from a 7B-scale model to a 13B-scale model by performing log probability algebra: Llama-2-base 13B + (Llama-2-chat 7B 1llmm----base 7B), where the first term represents the base log proba- bilities and the term in parentheses denotes the behavioral delta [41]. This behavioral delta can be weighted to adjust the balance between the pretrained knowledge and the transferred fine-tuned knowledge.
Peter is 78 years old this year. How old will he be in 500 years?
LLM
Pretrained:
Task-Specific SLM
SLM
Privious Work on Logit Arithmetic
LLM
FIRE
SLM
Incomprehensible Incomplete,
X Unaligne
50-500
Ours
G
LLM
Despite showing promise, logit arithmetic still exhibits a noticeable performance gap compared to directly fine-tuning large mod- els, primarily due to two reasons: Firstly, they statically prespecifies the weight of be- havioral delta at each decoding step identi- cally. However, the importance of fine-tuned knowledge varies significantly across differ- ent tasks, inputs, and even different decod- ing steps. For instance, a domain-so sppeiffc question-answering process, need more knowledge from fine-tuned small language models. Conversely, when decoding factual topics, we may need more knowledge from pretrained general models. Secondly, for un- seen tasks, the lack of pre-adjusted weights for the behavioral delta prevents logit arithmetic from executing effectively, making it challenging to transfer tuned knowledge, especially for complex tasks. As shown in Figure 1, when answering "Peter is 76 years old this year. How old will he be in 500 years?", using only the math expert does not ensure the result aligns with factual accuracy. Additionally, these techniques often assume that experts are trained and tested on the same data distribution, ignoring the heterogeneity of data that may be encountered at test time, rendering any single expert insufficient.
Adaptive Knowledfe Transfer Optimization
SLM
Pure math 76-500-576, reality humans are generally much shorter and Peter not necessarily alive after 500 years.
Comprehensible,
Aligned
Complete dnterpretable
Figure I:
Comparison between our work and previous work. Previous methods only use pre-tuned parame- ters a to transfer knowledge from single expert. In contrast, our method dynamically adjusts the propor- tion of knowledge transferred from multiple experts at each decoding step during inference.
In addressing these challenges, we reconsider the practice of logit arithmetic within new framework. Specifically, we work with a set of finely tuned SLM experts, each specializing in different tasks. At each decoding step, we dynamically allocate weights among these task-specific SLMs. However, it is non-trivial to effectively determine suitable weights for diverse tasks and SLM experts, as tranditional approaches such as grid search [29] or combinatorial search [34] suffer from costly search processes. To practically automate weight allocation, we reframe the problem of weight search as constrained distribution optimization problem. We tackle the fusion of multi-task knowledge by treating it as a centroid problem in Euclidean space using Hailbback-Leiblee divergence [24], which offers interpretability and explicit guidance.
2

---

We conduct thorough experiments on the LLaMA series to evaluate the effectiveness of our approach, applying our adaptive logit arithmetic method task fask-specific fine-tuning math, question-an -nswering. summarization, and multi-domain tasks. We also analyze the performance of multi-model fusion across seen and unseen tasks. Additionally, we discussed the feasibility of combining our method with in-context learning for single tasks and with task arithmetic for multi-task scenarios.
In our work, we make several key contributions. Firstly, we reassess existing logit arithmetic methods, highlighting the significant impact of fusion weights and the limitations imposed by a single small model on test performance. Secondly, we introduce a novel approach that autonomously learns fusion weights through constraint optimization, approximating the compute- intenssve results of fine- tuning large base model. Lastly, we conduct comprehensive experiments to validate our method, demonstrating substantial improvements in performance, generalization capabilities, and robustness.
2 Related Work
2.1 Efficient Specialization
Specializing pretrained model by fine-tuning for downstream tasks has become primary paradigm [7, 9, 10, 44]. However, with the increasing scale of models, full fine-tuning has become impractical due to the large number of parameters, requiring costly training resources [5, 54].
To tackle this challenge, researchers have developed parameter-efficient tuning methods [28]. These methods aim to achieve performance similar to full fine-tuning while reducing the GPU memory required. They typically involve freezin the original weights and adding task-specific trainable modules [15, 22, 26], low-rank matrices [12, 16, 23]. or bias [3]. Despite their benefits, these approaches still demand significant memory for updating gradients and might not be suitable for scenarios where training data are private or inaccessible. Another direction is model merging [17, 35, 38, 56, 58], which trains task-specific models on different domains, and then combines them into single model at deployment with weight average [57]. neuron permutation [1], interpolation [20, 39] or task vectors [17, 35, 59]. However, they typically suffer from parameter interference between different tasks and static optimal solutions may struggle with multiple task domains.
There a growing research interest specialized large models by eliciting existing knowledge, such as utilizing curated prompts via in-context learning [31] or employing small models to offer weak- to-strong supervision [6, 33, 41], generating weak labels or alignment signals. These methods are highly practical, requiring only the generated output (or logits). Our approach shares a similar weak- to-strong intuition and proposes adaptive strategies to achieve better balance between leveraging the pretrained general knowledge and acquiring task-specific knowledge. Furthermore, our method can be extended to multi-task scenarios.
2.2 Weak-to-Strong Generation
Different from knowledge distillation [4, 14, 49], where a more capable teacher guides a student model, we explore using weaker models teach stronger ones. Burns et al. [6] empirically demon- strate that weak supervisors can reliably elicit knowledge from much stronger models (e.g., supervis- ing GPT-4 with GPT-2). Such approaches represent promising direction, it is more practical and affordable to specialize in a much smaller model. Contrastive Decoding [27] enhances outputs by leveraging differences between large and small LMs,, as patterns like repetition and incoherence are often more pronounced in smaller ones. Speculative sampling [25, 48) speeds up inference using a lower- pewer-parameter version of the LLM as a draft model, exploiting the f'ct that draft models can approximate easier subtasks well. SpecInfer [40] goes further by employing a set of small models to generate drafts in parallel. Jin al. [19] train a projector to map the parameter of the weak expert to a larger version, but the projector needs to be trained and suffers from poor generation. Ji et al. [18] focuses on utilizing small models to rephrase unaligned answers from LLMs into more aligned ones. Mitchell et al. [41] and Liu et al. [33] leverage logits from small, fine-tuner models to inject specific knowledge into the pretrained LLM with the same vocabulary. Our approach differs from methods that require pretrained small model adapters [48] or mre-adjuseed parameters [33, 41]. Instead, we dynamically transfer the capabilities of small models to the large model at each decoding step, without needing access to the large model's parameters or training data.
3

---

3 Methodology
3.1 Problem Background
Autoregressive Language Models Modern autoregressive transformers generate continuations for input prompts token by token. Given a prompt 21sk-1 (denoted as x<*)). the model computes the logits for the k-th token, represented M (ar T<ck) R where V denotes the size of the vocabulary. A probability distribution P(æx Tess) is then obtained through softmax normalization: P(rk T(ck) softmax(M (<))). The next token Iy is subsequently sampled from this distribution, Le., Th. ~ P(rk Tzk)-
Distance Between Language Model Outputs Distribution We can utilize the Kullback- Leibler(KL) divergence to measure the similarity of two distributions P and Q generated from Iwo language models (with the same vocabulary), which can be viewed the distance of the two language models:
P( T(ck)
DEL x<c)= = Epee P(x xeck log
(I)
Q(x x<c)
=EV
this is implied by the context, we will omit the conditioning on | and simply use Dkr.(P](Q)
Logit Arithmetic Suppose we have two pretrained auto-regressive models with homogeneous architecture and the same vocabulary: small model with parameter set @ and a large model with parameter set 0L_ We aim to fine-tune the small model to obtain 0%₁ and transfer this fine-tuning knowledge to the large models. Previous work [33, 41] transferred fine-tuned knowledge to large model by designing arithmetic between logits, resulting in the output distribution P for the large model follows:
P(Zx|T<<) = Befttmax(M'inn (*x (xe)) + a (Ix/(e)))))) M (xx(
(2)
where M4, M and Mt, represent the logits of the large model, small model, and fine-tuned small model, respectively. Their corresponding normalized distributions denoted by P. Q. and Qfe The detailed theoretical proof supporting this logit arithmetic formula is provided in Appendix B. Here, is pre-adjusted hyperparameter that controls the extent of knowledge transfer from the small model. Our analysis from Appendix C demonstrates that logit arithmetic attempts to approximate the shift (Qre(za(e<s)) ya between the fine-tuned distribution and the pretrained distribution by controlling the parameter before inference.
3.2 Adaptive Knowledge Transfer Optimization
However, using pre-defined presents the issue that the resulting LLM's distribution P has fixed trajectory, which may not be suitable for every step decoding. So we need a dynamic a in the decoding steps. Nonetheless, solving this problem is non-trivial, so we attempt to transform it into an equivalent, optimizable objective. Here, we use the distance between language model outputs as defined in Equation (1) to represent the shift between distributions. We assume that, for different model sizes, at each decoding step, the distance between the fine-tuned model outputs and pretrained model output is the same:
DKt. wee) se Q s<ck), DKI. xeck) Ro (RI(() (<e)) (3)
Following the assumption in Equation (3), optimize the following expression each decoding step:
arg min (Dea.(P1P)- - -Ka..Q (DKI,(PIIP) - (Ka(Q11Q).))
(4)
Based on this optimization problem, we can adaptive transfer the knowledge of the corresponding SLM expert to the large model at each decoding step in a single-task setting. Our method has been proven to be effective in experiments. To guarantee symmetry, symmetrical term is incorporated:

---

3.3 Extending to the Fusion of Multiple SLMs
When dealing with complex tasks or new domains, a general LLM may lack the necessary expertise, and a single SLM might not provide sufficient specialized knowledge due to the capacity gap. To migrate this challenge, our method can be extended fuse multiple SLMs and leverage their knowledge to compensate for the shortcomings in individual domain-specine knowledge. We fine- tune the SLM @ on each domain to obtain multiple sasc-specific experts (0,) making easier to acquire knowledge and dynamic fused to the LLM. During decoding, knowledge from these tomain-specine SLMs transferred simultaneously to the L.M.. We modify the Equation (3) as follows:
DKL (P || P) DKL ((Qi(((1. Q) DKL(P) || P) DKL (Q Joint( ((Q..((Q....)))) (5)
where Qe represents the distribution of 0, from the t-th domain, bution the combined knowledge of Q1, Q2. Qr. When we impose constraints like Equation (5), it attempts to align the joint distributions of the logits from the domain-specifie small models. How- ever, the distributions the logits from the domain-specific small models are usually not independent of each other, so their joint distribution cannot be estimated through simple multiplication.
represents the distri-
Due to the difficulty in obtaining joint distribution for multiple expert models, we decompose the joint distribution constraint problem into multi-object marginal distribution optimization. The transformation process we prove detail in Appendix E. By aligning the distributions of each domain-specific small model, we can infer approximate optimal solution within the extension of Equation (2), as shown by:
arg min Σ (DX(III))D-x(Q111))) (DKL(PIP)-Dxx((III)
DD (xI(Q)))))
(6)
where softmaxx
M (4(IaEE + Σ at (M M (x
Here M to represent the logit of t-th expert. Our algorithm is outlined in pseudo-code in Algorithm in Appendix F.
Intuitively, this projects the KI. divergences between the logits into Euclidean space and finds central point with the minimum distance sum as the best KI. value. This optimal KI. value corresponds to the output distribution of the large model with multi-domain knowledge. In experiments, optimize a E RR to obtain the optimal KL value. Generally, a is between and 2. In a multi-task setting, can accelerate the optimization process by optimizing the boundaries for only one expert, restricting only one SLM expert to be applied the current decoding step. We will provide a more detailed explanation of this in experiments.
4 Experiments
In this paper, same vocabulary, enabling us to employ logits arithmetic easily. Here, we use TinyLLaMA-1.18 [60] and LLaMA2-7B as our pretrained small models, and LLaMA2-13B as the large model for transfer. We conduct tests in single-domain and multi-domain settings to demonstrate that our method can effectively transfer knowledge the large model in both scenarios.
the LLaMA2 [55] family of models to test method, which contains the
4.1 Datasets
Following the setting in Liu [33], we evaluate on the following datasets: mathmetical reasoning
(GSM8K [8];; factual accuracy (TruthfulQA [32]); realistic knowledge (TriviaQA [21]): multi-domain general knowledge (MMLU benchmark [13]); summarization Malli (CNN/DM) [47]). All datasets are tested using 0-shot setting. Detailed information is provided in Appendix G.
4.2 Implementation Details
For all tasks except TruthfulQA, we construct prompts based on the task type and employ supervised instruction tuning to train each task expert. Following the previous work [33], we use "Llama-2- 7b-chat-hf" as the 7B expert for TurhfulQA, and TmyLLaMA-chht--errt
the I. expert for
5

---

Table 1: Performance on single-task scenarios. Bold numbers indicate the best-performing model transferred from the same size. Underlines indicate whether the method outperforms the expert model being used. Notably, we are unable to obtain the LoRA adapter for LLAMA2-chat version. Therefore, we set the LoRA Tuning for the 13B model on TruthfulQA to match the same values as Full Fine-Tuning, e. g,g,, 61.93.
GSM8K (EM.)
TruthfulQA (Acc.)
TriviaQA (EM.)
CNN/DM (Rouge 2.)
MMLU (Acc.)
Model
Avg.
13B
Base Model
6.90
46.13
36.44
8.94
51.25
29.93
Full Fine-tuning
47.23
61,93
56.36
15.50
57.94
47.79
LoRA Tuning
41.54
61,93
61.89
17.18
60.46
48.60
Transfer from I.IB
Full Fine-tuning
12.51
29.01
33.66
14.22
37.26
25.33
Proxy Tuning Ours
16.91
31.48
48.74
13.23
39.88
31.74
18.27
37.05
53.81
14.48
48.32
34.86
Transfer from 78
Full Fine-tuning
37.07
60.02
52.10
15.21
56.23
44.13
Proxy Tuning Ours
37.68
61.02
52.81
14.37
56.24
44.43
39.34
61.56
57.11
15.31
57.15
46.09
Table 2: Performance on multi-task scenarios. "Base" denotes an untrained model. "Multi-Task Tuning" refers models trained using data mixing. Bold numbers represent the best-performing multi-task models among those using experts of the same size. The leftmost "Avg." represents the average performance of Seen Tasks and Unseen Tasks (57 tasks MMLU), calculated by averaging the mean performance on Seen Tasks and the performance on MMLU.
Seen Task
Unseen Task
Model
Avg.
GSMSK
TruthfulQA
TriviaQA
CNN/DM
Avg.
MMLU
Pre-trained model
1.1B (Base)
18.29
2,04
27.41
9.6
7,21
11.56
25.02
7B (Base)
29. 16
3.80
30.96
36.7
8.81
20.06
38.26
13B (Base)
37.93
6.89
46.13
36.5
8.94
24.61
51.25
Multi task with nuning 13B (Multi-Task Tuning)
45.68
39.03
44.39
62.79
16.95
40.78
50.58
Single task with taning
1.IB-Expert (GSM8K)
18.50
12.51
25.38
6.12
12.69
24.30
1.IB-Expert (TruthfulQA)
16.15
2.81
29.01
2.83
7.23
10.47
21.82
1.1B-Expert (TriviaQA)
21.72
3.26
26.25
33.66
8.03
17.80
25.63
1.1B-Expert (CNN/DM)
12.98
2.73
26.39
1.65
14.22
11.24
14.73
Multi-task transfer from 1.IB 1.18 Multi-Task Tuning)
23.90
14.40
25.76
35.05
14.26
22.37
25.42
Ours
32.59
18.65
36.33
18.84
9.38
20.80
44.38
Single task with taning
7B-Expert (GSMSK)
34.14
37.07
36.04
20.41
26.18
42.10
7B-Expert (TruthfuiQA)
33.23
8.26
60.02
0.17
11.02
19.86
46.61
BB-Expert (TriiiiQQA)
29.35
4.62
33.66
52.10
10.30
25.17
33.52
7B-Expert (CNN/DM)
22.29
4.39
34.57
0.19
15.21
13.59
30.98
Multi-tas Multi-task transfer from 7B
7B (Multi-Tan Task Tuning)
34.89
34.72
33.28
51.54
16.30
33.96
35.82
Ours
39.42
34.87
42.25
22.48
10.52
27.53
51.31
TurhfulQA. For full fine-tuning.
the batch size 128, learning rate 2e-5, optimizer Adam. For LoRA tuning, we set the rank to 64, learning rate to le-4, optimizer to Adam. We train for epochs. For multi-task tuning, perform full fine-tuning using mixed-seed training dataset.
During inference, we use greed decoding and set batch size to 256, top_p to 1.0 and temperature to 0.05. To accelerate inference, we use VLLM* synchronizing the signals of logits during logit arithmetic to achieve efficient inference. All experiments are performed on H100 GPUs.
httts://githbb.cm/////rrrroooo

---

1.8
GSM8K
Lower Bound
1.6
Up Bound
1.4
Performance
80
1.0
30
0.8
20
20
100
10
Decoding Step
GSM6K
TruthfuaAA
TriviaA
CNNDDM
MMLU
Figure The variation of a in knowledge trans- fer for the GSM8K expert. Lower Bound and Upper Bound represent the minimum and max- imum values obtained during the optimization process. In the
Figure 2: Compare the pre-defined a with the dynamic for different tasks.
4.3 Baselines
To demonstrate the effectiveness of our method efficiently transferring knowledge from a small model to a large model, we compare it against small model fine-tuned in single domain and a large model without fine-tuning. We also report the performance of large model fine-tuned on a single domain or fine-tuned using LoRA to demonstrate the feasibility of our method as replacement for fine-tuning approaches. Additionally, we compare our method with Proxy Tuning (setting a 1.0 as in the original work) to highlight the superiority of our method in the transfer process. We use the LLaM22-hha model as the expert for TruthfuiQA (therefore, there is no corresponding LoRA model). For other experts, we use models fine-tuned on the respective training sets. In multi-task scenarios, we follow the common training practice of mixing the training data, with the results serving as our multi-task bascline. More details can be found in the Appendix G.
4.4 Performance on Single-Tass Task Scenarios
As shown in Table our method improves upon the original experts when transferring knowledge from the 1.1B and 7B models to the 13B model. Across all tasks, our method transfers knowledge from 1.1B and 7B experts to the 13B model, achieving performance improvements of 37.6% and 4.4% over the original experts, respectively. This demonstrates that our method can leverage the existing knowledge in larger models to enhance the knowledge learned by the expert small models. Notably, observe significant improvements alone, indicating that our method effectively transfers expert knowledge while leveraging the inherent capabilities of the large model, such as stronger reasoning abilities and a richer knowledge base.
GSM8K and TriviaQA compared to the experts
Our method, which transfers knowledge from 1.1B and 7B experts, can close the performance gap by 72.9% and 96.4%, respectively, compared to 13B Full Fine- Tuniig,nin,, achieving improvements of 6.5% and 3.8% over Proxy Tuning. Compared to 13B Full LoRA Tuning, our method can close the gap by 71.7% and 90.7%, respectively, when transferring knowledge from 1. and 7B experts. Additionally, is worth noting that our method outperforms the 13B Full Fine-Tuning results on TriviaQA. Since our approach only requires fine-tuning smaller model, it demands less computational and memory resources compared to fine-tuning a large model, making our method highly promising.
4.5 Performance on Multi- Tasti Scenarios
In the multi-task scenario, we categorize the multi-domain MMLU task as an unseen task and the other four tasks as seen tasks. We then calculate the average performance on seen and unseen tasks to evaluate the overall generalization capability of our model. As shown in Table 2, our method achieves a 36.35% and 13.37% improvement over directly fine-tuning on 1. and 7B, respectively, using multi-domain fine-tuning. Furthermore, our results show that transferring knowledge from 7B outperforms 13B overall by 3.93%. Specifically, the performance improvement is 11.9% for seen tasks and 0.1 % for unseen tasks. This indicates that our approach can alleviate confficts between multiple
7

---

GSMSK
100
Table 3: The time required to train or inference 1,000 data points on single GPU. In the in- ference section, values in parentheses show the factor by which the inference speed is slower compared to the 13B FFT model. "FFT" de- notes Full Fine-Tuning, and "L.T" denotes LoRA Tuning. Our 1.1B/7B expert model use full fine- tuning.
0.
0.0
TrutsfuQQA
1.0
0.5
0.0
TrivisQA
1.0
0.5
0.0
1.0
CNNDDM
Model
Training
Inference
13B FFT 13B LTT
1176s
60s 60s
0.0
836s
Docoding Step
Proxy Tuning (from I.IB)
128s
142s (x2.36)
Ours (from I.IB)
128s
150% (x2.5)
Figure The variation of a for the four ex- perts during knowledge transfer an unseen task (MMLU: abstract algebra).
Proxy Tuning (from 7B)
588s
158s (x2.63)
Ours (from 7B)
588s
166s (x2.76)
tasks to certain extent and improve out-of-domain generalization by leveraging the capabilities of the large model itself.
In multi-task settings, our method achieves performance improvement of 71.3% and 86.3% over 13B multi-task tuning when transferring knowledge from 1.IB and 7B, respectively. can be observed that our method performs worse in domain-specific tasks compared multi-task tuning, e.g., TriviaA.. This is primarily because we cannot access the data needed by the experts, resulting in bottleneck in handling task conflicts. In the future, we will explore more effective ways to effortlessly transfer multiple small expert models to a large model. Notably, Proxy Tuning is designed for single-domain scenarios with known test distribution. It struggles test time with inputs from unknown distributions, making it difficult to use in multi-task scenarios. However, our method achieves the best results on unseen tasks, demonstrating its effectiveness in enhancing the generalization ability of large models on new tasks. Moreover, our method can effortlessly transfer knowledge from pretrained experts across multiple domains to the large model without requiring access to their training data.
5
Analysis
5.1
How a control the knowledge transfer from the small models?
We aim to understand how dynamically changes in single-task and multi-task scenarios, and what patterns it exhibits across different tasks. As shown in Figure 2, we compare our method with predefined a values of 0.5, 1.0, and 1.5. It can be observed that our method consistently outperforms the predefined a settings, demonstrating the effectiveness of adaptive adjustment at each decoding step. Additionally, the predefined method exhibits significant variation across different tasks, whereas our method addresses this issue and avoids the extensive effort required for parameter tuning.
As shown in Figures 3 and 4, we illustrate the variation of a for single expert and multiple experts, respectively. In Figure 3, it can be observed that a fluctuates between the lower bound and upper bound. At the lower bound, less expert knowledge is transferred, and it is more important to retain the large model's inherent capabilities. Conversely, at the upper bound, more expert knowledge is transferred, requiring more GSMsk-related capabilities for those steps. In Figure 4, when solving the "abstract algebra" problem in MMLU using four experts, it is evident that GSM8K transfers more knowledge overall, especially in the initial steps. Since "abstract algebra" is mathematically oriented problem, these results demonstrate that our method can effectively select and transfer expert knowledge during the decoding steps.
5.2 Efficiency Analysis
In batch inference with size of B, Proxy Tuning requires a complexity of approximately O(BV) for each decoding step. In contrast, our method requires O(nBV) complexity, where n (S 20) the number of parameter searches. Since < V, our complexity is comparable to Proxy Tuning.
8

---

In-Context Learning Enhaneed by Ours
Task Arithmetic Enhanced by Ours
60
50
55
Ours
Ours
ICL++urs
40
TA-Ours
50
30
45
40
35
10-
30
Single Domain
Multi Domain Preformance
All
Seen Task
Unseen Task Preformance
Avg.
(a) Enhance In Co-tontex Leariing (ICL) with methad from transferring 7B expert knowledge.
(b) Enhance Task Arithmetic (TA) with method from transferring 7B expert knowledge.
Figure 5: Enhance in-context learning and task arithmetic using our method.
In multi-task settings, performing parameter searches for each expert and then merging them would result in exponential growth, with complexity of O BV). To optimize this process, avoid merging parameters between experts at each step and instead select single expert for knowledge transfer. This reduces the algorithm complexity to (n BV). Additionally, we can constrain the parameter search range for each expert to achieve better efficiency and performance.
Here, complete the training and inference time taken. As shown in Table 3, our method achieves similar efficiency to Proxy Tuning during inference. Due the need for inference and communication between multiple models, our approach is approximately 2.5 times slower than direct inference with a 13B model. However, our method only requires training small model, which provides significant advantage in training efficiency.
H100 GPU for 1000 data, while recording the
5.3 Comparison the Weak-to-strong Approach with Other Methods
Comparison against In-Context Learning Enhancing in-context learning with logits arithmetic improve large language models. Since both methods work at the token level and need only black-box access, they can be combined to boost model performance. We categorize tasks as either Single- Bigle-Domminn (excluding MMLU) or Multi-Domain (MMLU). The "All" category averages the results of both. Notably, we use 5-shot information as in-context examples, and we apply different few-shot samples and prompts for various MMLU tasks. In our experiments, we enhance 13B model's 's performance using 7B expert and compare it to the 13B model using in-context learning. We present the results in Figure 5a. We observed that our method outperforms 5-shot in-context learning on both Single-Domain and Multi-Domain tasks. Furthermore, when combined with in- context learning, our method shows significant improvement on Multi-Domain tasks, resulting in an overall (All) increase of 18.3%. Specifically, combining our method with in-context learning does significantly impact performance on Single-Domain tasks, indicating that our method alone has already achieved capabilities similar to in-context learning for individual tasks.
Comparison against Task Arithmetic Task Arithmetic and logits arithmetic are similar in principle. both adjusting the shift between the expert and base model to control knowledge transfer. However, logits arithmetic isn't constrained by parameter size and can merge across model levels without needing access to specific parameters. Specifically, our method can be applied to combine experts from task arithmetic, integrating both approaches for multi-task learning. We use the same setup as our multi- Mallti-taskaaa scenarios, treating "All" as the average of Seen and Unseen Tasks. In figure 5b, our method performs well on Unseen Tasks, resulting in an overall improvement 5.7% compared to task arithmetic. When combine our method with task arithmetic,, we see improvements over task arithmetic alone, achieving the highest overall performance, with increases of 7.2% and 1.5% over task arithmetic and our method alone, respectively.

---

Conclusions
In this paper, we introduce a dynamic logit fusion approach in weak-to-strong specialization, which utilizes series of task-specific small models and allows for adaptive weight allocation among them. Through extensive experiments, we have demonstrated the effectiveness of our approach in both single-task and multi-task settings across various benchmarks. By transferring expertise from the 7B model to the 13B model, we have achieved significant performance improvements. closing the performance gap by 96.4% in single-task scenarios and by 86.3% in multi-task scenarios compared to full fine-tuning of the larger model. Our method also shows promising results on unseen tasks and can integrate in-context learning for single tasks and task arithmetic for multi-task scenarios.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant No. 622761 I0, No. 62172039 and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL). We thank the Shanghai AI Laboratory for supporting GPU resources. The authors would also like to thank the anonymous reviewers for their comments on improving the quality of this paper.
References
[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference Learning Representations, 2023.
[2] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2024.
[3] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efffcientt fine- tuning for cansformer-basee masked language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1220653//122222alllsshhh https://aclanthoolgge org/ 2022. www.sorrcl-shorr.....
URL
[4] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10925-10934,
2022.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Berbert-Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran As- sociates, Inc., 2020. URL https://proceed: File//447ccd6bfcc49977 41 11e44418abbbcccc 42f644-2ae6erra...
neur heurips... ips. wwtpcpprppppp..eceeecce lnmeer_filsse www.s//pppeee////0/0
[6] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker. Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, 1lya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023.
[7] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetunee language models. Journal of Machine Learning Research, 25(70) 2024.
10

---

[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171 141166, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19 1423. URL / /ttp//aaaaaantoo......... org/N19- 1423.
[10] Chenghao Fan, Wei Wei, Xiaoye Qu, Zhenyi Lu, Wenfeng Xie, Yu Cheng, and Dangyang Chen. Enhancing low-resource relation representations through multi-view decoupling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17688111 17976, Mar. 2024. doi: 10. 1609/ aaai. 2....222..... URL htw.h: //ojs. aaai org org/index..o http:////////aaarrrr////////////
[11] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseck-coder: When the large language model meets programming- - the rise of code intelligence, 2024.
[12] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint 2rXiv:2402.12354 2024.
[13] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. International Conference Learning Representations, 2020.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02553 2015.
[15] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790 2799. PMLR, 2019.
[16] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[17] Gabriel Ilharco, Mareo Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevieee wtw.ee/nnntoooffiiiieeee 6t0Kwf8-jr)
[18] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to weak-to-strong correction, 2024.
[19] Feihu Jin, Jiajun Zhang, and Chengqing Zong. Parameter-efficient tuning for large language model without calculating its gradients. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 321-330 Singapore. December 2023. Association for Computational Linguistics. doi: 10. http://w/222..mmppppmmmm
00g. http:/222...mmpmmmmmemmmmmmmmm smnlp-main.2 URL htttps://aclanthollottooo
[20] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro. and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2022.
[21] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1601-1611. 2017. 2017.

---

[22]
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. ceaumeter-fffcieen multi-task fine-tuning for transformers via shared hypernttorks. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 565-576, Online, August 2021, Association for Computational Linguistics. doi: 10. 10.11866/11/22211 www.////o......oonooo... acl--ong, URL /
org/2021 acl acl-long.44 47.
[23] Dawid Jan Kopiezko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL
[24] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
[25] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pages 19274-19286. PMLR, 2023.
[26] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 5582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021. acl-long.353. URL ht https: / ht///claantttt aclaatholoogyyyyy orgorrgg////o /22///221122222 wtt.:22211cccc.. 20121.acl-lom acl-long. 353.
[27] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki, editors, Proceedings of the 6/st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 22286-12322 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. (6653//1/003.acllloorr acl acl-long.68 687.
URL https: httpss////alaaahoooogggo wwwooog.oorrgooooo 2023.
[28] Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide parameter-efficient fine-tuning, 2023.
[29] Petro Liashchynskyi and Pavlo Liashchynskyi. Grid search, random search, genetic algorithm: big comparison for nas. arXiv preprint arXiv:1912.06059) 2019.
[30] Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost in pruning: The effects of pruning neural networks beyond test accuracy. Proceedings of Machine Learning and Systems, 139,-138, 2021.
[31] Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base 1lms: Rethinking alignment via in-context learning, 2023.
[32] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Compula- tional Linguistics (Volume 1: Long Papers), pages 3214-3252, 2022.
[33] Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. arXiv preprint vrXivv2401.08566 2024.
[34] Jialin Liu, Antoine Moreau, Mike Preuss, Jeremy Rapin, Baptiste Roziere, Fabien Teytaud, and Olivier Teytaud. Versatile black-box optimization. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pages 620-628, 2020.
[35] Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise model merging. arXiv preprint arXiv:2406. 15479, 2024.
12

---

[36] Zhenyi Lu, Jie Tian, Wei Wei, Xiaoye Qu Yu Cheng, Dangyang Chen, et al. Mitigating boundary ambiguity and inherent bias for text classification in the era of large language models. arXiv preprini arXiv:2406.07001, 2024.
[37] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023.
[38] Michael S Matena and Colin Raffel. Merging models with fisher-weighter averaging. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL hwwp::ttpp:/ote htttttss//oopeeeew.eeeeeeeeeeeee LSKlp_
[39] Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 2022.
[40] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree- based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architecturul Support for Programming Languages and Operating Systems, Volume 3, ASPLOS '24, ACM, April 2024. doi: 10.1 10.1145/3320666.3551111 URL. http: //t:://dxxx /dx. www.dooooooooooooooooooooooooo org/ 10. 1145/3620666.1 335.
[41] Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. An emulator for fine-tuning large language models using small language models. In The Twelfth International Conference Learning Representations, 2024. URL fttps://openreviee net/forum?id-Eo7kkBe..
[42] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. 2021.
[43] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secredy reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Cur- Advances in Neural Information Processing Systems, volume 36, pages saterrrffleeseeeeeeeeeeee heles//aper//// ran Associates, Inc., 2023. URL httpps://proceeedingss neurips. cc/ cc/ paper_ 2023/file (tw.e//0000000577777764062666660
www.rappeeeenneeeneeeeeeeeeeeeee pdd.
[44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL 00011220110 000000000000 html.
[45] Baptiste Rozière, Jonas Gehring. Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024.
[46] Timo Schick, Jane Dwivedi-Yuu Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer; Language models can teach themselves use tools. Advances in Neural Information Processing Systems, 36. 2024.
[47] Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18633/v 1099. URL /
13

---

[48] Shannon Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. Learning to decode collaboratively with multiple language models, 2024.
[49] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances Neural Information Processing Systems, 34:6906-6911 2021.
[50] Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, al. Living in the moment: Can large language models grasp co-temporal reasoning? arXiv preprint arXiv:2406.00077. 2024.
[51] Zhaochen Su, Jun Zhang, Xiaoye Qu. Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in Ilm. arXiv preprint arXiv:2408. 12076, 2024.
[52] Zhaochen Su, Jun Zhang, Tong Zhu, Xiaoye Qu, Juntao Li, Min Zhang, and Yu Cheng. Timo: Towards better temporal reasoning for language models. arXiv preprint 1xXv:24406.1192, 2024.
[53] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023.
[54] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini. Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[55] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashtykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[56] Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric Xing. and Mikhail Yurochkin. Fusing models with complementary expertise. In The Twelfth International Conference on Learning Representattons, 2024. URL https: /tppnr:////////e/e/eee www.//rrurr/eeee httpt//oruummiPPMMrCMiiii dPPhMrGCMIRL.
[57] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, 2022.
[58] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In Thirty-seventt Conference on Neural Informa- lion Processing Systems, 2023. URL htwps:tp/////ot http:////pppevveieeeee eview. net
[59] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, 2024. URL /tttps://operreeee net/forumiid= nZP6NgD3QY.
[60] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02388 2024.
14

---

[61] Tong Zhu, Daize Dong, Xiaoye Qu, Jiacheng Ruan, Wenliang Chen, and Yu Cheng, Dy- namic data mixing maximizes instruction tuning for caature-of-expertt arXiv preprint arXiv:2406. / /256, 2024.
[62] Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: Building mixture-of-experts from llama with continual pre-traine pre-training. arXiv preprint 2at1::2006.65544 2024.
15

---

A Model Architecture Diagram
As illustrated in Figure 6, approach dynamically calculates the fusion weights of the logits of SLMs at each decoding step, transferring the knowledge from multiple experts to LLM.
Kth Decoding Step
M
M
M
M
ax <MM M (M M + as(M M2
M
Adaptive Knowledge Transfer Optimization
Decoding
21
Z
P(ZA|01A-11 = softmar(M (1x))))))1)
Figure 6: The architecture of our method. The small Ilama represents the small model, while the large Ilama represents the large model. M /M denotes the logits the small/large language model. M represents the logits of a small expert language model for task t. The lower part of the figure illustrates our optimization in the decoding process, where each circle represents decoding step. The upper part of the figure shows how our method transfers the knowledge of experts in the kth step. At each decoding step, our method dynamically adjusts the {at) value for each expert. transferring knowledge from the small models to the larger model.
Proof of Logic Arithmetic
Reinforcement Learning from Human Feedback (RLHF) is commonly employed in optimizing language models, where learning process involves maximizing a reward objective while penalizing the KL divergence between the learned policy and reference policy with coefficient 3:
max B.w.D.yyrroolaallrraa. w)] - DKL. [TT | [x)] x)]
16

---

Following the DPO [43] framework, we can reformulate the above objective as:
Λ ( 2x) Tref (y (x)
max R E (vla))
[ra(x,,)) v)]
BEN Be (olo
8 log
(2 Blg ( x) (y xz)
max EgwD
Boy w (v)or)
log T(y x))
min B y
B (
Tref y (x)
log z(y x))
min Eg (ylz))
log exp
3 8T0(Z1 (x, v)
(y xx)
(8)
log TtV x))
min ExnD (ylzz)
Tref ())ex exp (3ro(1,)))
Tt x))
min E (vll))
log
Trel(y) x)) exp ( (z 2(1e) ((x)
( x))
min Een D (ynx())k)
log
log ((x)
Z(w) Fret (3) x) exp
(2 (2.
v) is the partition function. Note that the partition function is a function of only a and the reference policy Trets but does not depend on the policy mo- We can now define
where Z(x) Eg M x)) exp (2)
Teerl x)) exp (grolers))
( xx)
(9)
(xx)
Then the equation convert to
log <(ylx:) x (z]æz)
min EoND Eywa(via))
log Z(x)
(10)
min EynD (TL(a( x (v(l=)) log 2(x)]
Hence have the optimal solution:
7(x) nnd (yr(y exp (fro(os.))
क (ylx)) = nr" (y]ar)
(11)
for all x E DD
According to Rafailov et al. [43], reward models can be reparameterized as: r(z, y) = B log =(vlx) for some model T [ty w) and a given reference model Tee((y) .². Additionally, given pretrained language model and its fine-tuned counterpart Ta. the following relationship holds,
Th (y x)) T x))
Ta æ) = T x) exp(1og
T x) exp( (xxp))((1.8))
(12)
Implicit reward
This indicates that the fine-tuned model hn can be interpreted as the solution to constrained rein- forcement learning problem with constraint on the pre-trained model. Consequently, the theoretical framework is applicable to any fine-tuned model, providing an RL-based interpretation of fine-tuning.
We can further introduce S-size implicit reward for L-size finetuned models:
17

---

((x) x x)) exp 3 ( (x.
n T
x xt (y) x) (log
(13)
T
xx)
x xx (
By taking the logarithms, we derive the following logit arithmetic formula:
(x) (² (x) (x))
(14)
where M M and Mt, represent the logits of the large model, small model, and fine-tuned small model, respectively. This completes the derivation of the Equation (2).
C Proof of Method in Section 3.1
Based on the definitions of the probability distributions Q, Q/ P. and P, the model logits outputs M S fo and M satisfy the following relationships:
Q(Eu [<n) = softmax M (zn))<=))
Q5e(In x<c) voftmax(MMj (1
P(ak xec) = softmax (M (a [r()))
(15)
P(æx|(<e) = softmax(M (Ix (<x) + (Mfl(tn(Ze))) M
( (xx(x<)))
Suppose there vectors x and y of dimension n. The relationship between them is as follows:
e
y = softmar(z) プリ=1 e
(16)
Dj=1" e
For each value Iú, increases, 23 x; decreases, and - increases. Therefore, the corre- sponding ys is positively correlated with z;,
So we can know that in y x y oc æ. Therefore, based on Equation (15), we can derive the following relationship:
In Q(# IEx) & M (Za)T<)) In Q xeR) (xR((E))))) In P(** Izc)) x M (0I(T<R)
In P(x (xc) & (M4 (x)) + a (Mf,(xx/)e))) (x(xxx)2)))
(17)
x In P(ak x<c)) + a (In (I x<x) In Qn xzx))
Qu((xx xzch)
x in P(xx xek))
Q( x<c)
Thus, logit arithmetic can be viewed product-of-expert. ensemble as follows:
Qee(TK mek) Q(Ix xee))
P(** (eck) x P(zk Iec)
(18)
Essentially, assumes a predefined distribution shift @ ya between the fine-tuned and Q(mx|=<e) pretrained distributions of small model, which can construct the trajectory of P. It then attempts to approximate the shift for the fine-tuned and pretrained distributions of the large model P(zeeee
(mca)
18

---

D Efficiency Analysis of the Forward Process
In section 5.2, conducted Efficiency Analysis of logit Arithmetic. To better illustrate efficiency, we further analyze the overall efficiency of our method here.
Overall, during single forward pass, our method has similar time complexity to the static method. Given: current sequence length 8, large model dimension hr, small model dimension hs, number of layers in the large model LI, number of layers in the small model L2, batch size B, vocabulary size V. number of searches per decoding step n. Assume the FLOPs for a single forward pass of the large model and the small model are FLOPst and FLOPsg, respectively. The FLOPs can be calculated as: FLOPs = L1 (12Bsh + 2Bs h1) + -Bhh V ,ad F FLOPss = L2 (12Bsh' + 2Bs² hg) + B (here we ignore the kv cache). Therefore, the FLOPs for single forward pass of our method on a single task is: FLOPs, + FLOPss + nBV. Among these, only the nBV term (n 20) corresponds the additional computational cost of our method, which is much smaller compared to the previous term and can be considered negligible in the overall time. Additionally, in our efficiency analysis, as shown in Table our method is only 0.008 seconds slower per sample compared to the static method, which is negligible.
E Proof for the Fusion of Multiple SLMs Scenario
This section mainly explains how we extend the transfer problem to multiple small models. When transferring the knowledge of multiple expert SLMs to LLM,, we consider the following two aspects: 1. The fusion of knowledge from different domain experts. 2. The transfer of knowledge from SLM to LLM, i.e., the transfer of knowledge from a single expert, which was discussed in Section 3.2. Intuitively, we first focus the fusion of different domain experts' knowledge before performing the transfer. Here, we define the distribution of the combined knowledge of these small models as Therefore, we aim to achieve D (PI P) = DxL (Q]I).
Since solving for / is difficult, we propose constraining it based on the relationship between J and {Q.) approximate it. Here, we can transform Dx (QIlJ) into D (Q]IQ.) + (Q,) is the bias function from Q: to J. When we approximate J as the centroid of {Q:} on the KL-constrained plane, we can implicitly solve these bias functions. According the definition of the centroid, J can be solved by minimizing the sum of the squared distances to each point, as shown below:
(Q.),, where
T arg min
Σ (DKz(Q J) DxL (Q ll Q1))² 1=1
(19)
Since our goal is (PL(P) || P) DDk((QQ)))) substituting this into our equation gives us our final optimization objective:
T
arg min Σ (DLL(PI1 P) -xk (Q, || Q))²
(20)
To prove the reasonableness of our approximation, we provide a more rigorous proof below. Our initial objective is as follows:
T Σ [DDKL P) (J]()))
arg min
(21)
im1
By assuming (KI(Q = DRL (Q,) + C)())). we can transform the original problem
arg min DRL P P) ()z()))))²
(22)
P
into constrained optimization problems:
arg min(Dxx(P) I| P) - DxL (Q; || Q) -)((1))²
(23)
arg mim (DLL P) DKL -c,())
19

---

end for After jointly optimizing them, we have:
arg min
p
T
Σ (0
-=1
(24)
(DLL P) Dx) Q))² + E C)(i)))²
(DKL((P)))- (L(Q) Q)) + CJ-Q
Since CJ-Q is constant term independent of P, we can ignore it. Finally, we solve the original problem by optimizing this upper bound. When symmetrize the terms in the KL divergence, we can obtain a similar conclusion. Therefore, in the multi-task setting, we can solve using the following formula: (As shown in Equation (6)):
arg min
[KL(P] P) De (Qr Q)) + (KL(P) P) - (I
(25)
Pseudo Code
Algorithm Adaptive Lritts-Arithmetic
Require:
generation prompt. X, number of tokens to generate N, Large model 04, domain number T, expert small models (0.)/-1 M M and M, represent the logits outputs of the large model, small model, and (-th domain-specific small models. P. Q. and Q represent the outputs distribution of the large model, small model, and 1-th domain-specific small models.
I: & len(X)
2: + D0
3: + |
4: while (en(X) < and xt-1 * [EOS) do
5: for each domain i in do
Get domain expert logits for æa from of as Mf (Iul(TeN))
7:
8: For multitask scenario, perform search only for one task, for total of I' times. such [0.0,2.0], step is 0.1
for search a' R do
10:
P(xx (xex) t softmax M (xo((<e)) EE of (M (xulx<<)) M (xa(
6 ET, (KI(P)))) kI.(2-11))* (KI(PII))KKI(QQQQ)) (I(()1Q)))
12:
if C < m then
13:
a' m C
14:
16:
end for
17:
Calculate next token distribution for the large model
P
softmax M (mex) ΣT as (M M (xu (xw [x)))))
18:
Sample the next token æx P(xe(mee)
X x- (X;
20:
kc
21:
end while
22: return generated text X
20

---

Our algorithm is outlined in pseudo-code in Algorithm 1, we search for each task's a with step of 0.1.
G Dataset Details
GSM8K [8], dataset of 8.5K high- linguistically diverse grade school math word problems created by human problem writers. We evaluate using exact match (EM.).
TruthfulQA [32]. benchmark to measure whether language model is truthful in gen- erating answers to questions. There is no training set for this dataset. We evaluate using multiple-choice accuracy (Acc.).
TriviaQA [21], reading comprehension dataset containing 650K question-answer-
evidence triples. We evaluate using exact match (EM.).
MMLU [13]. a benchmark designed to evaluate the capabilities of language models. comprises approximately 16,000 multiple-choice questions across 57 tasks. We evaluate using multiple-choice accuracy (Acc.).
CNNlDailyMail (CNN/DM) [47], a dataset for text summarization. We evaluate using Rouge2.
The GSM8K, MMLU datasets are licensed under the MIT License. The TruthfulQA, TriviaQA, CNN-Dailymail datasets are licensed under the Apache-2.0 License. Thus, these datasets are available for non-commercial research purposes.
H
Case study
As shown in Table 4, our method achieves the correct output 18 when targeting a single task, such the GSM8K problem. Additionally, it can be observed that the model's output contains formatted information, e.g., "####" which matches the format learned in-context learning. This indicates that method achieves the ability of in-context learning the logit level.
For unseen tasks like MMLU's Abstract Algebra and Clinical Knowledge, our model generates correct answers by leveraging multi-expert knowledge transfer.
I Limitations and Future Works
Our approach shares the general limitations of existing logit-arithmetic-buee methods: (1) Since our method operates at the decoding step. this architecture mainly tests its ability in generative models, without exploring its effectiveness discriminative models; (2) Involving multiple models in the decoding step slow down inference compared inference with models of similar magnitude.
In the future, we will explore more efficient methods of logit arithmetic and investigate their potential application in other architectural models.
Broader Impacts
Our approach enables weak-to-strong knowledge transfer without accessing the model's 's parameters or expert-learning data. This is advantageous for protecting the privacy of expert data. Additionally, this data-free fine-tuning paradigm significantly reduces the cost of private deployment while enhancing data security, thus positively impacting societal data security. Our method does not exhibit any significant negative societal impacts.
21

---

Table 4: Case Study
GSM8K
Q: Janet201 9s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
LLAMA2-13B:
n
nAnswer:
LLAMA2-13B + In-Context Learning (5-shot): Janet cats eggs for breakfast every morning. She bakes muffins for her friends every day with eggs. She sells the remainder the farmers' market daily $2 per fresh duck egg.
So she makes $2 13 = 1<2213==6>26 every day at the farmers' market.
#### 26
Ours (LLAMA2-13B) GSM8K expert): Janet eats 3 eggs for breakfast and bakes eggs into muffins for a total of 3 + 4 4= <3+4=7>7 eggs per day. She has 16 = =16-7=9>9 eggs left to sell at the farmers' market. She sells each egg for $2, she makes x $2 $=9*2=188 per day the farmers' market.
#U## 18
MMLU: Abstract Algebra
Q: Statement Every group of order 42 has normal subgroup of order 7. Statement 2 Every group of order 42 has a normal subgroup of order 8.
A. True, True
B. False, False
C. True, False
D. False, True ?
11AMA2-13BB D. False, True
Question: Statement Every group order 42 has a normal subgroup of order Statement 2 Every group of order
Ours LLAMA2-13B- GSM8K expert + TruthfulQA expert + TriviaQA expert + CNN DM expert: C. True, False The reason is that the statement is true for the group of order 42, but not for the group of order
MMLU: Clinical Knowledge
Q: Wound drainage is described as serous if:
A. it contains red blood cells.
B. it is thick and yellow or green.
C. it contains microorganisms
D. it is clear or slightly yellow.
1LAMA2-13BB D
Question: Which of the following is not a characteristic of a normal blood film?
LLAMA2-13B + In-Context Learning (5-shot): D. it is clear or slightly yellow.
Ours (LLAMA2-13B + GSMSK expert TruthhuuQQ expert TriviaQA expert CNN DM expert):
D. it is clear or slightly yellow.
Table 5: Case Study(token routing)
GSM8K & TruthfulQA & Common
Q: A microwave oven is connected to an outlet, 120 V, and draws a current of 2 amps. At what rate energy being used by the microwave oven?
LLAMA2-7B: The rate which energy is being used by the microwave oven can be caleulated by multiplying the voltage (120 V) by the current (2 amps). This gives us power rating of 120 V x 2 A = 240 watts. An##### 240
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Appendix I, provide detailed report on our limitations and future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification: Appendix C, we provide the full set of assumptions and complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Appendix provides the pseudo-code of our method and Section 4.2 provides the training and test details of each method we experiment with.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our paper uses publicly available datasets and provides the complete code and execution scripts in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines ptatts::/niis..ee
(eolicc/giids/Coeee
ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips. cttpsccccppblp// ctuublic//ae /// desuidds/Caaa /eoeee/codeebiii sotibbmissiinn sswonPolicccccccecccc Piyy))lyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4.2 provides the training and test details of each method we experiment with.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not include error bars for the experiments. This omission is due to the high computational cost associated with calculating error bars for large language models. Additionally, the training-free algorithms used in the experiments are deterministic, ensuring reproducibility and consistency of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides computer resources and information needed to reproduce experiments in Section 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:rpp...puuiitthtccuuueeeeeee inne??
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms in every respect with the
NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We analyze both the positive and negative societal impacts the Appendix J.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Appendix G, all the creators original owners of assets used in the paper are properly cited and and provides licensing information for datasets used in our work.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We submit the code in the supplementary material with detail scripts and guidance.
Justification:
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "TRYh9jNR5j", "reviewer": "Reviewer_XMhg", "summary": "This paper focuses on the weak-to-strong generalization paradigm where the goal is to transfer knowledge from a small language model to larger one. The method they study is the one proposed by Mitchell et al. [1]: they use log probability algebra to combine the logits of the large model, the ones of a small model and the ones of a small model that has been finetuned. This combination involves a parameter $\\alpha$ that controls the contribution of the small model. The main contributions of this paper is to point to the limitations of using a static $\\alpha$ and to propose a method to adaptively learn such $alpha$. Their method consists in optimizing an objective based on the KL divergence and they show that their approach is consistently better than using a static alpha across a wide range of downstream tasks.\n\n\n\n\n[1] Mitchell, Eric, et al. \"An emulator for fine-tuning large language models using small language models.\" arXiv preprint arXiv:2310.12962 (2023).", "questions": "I would appreciate that the authors address the concerns I have regarding the theoretical justification of their optimization problem, the computational cost of their approach.", "limitations": "I think that the authors didn't clearly state the limitiations of their approach, which is regretful.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I find the paper is well written and the methodology well presented. The authors did a great job at presenting the problem, the limitations of the current methods to solve the problem and their method. They also did a good job at presenting their experiments.", "weaknesses": "Overall, my main concern is that I find the contribution limited and I have some doubts about the method. Here is a detailed list of my concerns:\n\n- **Computational feasability**: I think that the authors should be more transparent in the computational cost of their procedure. Solving the optimizaiton problem at every decoding step may be very expensive and it is not clear to me that when one needs to do many generations, a finetuned model with LoRA on the large model is cheaper than the procedure proposed by the authors. Also, is it important to update alpha at each decoding step? Can't one get a more efficient procedure by updating it only every 100 tokens or so? \n\n- **Not enough clear to me that the method does much better than the static $\\alpha$**: when I see the barplot of figure 2, it seems like $\\alpha=1$ is a bit below than the learnt $\\alpha$ but the gap is not huge. \n\n- **Theoretical justification for the optimization problem?**: So if I understand correctly, the authors objective function is to say \"I want the distance between the predictions of $\\tilde{P}$ and the large model is the same as the distance between the predictions of the finetuned small model and the small model\".   This looks like a reasonable belief. However, it would have been nice to have a theoretical justification. For instance, when you do RL finetuning, if you solve the problem exactly, the distribution you end up generating from ends up being p_bayes(generation) * exp(reward model you are training on) (usually done with ppo, dpo). When you take the logs, you get log(p) = log(p_bayes) + reward model.  Then you can estimate the reward model by taking the difference of logits for any model scale and in this case the intuition of the authors make sense to me and it is principled. However, in the standard finetuning case, when the authors apply this intuition at the level of tokens, it is not clear to me why it should work.\n\n- **Scaling experiments for studying the approach**: I know that the Llama suite starts at 7B but it would have been nice to study the behavior of the method with smaller models than 7b.  Understanding how robust the method is by varying the gap between the weak and strong models is important. There are maybe chances that the learnt $\\alpha$ approach shows bigger  gaps with respect to the static $\\alpha$ when the gap between the weak and strong models is large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:Solving the optimization problem at every decoding step is expensive?\n\nAs we analyzed in the \"Complementary to Efficiency Analysis\" section of the Global Rebuttal, our method only adds the term \"$nBV$\" compared to the static method. Optimizing $n$ times ($n \\le 20$)  during each forward pass is negligible compared to the overall forward pass time. In the experiments, as shown in Table 3, our method is only 0.008s slower per data point on average compared to the static method.\n\n  \n> Q2: A finetuned model with LoRA on the large model is cheaper?\n\nSFT with LoRA still requires forwarding the full large model thousands\n  of times (e.g., 13B) to tune LoRA during training. **In contrast, our\n  method does not require fine-tuning the large model at all, benefiting\n  from transferring from smaller models (7B or 1.1B) or using\n  pre-existing models (e.g., from Hugging Face).** As shown in Table 3, the time for LoRA tuning a 13B model is 836 seconds, while the time for fully fine-tuning a 7B or 1.1B model is 588s or 128s, respectively, which is significantly less than the time required to fine-tune the 13B model.  LoRA tuning on the smaller models further reduces the time to 364s (7B) and 128s (1.1B). This demonstrates that fine-tuning smaller models requires less hardware resources and less time. Furthermore, utilizing pre-trained smaller models from Hugging Face allows efficient transfer without extensive training. Therefore, our method is cheaper compared to fine-tune a large model.\n\n> Q3:Updating $\\alpha$ every 100 tokens is more efficient?\n\nAs we analyzed in the \"Complementary to Efficiency Analysis\" section of the Global Rebuttal, our method and the static method have almost the same time consumption per data point on average. As shown in the below table, reducing the update frequency of $\\alpha$ may negatively impact the final result. Therefore, we ultimately chose to optimize $\\alpha$ at each step.\n\n|update step|1|100|$+\\infty$|\n|-|-|-|-|\n| GSM8K| 39.34(0.166s/per sample) | 37.84(0.159s/per sample) | 37.68(0.158s/per sample) |\n\n\n\n> Q4: Not enough clear to me that the method does much better than the static $\\alpha$: when I see the barplot of figure 2, it seems like $\\alpha=1$ is a bit below than the learnt but the gap is not huge.\n\nActually, our method outperforms the static setting of $\\alpha=1.0$ on single tasks, with improvements of 4.4%, 0.9%, 8.1%, 6.5%, and 1.6% on GSM8K, TruthfulQA, TriviaQA, CNN/DM, and MMLU, respectively. Our method has a significant advantage over the static method, with an average improvement of 4.3%.\n\nTo better demonstrate the effectiveness of our method, we further compared it with our method with static $\\alpha$ in a multi-task setting. For the 4 seen tasks in our experiment, we set the corresponding expert coefficient to 0.25 ($\\alpha=1/4$, assigning the same proportion to each seen task expert). In the table below, it can be seen that our method, which dynamically adjusts the coefficients, significantly outperforms the static setting.\n\n\n|  | Seen Task | Unseen Task | Avg. |\n| --- | --- | --- | --- |\n| Ours (0.25 static) | 22.02 | 46.04 | 34.03 |\n| Ours | **27.53** | **51.31** | **39.42** |\n\n\n> Q5: The theoretical justification for the optimization problem?: It's not clear to me why it should work in standard finetuning case?\n\nTo demonstrate our optimization process, we can view the fine-tuning procedure as reinforcement learning (RL) with a KL-divergence constraint preventing divergence from a reference model.\n\n*According to the theory presented in the DPO[1]: Theorem 1. Under mild assumptions, all reward classes consistent with the Plackett-Luce (and Bradley-Terry in particular) models can be represented with the reparameterization* $r(x, y) = \\beta \\log \\frac{\\pi(y|x)}{\\pi_{ref}(y|x)}$ *for some model* $\\pi(y|x)$ *and a given reference model* $\\pi_{ref}(y|x)$.\n\nMeanwhile, According to [1,2] the optimal solution to the KL-constrained reward maximization objective is given by:\n\n$$\n\\begin{align}\n    \\pi_r(y|x)=\\frac{1}{Z(x)}\\pi_{ref}(y|x)exp(\\frac{1}{\\beta}r(x,y))\\\\\\\\\n  where\\quad Z(x)=\\sum_y \\pi_{ref}(y|x)exp(\\frac{1}{\\beta}r(x,y))\n\\end{align}\n$$\n\n\n  Combining the above theory, we can derive the following equation:\n\n  $\\pi_{r}(y|x)=\\frac{1}{Z(x)}\\pi_{ref}(y|x)exp(\\log \\frac{\\pi_r(y|x)}{\\pi_{ref}(y|x)})$\n\n  Since any language model can be viewed as the solution to KL-constrained RL with a constraint to the pre-trained model[1], this equation is applicable to fine-tuning scenarios. We can replace $\\pi$ in the parentheses with the small model's $\\pi$, resulting in the following equation:\n\n$\\pi_{L-ft}(y|x)=\\frac{1}{Z(x)}\\pi_{L-pt}(y|x)exp(\\log \\frac{\\pi_{S-ft}(y|x)}{\\pi_{S-pt}(y|x)})$\n\n- [1] Direct preference optimization: Your language model is secretly a reward model (NIPS2023)\n- [2] RL with KL penalties is better viewed as Bayesian inference (EMNLP2022)\n\n\n> Q6 Scaling experiments for studying the approach\n\nActually, we have chosen TinyLlama1.1B in our experiments.\nIn the table below, we show the improvements of our model compared to the static method on 1.1B and 7B models in single-task experiments. The numbers in the table represent the results of the static method, with the relative improvements of our method shown in parentheses. It can be noted that when the gap between the weak model and the strong model is larger, our method indeed better adjusts the capability of expert knowledge transfer.\n\n|  | GSM8K | TruthfulQA | TriviaQA | CNN/DM | MMLU | Avg. |\n| --- | --- | --- | --- | --- | --- | --- |\n| from 1.1B | 16.91(8.0%$\\uparrow$) | 31.48(17.7%$\\uparrow$) | 48.74(10.4%$\\uparrow$) |13.23(9.4%$\\uparrow$) | 39.88(21.16%$\\uparrow$) | 31.74(9.8%$\\uparrow$)\n| from 7B | 37.68(4.4%$\\uparrow$) | 61.02(0.9%$\\uparrow$) | 52.81(8.1%$\\uparrow$) |14.37(6.5%$\\uparrow$) | 56.24(1.6%$\\uparrow$) | 44.43(3.7%$\\uparrow$) |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5DqnikaTov", "reviewer": "Reviewer_YRcj", "summary": "The paper studies the problem of adapting large general language models via smaller expert language models fine-tuned on specific tasks. Prior work proposed the idea of mixing logits between a large model and the differencei in logits pre- and post- finetuning of a small model. The authors take this idea a step further and compute the mixing weights adaptively per-token, leading to better results.", "questions": "1.1 and 1.2: See weaknesses 1 and 2.\n\n2. What exactly do you mean by supervised instruction tuning: what are the smaller models fine-tuned on? Are these chain-of-thoughts for solving the task, e.g. GSM8k? Where do they come from? \n\n3. In Figure 3, qualitatively, what do the tokens (decoding steps) where we set the weight to the lower bound correspond to, and same for the upper bound? Are they qualitatively different?\n\n4. In Figure 3, why is the lower bound 0.8 and the upper bound 1.5? Are these tunable parameters? 0.8 seems quite high, shouldn't we want to set the lower bound to 0?", "limitations": "Limitations are adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The method is very simple: the authors match tune the weights to match the KL divergence between the small model before and after fine-tuning, for each token.\n\n2. The experiments are comprehensive with 5 tasks and 2 small models (1.1B and 7B). The authors also consider both single-task and multi-task scenarios.\n\n3. The results are good across all tasks, the proposed method outperforms proxy-tuning as well as full fine-tuning on the smaller model predictions, and recovers a large fraction of the ceiling performance achieved by directly finetuning the large model on ground truth.\n\n4. There are several ablations and understanding experiments in Section 5.", "weaknesses": "1. It is not intuitively obvious to me why matching the KL divergence is the right objective. Could the authors please provide some intuition? I imagine it is something like this: when the small model updates significantly for some token, we want the large model to also udpate significantly. That seems reasonable, but probably doesn't always work well: if the small model is unaware of some fact or makes an arithmetic mistake, it may need to update significantly on the corresponding tokens, while a large model does not need to update.\n\n2. The presentation is not always very clear. For example, in Eq. (5) it is not clear to me what the authors mean by the joint distribution of $Q_1, Q_2, \\ldots, Q_T$. How can we compute KL between this joint and $Q$?\n\n3. As the authors mention, the proposed method is 2.5 times slower at inference time compared to standard sampling from the same model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: It is not intuitively obvious to me why matching the KL divergence is the right objective. Could the authors please provide some intuition? I imagine it is something like this: when the small model updates significantly for some token, we want the large model to also udpate significantly. That seems reasonable, but probably doesn't always work well: if the small model is unaware of some fact or makes an arithmetic mistake, it may need to update significantly on the corresponding tokens, while a large model does not need to update.\n\nOur method has already considered this situation, which presents a challenge for static methods. Our goal is to control the shift of the fine-tuned model to be the same, and KL divergence is commonly used to describe the distance between distributions, making it more suitable for representing the shift between two distributions. \n\nUsing a static shift to match on a sentence level can indeed result in the incorrect transfer of some erroneous knowledge. Therefore, our method refines this process to each decoding step, allowing dynamic adjustment of knowledge transfer intensity to mitigate this issue. Notably, our method ultimately overlays the transferred knowledge onto the logits of the large model. **This means that when the small model generates errors, our method dynamically adjusts based on the large model's capabilities. When the large model can solve the problem independently, it will retain more of its own abilities.**\n\nAs shown in the results of Table 1, our method significantly improves the model's performance even when the effect of the 1.1B finetuned model is much lower than that of the 13B base model. For instance, on MMLU, the 1.1B model improves from 37.26 to 48.32. Compared to the static method's improvement from 37.26 to 39.88, **our method does not fully trust the capabilities of the small model but rather retains more of the large model's abilities to mitigate this issue.**\n\n\n\n> Q2: How can we compute KL between this $Q_1...Q_T$ and $Q$?\n\n\nThe term \"joint distribution\" in our paper refers to the fusion distribution obtained by combining the outputs of a series of smaller models. \n\n As shown in the \"Global Rebuttal\" section \"Supplementary Proof for the Fusion of Multiple SLMs Scenario,\" we transform the problem of approximating $J$ into a centroid problem (i.e., optimizing the upper bound of $KL(J||Q)$). Therefore, we can use equation (6) in the paper to calculate this KL divergence.\n\n> Q3: The proposed method is 2.5 times slower at inference time compared to standard sampling from the same model.\n\nIt should be noted that the 2.5x slower inference speed is compared to the 13B FFT (full fine-tuning). **However, our method does not require fine-tuning the large model (e.g., 13B), allowing it to benefit from smaller expert models, resulting in much lower hardware requirements.** As shown in Table 3, the time required for 13B FFT is 1176s, while the time required for 7B FFT or 1.1B FFT is only 588s or 128s, significantly less than the time required for 13B FFT.  Additionally, our method can leverage many pre-existing small expert models from Huggingface, further reducing training time.\n\nFurthermore, as noted in the \"Complementary to Efficiency Analysis\" section of our Global Rebuttal, the time consumed by our method is almost identical to that of the static method, while our method performs significantly better.\n\n\n> Q4: Supervised instruction tuning details.\n\nFor each task, we used the official dataset and trained our small model on the official training set. We conducted supervised instruction tuning without using chain-of-thoughts. When constructing the prompt, we used simple instructions for concatenation. For example, for gsm8k: \"Question: \" + [question] + \"\\\\nAnswer:\". For CNN/DM: [article] + \"\\\\n\\\\nSummarize the above article:\".\n\n\n> Q5:In Figure 3, what do the tokens where we set the weight to the lower bound correspond to, and what about the upper bound? Are they qualitatively different?\n\nWhen $\\alpha$ is high, the confidence in the logits generated by a specific expert will be higher, leaning towards the tokens that this expert is more certain about at the moment. For example, for gsm8k, a high $\\alpha$ will tend to generate mathematical symbols.\n\nFor the following question: {\"question\": \"A pen costs as much as a pencil and eraser combined. A pencil costs \\\\$1.20 and an eraser costs \\\\$ 0.30. How much will 8 pens cost?\"}\n\nThe answers obtained from our method are as follows (**bold** indicates $\\alpha$ is the upper bound, and `red` indicates $\\alpha$ to the lower bound):\n\n{Ours: \" **8** pencils will cost **8** * `$`1.2**0** = `$<<`8\\***1.2=9.60>>9.60**. 8 **erasers** `will cost` **8 \\*** `$`**0**.30 = **\\$<<8*0.30=2.40>>2.40.** `Thus`, **8** `pens will` cost \\$9.**60** **+** \\$**2.40** `= $<<`**9.6+2.4=12>>12**.\"}\n\n**As can be seen, when $\\alpha$ is at the upper bound, the response\nleans more towards mathematical reasoning; when $\\alpha$ is at the\nlower bound, the response tends to be more of a normal statement or\ninformation about the question.**\n\n> Q6:In Figure 3, why is the lower bound 0.8 and the upper bound 1.5? Are these tunable parameters? 0.8 seems quite high, shouldn't we want to set the lower bound to 0?\n\n  Sorry for the confusion, Figure 3 can indeed be misleading. In our experiments, $\\alpha$ was searched from 0 to 2.0. The values 0.8 and 1.5 in Figure 3 represent the minimum and maximum values obtained during the optimization process. In the GSM8K task, due to the large model's inherent capability bias, the overall trust in the expert knowledge is relatively high, resulting in higher values obtained during the optimization process. We will improve the depiction of the figure in the next\n  version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cws0BtUO4o", "reviewer": "Reviewer_frHX", "summary": "They tackle the problem of merging the logits from multiple models. To do so, they propose an objective that minimizes the squared loss of the KL between the two pairs of (student, teacher) models. This is solved via a random search.", "questions": "- Where is the baseline proxy tuning for multi-task tuning in Table 2? \n- From algorithm 1 in Appendix C, the optimization is done just via random search? Guessing values and storing the best? I couldn’t find the method to optimize the objective mentioned in the main paper. It should be more clearly stated in the main paper, and how it handles multitask setup. \n- For the efficiency analysis, the BV can be done in 1 forward pass all in parallel, but the n parameter searches require 20 forward passes. It really is n times more compute, which is not minimal. For example, decoding 100 tokens vs decoding 1 token would be a constant in the Big O, but they are different efficiency wise. \n- In eq (2), is the second term (multiplied by alpha) not normalized but in logit space? \n- In fig 4, are there some special about the tokens being generated at the timesteps where $alpha$ is high?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Paper well written and easy to follow \n- Nice ablations on alpha and efficiency \n- Works well in single task scenarios", "weaknesses": "- Using the squared error between two KL’s is not theoretically motivated (at least that I am aware of) \n- More description of the optimization method in main text since it is a big part of the method \n- Missing baseline in multi-task tuning setup", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:Using the squared error between two KL's is not theoretically motivated (at least that I am aware of)\n\nThe goal of our motivation is to enhance the constraints using KL divergence, aiming for the shift of the fine-tuned large model to be equal to the shift of the fine-tuned small models in each decoding step. Compared to global static transfer, we adjust an appropriate shift at each decoding step to achieve better transfer results. We use KL divergence as a distance function to measure this \"shift,\" (as shown in section 3.1) and the squared error between KL divergences helps us align these two shifts. Squared error is commonly used in various regression prediction approximations[1], and it is easy to solve, making it well-suited for our setting. Additionally, as shown in the \"Supplementary Proof for the Fusion of Multiple SLMs Scenario\" in the Global Rebuttal, due to the geometric properties and inequality characteristics of squared error, our method can extend more smoothly to scenarios involving multiple experts when using squared error.\n  \n- [1] Learning Theory for Distribution Regression (JMLR2016)\n\n\n> Q2:More description of the optimization method in main text since it is a big part of the method.\n\n  Thank you for your reminder. We will elaborate on this section in the next version. Our optimized method performs a linear search for $\\alpha$ and multiple Logi Arithmetic operations after obtaining the logits from all models to find the optimal situation described in Eq(6). During the search, we start from 0 and increment by 0.1 each time until we reach 2, resulting in 20 searches.\n  We perform this search at each decoding step, so, as shown in Figures 3 and 4, the $\\alpha$ varies for each decoding step.\n\nIn the multi-task setting, directly searching for every expert results in exponential complexity. For practical use, we accelerate this process by using only one small expert at each decoding step, thereby reducing the exponential complexity of the search process to linear complexity. Experiments have shown that this approach also yields good results (as shown in Table 2). \n\n\n> Q3: Where is the baseline proxy tuning for multi-task tuning in Table 2?\n\nIt is worth noting that Proxy Tuning, due to its lack of prior assumptions in a multi-task setting and the difficulty in presetting the transfer proportions for multiple experts, is not capable of handling multi-task scenarios. In contrast, our method can dynamically adjust the transfer proportions of expert knowledge, making it naturally suitable for multi-task settings.\n\nTo better demonstrate the effectiveness of our method, we further compared it with our method with static $\\alpha$ in a multi-task setting. For the 4 seen tasks in our experiment, we set the corresponding expert coefficient to 0.25 ($\\alpha=1/4$, assigning the same proportion to each seen task expert). In the table below, it can be seen that our method, which dynamically adjusts the coefficients, significantly outperforms the static setting.\n\n|  | Seen Task | Unseen Task | Avg. |\n| --- | --- | --- | --- |\n| Ours (0.25 static) | 22.02 | 46.04 | 34.03 |\n| Ours | **27.53** | **51.31** | **39.42** |\n\n\n> Q4:For the efficiency analysis, the BV can be done in 1 forward pass all in parallel, but the n parameter searches require 20 forward passes. It really is n times more compute, which is not minimal. For example, decoding 100 tokens vs decoding 1 token would be a constant in the BigO, but they are different efficiency wise.\n\n Actually, our method only performs one forward pass when doing logit arithmetic. As analyzed in the \"Complementary to Efficiency Analysis\" section of the Global Rebuttal, the $nBV$ term represents $n$ quick logit arithmetic operations to obtain the final logits, which only requires one forward pass and not $n$ forward passes. So overall, the time consumption is almost the same as that of the static method.\n\n\n> Q5:In eq (2), is the second term (multiplied by alpha) not normalized but in logit space?\n\n  The second term is not normalized. Normalization is performed after the entire logit arithmetic calculation is completed.\n\n\n> Q6:In fig 4, are there some special about the tokens being generated at the timesteps where alpha is high?\n\n  When $\\alpha$ is high, the confidence in the logits generated by a specific expert will be higher, leaning towards the tokens that this expert is more certain about it at the moment. For example, for gsm8k, a high $\\alpha$ will tend to generate mathematical symbols.\n\n  For the following question:\n{\"question\": \"A pen costs as much as a pencil and eraser combined. A pencil costs \\\\$1.20 and an eraser costs \\\\$ 0.30. How much will 8 pens cost?\"}\n\nThe answers obtained from our method are as follows (**bold** indicates $\\alpha$ is the upper bound, and `red` indicates $\\alpha$ to the lower bound):\n\n{Ours: \" **8** pencils will cost **8** * `$`1.2**0** = `$<<`8\\***1.2=9.60>>9.60**. 8 **erasers** `will cost` **8 \\*** `$`**0**.30 = **\\$<<8*0.30=2.40>>2.40.** `Thus`, **8** `pens will` cost \\$9.**60** **+** \\$**2.40** `= $<<`**9.6+2.4=12>>12**.\"}\n\n  As can be seen, when $\\alpha$ is at the upper bound, the response leans more towards mathematical reasoning; when $\\alpha$ is at the lower bound, the response tends to be more of a normal statement or information about the question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "przGmbj06Q", "reviewer": "Reviewer_Ttwe", "summary": "The article discusses existing weak-to-strong methods, noting that current approaches typically use a static knowledge transfer ratio and a single small model to convey complex knowledge, which results in suboptimal performance. Consequently, the article proposes a dynamic logit fusion method that employs a series of task-specific small models and adaptively allocates weights among these models at each decoding step. The weights are learned by optimizing a problem constrained by Kullback-Leibler divergence. The article conducts experiments on various benchmarks, including both multi-task and single-task scenarios.", "questions": "1. In the multi-task setting, which 7B model is used as the expert to implement your algorithm on the 13B model? I am very confused. If you are using different sets of experts to operate on the 13B model, isn't this weak-to-strong? Because the parameters of multiple 7B models exceed 13B.\n2. There seems to be a typo in formula (9) in Appendix B, it seems to be $\\propto$ rather than $=$.", "limitations": "The authors have adequately addressed the limitations.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The article reevaluates existing logit arithmetic methods, highlighting the significant impact of fusion weights and the limitations of a single small model on test performance.\n\n2. By using constrained optimization, the article autonomously learns fusion weights, thereby approximating the computationally intensive results of fine-tuning large foundational models.\n\n3. Experiments were conducted to validate the proposed method, demonstrating notable improvements in performance, generalization capability, and robustness.", "weaknesses": "1. In Section 3.2, the proposed method is based on an assumption that lacks clear supporting evidence. Does this somewhat undermine the theoretical foundation of the algorithm?\n\n2. In the multi-task experiments, the results for CNN/DM do not demonstrate the method's superiority. Additionally, in the experiments on unseen tasks, the method does not show significant improvement.\n\n3. In the experiments, all evaluations were conducted in a 0-shot setting. How would the evaluation results change if a 5-shot experiment were conducted?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: In the multi-task setting, which 7B model is used as the expert to implement your algorithm on the 13B model? I am very confused. If you are using different sets of experts to operate on the 13B model, isn't this weak-to-strong? Because the parameters of multiple 7B models exceed 13B\n\nIn the multi-task setting, we used all the experts for the seen tasks, i.e., four experts fine-tuned on each task (7B or 1.1B).\n\nIt needs to be clarified that our \"weak to strong\" approach aims to use weak model supervision to elicit the full capabilities of a much stronger model. In our setting, we use small fine-tuned models (weak models, e.g., 7B or 1.1B) that have transferred downstream task-specific knowledge to enhance a large model (strong model, e.g., 13B) without downstream knowledge. In our setting, \"weak\" or \"strong\" indicates the capability upper bound of a model, which is commonly enhanced by scaling model size. However, the capabilities of multiple small models are constrained by their model size and are generally weaker than the large ones, as shown by the performance of the 7B multi-task expert on unseen tasks in Table 2 (35.82<51.25). \n\nAdditionally, fine-tuning a large model is significantly more expensive than fine-tuning a small model, requiring more advanced hardware and more training time. In contrast, our weak-to-strong paradigm only needs to fine-tune the small models.\n\n\n> Q2: There seems to be a typo in formula (9) in Appendix B, it seems to be $\\propto$ rather than $=$.\n\nThanks for your valuable advice. We will carefully revise our paper based on your suggestion.\n\n\n> Q3:In Section 3.2, the proposed method is based on an assumption that lacks clear supporting evidence. Does this somewhat undermine the theoretical foundation of the algorithm\n\nOur work is based on verified theories [1, 2, 3]. First, using the shift of the fine-tuned model to accomplish our domain adaptation task is reasonable. To demonstrate our optimization process, we can view the fine-tuning procedure as reinforcement learning (RL) with a KL-divergence constraint preventing divergence from a reference model.\n\n  *According to the theory presented in the DPO[1]: Theorem 1. Under mild assumptions, all reward classes consistent with the Plackett-Luce (and Bradley-Terry in particular) models can be represented with the reparameterization* $r(x, y) = \\beta \\log \\frac{\\pi(y|x)}{\\pi_{ref}(y|x)}$ *for some model* $\\pi(y|x)$ *and a given reference model* $\\pi_{ref}(y|x)$.\n\n  Meanwhile, According to [1,2] the optimal solution to the KL-constrained reward maximization objective is given by:\n$$\n\\begin{align}\n  \\pi_r(y|x)=\\frac{1}{Z(x)}\\pi_{ref}(y|x)exp(\\frac{1}{\\beta}r(x,y))\\\\\\\\\n  where\\quad Z(x)=\\sum_y \\pi_{ref}(y|x)exp(\\frac{1}{\\beta}r(x,y))\n\\end{align}\n$$\n\n\n  Combining the above theory, we can derive the following equation:\n\n  $\\pi_{r}(y|x)=\\frac{1}{Z(x)}\\pi_{ref}(y|x)exp(\\log \\frac{\\pi_r(y|x)}{\\pi_{ref}(y|x)})$\n\n  Since any language model can be viewed as the solution to KL-constrained RL with a constraint to the pre-trained model[1], this equation is applicable to fine-tuning scenarios. We can replace\n  $\\pi$ in the parentheses with the small model's $\\pi$, resulting in the following equation:\n\n\n$\\pi_{L-ft}(y|x)=\\frac{1}{Z(x)}\\pi_{L-pt}(y|x)exp(\\log \\frac{\\pi_{S-ft}(y|x)}{\\pi_{S-pt}(y|x)})$\n\nIt can be seen that, based on the theory from previous work, it is reasonable to assume that the shifts between models are consistent for knowledge transfer. This is consistent with the form of the proof in Appendix B.\n\nCompared to global static transfer, we adjust an appropriate shift at each decoding step to achieve better transfer results. KL divergence[1, 2, 3] is commonly used to describe the distance between distributions (as shown in section 3.1), making it more suitable for representing the shift between two distributions. We use KL divergence as a distance function to represent the above shift, converting it into a KL-constrained problem, dynamically controlling the knowledge transfer by constraining each decoding step. Meanwhile, the squared error is commonly used in various regression prediction approximations[3], and it is easy to solve, making it well-suited for our setup. Additionally, as shown in the \"Supplementary Proof for the Fusion of Multiple SLMs Scenario\" in the Global Rebuttal, due to the geometric properties and inequality characteristics of squared error, our method can extend more smoothly to scenarios involving multiple experts when using squared error.\n\n- [1] Direct preference optimization: Your language model is secretly a reward model (NIPS2023)\n- [2] RL with KL penalties is better viewed as Bayesian inference (EMNLP2022)\n- [3] Learning Theory for Distribution Regression (JMLR2016)\n\n> Q4:In the multi-task experiments, the results for CNN/DM and unseen tasks do not show significant improvement\n\nActually, in our multi-task experiments, our method achieved a 16% improvement on CNN/DM compared to the 13B model (8.94->10.52). It is noticeable that after using multi-task tuning on the 13B model, the performance on unseen tasks even decreased (51.28->50.58). This indicates that unseen tasks are more challenging and that overfitting can occur when training on seen tasks. In contrast, our method shows an improvement on unseen tasks (51.28->51.31), demonstrating that our approach not only provides significant enhancements on seen tasks but also helps mitigate overfitting within the domain.\n\n> Q5: How would the evaluation results change if a 5-shot experiment were conducted?\n\n Actually, we have conducted 5-shot experiments. As mentioned in Section 5.3, our method can be combined with in-context learning(ICL) and can also enhance its effect. We used the 5-shot approach as the ICL setting. As shown in Figure 5(a), our method combined with 5-shot ICL achieves an overall improvement of 18.3% compared to using ICL alone. This is mainly due to our method's ability to integrate the knowledge possessed by the experts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Rebuttal

Dear reviewers,

We much appreciate for your acknowledgment of our work and helpful, insightful comments. Following the reviewers' suggestions, we have carefully revised the paper and conducted a series of new experiments to address the reviewers' concerns.

The below contains a rebuttal for remarks that are common to most reviewers.

### Complementary to Efficiency Analysis (for XMhg, YRcj, frHX)
In section 5.2 of our paper, we conducted an Efficiency Analysis of logit Arithmetic. To better illustrate our efficiency, we further analyze the overall efficiency of our method here.

Overall, **our method has a similar time complexity to the static method**.  Given: current sequence length $s$, large model dimension $h_L$, small model dimension $h_S$, number of layers in the large model $L_1$, number of layers in the small model $L_2$, batch size $B$, vocabulary size $V$, number of searches per decoding step $n$. Assume the FLOPs for a single forward pass of the large model and the small model are $FLOPs_L$ and $FLOPs_S$, respectively. The FLOPs can be calculated as: $FLOPs_L=L_1*(12Bsh_L^2+2Bs^2h_L)+Bsh_LV$ ,and $FLOPs_S=L_2*(12Bsh_S^2+2Bs^2h_S)+Bsh_SV$(here we ignore the kv cache). Therefore, the FLOPs for a single forward pass of our method on a single task is: $FLOPs_L + 2*FLOPs_S+nBV$. Among these, only the $nBV$ term ($n \le 20$) corresponds to the additional computational cost of our method, which is much smaller compared to the previous term and can be considered negligible in the overall time. Additionally, in our efficiency analysis, as shown in Table 3, our method is only 0.008 seconds slower per sample compared to the static method, which is negligible.

### Supplementary Proof for the Fusion of Multiple SLMs Scenario (for YRcj, frHX, Ttwe)

This section mainly explains how we extend the transfer problem to multiple small models. When transferring the knowledge of multiple expert SLMs to a LLM, we consider the following two aspects: 1. The fusion of knowledge from different domain experts. 2. The transfer of knowledge from SLM to LLM, i.e., the transfer of knowledge from a single expert, which was discussed in Section 3.2. Intuitively, we first focus on the fusion of different domain experts' knowledge before performing the transfer.
 Here, we define the distribution of the combined knowledge of these small models as $J$. **Therefore, we aim to achieve $KL(P || \tilde{P})=KL(Q||J)$**
 
  Since solving for $J$ is difficult, we propose constraining it based on the relationship between $J$ and $\{Q_i\}$ to approximate it. Here, we can transform $KL(Q||J)$ into $KL(Q||Q_i)+C_J(Q_i)$, where $C_J(Q_i)$ is the bias function from $Q_i$ to $J$. When we approximate $J$ as the centroid of $\{Q_i\}$ on the KL-constrained plane, we can implicitly solve these bias functions. According to the definition of the centroid, $J$ can be solved by minimizing the sum of the squared distances to each point, as shown below:

  $$\arg \min_{J} \sum_{i=1}^T (KL(Q \parallel J) - KL \left(Q \parallel Q_i \right))^2$$

  Since our goal is $KL(P \parallel \tilde{P})=KL(Q||J)$, substituting this into our equation gives us our final optimization objective:


$$\arg \min_{\tilde{P}} \sum_{i=1}^T (KL(P \parallel \tilde{P}) - KL \left(Q_i \parallel Q \right))^2$$


**To prove the reasonableness of our approximation, we provide a more rigorous proof below. Our initial objective is as follows:**


$$\arg \min_{\tilde{P}} \sum_{i=1}^T (KL(\tilde{P} \parallel P) - KL(J||Q))^2$$


  By assuming $KL(Q||J)=KL(Q||Q_i)+C_J(Q_i)$, we can transform the original problem $\arg \min_{\tilde{P}}  (KL(\tilde{P} \parallel P) - KL(J||Q))^2$ into $T$ constrained optimization problems:

$$\begin{align}\arg \min_{\tilde{P}} (KL(\tilde{P} \parallel P) - KL \left(Q_i \parallel Q \right)-C_J(Q_1))^2\\\\
...\\\\
\arg \min_{\tilde{P}} (KL(\tilde{P} \parallel P) - KL \left(Q_i \parallel Q \right)-C_J(Q_T))^2\end{align}$$




After jointly optimizing them, we have:

$$\begin{align}\arg \min_{\tilde{P}} \sum_{i=1}^T (KL(\tilde{P} \parallel P) - KL \left(Q_i \parallel Q \right)-C_J(Q_i))^2\\\\
\sum_{i=1}^T (KL(\tilde{P} \parallel P) - KL \left(Q_i \parallel Q \right)-C_J(Q_i))^2 \\\\\leq \sum_{i=1}^T (KL(\tilde{P} \parallel P) - KL \left(Q_i \parallel Q \right))^2+\sum_{i=1}^TC_J(Q_i))^2\\\\
=\sum_{i=1}^T (KL(\tilde{P} \parallel P) - KL \left(Q_i \parallel Q \right))^2+C_{J-Q}\end{align}$$

Since $C_{J-Q}$ is a constant term independent of $\tilde{P}$, we can ignore it. Finally, we solve the original problem by optimizing this upper bound. When we symmetrize the terms in the KL divergence, we can obtain a similar conclusion. Therefore, in the multi-task setting, we can solve it using the following formula (As shown in Equation (6) of the paper):

  $$\arg \min_{\tilde{P}} \sum_{i=1}^T \left[(KL(P \parallel \tilde{P}) - KL \left(Q_i \parallel Q \right))^2+(KL(\tilde{P} \parallel P) - KL \left(Q \parallel  Q_i\right))^2\right]$$

Meta-review: This paper proposes a dynamic logit fusion method for transferring knowledge from smaller, task-specific language models to a larger language model. The method adaptively learns the mixing weights between the models' logits at each decoding step by optimizing an objective based on KL divergence.

Reviewers found the method to be simple yet effective, achieving good results across various tasks. (YRcj: "The method is very simple... leading to better results."; frHX: "Paper well written and easy to follow... Works well in single task scenarios."). The experimental evaluation was considered comprehensive, covering multiple tasks, model sizes, and both single-task and multi-task scenarios. (YRcj: "The experiments are comprehensive... The authors also consider both single-task and multi-task scenarios."). The proposed method consistently outperformed baseline methods, including proxy tuning and full fine-tuning on smaller models, and approached the performance of directly fine-tuning the large model. (YRcj: "The results are good across all tasks... recovers a large fraction of the ceiling performance achieved by directly finetuning the large model on ground truth.")

However, the reviewers identified a few potential weakness in the initial reviews:
 - Limited theoretical justification: Initially, reviewers raised concerns about the theoretical motivation behind matching KL divergence as the objective function. (YRcj: "It is not intuitively obvious to me why matching the KL divergence is the right objective."; frHX: "Using the squared error between two KL’s is not theoretically motivated."). However, the authors addressed this concern in their rebuttal by providing a theoretical derivation based on KL-constrained reinforcement learning. (XMhg: "I thank the reviewers for their rebuttal and I appreciated their theoretical derivation. I believe that this important piece to be added to the paper.")
- Clarity of presentation: Some reviewers found certain aspects of the presentation to be unclear, particularly regarding the optimization method and the multi-task setup. (frHX: "More description of the optimization method in main text since it is a big part of the method... Missing baseline in multi-task tuning setup.") The authors addressed these concerns in their rebuttal by providing further details and clarifications.
 - Computational complexity: The method was noted to be slower at inference time compared to standard sampling. (YRcj: "As the authors mention, the proposed method is 2.5 times slower at inference time compared to standard sampling from the same model.") While the authors clarified that the increased cost is minimal compared to fine-tuning the large model, the efficiency aspect remains a potential limitation.

The paper presents a novel and effective method for weak-to-strong generalization in language models. The comprehensive experimental evaluation demonstrates the method's strong empirical performance. While some initial concerns regarding theoretical justification and computational cost were raised, the authors adequately addressed these concerns in their rebuttal.
Decision: Accept (poster)