Adversarial Moment-Matching Distillation of Large Language Models
Authors: Chen Jia
Venue: NeurIPS 2024 poster (2024)
Keywords: Knowledge distillation
Large language models
Imitation learning
Moment-matching
Adversarial training
Primary area: natural_language_processing

Abstract: Knowledge distillation (KD) has been shown to be highly effective in guiding a student model with a larger teacher model and achieving practical benefits in improving the computational and memory efficiency for large language models (LLMs). State-of-the-art KD methods for LLMs mostly rely on minimizing explicit metrics measuring the divergence between teacher and student probability predictions. Instead of optimizing these mandatory cloning objectives, we explore an imitation learning strategy for KD of LLMs. In particular, we minimize the imitation gap by matching the action-value moments of the teacher's behavior from both on- and off-policy perspectives. To achieve this moment-matching goal, we propose an adversarial training algorithm to jointly estimate the moment-matching distance and optimize the student policy to minimize it. Results from both task-agnostic instruction-following experiments and task-specific experiments demonstrate the effectiveness of our method and achieve new state-of-the-art performance.

Adversarial Moment-Matching Distillation of Large Language Models
Chen Jia SI-TECH Information Technology j jachennestlaake@@maill com
Abstract
Knowledge distillation (KD) has been shown to be highly effective in guiding student model with larger teacher model and achieving practical benefits in improving the computational and memory efficiency for large language models (LLMs). State-of-thh-eee KD methods for LLMs mostly rely minimizing explicit distribution distance between teacher and student probability predictions. Instead of optimizing these mandatory behavior cloning objectives, we explore an imitation learning strategy for KD of LLMs.. In particular, we minimize the imitation gap by matching the action-value moments of the teacher's behavior from both on- and off off-policy perspectives. To achieve this action-value moment-matching goal, we propose an adversarial training algorithm jointly estimate the moment- matching distance and optimize the student policy to minimize it. Results from both task-agnostic instruction-following experiments and task-specific experiments demonstrate the effectiveness of our method and achieve new state-of-the-an performance.
Introduction
Large language models (LLMs) like GPT-4 [1] and LLaMA [35] have revolutionized natural language processing, significantly enhancing the quality of text generation across various tasks. This success is largely due to the extensive scale of training data and the substantial increase in model parameters [19]. However, the high computational and memory requirements of these models present significant challenges for practical deployment. To address these issues, knowledge distillation (KD) [16] has emerged as a key technique. KD involves transferring knowledge from a large, complex teacher model to smaller, more efficient student model, thereby maintaining high performance while reducing resource demands. Most distillation methods for auto-regressive text generation models, including LLMs, employ metrics probability distribution distance, such as Lulbback-Leibler (KI.) divergence [20] and reverse KI. divergence [14], aiming to align the token-level probability distributions between the teacher and student models.
The distribution matching-baid based distillation methods can be viewed as behavior cloning on a decision- making problem from the perspective of imitation learning [24, 14, 2]. Based on this concept, early works based on the teacher- generated outputs [20] a supervised dataset [30] can be viewed as off policy approach. Recent works further incorporate an on-policy approach, training the student on its self-generated outputs [24], using KL-based divergence [14, 2, 21] and total variation (TV) distance [39]. Accordingly, such distribution matching- based methods face the sub-optimality problem. The objective functions aimed at aligning the probability distributions between the teacher and student models can be straightforward but cannot fully capture the goal of distilling language knowledge. First, intuitively, the correct output for input vary, and thus behavior cloning cannot capture the full knowledge teacher. Besides, there is no standardized definition for the quality of generated output given input, which makes it difficult to define the objective of knowledge distillation. This
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Yn
J1
(a) On-policy distributinn-atching distillation.
(b) Off-policy distribution- distribution-matching distillation.
(c) On-policy Q-value moment-match. distillation (ours).
(d) Off-policy Q-value moment-match. distillation (ours).
Figure 1: The comparison between the distribution-maccciin---ss
distillation and the action-value moment-matching distillation is outlined. we and х. denote the student policy and the teacher policy, respectively. For both on-policy (using student-generated outputs) and off-policy (using teacher- generated outputs) perspectives, our approach optimizes moment-man matthing of action-vall value functions (e-functions) instead of minimizing the distribution distance measured by M = KL, RKL, TV, etc.
imposes a significant limitation on the generalization performance of the student model through distillation.
To address the aforementioned issues, we employ a reinforcement learning (RL) formulation for the auto-regressive text generation problem and utilize the definition of imitation gap to describe high-level goal of knowledge distillation. Additionally, we address matching moments of the action-value function, which reflects the quality of token-level predictions for the entire output. In addressing the action-value function, we adopt the approach of Swamy et al. [33], considering two-player minimax game between the language policy and the action-val value functions, aiming to minimize an upper bound of the moment-matee mattentnttthinng objective. For this purpose, we introduce adversarial training algorithm based on the policy gradient to jointly optimize the on-/off-policy objectives. Figure illustrates the overall approach.
imitation gap for KD by
Theoretically, we compare the moment-matching objective with other distribution sererbution-matchinng mea- surements such as step-wise TV distance and analyze the convergence rate of our algorithm to an e-accurate stationary point for optimization. Empirically, we evaluate our approach on both costruction-following dataset and three task-specific datasets for text summarization, machine translation, and commonsense reasoning. Results demonstrate that the proposed adversarial moment- matching approach effectively optimizes the moment- maatnntntttcinng distance of the imitation gap and outperforms state-of-the- of-the-art KD methods and range of cetrriuutionmmtcchiin maacching-based methods. The code and implementation are released at https: ///g//gtttt//tt///// com/ wwcheneestaake//eeee
2 Related Work
Distillation of large language models. There has been an increasing interest in knowledge distillation (KD) auto-regressive LMs, especially concerning large language models (LLMs) [41, 42). This process effectively transfers elicited knowledge from teacher L.LMs to smaller student models, aiming to compress the large size of neural network parameters and make LLMs more efficient. Sequence- level KD (SeqKD) [20] is variation of supervised fine-tuning (SFT) in KD. It can be viewed the simplest method for distillation generated outputs. This method has been extensively used for LLMs and has achieved success [34, 6]. In contrast, distillation of white-box LLMs can make full use of internal information of the teacher model, such as logits [30, 39] and hidden states [23], for distribution alignment, making it more effective and efficient for KD. However, unlike previous work that explicitly clones the distribution of teacher LLMs into student models, this work learns auxiliary Q-value function guide KD.
black-box LLMs by fine-tuning the student model with teacher-
Distillation via distribution matching. Most promising results in the distillation of white-box L.LMs achieved by minimizing divergence between the probability distributions of the teacher model

---

and student models. Kullback-Leible (KL) divergence, reverse Kullback-Leibles (RKL) divergence, and Jensen- SaannnoShnnnnnn (JS) divergence are three widely used KD objectives for auto-regressive LMs [39. 14, 2, 21, 41]. Wen et al. [39] have shown the equivalent formulations of sequence-level KI., RKI., JS divergences, and the step-wise terms. Additionally, they also present the strong performance of step-wise total variation (TV) distance for KD which can upper bound the sequence-level term. As a result, most recent works focus on on-policy approaches for KD [2] and combine the real- time-generated outputs by students (on-policy) with the from supervised datasets) (off-policy). Following this line, Gu et al. [14] further propose a policy gradient-based method to address the high variance issues of RKL-based methods while Ko et al. propose a more efficient and effective method using a skew KL divergence loss and an adaptive off-policy approach. We also focus a combination of on-policy and off off- off-policy objectives for KD, but we introduce a more sophisticated moment-matchinn well-studied ceerriuttionmmtcchinn metrics such as KI., RKI., JS divergences, and TV distance.
outputs by teachers (or
approach instead of directly using the
Distillation via reinforcement learning. In a common formulation of RL in text generation [44, 26, 15], auto-regressive model can be viewed as a language policy, making decisions on the next token (action) based on the currently generated sequence (state). From this perspective, KD corresponds to behavior cloning in imitation learning [20. 7, 14, 2]. For imitation learning in text generation, early works such as SeqGAN [44] and TextGAIL [40] utilize a generative adversarial framework to balance between the reward model, optimized by discriminating generated/real word text, and the language policy, optimized by policy gradient- basedd methods using the reward model. Existing work on KD via imitation learning refers to ImitKD [24], which optimizes the student policy by learning from demonstrations of the teacher model. RL--based distillation can also be especially relevant for leveraging the feedback from the teacher to train student models [4, 9]. in which teacher models are used to generate the feedback data for training a reward model. We build our method upon an RL-based imitation learning framework. However, unlike previous work [20, 14, 2], we propose an adversarial moment-matching approach to enhance behavior cloning.
Method
3.1 Notations and Definitions
In this section, we consider the text generation task as a srciisionmmainng process and give a corre- sponding reinforcement learning (RL) formulation.
Text generation. Given an input æ, the auto-regressive generation task in our work aims generate a sequence of tokens as the output (y1..... yr), where ye comes from vocabulary V. For simplicity, we define y = (yo, ....... ..) as the full input-output sequence, where you x denotes the input. The generator modeled by a conditional probability distribution po æ)) IT po (Vt+1ly9s)) where yge denotes the prefix (yo, .1.....,u)). .t). {0, I, 13.
RL formulation. We model text generation as a Emite-horizon time-independent Markov de- cision process. At each time step t E {O...., T 1}. the policy Te takes an action (t): We+1 E V based on the current state ((): Uge E V, transits to the next state (1 + 1): ggt+1 E y and receives a reward (t): r(y< 1t-1) by reward function r y x V + R. We focus on
The policy corresponds to the generation model ne T0 yec) = po a (conditional) trajectory Ue.. " yT} =: T ~ Tolæ which refers to a se- quence of state-action pairs generated by given an initial state yo edly sampling an action yt-1 ~ (e(-1 and obtain the next state Ugr+1 ~ T( 3441)' for T time steps. In such case, the probability of (conditional) trajectory is formally rep- resented as p(r|a, TT) = II (yse+ilyse Meteelleee Vt+1) 1l function and Q-value function as V= (y<r)
Pa and then repeat-
We also define our value and where 7 (0, 1) denotes the discounting factor. We define the RL. objective in our generation task to maximize the performance J(mo) Em-ap Ey ice
E Ero
E1= 2 "
QT" (g M++1) [E
EF=1 " (e
-- r'r(y 1++)
In text generation, the state-transie natee-transiton is commonly assumed to be deterministic [44, 26). i... T'y se41llgee (t+1) = 1.
3

---

3.2 Knowledge Distillation as Moment-Matching Imitation Learning
Based on the RL formulation of auto-regressive generation, we can view the goal of knowledge distillation at high-level to bridge the performance gap between the teacher policy and the student policy.
Definition (Imitation gap). We define the imitation gap between the teacher policy and student policy as:
J(T.)- - J(To)
E m-Pe
Σ rr((gs
E a romgle
Σ
Arly (ol)) 1=0
(I)
From the perspective of imitation learning [33, 32), the objective of distillation from the teacher policy to the student policy Te can be represented as to minimize the imitation gap of Eq. (1) w.r.t the parameters of student policy 0. A direct idea from Eq. (1) to use moment matching over the reward to optimize the imitation gap [33]. However, we actually care about the long-term reward, each time step. we should consider the accumulated reward in the future output rather than the immediate reward to the fitness of previous tokens (prefix). To this end, we can alternatively use the Q-value function (def. in $3.1) for each timestep to represent the overall reward from the current timestep to the last timestep. Similar to [33], we can apply the Performance Difference Lemma (PDL) [18, 33] to expand the imitation gap in Eq. (I) into either off-policy or on-policy expressions.
Proposition (Off-policy bound of imitation gap [33]). Let FQ denote the set of Q-value functions induced by sampling actions from ne, then have: ((WSh.
[(T.) ((TO) S sup FEFQ
E
Σ
- E
[
(2)
=:Col((FOF))
In the following sections, we will use (ool J) to represent the off-policy moment-matching objective of milation learning for KD.
The off off-policy moment- noment-matching matching objective in Proposition only requires collected dataset of teacher-ee teacher-generated generated trajectories to be evaluated and minimized.
Proposition (On-policy bound of imitation gap [33]). Let Fo. denote the set of Ru'Q-vvvll functions induced by sampling actions from w. then we have:
J(T.) - ((To) <
sup
E an Torgle i
Σ 1=
[f(Uge ff-ff((w f(y Mett))
fEJq.
(l)
Con(ra
In the following sections, we will use (m f) to represent the on-policy moment-matching objective of an imitation learning for KD.
Proof. See Appendix A.I and Appendix A.2 for the complete derivations of Proposition and Proposition 2, respectively.
is notable from Proposition 2 that the on-policy moment-matching objective requires interactions with the teacher to tell us what action they would take in any state visited by the student as well as on-policy samples from the student's 's current policy 7 ~ To l.
In the remaining content of this section, we will explore the relationship between the moment- matching objectives and the existing citeriuution-matcchin draw a general formulation of the state-of-the -fee--artth-trtttt methods for distillation of LLMS [39, 14, 2, 21] that rely on citriiuution-matcchn between the student's and teacher's predictions, through minimizing the step-wise probability distribution distance between the teacher policy and student policy.
objectives [39]. At the beginning, we

---

Definition 2 (Generalized step-wise distribution distance). The aff-policy and on on-policy versions are defined as follows.
d' (T0, T.)
(M))) ;
Tor im
d' (To, Tx)
E (1 Fo
(5)
where M(...) denotes a distribution distance, consisting of total variation (TV) distance [39] and Ballback-eeible (KL)-based divergence [14, 2]. Detailed definitions for these distances refer to Appendix A.3. For simplicity, we directly replace M with TV, KL, RKL, etc in the following sections.
is notable from Wen et al. [39] that the sequence-level KL, RKL and JS divergences can be equivalently represented as the step-wise terms, and the sequence-level TV distance can be upper bounded by the step-wise terms, which can be actually implemented by algorithms. To make a connection with the step-wise distribution distance (Definition 2),
the following definition.
Definition 3 patrrbbutionmmatciing formulation of moment-matching objectives). Based on Definition 2. we can re-formulate the off off-policy and on on-policy moment- moment-matching (MM) objectives (Proposition and Proposition 2, respectively) via step-wise distribution-matchnee which can be defined as d (To, T.) and dMM (To, Tx) respectively, where the distance metric MM(...) can be defined as follows.
MM (T. (11)s)) (o(() (-Uge)
coffion)
y vc
ooff on
y v
y-
(-lyge
Off-policy: foll arg max Loll (TO, f); JEFQ
On-policy: fon arg max Con (To, FEFQ.
where Coft (T and (mT f) denote the off policy and on-policy moment-matching objectives, which are defined in Proposition 1 and Proposition 2, respectively.
Under Definition we observe that the main difference between the moment-matching objectives and other step-wise distribution distance, e.g., TV distance and KL-based divergences in formulation
comes from the optimal Q-value function sf roff(on) aiming maximize the discrepancy of its expec- tations based on v. (.19gt) v.s. =0(-1y<.) for each step E {0, 1,..., T - 1}. To look deeper, we draw connection between the nomeent-matcing objectives and step-wise TV distance using the following corollary.
Theorem 1 (Relationship between moment-matchinn objective and TV distance). Under constrain of uniform boundness on the class of Q-value functions for off-/on-policy learning: Fo = Fe. {f |[ < 1}, the coment-matthinng objectives in Proposition I and Proposition 2 can be upper-bounded by the step-wise TV distance, Formally, we have
J(T.)- (( S sup
Coll (TT (To, f) 2d (a2f/(TT) (TO, πx);
f:l1f1l0sS1
((T.) J(To) < sup
CTT f) < 2d (To,
(8)
for the off off-policy and on-policy perspectives, respectively.
Proof. See Appendix A.4 for the complete derivation.
We can observe from Theorem 1 that minimizing the step-wise TV distance can achieve sub- optimal results compared to optimizing the moment-matching objectives ( J),, L J) for off-policy and on-policy imitation learning, which are defined in Proposition and Proposition 2, respectively. Thus, optimizing the moment-matching objectives can potentially achieve better optimization results for imitation learning.
5

---

Algorithm 1: Adversarial training procedure
Input:
Dataset Day with inputs and ground-truth outputs
Teacher policy rx; Student policy Ho with initial parameters pretrained on Day; Off-policy Q-value function fe, and on-policy Q-value function fo, with initial parameters Φn and sbs, respectively; Step sizes K (outer), N (inner); Learning rate yi Controlling factor a; Off-ton-policy combination factor 8 Output: The optimized student policy no.
for k 2,,,2..... K do
for 0,
do Da and generate trajectory 7
Sample an input di <- Φi + a,,1,V #1 Lolt ( Ox: Sample an input ~ Da and generate trajectory 7 ne x de + d2 + a(1 - V Lo "r"" Oz. f
T |æ:
D maximize Lolf (TRR. S01) ) in Eq. (9)
v maximize (TT ( 102) in Eq. (9)
end
Sample an input Da and generate trajectories Tk off To aa and Tk ~ me|ær Ok-1 02 (- (-3goft (T 0k) + (1 B) (T 0s)) minimize L( for f0o) in Eq. (9)
end
3.3 Adversarial Training Algorithm
Optimization objective. As shown in previous work [14, 2, 21] incorporating both the off-policy and on-policy distillation benefits effectiveness and efficiency. We thus consider training objective to jointly minimize the off-policy moment-matching objective moment-matching objective in Proposition 2. Both the off-/on-policy objectives can be optimized by viewing the learning procedure as solving a game. More specifically, we consider a two-player minimax game between the student policy and the Q-value functions. To this end, we initialize two small networks of a single-layer MLP to estimate the off-/on her//onpplicy Q-value functions, respec- LM, the Q-value estimate module can be represented as tively. For example in causal/seq off(on) for any action token y E V. This estimates the Q-value furce (v< y) = (h, + Vy ofl()))) T "w
Proposition and the on-policy
function by taking the current t E {0,1..... - 1} hidden step of policy network h," E R (for off(on)) R. with linear next token prediction) combine with the feature vector of the token vy transformation by wy off(on) Rd for (ff(on))policy learning. Here, H represents the hidden size and the additional parameter cost is ( for Q-value estimation. Finally, combining off- and on-policy objectives with a factor 3 E (0, 1), the optimization problem can be represented as follows,
min max f (mo, Jor) + (1 B) (TTO J0z): 01.
=:C(no.for S0g)
where L( Sos: Jax) represents the overall training objective. To minimize the objective w.r. the policy parameters 0, we use a policy gradient approach and derive the policy gradient Appendix A.5, formally represented follows,
for = E
3 E
goff (off(T (T,0)] + (1- B)
E p'muToke
g
(rm(r'
.0]]
a~De
TNT
T-1
stt.
(oll(T,
(T,
Voo (oyl
(10)
( Wge
T-1
(on(t' = log no (Vl 1 (6
where Q,,, ) X IR denotes the empirical Q-value defined in Eq. (21). Besides, we stochastic gradient ascent (SGA) maximize the objective of [( for for) w.r.t parameters of the on on-policy Q-value function Φi and parameters of the off policy Q-value function d2.
Training procedure. The goal is to achieve equilibrium between minimizing the objective w.r.t.
the parameters of student policy 0 O and maximizing the objective w.r.t. the parameters of on-policy and off- off-policy Qaluee functions Φ1, d2 4, formally defined as ming max01,02 [( fer foo)

---

(Eq. (9)). To this end, we use an adversarial training strategy in Algorithm 1, by starting from student model fine-tuned on a dataset Day. In the training algorithm, we iteratively maximize the objective w.r.t the parameters of Q-value functions Saz Ság and simultaneously minimize the objective w.r.t. the parameters of student policy zp- In each iteration of policy updating, we first perform N steps of stochastic gradient ascent (SGA) w.r.t. the parameters of Q-value functions $1, 62. Then, the parameters of student policy are updated by stochastic gradient descent (SGD) with the estimated policy gradient with sampling policy gradients.
3.4 Convergence Analysis
We further provide convergence analysis for the algorithm proposed in $3.3. To deal with the challenges of non-convexity by certain reward structures, the algorithm is expected to obtain an c-accurate stationary point of the policy parameters A, E O satisfying that [ S c. We focus on policy optimization and directly use the optimized off-/on-poolcc Q-value functions in each outer-loop iteration & E (0,1,..., K - 1}. We denote Φ1 (0x) arg max L ( (2x))) = arg max do L [ Soz) as the inner-loop optimized functions and use [(0s) = [(TO, for (+2) (oa(Be) (def. in Eq. (9)) for simplicity in this section. We start with the following standard assumption [45].
Assumption 1. Suppose that the optimized Q- functions and the parameterized policy no satisfy the following conditions:
(i) The uniformly boundness of off/on-policy Q-value functions optimized by Algorithm 1, i.e.,
Il lloo llffzllo S 1.
(ii) The B- B-Lipschitzness and the L-smoothness of the parameterized policy, i.e., for any state- action pair (U M1+1) E y x V at any time step I. E (0, 1},
V log No (e U s1 < B, for any A E O
(11)
V log To, (1t+ilyg)) V log no, (3e+1 3 S L|01 -211. for any 0, O2 E θ (12)
Theorem 2 (Convergence rate of Algorithm to stationary points). Let {Ok]isksK be the sequence of parameters of the policy Th, given by Algorithm 1. Let the learning rate n
2 1-x7
V [1 (T~) Under Assumption 1, we have
Be
1 VK
min OSASK-
E
[LC()) SO
(13)
Proof. See Appendix A.6 for the complete derivation.
Theorem 2 illustrates that the output gradient norm square by Algorithm 1 can converge to neighborhood around zero with the rate of 1/VK. Furthermore, leveraging sufficient number of training iterations ((e-²), Algorithm 1 can obtain an e-accurate stationary point. This leads the following corollary on the computational complexity of the training procedure.
Corollary (Computational complexity of Algorithm 1). We formalize the policy as a softmax
function ne with linear transformation: softmax(0y <.) for any Ver E R. where 0 E R VixH and H denotes the hidden size. Then, to obtain an e-accurate stationary point by Algorithm 1, the complexity of gradient computation is 2 + T (V)).
Proof. See Appendix A.7 for the complete derivation.
Corollary shows that Algorithm has a polynomial computational complexity w.rt -,, N, [V], and T, to obtain an e-accurate stationary point for optimizing the training objective in Eq. (9).
4 Experiments
We consider task-agnostic cattucctionfollowing experiments and task-specific experiments, including text summarization, machine translation, and commonsense reasoning, We compare approach
7

---

Table 1: Comparison with state-of-the- KD methods the cnntrruttio-followwii dataset using fine-tuned OpenLL as the teacher and fine-tuned OpenI as the student. We format the best, the second best and worse than SFT results. The results based GPT-2 are available in Appendix C.I.
DollyEval
SelfInst
VicunaEval
S-NI
UnNT
Method
GPT-4
R-L
GPT-4
R-L
GPT44
R-L
R-L
R-L
OpenlJaaM2-78 (teacher)
58.811.2
32.5
56.7
21.610.2
46.2_0.
22.6_a..
36.3
38.5
SFT (student)
46.8
26.7
40.8
16.3
34.8
17.3_0.2
30.420.4
28.620.3
KD [16]
43.9
22.4
43.5
17.4
33.7
16.4_02
29.3
23.4
Seq [20]
50.2.0.6
26.210.4
46.8
15.8
38.8
18.0_a.6
29.7.0.3
27.8
ImitKD [24]
53.7
25.3
45.020.7
18.420.4
41.7=1.2
19.1=0.2
33.1
28.7
MiniLLM [14]
58.7
28.4
51.8
20.2:0066
44.2-1.1
20.7-0.5
37.410.4
37.5
GKD [2]
$7.611.
27.5
52.411.2
20.9.0.3
45.5
19.3_a.5
36.8
34.8
Li [21]
59.2+
29.5
53.4.1.0
20.8
46.3
20.4_a.3
37.2.0.1
38.2+0.1
Ours
59.8
30.7
54.2
21.7.103
47.8
21.4_
38.7
39.1
with various KD baselines, including: SFT, which fine-tunes the student model on the supervised dataset Day: KD [16]. which uses KL divergence on the supervised dataset Day: Seq [20], which applies SFT to the student model with teacher-generated outputs; ImitKD [24], which uses KL divergence on the student- student--enerated outputs; MiniL [14], which uses RKL divergence with a policy gradient method; GKD [2], which uses IS divergence with on-policy method; and DistiLLM [21]. which uses an adaptive training method for off-polics optimization of a skew KL divergence. Additionally, focus on step-wise distance optimization for KD and compare it with a range of well-known methods, including KI. divergence, RKL. divergence, JS divergence, and TV distance, as discussed by Wen al. [39]. All the reported results are the average across three random seeds.
4.1 Task-Agnostic Distillation
Experimental Setup. We follow the previous works [14, 21] for the implementation of the instruction- following experiment, aiming evaluate the distilled model's ability handle diverse tasks presented in the form of instructions. We construct the training data from databricks- -cks-dolly-155 [8]. where randomly select 15K samples for training and equally split 500 samples for validation and testing. We evaluate the trained model on five cessructtionfolllwviii datasets: DollyEval, SelfInst [36]. VicunaEval [6]. S-NI [37]. and UnNI [17]. Following the previous works [14, 21], we also add the Open WebText [13] corpus, consisting of long-document plain text, for joint training with a language modeling task. This has been shown to effectively improve the performance of instruction tuning [14]. The evaluation metrics include ROUGE-L [25] and GPT-4 feedback with the same prompts as in [21]. More details on experimental setup refer to Appendix B.
Main results. Table 1 illustrates the instruction-following performances. Compared with the SFT baseline, which indicates the student model without KD, KD and SeqKD hardly improve the performances. This indicates that using only supervised datasets or teacher-generated outputs does not benefit the KD of large language models. In contrast, utilizing the student-gemerated outputs with KL divergence [2]. RKL divergence [14], and JS divergence [2] shows effectiveness for KD in the instruction-following task. State-of-the-ar methods [14, 2, 21] tend combine the student- generated outputs with the teacher-gents generated output or supervised dataset to further improve the results of KD. This shows that a mixture optimization of both on-policy and off off-policy objectives can effectively improve the KD performance of large language models on the instruction followwig task. In particular, we use an adversarial moment- matching method and optimize both on- on-poiic and off-policy objectives for KD, thus achieving the best results on five test datasets with both GPT-4 feedback and ROUGE-L evaluation..
4.2 Task-Specific Distillation
Experimental Setup. We evaluated the KD models on three tasks consisting of text summarization, machine translation, and reasoning. For the text summarization task, we follow Ko et al. [21] to conduct experiments on the SAMSum [12] dataset. For the machine translation tasks, we follow Ko et al. [21] to conduct experiments on the IWSLT' (en-de) [5] dataset. For the commonsense reasoning task, we conduct experiments on the Strategy QA dataset [11] with chain-of-thouse augmentations
8

---

Table 2: Comparison with the offte-of- KD methods on text summarization, machine translation and commonsense reasoning datasets. We report the ROUGE-L, BLEU and accuracy for SAMSum, 17 (en-de) and StrategyQA, respectively. We format the best, the second best and worse than SFT results.
SAMSum
IWSLT 17 (en-de)
StrategyQA
Method
TS-Small
TS-Base
TS-Large
T5-Small
TS-Base
TS-Large
TS-Small
TS-Basc
TS-Large
TS-XL (teacher)
52.5
35.2:0.2
64.520.5
SFT (student)
40.6
47.3
49.8
21.3
30.1
33.7
52.4
57.5
60.7
KD [16]
39.2+0..4
46.5
47.4
21.7
29.820.
31.7
49.7
55.3+0.1
59.2+0.5
SeqKD [20]
39.7+0.a
47.7
49.3+0.4
21.2+0.3
29.220.2
32.9
50.620.7
57.5+111
61,5208
ImitKD (24)
41.8g..
48.620.7
51.2auu.
22.2au03
28.7
34.1
53.8gua
59.720.5
61.7
GKI) [2]
42.1
48.2+0.5
51.7.0.1
22.7+0.2
31.240.1
34.7
55.620.1
60.320.5
63.640.3
DistiLLM (21)
42.610.3
49.4
$2.1a0.
22.5
30.8
35.5an.1
56.3an
61.2a07
62.8102
Ours
43.7
50.4
20.7
23.7
32.4
36.0
58.2
62.9 20.5
65.3g0.7
[38]. For all of the task-specific experiments, we use T5-XL [29] as the teacher model and T5- T5-Large/- Base/-Small as the student model. For the machine translation experiments, we employ multilingual pretrained model, mT5 [43], to build the methods. For evaluation, we use ROUGE-L [25], BLEU [27]. and accuracy as the performance metrics SAMSum, IWSLT' 17 (en-de), and StrategyQA, respectively. More details about the experimental setup refer to Appendix B.
Main results. Table displays the performances on three task-specific datasets. Since the original work of MiniLLM [14] does not consider these tasks, we thus do not make comparisons with MiniLLM. The performance trend is similar to the instruct-following results, revealing that KD of large language models for specific tasks also benefits from the combination of on-policy objectives with
outputs and off- oiflpolicy objectives with sacher-generatee outputs or supervised datasets. Additionally, we observe that student models of different sizes all benefit from the KD methods to improve performance. Overall, our approach achieves the best results on all three task-specific datasets for student models of different sizes. This demonstrates the effectiveness of an adversarial moment-matching approach for KD of large language models on specific tasks.
4.3 Analysis on Step-Wiss Wise Distance Optimization
Comparison with distribution matching. We make compar- isons with different step-wise dis- tribution distances with a uni- form formulation of Definition 2, considering the on on-policy, off- policy objectives as well the joint form. Results on four tasks with a default combination fac- tor 3 = 0.5 are shown Fig- ure 2. More instruct-following results are available in Appendix C.2 and results with different val- ues of off-/on-policy combina- tion factor are available in Ap- pendix C.5. Compared with the KL divergence, RKL divergence, IS divergence and total variation distance, the proposed moment-matche distance achieves the best results under both the on- on-policy and off- off-policy training objectives, which shows that the proposed noment-matching approach is effective for KD of large language models. Besides, we observe that using a joint objective of both on-policy and off-policy can further significantly improve the performances. This shows that both on-policy and off off-policy moment-matchin objectives contribute to the minimization of the imitation gap and can thus benefit the KD of large language models.
12
(a) DollyEval.
(b) SAMSum.
(c) IWSLT' 17 (en-de).
(d) StrategyQA.
Figure 2: Performance of difference step-wise distribution dis- tances.

---

Ew
S47
241
10
100
241
1.2
CAI
24
LMN
-23
22
029
114
211
341
1.
141
11:
412
141
6000
2000
(a) Training loss and d against training step.
d
(b) On-policy moment-matchine distance d'M on the test sets.
(c) Off-policy moment matchign distance d on the test sets.
Figure Adversarial training procedure for optimizing the on-policy and off-policy moment- matching distances dMM d on the instruction-followingg dataset.
Adversarial training procedure. We present the training loss and moment- matihiingttiii distance against the adversarial training steps. As depicted in Figure (a). the training loss initially increases within the first 0-1,000 steps, indicating that initially, the Q-value functions are stronger than the policy in maximizing the loss function [( fe. faz) in Eq. (9). Concurrently, the policy gradient method contributes to minimizing the training loss, which eventually converges to a much lower stable value. Additionally, both the on-policy and off-policy and eventually reach a low value with only minor fluctuations. For more results and details on experimental setups, please refer Appendix C.3.
distances don and dMM decrease
Moment-matching distance optimization. We further illustrate the on-policy moment-matching distance d and the off-policy nooment-matching distance doft (defined in Definition 3) optimized by different step-wise distances in Figure 3 (b) and (c), respectively. Interestingly, we observe that the total variation (TV) distance obtains the second-best results average for both on-policy and off-policy distances. This finding suggests a similarity between the formulations of TV distance and mantenttmatching distances to some extent, supported by the theoretical result of Theorem test sets, our approach effectively optimizes both on-policy and
Across all instruction-follont off- off- off-policy moment- matehhig distances more than other step-wiss wise distribution distances used in KD, including KL divergence, RKL divergence, JS divergence, and TV distance. This observation also underscores the effectiveness of our policy gradient methods. Extensive results on the task-specific datasets are available in Appendix C.4.
5 Conclusion
In this work, we investigated a moment- moment-matching approach for knowledge distillation of large language models. Specifically, we formulated knowledge distillation from perspective of imitation learning and derived both on-policy and off- off-policy bounds for the imitation gap between the teacher model and student model via moment-matching distance. Additionally, we proposed an adversarial training algorithm to simultaneously estimate and minimize the joint objective of on-policy and off-policy moment-matching distances. In experiments, we evaluated the proposed algorithm on four cetructtionfoolowinn datasets and three task task-fpec datasets, comparing it with range of tfte--of-hhearr KD methods as well as four well-studied step-wise distribution distances for KD of regrerregees models. Results demonstrate that our approach can effectively leverage the policy gradient method to optimize the noment-matching distance and achieve the best results across all datasets.
Limitations and future work. The proposed adversarial training algorithm requires additional computational steps for the inner-loop gradient ascent, which may result in increased time complexity. Moreover, the proposed approach necessitates auxiliary networks to build the Q-value functions, which may incur additional memory costs. Besides, the experiments are conducted with limited LLM architectures, such as OpenI. LaMA and T5. Therefore, in future work, we aim to enhance the time and memory efficiency of our approach, and evaluate the proposed approach on wider range of architectures.
10

---

Acknowledgements
We thank the anonymous reviewers for their helpful comments and suggestions. This work was supported by SI-TECH Information Technology Co., Ltd.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08877, 2023.
[2] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self self-generated mistakes. The Twelfth International Conference on Learning Representations, 2024.
[3] James Bagnell, Sham M Kakade, Jeff Schneider, and Andrew Ng. Policy search by dynamic programming. Advances in Neural Information Processing Systems, 16, 2003.
[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
[5] Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuitho Sudoh, Koichiro Yoshino, and Christian Federmann. Overview of the iwslt 2017 evaluation campaign. In Proceedings of the 14th International Workshop on Spoken Language Translation, pages 2-14, 2017.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[7] Kamil Ciosek. Imitation learning by reinforcement learning. In International Conference on Learning Representations, 2021.
[8] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction- tunee 1lm, 2023.
[9] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023.
[10] Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama. URL: https://github 2023.
[11] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361. 2021.
[12] Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: human-annotates dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, 2019.
[13] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007.. http:///ww.....aaaoaa.e////tttte hot.ppeen/eeeeextoorpuu..s 2019.
[14] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference Learning Representations, 2024.
[15] Yongchang Hao, Yuxin I Liu, and Lili Mou. Teacher forcing recovers reward functions for text generation. Advances in Neural Information Processing Systems,
2022.
11

---

[16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1503.02513,
[17] Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409-14425
2023.
[18] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267- 274, 2002.
[19] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wxXiv20011.0866/ 2020.
[20] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2016.
[21] Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Youn Yun. Distillm: Towards streamlined distillation for large language models. In Forty-first International Conference Machine Learning 2024.
[22] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01664 2020.
[23] Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less more: Task-aware layer-wise distillation for language model compression. In International Conference Machine Learning, pages 20852-20867. PMLR, 2023.
[24] Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6121-6133, 2020.
[25] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Oul, pages 74-81, 2004.
[26] Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In 9th International Conference on Learning Representations, 2021.
[27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 31 311-318. 2002.
[28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):1 2019.
[29] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21
2020.
[30] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[31] Bharath Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf. and Gert RG Lanckriet. On integral probability metrics,) ) phi-diiergenncs and binary classification. arXiv preprint XXXiv:0901.2696 2009.
[32] Gokul Swamy, Sanjiban Choudhury, Bagnell, and Steven Z. Wu. Sequence model imitation learning with unobserved contexts. Advances Neural Information Processing Systems, 35:17665-17676, 2022.
12

---

[33] Gokul Swamy, Sanjiban Choudhury. Andrew Bagnell, and Steven Wu. Of moments and match- ing: game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022-10032. PMLR, 2021.
[34] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followig llama model. https://gtt gitggtthhbb.... com aomm/attu-laa 1ab/ attaaa//ssafoooeeeeeeeeeeeessast ord_ alpaca, 2023.
[35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023.
[36] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- seff-generated in- structions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), pages 13484-13508, 2023.
[37] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Supper-naturallinsttccioosss Generalization via declarative instructions on 1600+ nip tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-51094 2022.
[38] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
[39] Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10817-10834, 2023.
[40] Qingyang Wu, Lei Li, and Zhou Yu. Textgail: Generative adversarial imitation learning for text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14067-14075, 2021.
[41] Taiqiang Wu, Chaofan Tao, Jiahao Wang, Zhe Zhao, and Ngai Wong. Rethinking kullback- leibler divergence in knowledge distillation for large language models. arXiv preprint arXiv:2404.02657, 2024.
[42] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint 2xX1v:2402.13116, 2024.
[43] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: massively multilingual pre-trained text-to-text trans- former. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, 2021.
[44] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
[45] Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58(6):3586-3112 36125 2020.
13

---

A Proofs
A.1 Proof of Proposition 1
Proof. Similar to the proof of Performance Difference Lemma (PDL) [18, 3, 33], we have
J(T.) (TO)
E am
Σ (y (041)
E
[x))
xape
E
y
VV0 V~ (V
- E
[x))
g= ror in
a~pe
7' (1(Ugo + 7VV00 (Us(41) V (3)((1))))
E
vosale
(1(y + PH (y
- V (y)())))
i
E
(Q (V)
V
E
(T0(
E y
( sen wi]
ro in
1=0
sup E lEfo
Σ
[f S.. 8)
where (i) follows from Bellman equation and noting that the transition probability T( Bc+1) is deterministic in an auto-regressive text generation problem. This completes the proof.
A.2 Proof of Proposition 2
Proof. Similar to the proof of Proposition 1, we have
J(T.) - J(TO)
E nr 1 l0
y rr st Mc+1)
+ E ar-pa
[x)]
E
(W (((g018441)) v* ((z))))
E [v=((x))
aropa
yongle
7' (Uz)) - (r(y + vV= (1s++))))
E
s pm rivtgla a
(v* (v (r(U + B T( (U
E
(VT (Uz))) (U
(y - Q (U
E
(-
<
sup FEFQ.
E
Σ 7' 1=0
E
[[ f Me+1)) ((
wi
14

---

which completes the proof of Proposition 2.
A.3 Existing Step-Wiss Distribution Distance for Distillation
Definition 4 (Step-wise distribution distances for distillation (39]). Following Wen et al. [39], define four groups of well-studied probability distribution distances as follows,
Total variation (TV) distance. The token-level TV distance between the probabilities of teacher policy n. and student policy no given the current state ygt can be defined by the Ey-norm as follows,
TV(To(-lyg)) n. (. = 2
Σ NEV
T. (vlUg)) - To(UVEe
(14)
Kullback-Leible (KL) divergence. The token-level KL divergence between the probabili- ties of teacher policy as and student policy Te given the current state y can be defined as follows,
To (vlye))
(T = Σ Tu log BEV
(15)
Reverse Kullback-Leibler (RKL) divergence. The token-level RKL divergence be- tween the probabilities of teacher policy næ and student policy ne given the current state boldsymbolyge can be defined as follows.
To T
(KL((T((IW<))) T. (1)) = Σ Te(y)ye)) log (.
(16)
(y vE)
BEV
Jenson-Shannon (JS) divergence. The token-level JS divergence between the probabilities of teacher policy n. and student policy ne given the current state boldsymboly can be defined based on the KL divergence and RKL divergence follows,
To + T
Tr + To
JS(To(-IWe)) (- = KL((T,
RKKL(T
(17)
2
2
A.4 Proof of Theorem
Proof. We first derive upper bound for the on-policy nomentt-atchiin objective of Eq. (3). Set Fo. = [f: | I 1}, and by the definition L(mo, J') in Eq. (3), we have
(moo f)
sup
=1
[f f(Uge.v) 1
sup
E mopp i-agle
E (-v.(1lVg))
fllifll S1
sup
E miggle
E (lvgal
[f(yse
E
[F(yge:))]
f:llfllos
1=0
y
Then, we have
sup
Con (To.
1:11/1|a
]
E moo Toun gign im
sup
E
[f w)1
[f [f w)
f:11fll $1
(1
(181)
E
( TT)) (Def. & 4),
Tw im
1=0
15

---

where (i) follows from Jensen's inequality, (ii) follows from [31] and (iii) follows from the definition of TV distance.
Similarly, we can bound the off-policy version of Eq. (2) as follows,
sup f
E
L
(To. [)
Σ
n.. (ulv no(ulss))
rurelm
VEV
2d (mo, n.) (Def. 4),
which completes the proof of Theorem 1.
A.5
Derivation of Policy Gradient in Eq. (10)
Based on the definition of training objective in Eq. (9), we have
V (o f fox) = 3V C (No, fo,) + (1 B)V f0x)
(18)
Based on the definition of Coll (TB, fór) in Eq. (2), we have
cfl (
[F( ]
E
(lvs))
Tirte ja
1=0
E
[f
y~uT 11g)
(19)
T
E
To(ylye)) log Voggooll (Uge.))
(=0
NEV
E
v log (oll (y
Then, based on the definition of L (TO. fea) in Eq. (3), we have
V Lom (Te, f
[Fo - for (00
I
(1bg))
t=
T-1
T-1 xt
Σ 7 V'V log (o
E (-lns-)
[oo - S (y
(20)
where (i) follows from standard derivation of gradient policy (c.f. [22]). For simplicity, set
T-1
Q1,, (H vc-1)
Σ p
[f (y v)] for (H Bo+1)
(21)
(y-Ta(-Vee Bge)
the empirical Q-value given any draw of trajectory ne wo x, Pa in Eq. (20). Coming back to Eq. (18) and combining with Eq. (19) and Eq. (20), we have
VL(TO, for foz)
v log Moggolyl (
amia
y~ Dgd
T
+ (1 - 3)
E anip
Σ
y v'v log (e
roo im
0=0
16

---

Then, using the law of iterated expectations, we obtain the final formulation of policy gradient,
VL(TT, Sor f02)
R aerpPe
E
VVlo (o (y wi]
y Mgde
(1
E TloTTp
log 1/ ( (D'snn Bhrn))
t=0
which completes the derivation of policy gradient in Eq. (10).
A.6 Proof of Theorem
-3 (T. (1 B) Pm; T. |2, T' no ar, where L(0) = [( J S02) (def. in Eq. (9)) denote the objective w.r.t. the policy parameters 0 given any off-/on-policy Q-value functions So.
Lemma 1. Let [())) = -
8) denote the empirical policy gradient
given any trajectories
and foz. Then, under Assumption 1, we have || [V]O))]| - Be with
8(1 )B + 2(1 3) 7
Bc 1 1-7
(1 71-
Proof. By triangle inequality, have for any x pm; 7 Th [2, T' nol
|VL(1 s B (T, + (1 3) (on (T',
(22)
By the formulation of off off-policy gradient || (r, (r,0)l in Eq. (10) under the condition of optimized off- off-policy Q-value functions fo, by Algorithm 1, we have
| (T. 01
Σ
E (-
V log (vl (y
By Jensen's inequality, we have
| (T, 011
E v-
Polgoolllussll
Sox (w
By Assumption 1, we have
T-1
B(1 (1
golt (T. 0)
S S
Σ
7
(23)
1-7
Similarly, we can bound the on-policy gradient (7,, .)1l by Jensen's 's inequality follows,
| 0)| s T-1
| log To (v w @ (y M(+1)
Based on the definition of 2rog (Uer V1+1) in Eq. (21) and by Jensen's inequality, we have
@ (Uge Me+2) =
[oo y)] for (U (V )w+1))
E
(M w)1l
|s Vo Se w+
your (luger)
Then, by Assumption (i) that 11 1, we have
T-1 <
T-1
2(1 - TT)
| (0s.))442))
Σ t'ut
1 1-7
t'=0
17

---

Thus, we have
2(1 - (7.0)
(24)
(1 7)
Coming back the bound of [|V_(() in Eq. (22), we combine it with Eq. (23) and Eq. (24). Then, we have
|V((0) - B(1-7")
2(1 (1--8)(1 (1
+
1-7
(1-
Bc
which completes the proof of Lemma 1.
Lemma Under Assumption 1, the abjectivee C(0) is L smooth such that for any 8 E O,
((0) - C(0) (V20) + Lc - -
with the constant
L1 3 (1 (1- + L))
2(1 2(1 31) 31
yB²
1 1-7 +11
(1 7)2 1 +
Proof. Under the definition of policy gradient in Eq. (10), for any 01. B2 E O, we have
LL())) vL((2))l
(goft - (T,)((, (T,
B
+ (1 B) (ErimToy (m [TTH
EBI [ goo (T2, 022) 1)
Then, by Jensen's inequality and triangle inequality, we have
[VV
VL(02))
< E aerpPo
B
Goff (T, (T, - 02)1
-uutt (T, (r.0:)-gun
Toumaaa a
1
(25)
+ B1
E En
gom (TT( 01)
- E B la
gam (T))))))))
02)
Based on the definition off-policy gradient in Eq. (10) and using Jensen's 's inequality, we have for any w pay T TT|
1
2off (T, - goft (T. 02)
E
V log To, (vlVs.)fes (y
V log ne, (vWgg)) ( (M w)1
yomto, (1))
(26)
18

---

Then, by triangle inequality, we have for any E f0,1,..... T 1/.
E
V log na, (vlv (v
E ynsto Ugg
V log me, (vly<:)) (y (Uge. w)
Σ NEV
To, (uluer) V Vlog To, (e (v To, ( log To,
(o (u
Σ NEV
To, (ulv no, (vlue:) (vlv So (U w\|
+ muz (vly log Te, (vlugr) log To, To (ulve:))
(27)
By Taylor expansion of (e) have that for any t {0, 1, ..... T 1}.
Te, (vly ne, (u = (01 - T2) log V
0 00 V (
< 021 B ()(
where is vector lying between 0₃ and 02. i.e., there exists some À E [0, I] such that A01 + (1 2)62. Then, combining with Eq. (27). yields
E youtor s.)
log To, (ol (n
E mo
V log ne, (wll (v
Ba ne (e) - 021l + ne. (11 - 021
BEV
=B² 01 82ll
Then, combining with Eq. (26) yields
(1 - )BB + +L)
g (T, go (T, ( 02) s SB² + L)] 02l
Σ n'
10, 02
1-7
(28)
In addition, we can first bound In using Jensen's 's inequality and triangle inequality,
h =
E B
(T. (T, 01)
B notte a
Gon (T2 (2, O2)
2 (15e01+11)
II n==0
no, (Mr+1ly3 V log no, 104411y30
II
To, (ye +1 13 log Te, (ye (2e
dy dy dy
By triangle inequality and the boundess of Q1a,, (H v1+1)l < 2(1--")
1-7 we further have,
2(1 2(1 -(1-
12
(vr+1lyse)
Ta (Vr+1
V log Te, (20+113
I #=
Te (y (gge) V log ne, (1++1l8ge) V log ne, (10+1 dy ... dy ... dy
(29)
19

---

By Taylor expansion of I P==0 To Te(Ve+1ll we have
1-1
[-1
TT, (We++ 11s5))
II
Th (W0+1
P==D
1-1
(01
02)
V Viy (V (gge)
To +1 (mm))
P==
t-1
-
02l
V log F Ur s
(1er+
t'=0
#²=0
02l
t. B
T TV +1
#"=0
where & denotes a vector lying between 01 and 02. i.c., there exists some such that @ A01 + (1
X)02. Coming back to the boundness of 1z in Eq. (29), we have
12 5 2(1 2(1-2") T-1
To (ve+ (ve+1l ) 110, - -22ldyy ... .dy.. ..dy.
Σ
Bar !"=)
Ta(ue +1 () +1 + L II
1-7
t-=0
2(1 - - T-1
2( -
7B2
Σ
TB L) -221
1- +14 100 0211.
1 1-7 t=
(1 7)2
(30)
where the last incquality follows from the fact that
T-1
7 TTT + (T - T+1 7 -- - + 11
7(1 7(1 - 7?) (1 -
Σ ty
71
(1-7)2
Then, combining Eq. (25) with the boundness of L in Eq. (28) and the boundness of 12 in Eq. (30), obtain the final bound of
[V((0) vL()2) -1--77 (B² (1-7")( L) + (1 - 9) 2(1 - 7222 22)²
(1-7 + 1)) 110 - 021 021l
BB²
(31)
1-7
(1 - 7)2 7}²
Lr
Next, we have for any 0, 0 E O,
((0) - C(8') - (V (VC(()) 0 8')
< [C)) L(0') (V2(0), 0')1
(71( + +1(0- - 0')), 0 O')dt (VC(0)) 0 -0)
(0.1)
|VL(('' + ((0 - (') VC l'lldt
(0.1)
Then, by Eq. (31) and set 01 = 0' + t(0 - 0¹) and 02 = 0', we have
C(0) - C(8') (VC(8', 8*)
/ (0.1)
Le 0 011 , Lelle 12,
which completes the proof of lemma 2.
20

---

We prove Theorem as follows.
Proof of Theorem 2. Let 0r. 0z+1, t {O.1,..., , 1} adjacent parameters of policy ne
Th.s1 given by Algorithm 1. Then, using Lemma 2 by setting 0 0x+1, O' 02 for any k {0, 10......,4 1}, we have
(( Sor ( (02(0))))
1 fo (ox) (VL(T0, (9k)) (o2)))), 0k) + T. T.c10k+1 0xl1
Following from the updating rule 0x+1 02 10--VVLLC ((Ok) and Lemma I, we have
11x+1 0&1 //1 m/|l <Be
Then, we have
L( 11) (02+:)) (os(0o-)))
SL(TO sor(ow): Sos()))) (VL(To, (o100))))
We introduce probability measure space (2, P) and then 0g θ, k E (0. 1,.., K can be viewed as random variable on it. Let fo follk))ssssssss denote sequence of increasing sigma-algebras such that o00) (01) C 0..00000--0
F, we define the conditional expectation
(())(0)) of (0x)) as
E| 2[V2((e) ((0)) -Egwp [ -3,, In (T, 0x) + (1 (1-8) NTo ( 0b)
=VL(TT (0x) foon):/oziii foz(")))
where the second equality follows from the unbiased estimation property in Eq. (10). Then, taking the conditional expectation, we have
1 SL(Ta, Jo100) - - (o1)))) +²² " BBLc
Taking total expectation, rearranging the terms and making average on {0, (0.1.....,4 K 1}. we have
-1
K
BB for + 1
k=0
(L(TTO (o)( fos(0.) -
11 I.c
B.C.
2(1 77)
<
m1 K 2 BB
Let Br V (I-7)KLE and denote L(Ox) L( (. for simpilicity for any l
2 /
1-77
(0. 10.1...., K 1}. then we have
(1 (1-T))L =o - (V (32)
min oSkSK-1
[|V
Σ
E [IVC(W))
2
K
(1 (1-7)K
21

---

A.7
Proof of Corollary 1
Proof. Let the convergence rate in Eq. (32) satisfy that 2Bc
(1-9 (1- 1,-,,K
S then we have
K> 4(1 B(117)BBLC
which indicates that when the iteration number of policy updating satisfies K = O(e it can reach an e-accurate stationary point to optimize the objective Eq. (9), such that
min K
E [[VC(()) <
For simplicity, we define the policy as softmax function with a linear transformation of yst R with 0 E IRI Formally, for any trajectory т and any timestep E {0, 1..., - 1}. we have the probability of any y V.
exp(0,y5s)
To [o(
(33)
e exp(0
In the following, we will analyze the computational complexity in each policy updating iteration. First, we find that each inner-loop step Q-value function updating has gradient computation complexity of (H) given the linear formulation of Q-value functions. Accordingly, N inner-loop steps each policy updating iteration have a computational complexity of ((T)V)))) Second, for policy gradient computation, since computational complexity of V log To (yly is O(V)H). the computation complexity of policy gradient computation is ( VI + (VD) Overall, the total gradient computational complexity is (Oe (N + Corollary 1.
+ [))), which completes the proof of
B Experimental Setup
We use NVIDIA A40 GPUs with 40GB RAM to conduct all the experiments.
B.1 Instruction-Follovii Experiments
Base models. We conduct experiments both GPT-2 [28] and OpeAL.LMMA [10]. For the GPT-2 experiments, we use GPT-2 XL with 1.5B parameters to construct the teacher policy and GPT-23 with 117M parameters construct the student policy. For the OpenLLaMA experiments, we use OpenLLaMA-7B with 6.7B parameters to construct the teacher policy and OpenLLaMA-3B5 with 2.7B parameters construct the student model.
Training details. We fine-tune the OpenLLaMA-77 teacher model and the OpenLLaMA-33 student models the corresponding supervised dataset with 10,000 steps. The GPT-2 teacher and student models use the fine-tuned checkpoints by Gu et al. [14]. For the implementation of compared baselines,
the code by Ko al. [21] and re-run the results. The optimization protocol for KD training largely follows the previous work [14, 21]. In particular, we search for the learning rates among finite for each experiment obtain the best result. The batch size for each experiments is seleted to make full use of the 40GB RAM of an A40 GPU. To handle the adversarial training, we choose the number of adversarial steps K = 5 and the adversarial control factor the development experiments. We use a default off-/on-policy combination factor 3 = 0.5 for main experiments while exploring other values for analysis. The hyperparameters for training are listed in Table 3.
0.1 based on
22

---

Table 3: Hyperparameters for instruction-followiing exp.reeneeiinneeeeeeeeeeee
Hyperparameter
GPT-2
OpenI_LaMA
Max. Step Size (K)
10,000
10,000
Inner Step Size (N)
5
Batch Size (per GPU)
{8, 16, 32) 0.1
{4, 8)
Dropout Rate
0.1
Controlling Factor (a)
0.1
0.1
Discounting Factor (?)
[0.90, 0.95, 0.99)
(0.90, 0.95, 0.99)
Combination Factor (3)
{0, 0.5, 0.9, 0.99, 1.0)
(0, 0.5, 0.9, 0.99, 1.0] {5e le Se 5) 500
Learning Rate (7)
(Se 1e Se
Warmup Steps
1,000
Weight Decay
le
le 512
Max Seq. Length
512
Sampling (top-p)
1.0
1.0
Sampling (temperature)
1.0
1.0
Evaluation
Greedy Sampling
Greedy Sampling
#GPUs
2
4
B.2 Task-Specific Experiments
Base models. For the text summarization and commonsense reasoning experiments, we use T5- XL with 2.8B parameters to construct the teacher policy and construct the student policy with T5-Large' (770M parameters), T5-Base* (220M parameters) and T5-Small (60M parameters). For the machine translation experiments, mTSS-Large/-Base/SSallll to construct the student policy.
use mT5-XL [43] construct the teacher policy and
Training details. We initialize the corresponding teacher and student models using 10,000-step-fine- tuning checkpoints on the SAMSum dataset, 1.000-step-fine-tummee checkpoints on the IWSLT' (en-de) dataset and 5,000-step-fine-tning checkpoints on the StrategyQA dataset. We largely follow Ko et al. [21] to set the hyperparameters for training. In particular, we search for the learning rate from preset range obtain the best result for each baseline and our method. The batch size is selected to make full use of the RAM of GPUs We use a relatively larger maximum number of training steps for IWSLT' 17 (en-de) experiments to satisfy sufficient convergences for the machine translation task. We use beam search for the evaluation the IWSLT' 17 (en-de) dataset.
Table 4: Hyperparameters for three task-specific experiments.
Hyperparameter
SAMSum
IWSLT' 17 (en-de)
Strategy
Max. Step Size (K)
10,000
80,000
3,000
Inner Step Size (N)
5
2
5
Batch Size (per GPU)
[16, 32, 64)
(16, 32, 64) 0.3
(16, 32, 64) 0.1
Dropout Rate Controlling Factor (a)
0.0
0.1
0.1
0.1
Discounting Factor (7)
[0.90, 0.95, 0.99)
(0.90, 0.95, 0.99)
(0.90, 0.95, 0.99)
Combination Factor (3)
{0. 0.5, 0.9, 0.99, 1.0) [Se-5, 1. Se
(0, 0.5, 0.9, 0.99, 1.0] [le Se- le-3) 4,000 1e-4
[0, 0.5, 0.9, 0.99, 1.0)
Learning Rate (y)
1, 1)
Warmup Steps
1,000
300
Weight Decay
1e-2
1e-2
Max. Seq. Length
1,024
512
1,024
Sampling (top-p)
1.0
1.0
1.0
Sampling (temperature)
1.0
1.0
1.0
Evaluation
Greedy Sampling
Beam Search
Greedy Sampling
#GPUs
2
hattg///buuggnnaaeee.ooooog///// http::///bugginggaee..ooooog/// http://uuggunggaeee.ooooo/////// small
23

---

Table 5: Comparison with state-of-the-ae KD methods the instruction-follovee dataset using fine-tuned GPT-2 XL (1.5B) as the teacher model and tinettuned GPT-2 (O.1B) as the student model. We format the best, the second best and worse than SFT results.
DollyEval
SelfInst
VicunaEval
S-NI
UnNI
Method
GPT-4
R-L
GPT-4
R-L
GPT-4
R-L
R-L
R-L
GPT-2 XL (teacher)
45.5
28.2
34.711.6
14.320.2
32.7
16.2
27.6.0.3
32.2
SFT (student)
2..8
23.4
20.2+0.7
10.340.5
17.8
14.640.4
16.1
18.2+0.6
KD [16]
40.5
23.8+0.3
18.0+1.0
12.340.2
17.240.7
15.2+0.4
20.8+0.5
22.5+0.3
SDqKD [20]
29.8
24.240.2
18.240.8
11.610.4
18.240.7
15.5
15.540.6
20.1
ImitKD [24]
26.440.6
22.740.5
18.2+0.5
11.5
18.640.4
14.5
18.2+0.3
21.840.4
Minil.LM [14]
30.2+1.2
24.3+0.3
20.5
13.2++0.3
20.5
18.5+0.3
22.7+0.3
23.540.2
GKD [2]
29.2
23.640.2
20.7+0.6
12.7
20.22+0.6
17.7+0.2
25.1+0.3
25.940.1
DistiL.LM [21]
31.2.0.4
25.210.4
21.740.5
12.5
22.5
19.2.0.5
27.7.0.2
27 27.610.4
Ours
31.7
26.1
22.7.0.0
14.2.0.3
23.6.40.8
20.5
28.6
29.9
(a) EvllyEval.
(b) SelfInst.
(c) VicunaEval.
(d) S-NT.
(e) UnNI.
Figure 4: Performance of difference step-wise distribution distances on five instruction foollooii- datasets using OpenLLaMA-7B - OpenLL
C Additional Results
C.1 Results Based on GPT-2
In addition to the experimental results based OpenI.LaMA for instruction-following tasks, also conduct experiments based on GPT-2. Results are illustrated Table Compared with current state-of-the-art KD approaches, our method achieves the best results on five datasets with both GPT-4 feedback and ROUGE-L evaluations.
C.2 Comparisons on Step- Distribution Distance
Figure 4 and Figure 5 illustrate performance comparison with well-studied step-wise distribution distance, including KL. RKL, JS divergences and TV distances. Results show that the optimization of proposed moment-matching objectives outperforms other step-wise distribution distances via either on-policy distillation or off-policy distillation. Besides, jointly using on-policy and off-policy moment-matching further improves the performances and achieves the best results on five instruction- following datasets with KD from the OpenL.LaMA-78 to LaMA-BB model, and achieves the best results on three task-spec datasets with KD from the (m) to T5m T5-Basc model.
24

---

(a) SAMSum.
(b) IWSLT 17 (en-de).
(c) StrategyQA,
Figure 5: Performance of difference step-wise distribution distances on three task-specific datasets using (m)T5-XL + (a))5-BBsee
mainn l or polyy sstance 05
-22
poipety dstanee 42
0.3
1.0
-20
1.8
1.4
Dlaua not Coua
0.7
0.8
0.7-
0.6
0.5
1.0
3.6
05
0.4
2002
4000
0002
8000
10002
2000
4000
0002
0300
13000
Step
Step
(a) Instruct-following.
(b) SAMSum.
training Icau
trasiin los
pripely www.ce
20-200ly dxtance d2
1.
12
11-
1000 (trai))
Dalancc tonad
18
9.6
07-
2.4
05
5.25
10003
70000
30003
20200
30003
252
503
750
1000
1250
1500
1753
2000
Stop
Stop
(c) IWSLT' (en-de).
(d) StrategyQA.
Figure Training loss and d doff against training step on four datasets.
C.3 Adversarial Training Procedure
Figure illustrates the training loss and on-loff-policy moment-matching distances against the training steps the instruction-followee training losses on four datasets have a similar trend, increasing at the beginning and then converging to relatively lower level. The trend of loss function aligns with the characteristics of adversarial training with gradient descent ascent. contrast, both on-policy moment-matching distance d and the off-policy moment-min moment-matching distance dolf reduce the number of training steps increases, which shows the effectiveness of our adversarial training approach for moment- matchint.
dataset and three task-specific datasets. We can observe that the
25

---

d2) on five test sets
245
1:22
45
3.0
40
KKL
1.81
1.26
2:37
2:18
1.02
25
35
2.11
4.52
3.47
332
2.43
1.25
-0
15
2:42
2:13
2.42
143
2.0
25
1.5
20
-5
Ours
0.39
1.85
2.11
9.bb
0.2
1:24
Ours
0.75
1.45
111
5.72
0A2
0.35
30
Deaggbol
Setter
Wizaffool
6:08
Und
Sefffal
6:88
und
Avtrange
(a) On- On-policy moment-matching distance.
(h) Off-policy moment- matthing distance.
Figure 7: Moment-matching via distributiion-matcinng the dnstruction-followiiin dataset.
121
1.0
(a) On-policy motenng (b) On-policy motenn-matching (c) On-policy moment- matching
distance SAMSum.
distance IWSLT' 17 (en-de). distance StrategyQA.
zio
21
14
24
10
10
20
1.
2.1
L
Lage
(d) Off-policy moment-matching (e) Off-policy moment-matching (f) Off-policy comentt-machinn distance on SAMSum.
distance on IWSLT' 17 (en-de). distance on Strategy
Figure 8: Moment- matnhing via Instriuutionmmacciin on three task-specific datasets.
C.4 Moment-Matching via Distribution Matching
We investigate how the distribution matcohig methods via KL, RKL, JS divergences or TV distance can optimize the moment-Ne moment-matching matching distance Figure 7 and Figure 8. Results show that the proposed adversarial training algorithm is more effective in minimizing the moment-matching distance than the distribution- mattutiiinn-mmiiinngn methods.
C.5 Analysis on the Off-/On-Policy Combination Factor 3
We study the impact of on-policy and off-policy objectives with the combination factor (0.00, 0.25, 0.50, 0.75, 1.00} in Eq. (9), which denotes a linear combination coefficient of the on-policy and off-policy objectives. We observe that if 3 0.00, only the on-policy objective contributes to policy learning. As it increases from O to 1, the influence of off off-policy objective increases while that the on-policy objective decreases. Finally, when 3 1.00, only the off-policy objective contributes to policy learning. We conduct experiments across four datasets. Specifically, we evaluate ROUGE-L for OpenI.LaMA2-33 on the DollyEval dataset, ROUGE-L for T5-base on the SAMSum dataset, accuracy for T5-base the IWSLT' 17 dataset and accuracy for T5-base
26

---

Table 6: Effects of the off-/on-policy combination factor 3 four datasets.
0.00
0.25
0.75
1.00
Evll
28.8
31.2
30.7
29.8
27.4 27.4=0.4
SAMSum
48.
50,5
50,4
51.2.00.
48.7=a.1
IWSLT" 17 (en-de)
30.7+0.1
31.7=0.6
32.4=02
33.2+0.2
31.2-0.2
StrategyQA
59.710.4
61.410.2
62.910.4
62.710.4
60.8
the StrategyQA dataset. Results in Table 6 show that combination of on-policy and off-policy objectives outperforms using either on-policy or off-policy objectives only across four datasets.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: The claim of contributions in the abstract and introduction have been fully reflected in the sections of Methods and Experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes].
Justification:
Limitations are discussed in section of the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
27

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed
penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes].
Justification: Complete proof's for the theoretical results are available in Appendix A. All proofs are based on Assumption
Guidelines:
The answer NA means that the paper does include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: Detailed experimental setups such as datasets, models and hyperparameters used in implementing proposed algorithms are all described in detail. See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
28

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: All the datasets used in this work are publicly available. The code and implementation details are released at this GitHub URL: https://gi com/ j waweeenwweslaake//eeeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr http:/guddss//ssee
httpammsssiooooccyyo..oo.c for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips.
htt.:/gudddeeeeeeeumiiiiiee www.eooodieisiioooooooooooooo www.iiissssiiiiiiiio.oo.. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]..
Justification:
We provide the details of experimental settings in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
29

---

Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: All experimental results have error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes].
Justification: Available Appendix B. Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttp:://neurpps http://wrrr....cccpppiip/t/tttta cc.ppublllchhhiccccuueeeeeee inness??
Answer: [Yes].
Justification:
The research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
30

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: Our work mainly focuses on algorithm design and performance improvement, which has no relationship to societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification:
The paper has cited the original papers that produced the models, code packages datasets.
31

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
32

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
33


Reviews:
[{"review_id": "OeGy8W4va1", "reviewer": "Reviewer_M71H", "summary": "This paper proposed an adversarial moment-matching approach for knowledge distillation of LLM. The idea is to reformulate the knowledge distillation from an imitation learning perspective and derive both on-policy and off-policy bounds for the imitation gap between the teacher and student models. The authors proposed an adversarial training algorithm to estimate and minimize the on-policy and off-policy moment-matching distances. The moment-matching distance is evaluated by the value function and the student is updated using policy gradients to minimize this distance. Experiments on instruction-following and task-specific datasets show that the proposed approach outperforms other knowledge distillation methods.", "questions": "Overall, I think the idea is novel. However, as authors pointed out in the limitation part, the required time/memory cost/training efforts can assumed to be high.  I would recommend have a comparison here with other distribution matching methods or knowledge distillation methods.\n\nSee the weakness section for my questions.", "limitations": "See above question and weakness sections.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is novel to reformulate the knowledge distillation as a moment matching problem, where the matching distance is evaluated by the Q-value function.\n\nThe authors derive the imitation gap bound for both on-policy and off-policy setup, and optimize the gap to achieve the knowledge distillation.\n\nThe proposed method demonstrated good performance on both instruction-following and task-specific datasets. The seven baselines are either distribution matching based or supervised finetuned.\n\nI found the connection between the proposed moment-matching approach and distribution distances matching interesting. Specifically, the authors show that minimizing the total variation distance can achieve sub-optimal results for the moment-matching bounds", "weaknesses": "1. I would recommend having ablation studies and analysis of the impact of on-policy and off-policy objective. And analysis how each of them effect the overall performance.\n\n2. Solving Eq.(9) is involving optimizing the minmax problem. (a) First, the optimization requires additional computational steps for the inner-loop gradient update. How expensive is the computation? such as the time/memory cost (b) Is the optimization robust with respect of hyperparameter changes like $K$ and $\\alpha$.\n\n3. The method requires an auxiliary network for Q-value estimation. It make the training system even more delicated. What network is used for Q-value estimation? any analysis here?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper and for providing constructive comments. We hope our detailed responses will adequately address your concerns.\n\n**W1. Ablation studies and analysis of the impact of on-policy and off-policy objective. And analysis how each of them effect the overall performance.**\n> Thanks for your valuable suggestion! Please refer to the [global response (Q3)](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn), where we provide an analysis of a linear combination of on-policy and off-policy objectives, which demonstrates stronger performance compared to using only on-policy or off-policy objectives individually. Additionally, we analyze the impact of on-policy and off-policy objectives on other distribution-matching baselines, as depicted in Figure 2 of the paper.\n\n**W2. Solving Eq.(9) is involving optimizing the minmax problem. (a) First, the optimization requires additional computational steps for the inner-loop gradient update. How expensive is the computation? such as the time/memory cost (b) Is the optimization robust with respect of hyperparameter changes like $K$ and $\\alpha$.**\n\n>**a. (Time/Memory Cost)** Please refer to the [global response (Q2)](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn), where we analyze and compare our method with baseline methods in terms of both memory and computational complexities. Our analysis indicates that our method does not significantly increase memory or computational complexity in one iteration of policy updating.\n\n>**b. (Robustness w.r.t. $K$ and $\\alpha$)** In our adversarial training algorithm, the number of inner-loop steps $K$ and the learning rate controlling factor $\\alpha$ are hyperparameters that control the stochastic gradient ascent learning of the on-/off-policy $Q$-value functions. To demonstrate the robustness of $K$ and $\\alpha$ in policy optimization, we experimented with different values: $K \\in {1, 3, 5, 10, 15}$ and $\\alpha \\in {0.001, 0.01, 0.1, 0.2, 0.5, 1.0}$ for the instruction-following experiments. The results on DollyEval are shown below. The results indicate that as the number of inner-loop steps $K$ increases, the estimation of on-/off-policy $Q$-value functions becomes more effective, contributing to knowledge distillation. After the training objective converges, increasing $K$ further, for instance, $K \\geq 10$, does not significantly improve performance. A small controlling factor $\\alpha$, for instance, $\\alpha=0.001$ leads to a small learning rate for the $Q$-value function, i.e., $\\alpha \\eta$, making the $Q$-value in the minimax optimization too weak to serve as an effective critic for policy imitation learning. As $\\alpha$ increases to an appropriate level that balances the the $Q$-value critic and the policy, distillation performance improves. However, a large $\\alpha$, for example, $\\alpha \\geq 0.5$, can cause the $Q$-value critic to become too dominant, hindering effective policy learning during the adversarial training process, and leading to poor performance.\n>|$K$|\t1|\t3|\t5|\t10|\t15|\n|----|----|----|----|----|----|\n|ROUGE-L|\t28.5$_{\\pm 0.4}$|\t29.8$_{\\pm 0.4}$|\t30.7$_{\\pm 0.4}$|\t30.9$_{\\pm 0.6}$|\t30.2$_{\\pm 0.4}$|  \n\n>|$\\alpha$|\t0.001|\t0.01|\t0.1|\t0.2|\t0.5|\t1.0|\n|----|----|----|----|----|----|----|\n|ROUGE-L|\t27.7$_{\\pm 0.5}$|\t29.6$_{\\pm 0.5}$|\t30.7$_{\\pm 0.4}$|\t29.5$_{\\pm 0.6}$|\t28.4$_{\\pm 0.4}$|\t25.5$_{\\pm 0.6}$|\n\n**W3. The method requires an auxiliary network for Q-value estimation. It make the training system even more delicated. What network is used for Q-value estimation? any analysis here?**\n> Please refer to the [global response (Q1)](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn), where we provide a detailed discussion of the network used for $Q$-value function estimation and analyze the associated memory and computational complexities.\n\n**Thanks again for your effort in giving detailed and constructive comments. If you have more questions, feel free to discuss them with us. We will take your comments and suggestions in paper revision.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NbBj8onTXB", "reviewer": "Reviewer_8hQS", "summary": "The paper introduces a novel approach to knowledge distillation for Large Language Models (LLMs) using an adversarial training method that incorporates both on and off-policy distillation. The method jointly learns a critic that estimates Q-values while updating both the Q-function and the student model to more closely match the teacher model. The authors employ a policy gradient method to update the student model.", "questions": "+ How exactly is the Q-value function parameterized? Is it an extra head on the model, or a new model entirely?\n+ Regarding the use of policy gradients for training the student function:\n   a. Did you use a baseline to reduce variance?\n   b. What is the variance of these policy gradients? In applications such as RL, policy gradients typically have quite high variance compared to other methods.\n   c. Have you considered lightweight baseline methods, such as those presented in [1]?\n+ Can you provide an ablation of the different elements of the approach, such as investigating the relative importance of the two upper bounds?\n+ Can you provide an analysis -- even if it is brief -- on the computational cost and memory usage of using the additional Q-value critics while training?\n\n\n\n[1] Ahmadian, Arash, et al. \"Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.\" arXiv preprint arXiv:2402.14740 (2024).", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ As far as I am aware, a novel approach to knowledge distillation for LLMs -- although I am not an expert.\n+ A well-presented method, tying together some previous ideas on IRL into the distillation application.\n+ Demonstrates a boost to accuracy.", "weaknesses": "+ Some crucial details are unclear, particularly regarding the parameterization of the Q-value function (see questions section)\n+ Not very much discussion of the computational complexity or additional overhead in memory of having multiple models and requiring rollouts from the teacher and student model while training\n+ Lack of ablation studies -- the method is evaluated as a single monolithic method, when there are many variants that could be applied, such as a weighted combination of the two upper bounds. In particular, I'd like to see how the method using only the on-policy upper bound and the method using only the off-policy upper bound would compare against the method using the linear combination of the on and off policy upper bounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper and for providing constructive comments. We hope our detailed responses will adequately address your concerns.\n\n**Q1. How exactly is the Q-value function parameterized? Is it an extra head on the model, or a new model entirely?**\n> Please refer to the [global response (Q1)](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn), where we discuss the parameterization of the $Q$-value function, which is based on an additional trainable linear head on the student model.\n\n**Q2. Regarding the use of policy gradients for training the student function: a. Did you use a baseline to reduce variance? b. What is the variance of these policy gradients? In applications such as RL, policy gradients typically have quite high variance compared to other methods. c. Have you considered lightweight baseline methods, such as those presented in [1]?**\n> **ab. (Variance of Policy Gradient)** Since the student model was pre-trained with SFT, the variance of the policy gradient is sufficiently low to ensure stable training. Therefore, in consideration of computational efficiency, we did not employ a baseline model in our original experiments.\n\n> **c. (Baseline Model)** Thanks for your suggestion! Following the approach of Ahmadian et al. [1], we implemented two types of baseline models to reduce the variance of policy gradients. Specifically, the sampled baseline uses an average of $k=4$ trajectories draws to estimate the baseline, represented as $\\hat{b} = \\frac{1}{k} \\sum_{i=1}^k U^{\\rm on}(\\tau_i, f)$, $\\tau_1, \\ldots, \\tau_k \\sim \\pi_\\theta(\\cdot | x)$. The learned baseline involves optimizing a value network (a linear head) $b_\\psi$ with respect to the MLE loss $\\sum_s \\vert U^{\\rm on}(\\tau_s, f) - b_{\\psi}(\\tau_s) \\vert^2 $. Reuslts show that using a baseline can slightly improve performance, although it increases the computational complexity.\n|Dataset|\tw/o baseline|\tsampled baseline|\tlearned baseline|\n|----|----|----|----|\n|DollyEval (OpenLLaMA2-3B,ROUGE-L)|\t27.4$_{\\pm 0.4}$|\t**27.8**$_{\\pm 0.3}$|\t26.8$_{\\pm 0.2}$|\nSAMSum (T5-base, ROUGE-L )|\t48.2$_{\\pm 0.3}$|\t**48.7**$_{\\pm 0.2}$|\t48.5$_{\\pm 0.4}$|\nIWSLT’17 (T5-base, BLEU)|\t31.2$_{\\pm 0.2}$|\t30.7$_{\\pm 0.2}$|\t**31.8**$_{\\pm 0.1}$|\nStrategyQA (T5-base, Acc)|\t60.8$_{\\pm 0.3}$|\t**61.2**$_{\\pm 0.3}$|\t58.5$_{\\pm 0.3}$|  \n\n>[1] Ahmadian, Arash, et al. \"Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.\" arXiv preprint arXiv:2402.14740 (2024). \n\n**Q3. Can you provide an ablation of the different elements of the approach, such as investigating the relative importance of the two upper bounds?**\n> Thanks for your suggestion! Please refer to the [global response (Q3)](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn), where we present an ablation study on the effectiveness of the two upper bounds, using a linear combination of on-policy and off-policy objectives. The results demonstrate stronger performance compared to using either the on-policy or off-policy objective alone.\n\n**Q4. Can you provide an analysis -- even if it is brief -- on the computational cost and memory usage of using the additional Q-value critics while training?**\n> Thanks for your suggestion! Please refer to the [global response (Q2)](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn), where we analyze the computational and memory costs associated with using additional $Q$-value critics during training. Comparisons with baseline methods show that our method does not significantly increase memory or computational complexity for policy learning with the additional Q-value critics.\n\n**Thank you once again for your effort in providing detailed and constructive comments. If you have further questions, please feel free to discuss them with us. We will consider your comments and suggestions in our paper revision.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k9MUPmS7EI", "reviewer": "Reviewer_zEWr", "summary": "This paper applies a reinforcement learning (RL) framework to the problem of auto-regressive text generation, framing knowledge distillation as a task of minimizing the imitation gap between teacher and student policies. The authors provide a theoretical analysis demonstrating that the proposed momentum-matching method offers a tighter bound on this imitation gap compared to traditional distribution-matching approaches, potentially leading to improved optimization. To efficiently optimize the momentum-matching target, the paper introduces an adversarial training procedure that alternates updates between the student policy parameters and the Q-value functions, which are used to assess the imitation gaps. Experimental results showcased within the paper indicate that the proposed momentum-matching method outperforms existing distribution-matching baselines in terms of effectiveness.", "questions": "1. Could you provide a detailed comparison of resource consumption and memory costs relative to the baseline methods? Additionally, can you discuss the impact of the number of samples used to estimate the policy gradient on the performance of your method?\n\n2. It seems that the performance of the auxiliary models used to compute Q-value functions significantly impacts the effectiveness of your method. However, the paper does not provide details on the training of these models. I am curious to know whether these models are trained from scratch during the adversarial procedure or if they are pre-trained on certain datasets before inclusion. Is it possible to directly fine-tune teacher models to compute Q-value functions?", "limitations": "The paper briefly discusses the additional computational consumption and GPU memory costs associated with the proposed methods. However, it lacks a more thorough and detailed discussion of other potential limitations that may affect the applicability or scalability of the approach.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured and presents a clear, enjoyable narrative, facilitating ease of understanding for the reader.\n\n2. Utilizing an RL framework, the paper theoretically demonstrates that the momentum-matching target provides a tighter bound for minimizing the imitation gap compared to conventional distribution-matching targets. Furthermore, an adversarial training procedure is proposed to effectively optimize the momentum-matching target, aiming to approximate a Nash equilibrium between the parameters of the student policy and the Q-value functions.\n\n3. Comprehensive experiments empirically demonstrate that the proposed method outperforms existing distribution-matching methods in performance. Additionally, the presented analysis of training loss curves illustrates the stability of the proposed adversarial training procedure.", "weaknesses": "1. Unlike distribution-matching methods, the RL-based momentum-matching adversarial framework requires significantly more computational resources and runtime due to the necessity of calculating policy gradients and updating the parameters of auxiliary networks involved in Q-value functions. While the authors acknowledge this limitation in the conclusion section, the paper lacks quantitative analysis concerning the computation of policy gradients. It does not detail the resource consumption of the overall procedure. This omission limits the reader’s ability to assess the practical applicability of the method.\n\n2. The presented experiments primarily examines knowledge distillation performance within similar or identical architectural frameworks. However, it does not demonstrate the method's generalizability across models with different architectures, thus leaving the robustness of the approach across diverse settings untested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed and constructive comments, as well as your support of our work. We hope our responses adequately address your concerns.\n\n**Q1. a. Could you provide a detailed comparison of resource consumption and memory costs relative to the baseline methods? b. Additionally, can you discuss the impact of the number of samples used to estimate the policy gradient on the performance of your method?**\n> **a. (Computational and Memory costs)** Thanks for your suggestion! Please refer to the [global response (Q2)](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn), where we analyze and compare the memory and computational complexities of our method with baseline methods. The analysis shows that our approach does not significantly increase memory or computational complexity in a single iteration of policy updating.\n\n> **b. (Impact of the Number of Samples)** Thanks for your suggestion! To analyze the impact of sample size on the performance of policy learning, we conducted experiments on an instruction-following dataset with varying numbers of samples. The ROUGE-L results, based on OpenLLaMA2-3B as the student model, are summarized below. They show that as the number of training samples increases, performance on the five test sets improves rapidly at first but then stabilizes once a certain sample size is reached. This demonstrates the importance of sample size in optimizing the policy gradient.\n\n|number of samples| 1,000| 2,000| 4,000| 6,000| 8,000| 10,000| 15,000|\n|----|----|----|----|----|----|----|----|\n|DollyEval|8.7$_{\\pm 0.7}$ | 16.2$_{\\pm 0.8}$ | 21.5$_{\\pm 0.5}$ | 26.8$_{\\pm 0.4}$ | 28.3$_{\\pm 0.3}$ | 30.0$_{\\pm 0.4}$| 30.7$_{\\pm 0.4}$ |\n|SelfInst| 6.7$_{\\pm 1.1}$ | 10.2$_{\\pm 0.8}$ | 12.7$_{\\pm 0.7}$ | 15.5$_{\\pm 0.8}$ | 18.2$_{\\pm 0.6}$ | 21.8$_{\\pm 0.5}$ | 21.7$_{\\pm 0.5}$ |\n|VicunaEval| 2.8$_{\\pm 0.9}$  | 7.9$_{\\pm 0.7}$  | 12.0$_{\\pm 0.8}$ | 14.6$_{\\pm 0.7}$ | 19.7$_{\\pm 0.5}$ | 20.8$_{\\pm 0.3}$ | 21.4$_{\\pm 0.4}$ |\n|S-NI| 10.3$_{\\pm 0.7}$ | 19.7$_{\\pm 0.8}$ | 23.6$_{\\pm 0.5}$ | 30.2$_{\\pm 0.6}$ | 34.8$_{\\pm 0.5}$ | 38.2$_{\\pm 0.3}$ | 38.7$_{\\pm 0.4}$ |\n|UnNI| 15.2$_{\\pm 1.2}$ | 16.4$_{\\pm 1.3}$ |22.7$_{\\pm 0.6}$ | 30.6$_{\\pm 0.8}$ | 37.2$_{\\pm 0.5}$ | 38.3$_{\\pm 0.6}$ | 39.1$_{\\pm 0.3}$ |\n\n**Q2. a. Whether the $Q$-value functions are trained from scratch during the adversarial procedure or if they are pre-trained on certain datasets before inclusion. b. Is it possible to directly fine-tune teacher models to compute Q-value functions?**\n> **a. ($Q$-value Function Formulation)** Please refer to the [global response (Q1)](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn) for detailed formulations of the $Q$-value functions. These functions are constructed by adding linear layers to the student network and the parameters of these layers are trained from scratch during the adversarial procedure for Q-value function estimation. \n\n> **b.** Thanks for your suggestion! We agree that directly fine-tuning teacher models for $Q$-value function estimation is a novel idea. Our current approach uses an adversarial training strategy to estimate the $Q$-value function by solving a minimax optimization problem. Considering memory and computational complexity (as discussed in [global response Q2](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn)), we opted to train a linear layer on the student network for $Q$-value function estimation (as discussed in part **a**). However, fine-tuning the teacher model for $Q$-value function estimation, and subsequently using this $Q$-value for imitation learning, is an interesting direction that we may explore in future work.\n\nIn addition to addressing your insightful questions, we would also like to clarify the remaining weaknesses of our work. W1 has been addressed in Q1, so we will focus on W2 below.\n\n**W2. The presented experiments primarily examines knowledge distillation performance within similar or identical architectural frameworks. However, it does not demonstrate the method's generalizability across models with different architectures, thus leaving the robustness of the approach across diverse settings untested.**\n> Thank you for your insightful suggestions! In our experiments, we have evaluated decoder-based LM architectures, such as OpenLLaMA2 and GPT-2, as well as encoder-decoder-based architectures like T5 and mT5. Due to limited computational resources, we were unable to evaluate the proposed approach on a wider range of architectures, such as OPT, LSTM, and GLM. However, we plan to explore these in future work.\n\nWe also appreciate your comments on the weaknesses, and we will incorporate your suggestions into our paper revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CT80M1IVhh", "reviewer": "Reviewer_ejno", "summary": "To improve knowledge distillation for large language models, the authors first motivate an RL-based formulation that aims to minimize the imitation gap while matching on and off-policy moment bounds, and then introducing an adversarial training algorithm that achieves this by posing it as a two-player minimax game. They showcase the efficacy of the approach with instruction-following and task-specific experiments.", "questions": "- Is the comparison between SFT and the other methods fair in terms of compute budget? It might be interesting to see if increasing the number of epochs still leads to a performance improvement, or if it has plateaued.\n- Could there be any practical benefit to having a different $\\alpha$ for on and off-policy optimization in Algorithm 1? The intuition for this comes from the results of Section 4.3 and particularly Figure 2.", "limitations": "The authors discuss the limitations appropriately in Section 5 of the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**S1.** The problem is an important one and, as far as I can tell, the method is novel and well motivated.\n\n**S2.** The experiments are reasonable to establish the efficacy of the method.\n\n**S3.** The paper is well written and clear.", "weaknesses": "**W1.** *The paper is missing a more thorough comparison of the training costs of each of the compared methods.* The authors briefly mention in Section 5 that their method induces a larger computational and memory cost compared to some of the other baselines. This discussion should be fleshed out more and potentially backed by experimental evidence.\n\nMinor comments:\n- It does not have any practical influence, but the definition of $\\mathbf{y}$ in line 101 is not very elegant as $y_0$ will have a different dimensionality than $y_i$ for $i \\in \\\\{1,\\dots,T\\\\}$. It might be cleaner to define $\\mathbf{y} = \\mathbf{x} || (y_1,\\dots,y_T)$, where $||$ is a concatenation operator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed and constructive comments, as well as your support of our work. We hope our responses adequately address your concerns.\n\n**Q1. Is the comparison between SFT and the other methods fair in terms of compute budget? It might be interesting to see if increasing the number of epochs still leads to a performance improvement, or if it has plateaued.**\n> Thanks for your suggestion! In our experiments, we use sufficient numbers of training steps to ensure our method and the baselines converge. The comparisons on all the datasets are based on the converged results. To investigate if increasing the iteration steps can lead to a performance improvement, we extended the training epochs to 4, with a maximum of 20,000 iteration steps in the instruction-following experiments. We evaluated our method, SFT, and two strong baselines using checkpoints from different training steps. The results show that (i) with sufficient training steps, e.g., $\\geq 8,000$, all methods converge to a plateau with only small fluctuations, and (ii) under the premise of convergence and using the same amount of computation (e.g., training steps = 8,000, 10,000, 15,000, 20,000), our model still outperforms the other baselines.\n|Steps|\t2,000|\t4,000|\t6,000|\t8,000|\t10,000|\t15,000|\t20,000|\n|----|----|----|----|----|----|----|----|\n|SFT|\t14.7$_{\\pm 0.5}$|\t24.8$_{\\pm 0.3}$|\t26.8$_{\\pm 0.3}$|\t25.4$_{\\pm 0.3}$|\t26.7$_{\\pm 0.5}$|\t27.1$_{\\pm 0.2}$|\t26.6$_{\\pm 0.2}$|\nGKD|\t12.1$_{\\pm 0.4}$|\t20.7$_{\\pm 0.4}$|\t26.4$_{\\pm 0.3}$|\t27.0$_{\\pm 0.4}$|\t26.8$_{\\pm 0.4}$|\t27.3$_{\\pm 0.3}$\t|27.7$_{\\pm 0.2}$|\nDistiLLM|\t17.4$_{\\pm 0.5}$|\t25.9$_{\\pm 0.3}$|\t28.8$_{\\pm 0.3}$|\t29.7$_{\\pm 0.3}$|\t29.5$_{\\pm 0.2}$|\t28.9$_{\\pm 0.2}$|\t29.3$_{\\pm 0.1}$|\nOurs|\t16.9$_{\\pm 0.6}$|\t25.3$_{\\pm 0.4}$|\t29.5$_{\\pm 0.5}$|\t30.8$_{\\pm 0.3}$|\t30.7$_{\\pm 0.2}$|\t30.3$_{\\pm 0.2}$|\t30.6$_{\\pm 0.2}$|\n\n**Q2. Could there be any practical benefit to having a different for on and off-policy optimization in Algorithm 1? The intuition for this comes from the results of Section 4.3 and particularly Figure 2.**\n> Thanks for your suggestion! We empirically investigated the learning rate controlling factors $\\alpha$ for on-/off-policy $Q$-value function learning, respectively. We conducted instruction-following experiments with variants of off-policy $\\alpha$ and on-policy $\\alpha$ in ${0.05, 0.1, 0.15, 0.2} \\times {0.05, 0.1, 0.15, 0.2}$ and illustrated the results on the DollyEval set in the table below. We observed that the best result (ROUGE-L = 31.5) was achieved with off-policy $\\alpha = 0.15$ and on-policy $\\alpha = 0.10$, while the $\\underline{\\text{second best}}$ result (ROUGE-L = 31.2) was achieved with off-policy $\\alpha = 0.15$ and on-policy $\\alpha = 0.05$, outperforming the results (ROUGE-L = 30.7) under equal $\\alpha$ values for on- and off-policy optimization. This demonstrates the effects of using different $\\alpha$ for on- and off-policy optimization.\n|On-Policy $\\alpha$ ($\\downarrow$) Off-policy $\\alpha$ ($\\rightarrow$) |\t0.05|\t0.10|\t0.15|\t0.20|\n|----|----|----|----|----|\n|**0.05**|\t$30.5_{\\pm 0.2}$|\t$30.2 _{\\pm 0.4}$|\t$\\underline{31.2}_{\\pm 0.1}$|\t$29.2_{\\pm 0.2}$|\n|**0.10**|\t$29.2_{\\pm 0.5}$|\t$30.7_{\\pm 0.4}$|\t${\\bf 31.5}_{\\pm 0.4}$|\t$30.2_{\\pm 0.5}$|\n|**0.15**|\t$28.6_{\\pm 0.4}$|\t$29.6_{\\pm 0.6}$|\t$29.8_{\\pm 0.4}$|\t$28.8_{\\pm 0.5}$|\n|**0.20**|\t$27.9_{\\pm 0.4}$|\t$28.5_{\\pm 0.4}$|\t$28.2_{\\pm 0.6}$|\t$29.5_{\\pm 0.6}$|\n\nIn addition to addressing your insightful questions, we would also like to clarify the weakness as follows.\n\n**W1. The paper is missing a more thorough comparison of the training costs of each of the compared methods.**\n> Please refer to the [global response (Q2)](https://openreview.net/forum?id=0VeSCjRDBy&noteId=HgJrUiZdkn), where we analyze and compare the training costs, including both memory and computational complexities, of our method and the baselines. The analysis shows that our method does not significantly increase the memory or computational complexity in one iteration of policy updating.\n\nWe also appreciate your comments on the minor weakness. We will incorporate your suggestions into our paper revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Response to Common Questions
**Q1. The parameterized estimation of Q-value function.**
> The Q-value function $f_{\phi}(y_{<t}, y)$, where $y_{<t}$ denotes the state of the $0:t-1$ tokens and $y \in \mathcal{V}$ denotes an action of the $t$-th token, was estimated with a neural network, which can be represented as $f_{\phi}(y_{<t}, y) = (h_{t-1} + v_y)w_y^{\top}$. Here, $h_{t-1}$ denotes the feature representation of the state $y_{<t}$ derived from the $(t-1)$-th hidden step of an auto-regressive LM, $v_y$ represents the feature vector corresponding to the token $y$, while $w_y$ denotes the parameters of a linear layer. Thus, the $Q$-value function can be viewed as an additional linear layer on the model with the trainable parameters $\phi = ( v_y, w_y )_{y \in \mathcal{V}}$. Notably, the trainable parameters of the $Q$-value function were optimized starting from randomly initialized weights.

**Q2. Computational and memory costs of the minimax optimization with the Q-value estimation.**
>**a. (Memory Cost)** Compared to the baseline SFT and distribution-matching methods, the additional trainable parameters come from the $Q$-value function, which consists of $\mathcal{O}(H|\mathcal{V}|)$ parameters based on the response to **Q1**, where $H$ represents the hidden dimension and $|\mathcal{V}|$ represents the vocabulary size.

>**b. (Computational Cost)** For each iteration (one step of policy update) of our method, the computational complexity of our gradient-based algorithm (Alg. 1) consists of **(i)** the inner-loop gradient computation and updating: $\mathcal{O}(KT|\mathcal{V}|H)$, which is a multiple of $K$ steps in the inner-loop, the maximum number of sequence length $T$, the vocabulary size $|\mathcal{V}|$ and the hidden size $H$, and **(ii)** the policy gradient computation and updating: $\mathcal{O}(T|\mathcal{V}||\theta|)$, where $|\theta|$ represents the number of parameters in a LM. We observe that the computational complexity in other distribution-matching baselines, such as KL-, RKL-, JS-divergence and TV distance is equal to (ii) in our method. Thus, our method incurs additional computational complexity of (i) in compared with the baseline distribution-matching methods. Since the number of parameters in language model $|\theta|$ is much larger than the hidden dimension, e.g., $117M=|\theta|\gg KH= 3,840$ for GPT-2, $2.7B=|\theta|\gg KH= 16,000$ for OpenLLaMA2 and $220M=|\theta|\gg KH= 3,840$ for T5-base, **our method does not significantly increase the computational complexity in one iteration of policy gradient updating**.
||ours|baselines|
|----|----|----|
|memory complexity| $\mathcal{O}(H\vert\mathcal{V}\vert + \vert  \theta \vert)$ | $\mathcal{O}(\vert  \theta \vert)$ |
|computational complexity| $\mathcal{O}(KT\vert\mathcal{V}\vert H + T\vert\mathcal{V}\vert\vert\theta\vert)$ |  $\mathcal{O}(T\vert\mathcal{V}\vert\vert\theta\vert)$|

**Q3. Ablation study corresponding to the on- and off-policy learning.**
> We study the impact of on-policy and off-policy objectives within our method by defining the following linear combination policy gradient updating role: $ \theta \leftarrow \theta - 2\eta ( \gamma G_{\rm on}(\tau_{\rm on}, \theta) + (1-\gamma) G_{\rm off}(\tau_{\rm off}, \theta) ) $, where $\gamma \in [0,1]$ denotes the linear combination coefficient of the on-policy gradient $G_{\rm on}(\tau_{\rm on}, \theta)$ and the off-policy gradient $G_{\rm off}(\tau_{\rm off}, \theta)$. We observe that if $\gamma=0$, only the off-policy objective contributes to policy learning. As $\gamma$ increases from 0 to 1, the influence of on-policy objective increases while that of the off-policy objective decreases. Finally, when $\gamma=1$, only the on-policy objective contributes to policy learning. The following results shows that a combination of on-policy and off-policy objectives outperforms using either on-policy or off-policy objectives only across four datasets
|$\gamma$|0 (off-policy)|0.25|0.50|0.75|1.00 (on-policy)|
|----|----|----|----|----|----|
|DollyEval (OpenLLaMA2-3B, ROUGE-L)|27.4$_{\pm 0.4}$|	28.2$_{\pm 0.3}$|	**30.7**$_{\pm 0.4}$|	29.5$_{\pm 0.7}$|	28.8$_{\pm 0.7}$|
SAMSum (T5-base, ROUGE-L)|	48.2$_{\pm 0.3}$|	**50.8**$_{\pm 0.3}$|	50.4$_{\pm 0.3}$|	49.2$_{\pm 0.2}$|	48.7$_{\pm 0.2}$|
IWSLT’17 (T5-base, BLEU)|	31.2$_{\pm 0.2}$|	30.3$_{\pm 0.2}$|	**32.4**$_{\pm 0.2}$|	31.6$_{\pm 0.1}$|	30.7$_{\pm 0.1}$|
StrategyQA (T5-base, Acc)|	60.8$_{\pm 0.3}$|	**63.6**$_{\pm 0.5}$|	62.9$_{\pm 0.4}$|	60.5$_{\pm 0.5}$|	59.7$_{\pm 0.4}$|

Meta-review: This paper presents a RL framework to the problem of auto-regressive text generation, framing knowledge distillation of LLMs as a task of minimizing the imitation gap between teacher and student policies. After author rebuttal, it received scores of 5677. All the reviewers are generally happy about the paper, agreeing that (1) the proposed method is novel, (2) the paper is well written, and (3) the experiments are reasonable to establish the efficacy of the method.

Reviewers had also shown some common concerns, such as clarity regarding the parameterization of the Q-value function, and lack of quantitative analysis concerning the computation of policy gradients. The authors have done a good job in addressing these concerns during rebuttal. Overall, the AC would like to recommend accept of the paper.
Decision: Accept (poster)