On Teacher Hacking in Language Model Distillation
Authors: Daniil Tiapkin
Daniele Calandriello
Johan Ferret
Sarah Perrin
Nino Vieillard
Alexandre Rame
Mathieu Blondel
Venue: ICML 2025 poster (2025)
Keywords: knowledge distillation
teacher hacking
language model distillation
Primary area: deep_learning->large_language_models
TL;DR: We study teacher hacking: does over-optimization of the distillation objective harm the ground-truth performance?

Abstract: Post-training of language models (LMs) increasingly relies on the following two stages: (i) knowledge distillation, where the LM is trained to imitate a larger teacher LM, and (ii) reinforcement learning from human feedback (RLHF), where the LM is aligned by optimizing a reward model. In the second RLHF stage, a well-known challenge is reward hacking, where the LM over-optimizes the reward model, leading to degraded performance on the true objective, in line with Goodhart's law.
In this paper, we investigate whether a similar phenomenon, that we call teacher hacking, can occur during knowledge distillation. This could arise because the teacher LM is itself an imperfect approximation of the true distribution. To study this, we propose a controlled experimental setup involving: (i) an oracle LM representing the ground-truth distribution, (ii) a teacher LM distilled from the oracle, and (iii) a student LM distilled from the teacher.
Our experiments reveal the following insights. When using a fixed offline dataset for distillation, teacher hacking occurs; moreover, we can detect it by observing when the optimization process deviates from polynomial convergence laws. In contrast, employing online data generation techniques effectively mitigates teacher hacking. More precisely, we identify data diversity as the key factor in preventing hacking.
Overall, our findings provide a deeper understanding of the benefits and limitations of distillation for building robust LMs.

On Teacher Hacking in Language Model Distillation
Daniil Tiapkin' Daniele Calandriello Johan Ferret Sarah Perrin Nino Vieillard Alexandre Ramé? Mathieu Blondel
Abstract
Post-training of language models (LMs) increas- ingly relies on the following two stages: (i) knowl- edge distillation, where the LM is trained im- itate a larger teacher L.M, and (ii) reinforcement learning from human feedback (RLHF), where the LM is aligned by optimizing a reward model. In the second RLHF stage, well-known challenge is reward hacking, where the LM over-optimizes the reward model. Such phenomenon is in line with Goodhart's law and can lead to degraded per- formance the true objective. In this paper, we investigate whether a similar phenomenon, that we call teacher hacking, can occur during knowl- edge distillation. This could arise because the teacher LM is itself an imperfect approximation of the true distribution. To study this, we pro- pose controlled experimental setup involving: (i) oracle LM representing the ground-truth distribution, (ii) teacher LM distilled from the oracle, and (iii) a student LM distilled from the teacher. Our experiments reveal the following insights. When using a fixed offline dataset for distillation, teacher hacking occurs; moreover, we can detect by observing when the optimization process deviates from polynomial convergence laws. In contrast, employing online data genera- tion techniques effectively mitigates teacher hack- ing. More precisely, we identify data diversity as the key factor in preventing hacking. Overall, our findings provide deeper understanding of the benefits and limitations of distillation for building robust and efficient LMs.
Human expert
Teacher model
Student model
Oracle model
Teacher model
Student model
Figure 1. Overview of our controlled experimental setup. Usu- ally. the teacher model is trained expert data before being distilled into the student LM. In the controlled setup of this paper. the teacher is itself distilled from additional oracle model. This oracle model allows us to measure the quality of the distillation process into the student, and reveal "teacher hacking".
1. Introduction
Distillation for post-training LMs. Language models (LMs) have achieved remarkable success across a wide range of natural language processing tasks, such trans- lation, summarization, and reasoning. Notably, large LMs demonstrate impressive generalization capabilities, but their high computational cost poses a significant challenge, par- ticularly when deployed on resource-constee constrainn devices. Efficiency considcrations motivate the training of smaller LMs, that would ideally provide similar performance at a fraction of the computational cost. To this end, the most pop- ular approach is knowledge distillation (KD) (Hinton et al., 2015), in which smaller student LM is trained to imitate the larger teacher LM. Distillation is increasingly studied (Agarwal et al., 2024; Gu et al., 2024; Kim et al., 2024) and used, notably for the post-training pipelines of LMs (as demonstrated by Zephyr (Tunstall et al., 2023), Gemma-2 (Riviere et al., 2024), and DeepSeck-Vi (Liu et al., 2024)). just before the final reinforcement learning from human feedback (RLHF) (Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022) phase.
'CMAP. École Polytechnique, Palaiseau, France:
Work done during internship Google DeepMind.
Google DeepMind. Correspondence to: Daniil Tiapkin
<danill.iapkii @ @olee polytcchnique.cdu>.
Teacher as an imperfect proxy. However, a key under- studied limitation of KD is that the teacher model does not represent the ground-truti distribution but instead acts as
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On Teacher Hacking in Language Model Distillation
an imperfect proxy for it (Menon al., 2021; Zhang al., 2023). This viewpoint draws parallels to a well-studied phenomenon in RLHF known reward hacking (Amodei et al., 2016; Pan et al., 2022; Gao et al., 2023). Reward hack- ing is a manifestation of Goodhart's law arising when LMs over-optimize the reward model, trained to represent human preference, thus also an imperfect proxy for the true task objective. The consequences of reward hacking in RLHF can be significant, leading to models misleading humans and producing unsafe behaviors (Hendrycks et al., 2021; Wen et al., 2025).
2. Preliminaries
Let and y denote the spaces of all possible prompts and responses, assumed to be sentences in the vocabulary Σ. For two sets A and B. ((A) denotes the space of proba- bility measures over A, and ((A(B) represents the space of conditional probability measures over A given B. An auto-regressive language model π E LMs(A') is defined as the conditional probability of the next token or the end- ofseeeee token given a prompt x E X and a partial generation y E Y, expressed as v (w]at, y). where w E Σ. In practice, the probability [w]a, y) is defined using a soft- max with temperature It (w y) exp( 2 2z, v)), where 2( 2(|| y) are the logits output by the neural network and is temperature parameter.
Teacher hacking, Inspired by the analogy with reward hacking in RLHF, we define teacher hacking as a possible phenomenon during distillation in which the student L.M learns imitate the teacher, not by better approximating the true data distribution, but by exploiting imperfections in the teacher. This raises natural questions: (1) Does teacher hacking occur in practice? and if so, (2) when does it appear?, and (3) what strategies can miligate it?
For any language model т, the induced distribution over re- sponses is given by (r(()/ A I1 ( M:i), where lyl denotes the length of the response y. i.e., the num- ber of tokens (non-empty characters) in y, and Ya - (y1 3/....,, ;/t.1.)
Controlled experimental setup. To answer these questions, we propose a controlled experimental setup. Specifically, we introduce an oracle model that represents the ground- truth; see Figure 1 for illustration. In this setup, the distance between the student and oracle LMs serves as a "golden metric". Conversely, the distance between student and teacher LMs, optimized during the fine-tuning of the stu- dent, defines a "proxy metric". Our setup is semi-synthetic, in the sense that prompts are sampled from real datasets but responses are sampled from the oracle LM to learn the teacher, and from the teacher LM learn the student.
Distances between language model distributions. Let d E A(X) be distribution over prompts induced by the particular task dataset. To measure the convergence of one distribution, induced by a language model π, to distribution induced by a language model T', we use the (expected) forward and reverse KI. between the cor- responding conditional measures Pr and pr', defined as KI KLee (Tr, r') - E properties of the KL divergence, the sequence-level diver- gence can be estimated very efficiently using teken-level KL divergences. Additionally, we use a sequence-level Jensen- Shannon divergence JS (TT, x'). We provide more details in Appendix B.
(+++))) By the
We summarize our main contributions as follows.
We introduce and formally define the phenomenon of teacher hacking in LM distillation, providing a novel controlled experimental framework to systematically measure its importance.
Supervised fine-tuning. Let p E A()(X)) be a condi- tional response distribution that encodes the ground-truth response distribution. Solving downstream tasks such as summarization, translation, reasoning, instruction follow- ing is fundamentally equivalent to approximating p. There- fore, the ultimate goal of any post-training pipeline is to approximate in order to address these tasks effectively.
We show that teacher hacking occurs when distillation is performed on fixed offline dataset. Notably, we empirically found that the proxy metric initially follows a polynomial convergence law, and that teacher hacking emerges when the optimization process deviates from this law.
One of the common approaches to this problem is supervised fine-tuning (SFT). Let us assume that we have dataset of pairs (I, for d(-) and y ~ p(-12)). Then, to find a language model such that its conditional distribution Px approximates p. it is common to use a simple log-loss
We analyze strategies mitigate teacher hacking, No- tably, we show that teacher hacking is absent when using online data generation methods, sampling directly from the teacher or the student during distillation. We relate this success to increased diversity in the dataset. We then propose cheaper practical alternative strategies to mitigate teacher hacking, such as inereasing the diver- sity across prompts or using larger amount of offline Seacher-generatee data.
LSFT(T) & Ba log P. (2/2))
(I)
This loss is equal, up to a constant factor, to an expected sequence-level forward KL divergence between and px: Exnd( | (p.ppp)))))
2

---

On Teacher Hacking in Language Model Distillation
Language model distillation. We suppose that have access teacher language model, denoted We E (2), such that it approximates the ground-truth distribution p that is, Pt = Pn, = p. The goal of language model distilla- tion is to train a student language model, denoted s, so as to approximate the teacher model, that is, P = Pr. R Pt- We emphasize that the teacher-induced distribution pt is not equal ground-truth p but only approximates it.
and responses. Instead, it may include multiple responses per prompt, and the number of prompts is not necessarily equal to the total number of available prompts N. Each training batch is sampled directly from this fixed dataset.
Online data sources, by contrast, involve generating re- sponses dynamically through an online sampling proce- dure. For each training batch of prompts { of size B, sampled from D prompt. a corresponding model (either the teacher the student) generates batch of responses {u; ) 1=11 While sampling from the student model is often referred to as on-policy generation in the literature (Agarwal et al., 2024), we use the term online student to emphasize its parallel to the online teacher data source.
We usually distinguish between hard and soft distillation. In hard distillation, the student is only trained from the teacher's ' predicted next tokens, i.c., the student only sees the most likely tokens according to the teacher. In soft dis- tillation, the student is trained from the teacher's predicted next token distributions, giving much more information to the student. In this work, we focus on soft distillation, and the loss function for this procedure takes the form
The distinction between offline and online data is partic- ularly evident during subsequent epochs: for offline data sources, responses remain fixed across epochs, while for online data sources, new responses are generated for each epoch, independently drawn from the same distribution.
I beoken (aaken(TT(-Fe)) (Ts(-12, Ua), me(- v.))
(TD( 2
E
1
(2) where x ~ d(-) and y ~ v(-lz), v E A()() data source, and loken is token-level loss function between two distributions over the vocabulary.
Teacher hacking. As we already mentioned, the main goal of the post-training pipeline is to approximate the ground-truth distribution p by the student model. This goal could be achieved by SFT given access sufficiently many samples from p. However, recent works have shown that dis- tilling from a teacher model can actually work better since the whole next-token distribution contains much richer infor- mation than only sampled tokens. An understudied problem is that the teacher model is only an imperfect proxy for p. This may lead to a problematic situation where the student LM learns to imitate the teacher, not by better approximat- ing the true data distribution but by exploiting imperfections in the teacher. We call this phenomenon teacher hacking, and give a more formal definition below.
The token-level loss and the data source should satisfy two assumptions: (i) it is non-negative and satisfies the property brokon (p, v) = O if and only if p for any two distribu- tions p. q E (Σ)), and (ii) the support of vr- (z)) includes the support of teacher-induced conditions measure pe(-(z) for almost all x, i.e. d(x)) > Given these two assumptions, it is easy to show that a language model minimizes the loss (TD( if and only if TT. TT.
In particular, considering v(-(z)) induced by an offline dataset generated by a teacher and using broken (p: q) = (L q). we achieve the same expected loss as in the case of SFT. This approach corresponds to the works of (Hinton al., 2015; Sanh al., 2020). However, we could consider different token-level losses, such as reverse KL divergence (Gu et al., 2024), generalized Jensen-Shannol divergence (Agarwal et al., 2024), or skewed KL divergence (Ko et al., 2024). The data source can be induced by sampling on- line from the teacher model (Kim & Rush, 2016), sampling online from the student model (Gu et al., 2024), or combin- ing offline and online data (Lin al., 2020; Agarwal et al., 2024; Ko et al., 2024).
(Teacher hacking). Let {pt Ik=1 be a sequence of conditional response distributions in- duced during the training of a student model, Pt the distribution induced by the teacher model, and P the target human expert conditional distribution.
Definition 1
(0)
We say that {o% [ J exhibits the teacher hack- ing phenomenon with respect to a distance measure dist: A()2)) x A - R, if, as k -> +00, .tt) decreases while dist (8) .p) increases.
dist (ps (1)
Offfiin vs. online data sources. In this paper, we distin- guish between two different types of data sources: offline and online. Offline data sources are based on fixed dataset, denoted as Doffline = M where I; is sampled from D prompt: and y; ~ pt (-12.)) are responses generated by the teacher model. Importantly, the dataset Dominne does not need have a one-to-one correspondence between prompts
A simple example where this would occur is when the stu- dent model is initially closer to the target distribution than the teacher model. However, in more realistic scenarios that are closer to real-world applications, the teacher model is larger and provides a better approximation of p than the student model.
3

---

On Teacher Hacking in Language Model Distillation
Prompt dataset
Teacher model
Proxy metries, in contrast, do not rely on the oracle model and instead measure the alignment between the student and teacher models. These metrics use the same types of distances: the forward KL divergence KLgeq (T,, Ts), the reverse KI. divergence KT (T,, Te), and the Jensen- Shannon divergence JS TT). Similar to the golden metrics, proxy metrics are estimated using the validation set of prompts and sampling from the models involved.
X
SFT
Generation
Distillation
(x,y)
Oracle dataset
Oracle model
SFT
Student model
Training. The training procedure for our experiments con- sists of two stages.
Figure 2. Overview of the training pipeline. Two stages: (1) prompts from task-specific real dataset are used by the oracle model to generate the oracle pairs (a, y). and afterwards, this dataset is used to get initial SFT checkpoints for both teacher and student model; (2) prompts from the same distribution are used to perform knowledge distillation, where the teacher model serves as a proxy to train the student model.
In the first stage, supervised fine-tuning is performed on both the teacher and student models using a small oracle- generated dataset D Doracle [2 y1) Nagacle The prompts 24 are sampled from the task distribution d(-), and the re- sponses Vs ~ p(-(ar,) are generated using the oracle model. Our setup is semi-synthetic, in the sense that prompts are sampled from real datasets but responses (seen as labels) are sampled from the oracle LM to learn the teacher, and from the teacher LM learn the student. This stage is the only place where direct information from the oracle model is propagated to the teacher and student models. The fine- tunettunig process optimizes usual SFT loss (1) that in expectation equals to a sequence-level distance KT (14, iT). The best checkpoint is selected based on the estimate of this quantity over the validation set.
Overfitting vs. teacher hacking. We would like to clarify the difference between two related phenomena: classical overfitting and the newly-introduced teacher hacking. In the case of overfitting, the model continues to minimize the loss on the training set but fails to do so on the held-out validation set, which it never observes. In contrast, teacher hacking occurs when the model successfully achieves its objective from the teacher's perspective, even on the valida- tion dataset, but fails to improve in terms of approximating the ground-truth behavior. The ultimate outcome of both phenomena similar: the model fails to generalize to the ground truth, but the reasons differ.
In the second stage, distillation is conducted from the teacher to the student model by optimizing the soft- distillation loss (2). The distillation process uses a training dataset of unlabeled prompts Prrompt = {I: where Noraclo and different data sources that define a dis-
N >>>> tribution of y ~ v(-[ar) in the loss. The final pipeline is summarized Figure 2.
3. Methodology
To analyze the effect of teacher hacking, we require a method to estimate the distance between the student model and the ground-truth distribution. For this purpose, we in- troduce the oracle model, denoted as μ E LM),X),, which is assumed induce the target distribution p. i.e., Pμ= = p.
Evaluation. To investigate the teacher hacking phe- nomenon, we analyze two key types of curves: (1) the
dependence of the training loss, proxy metrics, and golden metrics on the number of epochs completed, and (2) the proxy-golden curve, which illustrates the relationship be- tween the golden metric (only accessible in our controlled experimental setup) and the proxy metric. Both proxy and golden metrics are computed using a held-out validation set of prompts. The final pipeline is summarized in Figure 3.
Golden and proxy metries. As outlined in Definition 1, evaluating the teacher hacking phenomenon requires com- puting two sets of metrics.
Golden metrics are computed using the oracle model and reflect the performance with respect to the true objective. Specifically, we use three types of divergences: the for- ward KL divergence KLaeq TT), the reverse KL diver- gence KLeeq (Ty: 14), and a Jensen-Shaue Shannon-likee distance JSang(Ts n1), which is closely related to the -hannnon divergence between the conditional distributions over the re- sponse space. These metrics are estimated using a held-out validation set of prompts, with sampling performed from the respective models.
The epoch-dependence plots provide insights into scaling law phenomena (Kaplan al., 2020) and help to understand overall training dynamics. Proxy-golden curves are crucial for visually assessing the presence of teacher hacking: a U- shaped curve serves as clear indicator. Indeed, we expect the proxy metric to be reduced during training, and if the golden metric first decreases and then increases, it directly shows teacher hacking. These proxy-golden plots can be compared to plots from Gao et al. (2023), with one essential

---

On Teacher Hacking in Language Model Distillation
Validation prompt dataset
XSum, Forward KL loss, T5-large to T5-base
Offline data source
19.8
Golden metric KLay (p.. ma)
19.6
Golden metric
Praxy metric
19.4
Epoch
Oracle model
Student model
Teacher model
19.2
Figure Overview
evaluation pipeline. use the vali- dation prompt dataset to measure the golden metric (the distance between the oracle and the student models) and the proxy metric (the distance between the teacher and the student models).
19.0
18.8
13
11
10
Proxy metric KLang (Te Te)
difference: our approach measures optimization progress as the distance the teacher model rather than the distance from an initial reference policy.
Figure 4. Proxy-Golden plot (offline data source). We distill TS-large teacher into T5-base student on the XSUM dataset. The token-level training loss is the forward KL, the proxy metric is distance to the teacher distribution and golden metric is the distance to the ground ground-truth (oracle) distribution (available thanks to our semi-synthetic controlled experimental setup). plot, the æ-axis (proxy metric) indicates optimization progress, and the y-axis shows the ground-trun truth performance (golden metric): lower is better. Teacher hacking occurs in the of offline data source: the orange curve has a U-type shape, indicating that during optimization, the orange metric starts increasing. whereas the proxy metric continues to decrease.
4. Experimental Results
Oracle, teacher, and student models. Our experiments use a family of encoder-decoder language models based on T5 (Raffel et al., 2020; Roberts et al., 2022). The or- acle model is the Flan-T5-XI. (Chung et al., 2024), 3B- parameter model fine-tuned on the Flan dataset (Wei al., 2021; Longpre et al., 2023) for instruction-uu based tasks. For the teacher and student models, we use pretrained check- points of 1.1 in three configurations: small (77M parame- ters), base (250M parameters), and large (800M parameters). We always use temperature sampling with a temperature pa- rameter 1 for generations from any model.
4.1. Does Teacher Hacking Appear?
We begin by investigating whether teacher hacking appears.
Setup. For the first experiment, we use only offline data sources: responses are pre pre-generated yi pe(x;) for all 'D prompt. and the dataset remains fixed throughout training. The learning rate for optimization is selected via a grid search over {10-4. 3,3 x 10 10-4, 10,,,--0) 100-3.
Datasets. Our experiments use three datasets for training and evaluation: the XSum summarization dataset (Narayan et al., 2018), the WMT-14 en-de translation dataset (Bojar et al., 2014), and the cnttructioo-followinn dataset Natural Instructions (Mishra et al., 2022; Wang al., 2022). In alignment with our experimental setup, prompt data from these datasets, supplemented with task- specific instructions as needed for each task.
The distillation procedure starts from the SFT checkpoints of the teacher and student models. Training is carried over 50 epochs analyze long-term convergence behavior.
only the
Results. The results of distilling the T5-large teacher model into the T5-base student on the XSum dataset, using forward KL loss, along with the corresponding golden and proxy metrics, are shown Figure 4.
For the first stage of the training pipeline, where the oracle dataset is build and used for SFT, we use Noracle = 25 000, 50 000, and 100 000 prompts from the XSum, WMT-14 en-de, and Natural Instructions datasets. respectively. For the second stage, which involves the knowledge distillation procedure, use N 200 000, 450 000, and 500 000 prompts from these datasets. A single epoch is defined as one complete pass through all N examples, corresponding to [N/B] training steps, where denotes the batch size. For XSum and WMT-14 en-de, we use batch size B = 32; for Natural Instructions, we use batch size B 64.
In this plot, the z-axis represents the optimization progress in terms of the distance the teacher model (from left to right), while the y-axis shows the golden metric. The scatter plot shows the exact values of proxy and golden metrics, where the color demonstrates at which epoch this measurement was performed. The curve itself shows a relationship between smoothed values of the proxy and
5

---

On Teacher Hacking in Language Model Distillation
XSum, Forward KL loss, T5-large to T5-base
Offline
Online (student)
Online (teacher)
Train loss
Proxy metric
Golden metric
0.489
21.3
KI lang (T Tal
train loss
0.254
13
19.7
KL 200
0.132
9.61
18.2
0.0688
7.08
16.8
0.3
3
90
0.3
50
0
3
50
epochs
epochs
epochs
Figure Impact using offline vs. online data sources. When using fixed offline dataset, though the proxy metric continues decrease, this is not visible in the golden metric, which continues to increase, phenomenon call teacher hacking. However, when using online response sampling, both from the teacher model or from the student model, this phenomenon does not occur.
golden metric, where smoothing is performed by Gaussian smoothing.
metrics is due to averaging over sentence length, which is used in training loss but not in proxy/golden metrics. The training loss indicates the token-level training convergence, whereas the proxy/golden metrics show the sentence-level validation convergence.
For offline data source, the plot exhibits a U-shaped curve. This behavior indicates teacher hacking: as optimization progresses, the ground-truth performance (golden metric) initially improves but eventually deteriorates. Overall, the conclusion of this experiments is following.
Results. The comparison results between offline and on- line data sources are shown in Figure 5. Additional com- parisons for other combinations datasets, student/teacher model sizes, and loss functions are provided in Appendix A.
Teacher hacking exists and emerges after extended training on fixed offline dataset.
Observation 1.
We analyze the epoch dependence of the training loss, proxy metric, and golden metric. We present the dependencies log-log scale to track possible polynomial convergence laws: the polynomial dependence of the metric on the training time. Overall, we make the following observations.
4.2. When Does Teacher Hacking Appear?
In this subsection, we investigate the conditions under which teacher hacking occurs.
Setup. For this experiment, we evaluate three distinct data sources: (I) Offline data source; (2) Online teacher data source: for each batch of prompts sampled from D prompts new response y/ pe(a:;) ) is dynamically generated by the teacher model; (3) Online student data source: responses are generated on-the-fly flyy Vi Pa(x;) using the current student model.
(i) In the offline data scenario, we validate the presence of teacher hacking: the proxy metric decreases while golden metric increases after a certain point. This phe- nomenon does oceur with online data sources.
(ii) We notice that the behavior of all curves for online and offline data sources is different. Overall, the training loss for the offline data sources decreases faster since the training loss is optimized multiple times over the same data, but the performance on the proxy/golden metrics is worse overall.
As in the previous experiment, we use the forward KL di- vergence as a token-level loss. We refer to Appendix A for different token-level loss functions, such reverse KL divergence and Jensen-Shannon divergence.
(iii) The proxy metric for online data sources follows linear trend the log-log scale, indicating polynomial convergence law. In contrast, teacher hacking in offline data coincides with deviations in the proxy metric compared to online data. It gives a mechanism to detect teacher hacking using only the proxy metric, that measurable even real scenarios (not only our controlled experimental setup).
We analyze the dynamics of proxy and golden metrics for each data source. For the proxy and golden metrics, we apply Gaussian smoothing to smooth the noisy behavior of the curves. We would also like to emphasize that the differ- ence scaling between the training loss and proxy/golden

---

On Teacher Hacking in Language Model Distillation
XSum, Forward KL loss, T5-large to T5-base
Offline
Offline (x0.5 prompts, 2x gen)
Offline (x0.2 prompts, 5x gen)
Train loss
Proxy metric
Golden metric
0.441
20.8
KT Kang (TT,. ita
KL (p. ma)
train loss
0.235
14.2
20.1
0.125
11.3
19.4
0.0667
8.94
18.8
0.3
3
0.3
1 50
0
3
1
epochs
epochs
epochs
Figure Impact of diversity offline data sources. We regulate the diversity of dataset by decreasing number prompts in 2/5 times and providing 2/5-times more generations for each existing prompt, while preserving the size of the dataset. Whereas the dynamics of the train loss and proxy metric are almost the same, the effect of teacher hacking becomes more evident with a less diverse dataset.
These results highlight that teacher hacking can harm ground-truth performance, particularly when training in- volves multiple epochs on same dataset. However, we would like to emphasize that this issue is not present when training limited to small number of epochs (e.g., 1-3), as the golden metric remains stable in these cases. We sum- marize the conclusions of these first experiments as follows.
other compared to k generations for different prompts. In our experiments, we apply this technique for k 2 and = 5.
Second, investigate how increasing the generation budget influences teacher hacking. For fixed integer N, generate responses for each prompt in D promptt resulting in a dataset of size m N. Despite the larger dataset size, we define epochs as passing through data points to ensure comparability across experiments. This setup enables us to interpolate between using an offline data source and an online teacher data source. We use values
Observation 2. Employing online data generation limiting training to few epochs effectively prevents teacher hacking.
and 3 for our experiments. The rest of the experimental setup follows the description in Section 4.1.
4.3. How to Mitigate Teacher Hacking?
In the next experiment, evaluate different methods for modifying the diversity and amount of offline data, in order to investigate how the properties of the offline data affect the teacher hacking phenomenon.
Diversity of offline data sources. We begin by examin- ing the impact of dataset diversity on training dynamics and the previously observed teacher hacking phenomenon. The results are shown in Figure 6. Notably, while the training loss and proxy metric dynamics remain nearly the same, the golden metric behaves differently: lower dataset diver- sity leads to worse golden metric performance, making the teacher hacking effect more evident.
Setup. For this experiment, we evaluate different ap- proaches constructing offline data source. We define an ordinary offline data source one that uses all available prompts and single generation for each prompt: Domine {(v,, y:) y: ~ Pe(- x:)) x.)) where Dprompt = {x:)
Generation budget. Next, we examine the effect of larger generation budget on training dynamics, as shown in Figure 7. Increasing the number generations per prompt uniformly improves both the proxy and golden metrics over time and, oppositely, alters the training loss dynamics. Over- all, this suggests an interpolation between the behavior of ordinary offline and online teacher data sources.
Our first objective is to study how the diversity of the offline dataset impacts the teacher hacking phenomenon under fixed dataset generation budget. Let & N be natural number. To construct dataset with reduced diversity, we sub-sample [N/k] prompts from Pprompt and generate k responses for each sampled prompt using the teacher model. The resulting dataset maintains the same generation budget of total responses but exhibits reduced diversity because the le generations for the same prompt x are closer to each
The results suggest two practical strategies to mitigate the effects of teacher hacking:
Discussion.

---

On Teacher Hacking in Language Model Distillation
XSum, Forward KL loss, T5-large T5-base
Offline
Offline (2x gen)
Offline (3x gen)
Online (teacher)
Train loss
Proxy metric
Golden metric
0.442
18.3
20.9
KL (T m
train loss
0.238
13.4
19.4
KL. (p.
0.128
9.72
18.1
0.0688
7.08
16.8
3
0.3
3
S
0.3
3
1 +
epochs
epochs
epochs
Figure 7. Impact of generation budget for offline data sources. As the number of generations per prompt increases, both proxy and golden metrics improve, suggesting that the effect of teacher hacking is decreasing.
2024).
Observation 3. Prioritize Prompt Diversity. When the generation budget for the distillation dataset is fixed, focusing increasing the diversity of prompts can help reduce the impact of teacher hacking.
Knowledge distillation (KD) was initially introduced as a method to compress a large teacher model into a smaller stu- dent model without much loss in performance (Buciluă al., 2006; Hinton al., 2015); has then been successfully used to create smaller language models such DistilBERT (Sanh et al., 2020), Zephyr (Tunstall et al., 2023), Gemma-2 (Riv- iere et al., 2024), Gemini-1.5 Flash (Georgiev et al., 2024), and DeepSeek-Ve (Liu et al., 2024). One of the actively used approaches to language model distillation sequence- level KD (Kim Rush, 2016). In our case, this approach corresponds to utilizing offline data sources and forward KI. token-level loss (thanks properties of the KI.). Other approaches focus matching different quantities of the teacher model by the student model, such as hidden states (Jiao et al., 2020; Zhang et al., 2024) or attention scores (Wang et al., 2020).
Observation 4. Expand the Dataset with Multiple Completions. If the prompt dataset is fixed, increas- ing the generation budget by generating multiple com- pletions per prompt also helps diminish the effects of teacher hacking.
5. Related Work
Goodhart's s law states: "When measure becomes target, it ceases to be good measure" (Strathern, 1997). In partic- ular, it manifests itself as reward hacking in RLHF (Amodei et al., 2016; Gao et al., 2023; Weng, 2024). A line of works studied reward hacking under controlled experimental se- tups (Gao et al., 2023; Rafailov et al., 2024). Our setup closely resembles that of Gao al. (2023), where types of reward models (RMs) used: golden reward model, which substitutes the ground-truth reward function, and proxy reward model, trained on golden RM-preferred gener- ations as ground-truth preferences. Specifically, we employ oracle and teacher models in the same roles as the golden and proxy RMs, respectively: the teacher model is trained on wracle-generated data, while the final student model is trained using the teacher's next-token distribution.
Most of the neviiously-mentioned works implicitly or ex- plicitly assume that the replication of the teacher model represents the final goal of the distillation process, contrary to the approach mentioned by Menon et al. (2021), where the teacher is only an imperfect approximation of the true data distribution. Based on this perspective, Zhang et al. (2023) developed a perturbed loss that can be seen as training from "proxy teacher". However, they do not further investigate the consequences over-optimizing the objective.
6. Conclusion
In this paper, we introduce and examine the phenomenon of teacher hacking in language model distillation by design- ing a semi-synthetic controlled experimental setup. This allows us to measure its effects, and validate experimen- tally its presence when using a fixed offline dataset for the
Given possible negative consequences of reward hacking (Hendrycks et al., 2021; Wen et al., 2025), another line of research attempts to mitigate its effects through better reward modeling or more robust training procedures (Chen al., 2024; Rame et al., 2024; Liu et al., 2025; Miao al.,
8

---

On Teacher Hacking in Language Model Distillation
distillation procedure.
tional Linguistics. doi: 10.315551//11433322 URL httts://aclartthol.e www.llgggyogggggoo.o 0t/g///14-3302/
Fortunately, practical outcome of our study, we were able to identify several strategies to mitigate teacher hack- ing: (1) utilize online generations during the distillation process, (2) when the generation budget is fixed, prioritize increasing the diversity of the prompt dataset, and (3) the prompt dataset is fixed and online generations are not feasi- ble, generate multiple offline completions per prompt ahead of time to expand the dataset. We hope that these practical and methodological insights provide valuable guidance in extending the applicability and effectiveness of language model distillation in real-world scenarios.
Buciluă, C., Caruana, R., and Niculescu-Mizil A. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Chen, L.,. Zhu, C., Chen, J., Soselia, D., Zhou, T., Gold- stein, T., Huang, H.. Shoeybi, M., and Catanzaro, B. ODIN: Disentangled reward mitigates hacking in RLHF. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Pro- ceedings of the 41st International Conference Ma- chine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 2024. Utthttps: /tte:s//prroeeee /tpooooooediccoo ngs. ml prrsss/ v235/ / chen2 4bn. html.
Impact statement
PMLR, 21-27 Jul
This paper presents work on language model distillation, which is actively used in the training of many modern lan- guage models. We identify a possible shortcoming of ex- isting distillation procedures, called teacher hacking, that can lead to the transfer of unsafe behaviors from teacher to student. Additionally, we proposed several strategies to reduce the effect of this phenomenon. We believe that under- standing and identifying such issues have positive societal consequences and allow the development of more reliable and safe language models.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang. X., Dehghani, M., Brahma, S., et al. Scaling instruction finetuned
restructionffinetuned language models. Journal of Machine Learning Research, 25(70): 2024.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835- PMLR, 2023.
References
Georgiev, P., Lei, V. I., Burnell, R., Bai, L.,, Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024. URL https: http///epppee.eeeeeeeeeee bet/forumm
Gu, Y., Dong, L., Wei, F., and Huang, M. MiniLLM: Knowl- edge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL /ttp:::////p//eeeeeeeeee wtteerrvveeweee//eeeee net/ forum? i id-5hogf7IB22.
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Mané, D. Concrete problems in Al safety. arXiv preprint 1606.06666656 2016.
Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. Unsolved problems in ML safety. arXiv preprint vrXiv:2109.13916 2021.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowl- edge in a neural network, 2015.
Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P.. Leveling, J., Monz, C., Pecina, P., Post, M., Saint- Amand, H., Sorieut, R., Specia, L., and Tamchyna, A. statistical machine
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. TinyBERT, Distilling BERT for natural language understanding. In Cohn, T., He, Y.. and Liu, Y. (eds.), Findings of the Association for Com- putational Linguistics: EMNLP 2020, pp. 4163-4174, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18655/v1/2020.fnddl http:///000..iinneeeemmil/
Findings of the 2014 workshop translation. In Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Monz, C., Post, M., and Spe- cia, L. (eds.), Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 12-58, Baltimore, Maryland, USA, June 2014. Association for Computa-
372. URL https: pettts::/aclae /tt/aaaaaaaaalggaaa/aaaaaaa wwwp:ogg...ggg..oo./...//

---

On Teacher Hacking in Language Model Distillation
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
Menon, A. K., Rawat, A. S., Reddi, S., Kim, S., and Kumar, S. A statistical perspective on distillation. In Interna- tional Conference on Machine Learning, pp. 7632-7642. PMLR, 2021.
Kim, G., Jang, D., and Yang. E. PromptKD: Dis- tilling student-friendly knowledge for generative lan- guage models via prompt tuning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Find- ings of the Association for Computational Linguis- ties: EMNLP 2024, pp. 6266-6282, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10. 10.18653/w1/2224.nnnngg
Miao, Y., Zhang. S., Ding, L., Bao, R., Zhang, L., and Tao, D. Inform: Mitigating reward hacking in rlhf via information-theone
reward modeling. In The Thirty- eighth Annual Conference on Neural Information Pro- cessing Systems, 2024.
Mishra, S., Khashabi, D., Baral, C.,, and Hajishirzi, H. Cross- task generalization via natural language crowdsourcing instructions. In ACL., 2022.
364. URL https: / ac findings-emmip. 364/.
Narayan, S., Cohen, S. B., and Lapata, M. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October-November 2018. Association for Com- putational Linguisics.. doi: 10. 10.11863/v1/D18-122 1206. URL https: / http:///aaaaaahllaaeooeo ort/
Kim, Y. and Rush, A. M. Sequence-level knowledge dis- tillation. In Su, J., Duh, K., and Carreras, X. (eds.), Proceedings the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pp- 1317-1327, Austin, Texas, November 2016. Association for Compu- tational Linguistics. doi: 10.18653/1/D11-1110 URL https: / aalaacaannttht hwtpaaaaaannnnnaoaaaaaaaaaooolla wwtp://////D/1111/11
Ko, J., Kim, S., Chen, T., and Yun, S.-Y. DistiLLM: To- wards streamlined distillation for large language mod- cls. In Forty-first International Conference on Machine Learming, 2024. URL https: / ht////ppeereeeeeeeeeeeee
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 2022.
Lin, A., Wohlwend, J., Chen, H., and Lei, T. Autoregres- sive knowledge distillation through imitation learning. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6121 6133, Online, November 2020. Association for Computa- tional Linguistics. doi: 10.18665/ 494. URL https: / htt///aaaantclll.c.raeeeoeeee emnlp-main 494/.
Pan, A., Bhatia, K., and Steinhardt, J.
The effects of reward misspecification: Mapping and mitigating mis- aligned models. In International Conference on Learning Representations, 2022. URL https: opwnrevie..eeeeeeee net
Rafailov, R., Chittepu. Y., Park, R., Sikchi, H. S., Hejna, J., Knox, B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems, 37: 37:126207 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- V3 Technical report. arXiv preprint arXiv:2412 2024.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67,200)
Liu, T., Xiong, W., Ren, J., Chen, L., Wu, J., Joshi, R., Gao, Y., Shen, J., Qin, Z., Yu, T. Sohn, D., Makarova, A., Liu, J. Z., Liu, Y., Piot, B., Ittycheriah, A., Kumar, A., and Salch, M. RRM: Robust reward model training mitigates reward hacking. In The Thirteenth International Confer- ence on Learning Representationss 2025, URL https: !/openreview www.reeiieww... net/ / forum? i d= BAS 5MQmm.C.
Rame, A., Vieillard, N., Hussenot, L., Dadashi-Tazeon Tazehozi. R., Cideron, G., Bachem, O., and Ferret, J. WARM: On the benefits of weight averaged reward models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp. F. (eds.), Pro- ceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42048 42073. PMLR, 21-27 Jul
I Longpre, S., Hou, I.., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le. Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference on Machine Learning, pp. 22631 1-2648.. PMLR, 2023.
10

---

On Teacher Hacking in Language Model Distillation
2024. URL https: wwt..htttssppoceeeeeeeeeoe http://pooeddddee .tr.ediinnsmmmm..........mmm mlr wwl..prrss////.......... v235/rame24a html.
Wang, Y., Mishra, S., Alipoormolabashi, P., Ko- rdi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A. S., Naik, A., Stap, D., et al. Super- via declarative instruc- tions on 1600+ tasks. In EMNLP, 2022.
Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L.., Mesnard, T., Shahriari, B., Ramé, A., et al. Gemma 2: Improving open language models practical size. arXiv preprint arXiv:2408. 001 18, 2024.
naturalinstrrctionssgnneerrr
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le. Q. V. Finetuned lan- guage models are zero-shot learners. In International Conference on Learning Representations, 2021.
Roberts, A., Chung, H. W., Levskaya, A.. Mishra, G., Brad- bury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., Hawthorne, C., Lewkowyez, A., Sal- cianu, A., Zee, M., Austin, J., Goodman, S., Soares, L. B., Hu, H., Tsyyashchenko, S., Chowdhery, A., Bast- ings, J., Bulian, J., Garcia, X., Ni, J., Chen, A., Kenealy, K., Clark, J. H., Lee, S., Garrette, D., Lee-Thorp. J., Raffel, C., Shazeer, N., Ritter, M., Bosma, M., Passos, A., Maitin-Shepard, J., Fiedel, N., Omernick, M., Sacta, B., Sepassi, R., Spiridonov, A., Newlan, J., and Ges- mundo, A. Scaling up models and data with t5x and seqio. arXiv preprint arXiv: 17189, 2022. URL https: http//aaxvvvvv..///iv org/ abs 221./2223.1111
Wen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., Bowman, S. R.,, He, H., and Feng, S. Lan- guage models learn to mislead humans via RLHF. In The Thirteenth International Conference on Learning Representations, 2025. URL https: htpp//ppenne..eeeeeeeee net net fort/formm?e (o-oummidd=x11iiPP55.
Weng, L. Reward hacking in reinforcement learn- ing. lilianweng.githubbie Nov 2024. URL
http:://iiaaweenggiittttt io/posts/ hackingg..aa.ig/./...
Sanh, V., Debut, L.,, Chaumond, J., and Wolf, T. DistilBERT, distilled version of BERT: smaller, faster, cheaper and lighter, 2020. URLhttps: /ttp://aaxvvvvvv www.iiiooo.gas/////ooo/op/p
Zhang, R., Shen, J., Liu, T., Liu, J., Bendersky, M., Najork, M., and Zhang, C. Do not blindly imitate the teacher: Using perturbed loss for knowledge distillation. arXiv preprint arXiv:2305.05010, 2023.
Song. F. Yu, B., Lang, H., Yu, H., Huang, F., Wang, H., and Li, Y. Scaling data diversity for fine-tuning language models in human alignment. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint Interna- tional Conference on Computational Linguistics, Lan- guage Resources and Evaluation (IREECCOLING 2024), pp. 14366, Torino, Italia, May 2024. ELRA and ICCL. URL https : / /aclaantollggy...../.... org/2024. Trec-main.122.
Zhang, S., Zhang, X., Sun, Z., Chen, Y., and Xu, J. Dual- space knowledge distillation for large language models. Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods Natural Language Processing, pp. 18164-18181, Miami, Florida. USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/0224. emnlp- main. 1010. URL htttp:://aclaattollllo org / 2024 emn po.main... in. 10000/
Stiennon, N., Ouyang, L.,, Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Strathern, M. 'improving ratings': audit in the british uni- versity system. European review, 5(3):305-321) 1997.
Tunstall, I., Beeching. F.,, Lambert, N., Rajani, N., Ra- sul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023.
Wang, W. Wei, F., Dong, L.., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distillation for aask-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33: 5776-5788, 2020.

---

On Teacher Hacking in Language Model Distillation
WMT-14 en-de, Forward KL loss, T5-large T5-base
Offline
Online (student)
Online (teacher)
Train loss
Proxy metric
Golden metric
0.338
22.8
27.8
KLag (T Teg
train loss
0.192
13
25.6
0.109
7.43
KT
23.6
0.0617
4.24
21.7
9.3
3
0.3
epochs
epochs
epochs
Natural Instructions, Forward KL loss, T5-large to T5-base
Train loss
Proxy metric
Golden metric
0.402
4.56
6.32
sly (Tt, Tal
train loss
0.182
3.18
5.67
KL (n,
0.0824
2.22
5.09
0.0373
1.54
4.57
0
3
10
50
0.
30
50
0.3
3
20
50
epochs
epochs
epochs
Figure 8. Impact of the dataset choice: offline vs. online data sources. We verify our claims on the presence of teacher hacking in the of offline data sources for different tasks: the translation task WMT-14 en-de (top row) and the instruction following task Natural Instruction (bottom row). In general, the behavior of the curves is the same across all the datasets: for online data sources, both proxy and golden metrics are decreasing. At the same time, for oflline data sources, the proxy metric is decreasing stagnating, whereas the golden metric clearly increasing.
A. Additional Experiments
This section presents additional experiments on teacher hacking across various datasets, model sizes, and loss types, as well as an additional experiment on the effect of mixing offline and online data.
A.1. The Impact of the Dataset
In this subsection, we validate our claims across various tasks, such as the translation task on the WMT-14 en-de dataset (Bojar et al., 2014) and the instruction-followingg task on the Natural Instructions dataset (Mishra et al., 2022: Wang et al., 2022). We consider the same pair of models: T5-large as the teacher model and T5-base as the student model, utilizing forward KI. token-level loss along with the corresponding proxy and golden metries.
Online offline data sources. The results of the comparison of online and offline data sources the WMT-14 en-de and Natural Instruction datasets are presented in Figure 9.
Overall, we observe the same phenomenon as noted in the summarization task in Section 4. The general patterns are consistent: for all data sources, the training loss decreases slowly for online data sources, as expected, while the proxy metric decreases or stagnates, showing no signs of classical overfitting. However, the golden metric continues to decline in the case of offline data sources, and it starts declining in the case of offline data source, indicating the presence of teacher hacking.
12

---

On Teacher Hacking in Language Model Distillation
WMT-14 en-de, Forward KL loss, T5-large T5-base
Offline
Offline (x0.5 prompts, 2x gen)
Offline (x0.2 prompts, 5x gen)
Train loss
Proxy metric
Golden metric
0.338
22.5
27.8
KLong (Tt.. nTa
soy Ma.
train loss
0.191
13.8
26
0.108
8.45
24.3
0.061
5.18
22.6
0
0.3
3
0.3
3
epochs
epochs
epochs
Natural Instructions, Forward KL loss, T5-large to T5-base
Train loss
Proxy metric
Golden metric
0.353
6.32
KI sly (Tt, Tal
train loss
0.165
3.46
5.79
KL (M.
0.0773
2.56
5.31
0.0362
1.9
4.86
0
3
50
30
50
0.
3
10
30
50
epochs
epochs
epochs
Figure 9. Impact of the dataset choice: dataset diversity. Across other datasets, we can notice that the impact of diversity still present for the cestuuttion-follwiing task but not for the translation task. can be explained the initially small diversity of the WMT-14 dataset.
Diversity of offline data sources. Next, we study the impact of the diversity of the dataset on teacher hacking, following the setup described in Section 4.3. The results are presented in Figure 9. We observe the same detrimental effect of the decreasing of the dataset diversity in the case of the instruction following task. However, in the case of the translation task, almost has no effect. We connect this effect to the small change in diversity of the initial dataset since contains only relatively short sentences. To validate this claim, we compute the quantitative measurement of diversity, used by Song et al. (2024), that is equal to ratio of unique bigrams all bigrams in the tokenized dataset, multiplied by the size the dataset, Table In particular, can that for WMT- 14 dataset, the decay of quantitative diversity with reducing number of unique prompts is less dramatic, explaining a little change from point of view of proxy/golden metries.
Generation budget. Finally, we verify the claims on increasing the generation budget for the offline data sources. The results are presented in Figure 10. In this case, for both tasks, we observe an improvement in the golden metric, especially for the translation task, and we observe a marginal improvement in the proxy metric. As in the case of the summarization task, it signals the decreasing teacher hacking effect.
A.2. The impact of student and teacher model sizes
In this subsection, we validate our findings across different student and teacher model sizes using the XSum dataset and forward KL token-level loss.
Online vs. offline data sources. We conduct distillation experiments from T5-base to T5llmall and from T5-large to T5-small, evaluating performance across offline and online data sources. The results are shown in Figure 11.
13

---

On Teacher Hacking in Language Model Distillation
WMT-14 en-de, Forward KL loss, T5-large T5-base
Offline
Offline (2x gen)
Offline (3x gen)
Online (teacher)
Train loss
Proxy metric
Golden metric
0.338
train loss
0.192
12.7
25.6
KL (T
KT (p.
0.109
7.35
23.6
0.0617
4.24
21.7
50
9.3
3
0.3
epochs
epochs
epochs
Natural Instructions, Forward KL loss, T5-large to T5-base
Train loss
Proxy metric
Golden metric
0.355
4.6
6.32
Kls (Tt, Ta.
train loss
0.167
3.2
5.67
KL (M,
0.079
2.22
5.09
0.0373
1.54
4.57
0.
3
10
0.3
20
50
03
3
50
epochs
epochs
epochs
Figure 10. Impact of the dataset choice: generation budget. We additionally confirm the claim on the positive impact a larger number of generations per prompt two other datasets.
For online data sources, both the proxy and golden metrics decrease monotonically, regardless of the model sizes, confirming our carlier observations. In contrast, for offline data sources, the golden metric consistently increases. Notably, in the case of distilling Targerge to T5-small, the proxy metric also shows slight increase, indicating standard overfitting rather than teacher hacking. Meanwhile, when distilling T5-base T5-small, the proxy metric stagnates, suggesting the presence of teacher hacking.
A.3. The Impact of Token-Level Loss Functions
In this subsection, examine how the choice of token-level loss function and proxy/golden metrics influences the teacher hacking phenomenon. The experiments are conducted using the XSum summarization dataset, with T5-base the student model and T5-large as the teacher model.
Online vs. offfine data sources. This experiment compares different data sources using two token-level loss functions: divergence and reverse KL divergence, alongside their corresponding proxy and golden metrics. The results shown Figure 12.
The observed behavior across all plots closely resembles that of the forward KL loss. Specifically, when using online data sources, both proxy and golden metrics decrease monotonically. In contrast, for offline data sources, proxy metrics continue to decrease, but the golden metric increases, signaling the presence of teacher hacking.
divergences with coefficients of 0.1 and 0.9. However, their performance was either comparable or worse than reverse forward KI. divergence for the respective
Additional experiments were conducted using generalized
14

---

On Teacher Hacking in Language Model Distillation
Table Diversity measurement. Following an approach Song et al. (2024), we report dataset diversity the ratio of unique bigrams in tokenized prompt-completio pairs to the total number of tokenized bigrams, multiplied by the dataset size. Additionally, we report pommptggeneaation ratio. These measurements explain why the influence
the dataset diversity ratio to dataset's diversity of diversity reduction WMT is much smaller: diversity changes in the prompt-completion dataset are not dramatic in the summarization sesluutioo--oolooingg datasets.
Prompt-Generation Ratio
Dataset
0.2:5
0.5:2
1:1
1:2
1:3
XSum
4813.3 (0.47)
7292.1 (0.72)
10140.1 (1.00)
10645.8 (1.05)
11112.9 (1.10)
WMT-14 48079.4 (0.77)
54511.5 (0.87)
62652.0 (1.00)
85855.5 (1.37)
105242.1 (1.68)
Natural Instructions
8464.1 (0.45)
13311.6 (0.71)
18751.5 (1.00)
19631.7 (1.05)
20430.0 (1.08)
XSum, Forward KL loss, T5-base to T5-small
Offline
Online (student)
Online (teacher)
Train loss
Proxy metric
Golden metric
0.636
23.7
31.2
train loss
0.384
17.7
28.1
Kloug (Te.
KI soy (1.
0.232
13.2
25.2
0.14
9.86
22.7
2
0.3
epochs
epochs
epochs
XSum, Forward KL loss, T5-large to T5-small
Offline
Online (student)
Online (teacher)
Train loss
Proxy metric
Golden metric
0.791
27.5
32.1
KLag [T Re)
train loss
0.453
20.8
28.1
KLog (e..
0.259
15.7
24.7
0.148
11.9
21.7
03
3
10
30
50
3
30
50
0.3
10
30
50
epochs
epochs
epochs
Figure Impact the model sizes: offline vs. online data sources. We examine the train loss and proxy/golden metrics for model size pairs: 75-base as the teacher and T5-small as the student (top), and T5-large as the teacher and T5-small as the student (bottom). For both pairs, no teacher hacking oceurs with online data generation. However, teacher hacking is observed during distillation from TS-base to T5-small, as the proxy metric stagnates while the golden metric decreases. In contrast, distillation from T5-large to T5-small shows behavior consistent with standard overfitting as the proxy metric slightly increases. This may be attributed larger difference model sizes.
proxy and golden metrics. As a result, they are excluded from the comparison.
15

---

On Teacher Hacking in Language Model Distillation
XSum, Jensen-Shannon Shannon loss, T5-large to T5-base
Offline
Online (student)
Online (teacher)
Train loss
Proxy metric
Golden metric
0.0945
3.52
4.26
Saa (Tg. (L)
train loss
0.0527
2.69
3.83
.SSug (Te
0.0294
2.05
3.44
0.0164
1.56
3.09
9.3
3
03
epochs
epochs
epochs
XSum, Reverse KL loss, T5-large to T5-base
Offline
Online (student)
Online (teacher)
Train loss
Proxy metric
Golden metric
0.479
20.9
28.2
KI oog [Ts. Rt.
train loss
0.253
14.9
25.1
Klam (Te
0.133
10.7
22.4
0.0705
7.6
19.9
10
50
0.3
10
30
50
0.3
10
30
50
epochs
epochs
epochs
Figure Impact loss type: offfine vs. online data sources. can observe the effect of teacher hacking appears regardless of the choice of loss function.
A.4. Offline-Online Data Mixtures
In this subsection, we examine how varying mixtures offline and online data influence the occurrence of teacher hacking. The experiment uses the XSum dataset, T5-base as the student model, T5-large the teacher model, and forward KL token-level loss, following Section 4.
We use the following procedure to generate each training batch: with probability the batch is sampled from a fixed offline dataset, and with probability a, it is generated by the current student model. This process is repeated at each training step. We evaluate three values for a: 0.1, 0.5, and 0.9, corresponding to 10%, 50%, and 90% offline data proportion, respectively. The results are shown Figure 13.
Our results show that increasing the proportion of online data in the distillation process (that is equivalent to decreasing the proportion of offline data) significantly improves the golden metric. Even with 10% online student data (corresponding to 90% offline data), the golden metric plateaus, thus effectively reducing teacher hacking. Higher proportions of online data further mitigate the effect, with 90% online student data resulting in training dynamics nearly identical to those observed when using only student-gent -geeeratd data.
These results suggest that exclusively using online- geiteeege data is not required to avoid teacher hacking. Instead, incorporating a fraction of maline-generatee data during the distillation process sufficient. In particular, as little as 10% online data can substantially reduce the impact of teacher hacking.
Discussion.
16

---

On Teacher Hacking in Language Model Distillation
XSum, Forward KL loss, T5-large to T5-base
Mixture, 10% offline
Mixture, 50% offline
Mixture, 90% offline
Offline
Online (student)
Train loss
Proxy metric
Golden metric
0.489
21.3
KI kla (Tt,. Tta
train loss
0.254
13.3
19.8
KL 200
0.132
9.73
18.3
0.0688
7.11
17
0.3
3
90
0.3
1
50
0.3
3
50
epochs
epochs
epochs
Figure Mixture of offline and online data. This plot compares strategies combining offline and online data during the distillation process. The results show that incorporating just 10% online student data significantly reduces the effect of teacher hacking, causing the golden metric to stabilize rather than increase. At the same time, the usage of at least 50% of the online generated data allows to avoid the effect of teacher hacking completely.
B. Additional Details
this section, we provide an exact formulation for sequence-level divergences
Kaseg (TT, x²) A Egond prrpp
Σ KT ya)), o (+a) v:)))
In particular, to estimate this KL-divergence, need to sample prompts and generate responses using language model Additionally, we introduce a sequence-based Jensen-Shannon divergence, defined
yl
1 Ie'l
JSseq (T,, x²) & (w
(L(T Ba), m( va)) +
_ KL (11, (.)),,((aa)) v.:1
2
2
(4)
where m(w]ar, v) A 0.5 [--|,, y) + 0.5 n (w]ax, y) is mixture of two language models. estimate this divergence, we need to use samples from both models and n' and compute an average of two token-level KL-divergences. Notice that computation of true Jensen-Shannon divergence between pa and per is computationally infeasible since the log-probabilities of the mixture 0.5 Ps (v/z) + 0.5 (y/()) does not satisfy chain rule in terms of and n².
C. Hyperparameters
In this section, we provide detailed information on the hyperparameters used for our experiments; see Table and Table
17

---

On Teacher Hacking in Language Model Distillation
Table 2. Hyperparameter details for summarization & translation tasks.
Hyperparameter
Value
Hyperparameter
Value
Oracle Dataset Size
100,000
Oracle Dataset Size
50,000
Distillation Dataset Size
200,000
Distillation Dataset Size
450,000
Training Steps
390,625
Training Steps
703,125
Batch Size
32
Batch Size
32
Task
XSum
Task
WMT-14 14 en-de
Dropout
0.0
Dropout
0.0
Warmup Schedule
100 steps
Warmup Schedule
100 steps
Optimal Learning Rate (LR)
0.0003
Optimal Learning Rate (LR)
0.0003
Input Length (Tokenized)
1024
Input Length (Tokenized)
80
Output Length (Tokenized)
128
Output Length (Tokenized)
80
Softmax Temperature
1.0
Softmax Temperature
1.0
Table 3. Hyperparameter details for instruction following task.
Hyperparameter
Value
Oracle Dataset Size
100,000
Distillation Dataset Size
500,000
Training Steps
390,625
Batch Size
64
Task
Natural Instructions
Dropout
0.0
Warmup Schedule
100 steps
Optimal Learning Rate (LR)
0.0003
Input Length (Tokenized)
2048
Output Length (Tokenized)
256
Softmax Temperature
1.0
18


Reviews:
[{"review_id": "CAgFzoMVit", "reviewer": "Reviewer_z8Ux", "summary": "This work investigates a novel phenomenon termed \"teacher hacking,\" where student language models (LMs) over-optimize to imperfections in the teacher model during knowledge distillation, leading to degraded performance on the true objective. The authors propose a controlled experimental setup involving an oracle model (ground-truth distribution), a teacher model distilled from the oracle, and a student model distilled from the teacher. Through systematic experiments, they demonstrate that teacher hacking occurs when using fixed offline datasets for distillation but can be mitigated using online data generation techniques. The study highlights data diversity as a critical factor in preventing teacher hacking and provides practical strategies to address the issue.\n\nPros:\n1) The paper identifies and formally defines the \"teacher hacking\" phenomenon, drawing an insightful analogy to \"reward hacking\" in reinforcement learning from human feedback (RLHF). This perspective bridges gaps in understanding distillation limitations.\n2) The semi-synthetic framework with an oracle model provides a rigorous way to measure ground-truth performance (golden metrics) and proxy metrics (teacher-student alignment). This setup allows clear detection of teacher hacking through U-shaped proxy-golden curves.\n3) The authors provide actionable solutions, such as online data generation, increasing prompt diversity, and expanding datasets with multiple completions. These strategies are validated across multiple tasks and model sizes.\n4) The study includes experiments across diverse tasks (summarization, translation, instruction following), model architectures (T5 variants), and loss functions (forward/reverse KL, Jensen-Shannon). This breadth strengthens the generalizability of findings.\n\nCons:\n1) Dataset Diversity Limitations:\n    * While the paper emphasizes data diversity, the experiments on the WMT-14 en-de translation task show minimal impact of dataset diversity on teacher hacking (Fig. 9). This inconsistency suggests the phenomenon may be task-dependent, with translation tasks being less sensitive to diversity manipulations.\n    * The \"x0.5 prompts, 2x gen\" and \"x0.2 prompts, 5x gen\" experiments (Fig. 6) reduce prompt diversity but increase generations per prompt. However, the analysis does not quantify the trade-off between prompt diversity and generation redundancy, leaving ambiguity about optimal resource allocation.\n2) Model Size and Capacity Mismatch:\n    * The experiments distilling T5-large to T5-small show proxy metric increases indicative of classical overfitting rather than teacher hacking (Fig. 11). This suggests the framework may conflate overfitting with teacher hacking when model capacity gaps are large, weakening the specificity of the teacher hacking diagnosis.\n    * The study focuses on T5-based models, limiting generalizability to other architectures (e.g., transformer-decoder LMs). The phenomenon's dependence on architectural differences remains unexplored.\n3) Experimental Design Gaps:\n    * The offline-online data mixture experiments (Fig. 13) use fixed α values (10%, 50%, 90%) but do not systematically vary α across a continuous range. This prevents identifying the minimum online data proportion required to suppress teacher hacking.\n    * The golden metric improvements from increased generation budgets (Fig. 7) are marginal for proxy metrics, suggesting diminishing returns. The paper does not analyze cost-benefit trade-offs for different generation strategies.\n4) Task-Specific Sensitivity:\n    * Teacher hacking effects are more pronounced in instruction-following tasks (Natural Instructions) than translation tasks (WMT-14 en-de). The paper does not investigate why certain tasks are more susceptible, potentially due to inherent dataset properties or evaluation metrics.\n5) Evaluation Metric Limitations:\n    * The golden metric (distance to oracle) is task-agnostic and may not correlate with downstream task performance. The paper lacks task-specific evaluations (e.g., BLEU for translation, ROUGE for summarization) to validate practical implications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer z8Ux for the detailed and valuable feedback! In the following, we address the issues raised in the review.\n\n## Dataset diversity limitations.\n\n> **The analysis doesn't quantify the trade-off between prompt diversity and generation redundancy**\n\n> **The experiments don't establish diversity as the sole or primary factor, as other aspects of data quality may also play roles**\n\n> **Dataset diversity: Use quantitative metrics to measure dataset diversity.**\n\n\nWe emphasize that our study focuses not just on the diversity of the prompts but on the diversity of the entire prompt-completion dataset. We found that our approach is the only one that satisfies the following three key properties:\n\n(1) Preserving the conditional distribution of completions given the prompt – ensuring that answer quality remains the same.\n\n(2) Reducing the total diversity of the dataset.\n\n(3) Preserving dataset size – eliminating data quantity as a confounding factor.\n\nWe are unaware of any experimental setup that maintains dataset size and generation redundancy while modifying prompt diversity. However, if the reviewer can suggest one, we would happily implement it. The closest related work, Song et al. (2024), varied prompt dataset diversity by increasing the number of prompts sampled. This violates (3) and prevents us from isolating the effect of data diversity from total data quantity. \n\nFollowing Song et al. (2024), we report dataset diversity as the ratio of unique bigrams in tokenized prompt-completion pairs to the total number of tokenized bigrams, multiplied by the dataset size. Additionally, we report the dataset diversity ratio to the dataset's diversity with a 1:1 prompt-generation ratio.\n\n| Dataset \\ prompt-generation ratio | 0.2:5       | 0.5:2       | 1:1         | 1:2         | 1:3         |\n|-----------------------------------|------------|------------|------------|------------|------------|\n| XSum                              | 4813.3 (0.47)  | 7292.1 (0.72)  | 10140.1 (1.0)  | 10645.8 (1.05) | 11112.9 (1.1)  |\n| WMT-14                            | 48079.4 (0.77) | 54511.5 (0.87) | 62652.0 (1.0)  | 85855.5 (1.37) | 105242.1 (1.68) |\n| Natural Instructions              | 8464.1 (0.45)  | 13311.6 (0.71) | 18751.5 (1.0)  | 19631.7 (1.05) | 20430.0 (1.08)  |\n\n\nThese measurements explain why the influence of diversity reduction on WMT is much smaller: diversity changes in the prompt-completion dataset are not as dramatic as in the summarization or instruction-following datasets. Another factor is that the prompts and completions in the WMT-14 dataset are much shorter than in XSum and Natural Instructions. We will integrate this discussion into the manuscript.\n\n#### Reference:\n\nSong et al. (2024). Scaling data diversity for fine-tuning language models in human alignment. COLING-2024.\n\n## Model Size and Capacity Mismatch\n\n> **[...] the framework may conflate overfitting with teacher hacking when model capacity gaps are large, weakening the specificity of the teacher hacking diagnosis.**\n\n> **Limited architectural exploration**\n\nWe agree that it would be interesting to study the phenomenon of teacher hacking on different architectures and with different model capacity gaps. However, we leave this direction for further work. \n\n\n## Experimental Design Gaps\n\n> **[...] The paper does not analyze cost-benefit trade-offs for different generation strategies.**\n\n> **The experimental analysis would benefit from more systematic variation of parameters in the offline-online data mixture experiments.**\n\n\nWe agree that it would be valuable to identify the minimal additional compute needed to suppress the effect of teacher hacking. We left the question of exploration of optimal trade-offs as a direction for further work.\n\n\n## Task-Specific sensitivity\n\n> **[...] The paper does not investigate why certain tasks are more susceptible, potentially due to inherent dataset properties or evaluation metrics.**\n\nWe agree that studying teacher hacking on a larger variety of datasets and tasks might be interesting, especially given the task-dependent strength of some of our recommendations. We leave this study as a promising direction for further work.\n\n\n## Evaluation metric limitations\n\n> **Task metrics: Include task-specific metrics for comprehensive assessment.**\n\nThe decision to use the distance to the oracle as the only ground-truth evaluation metric is connected to the final problem we are solving. In particular, ROUGE / BLEU metrics between oracle and student generations will serve only as a very noisy and imperfect proxy to the distance between student and oracle distributions, which we can evaluate using a less noisy estimation. At the same time, using these metrics with human references would violate the assumption of our setup that the oracle model is a source of ground-truth distribution.\n\n## Missing related work\nWe will happily add suggested references to the related work section.", "claims_and_evidence": "The claims in the submission are generally supported by clear and convincing evidence, though with some notable exceptions:\n\n**Supported Claims**\n- Teacher hacking occurs during knowledge distillation with fixed offline datasets: Strongly supported by experimental results showing U-shaped proxy-golden curves where golden metrics (distance to oracle) deteriorate while proxy metrics (distance to teacher) improve (Fig. 4, Fig. 5).\n- Teacher hacking can be detected by deviations from polynomial convergence laws: Convincingly demonstrated through log-log plots comparing online and offline training dynamics, where offline methods show clear deviations from expected convergence patterns (Fig. 5).\n- Online data generation effectively mitigates teacher hacking: Well-supported across multiple tasks and model sizes, showing consistent improvement in golden metrics when using online data sources (Fig. 5, Fig. 8).\n\n**Problematic Claims**\n- Data diversity is the key factor in preventing teacher hacking: While supported by experiments manipulating dataset diversity (Fig. 6, Fig. 9), this claim has limitations:\n    * The impact of diversity varies significantly across tasks (minimal effect on translation tasks)\n    * The analysis doesn't quantify the trade-off between prompt diversity and generation redundancy\n    * The experiments don't establish diversity as the sole or primary factor, as other aspects of data quality may also play roles", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for investigating teacher hacking in LM distillation:\n\n- Controlled experimental setup: Effectively isolates and measures teacher hacking through oracle, teacher, and student models.\n- Golden and proxy metrics: Capture both ground-truth performance and distillation alignment.\n- Diverse experimental scenarios: Demonstrate generalizability across tasks, model sizes, and loss functions.\n\nSome aspects could be strengthened with task-specific evaluations and more rigorous dataset diversity quantification. Overall, the methods and criteria are well-suited to the problem.", "theoretical_claims": "The paper makes several theoretical claims about teacher hacking, including its definition, detection via deviation from polynomial convergence laws, and mitigation strategies. These claims are logically consistent and supported by mathematical formulations and experimental evidence. The theoretical framework for measuring distances between language model distributions (forward/reverse KL divergence, Jensen-Shannon divergence) is sound and appropriately applied in the analysis. The paper does not present formal proofs requiring rigorous verification but rather builds its arguments on established information-theoretic measures and empirical validation. The theoretical claims appear correct within the context of the problem and experimental setup presented.", "experimental_designs_or_analyses": "Some minor issues could be addressed:\n- The impact of dataset diversity shows variability across tasks (particularly translation tasks), suggesting task-dependent effects that aren't fully explored.\n- The distinction between classical overfitting and teacher hacking could be more clearly established in some model size comparisons.\n- The experimental analysis would benefit from more systematic variation of parameters in the offline-online data mixture experiments.\n\nOverall, the experimental designs are robust and valid for investigating the teacher hacking phenomenon.", "relation_to_broader_scientific_literature": "The paper's key contributions relate to broader scientific literature as follows:\n- Teacher hacking analogy: Extends reward hacking research in RLHF, linked to Goodhart's law (Amodei et al., 2016; Gao et al., 2023).\n- Controlled setup: Similar to experimental designs in reward hacking studies (Gao et al., 2023), using golden/proxy metrics.\n- Mitigation strategies: Align with ML principles emphasizing data quality/diversity.\n- Convergence analysis: Relates to scaling law research (Kaplan et al., 2020).\n- Knowledge distillation foundations: Builds on established techniques (Hinton et al., 2015; Sanh et al., 2020).", "essential_references_not_discussed": "The paper adequately covers relevant prior work in knowledge distillation and reward hacking, but some of the recent works can be discussed, e.g., KD for LMs [1,2], reward hacking in RLHF [3]\n\n[1] Dual-Space Knowledge Distillation for Large Language Models. In Proc. of EMNLP 2024.\n[2] Revisiting Knowledge Distillation for Autoregressive Language Models. In Proc. of ACL 2024.\n[3] Mitigating Reward Hacking via Information-Theoretic Reward Modeling. In Proc. of NeurIPS 2024.", "other_strengths_and_weaknesses": "Strengths\n- Originality: The paper introduces the novel concept of \"teacher hacking\" in LM distillation, analogous to reward hacking in RLHF, and provides a systematic framework to study it.\n- Significance: The findings have practical implications for developing more reliable and safe language model distillation pipelines.\n- Clarity: The paper is well-structured with clear explanations of the experimental setup, methodology, and results.\n\nWeaknesses\n- Limited architectural exploration: The study focuses on T5-based models, limiting generalizability to other architectures.\nTask-specific evaluations: The paper lacks comprehensive task-specific metric evaluations to validate practical implications.", "other_comments_or_suggestions": "- Terminology: Ensure consistent terminology usage.  \n- Task metrics: Include task-specific metrics for comprehensive assessment.  \n- Architectures: Experiment with different model architectures.  \n- Dataset diversity: Use quantitative metrics to measure dataset diversity.", "questions_for_authors": "see weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dx6HMC5rpX", "reviewer": "Reviewer_KCu2", "summary": "This paper identifies and formally defines phenomenon of teacher hacking, which describes a trend for student LM to \"overfit\" to the teacher model instead of the ground-truth, golden oracle distribution we want it to learn. Authors identify the use of fixed offline dataset to be a key reason for teacher hacking to occur, and advocates the use of online data generation to avoid teacher hacking during LM distillation. They also highlight the data diversity as a key factor and suggest that teaching hacking can be effectively detected by observing when the optimization deviates from polynomial convergence laws.\n\n## Update after rebuttal \n\nI am satisfied with author's rebuttal and keep my score of 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer KCu2 for the helpful feedback! In the following, we answer the questions raised in the review.\n\n> **Q1: You designed two stages for the experiment, and in the first stage, oracle LM generates oracle dataset for SFT on both teacher and student models to provide an initial checkpoint. Why would it be necessary to do this for student model?**\n\nWe totally agree; it is indeed possible to distill directly, using the pretrained (PT) model as an initial checkpoint. There are two main reasons why we utilize SFT instead of PT:\n- The quality of the student generation should not be too poor in order to benefit from online student generations;\n- We want to draw a more direct analogy with RLHF, which is typically performed with an initial SFT checkpoint.\n\n> **Q2: Not enough analysis for absense of teacher hacking when using online data sources. Why the teacher hacking problem naturally disappear in the online data source distillation setting? [...]  Can authors further elaborate on this and provide some insights?**\n\nWe do not have a solid theoretical explanation for this behavior. In our understanding, when the student model observes the teacher’s logit in the same context for too long, it starts memorizing the teacher's logits rather than generalizing to the ground-truth behavior. However, the clear threshold between memorization and generalization in our example is not clear, and we left the identification of it as an interesting direction for further work. \nAdditionally, we can connect the observed behavior with the increase in diversity of the dataset, which is known to be highly beneficial for the generalization abilities of the network (see, e.g., Bukharin et al, 2024, Chen et al. 2024, Zhang et al., 2025).\n\n#### References\n\n\nAlexander Bukharin, Shiyang Li, Zhengyang Wang, Jingfeng Yang, Bing Yin, Xian Li, Chao Zhang, Tuo Zhao, and Haoming Jiang. 2024. Data Diversity Matters for Robust Instruction Tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3411–3425, Miami, Florida, USA. Association for Computational Linguistics.\n\n\n\nChen, H., Waheed, A., Li, X., Wang, Y., Wang, J., Raj, B., & Abdin, M. I. (2024). On the Diversity of Synthetic Data and its Impact on Training Large Language Models. arXiv preprint arXiv:2410.15226.\n\nZhang, C., Zhong, H., Zhang, K., Chai, C., Wang, R., Zhuang, X., ... & He, C. (2025). Harnessing Diversity for Important Data Selection in Pretraining Large Language Models. ICLR 2025", "claims_and_evidence": "The claims made by this paper is quite clear and evidence show satisfying support for their claims. I don't have questions regarding this part.", "methods_and_evaluation_criteria": "The evaluation criteria and dataset selection make sense to support the claim of this work.", "theoretical_claims": "This paper mainly discuss an empirical phenomenon and does not highlight theoretical analysis; there is some theoretical discussion based on JS divergence and KL divergence regarding the loss used in LM distillation and this part looks fine to me.", "experimental_designs_or_analyses": "The soundness of experimental settings are generally fine. I have following questions for the experimental design and result analysis:\n\nQ1: You designed two stages for the experiment, and in the first stage, oracle LM generates oracle dataset for SFT on both teacher and student models to provide an initial checkpoint. Why would it be necessary to do this for student model? Since the student model needs to learn from the teacher model in the second stage by distillation (and implicitly learn from the oracle model anyways). \n\nQ2: Not enough analysis for absense of teacher hacking when using online data sources. Why the teacher hacking problem naturally disappear in the online data source distillation setting? Intuitively, the more (diverse) data samples used in distillation, the more similarly should student model mimic the teacher's behaviour. It is not surprising that proxy metric continue to decrease with longer training, but it is surprising (and lacks explanation) why the golden metric can also generalize without degradation as well. Can authors further elaborate on this and provide some insights?", "relation_to_broader_scientific_literature": "The paper identifies degradation of learning effects in the context of language model distillation. The core idea is not very new and the results obtained here are not so surprising, but this work indeed provides some take-away messages for langauge model trainining practitioners.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Paper writing is well-structured, and the take-away information is quite clear and easy to consume, which is an advantage of this paper.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3W1wrXdxn", "reviewer": "Reviewer_jZCq", "summary": "The paper introduces the phenomenon of \"teacher hacking,\" where using a fixed offline dataset for distillation degrades performance, and proposes solutions like online data generation and increased data diversity to mitigate this issue.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer jZCq for the valuable feedback. In the following, we address the questions raised in the review.\n\n> **The paper lacks significant theoretical claims or proofs, focusing primarily on empirical observations. A more theoretical understanding of the teacher hacking phenomenon could strengthen the paper.**\n\nWe agree that the theoretical statements could strengthen the paper, but we find the derivation of the theory very challenging since the statement of teacher hacking highly relies on the presence of multiple models as well as the need to take into account the optimization procedure itself. We leave theoretical explanations as a promising direction for further work.\n\n> **The paper does not provide a comprehensive comparison with other distillation methods or techniques that could potentially mitigate teacher hacking. This limits the understanding of the relative effectiveness of the proposed strategies.**\n\nIn the Appendix, we provide an experiment with multiple distillation losses and data generation strategies, none of which were shown to be resistant to the effect of teacher hacking.\n\n> **While the authors use multiple datasets, the scope is limited to specific tasks such as summarization, translation, and instruction following. Expanding the scope to include other tasks and datasets could provide a more comprehensive understanding of the phenomenon.**\n\nWhile more experiments are always welcome, we believe our experiments encompass three relatively different tasks. \n\n> **The paper does not provide a clear differentiation between teacher hacking and reward hacking, nor does it explore how insights from one phenomenon could inform the other. This limits the understanding of the unique aspects of teacher hacking and its broader implications.**\n\nThe effects of teacher hacking and reward hacking are different because there is no reward function during the distillation procedure and no teacher during the reinforcement learning from human feedback. However, we can interpret teacher hacking as an analogy to reward hacking that happens during the other part of the post-training pipeline. Nevertheless, the roots of both effects come from over-optimization of imperfect proxy objectives, but the nature of proxies is different.\n\n\n> **The paper lacks detailed descriptions of the experimental procedures, and the code used for the experiments is not open-sourced. This makes it difficult for other researchers to reproduce the results and verify the findings, limiting the transparency and credibility of the research.**\n\nUnfortunately, we cannot open source our code. However, we believe we have provided ample experimental details in the appendix to reproduce all our experiments.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The key contributions of the paper are well-grounded in the existing scientific literature and provide new insights and practical strategies for improving the distillation process of language models. The paper effectively builds upon previous research while addressing an understudied limitation of knowledge distillation, contributing to the broader understanding of language model training and optimization.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths：\n\n1. The paper introduces a novel phenomenon, teacher hacking, and provides a systematic framework for its analysis. The findings have important implications for the development of more robust language models.\n2. The authors conduct extensive experiments on multiple datasets and model configurations, providing a thorough understanding of the teacher hacking phenomenon and its mitigation strategies.\n3. The paper offers practical strategies to mitigate teacher hacking, such as using online data generation and increasing dataset diversity, which can be directly applied in real-world scenarios.\n\nWeaknesses：\n\n1. The paper lacks significant theoretical claims or proofs, focusing primarily on empirical observations. A more theoretical understanding of the teacher hacking phenomenon could strengthen the paper.\n2. The paper does not provide a comprehensive comparison with other distillation methods or techniques that could potentially mitigate teacher hacking. This limits the understanding of the relative effectiveness of the proposed strategies.\n3. While the authors use multiple datasets, the scope is limited to specific tasks such as summarization, translation, and instruction following. Expanding the scope to include other tasks and datasets could provide a more comprehensive understanding of the phenomenon.\n4. The paper does not provide a clear differentiation between teacher hacking and reward hacking, nor does it explore how insights from one phenomenon could inform the other. This limits the understanding of the unique aspects of teacher hacking and its broader implications.\n5. The paper lacks detailed descriptions of the experimental procedures, and the code used for the experiments is not open-sourced. This makes it difficult for other researchers to reproduce the results and verify the findings, limiting the transparency and credibility of the research.", "other_comments_or_suggestions": "See weaknesses above.", "questions_for_authors": "See weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents and studies the phenomenon known as "teacher hacking." Essentially, the author observed that large language models (LLMs) can over-optimize for an imperfect teacher during knowledge distillation, similar to reward hacking in RLHF. The reviewers agree that this observation is both novel and insightful. Although the paper lacks a fundamental understanding of this phenomenon, I still believe it is valuable to share these findings with the community.
Decision: Accept (poster)