Dataset Distillation for Memorized Data: Soft Labels can Leak Held-Out Teacher Knowledge
Authors: Freya Behrens
Lenka Zdeborová
Venue: ICLR 2026 Poster (2026)
Keywords: distillation
memorization
generalization
learning theory
model transfer
privacy
Primary area: learning theory

Abstract: Dataset distillation aims to compress training data into fewer examples via a teacher, from which a student can learn effectively. While its success is often attributed to structure in the data, modern neural networks also memorize specific facts, but if and how such memorized information can be transferred in distillation settings remains less understood. While this transfer may be desirable in some applications, it also raises privacy concerns, where preventing such leakage is crucial. In this work, we show that students trained on soft labels from teachers can indeed achieve non-trivial accuracy on held-out memorized data they never directly observed. This effect persists on structured data when the teacher has not generalized. To understand this effect in isolation, we consider finite random i.i.d. datasets where generalization is a priori impossible and a successful teacher fit implies pure memorization. Still, students can learn non-trivial information about the held-out data, in some cases up to perfect accuracy. For multinomial logistic classification and single layer MLPs, we show this corresponds to the setting where the teacher can be recovered functionally -- the student matches the teacher's predictions on all possible inputs, including the held-out memorized data. We empirically show that these phenomena strongly depend on the sample complexity and the temperature with which the logits are smoothed, but persist across varying network capacities, architectures and dataset compositions.

Published as a conference paper at ICLR 2026
DATASET DISTILLATION FOR MEMORIZED DATA: SOFT LABELS CAN LEAK HULD-OUT TEACHER KNOWLEDGE
Freya Behrens & Lenka Zdeborová
Statistical Physics of Computation Laboratory École polytechnique fédérale de Lausanne (EPFL), Switzerland freya. www.ees@eerll......mmmm...
ABSTRACT
Dataset distillation aims to compress training data into fewer examples via a teacher, from which a student can lear effectively. While its success is often attributed to structure in the data, modern neural networks also memorize specific facts, but if and how such memorized information can be transferred in distillation settings remains less understood. While this transfer may be desirable in some applications, it also raises privacy concerns, where preventing such leakage crucial. In this work, we show that students trained on soft labels from teachers can indeed achieve non-trivial accuracy on held-out memorized data they never directly observed. This effect persists structured data when the teacher has not generalized. To understand this effect in isolation, we consider finite random i.i.d. datasets where generalization is priori impossible and successful teacher fit implies pure memorization. Still, students can learn non-trivial information about the held-out data, in some cases up to perfect accuracy. For multinomial logistic classification and single layer MLPs, we show this corresponds to the setting where the teacher can be recovered functionally - the student matches the teacher's predictions on all possible inputs, including the held-out memorized data. We empirically show that these phenomena strongly depend the sample complexity and the temperature with which the logits are smoothed, but persist across varying network capacities, architectures and dataset compositions.
INTRODUCTION
With the advent of foundation models, it has become of great interest to exploit and transfer their ca- pabilities to other models and finetuning and distillation make this possible in practice, In distillation, student model is trained on data derived from teacher model (Hinton et al. [2155] Xu et al. 2024);
dataset distillation specifically focuses on finding a minimal training set that achieves high perfor- mance for similarly sized teacher and student models (Cazenavette et al., 2023 Yu et al. 2024 Yang cct al. 2024). A central mechanism is the use of soft labels, where the teacher's ' logits are transformed into probability distributions that the student is trained match (Buciluá et al., 2006 Ba & Caruana 2014 Hinton et al. 2015). This simple idea has been remarkably effective and remains competitive for
modern architectures (Gou et al. 202 Yu et 2024 Xu al., (2024] Qin et al. (2024). While there
have been thcoretical attempts to explain the benefits of soft labels (Phuong & Lampert 2019 Sagli-
ctti & Zdeborova| 2022, Menon al. 202T. [Boix-Adsera 2024 Dissanayake al. 2025). ), remains unclear what exactly the "dark know Medgee"ge" (Hinton et al. 2015) in soft labels is, and how to quantify it.
Among the hypotheses on the regularizing benefits of soft labels (Müller al. 2019 Yuan et al. 2020) Zhou et al. (02T), one line of reasoning suggests that they are effective because they encode latent structure the data distribution (Phuong & Lampert [2019] Menon al., 2021). This view explains empirical successes in image classification and language, where soft labels regularize the student by exposing correlations in the teacher's predictions (Qin et al. 2024 Xu et al. 2024). However, success in large-scale models does not rely structure alone: such models not only generalize from data distributions but also memorize singular facts and associations (Chen et al. 2024). This raises the question of whether soft labels also convey memorized information, and if students can inherit it during distillation. So far, this issue has been studied mainly from a privacy perspective: soft labels and related outputs may leak memorized training information, enabling the recovery of private

---

Published as a conference paper at ICLR 2026
(A)
teacher 100% Loain atC.
(B)
indapendent 25% best acc. [svg. 3.2%
(C)
studerd
(D) dec sion bounder les teucher stadent
45% acc. lovg. 2,4%]
Figure 1: Information leakage via soft labels. We examine fully connected networks with ReLU activations and p = 100 hidden neurons and biases. A teacher network is trained on 2D input data DJ with i.i.d. random uniform labels drawn from {1. 2.3. (A) visualizes DJ and teacher decision boundaries which achieve 100% accuracy. Then, D. is partitioned into two disjoint sets DS train and Dis (60%, 40%). We examine 2 settings: Training student networks via cross-entropy (B) on the class label only. making the student independent from the teacher, and (C) on soft labels obtained from the teacher via softmax on the logits. While the independent model only achieves trivial accuracy of ~ 30%. students that fit the teacher's ' soft labels achieve non-trivial test accuracy of ~ 50%. Markers indicate data from the test set, and whether it was classified wrongly (x) or correctly (o). We report averages and the standard error on the mean over 5 runs. (D) The decision boundaries for teacher (black) and student (blue). Appendix A contains further examples.
attributes Ma et al. 2024 Cloud et al. 2025). Yet, these works focus on attack scenarios, and the question of how memorization behaves in benign distillation is still open. We therefore ask:
Do the teacher's 's soft labels encode memorized knowledge? And yes, can students pick up this non-trivial information?
To address this, we study memorization transfer in small empirical models that allow for precise control and measurement, rather than attempting to analyze large-scale systems directly. We isolate the role of memorization in distillation with soft labels by training teacher networks to memorize a finite training dataset of input-label pairs. We then distill their "memorized" knowledge into soft labels, to train students who see only fraction of those pairs, and are evaluated on the held-out remainder. We apply this protocol both to (i) small transformers on structured algorithmic tasks and (ii) fully connected networks uncorrelated in- and output. While in (i) exploit early stopping to obtain memorizing teachers, (ii) does not have a latent structure by design which always implies teacher memorization. Despite its simplicity, the controlled memorization-only setting (ii) has, to our knowledge, not been studied previously in the distillation literature.
For both cases we answer our original question positively: From training on the teacher's soft labels a student can indeed learn non-trivial information about held-out memorized data. A simple visual example for distillation in two dimensions is shown in Fig. T This has consequences both for the efficient transferability of memorized knowledge, as well as the leakage of private information. We summarize our specific contributions below
We demonstrate for both structured but memorized datasets and purely random i.i.d. data that students trained on teacher's is soft labels can consistently recover non-trivial - in some cases perfect - accuracy on data the teacher memorized but the student never saw.
We show that this effect depends strongly on the temperature with which the soft labels are created from the teacher logits and can be interpreted as a regularizer that interpolates between fitting the teacher function and recovering only the ground-truth training labels.
For random i.i.d. data, show that in logistic regression, simple closed-form capacity and identifiability thresholds separate distinct leakage regimes, and that these thresholds extend the multi-class case with similar qualitative behavior. For ReLU MLPs, the soft label memorizing and teacher-matche matching solutions are distinct; the student transitions from the former to the latter only once the teacher is identifiable, with a sudden jump in accuracy.
!Our results and the code to reproduce them are available at https://gitae thub. com http://ooogggroo/dacccoo wttpgrrroppppaeeeeeeeeeeeee dist at wwwommmmmmiii www.mmmmmrriioooooommm
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Soft labels have been central to knowledge distillation since its inception (Buciluă Hinton et al. 2015), and remain competitive across domains (Gou ct al. 2021b; Xu et al. 2024). Their effectiveness has been linked to regularization effects (Müller
Distillation.
et al.
2006 [Ba & Caruana 2214
al.
2019 Yuan et al. (020) and encoding statistical structure aligned with the data distribu-
tion (Phuong& & Lampert 2019 Menon et al. 2021). Theoretical investigations of their effectiveness
have considered simplified models such as deep linear networks or linear representations (Phuong
& Lampeet 2019; [Boix-Adsers 2024 Dissanayake et al.) 2025 Zhang et al. 2023). In particular, known that in the linear case distillation allows the student to efficienlly match the teacher functionally Phuong & Lamperi (2019) In similar setting as ours, (Saglietii & Zdeboroval (2022) analyze regularization transfer from teacher to student, but take a teacher as a generating model itself rather than letting it memorize a fixed dataset.
In parallel, dataset distillation constructs small datasets that transfer capabilities for faster train- ing (Wang et al., 2018 Yu et al., (2024), with similar effects observed even with arbitrary trans- fer sets (Yang et a 2024] Nayak et al.) (022)) These findings suggest that distillation success depends less input realism than on whether the teacher function can be inferred from supervi- sion (Cazenavette et al. 2023). In contrast, we analyze matched-capacity teachers and students on memorized data without input modification. While we do not modify the input distribution, our analysis shows that when the data is sufficient to functionally match the teacher, and softmax temperatures are high, the student can learn the teacher functionally rather than merely class labels. This contrasts with work on unlearning, where distillation is used to suppress specific capabilities robustly rather than retain unrelated capabilities (Lee et al., 2025). This difference highlights the importance of understanding when distillation preserves or erases information, as we do through different data regimes in toy examples.
Memorization. Zhang et al. (211)) famously showed that deep networks can fit easily random labels, demonstrating their large memorization capacity. We extend this observation by studying how such memorized information can be transferred via distillation with soft labels. This is relevant for modern large language models which do not memorize their training corpus, but simultaneously require factual recall (Chen et al. 2024).. However, memorizing additional facts incurs linear cost in model parameters (Lu et al. (024). Bansal et al. (2022) distinguish example-level and heuristic memorization, where the latter relies on shortcuts or spurious correlations, which is known to hurt generalization (Bayat et al. 2025). In our random data setup, correlations in the dataset arise only from its finiteness, and our analysis in the large data and parameter limit rules out any spurious effects incurred by the finiteness.
Privacy. In privacy, the goal is often to create models that are only weakly dependent individual training datapoints, preventing their recovery through queries or learning (Dwork et al.] (014). One
teacher training 9
specific setting focuses on hiding the labels of training data (Ghazi eet al. 2021). This goal inherently contrasts with memorization. which requires retaining labels for single examples (Ma al. 2024). While these objectives are rarely analyzed together, we consider a teacher that memorizes data and ask under what cireumstances the label information of held-out teacher data can remain hidden from the student during soft label training. A practical example is given in (Cloud et al., 2025), where student LLM is fine-tuned on random data sampled from a teacher that encodes a hidden trait. Training on unrelated memorized data causes the student acquire the hidden trait, mirroring our observations in toy models. This illustrates that while soft labels can efficiently convey memorized teacher information, they can also create potential privacy risks, further motivating the analysis of our similar toy setting.
pn
student training St
................................ ew
student testing 2
(1 p)n
validation gre
Figure 2: Data setting.
NOTATION AND EXPERIMENTAL SETTING
Data.
We consider input-output pairs in classification setting, where input coordinates are Rd and there are e possible distribute labels y. The data is available either through the finite set D (Section 4 or a generating model from which sample i.i.d. (Section 5). See Figure 2 For
3

---

Published as a conference paper at ICLR 2026
Y- 10.0
1-1.0
T-
(A)
teacher training on modular addition
1.0
5,0
0.8
accurity
0.6
0.4
02
acc
seele
0.0
40
60
80
200
training steps
0.50
0.25
0.75
0.25 0.50
0.75
0.00 0.23
0.50
0.75
Figure Information leakage via soft labels for structured data in transformers. (A) Loss curves for small transformers trained on the 30% of the modular addition task with 113. The models D, ② and 3 are stopped after different training times. (B) Students with matching
architecture trained on the respective teachers (rows) with different softmax temperatures T (columns)) We show the students train and test error, and their accuracy on D For comparison, we show the teachers validation accuracy as a horizontal line, marked with green star. Appendix C.1 describes architecture and training details. The same experiment is repeated for ReLU MLPs in Appendix C.3
training the teacher we define the finite teacher dataset of such samples (elements) from D as DT (xx", "xx",""" ,/"") To evaluate generalization of the teacher we consider D which is either D \ D or an independent sample. To train the student, the teacher dataset DJ is randomly partitioned into two disjoint subsets: the student training set D utain and the student test set DS Diest- We refer to D //n the student's training data fraction.
Models and Training. All models we consider are parameterized functions So Rd + R that map inputs class logits E R Predictions obtained by applying argmax over the output logits. We use the cross entropy loss for supervised classification. For y R being the one-hot encoded label vectors, for the teacher, the cross-entropy loss with temperature is
exp(zx/T)
LCE((x", hn) = Σ> (e log [or=1 (fo (x))))
->
(z(2).
Σj= exp(2//T)
R=1
To transfer the knowledge from teacher f to student fo train them using the teacher's soft labels. This is achieved using cross-entropy loss, but instead of the ground truth one-hot vector y" given teacher network's iss soft labels 3"h a- (x*)). We train using the Adam optimizer (Kingma & Ba 2015) with full batches and default Py settings (Paszke et al. 2019).
Evaluation. We report accuracies of the teacher and student: acc (teacher on D.), acc train (student on D accies (student on De acc (teacher on Dvat), and acc (student on Dval). We always the ground-truth labels to compute the accuracy for the student, and never the teacher predictions. When the teacher overfits the training data acc > acc we consider it to be memorizing. For D where the labels sampled uniformly random, independently of the input, both acc and accs reduce to random guessing at 1/c, so teacher memorization requires acc > 1/c.
4 LEAKING HELD-OUT MEMORIES WITH LATENT STRUCTURE
To complement our 2D toy setting from Fig. |
we now study whether the leakage of memorized information through soft labels also occurs in more realistic architectures and structured data. Specifi- cally, we the modular addition task and single layer transformer following Nanda et al. (2023). This setting training exhibits two phases: Even though the teacher quickly learns to fit the training set perfectly, generalization to the task is delayed. This allows isolate different settings:
4

---

Published as a conference paper at ICLR 2026
Teachers that memorize their training set without discovering structure, and those that generalize. From this, we examine how student learning varies based on what the teacher has learned and how this impacts the soft labels leakage of memorized information.
Memorization and generalization in modular addition. The modular addition task requires adding two integers a, b E [0, p] modulo p. We consider the case where this task is available as a dataset of tuples with one-hot encoded tokens
(a, p) E {0, 10.r with the label E [0. p 1]. For our experiments we consider only the case where = 113, so that the number of possible unique samples is [D] = 1 1132 12, 769. We train the teacher on 30% of this data, the set D. (n 3, 830). The other part of the original dataset D is kept aside for validation in Dval- When training student we split D into two disjoint sets D train and DS uest. where [D
pn.
We train transformer architectures with single layer (see Appendix C.I). To analyze teachers that have memorized the input data (ace." > acc to different degrees, we early stop the training, see Fig. 3 A). At checkpoint (I) and 2 there is memorization to different degrees, and at ( the teacher generalizes. In the following, we use these teachers to train students via soft labels, and contrary to their early-stopped teachers we always train them until convergence. For given experimental sweep, we select the maximum number of epochs such that the slowest converging setting run shows only small improvement, and apply this for all settings in this sweep. Since the phase transition in this specific setting where the student starts to be able to learn the data is at p R 0.75, a student with r - 0 would obtain perfect generalization. We use different temperatures to generate the soft labels; results as shown in Fig. 3 B). Importantly, we observe that acciraia is always 100%, regardless of p and r.
Soft labels may leak memorized information in transformers. In Fig. B B, first row, left), at 10 for teacher D, we observe that for small p. i.e. small training sets DS wain the student achieves higher acc... (orange) than acc (dashed green). This means that indeed the soft labels are leaking some information on the training set DI that accuracy on DS is higher than for D This indicates that the soft labels leak information specific to the teacher's ' training set D.JT and allow the student to recover held-out memorized samples, while they do not improve performance on D similarly strongly. As the fraction of seen teacher data grows, acc reaches 1.0, and acc approaches acc The teacher information leaking. A similar but more abrupt transition occurs for teacher 2 the same 10 (middle row, left).
These results parallel our earlier observations from Fig. 1 For some DS irain training on the teacher's soft labels leads to non-trivial accuracy on D which is importantly higher than that on v (analogous to random guessing previously). Unlike the 2D case, however, here the student can perfectly generalize to the held-out D At the same time, despite 5x longer training than the teacher, at 10, these students fail to generalize Dval when distilled from the non- non-generalizing teachers ! and @ Instead they match acc. This shows that while soft labels can leak memorized inputs, they can also prevent the student from learning latent structure that undertrained memorizing teachers have not discovered.
Higher temperatures are more data efficient for fitting the teacher. At lower temperatures where the soft labels resemble one-hot labels and contain less information about the teacher, the student can outperform the teacher and generalize to D As shown in Fig B. right column), at T 0.1 student performance even becomes independent of the teacher. The student either fails to generalize due to insufficient data (e.g., at p 0.7), or exhibits delayed generalization (learning curves Appendix C.2). Only for larger т = 10, learning from the generalizing teacher 3 requires less data with almost immediate generalization
D and for the memorizing teachers ! & 2 the students matches their function. This highlights that higher temperatures both improve data efficiency and convergence speed, and increase the leakage of teacher-specific memorized information.
5 LEAKAGE IN THE PURE MEMORIZATION SETTING
In the previous section, we used acc as a proxy for the amount of teacher memorization. However, a low Tc does not rule out that the model internally captures some underlying structure, even if was not predictive. To isolate memorization in controlled setting and to characterize the leakage behavior theoretically, we now consider a data model where there is no structure in the data a priori analogous to the introductory example from Fig. The entries of the input sampled i.i.d.
5

---

Published as a conference paper at ICLR 2026
(A.1)
BCC
3CC
(B)
phenomenology
1.0
1.0
0.8
0.-
0.9
0.9
D.-
0.8
0.8
0.7
0.7
0.7
0.5
202
0.5
0.6
02
1600
1600
1600
0.5
0.5
Figure 4: Binary logistic regression. (A.1) We show the training accuracy of the teacher on D. for p = 0.8 and the students training (A.2) and testing accuracies (A.3) of the student on the two partitions of D.. While the teacher is trained via Adam on the logistic loss, the student solutions are obtained from the teacher logits on D using the pseudo-inverse. The thresholds iiber Tua and (p) are highlighted in green, pink and blue. (B) depicts the different regimes of teacher/student learning as a function of p and the sample complexity a. The dimension is fixed at d = 1, 600 and is varied. We distinguish whether the student fits D with acc accin 2 0.99 (gray/bluc/green) or not (red/orange). In the regime where it fits the DT, gray implies that the student learns only close to trivial accuracy (acc < 0.55), blue that it is non-trivial (acc < 0.99) and green is perfect (acefest. 2 0.99). We measure the MSE loss directly the teacher logit (see Appendix D.I to evaluate whether the student learned the teacher (orange) or not (red) with a threshold set at 0.1.
from a Gaussian x2 ~ (00, 1) and the labels E (1....,c) c) are sampled uniformly and i.i.d. from e classes. The inputs and labels are independent by design, so any teacher needs to memorize the finite dataset D., failing to generalize to Deal-
In the following, we analyze logistic regression, where we can derive closed-form thresholds for limit. We consider its multi-class version and show
the recovery of D in the high-dimensionn how the same threshold scales in c. To estimate the impact of more complex non-linear teachers we analyze leakage in one hidden layer ReLU MLPs. In Appendix B we show that a teacher GPT-2 model (Radford et al. 2019) fine tuned on a dataset randomly associated sequences of tokens and classes can also exhibit non-trivial test accuracy acc on held-out sequences.
5.1 MULTINOMIAL LOGISTIC REGRESSION
For multinomial logistic regression we consider linear models (x) = W. x with W E Rexd that are trained via cross-entropy, known as multinomial logistic regression or softmax regression. In distillation this limits us to setting where teacher and student architecture match.
Formal analysis: leakage logistic regression. We first consider the case of only two classes, logistic regressior When we have direct access to the logit, the problem of recovering the teacher weights W under the square loss is equivalent to solving an (over- or under padder-pramttrrized) least squares problem, by means of the pseudo-inverse of the input matrix with the logits, i.c., W = =
where X E T xd; fw (X)= E R and rarain [DS [DS We consider different sample complexities on D.,, and the train and test accuracies of the student on DS training set size p = 0.8. We observe that the acc and aco start decaying from 1.0 at a given a. The test accuracy grows monotonically in from the trivial random guessing accuracy up to perfect accuracy, and at some point decreases again. The general phenomenology concentrates for large d and n, as function of p, resulting in three thresholds that can be defined in terms of a n/d:
n/d. Fig. ((A) shows the accuracy of the teacher
train and D as a function of a at fixed
a < lahel - teacher memorization capacity: The teacher can fit all input-class pairs in D.. In the proportional limit when d, > xo, Cover's Theorem (Cover) 1965) states that T < 2.
2 a (p) identifiability threshold: The student functionally matches the teacher using the logits. measured through the mean squared error loss on the teacher logits, which occurs at á 1/p, as the input matrix X becomes invertible.
S S (p) student memorization capacity: The student can fit all data from DS tnann via the input-logit pairs from the teacher.
"Here, we exceptionally consider f= Rd + R as this is the usual setup of the two class classification problem. The class is then determined by the sign of the single output logit

---

Published as a conference paper at ICLR 2026
The capacity of the teacher Jaae is independent of the distillation setting, stating how many randomly sampled imputs it can associate correctly to a randomly sampled but fixed label. In contrast, identifia- bility functional matching of the teacher is possible from agg(p) this threshold is identified for linear networks in Phuong & Lampert (2019). Finally, the student capacity (p) describes whether the student fits the teacher supervision from D train conrectly.
In our experiments, for finite sizes, we observe that the teacher memorization capacity (d = 1600) ~ 1.96 is already close to the infinite limit of a = 2. Beyond this threshold, the student cannot fit Piain perfectly anymore, as it is not memorized by the teacher and information corrupted. However, when the teacher does memorize D.' perfectly, the student obtains perfect accuracy on Di through the logit training set. In this case, we observe that the logits contain weak signal on the other held-out memorized data and allow the student to obtain acc 2 55% for large enough a and p. as shown in Fig. D A.3);; some information on the held-out data is leaking. In terms of a, acc grows monotonically, e.g. for a = 0.8, d 1600) = 1.26, where reaches acciest 2 0.99 even though fifth of the memorized data that was held-out. This means that the student can indeed recover the hidden memorized data by recovering the teacher weights W.
Fig. 4 B) shows the different phases can be delineated as a function of and p for finite fixed = 1600: low/no leakage where acc < 0.55, weak leakage of information acc E (0.55, 0.99), full recovery of the held-out memorized data acc 2 0.99 and failed teacher memorization beyond T We can separate the latter regime into two depending on p and a, whether the student is able label
to recover the (nonmmmmorizing) teacher or not, depending on X's invertibility.
The impact of temperature on memorization. In practical distillation with more expressive networks one cannot simply invert but instead one minimizes the cross-entropy loss on soft labels via gradient methods. Creating soft labels from a students logits requires choosing a temperature 7 in the softmax function equation T With r + one recovers the one hot encodings of the labels and thereby destroys any information that would have been embedded by the teacher. At the other limit, when t + oc, the soft labels become uniform and information about the labels and the teacher is destroyed.
For the case of multinomial regression with two classes Fig. 5ig.5.. 5|A) shows the relevant thresholds in terms of n/(dc) and on the temperature for fixed p 0.8 (for accuracies see Appendix D.2) Next to label. a (p,T).
T), and aasbe T), we introduce another threshold, S-shfffee (p. r), derived from a controlled experiment. For each input x with class in D we assign a soft label sampled from different teacher input x' within the same class (y y²). This procedure preserves the correct class identity- - the highest soft label entry still corresponds to y but removes any teacher-specific information about x. As result, the student sees noisy supervision: It is class-consistent but the correlation between the rest of the soft label and input is broken. We then define df point at which this noise prevents the student from learning the class signal.
S-shullle (p, 7) as the
In Fig. 5(A), we observe that label interpreting as hyperparameter that shifts the training objective between fitting soft labels and teacher function (high r) and recovering class identity (low T).
T S-shuffle transitions from and to llabel as T increases. This supports
Multiple classes > 2. As the number of classes increases, the student has c-sized soft label available per training sample, which can contain information about other samples. At the same time, the model size of both teacher and student scales with a factor of We observe empirically that the behavior for several classes is consistent with that for two classes: The student can learn non-trivial information about held-out memorized samples and achieve up tp 100% accuracy from the soft labels. In Fig. 5 B) we observe the scaling behavior of the four relevant thresholds in terms of the number of classes for a fixed p= = 0.8 and - 10, leading to a 1/c, the scaling of alabel alatel 1/ log 1/v0.. Naturally, only the scaling of ai is independent of p and Specifically
and a ai the scaling of a at 0 should arrive at alabel- Nonethcless, the order of the thresholds in remains the same, retaining the original dependence. In Appendix D.2 we confirm this for varying temperatures and a fixed 10, where the phenomena are consistent with c 2.
S-shuffle
7

---

Published as a conference paper at ICLR 2026
(8.1)
(B.2)
aCC
(8.3)
(B.4) accis
1.0
50
40
30
dz
10
0.50
0.75 1.00
8.0 mnoot
alog
alogc
number of classes c). (A) For the setting with c possible classes and d = 1000, we show the capacity and learning thresholds ol (green), lahel (pink), add (blue) and Label reported in Appendix D.2 (B) We take the number of classes as (2, 10, 20, 30, 40, 50), the larger the darker the color, and give train and test accuracies in varying scales a {c, vc, c). Here p 0.55 is fixed and d {100, 1000} is varied for computational efficiency depending on a and 10.
Figure 5: Impact of (temperature
S.S-shuffle (red) as a function of the softmax temperature T and the sample
complexity a; accuracies
5.2 TWo MECHANISMS FOR LEAKING MEMORIZED INFORMATION IN RELU MLPs
In this section, we show that ReLU MLPs already exhibit more complex behavior for the same random uncorrelated inputs and labels as before than the multinomial regression case. In Fig. 6
we consider a matched teacher and student, ReLU MIPs with a single hidden layer, with c = 100
classes. On the z-axis we vary the fraction p of D that observed by the student. In Fig. A the
acc and acc exhibit a similar phenomenon as before for the logistic regression in Fig. 4 D):
While the student memorizes its own training set perfectly, the accuracy ace on the held-out data non-trivial and increases monotonically more and more data from D is available. However, at higher sample complexity shown in panel (B) of the same figure, we observe two new phenomena: A first observation the presence of phase where acc slowly drops while the teacher accuracy ace.. remains perfect. Mcanwhile, acc is lower than for the same p at lower sample complexity a. This is inconsistent with the previous observation, where larger DDT' helped functionally matching the teacher better and therefore led to higher accuracy. A second observation is the marked jump after the drop in acc where both acc and acc Test immediately rise to 100% accuracy.
Memorization fails before teacher identification succeeds: oi < au To understand these phenomena better, we turn to a more complete picture of the phase space in Fig. Al.). Next to the regions already identified for the logistic regression in Fig. B), we split the regions where a weak leak- age is detected into two parts: The one where the student perfectly learns DS and the one where the student does not memorize the training data. In Fig. 7 B.2) it is further visible that acc decreases before it increases as function of the sample complex- ity a. To understand this behavior we func-
(A)
n n= 25,000
(B)
100, 000
1.0
0.8
acturary
0.6
+ accipn
+ acc
0.4
+ acciun
-|- acc!
0.2
0.0
0.2
0.4
0.6
0.8
0.2 0.4
0.6
0.8
Figure 6: Both teacher and student are MLPs with a single hidden layer of size p = 500 and ReLU activations. The inputs are d = 1000 and c 100. The teacher successfully memorizes a training set DTTT of size 25,000 (A) and 100,000 (B). We track the accuracies on both via acc and acc with the standard error on the mean reported for 10 runs. In all cases shown here, some information is leaked statistically, allowing the student to surpass trivial performance data not seen by the teacher (acc in some cases reaching up to 100% test accuracy.
observe acc and acc. tions of training time in Fig. C..3). There, a sudden jump in train and test accuracy occurs as a function of stu- dent training time at around ~ 100.
While before the jump, the training and testing accuracy are at different levels (and already non-trivial for the student), they jointly jump to 100% accuracy. In Appendix D.3 it is shown that this jump coincides with drop in the CE loss the teacher distribution and that just before the transition, accs. approaches that of a student trained on intra-class sampled soft labels. This suggests that for ReLU MLPss, there may be two distinct weight configurations: One where the student memorizes the
8

---

Published as a conference paper at ICLR 2026
(A)
phenomena
(8.1)
acc!
(6.2)
acC
(8.3)
acCive
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0,4
1.0
0.2
0.2
0.5
0.2-
0.2
0.0
0,0
0.0
50
100
150
150
memorizes Mia acCie::PaCCC
(C.2) 5 approximates 2i
(C.3)
5 learns T
(C.4)
acc 100%
2 100%
acc
DCC
1.0
(5200000
1.0
n=50.000
1.0
1.0
200.
acc
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
50
100
150
200
0
50
1.0
150
200
5.0
1.00
15.0
2000
50
1000
150
200
Figure Leakage 1-hidden layer ReLU networks. The teacher and student architectures match single hidden layer ReLU networks with 500 for varying settings of sample complexity a = nn/(c)) and student training fractions p. The number of samples n is changed while = 100 and d 1000 and the temperature
20 are fixed. Each experiment is repeated 5 times and average accuracies are reported. (A) Different regimes distinguish the type of generalization the student achieves: (blue) weakly with memorization of D (light blue) weakly but without memorization of D 'rain' (green) perfectly generalizing held-out memorized data; (orange) the teacher cannot memorize D.T but the student fits the teacher nonetheless; (red) the teacher cannot fit DJI and the student does not discover the teacher either. (B) acc! acc and acc (C) Accuracy as a marked with white circles in (A) and (B), varying n and keeping d = 1000. For comparison, we show averages of acc and aco val at the end of training horizontal lines.
function of training time t for fixed p 0.65 and different sample complexities
soft labels, and another where it functionally matches the teacher. This distinction was not present for multinomial logistic regression.
Memorizing the soft labels generalizing the teacher function. These observations suggest that the student can learn two functionally different solutions that both leak information about held-out memorized data, but differently: One solution memorizes the teacher's soft labels representing DS Daain and another generalizing solution matches the teacher functionally. This extends the picture from the multinomial regression, in that only weakly (and fully) learning the teacher function leads to non-trivial leakage on the held-out set, but also solution that truly memorizes the soft labels can capture some additional structure on held-out data. Whether one the other solution is learned depends non-triviall, ooiiiivially on the respective capacity thresholds, the algorithm, and the ratio between the teacher and student capacity. In Appendix D.44 we provide some additional ablations that explore how increasing the parameters via the hidden layer size and the relative capacity of teacher and student an unmatched setting impact acc.
Localizing the information in the soft labels. In Appendix D.4 we test the effect of removing an input class c; from DDS Prain and removing it from the soft labels by zeroing it out for all other classes C * c. We find that while removing the inputs can still lead to non-trivial accuracy on in DS removing the corresponding soft label entries detrimental for test performance. Likewise, zeroing out the smallest values in every soft label negatively affects acc This leads to hypothesize that the common practice of using only the top-k largest values may not allow for generalizing on the memorized information.

---

Published as a conference paper at ICLR 2026
6 DISCUSSION AND CONCLUSION
In this work, we study how memorized data influences distillation. We analyze both structured data with teachers at varying levels of memorization or generalization as well as teachers that memorized data without a latent structure, We show that students can acquire information about memorized teacher data that was held out during their training from the teachers soft labels. By evaluating performances across teacher and student, we identify distinct regimes: In some, the student memorizes soft labels via statistical leakage; in others, generalizes the teachers function. Our findings serve as a proof of concept to understand that memorized information can be transferred between models and extend prior results on linear students with binary labels by showing that this functional recovery behaviour persists across multiple classes and across nonlinear architectures. This broader empirical and theoretical picture indicates that the leakage mechanisms originally characterized for linear networks Phuong & Lampert (219)) apply more generally, which makes their implications for memorized data more concerning.
Limitations. Still, our analysis is restricted synthetic datasets that are either structured explicitly memorized and do not capture all aspects of natural data distributions. While this setup allows for precise control and analysis. it isolates the information contained in soft labels, which the focus of our theoretical contribution, it limits the immediate application of our findings to real-world tasks even though such pipelines, for example eenacy-constrained settings where the student has restricted data access and relies on a more permissive teacher, motivate the problem. Additionally, we focus on simple models such as logistic regression, single-layer ReLU MLPs and small transformers to enable theoretical clarity. Even though we show that leakage also occurs for one instance of large language model with synthetic data, it remains unclear to what extent the identified leakage regimes and thresholds translate deeper architectures. We did not analyze the influence of regularization or optimization dynamics, both of which can affect capacity and therefore leakage. Finally, we only consider memorization and soft labels, which excludes broader knowledge and dataset distillation settings where the teacher jointly learns generalizing structure and memorized data. Our results therefore isolate one mechanism through which soft labels convey information, rather than providing a full account of distillation in realistic settings.
Future work. On the theoretical side, our framework motivates a capacity analysis of multinomial logistic regression and single layer networks, to identify information-theoreiic and algorithmic capacity thresholds for classification. On the practical side, is important to understand how models represent both memorized and generalized content jointly, and in particular how this knowledge can be transferred for efficient dataset distillation, or whether it can be hidden for privacy reasons.
ACKNOWLEDGMENTS
We thank Luca Biggio for insightful discussions and David Bau for pointing us to relevant related work. This work was supported by the Swiss National Science Foundation under grants SNSF SMArtNet (grant number 212049).
REFERENCES
Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL
files//pperr2014/
Rachit Bansal, Danish Pruthi, and Yonatan Belinkov. Measures of information reflect memorization patterns. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL Id=CCahlqHogk
Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob, David Lopez-Paz, and Pascal Vincent. The pitfalls of memorization: When memorization hurts generalization. In The Thirteenth International Conference on Learning Representations, 2025. URL httttt::///ppneeeee net ffot//oomm
id=wVh2h97p1M
10

---

Published as a conference paper at ICLR 2026
Enric Boix-Adsera Towards theory of model distillation, May 2024.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizie Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 535-541, 2006.
George Cazenavette, Tongzhou Wang. Antonio Torralba. Alexei A. Efros, and Jun- Yan Zhu. Generaliz- ing dataset distillation via deep generative prior. In 2023 JEEE/CVF Conference on Computer Vision doi: 10. N..1099/CPPS2222 2023.00364.
and Pattern Recognition (CVPR), pp. 3739-3748, 2023.
Bowen Chen, Namgi Han, and Yusuke Miyao. A multi- perspective analysis of memorization in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 11190-1 1209, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.
URL https: Btl.:attss//ccllattooooooooooo
emmrpp-men main. 627/
Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Btlle, Jacob Hilton, Samuel Marks, and Owain Evans. Subliminal learning: Language models transmit behavioral traits via hidden signals data, 2025. URL /
14805
Thomas M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers, EC-14(3): 326 234,334, 1965. doi: 10. 1..1109/GEEC.165..6411111
Pasan Dissanayake, Faisal Hamman, Barproda Halder, Qiuyi Zhang, Ili Sucholutsky, and Sang- hamitra Dutta. Quantifying knowledge distillation using partial information decomposition. In The 28th International Conference on Artificial Intelligence and Statistics, 2025. URL
https://openreviie net htt/:/oormmmiieeee Foruum?id
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and trends@ theoretical computer science, 9(3 14:2211-100) 2014.
Badih Ghazi, Noah Golowich, Ravi Kumar, Pasin Manurangsi, and Chiyuan Zhang. Deep learning with label differential privacy. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, volume 34, pp. 27131-27145. Curran Associates, Inc.,
2021. URL ltttps://proceedinggn neurips wttpcpccpppppeeeeeeeeeeeeee wwweeerffillepppppeeeeee lespppppe//22111 E1 31le//aa3l669eecccffcc1333000003
Janping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision. 129(6); March 2021a. ISSN 1573-1405. doi: 10007/411263-022-1445333 URL. http://des doi wor./orrggo. 1007/
Jiuxiang Gou, Baoyuan Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789-1819, 2021b. doi: 10.1007/ sl 111263-021-01453-2.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network, March 2015.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http:
/ l/arxiv.oro hww.vorg/abb.. aba /1112 6980
Bruce W. Lee, Addie Foote, Alex Infanger, Leni Shor, Harish Kamath, Jacob Goldman-Wetzler, Bryce Woodworth, Alex Cloud, and Alexander Matt Tumer. Distillation robustifies unlearning. In The ninth Annual Conference on Neural Information Processing Systems, 2025. URL
https:://pennreiee..nettt htt.eeeeeee.ooooooooooooiii

---

Published as a conference paper at ICLR 2026
Xingyu Lu, Xiaonan Li, Qinyuan Cheng, Kai Ding, Xuanjing Huang, and Xipeng Qiu. Scaling laws for fact memorization of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 11263-1 1282. Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/20222 1a0.55///11000444.dnn-eeeeeaeea wnndiinge-en-ge........
URL
/
Findings -aaggg--emnppp. 6587
Olivia Ma, Jonathan
and Dmitrii Usynin. Efficient and private: Memorisation under differentially private parameter-efficien fine-tuning in language models, 2024.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7632-7642. PMLR, 18-24 Jul 2021. URL http:tttppp/ccceeet//e/
mlr. wtw.p/eess//w11m//menolleeeeeeee html
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In H. Wallach, H. Larochelle. A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Asso- ciates, Inc., 2019. /tt
wwtpppprppppppeeteeeeee fi wwwfflllssii.o 1esp/pper////// pdf
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023. Utttttss:://aat http:////a.www...//.aeeeo
hwtpx/iwwwrgr///////////// /bb///a///// aataabb/300 2301
05217
Gaurav Kumar Nayak, Konda Reddy Mopuri, and Anirban Chakraborty, Effectiveness of arbi- trary transfer sets for data-free knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1430-1444 1438, January 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style. high-pperfrmance deep learning library, 2019. URL https::/t/// wwtpxiivwrrg/b////o www./bog/////o..o.. 912. 01703.
Mary Phuong and Christoph Lampert. Towards Understanding Knowledge Distillation. In Proceed- ings of the 36th International Conference on Machine Learning, pp. 5142-5151. PMLR, May 2019.
Tian Qin, Zhiwei Deng, and David Alvarez-Melis. A label is worth a thousand images in dataset distillation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Luca Saglietti and Lenka Zdeborova. Solvable model for inheriting the regularization through knowledge distillation. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova (eds.), Proceed- ings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pp. 809-846. PMLR, 16-19 Aug 2022. URL https: httpss//ppccceeiiege.e mll www......../.i.... http:///////aiiittiiiiii.. html
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arxiv:/81/ 10959, 2018
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A Survey on Knowledge Distillation of Large Language Models, October 2024.
William Yang, Ye Zhu, Zhiwei Deng, and Olga Russakovsky. What is dataset distillation learning? In Forty-first International Conference on Machine Learning, 2024. bttttttss: //tp::::/p/p////e/e net fttt/forummiix7788yc3333
12

---

Published as a conference paper at ICLR 2026
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence. 46(1):150-170, 2024. doi: 10.1 00.100//AAMM2222.333333
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3902-3910, 2020. doi: 00.1199/CVP4226000000000000
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. URL httpps://pppeeeeiieeest net. Sy
Quanshi Zhang, Xu Cheng, Yilan Chen, and Zhefan Rao. Quantifying the knowledge a dnn to explain knowledge distillation for classification. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(4):5099-51 13, 2023, doi: 10. 10.110//PAAM.22222200444
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: bias -ariine tradeoff perspective. In Inter- national Conference on Learning Representations, 2021. URL //tprreeee.//.............. net/
Forum?id=g1Td-55333 5X324
13

---

Published as a conference paper at ICLR 2026
A VISUAL EXAMPLE FOR d = 2 AND c 20
As another example that can be visualized analogously to Fig. we show random dataset again in two dimensions but with 20 classes in Fig.
(A)
teacher
indapendent
(D)
dec sion bounderie
66% (avg.
Figure 8: Information leakage via soft labels for == 20. We examine fully connected networks with ReLU activations and p = 300 hidden neurons. A teacher network is trained on 2D input data D. with i..dd random uniform labels drawn from 20 classes visualized in a spectrum of colors. (A) Visualizes D. and teacher decision boundaries which achieve 100% accuracy. Then, teacher data is partitioned into two disjoint sets D and D at (60%, 40%) ratios. We examine 2 settings: Training student networks via cross-entropy (B) on the class information only, making the student independent from the teacher, and (C) on soft labels obtained from the teacher via softmax on the logits with temperature T = 20. While the independently trained model only achieves close to trivial accuracy of ~ 6%, students that fit the teacher's soft labels achieve non-trivial test accuracy of ~ 66%. Red and green indicate data from the test set, and whether it was classified wrongly or correctly. We show the average test accuracy and standard error the mean over 5 runs. (D) The decision boundaries between teacher (black) and student (purple) correspond very well.
B DATASET DISTILLATION FOR FINETUNED GPT-2 CLASSIFIERS ON RANDOM SEQUENCES
In order test whether the phenomena observed in Section [ extends also to random sequence data, we examine similar setting with a GPT- 2 architecture (Radford et al. 2019). We con- where
10
sider sequences æ = 1429_3507_344 each sequence concatenates three random num- bers sampled uniformly and i.i.d. between and 1000, with random class y out of 1000 pos- sible classes. In our setting DJ contains 6000 samples of such sequences and their classes. We equip the next-token prediction backbone GPT-2 with a linear classifier head. We use the stan- dard tokenizer and train the teacher on DTTT for 100 epochs using Adam W with a learning rate of 5 x 10-4
eccuracy
0.6
0.4
0.2
test
0.5
train
0.8
0.0
200
epochs
Figure 9: Leakage in large language model for synthetic data. acc and acc for GPT- 2 classifier students trained using different frac- tions of random sentences memorized by a teacher with the same architecture (both pre-trained). The size of the memorized training set is n = 6000 sentences made of three random numbers up to 1000, each with one of 1000 classes assigned ran- domly.
After successful training, when the teacher mem- orizes the sentences with 100% accuracy, we ex- tract the teacher's logits for its training data and create soft labels. with temperature 5 20. We train different the students for different fractions p (0.2, 0.5, 0.8) for 200 epochs, but otherwise use the same settings as for the teacher. After con- vergence all three students reach acci 99.5% and acc$ (0.213, 0.524, 0.652) (Fig. while the test accuracy of random guessing is approximately acc 0.1%. For seeing only 80% of the teachers data, the student achieves > 60% accuracy on the held-out data. This suggests that, similar to single-layer models, over-parameterized language model may recover non-trivial
14

---

Published as a conference paper at ICLR 2026
fraction of the teacher's held-out memorized data. However, despite some exploration of different parameters, we did not yet observe a setting where the teacher function is exactly recovered as for the MLPs, i.. where the student reaches acc test 100%.
C SUPPLEMENTARY MATERIAL FOR MODULAR ADDITION
C.I IMPLEMENTATION DETAILS FOR THE TRANSFORMER
As an architecture we consider the single layer transformer from Nanda et al. (2023). It embeds the
p + tokens into 128 dimensions. There is dot-product attention layer with 4 heads followed by
ReLU MLP with single hidden layer of 4 128 dimensions. A readout layer maps its outputs to the
p classes.
By design the prediction is autoregressive, but in this case only the last predicted token is relevant and included in the training loss. It becomes a parameterized function f 3R b> IRP where c and d 3p.
During training we use weight decay set to 1.0 and use full batches, which recovers the setting in which grokking was observed (Nanda et al. 2023). We train using the Adam optimizer and learning rate of 0.001.
C.2 GENERALIZATION SPEED
We examine how fast (in terms of training time) student reaches perfect accuracy when trained on the soft labels from a perfectly generalizing teacher. In Fig. [0 we show exemplary learning curves for the results from Fig. with teacher 3 in the main Section
training set, large temperature p=
large training small temperature p=
100
300 400
200
400
training
Figure 10: Comparison of the training, test and validation accuracy for transformer students (training and test data were seen by the teacher), for single run from data used in Fig. 3 A). (Left) The student learning from small training set (p 0.3) with high temperature (r 10) not only memorizes its training data fast but generalizes on the held-out teacher train set and validation after only few epochs. (Right) A student that sees large training set at smaller temperature however, exhibits grokking in a similar fashion as the original teacher (see Fig. 3(A)) )
C.3 MODULAR ADDITION WITH MIPS
In addition small transformers, we repeat the experiment from the main text Fig. with 2-hidden layer ReLU MLPs with 200 hidden neurons each in Fig.
We also stop the models training at three different points D, ② and @ which exhibit almost none, very weak and good generalization. In analogy to the transformer, at low temperatures the student's 's behavior becomes independent the time at which the teacher training was stopped. Also, for higher temperatures and memorizing students, the acc reaches higher values than acc indicating that the soft labels only transfer information about the memorized labels but not the tasks structure and prevent the students from generalizing.
15

---

Published as a conference paper at ICLR 2026
(8)
t=10.0
T=100
7= 0.1
(A) teacher training on modular addition
accurso
0.5
0.0
acC
acc
100
150
actiun
training steps
acc
0.00 0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.5
0.50
0.75
Figure 11: Same experiments in Fig. but with teacher and student architectures that 2-hidden layer ReLU MLPs. (A) Teacher training that was stopped at different points in training, leading to the teachers ①, 2 and 3. (B) Accuracies after training from the student for learning from the different teachers, at different and T.
D SUPPLEMENTARY MATERIAL FOR RANDOM DATA
D.1 LOGISTIC REGRESSION: TRAINING STUDENTS FROM THE LOGIT
We define the mean squared error between the teacher fr and the student f on the dataset D [x", y=
J==1 as:
1
Let the matching accuracy of the student with respect to the teacher on the student test set be
Σn n ==1
arg max fy (x*) = arg max f7 (xe*)
acc T
(2)
where f.f (x) and f (x) denote the logits assigned to class j by the teacher and student, respectively.
Fig. 12 shows the different values of acc acc acc accs match-T; msee SS D and
mse Ss, D for p and a = n/d.
16

---

Published as a conference paper at ICLR 2026
acc.
BCC
acc
1.0
1.0
1,00
0.8
0.8
0.6
0.6
0.6
0.85
0.4
0.4
0.80
02
0.2
0.2
0.2
0.75
0.0
0.0
BCC
msetf m (D
mse(f" (ss()"(D
0.8
0.8
0.6
0.6
10-9
10-2
0.4
0.4
10-1
0.2
0.2
0.2
4
0.0
Figure 12: We train logistic regression teacher to fit D and recover student via the teacher logit via a pseudo-inverse. We change the number samples and keep d 1600 while we change n d. Every point reports the mean of 5 experiments with different teacher, student and model initializations. For training the teacher we used Adam, learning rate 0.001 and for 10, 000 steps.
D.2 VARYING THE TEMPERATURE FOR MULTINOMIAL LOGISTIC REGRESSION
In Fig. we show that the temperature influences whether or not held-out teacher data is leaked to the student for different and in multinomial logistic regression with 2. In addition, the original measurements that lead to the phase diagram for c in Fig. ((A) are shown in Fig. 14 We show the same experiment for 10 in Fig. 15
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.3
1.0
1.5
0.
1.0
1.5
0.3
1.0
1.5
Figure 13: For multinomial regression with two classes and a =/(dc), and d = 1000 we report the impact of different temperature in the softmax on acc Experiments are repeated times. We train the teacher with Adam and learning rate 0.0001 for 1000 epochs and the student with learning rate 0.001 for 5000 steps.
17

---

Published as a conference paper at ICLR 2026
16F-
28
acc
364
022
38P
24
1.8
12
1.4
1.5
24
04
56
14
12
14
3.5
Figure 14: For multinomial regression with two classes 0.8 and 1000 we report accuracies for teacher and training data for two experimental settings. The top row is standard setting and the bottom row re-shuffles the input-soft label assignment within the classes in 2
train
XP-
36
36
axx
0.2
Figure 15: Same setting Fig. 14 but with 10 classes instead of 2. The lines in (B) the T
shullle (red) as a function of the softmax thresholds llaeel (green), auttel (pink), agd (blue) and a d temperature and the sample complexity a.
D.3 SUPPLEMENTARY TRAINING INFORMATION
We supplement the accuracy curves from Fig. for the single hidden layer ReLU MLP with the corresponding losses in Fig. 17 for different sample complexities a n/(dc). It is visible, that
around the same moment where the jump in accuracy occurs for Fig. 16 (C3), the loss also drops
significantly.
In Fig. [6 compare the learning curves from the students in Fig. (C) with those from the students which trains on the within-class shuffled soft labels. In the third panel it is visible, that before the student generalizes to the teacher function, it obtains the same accuracy as the shuffled student. Before this third panel, the accuracy is matching well, whereas for the forth panel the difference comes about quickly. This supports the intuition that the student first attempts to truly memorize, and then generalizes the teacher structure in the third panel.
18

---

Published as a conference paper at ICLR 2026
S memorizes Dist acc Pacc
5 approximates Diss Pacci
5 leams 1100%
acc 100% acc accie
0.8
0.4
0.0
200 100
150
50
1.00
200
Figure 16: Accuracies for ReLU MLPs from Fig. C), but this time compared with runs where the input data was shuffled within classes, red. Again, the input dimension d = 1000 with c 100 classes and from left to right the teacher saw n = 10 x (15, 50, 100, 200} samples, of which the student was trained with a p = 0.65 fraction. There are runs for the normal student, and 2 for the student that receives the altered teacher data for training.
memorizes Dewn acc.epaccr)
appreximates Drai accl.g accc
leams a 1100%
acc!
acc mi act
0.850
Inse
4.800
4.550
4.500
100
150
100
150
50
150
200
50
200
Figure 17: Cross entropy loss on DS and DS for the results from Fig. 16 C). The input dimension
is d = 1000 with 100 classes and from left to right the teacher saw = 10 x {15, 50, 100, 200} samples, of which the student was trained with p 0.65 fraction. We show the average (wide line) and single runs (thin lines) for 5 runs each.
D.4 ABLATIONS FOR RELU MLPs
D.4.1 VARYING THE SOFT LABEL CONTENT
In this section we conduct several ablations, to understand which information in the soft labels is crucial for obtaining good acc. We conduct three experiments, where we:
Remove small soft label entries: We zero out the entries of the smallest & values of the soft labels for a given input. This leads to the rest the vector not summing to one anymore, but the cross-entropy loss can still be computed.
Remove single class from the training data: We remove the class from the training data, and evaluate on the test data.
Remove single class from other classes soft label vectors: We specifically zero the class e value in the teacher soft labels for all classes c' a
In Fig- (8(A) A) we observe the effect of removing parts of the logits. Already removing a single entry is critical when the normal student would otherwise have learned the teacher. Removing more deteriorates performance accuracy quickly, indicating that accessing the complete soft label is important to recover held-out memorized items.
In Fig. 18 B) we observe that removing the class e from the soft labels is detrimental to the accuracy on that class in the test set (ace. but maintained almost at normal level for other classes. The average performance on the other hand affected only little. contrast, when we remove the class c completely from the training but leave it intact in other classes' soft labels, the held-out sample
19

---

Published as a conference paper at ICLR 2026
accuracy remains at high level for class c, well as the others. This further emphasizes that lot of information contained in the soft labels, and that especially the relational information to the class can help lot.
0.8
zero out smallest soft label values noraal
training
0.6
accurey
remove class c soft label entry
accipt
0.4
... acc
0.2
0.0
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Nid
Nld
Figure 18: We consider ReLU MLPs with single hidden layer where student and teacher have p = 500. We keep 1000, c 100 and the fraction on which the student trained is = 0.7, and
temperature of 20. (A) We compare zeroing out the smallest i E {1, 10, 50) out of the 100 values from the soft label vector in the training data with a student trained on the unaltered data. (B) We also compare removing samples with class (here = 1 w.l.o.g.) from the training data completely, removing the soft labels of all training data DS whenever the true labels is not c. train Every point is the average of 5 random datasets and initializations, with the standard error the mean shown as bars
20

---

Published as a conference paper at ICLR 2026
D.4.2 VARYING THE HIDDEN LAYER SIZE OF TEACHERS AND STUDENTS
In the following, vary the sizes of the hidden layers in the teacher and student single layer ReLU MLPs, calling them p² and p¹ respectively. We first keep p 0.45 fixed and keep the temperature 20. In Fig. 19 and 20, we vary p" and p isolation or together respectively.
In running our experiments for Fig. 19 we keep d constant and vary and p² = p², but instead of plotting the resulting accuracies over a n/(de) as the main we plot them over a/p". This incorporates the hidden layer size in the denominator that represents the number of parameters of the model, and indeed the curves fall together quite accurately.
In Fig. 19 we repeat the same experiment, but now keeping pT = 500 and varying the student pa. As we can see, this improves accirain for e.g. P = 0.7. This expected as the student with more neurons has higher capacity to learn the soft labels, which is the phase for the given p.
hidden p p
-
200
350
650
acd
acc.
0.4
0.2
0.000
0.001
0,002
0.063
0.004
0.005
0.006
0.007
milcco"
Figure 19: For ReLU MLPs we vary the teacher and students hidden layer sizes p" = = jointly. We keep p 0.45, 100, d 1000, and vary d {12.5, 25, 50, 75, 100, 125, 150, 17,200), with hidden layer sizes as in the legend. Experiments here are repeated once.
0.8
0.6
hiddan size p
400
ace5
ac03
600
acc
1.0
Figure 20: For ReLU MLPs we keep the teacher hidden layer sizes pT = 500 and vary the studens hidden layer size D We keep p 0.45, 100, d = 1000, and vary n. with hidden layer sizes as in the legend. Experiments here are repeated once.
21

---

Published as a conference paper at ICLR 2026
In Fig. 21 we zoom into the phase where the student is not matching the teacher but still finding non-trivial acc > acc on the held-out memorized teacher data. While we keep the teacher size fixed at p² 500 we observe that lowering the capacity of the student is optimal for sizes of the dataset. On the other hand, for larger p³ the accuracy decreases but not in a linear way e.g. for 0.3 students with p² 250 are as good as students with p = 1000.
10a
Sudeee hidden sizze
accuracy
0.100
0.075
0.050
0.025
acc
acc
0.000
10a
0.2
0.4
0.6
0.8
Figure 21: The teacher hidden layer size for the ReLU MLP kept at p² 500, while vary the data size fraction and the student hidden layer size pd (100, 200, 250, 300, 400, 500, 600, 750, 1000, 2000}. We report the test accuracy and acc over the student with pd 500. Experiments are repeated 5 times and the standard error on the mean is reported.
22

---

Published as a conference paper at ICLR 2026
D.4.3 VARYING THE NUMBER OF CLASSES
Finally, we examine the effect of the number of classes in Fig. 22 When we scale the r-axis n/d, the behavior we described in the main for the different classes remains similar.
---
20
100
110
acC
022
50
150
200
Figure 22: For ReLU MLPs with p 500 for both teacher and student, vary the number of classes c E {20, 50, 90, 100, 1 110) and show the student's accuracy on DS DS compare with the teacher accuracy.
train test single run, and
E COMPUTATIONAL RESOURCES
All experiments can be run both on a CPU or GPU for multinomial logistic regression CPU may be faster than GPU,
The most computational intensive were the phase diagrams Fig. 4a) with 13 compute days, and Fig. 6a) with roughly compute days on GPU. In both cases though, we ran the full pipeline for parallel experiments on single machine with an NVIDIA RTX A5000 within roughly two days. Since many of the experiments are run for different seeds to obtain error bars, running the experiments once is roughly a factor 5 faster than running all. Since experiments do not require large resources they can parallelized easily.
23


Reviews:
[{"review_id": "iMKz6kvQas", "reviewer": "Reviewer_h94i", "summary": "This paper studies how memorized information can be transferred from a teacher to a student during dataset distillation through soft labels. The authors show that even when the teacher has not generalized, students trained on its soft labels can recover non-trivial information about held-out samples, both in structured and random data settings. Using controlled experiments with transformers, logistic regression, and ReLU MLPs, the paper identifies distinct regimes that separate different forms of information leakage, depending on sample complexity, temperature, and model capacity.", "questions": "1.  Throughout the paper, the authors report the student's train, test, and validation accuracy. To be precise, are these accuracies calculated against the *original* ground-truth labels from the dataset, or against the teacher's (potentially incorrect) labels? The reviewer assumes the former, but it should be written explicitly.\n2.  The paper uses several terms to define the data partitions ( $\\mathcal{D}^{T}$ , $\\mathcal{D}_{train}^{S}$, etc.. ). A small diagram visualizing the relationship and partitioning of these datasets, perhaps in the appendix, would greatly aid in understanding the experimental setup.\n3.  The authors mention training models \"until convergence\" (e.g., in the context of Line 222). Could the authors please specify the exact convergence criteria used? For example, was it a fixed number of steps, a loss-based patience threshold, or another metric?\n4.  In a similar vein to the user note on Line 215, the phrasing \"the size of the complete data distribution\" is ambiguous. Are the authors referring to the cardinality of the sample space, or something else?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "This paper tackles a very cool and relevant topic: whether dataset distillation, specifically using soft labels, can transfer memorized, held-out information from a teacher to a student. The work makes several very interesting observations on this front, and its primary strength lies in the careful experimental design used to isolate this phenomenon. The modular addition experiments, for instance, provide a compelling and clean demonstration of the difference in transfer outcomes between a generalizing teacher and a non-generalizing one. This, combined with the analyses on i.i.d. data using logistic regression and MLPs, creates a strong, controlled environment to probe the core questions of this work. The paper is exciting because it surfaces several intriguing phenomena—such as the clear importance of soft labels highlighted by the shuffling experiment and the complex behaviors hinted at in figures like 4A—which all point toward deeper mechanisms at play in distillation. The initial visualization in Figure 1 is also a very effective and intuitive way to ground the reader in the central concept.", "weaknesses": "The paper's primary weaknesses lie in the clarity of its presentation and the limited scope of the experiments.\n\n* **Clarity and Focus:** The paper presents a wide array of experimental setups (modular addition, logistic regression, MLPs, varying temperature, classes, $\\rho$, etc.) but struggles to synthesize them into a cohesive narrative. Many interesting results are condensed into dense paragraphs, making it difficult for the reader to dissect the core takeaways from each experiment.\n* **Suggestions for Focus:** The paper's impact could be significantly improved by streamlining the main body. The authors might consider focusing on a few core experiments and analyzing them more deeply. For example, a clearer narrative could be built around:\n    1.  The modular addition task, emphasizing the difference between generalizing and non-generalizing teachers.\n    2.  The multinomial logistic regression with ReLU MLPs with a more thorough analysis of the role of $\\rho$ (data fraction) and model scale (partially covered by the $\\alpha$ experiment) in these settings.\n    * Other analyses, such as the effect of the number of classes, seem less central and could be moved to the appendix to improve flow.\n* **Unclear Notation and Definitions:** The theoretical discussion is difficult to follow due to ambiguous notation. The symbols $\\alpha_{label}$ and $\\alpha_{id}$ are introduced, but their practical meaning and the distinction between the \"identifiability threshold\" and \"student memorization capacity\" are not clearly explained.\n    * To give an example, quoting: \"*identifiability threshold: The student can identify the teacher using the logits, measured through the mean squared error loss on the teacher logits, which occurs at $\\alpha_{id} = 1/\\rho$, as the input matrix X becomes invertible.*\"\n    * The phrasing \"the student can identify the teacher\" is imprecise. It would be clearer to describe the empirical observation, such as \"the student's predictions on the test set functionally match the teacher's predictions.\"\n    * Similarly, naming $\\alpha$ the \"sample complexity\" is not very intuitive; a term like \"inverse overparameterization factor\" might be more descriptive. A diagram illustrating the experimental setup and the relationship between these different $\\alpha$ thresholds would be highly beneficial.\n* **Figure Density:** Many figures, particularly Figure 4, are very dense and confusing. While Figure 4A seems to hint at a very interesting phenomenon, the key insight does not come through clearly from the visualization or the accompanying text.\n* **Lack of Scale:** The authors acknowledge this as a limitation, but it is a significant one. All experiments are conducted on very small models. Without even medium-scale experiments, it is unknown whether these observed behaviors generalize to the deeper, more complex architectures where distillation is commonly applied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5EpiNsxpAM", "reviewer": "Reviewer_hkBJ", "summary": "This submission concerns information leakage in model distillation: what does the teacher pass to the student? In particular, when the teacher has memorized some labels from its training data, does this information pass through? The paper shows that the answer is yes. The main results are empirical results in controlled synthetic settings. There is some theoretical analysis as well.\n\nHere's the main setup. We have some classification dataset $(X,Y)$ and train a teacher model $f(x)$ it. We split the data into $X=X_1 \\cup X_2$ and $Y=Y_1\\cup Y_2$. We train a student model on $(X,f(X))$. The experiments show how the student model can memorize the labels $Y_2$ without ever directly seeing them, i.e., the information comes solely through the teacher's labeling.\n\nThe teacher's predictions take the form of a probability distribution over classes and are controlled by a temperature parameter: in low-temperature limit the teacher makes hard choices and in the high-temperature limit the distribution is uniform. Clearly, in the high-temp limit, no information about $Y_2$ gets through to the student. On the other hand, if the teacher predicts the correct labels on its training data, then the low-temp limit corresponds to giving the student $(X_1,Y_1)$, which is independent of $Y_2$. But in between the soft labels seem to convey a lot of information!\n\nThe paper has lots of experiments on this phenomenon in different synthetic settings, with lots of interrelated results. It's not clear to me what the one big takeaway is, other than the fact that dramatic leakage can occur.", "questions": "Is my understanding correct about the implications of Phuong & Lampert's Theorem 1?\n\nHere is another natural setting for these experiments: the teacher trains model $f$ on $(X_T,Y_T)$ and the student trains on $(X_S, f(X_S))$, where $X_S$ and $X_T$ are independently drawn. Have you tried this? Would you expect very different behavior in this setting?\n\nAt points the experiments seem to switch away from cross-entropy, e.g., around line 295. Can you explain why we make that switch?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "I enjoyed reading the submission. The experiments seem well-executed and are designed well: they do a good job probing the phenomenon.\n\nI will likely study this paper further, and I expect many other people will find it interesting.", "weaknesses": "My main criticism of the paper is that the main takeaway, \"soft labels can leak held-out teacher knowledge,\" appears to be known. Theorem 1 of Phuong and Lampert (2019), who the authors do cite as related work, shows that soft labels in a linear classification setting lead to the student recovering the teacher's weights exactly. \n\nNow, this submission has a lot of results beyond the headline, so the work is still valuable. But the takeaways are less clear-cut. With so many experiments, the main part of this paper should probably be twice as long. The results are quite compressed and we jump around a lot. This paper would improve with additional work to craft the story.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aHFWlME4ec", "reviewer": "Reviewer_Z77o", "summary": "This paper studies an exotic problem: when a teacher trained to memorize a dataset and a student trained by distillation on a smaller training partition (compared to the teacher), the student generalizes better than a model trained on the same partition by standard cross entropy.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors have done a meticulous and rigorous job in terms of clearly defining their hypothesis and constructing a specific set experiments which is used to study its validity.\n- Although the experimental framework is constrained (the tasks studied and the networks used), the results presented seem convincing in terms of proving the authors' hypothesis", "weaknesses": "- Motivation: It is not clear what is the motivation of the presented experiments. Even if the paper's hypothesis is true what does that mean in practice and that are the applications? Why should the community pay attention to the results of this paper? While this is somewhat described in privacy paragraph of section 2, more clarity is needed to understand the paper's motivation.\n- Impact of presented results: Can the paper's results be applied to real-world tasks, datasets and networks? Unfortunately, there's no evidence of that as also outlined in paper's limitations. \n- What's the purpose of training on random data? Can this have some real-world implication or application?\n- There's a very large number of experiments and cases analysed that makes the paper difficult to follow\n- Better paper proof reading: e.g. line 235", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)