Generative Diffusion Prior Distillation for Long-Context Knowledge Transfer
Authors: Hewa Dehigahawattage Nilushika Udayangani
Nandakishor Desai
Marimuthu Palaniswami
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Distillation
Time Series Classification
Partial Classifcation
Diffusion Models
Generative Prior Knowledge
Inverse Diffsuion Reconstruction
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a KD framework that models teacher knowledge as a diffusion-based generative prior, relating student features from partial sequences to teacher features from full sequences as degraded-to-clean signals.

Abstract: While traditional time-series classifiers assume full sequences at inference, practical constraints (latency and cost) often limit inputs to partial prefixes. The absence of class-discriminative patterns in partial data can significantly hinder a classifier’s ability to generalize. This work uses knowledge distillation (KD) to equip partial time series classifiers with the generalization ability of their full-sequence counterparts. In KD, high-capacity teacher transfers supervision to aid student learning on the target task. Matching with teacher features has shown promise in closing the generalization gap due to limited parameter capacity. However, when the generalization gap arises from training-data differences (full versus partial), the teacher’s full-context features can be an overwhelming target signal for the student’s short-context features. To provide progressive, diverse, and collective teacher supervision, we propose Generative Diffusion Prior Distillation (GDPD), a novel KD framework that treats short-context student features as degraded observations of the target full-context features. Inspired by the iterative restoration capability of diffusion models, we learn a diffusion-based generative prior over teacher features. Leveraging this prior, we posterior-sample target teacher representations that could best explain the missing long-range information in the student features and optimize the student features to be minimally degraded relative to these targets. GDPD provides each student feature with a distribution of task-relevant long-context knowledge, which benefits learning on the partial classification task. Extensive experiments across earliness settings, datasets, and architectures demonstrate GDPD’s effectiveness for full-to-partial distillation.

Published as a conference paper at ICLR 2026
GENERATIVE DIFFUSION PRIOR DISTILLATION FOR LONG-CONTEXT KNOWLEDGE TRANSFER
Nilushika Udayangani,, Kishor Nandakishor & Marimuthu Palaniswami Department of Electrical and Electronic Engineering
University of Melbourne
Parkville, VIC 3052, Australia
ABSTRACT
While traditional time-series classifiers assume full sequences at inference, practi- cal constraints (latency and cost) often limit inputs to partial prefixes. The absence of dinsmmiisrrmmiettiee patterns in partial data can significantly hinder a classi- fier's ability to generalize. This work uses knowledge distillation (KD) to equip partial time series classifiers with the generalization ability their full-sequence counterparts. In KD, high-capacity teacher transfers supervision to help student learning on the target task. When the generalization gap is due to limited param- eter capacity, matching with teacher features has shown promise. However, when the generalization gap stems from training-data differences (full versus partial), the teacher's s full-context features can be overwhelming target signal for the student's s short-context features. To provide progressive, diverse, and collective teacher supervision, we propose Generative Diffusion Prior Distillation (GDPD), a novel KD framework that treats short-context student features degraded ob- servations of the target full-context features. Inspired by the iterative restora- tion capability of diffusion models, we learn a diffusion-based generative prior over teacher features. Leveraging this prior, we (posterior- ssampe target teacher representations that could best explain the missing raage information in the student features and optimize the student features to be minimally degraded rel- ative to these targets. GDPD provides each student feature with distribution task-relevant long-context knowledge, which benefits learning on the partial classification task. Extensive experiments across earliness settings, datasets, and architectures demonstrate GDPD's effectiveness for full-to-partial distillation.
INTRODUCTION
Many real-world applications in healthcare and industrial automation rely supervised classifica- tion of time series, where the goal is to accurately assign a class label to a given sequence. While traditional models assume access to the entire sequence during inference, this assumption often breaks down in practical settings. In many scenarios, models see only a prefix of the time series due constraints such latency, cost, sensor dropout. For instance, in emergency arrhythmia detection from ECG, decisions may need to be made from 5-10 seconds of data rather than full 60- second recording. However class-discriminative patterns may emerge any point in the sequence, and missing these patterns under partial observability reduces class separability, causing classifiers trained and operated on partial data to generalize poorly. This work investigates how supervised classifiers, trained and operated on partial time series can be effectively equipped with the capacity to generalize from full-length series.
For partial (prefix) time series, the true class is ambiguous since multiple classes can appear iden- tical in the early timesteps before diverging later. Therefore, when training a classifier on partial data, hard-label supervision alone can be misleading, causing the model to overfit to spurious early cues and form unstable decision boundaries. To prevent this, we propose to provide additional reg- ularization signal from teacher model trained on full- full-length sequences, inspired by Knowledge
"Corresponding Author. Email: hewadehigaha @ www.eet..ninnen..cd....

---

Published as a conference paper at ICLR 2026
Distillation (KD). KD, first introduced in Buciluu et al. (2006); Hinton et al. (2015), is train- ing paradigm in which knowledge is transferred from a teacher to guide the training of a student network. The teacher is model that learns representations which generalize well an ability ac- quired through computationally intensive training or greater architectural capacity. This ability can be distilled into student, which may lack the inductive biases to discover such representations from training data alone, often due to limited training resources or parameter capacity. The most widely adopted approach to KD trains the student match the teacher's output logits, providing an additional regularization signal during optimization on the target task (Hinton et al. (015)). Later works (Romero et al.) 2015 Park et al., 2019 Zagoruyko & Komodakis 2016) extended this idea to match intermediate features beyond the output logits.
However, these direct feature/logit matching KD methods were proposed to address the general- ization gap arising from differences in parameter capacity, with both the student and the teacher privileged to see the same data. In contrast, when distilling knowledge from a teacher trained on full-length sequences to a student trained on partial sequences, several fundamental concerns arise: 1) Can the distillation technique transfer the teacher knowledge effectively? Even when the teacher is highly capable model with strong representational quality, the student may fail to prop- erly comprehend the transferred knowledge, leading to limited gains from KD (Cho & Hariharan 2019 Mirzadeh al., [2200] Qiu et al.) 2022; Sttnton et al. (02T). Prior research has attributed this issue to the capacity/architectura gap between teacher and student, and proposed intermedi- ate "teacher-assistant models Mirzadeh et al. (2020); Sonn et al. (202I) and student-friendly teacher training (Park et al. [221]] Rao et al. 2003] Cho & Hariharan 2019). However, when the teacher and student are exposed to different input spaces (e.g.. full versus partial data), an inherent representa- tional gap is introduced even when the models have identical parameter capacity. In such scenarios, if the distillation loss is poorly designed by directly enforcing alignment with the teacher's full- context features, it can overwhelm the student, which only encodes partial-context features, and thereby limit its ability to effectively absorb the transferred knowledge.
2) Is single teacher's perspective diverse enough? Exposing students to diverse yet consistent views of the same underlying information enhances generalization and fosters robust inductive bi- ases You et al. (2017); Allen-Zhu & Li (2020); Hossain et al. (2025). While existing works promote |
diversity through teacher ensembles Allen-Zhi & Li (2020): You et al. (2017) or mutual supervi- sion among student ensembles Zhang et al. (2018); Furlanello et al. (2118), key concern remains whether supervision from a single model provides sufficient diversity of knowledge. (Hossain et al. 2025) demonstrate that multiple augmented teacher views can be generated from single model by perturbing its features with random noise. This increases knowledge diversity while avoiding the cost of retraining multiple models. Providing diverse perspectives of teacher knowledge is partic- ularly important in full-to-partial distillation, where student fcatures are degraded, incomplete, or ambiguous compared to those of the teacher, as we do not want to overcommit to a single possible interpretation of the missing or noisy information.
3) Is the knowledge faithful? KD transfers limited knowledge leading students with very different predictive distributions from their teachers, hindering safe substitution (Stanton et al.) 2021 Lamb et al.] 2023). Even though knowledge improves predictive accuracy, achieving good fudelity the ability of the student to match teacher predictions, with existing methods is extremely difficult (Stanton et al. 202I). Stanton et al. (202I) observe that augmenting the distillation set with data samples not present in the teacher's 's training data increases the drop in distillation fidelity. Similarly, the training-date data mismatch that arises when the teacher is exposed to full-length data while the student is exposed to partial data can make it more challenging for the student to match the teacher's 's predictive distribution.
With these concerns in mind, rethink distillation from a different lens and propose Generative Diffusion Prior Distillation (GDPD). In GDPD, we view student representations learned from partial sequences as degradations (partial measurements) of target teacher features derived from full-length sequences. Inspired by the iterative restoration power of diffusion models (Kawar et al.) 2022), we train diffusion model to serve as generative prior over teacher features, capturing and storing their statistical structure. Using this prior, we search within the space of teacher features for target representations with optimal teacher knowledge, and train student features to become minimally degraded relative to these discovered targets. Unlike conventional KD, which provides single teacher signal, we model knowledge as a distribution over target teacher signals. We discuss how this distributional knowledge helps GDPD to address above three concerns of KD, exacerbated
2

---

Published as a conference paper at ICLR 2026
in full-to- full-to-partial partial distillation, by generating teacher signals that 1) are dynamic and progressive with respect to the student's current capability, 2) provide stochastic diversity of the same features, and 3) complete optimal knowledge through collective aggregation (Section 3.4.
In short, we make following contributions: I) demonstrating that KD can equip early time-series classifiers, operating partial time series, with the generalization ability of classifiers trained on full-length time series, establishing this direction as a pioneer effort, 2) being the first to model teacher knowledge lationship as an ill-posed problem 3) introducing a novel KD framework, GDPD, to provide dy- namic, diverse, and collective knowledge for effective full-to- tolatttaaatall distillation, and providing an in-depth analysis and discussion evaluating GDPD and baseline KD methods in full-to-partid distillation.
generative distribution, formulating the target teacher-student feature re-
2 PRELIMINARY
Knowledge Distillation. KD seeks optimal student parameters by jointly minimizing the task loss Task and a distillation loss LKD that aligns the student with a pre-trained teacher:
0. = arg min Task Task (0) + AKD LKD (0),
(I)
0
where Task and AςD control the relative contributions of the two terms.
Diffusion Models. Given samples from the data distribution Pdata diffusion models are capable of learning parameterized distribution P& that approximates Páata and is easy to sample from Song et al. (020bb). This is achieved through forward diffusion and reverse denoising processes. The Forward Process is Markov chain that gradually corrupts data Zo Pduta until it approaches
Gaussian noise zy ~ Piment = (0, I) after T' diffusion steps. Corrupted latent variables -,, are sampled from Pasm with diffusion process defined as a chain of Gaussian transitions:
T II g(z 1
(ZZI.T zo)
g(z 2 N(zii / 1- Br 1: Be1),
with a fixed or learned variance schedule { ing process is that any marginal at step has closed form Ho al. (2020): q(z: zo) N(zz; vā. Zo: (1 a.) with 0ę 1 3r and ā₂ = II==1 ag.. Equivalently, any step z, can be directly sampled from Zo: z, vāa Zo + VI- E, with AO I). The Reverse Pro- cess is Markov chain that iteratively denoises sampled Gaussian noise to a clean data. Starting from zr (Zzi I,I,, we learn parameterized reverse process from latent Zy clean data Zo. as chain of Gaussian transitions:
important property of the forward nois-
T
Po(Zor) p((T)( II (2 2z), Pa(2 2.) M(2 Meg (Ze, 1), Egl).
The mean us (Z,, f) is primarily what we want to learn using neural network Ho et al. 22020). The
variance Eg can be either time-dependent constants or learnable Nichol & Dhariwal (202 (). A func-
tion approximator Es predicts the noise from 2, and sets: Pt (g(2,, t) you 2
vT E (2 1)
9.
Training minimizes the & loss between the true forward noise e and the predicted noise co (Zc, t):
Ldittusion (Φ) |E| || Ez (o(,,,))1]]
Sampling/Guided Sampling. During inference, sampling is performed by running the learned reverse process starting from Gaussian noise. Guided sampling augments this process with external signals (e.g., labels or features) to steer generation toward desired conditions Dhariwal & Nichol 2021).
Inverse Diffusion Problem. Given a degraded measurementy D(z.), D defines the degradation of the clean signal Zp, the objective to recover Zn by sampling from the posterior p(z, y) oc p(y 20) po po (Zn), where (2(2)) is diffusion prior learned from data (Kawar al. 2022).
3

---

Published as a conference paper at ICLR 2026
3 METHOD
3.1 PROBLEM FORMULATION
Let D y') ====1,--- 1,....,,, N) denote time series dataset with N samples, where x, R is a time series with M channels and L time steps, and y; E RC is the one-hot encoded label corresponding to the ground-truth class
{1....,C) C). We henceforth write generie sample as x. We denote by X E RMxxe a partially observed time series containing only the first e < L time steps. While our main focus is on timestep-wise partialness, i.e., observing only a prefix of the sequence, we also evaluate the case where channel-wise partialness is present, with only subset of channels m < M observed, with X E Rm The goal of this work is learn student classifier So that can effectively map early, partially observed inputs x. to their corresponding labels y, while leveraging the knowledge of teacher TI trained on the full-length sequences x. We seek optimal student parameters 0: by minimizing
CTssk E y ((CE(S)(Xx)) y)], and (KD(O) Ex-D Ex-D|e((rree (x)) (x 0))
Here, tCE is the cross-entropy loss, and Φz. Os are functions to be determined that capture teacher and student behavior on full and partial inputs, respectively. By minimizing the discrepancy measure ((.,)), we encourage the student to behave on partially observed inputs as the teacher would on the full-length sequences.
3.2 GENERATIVE DIFFUSION PRIOR DISTILLATION
We write the student model as Sg = Shead Steat where Steat denotes the mapping up to the feature extraction layer k, and Shead denotes the subsequent mapping from these features to the final predic- tion. Training the student on the partial sequences D. = {(xx. y)) }NN, we define Stont (x.) = Zehort as the intermediate feature of the partially observed input x,, referred to as the short-context feature. Let us assume there exists feature "dong-ideal> which encodes the optimal long-range information required for making accurate predictions from Xe, as would be obtained if the model had access to and were ideally trained on full-length sequences. During training, our goal is to guide the student model such that it produces features Zahort that resemble Llong-ideal as closely as possible, as if its predictions were informed by the full-length sequences, with the aim that this behavior generalizes to partial sequences at inference time.
We denote by Zlong the features of a teacher model trained on the full-length sequences D, referred to as the long-context features. To capture the teacher's knowledge, we train diffusion model on Zlong to approximate the distribution of long- context features, p( p( We assume there exist features (possibly multiple) within the teacher's feature manifold that can provide useful hints of Long-ideal- We call these hini features and denote them by blong-hint ~ p(Ztong). We start by viewing Zahort degraded partial measurement of the underlying clean feature Zoongiideal which contains the optimal long-context knowledge and each Zing as a valid approximation (or completion) of Long-ideal- The diffusion model trained on the teacher feature space serves as an effective prior P& (Zong),, capturing the statistics of plausible long-context features. Using this prior knowledge, our goal is to guide the student model to produce Zshort that retains as much information as possible about its underlying clean feature zoong-ideal- Accordingly, we aim for the student features to be minimally degraded relative to the hint features Zlong-hint. and thereby closer to Zong
Teacher Knowledge as Generative Prior.
idng-ideal' ideal'
To define the relationship between student features and hint features, we model the diffusion pos- terior sampler Past (Zong Zshort)) where we utilize the pre-trained generative diffusion prior ps (Zlon) to search in the space of Zlong, for an optimal Zlong that best matches Zshort, regarding Zehort as degraded observation of Ziong. A posterior reconstruction sample 2log ~ Pai (Zing Zshort.) represents a plausible completion (clean signal) consistent with the partial information present in Zehort. We argue that, if the student produce features that preserve sufficient informa- tion of hint features, that is when Zshort is minimally degraded to Zlong-hint. then student features should recover hint features as their posterior reconstruction samples. In other words, if Zshore are sufficiently informative with valid long- contect knowledge, then they should provide the right con- ditioning to recover one such representation, Zlong-hiut: as their plausible completion:
2 Tong
paitt [Zongg Zshorr; O.)
Mong
Zlong hint:
(3)
4

---

where 0, is the optimal student parameters, what we are after. Recall that hint features are function- ally defined teacher features that contain relevant long-context information necessary to assign the correct prediction to x,, We characterize hint features by this predictive property, which posterior reconstructions are trained to emulate under optimal student parameters. Therefore, during training Published as a conference paper at ICLR 2026
X
Laiflusion
Diflusion Prior Pla)
2 pizy
Rustr))
-W
zye's
%
S(e)
Laask
LGOPD
S: Student
Teacher W: L hase weights
Q WWighte for nokee fuzing block
KD: direct
GDPD: uses collection of
X: inputs X- Pertwilnpuis V: Targats
comseponning teacher signal ressive teacher sigrali
GDPD
GDPD
Figure/ 1: Depiction of (a) the proposed GDPD framework, and (b) a comparison of KD vs. GDPD. KD provides single static teacher signal, whereas GDPD provides each student feature with collection of diverse and progressive teacher signals over the course of training.
we optimize the student features by constraining their posterior reconstructions to output the correct label:
LCDPD( (x [ECT( Shead (2())
.}]
Monn ~ Paiff (Zlogg
Zehort
Sg (xx))
Conditional Generation with Unconditional Prior. To enable sampling from p(Ziong Zshort using an unconditional diffusion model, we adapt the reverse process to condition on Zshort during inference, in line with guided sampling. Typically, guided sampling inverse diffusion modifies the score function at each reverse step using approximation of the likelihood gradients Chung et al. (2023). In contrast to those settings, where the degraded measurements are fixed, our conditional signals subject to optimization, which prompts to require simple and direct form of guidance from them (see Section A.4.I for a detailed discussion). Therefore, we adopt straightforward con- ditioning strategy by initializing the reverse diffusion process directly based Zahort Specifically we match each student feature 4short to the initial noisy step T by fusing Gaussian noise, and use this initialization start the reverse process:
Zlong. Zshoot + a1- E, (~N(0.,T
(5)
where a is the fusion weight between the two terms, which can be treated as fixed hyperparameter learned feature-wise during the distillation process. We choose leam a during distillation, as this allows different features to be fused with different noise levels so that each is mapped appro- priately the initial noise step. With this initialization, reverse process is conditioned on Zehort allowing the algorithm to explore the feature manifold, ideally staying close to the starting point Zshort, so that converges a plausible clean feature Rzong consistent to initialization. Over the course of optimization, posterior sampling connects different long contect features to student fea- ture (Equation 44). This enables student feature to evolve toward optimality by leveraging their collective knowledge, which can better approximates the knowledge of ionggideall- An overview of this proposed method is illustrated in Figure [ (a).
Training. Our student training proceeds in two phases, separated by warm-up epoch Ewarm- For ep Ewarrm we train only the diffusion prior on teacher features with the student initialized on the partial classification task. For ep Ewarm- the student is optimized extract long-context knowledge using the learned diffusion prior.
(aaak (0) Ldiffusion ()),
ep < Ewarm:
Lirsin
(6)
ATask L (0) ARD LODPD((), ep Ewarm
5

---

Published as a conference paper at ICLR 2026
3.3 "KNOWLEDGE" AS A DISTRIBUTION
Conventional KD (deterministic / point knowledge). KD treats the teacher signal that each student state (feature, soft label, relation) should match point larget. For a student state Z, = Zz. the "knowledge" is taken to be a single teacher state: k* 22, with 2, the corresponding observed teacher state (equivalently, Pkiz- =z, = 6x-). Supervision then enforces alignment of the student state with this single target, with discrepancy loss: f(z,; 0,k')
Common instances include:
(Feature KD) 2₁ = fx(x;0), kt = f((x), Ł= (=11f((x;0) - f((x)]1;
(Logit KD) Z. - --.(- x; 0), k* - pi(: x); t= KK-Kppp-1 1x) plpp( xx; 0));
(Relational KD) 2, r<,((x 0), (x:: k' +==((((x,
(= (=,z--k)) k*)²
GDPD (generative / distributional knowledge). In contrast, GDPD provides distribution of plausible teacher signals for each student state, consistent with what the student currently knows. Rather than treating knowledge as single target, GDPD models it as distribution from which the student can learn to sample in order to acquire optimal and diverse task-relevant knowledge. Formally, for a student state 2s, the "knowledge" is a distribution over teacher states: ~ p(K Z, V.). More robust supervision can be defined as the expected loss (approximated by Monte Carlo average over samples) under this distribution:
[((zz) k)] R 1 (2: kDt) p(. 12.)
Since each forward pass in GDPD explores a different noise trajectory, the stochasticity across training naturally covers multiple samples over time. Therefore, it is sufficient in practice to use J= 1. See ablation over J in the Section
3.4 How GDPD ADDRESSES FUNDAMENTAL KD CONCERNS, EXACERBATED TULL-TO-PARTIA DISTILLATION
How Does GDPD Transfer the Teacher Knowledge Effectively? Since features derived from full and partial observations are not directly aligned in representation space, directly enforcing the partial-context student states 2, to match full-context teacher states k* z, as point targets can overwhelm the student. GDPD alleviates this gap by providing dynamic and progressive teacher signals. Each student state 2, samples its target teacher signal k ~ p(K Z, = 2,; 0¹) from the teacher manifold, consistent with the student's current knowledge forward pass. In this way, the teacher signal adapts over the course of training, allowing the student progressively refine ability to recover correct teacher features and ultimately absorb richer knowledge.
reflected in z,; O during each
How GDPD Provide Diverse Teacher Perspectives Using Single Model? When the student operates with degraded, incomplete features compared to the teacher, overfitting to a single teacher perspective can risky, i.e., single set of features (or logits) binds only one possible interpretation of the missing or noisy information. One possible way to create diverse perspectives from a single teacher through stochastic diversity, i.e., generating multiple noise- perturrer variants of the same feature (Hossain et al.) 2025). However, hand-designed perturbations may produce distant, irrele- vant teacher signals, outside the meaningful teacher manifold. GDPD use diffusion models which naturally works by generating samples in the close vicinity of the target distribution Chen et al. (2024). Unlike randomly perturbed feature variants, the diversity of diffusion -eeneiinngeeeetted teacher signals in GDPD is controlled: they not arbitrary but sampled as plausible completions of the student features, k ~ p(K] Z, Z.hoo))
How Does GDPD Transfer Faithful Knowledge? When the teacher's training data and the dis- tillation set differ, the knowledge from direct corresponding teacher signals z, alone becomes very limited, making it difficult faithfully replicate (predictive distributions of) the teacher (Stanton et al. 2024). In GDPD, as the student is trained toward optimal states,
et al. 2021 each student state interacts with a collection of teacher signals { }: k() p(K Z, = z.),
which collectively construct comprehensive long-range knowledge for 2g. See Figure [(b. Unlike
(b).

---

Published as a conference paper at ICLR 2026
Agreement 80
50
70
(a) Earliness 0.2
(b) Earliness 0.4
(c) Earliness = 0.5
(d) Earliness 0.8
Figure 2: Fidelity comparisons across methods and earliness levels, with each boxplot summarizing 12 UCR datasets. Fidelity is measured as teacher-studen. top-1 agreement on the test set.
Table 1: Summary of performance across different earliness levels on 12 UCR datasets. Best values are bold. Rows marked with + indicate lower-is-
E2
Earliness=0.44
Earliness=0.6L.
Earliness=0088L
Base BascKD
Fits
GDPD
Base
BaseKD
Fits
GDPD
Base
BaseKD
Fits
GDPD
Base
BaseKD
Fits
GDPD
Avg-AUC-PER
63.64
69.23
67.47
73.83
70.44
78.03
75.36
81,70
76.79
83,70
81.15
86.00
77.79
84.78
82.74
89,02
Avg. Rank L
3.50
2.42
2,92
1.17
3.58
2.50
2.67
1,25
3.58
2.58
2.42
1.33
3.67
2.42
2.83
1,08
Nuin Top
0
2
0
10
0
1
1
10
0
3
2
0
0
1
11
Wins/Draws
12
10
12
11
10
11
4
the supervision from single corresponding teacher signal k* 24, this collection better reveals the statistical structure of teacher features (class separability, geometric relationships between features). We validate faithful knowledge empirically in Section 4.1
EXPERIMENTS
To reach generalizable conclusions across partialness levels, architectures, datasets, and runs, evaluate multiple settings. Experiments are conducted on the UCR univariate (Dau et al.) 2019). UEA multivariate (Bagnall al. 2018), and real-world PhysioNet mortality dataset (Silva al.) 2012). Notation Ne/l + Nel2 denotes teacher-student distillation. All students use the same training protocol, and results are averaged five runs. Full experimental setup details in Section A.2
4.1
MAIN RESULTS
GDPD Effective Across Varying Degrees of Partialness. We generate time series with varying earliness by truncating at {0.2L, 0.4L, 0.5L, 0.6L, 0.8L, L}. where L is the full length. The teacher is trained on full-length series, while students are trained on truncated series using KD from logits (Base-KD) KD) (Hinton et al., 2013). features (Fits) (Romero et al. 2015), and GDPD, along with a baseline student (Base) trained without distillation. Results for LSTM3-100 - LSTM3-100 on 12 UCR datasets shown in Table and Table 19}. At each earliness level, distilled students achieve higher AUC-PRO and lower rank than the Base student, showing that full-context teacher knowledge improves partial classification. GDPD attains the best AUC-PRC and rank, winning on over 80% of datasets, demonstrating its effectiveness over direct feature- and logit-KD across partialness levels.
GDPD Outperforms Existing KD Variants. To further compare GDPD with different KD vari- ants, we construct students using RKD (Park et al., 2019), Attention (Zagrrukko ko & Komodakis,
2016 DKD (Zhao al. 2022). DT2W QQio al. 2023). VID Ahn et al. 2019 PKT (Passalis et al. 2020), TeKAP (Hossain et al. 2025). TTM (Zheng & Yang 2024), Base-Ki KD D, Fits, and Base. Performance is evaluated at partialness
0.5Z on 12 UCR datasets for LSTM3- 100 - L.TM3-- 100 (Table 2p. All students improve over the Base, showing that they benefit from long-context knowledge. GDPD outperforms all methods, achieving top-3 performance on 80% datasets with an average rank of 2.25, while no other method achieves rank close to 4.
GDPD Improves Student Fidelity Beyond generalization across earliness levels, we also assess whether GDPD yields students with higher fidelity. Figure 2 summarizes student fidelity, measured as average top-1 agreement over 12 UCR datasets for LSTM3-100 - LSTM3-1000 at different ear- liness levels. This metric is the percentage of instances where the student's top-1 prediction (from

---

Published as a conference paper at ICLR 2026
Table Comparison of distillation methods at earliness 0.5L on 12 UCR datasets. Test-fidelity is reported as average top-1 agreement. Rows marked with indicate sower-is-bettee
Base
Buse KD
Fits
VID
DKD
Attention
DT2W
RKD
RKDle
PKT
TeKAP
TTM
GDPD
Avg. AUC-C
71.6
78.14
77.28
77.19
74.29
75.47
55,32
79.89
79.26
76.86
76.54
78.05
84.64
Fidelliyy
72.13
71.32
52.59
Avg. Rank
9.67
5.83
7:58
6,33
8.25
7.25
11.42
6.33
6.42
6.67
6.75
6.17
2.25
Num. Top-1
0
2
0
0
0
2
0
Num. T
WineDraws
12
10
12
11
12
10
11
11
Losses
0
1
0
0
I
Table 3: Results on 12 multivariate datasets under: 1) time-wise (e = 0.5L), and 2) time+channel
(e 0.5L, 0.5.) partialness. Each cell reports Avg. AUC-PRC/ Avg. Rank / #Top-1 wins.
Inception55-32 Inception55-32
LSTM3-100 LSTM3-100
Time-wise
Time++Chamel
Time wine
Time+Chamnel
Base
86,22 3.25 /0
0..92/2.92/0
79.41 3.500/
68.75/3.42/0
Base-KD
88.77 2.00
77.92. 2.08
81.961 2.50/3
71.11/2.75/1
Fits
3.17 /.140/
75.34/3.42/
81,95 / 2.42/2
71.69 /2
GDPD
90.82 1.33/
80.24/ 1.33
83.87 / 1.5877
73.60 1942/9
Table 4: Model compression results under two carliness levels (c 0,5L, L) and two compression targets. Each cell reports Avg. AUC-PRC/ Avg. Rank Num. Top-1 wins across 12 UCR datasets.
LSTM3-100 - LSTMI-8
LSTM3-100 LSTM2-32
0.5L
L
e 0.5L
e L
Base
58.58/3.42/0
74.92/3.33/0
72.98/3.33/4
84:34/3.67
Base-KD
2..62/2.33/3
/211//
Fits
59.84/2.67 /O
76.49 /
78.26 78.26/2.75/
/599//2.75
GDPD
72.76/ 1.17 111
78.83
1.42
83.67 1.33/9
89.84/ 1.25 /9
partial data) matches the teacher's top-1 prediction (from the full sequence), directly quantifying how well the student replicates the teacher's behavior Sttanton et al. (202I). In all settings, GDPD students achieve higher fidelity than baseline KD, demonstrating its effectiveness in faithfully trans- ferring desirable teacher behavior under different degrees of partialness.
GDPD is Robust with Channel-wise Partialness. We evaluate GDPD's robustness under channel- wise partialness using 12 UEA multivariate datasets with two settings: 1) time-wise partialness only (e = 0.57.), and 2) combined time- and channel-wist partialness (e = 0.57. m 0.5M), where half the channels removed (Table GDPD consistently achieves the highest AUC-PRC, lowest average ranks, and most top-1 wins, confirming its robustness to channel-wise partialness.
GDPD Effective in Model Compression. We evaluate GDPD's effectiveness for model com- pression under two scenarios: 1) compression with partialness (e = 0.5L) and 2) compression only (e L). Results for compression targets in Table show GDPD consistently achieves the high- est AUC-PRC, lowest rank, and most wins, confirming its effectiveness for compressed students.
GDPD Provides Effective Self-Distillation. When teacher and student architectures are identical and no partialness is involved (e = L), our experiments correspond to self-distillation (Pham et al. 2022]. We evaluate GDPD under this setting Table [ GDPD achieves higher AUC-PRC and lower rank than the teacher, surpassing them on most datasets. GDPD proves more effective for self-distillation than vanilla KD. Justification for improvements: Even without teacher-student representational gap, GDPD exposes each student feature to multiple vicinal features (analogous to feature augmentation), which helps the model generalize better while avoiding overconfidence.
GDPD Across Different Network Architectures. We also evaluate GDPD under 1) similar teacher-student and 2) cross-architecture distillation. GDPD consistently achieves the highest per- formance gains, demonstrating effectiveness in both settings (see Table 18).
GDPD Transfers Long-context Knowledge. To assess whether GDPD students learn representa- tions that generalize from full-context teacher knowledge, evaluate representation transferability to the suffix (last 0.5L) of each time series. We use two protocols: 1) Linear probe on frozen backbone: Train the student on prefix data (first 0.5..), freeze its backbone, then train linear clas-
8

---

Published as a conference paper at ICLR 2026
Table 5: Self-distillation results under three network architectures (e L). Each cell reports Avg. AUC-PRC/ Avg. Rank across 12 UCR datasets.
LSTM3-100 100 - LSTMM-1100 100
Inception55-32 32 + Inception5--32
ResNet33 64 ResNet32-64
Teacher
87.32/3.33
96.96 2.83
98.43 /2333
Base-KD
89.29 / 2,08
97. 80.8 1.50
98.52/ 2.25
Fils
88.73
93.90 3.92
98.55 2.33
GDPD
Table Transferability from prefix-trained students to suffix inputs on the StarLightCurves dataset. We report AUC-PRC; best values per row are in bold.
Base
GDPD)
Linear probe
65.57
74.86
65.74
76.88
Zero-shot 35.21
66.21
48.05
66.70
Table 7: Results on the PhysioNet case study in terms of AUC-ROC, AUC-PRC, and Accuracy. Four settings under = 0.5L are considered: (I) main-task distillation, (2) downstream task (survival 2 100 prediction), (3) channel-wise partialness, and (4) balanced-teacher to imbalanced-student.
In-hospital Mortality
Survival 2:00
Channel Partialness
Balanced - Imbalanced
0.5L)
0.5L)
(c 0.5L, m 0.5M)
0.5L)
ROC
PRC
Acc.
ROC
PRC
Aec.
ROC
PRC
Acc.
ROC
PRC
Acc.
Teacher
76.04
76.12
70.72
76.04
76.12
70.72
76.04
76.12
70.72
76.04
76.12
70.72
Base
70
65.58
62.78
59.52
75.96
72.18
66.61
59.79
GDPD
74.45
73.74
68.56
70.52
69.74
65.13
61.31
62.18
57.40
76.88
65.84
86.10
sifier on features extracted from suffix inputs. This tests whether prefix-learned representations are linearly useful for classifying suffix inputs. 2) Zero-shot suffix evaluation: Evaluate the frozen prefix-traines student (backbone + original head) directly on suffx inputs without any additional training. Results on the StarLightCuures dataset for LSTM3-100 + LSSM3-100 are presented in Table 6 These results indicate that representations learned with GDPD exhibit strong transferabil- ity: GDPD achieves the best linear-probe and zero-shot performance on suffix inputs, evidencing that it effectively acquires and transfers long-context temporal knowledge.
GDPD Degrades Gracefully under Weak Teacher Supervision. Under progressively degraded teacher supervision, GDPD consistently achieves the highest performance and the slowest decline (Section A.3.5).
Real-World Case Study: Predicting In- In-Hospital Mortality. The PhysioNet Silva al. (2012) dataset contains electronic health records from ICU patients. The main task is to predict in hospiaal mortality using first 48 hours recordings after admission. We also derive auxiliary downstream task, survival 2 100 prediction, to evaluate cross-task distillation (see Section A.2.I]. We train the teacher on the main task using a balanced set of full-length data, In the first scenario, stu- dents are trained on the same task with partialness (e = 0.5L). We then consider three addi- tional scenarios introducing further teacher-student student heterogeneity: 1) training students on down- stream task (survival 2 100 prediction) with e 0.5L; 2) training under channel wise partialness (e 0.5L, 0.5M)); and 3) training on an imbalanced dataset with 0.5L. Table 7 Jindicates GDPD consistently outperforms Base and feature-KD across time- and channel-wise partialness, cross-task distillation, and when the class distribution of the distillation set differs from the teacher's training data, highlighting its robustness under heterogeneous real-world conditions. Cross-task gains show GDPD extracts task-relevant knowledge more effectively than direct feature matching.
4.2 ABLATIONS AND HYPERPARAMETER STUDY
To gain insights into the role of each component in Equation (6) we conduct an ablation study on the StarLighiCurves dataset for LSTM3-100 - LSTM3-100 100 with earliness set to 0.5L.
Ablation on Phase Scheduling. Under disabled warm-up and diffusion training (Ewarm 0, Laiifusion (Φ) 0), freezing yields 83.12 AUC-PRC and training without freezing yields 85.56, both near the Base (83.90), indicating GDPD provides no benefit without proper diffusion prior training. With warm-up disabled, jointly training the diffusion prior (Laiffusion + LGDPD)

---

Published as a conference paper at ICLR 2026
Table 8: Effect warm-up epochs (Ewarm) on GDPD performance (AUC-PRC).
Ewarm
(frozen (b)
(unfrozen db)
(joint)
50 100
300 400
95.31
96.78
yields 89.72 AUC-PRC, suggesting coupling helps even without a warm-up. Next, sweep Ewarm in Table [ At Ewarm = 600 = total epochs, training reduces to the task loss only, collapsing to the Base (83,90). The best results emerge when the warm-up occupies roughly half of the training epochs ge. 97.26 at Fwarm 300). These findings show that 1) the diffusion prior is essential, 2) phase-wise training improves GDPD, and 3) mid-range warm-up durations yield the highest gains.
Ablation on Loss Terms. Setting AKD = 0 yields the Base (83.90). Using only the GDPD signal (ATask = 0) gives limited improvement (2),2), while combination (ATask = 1, AKD = 1) achieves the best result (97.26). With ÀTask 1, we sweep ARD in Table 13 where any non-zero GDPD contribution improves over ÅKD 0. To verify that GDPD drives this gain, we replace LGDPD (Equation
( with logit-KD, KD,, which gives 94.51, close to Base-Ki KD (95.20) but inferior GDPD. Substituting feature-KD yields 83.81, similar to Fits (81.87. This suggests that, instead matching single static teacher signal, GDPD's diverse and progressive signals drives performance gains. We further validate this by increasing diversity, estimating LGDPD with multiple posterior samples J= 2,2,3, 3, 4, (Equation ). which yield 97.26, 97.34, 97.13, 97.87, and 97.78, respectively. The gain for > can be attributed to the additional diversity introduced within each mini-batch. Since
1 already achieves strong performance, we adopt for a better efficiency-accuracyy accuracy trade-off. We also ablate alternative GDPD implementations that reduce diversity in Section A.3.I Ablation on Diffusion Controls are provided in Section A.3.1
Computational Cost. GDPD incurs only modest training time overhead (0.24 s/epoch over Base- KD) and does not affect inference. Its computational cost comparable established feature KD (RKD, VID), and memory footprint is far below memory- memory-intensive methods (RKD). Detailed wall- clock and step-level analyses are provided in Section A.3.2
CONCLUSION
This paper proposes a novel KD framework for efficient knowledge transfer to bridge the general- ization gap from partial data. In conventional KD, directly matching a single set of teacher features can result in incomprehensible knowledge due to data gaps, limited and brittle knowledge tied to one perspective, and unfaithful knowledge from training-distillation distillation differences. To address this, we propose capturing teacher knowledge as a generative diffusion prior that serves as reservoir from which the student can progressively sample diverse and faithful knowledge. We conduct extensive evaluations of GDPD) across partialness levels and distillation settings, demonstrating consistent im- provements over existing KD approaches on benchmark datasets. Additionally, we validate GDPD on real-world dataset under challenging heterogeneous conditions. This paper opens new search direction by proposing teacher knowledge as an effective form of generative prior.
ACKNOWLEDGMENTS
This research was supported by The University of Melbourne's 's Research Computing Services and the Petascale Campus Initiative. We also gratefully acknowledge the authors and maintainers of the UCR, UEA, and PhysioNet time series repositories for making their datasets publicly available.
REPRODUCIBILITY STATEMENT
The source code for all models, training scripts, and experiments is available at https: SaapaooeeiiaaaaaCC___CCCRR Details of datasets, preprocessing steps, model configurations, and training procedures provided in the main text (Section 4 and Ap- pendix (Section A.2). Additional results are included in the supplementary materials to further support reproducibility.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, pp. 9163-9171, 2019.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning, arXiv preprint arXhv:2012.09816, 2020.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.0009 2018.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil, Model compression. SIGKDD, pp. 535-541, 2006.
Ekansh Chauhan, Swathi Guptha, Likith Reddy, and Bapi Raju. Lrh-net: A multi-level knowledge distillation approach for low-resource heart network. In International Workshop on Distributed, Collaborative, and Federated Learning, pp. 190-201. Springer, 2022.
Minshuo Chen, Song Mei, Jianqing Fan, and Mengdi Wang. An overview of diffusion models: Ap- plications, guided generation, statistical rates and optimization. arXiv preprint arXiv:240.00777, 2024.
Jang Hyun Cho and Bharath Hariharan. On the efficacy knowledge distillation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4794 4802, 2019.
Hyungjin Chung. Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh Interna- tional Conference on Learning Representations, 2023. URL ptttpp://epeereview.u
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. JEEE/CAA Journal of Automatica Sinica, 6(6):1293-1355, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 8794, 2021.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML, pp. 1607-1616. PMLR, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 103.0033, 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
S Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
Md Imtiaz Hossain, Sharmen Akhter, Choong Scon Hong, and Eui-Nam Huh. Single teacher, mul- tiple perspectives: Teacher knowledge augmentation for enhanced knowledge distillation. In The Thirteenth International Conference on Learning Representations, 2025.
Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowl- edge diffusion for distillation. Advances in Neural Information Processing Systems, 36:65299- 65316, 2023.
Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petit- jean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, (436))133311988 2020.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in neural information processing systems, 35:23593-23600 2022.
11

---

Published as a conference paper at ICLR 2026
Tom A Lamb, Rudy Brunel, Krishnamurthy DJ Dvijotham, M Pawan Kumar, Philip HS Torr, and Francisco Eiras. Faithful knowledge distillation. arXiv preprint arXiv:2306.0443) 2023.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Usue Mori, Alexander Mendiburu, Eamonn Keogh, and Jose A Lozano. Reliable early classifica- tion of time series based on discriminating the classes over time. Data mining and knowledge discovery, 31(1):233-263, 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International conference on machine learning, pp. 8162-8171, PMLR, 2021.
Amin Parchami-Araghi, Moritz Böhle, Sukrut Rao, and Bernt Schiele. Good teachers explain: ialaamattio--nhanneee knowledge distillation. In European Conference on Computer Vision, pp- 293-310. Springer, 2024.
Dae Young Park, Moon-Hyur Cha, Daesin Kim, Bohyung Han, et al. Learning student-friendh teacher networks for knowledge distillation. Advances in neural information processing systems, 34:13292-13300 133033, 2021.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, pp. 3967-3976, 2019.
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Probabilistic knowledge transfer for lightweight deep representation learning. IEEE Transactions on Neural Networks and learning systems, 223502000-, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NeurlPS, 32, 2019.
Minh Pham, Minsu Cho, Ameya Joshi, and Chinmay Hegde. Revisiting self-distillation. arXiv preprint arXiv:2206.08// 2022.
Zhongzheng Qiao, Minghui Hu, Xudong Jiang. Ponnuthurai Nagaratnam Suganthan, and Ra- masamy Savitha. classinncremenaa learning on multivariate time series via shape-aligned tem- poral distillation. In ICASSP, pp. 1-5. IEEE, 2023.
Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Bet- ter teacher better student: Dynamic prior knowledge for knowledge distillation. arXiv preprint arXiv:2206.06066 060677 2022.
Jun Rao, Xv Meng, Liang Ding, Shuhan Qi, Xuebo Liu, Min Zhang, and Dacheng Tao. Parameter- efficient and suuent--frenddy knowledge distillation. IEEE Transactions on Multimedia, 26:4230- 4241, 2023.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015.
Patrick Schäfer and Ulf Leser. Teaser: early and accurate time series classification. Data mining and knowledge discovery, (4(5):1336-1366) 2020.
Majid Sepahvand and Fardin cation from 12-lead ecg signals to single-lead ecg with minimal loss of accuracy through teacher- student knowledge distillation. Information Sciences, 593:64-77, 2022.
A novel method for reducing arrhythmia classifi-
Ikaro Silva, George Moody, Roger Mark, and Leo Anthony Celi. Predicting mortality of ieu patients: The physionet/compputig in cardiology challenge 2012. PhysioNet/CinC Challenge v1.0.0, Jan- uary 2012. Version 1.0.0, January 20, 2012.
Wonchul Son, Jaemin Na. Junyong Choi, and Wonjun Hwang. Densely guided knowledge distilla- tion using multiple teacher assistants. In Proceedings of the IEEE/CVP International Conference Computer Vision, pp. 9395-9404, 2021.
12

---

Published as a conference paper at ICLR 2026
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.134555 2020b.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? NeurIPS, 34:6906-699 6919, 2021.
Zhiguang Wang. Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In I/CNN, pp. 1578-1558 IEEE, 2017.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1285-1294, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv:/6/2.0300 16/2.03928, 2016.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4320-4328, 2018.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, pp. 11953-11962, 2022.
Kaixiang Zheng and En-Hui Yang. Knowledge distillation based on transformed teacher matching. arXiv preprint 2rXiv:2402.11144 2024.
Yi Zhu, Ye Wang, Jianyuan Mu, Yun Li, Jipeng Qiang, Yunhao Yuan, and Xindong Wu. Short text classification with soft knowledgeable prompt-tuning. Expert Systems with Applications, 246: 123248, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 RELATED WORK
Knowledge Distillation. KD, introduced by Buciluu et al. (2006); Hinton et al. (2015), demon- strates that smaller models can achieve comparable or superior performance through knowledge transfer from larger models. This process involves matching the teacher's 's softened logits with those of the student, adjusted by a temperature hlper-parameter to amplify the contribution of negative logits. Incorporating intermediate feature representations alongside final-layer logits has further improved performance, establishing state-of-the-arr results (oomeo et al. (2015); ZZaoruyko Ko- modakis (016)) Ahn et al. (2019); Park et al. (2019).
However, recent works observe that KD often fails to meet its conventional promise due to several concerns: 1) Distillation may not transfer teacher knowledge effectively because of the capacity or architectural gap between teacher and student. To address this, recent studies have proposed inter- mediate
models Mirzadeh et al. (2020); [Son et al. (2021) and student-
teacher training (Park et af., 2021 Rao et al.] 2023] Cho & Hariharan 2019). 2) Knowledge from single teacher is often not diverse enough, it reflects only one perspective. To address this,
isting works promote diversity through teacher ensembles (Allen-Zhr & Li 2020 You et al. 2017) or mutual supervision among student ensembles (Zhang et al.) |018|] Furlanello et al. 2018). More
recently. Hossain et al. (2025) generate multiple augmented teacher views from a single model by perturbing features with random noise, thereby increasing knowledge diversity while avoiding the cost of retraining multiple models. 3) Knowledge is not always faithful: recent works observe that KD often transfers limited knowledge, leading students to learn predictive distributions very differ- ent from their teachers, which hinders their safe substitution (Stanton et al. 2021] Lamb et al.) 2023))
To mitigate this, recent studies have proposed transferring properties beyond direct logits or features, which has been shown to improve student fidelity (Parchami-Arace When distilling knowledge from a teacher trained on full-length data to student operating on par- tial data, all of these concerns further exacerbated by the additional training-distiine distillation data mismatch (Stanton et al. 2021
al. 2024 Lamb al. 2023),
Partial Time-Series Classification There is a related but distinct line of research called early time-series classification (eTSC) (Mori et al. 2017 Schäfee & Lesen 2020), which aims predict as early as possible without observing the full sequence. The model processes growing prefix and decides at each step whether to predict or wait, trading off earliness and accuracy. In contrast, clas- sification with partial time series assumes only fixed prefix is available by constraint (e.g.. latency, cost, or sensor dropout). Unlike eTTC, there is no option to defer prediction, and the model must classify directly from incomplete and ambiguous data. This work investigates whether model op- crating on partial time series can benefit from the generalization of a model trained on full-length sequences, which may have learned robust representations across time. While none of the existing works specifically address prefix- based partialness, a few approaches mitigate the generalization gap from channel-wil -wiie partialness by distilling multi-leat ECG classifiers single-Icad models (Sepah- vand Modali-Mohammddi 2022 Chauhan et al. 2022). However, these application-speeiiee meth- ods rely direct feature- or logit-level KD) and overlook the dritiingdiitillatioo data mismatch.
This work address this problem in broad and innovative manner by modeling teacher-student feature relations as degraded-to-cles counterparts and leveraging a generative prior recover long-range temporal discriminative cues.
A.2 EXPERIMENTAL SETUP
Teacher and Student Models In our experiments, we primarily use Long Short-Term Memory (LSTM) network (Hochreiter 1997) (built upon recurrent blocks), ResNet (Wang et al. 2017) 7 (a
network primarily composed of convolutional layers), and an InceptionTime network (Ismail Fawaz et al. 2020), which is among the current oftt--of-the-an for TSC. For experiments involving model compression, construct smaller variants of LSTM under different compression levels by varying the number of layers and output dimensions. The total number of parameters, model sizes, and network configurations for all constructed models are summarized in Table
14

---

Published as a conference paper at ICLR 2026
Table 9: Configuration of networks used for student and teacher models. The output dimension indicates the hidden size for the LSTM and the output dimension of the first convolutional layer for Time Inception Time (Tsmail Fawaz et al. 2020) and ResNet (Wang al. 2017).
Network
Num. Layers
Output Dim.
Total Param.
Model Size (MB)
Inception5--22
55
32
978440
0,9361
Resnet32-64
32
2016008
1,9315
LSTMS-100
812008
0.7744
LSTM2-32
2
32
51976
0.0496
0.0014
Datasets. We conducted experiments using 12 univariate time-series datasets from the UCR- 2015 archive (Dau et al., 2019) and 12 multivariate datasets from the UEA archive (Bagnall et al, 2018). Details of the selected datasets are provided in Table TO and Table [1] respectively. All series were standardized to length 100 via linear interpolation, z-normalized, and evaluated with the original train/test split with 20% validation.
Table 10: Summary of univariate UCR benchmark datasets used in our experiments.
Dataset
Type
Train
Test
Variables (M)
Length (I.)
Categories (C)
CBF
Simulated
30
900
128
3
Coffee
Spectro
28
28
286
2
ECG200
BCG
100
ECGFiveDays
FCG
GunPaint
Motion
50
150
2
Face All
Image
560
1690
131
14
Pewer
Sensor
67
1029
24
2
ECG
1800
1965
750
42
StarLiggtCurres
Scnsor
1000
8236
1024
SyntheticCottro
Simulatedd
Trace
Sensor
100
100
275
TwoLeG
ECG
11399
Table 11: Summary multivariate UEA benchmark datasets used in our experiments.
Dataset
Train
Tost
Variables (M)
Length (L)
Categorics (C)
Articulary WordRocongiiion
275
144
RasicMations
40
6
100
Cricket
108
12
6
1197
12
ERing
30
270
4
65
Japanese Vowels
270
370
12
29
9
Libras
180
NATOPS
PenDigits
7494
3498
PEMS SF
267
173
963
144
RacketSports
151
152
30
Sell SellRegalatiomCCP)
268
293
896
UWWaeGestureL.bbaay
120
320
315
Implementation Details. We select the best teacher model from five random initializations based on the validation area under the precision-recall curve (AUC-PRC) Wang al. (2017). All students involving KD are trained using a combination of the task loss and the distillation loss:
Liaain (0) >Task Crack (0) AKD (k),
where Arask and ARD determine the contributions of the classification loss Crask (cross-entropy) and the distillation loss LKD. respectively. For all experiments, ATask fixed at 1, while AKD is optimized via grid search over (0.1, 10). Models are implemented in PyTorch Parcheeaszke et al. (2019) and trained with the Adam optimizer using a batch size of 64 for a maximum of 600 epochs, with the best weights selected based on validation loss. For GDPD students, the warm-up epoch is set to Ewarm = 300, 350. nearly half of the total epochs. A learning rate decay of 0.5 is applied at epochs 25, 30, and 35, with initial learning rates to 0.01 for the LSTM2 100 and LSTM2. 32 models, and 0.1 for the other models. All student results are reported as the average over five runs with different random initializations.
15

---

Published as a conference paper at ICLR 2026
Implementation of GDPD. We adopt lightweight DDIM implementation together with the noise fusing block proposed by Huang et al. (2023) for diffusion prior training, using a total 1000
diffusion steps. All the hyperparameters used to implement GDPD are listed in Table T2
Table 12: Default hyperparameters used for implementing GDPD.
Parameter
Value
Diffusion steps (T)
1000
Number of NFEs (sampling steps)
5
Knowledge distillation weight (AKD)
Best among (0.1. I, 103
weight (Atwsl))
Number posterior sumples (J)
Total training epochs
600
Warm-up epochs (warm))
Bost among (300, 350)
Batch size
64
Optimizer
Adam
Evaluation Metries. Model performance was primarily evaluated using area under the receiver operating characteristic curve (AUC-ROC), average AUC-PRC, and accuracy on the test set. We adopt metric from Stanton et al. (02T)) to measure model fidelity: the average agreement between the student's s and teacher's top-1 predictions:
Average Top-1 Agreement 1 N
Σ IF (vr, Va.s)
win/draw/loss calculation was employed, where model 'wins' on a dataset, achieves the highest AUC-PRC. We prioritized AUC-PRC other metries due its robustness class im- balance.
The reported metrics in Table and Table are:
Avg. AUC-PRC: The average AUC-PRC across all datasets.
Avg. Test-Fidelity: The average teacher-student agreement across all datasets.
Avg. Rank: The average ranking of a method compared all baselines (lower better).
Num. Top-1: The number of datasets where the method achieves the highest performance (AUC-PRC) among all baselines.
Num. Top-3: The number of datasets where the method ranks within the top three in performance.
The number of datasets where GDPD achieves equal or better performance compared to all baselines.
Wins/Draws:
Losses: The number of datasets where GDPD underperforms compared to baselines.
A.2.1 PREdICTING IN IN-HOSPITAL MORTALITY ON PHISONET DATA
The PhysioNet Silva et al. (012)) dataset contains medical records collected during the first 48 hours after patients admitted to intensive care unit. A total of 37 variables more times for each patient, along with labels indicating length of stay (days), survival (days), and in-hospital death. Omitting categorical variables, we use time series variables:
observed
DiasABP Invasive diastolic arterial blood pressure (mmHg)
FiO2: Fractional inspired oxygen (0-1)
HR: Heart rate (bpm)
MAP: Invasive mean arterial blood pressure (mmHg)
NIMAP: Non-invasive mean arterial blood pressure (mmHg)
SaO2: Oxygen saturation in hemoglobin (%)
RespRate: Respiration rate (bpm)
NISysA BP: Non-invasive systolic arterial blood pressure (mmHg)
16

---

Published as a conference paper at ICLR 2026
SysABP: Invasive systolic arterial blood pressure (mmHg)
Glucose: Serum glucose (mg/dL)
Temp: Temperature (*C)
Therefore each sample is time series with M = 11 variables and L 48 timesteps. We impute missing values using forward/backward filling, with train-set feature means for entirely missing channels.
Downstream Task. In addition to the in-hospital mortality classification task (label = in-hospital death). we define downstream task from the multi-label annotations: survival 200 days prediction (label = I if survival == -1 or survival 2 2100; else 0), to evaluate performance under cross-task distillation.
A.3 ADDITIONAL. RESULTS
A.3.1 FURTHER ABLATION STUDIES
Ablation on Loss Terms. To further verify that the diversity of GDPD's teacher signals drives
the gain, we modify LGDPD (Equation (4 as (xxy)>> Boog oong ||²) where posterior recon- struction samples are constrained to match the corresponding direct teacher feature zoong of the same training sample. This reduces knowledge diversity, forcing exact reconstruction, with a reduced re- sult of 96.21 that confirms the limitation.
We present ablation for Distillation ratio (AKD) Table 13
Table 13: Ablation of distillation ratio: GDPD performance measured in terms of AUC-PRC.
Distillation ratio (AKD)
0.01
0.1
AUC-PRC
86.41
Ablation Diffusion Controls. We ablate the number of forward diffusion steps T in Cdiffusion (Ø) with {100, 500, 800, 1000), obtaining 93.29, 95.33, 92.49, and 97.26, and set T 1000 in experiments. Following Huang al. (2023), use DDIM (Song et al. 2020a), which accelerates denoising compared to early diffusion models and allows sampling with far fewer score function evaluations (NFEs) < T. We ablate the NFEs in the LGDPD with {0, obtaining AUC-PRC values of 83.90, 95.98, 97.31, 11, 94.96, 97.26, and 97. 97.90, respectively. Even with single step, GDPD achieves substantial gain over the Base model (83.90), and only few steps are sufficient to reach near-optimal performance; hence, we set NFEs to in our experiments.
10} steps,
A.3.2 COMPUTATIONAL COST ANALYSIS OF GDPD
We evaluate the computational overhead of GDPD by quantifying the training cost on the Starl.ightCurves dataset for LSTM3-100 LSTM3-100 under the partialness level 0.57.
Table 14: Training, memory, and inference cost comparison across distillation methods for LSTM3- 100 + LSTM3-100.
Student Params (M)
Additional Params (M)
Epoch Time (s)
Step Time (ms)
GPU Inference
Method
Train (h)
(GB)
(ms)
Base
0.20
0
0.10
0.61
46.71
0.17
0.02
KD
0.20
0.10
46.88
0.17
0.02
Fits
0.20
0.01
0.10
0.62
48.
0.22
0.02
VID
0.20
0,03
0.11
0.68
52,00
0.25
0.02
DKD
0.20
0
0.10
0.62
47.98
0.17
0.02
Attention
0.20
0
0.11
0.64
49.00
0.17
0.02
RKD
0.20
0.13
0.80
61.61
1,03
0.02
GDPD
0.20
0.21
0.14
0.85
65.14
0.25
0.02
warm-upp
0.21
0.07
62.46
0.24
GDPD guided phuse
0.20
0.21
0.08
0.89
66.21
0.25
0.02
17

---

Published as a conference paper at ICLR 2026
Table 14
summarizes the results obtained using a single RTX-A6000/3000 claas GPU. Below, discuss the computational cost of each phase of GDPD.
Teacher Training. Training the teacher is identical to any standard KD pipeline and introduces additional cost GDPD.
Diffusion-prior Training. Trained in feature space, where the dimensionality far lower than in the input domain, GDPD's diffusion training becomes significantly cheaper computationally. The diffusion prior is lightweight (only 0.206M trainable parameters, including the noise adapter) and is trained during the student's 's warm-up phase. The warm-up stage costs 0.81 s/epoch, compared to 0.61 s/epoch for the Base student. With a warm-up duration 300 epochs, this adds only ~ minute of extra training in the evaluated setting. In practice, diffusion-prior training is comparable to train- ing one additional Base classifier and remains far cheaper than ensemble-teacher distillation, while still providing the benefit of knowledge diversity.
Diffusson-guiieed Training. During GDPD's main distillation phase, posterior sampling uses only NFEs, and all sampling is performed feature space, making each reverse-diffusion step ex- tremely cheap. The diffusion-guided stage costs 0.89 s/epoch, compared 0.61 s/epoch for the Base student, an overhead of only ~0.28 per epoch. Over 300 epochs, this amounts to approximately 1.2 minutes of additional training time the evaluated dataset.
Overall Training Cost. In the evaluated setting, training the Base and Base-KD requires 0.10 h, whereas GDPD (warm-up up guided phase) requires 0.14 h, adding only ~-24 minutes of extra train- ing. The overall training cost GDPD is comparable to widely adopted ceature-distillaan such as RKD and VID, while delivering substantially higher performance (Table 2). GDPD also maintains a low memory footprint of 0.24-0.25 GB, similar Fits and VID, only slightly above basits-based KD (0.17 GB), and far below memory memory-intensive methods such as RKD (1.03 GB). De- spite incorporating a diffusion prior, GDPD introduces minimal memory overhead. This modest training overhead is well justified by the consistent and significant performance improvements over conventional KD baselines and the Base classifier (Table 2p.
methods
Inference Cost. Inference cost is unchanged: the GDPD achieves 0.02 ms/sample, identical to the Base. All additional computation occurs only during training, while the deployed model remains as efficient as the Base classifier.
Table 15: Effect of inference diffusion steps on training cost, memory and inference time.
Inference Steps
Total Train (h)
Epoch (s)
Step Time (ms)
GPU Mem
Inference
AUC-PRC
(GB)
(ms)
(%)
0.61
0.13
0.77
59.03
0.24
0.02
95.98
0.24
0.13
0.81
62.45
0.25
0.02
94,96
0.14
0.85
65.14
0.25
97.26
10
0.16
0.95
72.79
0.25
0.02
97.90
Training Cost vs. Inference Steps. Table reports how the computational cost varies with the number of inference steps (equivalently, the NFEs in our implementation). Increasing inference steps (our default), epoch time
steps slightly raises per-epoch time and memory. From creases from 0.77 to 0.85 and memory from 0.24 GB to 0.25 GB. Even at 10 steps, memory re- mains at 0.25 GB and total training stays below 0.16 h. We adopt steps the best port-performance trade-off.
Training Cost vs. Posterior Samples. Table 16 summarizes how training cost scales with the number of posterior samples. The cost grows roughly linearly, as each sample requires an additional draw. Epoch time rises from 0.85 (I sample) to 1.44 (5 samples), and total training from 0.14 to 0.24 h, while GPU memory remains fixed at 0.25 GB. We adopt a single sample as the best cost-performance trade-off.
18

---

Published as a conference paper at ICLR 2026
Table 16: Effect of the number of posterior samples on training cost, memory and inference time.
Posterior Samples ())
Total Train (h)
Epoch Time (8)
Step Time (ms)
GPU Mem
Inference
AUC-PRC
(GB)
(%)
0.10
0.61
46.71
0.17
83.90
0.14
0.85
65.14
0.25
0.02
97.26
0.17
0.99
76.05
0.25
97.34
0.19
1.11
85.99
0.25
97.13
0.22
1.30
100.17
0.25
0.02
97.87
0.24
1.44
0.25
Both ablations show that GDPD's diffusion controls offer flexible cost-performance trade-off with minimal memory overhead and no effect on inference speed.
A.3.3 EFFECT OF MAACHER-STUDRRE LAYER SELECTION.
To assess the impact of layer choice, perform layer-wise ablation by applying GDPD across all teacher-student layer combinations in the LSTM3-100 - LSTM3-100 under the partialness level 0.5L (Table 17 We additionally evaluate "1+2+3" variant in which the diffusion prior trained on features from all three teacher layers. Across these ten configurations, we compute the average rank and mean AUC-PRC over eight UCR datasets. The final-layer distillation (3-3) reports the best performance, while distillation involving shallow layers (1 or 2) performs slightly lower yet remains close. The multi-layer prior (I+2+3) performs comparably but better than final-layer distillation, suggesting that deep teacher features alone are the most effective.
We further report layer-wise averages and rank calculations for both teacher and student, and the results indicate that the final layer is the strongest choice for both. Accordingly, all our experiments use the layer features.
Table 17: Cross-layer GDPD performance for LSTM3-100 LSTM3-100 on eight UCR datasets. Each cell reports Avg. Rank / Avg. AUC-PRC 1. Layer-wise averages and rank calculations are also provided for each layer. "1+2+3" denotes diffusion prior trained on feature representations from all three teacher layers.
Student Layer 2
Teacher Layer Avg.
Teacher Layer
8.8% 81.35
6.63 / 82.02
5.50 / 82.
2.42 81.83
6.00/
5,25 /2.84
/
1,83 82.72
5.38 82.25
4.13 /82.92
3.89/ 84.03
1.75 83.07
1+2+3
5.38 / 81.56
Student Layer Avg.
2.38/ 80.88
1.88 /8.59
1.75 83.15
A.3.4 GDPD is ROBUST ACROSS DIFFERENT NETWORK ARCHITECTURES.
In Table g we report results for two distillation settings: 1) similar teacher-studens architec- tures (Inception55-32 - Inception55-32, Resnet32-64 > Resnet32-64) and 2) cross architectures (Inception55-32 > Resnet32-64, Resnet32-64 - Inception55-32). Across both settings, GDPD consistently achieves the highest performance gains, measured by average AUC-PRC, lowest aver- age rank, and the greatest number of top-1 wins, demonsirating its effectiveness for both similar- and cross- wross-acchitectrre distillation.
Table 18: Summary of similar- and cross-architecture distillation results on 12 UCR datasets. Each cell reports Avg. AUC-PRC/ Avg. Rank Num. Top-I wins.
Inception - Inception
Inception -> ResNet
ResNet - ResNet
ResNet -> Inception
Base-KD
26/ 2266/3.5
89.37
15.08 2.42
Fits
88.98 2.558 /
82.47
3.33/0
2,42 89.26
77.06 2.75
GDPD
92.29 1.17 11
87.79 / 1.17 10
90.85 1.58/
$1.05/ 1.08 11
19

---

Published as a conference paper at ICLR 2026
A.3.5 GDPD DEGRADES GRACEFULLY UNDER WEAK TEACHER SUPERVISION.
To assess robustness when the teacher provides poorly structured feature spaces, we construct sequence of increasingly degraded teachers on the StarLightCurves dataset by reducing the amount of training data and injecting label noise. We train four weak teachers: (WT-1) 0% training data reduction and 0% label noise, (WT-2) 25% training data reduction and 0% label noise, (WT-3) 25% training data reduction and 10% label noise, and (WT-4) 50% training data reduction and 25% label noise. Figure 3 reports how each baseline responds to these progressively degraded supervision for Inception55-23 - Inception55-32 under the partialness level e 0.5 Across the four weak-teacher settings, all methods degrade supervision quality decreases, but GDPD consistently maintains the highest performance and shows the slowest rate of decline.
Average AUC-PRC
0.98
0.97
0.96
0.95
Base-KD
RR:S
0.94
GOPD
WT-1
W72
WT:3
WT4
Weak-Teacher Setting
Figure 3: Performance comparison under four increasingly degraded teacher configurations (WT-1
to WT-4),, showing how each method responds to weak-teacher supervision.
A.3.6 GDPD IS ROBUST ACROSS DIFFERENT EARLINESS LEVELS.
repoort results for earliness levels 0.57. and L,, which are complementary to the main results presented in Table
Table 19: Summary of performance at earliness levels 0.5L. and L. (full length) 12 UCR datasets. Best values are in bold. Rows marked with 4. indicate lower-is-better GDPD achieves the highest AUC-PRC, the lowest rank, and the largest number of Top-1 wins.
Earliness:0.5L.
Earliness=L (full)
KD
GDPD
Base
Avg. AUC.PRC
71.60
78.14
77.28
84.64
Avg. AUC.PRC
87.32
89.29
88.73
91,21
Avg.
Avg. 3.33
2.75
Num. Top-1
10
Num. Top-1
0
5
Wins/Draws
12
10
12
Wins/Daws
11
8
10
Losses
Losses L 1
2
A.3.7 FULL RESULTS FOR SUMMARIES REPORTED IN THE MAIN TEXT
Robustness to Different Earliness Levels. Table 20 provides the complete results of evaluating GDPD under different earliness levels on 12 UCR datasets.
Comparison with Different Distillation Objectives. The full results comparing GDPD with ex- isting KD variants, evaluated at carliness 0.5L, are provided Table 21
Time-wise and Channel-wis Partialness for Multivariate Datasets. The detailed results for time-wise partialness (evaluated at e = 0.5L) and time+channel-wise partialness (evaluated at e
0.5L, 0.5.M) in the case of LSTM3-100 - LSTM3-100 distillation are provided in Table 22 The corresponding full results for Inception55-32 + Inception55-32 distillation under the same earliness settings are reported in Table 23
GDPD in Model Compression.
The complete results for two compression targets under two ear-
liness levels presented in Table 24
GDPD in Self-distillatio.. The detailed self-distillation results (with similar model capacity and without earliness) are summarized in Table 25
20

---

Published as a conference paper at ICLR 2026
Table 20: Detailed UCR results at earliness e E {0.2L, 0. 0.5L, 0.84, L for LSTM LSTM. Best per row in bold.
(a) c = 0.2L
0.4L
(c) c 0.5L
K.D
KD
GDPD
Base
GDPD
CBP
64.08
69.24
68.44
68.96
93.87
89.
88.11
94.35
90.29
97.39
95.04
95.44 44.44
Coffee
90.76
98.88
91.07
99.34
64.95
88.28
90.13
71.26
83.28
82.98
85.28
ECG200
66.U8
76.88
79.81
80.19
68.26
78.93
70.34
80.15
79.31
83.61
79.30
82.95
Days
56.78
57.19
56.32
57.70
55.40
56.38
56.30
55.82
48.33
61.44
66.72
73.85
Gun
77.38
75.18
84,80
74.87
68.31
84.59
74.21
79.22
78.90
FaceAll
31,90
48.29
41.88
52.84
55.67
69.90
72.44
73.78
53.09
71.48
70.09
76,08
Pemeenand
67.45
69.34
66.36
73.30
79.74
79.43
79.93
81.84
82.53
87.26
82.53
91.71
NonInvasiveFatall
51.92
48,
50.16
67.34
72.23
72.20
72.12
StarLightCurves
77.35
93.40
86,80
94.83
93.68
97.13
95.
97.43
83.90
95.20
81.87
97.26
synihetic.conirol
81.02
87.57
83,20
87.52
82.32
81.66
86. 59
90.87
63.82
61.46
63.08
82.94
Tince
54,94
71.78
74.05
72.52
TwoLeadECG
59.22
54.56
63.88
64.20
69.10
79.47
64.93
85.17
77.19
82.07
82.45
88.44
Avg. AUC-PRC
63.64
69.23
67.47
73,83
81.70
78.03
75.36
81.70
71,60
78.14
28
84.64
Avg. Rank
3.50
2,42
2.92
1.17
3.58
2,50
2.67
1,25
3.50
2.33
2.92
1.17
Num. Top-1
Wins/Draws
12
10
12
Losses
0
2
0
(d) e = 0.6L
(c) = 0.8L
(f)
KD
KD)
CBF
82.61
90.68
82.76
89.88
93.37
93.44
95.46
95.26
95.08
99.50
99.26
Coffee
74.35
100
90.64
100
98.23
99.51
98.34
100..0
99.67
99.67
100.00
100.00
ECG200
76.09
80.78
83.55
81.26
83.32
77.94
82.69
85.01
78.75
$0.36
77.30
79.18
BCyy
78.08
86.82
81.46
89.90
59.33
74.01
66.29
79.09
92.62
90.52
85.44
95.37
Gun_Point
69.81
74.10
77.99
76.36
92.29
92.71
92.82
93.20
96.11
92.20
96.54
94.11
FaceAll
69.21
69.68
70.64
75.61
49.31
73.71
73.33
78.59
78.83
83.53
82.84
85.94
DemanDannnd
93.04
92.46
93.02
95.32
92.90
96.37
17
98.07
98.68
99.21
98.55
99.21
NonlavasiweFatall
68.91
70.97
73.53
74.25
48.75
63,70
69.58
72.81
84.49
88.38
85.44
88.35
StarLightCurves
81.15
94.02
95.23
96.97
83,34
96.45
80.79
96.46
96.12
97.19
96.92
97.45
synthetic contml
99.69
99
99.86
99.87
98.09
99.11
99.05
99.33
99.08
99.73
99.59
99.66
Trace
50.18
67.41
55.64
65.34
48.52
60.82
54.29
75.38
60.81
72.30
77.77
77.26
TWoLeadECC
78.32
77.79
69.51
87.27
86.05
89.56
86.04
95.09
67.62
68.84
72.95
78.71
Avg. AUC- PRC
76.79
83.70
81.15
86.00
77.79
84.78
82.74
89.02
87.32
89,
88.73
91.21
Avg. Rank
3.58
2,58
2.42
1.33
3.67
2.42
2,83
1,08
3,33
2,08
2.75
1,58
Top-1
Wins/Drws
11
10
Losses
2
Table 21: Performance across 12 UCR datasets for LSTM LSTM. Best per row bold; second best is underlined. (For Avg. Rank, lower is better.)
Base-KD
Attention
CBF
90.29
97.39
95.04
84.18
90.90
80,20
46.60
86.67
80.12
75.95
79.70
86.41
95.44
Coffee
71.26
83.28
82.98
86.33
84.41
82.59
72.51
83.18
96.47
85.67
84.53
85.02
85.28
ECG200
79.31
83.61
79.30
81.01
79.80
83.28
81.87
79.13
$1.08
80.22
81.47
82.94
82.95
ECGFiveDays
48.33
61.44
66.72
66.97
82.85
71.95
Gun, Point
74.21
79.22
78.90
76.42
63.98
65.99
$6.31
84.58
70.85
64.36
68.55
74.09
93.55
FaceAll
53.09
71.48
70.09
72.73
62.24
71.08
13.97
70.63
68.93
73.92
74.90
68.05
76.08
82.53
87.26
82.53
88.94
84.03
84.77
90.61
93.34
94.77
80.13
88.07
88.87
91.71
71.68
72.12
71.89
55.39
51.22
61.79
16.61
59. 14
57. 57.48
63.45
65.73
62.30
73.32
Starl LightCurves
83.90
95.20
81.87
89.81
89.24
87.65
72.41
92.47
93.44
91.87
87.57
97.26
synthetic.om control
63.82
61.46
63.08
74.58
65.13
67.73
48.87
98.41
96.76
78.07
63.13
70.39
82.94
Trace
63.59
TwaLGG
77.19
82.07
82.45
81.93
85.37
84.37
$7.95
63.97
76.53
80. 10
79.05
82.05
88.44
AUC-PRC
71.60
78.14
77.28
77.19
74.29
75.47
55.32
79.89
79.26
76.86
76.54
78.0S
84.64
Test-Fidelity
66.75
73.13
69.39
71.32
69.36
20
$2.59
74.34
71.22
71.05
69.53
70.24
77.58
Avg. Rank
5.83
7.58
6.33
11.42
Numn. Top
0
2
0
0
0
0
0
2
0
D
Num. Top-3
Wins/Draws
12
10
12
11
11
11
12
10
11
11
Losses
21

---

Published as a conference paper at ICLR 2026
Table 22: Detailed results for LSTM LSTM on multivariate datasets under Time-wise partialness
(e 0.5L) and T'me+CChane partialness (e = 0.5L, m = 0.5.M.. Best result per row in bold.
(a) Time-wise partialness
(b) Thme+Channel partialness
K.D
GDPD
Dalaset
Buse
Base
Fits
GDPD
ArticularyWore WordResog.
87.55
90.26
89.66
89.88
Articulary eondRe
60.41
64.97
69.41
66.57
BasicMotions
93.54
91.89
92.67
97.33
BasicMotions
18
94.85
94.31
97.44
Cricket
83.39
85.27
75.77
87.95
Cricket
66.27
67.46
70.12
71.69
ERing
64.25
62.69
66.74
67.55
ERing
46.52
52.98
46.42
55.23
JapaneseVowel
94.39
96.49
95.98
96.99
Japanese Vowels
92.26
91.70
92.53
93.04
Libras
49.49
65.91
64.70
66.94
Libras
35.16
37.35
38.32
41.27
NATOPS
80.15 14
$1.08
$2.31
84,93
NATOPS
86.86
85.27
86.86
87.64
PEMS-SF
71,70
76.38
73.79
75.50
PEMS-SF
66.98
73.57
PenDigits
95.95
96.14
95. 99
95.14
PenDigits
75.46
75.61
75.55
70.91
SporttSports
82.36
85,20
87.82
86.64
RackeiSSprtts
74.32
74.45
77.35
79.18
SelFRegulationCPP
86.88
SelfRegilationCCPI
83.97
U UWaxeGesuueLiibrry
65.82
68.98
72.08
70.73
UWaveGesureLibbary
$1.37
51.12
56.29
53.56
Avg. AUC-PRC
79.41
81,96
81.95
83.87
Avg. AUC-PRC
68.75
71.11
71.69
73.60
Avg. Rank
3.50
2.50
2.42
1.58
Avg. Rank
3,42
2.75
2.33
1.42
Num. Top-1
Num. Top-1
0
Wins/Dras
Wins/Drass
11
11
Losses
I Losses
Table 23: Detailed results for Inception + Inception multivariate datasets under Time-wist partialness (e = 0.5L) and Time+Channel partialness (c 0.5L, m = 0.5.M). Best result per row in bold.
(a) Time-wise partialness
(b) Time+Channel partialness
Dataset
Buse
Base KD
Fits
GDPD
Dalaset
Base
Base H.Dee-KDD
Fits
GDPD
Articulary ArticularyWordkeeee WordRecog.
92.23
96.32
84.62
96,92
Articulary WordRecog.
69.44
76.01
68.16
79,56
BasicMotions
99.78
100
100
100
BasicMotins
76.78
100
140
100
Cricket
97.75
98.96
97.03
98.
Cricket
84.03
87.61
81.90 11.96
86.66
ERing
76.65
79.26
70.15
80.21
ERing
71.87
71.39
70.40
72.42
Japanese Vowels
98.43
99.04
96.26
99.19
Japunese Nowels
94.95
96.40
93.95
96.65
L.bras
68.78
81.28
81.20
82.70
I Jihras
39.60
39.31
40.96
45.05
NATOPS
86.94
84.78
82.01
89.65
NATOPS
87.48
88.51
87.20
90.35
PEMS SF
$7.84
65.07
65.45
77.70
PEMS SF
63.80
64. 70
61.31
75.71
PenDigits
90.68
95.37
95.39
94.69
PenDigits
75.05
75.49
74.73
71.90
RacketSports
RackeiSports
SelfRegulationSCPI
93.13
96.96
95.71
96.91
SelfRegulationSCPI
95.32
96.45
95.70
96.85
UWaveGesturLLibrary
81,55
81.66
71.49
84.27
UWaveGestureLibrary
58.43
56.87
53.91
64,05
Avg. AUC-PRC
86, 22
88,
85 40
90.82
Avg. AUC-PRC
76.58
97.92
75.34
80.24
Avg.
1.33
Avg. Rank
2.92
2,08
3.42
Num. Top-1
2
9
Num. Top
D
3
Wins/Draws
12
11
Wins/Draws
11
10
II
Losses
2
A.4 DISCUSSION
A.4.1 CONDITIONING STRATEGY IN GDPD
GDPD operates in an inverse diffusion setting, where the guidance is provided by the measurement or partial observation. The conditioning mechanism in inverse diffusion is fundamentally different from classifier- or classifier-free guidance used in guided generation. Below, we justify our use of simple initialization-basee conditioning GDPD by outlining why other common conditioning strategies in the inverse-diffusion are unsuitable for our setting.
Conditional Denoisers. A common approach is to feed the measurement (Zattrt into the denoiser during both training and sampling, allowing the denoiser to learn how to influence the output Zlong- This resembles classifier-free guidance in guided generation. However, this incompatible with GDPD since conditioning signals are non-stationary. During training, the student keeps learning, and its Zshort features change continuously. quickly becomes invalid, because it no longer reflects the student's updated representation. Keeping aligned would require retraining the denoiser at every student update. For this reason, GDPD cannot rely conditional denoisers.
conditional denoiser trained on earlier student states
Projection or Data-consistency Updates. These methods apply an unconditional denoiser and then project the output onto the set of measurements to enforce consistency with the observation,
22

---

Published as a conference paper at ICLR 2026
Table 24: Complete model compression results under two earliness levels 0.5L, L) and two
compression targets (LSTM3-100-LLSTT -
Best per row is in bold.
LSTMB-100 LSTMI-8
LSTM3-100
LSTM2-32
Earliness
Dataset
Base-KD
GDPD
Base-D
Fits
GDPD
CBF
58.95
60.45
58.95
70.17
90.40
90.58
92.61
Coffee
71.12
69.93
71.45
78.00
80.29
86.16
77.00 76
84
ECG200
83.51
84,94
84.76
84,06
81.54
79.39
79.79
82.28
ECyFiveDays
63.31
73.51
06.44
77.41
74.47
63.53
74.23
75.10
Gun Point
60.93
66.56
61.06
75.27
62.93
78.14
76.75
80.19
0.5.L
FaceAll
26.84
28.81
26.84
47.19
53.96
62.95
68.04
74.60
Delaeon
85,00
86.41
85.00
93.35
92,10
92.76
90.27
92.37
Non
18.20
33.03
50.63
36.91
88
76.27
Starl ightCurves
75.63
77.66
76.11
80.33
77.18
83.13
93.92
94,22
synthetic.cumird
93.73
Trace
49.02
48.81
49.02
73.08
67.17
69.78
62.10
73.23
TwoLeadECG
56.67
70.90
56.67
86.60
83.50
87.10
92.21
68
Avg. AUC-PRC
58.58
61.97
59.84
72.76
72.98
74.92
78.26
83.67
Arg. Rank
3.42
2.33
2.67
1.17
3.33
2.58
2.75
1.33
Num. Top-1
0
11
9
Wins/Draws
12
12
Losses
0
CBF
65.44
61.32
64.41
65.89
77.17
79.80
78.50
86.78
Coffee
87.08
87.79
97.38
99.51
99.67
100
RCG200
70.71
70.67
70.92
74.09
72.68
73.33
76.15
801.5
Bays
75.76
93.22
Gun Point
94.75
96.32
95.63
96.77
91.40
98.47
94.95
98.00
FaceAll
42.91
45.82
47.31
76.09
82.59
InwyPowerDeeman
98.46
99.03
98.46
99
98. 18
98.67
9R. 24
98.68
NonInvasivefataal
43.81
44.53
43.45
44.78
86.07
88.88
86.47
88,
SturLightCurvess
93.20
92.62
93.65
93,27
96.40
97.24
96.60
97.20
synthetic control
87.73
96.99
93.36
97.83
99.50
99.38
99.44
99.54
Trace
69.89
75.10
69.64
75.44
69.52
75.58
78.00
78.00
TwoLeadECG
69.25
72.99
67.62
72.03
73.28
72.02
74.03
75.25
Avg. AUC-PRC
74.92
77.62
76.49
78.83
84.34
88.11
85.93
89.84
Arg.
1.42
2.75
1.25
Num. Top-1
0
3
2
7
D
1
9
Wins/Draws
Losses
0
2
0
Table 25: Detailed self-distillation results on 12 UCR datasets under three architectures. Best per row is in bold.
LSTM + LSTM
Inception - Inception
ResNet + ResNet
Datasct
Teacher
Base-KD
Fits
GDPD
Teacher
Base-KD
Fits
GDPD
Teacher
Base-KD
Fits
GDPD
CBF
95.08
99.50
91.49
99.26
99.19
99.91
96.51
99.94
99.72
99.69
99.61
99.76
Coffee
99.67
100
99.75
100
100
100
100
ECG200
78.75
80.36
77.30
79.18
91.16
92.21
91.14
92. 16
92.19
92.91
93.37
93.09
ECGFiveDays
92.62
90.52
85.44
95.37
95,94
100
94.51
100
98.25
97.29
97.84
97.63
Gun Point
96.11
92.20
96.54
94.11
99.71
100
99.72
99.99
100
100
100
100
FaceAll
78.83
83.53
82.84
85,94
86.72
89.49
71.69
90.87
96.38
97.19
96.80
97.25
ItalPooweDDemand
99.21
98. 90
99.05
98.01
99.05
98. 98.99
98.96
FonInvasiveeaaalRGG
$4.49
$8.18
85.44
88.35
94.85
95.00
84.53
95.33
97.45
97.44
97.41
97.48
Starl SihhCuuissssevs
97.19
synthetic control
99.08
99.73
99.59
99.66
99.97
99.95
99.29
100
99.98
99.99
99.94
99.99
Trace
72.30
77.77
77.26
100
100
100
100
TaoLGG
67.62
68.84
72.93
78.71
99.55
99.92
99 99.42
99. 39
100
99.99
100
100
Avg. AUC-PRE
87.32
89. 29
88. 73
91.21
96.96
97. 80
93 03.90
97.97
98.43
98.52
98.55 55
98.58
Avg. Rank
3.33
2.08
2.75
2.83
1,50
3.92
1.25
2.33
2.25
2.33
1.25
Num. Top-1
5
7
9
5
Wins/Draws
10
12
10
Losses
2
2
typically assuming a known, fixed, and linear degradation model. In GDPD, the "degradation" from Flong to zsort is neither known, fixed, nor linear. The student features zshort evolve throughout training, making any strict projection- based constraint incompatible with their non-stationary nature.
Bayesian fikelihood-gradiee Conditioning. These methods use an unconditional denoiser and modify the score at each diffusion step with the gradient of the likelihood how likely the current sample would produce the observed measurement. Because this likelihood is intractable, inverse-diffusion methods approximate it using assumptions tied to their specific degradation setting
23

---

Published as a conference paper at ICLR 2026
(linear, nonlinear, noisy, etc.). This requires explicit knowledge of the forward degradation and introduces significant computational overhead. In GDPD, however, the degradation from Zlong to Shon is unknown, dynamic, and not tied to any fixed physical process. Because the GDPD loss already supervises the outputs and the conditioning signal itself evolves during training, we treat this mapping general and do not enforce strict measurement consistency at every diffusion step. Imposing such exact reconstruction constraints would be both unnecessary and computationally costly.
Initialization-base based Conditioning. We adopt initialization-based conditioning because it accom- modates the evolving nature of the student features (with the appropriate noise level) without re- quiring any retraining or modification of the conditioning mechanism. As the student updates, its Ehhont features directly influence the diffusion trajectory, ensuring that the conditioning always re- flects the student's 's current knowledge. Although this approach does not enforce strict measurement consistency, this is acceptable: the GDPD loss already supervises the reconstructed features, and the conditioning signal only needs to steer the generation process rather than impose exact reconstruc- tion constraints.
A.4.2 LIMITATIONS AND FUTURE WORK.
Modeling teacher knowledge through a generative prior opens up a broad space of potential distil- lation objectives to foster many desirable teacher properties. In this work, we instantiate only one such formulation, and exploring alternative distillation objectives grounded in the generative prior remains promising direction for future rescarch. The computational cost of GDPD is higher than simple logits-basce distillation methods (e.g., Base-KD, DKD), but it is comparable to established feature-based distillation objectives such as VID and RKD.
Applicability Beyond Time Series Classification. This work establishes GDPD as principled distillation framework for partial time-series classification, validated across standard benchmarks and a real-world case study, and accompanied by practical guidance on key hyperparameter con- trols. The current implementation supervises posterior reconstructions using classification loss and is therefore instantiated for classification. However, modeling teacher knowledge as generative prior provides a novel and broadly applicable perspective on knowledge distillation, leaving several promising extensions of GDPD for future work. First, GDPD may be applicable to forecasting or multimodal learning with missing modalities by substituting the classification vision with the relevant task-specifi objective. Second, the proposed framework can, in principle, be applicd to partial-inpl input classification in domains beyond time series, such as short-text classifi- cation (e.g., headlines snippets) (Zhu et al., 2024). In textual data, topic shifts and the discrete nature of token representations make short contexts more ambiguous than temporally continuous signals (Zhu al.) (0024). Thus, when extending GDPD textual domains, feasible earliness levels must be carefully validated, and potential mismatches in teacher-student vocabularies (reflected in their embedding layers) should be addressed. These considerations represent promising directions for future exploration.
posterior super-
THE USE OF LARGE LANGUAGE MODELS (LLMs)
In preparing this manuscript, we used OpenAl's ChatGPT (GPT-4) writing assistance tool. Its role was limited polishing language for improved clarity, grammar, and readability in certain sections of the paper and appendix. The model did not contribute to research ideation, method- ological design, experimental execution, data analysis, or interpretation of results. All scientific content, technical contributions, and conclusions are solely the work of the authors. We take full responsibility for the accuracy and integrity of the paper.
24


Reviews:
[{"review_id": "pBGqJUVy9j", "reviewer": "Reviewer_uuK6", "summary": "This paper studies classification from partial time-series prefixes and proposes GDPD, a distillation framework that learns a diffusion prior over teacher features and uses posterior samples—conditioned on the student’s short-context features—to supervise the student. The aim is to provide progressive, diverse, and distributional targets rather than single-point matches. Across UCR/UEA benchmarks and a PhysioNet case study, GDPD improves accuracy and teacher–student fidelity under multiple earliness levels, channel-wise partialness, compression, and self-distillation.", "questions": "- How is $z_{\\text{long-hint}}$ defined for a given example—what subset of the teacher feature manifold counts as useful hints of $z^*_{\\text{long-ideal}}$? If $z_{\\text{long-hint}}\\sim p(z_{\\text{long}})$ is just a generic draw, how does it differ from standard teacher features, and why doesn’t this construct appear later in the method/experiments? Please clarify the relationship between the posterior reconstruction $\\hat z_{\\text{long}}$ and the hints, or state that $z_{\\text{long-hint}}$ is expository only.\n- Which teacher/student layers define $z_{\\text{long}}$ and $z_{\\text{short}}$ across architectures? Have you tried cross-layer modeling (multiple blocks) or multi-scale priors?\n- How much training time does GDPD add on typical settings, and how sensitive are results to diffusion steps/samples and key hyperparameters? Any guidance on trade-offs between speed and accuracy?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work focuses on full-context → partial-context distillation and explains why direct feature/logit matching can overwhelm the student under representation mismatch.\n- Treating teacher representations as a diffusion-learned prior and supervising with distributional, progressive targets is a substantive shift from standard KD.\n- Consistent gains over Base, Fits, and other KD baselines across earliness settings; better teacher–student agreement; robustness to channel-wise partialness; and utility for compression and self-distillation.\n- Warm-up scheduling, loss-weight trade-offs between task and GDPD, and the effect of sampling multiplicity J provide insight into what drives improvements.\n- The PhysioNet ICU study suggests the approach remains effective under heterogeneous conditions (partial channels, imbalance, cross-task).", "weaknesses": "- Training adds a diffusion prior and posterior sampling within student optimization. Please report wall-clock and step-level overhead versus Fits/Logit-KD (e.g., epochs × diffusion steps, training/inference cost).\n- Performance may depend on which teacher layer is modeled and which student features are aligned. Clear guidance, cross-layer experiments, or robustness checks would help practitioners.\n- Please discuss applicability beyond time series (e.g., images or text under partial inputs) and clearly state assumptions and limitations to contextualize the results and guide follow-up work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wjpN4QKkt0", "reviewer": "Reviewer_y3wg", "summary": "This paper proposes a new method to train time series classifiers that operate on partial sequence prefixes to achieve similar generalization performance to those trained on full sequences. The authors argue that the generalization gap in this case is due to a representation gap, as the full context features from teacher models cannot be directly distilled into the partial context features from student models. To solve this, a diffusion model is trained to obtain a generative prior over full context features, which can then be used to posterior-sample completed features given the partial features from the student model. The student model is then optimized to produce partial features such that its corresponding posterior full feature samples have high classification performance.", "questions": "1. Could the authors comment on and provide some analysis of the trade-off between model performance and computational cost for the proposed method and other conventional distillation methods?\n\n2. Could the authors clarify why the noise-fusing approach is used for diffusion model conditioning? How does it compared to classifier-free/guided guidance in this setting?\n\n3. Could the authors discuss limitations of the proposed method in the paper?\n\n4. Does z_long and z_short have the same dimensionality? From the paper, it is unclear whether the posterior sampling procedure is an impainting process that fills in the missing postfix in z_short or a transformation that modifies the entirety of z_short into z_long.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces an interesting formulation for the partial time series prefix classification problem, which identifies the representation gap as a key reason why conventional distillation methods do not work as expected.\n\n2. The proposed method is novel, which leverages powerful diffusion priors to obtain posterior full features given partial features for the downstream classification task. The learning objective is effective since it directly optimizes to student model to achieve highly discriminative performance given the posterior full feature samples.\n\n3. The authors provide some reasonable justifications for why the proposed method works, which is much appreciated.\n\n4. Extensive experiments with ablation studies are performed the validate the proposed method's empirical performance. However, I cannot evaluate the experimental results, since I'm not familiar with time series classification benchmarks and baselines.", "weaknesses": "1. My main concern is the computational complexity of the proposed method in all training stages. First, in the pre-training stage, it involves pre-training (i) a teacher classifier on full sequences; (ii) a diffusion model on the full features extracted from one of the middle layers of the teacher classifier. Then, when training the student model, each training step requires sampling from the posterior distribution, which involves simulating the reverse diffusion process of the diffusion model for multiple steps. This is significantly more costly than traditional distillation methods.\n\n2. It is unclear why the noise-fusing approach is used for the conditioning mechanism for the diffusion model, given that classifier-guided/free guidance is more principled and widely used in many applications.\n\n3. Limitations of the proposed method is not discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BrSFkzmp8G", "reviewer": "Reviewer_yLkg", "summary": "The paper tackles classification from partial time-series prefixes by distilling knowledge from a teacher trained on full sequences.\nExtensive experiments across earliness settings, datasets, and architectures demonstrate GDPD’s effectiveness for full-to-partial distillation", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Generally clear and readable; figures and tables are informative; Codes are publicly available.", "weaknesses": "1. Training a diffusion model may additionally introduce significant computation and memory\n2. compared baselines (KD and FitNet) are significantly out-of-dated\n3. the paper lacks a formalization of the posterior being sampled", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVFBLkJP78", "reviewer": "Reviewer_VsYh", "summary": "The paper proposes Generative Diffusion Prior Distillation (GDPD), a novel knowledge distillation framework that treats the teacher’s full-context representation as a generative prior and uses diffusion models to approximate the distribution of teacher features. During distillation, the student’s partial observations are regarded as degraded evidence, and posterior sampling from the diffusion prior produces plausible teacher feature reconstructions that serve as soft supervision. This approach effectively bridges the gap between full-context teachers and partial-context students in time-series or incomplete-modality settings. Extensive experiments on UCR, UEA, and PhysioNet datasets demonstrate consistent gains over standard KD and feature-based methods, accompanied by detailed ablation studies and reproducibility details.", "questions": "1. To what extent does the diffusion prior capture epistemic uncertainty about the teacher’s manifold, as opposed to merely adding stochastic regularization? Could visualizing the learned diffusion trajectories help clarify this distinction?\n2. The GDPD assumes the teacher’s feature space forms a coherent generative manifold. How sensitive is the approach to teacher architectures where intermediate representations are poorly structured?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper frame the teacher’s representation as a learnable generative prior, leveraging diffusion modeling to capture the intrinsic uncertainty and diversity of knowledge transfer. This view enriches the typically deterministic KD paradigm with a stochastic, distributional interpretation.\n2. The experiments are extensive and systematically designed, with convincing improvements and well-controlled ablations.", "weaknesses": "1. While GDPD is conceptually elegant, the paper’s treatment of the diffusion prior remains largely phenomenological—its success is empirically shown, yet its mechanistic role in shaping the student’s feature space is underexplored. The work stops short of analyzing what kind of uncertainty the diffusion prior captures (epistemic vs. aleatoric), or how this aligns with the teacher’s representational manifold. Consequently, the “why” behind the gains remains somewhat opaque.\n2. Moreover, the approach introduces additional training overhead through diffusion sampling and warm-up phases, but the paper does not quantify these costs or discuss trade-offs between sample diversity and efficiency. Finally, while the method is positioned as general, all evaluations are within classification tasks; its potential limitations on forecasting, regression, or multimodal fusion remain untested, leaving open questions about scalability beyond the current scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)