Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation
Authors: Muquan Li
Hang Gou
Dongyang Zhang
Shuang Liang
Xiurui Xie
Deqiang Ouyang
Ke Qin
Venue: NeurIPS 2025 poster ()
Keywords: Dataset distillation
Inner-loop optimization
Primary area: applications
TL;DR: This paper proposes a novel inner-loop dataset distillation framework , which automatically adjusts the truncation position and window size, and utilizes a low-rank Hessian approximation to reduce computational overhead.

Abstract: The growing demand for efficient deep learning has positioned dataset distillation as a pivotal technique for compressing training dataset while preserving model performance. However, existing inner-loop optimization methods for dataset distillation typically rely on random truncation strategies, which lack flexibility and often yield suboptimal results. In this work, we observe that neural networks exhibit distinct learning dynamics across different training stages—early, middle, and late—making random truncation ineffective. To address this limitation, we propose Automatic Truncated Backpropagation Through Time (AT-BPTT), a novel framework that dynamically adapts both truncation positions and window sizes according to intrinsic gradient behavior. AT-BPTT introduces three key components: (1) a probabilistic mechanism for stage-aware timestep selection, (2) an adaptive window sizing strategy based on gradient variation, and (3) a low-rank Hessian approximation to reduce computational overhead. Extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K show that AT-BPTT achieves state-of-the-art performance, improving accuracy by an average of 6.16\% over baseline methods. Moreover, our approach accelerates inner-loop optimization by 3.9 × while saving 63\% memory cost.

Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation
Muquan Li', Hang Gou', Dongyang Zhang' Shuang Liang', Xiurui Xie Deqiang Ouyang', Ke Qin
University of Electronic Science and Technology of China "Chongqing University
(muquan112023, gouhang) @std. uestc. edu.
(dyzhang, shuangliang minanglling, xiexiurui, qinke) wwwluuueet....... cewsst..edd.com
deqiangouyanggoqu. edu.
Abstract
The growing demand for efficient deep learning has positioned dataset distillation as pivotal technique for compressing training dataset while preserving model performance. However, existing inner-loop optimization methods for dataset dis- tillation typically rely on random truncation strategies, which lack flexibility and often yield suboptimal results. In this work, observe that neural networks exhibit distinct learning dynamics across different training stages stages-early middle, and fate-making random truncation ineffective. To address this limitation, we propose Automatic Truncated Backpropagation Through Time (AT-BPTT), novel framework that dynamically adapts both truncation positions and window sizes according to intrinsic gradient behavior. AT-BPTT introduces three key compo- nents: (1) a probabilistic mechanism for stage-aware timestep selection, (2) an adaptive window sizing strategy based on gradient variation, and (3) low-rank Hessian approximation reduce computational overhead. Extensive experiments on CIFAR-10, CIFAR- 100, Tiny-lmageNet, and ImageNet- K show that AT-BPTT achieves state-of- the-art performance, improving accuracy by an average of 6. 1.6 over baseline methods. Moreover, our approach accelerates inner-loop optimization by 3.9 x while saving 63% memory cost.
1 Introduction
The unprecedented success deep learning [8] has been fundamentally driven by the availability of large-scale datasets and computational resources. However, this data-centric paradigm poses scalability challenges, particularly in storage efficiency, computational overhead, and environmental sustainability [5]
33 20 These limitations have catalyzed the emergence of dataset distillation 48 19 (DD) as transformative approach to data-efficient learning. By synthesizing compact surrogate datasets that preserve the essential characteristics of their original counterparts, DD enables efficient model training without compromising performance.
[45] (33
10
consisting of inner-loop and outer-loop optimization. The outer-loop optimizes the synthetic dataset to minimize discrepancies between models trained on distilled and original data, and the inner-loop simulates the training dynamics neural network on distilled dataset [9]. Previous DD methods focus on approximating the outer-loop directly align the final performance metric [53 51 24 [7] [2]
Dataset distillation is inherently formulated as bilevel optimization problem 36
9.
(22]. However, outer-loop methods typically rely on surrogate objectives or heuristic measures to indirectly capture the effects of inner-low loop training, thereby failing accurately reflect the model's training dynamics. Popular inner-loop optimization techniques, such as Backpropagation Through
"Corresponding to: Dongyang Zhang
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

70
Truncation Pruncalionoaatit Posttin
Apriney (5
Autemee 0%
Accancy (5))
50
S. 8
16
Farty-Pallmnrrry
Midit--hhiiittr
alc-Prelimrrrryy
Early Post
30
Middle Post
30
Late Post
200
600
650
100
300 Epezh
500
100
300
600
Eprsh
(d)
(b)
(G)
Early Stage
Middle Stage
Late Stage
Whxlow Size
(1
Acuaa
Accuaay
30
40
Each Original
Eart--Sraller
80
Middie-Criginel
SmailerSmmlerr
Lute- Originall
Late-Smaller
Eoefy-Laanger
30
Late-Largee
1000
2000
100
400
500
600
100
200
300
4001
500
100
100
200
500
4001
5to)
600
Spech
Epoch
Epodd
(d)
(e)
(9)
Figure 1: Hypothesis verification for the influence of truncation strategies and window size. (a)(b((c) show experiments where the preliminary or post truncation positions are implemented at early, middle and late stages, respectively, and (f)(e)(f) present experiments where the window size is changed after fixing the truncation position. For example, Early-Preliminary (a) means that randomly select preliminary phase (0-100) timesteps in early training stage (0-200 epochs).
Time (BPTT) 46 and its truncated variants (T-BPTT 31], RaT-BPTT 19),, adopt rigid truncation
strategies to partially address performance bottlenecks. Nevertheless, these methods uniformly apply fixed or random truncation across all training stages, disregarding the distinct learning patterns of neural networks: early stages prioritize simple patterns and later stages refine complex features [I]. Consequently, the rigid truncation of learning trajectories limits model performance.
To address these limitations, we posit that automatic adjustment of truncation strategies across distinct training stages could better align with the intrinsic learning dynamics of deep neural networks (DNNs) [T]. We verify this perspective through a controlled variable experiment, which partitions the training process into three stages, and selects preliminary post timesteps unfolding in each epoch as the truncation position. Our experimental analysis reveals three critical findings: (1) early- stage truncation of preliminary timesteps yields 2.9% improvement in test accuracy (Fig. (b). (2)
middle-stage performance demonstrates negligible sensitivity to truncation position (A < 0.3%,
Fig. (p). and (3) late-stage prioritization of post timesteps enhances accuracy by 1.8% (e). These results substantiate the suboptimality of random truncation approaches, and motivate the formulation of secondary perspective: adaptive window sizing could simultaneously optimize computational efficiency and gradient preservation. Subsequent experiments with fixed optimal truncation positions (Fig. I-f) demonstrate that larger window in the early stage achieves a 2.5% accuracy gain, while the middle- and late-stage variations show insignificant impacts (A < 0.2%). These hypotheses and their validation are detailed in Section
Based on the above observation, we propose an effective DD method called Automatic Truncated Backpropagation Through Time (AT-BPTT). This method integrates three key mechanisms: dynamic truncation position, adaptive window size and low-rank Hessian approximation. Dynamic truncation position employs gradient magnitudes to probabilistically determine better truncation timesteps across different training stages. Adaptive window size leverages the magnitudes of gradient variation to modulate the truncation window, ensuring that critical gradient information retained when fluctuations are high. Low-rank Hessian approximation addresses the computational time and storage cost issues of Hessian matrix calculation in second-order optimization. Extensive experiments illustrate that AT-BPTT achieves state-of-the-an (SOTA) performance on CIFAR-I0 [15), CIFAR-100 [16] and ImageNet- 1K (29]. outperforming the leading inner-loop method by an average of 6. 6.16% while delivering a 3.9 x faster training speed and a 63% memory reduction.
[5],
2 Related Work
Dataset distillation is proposed with the aim of generating a compact synthetic dataset that effectively substitutes for the original dataset. Dataset distillation can be formulated as a bilevel optimization problem, with current mainstream approaches broadly categorized into outer-loop optimization

---

methods (data matching methods) and inner-loop optimization methods (meta- methods). Outer-loop optimization methods achieve dataset distillation by aligning surrogate objectives between the original and synthetic datasets. Gradient matching [5]] aligns the gradients of synthetic and original data during training to mimic the optimization behavior of the original dataset. Trajectory aligns parameter trajectories throughout the training process. overcoming
matching (2. 6 12 22 4
the short-term matching limitations of gradient matching. Distribution matching [41 52 44 aligns
distributions in the feature space, circumventing the complexity of bilevel optimization. However, outer-loop methods rely on surrogate models to indirectly capture the effects inner-loop training, failing to accurately reflect the true training dynamics of the model.
Inner-loop optimization methods treat the synthetic dataset as hyperparameters, minimizing the risk of models trained on synthetic data on the target dataset. A prominent inner-loop approach is Backpropagation Through Time [46], which simulates the model training process by performing multi-step gradient descent in the inner loop and optimizes the synthetic data in the outer loop to achieve dataset distillation. Subsequently, T-BPTT truncating the unrolled time steps. Recently, RaT-BPTT to enhance BPTT's distillation performance, partially reducing computational costs. However, its random truncation strategy fails to align with the learning characteristics of deep neural networks, limiting its performance.
improves distillation performance by employs random truncation strategy
Besides optimization based techniques, there are several other DD methods. Diffusion-based methods B5 generate compact samples that capture key features of the original dataset by optimizing
representations in the latent space or directly adjusting the diffusion process, excelling in high-quality
image generation and high-resolution tasks. Decoupled optimization methods [47 [32 3X decompose
the complex distillation process into independently optimizable subtasks, reducing computational complexity and enhancing the capability to handle large-scale datasets.
3 Preliminary
Dataset distillation ( aims to compress large training set into a smaller set S such that training on achieves comparable performance to training on D. The process involves bilevel formulation:
min (S),(S), D) s.t. 0 = A(O,, S,T),
(1)
$
where A represents the inner-loop learner over T steps with initialization 0g and synthetic dataset S.
Backpropagation Through Time (BPTT). BPTT 46 is the standard method for solving bilevel optimization reverse mode. When A follows gradient descent with learning rate a, the meta- gradient with respect to every unrolling learning trajectory is computed using the chain rule:
@L(Or(S), D)
T T-1
GBPTT -a
Σ
II [1 -all,gg =1
00
where H, 8 (S), represents the Hessian matrix at timestep and 9, (C(): PS) denotes 08
De0s
the value with respect to the parameter 0 and the mixed partial derivatives of S at timestep i. The detailed derivation of the formula shown in the Appendix
Truncated BPTT (T-BPTT). To alleviate the memory burden, T-BPTT 31 propagates gradients
through a smaller unrolling window of M steps instead of the full trajectory:
GT = (S) D)
T-1
Σ
T-1
II
1 [1-HH aHz
00
E
M
j=1-1
This truncation omits the first T M + terms, reducing the number of Hessian products.
Random Truncated BPTT (RaT-BPTT). RaT-BPTT 9)] extends T-BPTT by randomly positioning the truncated window along every unrolling learning trajectory in the training process. The meta- gradient of RaT-BPTT with random truncation position N is:
ac (C(0x(S), D)
N-1
T-1
G GRAT-BPTT
Σ
H-
(4)
00
N
j=++1
which differs from T-BPTT by randomly sampling M timesteps and omitting shared Hessian products
3

---

4 Methodology
4.1 Hypothesis Verification
We challenge the practice of applying random truncation uniformly across the entire training process in RaT-BPTT [2] based on two key observations, First, it is well established that deep neural networks (DNNs) tend to learn simple patterns during the early training stages before progressively acquiring more complex patterns []. Second, T-BPTT [II] exclusively utilizes the last T M + timesteps of each epoch, while the impact of restricting updates to early timesteps remains unexplored. These observations motivate our Hypothesis I: Might sampling specific timesteps from distinct stages lead to better performance, rather than uniformly applying random truncation?
To verify this hypothesis, we partition the epochs throughout the training process into three equal stages (early, middle, and late stages) while bisecting the timesteps within the unfolding learning trajectory for every epoch (preliminary and post phases). Our validation involves controlled ex- periments where we randomly truncate either preliminary or post timesteps during the early stage, maintaining standard RaT-BPTT implementation in subsequent two stages. Similar validations are implemented on the middle and late training stages, The experiments are conducted on the CIFAR-10 [15 dataset using the ConvNets architecture. Our experimental results shown in Fig. T reveal three key observations: (1) preliminary phase truncation in early stage enhances validation accuracy by an average of 2. 2.9%; (2) middle stage shows negligible performance variance (A < 0.5%) between phase choices; (3) post phase selection in late stage demonstrates 1.8% accuracy improvement. These findings not only validate our initial hypothesis but also raise a new Hypothesis II: Can dynamicully adjusting the truncation window size further enhance distillation performance?
To further investigate the impact of truncation window size, we adopt roughly adaptive truncation strategy based on the experiments mentioned above: selecting preliminary timesteps during the early training stage, applying random truncation in the middle stage, and seclecting post timesteps in the late stage. Subsequently, we conduct experiments in which each stage alternated between using the original window (used in RaT-BPTT), a larger window, and a smaller window (modifying the original window size by +10 timesteps). As shown in Fig. [ large window leads to a 2.5% accuracy improvement in the early stage, while variations window size during the middle and late stages have a negligible effect (A < 0.2%). Based on these, we conclude that selecting appropriately sized truncation windows for specific timesteps at different training stages yields better performance.
4.2 Automatic Truncated BPTT
Building on the above findings, we propose an Automatic Truncated Backpropagation Through Time (AT-BPTT) framework designed to optimize the inner-loop for dataset distillation. This automatic mechanism consists of three components: dynamic truncation position, adaptive window size and a low-rank Hessian approximation, as shown in Fig. 3
Gradicnt Magnitude and Gradicnt Variation Magnitude
Dynamic Truncation Position. The experimental val- idation of Hypothesis I raises a question: How to estab- lish an evaluation metric that enables staged truncation of timesteps during different training stages? Our analy- sis begins with the gradient accumulation mechanism ex- pressed in Eqn. which reveals that parameter updates represent cumulative contributions from all timestep gra- dients, with each timestep's influence modulated by its temporal position and subsequent gradient dynamics. To quantify these temporal effects, we record gradient mag- nitudes across all timesteps during training iterations, with averages visualized in Fig. 2 It is observed that gra- dient updates are initially large but gradually diminish as timesteps progress. Interestingly, this observation and the verification of Hypothesis I align with Arpit et al. [''s seminal work on deep learning dynamics: DNNs tend to learn simple, easily identifiable patterns during the early stages of training before progressively shifting toward more complex and fine-graines patterns in later stages. This is attributable to that simple
3.0
Magnitode
08
Gradient Variatio Magrinude
07
2,5
Gralien Varscim Magiinil
Ga Mazniute
0.6
2.0
0.5
1.5
0.2
03
1.0-
0.5
ihad
Ube Ands anthu aad ou
50
100
150
200
Timestop
Figure 2: Illustration of the gradient and gradient variation average magnitudes each timestep during training process. The entire timesteps are roughly averaged into preliminary and post phases.
patterns often dominate the data and exhibit

---

Inner-loop
Prefiminary
Post
Tizessega
Uncolling Lesming Trajectory
Synthetic dataset
Semantic Preservation
Gradient Magnitude
Eady
Gradient Variation Magnitade
X
S
Pe (0
Guav
Middle
Thrshhald guided
P... 18) 023%%
Low Rank Hessian Apgruximation
Update
Stage Transition
Late
Truncation Position
Pax 10 0.51%
Window Size
Outer-loop
Figure 3: Overall framework of our proposed AT-BPTT, The distilled data flows through our patch- wise semantic preservation module the inner-loop optimization. The dynamic truncation position and adaptive window size then jointly optimize inner-loop training dynamics. The low-rank Hessian approximation is utilized to reduce computational cost.
high saliency, allowing large gradient updates the preliminary phase to align effectively with these patterns, leading to rapid performance improvement in the early stage. Although the gradients of timesteps the post phase are smaller, they play a crucial role in fine-tuning the network, ultimately leading to significant performance gains in the late stage.
This staged behavior suggests the gradient magnitude serves as an intrinsic metric for training stage identification. We thus formalize our truncation probability mechanism through temperature- controlled softmax normalization:
exp( V
P Pirunc (t) Σ exp Vxp((
where t represents the annealing temperature parameter. Therefore, we define a three-stage strategy to determine dynamic truncation positions: (1) In the early stage, each timestep t is selected truncation position with probability Purnn(1) (2) In the middle stage, truncation applied randomly; 1- (t)
(3) In the late stage, the probability of each timestep is
exp( (oCalla/r)
if Early Stage, if Middle Stage,
S exp (12l//))))
Pirune (t)
1/T,
(6)
exp(TT6r11100) 2 (92
if Late Stage.
T-1
Adaptive Window Size. Based on the three-stage framework, we further propose an adaptive window size mechanism to address Hypothesis II. Revisiting Eqn[4] conventional inner-loop optimizations employ fixed window size truncation to reduce computational overhead by partial Hessian matrix computation. This Hessian matrix, obtained via second-orde diffecenniiaion of the loss function, captures the temporal variations in gradients across different timesteps. We thus record the gradient variation magnitude/ VoLa
L || each timestep during training as visualized in Fig. This reveals a type of staged pattern: early training stages exhibit high- siggitagn oscillatory gradients, while middle and late stages demonstrate stabilized gradient variations.
This observation aligns with Hypothesis verification experiments, suggesting that substantial gra- dient variations indicate active adjustment of model decision boundaries at corresponding timesteps. To obtain an adaptive window size aligned with gradient variation magnitude, we introduce a cemperature-conttolledd soft normalization of gradient variations obtain the size weight 17(t) at t:
exp(I VoL. 12l
(()) Σ exp Vol - | (o
The adaptive window size W (t) at timestep / is then computed through linear transformation of the original window size W in RaT-BPTT: (t) w + 2d v7(4) with a range [ d. W + d] and d > 0. This adaptive adjustment enables the model automatically allocate larger windows for
5

---

timesteps with significant gradient variations while contracting windows for stable phases, achieving better balance between computational efficiency and model performance.
Threshold-guides Stage Transition. In contrast to the rough equal partitioning for hypothesis verification in Section 4.1] adopt threshold-guided stage transition approach to stably switch between early, middle, and late training stages. Since the convergence of optimization processes in visual tasks is ultimately determined by the trend in model performance metrics, we utilize accuracy variation at each timestep to automatically regulate training stages. We define the accuracy variation as: AA, A, At- Two zero-initialized counters Cearly-mided and Cmiddie-latt late are defined to record the number of times AA, < M and AA, < N, respectively. The rule from early to middle stage and from middle late stage are:
T2 Σ 1=
Cearly-midd'
1I(AA < M) X Cmiddie-late 1=1
1(A, < <) NY, (8)
where 1(.) denotes the indicator function that returns the condition is satisfied and 0 otherwise. This mechanism triggers stage transition when either: (1) X times of accuracy variation fall below M for early-to-middle transition, or (2) Y times fall below N for middle-to-late transition. The accuracy variation reflects model learning dynamics, and employing multiple threshold evaluations effectively mitigates training noise, thereby ensuring stable transitions.
4.3 Low-Rank Hessian Approximation
In addition to random truncation, RaT-BPTT's unfolding learning trajectory approach also suffers from high computational overhead. This primarily stems from the frequent computation of implicit Hessian matrix products in Eqn. [4] which are required for evaluating inner-loop optimization perfor- mance. Therefore, introduce low-rank Hessian approximation (LRHA) reduce both time and memory complexity while preserving the gradient direction by leveraging the low-rank structure of the Hessian matrix. Given the Hessian at timestep we approximate it by rank-kj factorization:
H, = T ss H1j,
(9)
where Uj. V E ddd and Σ, E x We adaptively determine ky based on the normalized gradient magnitude, with lower bound:
V e max;
/ )):
ky = max
kmax
(10)
where kinxx 0.1 d and kmin > O) ensures numerical stability. Instead of materializing Hj. we apply randomized singular value decomposition (SVD) using Hessian-vector products (HVP) and QR factorization:
y () Ω N(,,11
( = HVP H H1 y
9= 1,2,
(11)
2 R, = QR(Y(2))
B, Q H1
Ú, Ej. V, SVD(B)).
We then reconstruct new Hy:
A, = (Q,0)) E, TQ,
(12)
The final meta-gradient of AT-BPTT is formulated as:
GATBPTT (C())
N-1
T-1
Σ
II [ aHt)
Prume Piume(i)gg
(13)
00
(Q)
Let denote the dimensionality of the model parameter vector. Through our proposed LRHA, the
time and memory complexity drop from O(p²) to O(p kg + kg)) and from O(r²) to O(pkk, k2), respectively, greatly reducing computational resource consumption.

---

4.4 Patch-wise Semantic Preservation
To address the poor performance of inner-loop DD on high-resolution datasets, we propose lightweight patch-wise semantic preservation (PSP) to enhance generalization capability. Given an input image R #xx33 we split into n x n non overlaappp patches:
P 1 Fig = zli (i + 1) s,j s: + 1). s),
[H/n]}
(14)
where controls the granularity of local processing and we set n = 4. Each patch æiy is independently distilled using AT-BPIT to generate local synthetic prototypes:
Sij AT-BPPT((..) P
(15)
where Osoaal denotes the parameters of the local distillation network applied to each patch. To ensure semantic coherence, we perform prototype centroid matching against the global prototype set Sglobal "
Calign
Σ (1n())))) p pL(Sglobll) I|2.
(16)
where vd-) calculates the prototype centroid. The final objective combines original distillation loss and alignment loss: Lusal LXI-BETT + aatignn where balances the two objectives.
Experiments
5.1 Experimental Setup
Dataset. We adhere to the conventional procedure adopted in dataset distillation 9. We select three standard datasets: CIFAR-10 [13] (10 classes, 32x CIFAR-100 [5] (100 classes, 32x32), and Tiny- ImageNet [16] (200 classes, 64x64). To further show the effectiveness of AT-BPTT on high-resolution images, scale up the dataset to ImageNet- I 29 (1,000 classes, 224x224), For CIFAR-10 10 and C10,R-100, we distill datasets with 1, 10, and 50 images per class (IPC = 1, 10, 50), while for Tmay-Im and ImageNet-IK, K, we use and 10 images per class (IPC 1, 10).
Baselines. We compare our AT-BPTT with the most representative baselines categorized into two distinct paradigms: one category contains methods that approximate or optimize the outer-loop of bilevel optimization, and the other category includes methods that directly improve the inner-loop of bilevel optimization. More details baseline methods are introduced in Appendix B
Implementation Details. Following Rat-BPTT we employ standardized convolutional neural network [I (CNN) architectures with depth adapted to dataset specifications: 3-layer CNN (Conv- 3) for CIFAR-IO/CIFAR 100 synthetic datasets, and a 4-layer CNN (Conv-4) for Tiny-ImageNer and high-resolution ImageNet-1K K datasets. More hyperparameter settings are detailed in Appendix B
5.2 Comparison with Previous Methods
Tab. T demonstrates that the proposed AT-BPTT framework establishes new state-of-the-a art perfor- mance on three benchmark datasets (CIFAR-10 [5), CIFAR- [15], and Ting-ImaeeNet [16)), outperforming existing inner-loop and outer-loop optimization approaches by a significant margin. Most remarkably, AT-BPTT achieves an average accuracy gain of 166% over RaT-BPTT, the SOTA inner-loop DD method. This substantial improvement low-resolution image distillation stems from the combination of our novel dynamic truncation strategy and adaptive window size mechanism, which collectively enhance the model's capacity to capture multi-level feature correlations through dynamically adjusted learning trajectories. The visualizations of these synthetic datasets are shown in Appendix
For the nighressolution dataset ImageNet-IK, Tab. shows that AT -PTT also exhibits superior competitiveness compared leading methods, achieving 30.6% accuracy under the IPC=10 setting. This exceptional performance is attributed to the transformative role of PSP segmentation strategy, which effectively transfers AT-BPTT's strengths low-resolution domains to high-resolution scenar- ios. These results not only confirm AT-BPTT's capability in processing high-resolution data but also highlight the scalability of the PSP strategy for more advanced high-resolution applications. More experimental results are provided in Appendix C
7

---

Table 1: Comparison with the SOTA baseline dataset distillation methods. Following previous
methods, the ConvNet used for distillation are Conv-3 on CIFAR-10 [15] and CIFAR-100 [T5], Conv-4 on Inn-ImmgeNet [16] and ImageNet-Il K (2)]. Each reported result is the average of experiments. Entries with are absent due to scalability problems.
Dataset
CIFAR-I0
CIFAR- 100
Tiny ImageNet
ImageNet-Ie
Img/class(IPC)
10
50
10
50
10
10
Outer-loop Optimization
DSA
53
28.8+0.7
52.1
60.640.5
13.9=0.3
32.340.3
42.840.4
6.640.2
14.4+2.0
1.1
3.2:0.3
CAFE
41
30.3+111
46.320.6
53.540.6
12.920.3
27.8+0.3
37.940.3
MTT
2
46.2+0.8
65.420.7
71.6+0.2
24.3+0.3
39.7+0.4
47.7+0.2
8.8+0.3
23.2+0.2
TESLA
48.5=0.8
66.420.8
72.6+0.7
24.8=0.4
41.7=0.3
47.9+0.3
7.7+1.0
17.8+0.5
DATH
12
46.9=0.5
66 66.820.2
16.1+0.3
27.9=0.2
47.2+0.4
55.0+0.2
17.1=0.3
31.1+0.3
ATT
22
48.3+1.0
67.7=0.6
74.5+0.9
26.1
44.2+0.8
51.2+0.2
11.0#0.4
25.8+0.7
4.7+14
8.7+10
MCT
54
48.5=0.2
66.0=0.3
72.3+0.3
24.5=0.5
42.5+0.5
46.8+0.2
9.6+0.5
22.6+0.8
ICFM
44
49.5=0.3
71.8
77.4+0.3
34.420.5
48.7=0.3
54.7=0.2
18.2+0.5
26.8+0.6
Inner-loup Optimization
BPTT 46
49.1=0.6
62.4e1).4
70.5+0.4
21.340.6
34.7+11.
1.1+0.7
2.3+0.9
FRePO 55
45.6+0.1
63.5e0.1
70.7+0.1
26.3+0.1
41.3+0.1
41.5+0.1
16.9+0.1
22.4+0.1
RCTG 24
49.6+1.2 66.8e0.3
35.5+0.7
22.4+0.3
RaT-BPTH
53.240.7
69.4:0.4
75.340.3
35.340.4
47.5+0.2
50.640.2
20.1
24.4:0.2
5.241.1
13.0+0.9
Teddy 49
30.1=14
53.020.5
66.140.44
13.520.4
33.420.7
49.420.5
34.1+0.8
Ours
54.420.6
72.420.3
78.7+0.2
36.920.5
49.020.6
55.920.1
24.3+0.4
32.7+0.5
14.7=0.7
30.6+0.3
Full dataset
84.8
56.2
37.6
33.8
5.3
Computational Efficiency Comparison
85
To evaluate the computational efficiency of our approach, we compare the average GPU memory consumption and training time among three leading data distillation meth-
DATM RaT--PTT
Metrory 0.633
Accuety (%6
75
Ours
23:30
16.7G
ods: DATM [12, RaT--PTT
[ and our proposed AT-
65
TEC
BPTT. All experiments are conducted on NVIDIA A800
55
hal-BTTI
185
21.6
26.3
GPUs with IPC setting 1, 10, and 50. As demonstrated in Fig. 4 AT-BPTT achieves superior computational efficiency while maintaining competitive performance, delivering 63% reduction in memory usage and 3.9x speedup compared to RaT-BPTT. These results under- score the exceptional scalability and practical viability of our method for real-world deployment.
AT.8PT
51
53
75
Spnap
177
25x
26x
45
1
10 Image Per Class (IPC)
50
Figure 4: Comparison of performance, GPU memory usage, and speedup between the SOTA DD methods and our AT-BPTT.
5.4 Ablation Study
Component Combination Evaluation. this section, we evaluate the individual and combined contributions of three performance components: dynamic truncation position (DTP), adaptive window size (AWS), and patch-wise wiie semantic preservation (PSP). Ablation studies are conducted under the IPC=10 setting across CIFAR- 1R-100/100 15). Tiny-ImageNer [16] and ImageNet-IK KK (29). Tab. 2 shows that the integrated application DTP and AWS yields a combined optimization effect, resulting in a 2.8% accuracy improvement, which surpasses the sum of the individual contributions. This arises from the complementary information processing mechanisms of DTP and AWS, which enhance the model's ability to capture key features of the original dataset. Regarding the PSP component, Tab, reveals that its accuracy improvement is marginal on CIFAR- 10/00000000---00 yet it is substantial on two datasets with higher resolution. When combined with DTP and AWS, it leads to remarkable gains of 8.3% and 17.6%. This is attributed to PSP's ability to segment high-resolution images into smaller patches, thereby fully leveraging AT-BPTT's strengths across various resolutions.
Stage Transition Threshold Analysis. achieve precise control over stage transitions during model training, we study four critical hyperparameters: M (the accuracy variation threshold in the early stage), X (the counter threshold for early stage), N (the accuracy variation threshold the middle stage), and Y (the counter threshold for middle stage). Through experiments with the CIFAR-10 [1]

---

Table 2: Ablation study for the contribution of differ- ent components in our framework. The improvements denoted by red numbers with respect to baseline.
Table 3: Ablation study for d corresponding to window size. The improvements denoted by red numbers with respect to baseline.
DTP AWS PSP
CIFAR-10
CTFAR-100
Tragefiet-18
Accuracy (%)
Time (hours)
Memory (GB)
0
70.56
4.9
5.34
V
70 1.3)
70.2
0.89
47.9 0.4)
26.6 (1 2.23
17.9
6.73
4.99
71.43 (1 0.57)
(1 0.2)
5.62 0.28)
69.5
0.13
47.8 (T
25.4
17.2
4.25
10
72.44
(1 1.88)
5.5 0.6)
6.14 0.80)
72.2
15
72.76
(1 2.2)
(1 0.9)
7.38 (1 2.00)
3.0)
1.30
30.6
20
72.95
2.39)
6.2 1.3)
9.16 3.82)
Accursey
22.5
72.5
20
69
1M
21:01
88.
63:61
64.57
42.0
-2.0
69.72
70-93
71.58
11.87
71.46
73.46
69.45
69.72
70,02
69.68
69.37
66.01
31.5
71.5
1.6
71.72
71.37
71.04
70.99
M
21.0
71.0
1,4
69.52
70.88
71.61
71.33
71.52
70,09
70.26
71:47
72.25
71.21
69.99
68.50
71.07
68.59
69:22
69.55
09:83
09:44
69:14
69.53
69.75
70.24
69.59
69.32
08:37
7000
20
22
Figure 5: Ablation study for the stage transition threshold. The left and right matrices reflect the effect of X and M, and Y and N on the accuracy. respectively. Darker colored squares indicate higher accuracy under the synergistic influence of horizontal and vertical coordinates.
under IPC=10, we identify that suboptimal parameter configurations induce two failure modes: the plateaus in the initial training phase or too early transition to the final stage, both of which disrupt the staged paradigm and cause substantial performance degradation. Fig. demonstrates optimal parameterization when X corresponds to 5% of total training epochs and Y to 4%, with M=1.5 and N=1.0. This configuration achieves balanced stage duration distribution, stable convergence patterns, and better AT- -T-P performance.
Window Size Analysis. The hyperparameter d in Section 4.2 governs the adjustment range of the truncation window, demonstrating a trade-off between model performance and computational efficiency. We conduct ablation study CIFAR-10 [5] with IPC=10 and quantitatively assess the dual effects of varying values. As Tab shows, progressive increases d simultancously enhance test accuracy and computational demands. Notably, compared to the baseline (d 0), 10 achieves a 1.88% accuracy improvement with modest resource increments, with only 0.6 hours additional training time and 0.8 GB GPU memory usage. However, when d 10, the accuracy gains diminish while the computational cost rises substantially. This mainly stems from the need for longer timesteps and the corresponding exponential growth in the dimensionality of the Hessian matrix as d increases. Based on a trade-off analysis between performance gains and computational costs, the value of d to 10 and report the results obtained with this setting. More ablation studies are provided Appendix(C
Conclusion
This paper proposes AT-BPTT, novel dataset distillation framework that optimizes inner-loop training through staged adaptation. Our analysis reveals that neural networks prioritize distinct learning patterns across training stages, necessitating adaptive truncation mechanisms. AT-BPTT resolves this by dynamically aligning truncation positions with gradient magnitude distributions, adjusting window sizes based on gradient stability, and reducing computational cost by low-rank Hessian approximation. The experiments across four benchmark datasets confirm that our method significantly outperforms existing dataset distillation method, achieving superior accuracy with efficient training. Furthermore, the formalization of gradient dynamics as stage indicator provides theoretical foundation for future research in bilevel optimization. Future work will focus on extending AT- AT-BPTT to recurrent architectures and federated learning scenarios.

---

Acknowledgments
This work is supported in part by the Natural Science Foundation of Sichuan Province (Grant No. 20252NSFFSC1464) the China Postdoctoral Science Foundation (Grant No. 2024M760357), the Postdoctoral Fellowship Program of CPSF (Grant No. GZB20240115), the Sichuan Science and Technology Program (granted No. 20242DZX0011), the Fundamental Research Funds for the Central Universities No.ZYGX2025XI042 and the Natural Science Foundation of China (Grant No. 62406057).
References
[1] D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger. E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. C. Courville, Bengio, and S. Lacoste-Julien. A closer look at memorization in deep networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML,, volume 70 of Proceedings of Machine Learning Research, pages 233-242. PMLR, 2017.
[2] G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J. Zhu. Dataset distillation by matching training trajectories. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pages 10708-10717. IEFF, 2022.
[3] M. Chen, Du, B. Huang, Y. Wang, X. Zhang, and W. Wang. Influence-guided diffusion for dataset distillation. In The Thirteenth International Conference on Learning Representations, ICLR, 2025.
[4] Y. Chen, G. Chen, M. Zhang, W. Guan, and L. Nie. Curriculum coarse-to-fine selection for high-ipe dataset distillation. CoRR, 0ls/2503.18872. 2025.
[5] J. Cui, R. Wang, S. Si, and C. Hsieh. DC-BENCH: dataset condensation benchmark. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, 2022.
[6] Cui, R. Wang, S. Si, and C. Hsich. Scaling up dataset distillation to imagenet- IK with constant memory, In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML, volume 202 of Proceedings of Machine Learning Research, pages 3565-6590 PMLR, 2023.
[7] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pages 1181-4186. Association for Computational Linguistics, 2019.
[8] Du, Y. Jiang, V. Y. Tan, J. Zhou, and H. Li. Minimizing the accumulated trajectory error to improve dataset distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pages 3749-3758. IEEE, 2023.
[9] Y. Feng, R. Vedantam, and Kempe. Embarrassingly simple dataset distillation. In 12th International Conference on Learning Representations, ICLR, 2024.
[10] J. Geng, Z. Chen, Y. Wang, H. Woisetschlaeger, S. Schimmler, R. Mayer, Z. Zhao, and C. Rong. survey on dataset distillation: Approaches, applications and future directions. Proceedings of the Thirty-Second International Joint Conference Artificial Intelligence, IJCAI, pages 6610-6618. ijcai.org, 2023.
[11] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting, In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 4367-4375. Computer Vision Foundation / IEEE Computer Society, 2018
[12] Z. Guo, K. Wang, G. Cazenavette, H. Li, K. Zhang, and Y. You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
10

---

[13] J. Howard and S. Gugger. Fastai: a layered api for deep learning. Information, 11(2):108, 2020.
[14] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In F. R. Bach and D. M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML, volume 37, pages 448-456, 2015.
[15] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report Citeseer, 2009.
[16] Y. Le and X. Yang. Tiny imagenet visual recognition challenge. Technical Report, 7(7):3, 2015.
[17] Y. Lee and H. W. Chung. Selmatch: Effectively scaling up dataset distillation via selection-basee initialization and partial updates by trajectory matching. In Forty-first International Conference on Machine Learning, ICML, 2024.
[18] S. Lei and D. Tao. A comprehensive survey of dataset distillation. IEEE Trans. Pattern Anal. Mach. Intell., 46(1): 2024.
[19] M. Li, D. Zhang, Q. Dong, X. Xie, and K. Qin. Adaptive dataset quantization. In AAAI-25, pages 12093-12100 2025.
[20] M. Li, D. Zhang, He, X. Xie, Y. Li, and K. Qin. Towards effective data-free knowledge distillation via diverse diffusion augmentation. In Proceedings of the 32nd ACM International Conference on Multimedia, MM, pages 4416-4425, 2024.
[21] Z. Li, Z. Guo, W. Zhao, T. Zhang, Z. Cheng, S. Khaki, K. Zhang, A. Sajedi, K. N. Platamiotis, K. Wang, and Y. You. Prioritize alignment dataset distillation. CoRR, 0ass2248.033666 2024.
[22] D. Liu, J. Gu, H. Cao, C. Trinitis, and M. Schulz. Dataset distillation by automatic training trajectories. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, editors, Computer Vision ECCV, volume 15145 of Lecture Notes in Computer Science, pages 334-351. Springer, 2024.
[23] N. Loo, R. M. Hasani, A. Amini, and D. Rus. Efficient dataset distillation using random feature approximation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, 2022.
[24] N. Loo, R. M. Hasani, M. Lechner, and D. Rus. Dataset distillation with convexified implicit gradients. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML, volume 202 of Proceedings of Machine Learning Research, pages 22674, PMLR, 2023.
[25] Y. Ma, K. Qin, and S. Liang. Beta-LR: Interpretable logical reasoning based on beta distribution. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024.
[26] A. Maekawa, N. Kobayashi, K. Funakoshi, and M. Okumura. Dataset distillation with attention labels for fine-tuning BERT. In A. Rogers, J. L. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), ACI., pages 119-127. Association for Computational Linguistics, 2023.
[27] A. Maekawa, S. Kosugi, K. Funakoshi, and M. Okumura. Dilm: Distilling dataset into language model for text-level dataset distillation. In K. Duh, H. Gómez-Adorno, and S. Bethard, editors, Findings of the Association for Computational Linguistics: NAACL, pages 3138-3153, Association for Computational Linguistics, 2024.
[28] T. Nguyen, R. Novak, L. Xiao, and J. Lee. Dataset distillation with infinitely wide convolutional networks. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, pages 5186-5198, 2021.
[29] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L.. Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):111-252, 2015.

---

[30] A. Sajedi, S. Khaki, L. Z. Liu, E. Amjadian, Y. A. Lawryshyn. and K. N. Plataniotis. Data-to- model distillation: Data-efficient learning framework. In European Conference on Computer Vision, pages 338-457, 2024.
[31] A. Shaban, C. Cheng, N. Hatch, and B. Boots. mization. In K. Chaudhuri and M. Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS, volume 89 of Proceedings of Machine Learning Research, pages 772--7322 PMLR, 2019.
Truncated back prok-ppapggttin for bilevel opti-
[32] Z. Shen, Sherif, Z. Yin, and S. Shao. DELT: A simple diversity- earlylate training for dataset distillation. CoRR, abs/2411.19946, 2024.
[33] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR, 2015.
[34] D. Su, J. Hou, W. Gao, Y. Tian, and B. Tang. D 4: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5809-5818, 8, 2024.
[35] D. Su, J. Hou, W. Gao, Y. Tian, and B. Tang. D4m Dataset distillation via disentangled diffusion model. IEEE/CVE Conference on Computer Vision and Puttern Recognition, CVPR, pages 5809-5818. IEEE, 2024.
[36] Sun, Shi, D. Yu, and Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In IEEE/CVF Conference on Computer Vision and Putlern Recognition, CVPR, pages 9390-9399, IEEE, 2024.
[37] Y. Tao, L. Kong, A. Kan, and L. Callot. Textual dataset distillation via language model embedding. Y. Al-Onaizan, M. Bansal, and Y. Chen, editors, Findings of the Association for Compulational Linguisties: EMNLP, pages 12557-12569, Association for Computational Linguistics, 2024.
[38] M. Tran, T. Le, X. Le, T. Do, and D. Phung. Enhancing dataset distillation via non-critical region refinement. CoRR, abs/2503. 18267, 2025.
[39] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR, 2019.
[40] K. Wang. Z. Li, Z.-Q. Cheng, S. Khaki, A. Sajedi, R. Vedantam, K. N. Plataniotis, A. Hauptmann, and Y. You. Emphasizing discriminative features for dataset distillation in complex scenarios. arXiv preprint arXiv:2410.
2024.
[41] K. Wang, B. Zhao, X. Peng. Z. Zhu, S. Yang, S. Wang, G. Huang, H. Bilen, X. Wang, and Y. You. CAFE: learning to condense dataset by aligning features. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pages 12186-12195. IEFE, 2022.
[42] R. Wang, S. Liang, Q. Chen, Y. Huang, M. Li, Y. Ma. D. Zhang, K. Qin, and M.-F. Leung. Graphcogent: Mitigating llms' working memory constraints via multi-agent collaboration in complex graph understanding, 2025.
[43] R. Wang. S. Liang, Q. Chen, J. Zhang, and K. Qin. Graphtool-instruutinn: instruction:: Revolutionizing graph reasoning in Ilms through decomposed subtask instruction. In Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining, V.I, KDD, pages 1492-1503, 2025.
[44] Wang, Y. Yang, Z. Liu, C. Sun, X. Hu, C. He, and L. Zhang. Dataset distillation with neural characteristic function: A minmax perspective. CoRR, abs/2502.200633, 2025.
[45] T. Wang, J. Zhu, A. Torralba, and Efros. Dataset distillation. CoRR, abs/ abs/1811 /////1811..1000 10959, 2018.
12

---

[46] P. J. Werbos. Backpropagation through time: what it does and how to do it. Proc. IEEE, 78(10):1500-150, 1990.
[47] Z. Yin, E. Xing, and Z. Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from A new perspective. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems, 2023.
[48] R. Yu, S. Liu, and X. Wang. Dataset distillation: A comprehensive review. IEEE Trans. Pattern Anal. Mach. Intell., 46(1):150 10001500110000 170, 2024.
[49] R. Yu, S. Liu, J. Ye, and X. Wang. Teddy: Efficient large-scale dataset distillation via taylor- approximated matching. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, editors, Computer Vision ECCV, volume 15104 of Lecture Notes in Computer Science, pages 1-17. Springer, 2024.
[50] X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, pages 649-657, 2015.
In Cortes,
[51] B. Zhao and H. Bilen. Dataset condensation with differentiable siamese augmentation. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML, volume 139 of Proceedings of Machine Learning Research, pages 12674- 12685. PMLR, 2021.
[52] B. Zhao and H. Bilen. Dataset condensation with distribution matching. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV. pages 6503-6512. IEEE, 2023.
[53] B. Zhao, K. Mopuri, and H. Bilen. Dataset condensation with gradient matching. In 9th International Conference on Learning Representutions, ICLR, 2021.
[54] W. Zhong, H. Tang, Q. Zheng, M. Xu, Y. Hu, and L. Nie. Towards stable and storage-efficient dataset distillation: Matching convexified trajectory. CoRR, abs/2406. 19827, 2024.
[55] Y. Zhou, E. Nezhadarya, and J. Ba. Dataset distillation using neural feature regression. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the contributions of AT-BPTT, including dynamic truncation positions, adaptive window sizing, low-rank Hessian approxi- mation, and patch-wise semantic preservation.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Appendix D discusses limitations: (1) computational overhead due to inner- loop unrolling, and (2) reliance on simple CNN architectures. Future work directions address these limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Theoretical derivations for BPTT and meta veta-ggrdieet computation are provided in Appendix A Assumptions and proofs are included.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Implementation details (data preprocessing, hyperparameters, architectures) are described in Section and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: Experiments use publicly available datasets (CIFAR-10, ImageNet- IK, etc.),
which standard benchmarks.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/niiss wtt.cc/pwlccgpceeeeeeeeo htte:gggudddeeoeSSuubbiiiiiii swiiissssoppiiiyoy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training protocols, hyperparameters, and dataset splits are detailed in Section and Tab. 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Results in Tab. [ report mean accuracy with standard deviations across independent runs. Error bars in Tab. aree explained as variations across experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: GPU memory usage and training time compared in Fig. Experiments were conducted on NVIDIA A800 GPUs in Section 5
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips: cttpr/rrppppppppppppc/ccccccccia ctcceeeeeeeccsccceeeeeeeeee
[Yes]
Justification: Appendix G states compliance with NeurIPS ethics guidelines. No human subjects, sensitive data, or fairness concerns are involved.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix discusses positive impacts pprivacy-preserving ML) and potential risks (misuse of generative models), with mitigation suggestions.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: While societal risks are acknowledged, no specific safeguards are detailed for the proposed framework.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite and intro the creators of all baselines and dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, datasets.paperssiii paperswi thcode. www.coo//aaaaa com/datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
There is no new asset like datasets or platforms proposed in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The work does not involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
No human subjects research conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are not used in the methodology or experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

Algorithm 1: Automatic Truncated Backpropagation Through Time
Input: Training dataset D, model parameters 0, total training timesteps T Params: Accuracy variation thresholds M, N, transition counters X, Y. temperature parameter T.
1
Initialize zero-initialized counters C1, C2.
2 for f = 1, do
Segment images through patch-wise semantic preservation. Compute accuracy variation AA, A At-1-
5 Update counters: C1 + C1 + 1(AA, < M), C2 + C2 + 1(4 < N).
if C 2 then
Switch to Middle Stage.
8
if C2 2 Y then
Switch to Late Stage.
Compute gradient magnitude /
Computes probability Pinume
Adjust window size (t) computed with d.
13
Compute GAU-BFTT through low-rank Hessian approximation.
Compute inner-loop loss (S)r
15
Update model: 0z+1 +- 0; - VVL..
Compute outer-loop loss Lmet on validation set.
17
Compute meta-gradient VgLmmeu-
Update distilled dataset: S t S a' VV SLmeta
Output: Distilled dataset S.
A The Mathematical Derivation for Equation.2 in Main Text
We minimize the test loss L(Or(S), D), where 0 (S) the model parameter obtained through inner optimization on the synthetic dataset S. Update parameters via gradient descent:
0 = 0, a S))
(17)
for T steps, resulting in (S). The goal BPTT compute the gradient of the outer loss with respect to i... the meta-gradient 0C We express 0r the result of updates from the initial parameters 0p
T-1
f = 00
V VoL(0,(S),S)
(18)
t=0
The meta-gradient is decomposed into:
OC
0C
00T as
(19)
as
00,
Outer gradient
Inner gradient propugation
We then expand the parameter update process recursively:
00
00:
00,
00, V2C(O,, as
V)L(O,,))
(20)
as
as
aas
as
Mixed derivative torm
Hessian
We expand the recursion into an explicit summation:
00r
a T-1 -
(C((0(())
8 (CC((((S) deas
(21)
as
O92
The final BPTT Meta-Gradient is formulated as:
GBPTT AL(Or(S),2 00T
8 ((),
8 ((O,
Σ
(22)
802
000S
j=tt1
21

---

B More Implementation details
To validate the effectiveness of our method while ensuring experimental rigor, we employ controlled variable approach in designing the experimental protocol. The critical experimental parameters and configurations are rigorously aligned with those of RaT-BPTT [ specifically encompassing essential aspects including data preprocessing procedures, model convergence acceleration mechanisms, gradient explosion mitigation strategies, and label processing methodologies. Each reported result is the average of 5 experiments.
Data Preprocessing. In the experimental setup, all datasets were subjected to unified geometric augmentation strategy comprising random rotation and horizontal flipping operations. The augmented data subsequently underwent ZCA whitening processing with regularization coefficient À 0.1. For synthetic data preprocessing, the initial data generation was performed through a Gaussian distribution-based random initialization method, followed by normalization operations ensure numerical distribution consistency.
Model Convergence Acceleration Mechanisms. During the distillation stage, three acceleration strategies are implemented to enhance optimization efficiency: (1) An Exponential Moving Average (EMA) technique is adopted to accelerate model convergence while improving stability and general- ization capabilities; (2) The Higher framework is employed for efficient meta-gras veta-gradient computation; (3) An Adam optimizer with learning rate 0.001 is utilized for precise inner-loop updates.
Gradient Explosion Mitigation and Label Processing. To address the issue of gradient explosion, our methodology incorporates a meta gradient clipping strategy while ensuring gradient stability through sufficiently large batch sizes for each dataset. Consistent with the experimental configuration of RaT-BPTT, approach maintains architectural coherence by employing positive real-value representations rather than implementing normalization on label probability distributions.
Hyperparameter Settings. In the experiments, several hyperparameters require tuning, they directly influence the distillation performance of the method. Accordingly, we detail the hyperparam- eter settings used for performance evaluation in the main text presented in the Tab, g Specifically, window denotes the initial window size, totwindow represents the total number of unrolled time steps, d controls the range of the truncation window size, Ir indicates the learning rate, architecture refers to the adopted network architecture, batch size determines the number of training samples used in single parameter update, and epoch specifies the number of iterations.
Table Description of Hyperparameters in Experiments.
Dataset
CIFAR-10
CIFAR-100
ImageNet-IK ImngeNm
window
100
totwindow
200
200
300
400
d
10
20
40
50
0.001
0.001
0.0003
0.0001
urchitecture batch size
Con3
Cet3
ConvNet4
ConvNet5
1000
1000
500
250
epoch
600
600
1000
1000
C More Experimental Results
C.1 Hypothesis Verification on Tiny-ImageNet
We conduct similar verification experiments on Tiny-ImageNet [16] As shown in Fig. 6 the Tiny- ImageNet exhibits three-stage characteristics consistent with CIFAR-10: (1) During the early stage, models preferencely select preliminary truncation positions with larger window sizes; (2) middle stage demonstrates limited performance sensitivity to variations in both truncation positions and window sizes; (3) the late stage, models shift preference towards post truncation positions while window size adjustments show diminishing impact. This cross-dataset consistency substantiates the generalizability of our findings and confirms the effectiveness of the AT-BPTT across diverse datasets.
22

---

30-
25-
Trunration Poataiinoeaotooo
Azcunoe (55
Accanay
Party--raiinrarry
Midbll-Phalimirrry
Early Post
Middile Post
Late Pool
200
400
600
300
Farch
Eprach (b)
Epacc (c)
(+]
Early Stage
Midklle Stage
Late Stage
1so
Whrdow Stze
25
Acuaa (0
20
20-
tont--rratler
Miidle-Smleer Middie-Criginll
Late-Smaalll
Each Original
Lute- Original
Lange-Laarger
1000
2000
700
400
500
600
100
200
300
4001
500
100
100
200
500
4001
Sto)
600
Spoch
Epoch
Epodh
(d)
(e)
(f)
Figure 6: Hypothesis verification for the truncation strategies and window size on Tiny-ImageNe [6]. (a)()(c) show experiments where the preliminary or post truncation positions are implemented at early, middle and late stages, respectively, and (d)(e)(f) present experiments where the window size is changed after fixing the truncation position. For example, Early- Baryy-Prelimnnary in (a) means that randomly select preliminary phase (0-100) timesteps in early training stage (0-200 epochs).
Table 5: Computational efficiency and performance comparison with low-rank Hessian approximation (LRHA) CIFAR-10 when IPC=10.
Method
GPU Memory (GB)
Training Time (h)
Accuracy (%)
Memory Saving Speedup
RaT-BPTT 9
18.9
21.6
69.4
1.0x
AT-BPTT (wfo LRHA)
15.3
18.4
72.7
19.0%
1.17x
AT-BPTT (w/ LRHA)
6.94
5,5
72.4
63.3%
3.9x
C.2 Impact of Low-Rank Hessian Approximation
To rigorously evaluate the impact of our low-rank Hessian approximation (LRHA) mechanism, we conduct controlled experiments comparing three configurations CIFAR-10 10 with IPC=10: (1) Baseline RaT-BPTT with full Hessian computation, (2) AT-BPTT without LRHA, and (3) Full AT- BPTT with LRHA enabled. All experiments performed NVIDIA A800 GPUs with identical hyperparameters and ConvNet-3 architecture. The results Tab. Tab. demonstrate that LRHA provides substantial computational benefits while preserving model performance.
C.3
Generalization Experiments on Wide Networks
Previous studies 55 24 have sought to narrow the discrepancy between proxy training environments and real training scenarios by adopting wider network architectures. Notably, the RaT-BPTT [9] method similarly employs a paradigm of training narrow networks while evaluating on wide networks (4 times wider ConvNet). This study systematically evaluates the performance of AT- BPTT under wide network architectures through experiments. Experimental results as shown in demonstrate that AT-BPTT maintains its comprehensive leading advantage over RaT-BPTT in wide network configurations, outperforming RaT-BPTT by average of 3,4%. Compared with narrow network configurations, AT-BPTT achieves measurable performance improvements in wide architectures. This outcome successfully validates prior hypotheses and directly stems from AT- BPTT's phased training strategy with dynamically adjusted windows, confirming the effectiveness of its adaptive mechanism.
C.4 Comparison with Diffusion Model-Based Methods
In addition to optimization methods based on the inner- and outer-loop, generative model-based methods have been developed for synthesizing compact datasets. These methods leverage Generative Adversarial Networks (GANs) diffusion models to produce high-quality synthetic data, gaining
23

---

Table 6: Comparison with the baseline dataset distillation methods wide networks. The improve- ments denoted by red numbers are with respect to our baseline RaT-BPTT". Each reported result is the average of 5 experiments.
Dataset
CIFAR-10
CIFAR-100
Tiny IntyIemaeNNet
(Ig/class(PC))
KIP'
28
49.9
62.740.3
68.6*0.2
15.7=0.2
28.3+0.1
READ"
53.6+1.2
66.3+0.5
71.140.4
26.3=1.1
33.0+0.3
FRePO"
55
46.8+0.7
65.5+0.6
71.7=0.2
28.7=0.1
42.5+0.2
44.3+0.2
15.4+0.3
25.420.2
42.5
RCIG'
24
53.9&1.0
69.1+0.4
73.5+0.3
39.3=0.4
44.1a0.4
46.7a0.1
25.640.5
29.4
47.7
RaT-BPTT"
54.1#0.4
71.0+0.2
75.4+0.2
36.5=1.0
47,9+0.2
51.0#0.6
20.3+0.9
24.9+1.3
47.6
Ours
55.6+0.3
71.8+0.2
78.7+0.2
37.2+0.5
49,9+0.7
54.7+0.2
26.8+0.8
33.4+0.6
51.0
A%
+1.5
+3.3
+0.7
+3.7
+6.5
+8.5
+3.4
significant popularity and advancements in recent years. We select two leading diffusion model-based methods, D'M [34] and D [30), as baselines and evaluate the distillation performance of AT-BPTT under identical experimental settings. As shown in Tab. 7 AT-BPTT maintains leading position in distillation performance, achieving a average of 6.7% performance improvement across various datasets.
Table Comparison with decoupled optimization- distillation methods using CIFAR-10/100 and Tiny- Met datasets, with bolded values indicating the highest test accuracy.
Dataset
CIFAR-10
CIFAR-100
Tiny-ImageNet
Img/class(IPC)
10
50
10
50
1
10
50
D*N
56.2
45.0
48.8
46.8
D 30
50.2
67.8
74.4
29.8
46.6
51.2
16.7
26.1
30.1
Ours
54.4
72.4
78.7
36.9
49.0
55.9
24.3
32.7
48.7
C.5 Comparison with Baseline on Other Datasets
To further demonstrate the effectiveness of AT-BPTT on high-resolution dataset, we conduct experi- ments ImageNet subsets [13] including ImageNette, Image ImageMeow, and ImageFruit.
The experiments compare AT-BPTT with baseline methods, including Random MTT (2). RDED [6), FTD (8). DATM 12. EDF 40) and NCFM 44. As presented in the Tab. 8 AT-BPTT consistently achieves the highest accuracy across all datasets and IPC settings, for example, attaining 79. % on ImageNet (IPC=10) and 47.6% on ImageFruit (IPC=1). These results significantly outperform leading methods, demonstrating that AT-BPTT exhibits superior generalization and performance stability across diverse datasets and data scales. This advantage is particularly pronounced in data-constrained scenarios (IPC=1), underscoring its efficacy in image classification tasks.
Table 8: Comparison with baseline methods using ImageNette, Image Woof, ImageMeow, and ImageFruit dataset [13] with bolded values indicating the highest test accuracy.
Dataset
ImageNette
ImageWoof
ImageMeow
ImageFruit
(IP)
10
10
Random
S
23.5
47.7
14.2
27.0
13.8
29.0
13.2
21.4
NTT 2
47.7
63.0
28.6
35.8
30.7
40.4
26.6
40.3
33.8
63.2
18.5
40.6
FID S
52.2
67.7
30.1
38.8
33.8
43.3
29.1
44.9
DAIM 12
52.5
68.9
30.4
40.5
34.0
48.9
30.9
45.5
EDP
40
52.6
71.0
30.8
41.8
34,5
52.6
32.8
46.2
NCPM 44
53.4
77.6
27.2
48.4
34.6
58.2
29.2
44.8
Ours
31.8
60.8
C.6 Experiments Language Tasks
To assess AT-BPTT's generalization capability for language tasks, we evaluate our method on three standard text classification benchmarks: SST-2 [39], MNLI-m (39], and AGNews [50]. As

---

demonstrated in 9 AT-BPTT achieves consistent improvements over existing textual dataset distillation approaches, with an average accuracy gain of 3.2% across all evaluated datasets. The results demonstrate that our method is also effective on non-visual datasets.
Table 9: Comparison with the SOTA textual dataset distillation methods BERT 39
across SST-2
MNLI-m 39], and AGNews [50].
Dataset
SST-2 39
MNLI-m
AGNews 550
Img/cluss(IPC)
1
10
50
10
50
10
50
DDAL 26
64.5
69.7
73.5
34.1
38.8
41.5
26.1
28.9
34.2
DILM 27
72.5
76,3
80.3
39.7
44.8
48.7
27.8
30.9
36.5
DaLLME [37
72.3
77.5
80.9
42.7
47.2
51.4
27.6
30.8
36.6
Ours
73,2
79.4
82,4
44.9
48.1
53.2
28.8
31.1
39,3
Full dataset
92.7
86.7
94.6
D Difficulty Sample Conjecture
Our theoretical analysis reveals the learning characteristics of deep neural networks (DNN) M: during the initial training stage, the networks tend to rapidly capture easily identifiable simple patterns in data, while progressively shifting their focus to more complex and fine-grained feature repre- sentations as training advances. This discovery aligns theoretically with the data difficulty scoring mechanism proposed in PAD (2 based on the trajectories matching framework (2] Specifically, the PAD innovatively introduces a difficulty scoring function that effectively quantifies the learning complexity of data samples for DNNS, thereby establishing a dynamic curriculum learning mech- anism. Through intelligent scheduling strategies, this mechanism gradually introduces samples of increasing difficulty during expert trajectory training, demonstrating inherent consistency with the phased training paradigm of the AT-BPTT algorithm.
Building upon these theoretical connections, propose the following hypothetical improvement: systematically ordering the original dataset through the difficulty scoring function and combining it with phased progressive training strategy construct stepwise learning path from low-difficulty to high-diffiultyy samples. We suggest focusing on low-scoring samples during the initial training stage and progressively incorporating high-scoring complex samples model capability improves. This structured knowledge progression mechanism is expected to establish a more refined model learning paradigm, thereby enhancing overall training effectiveness. The development of a more sophisticated and refined data preprocessing framework presents promising direction for further investigation, which constitutes one of our key research priorities in future studies.
E
Discussion
Although we evaluate AT-BPTT in the context of dataset distillation, the proposed algorithm is generally applicable to a broad range of bilevel optimization problems that involve unrolled gradient- based inner loops. In particular, AT-BPTT can serve as drop-in replacement for standard or truncated BPTT in applications such as meta-learning logical reasoning
(25). tool learning (3). agent [42), differentiable neural architecture search (NAS), and personalized federated learning. These applications often rely on fixed-length unrolling or heuristic truncation strategies, which can be suboptimal or inefficient. In contrast, AT-BPTT adaptively adjusts both the truncation points and the weighting of intermediate gradients based on the optimization landscape, potentially leading to improved stability, reduced memory and compute costs, and better overall performance. We believe AT-BPTT offers principled and generalizable framework for efficient bilevel optimization, and we leave its integration into these broader domains promising directions for future work.
AT-BPTT also has some drawbacks that need be addressed. First, this inner-loop optimization inevitably involves unfolding the learning trajectory during training. Although we have simplified the computation of the Hessian matrix product using a low-rank Hessian approximation method, the computational cost remains significant, leading to higher time complexity compared to NCFM [44). Second, our base network is still limited to relatively simple models like CNNs, results in practical limitations. Therefore, our future work will focus improving the computational efficiency of
25

---

the algorithm or combining the inner- and outer-loop optimization to reduce the unfolding of the learning trajectory. and exploring the implementation of algorithm on larger-scale models, such as diffusion models and transformers.
F Visualizations of Distilled Dataset
In this section, present visualizations of the distilled datasets obtained from various datasets. The IPC the presented datasets is uniformly set to 10, with Fig. iillustrating the visualization results on CIFAR-10, 10, respectively.
Figure 7: Visualization results of our proposed method for CIFAR-10 [5 with IPC=10.
G Ethical Statement
This research proposes a general optimization algorithm for bilevel problems that improves the efficiency and adaptability of truncated backpropagation through time. The work was conducted with integrity, transparency, and academic rigor. No human subjects, sensitive data, or personally identifiable information were involved. The method does not raise concerns regarding fairness, bias, or misuse its current form. have also ensured reproducibility by providing full methodological details and plan to relcase code under an open-source license upon publication.
26

---

H
Broader Impacts
This paper introduces an adaptive and efficient method for truncated gradient backpropagation in bilevel optimization, which has the potential to significantly reduce computational costs in range of machine learning applications, including meta-learnin, federated learning. neural architecture search, and dataset distillation. The positive societal impact includes making advanced machine learning more accessible and sustainable by lowering the hardware and energy requirements of training large models. This aligns with ongoing efforts in the community to improve the environmental footprint of ML research.
27


Reviews:
[{"review_id": "KFFK6Frhbw", "reviewer": "Reviewer_aCEq", "summary": "This paper advances Rat-BPTT, by introducing an automatic approach to determine the alignment range. The authors propose a dynamic truncation position selector that uses gradient magnitudes to probabilistically choose where to start backpropagation, and also useing adaptive window sizing to adjusts the truncation window length. The method is further extended to high-resolution images with a patch-wise distillation strategy. Extensive experiments show that AT-BPTT achieves SOTA results among inner-loop methods and is highly competitive with other methods across benchmarks.", "questions": "1. Lack justification and ablation of split the learning process into three stages. What if we only define two stages, or more stages? Some ablation experiments would help.\n2. What is the actual lengths of the three stages for each experiment?\n3. What is the approximation accuracy of LRHA? Will the errors affect overall distillation performance?\n4. Why is the PSP module not working well with small resolution images.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n- The paper is well motivated. The authors find the core obstacle in RBTT paradigm, and propose automatic  solution to enhance the current random truncation solution.\n- The method achieves substantial improvements in both accuracy and computational efficiency.\n- The paper is very clearly written with sufficient implementation details. \n\n## Weaknesses\n\n\nMy main concern is the lack of justification to the  algorithm design. The motivation is sound, but the authors' solution is complex and some design choice is empirical. But overall, I tend towards acceptance due to the good motivation and performance.", "was_revised": true, "final_justification": "I appreciate the authors' feedback and my concerns are well addressed. The additional ablation is a strong complement to the paper. I would raise my rating to \"Accept\".", "rebuttal": "Thank you for appreciating our clear motivation, detailed presentation, and the significant accuracy and efficiency gains of our method. Below we will address specific questions.\n\n> ***W1: Lack of justification to the algorithm design***\n\nThank you for recognizing our strong motivation and gains. Based on Q1, we infer that your concerns come from the three-stage strategy, which we will address in the next section.\n\n> ***Q1: Justification and ablation of three stages***\n\nThank you for your suggestion regarding spliting the learning process into three stages. This is a question that interests Reviewers aCEq, zMmc and B86A. We address it with a Justification and Ablation Study:\n\n### Justification: \nBased on the theory in paper [1] that is cited as [1] in the main text (early stages prioritize simple patterns and later stages refine complex features), we initially divided the stages into two parts (early and late). However, such improvements in results are limited, as shown in the the table below. This inspires us to explore a more refined staged strategy. Additionally, according to paper [2] that is cited as [9] in the main text, overfocusing the truncted window on a certain stage is not conducive to global learning. Therefore, it is crucial to retain a period of random windowing to cover the entire trajectory. Together, a three-stage strategy is needed. According to the analysis of **Figures 1** and **3** in the main text, this extra stage is defined as the middle stage, and finally boosts performance. \n\nIn addition, why not more stages, such as four stages? First, each extra stage adds two new hyperparameters in the stage transition, which complicates tuning and deployment; Moreover, according to the analysis in **Figure 1** in the main text, middle-stage performance demonstrates negligible sensitivity to truncation position (lines 46). A finer partitioning thus produces negligible change. Therefore, we ultimately chose to use a three-stage strategy.\n\n### Ablation Study:\n\nWe conduct ablation experiments on three variants to verify the effectiveness of three-stage stragety. Using CIFAR-10 with IPC=10 (600 epochs) as an example, the setups are as follows:\n\n- **Two-stage variant:** The training is split into two stages of 300 epochs each. In the first stage, truncations position follow $ P_{trunc}(t) $ for early stage in **Equation 6**. In the second stage, truncation positions follow $ P_{trunc}(t) $ for late stage in **Equation 6**.\n- **Three-stage variant:** The training is split into three stages of 200 epochs each. The first and third stages are identical to the first and second stages of the *two-stage variant*, respectively. The second stage employs random truncation.\n- **Four-stage variant:** The training is split into four stages of 150 epochs each. The strategies for the first and third stages align with the first and second stages of the *two-stage variant*, while the second and fourth stages employ random truncation.\n\nThe experimental results are shown in the table below:\n\n| Two-stage variant | Three-stage variant | Four-stage variant |\n| :---------------: | :-----------------: | :----------------: |\n|     69.6±0.2      |      70.5±0.3       |      70.3±0.2      |\n\n\nTable above demonstrates that the three-stage variant consistently outperforms both other two variants. \n\nIn conclusion, both Justification and Ablation Study support our choice of a three-stage design as an effective and balanced truncation strategy.\n\n> ***Q2: Lengths of the three stages***\n\nTable below reports the average number of epochs in each stage for IPC = 10 across our four benchmarks.\n\n| Dataset | CIFAR-10 | CIFAR-100 | Tiny-ImageNet | ImageNet-1K |\n| :-----: | :------: | :-------: | :-----------: | :---------: |\n|   *IPC*   |    10    |    10     |      10       |     10      |\n|  Early  |   149    |    185    |      244      |     417     |\n| Middle  |   233    |    252    |      405      |     320     |\n|  Late   |   218    |    163    |      351      |     263     |\n\nAs detailed in **Threshold-guided Stage Transition** mechanism (lines 195-207), the transitions between stages are triggered automatically by the model's performance dynamics during training. Therefore, each stage’s duration naturally adapts to the dataset’s learning dynamics under a given IPC.\n\n\n> ***Q3: Impact of LRHA***\n\nWe have evaluated the impact of our LRHA in **Appendix C.2** (lines 538-544). As shown in **Table 5**, we test the experimental results before and after the introduction of LRHA, using CIFAR-10 IPC=10 as an example. After the introduction of LRHA, the accuracy is reduced from 72.7% to 72.4%, with a negligible drop of just 0.3 %. In contrast, LRHA cuts peak memory usage by 63.3 % and speeds up training by  $3.9 \\times$. These results demonstrate that LRHA introduces minimal approximation error while delivering substantial computational savings. Additionally, the experimental results reported in the main text are the results after applying LRHA.\n\n> ***Q4: Performance of PSP***\n\nWe appreciate your interest in the behavior shown in **Table 2**. In fact, PSP performs better on high-resolution datasets, while PSP has only negligible improvements on low-resolution datasets. This discrepancy stems from low-resolution patches containing too little semantic content for PSP to exploit.\n\nFor low-resolution images (e.g., CIFAR-10, $32 \\times 32$), PSP results in each patch being only $8 \\times 8$ pixels in size. An $8 \\times 8$ pixel block contains limited semantic information, making it difficult for the model to learn meaningful representations. Therefore, for datasets like CIFAR-10 or CIFAR-100, distilling the complete $32 \\times 32$ image directly is a much more effective strategy. For high-resolution images (ImageNet-1K, $224 \\times 224$), following the same $4 \\times 4$ division rule, a $224 \\times 224$ image is segmented into 4 patches, each of size $56 \\times 56$ pixels. A $56 \\times 56$ patch is large enough to preserve rich local structure critical for distillation. Therefore, PSP is a specialized component designed to tackle the challenges of distilling high-resolution images.\n\n\n>***Paper Formatting Concerns:***\n\nThank you for your suggestion on paper formatting. We will fix the $ P_{trunc} $ in **Equation 6** to $P*_{trunc}$.\n\n**References:**\n\n[1] A closer look at memorization in deep networks. ICML 2017.\n\n[2] Embarrassingly simple dataset distillation. ICLR 2024."}, {"review_id": "VgElSzTTCt", "reviewer": "Reviewer_KQDB", "summary": "This paper proposes Automatic Truncated Backpropagation through Time (AT-BPTT) to improve upon previous bilevel optimization methods by accounting for the training dynamics of both the inner loop and the outer loop.  To account for inner loop learning dynamics (i.e. learning the parameters of the ConvNet on the distilled data), the truncation window is determined probabilistically by utilizing the inner gradient magnitude and variations. The exact formulation of the probabilistic mechanism is depends on the training stage of the outer loop (i.e. learning the distilled data). Additionally, to reduce the memory overhead, a low Hessian approximation is used. Results on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K shows SOTA dataset distillation performance.", "questions": "1. How is the distilled data initialized?\n2. How does it perform in large IPC settings (50 pic) for ImageNet-1K?\n3. What does the distilled ImageNet data look like?\n4. How well does this method work using ResNets as the model for dataset distillation? These models tends to be more difficult due to the noisy gradients within the inner loop.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength\n1. Addressing the noisy nature of the inner loop in the bilevel optimization problem for dataset distillation is important and critically understudied. This paper provides some much needed insights on the training dynamics of bilevel optimized distilled data.\n2. The proposed algorithm AT-BPTT uses a novel idea of dynamically adjusting the inner loop behavior based of the magnitude and variation of inner loop gradients\n3. Low-rank Hessian approximation and Patch-wise Semantic Preservation are important contributions towards making dataset distillation less resource intensive \n4. The proposed AT-BPTT achieves SOTA performances in CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet-1K\n5. The ablation studies and evaluation is comprehensive. The paper show that the strategy for selecting window size and location as well as the patch-wise semantic preservations all contribute to the final improvement. Cross architecture generalization and high IPC experiments are also included in the appendix.\n6. Code is anonymized and provided\n \nWeaknesses and typos\n1. The presentation of the algorithm itself should be included in the main paper instead of appendix and can be improved: outline in Algorithm 1 is a bit confusing-the inner loop should be more explicitly written\n2. Missing BPTT citation in the context of dataset distillation [1]. \n3. Segmennt typo in Algorithm 1\n\n[1] Deng, Zhiwei, and Olga Russakovsky. \"Remember the past: Distilling datasets into addressable memories for neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 34391-34404.", "was_revised": true, "final_justification": "Authors answered my confusions and will maintain my score.", "rebuttal": "Thank you for acknowledging our novel dynamic adjustment, resource‑saving Hessian and semantic preservation contributions, SOTA results across benchmarks, and comprehensive evaluations with available code. Below we will address specific questions.\n\n\n> ***W1/W2/W3: Presentation of algorithm, missing citation and typo***\n\nThank you for your suggestions regarding the presentation of algorithm, missing citation and typo. If the paper is accepted, we will adjust the layout to place the algorithm in the main text; We will clearly indicate the positions of the inner and outer loops using comments; We will add the BPTT reference; We will correct the typo of \"Segmennt\" to \"Segment\".\n\n\n> ***Q1: Data initialization***\n\nWe have reported the data initialization procedure in **Appendix B** (lines 507-509). Specifically, We draw synthetic images from a standard Gaussian distribution and assign labels so that each class has an equal number of samples. Following this random generation, we apply a norm-normalization step to unit length. This initialization creates balanced, model‑agnostic starting data that is comparable across experiments.\n\n\n> ***Q2: Performance of high-IPC setting***\n\nWe appreciate your interest in the high‑IPC settings on ImageNet-1K. We have completed this experiment during the rebuttal period. With IPC=50, AT-BPTT achieves a test accuracy of 46.5% on ImageNet-1K. This accuracy is substantially higher than the 30.6% at the IPC=10 setting. It provides strong evidence that our method's performance scales robustly with an increasing IPC, rather than being effective only at low-IPC settings. Additionally, we have reported the results on other three datasets (CIFAR-10/100, Tiny-ImageNet) with high-IPC settings in **Appendix C.5** and **Table 8**.\n\n> ***Q3: Data visualization***\n\nWhile the rebuttal system does not support displaying figures, we will include a detailed ImageNet-1K data visualization in the final version, including:\n- We will present a side‑by‑side grid of $5 \\times 2$ distilled samples alongside Teddy [1] and other high‑resolution baselines.\n- Compared to low‑resolution distilled datasets, our ImageNet‑1K distillates exhibit sharper, more recognizable object outlines.\n- The backgrounds become washed‑out or texture‑like, since the model integrates more class features.\n\n\n\n> ***Q4: Performance on ResNets***\n\nWe have conducted experiments using the ResNet-18 architecture in **Appendix C.4** and **C.5**. \n\n- In **Appendix C.4** (lines 557-564), we have evaluated the distilled CIFAR-10 dataset with IPC=10 and 50 across four distinct architectures, including ResNet-18. As shown in **Table 7**, the performance of our method on ResNet-18 is lower than on ConvNet, which is the same as the judgement you mentioned. In this case, our AT-BPTT still outperforms all baselines, achieving the current SOTA accuracy of 75.6% under the IPC=50 setting. This provides strong evidence that AT-BPTT has excellent cross-architecture generalization.\n\n- In **Appendix C.5** (lines 565-578), we have further conducted experiments with high-IPC settings, consistently using ResNet-18. The results in **Table 8** demonstrate that AT-BPTT comprehensively surpasses all baselines, across CIFAR-10/100 and Tiny-ImageNet. \n\nIn conclusion, these findings confirm AT‑BPTT’s robust cross‑architecture generalization, even on ResNets.\n\n**References:**\n\n[1] Teddy: Efficient large-scale dataset distillation via taylor-approximated matching. ECCV 2024."}, {"review_id": "WMyFpReRT2", "reviewer": "Reviewer_zMmc", "summary": "This paper proposes AT-BPTT, a novel framework to enhance inner-loop optimization in dataset distillation. The key motivation stems from the observation that deep neural networks exhibit different learning behaviors at early, middle, and late training stages, making existing random or fixed truncation strategies suboptimal. AT-BPTT introduces a dynamic truncation strategy guided by gradient magnitude and training stage, an adaptive window size based on gradient variation to retain important updates, a low-rank Hessian approximation to reduce computation and memory costs, and a patch-wise semantic preservation module to enable effective distillation on high-resolution datasets.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, I did not identify any major weaknesses. This is a strong and well-executed paper with solid technical contributions and extensive empirical validation. While the core ideas extend prior work (e.g., RaT-BPTT) in a relatively incremental manner, the proposed framework integrates multiple innovations (e.g., dynamic truncation, adaptive window sizing, and low-rank Hessian approximation). The empirical results are compelling, and the method demonstrates strong scalability and practical efficiency across diverse datasets, including high-resolution benchmarks.\n\nMinor Weaknesses:\n1. The paper does not provide a detailed theoretical justification or ablation for how the three training stages (early, middle, late) are precisely defined or separated beyond empirical heuristics.\n\n2. The adaptive window size relies on gradient variation, but its parameterization (e.g., range of adjustment d) may require manual tuning per dataset; clearer guidelines or automated adaptation would improve practical usability.\n\n3. The method introduces several new hyperparameters (e.g., stage transition thresholds, temperature for softmax), which could limit its deployment in real-world settings without additional tuning efforts.\n\n4. The paper lacks comparison with recent SOTA generative dataset distillation methods (e.g., Minimax [1], IGD [2]).\n\n5. While image classification results are strong, it would strengthen the paper to show effectiveness on other downstream tasks such as neural architecture search and continual learning.\n\n[1] Efficient Dataset Distillation via Minimax Diffusion\n\n[2] Influence-Guided Diffusion for Dataset Distillation", "was_revised": true, "final_justification": "I have carefully read the authors' response as well as the comments from other reviewers. The detailed content has adequately addressed my concerns. I believe this is a strong paper in the field of dataset distillation and will maintain my original score of “Accept.” While the performance on ImageNet-1K is slightly below that of diffusion-based methods, the superior results on smaller-scale datasets are convincing. I recommend that the authors include these additional results in the revised version.", "rebuttal": "Thank you for highlighting our solid technical contributions, integrated innovations, extensive validation, and scalable efficiency across diverse datasets. Below we will address specific questions.\n\n> ***W1: Justification and ablation of three stages***\n\nThank you for your suggestion regarding spliting the learning process into three stages. This is a question that interests Reviewers aCEq, zMmc and B86A. We address it with a Justification and Ablation Study:\n\n### Justification: \nBased on the theory in paper [1] that is cited as [1] in the main text (early stages prioritize simple patterns and later stages refine complex features), we initially divided the stages into two parts (early and late). However, such improvements in results are limited, as shown in the the table below. This inspires us to explore a more refined staged strategy. Additionally, according to paper [2] that is cited as [9] in the main text, overfocusing the truncted window on a certain stage is not conducive to global learning. Therefore, it is crucial to retain a period of random windowing to cover the entire trajectory. Together, a three-stage strategy is needed. According to the analysis of **Figures 1** and **3** in the main text, this extra stage is defined as the middle stage, and finally boosts performance. \n\nIn addition, why not more stages, such as four stages? First, each extra stage adds two new hyperparameters in the stage transition, which complicates tuning and deployment; Moreover, according to the analysis in **Figure 1** in the main text, middle-stage performance demonstrates negligible sensitivity to truncation position (lines 46). A finer partitioning thus produces negligible change. Therefore, we ultimately chose to use a three-stage strategy.\n\n### Ablation Study:\n\nWe conduct ablation experiments on three variants to verify the effectiveness of three-stage stragety. Using CIFAR-10 with IPC=10 (600 epochs) as an example, the setups are as follows:\n\n- **Two-stage variant:** The training is split into two stages of 300 epochs each. In the first stage, truncations position follow $ P_{trunc}(t) $ for early stage in **Equation 6**. In the second stage, truncation positions follow $ P_{trunc}(t) $ for late stage in **Equation 6**.\n- **Three-stage variant:** The training is split into three stages of 200 epochs each. The first and third stages are identical to the first and second stages of the *two-stage variant*, respectively. The second stage employs random truncation.\n- **Four-stage variant:** The training is split into four stages of 150 epochs each. The strategies for the first and third stages align with the first and second stages of the *two-stage variant*, while the second and fourth stages employ random truncation.\n\nThe experimental results are shown in the table below:\n\n| Two-stage variant | Three-stage variant | Four-stage variant |\n| :---------------: | :-----------------: | :----------------: |\n|     69.6±0.2      |      70.5±0.3       |      70.3±0.2      |\n\n\nTable above demonstrates that the three-stage variant consistently outperforms both other two variants. \n\nIn conclusion, both Justification and Ablation Study support our choice of a three-stage design as an effective and balanced truncation strategy.\n\n\n> ***W2/W3: Hyperparameter fine-tuning***\n\nThank you and Review B86A for your suggestions regarding hyperparameter fine-tuning. Based on your comments and Reviewer B86A's W2/Q2, we provide following comprehensive reply.\n\nWe agree that clarifying the hyperparameter sensitivity, computational overhead, and practical strategies for setting these parameters is crucial for reproducibility and real-world deployment. Therefore, we provide clear guidelines to minimize tuning effort. Our analysis divides these hyperparameters into three categories based on dataset sensitivity, and we discuss empirical basis, practical strategy and computational cost.\n\n\n### Sensitivity Category\n- Low sensitivity: Fixed defaults suffice across datasets due to universal gradient dynamics\n- Medium sensitivity: Adjusted via simple scaling rules, minimizing need for per-dataset tuning\n- High sensitivity: Only M and N require dataset-specific tuning\n\n### Empirical Basis and Practical Strategy\n|Hyperparameter|Sensitivity|Empirical Basis|Practical Strategy|\n|:-:|:-:|:-:|:-:|\n|$\\tau$ (temperature for dynamic truncation and adaptive window)|Low Sensitivity|Gradient magnitude distributions are similar across dataset, which makes $\\tau$ less sensitive to dataset. We conduct ablation study from CIFAR-10 to ImageNet-1K with $\\tau$ = 1.0 without re-tuning. Accuracy varied <±0.5% when $\\tau$ is perturbed ±0.5 on CIFAR-10/ImageNet-1K.|$\\tau$ = 1.0 for all datasets.|\n|k_min (LRHA min rank)|Low Sensitivity|k_min is invariant to dataset. k_min = 1 worked for all tested datasets. Higher k_min will add unnecessary cost.|k_min = 1 for all datasets.|\n|k_max scale (LRHA max rank factor)|Low Sensitivity|Fixed scaling factor.|Not a tunable hyperparameter.|\n|$d$ (window adjustment range)|Medium Sensitivity|Window size exhibits a direct correlation with the total number of timesteps $T$. $T$ is related to dataset scale. Larger datasets require larger windows to capture complex dynamics. |$d \\in [10,50]$ for the scale of CIFAR-10 to ImageNet-1K|\n|X, Y (transition counters)|Medium Sensitivity|X and Y depend on training duration. We thus set X, Y proportionally to total epochs.|X = 15% of total epochs; Y = 12.5% of total epochs.|\n|M, N (accuracy variation thresholds)|High Sensitivity|M and N depend on convergence speed of dataset. Different datasets require different thresholds: e.g., M=1.5 for CIFAR-10, but M=1.0 for Tiny-ImageNet due to slower early convergence.|Ablation study in **Stage Transition Threshold Analysis** (lines 295-304).|\n\n### Computational Cost of Hyperparameter Tuning\n\nThe computational cost for tuning AT-BPTT hyperparameters is low, as most parameters are low-sensitivity or medium-sensitivity. As shown in table above, the costs of low-sensitivity or medium-sensitivity hyperparameters are negligible. The hyperparameter tuning is thus focused on M and N. And this process for other datasets is similar to that of ablation study in **Figure 5**. The computational costs on Nvidia A800 are shown in table below:\n\n| Dataset |CIFAR-10  | CIFAR-100| Tiny-ImageNet | ImageNet-1K| \n|:-:|:-:|:-:|:-:|:-:| \n|Time (h) | <25 | <27 | <40 | <80 | \n|Memory (GB) | <7 | <9 | < 18 | < 44 |\n\nIf the paper is accepted, we will provide the analysis above in the camera-ready version.\n\n\n> ***W4: Comparison with generative DD methods***\n\nThank you for your suggestion regarding the comparison with SOTA generative DD methods. We provide the experimental results in the table below:\n\n| Dataset |      | CIFAR-10 |      |      | CIFAR-100 |      |      | Tiny-ImageNet |      |     ImageNet-1K||\n |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n | IPC     | 1    | 10       | 50   | 1    | 10        | 50   | 1    | 10            | 50   |1|10|\n  | Minimax [3]     | -    | -    | -| -    |-     | - | - | -         |- |-|44.3|\n  | IGD  [4]   | -    | -        | 66.8 | -    | 45.8      | 53.9 | -    | -             | -    |-|46.2|\n  | Ours    | 54.4 | 72.4     | 78.7 | 36.9 | 49.0      | 55.9 | 24.3 | 32.7          | 48.7 |14.7|30.6|\n\nThrough comprehensive comparison, we get several conclusions:\n- The results on ImageNet-1K of Minimax and IGD are both 44.3 and 46.2, which are higher than our results. \n- These two generative methods can not generalize across datasets with various resolutions like CIFAR-10, CIFAR-100, and Tiny-ImageNet, while our AT-BPTT achieves comparable results across these datasets, especially in low-resolution datasets.\n- These two generative methods necessitate the incorporation of diffusion models for distillation, thereby incurring substantial GPU memory consumption and additional computational overhead. In contrast, our method with a LRHA module achieves a $3.9 \\times$ training speedup and 63% memory savings compared to the baseline.\n- As shown in **Table 8** in **Appendix C.5**, our AT-BPTT extends the scale of synthetic dataset to high-IPC cases and achieves expressive performance.\n\nIf the paper is accepted, we will add the new comparison table to the camera-ready version.\n\n\n> ***W5: Application of downstream tasks***\n\nThank you for highlighting the importance of downstream evaluations. As stated in **Appendix F Discussion**, we outline several downstream tasks as our future work, like neural architecture search and federated learning. To initially verify the effectiveness of our method for downstream tasks, we have extended language tasks in **Appendix C.8** and pretrained model fine-tuning in **Appendix C.9**. For instance, the results shown in **Figure 7** demonstrate that pre-training on data distilled by AT-BPTT significantly outperforms random selection. These experiments demonstrate that our method preserves critical representations that are transferable to complex downstream tasks beyond simple classification. Additionally, we further discuss AT‑BPTT’s theoretical benefits for diverse applications in **Appendix F Discussion**. Therefore, we will continue to explore AT-BPTT to other downstream tasks in future work.\n\n**References:**\n\n[1] A closer look at memorization in deep networks. ICML 2017.\n\n[2] Embarrassingly simple dataset distillation. ICLR 2024.\n\n[3] Efficient Dataset Distillation via Minimax Diffusion. CVPR 2024.\n\n[4] Influence-Guided Diffusion for Dataset Distillation. ICLR 2025."}, {"review_id": "swKS7zRpzT", "reviewer": "Reviewer_B86A", "summary": "This paper focuses on dataset distillation. This technique aims to create a compact dataset that preserves model accuracy. Dataset distillation is formulated as a bilevel optimization problem where the inner loop is responsible for training on a small, synthetic dataset, and the outer loop minimizes the generalization error on the target dataset. This paper builds on RaT-BPTT which proposed to truncate and apply a window on inner loop gradient calculation.\nThe authors propose to dynamically set the truncation position based on gradient magnitude. They also introduce an adaptive window sizing strategy determined by gradient variation. Finally, they propose using a low-rank Hessian approximation to increase training speed and reduce memory cost. Extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K show substantial gains compared to baseline methods.\n\nWe can formulate 2 claims from the paper:\n1. \"Sampling specific timesteps from distinct stages lead to **better** performance, rather than uniformly applying random truncation\" (n.b. optimal is misused in the paper)\n2. \"Adjusting the truncation window size further enhance distillation performance\"\n\nThen we can summarize the contributions as follows:\n1. Introduce a dynamic truncation strategy and window sizing based on outer loop stages\n2. Introduce a low-rank Hessian approximation to reduce memory and computational cost\n3. Patch-wise semantic preservation to ensure semantic consistency in high-resolution images", "questions": "Question 1: For clarity, could the authors explicitly define the scope of the 'Early,' 'Middle,' and 'Late' stages versus the 'Preliminary' and 'Post' phases? My understanding, inferred from the text (e.g., Section 4.1 lines 122-124 ), is that 'Early,' 'Middle,' and 'Late' refer to distinct stages within the outer-loop training process (epochs), while 'Preliminary' and 'Post' describe sub-phases within the unrolled learning trajectory of each inner-loop epoch (timesteps). Usage of \"inner-loop\" and \"outer-loop\" terminology would really help clarify this distinction.\n\nThen, if I'm not mistaken,\n\nQuestion 2: To strengthen the reproducibility and practical applicability of AT-BPTT, could the authors provide a more detailed analysis or discussion on the hyperparameter tuning process? Specifically, given the numerous new hyperparameters introduced (temperature τ and T for dynamic truncation, temperature τ and d for adaptive window size, thresholds M,N,X,Y for stage transition, and kmax​,kmin​ for LRHA ), how were these values determined for CIFAR-100, Tiny-ImageNet, and ImageNet-1K, considering the ablation studies in Section 5.4 only focus on CIFAR-10?\n\nIt is stated that AT-BPTT is 'automatic', yet the extensive set of tunable parameters suggests a potentially high computational cost and time investment for hyperparameter search on new datasets.  We would appreciate a discussion or, ideally, an empirical evaluation of:\n\n    The sensitivity of AT-BPTT's performance to these hyperparameters across different datasets.\n    The computational overhead associated with tuning these hyperparameters for each dataset.\n    Any strategies or heuristics employed to efficiently set these parameters for new datasets without extensive grid search, perhaps leveraging insights from the intrinsic gradient behaviors observed.\n\nThis would greatly clarify the true 'automatic' nature and practical deployment feasibility of the proposed framework.\n\nQuestion 3: In the experiments, the method \"Teddy\" seems to be performing well. Although, the authors do not mention Teddy in the related work section. It's also not compared in the table 7 which seems to be pretty close to the experiments in the Teddy paper. Could you detail how Teddy relates to your work and conducts additional experiments to compare with this paper?\n\nQuestion 4: In all the results, and espacially on Tiny Imagenet and Imagenet 1K in table 1 (but also in annexes), the authors should report the accuracies with and without patch-wise semantic preservation. They should also display the results of one of the methods (RaT-BPTT) with and without patch-wise semantic preservation. This would help to understand each individual contribution on high resolution datasets.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe results are quite significant, with AT-BPTT outperforming RaT-BPTT and other baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K.\n\nWeaknesses:\n\nClarity could be improved as mentioned on the questions below. It's not clear to me what is the link between paper cited as [1] on learning dynamics and the proposed method. It seems to be more of an intuitive connection rather than a formal one.\n\nDependence on hyperparameters is not well discussed. The paper claims that AT-BPTT is 'automatic', yet the extensive set of tunable parameters suggests a potentially high computational cost and time investment for hyperparameter search on new datasets.\n\nPatch-Wise semantic preservation is a bit \"on the side\" of the main contribution, and it is not clear how it relates to the main claims of the paper. The issue is that it's hard to differentiate between the contribution of the patch-wise semantic preservation and the dynamic truncation and adaptive window sizing. The paper does not provide a clear separation of these contributions, making it difficult to assess their individual impacts on performance.\n\nFormal issues:\nHypotheses given (1 and 2) are in fact research questions.", "was_revised": true, "final_justification": "I have engaged a conversation on papers weaknesses and the authors have addressed my concerns.", "rebuttal": "Thank you for recognizing the significant performance gains of AT‑BPTT. Below we will address specific questions.\n\n> ***W1/Q1: Clarity and concerns about the link***\n\nThank you for pointing out the need to clarify the definitions and the link between [1] and our method.\n\nWe appreciate that your interpretation is consistent with our formal definitions of stages and phases. For clarity, we will provide the descriptions below to explicitly define three stages and two phases, and these will be provided in the main text:\n- **Outer-loop Training Stages:** We use **Early**, **Middle**, and **Late** to describe the global stages of the entire dataset distillation process. These stages are counted by epochs.\n- **Inner-loop Learning Phases:** We use '**Preliminary**' and '**Post**' to describe sub-phases within the unrolled learning trajectory. These phases are counted by timesteps.\n\nAdditionally, we are sure that there is a profound inherent link between the paper [1] and our method. The essence of our three-stage division lies in focusing on the learning priorities of the model at different stages. This aligns exactly with the conclusion in [1]: early stages prioritize simple patterns and later stages refine complex features. Based on Reviewer aCEq's Q1, we can confidently state that our method not only has a strong formal link with paper [1], but also extends its theoretical framework to dataset distillation.\n\n> ***W2/Q2: Hyperparameter fine-tuning***\n\nThank you and Review zMmc for your suggestion regarding hyperparameter fine-tuning. Based on your comments and Reviewer zMmc's W2/W3, we offer the following comprehensive reply.\n\nWe agree that clarifying the hyperparameter sensitivity, computational overhead, and practical strategies for setting these parameters is crucial for reproducibility and real-world deployment. Therefore, we provide clear guidelines to minimize tuning effort. Our analysis divides these hyperparameters into three categories based on dataset sensitivity, and we discuss empirical basis, practical strategy and computational cost.\n\n\n### Sensitivity Category\n- Low sensitivity: Fixed defaults suffice across datasets due to universal gradient dynamics\n- Medium sensitivity: Adjusted via simple scaling rules, minimizing need for per-dataset tuning\n- High sensitivity: Only M and N require dataset-specific tuning\n\n### Empirical Basis and Practical Strategy\n|Hyperparameter|Sensitivity|Empirical Basis|Practical Strategy|\n|:-:|:-:|:-:|:-:|\n|$\\tau$ (temperature for dynamic truncation and adaptive window)|Low Sensitivity|Gradient magnitude distributions are similar across dataset, which makes $\\tau$ less sensitive to dataset. We conduct ablation study from CIFAR-10 to ImageNet-1K with $\\tau$ = 1.0 without re-tuning. Accuracy varied <±0.5% when $\\tau$ is perturbed ±0.5 on CIFAR-10/ImageNet-1K.|$\\tau$ = 1.0 for all datasets.|\n|k_min (LRHA min rank)|Low Sensitivity|k_min is invariant to dataset. k_min = 1 worked for all tested datasets. Higher k_min will add unnecessary cost.|k_min = 1 for all datasets.|\n|k_max scale (LRHA max rank factor)|Low Sensitivity|Fixed scaling factor.|Not a tunable hyperparameter.|\n|$d$ (window adjustment range)|Medium Sensitivity|Window size exhibits a direct correlation with the total number of timesteps $T$. $T$ is related to dataset scale. Larger datasets require larger windows to capture complex dynamics. |$d \\in [10,50]$ for the scale of CIFAR-10 to ImageNet-1K|\n|X, Y (transition counters)|Medium Sensitivity|X and Y depend on training duration. We thus set X, Y proportionally to total epochs.|X = 15% of total epochs; Y = 12.5% of total epochs.|\n|M, N (accuracy variation thresholds)|High Sensitivity|M and N depend on convergence speed of dataset. Different datasets require different thresholds: e.g., M=1.5 for CIFAR-10, but M=1.0 for Tiny-ImageNet due to slower early convergence.|Ablation study in **Stage Transition Threshold Analysis** (lines 295-304).|\n\n### Computational Cost of Hyperparameter Tuning\n\nThe computational cost for tuning AT-BPTT hyperparameters is low, as most parameters are low-sensitivity or medium-sensitivity. As shown in table above, the costs of low-sensitivity or medium-sensitivity hyperparameters are negligible. The hyperparameter tuning is thus focused on M and N. And this process for other datasets is similar to that of ablation study in **Figure 5**. The computational costs on Nvidia A800 are shown in table below:\n\n| Dataset |CIFAR-10  | CIFAR-100| Tiny-ImageNet | ImageNet-1K| \n|:-:|:-:|:-:|:-:|:-:| \n|Time (h) | <25 | <27 | <40 | <80 | \n|Memory (GB) | <7 | <9 | < 18 | < 44 |\n\nIf the paper is accepted, we will provide the analysis above in the camera-ready version.\n\n>***W3/Q4: Ablation study of PSP (or LRHA?)***\n\nThank you for your comments on the Patch-wise Semantic Preservation (PSP) module. Our intention is not to design PSP as an inseparable component of AT-BPTT, but rather as a plug-and-play paradigm solving the challenges when applied to high-resolution data. To clearly differentiate the individual impact of three components, we have already conducted a detailed ablation study (lines 283-294), with the results presented in **Table 2**. In fact, the results of Tiny-ImageNet and ImageNet-1K are included in **Table 2**. Additionally, the results of RaT-BPTT with and without PSP are also included in the second and fifth lines in **Table 2**, since RaT-BPTT is our backbone.\n\nHowever, based on the questions you've raised, particularly \"the authors should report the accuracies with and without patch-wise semantic preservation\", we infer that there might be a typo in your comments. Could you please clarify whether you are referring to the Low-rank Hessian approximation (LRHA) instead of PSP in W3/Q4?\n\nTo fully address your concerns, we have further conduct relevant experiments for LRHA that might meet your requirements. Since we have conducted its ablation study for low-resolution datasets in **Appendix C.2**, the results shown below are the extention of **Table 5**. \n\n| Dataset       | IPC | Method            | GPU Memory (GB) | Training Time (h) | Accuracy (%) |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Tiny-ImageNet |   10  | RaT-BPTT    | 43.4    | 37.6    | 24.2   |\n|     |     | RaT-BPTT(w/ LRHA)   | 24.7    | 13.6     | 23.3    |\n| |   | AT-BPTT(w/o LRHA) | 38.7    | 28.5    | 33.2   |\n|     |     | AT-BPTT(w/ LRHA)  | 17.6   | 7.8    | 32.7  |\n| ImageNet-1K  |  10   | RaT-BPTT  | 85.4   | 71.4      | 12.8  |\n|    |     | RaT-BPTT(w/ LRHA)          | 51.4       | 22.5       | 11.2   |\n|    |   | AT-BPTT(w/o LRHA) | 72.8    | 62.4       | 31.4         |\n|      |     | AT-BPTT(w/ LRHA)  | 43.3    | 15.6      | 30.6         |\n\nTable above demonstrates that LRHA can significantly reduce memory and training time while only slightly reducing accuracy. If the paper is accepted, we will provide the ablation experiments above in the camera-ready version.\n\n>***Q3: Relationship and comparison with Teddy***\n\nThank you for your suggestion to add details about Teddy [2] method. Teddy is an interesting DD method, and both it and our method have their own strengths. We will elaborate on the relationship and comparison between Teddy and our work.\n\n### Relationship:\nBoth methods have made improvements to reduce computational resource consumption in the meta-learning optimization objective. For the second-order derivative in the meta-learning objective, Teddy converts it to a first-order derivative using a Taylor approximation, while AT-BPTT approximates the Hessian matrix generated in the second-order derivative using the LRHA module.\nThe methodologies of the two are complementary. Teddy uses Taylor approximation and model pooling to optimize global distillation, making it suitable for distillation with large-scale data; while AT-BPTT uses dynamic truncation to focus on fine-grained control of training dynamics, making it suitable for various inner-loop optimization scenarios.\n\n### Comparison:\nSince Teddy is not open-sourced, we use the results of our reproduction for Teddy. We compare Teddy with our AT-BPTT in three aspects: test accuracy, cross-architecture, and high-IPC settings.\n\n- **Test accuracy:** As shown in **Table 1**, although our method is lower than Teddy on high-resolution datasets, it significantly outperforms Teddy on various low-resolution datasets. \n\n- **Cross-architecture:** Under similar settings in **Appendix C.4**, we conduct cross-architecture evaluations for Teddy and ours. Table below shows that AT-BPTT consistently outperforms Teddy across four tested architectures.\n\n| Architecture | ConvNet |      | VGG11 |      | AlexNet |      | ResNet18 |      |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| *IPC*          | 10      | 50   | 10    | 50   | 10      | 50   | 10       | 50   |\n| Teddy        | 53.0    | 66.1 | 46.2  | 57.7 | 39.3    | 55.4 | 51.8     | 63.4 |\n| Ours         | 72.3    | 78.9 | 68.9  | 77.1 | 68.5    | 76.3 | 69.3     | 75.6 |\n\n- **High-IPC settings:** Under similar settings in **Appendix C.5**, we conduct evaluations in high-IPC settings for Teddy and ours. Table below shows that AT-BPTT consistently outperforms Teddy across various high-IPC cases.\n\n| Dataset |   | CIFAR-10 |  |  |  | CIFAR-100 |  |  | Tiny-ImageNet |  |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|   *IPC*   |  250 |    500   | 1000 | 1500 |  25  |     50    |  100 |  150 |       50      |  100 |\n|  Teddy  | 81.7 |   84.2   | 86.5 | 90.4 | 51.5 |    54.1   | 61.9 | 66.7 |      45.8     | 51.3 |\n|   Ours  | 89.4 |   93.3   | 94.9 | 95.3 | 66.1 |    72.9   | 75.8 | 76.5 |      57.3     | 60.7 |\n\n>***W4: Formal issues***\n\nThank you for your suggestion regarding the format. Hypotheses (1) and (2) will be more appropriately expressed as research questions; we will fix \"optimal\" to \"better\" in the main text.\n\n**References:**\n\n[1] A closer look at memorization in deep networks. ICML 2017.\n\n[2] Teddy: Efficient large-scale dataset distillation via taylor-approximated matching. ECCV 2024."}]

Author rebuttal:

Meta-review: This paper introduces AT-BPTT, a novel framework for efficient backpropagation featuring several key innovations: a probabilistic stage-aware timestep selector, an adaptive window size strategy driven by gradient variation, a low-rank Hessian approximation (LRHA) for computational efficiency, and a patch-wise semantic preservation (PSP) module for handling high-resolution data. Comprehensive experiments on multiple image datasets demonstrate that AT-BPTT achieves a significant average accuracy improvement of over 6%, alongside a 3.9x speedup in optimization and a 63% reduction in memory cost.

The reviewers' primary concerns centered on the motivation for the proposed three-stage splitting strategy and the transferability of the method's hyperparameters across different models and datasets. The authors' rebuttal effectively addressed these points by providing additional ablation studies and a detailed cross-dataset analysis, which satisfied all reviewers. Further concerns regarding comparisons with state-of-the-art data distillation algorithms, the individual impact of the LRHA and PSP components, and the role of high Images-Per-Class (IPC) were also adequately clarified.

Given the persuasive author response, all reviewers have updated their scores to reflect a strong consensus for acceptance. This paper presents a method of considerable technical novelty, demonstrated through rigorous empirical validation and shown to have broad applicability. The AC strongly recommends acceptance.
Decision: Accept (poster)