Title: Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dataset distillation

Inner-loop optimization
Abstract: The growing demand for efficient deep learning has positioned dataset distillation as a pivotal technique for compressing training dataset while preserving model performance. However, existing inner-loop optimization methods for dataset distillation typically rely on random truncation strategies, which lack flexibility and often yield suboptimal results. In this work, we observe that neural networks exhibit distinct learning dynamics across different training stages—early, middle, and late—making random truncation ineffective. To address this limitation, we propose Automatic Truncated Backpropagation Through Time (AT-BPTT), a novel framework that dynamically adapts both truncation positions and window sizes according to intrinsic gradient behavior. AT-BPTT introduces three key components: (1) a probabilistic mechanism for stage-aware timestep selection, (2) an adaptive window sizing strategy based on gradient variation, and (3) a low-rank Hessian approximation to reduce computational overhead. Extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K show that AT-BPTT achieves state-of-the-art performance, improving accuracy by an average of 6.16\% over baseline methods. Moreover, our approach accelerates inner-loop optimization by 3.9 × while saving 63\% memory cost.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on dataset distillation. This technique aims to create a compact dataset that preserves model accuracy. Dataset distillation is formulated as a bilevel optimization problem where the inner loop is responsible for training on a small, synthetic dataset, and the outer loop minimizes the generalization error on the target dataset. This paper builds on RaT-BPTT which proposed to truncate and apply a window on inner loop gradient calculation.\nThe authors propose to dynamically set the truncation position based on gradient magnitude. They also introduce an adaptive window sizing strategy determined by gradient variation. Finally, they propose using a low-rank Hessian approximation to increase training speed and reduce memory cost. Extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K show substantial gains compared to baseline methods.\n\nWe can formulate 2 claims from the paper:\n1. \"Sampling specific timesteps from distinct stages lead to **better** performance, rather than uniformly applying random truncation\" (n.b. optimal is misused in the paper)\n2. \"Adjusting the truncation window size further enhance distillation performance\"\n\nThen we can summarize the contributions as follows:\n1. Introduce a dynamic truncation strategy and window sizing based on outer loop stages\n2. Introduce a low-rank Hessian approximation to reduce memory and computational cost\n3. Patch-wise semantic preservation to ensure semantic consistency in high-resolution images\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe results are quite significant, with AT-BPTT outperforming RaT-BPTT and other baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K.\n\nWeaknesses:\n\nClarity could be improved as mentioned on the questions below. It's not clear to me what is the link between paper cited as [1] on learning dynamics and the proposed method. It seems to be more of an intuitive connection rather than a formal one.\n\nDependence on hyperparameters is not well discussed. The paper claims that AT-BPTT is 'automatic', yet the extensive set of tunable parameters suggests a potentially high computational cost and time investment for hyperparameter search on new datasets.\n\nPatch-Wise semantic preservation is a bit \"on the side\" of the main contribution, and it is not clear how it relates to the main claims of the paper. The issue is that it's hard to differentiate between the contribution of the patch-wise semantic preservation and the dynamic truncation and adaptive window sizing. The paper does not provide a clear separation of these contributions, making it difficult to assess their individual impacts on performance.\n\nFormal issues:\nHypotheses given (1 and 2) are in fact research questions.\n\nQUESTIONS:\nQuestion 1: For clarity, could the authors explicitly define the scope of the 'Early,' 'Middle,' and 'Late' stages versus the 'Preliminary' and 'Post' phases? My understanding, inferred from the text (e.g., Section 4.1 lines 122-124 ), is that 'Early,' 'Middle,' and 'Late' refer to distinct stages within the outer-loop training process (epochs), while 'Preliminary' and 'Post' describe sub-phases within the unrolled learning trajectory of each inner-loop epoch (timesteps). Usage of \"inner-loop\" and \"outer-loop\" terminology would really help clarify this distinction.\n\nThen, if I'm not mistaken,\n\nQuestion 2: To strengthen the reproducibility and practical applicability of AT-BPTT, could the authors provide a more detailed analysis or discussion on the hyperparameter tuning process? Specifically, given the numerous new hyperparameters introduced (temperature τ and T for dynamic truncation, temperature τ and d for adaptive window size, thresholds M,N,X,Y for stage transition, and kmax​,kmin​ for LRHA ), how were these values determined for CIFAR-100, Tiny-ImageNet, and ImageNet-1K, considering the ablation studies in Section 5.4 only focus on CIFAR-10?\n\nIt is stated that AT-BPTT is 'automatic', yet the extensive set of tunable parameters suggests a potentially high computational cost and time investment for hyperparameter search on new datasets.  We would appreciate a discussion or, ideally, an empirical evaluation of:\n\n    The sensitivity of AT-BPTT's performance to these hyperparameters across different datasets.\n    The computational overhead associated with tuning these hyperparameters for each dataset.\n    Any strategies or heuristics employed to efficiently set these parameters for new datasets without extensive grid search, perhaps leveraging insights from the intrinsic gradient behaviors observed.\n\nThis would greatly clarify the true 'automatic' nature and practical deployment feasibility of the proposed framework.\n\nQuestion 3: In the experiments, the method \"Teddy\" seems to be performing well. Although, the authors do not mention Teddy in the related work section. It's also not compared in the table 7 which seems to be pretty close to the experiments in the Teddy paper. Could you detail how Teddy relates to your work and conducts additional experiments to compare with this paper?\n\nQuestion 4: In all the results, and espacially on Tiny Imagenet and Imagenet 1K in table 1 (but also in annexes), the authors should report the accuracies with and without patch-wise semantic preservation. They should also display the results of one of the methods (RaT-BPTT) with and without patch-wise semantic preservation. This would help to understand each individual contribution on high resolution datasets.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AT-BPTT, a novel framework to enhance inner-loop optimization in dataset distillation. The key motivation stems from the observation that deep neural networks exhibit different learning behaviors at early, middle, and late training stages, making existing random or fixed truncation strategies suboptimal. AT-BPTT introduces a dynamic truncation strategy guided by gradient magnitude and training stage, an adaptive window size based on gradient variation to retain important updates, a low-rank Hessian approximation to reduce computation and memory costs, and a patch-wise semantic preservation module to enable effective distillation on high-resolution datasets.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I did not identify any major weaknesses. This is a strong and well-executed paper with solid technical contributions and extensive empirical validation. While the core ideas extend prior work (e.g., RaT-BPTT) in a relatively incremental manner, the proposed framework integrates multiple innovations (e.g., dynamic truncation, adaptive window sizing, and low-rank Hessian approximation). The empirical results are compelling, and the method demonstrates strong scalability and practical efficiency across diverse datasets, including high-resolution benchmarks.\n\nMinor Weaknesses:\n1. The paper does not provide a detailed theoretical justification or ablation for how the three training stages (early, middle, late) are precisely defined or separated beyond empirical heuristics.\n\n2. The adaptive window size relies on gradient variation, but its parameterization (e.g., range of adjustment d) may require manual tuning per dataset; clearer guidelines or automated adaptation would improve practical usability.\n\n3. The method introduces several new hyperparameters (e.g., stage transition thresholds, temperature for softmax), which could limit its deployment in real-world settings without additional tuning efforts.\n\n4. The paper lacks comparison with recent SOTA generative dataset distillation methods (e.g., Minimax [1], IGD [2]).\n\n5. While image classification results are strong, it would strengthen the paper to show effectiveness on other downstream tasks such as neural architecture search and continual learning.\n\n[1] Efficient Dataset Distillation via Minimax Diffusion\n\n[2] Influence-Guided Diffusion for Dataset Distillation\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Automatic Truncated Backpropagation through Time (AT-BPTT) to improve upon previous bilevel optimization methods by accounting for the training dynamics of both the inner loop and the outer loop.  To account for inner loop learning dynamics (i.e. learning the parameters of the ConvNet on the distilled data), the truncation window is determined probabilistically by utilizing the inner gradient magnitude and variations. The exact formulation of the probabilistic mechanism is depends on the training stage of the outer loop (i.e. learning the distilled data). Additionally, to reduce the memory overhead, a low Hessian approximation is used. Results on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K shows SOTA dataset distillation performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. Addressing the noisy nature of the inner loop in the bilevel optimization problem for dataset distillation is important and critically understudied. This paper provides some much needed insights on the training dynamics of bilevel optimized distilled data.\n2. The proposed algorithm AT-BPTT uses a novel idea of dynamically adjusting the inner loop behavior based of the magnitude and variation of inner loop gradients\n3. Low-rank Hessian approximation and Patch-wise Semantic Preservation are important contributions towards making dataset distillation less resource intensive \n4. The proposed AT-BPTT achieves SOTA performances in CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet-1K\n5. The ablation studies and evaluation is comprehensive. The paper show that the strategy for selecting window size and location as well as the patch-wise semantic preservations all contribute to the final improvement. Cross architecture generalization and high IPC experiments are also included in the appendix.\n6. Code is anonymized and provided\n \nWeaknesses and typos\n1. The presentation of the algorithm itself should be included in the main paper instead of appendix and can be improved: outline in Algorithm 1 is a bit confusing-the inner loop should be more explicitly written\n2. Missing BPTT citation in the context of dataset distillation [1]. \n3. Segmennt typo in Algorithm 1\n\n[1] Deng, Zhiwei, and Olga Russakovsky. \"Remember the past: Distilling datasets into addressable memories for neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 34391-34404.\n\nQUESTIONS:\n1. How is the distilled data initialized?\n2. How does it perform in large IPC settings (50 pic) for ImageNet-1K?\n3. What does the distilled ImageNet data look like?\n4. How well does this method work using ResNets as the model for dataset distillation? These models tends to be more difficult due to the noisy gradients within the inner loop.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper advances Rat-BPTT, by introducing an automatic approach to determine the alignment range. The authors propose a dynamic truncation position selector that uses gradient magnitudes to probabilistically choose where to start backpropagation, and also useing adaptive window sizing to adjusts the truncation window length. The method is further extended to high-resolution images with a patch-wise distillation strategy. Extensive experiments show that AT-BPTT achieves SOTA results among inner-loop methods and is highly competitive with other methods across benchmarks.

STRENGTHS AND WEAKNESSES:
## Strengths

- The paper is well motivated. The authors find the core obstacle in RBTT paradigm, and propose automatic  solution to enhance the current random truncation solution.
- The method achieves substantial improvements in both accuracy and computational efficiency.
- The paper is very clearly written with sufficient implementation details. 

## Weaknesses


My main concern is the lack of justification to the  algorithm design. The motivation is sound, but the authors' solution is complex and some design choice is empirical. But overall, I tend towards acceptance due to the good motivation and performance.

QUESTIONS:
1. Lack justification and ablation of split the learning process into three stages. What if we only define two stages, or more stages? Some ablation experiments would help.
2. What is the actual lengths of the three stages for each experiment?
3. What is the approximation accuracy of LRHA? Will the errors affect overall distillation performance?
4. Why is the PSP module not working well with small resolution images.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03