Title: FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Efficient Learning

Dataset Distillation

Data-Level Residual Connections
Abstract: Residual connection has been extensively studied and widely applied at the model architecture level. However, its potential in the more challenging data-centric approaches remains unexplored. In this work, we introduce the concept of ***Data Residual Matching*** for the first time, leveraging data-level skip connections to facilitate data generation and mitigate data information vanishing. This approach maintains a balance between newly acquired knowledge through pixel space optimization and existing core local information identification within raw data modalities, specifically for the dataset distillation task. Furthermore, by incorporating training-time refinements, our method significantly improves computational efficiency, achieving superior performance while reducing training time and peak GPU memory usage by 50\%. Consequently, the proposed method **F**ast and  **A**ccurate  **D**ata **R**esidual **M**atching for Dataset Distillation (**FADRM**) establishes a new state-of-the-art, demonstrating substantial improvements over existing methods across multiple dataset benchmarks in both efficiency and effectiveness. For instance, with ResNet-18 as the student model and a 0.8\% compression ratio on ImageNet-1K, the method achieves 48.4\% test accuracy in single-model dataset distillation and 50.9\% in multi-model dataset distillation, surpassing RDED by +6.4\% and outperforming state-of-the-art multi-model approaches, EDC and CV-DD, by +2.3\% and +4.9\%.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FADRM that introduces data-level residual connections to preserve important information during synthetic data generation. \nThe FADRM combines multi-resolution optimization and adjustable residual merging to improve both efficiency and accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n**S1**: The extension of the residual mechanism from model-level to data-level is conceptually novel.\n\n**S2**: The method achieves impressive performance across various datasets and architectures.\n\n\nWeaknesses:\n\n**W1**: \nFigure 2 is not clear.  \nWhether every **Data Residual Block** includes a **Recover**? \nWhy **Downsample** upsize the image size in **Resampler**, and vice versa?  \nThis increases the difficulty of understanding. \n\n**W2**: \nThis paper claims FADRM achieves Fast and Accurate matching.\nRDED is a relevant and efficient baseline, which is missing from Figure 1 and Table 2.\n\n**W3**:\nHow does the author explain the observation in Figure 3, where information vanishing still occurs even with ARC? Furthermore, why is there such a big shock compared to without ARC? This raises questions about whether ARC introduces instability into the distillation process. These phenomena should be discussed in detail.\n\n**W4**: \nTheorem 2 relies on the assumption in Eq.(8), which is crucial for concluding that FADRM generalizes better.  \nHowever, the paper does not provide empirical validation or intuitive justification for when this assumption holds.  \nThus, the theoretical advantage is conditional and its practical applicability is unclear.\nThe paper reports accuracy and runtime but never measures either side of Eq. (8), such as empirical Rademacher estimates, margin distributions or PAC-Bayes surrogates.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies the information vanishing problem in uni-level frameworks and addresses it with Adjustable Residual Connection (ARC), which applies the idea of model-level skip connections at the data level. It also introduces Multi-Resolution Optimization (MRO), which distills datasets at lower resolutions to improve training efficiency. Mixed Precision Training (MPT) is used to further reduce training time and memory use. The proposed method, FADRM, achieves state-of-the-art results in both accuracy and efficiency across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n(+) The paper provides a clear theoretical justification and empirical validation for the phenomenon of information vanishing in uni-level frameworks, effectively defining and analyzing this critical issue.\n\n(+) The authors present an original approach by interpreting model-level skip connections as data-level skip connections, successfully preserving critical information from the original dataset during the distillation process.\n\nWeakness\n\n(-) It remains unclear whether the reported improvements in computational efficiency attributed to MRO are genuinely derived from this approach itself, or if they primarily result from MPT.\n\n(-) Although Table 3 demonstrates notable performance gains when comparing FADRM to SRe^2L++, the reported efficiency and GPU memory are same. Thus, it is questionable whether FADRM truly offers substantial efficiency enhancements over SRe^2L++.\n\n(-) Table 1 indicates minor performance gains for ImageNet-1K with ResNet-18. However, without reported statistical variability from multiple runs, commonly provided in other baselines, assessing the statistical significance of these improvements and making a rigorous comparison with baselines is difficult.\n\nQUESTIONS:\nCould the authors clarify why the evaluation criteria used for time efficiency comparisons differ between Table 2 (left) and Table 3?\nLines 289-291 indicate that MRO reduces peak memory usage. However, Table 3 shows that MPT alone reduces peak memory to 2.9GB. Could the authors clarify whether the observed memory reduction is indeed due to MRO rather than MPT?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel data residual matching strategy for database distillation, aiming to significantly reduce training time and memory usage. The approach demonstrates substantial improvements in both computational efficiency and model performance, reducing resource demands by 50% compared to baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**(S1)** The proposed method yields clear performance gains over baseline distillation techniques.\\\n**(S2)** Training time and memory consumption are significantly reduced, making the approach highly practical for real-world applications where efficiency is crucial.\n\n\n**Weaknesses**\n\n**(W1)** There seems a labelling error in Figure 2, where the “downsample” and “upsample” lables in the resampler module appear to be mistakenly swapped.\\\n**(W2)** The evaluated task is not clearly stated. While it appears to be image classification, this must be explicitly mentioned for clarity.\\\n**(W3, Major)** The models used for training and evaluation are not clearly described. Table 1 suggests the use of ResNet-based architectures, but the paper should clearly state which models were used and whether the approach generalizes to non-ResNet architectures. It is unclear whether the selection here is related to the design of the residual blocks in the proposed framework. This lack of clarity raises a major concern regarding the paper’s methodology.\\\n**(W4 - Minor)** Figure 2 could be misleading, as it shows an equal number of patches for the original and distilled datasets. Typically, the distilled set should contain significantly fewer samples.\\\n**(W5)** The peak memory results for FADRM+ in Tables 2 and 4 differ, while those for the other models remain consistent. Could you clarify the reason for this inconsistency?\n\nQUESTIONS:\n**(Q1)** What task was concerned here for performance evaluation, and why was it chosen? Was it image classification?\\\n**(Q2)** What models were trained using the distilled datasets? Were the training configurations (e.g., hyperparameters, epochs) kept consistent across all methods/models?\n\nLIMITATIONS:\nThe paper does not clearly discuss its limitations. In particular, the generalizability of the method to architectures beyond ResNet is a concern that should be addressed, especially when the proposed approach may rely on residual connections.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FADRM (Fast and Accurate Data Residual Matching), a novel dataset distillation framework that extends residual connections to the data level to address the issue of information vanishing in synthetic data. The method uses data-level skip connections and multi-resolution optimization (MRO) to preserve both local and global information while significantly reducing memory and time costs. By injecting residuals from the original patches during the iterative optimization process, FADRM ensures better feature retention and faster convergence. Empirical results across CIFAR-100, Tiny-ImageNet, and ImageNet-1K show FADRM's advantage in accuracy, efficiency, and cross-architecture generalization.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The core idea of extending residual connections from model-level to data-level is novel. It aims to address the information vanishing issue common in uni-level dataset distillation.\n\nS2: The proposed Multi-Resolution Optimization (MRO) and Mixed Precision Training (MPT) reduce computational costs and makes the method scalable to large datasets like ImageNet-1K.\n\nS3: The method consistently outperforms prior SOTA on several benchmarks across both accuracy and efficiency, with ablations and cross-architecture generalization evaluations.\n\nW1: The method requires pretrained feature extractors (with fixed BatchNorm) to work. It’s unclear how robust the method is if the pretrained model is weak or mismatched to the downstream task.\n\nW2: The theorems give motivation and justification for ARC (and MRO), but they aren't tightly integrated into the method design or used to guide any decisions so it reads a bit post-hoc. It would help if the authors can discuss how to make ARC adaptive or grounded the merge ratio in theory or training signals.\n\nW3: The paper doesn't discuss where FADRM fails or when ARC/MRO might hurt performance. This makes the method look overfitted to certain hyperparameters.\n\nW4: The paper talks a lot about information vanishing and density but lacks a well-defined, generalizable metric or visualization to evaluate it quantitatively.\n\nQUESTIONS:\nHow sensitive is the method to the choice of merge ratio and number of residual injections, and could these parameters be learned or adapted during training?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research theoretically points out that the mutual information between original dataset and synthetic dataset generated by uni-level decoupled dataset distillation methods has upper bound. To mitigate this problem, this research suggests the FADRM framework, which leverages the data-level skip connection concept. FADRM framework has two components: Multi-resolution optimization (MRO) to enhance efficiency and Adjustable residual connection (ARC) to mitigate the information vanishing. Through the diverse experimental results, FADRM shows the efficiency and effectiveness improvement.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The theoretical analysis of the uni-level decoupled framework in Theorem 1 is impressive because it has an intuitive interpretation and clearly shows the limitations of uni-level methods.\n\n2. The basic concept of FADRM, Data-level skip connection is a novel idea that has not been proposed.\n\n3. This research has conducted many detailed experiments to prove the effectiveness of the proposed idea and shows high efficiency and performance improvement.\n\n**Weaknesses**\n1. Although this research have theoretically analyzed the limitations of the uni-level decoupled framework, the authors have not theoretically demonstrated the effectiveness of the proposed methodology FADRM in the same framework.\n\n2. (Minor) The simple application of Mixed Precision Training (MPT) for the sake of efficiency falls short in terms of originality and contribution.\n\n3. (Minor) If I understand correctly, budget $b$ or $\\mathcal{B}$ refers to the optimization steps. However, in dataset distillation, budget usually refers to storage budget in relation to IPC, so this usage can be confusing and I suggest replacing it with the appropriate word.\n\nQUESTIONS:\nI like the paper overall and would be happy to raise my score if the authors address the few weaknesses and questions.\n\n1. Is there any reason why $\\tilde{x}$ should be optimized in the form of Cross-entropy + BN matching in Theorem 1? I also checked the proof in the appendix, but that condition is not used. I wonder if $f_\\theta$ trained and frozen with $\\mathcal{O}$ is sufficient to use the data processing inequality.\n\n2. Any further clarification on Theorem 2 would be appreciated. There is no sufficient interpretation of Theorem 2, while it contains many statements.\n\n3. I am curious about the advantages of MRO compared to synthetic image parameterization methods [1,2,3] and whether the FADRM framework can be used with synthetic image parameterization. I speculate that the synthetic image parameterization method might bypass the problem of carefully tuning the down-sampled size $D_{ds}$ of MRO as claimed by the authors. In particular, DDiF [4], with its resolution-agnostic nature, can enable a dynamically adjusted $D_{ds}$ rather than a static predefined $D_{ds}$.\n\n[1] Dataset Condensation via Efficient Synthetic-Data Parameterization\n\n[2] Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks\n\n[3] Dataset Distillation via Factorization\n\n[4] Distilling Dataset into Neural Field\n\nLIMITATIONS:\nSee weaknesses and questions", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}