FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation
Authors: Jiacheng Cui
Xinyue Bi
Yaxin Luo
Xiaohan Zhao
Jiacheng Liu
Zhiqiang Shen
Venue: NeurIPS 2025 poster ()
Keywords: Efficient Learning
Dataset Distillation
Data-Level Residual Connections
Primary area: general_machine_learning
TL;DR: We propose a data-centric residual matching method that significantly improves dataset distillation efficiency and accuracy.

Abstract: Residual connection has been extensively studied and widely applied at the model architecture level. However, its potential in the more challenging data-centric approaches remains unexplored. In this work, we introduce the concept of ***Data Residual Matching*** for the first time, leveraging data-level skip connections to facilitate data generation and mitigate data information vanishing. This approach maintains a balance between newly acquired knowledge through pixel space optimization and existing core local information identification within raw data modalities, specifically for the dataset distillation task. Furthermore, by incorporating training-time refinements, our method significantly improves computational efficiency, achieving superior performance while reducing training time and peak GPU memory usage by 50\%. Consequently, the proposed method **F**ast and  **A**ccurate  **D**ata **R**esidual **M**atching for Dataset Distillation (**FADRM**) establishes a new state-of-the-art, demonstrating substantial improvements over existing methods across multiple dataset benchmarks in both efficiency and effectiveness. For instance, with ResNet-18 as the student model and a 0.8\% compression ratio on ImageNet-1K, the method achieves 48.4\% test accuracy in single-model dataset distillation and 50.9\% in multi-model dataset distillation, surpassing RDED by +6.4\% and outperforming state-of-the-art multi-model approaches, EDC and CV-DD, by +2.3\% and +4.9\%.

FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation
Jiacheng Cui Xinyue Bi² Yaxin Luo', Xiaohan Zhao', .Jacheng Liu', Zhiqiang Shen
1 VILA Lab, MBZUAI
University of Ottawa
Equal Contribution *Corresponding Author
Code: FADRM (GitHub)
Abstract
Residual connection has been extensively studied and widely applied at the model architecture level. However, its potential in the more challenging data-centric approaches remains unexplored. In this work, we introduce the concept of Data Residual Matching for the first time, leveraging data-level skip connections facilitate data generation and mitigate data information vanishing. This approach maintains a balance between newly acquired knowledge through pixel space opti- mization and existing core local information identification within raw data modal- ities, specifically for the dataset distillation task. Furthermore, by incorporating training-time refinements, our method significantly improves computational effi- ciency, achieving superior performance while reducing training time and peak GPU memory usage by 50%. Consequently, the proposed method Fast and Accurate Data Residual Matching for Dataset Distillation (FADRM) establishes new state- of-the-art, demonstrating substantial improvements over existing methods across multiple dataset benchmarks in both efficiency and effectiveness. For instance, with ResNet-18 as the student model and 0.8% compression ratio on ImageNet-Ie the method achieves 48.4% test accuracy in single-model dataset distillation and 50.9% in multi-mode dataset distillation, surpassing RDED by +6.4% and outperforming state-of-the-es multi-model approaches, EDC and CV-DD, by +2.3% and +4.9%
Introduction
FADRM (Ours) #FADRM (Ours) -SRe2L
EDC +G-VDSM
In recent years, the computer vision and natural language processing communities have predom- inantly focused model-centric research, driv- ing an unprecedented expansion in the scale of
Accuracy (%)
neural networks. Landmark developments such as LLMs and MILMs in ChatGPT [26. 1], Gem-
ini [36]. DeepSeek [19] and other large-scale foundation models have shown the tremendous potential deep learning architectures. How- ever, as these models grow in complexity. the dependency on high-quality, richly informative datasets has become increasingly apparent, set- ting the stage for paradigm shift towards data- centric approaches. Historically, the emphasis on building bigger and more complex models has often overshadowed the critical importance of the data. While model-centric strategies have
IPC-1 IPC-10
IPC-50
100
150
200
250
Training Time (GPU Hours)
Figure 1: Total training hours single RTX- 4090 vs. test set accuracy, comparing prior state- of-the-art methods with our proposed framework (+ denotes modei distillation).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Original Patches
Distilled Images
Data Residual Block
Residual Block
Data Residual Block
Recover
200=200
224x224
224x224
Recover
Resampler
Residnal
Poorppe
Danmamypl
Resádual
Residual
Residual Merge
Dowmsangle
Lee +
Original Pradaos
update
Rreg
200-200
200-200
Figure 2: Overview of FADRM. It starts by downsampling the real data patches (both I and 2x2 [33] can be used as initialization and perform well in our experiments, meanwhile imposing downsampling to reduce cost). These downsampled images are subsequently processed through series of proposed Data Residual Blocks. Each block utilizes pretrained model to optimize the images within a predefined optimization budget, resamples them to a target resolution, and incorporates residual connections from the original patches via mixing ratio a. Finally, the images undergo an additional recovery stage, without residual connections, to produce the final distilled data.
delivered impressive results, they tend to overlook the benefits of optimizing data quality, which essential for achieving higher performance with lower data demands. Recent advancements in data-centric research highlight the importance of improving information density, reducing the volume of required data, and expediting the training process of large-scale models, thus presenting a more holistic approach performance enhancement.
Within this evolving landscape, dataset distillation [39], also called dataset condensation [14, 47, 43] has emerged as pivotal area of research. The goal of dataset distillation is to compress large-scale datasets into smaller, highly informative subsets that retain the essential characteristics of the original data. This approach not only accelerates the training process of complex models but also mitigates the storage and computational challenges associated with massive datasets. Despite significant progress, many existing tate-of-thheart methods in dataset distillation still struggle with issues related to scalability, generalization across diverse data resolutions, realism and robustness.
While residual connections have been well studied and widely implemented in the model architecture design field, primarily to prevent gradient vanishing and ensure effective feature propagation, their potential within data-centric paradigms remains largely unexplored. At the model level, residual connections help maintain the flow of gradients and enable deeper network architectures. contrast, at the data level, similar connections can potentially prevent the loss of critical original dataset information and improve scalability and generalization across architectures during the data distillation process. This observation and design introduce novel perspective on leveraging residual mechanisms beyond traditional model optimization, especially in the challenging domain of dataset optimization.
In this work, we introduce for the first time the concept of Data Residual Matching for dataset distillation. Our approach leverages data-level skip connections, novel idea for data-centric task to prevent real data information vanishing in multi-block data synthesis architecture. We call our method Fast and Accurate Data Residual Matching (FADRM), which, as shown in Fig. employs multi-resolute
image recovery scheme that utilizes image resolution shrinkage and expansion a residual manner, thereby capturing fine- graigga details and facilitating the recovery of both global and local information. This balance between newly acquired knowledge through pixel space

---

optimization and the preservation of existing core local information within raw data modalities marks significant advancement in dataset distillation. By integrating these data-level residual connections. our approach enhances the generalization and robustness of the distilled datasets.
Exhaustive empirical evaluations of proposed FADRM CIFAR-100 [15], Tiny-ImageNer [16), ImageNet-IK K [8] and its subset demonstrate that it not only accelerates the dataset distillation process by 50% but also achieves superior accuracy that beats all previous state-of-the-ar methods on both accuracy and generation speed. This approach effectively bridges the gap between model-centric and data-centric paradigms, providing a robust solution to the challenges inherent in -igh-quaitty data generation. Our contributions in this paper are as follows:
We extend conventional residual connection from the model level the data level area, and present for the first time simple yet effective, theoretically grounded residual connection design for data generation enhance data-centric task.
We introduce novel dataset distillation framework based on the proposed data residual matching, incorporating multi-scale residual connections in data synthesis to improve both efficiency and accuracy.
Our approach achieves state-of siiee-ofthle of-the-art results across multiple datasets, such as CIFAR- 100, Tiny-Imagel and ImageNet- K, while being more efficient and requiring less computa- tional cost than all previous methods.
2 Related Work
Dataset Distillation aims to synthesize a compact dataset that retains the critical information of larger original dataset, enabling efficient training while maintaining performance comparable to the full dataset. Overall, the matching criteria include Meta-Model Matching [39, 25, 23, 49, 9, 12]. Gradient Matching [47, 45, 18, 14, 48], Trajectory Matching [4. 7, 5, 10]. Distribution Matching [46, 38, 21, 17, 27, 32, 411, and Uni-level Global Statistics Matching [43, 30, 31, 42, 6, 40]. Dataset distillation on large-scale datasets has recently attracted significant attention from the community. For a detailed overview, it can be referred to the newest survey works [29, 20] on this topic.
Efficient Dataset Distillation. Several methods improve the computational efficiency of dataset distillation. TESLA [7] accelerates MTT [4] via batched gradient computation, avoiding full graph storage and scaling to large datasets. DM [46] sidesteps bi-level optimization by directly matching feature distributions. L. [43] adopts a Uni-Levvl Framework that aligns synthetic data with pretrained model statistics. G- G-VBSM [30] extends this by using lightweight model ensembles. EDC [31] further boosts efficiency through real data initialization, accelcrating convergence.
Residual Connection in Network Design. Residual connections have played a pivotal role in advancing deep learning. Introduced ResNet [11] to alleviate vanishing gradients, they enabled deeper networks by improving gradient flow. This idea was extended in iiceptionReesee [34] through multi-scale feature integration, and further generalized in DenseNet [13] via dense connectivity and feature reuse. Residual designs have also been central to Transformer architectures [37].
3 Approach
Preliminaries. Let the original dataset be denoted by O {( y:) and let the goal of dataset
distillation be to construct compact synthetic dataset C = [(²,, 0;)} the model for trained on C exhibits similar generalization behavior to the model foo trained on O. This objective can be formulated as minimizing the performance gap over the real data distribution:
} j=1 with [C] < |O), such that
[c|
min sup
|C [foo (z). v) L (ffe
(1)
C,CC (a.w)~0
where the parameters to and 6e obtained via empirical risk minimization:
to = arg min EmiinE [l 0
6c = arg min E EannE
(2)
The goal is to generate C in order to maximize model performance with minimal data. Among existing methods, notable class directly optimizes synthetic data without access to the original dataset,
3

---

referred to uni-level optimization. While effective, this approach faces two key limitations: (1) progressive information loss during optimization, termed information vanishing. and (2) substantial computational and memory costs for large-scale synthesis, limiting real-world applicabilityy.
Information Vanishing. In contrast images distilled using bi-level frameworks, the information content in images generated by uni-level methods (e.g., EDC [31]) is fundamentally upper- bouered as the original dataset the information density initially increases but eventually deteriorates due to the accumulation of local feature loss. This degradation leads information vanishing (see Fig. 3), which significantly reduces the fidelity of the distilled images and limits their effectiveness in downstream tasks.
utilized during synthesis (see Theorem 1). As optimization progresses,
Theorem 1 (Proof in Appendix A.2). Let fo denote a pretrained neural network on the original dataset O with fixed parameters and corresponding BatchNorm statistics BN and BN Let be a synthetic sample obtained by minimizing general objective L(fo; x) that relies exclusively information extracted from the pretrained model fo (e.g., its predictive output distribution or internal feature statistics alignment), that is, 3 arg min, L(fo; Define
H(fo)
sup H(fo(z)) xEsupp(O)
as the maximum per-sample Shannon entropy of the model's output. Then, the mutual information between the optimized distilled dataset C = {(0}, 05) 0;)1 and the original dataser is bounded by
T(C: S |C| H()))
(4)
The insight of this theorem is that if the pretrained model fo overly confident on all inputs (low maximum entropy), then ((fe) is small, and thus the distilled set, no matter how we optimize it, cannot encode large amount of information about O.
6.60
ARC Momeuts
6.55
Average Information Score
3.2
Information Density
3.1
3.0
SRe2L
W/O ARC
2.9
FADRM
6.35
100
500
1000
1500
2000 2500 3000 3500 4000
2.8
200
600
1000
Optimization Step
Class Index
Figure 3: The above figures illustrate the phenomenon of Information Vanishing. The Left Figure shows the evolution of information density across optimization steps, quantified through feature-level entropy using a pretrained ResNet-18 [11], comparing uni-level optimization (W/O ARC) with our FADRM (W/ ARC). The gray lines highlight the information density enhancement achieved through residual connection. The Right Figure shows the comparison of information scores (higher is better) across different classes, measured by pixel-level entropy, among FADRM. SRe*L++, and RDED. All experiments are conducted on distilled ImageNet- IK dataset with IPC=10.
Computational Challenges. Although uni-level frameworks exhibit scalability to large-scale datasets, the overall time required to generate a large distilled dataset remains prohibitively expensive. As illustrated in Fig. 1, EDC [31] requires nearly 70 hours to generate 50 IPC distilled dataset, which limits its applicability in contexts involving repeated runs, large-scale data synthesis, comprehensive empirical analysis. This motivates the need for more computationally efficient optimization strategies.
3.1 Overview of FADRM
The proposed FADRM framework, illustrated in Fig. 2 and detailed in Algorithm 1, addresses the limitations of existing uni-level optimization frameworks by integrating three proposed components: (1) MPT: a preciion training scheme that accelerates optimization and reduces computation

---

by casting model parameters to lower-precision formats, (2) MRO: multi-resolution optimization that improves computational efficiency, and (3) ARC: an adjustable embedded residual mechanism designed to seamlessly integrate essential features from the original dataset. This framework ensures both efficiency and generation fidelity in the optimization process.
Algorithm 1 FADRM: Residual Matching for Dataset Distillation
Require: Recover Model Sa, Total Training Iters Nier Real Patches P., Merge Ratio a, Downsam- pled Resolutions Da. Original Resolutions Dorig, Number of ARCs k:
Ensure: Distilled image Nia
1: Piter f [Nerr/ (Nie/(k++)).. (k+1)l,
To < RESAMPLE(P.. Das)
2: for 1 to & do
3:
for == 1 to fliter do
4:
1)imi))mar+t t GRADSTHP(Fo:: (i- Optimize & to align the property of fo 5: end for
RESAMPLE Timner Dorig), if Shape( Dds
6: Firtzer RESAMPLEE (Frsmne: Das); otherwise
7:
Tinm t arTi + (1-a) RESAMPLE(P, Swwpshappppppee (Binee)
8: end for
9:
for 1= to Niter- - kTiiter do
10:
Tkeze +tt GRADSTEP((Rr (xmnatt-1)
11: end for
12: return N
3.2 Mixed Precision Training for Data Generation
Previous uni-level frameworks typically retain a fixed training pipeline, seeking efficiency through architectural or imitialization-levee changes. In contrast, we explicitly optimize the training process by incorporating Mixed Precision Training (MPT) [24]. Specifically, we convert the model parameters from FP32 to FP16 and utilize FP16 for both logits computation and cross-entropy loss evaluation. To preserve numerical stability and ensure accurate distribution matching, we retain the computation of the divergence to the global statistics (Appendix D). as well as the gradients of the total loss with respect to 7 in FP32. By integrating MPT, our framework significantly reduces both GPU memory consumption and training time by approximately 50%, thereby significantly enhancing efficiency.
3.3 Multi-resolution Optimization
Optimization (MRO) enhances computational efficiency by optimizing images across multiple resolutions, unlike conventional methods that operate fixed input size. Naturally, low-resolution inputs can reduce computational cost for the model, they often come at the expense of performance. To mitigate this, our method periodically increases the data resolution back at specific stages, resulting in mixed--rsollution optimization process, as illustrated in Fig. 2 (bottom- right). This approach is particularly beneficial for large-scale datasets (e.g., ImageNet- IK), where direct high-resolutie significantly with input size for large datasets but remains stable for smaller ones (input size & 64). Thus, MRO is applied exclusively to large-scale datasets, as downscaling offers no efficiency gains for smaller ones. Specifically, given an initialized image P. E Pus x Dou XC first downsample it into a predefined resolution Das utilizing bilinear interpolation (detailed in Appendix E):
optimization is computationally inefficient. Notably, optimization time scales
Zg = Resample(P... D&), Das < Donig
(5)
The downscaled images ãn are optimized over Witter [Niter/ /Niner(k+11 [k+1)J iterations, yielding the refined result Enzz. Subsequently, Tra. is upscaled to its original resolution,
Traa = Resamplel (i Darig)
The upscaled image T.y.. is further optimized within the same budget Nitrr to recover information lost during the downscaling and upscaling processes. This iterative procedure (downscaling optimization
5

---

and upscaling optimization) is repeated until the total optimization iteration Nier is exhausted. To ensure MRO attains efficiency gains without quality loss, selecting an appropriate Da is crucial. Too small a Das causes information loss, while too large yields minimal efficiency improvement. Hence, Das should be carefully balanced for efficiency and effectiveness.
Saved Computation by MRO. Assume the forward computation cost scales as O(D²C). The baseline method performs all Niter steps at full resolution Dorig. yielding: Costbaseline N ( C). FADRM performs k alternating-resolurio stages of Thiter = [Nierr steps, with approximately half at downsampled resolution Das Let r (orig) The cost ratio is:
COStMRO
Titer
([ (1 - (1-r))
Costraseline
Niter
Under fixed Nier and k, the cost ratio decreases linearly with 1 - Smaller (i.c., more aggressive downsampling) yields greater savings, but may compromise data fidelity.
3.4 Adjustable Residual Connection
In uni-level optimization, the absence of the original dataset leads to information vanishing which significantly degrades the feature repre- sentation of the distilled dataset. To mitigate this issue, we introduce Adjustable Residual Con- nection (ARC), a core mechanism that mitigates information vanishing (see Fig. 3) and improves the robustness of the distilled data (see Theo- rem 2 and Theorem 3). Essentially, ARC iter- atively fuses the intermediate optimized image i E IR D2 x D C at iteration t with the resized initialized data patches Pt, which contain subtle details from the original dataset. Formally, the update rule is defined as:
Volcano Airliner Samoyede Ice Bear
a a 0.7 a 0.8 a 0.9 a= 1.0
Ge = a. + (1 a:) Resample D.)
(8)
Figure 4: Visualization of the distilled images with varying merge ratios using FADRM.
where [0, 1] is tunable merge ratio gov- erning the contribution of original dataset infor-
mation. A smaller strengthens the integration of details from P., whereas a larger a prioritizes the preservation of the global features in the é, This trend is visualized Fig. 4. ARC introduces a hyperparameter k, which determines the frequency of residual injections. Given total optimization budget of Niee. the training process is divided into k: + 1 segments, where residual connections occur /Nuer (k: + 1)J iterations. The update follows:
after every Niter
P iaar Resample (P Dinn..) Tirner
a. (1 a irme
where i e 21, k} denotes the index of the residual injection stage, and Dinee indicates the spatial resolution of the intermediate image the corresponding iteration in final phase involves pure optimization without further residual injections. Notably, ARC conducts per-element weighted fusion of two image tensors with negligible overhead. With complexity of O OHHWW,C). scales linearly with pixel and channel counts, making it suitable for high-resolution data.
Theorem 2 (Proof in Appendix A.3.1). Let H be a class of functions h R2d + R. and let be Lipschizz-contiuuuss with constant Ln. > O, and the loss function é be Lipschiz--onninnon with constant L > and bounded within a finite range [0, B). Consider:
Optimized per- 2. residual injected dataset 3. patches selected from the original dataset: O = =xx, Vi y # + 13pes - xzll. Let hres E H. denote the hypothesis trained on Čres, and hEADRM H be trained CFADRM- Define the corresponding empirical risks: Eros: := E (ātee) jee)), LEADRM: == D ((FFADRM (ā:), Di). Assume:
turbation added to the original data: Črea = {ires, " gre*] (FADRM): ČFADRM = {āi, Di h
discrepancy = 1
Ri, (H O) < Rr Ōroo)
(10)

---

Then the generalization bound of hEADRM is rigorously shown to be righter than that of free. i.e.,
ÉAADRM + 2B RR, CFADRM) < <res 2B R., o (ree).
(11)
The key insight of this theorem is that when synthetic data is heavily optimized, thereby increasing the complexity the hypothesis space and making the model more susceptible to overfitting, combining with the more structured original data can yield tighter generalization bound. We further demonstrate the advantage of FADRM over the Uni-level framework through Theorem 3.
Theorem (Proof in Appendix A.3.2). Let Emi denote the image generated by the uni-level frame- work, and fudmm denote the image generated by FADRM using both fa and real image patches O. Then the mutual information between the generated image and the original data satisfies,
( O) > [Grui O)-
This theorem shows that FADRM's improvement primarily arises from its direct access to the original dataset during optimization, thereby enhancing the amount of captured information.
Experiments
4.1 Datasets and Experimental Setup
Datasets. We conduct experiments across datasets with varying resolutions, including CIFAR-100 (32x32) [15], Imay- (64 (64x64) [16), ImageNet- IK (224x224) [8], and their subsets.
Baseline Methods. To evaluate the effectiveness of our proposed framework, we conduct a compre- hensive comparison against three state-of-the-art dataset distillation baselines. The first baseline is RDED [33], which selects cropped patches directly from the original dataset and is therefore catego- rized as involving full participation of the original data. The second method, EDC [31], retains high degree of original data participation by optimizing selected patches with an extremely small learning rate, producing synthetic images that are close to the original samples. The third method, CV-DD [6], aligns global BatchNorm statistics with sufficient optimization by updating initialization, resulting in minimal original data involvement despite initialization from real patches. These baselines exhibit varying degrees of original data involvement, providing solid basis for evaluating FADRM.
4.2
Main Results
Results Analysis. As shown Table our framework consistently achieves state-of-the-an perfor- mance across various settings. For instance, on ImageNet-1K with IPC=10 and ResNet-101 as the student model, the ensemble-enhanced variant FADRM+ attains an accuracy 58. %, outperform- ing EDC and CV-DD by a substantial margin of +6.4%. Notably, RDED underperforms FADRM, underscoring the limitations of relying solely on the original dataset without further optimization. Furthermore, CV-DD is inferior to FADRM+, highlighting the drawbacks of largely excluding origi- nal data during synthesis. Lastly, the consistent outperformance of FADRM+ over EDC validates the efficacy of our framework in harnessing original data via data-level residual connections.
Information Vanishing. To quantify the issue of informa- tion vanishing, we report the generalization accuracy of models trained on distilled images generated by FADRM (w/ and w/o ARC) across different optimization steps. As
Iterations
w/ ARC (%)
w/o ARC (%)
500
40.0
39.7
1000
45.2
44.8
1500
47.2
46.5
shown in Table 1, prolonged optimization without ARC leads to accuracy degradation, indicating information loss.
2000
47.7
46.1
2500
In contrast, applying ARC stabilizes the optimization pro- cess and maintains steady performance gains, highlighting its effectiveness in mitigating information vanishing and retaining essential representational cues.
3000
48.0
45.4
3500
48.2
44.5
4000
48.4
43.1
Table 1: Generalization accuracy (%) across varying optimization steps.
Efficiency Comparison. Table 3 (Left) highlights the superior efficiency of our framework compared to existing Uni-level frameworks. Bi-level frameworks are excluded from this comparison due to their inherent limitations scalability for large-scale datasets. Specifically, FADRM+ achieves reduction of 3.9 seconds per image in optimization time
7

---

RosNee 18
ResNer S0
ResiNes 101
(Ratio)
ROLD
CV-DD
FADRMG
FADRN
BXDRM-
CVDD FADRM FADRM-
(0.2%)
17.1
29.7
28.3
31.8
40.6
10.9
36.1
28.7
27.3
57.4
11.2
32.3
29,0
29.2
48.1
CIFAR-I0
50 (30.09%)
(2.0%%
56.9
66.8
63.7
68.6
62.7
67.1
67.4
11.0
71.3
67.9
64.0
41.6
62.1
69.4
68.2
61.5
66.5
71.5
72.1
67.4
54,1
67.9
68.5
61.7
63.8
67.6
08.3
71.9
68
72.1
Whole Datsser
79,9
79.5
(50)
11.8
19.2
28.6
40.4
35.0
18.4
40.6
78.0
27.9
Tay ImageNet
(10.0%)
(2.0%%
58.2
41.9
51.2
57.2
47.8
56.1
56.4
48.9
58.7
$2.8
45.6
38.4
50.2
58.8
54.7
43.8
42.3
57.0
53.7
09.3
22.9
41.2
51.6
58.6
47.6
54.1
47.8
57.2
68.8
53.6
Whole Duttset
68.9
71,5
7006
(0.11
35.8
59.2
31.1
25.3
26.3
ImngeNent
50 (1.09%)
69.0
54.0
61.0
61.9
50.0
Whole Dunset
93.8
(0.T5))
20.8
21.4
21.0
12.8
17.8
19.1
19.5
19.9
19.6
19.9
20.0
21.8
Woog
50 (2.35% 3%5
(1.155)
68.5
36.5
71.9
49.3
72.3
44.5
72.6
57.3
67.0
35.2
71.2
47.8
71.0
44.9
71.7
54.1
59.1
31.3
69.9
42.6
40.4
70.3
70.6
51.4
Whole Datsact
88.2
77.8
82.7
(0.158) 183)
6.6
14.7
13.3
12.2
7.0
ImageNer-1K IK
(3.9%%
(0.85)
56.5
42.0
38.0
48.6
46.0
39.5
60.1
48
61.2
50.9
49.7
62.0
64.3
63.9
05.6
57.5
66.9
61.2
64,9
51.7
62.7
51.7
66.0
54.8
67.0
Whole Dumzee
78.6
79.8
Table 2: Post-evaluation performance comparison with SOTA baseline methods. All experiments follow the training settings established in EDC [31]: 300 epochs for Tiny-ImageNet (IPC=10. 50), ImageNet- IK, and its subsets, and 1,000 epochs for CIFAR- 100, Tiny-ImageNet (IPC=1). For fair comparison with single-model distillation (RDED) and ensemble-basee based methods (CV-DD, EDC), we evaluate both the single- model version (FADRM only utilized ResNet- 18 [11] for distillation) and the mnsemble-enhanced enhanced version (FADRM+). This evaluation strategy ensures equitable benchmarking while maintaining methodological consistency across all experiments.
Model
*Parums
RDED
EDC
CV-DD
FADRM-
Method
Time Cost (8)
Peak Memory (GB)
ResNet- 18 [11]
11.7M
42.0
48.6
46.0
50.9
SRc L [6]
2.52
5.3
ResNet-50 [1]]
25 25.6M
49.7
54.1
$1.3
57.5
ResNet-101 [1]]
48.3
58.1
FADRM
EfficientNet-BO [35]
39.6M
42.8
51.1
43.2
51.9
MobileNetV2 [28]
3.4M
34.4
45.0
39.0
45.5
G-VBSM [30]
17.28
21.4
ShuffteNe [44]
19.6
30.2
Swin-Tiny [22]
28.0M
29.2
38.3
39.1
CV-DD [6]
8.20
23.4
Wide ResNer-50-2 [11]
68.9M
50.0
53.9
59.1
EDC|311 [31]
4.99
17.9
DenseNet121 [13]
8.0M
49.4
50.9
55,4
DenseNet169 [13]
14.2M
50.9
53.6
58.5
FADRM+
1.09
11.0
DenseNet201 [13]
20.0M
49.0
59.7
Table 3: Left: Efficiency comparison between various optimization- bbssdd methods and our approach when distilling ImageNet-IK. The time cost measured in seconds, representing the duration required to generate a single image on single RTX 4090 GPU. Right: Top-1 accuracy (%) on ImageNet- IK for cross-architecture generalization with IPC=10.
compared to EDC [31], culminating in a total computational saving of 54 hours when applied to the 50 IPC ImageNet-1K 1K dataset. Similarly, FADRM demonstrates 28.5 hours reduction in training time relative to SRe2L++ for the same task. Additionally, our framework significantly reduces peak memory usage compared to other frameworks, enabling efficient dataset distillation even in resource-constraiied scenarios. These results underscore the scalability and computational efficiency of our approach, which not only accelerates large-scale dataset distillation but also substantially lowers associated computational costs.
4.3 Ablation Study
Impact of Patch Numbers for Initialization and Residuals. To assess the effect of different patch configurations during both the initialization and resid- injection stages, we conduct an ablation study, as shown in Table 4. The results suggest that both 1 x and 2 x patch settings are effective for gener- ating distilled data. However, the 1 1 configuration consistently delivers the better overall performance, making it the preferred choice in practice.
IPC=10
IPC-50
FADRM
FADRM+
FADRM+
1x1
48.4
50.9
60.1
61.2
xx2
47.7
50.0
59.8
60.1
Table 4:
Comparison of student model (ResNet-18) generalization performance when trained distilled datasets generated using 1 x and 2 x 2 patch configurations during initialization and residual injection.
Impact of Mixed Precision Training (MPT). Our ablation study in Table 5 shows that MPT preserves distilled dataset quality while significantly reducing peak memory usage and improving optimization efficiency, making it an effective strategy for accelerating distillation.
8

---

FADRM
FADRM+
SRe
G-VBSM
W/MPT
W/MPT
W/O
WPMPT
ResNet-18 (Stident
47.7
47.8 %
50.0
49.6%
43.1 %
43.1
30.5 %
30.7 %
Emiciency
0.26
0.63 ms
0.58
0.96
0.26 ms
0.63
2.65
4.32 ms
Peak GPU Mernory
GB
GB
11.0 GB
23.0 GB
2.9 GB
5.3 GB
11.8 GB
21.4 GB
Table 5: Comparison of model generalization performance, optimization efficiency (milliseconds per image per iteration, measured under 100 batch size and 224 as input size), and peak GPU memory usage with and without mixed precision training under ImageNet- I IPC=10.
Configuration
Accuracy (8)
Time Cost
Configaration
Accurac
Time (6)
FADRM (WADARCC+ W/O MRO)
46.4
0.52
FADRM+ ARC + W.O MRO)
48.7
L.16
FADRM (W2D ARC W/ WYOM
46.2
0.47
FADRM+ (WID ARC W/ MRO)
1.09
FADRM AWC 0.5) + W/ W.MRO)
45.7
0.47
FADRM+ (W/ARC 0.9) W/ MRU)
48.5
L.09
FADRM (WL (a- (.8) W/ MRO)
46.4
0.47
FADRM+ (W/ARC'C (o 0.8) a W: MRU)
48.0
1.09
FADRM
47.6
0.47
(W/RRC 0.7) MRO)
FADRM (OORMIWWARRC (ax (180) WV.MRO)
47.3
0.47
FADRM+ (W/ARC (a (.G) W/ MRU)
49.3
1.09
FADRM (W/-RRC 0.5) W/)MRO)
47.7
0.47
FADRM+ (W/ARC (a. 0.55 MRO)
50.0
1.09
FADRM AWCARCC 0.0 WR)
47.4
0.47
FADRM+ (W/ARC 0.4) MRU)
1.09
Table 6: Performance comparison of R8 as the student model trained with distilled ImageNet- IK (IPC=10) datasets generated with different merge ratios (a), fixed D&.2000 and k=3. The efficiency measured in seconds per image generation. Left presents the ablation results for single-model distillation, while Right shows the corresponding results for multi-model distillation.
2
5
6
Da,
160
180
200
224
ImageNet. IK CIFAR-100
47.1
47.6
47.8
47.6
47.4
47.3
Post Eval (5)
47.2
47.5
47.7
47.7
59.2
60.9
61.5
60.5
59.4
57.9
Time Cost (s)
0.42
0.44
0.47
0.52
Table 7: Left presents the ablation results for (frequency of residual connections) using FADRM with Da,2000, a=0.5, while Right shows the ablation results for Da on ImageNet-1K K IPC=10. Efficiency is measured as the total time required to optimize a single image under a fixed budget of 2,000 optimization iterations using FADRM with a=0.5, k=
Impact of Components in FADRM. To assess the contribution of each component (MRO and ARC), we conduct comprehensive ablation study. As shown in Table incorporating MRO initially leads to a performance drop relative to the baseline (w/o ARC and MRO). This degra- dation primarily stems from the loss of critical details during resampling. However, integrating ARC and reducing the merge ratio a, thereby pri- oritizing original patches during merging, sub- stantially enhances performance relative to the baseline, which remains susceptible to informa- tion vanishing. The optimal performance is ob- served at a merge ratio of 0.5 for both settings, suggesting that equal fusion of original and intermediate optimized patches yields the most favorable results. Crucially, the findings clearly show that ARC effectively mitigates information vanishing and robustly recovers missing details during resampling, enabling a computationally efficient yet highly effective framework.
FADRM LRe
Cat
Toy
Goat
Lion
Dog
Car
SRe' L*+
FADRM
Golden Fish
Snow Mobile
Figure 5: Visualization of dataset distilled by FADRM and SRe'L++ on Tiny-Imagel two rows) and ImageNet- (bottom two rows).
(top
Impact of varying k. To examine the effect of & on distilled dataset quality, we perform an ablation study shown in Table 7 (Left). The results indicate that A = 3 achieves the best performance. Accuracy improves as l increases from to 3 but drops beyond this point, suggesting that excessive residual connections introduce redundant information and hinder the learning of coherent structures.
Impact of Downsampled Input Size in MRO. To determine the optimal downsampled input size (Da) for MRO, we conduct an ablation study, as presented in Table 7 (Right). Our results demonstrate that 2200 achieves the most optimal performance. Notably, using other sizes leads to a degradation in the quality of the distilled dataset compared to optimizing with the original input size of 224.

---

Top- Accuracy (%)
Top- Accuracy (%)
48
40-
32-
FADRM
FADRM
44
RDED
RDED
24
40
120
200
100 120
140
160
180
200
Number of Classes
Number of Classes
Figure 6: Five-step and Ten-step class-incremental learning on ImageNt with IPC=50.
4.4 Further Analysis
Crost-Arc Generalization. A fundamental criterion for evaluating the quality of distilled data is its ability to generalize across diverse network architectures, which significantly enhances its practical utility in real-world applications. As illustrated in Table 3 (Right), FADRM+ consistently outperforms all existing saate-of-the-ee methods across models of varying sizes and complexities, demonstrating superior generalization capabilities and robustness diverse scenarios.
Empirical Rademacher Complexity. A key assumption (Assumption 10) of our framework is that the empirical Rademacher complexity of the optimized patches exceeds that of the original ones. To verify this, conduct detailed empirical analysis. As shown Table 8, the em- pirical Rademacher complexity of the optimized patches consistently surpasses that of the originals, thereby sup- porting our theoretical claim and performance gains.
Models
R 0)
ResNet-18
4.89 = 0.05
4.94 i: 0.04
MobileNet-V V2
4.76 0.01
4.78 + 0.01
ShuffeNet-
4.72 0.02
4.75
Table 8: Empirical Rademacher Com-
plexity of optimized vs. original patches across different architectures.
4.5 Distilled Image Visualization
Fig. 5 compares distilled data from FADRM and SRe2L++ [6], both using ResNet- 18 with identical initial patch images, differing only in FADRM's incorporation of residual connections. As shown, FADRM more faithfully and effectively preserves the critical features of the original patches and consistently and visibly retains significantly more details than SRe This highlights the advantage of residual connections in enhancing information density and improving the quality of distilled data.
4.6 Application: Continual Learning
Leveraging continual learning to verify the effectiveness of distilled dataset generalization has been widely used in prior work [43, 30, 46]. Following these protocols and utilizing the class-incremental learning framework as in DM [46], we conduct an evaluation on Tiny-ImageNet IPC=50 using a 5-step and 10-step incremental setting, as shown in Fig. 6. The results clearly indicate that FADRM consistently surpasses RDED in various settings, demonstrating its effectiveness.
Conclusion
We proposed FADRM, novel framework for dataset distillation designed to generate high-quality dis- tilled datasets with significantly reduced computational overhead. Our work identifies and addresses the critical challenge of vanishing information, fundamental limitation in Uni-Level Framework that heavily undermines the information density of distilled datasets. To address this, we introduce data-level residual connections, a novel mechanism that balances the operations preserving critical original features and integrating new information, enriching the distilled dataset with both original and new condensed features and increasing its overall information density. Furthermore, by integrating parameter mixed precision training and input multi-resolutin optimization, our framework achieves significant reductions both Peak GPU memory consumption and training time. Extensive exper- iments demonstrate that FADRM outperforms existing ofate-of-the-art methods in both efficiency and accuracy across multiple benchmark datasets. For future work, we aim extend the idea of data-level residual connections to broader modalities and applications of dataset distillation tasks.
10

---

Acknowledgments
This work was supported by the MBZUAI-WIS Joint Program for AI Research.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 482, 2002.
[3] Normand J Beaudry and Renato Renner. An intuitive proof of the data processing inequality. arXiv preprint 1rXiv:1107.07400 2011.
[4] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 4750 4759, 2022.
[5] Mingyang Chen, Bo Huang, Junda Lu, Bing Li, Yi Wang, Minhao Cheng, and Wei Wang. Dataset distillation via adversarial prediction matching. arXiv preprint arXiv:2312.089 2023.
[6] Jiacheng Cui, Zhaoyi Li, Xiaochen Ma. Xinyue Bi, Yaxin Luo, and Zhiqiang Shen. Dataset distillation via committee voting. arXiv preprint arXiv:2501.07572 2025.
[7] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation imagenet- IK with constant memory. In International Conference on Machine Learning, pages 6565-6590. PMLR, 2023.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[9] Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. arXiv preprint arXiv:2206. 220102200.002166, 2022.
[10] Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. To- wards lossless dataset distillation via aligged trajectory matching. In The Twelfth International Conference on Learning Representations, 2024.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Yang He, Lingao Xiao, Joey Tianyi Zhou, and Ivor Tsang. Multisize dataset condensation. arXiv preprint arXiv:2403.0607 06075, 2024.
[13] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[14] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data param- eterization. In International Conference on Machine Learning, pages 11102-11118 PMLR, 2022.
[15] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, ON, Canada, 2009.
[16] Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[17] Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing, arXiv preprint arXiv:2208.10499 2022.
11

---

[18] Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pages 12352-12364. PMLR, 2022.
[19] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprini arXiv:2412. 19437, 2024.
[20] Ping Liu and Jiawei Du. The evolution of dataset distillation: Toward scalable and generalizable solutions. arXiv preprint arXiv:2502.05677 2025.
[21] Songhua Liu, Kai Wang, Xingyi Yang. Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. Advances in Neural Information Processing Systems, 35:1 111010001111111 2022.
[22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[23] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems, 35: 13877-13891, 2022.
[24] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv: 1oXiv:1710..0 171.003740 2017.
[25] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 14:116-5198,, 2021.
[26] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[27] Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Kon- stantinos N. Plataniotis. Datadam: Efficient dataset distillation with attention matching, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17097-17107, October 2023.
[28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510 2018
Liang-Chich Chich Chen.
[29] Xinyi Shang, Peng Sun, Zhiqiang Shen, Tao Lin, and Jing-Hao Xue. Dataset distillation in the era of large-scale data: Methods, analysis, and future directions. Authorea Preprints, 2025.
[30] Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16709-16718, 2024.
[31] Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation. Advances in Neural Information Processing Systems, 37:99161-99201,, 2024.
[32] Donghyeok Shin, Seungjae Shin, and II-Chul Moon. Frequency dosain-based dataset distilla- tion. Advances in Neural Information Processing Systems, 36:70033-70044, 2023.
[33] Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1390-9399 2024.
[34] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
12

---

[35] Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning, pages 10096-10106. PMLR, 2021.
[36] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Bapisste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[38] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinehao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[39] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 181.v: 10959, 2018.
[40] Lingao Xiao and Yang He. Are large-scale soft labels necessary for large-scale dataset distilla- tion? arXiv preprint arXiv:2410.15919 2024.
[41] Eric Xue, Yijiang Li, Haoyang Liu, Peiran Wang. Yifan Shen, and Haohan Wang. Towards adversarially robust dataset distillation by curvature regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 9041-9049, 2025.
[42] Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. arXiv preprint arXiv:2311.18888, 2023.
[43] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation imagenet scale from new perspective. Advances in Neural Information Processing Systems, 36, 2024.
[44] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9848-6856, 2018.
[45] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674-12685. PMLR, 2021.
[46] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, January 2-7, 2023, 2023.
[47] Bo Zhao, Konda Reddy Mopuri. and Hakan Bilen. Dataset condensation with gradient matching. In International Conference Learning Representations, 2021.
[48] Binglin Zhou, Linhao Zhong, and Wentao Chen. Improve cross-architecture generalization dataset distillation. arXiv preprint arXiv:2402.1 1300077777.77 2024.
[49] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression, Advances in Neural Information Processing Systems, 35:9813-9827, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims the abstract and introduction accurately reflect the paper's s contributions and scope. Specifically, the abstract summarizes proposed method, key findings, and empirical improvements over prior work. The main contributions are explicitly listed in Sec. and Sec. 4 provides experimental validation supporting these contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, do talk about limitations of this work, Sec. F for more details.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "L-mmittioon" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: For each theoretical result, we provide the full set of assumptions and complete, correct proofs. All theorems, formulas, and proofs numbered and cross-refenee ress-reefre..... Assumptions are clearly stated within each theorem statement. Proofs are presented in Appendix A. We also properly reference all lemmas or theorems used in our analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide all necessary information to enable reproduction of our main experimental results. This includes comprehensive descriptions of the model architecture, optimization settings, hyperparameters, training and evaluation protocols, and dataset speci- fications. reproducibilitt-reeeaaa relevant details are presented in Sec. and Appendices G and G.2, ensuring that the key claims and conclusions can be independently verified, even in the absence of released code. We also plan to release the code and pretrained models upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We intend to release the full codebase and data preparation scripts once the double blind review is finished. In the Appendices and G.2, we include detailed instructions for reproducing our main experimental results. These materials are designed to ensure faithful reproduction of both our proposed method and all baseline comparisons.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /nips/ips..... cc/ for more details.
pub public// guides / /Codeeeodelaaa
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines thttps: //nips tttpcc/ppppbta wwwpc//ppppcc.ee.......c.c..tttt htalllic/ciiii ////guuieess quides//o CodeCoodSSiii Submi as Loo oolicyyyyy. icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We specify all training and test details necessary to understand and reproduce the results, including optimizer types, learning rates, batch sizes, and other hyperparameters. These details are comprehensively documented in Sec and Appendix G.2, allowing readers to fully appreciate the experimental setup and outcomes.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
16

---

Answer: [No]
Justification: We do not report error bars or standard deviations, as our method consistently outperforms prior baselines by large margin across all benchmarks (see Sec 4), making the results statistically robust without requiring additional significance metrics. The substantial performance gap renders the conclusions reliable even without variance reporting. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information on the computational resources used for all experiments, including GPU type, number of GPUs, memory, and runtime for different setups. These details are included in Sec 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htt /tturr//////iieeee ps wwtppcppppccccccctccctc..ct... htte:/pblllc//etteee caGui de ness
Answer: [Yes]
Justification: Yes, we did follow the code of Ethics mentioned in the website.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We explicitly discuss both the potential positive and negative societal impacts our work, more details can be found in Appendix F.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work focuses on dataset distillation for image classification using standard, publicly available datasets. We do not release models or data with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
18

---

Answer: [Yes]
Justification: We properly credited them in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use the package should be provided. For popular datasets, paper seeerwie rswi thcode com/ datasee has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new assets, including distilled datasets and pretrained models. We will include them in the supplemental material. Documentation will be provided alongside the released assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects, and thus no IRB approval is required.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Our core methods do not involve large language models (LLMs) as an impor- tant, original, non-standard component.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non- sta-standard components.
Please refer to our LLM policy (https: : /t////nerispp//eee//e wcc/cnneeeencec///eeeeeeeeececee 2025/ LLM) for what should should not be described.
20

---

Appendix of FADRM
Contents
A Theoretical Derivation
22
A.1 Preliminary
22
A.2 Bounded Information BN-Aligned Synthetic Data
................................
24
A.3 ARC improves the robustness of the distilled images
24
A.3.1 Generalization Bound Analysis.
24
A.3.2 Data Processing Inequality Analysis
27
Empirical Rademacher Complexity Estimation
27
Additional Experiments
28
C.1 Adaptive Multi- RestiiRneoo Optimization
28
C.2 Distillation Performance vs. Model Confidence
29
C.3 More Experimental Results of ARC Small Datasets
................................
29
C.4 Stability of ARC
30
C.5 Contribution of Individual Components on Efficiency Gains
30
Optimization Details
30
Resampling via Bilinear Interpolation
31
Limitations
31
G Experimental Setup
32
G.I Experiment Pipeline
32
G.2
Hyper-parameters Setting
32
G.2.1
CIFAR-100
33
G.2.2
Tiny-ImageNet
33
G.2.3
ImageNette
35
G,2.4
ImageWoof
35
G.2.5
ImageNet-IK
35
Additional Distilled Data Visualization
37
21

---

A Theoretical Derivation
A.1 Preliminary
Lemma (Data Processing Inequality (3]). Let X + Y + Z form Markov chain. Then the mutual information between X and upper bounded by that between X and Y:
I(X; 2) S I(XY).
(12)
In particular, no post-processing Y can increase the information that Y contains about X.
Theorem 4 (Temperature-scuul KL divergence is bounded and Lipschitz-continuuus, Fix integers k 2 2 and constant C' >> 0. For any temperature T > 0 let
exp(21/T)
2 (Z1,...,,E5) =x) E [--c,0)
h
(i 1i=1....,A k).
(13)
exp(2//11
Let p = (pI....,PH .Pk) E As be an arbitrary target probability vector (e.g. it may come from another soft-max with its own temperature). Define the loss
&
(t) = kup 11 (z) = Σ
pi
log
Pr (T)
(14)
1=1
9i
Then the following hold:
1. (Bounded range) For every admissible pair (p, 2).
2C
s E(p.g S B,
B = log k + T
(15)
2. -Lipschht continuity in logits) The map Ep. (T)
(z)) is L-Lipschitz wart. the lox. norm with L 1. T Consequently, it is Vk/TT-Lipshhin w.r.. the Euclidean norm.
proof of Theorem 4. (i) Boundedness. Write
k
kT<p l1 g ())
Σ
Pa log Σ Pe log q TT)
(16)
Since æ + x log x is non-positive on [0, 1], the first term is at most 0, so
(1 s p. log (T)
(17)
For the soft-max, log q; (T) 2 log U, where Z = E/= exp(2,/T). Hence
k
log (T)
pizi + +og Z.
(18)
T
1=1
Because each 2 E [-C,C] and E.P. Pi 1,
C T
Σ
pez:
(19)
T
Moreover, =; < C' implies Z S Eeep((C/T) and thus log 7 < log k + T.. Combining the two parts yields the desired upper bound log & + 2C/T. Non-negat KI. divergence gives the lower bound 0.
22

---

(ii) Lipschitz continuity. Differentiate é w.r.t 2z:
(T) T
d ( (z)) = Pi
(20)
Because pp; 1 S 1, we have [0,,4| S 1/T for every coordinate. Thus | S 1//,, and by
the mean-value theorem,
1
(( (=)) - u(p l (7 ( S +llz - oo0
-
ve,2''
(21)
so = 1/T in the Lo norm. Since ||l||2 v v /lllelses the Euclidean Lipschitz constant is at most Vk/T.
Lemma 2 (Generalization Bound via Rademacher Complexity (2]). Let H be a class of functions mapping X' - 00, B), and let S = {I1,... ,,Tn) be i.i.d. sample from distribution D. Then, for any 0, with probability at least - 6, the following inequality holds for all h E H:
log(1/0)
B)wo((())]= S
Σ 7
Max)) + (H,,.HH + By
(22)
2n
Lemma 3 (Empirical Risk Proximity). Let ==
== ašt + (1 (1- a)z; with a E (0, and let the
corresponding dutasets be Crea == { W)))== CFADRM == ((0,, v)) Then for any model E H. the empirical risk difference is bounded by a negligible value:
1 n
Crea(1)) (h) - -EAADRM(h) s (1 a). A1, where A ==
Σ |
xill
(23)
n
Proof of Lemma 3. We begin by computing the pointwise difference in the loss:
[((h(z)"*) ), y:) E(h(E.), 34)1
(24)
Since l is Le-Lipschitz in the model output, and h is -ipischtt in the input, have:
|e(h(0;-)) v4) ((h(5)), v:)1 T4 [ h(x.)l S L.gh 1 f,ll-
(25)
Note that:
ã az, aes (1 a)x;
x i (1 - (zi xx))
(26)
So:
|: -:ll = (1 - -a]ll - =zill-
(27)
Therefore,
(((h( v1) E(h(os), vi) & Le - a.) - xill-
(28)
Averaging over n samples:
(1))a - (EA
1 Σ (1 - -zil = L1 - -a) Az.
(29)
im
Corollary (Lipschitz Convex Combination Bound). Let h : Rd + R be an L-Lipschitz function.
For any x, y E Rd and a (0, 1), define az + (1 a)y. Then:
[h(z) - (ah(x) + (1-aa)(1)) < La(1 -a]l yl
(30)
In particular, this implies:
h(=) < ah(x) + (1 - a)h(y) + La(1 a) - y/
(31)
h(2)) 2 ah(z) + a}h(y) La(1- a)
(32)
23

---

A.2 Bounded Information in BN-Aligned Synthetic Data
Proof of Theorem I. Let O denote the original dataset. From it, a pretrained model fo is derived, which includes BatchNorm statistics {14, o.) J. Each synthetic image i; in the distilled dataset C is generated by minimizing an objective function depending only on fe and a fixed label V/.
We assume that each ž is generated independently given fo. and that so is deterministic function of O. Then, for each sample (.zz új), have the Markov chain:
O - fo -
(33)
By applying Lemma 1, we get:
[a;; 0) < (HHoOO)H((J))
=
(34)
Now, by the chain rule of mutual information:
[ S Σ j=1
O)= -
S [C|- -( H(fo).
(35)
where we used the fact that ūj is fixed and independent of O and the independence assumption across samples. Thus, the total information that the synthetic dataset C can retain about the original dataset O is bounded by the product of its size and the entropy of the model fa.
A.3 ARC improves the robustness of the distilled images
We evaluate the performance gain of FADRM over the uni-level framework from two complementary perspectives: (1) Generalization Bound Analysis, and (2) Data Processing Inequality.
A.3.1 Generalization Bound Analysis.
We first analyze the advantage of FADRM through the lens of generalization bounds. The central insight of this analysis is that when the synthetic data is optimized, exhibiting low cross-entropy loss and strong feature alignment, it becomes overly easy for the model to fit. Such data provides limited regularization, making the student model susceptible to overfitting and resulting in degraded generalization to unseen samples. In contrast, when synthetic data is combined with real data under the FADRM framework, the real data introduces natural variability and subtle distributional nuances that act as an implicit regularizer. This hybrid training scheme mitigates overfitting by constraining the hypothesis space, thereby improving the model's generalization performance.
Analysis of Assumption 10. To facilitate subsequent theoretical analysis, we first provide a prelimi- nary interpretation of Assumption 10. We begin with mild and empirically supported assumption that the Rademacher complexity of the optimized patches exceeds that of the original ones, i.c.,
R. (H (+os) > (H o O),
(36)
where O denotes the original data patch and Cres represents its optimized counterpart. Building upon this preliminary inequality, we introduce small constants to achieve a more general analytical form, taking into account the bounded smoothness and Lipschitz continuity of l and h. In practice, these constants remain small due to the use of high-temperature KL loss and strong regularization (e.g.. CutMix). Accordingly, the assumption can be equivalently expressed as
Ln
La
Rn(H a Eres) R, (H O)>
A,
(37)
2B
2
where Le and LA denote the Lipschitz constants of the loss and hypothesis functions, respectively. Since both La and Le are small in typical settings, the coefficient (Lag + La) becomes negligible, allowing us to simplify subsequent theoretical derivations.
Proof of Theorem 2. Let ares be a perturbation generated via distribution (running statistics) match- ing and prediction (cross entropy) matching, and let x4 be real image from the original dataset.
Define the residual-injected sample ã; as:
c a..ee + (1 aa) a (0,1)
(38)
Define the datasets:
24

---

Cres { " h perturbation generated via distribution (running statistics) matching and prediction (cross entropy) matching.
{Zi, yi Ni selected patches from the original dataset,
CPADRM { Vs 1 residual-injected dataset.
We begin by bounding the Rademacher complexity of the -siidual-ijjectd dataset ĆEADRM = (5x )== where σ; axiee + a.),,, using Lemma 2 and Corollary
From the definition:
n ((3sss)
R.,H CFADRM) E.
sup = NEH 72
(39)
i=1
By Corollary we have for each term:
h(#;) S ah( a))((a))) + Ee:
where |esl S Tu ae11 - -a)l
xz
(40)
Therefore:
Σ o,,) S Σ o. (ah(E + (1 (1-a))(zz;) + |o,
(41)
Using |od| = 1, we get:
n
losedl s Tna(1 - a)
Σ
|
x.ll = n Lna(1 - -a)
(42)
Divide by n, take supremum and expectation:
Ry (H CFADRM) a RR o (ree) a1 (?) aa) -A
(43)
Rearrange the Inequality:
Ri (HoCCADD)))) - RR, (HoCre)) & (1 (1-a) Ri (HL - Ry, (H (+e)) + Ln) (44)
Multiply 2B both sides and add a negligible positive value to the LHS:
2B (R CFADRM) RR (7, o Cress]
2B(1 a:) [ (H O) RR, (H (+as)
(45)
+ 2BL, a) A+e
As validated in Theorem 4, when T> O, KL-divergence becomes a bounded B-range loss, which we then apply Lemma to formulate generalization error:
Cgen(h) < [(h) + 2B Ry RH o S)
(46)
Apply both models:
Lygnn (Ryea) S Sres + 2B RR, Öres)
(47)
Cyen (bFADRM) LEADRM 2B R., (H CFADRM)
(48)
Recall the lower bound for the difference of two ERMs established in Lemma 3, we then have:
Eres (EADRM(h) 2 LL A, where A Σ 2
rzll-
(49)
Given the Equation (37), we can then derive:
LL ()-4)))))) A> 2B (1 (1-a) a) R. (HL0) - (7 c L ha(l - a) a) (50)
25

---

where the RHS in Equation (50) is the upper bound for the difference in Rademacher Complexity, we then derive the following inequality:
Ens LPADRM > 2B (M a CPADRM) - Hi (H Cres)
(51)
which shows:
Eres + 2B Rn,HH o Cree) EFADRM 2B RH, CFADRM)
(52)
Analysis of Merge Ratio a. We further conduct formal analysis to determine the optimal merge ratio a. As demonstrated in the following analysis, the optimal configuration achieved when approaches 0.5, yielding balanced contribution between the two distillation components and promoting both stable convergence and improved generalization. To theoretically substantiate this choice, we examine the influence of on the generalization bound of FADRM, defined as follows:
((a)= = LFADRM + 2B R. (H ČrAdrm),
(53)
where ÉFADRM denotes the empirical loss, and R.,())
(-) is the empirical Rademacher complexity.
Assuming the loss function is A-strongly convex and denoting A 1 Σ, || zz we obtain:
LADDRM S aCCns (1- a) 3 a(1 a)
(54)
where Eres and Eo are the empirical losses of the residual and original components, respectively, Following the derivation in Equation 43, the empirical Rademacher complexity can be bounded as:
R2 (H CAADrR) S aRR, (H Cren) + (1 - a.,, O) + L, . a(1 a))
(55)
where Ln is the Lipschitz constant of the hypothesis class H. Substituting both inequalities, we obtain an upper bound on D(a):
D(a) Xa Y Z(a 1)a,
(56)
where the coefficients are defined as:
X = Cres + 2B Ry (H, o Cres;
Y = Lo + 2B Rr (H O),
X ~ 2BL
=
2
Minimizing the quadratic bound ((x) with respect to a yields:
2 + Y-X
a"
(57)
27
In practice, FADRM employs temperature-scaled KL divergence, under which the strong convexity constant satisfies
x (C- (-4
(58)
where C is the number of classes, d the embedding dimensionality, and T the temperature parameter. As T increases, À grows approximately linearly with C., and the bias term Y-X becomes negligible 22
compared to dominant term. Therefore, the optimal merge ratio a* approaches 0.5, implying that equal weighting between the residual (synthetic) and original (real) components achieves near- optimal generalization. This theoretical conclusion aligns with our empirical findings in Table where 0.5 consistently yields strong performance across datasets.
26

---

A.3.2 Data Processing Inequality Analysis
We further show that FADRM will improve the bound established in Theorem 1.
Proof of Theorem 3. We analyze the effect FADRM, particularly the Ajustable Residual Connec- tion (ARC) component, in mitigating the limitations of the uni-level framework.
In FADRM, the generated image is produced using both the learned model So and real image patches from O, by iteratively fusing these patches into the intermediate optimized image. This process introduces new dependency structure:
+ fo.
(59)
As result, the conditional independence assumption
TT
O fo
(60)
no longer holds. Consequently, the Markov chain underlying the standard Data Processing Inequality (DPI) for the uni-level framework is violated.
By the chain rule of mutual information, have:
(ā; O) = I(fo; ((;; O fe).
(61)
In the uni-level framework, the generation process depends solely on fa. implying
[(2; O fe) 0.
(62)
However, in the FADRM setting, real image patches directly influence the generation of ê, leading to strictly positive conditional mutual information term:
T(:; O fo) O.
(63)
Therefore,
i - I(f: + I(:: O fo) > I(fo; = I(Zuii O)
(64)
which proves the theorem.
This result demonstrates that FADRM fundamentally breaks the conditional independence constraint inherent in the uni-level framework, allowing additional information from the original dataset to flow into the generated samples. Consequently, FADRM captures richer and more diverse information, leading to superior knowledge distillation performance.
B Empirical Rademacher Complexity Estimation
Formulation. Given fixed dataset O = {= and hypothesis class H (ho X + R], the empirical Rademacher complexity quantifies the expressive capacity of H on O:
(H, () E sup bER
Lid Unif(-1,+1).
(655
Here, σ (o1...., Gn) is a vector of independent Rademacher random variables representing a random +1 labeling of the data. From data-centric perspective, for fixed hypothesis class H. (H O) measures how easily the data O admits spurious alignment with random #1 labels; larger value indicates that the same model family can fit these data more readily, implying higher overfitting propensity and weaker implicit regularization of the dataset.
Functional interpretation. For fixed random draw define the functional
1 Σ a,(()))
( sup NEH
(66)
which captures the maximum correlation between the random sign pattern o and the predictions M(xx) over all admissible h E H. In other words, Φ(ar) represents the single-trial response of H to random noise labeling of O. function class with high capacity will yield larger Φ((r) since can align more closely with arbitrary noise patterns.
27

---

true Monte Carlo approximation.
Because the expectation in Eq. (66) is intractable, we estimate it via
M i.dd. Monte Carlo samples (o
1 Jm=1
(M) (HoO) Hi
Σ M(alm)) M
1
((a(m)
sup NeW
(m)
Mxx))
(67)
Each 6(o (()())) can be viewed stochastic probe of the capacity of H. and their empirical average approximates the true expectation E, [o(or]]
Optimization-aasee based inner supremum. The inner supremum SuPkeи in Eq. (66) is approximated via empirical optimization. Given parameterized model ho(z)) B (gnh(gg(z)) bounded to [- B, BJ. we maximize () Σ; ozho (x:) by minimizing its negation:
((0:0) - Σ az)o(
(68)
We train 0 for E epochs using SGD Adam and record the best achieved value
(a) = max E o, hoco (x;) sup Σ o,ho(
(69)
Since the optimization may not reach the global maximum, o² (or serves lower bound on the true supremum, and the final N, estimate is thus a conservative junder-Japproximatio complexity.
Empirical estimation for optimized and original patches. we compute an analogous quantity:
For the optimized image patches Cres
M
RA (H o Cren
Σ
max ISes
(m) hate) (T.
(70)
M
and compare it to that computed on the original patches O. Under the theoretical assumption in Eq. 10, if the residual optimization step enriches the feature expressivity of the synthetic data, we expect the following relationship to hold:
Ri, (H. Cres) > R. (H.
(71)
Statistical reporting. Let 2a (m)) denote the best objective obtained for each Rademacher draw. We report the sample mean and standard error (SE) across M trials:
M
DM (cm - 7)2
1 M
Ri
U
SE
(72)
VM-
This Monte Carlo-based based estimator provides reproducible and statistically sound measure of em- pirical R,, thereby linking theoretical generalization guarantees with observed improvements in data-level residual optimization.
C
Additional Experiments
C.1 Adaptive Multi-Resolution Optimization
To avoid manually selecting the downsampled dimension Das, we apply an adaptive multi-resolution strategy that automatically adjusts the effective image resolution according to the smoothness of the input. The underlying idea that smoother images can tolerate stronger downsampling without significant loss of fidelity, whereas images with rich textures require higher resolutions.
We estimate image smoothness using the total variation (TV) of a batch E R BxCxHXW
B
TV(x)
Σ
[| :] aali - 11: + || 1 2./a:: - 1]]li)
(73)
28

---

Based on the TV value, we compute an adaptive shrink ratio:
0.5
ShrinkRatio = min 0.8, max,TV, 10
(74)
and the corresponding downsampled dimension:
Da max (LD ShrinkRatioJ, 128)
(75)
This adaptive adjustment enables automatic control of the resolution level during training without manual tuning. As shown in Table 9, the adaptive strategy achieves comparable performance to manually selected resolutions.
Dataset
Manually Selected
Adaptive
ImageNet-1K
47.7%
47.5%
Table 9: Comparison between manually selected and adaptive multi resouuiissoltion optimization.
This adaptive method effectively removes the need for manual hyperparameter search while maintain- ing model performance.
C.2 Distillation Performance vs. Model Confidence
As shown in Theorem 1, FADRM, similar other uni-level distillation methods, is most effective when the pretrained model is not overly confident, i.e.. when its output distribution exhibits higher entropy. In this regime, the mutual information upper bound becomes looser, allowing the distilled data to better capture information from the original dataset.
To empirically verify this relationship, vary the number of training epochs used to obtain the teacher model, which directly affects its output confidence. As the teacher becomes more confi- dent (lower cross-entropy), the effectiveness of FADRM decreases, consistent with the theoretical prediction. The results are summarized in Table 10.
Squeezed Epochs
Cross-Entropy
FADRM Accuracy (%)
50
0.0338
61.5
100
0.0054
59.6
150
0.0031
58.7
200
0.0028
58.3
Table 10: Relationship between teacher confidence (measured by cross-entropy) and FADRM performance. Lower cross-entropy indicates higher confidence.
These results align with the theoretical insight that moderate uncertainty the teacher model facili- tates more informative knowledge transfer, while overly confident teachers limit the representational diversity distilled into the student.
C.3 More Experimental Results ARC on Small Datasets
To further evaluate the performance of ARC on small-scale datasets, we conduct experiments on CIFAR-100. As shown in Table 11, results indicate that a 0.5 yields the best performance, consistent with our observation on the large-scale dataset (ImageNet-IK)
0.9
0.8
0.7
0.6
0.5
0.4
0.3
FADRM Accuracy (%)
58.0
58.9
58.7
59.2
58.6
Table 11: Performance of ARC on CIFAR- across different values of The optimal setting
= 0.5 is consistent with the observation on ImageNet-IK.
29

---

C.4 Stability of ARC
To verify that ARC maintains semantic consistency rather than introducing disruptive information, we conduct two complementary analyses. First, we compute the cosine similarity between features before and after ARC injection across stages. Second, we assess the accuracy of the distilled images before and after applying ARC using pretrained model as a verifier. As shown in the Table 12, the feature-level cosine similarity remains high (2 0.88) across all stages, indicating strong semantic preservation. Although verifier accuracy shows slight decline (from 100% to 97%) at the initial injection stage, quickly recovers in subsequent stages, suggesting that ARC adds semantically aligned residual content rather than harmful noise. Overall, these results confirm that incorporating ARC is stable and does not negatively impact the optimization process.
Stages
Feuture Cosine Similarity
Before ARC Accuracy (%)
After ARC Accuracy (%)
Residual Injection Stage #1 Residual Injection Stage #2
0.89
100
97
0.88
100
100
Residual Injection Stage #3
0.88
100
100
Table 12: Feature stability and accuracy before and after ARC injection across different residual stages CIFAR-100.
C.5 Contribution of Individual Components Efficiency Gains
To clarify the individual contributions of Mixed Precision Training (MPT) and Multi-Resolution Optimization (MRO) to computational efficiency, we conducted ablation study shown in the table below, the experiment is conducted on ImageNet- I where efficiency is measured based on the time needed for generating a single distilled image. As shown in Table 13, MRO is primarily intended to reduce the overall wall-clock time required to optimize a single image by iteratively performing training on downsampled versions. Since the process still involves full-resolution updates, the peak GPU usage remains unchanged. In contrast, MPT is designed to reduce both computation time and peak memory consumption by leveraging lower-precision operations throughout.
While MPT accounts for the majority of the efficiency improvement, MRO provides complementary gains. For instance, when comparing the configurations with and without MRO under MPT, observe a reduction of 0.05 seconds per image. Although this difference may appear marginal at the individual level, it becomes meaningful when scaled. For example, generating 50 IPC ImageNet-IK dataset (i.e., 50,000 images) results in,
50, 000 x 0.05seconds = 2, 500seconds R 42minutes,
(76)
In summary, MPT is the primary contributor to efficiency, and MRO further improves performance by iteratively optimizing the images at a smaller resolution.
Settings
Time Cost (s/ img)
Minimum GPU Usage (GB)
Peak GPU Usage (GB)
w/o MPT, w/o MRO
1.26
5.3
5.3
w/o MPT. w/ MRO
0.84
4.0
5.3
w/ MPT, w/o MRO
0.52
2.9
2.9
w/ MPT, w/ MRO
0.47
2.5
2.9
Table 13: Efficiency ablation on ImageNet-1K. Decomposition of the contributions from Mixed Precision Training (MPT) and Multi-Reesolutinn Optimization (MRO). We report wall-clock time per distilled image (s/img) and GPU memory usage. MPT chiefly reduces both time and peak memory, while MRO provides additional wall-clock savings with unchanged peak memory.
D Optimization Details
Formally, the optimization process adheres to the principle of aligning the synthesized data with both the predictive behavior and the statistical distribution captured by a pretrained model fg. Specifically, given a synthesized image ū; at iteration t, the optimization objective is defined as:
arg min (..)( + Detobaal (ft),
(77)
S,
30

---

where l(fe ( enforces consistency with the target predictions, while D giaobal (2x) ensures align- ment with the statistical distribution. Importantly, the parameters fo remain fixed throughout the optimization, and only 32 is updated.
The prediction alignment term is formulated the cross-entropy loss computed the synthesized batch:
C(Jo
ΣΣ N
1 N
Nn,i log fo(Et)m.i:
(78)
n=1 s=1
where N denotes the batch size, and C represents the total number of classes. The alignment to the distribution in pretrained model is calculated as follows:
D glohal (Tx)
= Σ [(x( 1|u1
+ Σ -
Σ (Im(())) BN 1l
+ Σ
- -
where O denotes the original dataset, and 1 indexes the layers of the model. The terms BN and BN correspond to the running mean and running variance of the Batch Normalization (BN) statistics at layer I. By minimizing Dglobal ( the synthesized data is encouraged to exhibit statistical characteristics consistent with the original dataset, thereby preserving global information.
E Resampling via Bilinear Interpolation
Given an original image 1 2² + E defined on discrete pixel coordinates, the continuous extension I: IR2 + Rº at non-integer location (i', E IR? is computed via bilinear interpolation as follows:
i(('j') = Σ Σ
Wm,n [(i + m.j n.),
(79)
m==0 02
where i = - = a -iE 1), j -j E [0. 1), and the interpolation weights are defined by:
Wm,n = m + (-1)
(-1)"
(80)
Explicitly, Equation (79) expands to:
i( = (1- - 3) [(i.]) + a(1 - 3) T(1 1,5) + (1- I(i,j + + a3 I(i 1(i+1,j 1,j++1
(81)
This interpolation scheme can be viewed as a separable approximation to the continuous image function, with weights derived from tensor-product linear basis functions over the unit square. preserves differentiability with respect to the fractional coordinates (i', making it particularly amenable to gradient-based optimization frameworks.
F
Limitations
While FADRM offers substantial improvements in computational efficiency and performance for dataset distillation, also introduces several limitations. First, the method relies on the assumption that residual signals between synthetic and real data capture critical learning dynamics, which may not generalize across domains with highly abstract or non-visual modalities such as natural language or time-series data. Second, the use of distilled datasets can inadvertently reinforce biases present in the original data if not carefully audited, potentially leading to fairness concerns downstream applications. From a broader societal perspective, while FADRM reduces the computation and
31

---

resource demands of training large models, thereby contributing positively to sustainability, it may also facilitate the deployment of powerful models in low-resource or surveillance scenarios without adequate ethical oversight. Thus, responsible deployment and continued research into bias mitigation and cross-domane generalization are essential to ensure the safe and equitable application of FADRM.
G Experimental Setup
G.1 Experiment Pipeline
Our experiments start by generating synthetic images from pretrained teacher models using the proposed distillation framework. The resulting images form the distilled dataset, which is subse- quently used to train a student model from scratch under the training configurations detailed in Appendix G.2. To evaluate the effectiveness of the distilled datasets, we adopt standard image classification task. Each distilled dataset used to train a student model from scratch, and the resulting model is evaluated the original test set of the corresponding dataset (e.g., CIFAR- 100, Tiny-ImageNet, ImageNet-IK) The evaluation metric is top-1 classification accuracy, computed using the final checkpoint after full training. This protocol ensures that the measured performance differences stem from the representational fidelity and diversity of the distilled data rather than differences in optimization configurations model initialization.
G.2 Hyper-parameters Setting
Our method strictly follows the training configuration established in EDC to ensure a fair and consistent comparison across all evaluated approaches. Additionally, we re-run RDED and CV-DD under the same configuration and report the highest performance obtained between their original setup and the EDC configuration. This methodology guarantees rigorous and equitable evaluation by accounting for potential variations training dynamies across different settings.
Full Dataset Training. To establish an upper bound on performance across different backbone architectures (representing the results achieved when training models on the full original dataset) we adopt the hyperparameters specified in Table 14. These hyperparameters are carefully chosen to ensure full model convergence while effectively mitigating the risk of overfitting, thereby providing reliable reference for evaluating the performance of distilled datasets.
Hyperparameters for Training the Original Dataset
Optimizer
SGD
Learning Rate
0.1
Weight Decay
le-4
Momentum
0.9
Batch Size
128
Loss Function
Cross-Entropy
Epochs
300
Augmentation
RandomResizedCropp
Horizontal Flip, CutMix
Table 14: Hyperparameters for Training the Original Dataset.
In summary, the synthesis of distilled data follows consistent hyperparameter configurations, as outlined in Table 15. Variations in hyperparameters (1) the model Pre-train Pre-training phase. and (2) the post-evaluation phase. These adjustments are carefully tailored based the scale of the models and the specific characteristies of the datasets used. During the post-evaluation phase, we have four hyperparameter combinations, as detailed in Table 16. Among these parameters, 1) plays critical role in controlling the decay rate of the learning rate, as defined by the cosine learning rate schedule in Equation 82. Specifically, a larger value of results slower decay rate, thereby preserving a higher learning rate for longer duration during training.
introduced exclusively during two phases:
step
Learning Rate = 0.5 x 1 cos π epochs X
(82)
32

---

Hyperparameter
Value
Optimizer
Adam
Learning rate
0.25
Beta
(0.5,0.9)
Epsilon
1 x 10
Batch Size
100 10 (if C 100)
Iterations (Nise)
2,000
Merge Ratio (a)
0.5
Number of ARC (k)
Downsampled Size (Dos)
200 (ImageNet-1k K and Its subsets), Original Input Size (CIFAR-100,
Imn-eengeNee)
FADRM Recover Model FADRM+ Recover Model
ResNet18
ResNet18 DenseNer121 Shuf-
flc.NtV2 MobileNetV2
Scheduler Augmentation
Cosine Annealing
RandomResizedCrop. Horizontal
Flip
Table 15: Hyperparameters for generating the distilled datasets.
Setting
Learning Rate
S1
0.001
S2
0.001
S3
0.0005
S4
0.0005
Table 16: Hyperparameter settings with learning rate and 17.
G.2.1 CIFAR-100
This subsection outlines the hyperparameter configurations employed in the CIFAR-100 experiments, providing the necessary details to ensure reproducibility in future research.
Pre-training phase. Table 17 provides comprehensive summary of the hyperparameters employed for training the models on the original CIFAR-100 dataset for generating the distilled dataset.
Hyperparameters for Model Pre-training
Optimizer
SGD
Learning Rate
0.1
Weight Decay
1e-4
Momentum
0.9
Batch Size
128
Epoch Scheduler Augmentation
50
Cosine Annealing
RandomCrop. Horizontal Flip
Loss Function
Cross-Entropy
Table 17: Hyperparameters for CIFAR-100 Pre-trained Models.
Table 18 outlines the hyperparameter configurations employed for the post- evaluation phase on the Distilled CIFAR-IOW dataset.
Evaluation Phase.
G.2.2 Tiny-ImageNet
This part describes the hyperparameter settings used in the Tiny-ImageNet experiments, offering comprehensive details to facilitate reproducibility for future studies.
33

---

Hyperparameters for Post-Eval on R18, R50 and R101
Optimizer
AdamW
S1
IPCI (R50), IPCS0 (R18,R50)
S2
IPCI0 (R18, R50)
S3
IPC1 (R101), IPC10 (R101),
IPC50 (R101)
S4
IPCI (R18)
Soft Label Generation
BSSL
Loss Function
KL-Divergenee
Batch Size
16
Epochs
1000
Augmentation
RandomResizeCCrop.
Horizontal Flip, CutMix
Table 18: Hyperparameters for post-evaluation task ResNet18, ResNet50 and ResNet101 for CIFAR-100.
Pre-training phase. Table 19 presents detailed overview of the hyperparameters used for model training on the original Imag- dataset.
Hyperparameters for Model Pre-training
Optimizer
SGD
Learning Rate
0.1
Weight Decay
le-4
Momentum
0.9
Batch Size
64
Epoch Scheduler Augmentation
150
Cosine Annealing
RandomCrop, Horizontal Flip
Loss Function
Cross-Entropy
Table 19: Hyperparameters for Imay-ema Pre-trained Models.
Hyperparameters for Post-Eval on R18, R50 and R101
Optimizer
AdamW
IPC50 (R18)
S2
IPC1 (R18) IPC10 (R18)
S3
IPC50 (R50, R101)
S4
IPC1 (R50, R101) IPC10 (R50,
R101)
Soft Label Generation
BSSL
Loss Function
KL-Divergence
Batch Size
16
Epochs
300 (IPC10. IPC50), 1000 (IPCI)
Augmentation
RandomResizedCrop.
Horizontal Flip, CutMix
Table 20: Hyperparameters for post-evaluation task ResNet18, ResNet50 and ResNet101 for Imag-em
34

---

Evaluation Phase. Table 20 details the hyperparameter settings used during the post-evaluation phase on the Distilled Tiny-Imagel dataset.
G.2.3 ImageNette
This subsection describes the hyperparameter settings utilized in the ImageNette experiments, offering detailed information facilitate reproducibility for subsequent studies.
Pre-training phase. Table 21 summarizes the hyperparameters used for training models on the original ImageNette dataset to generate the distilled dataset, ensuring clarity and reproducibility.
Hyperparameters for Model Pre-training
Optimizer
SGD
Learning Rate
0.01
Weight Decay
lc-4
Momentum
0.9
Batch Size
128
Epoch
300
Scheduler
Cosine Annealing
Augmentation
RandomReizecrop Horizontal Flip
Loss Function
Cross-Entropy
Table 21: Hyperparameters for ImageNette Pre-trained Models.
Evaluation Phase. Table 22 details the hyperparameter settings applied during the post-evaluation phase on the Distilled ImageNette dataset.
Hyperparameters for Post-Eval on R18, R50 and R101
Optimizer
Adam
S2
IPC50 (R101)
S3
IPC10 (R18, R50) IPC50(R50)
S4
IPC1(R18, R50, R101) IPC10
(R101) IPC50 (R18)
Soft Label Generation
BSSL
Loss Function
KL-Divergenee
Batch Size
16
Epochs
300
Augmentation
RandomResizedCrrp.
Horizontal Flip. CutMix
Table 22: Hyperparameters for post-evaluation task on ResNet18, ResNet50 and ResNet101 for ImageNette.
G.2.4 Imog
This section describes the hyperparameter settings used the Imof experiments, offering detailed information to facilitate reproducibility for future studies.
Pre-training phase. Table 23 presents detailed overview of the hyperparameters utilized for training models on the original Image Woof dataset to produce the distilled dataset.
Evaluation Phase. Table 24 presents the hyperparameter settings utilized during the post-evaluation stage on the distilled Image Woof dataset, detailing the configurations applied for performance assessment.
G.2.5 ImageNet-1K
This subsection outlines the hyperparameter configurations employed in the ImageNet- experi- ments, providing the necessary details ensure reproducibility in future research.
35

---

Hyperparameters for Model Pre-training
Optimizer
SGD
Learning Rate
0.1
Weight Decay
le-4
Momentum
0.9
Batch Size
128
Epoch
50
Scheduler
Cosine Annealing
Augmentation
RundomResizedCrop. Horizontal Flip
Loss Function
Cross-Entropy
Table 23: Hyperparameters for Image Woof Pre-trained Models.
Hyperparameters for Post-Eval R18, R50 and R101
Optimizer
AdamW
SI
IPCI (R101)
S2
IPC50 (R18)
S3
IPC10 (RI8, R50) IPC50 (R50,
R101)
S4
IPCI (R18, R50) IPC10 (R101)
Soft Label Generation
BSSL
Loss Function
KL-Divergence
Batch Size
16
Epochs
300
Augmentation
RandomResizedCrop.
Horizontal Flip, CutMix
Table 24: Hyperparameters for post-evaluation task on ResNet18, ResNet50 and ResNet101 for ImageWoof.
Pre-training phase. For ImageNet-IIK, we employed the official PyTorch pretrained models, which have been extensively trained on the full ImageNet- IK dataset.
Hyperparameters for Post-Eval on R18, R50 and R101
Optimizer
AdamW
SI
IPC50 (R18, R50)
$2
IPC1 (R18) IPC10 (R18, R50, R101)
S3
IPC50 (R101)
$4
IPCI (R50, R101)
Soft Label Generation
BSSI.
Loss Function
KL-Divergence
Batch Size
16
Epochs
300
Augmentation
RandomResizedCrop.
Horizontal Flip, CutMix
Table 25: Hyperparameters for post-evaluation task on ResNet18, ResNet50 and ResNet101 for ImageNet-IK.
Evaluation Phase. Table 25 provides a detailed overview of the hyperparameter settings used during the post-evaluation phase the distilled ImageNet-Ie IK dataset.
36

---

H
Additional Distilled Data Visualization
Additional visualizations of the distilled data generated by FADRM are provided in Fig. 7 (CIFAR- 100), Fig. 9 (Tiny-Image Fig. 11 (ImageNette), Fig. 13 (ImageWoof), and Fig. 15 (ImageNet-Ih IK). Furthermore, enhanced versions FADRM+ are presented in Fig. 8 (CIFAR-100) 100), Fig. 10 (Tiny- ImageNet), Fig. 12 (ImageNette), Fig. 14 (ImageWoof), and Fig. 16 (ImageNet-IK).
Figure 7: Visualization of synthetic data CIFAR-100 generated by FADRM.
37

---

Figure 8: Visualization of synthetic data CIFAR- 100 generated by FADRM+.
38

---

Figure 9: Visualization of synthetic data on Finy-ImaeeNet generated by FADRM.
39

---

Figure 10: Visualization of synthetic data on Tiny-Imagen generated by FADRM+.
40

---

000
Figure 11: Visualization of synthetic data ImageNette generated by FADRM.
41

---

Figure 12: Visualization of synthetic data ImageNette generated by FADRM+
42

---

Figure 13: Visualization of synthetic data ImageWoof generated by FADRM.
43

---

Figure 14: Visualization of synthetic data ImageWoof generated by FADRM+
44

---

Figure 15: Visualization of synthetic data on ImageNet-IK IK generated by FADRM.
45

---

Figure 16: Visualization of synthetic data on ImageNet- 1K generated by FADRM+.
46


Reviews:
[{"review_id": "6xWd2PaWDI", "reviewer": "Reviewer_xpuv", "summary": "This research theoretically points out that the mutual information between original dataset and synthetic dataset generated by uni-level decoupled dataset distillation methods has upper bound. To mitigate this problem, this research suggests the FADRM framework, which leverages the data-level skip connection concept. FADRM framework has two components: Multi-resolution optimization (MRO) to enhance efficiency and Adjustable residual connection (ARC) to mitigate the information vanishing. Through the diverse experimental results, FADRM shows the efficiency and effectiveness improvement.", "questions": "I like the paper overall and would be happy to raise my score if the authors address the few weaknesses and questions.\n\n1. Is there any reason why $\\tilde{x}$ should be optimized in the form of Cross-entropy + BN matching in Theorem 1? I also checked the proof in the appendix, but that condition is not used. I wonder if $f_\\theta$ trained and frozen with $\\mathcal{O}$ is sufficient to use the data processing inequality.\n\n2. Any further clarification on Theorem 2 would be appreciated. There is no sufficient interpretation of Theorem 2, while it contains many statements.\n\n3. I am curious about the advantages of MRO compared to synthetic image parameterization methods [1,2,3] and whether the FADRM framework can be used with synthetic image parameterization. I speculate that the synthetic image parameterization method might bypass the problem of carefully tuning the down-sampled size $D_{ds}$ of MRO as claimed by the authors. In particular, DDiF [4], with its resolution-agnostic nature, can enable a dynamically adjusted $D_{ds}$ rather than a static predefined $D_{ds}$.\n\n[1] Dataset Condensation via Efficient Synthetic-Data Parameterization\n\n[2] Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks\n\n[3] Dataset Distillation via Factorization\n\n[4] Distilling Dataset into Neural Field", "limitations": "See weaknesses and questions", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The theoretical analysis of the uni-level decoupled framework in Theorem 1 is impressive because it has an intuitive interpretation and clearly shows the limitations of uni-level methods.\n\n2. The basic concept of FADRM, Data-level skip connection is a novel idea that has not been proposed.\n\n3. This research has conducted many detailed experiments to prove the effectiveness of the proposed idea and shows high efficiency and performance improvement.\n\n**Weaknesses**\n1. Although this research have theoretically analyzed the limitations of the uni-level decoupled framework, the authors have not theoretically demonstrated the effectiveness of the proposed methodology FADRM in the same framework.\n\n2. (Minor) The simple application of Mixed Precision Training (MPT) for the sake of efficiency falls short in terms of originality and contribution.\n\n3. (Minor) If I understand correctly, budget $b$ or $\\mathcal{B}$ refers to the optimization steps. However, in dataset distillation, budget usually refers to storage budget in relation to IPC, so this usage can be confusing and I suggest replacing it with the appropriate word.", "was_revised": true, "final_justification": "This paper introduces a new problem in previous studies, information vanishing, with theoretical analysis and proposes a simple but effective methodology to alleviate it. I requested a further theoretical analysis and expandability of the proposed methodology, as well as additional explanations, and the authors' responses addressed most of my concerns. Therefore, I believe the paper is good enough to be accepted.", "rebuttal": "We sincerely thank you for your constructive comments. We are encouraged that you find our work novel with impressive theoretical insights and strong empirical validation. We would like to address the comments and questions below.\n\n\n> W1. While the limitations of the uni-level decoupled framework are discussed, the theoretical effectiveness of FADRM within this framework remains unproven.\n\nThanks for raising this point. We will formally analyze the impact of FADRM, particularly ARC to mitigate the limitations of the uni-level framework. \n\nIn FADRM's setting, $\\hat{x}$ is generated by using both $f\\_\\theta$ and real image patch from $\\mathcal{O}$ by iteratively fusing the real image patch to the intermediate optimized image. This introduces a new direct dependency:\n\n$$\n\\mathcal{O} \\rightarrow \\hat{x} \\leftarrow f\\_\\theta\n$$\n\nThis breaks the conditional independence:\n\n$$\n\\hat{x} \\text{ is not conditionally independent of } \\mathcal{O} \\text{ given } f_\\theta\n$$\n\nThus, the Markov structure assumed in Data Processing Inequality no longer applies. By the chain rule:\n\n$$\nI(\\hat{x}; \\mathcal{O}) = I(f\\_\\theta; \\hat{x}) + I(\\hat{x}; \\mathcal{O} \\mid f\\_\\theta)\n$$\n\n- In the uni-level framework:\n\n$$\nI(\\hat{x}; \\mathcal{O} \\mid f\\_\\theta) = 0\n$$\n\n- In our FADRM setting, the second term is strictly positive:\n\n$$\nI(\\hat{x}; \\mathcal{O} \\mid f\\_\\theta) > 0\n$$\n\nHence the mutual information is strictly increased in our FADRM settings.\n\n\n> W2. The use of Mixed Precision Training (MPT) primarily for efficiency lacks sufficient novelty.\n\nWe thank the reviewer for asking this question. We would like to clarify that our contribution lies not in the use of MPT itself, but in introducing optimization-level efficiency into dataset distillation, an area where most prior work focuses on architectural or initialization-level shortcuts (e.g., lighter backbones or real-image initialization). Our use of MPT is thus novel within this setting and highlights a promising direction for improving efficiency in future research.\n\n> W3. Misleading notation of \"budget\".\n\nThank you for the suggestion. We will revise the terms $b$ and  $\\mathcal{B}$ to $n\\_{\\text{iter}}$ and $N\\_{\\text{iter}}$, respectively, to improve clarity and avoid confusion.\n\n\n> Q1. Loss condition not used in the proof.\n\nWe thank the reviewer for raising this concern. The specific loss function (cross-entropy + BN matching) is not required for the proof; it is included to reflect typical objectives used in uni-level frameworks. Theorem 1 remains valid regardless of the loss formulation, as long as the original dataset $\\mathcal{O}$ is not involved and the pretrained model is kept frozen during optimization, following directly from the data processing inequality.\n\n> Q2. Further clarification on Theorem 2.\n\nWe appreciate the reviewer’s request for further clarification on Theorem 2. Below we provide an intuitive and formal explanation, focusing in particular on Equation 8.\n\nTheorem 2 aims to theoretically justify the generalization advantage of FADRM over training with uni-level framework's distilled data alone. Specifically, Equation 8 establishes a complexity gap between the original dataset $\\mathcal{O}$ and the uni-level framework's distilled data $\\tilde{\\mathcal{C}}^{\\text{res}}$:\n\n$$\n\\mathfrak{R}\\_n(\\mathcal{H} \\circ \\mathcal{O}) - \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\tilde{\\mathcal{C}}^{\\text{res}}) < -\\frac{L\\_h \\Delta (L\\_l + 2B\\alpha)}{2B}\n$$\n\nThis inequality reflects the assumption that uni-level framework's distilled samples $\\tilde{\\mathcal{C}}^{\\text{res}}$ being directly constructed to match the behavior of the target model, are easier to fit and therefore induce **higher Rademacher complexity** than orginal images. In contrast, $\\mathcal{O}$ contains richer semantic variability, which naturally regularizes overfitting, resulting in a lower complexity.\n\nTheorem 2 then shows that **combining uni-level framework with the structured real data** leads to a better generalization bound. Under the assumption, the generalization bound of $h\\_{\\mathrm{FADRM}}$ is rigorously shown to be tighter than that of $h\\_{\\mathrm{res}}$, i.e.,\n\n$$\n\\widehat{\\mathcal{L}}\\_{\\mathrm{FADRM}} + 2B \\cdot \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\tilde{\\mathcal{C}}\\_{\\mathrm{FADRM}}) \n< \\widehat{\\mathcal{L}}\\_{\\mathrm{res}} + 2B \\cdot \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\tilde{\\mathcal{C}}^{\\text{res}}).\n$$\n\nThe insight of this theorem is that, when synthetic data is highly optimized and thus induces greater hypothesis complexity, combining it with the more structured and regular original data can lead to a tighter generalization bound, accounting for both empirical risk and theoretical analysis.\n\n\n> Q3 (1): Comparison betwwen image parameterization and MRO.\n\nWe thank the reviewer for this insightful question. **Multiple Resolution Optimization (MRO)** is a *training efficiency technique* that reduces the computational cost of pixel-level optimization by periodically operating at lower resolutions. In contrast, **image parameterization** replaces pixel optimization with a latent code and a shared decoder, aiming to reduce parameter count and improve representational compactness.\n\nKey advantages of MRO over image parameterization include:\n\n- **Lower Hyperparameter Sensitivity**: MRO introduces only a single downsampling factor $D_{\\text{ds}}$, while parameterized methods require careful tuning of latent size, decoder architecture, and regularization.\n\n- **Preservation of Fine Details**: MRO retains full-resolution pixel expressiveness, which is crucial for high-resolution datasets like ImageNet.\n\n- **Simplicity and Broad Compatibility**: MRO is easy to implement, requires no auxiliary modules, and can be seamlessly integrated into any optimization-based distillation framework.\n\n\n> Q3 (2) Apply image parameterization within FADRM Framework.\n\nWe thank the reviewer for this insightful question. **FADRM is compatible with image parameterization techniques such as HaBa, etc**. In this setting, instead of initializing pixel-space learnable synthetic images, we initialize trainable latent bases and trainable hallucinator networks. During training, images are decoded via the hallucinators, and the FADRM loss is computed on the decoded outputs. Gradients are backpropagated to update both components. Residual feature injections can still be applied periodically (e.g., every 500 iterations) to enhance information retention. Additionally, MRO can be integrated by performing forward passes at lower resolutions to reduce training cost.\n\nWe appreciate the reviewer’s suggestion and believe that combining image parameterization with uni-level frameworks opens up promising directions for future research.\n\n\n> Q3 (3) Adaptive $D_{\\text{ds}}$.\n\nWe thank the reviewer for the thoughtful question. While DDiF is resolution-agnostic during generation, it still requires a fixed output resolution and cannot dynamically select the optimal scale for each image. Therefore, it does not directly help determine the downsampling factor $D_{\\text{ds}}$ automatically in our setting.\n\nTo avoid manually select $D_{\\text{ds}}$, we use an adaptive strategy based on total variation (TV) to estimate image smoothness. Smoother images can tolerate greater downsampling without loss of fidelity. Given a batch $x \\in \\mathbb{R}^{B \\times C \\times H \\times W}$:\n\n$$\n\\text{TV}(x) = \\frac{1}{B} \\sum_{i=1}^{B} \\left( \\| x_i[:, 1:, :] - x_i[:, :-1, :] \\|_1 + \\| x_i[:, :, 1:] - x_i[:, :, :-1] \\|_1 \\right)\n$$\n\nWe compute:\n$$\n\\text{ShrinkRatio} = \\min\\left(0.8,\\ \\frac{0.5}{\\max(\\text{TV}, 10^{-4})} \\right),\\quad D_{\\text{ds}} = \\max\\left( \\lfloor D \\cdot \\text{ShrinkRatio} \\rfloor,\\ 128 \\right)\n$$\n\nAs shown below, this adaptive strategy achieves performance on par with manual tuning:\n\n|                  | Manually Selected |   Adaptive   |\n|:----------------:|:-----------------:|:------------:|\n|  ImageNet-1K     |      47.7%        |    47.5%     |\n|  CIFAR-100       |      61.5%        |    61.2%     |\n\nWe will include these results in the revision and thank the reviewer for inspiring this direction."}, {"review_id": "ioCirr2iKq", "reviewer": "Reviewer_ccAs", "summary": "This paper introduces FADRM (Fast and Accurate Data Residual Matching), a novel dataset distillation framework that extends residual connections to the data level to address the issue of information vanishing in synthetic data. The method uses data-level skip connections and multi-resolution optimization (MRO) to preserve both local and global information while significantly reducing memory and time costs. By injecting residuals from the original patches during the iterative optimization process, FADRM ensures better feature retention and faster convergence. Empirical results across CIFAR-100, Tiny-ImageNet, and ImageNet-1K show FADRM's advantage in accuracy, efficiency, and cross-architecture generalization.", "questions": "How sensitive is the method to the choice of merge ratio and number of residual injections, and could these parameters be learned or adapted during training?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1: The core idea of extending residual connections from model-level to data-level is novel. It aims to address the information vanishing issue common in uni-level dataset distillation.\n\nS2: The proposed Multi-Resolution Optimization (MRO) and Mixed Precision Training (MPT) reduce computational costs and makes the method scalable to large datasets like ImageNet-1K.\n\nS3: The method consistently outperforms prior SOTA on several benchmarks across both accuracy and efficiency, with ablations and cross-architecture generalization evaluations.\n\nW1: The method requires pretrained feature extractors (with fixed BatchNorm) to work. It’s unclear how robust the method is if the pretrained model is weak or mismatched to the downstream task.\n\nW2: The theorems give motivation and justification for ARC (and MRO), but they aren't tightly integrated into the method design or used to guide any decisions so it reads a bit post-hoc. It would help if the authors can discuss how to make ARC adaptive or grounded the merge ratio in theory or training signals.\n\nW3: The paper doesn't discuss where FADRM fails or when ARC/MRO might hurt performance. This makes the method look overfitted to certain hyperparameters.\n\nW4: The paper talks a lot about information vanishing and density but lacks a well-defined, generalizable metric or visualization to evaluate it quantitatively.", "was_revised": true, "final_justification": "Thanks for the authors' response, and they have addressed most of my concerns - I've raised the score accordingly.", "rebuttal": "We sincerely thank you for your constructive comments. We are encouraged that you find our work novel with strong empirical performance on both efficiency and effectiveness. We would like to address the comments and questions below.\n\n> W1: Concern if weak or mismatched, distillation may degrade.\n\nWe thank the reviewer for the insightful comment.\n\nAs shown in Theorem 1, FADRM, like other uni-level distillation methods, is most effective when the pretrained model not too confident (relative weak), i.e., when its output has high entropy. In this regime, the mutual information upper bound is looser, allowing distilled data to better capture information from the original dataset. This is supported by both theory (Theorem 1) and experiments (see table below).\n\n|  Squeezed Epochs | Cross-Entropy   | FADRM Accurracy (%) |\n|:----------------:|:---------------:|:---------------:|\n|  50              |      0.0338     |         61.5    |\n|  100             |      0.0054     |         59.6    |\n|  150             |      0.0031     |         58.7    |\n|  200             |      0.0028     |         58.3    |\n\nRegarding architectural mismatch, our original submission includes cross-architecture results (Table 2, right), where FADRM consistently outperforms baselines. We further reinforce this with additional experiments in the table below on ImageNet-1k IPC=50.\n\n|  Models         | RDED | FADRM  | EDC | FADRM+ |\n|:---------------:|:----:|:------:|:---:|:------:|\n|  DeiT-Tiny      | 44.5 |**46.3**| 55.0|**55.3**|\n|  ConvNext-Tiny  | 65.4 |**66.8**| 66.6|**67.4**|\n|  Swin-Tiny      | 56.9 |**58.2**| 63.3|**64.7**|\n|  MobileNet-V2   | 53.9 |**54.1**| 57.8|**58.6**|\n|  ShuffleNet-V2  | 30.9 |**31.2**| 45.7|**46.3**|\n|  EfficientNet-B0| 57.6 |**58.4**| 60.9|**61.9**|\n\n\n> W2: Ground the merge ratio in theoretical anlysis.\n\nWe thank the reviewer for the question. To ground the choice of merge ratio $\\alpha$ in theory, we define the generalization bound of FADRM as:\n\n$$\nD(\\alpha) =  \\widehat{\\mathcal{L}}\\_{\\text{FADRM}} + 2B \\cdot \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\tilde{\\mathcal{C}}\\_{\\text{FADRM}})\n$$\n\nAssuming the loss is $\\lambda$-strongly convex and letting $\\Delta := \\frac{1}{n} \\sum\\_i \\|\\tilde{x}\\_i^{\\text{res}} - x\\_i\\|$, we have:\n\n$$\n\\widehat{\\mathcal{L}}\\_{\\text{FADRM}} \\le \\alpha \\widehat{\\mathcal{L}}\\_{\\text{res}} + (1 - \\alpha) \\widehat{\\mathcal{L}}\\_{\\mathcal{O}} - \\frac{\\lambda}{2} \\alpha(1 - \\alpha)\\Delta\n$$\n\nand, as shown in Appendix A.3:\n\n$$\n\\mathfrak{R}\\_n(\\mathcal{H} \\circ \\tilde{\\mathcal{C}}\\_{\\text{FADRM}}) \\le \\alpha \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\tilde{\\mathcal{C}}\\_{\\text{res}}) + (1 - \\alpha) \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\mathcal{O}) + L\\_h \\cdot \\alpha(1 - \\alpha)\\Delta.\n$$\n\nSubstituting both yields:\n\n$$\nD(\\alpha) \\le X\\alpha + Y(1 - \\alpha) + Z(\\alpha - 1)\\alpha,\n$$\n\nwhere:\n- $X := \\widehat{\\mathcal{L}}\\_{\\text{res}} + 2B \\cdot \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\tilde{\\mathcal{C}}\\_{\\text{res}})$  \n- $Y := \\widehat{\\mathcal{L}}\\_{\\mathcal{O}} + 2B \\cdot \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\mathcal{O})$  \n- $Z := \\frac{\\lambda}{2} \\Delta - 2B L_h \\Delta$\n\nMinimizing this quadratic yields the optimal merge ratio:\n\n$$\n\\alpha^\\star = \\frac{1}{2} + \\frac{Y - X}{2Z}.\n$$\n\nSince in practice we use temperature-scaled KL divergence, the strong convexity constant satisfies $\\lambda \\approx 1 + (C - 1)\\exp(-\\frac{d}{T})$, which approaches the number of classes $C$ as the temperature $T$ increases. Consequently, the bias term $\\frac{Y - X}{2Z}$ becomes negligible relative to the dominant $\\lambda$ term, and the optimal merge ratio $\\alpha^\\star$ remains close to 0.5.\n\nThis aligns with our empirical results in Table 4, where $\\alpha = 0.5$ performs consistently well across datasets. Thus, our selection is both theoretically grounded and empirically validated.\n\n\n> W3: Discussion where FADRM fails.\n\nWe appreciate the reviewer’s comment and would like to clarify that the potential failure modes of FADRM to certain hyperparameters are indeed discussed:\n\n- MRO (lines 240–243): We show that applying MRO without ARC leads to performance degradation, as the reduced resolution impairs the representational fidelity without the residual correction.\n- Merge ratio $\\alpha$ (lines 243–248): We demonstrate that a large $\\alpha$ leads to excessive reliance on the optimized data, reducing robustness to overfitting.\n- Residual frequency $k$ (lines 262–267): We observe that overly frequent residual injection introduces redundancy and leaves insufficient iterations per stage, leading to degraded synthetic images.\n\nWe agree these points deserve further emphasis and we will include an explicit subsection in the revision to summarize failure case. Thank you again for pointing this out.\n\n> W4: Lacks a clear metric to quantify information vanishing.\n\nWe appreciate the reviewer’s suggestion. To better quantify the issue of information vanishing, we report the generalization accuracy of models trained on distilled images across varying optimization steps (500–4000 iterations). As shown in the table below, prolonged optimization without ARC leads to performance degradation, indicating the issue of information vanishing. In contrast, applying ARC enables continued performance gains, demonstrating its ability to extract additional information during extended optimization. We will include and visualize this metric in the revision.\n\n| Iterations | FADRM w/ ARC (%) | FADRM w/o ARC Accuracy (%) |\n|:----------:|:------------------:|:--------------------:|\n| 500        |    40.0            |    39.7              |\n| 1000       |    45.2            |    44.8              |\n| 1500       |    47.2            |    46.5              |\n| 2000       |    47.7            |    46.1              |\n| 2500       |    48.0            |    46.5              |\n| 3000       |    48.0            |    45.4              |\n| 3500       |    48.2            |    44.5              |\n| 4000       |    48.4            |    43.1              |\n\n\n> Q1 (1): Sensitiveness of the Hyper-parameter.\n\nWe thank the reviewer for the question. We clarify that FADRM is robust to both the merge ratio $\\alpha$ and the number of residual injections $k$.\n\nFor $\\alpha$, our theoretical analysis (see response to W2) suggests that the optimal value lies near 0.5. Empirically (Table 4), performance remains stable around this value, with less than 0.7% variation. Notably, $\\alpha = 0.5$ achieves the best result on ImageNet-1K, and consistent trends are observed on CIFAR-100 (see table below), where $\\alpha = 0.5$ also yields the strongest performance.\n\n|  $\\alpha$        | FADRM Accurracy (%)|\n|:----------------:|:---------------:|\n|  0.9             |       58.0      |\n|  0.8             |       58.9      |\n|  0.7             |       58.7      |\n|  0.6             |       59.6      |\n|  0.5             |     **61.5**    |\n|  0.4             |       59.2      |\n|  0.3             |       58.6      |\n\n\nFor $k$, moderate values (e.g., 2–4) balance optimization depth and information retention. Performance varies within 0.2% (ImageNet-1K) and 1% (CIFAR-100) as shown in left panel of table 5, with $k{=}3$ performs the best across diferent datasets.\n\nOverall, FADRM requires minimal tuning and $k=3$ and $\\alpha=0.5$ performs stably across different datasets.\n\n\n> Q1 (2): Adaptive Hyper-parameters during training.\n\nWe thank the reviewer for this insightful question. While our method supports adaptive adjustment of hyperparameters, we emphasize that the default configuration, merge ratio $\\alpha = 0.5$ and number of residual injections $k = 3$, already yields strong and stable performance across diverse settings.\n\nAs established in our theoretical analysis, the optimal merge ratio $\\alpha$ lies near 0.5. The remaining two hyperparameters are the downsampled input resolution $D\\_{\\text{ds}}$ and the number of residual injections $k$, which determines when residual information from the original dataset is injected.\n\nTo avoid *information vanishing* caused by prolonged optimization in each stage, we inject residuals only when the current synthetic image is deemed converged. We adopt the following convergence criterion:\n\n- Cross-entropy loss $\\mathcal{L}\\_{\\text{CE}} < 0.01$  \n- BatchNorm regularization $\\mathcal{L}\\_{\\text{BN}} < 60$\n\nTo adaptively determine $D\\_{\\text{ds}}$, we use total variation (TV) to quantify image smoothness. Smoother images can be optimized at lower resolutions without loss of fidelity. Specifically, for a batch of images $x \\in \\mathbb{R}^{B \\times C \\times H \\times W}$:\n\n$$\n\\text{TV}(x) = \\frac{1}{B} \\sum\\_{i=1}^{B} \\left( \\| x\\_i[:, 1:, :] - x\\_i[:, :-1, :] \\|_1 + \\| x\\_i[:, :, 1:] - x\\_i[:, :, :-1] \\|\\_1 \\right)\n$$\n\nThe downsampling factor is then computed as:\n\n$$\n\\text{ShrinkRatio} = \\min\\left(0.8, \\frac{0.5}{\\max(\\text{TV}, 10^{-4})} \\right), \\quad D\\_{\\text{ds}} = \\max\\left( \\lfloor D \\cdot \\text{ShrinkRatio} \\rfloor,\\ 128 \\right)\n$$\n\nAs shown in the experiments blow on IPC=10, this procedure achieves performance comparable to manually tuned configurations. We will include these results in the revision. We agree this is a promising direction and sincerely thank the reviewer for highlighting it.\n\n|                  | Manually Selected |      Adaptive    |\n|:----------------:|:-----------------:|:----------------:|\n|  ImageNet-1k     |       47.7 %      |      47.5 %      |\n|  CIFAR-100       |       61.5 %      |      61.2 %      |"}, {"review_id": "4XDy1DD4WO", "reviewer": "Reviewer_Ujgz", "summary": "This paper introduces a novel data residual matching strategy for database distillation, aiming to significantly reduce training time and memory usage. The approach demonstrates substantial improvements in both computational efficiency and model performance, reducing resource demands by 50% compared to baseline methods.", "questions": "**(Q1)** What task was concerned here for performance evaluation, and why was it chosen? Was it image classification?\\\n**(Q2)** What models were trained using the distilled datasets? Were the training configurations (e.g., hyperparameters, epochs) kept consistent across all methods/models?", "limitations": "The paper does not clearly discuss its limitations. In particular, the generalizability of the method to architectures beyond ResNet is a concern that should be addressed, especially when the proposed approach may rely on residual connections.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n**(S1)** The proposed method yields clear performance gains over baseline distillation techniques.\\\n**(S2)** Training time and memory consumption are significantly reduced, making the approach highly practical for real-world applications where efficiency is crucial.\n\n\n**Weaknesses**\n\n**(W1)** There seems a labelling error in Figure 2, where the “downsample” and “upsample” lables in the resampler module appear to be mistakenly swapped.\\\n**(W2)** The evaluated task is not clearly stated. While it appears to be image classification, this must be explicitly mentioned for clarity.\\\n**(W3, Major)** The models used for training and evaluation are not clearly described. Table 1 suggests the use of ResNet-based architectures, but the paper should clearly state which models were used and whether the approach generalizes to non-ResNet architectures. It is unclear whether the selection here is related to the design of the residual blocks in the proposed framework. This lack of clarity raises a major concern regarding the paper’s methodology.\\\n**(W4 - Minor)** Figure 2 could be misleading, as it shows an equal number of patches for the original and distilled datasets. Typically, the distilled set should contain significantly fewer samples.\\\n**(W5)** The peak memory results for FADRM+ in Tables 2 and 4 differ, while those for the other models remain consistent. Could you clarify the reason for this inconsistency?", "was_revised": true, "final_justification": "My concerns have been properly addressed by the authors, and after reviewing the comments from other reviewers, I have decided to raise my rating from 3 to 4.", "rebuttal": "We sincerely thank you for your constructive comments. We are encouraged that you found our method both novel and practically valuable, and appreciate your recognition of its performance and efficiency improvements. We would like to address the comments and questions below.\n\n> W1: Labelling Error.\n\nWe thank the reviewer for pointing this out. We will eliminate this error in the revised version to accurately reflect the intended operations within the resampler module.\n\n> W2: Evaluation task clarification.\n\nWe thank the reviewer for pointing this out. We confirm that the evaluated task throughout the paper is indeed image classification, and we will explicitly clarify this in the revised version to avoid any ambiguity.\n\n> W3: Ambiguity in model usage and generalization claims.\n\nWe thank the reviewer for raising this important concern. To clarify:\n\n1. **Training vs. Evaluation Models:**  \n   If we understand the question correctly, *training* refers to the models used to optimize the distilled dataset, while *evaluation* refers to the models trained on the distilled data to assess generalization. We have already provided this information in the original submission:\n   - For single-model distillation (FADRM), the training model is stated in the caption of **Table 1**.\n   - For multi-model distillation (FADRM+), the training models are detailed in **Table 7** of the Appendix.\n   - The evaluation models used are listed in the **top row of Table 1**.\n\nWe **fully agree that these details are crucial** to the clarity and reproducibility of our method. To avoid any ambiguity, we will consolidate this information and explicitly state it in the main paper.\n\n2. **Generalization Beyond ResNet:**  \n   Our distilled datasets are designed to generalize beyond the ResNet family. As shown in the **right block of Table 2** in the main paper, we have already conducted cross-architecture evaluations to demonstrate that our method generalizes beyond the ResNet family. To further strengthen this point, we also include **additional results** in the table (ImageNet-1k IPC=50) below to more comprehensively support the cross-architecture generalizability of our approach.\n   \n|  Models         | RDED | FADRM  | EDC | FADRM+ |\n|:-----------------:|:------:|:--------:|:-----:|:--------:|\n|  DeiT-Tiny      | 44.5 |**46.3**| 55.0|**55.3**|\n|  ConvNext-Tiny  | 65.4 |**66.8**| 66.6|**67.4**|\n|  Swin-Tiny      | 56.9 |**58.2**| 63.3|**64.7**|\n|  MobileNet-V2   | 53.9 |**54.1**| 57.8|**58.6**|\n|  ShuffleNet-V2  | 30.9 |**31.2**| 45.7|**46.3**|\n|  EfficientNet-B0| 57.6 |**58.4**| 60.9|**61.9**|\n\n\nWe appreciate the reviewer’s thoughtful feedback and will revise the manuscript accordingly to enhance clarity and completeness.\n\n> W4: Clarification for Figure 2.\n\nWe thank the reviewer for raising this point. We would like to clarify that in our framework, the original data patches shown in Figure 2 are not meant to represent the full original dataset, but rather a subset **selected as initialization** for the distilled dataset. Specifically, for a target of *N* images per class (e.g., 50 IPC), we first select *N* patches per class from the original dataset, which are then used as the initialization for optimization. Therefore, the number of original and distilled patches is intentionally the same at this stage. To avoid confusion, we will revise the caption of Figure 2 to better explain this initialization process.\n\n> W5: Inconsistent memory results.\n\nWe sincerely thank the reviewer for pointing this out. Upon careful re-examination, we confirm that this was indeed a reporting error, the correct peak GPU memory usage should be 11.0 GB. We apologize for the oversight. Importantly, this correction does not affect any of the experimental conclusions or comparisons in the paper. We will update the result accordingly in the revision.\n\n> Q1: Performance evaluation task.\n\nWe thank the reviewer for the question. Yes, the downstream task used for performance evaluation throughout the paper is **image classification**.\n\nThis task was chosen because, given only class labels and distilled images, image classification provides a standardized way to assess whether the synthetic data captures sufficient discriminative information for training a performant classifier. It is also the **standard evaluation protocol** in the dataset distillation literature, particularly for the image modality. Therefore, it provides a fair and widely accepted basis for comparison across methods.\n\n> Q2: Trained Models and hyper-parameter configurations.\n\nWe appreciate the reviewer’s question.\n\nIn our main experiments (Table 1), we trained **ResNet-18**, **ResNet-50**, and **ResNet-101** on the distilled datasets. To further evaluate the generalizability of our method across architectures, we conducted **cross-architecture experiments (Table 2 right)** involving:\n- **Transformer-based models**: Swin-Tiny, ConvNext-Tiny, DeiT-Tiny\n- **Lightweight CNNs**: MobileNetV2, ShuffleNetV2  \n- **High-capacity models**: EfficientNet, DenseNet201, Wide ResNet50-2\n\nAll models were trained following the configuration used in EDC: 1000 epochs for CIFAR-100 and Tiny-ImageNet IPC=1, and 300 epochs for all other datasets, to ensure consistency and fair comparison across methods. Other hyperparameters (e.g., learning rate, optimizer) were selected based on model capacity and the complexity of the distilled dataset. Detailed training configurations are provided in Appendix Section F."}, {"review_id": "f8vHAp1udT", "reviewer": "Reviewer_Ne6j", "summary": "This paper identifies the information vanishing problem in uni-level frameworks and addresses it with Adjustable Residual Connection (ARC), which applies the idea of model-level skip connections at the data level. It also introduces Multi-Resolution Optimization (MRO), which distills datasets at lower resolutions to improve training efficiency. Mixed Precision Training (MPT) is used to further reduce training time and memory use. The proposed method, FADRM, achieves state-of-the-art results in both accuracy and efficiency across multiple benchmarks.", "questions": "Could the authors clarify why the evaluation criteria used for time efficiency comparisons differ between Table 2 (left) and Table 3?\nLines 289-291 indicate that MRO reduces peak memory usage. However, Table 3 shows that MPT alone reduces peak memory to 2.9GB. Could the authors clarify whether the observed memory reduction is indeed due to MRO rather than MPT?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n\n(+) The paper provides a clear theoretical justification and empirical validation for the phenomenon of information vanishing in uni-level frameworks, effectively defining and analyzing this critical issue.\n\n(+) The authors present an original approach by interpreting model-level skip connections as data-level skip connections, successfully preserving critical information from the original dataset during the distillation process.\n\nWeakness\n\n(-) It remains unclear whether the reported improvements in computational efficiency attributed to MRO are genuinely derived from this approach itself, or if they primarily result from MPT.\n\n(-) Although Table 3 demonstrates notable performance gains when comparing FADRM to SRe^2L++, the reported efficiency and GPU memory are same. Thus, it is questionable whether FADRM truly offers substantial efficiency enhancements over SRe^2L++.\n\n(-) Table 1 indicates minor performance gains for ImageNet-1K with ResNet-18. However, without reported statistical variability from multiple runs, commonly provided in other baselines, assessing the statistical significance of these improvements and making a rigorous comparison with baselines is difficult.", "was_revised": true, "final_justification": "The authors’ clarifications, especially on efficiency, convinced me that MPT and MRO, though not major novelties individually, play meaningful roles in improving overall efficiency. The thorough responses have addressed my main concerns, leading to a borderline accept.", "rebuttal": "We sincerely thank you for your encouraging feedback. We appreciate your recognition of the clear theoretical justification and the novelty of our data-level residual design. We address the remaining concerns in the detailed responses below.\n\n> W1: Contribution of individual componenets on efficiency gains.\n\nThank you for the suggestion. To clarify the individual contributions of Mixed Precision Training (MPT) and Multi-Resolution Optimization (MRO) to computational efficiency, we conducted an ablation study as shown in the table below, the experiment is conducted on ImageNet-1k, where efficiency is measured based on the time needed for generating a single distilled image.\n\n| Settings           | Time Cost (s/img) | Minimum GPU Usage (GB) | Peak GPU Usage (GB)  |\n|:------------------:|:------------------:|:----------------------:|:--------------------:|\n| w/o MPT, w/o MRO   | 1.26               | 5.3                    | 5.3                  |\n| w/o MPT, w/ MRO    | 0.84               | 4.0                    | 5.3                  |\n| w/ MPT, w/o MRO    | 0.52               | 2.9                    | 2.9                  |\n| w/ MPT, w/ MRO     | 0.47               | 2.5                    | 2.9                  |\n\nAs shown, MRO is primarily intended to reduce the overall wall-clock time required to optimize a single image by iteratively performing training on downsampled versions. Since the process still involves full-resolution updates, the peak GPU usage remains unchanged. In contrast, MPT is designed to reduce both computation time and peak memory consumption by leveraging lower-precision operations throughout.\n\nWhile MPT accounts for the majority of the efficiency improvement, MRO provides complementary gains. For instance, when comparing the configurations with and without MRO under MPT, we observe a reduction of 0.05 seconds per image. Although this difference may appear marginal at the individual level, it becomes meaningful when scaled. For example, generating a 50 IPC ImageNet-1K dataset (i.e., 50,000 images) results in:\n\n$$\n50{,}000 \\times 0.05\\ \\text{seconds} = 2{,}500\\ \\text{seconds} \\approx 42\\ \\text{minutes}\n$$\n\nIn summary, MPT is the primary contributor to efficiency, and MRO further improves performance by iteratively optimze the images on a smaller resolution.\n\n\n> W2: Efficiency enhancements over SRe$^2$L++.\n\nThank you for the question. We clarify that SRe$^2$L++ does not adopt Mixed Precision Training (MPT) or Multi-Resolution Optimization (MRO), and requires 4000 iterations to optimize each image. In contrast, our proposed method FADRM applies both MPT and MRO, and completes optimization in only 2000 iterations. This results in substantial efficiency gains, as shown in Table 2 (left), including a reduction of approximately 2.05 seconds in wall-clock time per image and a significant decrease in peak GPU memory usage.\n\nTo illustrate the impact at scale, generating a 50 IPC ImageNet-1K distilled dataset (50,000 images) would lead to:\n\n$$\n50{,}000 \\times 2.05\\ \\text{seconds} = 102{,}500\\ \\text{seconds} \\approx 28.5\\ \\text{hours}\n$$\n\nThis demonstrates the practical advantage of FADRM over SRe$^2$L++ in large-scale settings. Additionally, we clarify that Table 3 serves a different purpose: it evaluates the impact of applying MPT across various methods and confirms that MPT does not degrade the quality of the distilled dataset. The analysis is not limited to FADRM and is intended to show the broader applicability of MPT in the field of dataset distillation.\n\n> W3: Clarification for performance gains.\n\nWe thank the reviewer for raising this important point.\n\nFirstly, we would like to clarify that the performance gains in the ImageNet-1K ResNet-18 setting are substantial. For single-model distillation, our method FADRM achieves clear improvements over the strongest baseline RDED. For example, at IPC = 10, FADRM outperforms RDED by +5.7%, which we believe is both meaningful and robust. For multi-model distillation, FADRM+ also consistently surpasses other approaches across all IPC levels, as reported in Table 1. The only case of relatively moderate improvement is at IPC = 50, where FADRM slightly outperforms CV-DD by +0.6%. \n\nTo confirm the consistency and statistical reliability of our results, we have conducted additional runs and report the mean and standard deviation across 3 independent trials on ImageNet-1K with ResNet-18, as shown in the table below (CV-DD didn't report statistical variations). The results further demonstrate the stability and robustness of FADRM across runs.\n\n| IPC | RDED (single) | FADRM (single) | CV-DD (multi) | EDC(multi) | FADRM+(multi) |\n|:-----------------:|:----------:|:----------:|:----------:|:----------:|:---------------:|\n| 1        | 6.6 $\\pm$ 0.2    |  7.1 $\\pm$ 0.3    | 9.2     | 12.8 $\\pm$ 0.1  | **14.3** $\\pm$ 0.2 |\n| 10       | 42.0 $\\pm$ 0.1   |  47.8 $\\pm$ 0.2   | 46.0    | 48.6 $\\pm$ 0.3  | **50.4** $\\pm$ 0.1|\n| 50       | 56.5 $\\pm$ 0.1   |  59.9 $\\pm$ 0.2   | 59.5    | 58.0 $\\pm$ 0.2  | **60.4**$\\pm$ 0.3 |\n\n> Q1: Clarification on evaluation criteria of efficiency.\n\nThank you for the question. There are two distinct evaluation criteria used to assess time efficiency in the paper: (1) seconds per image generation, as shown in Table 2 (left), and (2) milliseconds per image update, as shown in Table 3. The choice of metric depends on the evaluation objective.\n\nIn Table 2 (left), we report seconds per image generation, which captures the total wall-clock time required to generate a single distilled image. This metric accounts for various factors, including the total number of optimization iterations, the varying update time across different image resolutions introduced by MRO, and additional overhead such as loading precomputed statistics in EDC. We choose this metric because, under the MRO setting, image resolution changes throughout the optimization process, making it difficult to fairly measure efficiency using a single per-update timing. In contrast, seconds per image generation provides a more comprehensive and resolution-agnostic view of overall efficiency.\n\nIn contrast, Table 3 focuses on optimization-level efficiency under fixed-resolution settings, where only MPT is applied and MRO is excluded. In this case, we use milliseconds per image update to isolate and assess the efficiency gains specifically attributable to MPT.\n\nIn summary, we adopt different metrics to match the evaluation focus: Table 2 reflects overall generation efficiency, accounting for all contributing factors, while Table 3 isolates the per-iteration benefits of MPT without the confounding effects introduced by varying image resolutions.\n\n\n> Q2: Clarification on Peak Memory Usage.\n\nThank you for the question. We would like to clarify that in line 289, the phrase \"optimization-level refinement and MRO\" refers collectively to two efficiency enhancing components used in our method: optimization-level refinement corresponds to Mixed Precision Training (MPT), while MRO refers to Multi-Resolution Optimization. The sentence aims to highlight that the integration of both techniques leads to improved efficiency.\n\nTo answer the question directly: the observed reduction in peak GPU memory usage is primarily attributed to MPT, which reduces memory consumption by employing lower-precision arithmetic during optimization. In contrast, MRO is designed to improve overall training efficiency by reducing computation time, as discussed in our response to Weakness 1.\n\nWe hope this clarifies that the efficiency gains mentioned in line 289 result from the combined use of MPT and MRO, with MPT responsible for the peak memory reduction."}, {"review_id": "aQxR0PhQDh", "reviewer": "Reviewer_kJwL", "summary": "This paper proposes FADRM that introduces data-level residual connections to preserve important information during synthetic data generation. \nThe FADRM combines multi-resolution optimization and adjustable residual merging to improve both efficiency and accuracy.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n**S1**: The extension of the residual mechanism from model-level to data-level is conceptually novel.\n\n**S2**: The method achieves impressive performance across various datasets and architectures.\n\n\nWeaknesses:\n\n**W1**: \nFigure 2 is not clear.  \nWhether every **Data Residual Block** includes a **Recover**? \nWhy **Downsample** upsize the image size in **Resampler**, and vice versa?  \nThis increases the difficulty of understanding. \n\n**W2**: \nThis paper claims FADRM achieves Fast and Accurate matching.\nRDED is a relevant and efficient baseline, which is missing from Figure 1 and Table 2.\n\n**W3**:\nHow does the author explain the observation in Figure 3, where information vanishing still occurs even with ARC? Furthermore, why is there such a big shock compared to without ARC? This raises questions about whether ARC introduces instability into the distillation process. These phenomena should be discussed in detail.\n\n**W4**: \nTheorem 2 relies on the assumption in Eq.(8), which is crucial for concluding that FADRM generalizes better.  \nHowever, the paper does not provide empirical validation or intuitive justification for when this assumption holds.  \nThus, the theoretical advantage is conditional and its practical applicability is unclear.\nThe paper reports accuracy and runtime but never measures either side of Eq. (8), such as empirical Rademacher estimates, margin distributions or PAC-Bayes surrogates.", "was_revised": true, "final_justification": "I remain concerned about the novelty of this work, especially in terms of multi-resolution optimization, skip connections, and mixed-precision training, which are established techniques in the field. Reviewer Ne6j also noted this point. Therefore, I believe the manuscript requires further development before it can be considered suitable for publication.", "rebuttal": "We sincerely thank you for your constructive comments. We are encouraged that you found our method novel, and appreciate your recognition of its performance and efficiency improvements. We would like to address the comments and questions below.\n\n\n> W1: Figure 2 is not clear.\n\nWe sincerely thank the reviewer for the thoughtful question. To clarify, each Data Residual Block does indeed include a Recover step. We also appreciate the reviewer for pointing out the labeling issue in Figure 2, upon review, we confirm that the downsample and upsample labels were inadvertently swapped, which may have caused confusion. We will correct the labels and revise the figure to improve clarity in the final version.\n\n\n> W2: Comparison to non-optimization based methods.\n\nWe thank the reviewer for the question. RDED is a non-optimization-based framework that selects representative patches using a pretrained model, without involving any further image-level optimization. In contrast, our method is optimization-based and involves iterative refinement of synthetic images, which inevitably incurs higher computational cost. Nonetheless, such optimization typically leads to superior performance over non-optimization approaches.\n\nTo ensure a fair comparison of efficiency, we report results against optimization-based baselines that involve similar iterative procedures and are scalable to large-scale datasets such as ImageNet-1K. As a result, RDED is not included in Table 2.\n\n> W3: Further explanation of Information Vanishing.\n\nWe thank the reviewer for the question. First, we would like to clarify that our goal is not to eliminate information vanishing entirely, but to mitigate it. As optimization progresses, fine-grained local details are gradually suppressed in favor of dominant global structures. Once global information becomes saturated, continued suppression of local details inevitably leads to information vanishing.\n\nThe sharp surge observed in Figure 3 occurs as ARC injects rich fine-grained details back into the data. This sudden reintroduction of diverse semantic content naturally leads to a spike in feature-level entropy. Importantly, this surge reflects the restoration of previously suppressed local details, rather than instability or noise.\n\nTo verify that ARC preserves semantic consistency rather than introducing disruptive information, we conduct two complementary analyses. First, we measure the **cosine similarity of features** before and after ARC injection across three ARC stages ($k=3$). Second, we evaluate the **accuracy of the distilled images** before and after ARC using a pretrained model as a verifier.\n\nAs shown in the table below, feature-level cosine similarity remains high (≥ 0.88) across all stages, indicating that the semantic structure is well preserved. While verifier accuracy drops slightly (from 100% to 97%) at the first injection stage, it quickly recovers in later stages, suggesting that ARC introduces **semantically aligned residual content** rather than harmful noise. These results demonstrate that the observed entropy increase stems from meaningful details, not instability in the distillation process.\n\n|     Stages                    |  Feature Cosine Similarity   |   Before ARC Accuracy  |   After ARC Accuracy  |\n|:---------------------------:|:----------------------------:|:----------------------:|:---------------------:|\n| Residual Injection Stage #1 |           0.89              |         100%           |         97%           |\n| Residual Injection Stage #2 |           0.88              |         100%           |         100%          |\n| Residual Injection Stage #3 |           0.88              |         100%           |         100%          |\n\n\n> W4: More analysis on assumption 8.\n\nWe thank the reviewer for the insightful suggestion. We first elaborate on the intuition behind Assumption 8 and then empirically verify whether it generally holds. Starting from Assumption 8 and rearranging the right-hand side, we obtain:\n\n$$\n\\mathfrak{R}\\_n(\\mathcal{H} \\circ \\tilde{\\mathcal{C}}^{\\text{res}}) - \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\mathcal{O}) > \\left( \\frac{L\\_h L\\_\\ell}{2B} + \\frac{L\\_h}{2} \\right) \\cdot \\Delta,\n$$\n\nwhere $\\mathcal{O}$ denotes the original data patch, and $\\tilde{\\mathcal{C}}^{\\text{res}}$ refers to the class induced by the optimized version of the original patch.\n\nSince both $L\\_\\ell$ and $L\\_h$ are small due to the use of high-temperature KL loss and strong regularization (e.g., CutMix), the coefficient $\\left( \\frac{L\\_h L\\_\\ell}{2B} + \\frac{L\\_h}{2} \\right)$ is sufficiently small. As a result, the right-hand side becomes negligible in our setting and can be conservatively omitted to simplify the analysis, which is:\n\n$$\n\\mathfrak{R}\\_n(\\mathcal{H} \\circ \\tilde{\\mathcal{C}}^{\\text{res}}) > \\mathfrak{R}\\_n(\\mathcal{H} \\circ \\mathcal{O}),\n$$\n\nThe intuitions is that if the Rademacher complexity of the optimized patches exceed that of the original ones, theorem 2 typically holds.\n\nTo validate if this inequality holds empirically, we estimate the **empirical Rademacher complexity**, defined as:\n\n$$\n\\widehat{\\mathfrak{R}}\\_n(\\mathcal{H} \\circ \\mathcal{S}) = \\mathbb{E}\\_{\\sigma} \\left[ \\sup\\_{h \\in \\mathcal{H}} \\frac{1}{n} \\sum\\_{i=1}^n \\sigma\\_i h(x\\_i) \\right],\n$$\n\nwhere $\\sigma\\_i$ are i.i.d. Rademacher variables from $\\{-1, +1\\}$ and $\\mathcal{S}$ is the dataset. Although computing the population complexity is intractable, it is well known that:\n\n$$\n\\lim\\_{n \\to \\infty} \\widehat{\\mathfrak{R}}\\_n(\\mathcal{H}) = \\mathfrak{R}\\_n(\\mathcal{H}),\n$$\n\nso the empirical proxy becomes more accurate as $n$ increases.\n\nIn practice, we approximate this quantity via *random label memorization* on a IPC=50 dataset (n=50,000). A standard empirical estimate only involves binary classification, which we found to be too simple and less discriminative in our setting. To address this, we increase task difficulty by assigning **random labels uniformly from a large label space** (e.g., 500 and 1000 classes), making the memorization task substantially harder.\n\nConcretely, for a given dataset $\\mathcal{D} = \\{x\\_i\\}\\_{i=1}^n$, we construct a randomized label version $\\mathcal{D}\\_{\\text{rand}} = \\{(x\\_i, y\\_i^\\text{rand})\\}\\_{i=1}^n$ by sampling:\n\n$$\ny\\_i^\\text{rand} \\sim \\mathcal{U}\\{1, 2, \\dots, K\\}, \\quad \\text{where } K \\in \\{500, 1000\\}.\n$$\n\nWe apply this procedure to both the original patch set and its optimized counterpart, and train three distinct models on each using cross-entropy loss. The minimum training loss across runs is reported as a proxy for Rademacher complexity: lower loss indicates higher capacity to fit random labels, and thus higher empirical complexity.\n\nAcross all settings, whether under moderate (500-way) or challenging (1000-way) classification tasks, we consistently observe that the optimized data achieves **lower training loss** than the original. This suggests that the optimized data is easier to fit, reflecting a higher capacity of the hypothesis class to memorize random labels, thereby indicating a higher empirical Rademacher complexity.\n\n|  Models         | $\\mathcal{O}$ loss | $\\tilde{C}\\_{res}$ loss |\n|:-------------------------:|:----------:|:------:|\n|  ResNet-18 (500-way)      |   0.0022   |  0.0016 | \n|  ResNet-18 (1000-way)     |   0.0034   |  0.0026 | \n|  MobileNet-V2 (500-way)   |   0.0016   |  0.0004 | \n|  MobileNet-V2 (1000-way)  |   0.0019   |  0.0006 | \n|  ShuffleNet-V2 (500-way)  |   0.0457   |  0.0414 | \n|  ShuffleNet-V2 (1000-way) |   0.0309   |  0.0234 | \n\nThese findings offer strong empirical support for Assumption 8, suggesting that optimized patches indeed yield *higher empirical Rademacher complexity*. Therefore, the theorem 2 becomes applicable to real-world scenarios."}]

Author rebuttal:

Meta-review: This paper proposes Fast and Accurate Data residual Matching, a dataset distillation to address the information vanishing problem in synthetic data. 

Most reviewers acknowledge the novelty and, after rebuttal, the experimental support provider; as such, they lean towards acceptance
One reviewer insists on the incremental nature of the approach, when it comes to optimization approaches. While individual models might be incremental, the rest of reviewers agree that the paper merits acceptance.
Decision: Accept (poster)