Understanding Dataset Distillation via Spectral Filtering
Authors: Deyu Bo
Songhua Liu
Xinchao Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Dataset Distillation
Spectral Filtering
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A theoretical analysis of dataset distillation from the perspective of spectral filtering.

Abstract: Dataset distillation (DD) has emerged as a promising approach to compress datasets and speed up model training. However, the underlying connections among various DD methods remain largely unexplored. In this paper, we introduce UniDD, a spectral filtering framework that unifies diverse DD objectives. UniDD interprets each DD objective as a specific filter function applied to the eigenvalues of the feature-feature correlation (FFC) matrix to extract certain frequency information of the feature-label correlation (FLC) matrix. In this way, UniDD reveals that the essence of DD fundamentally lies in matching frequency-specific features. Moreover, we characterize the roles of different filters. For example, low-pass filters, \eg, DM and DC, capture blurred patches, while high-pass filters, \eg, MTT and FrePo, prefer to synthesize fine-grained textures and have better diversity. However, existing methods can only learn the sole frequency information as they rely on fixed filter functions throughout distillation. To address this limitation, we further propose Curriculum Frequency Matching (CFM), which gradually adjusts the filter parameter to cover both low- and high-frequency information of the FFC and FLC matrices. Extensive experiments on small-scale datasets, such as CIFAR-10/100, and large-scale ImageNet-1K, demonstrate the superior performance of CFM over existing baselines and validate the practicality of UniDD.

Published as a conference paper at ICLR 2026
UNDERSTANDING DATASET DISTILLATION VIA SPECTRAL FILTERING
Deyu Bo' Songhua Liu Xinchao Wang
National University of Singapore "Shanghai Jiao Tong University deyu.bo@nee edd. sg, liusonghuals jtu. edu. xinchao@nus, edu. sg
ABSTRACT
Dataset distillation (DD) has emerged as a promising approach to compress datasets and speed up model training. However, the underlying connections among various DD methods remain largely unexplored. In this paper, we introduce UniDD, a spectral filtering framework that unifies diverse DD objectives. UniDD interprets each DD objective specific filter function applied to the eigenvalues of the feature-feature correlation (FFC) matrix to extract certain frequency information of the feature-label correlation (FLC) matrix. In this way, UniDD reveals that the essence of DD fundamentally lies in matching frequency-spect Moreover, we characterize the roles of different filters. For example, low-pass filters, e.g., DM and DC, capture blurred patches, while high-pass filters, e.g., MTT and FrePo, prefer synthesize fine-grained textures and have better diversity. However, existing methods can only learn the sole frequency information as they rely on fixed filter functions throughout distillation. To address this limitation, we further propose Curriculum Frequency Matching (CFM), which gradually adjusts the filter parameter to cover both low- and high-frequency information of the FFC and FLC matrices. Extensive experiments 10/100, and large-scale ImageNet- K, demonstrate the superior performance CFM over existing baselines and validate the practicality of UniDD.
features.
small-scale datasets, such as CIFAR-
INTRODUCTION
The exponential growth of data presents significant challenges to the efficiency and scalability of training deep neural networks. Dataset distillation (DD) has emerged solution these challenges, aiming to condense large-scale real datasets into compact, synthetic ones without compromising model performance (Wang al., 2018: Yu et al., 2024; Lei & Tao, 2024; Geng al., 2023; Sachdeva McAuley, 2023). This approach has shown promise across range of domains, including image (Zhao al., 2021; Zhao & Bilen, 2023; Cazenavette et al., 2022), time series (Liu al., 2024b; Ding et al., 2024), and graph (Jin et al., 2022; Yang et al., 2023; Liu et al., 2024a).
Existing DD methods vary in their optimization objectives, which can be grouped into four main categories. Statistical matching (Zhao & Bilen, 2023; Zhao et al., 2023) aligns key statistics, such as mean and variance, between real and synthetic datasets. Gradient matching (Zhao et al., 2021; Kim et al., 2022; Zhao & Bilen, 2021) minimizes the direction of model gradients across real and synthetic data during training. Trajectory matching (Cazenavette et al., 2022; Guo et al., 2024) enforces the synthetic data emulate the update trajectories of real model parameters. Kernel-based approaches (Zhou et al., 2022; Loo et al., 2022) employ a closed- form solution to bypass the inner optimization and improve distillation efficiency. Despite differences in their objectives, all methods strive to minimize the discrepancy between real and synthetic datasets from certain perspectives. This raises some key questions: Are these DD methods related? so, is there a unified framework that can encompass and explain the objectives of various DD methods? These questions are crucial as unified framework can deepen our understanding of DD, uncover the essence of existing methods, and offer potential insights for advanced distillation technologies.
As the first contribution our work, we theoretically analyze some representative DD methods, including DM (Zhao& & Bilen, 2023), DC (Zhao et al., 2021), MTT (Cazenavette et al., 2022), and
"Corespondingg Author

---

Published as a conference paper at ICLR 2026
FrePo (Zhou et al., 2022), and summarize them into spectral filtering framework, termed UniDD, uncovering their commonalities and differences. Specifically, these DD methods are all proven to match the feature-feature correlation (FFC) and feature-label correlation (FLC) matrices between real and synthetic datasets. Their difference lies in the filter function applied to the FFC matrix, which changes the eigenvalues of the FFC matrix to extract certain frequency components of the FLC matrix. Based on their filtering behaviors, we classify existing DD methods into lowffrequecyy matching (LFM) and
matching (HFM). Our experimental investigations, detailed in Section 4, demonstrate that LFM-based methods, e.g., DM and DC, tend to learn coarse-grained colors and blur the synthetic images. In contrast, HFFMbased methods, e.g., MTT and FrePo, focus on fine-grained textures, which have better diversity.
The connections between DD objectives and filter functions further enable us to identify the weak- nesses of existing methods. Traditional DD methods typically select the matching objectives based on intuition, while UniDD shifts this paradigm filter design, making the new DD methods more interpretable and reliable. Based on the guidance of UniDD, we find that existing DD methods only use fixed filter functions during distillation and can only learn a single frequency component, which limits their adaptability. Therefore, we propose Curriculum Frequency Matching (CFM), which gradually adjusts the filter parameter to increase the ratio of high-frequency information the synthetic data, thereby covering a wider range of frequency components. To verify its effectiveness, we conduct extensive experiments on CIFAR-IO/ 10/10, Tiny-ImageNet, and ImageNet- Ik. The results demonstrate that CFM consistently outperforms baselines by substantial margins and yields better cross-architecture generalization ability. The contributions of this paper are summarized below:
We introduce UniDD, spectral filtering framework that interprets each DD objective specific filter function applied to the FFC and FLC matrices, thus unifying diverse DD methods as a frequency-matchine problem.
We classify existing methods into low-frequency and high-frequency matching based on their filter functions, highlighting their respective roles in encoding global colors and local textures.
We propose CFM, a novel DD method with dynamic filters that encode both low- and high- frequency information. Extensive experiments across diverse benchmarks demonstrate its superior performance over existing baselines.
2 PRELIMINARY
Notations Let T = (H, Y) denote a real dataset, where H represents the original data with [H] = n samples, and Y E xa is the one-hot label matrix for classes. The goal of DD is to learn synthetic network S = (Ha, Y2), where [Ha| = m < n and Y, E mme,, such that models trained on T and S have comparable performance. In addition, there is a pre-trained distillation network o(.). such as ConvNet (Zhao et al., 2021) or R8 (He et al., 2016) for the image datasets. We use X- = d(H) Raxd and X, Φ H.)) E m indicate the data representations learned by the distillation network on real and synthetic datasets, where d is the dimension.
FFC and FLC matrices. For the real and synthetic datasets, the FFC matrices are defined as X X X, E Rdddd respectively. These matrices capture correlations among feature channels and serve as important statistical descriptors of the datasets. Similarly, the FLC matrices are denoted [ Y, E dx respectively. Each column corresponds to the average representation of a as X X,
class, capturing the alignment between channels and labels.
Spectral Filtering, As X X is positive semi definite, it can be decomposed into X X = T.UT. where U is the eigenvectors and A is the diagonal eigenvalue matrix with X, Aq. The filtering function f()) acting on X X is equivalent to acting on its eigenvalues, i.e., x U U(A)U Without loss of generality, we assume that the eigenvalues are sorted in ascending order, i.e., 0 <
\ .... < Aa. Jolliffe (2011) points out that eigenvectors associated with large eigenvalues encode low-frequency information, while small eigenvalues correspond to high-frequency components. Therefore, a high-pass filter be defined as f(x;) 2 f(A+11) Vi [1, 1]. as it will assign larger amplitudes to gighfrreueecy eigenvectors, while the low-pass filters will do the opposite.
Spectral-domain Dataset Distillation. There are some DD methods operating in the spectral domain, such as FreD (Shin et al., 2023) and NSD (Yang et al., 2024a). UniDD differs from them in two key aspects: First, FreD and NSD analyze the frequency of the input data itself, e.g., images in the
2

---

Published as a conference paper at ICLR 2026
Table An overview the objective and filter functions of four representative DD methods.
Matching
Method
Objective Function
Filtering Fumction
DM
X - xT x - x7 - 11 ax xx -1-0xxxx,,
f()) =1 f(A) {L,A}
Low-freuunccy
MTT
E ax XX'XXXXX Eo ax, XX,xxX x. y
A(A) = (1 ad)(ma)
High-frequency
FRePo
xxxxx + x Xx, B1) xT
A(A) (2+
Fourier domain, while UniDD operates on the spectral properties of the FFC matrix, making it more general and data-agnostic. Second, FreD and NSD) remain natatearramttiintioonn methods whose optimization relies existing objectives, while UniDD formulates framework of DD objectives in the representation space, enabling the principled design of new distillation loss functions.
3 SPECTRAL FILTERING OF DATASET DISTILLATION
Before delving into the details, we first formally formulate UniDD, spectral filtering framework that unifies diverse DD objectives.
Theorem There exists a wide range of dataset distillation methods that can be unified into spectral filtering framework, defined as:
min ||(x x)e(X -XX XX) (Y)))l
where the FFC and FLC matrices serve as filter and signal, respectively, J()) is the filterng function, and g(-) is a binary function with YY or
We analyze four representative DD methods the following and explain their relationship with different filter functions. Based on the filter behaviors, they can be divided into LFM-based and HFF--based methods. See Table for a quick overview and Appendix B for more detailed derivations.
3.1 NOW-FREQUENCY MATCHING
Methods belonging to LFM tend to capture the principal components of the FFC matrix. Generally, these methods have quick convergent rate and perform well with lower synthetic budgets. However, they fail to learn fine-grained information and have poor diversity. Here, we analyze two sub- categories: statistical matching and gradient matching.
Statistical Matching, Matching important statistics between the real and synthetic datasets is a straightforward way distill knowledge. DM (Zhao & Bilen, 2023) proposes to match the average representations of each class, whose objective function can be defined as:
X X. Y.l
(2)
where the corresponding filtering functions are s(XT X) = I and y(XTY) = XTy, respectively. For clarity, omit the mean normalization here, but this does not affect the definitions of and g-
However, DM does not perform well on complex distillation backbones, e.g., ResNet-18, as it only matches the statistic of the last layer of backbones. To overcome this limitation, L (Yin et al., 2023) proposes to match the mean and variance information in each Batch Normalization (BN) layer. The objective function is defined as:
[iag(XX
x) TxT)
diag(X] xx) [[avg(XX) avg(X..)l
where diag(-) and avg(-) indicate diagonal and average operations. Notably, SRe replaces the class representation X y with average sample representation avg(X), thus losing the category information. It then adds cross-entropy elassification loss, i.e., Loo H Y,) compensation.
3

---

Published as a conference paper at ICLR 2026
Gradient Matching. Another LFM example is gradient matching (Zhao et al., 2021), which minimizes the differences of model gradients the real and synthetic datasets. The gradients of the linear classifier calculated as:
Vw = x - Y). vi = x, (X.W-Yx))
(4)
By deriving the upper bound of minimizing the gradient differences, we unify gradient matching into our framework:
V V I & I X__ -X
(5)
where the corresponding filtering functions are f(XTx) X when y I, and f(X X) = / when g(X Y) = Y In addition, Deng et al. (2024) adds a covariance matching loss DM to explore the inter-feature correlations, which can also be viewed as special case of gradient matching.
3.2 HIGH- FREQUENCY MATCHING
Instead of directly matching the principle components, HFM--base methods typically apply high-pass filters the FFC matrix enhance the high- freeufry information and improve diversity. They usually perform better than LFM-base methods but also bring more computations. We analyze two sub-categories: trajectory matching and kernel ridge regression (KRR).
Trajectory Matching. Matching the long-range training trajectories has been proven to be an effective approach for DD. We take MTT (Cazenavette et al., 2022) example, which minimizes the differences between network parameters trained on the real and synthetic datasets. The objective function can be formulated as:
| W 1
(6)
where WP and W indicate the parameters trained on the real and synthetic datasets for and Q iterations, respectively. For a full-batch gradient descent, have:
W W V (T a X)WO + +aXTy,
where a is the learning rate and 8 indicates the initial parameters. For K iterations, we have:
K-1
W K = (1 -1
Σ a(l Eal--x-xxixTx
(8)
km0
As W and W have the same initialization W the objective function be reformulated as:
W - w - || -
ax x) - - -I-ax x,)911
(9)
Σ - -I- ax xXXXXT
(1-ax; X.XxX.XXXXXX.
where the corresponding filtering functions are ((XT)) (I ax P/ when g and f(x (x)= E - X)p/ when g(X X Y) = x
Subsequent works improve MTT from different perspectives, such as memory overhead (Cui et al., 2023), trajectory training (Du et al., 2023), and trajectory selection (Guo al., 2024). However, they all follow the same objective function so that they can be naturally included in our unified framework.
KRR. The carly-stage DD algorithms usually update the synthetic data by solving two-level optimization problem. To reduce computation and memory costs, KRR-based methods are proposed replace the inner-optimization with closed-form solution:
Y Krs (1C.s + +BT) -y
(10)
where K.,- x,),X.) E B and IC,, = xC(X,,, X.) E mR are two Gram matrices. There are many choices of kernel functions. When using the linear kernel, we have:
|| XX (X,X. 1
(11)
4

---

Published as a conference paper at ICLR 2026
0.9
Low-Pass His-
GM
0.8
Pooort
K3R
107
0.7
00
0.6
50
100
150
200
Spatial Frequecc
Class Index
(a) Raw
(b) LP
(c) HP
Figure 1: Power Spec- Figure 2: Intra-class Figure 3: Comparison between real and synthetic
tral Density
Similarity.
images. LP: Low-pass filter; HP: High-pass filter.
where (X, + +,31)-ly, be regularization term B3 F. the optimal solution of W, becomes:
trainable weight matrix W, Rdxe, By adding
W = (X X 31) '
(12)
Then the objective function of KRR be reformulated as:
[[x X + x xx (X,X, + BI) 'Y
(13)
By applying a matrix identity transform (Welling, 2013) on the second term, the objective function can be unified into UniDD:
X, -1 where the corresponding filtering functions are f(X x) (x (1) and 9 Y
(14)
It is worth noting that only the linear kernel case can be unified into our framework, e.g., KIP (Nguyen al., 2021a;b and FrePo (Zhou et al., 2022). We leave the analysis of the non-linear kernel, e.g., Gaussian and polynomial, as further work.
4 IN-DEPTH ANALYSIS OF FILTERS
The above analysis reveals that DD methods adopt different filters, e.g., ., low-pass and high-pass. This section further justifies our claim and analyzes the roles of different filters through experiments the Tiny-ImageN dataset.
Power Spectral Density. We first verify that SM and GM behave low-pass filters, whereas TM and KRR exhibit high-pass filtering. Specifically, Figure visualizes the radially averaged power spectral density of the synthetic images learned by these four methods. The results show that SM and GM have more energy the low-frequency band, while TMM and KRR allocate greater power to high-frequency components, thereby supporting our claim.
Intra-class Similarity. We further investigate the influence of low-pass and high-pass filters on the synthetic data. Following G-VBSM (Shao al., 2024), we compute the intra-class cosine similarity of synthetic datasets distilled by different filters. As shown in Figure 2, the low-pass filter increases intra-class similarity of synthetic data and preserves the consistency of DD, while the high-pass filter improves data diversity by reducing intra-class similarity.
Visualization. Finally, we visualize the raw images and the synthetic images distilled by low-pass and high-pass filters. We observe that synthetic images produced by low-pass filters (Figure 2b) appear noticeably blurrier than the original images (Figure 2a), low-pass filters assign higher weights to eigenvectors associated with larger eigenvalues. These eigenvectors represent the low- wwwfrrquency components of the original data, e,g., coarse-graines shapes. In contrast, high-pass filters emphasize eigenvectors with smaller eigenvalues, thereby enhancing fine-grained and complex textures, as shown in Figure 2c.
THE PROPOSED METHOD
It is crucial to preserve both the consistency and diversity of the synthetic datasets. However, existing DD methods only have fixed filter functions and cannot capture the diverse frequency information of the real datasets, which motivates the design of our model. The framework is shown in Figure 4.
5

---

Published as a conference paper at ICLR 2026
Synthetic Data E ww)
Real Data T E (nt, k,
FLC of (d.c)
FLC 4' E (d.c)
Siggnl
1-th layer
layer
FFC
FFC ui E (d,d)
3x3 Conv
1 (d.a)
Liiter
3x3 Conv
Feuture x (m, d, w)
Fenture (n,d, h, w)
ReLU
Filtering ( (2+ (2+P)) Pb)
ReLU
Batch Norm
Batch Norm
Aets
Label V
(m,,
Batch,
Batchy Bg = 0.5
Hatcha
Label
Bg 1.0
Bg 0.01
Figure 4: Framework of CFM. The real and synthetic data is first fed into a backbone to calculate their corresponding FFC and FLC matrices. The filter f(A) (X+ 3,)- 11 then applied on FFC to extract certain frequencies. CFM uses different values of 3% in each batch to cover both the low-frequency and high-frequeney information.
Implementation of FFC and FLC. We calculate the FFC and FLC matrices based on the feature map of each convolutional layer. Specifically, the feature map of the /-th layer denoted as X Im xx indicating the number of synthetic images, channels, height, and width, respectively. For the FFC matrix, we reshape the feature map into channel-level representations, and for the FLC
matrix, we average it along the height and width dimensions:
X = reshape (X)) Rmhwxdd
X = avg X!) E mgd
(15)
In practice, directly computing the FFC matrices suffers from numerical overflow problems because the reshape operator significantly increases the number of samples from m to mhw. Therefore, we use the covariance matrix and mean representation as substitutes stabilize the training process
V 1
m/w (X) (X)
x (X) -X))
Y.:
(16)
where V and Φ. denote the normalized FFC and FLC matrices of the synthetic dataset. Similarly, we can calculate Y' and q' for the real dataset.
Exponential Moving Updating (EMU). The computation of y and o is affordable for real datasets but much more complex for synthetic datasets. In practice, we can only calculate V and & in mini-batch manner, which differs from the full-batch assumption in the theoretical analysis. To mitigate this gap, we adopt a moving update technology (Shao al., 2024; Loo et al., 2024) to cache the result of each batch and ultimately approximate the statistics of the real datasets:
T vb. + (1 1 0
1 -ーーー + + -1- 1
-
(17)
where 4/.6 and - indicate the FFC matrices of the current b-th batch and the previous batch. respectively. The symbols of the FL.C matrices are the same as above.
Curriculum Frequency Matching. After calculating the FFC and FLC matrices, we need to design filtering function to match the crucial frequency information between the real and synthetic datasets. Specifically, we consider a filter (()) (A + 3)-1, whose shape is controlled by the value of B. Generally, as decreases, the proportion of high-frequency information gradually increases, thus improving the diversity of synthetic datasets. However, high-frequency information also introduces additional noise, resulting in performance degradation, shown in Figure 6. Therefore, define scheduler to dynamically adjust the value of /3 in different batches to balance the consistency and diversity of the synthetic dataset
(18) where 3 is hyper- hyper-parameter that controls the maximum frequency, and B denotes the total number of batches. Notably, f(V+-") = (W-. h + (.))= has different characteristics. For example, W. corresponding to 31, preserves consistency and V..B corresponding 3n. emphasizes diversity.
3g = 3 * (1 + cos(Tb/B))/2

---

Published as a conference paper at ICLR 2026
Table 2: Performance (%) of different DD methods in CIFAR-10/100. The best performance is highlighted in bold. Results are taken from the original papers, and indicates missing data.
ComNNe-12 128
ResNet-18
Dutaset
IPC
DM
MTT
PrePo
G-VBSM
DWA
CFM
SRe"1.
G-VBDM
DWA
CFM
10
48.920.6
44920.55
65.310.7
65.510.4
46.520.7
45.0104
52.120.5
27.2a0.5
53.510.6
32,6104
57.020.3
CIFAR-10
50
63.0s004
53,900.
71.620.2
71.7a0.2
$4.320.3
63.350.7
64.0s0.
47.5at16
59.2:0.4
53 33.180.3
82.3e0.4
10
29.7=0.3
25.2=0.3
33.120.4
42.54022
38.7=0.2
47.66044
58.3:0.4
31.640.5
59.540.4
69.6
64.620.4
CIFAR-100
100
43.620.4
44.3002
45.740.4
59.0.0.1
67.1.0.3
49.540.3
Table 3: Performance (%) of different DD methods in Tiny-ImageNet and ImageNet-Ik.
ResNet-18
ResNet-101
Dalasel
IPC
SRc
CDA
G-VBSM
RDED
DWA
CFM
SRc"L
RDED
DWA
CFM
41.4
48.8
47 47.640..
58.2:0.1
52.8=0.2
58.0=0.2
42.5+0.2
41.2+0.1
54.740.3
60.4:0.2
T-ImageNet
100
49.7a0.3
53.2
51.010.4
56.020.2
59.2s0.1
51.500.3
57.420.3
61.120.2
10
21.3+0.6
31.400.5
42.020.1
37.920.2
40.625.3
30.940.1
48.3=1.0
46.91114
47.220.5
ImageNet- Ik
50
46.8+0.2
53.5
51,840.4
56.5+0.1
55.2=0.2
57,3+0.2
60.8+0.5
61.2+0.4
63.340.7
63.5e0.2
100
$2.820.3
58.0
35.720.4
59.220.3
59.520.2
62.841.2
66.740.2
66.920.3
Loss Function. After defining the filtering function f. another consideration is the choice of which restricted g "X Y) = or X Y. The former corresponds to filter-matching loss, while the latter leads to signal-matching loss. In the context of CFM, we define the loss functions as:
Litter
l1 + +.Bol)+ (v. + 11,
b=1 1=1
(19)
Esignal
[[ + -.11))1)-'---1 (w!D Bo11-
By combining these functions with the basic classification loss, we get the final loss function:
L Lets Hs, Y.) + Cfiitter + 1/Lrsignal
(20)
where 1) = 0.1 a hyperparameter for all datasets. See Appendix C for more details, such as the pseudo algorithm.
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Datasets. We consider four datasets, including CIFAR-10/100 (Krizhevsky et al., 2009) (32x32, 10/100 classes), Tiay-ImageNet (Le & Yang, 2015) 664x64, 200 classes), and ImageNet- (Deng et al., 2009) (224x224, 1000 classes).
Network Architectures. We use ResNet-18 (He al., 2016) as the distillation network for all datasets. For fair comparison with traditional DD methods, we also use ConvNet- (Zhao et al., 2021) in CIFAR- 10/100, as suggested by G- G-VBSM (Shao et al., 2024). We adopt various memory budgets for different datasets, including images per class (IPC)-10, 50, and 100.
Baselines. Traditional DD methods work on small-scale datasets, e.g., CIFAR-10/100, but do not perform well in the large-scale ImageNet-1 K. In this case, we select DM (Zhao & Bilen, 2023), DC (Zhao et al., 2021), MTT (Cazenavette al., 2022), and FrePo (Zhou et al., 2022) as baselines in report the performance of some more recent DD methods, including
1IFAR-10/109 Morcover, S (Yin et al., 2023), CDA (Yin & Shen, 2023), G-VBSM (Shao et al., 2024), RDED (Sun et al., 2024), and DWA (Du et al., 2024).
Evaluation. In the evaluation phase, we adopt the Fast Knowledge Distillation (FKD) strategy sug- gested by L, which has been proven to be useful for large-scale datasets. For fair comparison, we strictly control the hyperparameters of FKD to be the same as those of previous methods. See Appendix C.3 for more details.
7

---

Published as a conference paper at ICLR 2026
Table 4: cross-architectue performance (%) of different DD methods. indicates that the results our evaluation; otherwise, we adopt the results from the original paper. The results of DWA are not reported due the lack of source code and synthetic images.
Validaton Model
ImageNet-Ik (IPC=50)
ResNet-18
ResNet-50
ResNet-101
DenseNet-121
RegNet-Y-SGP
ConvNeXI-Tiny
S
46.80
55,60
60.81
49.47
60.34
53.53
CDA
53.45
61.26
61.57
57.35
63.22
62.58
G-VBSME
51.8
58.7
58.47
62.02
61.93
CFM
57.32
63.23
63.46
60.91
64.03
64.89
Table 5: Ablation studies on the loss functions of CFM. Experiments are conducted on three datasets with IPC=50.
Accursay
IPC=50
IPG+50
Loss Functions
Datasets
IPC=100
Cilter
Laigual
Les
CIFAR
Tiny
ImageNet
Loweass
R= --111
1=1e-2
3-30-3
Crs
Loepas
1-le-1
B-1e-2
3- 6
64.95
$1,20
53.43
/
67.32
$1.92
55.76
Table 6: Ablation studies on filters. Left: Tiny- ImageNet; Right: ImageNet-IK.
72.48
54.84
57.32
6.2 QUANTITATIVE RESULTS
We conduct experiments both small-scale and large-scale datasets. The results are shown in Tables and 3, respectively, from which we have the following observations:
CIFAR-10/100 Traditional DD methods perform well on datasets with fewer classes, i.e., CIFAR- 10, and shallow models, i.e., ConvNet- 128. The advanced DD methods are more effective when the dataset becomes larger and the network goes deeper. We observe that CFM consistently outperforms all advanced DD methods by a large margin. Notably, ResNet-18 trained on the original datasets has accuracy of 94.25% and 77.45%
CIFAR-10/100 The performance of existing methods is far from achieving optimal results, while CFM is significantly approaching, demonstrating its effectiveness in distilling low-resolution data.
Imageee & ImageNet-Ik. In the large-scale datasets, we use ResNet- 18, 101 to evaluate the performance of different methods. In Tin--ImageNet, CFM outperforms DWA by 3% on average and is slightly lower than RDED when IPC=50. When IPC increases to 100, CFM achieves the best performance, demonstrating its effectiveness. the ImageNet-1k part, we can see that CFM consistently surpasses all training-based DD methods in IPC=(50, 100). validating its superiority over the advanced DD methods. Moreover, we observe that RDED performs better than CFM in 10, We suspect that the synthetic data cannot effectively cover the useful frequency features under the limited memory budget, leading to the performance degeneration of CFM.
6.3 CROSS-ARCHITECTURE GENERALIZATION
In addition to classification accuracy, cross-architecture generalization also crucial for DD. Ideally, synthetic datasets should encode the knowledge of real datasets rather than overfitting to a specific model architecture. Therefore, we use diverse networks, including ResNet-50/101, DenseNet- 121 (Huang et al., 2017), RegNet- Y-SGF (Xu al., 2023), and ConvNeXt-Ting (Liu et al., 2022b), to evaluate the generalization ability different DD methods on the ImageNet- 1k dataset with IPC=50. The distillation network is uniformly set ResNet-18 18. Results are shown in Table 4, from which can see that CFM achieves the best performance by substantial margin across different architectures.
6.4 ABLATION STUDIES
We conducted two experiments to demonstrate the importance of each module in the proposed model. The first experiment, shown in Table is used identify the effectiveness of different loss functions.
8

---

Published as a conference paper at ICLR 2026
Table 7:
Time overhead of three DD methods.
Table 8: Space overhead of three DD methods.
Time (s)
Gels
+LiN
+amm
+ +iiner + Leggal
Space (MB)
CaN
1 Caome
Litter 1
Erignol
SRe²L
0.16
0.26
SR G-VBSM
17186
22522
G-VBSM
0.16
0.35
17186
23170
CFM
0.16
0.31
CFM
17186
21450
Neay-ImaeNNet
ImageNet-Ik
SRe2L
G-VBSM
CFM
GoldFish
Jellyfish
Snail
GoldFish
Cucumber
Pizza
Figure 5: Visualization of the images synthesized by different DD methods.
Specifically, we sequentially remove the classification loss and signal matching loss, and evaluate CFM on three datasets, including CIFAR- 100, Imageem and ImageNet-Ik. We can observe that all three loss functions contribute to the performance of CFM, and the filter-macching loss plays fundamental role. This discovery suggests that important knowledge of the real datasets may be encoded in the FFC matrix of the feature maps.
The second experiment is used verify the effectiveness of the curriculum strategy. Specifically, evaluate the performance of different filters, including low-pass filter, high-pass filter with parameter 1e1eee),,, "), and high-pass filter with CFM. The results are shown in Figure 6. from which we can find that the performance of the low-pass filter is far away from the high-pass filters, indicating that the high-frequency information more important for DD. However, the extremely high frequency will also lead to performance degradation. For example, the performance of 3 = le not as good as other parameters. On the other hand, CFM balances the ratio between low-frequency and high-frequency by setting proper frequency band, consistently outperforming other filters.
{le
1e 1e
6.5 TIME AND SPACE OVERHEAD
We compare the time and space overhead of CFM with two representative DD methods: SRe2L Yin al. (2023) and G-VBSM Shao al. (2024) the ImageNet- 1k dataset with ResNet-18 18 backbone. To ensure fairness, all methods optimize 500 synthetic images per step. The results are averaged over 1,000 iterations on a RTX 4090 GPU. Results are shown in Tables 7 and 8.
All methods use the basic classification loss. SRe2I. proposes batch normalization loss, LBN, to align the mean and variance statistics. Building on this, G-VBSM further adds a densification loss, Laense. to improve the diversity of the synthesized images. Unlike them, CFM uses the filter-matching and eggnal-macching losses. Compared to SRe2L, CFM has a slightly higher time overhead, but is still lower than G-VBSM. On the other hand, CFM has the lowest space overhead, the number of convolutional layers is less than that of batch normalization, demonstrating the efficiency of CFM.
6.6 VISUALIZATION
To provide a more intuitive comparison of various DD methods, we visualize the synthetic images generated by LR G-VBSM, and CFM in Figure 5. It can be observed that L and G- B-VB struggle to produce meaningful images, particularly with the low-resolution Tiny-ImageNe dataset. In contrast, the images synthesized by CFM exhibit clear semantic information, e.g., recognizable

---

Published as a conference paper at ICLR 2026
shapes, demonstrating the effectiveness of matching frequency-specien features between the real and synthetic datasets. See Appendix D for more visualizations.
7 RELATED WORK
Data Parameterization. Traditionally, the synthetic images are optimized in the pixel space, which is inefficient due to the potential data redundancy. Therefore, some methods explore how to parameterize the synthetic images. IDC (Kim et al., 2022) distillates low-resolution images to save budget and up-samples them in the evaluation stage. Haba (Liu et al., 2022a) designs a decoder network to combine different latent codes for diverse synthetic datasets, RDED (Sun et al., 2024) stitches the core patches of real images together as the synthetic data. Some methods use generative models to decode the synthetic datasets from latent codes. including GAN- based methods (Zhao & Bilen, 2022; Liu Wang. 2023), diffusion-basee methods (Gu et al., 2024; Su al., 2024), and implicit function- based methods (Shin et al., 2023). Generally, data parameterization can be incorporated with different DD objectives to improve their efficiency.
Model Augmentation. The knowledge of the datasets is mostly encoded in the pre- pre-trained distillation networks. Therefore, some methods focus on training a suitable network for DD by designing some augmentation strategies. FTD (Du et al., 2023) constrains model weights to achieve flat trajectory and reduces the accumulated errors. Zhang et al. (2023) use early-stage models and parameter perturbation increase the search space of the parameters. More recently, DWA (Du et al., 2024) employs directed weight perturbations on the pre-training model that maintain the unique features of each synthetic data.
Theory. Some works tend to explore DD from a theoretical perspective. For example, Maalouf et al. (2023) analyzes the size and approximation error of the synthetic datasets. Cui al. (2024) explores the influence of spurious correlations in DD. Yang et al. (2024b) explains the information captured by the synthetic datasets. Kungurtsev al. (2024) provides a formal definition of DD and gives foundation analysis on the optimization of DD. These methods have some important insights, but fail explain the roles of existing methods. contrast, UniDD establishes the relationship between DD objectives and spectral filtering.
CONCLUSION
In this paper, we introduce UniDD, a framework that unifies various DD objectives through spectral filtering. UniDD demonstrates that each DD objective corresponds to filter function applied to the FFC and FLC matrices. This finding reveals the nature of various DD methods and inspires the design of new methods. Based on UniDD, we propose CFM to encode both low- and high-free gegh-frequenc information by gradually changing the filter parameter. Experiments conducted various datasets validate the effectiveness and generalization the proposed method. A promising future direction is generalize UniDD to the distillation of unsupervised and multi-modal datasets.
ACKNOWLEDGMENT
This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-000
ETHICS STATEMENT
This work aims to provide unified view of dataset distillation and deepen our understanding of data efficiency. There are many potential socictal consequences of our work, none which we feel must be specifically highlighted here.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
To ensure reproducibility, we provide additional implementation details in Appendix C. During the review process, we submit the source code as supplementary material, and we promise to release the code publicly upon acceptance of the paper.
REFERENCES
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun Yan Zhu. Dataset distillation by matching training trajectories. CVPR, pp. 10708-10717. IEFF, 2022.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation imagenet- Ik with constant memory. In JCML, volume 202 Proceedings of Machine Learning Research, pp. 6565-6590. PMLR, 2023.
Justin Cui, Ruochen Wang, Yuanhao Xiong, and Cho-Jui Hsich. Ameliorate spurious correlations in dataset condensation. In ICML. OpenReview.net, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. IEEE Computer Society, 2009.
Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. In CVPR. IEEE, 2024.
Jianrong Ding, Zhanyu Liu, Guanjie Zheng, Haiming Jin, and Linghe Kong. Condtsf: One-line plugin of dataset condensation for time series forecasting. NeurIPS, 2024.
Jiawei Du, Yidi Jiang, Vincent Y. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error improve dataset distillation. In CVPR, pp. 3749-3758. IEBE, 2023.
Jiawei Du, Xin Zhang, Juncheng Hu, Wenxing Huang, and Joey Tianyi Zhou. Diversity-drive -riven synthesis: Enhancing dataset distillation through directed weight adjustment. In NeurlPS, 2024.
Jiahui Geng, Zongxiong Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao, and Chumming Rong. survey on dataset distillation: Approaches, applications and future directions. In IJCAI, pp. 6610-6618. ijcai.org, 2023.
Janyang Gu. Saeed Vahidian, Vyacheslav Kungurtsev. Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In CVPR, pp. 15793-15803, IEEE, 2024.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via cefficullt--aignnee trajectory matching. In ICLR. OpenReview.net, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778. IEEE Computer Society, 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, pp. 2261-2266 IEEE Computer Society, 2017.
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. In ICLR. OpenReview.net. 2022.
Ian Jolliffe. Principal component analysis. In International encyclopedia of statistical science, pp. 1094-1096. Springer. 2011.
Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha. and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 1110-11118. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky, Geoffrey Hinton, and et al. Learning multiple layers of features from tiny images. 2009.
Vyacheslav Kungurtsev, Yuanfang Peng. Jianyang Gu, Saeed Vahidian, Anthony Quinn, Fadwa Idlahcen, and Yiran Chen. Dataset distillation from first principles: Integrating core information extraction and purposeful learning. CoRR, 0tss/209.004411 2024.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)3, 2015.
Shiye Lei and Dacheng Tao. A comprehensive survey of dataset distillation. IEEE Trans. Puttern Anal. Mach. Intell., 46(1):17-32, 2024.
Songhua Liu and Xinchao Wang. MGDD: meta generator for fast dataset distillation. In NeurIPS, 2023.
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In NeurlPS, 2022a.
Yang Liu, Deyu Bo, and Chuan Shi. Graph distillation with eigenbasis matching. In ICML. OpenRe- view.net, 2024a.
Zhanyu Liu, Ke Hao, Guanjie Zheng, and Yanwei Yu. Dataset condensation for time series classifica- tion via dual domain matching. In KDD, pp. 1980-1991, ACM, 2024b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, pp. 11966-11976. IEEE, 2022b.
Noel Loo, Ramin M. Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In NeurlPS, 2022.
Noel Loo, Alaa Maalouf, Ramin M. Hasani, Mathias Lechner, Alexander Amini, and Danicla Rus. Large scale dataset distillation with domain shift. In ICML. OpenReview.net, 2024.
Alaa Maalouf, Murad Tukan, Noel Loo, Ramin M. Hasani, Mathias Lechner, and Daniela Rus. On the size and approximation error of distilled sets. In NeurIPS. OpenReview.net, 2023.
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge- regression. In ICLR. OpenReview.net, 2021a.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. In NeurlPS, pp. 5186-5198, 2021b.
Noveen Sachdeva and Julian J. McAuley. Data distillation: A survey. CoRR, 0abs/2301.0427/ 2023.
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In CVPR, pp. 16709-16718. IEEE, 2024.
DongHyeok Shin, Seungjae Shin, and 11-Chul Moon. Frequency domain-based dataset distillation. In NeurlPS, 2023.
Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D Dataset distillation via disentangled diffusion model. In CVPR, pp. 5809-5818 18. IEEE, 2024.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In CVPR, pp. 9390-9399, IEEE, 2024.
Tongzhou Wang, Jun Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. CoRR, abs/1811.10095, 2018
Max Welling. Kernel ridge regression. Max Welling's 's classnotes in machine learning, pp. 1-3, 2013.
Jing Xu, Yu Pan, Xinglin Pan, Steven C. H. Hoi, Zhang Yi, and Zenglin Xu. Regnet: Self-regulated network for image classification. IEEE Trans. Neural Networks Learn. Syst., 34(11):9562-956 2023.
12

---

Published as a conference paper at ICLR 2026
Beining Yang, Kai Wang. Qingyun Sun, Cheng Ji. Xingcheng Fu, Hao Tang, Yang You, and Jianxin Li. Does graph distillation see like vision dataset counterpart? In NeurIPS, 2023.
Shaolei Yang, Shen Cheng, Mingbo Hong. Haoqiang Fan, Xing Wei, and Shuaicheng Liu. Neural spectral decomposition for dataset distillation. ECCV, volume 15110, pp. 275-290. Springer, 2024a.
William Yang, Ye Zhu, Zhiwei Deng, and Olga Russakovsky. What is dataset distillation learning? In ICML. OpenReview.net, 2024b.
Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. CoRR, abs/2311. 2023.
Zeyuan Yin, Eric P. Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from A new perspective. In NeurlPS, 2023.
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. IEEE
Trans. Pattern Anal. Mach. Intell., 46(1): 2024.
Lei Zhang, Jie Zhang, Bowen Lei. Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In CVPR, pp. 11950-11955 1959. IEEE, 2023.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 12674-12685, PMLR, 2021.
Bo Zhao and Hakan Bilen. Synthesizing informative training samples with GAN. CoRR, abs/2204.07513, 2022.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, pp. IEEE, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR. OpenReview.net, 2021.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In CVPR, pp. 7856-7865. IEEE, 2023.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. In NeurlPS, 2022.
A STATEMENT ON LLM USAGE
In drafting this manuscript, LLMs were used exclusively for polishing language and checking grammar. The content was not directly copied from LLM outputs, and scientific concepts, analyses, and conclusions represent the authors' original work.
B DERIVATION OF UNIDD
The derivation of statistical matching, gradient matching, and trajectory matching is relatively intuitive and has been fully introduced in the main text. Here, we mainly derive the conclusion of KRR, which closely related to the proposed model. Generally, the objective of KRR defined as:
LKRR = ||CW Y 1 + TWW,
(21)
where K. E Rax= is the Gram matrix of input data.
By taking the derivative with respect to W, we have:
Vw KCT (KCW - -Y) + B(CW MACW+KTTT WKT.).
(22)
13

---

Published as a conference paper at ICLR 2026
When set the gradient zero, have:
W (K (K++31)-
(23)
This equation gives an optimal solution of W, which can used to replace the inner loop of DD. implemented with the linear kernel, we have:
W. = (X,X, +B
(24)
The outer loop of DD treats W, classifier to minimize the classification error in the real datasets:
Y Krs 1. || xxX (X,X, +
(25)
which derivatives Equation in the main text.
Another detail is the identity transformation, formulated as
P(QP +I)- (PQ + +
(26)
By setting X = P and X, = Q we have:
x_ (X 3T) (x, X, + -T.
(27)
Based on this equation, we successfully transform the Gram matrix into the FFC matrix and unify KRR-based methods into UniDDD.
C IMPLEMENTATION DETAILS.
C.I ALGORITIMM
Algorithm illustrates the distillation process of CFM.
Algorithm 1 Curriculum Frequency Matching
Input: Distillation network d, real dataset T (H, Y), minimum parameter Smins maximum step
T. number of iteration Z, batch size [B].
Output: Synthetic dataset S (Hs, Ys)
1: Feed H into for forward pass
2: for layer index = 1.... L do
3: Calculate / and 4' based on Eq. 16
4: end for
5: for batch index b = 1, B do
6: Initialize H with randomly sampled real data
7: Update 8, based Eq. 18
8: repeat
9:
Feed H. into for forward pass for layer index | = 1,... do
10:
Calculate T and qi.b based on Eq. 17
12:
Calculate Litter and Exignal based on Eq. 19
end for
14:
Calculate Lee and back-propagate C Update H.
16:
until Reached the number of iteration I
17:
end for
C.2 ENVIRONMENT
All experiments conducted single GeForce RTX 4090.
14

---

Published as a conference paper at ICLR 2026
Table Statistics datasets.
CIFAR-10 CIFAR-I00
Tiny-lmageNet
ImageNet- lk
Classes
100
200
1000
Training
50,000
50,000
100,000
1,281,167
Validation
10,000
10,000
10,000
50,000
Resolution
64x64
64x64
64x64
224x224
Table 10: Squeeze Configurations of CFM.
Config
CIFAR-10/100
Tiny ImageNet
ImageNet-1K 11
Epoch
200 100
50
90
Optimizer
SGD
SGD
SGD
Learning Rate
0.1
0.2
0.1
Momentum
0.9
0.9
0.9
WeightDecay
5e-4
lc-4
le-4
BatchSize
128
256
256
Scheduler
Cosine
Cosine
Step (0.1 30 epochs)
RandomCrop HorizontalFlip
RandomResizedCrop HorizontalFlip
RandomResizedCrop
Augmentation
HorizontalFlip
Table 11: Recover Configurations of CFM.
Config
CIFAR-10/100
Tiny-ImageNet
ImageNet-1k
B
0.1
1.0
0.1
Iteration
1000
1000
1000
Optimizer
Adam
Adam
Adam
Learning Rate
0.25
0.1
0.1
Betas
(0.5, 0.9)
(0.5, 0.9)
(0.5, 0.9)
BatchSize
10/100
200
500
Scheduler
Cosine
Cosine
Cosine
RandomResizedCrop
RandomResizedCrop
RandomResizedCrop
Augmentation
HorizontalFlip
HorizontalFlii
HorizontalFlip
Table 12: Recover Validation Configurations of CFM.
Config
CIFAR-10/100
Imag-Nt
ImageNet-1k
Epoch
1000
300
300
Optimizer
Adam SGD
SGD
Adam
Learning Rate
le-3 / le-1
2e-1
le-3
Parameters
/ Mom=0.9
Mom=0.9
WeightDecay
5e-4
le-4
le-4
BatchSize
64
64
100
Scheduler
Cosine
Cosine
Cosine
Temperature
30
20
20
RandomCrop HorizontalFlip
RandomResizedCrop HorizontalFlip
RandomRessizedCrop
Augmentation
HorizontalFlip
15

---

Published as a conference paper at ICLR 2026
C.3 EXPERIMENTAL SETUP
We use the squeeze, recover, and relabel pipeline, introduced by SRe*L, for the distillation of CFM. The details of datasets, setup (e.g., optimizer), and hyper-parameters are listed in Tables 9, 10, 11, and 12.
D VISUALIZATION
Finally, we visualize more synthetic images ImageNet-Ik Figure to provide a comprehensive exhibition. Specifically, each column represents the images sampled from the same batch. From left to right, the value of 3d decreases, and the high-frequency information gradually increases. It can be observed that images on the left side are somewhat blurry, while images on the right side have more complex textures, validating the conclusions in Section 4.
Goldfish
Cock
Castle
Figure Synthetic images. From left right, the high-frequency gradually increases.
16


Reviews:
[{"review_id": "fAoF3BjKM2", "reviewer": "Reviewer_zry6", "summary": "This study points out that existing dataset distillation objective functions have developed in a fragmented manner, and their relationships have not been investigated. This study proposes a new framework, UniDD, utilizing spectral filtering. Furthermore, UniDD enables to categorize previous researches based on the frequency bands the filter function focuses upon. Additionally, this work proposes Curriculum frequency matching (CFM), enabling the learning of both low and high-frequency information, thereby mitigating the limitations of prior research that only learned information from specific frequency bands.", "questions": "1. To support the claim that DM and DC are low-frequency matching methods while MTT and KRR are high-frequency matching methods, I suggest plotting the spectral density for the synthetic images generated by each methodology.\n\n2. I believe the synthetic label $Y_s$ need not necessarily be a one-hot label; treating it as an optimizable parameter does not alter the overall logic. Furthermore, in the experiments, soft labels based on fast knowledge distillation (FKD) were employed. I would be interested whether CFM remains effective when using optimized synthetic soft-labels.\n\n3. When using soft labels as synthetic labels, more memory is required than when storing only synthetic inputs, as the labels themselves must also be saved. Furthermore, the memory usage of synthetic labels increases significantly as the number of classes grows, to a point where it becomes non-negligible[4]. Therefore, I am curious how the utilized memory budget was configured to ensure a fair comparison with the baseline.\n\n4. I am interested in ablation studies concerning various frequency curriculum scheduling approaches. I am also curious whether this can be applied to parameterization methodologies such as FreD[1] and NSD[2], which only involve fixed frequency components in learning.\n\n5. I would be interested whether CFM exhibits high performance even when IPC=1.\n\n[4] Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is both highly impressive and fascinating that reinterpreting the objective function of existing dataset distillation from a spectral filtering perspective and proposing a unified framework.\n\n2. The proposed UniDD is a simple form, and the logical flow of expressing prior research using UniDD is also readily comprehensible.\n\n3. The motivation and core idea of CFM, which adaptively learns multiple frequency information based on UniDD, is natural and intuitive.", "weaknesses": "1. UniDD was proposed under assumptions that are prone to violation. Overall, this work develops UniDD’s logic under the assumption that the parameters of the feature extractor $\\phi$ are fixed and not considered. Consequently, both gradient matching and trajectory matching only considered the parameters of a single layer following the feature extractor. However, in dataset distillation, the parameters of the feature extractor are typically not fixed, meaning the UniDD framework may not readily satisfy these conditions. This suggests that UniDD has limitations in terms of its applicability.\n\n2. The objective functions for gradient matching and trajectory matching represent lower bounds for UniDD. This implies that achieving gradient matching and trajectory matching can be accomplished through low-frequency matching and high-frequency matching respectively; however, it is difficult to accept that the converse holds true. Therefore, the claim that gradient matching and trajectory matching correspond to low-frequency matching and high-frequency matching requires further clarification.\n\n3. The theoretical analysis and experimental design in this study do not align well. UniDD integrates statistical matching, gradient matching, trajectory matching, and KRR from a spectral filtering perspective, and proposes CFM to learn diverse frequency information. However, the baseline used in all experimental results except Table 2 is the decoupled method, which is difficult to consider as included in UniDD. To demonstrate the superiority of CFM, I believe that it is necessary to conduct diverse performance comparison experiments against the recent works for each component included within UniDD.\n\n4. Key previous literature is omitted. FreD[1], NSD[2], and NCFM[3] are prior studies addressing the frequency domain in dataset distillation; therefore, the relevance of the proposed idea must be discussed in the related works section.\n\n[1] Frequency Domain-based Dataset Distillation\n\n[2] Neural Spectral Decomposition for Dataset Distillation\n\n[3] Dataset Distillation with Neural Characteristic Function: A Minmax Perspective", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4mccfiugKy", "reviewer": "Reviewer_UzEG", "summary": "The paper proposed the combination technique, which produce both blurring (low-pass filter) and fine-grained (high-pass filter) image for dataset distillation. Experiments across various benchmarks demonstrate its superior performance over existing baselines.", "questions": "See weaknesses section!", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to follow.\n2. The main idea is clear, and it is easy to understand how it can lead to performance improvement.", "weaknesses": "1. The idea of using both blurring and fine-grained techniques for DD is not novel, as it has already been explored in [1].\n2. The proposed method only works under high IPC settings (greater than 1 IPC).\n3. The paper lacks comparisons with several state-of-the-art methods, such as [1] and [2].\n\n[1] Enhancing Dataset Distillation via Non-Critical Region Refinement. CVPR 2025\n[2] Dataset Distillation via the Wasserstein Metric. ICCV 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pDYbMdFPQi", "reviewer": "Reviewer_71it", "summary": "This work explores dataset distillation from a spectral perspective and introduces a new method called Class-wise Frequency Matching (CFM), which aligns the frequency characteristics of real and distilled data. The authors argue that dataset distillation naturally acts like a frequency filter, keeping the important, discriminative details while removing redundant information. Experiments on CIFAR-10 and ImageNet-100 support this idea and show that CFM can improve generalization performance, lending further evidence to their spectral interpretation. To the best of my knowledge, this is the first work that attempts to unify the theoretical understanding and methodological design of dataset distillation under a single spectral framework.", "questions": "Please check Weaknesses for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents, to the best of my knowledge, the first theoretical framework that systematically analyzes dataset distillation. The spectral analysis connects data condensation with the preservation of high-frequency, discriminative components. The derivations are careful and mathematically consistent, giving a logically sound and interpretable explanation of how synthetic data behave as spectral filters during training.\n\n2. The proposed CFM algorithm makes the theoretical idea practical. CFM matches the class-wise frequency distributions between real and distilled datasets to maintain frequency balance during training (e.g., Eq. 10). This turns the paper’s insight into a concrete algorithm, and the results show steady accuracy gains across different baselines, suggesting that frequency regularization can help dataset distillation generalize better.\n\n3. The writing is clear and the work is easy to reproduce. The mathematical reasoning is well organized, and the authors describe the implementation details for spectral energy computation in sufficient depth. They also release source code for both the analysis and CFM method.", "weaknesses": "The experiments are mainly conducted on smaller datasets like CIFAR-10 and ImageNet-100, so it remains uncertain whether the same spectral behavior and CFM improvements would hold on larger datasets such as ImageNet-21K. Adding results on a larger benchmark would make the conclusions more broadly convincing and show the robustness of the proposed approach. In addition, while CFM brings consistent improvements, its performance is still slightly behind the strongest existing methods. As shown in Table 3, the gains are modest and sometimes below strong baselines like RDED or FRePo, suggesting that there is still room for further refinement and optimization of the spectral alignment idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XqWSyR0SjJ", "reviewer": "Reviewer_b6oX", "summary": "This paper proposes a novel framework, UniDD, that unifies existing Dataset Distillation (DD) methods from the perspective of spectral filtering. The authors interpret different distillation objective functions as filtering functions applied to the feature-feature correlation matrix (FFC) and the feature-label correlation matrix (FLC), revealing that the essence of dataset distillation is \"matching frequency-specific features.\" To address the limitations that existing methods use fixed filters and thus cannot simultaneously learn across multiple frequency bands, the authors further propose Curriculum Frequency Matching (CFM). This method dynamically adjusts filter parameters, gradually covering from low to high frequencies during training, thereby achieving a balance between consistency and diversity in the synthetic data.", "questions": "1.\tWhat are the advantages of the proposed “spectral filtering” compared to transforming an image from the spatial domain to the frequency domain?\n2.\tCan the authors directly demonstrate the effects of what they define as low-frequency matching and high-frequency matching, rather than only visualizing them via filtered images as in Fig. 1?\n3.\tPlease provide a comparison of computational complexity metrics.\n4.\tCan the ablation study include comparisons against baseline methods? Moreover, in Table 5, the performance improvement of the designed loss over the basic classification loss seems relatively weak.\n5.\tIn Table 2, CFM achieves very strong performance on CIFAR-100. Does it surpass training on the full real dataset (Whole Dataset), or how close does it get?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The presentation is clear and easy to follow.", "weaknesses": "1.\tThe paper lacks sufficient rigorous theoretical justification, and UniDD's unified analysis relies on idealized assumptions. For example, the derivation of the DC method considers only linear classifiers, while the entire network is actually considered, which reduces readability and persuasiveness.\n2.\tCFM requires computing and matching FFC/FLC on each feature layer and applying dynamic filtering, which inevitably incurs significant computational overhead when the network is deep.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)