Frequency Domain-Based Dataset Distillation
Authors: DongHyeok Shin
Seungjae Shin
Il-chul Moon
Venue: NeurIPS 2023 poster (2023)
Keywords: Dataset distillation
Frequency domain
Dataset condensation
TL;DR: This paper proposes an utilization framework of frequency domain for dataset distillation.

Abstract: This paper presents FreD, a novel parameterization method for dataset distillation, which utilizes the frequency domain to distill a small-sized synthetic dataset from a large-sized original dataset. Unlike conventional approaches that focus on the spatial domain, FreD employs frequency-based transforms to optimize the frequency representations of each data instance. By leveraging the concentration of spatial domain information on specific frequency components, FreD intelligently selects a subset of frequency dimensions for optimization, leading to a significant reduction in the required budget for synthesizing an instance. Through the selection of frequency dimensions based on the explained variance, FreD demonstrates both theoretical and empirical evidence of its ability to operate efficiently within a limited budget, while better preserving the information of the original dataset compared to conventional parameterization methods. Furthermore, Based on the orthogonal compatibility of FreD with existing methods, we confirm that FreD consistently improves the performances of existing distillation methods over the evaluation scenarios with different benchmark datasets. We release the code at https://github.com/sdh0818/FreD.

Frequency Domain-based Dataset Distillation
Donghyeok Shin'
Seungjae Shin' KAIST tmmkko0910kkasst.ac.. kr
I1-Cuul Moon KAIST, Summary.Al cmoon@kaist.ea ac. kr
KAIST
tlsehdgurO@kaist.aac
Abstract
This paper presents FreD, novel parameterizattonn method for dataset distillation, which utilizes the frequency domain to distill small-sized synthetic dataset from large-sized original dataset. Unlike conventional approaches that focus the transforms to optimize the fre-
spatial domain, FreD employs frequency-base quency representations of each data instance. By leveraging the concentration of spatial domain information on specific frequency components, FreD intelligently selects subset of frequency dimensions for optimization, leading to significant reduction in the required budget for synthesizing an instance. Through the selec- tion of frequency dimensions based on the explained variance, FreD demonstrates both theoretical and empirical evidence of its ability to operate efficiently within limited budget, while better preserving the information of the original dataset compared to conventional parameterization methods. Furthermore, based on the orthogonal compatibility of FreD with existing methods, we confirm that FreD consistently improves the performances of existing distillation methods over the evaluation scenarios with different benchmark datasets. We release the code https:// thub. com sch0818h8881 1008888FFG 8/FreD.
Introduction
The era of big data presents challenges in data processing, analysis, and storage; and researchers have studied the concept of dataset distillation [20, 26, 14] resolve these challenges. Specifically, the objective of dataset distillation is to synthesize a dataset with smaller cardinality that can preserve the performance of original large-sized datasets in machine learning tasks. By distilling the key features from the original dataset into a condensed dataset, less computational resources and storage space are required while maintaining the performances from the original dataset. Dataset distillation optimizes small-sized variable to represent the input, not the model parameters. This optimization leads the variable to store a synthetic dataset, and the variable defined the memory space with the constraint of limited capacity. Since dataset distillation involves the optimization of data variables with limited capacity, distillation parameter designs, which we refer to as parameterization, could significantly improve corresponding optimization while minimizing memory usage.
Some of the existing distillation methods, i.e. 2D image dataset distillations [24, 14, 3, 25, 27], naively optimize data variables embedded on the input space without any transformation encoding. We will refer to distillation on the provided input space as spatial domain distillation, as an opposite concept of frequency domain distillation that is the focus of this paper. The main drawback of spatial domain distillation would be the difficulty in specifying the importance each pixel dimension on the spatial domain, so it is necessary to utilize the same budget as the original dimension for representing a single instance. From the perspective of data variable, which needs to capture the key information of the original dataset with limited budget, the variable modeling with whole dimensions becomes significant bottleneck that limits the number distilled data instances. Various spatial domain-based
Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

parameterization methods [7, 10] have been proposed to overcome this problem, but they are either highly vulnerable to instance-specific information loss [7] or require additional training with an auxiliary network [10].
This paper argues that the spatial domain distillation has limitations in terms of 1) memory efficieney and 2) representation constraints of the entire dataset. Accordingly, we propose a novel Frequency domain-based dataset Distillation method, coined FreD, which contributes to maintaining the task performances of the original dataset based on the limited budget. FreD employs frequency- transform to learn the data variable in the transformed frequency domain. Particularly, our proposed method selects and utilizes subset of frequency dimensions that are crucial for the formation of an instance and the corresponding dataset. By doing so, we are able achieve better condensed representation of the original dataset with even fewer dimensions, which corresponds to significant efficiency in memory. Throughout the empirical evaluations with different benchmarks, Frel shows consistent improvements over the existing methods regardless of the distillation objective utilized.
Preliminary
2.1
Basic Notations
This paper primarily focuses on the dataset distillation for classification tasks, which is a widely studied scenario in the dataset distillation community [20, 26]. Given C' classes, let X' E Rd denote the input variable space, and y = {1, 2.., C} represent the set candidate labels. Our dataset is D ((,, =)) S X' V. We assume that each instance (I, y) is drawn i.d from the data population distribution P. Let deep neural network de X + y be parameterized by E θ. This paper employs cross-entropy for the generic loss function, f(x, 0).
2.2 Previous Researches: Optimization and Parameterization of Dataset Distillation
Dataset Distillation Formulation. The goal of dataset distillation to produce a cardinality- reduced dataset S from the given dataset D, while maximally retaining the task-relevant information of D. The objective of dataset distillation formulated as follows:
min (z,w)e ((( y 0]]] where 0g = arg min
Σ (s..))))
f(x,, ye:
(1)
S
However, the optimization of Eq. (1) is costly and not scalable since it is a bi-level problem [26, 17].
Fowerd Pees Backepppegetin
D
To avoid this problem, various proxy objectives are
Original
Hooling Less
proposed to match the task-relevant information of
D such as gradient [26], features [25], trajectory [3],
NotLeamable Freezeee
etc. Throughout this paper, we generally express
Condensed
Noural Network
these objectives as (D,
(a) Dataset distillation on input- sized variable.
Parameterization of S. Orthogonal to the inves- tigation on objectives, other researchers have also examined the parameterization and corresponding dimensionality of the data variable, By param- eterizing S in directly storing input-sized instances, is possible to distill more instances and enhance the representation capability of 5 for [7, 10]. Figure divides the existing methods into three categories.
$
Original
Passiing Loss
Ry(S)
efficient manner, rather than
Condensed
Transfom
Transforedd
Neutal Nitwork
(b)
Datasct distillation on variable with parameterized transform.
As a method of Figure lb, HaBa [10] proposed a technique for dataset factorization, which involves breaking the dataset into bases and hallucination net- works for diverse samples. However, incorporating an additional network in distillation requires a sepa- rate budget, which is distinct from the data instances.
Original
Vattanng fastaOxi9
g
e(S)
Φ
Condensed
Stalk Transfom
Transfamed
Network
(c) Dataset distillation on variable with static transform.
As method of Figure 1e, IDC [7] proposed the uti- lization of an upsampler module in dataset distillation
Figure Comparison of different parameter- ization strategies for optimization of S.

---

to increase the number of data instances. This parameterization enables the condensation of a single instance with reduced spatial dimensions, thereby increasing the available number of instances. How- ever, the compression still operates on the spatial domain, which results in significant information loss per instance. Please refer to Appendix A. for the detailed literature reviews.
Instance level (Value)
Reconstructed Instance
Dataset [Expisined rana)
Instance level Dataset Vs (s,
Cn ()
Spatial Doman
200 dimensions
Goode Dorrann
low
high
200 dimensions
low
high
Frequency Domain
Frequency Domain
Inverse FT
(a) Statistical properties of dataset on spatial and frequency domain. (b) Tendency of dataset distillation loss.
Figure 2: Visualization of the information concentrated property. (a) On the frequency domain, large proportion of both amplitude (left) and explained variance ratio (right) are concentrated in few dimensions. (b) The magnitude of the gradient for LDD D) is also concentrated in a few frequency dimensions. We utilize trajectory matching for LnD (S, D). A brighter color denotes higher value. Best viewed in color.
Methodology
3.1
Motivation
The core idea of this paper is to utilize a frequency domain to compress information from the spatial domain into small number of frequency dimensions. Therefore, briefly introduce the frequency transform, which is a connector between spatial and frequency domains. function, denoted F, converts an input signal the frequency domain. It results in the frequency representation, ʃ = F(x) E Idixdd? The element-wise expression of f with x is as follows:
xd2
di 11-1d--1
fu, Σ a=0
Fa ((a, b, v)
Here, oba, b, u. v) is basis function for the frequency domain, and the form of Φ(a, b, u, v) differs by the choice of specific transform function, F. In general, an inverse of F-1, exists so that enables the reconstruction of the original input z, from its frequency components f i.c. F (F(x)).
characteristic of the frequency domain is that there exist specific frequency dimensions that encapsulate the major information of data instances from the other domain. According to [1. 21]. natural images tend to exhibit the energy compaction property, which is a concentration of energy in the low-frequency region. Consequently, it becomes feasible to compress the provided instances by exclusively leveraging the low-frequency region. Remark states that there exists a subset of frequency dimensions that minimize the reconstruction error of given instance follows:
Remark 1. ([16)) Given d-dimension data instance of signal x [To...,, Ia)"" let
[Fo,.., fay² be its frequency representation with discrete cosine transform (DCT), i.e. f DCT(x). Also, let (() denote the elements of while other elements are 0. Then, the minimizer of the reconstruction error || DCTT-((( (f()) is [fo... 0,...0.
Figure 2a supports the remark. When we apply a frequency transform to an image, we can see that the coefficients in the now-frequency region have very large values compared to the coefficients in other regions. Also, shown in Remark 1, the reconstructed image is quite similar the original image while only utilizing the lowest frequency dimensions. Beyond the instance level, The right side of Figure represents that certain dimensions in the frequency domain exhibit concentration of variance. It suggests that the frequency domain requires only a small number of dimensions to
Although some frequency transform handles the complex space, the real space for brevity.
3

---

EVR
4
1000
12
Overlapping dimenslons
30
30
800
20
20
20
600
(a) Original dataset
(b) DC (0.0411)
(c) DSA (0.2683)
400
Originw dideeel
DC
DM
of
TM
30
30
0
200
400
600
800
1000
10
20
30
0
10
20
90
10
20
50
Number of selected dimensions EVR
(g) Preservation property of trained S
(d) DM (0.0534)
(e) IDC (0.
(l) TM (0.0242)
Figure 3: (f)-(f) The log of explained variance ratio each frequency domain dimension (surface) and the degree of overlap between D and S for the top-200) dimensions (floor). The parenthesis is l-2 difference of explained variance ratio between D and S. (g) It can be observed that trained S mostly preserves the top-k explained variance ratio dimensions of D, regardless of k.
describe the total variance of dataset. On the other hand, the spatial domain does not exhibit this behavior; poor quality of reconstruction, and similar variance ratio across all dimensions.
To investigate whether this tendency is maintained in the dataset distillation, we conducted frequency analysis of various dataset distillation losses (D D). Figure 2b shows the magnitude of gradient for (DD(S, D) i.e. (DD,, D)Il In contrast to the spatial domain, where the gradients w.r.t the spatial domain are uniformly distributed across each dimension; the gradients w.r.t frequency domain condensed into certain dimensions. This result suggests that many dimensions spatial domain to reduce the loss, but only a few specific dimensions are required in the frequency domain. Furthermore, we compared the frequency domain information of D and S which were trained with losses from previous research. As shown in Figures 3a to 3f, the distribution of explained variance ratios in the frequency domain is very similar across different distillation losses. Also, Figure 3g shows that trained S mostly preserves the top-k explained variance ratio dimensions of D in the frequency domain, regardless of k and the distillation loss function. Consequently, if there exists frequency dimension on D with a low explained variance ratio of frequency representations, it can be speculated that the absence of the dimension will have little impact on the optimization of S.
needed in the
3.2 FreD: Frequency domain-based Dataset Distillation
We introduce the frequency dossin-based dataset distillation method, coined FreD, which only utilizes the subset of entire frequency dimensions. Utilizing FreD on the construction of S has several advantages. First of all, since the frequency domain concentrates information in a few dimensions, FreD can be casy to specify some dimensions that preserve the most information. As a result, by reducing the necessary dimensions for each instance, the remaining budget can be utilized to increase the number of condensed images. Second, FreD can be orthogonally applied to existing spatial- based methods without constraining the available loss functions. Figure 4 illustrates the overview of FreD with the information flow. FreD consists of three main components: 1) Synthetic frequency memory, 2) Binary mask memory, and 3) Inverse frequency transform.
Synthetic Frequency Memory F. Synthetic frequency memory F consists of [F] frequency-label data pair, i.e. F ())(), |0)) Each ((i) e Rd is initialized with frequency representation acquired through the frequency transform of randomly sampled (x.()),,( D, i.e. f() = F))()).
Binary Mask Memory M. To filter out uninformative dimensions in the frequency domain, we introduce a set of binary masks. Assuming the disparity of each class the frequency domain, we utilize class-wise masks M {M(( ...., ,.., M()), ), where each M() E {0, 1)d denotes the binary mask of class c. To filter out superfluous dimensions, each M is constrained to have k non-zero elements, with the remaining dimensions masked zero. Based on M we acquire the class-wise

---

Forward Pass
Back propagadon Lea masie Learaalle
M(
Original DalasetD
Matching Loss (S,(S
Mi+
f())
Transformed Instance
Class wise Binary Mask
Frequency Repessentaton
Condensed DatasetS
Inverse Frequency Transfrm
Transforedd Dotsset
Neural Network
(b) Inverse frequency transform with binary masking.
(a) Overall structure FreD.
Figure 4: Visualization of the proposed method, Frequency domain-based Dataset Distillation, FreD.
filtered frequency representation, (( @ f((), which takes & frequency coefficients as follows:
fur
if M =1
M u,v
O f
otherwise
PCA-based variance analysis is commonly employed for analyzing the distribution data instances. This paper measures the informativeness of each dimension in the frequency domain by utilizing the Explained Variance Ratio (EVR) of each dimension, which denote as ?hu, Ent ohe
Here, of is the variance the u, v-th frequency dimension. We construct mask, M which only utilizes the top-k dimensions based on the Tu, to maximally preserve the class-wise variance of D as follows:
M {
if Mu, is among the top-k values
otherwise
This masking strategy is efficient as it can computed solely using the dataset statistics and does not require additional training with deep neural network. This form of modeling is different from traditional filters such as low/high-pass filters, or band-stop filters, which are common choices in seaenncyyrannebaee
filters pass only certain frequency ranges by utilizing the stylized fact that each instance contains varying amounts of information across different frequency ranges. We conjecture that this type of class-agnostic filter cannot capture the discriminative features of each class, thereby failing to provide adequate information for downstream tasks, i.c. classification. This claim is supported by empirical studies in Section 4.3.
image processing. These
Inverse Frequency Transform F-1. We utilize the inverse frequency transform, F-1, to transform the inferred frequency representation to the corresponding instance on the spatial domain. The characteristics of the inverse frequency transform make it highly suitable as a choice for dataset distillation. First, F-1 is a differentiable function that enables the to the corresponding frequency domain. Second, F-1 is a static and invariant function, which does not require an additional parameter. Therefore, F-1 does not require any budget and training that could lead to inefficiency and instability. Third, the transformation process is efficient and fast. For d-dimension input vector, FF1 requires O(d log d) operation while the convolution layer with r-size kernel needs Odr). Therefore, in the common situation, where log d < r, F-1 is faster than the convolution layer.² We provide ablation study compare the effectiveness of each frequency transform in Section 4.3 and Appendix D.8.
of the gradient
Learning Framework. Following the tradition of dataset distillation [20, 26, 7], the training and evaluation stages FreD are as follows:
F = arg min LDD(Š, D) where F= C F)
(Training)
(5)
0" arg min L(Š*;0) where = F F
(Evaluation)
(6)
'There are several researches which utilize this property to replace the convolution layer with frequency transform [2, 12, 15].
5

---

Algorithm FreD: Frequency domain-based Dataset Distillation
1: Input: Original dataset D; Number of classes C: Frequency transform F; Distillation loss Lon;
Dimension budget per instance k; Number of frequency representations [F]; Learning rate a
2: Initialize F == Ø
3: for to do
4: F(o) - ((F(x())) y.(0))))))))))
) from a class-wise mini- {(z(1)
y ) D(a)
5: Initialize M(e) by Eq. (4) of the main paper
6: end for
7: repeat
8:
F o Bp), from mini-batch BD ~ D and Bp F
9: until convergence
10: Output: Masked frequency representations M @ F
where M @ F denotes the collection of instance-wise masked representation i.c. M O
((M()())) (f(i), ) y By estimating F from the training with Eq (5), evaluate the ef- fectiveness of F by utilizing 0*, which is a model parameter inferred from training with the transformed dataset, F- @ F*). Algorithm provides the instruction of FreD.
Budget Allocation. When we can store n instances which d-dimension vector for each class, the budget for dataset distillation is limited by n x d. In FreD, we utilize k < d dimensions for each instance. Therefore. we can accommodate |F] = [m( > instances with the same budget. After the training of FreD, we acquire M @ F", which actually shares the same dimension the original image. However, we only count k non-zero elements on each because storing 0 in d - A dimension negligible by small bytes. Please refer to Appendix E.3 for more discussion on budget allocation.
3.3 Theoretic Analysis of FreD
This section provides theoretical justification for the dimension selection of FreD by EVR, 17. For validation, we assume that F is linearly bijective.
Let domain A and B be connected by linear bijective function, W. The sum of over a subset of dimensions in domain for a dataset X is equal to the sum of 1) for the dataset transformed to domain B using only the corresponding subset of dimensions.
Proposition 1.
Proposition claims that if two different domains are linearly bijective, the sum of EVR that utilizes only specific dimensions remains the same even when transformed into different domain. other words, if the sum of EVR of specific dimensions in the frequency domain is high, this value can be maintained when transforming dataset only with those dimensions into the other domain.
Corollary 1. Assume that two distinct domains, and C, are linearly bijective with domain by Wn. and We. let X be dataset in domain A, and Xn and Xc be the datasets transformed to domains and C, respectively. Let Vi. and Vč be the set of k dimension indexes that maximize each domain. Let B and Tic be the corresponding sum of n for each domain. If 2 Tickk then the sum of TI for WVs. B. XB is greater than that of Wve. Xc.
Corollary claims that for multiple do- mains that have bijective relationship with a specific domain, a certain do- main can obtain the high value of sum of EVR with the same number of dimen- sions, then when inverse transformed, it can improve the explanation in orig- inal domain. Based on the informa- tion concentration of the frequency do- main, FreD can regarded as method that can better represent the distribution of the original dataset, D, based on PCA, which sets the principal components of the given dataset as new axes, can idcally preserve 7).
60
Acourasy dimereions
Test sccuracc
45
0.8
40
0.2
Speiial Frequency
64
250
500
750
1000
100
200
300
400
500
Number selected dioosions by EVR
Number of selected dimenstons by EVR
(a) Comparison of domain
(b) Efficiency of EVR selection
Figure 5: Empirical evidence for EVR-based selection.

---

Table 1: Test accuracies (%) on SVHN, CIFAR 10, and CIFAR-100. "IPC" denotes the number of images per class. "#Params" denotes the total number of budget parameters. The best results and the second- best result are highlighted in bold and underline, respectively.
SVHN
10 CIFAR-IO
CIPAR-100
IPC
10
50 1536k
1
10
30.72k
30.72k
30.72k
Coreset
Random
14.6
35.1 14.1
70.9
10.9
14.4 12.0
26.0 2:1
43.4
4.2 10.3
14.6 1:0.5
30.0 10.4
Herding
20.9 11.3
50.5 43.3
72.6
21.5
11.3
31.6400
40.4 20.6
844 10.3
17.3 4.0.3
33.7 4005
DC
76.1 +0.00
82.3 +0.3
28.3 +0.5
12.8 +0.3
+0.3
DSA
27.5 11.4
79.2 10.5
10.5
84.4 10.4
28.8 10.7
52.1 10.5
60.6 .0.5
13,9 10.3
32,3 10.3
42.8 10.4
purmmeterization
Input-sized
CAFE+DSA
DM
TM
58.5 #1.4
70.8 #1.8
82.3
85.7 #:1.1
46.3 #0.8
26.0 20.8
65.1 =0.7
48.9 =0.6
71.6 =022
62.3
=0.4
24.3 #0.2
11.4
29.7 1:0.2
40.1 +01.4
10.2
43.6 4:1.4
47.2 #1l2
KIP
57.3 +0.11
75.0 +0.1
80.5 +0.1
49.9 +0.2
62.7 +0.3
68.6 -02
15.7 +0.2
28.3 +0.1
71.7
10.1
10..2
IDC
68.1 +0.1
87.3 +:1.2
90.2 +0.1
50.0 #0.
67.5 =0.5
74.5 =0.1
Parameteriiattin
HaHa
09.8 11.33
83.2 1:04
88.3 10.1
48.3 +0.8
69.9 =044
74.0=02
33.4 +0.4
40.2 +0.2
47.0 +0.2
Fntire original dotaset
95.4 +0.1
84.8 =0.1
56.2 +0.3
dccoded instances
Increment of
HaBa
Frel
IDC
x16
x5
X5
x5
x-5
K5
K5
x16
X5
X5
x6.4
X5
X5
x5
x5
x2.56
x5
x2.50
x5
Binary Mask
Ours (FreD)
Binary Mask
Ours (FreD)
Binary Mask
Ours (FreD)
Baseline (TM)
Baseline (TM)
Baseline (TM)
(a) Class: Airplane
(b) Class: Automobile
(c) Class: Horse
Figure 6: Visualization of the binary mask, condensed image from TM, and the transformed images of FreD on CIFAR-I0 with IPC=1 (#Params=30.722 Binary masks and trained synthetic images are all the same size. The size of mask and baseline image were enlarged for a better layout.
However, PCA, being data-driven transform function, cannot be practically utilized as a method for dataset distillation. Please refer to Appendix E. for supporting evidence of this claim.
Figure 5a shows the sum of 1) in descending order in both the spatial and frequency domains. First, we observe that the number selected dimensions the frequency domain increases. frequency domain is much faster in converging to the total variance than the spatial domain. This result shows that the EVR-based dimension selection the frequency domain is more effective than the selection in the spatial domain. In terms of training. Figure 5b also shows that the performance of a dataset constructed with a very small number frequency dimensions converges rapidly the performance of dataset with the entire frequency dimension.
Experiments
4.1 Experiment Setting
We evaluate the efficacy of FreD on various henchmark datasets, i.e. SVHN [13], CIFAR-10, CIFAR-
100 [8] and ImageNet-Subset [6, 3, 4]. Please refer to Appendix D for additional experimental results on other datasets. We compared FreD with both methods: 1) method to model S as an input-sized variable; and 2) method, which parameterizes differently. As the learning with input-sized S, we chose baselines as DD [20], DSA [24], DM [25]. CAFE+DSA [19], TM [3], KIP [ [14] and FRePo [27]. We also selected IDC [7] and HaBa [10] as baselines, which is categorized the parameterization on S. we also compare FreD with core-set selection methods, such as random selection and herding [22]. We use trajectory matching objective (TM) [3] for LDD as a default although FreD can use any dataset distillation loss. We evaluate each method by training 5 randomly initialized networks from scratch on optimized S. Please refer to Appendix for detailed explanation of datasets and experiment settings.
7

---

Table 2: Test accuracies (%) on CIFAR- under various dataset distillation loss and cross- cross-architecture. "DC/DM/TM" denote the gradien/feeturr//riettoy matching for dataset distillation loss, respectively. We utilize AlexNet [9], VGGI1 [18], and ResNet18 [5] for cross- aroos-architecture.
DC
DM
TM
IPC
11
51
11
2
#Params
61.44k
337.92k
1566.72k
61.44k
337.92k
1566.723
61.44k
337.92k
1566.72k
Vanilla
31.4.10.2 1.22
45.3 10.3
54.2
10.8
34.6 10.5
50.4 20.1
62.0 10.3
50.6 =1.0
63.9 10.3
69.8 10.5
ConvNet
w/ IDC
35.2 +0..
53.8
=0.4
56.4
10.4
45.1
+0.5
59.3
=0.4
64.6
10.3
56.1
=0.4
60.9 10.4
71.1
10.4
HaRa
14.1
+0.5
58.9
37.3 +0.1
56.8 +0.1
64.4 +0.4
-0.4
69.5 +0.3
23.1 +0.2
w/ FreD
45.3-40.5
55.8
#0.4
59.8
#0.5
55.9
#0.4
61.3
=0.8
66.6 #0.6
61.4
=0.3
70.7 #0.5
75.5 #0.2
Vanilla
22.0
29.2 +0.9
34.1 10.6
21.5 +2.2
39.5
52.6 10.7
33.1 =1.1
43.9 +1.4
55.0
Average of Cress- Architectures
w/ IDC
28.7 +1.2
35.4 +0,6
40.2 +0.7
37.3 +1.1
50.5 +0.6
61.3 +0.5
42.5 -1.5
48.7 +1.8
61.5 +1.0
HalBa
25.4
+0.7
35.5
40.6
47.0
60.1 40.6
55.8
+0.9
w/ FreD
37.3 10.9
37.4 10.7
42.7 20.8
48,1 100.7
57.3 20.8
65.0
100.
49.7 110
60.1
1.077
69.1 10.7
Table 3: Test accuracies (%) on ImageNet-Subset (128 x 128) under Table 4: Test accuracies (%) on 3D MNIST.
IPC-2 (#Params-983.00 00k)).
Model
ImgNette
ImgWoof
IngFruit
ImgYellow
Imo
ImgSquawk
IPC
*Parmms
40.50
409.6
2083
TM
55.2 =111
30.9 =13
31.8 =16
49.7 #1.4
35,3 =2.2
43.9 20.6
Random
17.2
49.6 20.7
60.3 #0.7
IDC
65.4 212
37.6 #16
43.0
62.4
55.5 11.
DM
43.5 -0.
15.6 -0.
64.2 :0.5
w/ HaBa
51.9 +1.7
32.4 -0.7
34.7 #11
50.4 41.6
36.9 +0.9
41.9 +1.4
wi IDC
51.9-1.5
360
20.5
5588
20.3
69.0 +0.9
40.0
46.3 -122
66.3 41.1
=1.7
62.0
w/ Frel
62. 1033
66.6 107
danaset
78.
4.2 Experimental Results
Performance Comparison. Table presents the test accuracies of the neural network, which is trained on S inferred from each method. FreD achieves the best performances in all experimental settings. Especially, when the limited budget is extreme, i.e. IPC=1 (#Params=30.77 30k)2k);; FreD shows significant improvements compared to the second-best performer: 12.4%p SVHN and 10.6%p in CIFAR-10. 10. This result demonstrates that using the frequency domain, where information is concentrated specific dimensions, has positive effect on efficiency and performance improvement, especially in situations where the budget is very small. Please refer to Appendix D) for additional experimental results on other datasets.
Qualitative Analysis. Figure 6 visualizes the synthetic dataset by FreD on CIFAR- with IPC=1 (#Params=30.722) 32k... In this setting, we utilize 64 frequency dimensions per channel, which enables the construction of 16 images per class under the same budget. The results show that each class contains diverse data instances. Furthermore, despite of huge reduction in dimensions i.e. 64/1024 6. 6,,5% each image contains cuiss--dscriminatiie features. We also provide the corresponding binary masks, which are constructed by EVR value. As a result, the low- owwffrquenccy dimensions in the frequency domain were predominantly selected, supports that the majority of frequency components for image construction concentrated in the low-frequency region [1, 21]. Furthermore, it should be noted that our EVR-based mask construction does not enforce keeping the low-frequency components, what EVR only enforces is keeping the components with higher explanation ratio on the image feature. Therefore, the constructed binary masks are slightly different for each class. Please refer to Appendix D.10 for more visualization.
Compatibility of Parameterization. The parameterization method in dataset distillation should show consistent performance improvement across different distillation losses and test network architectures. Therefore, we conduct experiments by varying the dataset distillation loss and test network architectures. In the case of HaBa, conducting an experiment at IPC=1 is structurally impossible due the existence of hallucination network. Therefore, for fair comparison, we basically follow HaBa's IPC setting such as IPC=2,11,51, In Table 2, FreD shows the highest performance improvement for all experimental combinations. Specifically, FreD achieves substantial performance gap to the second-best performer up to 10.8%p in training architecture and 10.6%p in aross-arch generalization. Furthermore, in the high-dimensional dataset cases, Table verifies that FreD consistently outperforms other parameterization methods. These results demonstrate that the frequency domain exhibits high compatibility and consistent performance improvement, regardless of its association with dataset distillation objective and test network architecture.
8

---

Table 5: Test accuracies (%) on ImageNet-Subset--C Note that the TM on ImgSquawk is excluded because the off-the- synthetic dataset is not the default size of 128 x 128.
60 50
Todt 4e
*Paraas
Model
ImgNeme-C
ImgWoot-CC
ImgFrait-C
Ing Yellow-C
ImgMeow-CC
ImgSquawi--C
DSA
Tu
401520 (PC=1)
TM
IDC
34.5 20.6
38.0 21.6
23.8
18.7 29.4
22.7 21.1
28.5 10.9
35.6::77
36.8 214
23.3 21.1
222 21.2
26.8 20.5
323
M0.3
Figure 7: Test accuracies
IPC-1
72k) (6Param=-377 24)
IPC-10
(ICC-10)
4915200
W/ Frel
IDC
30.4 21.0
55.2 =0.8
509
21.9 2033
33.8 80.8
30.9
32.3 20.8
35.7 +0.6
32.2 10.7
47.9 =0.4
45.6
39.6 -0.5
31.3 =0.9
23.9 20.8
30.1 =0.5
41.
40.5 20.7
52.5 =0.8
(%) on CIFAR-1O-C.
3D Point Cloud Dataset. As the spatial dimension of the data increases, the required dimension budget for each instance also grows exponentially. To validate the efficay of Frel) on data with dimensions higher than 2D, we assess FreD on 3D point cloud data, 3D MNIST:* Table shows the test accuracies on the 3D MNIST dataset. FreD consistently achieves significant performance improvement over the baseline methods. This confirms the effectiveness of FreD in 2D image domain as well as 3D point cloud domain.
Robustness against Corruption. Toward exploring the application ability of dataset distillation, we shed light on the robustness against the corruption of trained synthetic dataset. We utilize the following test datasets: CIFAR- and CIFAR-IO-C for CIFAR- 10, ImageNet-Subseee C for ImagNet-Subsett Subset. For CIFAR-10. and CIFAR-IO-C experiments, we utilize the off-the-sheli synthetic datasets which are released by the authors of each paper. We report the average test accuracies across 15 types of corruption and 5 severity levels for CIFAR-10-C and Image Subset-C.
Figure 7 and Table 5 show the results of robustness on CIFAR- and ImageNet- tively. From both results, FreD shows the best performance over the whole setting which demonstrates the superior robustness against corruption. We want to note that IDC performs worse than the baseline in many ImageNet-Subseed on the ImageNet-Sutee Subset (see Table 3). On the other hand, FreD consistently shows significant performance improvement regardless of whether the test dataset is corrupted. It suggests that the frequency domain- based parameterization method shows higher domain generalization ability than the spatial domain- based parameterization method. Please refer to Appendix D.5 for the results of CIFAR-10. 10.1 and detailed results based on corruption types of CIFAR-1O-C.
respec-
experiments (see Table 5) although it shows performance improvement
To explain the rationale, corruptions that diminish the predictive ability of a machine learning model often occur at the high-frequency components. Adversarial attacks and texture-based corruptions are representative examples [11, 233. Unlike FreD, which can selectively store information about an image's 's frequency distribution, transforms such as factorization or upsampling are well-known for not preserving frequency-basse based information well. Consequently, previous methods are likely to suffer a decline in predictive ability on datasets that retain class information while adding adversarial noise. In contrast, FreD demonstrates relatively good robustness against distribution shifts by successfully storing the core frequency components that significantly influence class recognition, regardless of the perturbations applied to individual data instances.
Collaboration with Other Parameterization. The isting method either performs resolution resizing in the spatial domain or uses a neural network to change the di- mension requirement of the spatial domain. On the other hand, FreD optimizes the coefficient of the frequency do- main dimension and transforms it into the spatial domain through the inverse frequency transform. Therefore, FreD can be applied orthogonally to the existing spatial domain- based parameterization methods. Table 6 shows the per- formance of different parameterizations applied to HaBa. From the results, we observed that FreD further enhances the performance of HaBa. Furthermore, it is noteworthy that the performance of HaBa integrated with FreD higher than the combination of IDC and HaBa. These results imply that FreD can be well-integrated with spatial domain-based parameterization methods.
Table 6: Test accuracies (%) of each col- laboration on CIFAR-10.
IPC #Params
2
11
61.44k
337.92k
TM
50.6:1:0.
63.9 20.3
w/ HaBa
56.8 10.4
69.5 1033
w/ IDC & HaBa
61.3 10.3
70.9 #0.4
FreD & HaBa
62.3 +0.1
72.9 =0.2
'https://wwe www.:wwwwwg.....o..o...........o www.glee.com/tttsstt/taaaaaeeeee mnist

---

4.3 Ablation Studies
Effectiveness of Binary Mask M.
We conducted comparison experi- ment to validate the explained vari- ance ratio criterion for the selec-
70
Test accuracy (%)
Test accuracy (%%
80
tion of frequency dimensions. We se- lected the baselines for the ablation study as follows: Low-pass, Band- stop, High-pass, Random, and the magnitude of the amplitude in the fre- quency domain. We fixed the total budget and made k the same. Figure 8a illustrates the ablation study on dif- ferent variations of criterion for con- structing M. We skip the high-pass mask because of its low performance: 14.32% in IPC=1 (#Params=30.77 3kk and 17.11% % IPC=10 (#Params=307.22 2k). While Low-pass and Amplitude- based dimension selection also improves the performance of the baseline, EVR-based dimension selection consistently achieves the best performance.
-3 5
wi Low
vel Bans-alop
Wi Armp
Wi DWT We DCT (Dafall))
vel EVR (Dufeed)
30
IPC-1
IPC-10 (#Parame-337 2k)
IPC=1
IPC-10 [PPareme-307 2k)
(AParams-30.722 72k)
(4Parmm-300772 728)
(a) Binary mask M
(b) Frequency transform FF
Figure 8: Ablation studies on (a) the binary mask M, and (b) the frequency transform F.
Effectiveness of Frequency Transform F. We also conducted an ablation study on the frequency transform. Note that the FreD does not impose any constraints the utilization of frequency transform. Therefore, we compared the performance of FreD when applying widely used frequency transforms such the Discrete Cosine Transform (DCT), Discrete Fourier Transform (DFT), and Discrete Wavelet Transform (DWT). For DWT, we utilize the Haar wavelet function and low-pass filter instead of an EVR mask. As shown in Figure 8b, observe significant performance improvement regardless of the frequency transform. Especially, DCT shows the highest performance improvement than other frequency transforms. Please refer to Appendix D.8 for additional experiments and detailed analysis of the ablation study on frequency transform.
Budget Allocation. Dataset distillation aims to include as much information from the original dataset as possible on limited budget. Frel) can increase the number of data Fl by controlling the dimension budget per instance k. and FreD stores the frequency coefficients selected by EVR as k-dimensional vector. For example, with small value of k, more data can be stored i.e. large |F). This is way to increase the quantity of instances while decreasing the quality of variance and reconstruction. By utilizing this flexible trade-off, we can pick the balanced point between quantity and quality to further increase the efficiency of our limited budget. Figure 9 shows the performance of selecting the dimension budget per channel under different budget situations. Note that, for smaller budgets i.e. IPC=1 (#Params=30.725 increasing |F performs better. For larger budget cases, such as IPC=50 (#Params=1536k), allocating more dimensions better i.e. large k. This result shows that there is trade- off between the quantity and the quality of data instances depending on the budget size,
Test accuracy (%)
50
40
30
20
IPC=1
IPC-10
PC
dmm/64 (x 16)
dim:128
dim:256
dim:512
(x
(x 4)
(x 2)
Dimension budget per channel
Figure 9: Ablation study on budget al-
location of FreD with DM. Dashed line indicates the performance of DM.
each instance performs
5 Conclusion
This paper proposes a new parameterization methodology, FreD, that utilizes the augmented frequency domain. FreD selectively utilizes a set of dimensions with high variance ratio in the frequency domain, and FreD only optimizes the frequency representations
corresponding dimensions in the junction with the frequency transform. Based on the various experiments conducted on benchmark datasets, the results demonstrate the efficacy of utilizing the frequency domain in dataset distillation. Please refer to Appendix G for the limitation of the frequency domain-basse dataset distillation.
10

---

Acknowledgement
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (N..202RRAA222002 * MSIT: Ministry of Science and ICT
References
[1] Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions on Computers, 100(1):90-92 1974.
[2] Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards ai. Large-scale kernel machines, 34(5):1-41, 2007.
[3] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. Proceedings of the IEEE/CVF Confer- 2022.
ence on Computer Vision and Puttern Recognition, pages
[4] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3739-3748, 2023.
[5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[6] Jeremy Howard. A smaller subset of 10 easily classified classes from imagenet and little more french. URL http:://github. com/fastai/imagenett,, 2019.
[7] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Wop Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data param- eterization. In International Conference Machine Learning, pages 11102-11118. PMLR. 2022.
[8] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[9] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[10] Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. arXiv preprint 1rXiv:2210.16774, 2022.
[11] Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency domain model augmentation for adversarial attack. In European Conference on Computer Vision, pages 549-566. Springer, 2022.
[12) Michael Mathieu, Mikael Henaff, and Yann LeCun. Fast training of convolutional networks through ffts. arXiv preprint 13 2013.
[13] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[14] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186-5198, 2021.
[15] Harry Pratt. Bryan Williams, Frans Coenen, and Yalin Zheng, Fenn: Fourier convolutional neural networks. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part I 17, pages 786-798. Springer, 2017.
[16] Jorge Rebaza. first course in applied mathematics. John Wiley & Sons, 2021.

---

[17] Noveen Sachdeva and Julian McAuley. Data distillation: arXiv:2301.04272, 2023.
A survey.
arXiv preprint
[18] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 2014.
[19] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang. Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12196-12205, 2022.
[20] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 1891, 2018.
[21] Ye Wang, Miikka Vilermo, and Leonid Yaroslavsky. Energy compaction property of the mdet in comparison with other transforms. In Audio Engineering Society Convention 109. Audio Engineering Society, 2000.
[22] Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual Interna- tional Conference Machine Learning, pages 1121-1128, 2009.
[23] Chenglin Yang, Adam Kortylewski, Cihang Xie, Yinzhi Cao, and Alan Yuille. Patchattack: black-box texture-based attack with reinforcement learning. In European Conference on Computer Vision, pages 681-698. Springer, 2020.
[24] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 16674-12685 PMLR, 2021.
[25] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6514-6523, 2023.
[26] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint Stti::20000059222) 2020.
[27] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.007// 2022.
12


Reviews:
[{"review_id": "mTGKQzKumc", "reviewer": "Reviewer_ajn2", "summary": "This paper introduces a new parameterization method for dataset distillation, called FreD, that utilizes the frequency domain to distill a compact, synthetic dataset from a large-sized original one. This approach differs from conventional methods that focus on the spatial domain. It employs frequency-based transforms to optimize the frequency representations of each data instance. By leveraging the concentration of spatial domain information on specific frequency components, FreD is able to select a subset of frequency dimensions for optimization. The paper provides both theoretical and empirical evidence of FreD's efficiency compared to conventional parameterization methods. In addition, it highlights the compatibility of FreD with existing distillation methods and its consistent improvement in performance over different benchmark datasets. This work addresses the challenges posed by big data and it improves upon the current dataset distillation methods by offering an optimized and memory-efficient solution.", "questions": "1. In section 3.3, can the assumption that the function F is linearly bijective limit the applicability of the method? Is there any situation where this assumption might not hold and how would that affect the results?\n2. How does the approach handle high-dimensional datasets? Is there a limit on the dimensionality that the method can effectively handle?", "limitations": "The authors did not include the limitation section, it will be good to have a discussion on cross architecture generalization ability of the proposed method.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors have done a great job of describing their proposed method, FreD. They offer explicit step-by-step explanations of how each element of the method operates, from the synthetic frequency memory to the binary mask memory and the inverse frequency transform. It helps the reader can follow along with their reasoning.\n2. This work introduces an innovative parameterization based dataset distillation method, and is the first work of studying from a frequency perspective. It lays a foundation for further research and discussion in this frequency-oriented parameterization methods.", "weaknesses": "1. While this work demonstrates the performance increased relative to several previous methods, it did not compare with another parameterization method [1], where the performance seems to be better than this work. It would be good to include this comparison and discuss the potential limitations of the frequency-based parameterization approach.\n2. it's unclear how the choice of transform (Discrete Cosine Transform (DCT), Discrete Fourier Transform (DFT), or Discrete Wavelet Transform (DWT)) might influence the performance or effectiveness of their approach. This aspect is not thoroughly investigated or discussed.\n3. The Spectral Bias could impact the network's ability to generalize to new data, there is little discussion on this which could hurt the effectiveness of the frequency domain-based approach presented in the paper.\n\n[1] Deng, Zhiwei, and Olga Russakovsky. \"Remember the past: Distilling datasets into addressable memories for neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 34391-34404. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your constructive review on our manuscript.\n\n$\\textbf{[W1. Comparison with [1]]}$\n\nThank you for your valuable suggestion. [1] introduces a new parameterization method called memory addressing (MA). The key concept is to create a common representation by encapsulating the features shared among different classes into a set of bases. As the reviewer noted, the reported performances of [1] show mixed results under various settings. However, the performance of [1] is not solely due to the MA but also includes the effects of other components. For a fair comparison between MA and FreD, we standardized the distillation loss and evaluated their performances.\n\nTable 1 in the rebuttal material shows that FreD and MA exhibit competitive performances with each other under the implemented settings of DM and TM losses. In the same Table,  we further assessed the transferability of each approach by evaluating the robustness of the synthetic datasets against diverse distribution shifts. FreD particularly shows better performances than MA in most corrupted versions of datasets. We conjecture that because FreD selects informative dimensions in the frequency domain, it has good robustness to corruptions that typically occur in the high-frequency domain. Please refer to the response to Reviewer Xfn3. In terms of computational cost, MA requires roughly twice the computation time of FreD. We will include further comparisons with MA in the revised version.\n\n$\\textbf{[W2. The impact of utilizing different frequency transforms on FreD]}$\n\nPlease refer to Q1 in the global response.\n\n$\\textbf{[W3. Discussion on Spectral bias and the influence on the frequency domain-based approach]}$\n\nWe appreciate the reviewer's comment. Spectral bias refers to the phenomenon that neural networks are prone to prioritize learning the low-frequency components over relatively higher-frequency components [2,3,4]. For FreD, even though we employed a mask based on the explained variance ratio (EVR), which is not a simple low-frequency filter: the low-frequency components in the frequency domain were predominantly selected in most experimental scenarios. It should be noted that our EVR does not enforce to keep the low-frequency components, unlike the neural network's spectral bias; what EVR only enforces is keeping the components with higher explanation ratio on the image feature. \n\nHowever, this characteristic can become a risk to performance if the task-specific information of data is mostly found in the high-frequency components. These cases include 1) medical imaging on fine details like tumors [5] and 2) digital watermarking [6]. In such cases, there may be a requirement for a new masking that allows FreD to capture important high-frequency components. The masking strategy of FreD can be flexibly operated, and depending on the characteristics of the given dataset and task, it can readily employ other strategies as needed. \n\n$\\textbf{[Q1. The impact of linear bijectivity of $\\mathcal{F}$ on the applicability and performance of FreD]}$\n\n$\\textbf{Impact of linear bijectivity}$ As the reviewer has pointed out, for the theory presented in section 3.3 to hold, the function $\\mathcal{F}$ must be linearly bijective. Note that FreD could utilize any kind of frequency transform, such as DCT, DFT, and DWT. Since these transforms are all linearly bijective in theory, the proposed theorems could be applicable without any limitation.\n\nHaving said that, we elaborate the potential issues that might emerge, when integrating not linearly bijective transforms into the framework of FreD, as follows:\n\n$\\textbf{When $\\mathcal{F}$ is not bijective}$ If $\\mathcal{F}$ is not bijective, then its inverse $\\mathcal{F}^{-1}$ does not exist. This absence makes the process of transforming to another domain and then restoring back to the original domain infeasible. It potentially results in information loss that interferes accurate reconstruction of the original image. An alternative solution could be separating the transform into distinct encoder and decoder components, which enable a procedure for one-to-one mapping. However, the construction of such components necessitates additional training costs, when $\\mathcal{F}$ of FreD does not require any kind of additional training. Furthermore, if the encoding is not one-to-one, information might be lost during the encoding process.\n\n$\\textbf{When $\\mathcal{F}$ is not linear but bijective}$ FreD employs a subset of dimensions in the frequency domain. The choice is feasible as the linearly bijective transform maintains the EVR in the selected dimensions. Proposition and theorem in section 3.3 theoretically support this attribute, where these do not apply to nonlinear bijective transforms. For 2D-image processing, nonlinear bijective transforms include 1) Log-Polar Transform, and 2) Radial Basis Function (RBF) transform. Domains from these transforms do not exhibit the concentration of the original dataset's variance on specific dimensions. \n\n$\\textbf{[Q2. Applicability and Efficacy of FreD on high dimensional datasets]}$\n\nPlease refer to Q2 in the global response.\n\n$\\textbf{[L1. Limitation of Frequency domain-based approach]}$\n\nPlease refer to Q3 in the global response.\n\n$\\textbf{[L2. Cross Architecture Generalization of FreD]}$\n\nPlease refer to Table 2 in the main paper for the performance of cross architecture generalization. FreD overwhelms other baselines over the whole setting.\n\n\n[1] Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks, NeurIPS2022\n\n[2] On the spectral bias of neural networks, ICML2019\n\n[3] Overview frequency principle/spectral bias in deep learning, arXiv2022\n\n[4] Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions, CVPR2020\n\n[5] MRI based medical image analysis: Survey on brain tumor grade classification, BSPC2018\n\n[6] Digital watermarking for deep neural networks, IJMIR2018\n\n"}, {"review_id": "HrSKrHEP16", "reviewer": "Reviewer_tks8", "summary": "This work introduces Frequency-based Dataset Distillation as a new means of parameterizing the distilled dataset that requires a much smaller memory footprint. By learning the distilled dataset in the frequency space (defined by the discrete cosine transform by default), a binary mask based on the class-wise EVR can be used to select only certain frequency components with minimal performance tradeoff. The memory saved by masking out these components can be allocated towards distilling more total samples.", "questions": "I really like the paper overall and would be happy to raise my score if the authors address the few weaknesses described above.\n\n", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This work is well written and includes an exhaustive suite of experiments showing the effectiveness of the method.\n\nNearly all questions I had while reading were addressed later on in the paper.\n\n", "weaknesses": "It is not clear which dataset distillation loss is used in Figure 2. Likewise, it is unclear which distillation loss is used by FreD in Table 1 (although I think it is TM based on info in the appendix).\n\nI would recommend removing IPC from the tables all-together and including the number of parameters instead. For example, instead of IPC = 10 for CIFAR-10, you could have #Params = 30,720 (10x3x32x32). IPC can be misleading when comparing to re-parameterization methods since the IPC simply isn't true anymore.\n\nWould it be possible to see any visualizations for 3D MNIST? I searched through the appendix but only found visualizations for the 2D datasets. I've been very curious to see dataset distillation done on 3D data.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your constructive review on our manuscript.\n\n$\\textbf{[W1. Specifications of the distillation loss utilized in the experiment and figures]}$\n\nThanks for pointing out our lack of explanation. Figure 2 in the main paper utilized DM as a distillation loss to compute the gradient. To demonstrate that our observation is not confined to a specific loss, Figure 1 in the rebuttal material illustrates the magnitude of the gradient in the frequency domain across different distillation losses, which are DC and TM. It shows that the concentration of gradient exhibits a consistent pattern regardless of the type of distillation loss. In the revised paper, we will clarify the information on the utilized dataset distillation loss in Figure 2 of the main paper.\n\nRegarding Table 1 in the main paper, we acknowledge that the reviewer found the answer from section D.3 in the supplementary material. Yes. Table 1 utilizes TM as a distillation loss. We will add the description of utilized distillation loss in the revised paper.\n\n$\\textbf{[W2. Suggestion on representation unit of memory budget]}$\n\nThank you for your valuable suggestion. We acknowledge that the previous researches have utilized Image-per-Class (IPC) as a measure of memory budget, but we agree that the IPC unit is not suitable for analyzing a parameterization method like FreD. We also want to take into account class-shared parameterization, such as the memory addressing in [1], and instead of writing the number of parameters per class, we want to write the total number of parameters in that setting. Specifically, we will replace all IPC with $NumParams$, which follows the form of $Number \\quad of \\quad instances \\times Parameters \\quad of \\quad instance$. For example, IPC=1 for CIFAR-10 will be replaced to $NumParams=10\\times3072$. We strongly agree with changing the way memory budgets are expressed, but to avoid unnecessary confusion, we're sticking with IPC until the discussion period. After discussion period, we will reflect it in the revised paper.\n\n$\\textbf{[W3. Visualizations of 3D-MNIST]}$\n\nAs the reviewer has suggested, we provide Figure 2 in the rebuttal material, which displays the original image and a set of trained synthetic data through each distillation method. To enable visualization of the $16\\times16\\times16$ dimension point cloud, we sliced each instance's depth dimension into 16 images and displayed them separately. The image located at the top left represents the frontmost view, while the image at the bottom right corresponds to the rearmost view. From Figure 2 in the rebuttal material, FreD effectively captures the class-discriminative information that class 0 should possess. It indicates that the proposed frequency-based dataset distillation framework is applicable to higher-dimensional data than two-dimensional data. Furthermore, compared to the DM and IDC, the synthesized instance by FreD shows more clear boundary in dimensions 6~11 which is the key class-discriminative information of 0. This result demonstrates that the selection of informative dimensions in the frequency domain is effective. In the revised paper, we will add the visualization of the 3D-MNIST cloud synthesized by each method.\n\n[1] Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks, NeurIPS2022"}, {"review_id": "o9dcTWki3J", "reviewer": "Reviewer_5mRL", "summary": "The paper proposes a novel way (code named FreD) to perform parameterization on dataset distillation tasks. Different from other methods which all operate on the spatial domain, this paper proposes to convert the spatial domain into frequency domain which are more concentrated and then perform distillation on these concentrated frequency components. \n\nEmpirical evaluations show that FreD is able to achieve SOTA compared to previous methods.\n\n============\n\nI have read the author's responses. Part of my concerns are addressed such as the questions regarding IPC while some others remain debatable. Overall, I would still keep my original score.\n\n============", "questions": "See my comments on weakness. I am willing to raise the score if the above questions are answered so that I can get a better idea of how FreD works beyond what's currently shown in the main paper.", "limitations": "- the evaluation in the main paper is only done on datasets with low resolution, thus the scalability is hard to verify\n- same with a lot of previous parameterization methods, FreD also seem to work best with smaller IPCs  such as IPC 1(where generating more images can boost the performance a lot). This can be seen on CIFAR-100 IPC 50 that even with 2.56 times more images, the performance is close to methods without parameterization.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## originality\n- the paper proposes an interesting new way to perform dataset distillation which has not be explored previously to the best of my knowledge\n\n## quality\n- the paper shows the effectiveness of the proposed methods through an adequate amount of experiments \n- the method is also theoretically proved through proposition 1 and corollary 1.\n\n## clarity\n- the paper is well written and easy to follow\n- the paper categorizes baseline methods clearly which makes the evaluation results easy to follow\n- the increment of decoded instances are also clearly indicated in table 1\n- distilled results are visualized and showed in the paper\n\n## significance\nTo the best of my knowledge, previous parameterization methods mostly perform on the spatial domain. This paper is the first to propose converting spatial domain to frequency domain which concentrates the information better. It could potentially open up a new directions for dataset distillation/condensation methods.", "weaknesses": "- The IPCs used in table 1 are 1/10/50, but 2/11/51 in table 2. The motivation for doing so is not clear.\n- In Figure 2, does the image used come from the evaluation datasets? If not, it's confusing to use an image that is not related to the  evaluation datasets in this paper.\n- The evaluation datasets are all small datasets such as 32 * 32 , in the appendix, only IPC 1 for TinyImageNet is reported. What's the reason for the missing numbers on IPC 10/50? Is it scalability issues or performances issues or something else?\n- In [1], IPC 1 and 10 are used to evaluate the performance on ImageNet subsets, but IPC 2 is used in Fred, what's the specific reason to choose a different IPC than previous methods? Can we see some results on IPC 1 and 10 as well because it will be good to see how FreD performs on higher resolutions.\n\n[1] Dataset Distillation by Matching Training Trajectories", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your constructive review on our manuscript.\n\n$\\textbf{[W1. Differences in IPC settings for each table]}$\n\nWe apologize for any confusion caused to the reviewer by utilizing different IPC values across each table. In the case of HaBa, which is FreD's main baseline, conducting an experiment at IPC=1 is structurally impossible due to the characteristics of HaBa. HaBa requires the learning of two components during the distillation process: the hallucination networks and the bases. Even though they adopted lightweight networks, the utilized hallucination network occupies the budget corresponding to IPC=1, which disables the learning bases on the setting of IPC=1. This is reflected in HaBa's original paper, where they reported the performance at IPC=2/11/51 due to this limitation. For your reference, in Table 1 of the main paper, HaBa's IPC=1 performance is actually measured with an IPC=2 budget. We have identified that this is not clearly written and will add an explanation in the revised paper.\n\nWhile FreD and IDC do not have this constraint, we conducted the experiments at IPC=2/11/51 in Table 2-3 in the main paper and Table 2 in the supplementary material to facilitate a fair comparison with HaBa. For your information, you can check the performance of FreD on CIFAR-10, IPC=1/10 case in Table 3 in the rebuttal material. Please see the \"w/ DCT\" case.\n\n$\\textbf{[W2. Specification of the image on Figure 2 in the main paper]}$\n\nWe utilized an instance of airplane class in CIFAR-10 for Figure 2 in the main paper. For the convenience of analysis, it has been converted to a gray image. It does not hurt the applicability of FreD, because the transformation to the frequency domain is applied independently for each channel.\n\n$\\textbf{[W3. Questions on additional experiments on TinyImageNet and the presence of scalability issues]}$\n\nWe report the experiment results on high-dimensional data, such as Tiny-ImageNet ($3\\times64\\times 64$), ImageNet-Subset ($3\\times128\\times 128$), and LSUN ($3\\times128\\times128$) in the supplementary material (Table 1, 2) and rebuttal material (Table 4 and Figure 4). Please check the Q2 of the global response for the details. \n\nIn an experiment with the Tiny-ImageNet at a setting of IPC=10, the TM model recorded an accuracy of 23.2%, while FreD showed an improvement with an accuracy of 24.15%. In the IPC=50 case, the TM recorded an accuracy of 28.0%, while FreD recorded 26.4%. \n\nFreD's motivation lies in leveraging select important dimensions of the frequency domain, which can contain much of the spatial domain's information, to utilize the given memory budget more efficiently. This efficiency manifests greater utility when the available memory budget is more limited, and as the reviewer pointed out, FreD demonstrates more substantial performance improvement in most experiments with an IPC=1 setting.\n\nTinyImageNet is originally a dataset with 500 instances per class, and the IPC=50 setting for this dataset could be considered a not-so-drastic reduction.\n\nIn situations where such a significant reduction doesn't occur, FreD's motivation may be weakened. Excluding this particular setting, FreD consistently demonstrates superior performance compared to the baseline across evaluations with more diverse datasets that have high resolution and large volumes. Please refer to the supplementary material (Table 1, and 2) and the rebuttal material (Table 4 and Figure 4) for the results. \n\nWe conjecture that the FreD has small computation complexity in terms of data retrieval. FreD consists of two operations: EVR masking and inverse frequency transform. Both operations do not require heavy computation costs. We demonstrate that FreD's complexity of single-image retrieval in the response of W3 to Reviewer Xfn3. For empirical evidence, Table 2 in the rebuttal material indicates the empirical wall-clock time of single-image retrieval for parameterization methods. As a result, FreD achieves the second-best single-image retrieval complexity.\n\n$\\textbf{[W4. Question on the selection of different IPC than previous methods and request on the additional experiments]}$\n\nPlease refer to the response of W1 for the specific reason for choosing IPC=2 for performance comparison. We additionally report the performance of IPC=1/10 on ImageNet-Subset in Table 4 (Top) in the rebuttal material. Due to the time constraint imposed by the rebuttal deadline, we reduced the number of training iterations for each baseline and FreD from 15000 to 3000. It should be noted that the performance of FreD at IPC=1/10 already overwhelms the performance of HaBa at IPC=2/11, respectively. In the revised paper, we will reflect on the final performance based on the ultimate iteration of training."}, {"review_id": "CJqKkIscZz", "reviewer": "Reviewer_Xfn3", "summary": "This paper proposes a frequency-domain dataset distillation approach that can help to select the most important information in images to help distillation. To do this, the approach starts with a DCT to the input $x$ and then performs loss matching using existing dataset distillation approaches. Experiments demonstrate its advantages over state-of-the-art approaches.", "questions": "See weakness", "limitations": "See weakness", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using frequency domain information is novel and helpful.\n2. The experiments are good and solid.\n3. The visualization helps to understand the algorithm and results better.", "weaknesses": "1. Experiments are solid, but not enough. I think that dataset distillation is useful for large volume datasets. But experiments do not contain any large datasets. So, the experiments are not useful to help interprete the usefulness of the algorithm.\n2. There are no transfer experiments. Since dataset distillation is used to store a proxy of large datasets, and large datasets are often act as the source domain for transfer learning. Therefore, authors should conduct experiments on transfer learning datasets to show the advantage of the proposed approach. Only doing experiments on CIFAR-level datasets are not enough since to be honest, CIFAR is small and does not need to distill.\n3. Any complexity analysis? How efficient is the algorithm?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your constructive review on our manuscript.\n\n$\\textbf{[W1. A request for experiments on a large volume dataset]}$\n\nWe appreciate the valuable comments of the reviewer. The volume of the dataset could increase by two factors: either 1) the dimension of each instance in the dataset grows, or 2) the overall size of the dataset increases. \n\nFirstly, we conducted the experiments on several high-dimensional datasets such as Tiny-ImageNet, ImageNet-Subset, and LSUN. Please refer to the details in the Q2 of the global response. The results suggest that FreD may be more efficient than spatial domain-based methods as the data dimension increases, and this is empirically demonstrated through experiments.\n\nSecondly, previous researchers in the dataset distillation only focus on investigating the effect of dimension. Therefore, we also investigate the effect of the overall size of the dataset. We choose LSUN dataset [1] as the large dataset. To verify the performance during the rebuttal period, the original dataset was randomly sampled with 10,000/25,000 instances per class which results in a total 100,000/250,000, and experiments were conducted after downsizing each instance to a $3\\times128\\times128$ size. Figure 4 in the rebuttal material provides performances of FreD and other baselines on the LSUN dataset. As a result, FreD achieves the best performance compared to the implemented baselines.\n\n$\\textbf{[W2. Investigation on transferability of synthetic datasets]}$\n\nWe are truly thankful for the feedback on this aspect. It is highly important to verify the transferability of a synthetic dataset learned through dataset distillation, where previous works did not explore the aspect. \n\nAs an evaluation on the transferability of synthetic datasets across different methods, we refer to a test-time adaptation scenario, which is an example case of transfer learning. For an evaluation of the transferability of the synthetic dataset which was trained on CIFAR-10 and ImageNet, we utilized the following target datasets:\n\n* CIFAR-10.1 [2] consists of 2000 new test images which have the same classes as CIFAR-10.\n* CIFAR-10-C and ImageNet-C [3] aim at measuring the robustness of object recognition based on CIFAR-10 and ImageNet, respectively. They have 15 types of corruption and each corruption has five levels with level 5 indicating the most corrupted. \n\nWe created ImageNet-Subset-C by selecting data from ImageNet-C that matches the ImageNet-Subset classes. To examine the transferability of dataset distillation methods, our experiment was conducted in three stages:\n\n1) Distill the synthetic dataset from a source dataset.\n2) Train a neural network from scratch using the synthetic dataset. \n3) Test the network on the target dataset.\n\nFigure 3 in the rebuttal material shows the results of transferability on CIFAR-10.1 and CIFAR-10-C. FreD shows superior accuracy over the whole setting. Furthermore, Table 4 (bottom) in the rebuttal material also shows that FreD achieves better performances in ImageNet-Subset-C. These experimental results indicate that FreD is an efficient dataset distillation method in terms of transferability.\n\nTo explain the rationale, corruptions that diminish the predictive ability of a machine learning model often occur at the high-frequency components. Adversarial attacks and texture-based corruptions are representative examples [4,5]. Unlike FreD, which can selectively store information about an image's frequency distribution, transforms such as factorization or upsampling are well-known for not preserving frequency-based information well. Consequently, previous methods are likely to suffer a decline in predictive ability on datasets that retain class information while adding adversarial noise. In contrast, FreD demonstrates relatively good robustness against distribution shifts by successfully storing the core frequency components that significantly influence class recognition, regardless of the perturbations applied to individual data instances. \n\n$\\textbf{[W3. Complexity and efficiency of each algorithm]}$\n\nAs a complexity analysis, we consider the computation time for single image retrieval based on FreD and other parameterization methods. For $H\\times W$ sized 2D image in a spatial domain, IDC utilizes a resizing method as a parameterization, so its complexity is $\\mathcal{O}(HW)$. HaBa, which requires an additional network for computation, inherits the complexity, $\\mathcal{O}(HW{F}^{2})$, where $F$ is the filter size of the convolution neural network. MA [6] operates by performing matrix multiplication between the matrix and downsampled bases. Given that $K$ represents the number of bases and s is the downsampling scale, MA's complexity is $\\mathcal{O}(HW \\times \\frac{K}{s^2})$. FreD, when used with DCT, involves two operations: masking and inverse frequency transform. The complexities of these steps are $\\mathcal{O}(HW)$ and $\\mathcal{O}(HW\\log W)$ respectively. Thus, the total complexity of FreD is $\\mathcal{O}(HW\\log W)$.\n\nBased on the above complexity, Table 2 in the rebuttal material shows the empirical wall-clock time of single-image retrieval for each method. As a result, we show that FreD inherits the second-best single-image retrieval complexity. Although IDC is most efficient in time complexity, we empirically demonstrated that the parameterization based on IDC falls behind FreD in terms of performance in most settings. \n\n[1] LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop, arxiv2015\n\n[2] Do CIFAR-10 Classifiers Generalize to CIFAR-10?, ICML 2019 \n\n[3] Benchmarking NEURAL NETWORK ROBUSTNESS TO COMMON CORRUPTIONS AND PERTURBATIONS, ICLR 2019\n\n[4] Frequency Domain Model Augmentation for Adversarial Attack, ECCV 2022\n\n[5] PatchAttack: A Black-Box Texture-Based Attack with Reinforcement Learning, ECCV 2020\n\n[6] Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks, NeurIPS2022"}]

Author rebuttal:
$\textbf{[Q1. The impact of utilizing different frequency transforms on FreD]}$

We utilized three frequency transforms as the reviewers noted: DCT, DFT, and DWT. We especially want to highlight the energy compaction property of DCT, where most of the signal information tends to be concentrated in a few low-frequency components (Please refer to Figure 2 in [1] and Figure 1 in [2]). This characteristic aligns well with the motivation of FreD, and Figure 7-(b) in the main paper demonstrates that DCT is the best choice among the possible options.

To further analyze the effect of frequency transforms on FreD, we have conducted various experiments. Table 3 in the rebuttal material presents the results as follows:

* Across all settings, we observe improved performances of FreD than the baseline regardless of the type of frequency transform employed. 

* DCT outperforms DFT and DWT in most cases, highlighting the effective exploitation of DCT's energy compaction property within the FreD framework.

* DFT exhibits relatively lower performance in comparison to DCT and DWT. This discrepancy is attributed to the complex-valued nature of DFT. Unlike DCT and DWT, which operate in real frequency space, DFT requires additional resources to represent a single instance due to its complex space. As a result, the quantity of synthetic instances that can be generated within an identical budget is reduced by half than others, leading to lower performance.

$\textbf{[Q2. Applicability and Efficacy of FreD on High dimensional dataset]}$

In the spatial domain, discerning the informative dimensions is fundamentally challenging. As a result, dataset distillation must consider all dimensions within the spatial domain during the optimization. If all dimensions are not taken into account, it inevitably leads to a loss of information. In contrast, FreD selectively optimizes the essential dimensions by utilizing the transform to the frequency domain, thus it potentially magnifies its efficiency as the data dimension increases. We conjecture that this property is the main reason for the improved performance of FreD than other baselines.

To support our conjecture, we report the performances of FreD and other methods on Tiny-ImageNet ($3\times64\times64$) and ImageNet-Subset ($3\times128\times128$) in Table 1,2 in the supplementary material. From the results, FreD outperforms the baselines by a large margin in whole settings. Additionally, in the rebuttal material, we also report the performance on LSUN dataset ($3\times128\times128$) and additional results on ImageNet-Subset in Figure 4 and Table 4, respectively. FreD empirically shows consistently improved performances than the baselines. These results demonstrate the effectiveness of frequency domain-based perspective in dataset distillation.

$\textbf{[Q3. Limitation of FreD]}$

$\textbf{1. Efficacy differences depending on the applied domain}$ The applicability and efficacy of FreD's frequency transform are demonstrated specifically within the spatial domain of 2D/3D images. Among the data domains commonly used in machine learning framework, natural language domain would likely be challenging to connect with the frequency domain directly. While Fourier Transform can still be applied to text after it's been converted into a numerical format, such as a time series, this conversion is non-trivial and the resulting frequency domain representation may not be as intuitively meaningful. Thus, for certain domains, the effectiveness of frequency transform may not be as substantial as it is for 2D/3D images.

However, in multi-modal tasks, major components like audio signals and video data naturally align with the frequency domain. Tools such as the 1D Fourier transform for audio signals and 3D Fourier transform for video data already exist to process these types directly. Excluding a few specific domains, FreD would be a framework that can be applied across a broader range of domains.

$\textbf{2. Performances highly dependent on masking strategy}$ FreD's frequency-based parameterization is motivated by the fact that the spatial domain information of a 2D image can be concentrated in specific components of the transformed frequency domain. The explained variance ratio (EVR) based masking selects important dimensions from the frequency domain, consistently showing strong performances across the various experiments conducted in this study. 

Having said that, results in Figure 7-(a) of the main paper demonstrate that there could be significant performance disparities depending on the masking strategy employed. Therefore, there could be substantial issues if the chosen masking strategy fails to select the important dimensions accurately. For certain datasets or tasks, essential task-specific information might be contained in the high-frequency region. These cases could include 1) medical imaging on fine details like tumors [3] and 2) digital watermarking [4]. This highlights a limitation that EVR masking may not be suitable for all data and tasks.

It should be noted that the masking strategy of FreD can be flexibly operated, and depending on the characteristics of the given dataset and task, it is not restricted to using an EVR-based mask and can readily employ other strategies as needed. 

One potential solution to identify task-related frequency components is to utilize the gradient of the given task loss. If specific frequency components have a substantial gradient distribution, it indicates that the component greatly influences the task. By substituting with gradient-based masking, we could address the potential limitations that EVR mask might have.

[1] Discrete Cosine Transform, IEEE Trans. Comput. 1974

[2] Compression, Restoration, Resampling, “Compressive Sensing”: Fast Transforms in Digital Imaging, arxiv2015

[3] MRI based medical image analysis: Survey on brain tumor grade classification, BSPC2018

[4] Digital watermarking for deep neural networks, IJMIR2018

Meta-review: The manuscript has been reviewed by four reviewers, where the ratings are respectively 4,5,6,8.

The 4-rating review is the only one on the negative side. The only remaining concern from this reviewer is the FreD + Bptt. 

The authors have demonstrated good performance on this setting in the rebuttal (despite the review didn't reply to this last post).

As such, it is considered all the reviewers all in general satisfied with the manuscript. The AC agrees with this consensus and finds no basis to overturn the convergence.

Congrats!
Decision: Accept (poster)