Title: Frequency Domain-Based Dataset Distillation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dataset distillation

Frequency domain

Dataset condensation
Abstract: This paper presents FreD, a novel parameterization method for dataset distillation, which utilizes the frequency domain to distill a small-sized synthetic dataset from a large-sized original dataset. Unlike conventional approaches that focus on the spatial domain, FreD employs frequency-based transforms to optimize the frequency representations of each data instance. By leveraging the concentration of spatial domain information on specific frequency components, FreD intelligently selects a subset of frequency dimensions for optimization, leading to a significant reduction in the required budget for synthesizing an instance. Through the selection of frequency dimensions based on the explained variance, FreD demonstrates both theoretical and empirical evidence of its ability to operate efficiently within a limited budget, while better preserving the information of the original dataset compared to conventional parameterization methods. Furthermore, Based on the orthogonal compatibility of FreD with existing methods, we confirm that FreD consistently improves the performances of existing distillation methods over the evaluation scenarios with different benchmark datasets. We release the code at https://github.com/sdh0818/FreD.

Frequency Domain-based Dataset Distillation

Abstract

This paper presents FreD, a novel parameterization method for dataset distillation, which utilizes the frequency domain to distill a small-sized synthetic dataset from a large-sized original dataset. Unlike conventional approaches that focus on the spatial domain, FreD employs frequency-based transforms to optimize the frequency representations of each data instance. By leveraging the concentration of spatial domain information on specific frequency components, FreD intelligently selects a subset of frequency dimensions for optimization, leading to a significant reduction in the required budget for synthesizing an instance. Through the selection of frequency dimensions based on the explained variance, FreD demonstrates both theoretical and empirical evidence of its ability to operate efficiently within a limited budget, while better preserving the information of the original dataset compared to conventional parameterization methods. Furthermore, based on the orthogonal compatibility of FreD with existing methods, we confirm that FreD consistently improves the performances of existing distillation methods over the evaluation scenarios with different benchmark datasets. We release the code at https://github.com/sdh0818/FreD .

1 Introduction

The era of big data presents challenges in data processing, analysis, and storage; and researchers have studied the concept of dataset distillation [20, 26, 14] to resolve these challenges. Specifically, the objective of dataset distillation is to synthesize a dataset with a smaller cardinality that can preserve the performance of original large-sized datasets in machine learning tasks. By distilling the key features from the original dataset into a condensed dataset, less computational resources and storage space are required while maintaining the performances from the original dataset. Dataset distillation optimizes a small-sized variable to represent the input, not the model parameters. This optimization leads the variable to store a synthetic dataset, and the variable is defined in the memory space with the constraint of limited capacity. Since dataset distillation involves the optimization of data variables with limited capacity, distillation parameter designs, which we refer to as parameterization, could significantly improve corresponding optimization while minimizing memory usage.

Some of the existing distillation methods, i.e. 2D image dataset distillations [24, 14, 3, 25, 27], naively optimize data variables embedded on the input space without any transformation or encoding. We will refer to distillation on the provided input space as spatial domain distillation, as an opposite concept of frequency domain distillation that is the focus of this paper. The main drawback of spatial domain distillation would be the difficulty in specifying the importance of each pixel dimension on the spatial domain, so it is necessary to utilize the same budget as the original dimension for representing a single instance. From the perspective of a data variable, which needs to capture the key information of the original dataset with a limited budget, the variable modeling with whole dimensions becomes a significant bottleneck that limits the number of distilled data instances. Various spatial domain-based

parameterization methods [7, 10] have been proposed to overcome this problem, but they are either highly vulnerable to instance-specific information loss [7] or require additional training with an auxiliary network [10].

This paper argues that the spatial domain distillation has limitations in terms of 1) memory efficiency and 2) representation constraints of the entire dataset. Accordingly, we propose a novel Frequency domain-based dataset Distillation method, coined FreD, which contributes to maintaining the task performances of the original dataset based on the limited budget. FreD employs a frequency-based transform to learn the data variable in the transformed frequency domain. Particularly, our proposed method selects and utilizes a subset of frequency dimensions that are crucial for the formation of an instance and the corresponding dataset. By doing so, we are able to achieve a better condensed representation of the original dataset with even fewer dimensions, which corresponds to significant efficiency in memory. Throughout the empirical evaluations with different benchmarks, FreD shows consistent improvements over the existing methods regardless of the distillation objective utilized.

2 Preliminary

2.1 Basic Notations

This paper primarily focuses on the dataset distillation for classification tasks, which is a widely studied scenario in the dataset distillation community [20, 26]. Given C classes, let X ∈ R d denote the input variable space, and Y = { 1 , 2 , ..., C } represent the set of candidate labels. Our dataset is D = { ( x i , y i ) } N i =1 ⊆ X × Y . We assume that each instance ( x, y ) is drawn i.i.d from the data population distribution P . Let a deep neural network ϕ θ : X → Y be parameterized by θ ∈ Θ . This paper employs cross-entropy for the generic loss function, ℓ ( x, y ; θ ) .

2.2 Previous Researches: Optimization and Parameterization of Dataset Distillation

Dataset Distillation Formulation. The goal of dataset distillation is to produce a cardinalityreduced dataset S from the given dataset D , while maximally retaining the task-relevant information of D . The objective of dataset distillation is formulated as follows:

However, the optimization of Eq. (1) is costly and not scalable since it is a bi-level problem [26, 17]. To avoid this problem, various proxy objectives are proposed to match the task-relevant information of D such as gradient [26], features [25], trajectory [3], etc. Throughout this paper, we generally express these objectives as L DD ( S, D ) .

Parameterization of S . Orthogonal to the investigation on objectives, other researchers have also examined the parameterization and corresponding dimensionality of the data variable, S . By parameterizing S in a more efficient manner, rather than directly storing input-sized instances, it is possible to distill more instances and enhance the representation capability of S for D [7, 10]. Figure 1 divides the existing methods into three categories.

As a method of Figure 1b, HaBa [10] proposed a technique for dataset factorization, which involves breaking the dataset into bases and hallucination networks for diverse samples. However, incorporating an additional network in distillation requires a separate budget, which is distinct from the data instances.

As a method of Figure 1c, IDC [7] proposed the utilization of an upsampler module in dataset distillation

𝐷

𝑆

𝜙𝜃

Matching Loss

ℒ𝐷𝐷(𝑆,𝐷)

Neural

Network

Condensed

Original

𝜙𝜃

Forward Pass

Back-propagation

Learnable

Not Learnable

or Freeze

(a) Dataset distillation on input-sized variable.

𝑔𝜓

𝑆

Parameterized

Transform

Condensed

𝐷

𝜙𝜃

Matching Loss

ℒ𝐷𝐷(𝑔𝜓(𝑆), 𝐷)

Transformed

Original

𝜙𝜃

𝑔𝜓(𝑆)

Neural

Network

(b) Dataset distillation on variable with

parameterized transform.

𝑔

𝑆

Static

Transform

Condensed

𝐷

𝜙𝜃

Matching Loss

ℒ𝐷𝐷(𝑔(𝑆),𝐷)

Transformed

Original

𝜙𝜃

𝑔(𝑆)

Neural

Network

(c) Dataset distillation on variable with

static transform.

to increase the number of data instances. This parameterization enables the condensation of a single instance with reduced spatial dimensions, thereby increasing the available number of instances. However, the compression still operates on the spatial domain, which results in a significant information loss per instance. Please refer to Appendix A.1 for the detailed literature reviews.

low

high

Instance level

(Value)

Reconstructed

Instance

Dataset level

(Explained ratio)

200 dimensions

200 dimensions

Spatial

Domain

Frequency

Domain

low

high

Inverse FT

Instance level

∇𝑆𝑖

ℒ𝐷𝐷

𝑆, 𝐷

Spatial

Domain

Frequency

Domain

Dataset level

∇𝑆ℒ𝐷𝐷

𝑆, 𝐷

3 Methodology

3.1 Motivation

The core idea of this paper is to utilize a frequency domain to compress information from the spatial domain into a small number of frequency dimensions. Therefore, we briefly introduce the frequency transform, which is a connector between spatial and frequency domains. Frequency-transform function, denoted as F , converts an input signal x to the frequency domain. It results in the frequency representation, f = F ( x ) ∈ R d 1 × d 2 . 2 The element-wise expression of f with x is as follows:

Here, ϕ ( a, b, u, v ) is a basis function for the frequency domain, and the form of ϕ ( a, b, u, v ) differs by the choice of specific transform function, F . In general, an inverse of F , F -1 , exists so that it enables the reconstruction of the original input x , from its frequency components f i.e. x = F -1 ( F ( x )) .

A characteristic of the frequency domain is that there exist specific frequency dimensions that encapsulate the major information of data instances from the other domain. According to [1, 21], natural images tend to exhibit the energy compaction property, which is a concentration of energy in the low-frequency region. Consequently, it becomes feasible to compress the provided instances by exclusively leveraging the low-frequency region. Remark 1 states that there exists a subset of frequency dimensions that minimize the reconstruction error of a given instance as follows:

Remark 1. ([16]) Given d -dimension data instance of a signal x = [ x 0 , ..., x d ] T , let f = [ f 0 , ..., f d ] T be its frequency representation with discrete cosine transform (DCT), i.e. f = DCT ( x ) . Also, let f ( k ) denote the k elements of f while other elements are 0. Then, the minimizer of the reconstruction error ∥ x -DCT -1 ( f ( k ) ) ∥ 2 is [ f 0 , , , , f k , 0 , ... 0] .

Figure 2a supports the remark. When we apply a frequency transform to an image, we can see that the coefficients in the low-frequency region have very large values compared to the coefficients in other regions. Also, as shown in Remark 1, the reconstructed image is quite similar to the original image while only utilizing the lowest frequency dimensions. Beyond the instance level, The right side of Figure 2a represents that certain dimensions in the frequency domain exhibit a concentration of variance. It suggests that the frequency domain requires only a small number of dimensions to

























(a) Original dataset

























(b) DC (0.0411)

























(c) DSA (0.2683)

























(d) DM (0.0534)

























(e) IDC (0.1020)

























(f) TM (0.0242)









































(g) Preservation property of trained

S

describe the total variance of a dataset. On the other hand, the spatial domain does not exhibit this behavior; poor quality of reconstruction, and a similar variance ratio across all dimensions.

To investigate whether this tendency is maintained in the dataset distillation, we conducted a frequency analysis of various dataset distillation losses L DD ( S, D ) . Figure 2b shows the magnitude of gradient for L DD ( S, D ) i.e. ∥∇ S L DD ( S, D ) ∥ . In contrast to the spatial domain, where the gradients w.r.t the spatial domain are uniformly distributed across each dimension; the gradients w.r.t frequency domain are condensed into certain dimensions. This result suggests that many dimensions are needed in the spatial domain to reduce the loss, but only a few specific dimensions are required in the frequency domain. Furthermore, we compared the frequency domain information of D and S which were trained with losses from previous research. As shown in Figures 3a to 3f, the distribution of explained variance ratios in the frequency domain is very similar across different distillation losses. Also, Figure 3g shows that trained S mostly preserves the topk explained variance ratio dimensions of D in the frequency domain, regardless of k and the distillation loss function. Consequently, if there exists a frequency dimension on D with a low explained variance ratio of frequency representations, it can be speculated that the absence of the dimension will have little impact on the optimization of S .

3.2 FreD: Frequency domain-based Dataset Distillation

We introduce the frequency domain-based dataset distillation method, coined FreD, which only utilizes the subset of entire frequency dimensions. Utilizing FreD on the construction of S has several advantages. First of all, since the frequency domain concentrates information in a few dimensions, FreD can be easy to specify some dimensions that preserve the most information. As a result, by reducing the necessary dimensions for each instance, the remaining budget can be utilized to increase the number of condensed images. Second, FreD can be orthogonally applied to existing spatial-based methods without constraining the available loss functions. Figure 4 illustrates the overview of FreD with the information flow. FreD consists of three main components: 1) Synthetic frequency memory , 2) Binary mask memory , and 3) Inverse frequency transform .

Synthetic Frequency Memory F . Synthetic frequency memory F consists of | F | frequency-label data pair, i.e. F = { ( f ( i ) , y ( i ) ) } | F | i =1 . Each f ( i ) ∈ R d is initialized with a frequency representation acquired through the frequency transform of randomly sampled ( x ( i ) , y ( i ) ) ∼ D , i.e. f ( i ) = F ( x ( i ) ) .

Binary Mask Memory M . To filter out uninformative dimensions in the frequency domain, we introduce a set of binary masks. Assuming the disparity of each class in the frequency domain, we utilize class-wise masks as M = { M (1) , ..., M ( C ) } , where each M ( c ) ∈ { 0 , 1 } d denotes the binary mask of class c . To filter out superfluous dimensions, each M ( c ) is constrained to have k non-zero elements, with the remaining dimensions masked as zero. Based on M ( c ) , we acquire the class-wise

+ 𝑓

+

Original

Dataset

𝐷

Class-wise

Binary Mask

Frequency

Representation

Matching Loss

ℒ𝐷𝐷(𝑆, 𝐷)

Transformed

Dataset

𝑓 𝑓

𝑀 𝑐

𝑓 𝑓

𝑓

𝑖

𝑓 𝑓

𝑓 𝑓

𝑓 𝑓

𝑓 𝑓

𝑓 𝑓 𝑥

Condensed Dataset

𝑆

𝑓 𝑓 𝑥

Inverse

Frequency

Transform

𝜙𝜃

𝜙𝜃

Neural

Network

Forward Pass

Back-propagation

Learnable

Not Learnable

or Freeze

(a) Overall structure of FreD.

𝑀𝐶

⨀

𝑀 𝑐

𝑓

𝑓

𝑖

Transformed

Instance

(b) Inverse frequency transform with binary masking.

filtered frequency representation, M ( c ) ⊙ f ( i ) , which takes k frequency coefficients as follows:

PCA-based variance analysis is commonly employed for analyzing the distribution data instances. This paper measures the informativeness of each dimension in the frequency domain by utilizing the Explained Variance Ratio (EVR) of each dimension, which we denote as η u,v = σ 2 u,v ∑ u ′ ,v ′ σ 2 ′ ′ .

u

,v

Here, σ 2 u,v is the variance of the u, v -th frequency dimension. We construct a mask, M ( c ) u,v , which only utilizes the topk dimensions based on the η u,v to maximally preserve the class-wise variance of D as follows:

This masking strategy is efficient as it can be computed solely using the dataset statistics and does not require additional training with a deep neural network. This form of modeling is different from traditional filters such as low/high-pass filters, or band-stop filters, which are common choices in image processing. These frequency-range-based filters pass only certain frequency ranges by utilizing the stylized fact that each instance contains varying amounts of information across different frequency ranges. We conjecture that this type of class-agnostic filter cannot capture the discriminative features of each class, thereby failing to provide adequate information for downstream tasks, i.e. classification. This claim is supported by our empirical studies in Section 4.3.

Inverse Frequency Transform F -1 . Weutilize the inverse frequency transform, F -1 , to transform the inferred frequency representation to the corresponding instance on the spatial domain. The characteristics of the inverse frequency transform make it highly suitable as a choice for dataset distillation. First, F -1 is a differentiable function that enables the back-propagation of the gradient to the corresponding frequency domain. Second, F -1 is a static and invariant function, which does not require an additional parameter. Therefore, F -1 does not require any budget and training that could lead to inefficiency and instability. Third, the transformation process is efficient and fast. For d -dimension input vector, F -1 requires O ( d log d ) operation while the convolution layer with r -size kernel needs O ( dr ) . Therefore, in the common situation, where log d < r , F -1 is faster than the convolution layer. 3 We provide an ablation study to compare the effectiveness of each frequency transform in Section 4.3 and Appendix D.8.

Learning Framework. Following the tradition of dataset distillation [20, 26, 7], the training and evaluation stages of FreD are as follows:

Algorithm 1 FreD: Frequency domain-based Dataset Distillation

1: Input: Original dataset D ; Number of classes C ; Frequency transform F ; Distillation loss L DD ; Dimension budget per instance k ; Number of frequency representations | F | ; Learning rate α

2: Initialize F = ∅

3: for c = 1 to C do

4: F ( c ) ←{ ( F ( x ( i ) ) , y ( i ) ) } | F | C i =1 from a class-wise mini-batch { ( x ( i ) , y ( i ) ) } | F | C i =1 ∼ D ( c )

5: Initialize M ( c ) by Eq. (4) of the main paper

6: end for

7: repeat

8: F ← F -α ∇L ( F ( M ⊙ B ) , B ) from a mini-batch B ∼ D and B ∼ F

9: until convergence

10: Output: Masked frequency representations M ⊙ F

where M ⊙ F denotes the collection of instance-wise masked representation i.e. M ⊙ F = { ( M ( y ( i ) ) ( f ( i ) ) , y ( i ) ) } | F | i =1 . By estimating F ∗ from the training with Eq (5), we evaluate the effectiveness of F ∗ by utilizing θ ∗ , which is a model parameter inferred from training with the transformed dataset, ˜ S = F -1 ( M ⊙ F ∗ ) . Algorithm 1 provides the instruction of FreD.

Budget Allocation. When we can store n instances which is d -dimension vector for each class, the budget for dataset distillation is limited by n × d . In FreD, we utilize k < d dimensions for each instance. Therefore, we can accommodate | F | = ⌊ n ( d k ) ⌋ > n instances with the same budget. After the training of FreD, we acquire M ⊙ F ∗ , which actually shares the same dimension as the original image. However, we only count k non-zero elements on each f because storing 0 in d -k dimension is negligible by small bytes. Please refer to Appendix E.3 for more discussion on budget allocation.

3.3 Theoretic Analysis of FreD

This section provides theoretical justification for the dimension selection of FreD by EVR, η . For validation, we assume that F is linearly bijective.

Proposition 1. Let domain A and B be connected by a linear bijective function, W . The sum of η over a subset of dimensions in domain A for a dataset X is equal to the sum of η for the dataset transformed to domain B using only the corresponding subset of dimensions.

Proposition 1 claims that if two different domains are linearly bijective, the sum of EVR that utilizes only specific dimensions remains the same even when transformed into a different domain. In other words, if the sum of EVR of specific dimensions in the frequency domain is high, this value can be maintained when transforming a dataset only with those dimensions into the other domain.

Corollary 1. Assume that two distinct domains, B and C , are linearly bijective with domain A by W B and W C . let X be a dataset in domain A , and X B and X C be the datasets transformed to domains B and C , respectively. Let V ∗ B,k and V ∗ C,k be the set of k dimension indexes that maximize η in each domain. Let η ∗ B,k and η ∗ C,k be the corresponding sum of η for each domain. If η ∗ B,k ≥ η ∗ C,k , then the sum of η for W V ∗ B,k X B is greater than that of W V ∗ C,k X C .

Corollary 1 claims that for multiple domains that have a bijective relationship with a specific domain, if a certain domain can obtain the high value of sum of EVR with the same number of dimensions, then when inverse transformed, it can improve the explanation in original domain. Based on the information concentration of the frequency domain, FreD can be regarded as a method that can better represent the distribution of the original dataset, D , based on S .





























































(a) Comparison of domain (b) Efficiency of EVR selection Figure 5: Empirical evidence for EVR-based selection.

PCA, which sets the principal components of the given dataset as new axes, can ideally preserve η .

DD -1 F D D F

Binary Mask

Baseline (TM)

Ours (FreD)

(a) Class: Airplane

Binary Mask

Baseline (TM)

Ours (FreD)

(b) Class: Automobile

Binary Mask

Baseline (TM)

Ours (FreD)

(c) Class: Horse

Figure 6: Visualization of the binary mask, condensed image from TM, and the transformed images of FreD on CIFAR-10 with IPC=1 (#Params=30.72k). Binary masks and trained synthetic images are all the same size. The size of mask and baseline image were enlarged for a better layout.

However, PCA, being a data-driven transform function, cannot be practically utilized as a method for dataset distillation. Please refer to Appendix E.1 for supporting evidence of this claim.

Figure 5a shows the sum of η in descending order in both the spatial and frequency domains. First, we observe that as the number of selected dimensions in the frequency domain increases, frequency domain is much faster in converging to the total variance than the spatial domain. This result shows that the EVR-based dimension selection in the frequency domain is more effective than the selection in the spatial domain. In terms of training, Figure 5b also shows that the performance of a dataset constructed with a very small number of frequency dimensions converges rapidly to the performance of a dataset with the entire frequency dimension.

4 Experiments

4.1 Experiment Setting

We evaluate the efficacy of FreD on various benchmark datasets, i.e. SVHN [13], CIFAR-10, CIFAR100 [8] and ImageNet-Subset [6, 3, 4]. Please refer to Appendix D for additional experimental results on other datasets. We compared FreD with both methods: 1) a method to model S as an input-sized variable; and 2) a method, which parameterizes S , differently. As the learning with input-sized S , we chose baselines as DD [20], DSA [24], DM [25], CAFE+DSA [19], TM [3], KIP [14] and FRePo [27]. We also selected IDC [7] and HaBa [10] as baselines, which is categorized as the parameterization on S . we also compare FreD with core-set selection methods, such as random selection and herding [22]. We use trajectory matching objective (TM) [3] for L DD as a default although FreD can use any dataset distillation loss. We evaluate each method by training 5 randomly initialized networks from scratch on optimized S . Please refer to Appendix C for a detailed explanation of datasets and experiment settings.

4.2 Experimental Results

Performance Comparison. Table 1 presents the test accuracies of the neural network, which is trained on S inferred from each method. FreD achieves the best performances in all experimental settings. Especially, when the limited budget is extreme, i.e. IPC=1 (#Params=30.72k); FreD shows significant improvements compared to the second-best performer: 12.4%p in SVHN and 10.6%p in CIFAR-10. This result demonstrates that using the frequency domain, where information is concentrated in specific dimensions, has a positive effect on efficiency and performance improvement, especially in situations where the budget is very small. Please refer to Appendix D for additional experimental results on other datasets.

Qualitative Analysis. Figure 6 visualizes the synthetic dataset by FreD on CIFAR-10 with IPC=1 (#Params=30.72k). In this setting, we utilize 64 frequency dimensions per channel, which enables the construction of 16 images per class under the same budget. The results show that each class contains diverse data instances. Furthermore, despite of huge reduction in dimensions i.e. 64 / 1024 = 6 . 25% , each image contains class-discriminative features. We also provide the corresponding binary masks, which are constructed by EVR value. As a result, the low-frequency dimensions in the frequency domain were predominantly selected. It supports that the majority of frequency components for image construction are concentrated in the low-frequency region [1, 21]. Furthermore, it should be noted that our EVR-based mask construction does not enforce keeping the low-frequency components, what EVR only enforces is keeping the components with a higher explanation ratio on the image feature. Therefore, the constructed binary masks are slightly different for each class. Please refer to Appendix D.10 for more visualization.

Compatibility of Parameterization. The parameterization method in dataset distillation should show consistent performance improvement across different distillation losses and test network architectures. Therefore, we conduct experiments by varying the dataset distillation loss and test network architectures. In the case of HaBa, conducting an experiment at IPC=1 is structurally impossible due to the existence of a hallucination network. Therefore, for a fair comparison, we basically follow HaBa's IPC setting such as IPC=2,11,51. In Table 2, FreD shows the highest performance improvement for all experimental combinations. Specifically, FreD achieves a substantial performance gap to the second-best performer up to 10.8%p in training architecture and 10.6%p in cross-architecture generalization. Furthermore, in the high-dimensional dataset cases, Table 3 verifies that FreD consistently outperforms other parameterization methods. These results demonstrate that the frequency domain exhibits high compatibility and consistent performance improvement, regardless of its association with dataset distillation objective and test network architecture.



































Figure 7: Test accuracies (%) on CIFAR-10-C.

3D Point Cloud Dataset. As the spatial dimension of the data increases, the required dimension budget for each instance also grows exponentially. To validate the efficay of FreD on data with dimensions higher than 2D, we assess FreD on 3D point cloud data, 3D MNIST. 4 Table 4 shows the test accuracies on the 3D MNIST dataset. FreD consistently achieves significant performance improvement over the baseline methods. This confirms the effectiveness of FreD in 2D image domain as well as 3D point cloud domain.

Robustness against Corruption. Toward exploring the application ability of dataset distillation, we shed light on the robustness against the corruption of a trained synthetic dataset. We utilize the following test datasets: CIFAR-10.1 and CIFAR-10-C for CIFAR-10, ImageNet-Subset-C for ImagNet-Subset. For CIFAR-10.1 and CIFAR-10-C experiments, we utilize the off-the-shelf synthetic datasets which are released by the authors of each paper. We report the average test accuracies across 15 types of corruption and 5 severity levels for CIFAR-10-C and ImageNet-Subset-C.

Figure 7 and Table 5 show the results of robustness on CIFAR-10-C and ImageNet-Subset-C, respectively. From both results, FreD shows the best performance over the whole setting which demonstrates the superior robustness against corruption. We want to note that IDC performs worse than the baseline in many ImageNet-Subset-C experiments (see Table 5) although it shows performance improvement on the ImageNet-Subset (see Table 3). On the other hand, FreD consistently shows significant performance improvement regardless of whether the test dataset is corrupted. It suggests that the frequency domain-based parameterization method shows higher domain generalization ability than the spatial domain-based parameterization method. Please refer to Appendix D.5 for the results of CIFAR-10.1 and detailed results based on corruption types of CIFAR-10-C.

To explain the rationale, corruptions that diminish the predictive ability of a machine learning model often occur at the high-frequency components. Adversarial attacks and texture-based corruptions are representative examples [11, 23]. Unlike FreD, which can selectively store information about an image's frequency distribution, transforms such as factorization or upsampling are well-known for not preserving frequency-based information well. Consequently, previous methods are likely to suffer a decline in predictive ability on datasets that retain class information while adding adversarial noise. In contrast, FreD demonstrates relatively good robustness against distribution shifts by successfully storing the core frequency components that significantly influence class recognition, regardless of the perturbations applied to individual data instances.

Collaboration with Other Parameterization. The ex-

isting method either performs resolution resizing in the spatial domain or uses a neural network to change the dimension requirement of the spatial domain. On the other hand, FreD optimizes the coefficient of the frequency domain dimension and transforms it into the spatial domain through the inverse frequency transform. Therefore, FreD can be applied orthogonally to the existing spatial domainbased parameterization methods. Table 6 shows the performance of different parameterizations applied to HaBa. From the results, we observed that FreD further enhances

the performance of HaBa. Furthermore, it is noteworthy that the performance of HaBa integrated with FreD is higher than the combination of IDC and HaBa. These results imply that FreD can be well-integrated with spatial domain-based parameterization methods.

4.3 Ablation Studies

Effectiveness of Binary Mask M . We conducted a comparison experiment to validate the explained variance ratio as a criterion for the selection of frequency dimensions. We selected the baselines for the ablation study as follows: Low-pass, Bandstop, High-pass, Random, and the magnitude of the amplitude in the frequency domain. We fixed the total budget and made k the same. Figure 8a illustrates the ablation study on different variations of criterion for constructing M . We skip the high-pass mask because of its low performance:

































(a) Binary mask

M





























(b) Frequency transform

F

14.32% in IPC=1 (#Params=30.72k) and 17.11% in IPC=10 (#Params=307.2k). While Low-pass and Amplitude-based dimension selection also improves the performance of the baseline, EVR-based dimension selection consistently achieves the best performance.

Effectiveness of Frequency Transform F . We also conducted an ablation study on the frequency transform. Note that the FreD does not impose any constraints on the utilization of frequency transform. Therefore, we compared the performance of FreD when applying widely used frequency transforms such as the Discrete Cosine Transform (DCT), Discrete Fourier Transform (DFT), and Discrete Wavelet Transform (DWT). For DWT, we utilize the Haar wavelet function and low-pass filter instead of an EVR mask. As shown in Figure 8b, we observe a significant performance improvement regardless of the frequency transform. Especially, DCT shows the highest performance improvement than other frequency transforms. Please refer to Appendix D.8 for additional experiments and detailed analysis of the ablation study on frequency transform.

Budget Allocation. Dataset distillation aims to include as much information from the original dataset as possible on a limited budget. FreD can increase the number of data | F | by controlling the dimension budget per instance k , and FreD stores the frequency coefficients selected by EVR as k -dimensional vector. For example, with a small value of k , more data can be stored i.e. large | F | . This is a way to increase the quantity of instances while decreasing the quality of variance and reconstruction. By utilizing this flexible trade-off, we can pick the balanced point between quantity and quality to further increase the efficiency of our limited budget. Figure 9 shows the performance of selecting the dimension budget per channel under different budget situations. Note that, for smaller budgets i.e. IPC=1 (#Params=30.72k), increasing | F | performs better. For larger budget cases, such as IPC=50 (#Params=1536k), allocating more dimensions to each instance performs better i.e. large k . This result shows that there is a trade-





×







×







×







×























off between the quantity and the quality of data instances depending on the budget size.

5 Conclusion

This paper proposes a new parameterization methodology, FreD, that utilizes the augmented frequency domain. FreD selectively utilizes a set of dimensions with a high variance ratio in the frequency domain, and FreD only optimizes the frequency representations of the corresponding dimensions in the junction with the frequency transform. Based on the various experiments conducted on benchmark datasets, the results demonstrate the efficacy of utilizing the frequency domain in dataset distillation. Please refer to Appendix G for the limitation of the frequency domain-based dataset distillation.

References

Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions on Computers , 100(1):90-93, 1974.

Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards ai. Large-scale kernel machines , 34(5):1-41, 2007.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4750-4759, 2022.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3739-3748, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jeremy Howard. A smaller subset of 10 easily classified classes from imagenet and a little more french. URL https://github. com/fastai/imagenette , 2019.

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning , pages 11102-11118. PMLR, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. arXiv preprint arXiv:2210.16774 , 2022.

Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency domain model augmentation for adversarial attack. In European Conference on Computer Vision , pages 549-566. Springer, 2022.

Michael Mathieu, Mikael Henaff, and Yann LeCun. Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.5851 , 2013.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems , 34:5186-5198, 2021.

Harry Pratt, Bryan Williams, Frans Coenen, and Yalin Zheng. Fcnn: Fourier convolutional neural networks. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part I 17 , pages 786-798. Springer, 2017.

Jorge Rebaza. A first course in applied mathematics . John Wiley & Sons, 2021.

Noveen Sachdeva and Julian McAuley. Data distillation: A survey. arXiv preprint arXiv:2301.04272 , 2023.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12196-12205, 2022.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

Ye Wang, Miikka Vilermo, and Leonid Yaroslavsky. Energy compaction property of the mdct in comparison with other transforms. In Audio Engineering Society Convention 109 . Audio Engineering Society, 2000.

Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 1121-1128, 2009.

Chenglin Yang, Adam Kortylewski, Cihang Xie, Yinzhi Cao, and Alan Yuille. Patchattack: A black-box texture-based attack with reinforcement learning. In European Conference on Computer Vision , pages 681-698. Springer, 2020.

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning , pages 12674-12685. PMLR, 2021.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6514-6523, 2023.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929 , 2020.

Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00719 , 2022.

Peer reviews:
comments: SUMMARY:
This paper proposes a frequency-domain dataset distillation approach that can help to select the most important information in images to help distillation. To do this, the approach starts with a DCT to the input $x$ and then performs loss matching using existing dataset distillation approaches. Experiments demonstrate its advantages over state-of-the-art approaches.

STRENGTHS:
1. The idea of using frequency domain information is novel and helpful.
2. The experiments are good and solid.
3. The visualization helps to understand the algorithm and results better.

WEAKNESSES:
1. Experiments are solid, but not enough. I think that dataset distillation is useful for large volume datasets. But experiments do not contain any large datasets. So, the experiments are not useful to help interprete the usefulness of the algorithm.
2. There are no transfer experiments. Since dataset distillation is used to store a proxy of large datasets, and large datasets are often act as the source domain for transfer learning. Therefore, authors should conduct experiments on transfer learning datasets to show the advantage of the proposed approach. Only doing experiments on CIFAR-level datasets are not enough since to be honest, CIFAR is small and does not need to distill.
3. Any complexity analysis? How efficient is the algorithm?

QUESTIONS:
See weakness

LIMITATIONS:
See weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel way (code named FreD) to perform parameterization on dataset distillation tasks. Different from other methods which all operate on the spatial domain, this paper proposes to convert the spatial domain into frequency domain which are more concentrated and then perform distillation on these concentrated frequency components. \n\nEmpirical evaluations show that FreD is able to achieve SOTA compared to previous methods.\n\n============\n\nI have read the author's responses. Part of my concerns are addressed such as the questions regarding IPC while some others remain debatable. Overall, I would still keep my original score.\n\n============\n\nSTRENGTHS:\n## originality\n- the paper proposes an interesting new way to perform dataset distillation which has not be explored previously to the best of my knowledge\n\n## quality\n- the paper shows the effectiveness of the proposed methods through an adequate amount of experiments \n- the method is also theoretically proved through proposition 1 and corollary 1.\n\n## clarity\n- the paper is well written and easy to follow\n- the paper categorizes baseline methods clearly which makes the evaluation results easy to follow\n- the increment of decoded instances are also clearly indicated in table 1\n- distilled results are visualized and showed in the paper\n\n## significance\nTo the best of my knowledge, previous parameterization methods mostly perform on the spatial domain. This paper is the first to propose converting spatial domain to frequency domain which concentrates the information better. It could potentially open up a new directions for dataset distillation/condensation methods.\n\nWEAKNESSES:\n- The IPCs used in table 1 are 1/10/50, but 2/11/51 in table 2. The motivation for doing so is not clear.\n- In Figure 2, does the image used come from the evaluation datasets? If not, it's confusing to use an image that is not related to the  evaluation datasets in this paper.\n- The evaluation datasets are all small datasets such as 32 * 32 , in the appendix, only IPC 1 for TinyImageNet is reported. What's the reason for the missing numbers on IPC 10/50? Is it scalability issues or performances issues or something else?\n- In [1], IPC 1 and 10 are used to evaluate the performance on ImageNet subsets, but IPC 2 is used in Fred, what's the specific reason to choose a different IPC than previous methods? Can we see some results on IPC 1 and 10 as well because it will be good to see how FreD performs on higher resolutions.\n\n[1] Dataset Distillation by Matching Training Trajectories\n\nQUESTIONS:\nSee my comments on weakness. I am willing to raise the score if the above questions are answered so that I can get a better idea of how FreD works beyond what's currently shown in the main paper.\n\nLIMITATIONS:\n- the evaluation in the main paper is only done on datasets with low resolution, thus the scalability is hard to verify\n- same with a lot of previous parameterization methods, FreD also seem to work best with smaller IPCs  such as IPC 1(where generating more images can boost the performance a lot). This can be seen on CIFAR-100 IPC 50 that even with 2.56 times more images, the performance is close to methods without parameterization.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work introduces Frequency-based Dataset Distillation as a new means of parameterizing the distilled dataset that requires a much smaller memory footprint. By learning the distilled dataset in the frequency space (defined by the discrete cosine transform by default), a binary mask based on the class-wise EVR can be used to select only certain frequency components with minimal performance tradeoff. The memory saved by masking out these components can be allocated towards distilling more total samples.

STRENGTHS:
This work is well written and includes an exhaustive suite of experiments showing the effectiveness of the method.

Nearly all questions I had while reading were addressed later on in the paper.

WEAKNESSES:
It is not clear which dataset distillation loss is used in Figure 2. Likewise, it is unclear which distillation loss is used by FreD in Table 1 (although I think it is TM based on info in the appendix).

I would recommend removing IPC from the tables all-together and including the number of parameters instead. For example, instead of IPC = 10 for CIFAR-10, you could have #Params = 30,720 (10x3x32x32). IPC can be misleading when comparing to re-parameterization methods since the IPC simply isn't true anymore.

Would it be possible to see any visualizations for 3D MNIST? I searched through the appendix but only found visualizations for the 2D datasets. I've been very curious to see dataset distillation done on 3D data.

QUESTIONS:
I really like the paper overall and would be happy to raise my score if the authors address the few weaknesses described above.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new parameterization method for dataset distillation, called FreD, that utilizes the frequency domain to distill a compact, synthetic dataset from a large-sized original one. This approach differs from conventional methods that focus on the spatial domain. It employs frequency-based transforms to optimize the frequency representations of each data instance. By leveraging the concentration of spatial domain information on specific frequency components, FreD is able to select a subset of frequency dimensions for optimization. The paper provides both theoretical and empirical evidence of FreD's efficiency compared to conventional parameterization methods. In addition, it highlights the compatibility of FreD with existing distillation methods and its consistent improvement in performance over different benchmark datasets. This work addresses the challenges posed by big data and it improves upon the current dataset distillation methods by offering an optimized and memory-efficient solution.\n\nSTRENGTHS:\n1. The authors have done a great job of describing their proposed method, FreD. They offer explicit step-by-step explanations of how each element of the method operates, from the synthetic frequency memory to the binary mask memory and the inverse frequency transform. It helps the reader can follow along with their reasoning.\n2. This work introduces an innovative parameterization based dataset distillation method, and is the first work of studying from a frequency perspective. It lays a foundation for further research and discussion in this frequency-oriented parameterization methods.\n\nWEAKNESSES:\n1. While this work demonstrates the performance increased relative to several previous methods, it did not compare with another parameterization method [1], where the performance seems to be better than this work. It would be good to include this comparison and discuss the potential limitations of the frequency-based parameterization approach.\n2. it's unclear how the choice of transform (Discrete Cosine Transform (DCT), Discrete Fourier Transform (DFT), or Discrete Wavelet Transform (DWT)) might influence the performance or effectiveness of their approach. This aspect is not thoroughly investigated or discussed.\n3. The Spectral Bias could impact the network's ability to generalize to new data, there is little discussion on this which could hurt the effectiveness of the frequency domain-based approach presented in the paper.\n\n[1] Deng, Zhiwei, and Olga Russakovsky. \"Remember the past: Distilling datasets into addressable memories for neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 34391-34404.\n\nQUESTIONS:\n1. In section 3.3, can the assumption that the function F is linearly bijective limit the applicability of the method? Is there any situation where this assumption might not hold and how would that affect the results?\n2. How does the approach handle high-dimensional datasets? Is there a limit on the dimensionality that the method can effectively handle?\n\nLIMITATIONS:\nThe authors did not include the limitation section, it will be good to have a discussion on cross architecture generalization ability of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}