Distilling Dataset into Neural Field
Authors: Donghyeok Shin
HeeSun Bae
Gyuwon Sim
Wanmo Kang
Il-chul Moon
Venue: ICLR 2025 Poster (2025)
Keywords: Dataset distillation
Dataset condensation
Neural field
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: This paper proposes an utilization framework of neural field for dataset distillation.

Abstract: Utilizing a large-scale dataset is essential for training high-performance deep learning models, but it also comes with substantial computation and storage costs. To overcome these challenges, dataset distillation has emerged as a promising solution by compressing the large-scale dataset into a smaller synthetic dataset that retains the essential information needed for training. This paper proposes a novel parameterization framework for dataset distillation, coined Distilling Dataset into Neural Field (DDiF), which leverages the neural field to store the necessary information of the large-scale dataset. Due to the unique nature of the neural field, which takes coordinates as input and output quantity, DDiF effectively preserves the information and easily generates various shapes of data. We theoretically confirm that DDiF exhibits greater expressiveness than some previous literature when the utilized budget for a single synthetic instance is the same. Through extensive experiments, we demonstrate that DDiF achieves superior performance on several benchmark datasets, extending beyond the image domain to include video, audio, and 3D voxel. We release the code at \url{https://github.com/aailab-kaist/DDiF}.

Published as a conference paper at ICLR 2025
DISTILLING DATASET INTO NEURAL FIELD
Donghyeok Shin', HeeSun Bae', Gyuwon Sim', Wanmo Kang' & II-Chul Moon Koorea Advanced Institute of Science and Technology (KAIST), "summary.ai {t (tlsehdgurO, cat2507, gkwlaks 686, wanmo. kang,
kr.
ABSTRACT
Utilizing large-scale dataset is essential for training righ-performance deep learning models, but it also comes with substantial computation and storage costs. To overcome these challenges, dataset distillation has emerged solution by compressing the large-scale dataset into a smaller synthetic dataset that retains the essential information needed for training. This paper proposes a novel parameterization framework for dataset distillation, coined Distilling Dataset into Neural Field (DDiF), which leverages the neural field to store the necessary information of the large-scale dataset. Due to the unique nature of the neural field, which takes coordinates as input and output quantity, DDiF effec- tively preserves the information and easily generates various shapes of data. We theoretically confirm that DDiF exhibits greater expressiveness than some pre- vious literature when the utilized budget for single synthetic instance is the same. Through extensive experiments, we demonstrate that DDiF achieves su- perior performance several benchmark datasets, extending beyond the im- age domain to include video, audio, and 3D voxel. We release the code at attps://githun com aailallla wwwillab-aaistttie kaist/DDiF
promising
INTRODUCTION
High performances from recent deep learning models are mainly driven by scaling laws (Bengio al. 2007 Kaplan al. 2020),, which heavily rely on a large-scale dataset. However, utilizing the large-scale dataset incurs significant computation and storage costs. Dataset distillation has been proposed as a potential solution to address these challenges (Wang et al. 2018). The goal of dataset distillation is to synthesize a small-scale synthetic dataset that encapsulates the essential information needed to train a deep learning model on a large-scale dataset. Naturally, one the research direc- tions in dataset distillation is defining the essential information and developing efficient methods to learn it (Zhao et al. 2020 Zhao Bilen 2023 Cazenavette et al. 2022).
In parallel, another crucial research direction is parameterizing the small-scale synthetic dataset un- der the limited storage budget. The naive parameterization method constructs the synthetic instance in the same structure as the original instance. Due to the limitations of scalability and redundancy in this approach, various parameterization methods have been proposed to improve efficiency within the limited storage budget. Generally speaking, parameterization methods commonly employ low- dimensional codes and decoding functions that transform a code in reduced dimensions into a data instance of the original input space. Conceptually, the decoding functions can be categorized into 1) static decoding (Kim et al., 2022; Shin et al. (022)): 2) parameterized decoding (Deng & Rus- sakovsky 2022 Sachheeva al.[|202 Lee
2022 Liu et al. 2022) Wei et al. 2024); and 3) deep generative prior (Cazenavette et al. 20033] Su et al.) 2024a b. Zhong et al. 2024). Although these methods have shown promising results, they still exhibit limitations coding efficiency, expressive- ness, and applicability to diverse data structures. Additionally, there has been limited exploration of the theoretical foundations underlying these methods.
This paper introduces new parameterization framework for dataset distillation that stores infor- mation into synthetic neural fields under the limited storage budget, coined Distilling Dataset into Neural Field (DDiF). A field is function that takes a coordinate sponding quantity, and a neural field parameterizes the field using a neural network. DDiF employs the synthetic neural field as container of distilled information that utilizes small budget, and it decodes a synthetic instance by inputting a set of coordinates corresponding to the original instance. We emphasize that the neural field has a structural difference compared to conventional decoding
input and returns a corre-

---

Published as a conference paper at ICLR 2025
functions in parameterization methods, which map low-dimensional space to instance- space. Thanks to the continuous and coordinate-oriented nature of the neural field, DDiF effectively encodes information from -igh-dimessional data, which is crucial challenge in dataset distillation. Furthermore, due to the inherent flexibility of neural networks (Lu et al. 2017 Raghu et al. 2017), the synthetic instance decoded by DDIF exhibits high expressiveness. In addition, DDIF capable of encoding grid-based data from various modalities and decoding data instance of resolution that was not encountered during the distillation process. We present a theoretical analysis of parameter- ization methods by investigating the expressiveness of the decoded synthetic instance through the feasible space view. Based on this theoretical understanding, we demonstrate that DDiF possesses a larger feasible space compared to some previous literature under the same utilized budget for a sin- gle synthetic instance. Across various evaluation scenarios, DDiF consistently exhibits performance improvements, generalization, robustness, and adaptability on diverse modality datasets.
In summary, our contributions are as follows:
We propose new parameterization framework for dataset distillation, Distilling Dataset into Neural Field (DDiF), which employs a neural field to parameterize synthetic instance.
We theoretically analyze the expressiveness of DDiF by investigating the feasible space covered by its decoded synthetic instances. Furthermore, we confirm that DDiF exhibits high expressiveness by comparing it with previous methods under the same utilized budget for single synthetic instance.
Through extensive experiments, we demonstrate that DDiF consistently achieves high per- formance on diverse grid-based data, including image, video, audio, and 3D voxel. More- over, we present new experimental design, cross-resolution generalization, capable of measuring generalization performance on resolutions not encountered during training.
PRELIMINARY
2.1 PROBLEM FORMULATION
This paper focuses on dataset distillation for classification tasks. We define given large dataset that needs to be distilled as 'T (X,, YT) (... where Xy {ax [ar) denotes set of
[T]
D-dimensional input data I; E X C RD, and Y7 = {3))=1 denotes a set of corresponding labels
among C-classes y/ y 11, 1,, C}. We assume that each data pair (2t,, ys) is drawn i.i.d from the distribution P. Let classifier fa X + y be a neural network parameterized by 0 E O. We also define a loss function y x y J.
The main goal of dataset distillation is to synthesize a small dataset such that a model trained on this synthetic dataset can generalize well large dataset. Formally, given synthetic dataset 1S)
= (Xs, Ys) = {(t,, ū,)],,- where Xs == (2,) Ys = {ūا} and (S] objective of dataset distillation formulated as follows:
< <T.. the
min E(zg)>~ [e(fes (x), w)] subject to Os arg min 5
e(fo(z)).
(I)
(2,
Nonetheless, the optimization of Eq. | both computationally intensive and lacks scalability, as entails bi-level optimization problem for both 0 and S (Zhao et al., 2020 Borsos et al. 2020). To overcome these issues, several studies have suggested the surrogate objectives to effectively capture essential information needed for training the neural network on T, such matching gradient (Zhao) et al. 2020), distribution (Zhao & Bilen] 2023), and trajectory (Cazenavette et al. 2022). For the sake of brevity, denote these objectives L(T, throughout this paper.
2.2 PARAMETERIZATION OF DATASET DISTILLATION
Input-sized parameterization, which sets synthetic instance in the same format as a real instance, suffers from scalability as the dimension of given instance increases. Also, input-sized parame- terization does not utilize the storage budget efficiently because it contains redundant or irrelevant information (Lei & Tao 2023 Yu et 2023 Sachdeva & McAuley 2023).
Addressing these concerns, several studies have proposed parameterization methods to enhance the efficiency of synthetic dataset. In general, parameterization methods employ 1) codes Z |2
2

---

Published as a conference paper at ICLR 2025
where 2j E R and 2) decoding function 9o RR'' + R to construct the synthetic dataset Under this framework, a decoded synthetic instance is represented by a combination of code and decoding function i.c. ž g0(2)), Also, set of decoded synthetic instances become Xs {ga(=)|= ε 2). Typically, parameterization methods use Z and/or Φ as targets for optimization and storage. Therefore, the storage budget for the parameterization is calculated based on the total number of parameters comprising Z and/or Id. and it is adjusted to ensure the budget constraint.
Based on the structure of the decoding function, they can be broadly categorized into 1) static de- coding, 2) parameterized decoding, and 3) deep generative prior. Static decoding employs a non- parameterized decoding function g. such as resizing (Kim et al., 2022) and frequency transform (Shin et al.) 2024). These methods are fast, easy to apply. and do not require budget for the decod- ing function. However, since this decoding function fixed, the structure of code z becomes limited without the ability to adaptively transform g. Also, using a static decoding function inevitably re- duces expressiveness from a generalization perspective, leading to information loss.
Parameterized decoding utilizes linear combination with learnable coefficients (Deng & Rus- sakovsky, 2022 Sachdeva et al., 2023), decoder (Lee et al., [202]] ZZhng et al. (023), autoencoder
(Liu et al. 2022, Duan et al., 2023). or transformer structure (Wei et al.] (024) as the decoding function Jg. Although flexible decoding functions are employed, the optimized parameters of the decoding function also be stored within the limited budget, necessitating the use of a simple struc- ture (i.e., linear combination) or a lightweight neural network. It results in limited flexibility and presents challenges when extending to complex data types, such as video, 3D, and medical images.
Deep generative prior leverages a pretrained deep generative model without additional training, fo- cusing only on optimizing the latent vector (Cazenavette et al.. 2023 Su et al.. 2024a b) [Zhong eet al. 2024). This framework encourages better generalization to unseen architecture and scale to the nigh-dimensional dataset. However, it assumes easy access which might restrict the range of applications. Also, since the deep generative model contains a large number of parameters, the decoding process and backward update process take a long time.
well-trained generative model,
2.3 NEURAL FIELD REPRESENTATION
In physics and mathematics, field F is a function that takes a coordinate in space and outputs quantity (Xie et al. (022). If we apply the concept of the field to data modeling, a single grid-based data can be regarded locations to pixel intensity. Similarly, a video datatype is a function that additionally takes time as input, and 3D datatype is function that outputs occupancy value on the 3D coordinate system.
field. For example, an RGB image is a function that maps from pixel
According to the universal approximation theorem (Cybenko 1989), any field can be parameterized by neural network, which is referred to neural field F It implies that grid-based data can be expressed as a neural network. Let C == (G))iez be a set of coordinates of grid-based data and Q == fgz be set of corresponding quantities. To encode single grid-based data using a neural field Fg. we minimize a distortion measure, such as squared error, over all given coordinates as:
min Σ [lF((c)) 9:1l2
IET
Recently, the neural field has been adopted to many applications, such as representation learning (Park et al. 2019: Mildenhall et al., 2021), generative modeling (Skorokhodov et al. [021] Dupont
et al. 2021 medical imaging (Shen et al. 2022 Zang et al. 2021). and robotics (Li et al. 2022).
METHODOLOGY
This section proposes a new parameterization framework for dataset distillation that stores infor- mation of real dataset in synthetic neural fields under a limited storage budget, coined Distilling Dataset into Neural Field (DDiF). The core idea of this paper is to store the distilled information in the synthetic function. Although there are several candidates for the form of synthetic function, we primarily focus on (neural) field. Figure Illustrates the overview of DDiF. In the following, we begin by explaining the reasons for choosing neural field as the form of synthetic function. Then, we introduce our framework, DDiF, which parameterizes a synthetic instance using a neural field. Finally, we provide a theoretical analysis for a better understanding of parameterization and DDiF.
Atthouhh some studies (Deng& & Russakovsky 2022 Moser et al. 2024) use both ÿ and as inputs for 96. we expressed as (2) for the sake of uniformity.
3

---

Published as a conference paper at ICLR 2025
Forward Path
Back Propagation
S
T
Original (128 128) lanore
Downstream Task classification)
Paramettriization
CTT
Image 164 x 64)
Image
Neural Field
A.TS;
(rupwb))
Fi
Image (256 256)
Video
Video
(...2)
8/T,8,
Coordinate set
Ry
Quantity
Dataset
30
Coordinates
Input
Image 512)
Distillation Stage
Evaluation Stage
Figure/ 1: Overview of DDiF. Each decoded synthetic instance is constructed by the output of each synthetic neural field F& by inputting coordinate set (left). DDiF optimizes only the parameters vh, as coordinate set C does not require optimization or storage. Also, DDiF is capable of encoding grid-based data from various modalities. In the evaluation stage (right), DDiF can decode the data with sizes that were not encountered during the distillation stage by adjusting the input coordinates.
3.1 WHY NEURAL FIELD DATASET DISTILLATION?
Although the neural field is widely adopted, no research has yet been conducted on integrating it into dataset distillation. Herein, provide several properties, which are beneficial for dataset distillation due to its unique structural characteristic.
Coding Efficiency. The neural field efficiently encodes information of high-dimensional grid-based data. Distilling the high-dimensional dataset remains crucial challenge in expanding the applicability of dataset distillation (Lei &
9q. RR
R
D D=CxH
D -
Tao 2023 Input-sized parameterization typically scales poorly with resolution due to the curse of discretization (Mescheder 2020). While employing the decoding func- out-
(a) Previous decoding function Fgg: D
D could improve scalability, put ultimately depends on the data dimension D. It implies that as the data dimension D) increases, a more complex de- coding function Je is required, which becomes problematic given a limited storage budget. In contrast, the neural field stores information independently of the data dimension D. Its input and output dimensions are determined by the di- mensions of the coordinate space and the quantity space, respectively, as depicted in Figure 2 For high-dimensional data, the neural field only requires larger set of input co- ordinates. Moreover, we emphasize that the neural field can represent complex grid-based data, such videos and 3D voxels, which are limited the image domain. Since such data are typi- cally high-ding sigh-dimensional, the high coding efficiency of the neural field is particularly advantageous for complex grid-bacc based dataset distillation. Therefore, the neural field provides unified and efficient representation of high-dimensional grid-based data.
tion 9o D
C U.
F V
Coordiratos
(b) Neural field
Figure 2: Illustration of structural dif- ference between conventional decod- ing function and neural field.
Resolution Adaptiveness. The neural field robust to diverse resolution. the real world, is often necessary to resize data depending on downstream tasks (Wang et al. (2020b) [Shorten & Khoshgoftaar 2019). Existing methods can only apply postprocessing on the optimized synthetic dataset, leading to insufficient or distorted information. Meanwhile, the neural field easily obtains various sizes of data by adjusting the set input coordinates due to the continuous nature. Further- more, since the neural field is continuous function, it provides more accurate values for unseen coordinates. Please refer to Section 4.1 for the empirical evidence this claim by introducing a new experiment design, coined cross-resolution generalization.
4

---

Published as a conference paper at ICLR 2025
3.2 DDIF: DISTILLING DATASET INTO NEURAL FIELD
In this section, we introduce a basic framework for integrating the neural field into dataset distilla- tion. Specifically, DDiF parameterizes a single synthetic instance ē as a single neural field Føy. DDiF consists of two main components: 1) Coondinate set C and 2) Synthetic neural fields Fv.
Coordinate Set C. To define the function, it is first necessary to define the input space of the function. Fundamentally, a decoded synthetic instance by the parameterization method is the same shape the real instance. Therefore, our synthetic function must be defined in the space where the information of the real instances is stored. Suppose that the given real instance x E XT is n-dimensional grid representation with resolution Nz. k 1, ..., n, and each element contains m values, i.e. m = 3 of RGB values. Formally, the real instance x is element in Rm*N, x...NN Then, the coordinate set C, where the values of are stored, is defined by a bounded set of lattice points:
{ (2 E (0,1, No). Ak: :0.:...... 1,....1
C c=
Note that there are several properties of coordinate set C. which become advantages in dataset dis- tillation. First, since every real instance x E Xy is defined on the same coordinate set C (the only difference is the value on each coordinate). we do not need to consider the coordinate individually. Also, C is easily obtained if only the shape of the decoded instance is defined, without any additional information. For instance, assume that we want to get N x N-shaped data instances. Then, C is a set of lattice points in [0, N] [0, N]. Due to these static and bounded characteristics. DDiF does not need to optimize or store the coordinate set C. Based on this property. DDiF is a decoder-onl parameterization framework that leverages flexible decoding function Jø without inferring codes Z, which structurally differs from previous methods.
Synthetic Neural Fields Fg. DDiF utilizes neural field Fg RD + Th to obtain the decoded synthetic instance è by inputting the coordinate set cP Specificcaly.. given coordinate set C, the de- coded synthetic instance by DDiF is म = (C) == (c)lec- In DDiF, since a decoded synthetic instance 전 and synthetic neural field F. have one-to-one correspondence, obtaining K decoded synthetic instances requires K synthetic neural fields. We denote the parameter set of synthetic
neural fields as V == { j=1 and the set of synthetic neural fields as Fg = =(F, | structure of the synthetic neural field Fø, we follow the tradition of the neural field (Mildenhall let al. 2021 Tancik et al. 2020), which utilizes simple L-layer neural network:
[j=1 For the
(c) = (1) ....o (0) (c) + 6(0),
(C - (WW") (
where ((O) E Rddxde-1, (()) E dd are weights and bias at layer L. g(1) denotes nonlinear activation function. Under this formulation, v becomes ( [Wr0,,00]] b00
To avoid the spectral bias (Rahaman et al. 2019: Xu et al. (2099) that limits the expressiveness of neural fields, we employ sine activation function (Sitzmann et al. 2020) by default, which is widely used in neural fields.
Learning Framework. Given coordinate set C and a parameter set of synthetic neural fields W set of decoded synthetic instances is represented by Xss (Fe, (C) Under the arbitrary dataset distillation loss L(T,S), the overall optimization of DDiF is formulated as follows:
min C(T where S = {(Fo, (C), (
(3)
DDiF has limitations in applying optimizable soft label (Sucholutsky & Schonlau, 2021 Cui
et al. 2023), but we utilize predefined one-hot labels to ensure consistency with previous parame- terization i.e. ăj = My YT. Please refer to Appendix C.3 for the compatibility with soft label. In practice, dataset distillation commonly utilizes randomly sampled real instance & E XT as the initialization of synthetic instance ă. In the same context, we conduct the warm-up training for synthetic neural fields Fy. Concretely, we train Fy using Eq. (2) with randomly selected / I/C samples for each class. Appendix [B.5 specifies training procedure and decoding process of DDiF.
We defined C for dataset distillation training, but the domain of F is the entire n-dimensional space R"
5

---

Published as a conference paper at ICLR 2025
Budget calculation. As Eq.
(3) demonstrates, the optimization target of DDiF is v { 191 E
In detail, each synthetic neural field Fe utilize + - dc(d)-+1) parameters. We emphasize that b does not depend on the data dimension D, so high resolution does not necessarily increase the budget of Fv.. When the storage budget is limited to B, we configure the structure of Fgz, such as the width di and the number of layers I,, to satisfy [ x b & B.
+ m(dr_1 +
3.3 THEORETICAL ANALYSIS
Even though several parameterization methods for dataset distillation are proposed, there has been little discussion regarding the theoretical understanding of their methods. In this section, we provide theoretical analysis of parameterization methods by investigating the expressiveness of the decoded synthetic instance. Next, we analyze the expressiveness of DDiF when employing the sine activation function. Lastly, we demonstrate that DDiF exhibits higher expressiveness than the previous work, FreD (Shin et al. 2024), under the same utilized budget for a single synthetic instance.
We conjecture that the expressiveness of the decoded synthetic instance corresponds to the coverage of its respective data space. Accordingly, the optimization feasibility of dataset distillation depends on the coverage or feasible space of the decoded synthetic instance. Based on this conceptual idea, Proposition 3.1 characterizes the relationship between the feasible space of decoded synthetic in- stances and the optimal value of the dataset distillation objective, when the number of decoded synthetic instances is the same. Herein, we assume only synthetic inputs Xs as the optimization variable, Consequently, dataset distillation loss L(T, ,S) is simply expressed as a function of Xs i.e. L(T, .S is represented by ((Xs).
Proposition 3.1. Consider two functions g1, where gi Z; + RD for 1,2. Also, consider
two matrix variables Z; == [i.1..., ZiM] where their columns 2ij E Z, for = 1,2 and j = M.....M. We denotes (Z,) = (Z(1,,,... .... (i(zM))] for = 1,2. Set G. = {olg: Z, + } for i = 1,2. If 91 (2)) S (2(22) for any 9i E In and 92 E G2. then min 2.z2 (Z)) 2 min 926. (())(Z2)).
Please refer to Appendix A.I for proof. Proposition 3.1 demonstrates that the optimal value of dataset distillation loss becomes lower as the feasible space of decoded synthetic instances becomes larger, when the number of decoded synthetic instances is fixed. The previous experimental finding supports Proposition 3.1 when Gj = G2: an increase in frequency dimension leads to improved
dataset distillation performance (Shin et al. 2024).
Now, we investigate the feasible space of DDiF. We consider a neural field F. R - with two hidden layers, each having width of dl and employing sine activation function. Extending the result of NNvello 2022) using trigonometric identities, the output space of DDiF, which corresponds to the feasible space of the decoded synthetic instance, is represented as sum of cosines:
d
Px(x)) = 11(2) + Σ kEZd t=1
A cos (WE. + (kh)) where sk, = Pkk
π 2
(4)
where Wi=- (k. ( Pkki = (k. ) + D2 (1)
ak = Ein AK, sin (ppx:)) and 88 =
ES Ae. cos (P.,,)). Also, AR.s == w: I1; Jk, (( where JR, denotes Bessel function of the first kind of order kz. Please refer to Appendix A.2 for derivation. According to Eq. (4) it
should be noted that the amplitudes Ar, frequencies Wke phases sp',, and shift ((0) are tunable as the combination of neural network parameters v = (w(), b00 L / in DDiF.
The feasible space of DDiF in Eq. (4) has a similar form of the feasible space of previous work, FreD (Shin al.) 2024), one of the static parameterization method in dataset distillation. FreD optimizes frequency coefficients, which are selected by the explained variance ratio. They basically use the inverse discrete cosine transform (IDCT) to decode synthetic instances from the frequency domain. Suppose that FreD utilizes IDCT with N equidistant locations on R. Also, when U c CN == [0, ...,... N - 1} is the index set of selected frequency dimension; the optimized frequency coefficients is denoted as r == (ra/u E U). Then, the feasible space of decoded synthetic instance by FreD is expressed in the form of function over Cx:
(x;; D) = Σ
The cos
N
+ TU
2
where E CN
(5)
NEW

---

Published as a conference paper at ICLR 2025
Now, we turn attention to compare the feasible spaces of DDiF and FreD. Theorem 3.2 provides the relationship between the feasible spaces of DDiF and FreD when the same budget is utilized.
Theorem 3.2. Consider the truncation of Eq. (4 over Ke
[k| I [} i.e. Folx)
(((2) EnEke Σi_1 Ag,i cos (wye + (2,,) Suppose that FreD and DDiF utilize the same number
of parameters, i.e., [2]] and the number of parameters in 1 are fixed given value B. If B 6
and < } exp [[3+B-2] - -1). then g(z; I) S Fu(a)) for any z E CN-
Please refer to Appendix A.3 for proof. Theorem 3.2 implies that the feasible space of decoded val- ues in DDiF is larger than that in Frel when the parameters used for single synthetic instance fixed. Consequently, according to Proposition [3.1] when the number of decoded synthetic instances equal, DDiF achieves lower optimal value for the dataset distillation loss than FreD.
To support Theorem 3.2 we investigate
the performance of the reconstruction task by varying the utilized budget. We spec- ulate that the low reconstruction error is due to the large feasible space. For the re- construction task, we utilize 10 randomly
Reconstruction Error
0.8
IDC
445
DDIF
Test Test accuracy
0.6
40
0.4
35
IDC
selected images per class on ImageNet- Subset (Cazenavette et al.) 2022; 2023)
0.2
FreD
DDiF
0.0
with 128 resolution. For the dataset distil- lation task, we employ TM (Cazenavette eet al., 2022) under image per class, and the same dataset with the reconstruc- tion task. As shown in Figure 3 DDiF achieves higher performance than FreD in both tasks, which indicates greater expres- siveness under the same utilized budget. Furthermore, we emphasize that DDiF maintains high performance even with a small budget, whereas FreD shows significant performance degradation as the allocated budget for each instance decreases. These experimental results provide further evidence that DDiF exhibits greater expres- siveness. In addition, although do not provide theoretical comparison with IDC (Kim et al.) 2022). Figure 3 presents empirical evidence demonstrating that DDiF exhibits greater expressive- ness than IDC. We believe that the proposed theoretical analysis will serve starting point for future theoretical comparisons of parameterization methods in the dataset distillation domain.
10"
10"
10"
10"
Utllized budget per Instance B
Utlized budget per instance
(a) Reconstruction task
(b) Dataset distillation
Figure 3:
Performance curve on (a) reconstruction task
and (b) dataset distillation under the same utilized bud- get. Pearson correlation coefficient of reconstruction error and test accuracy is -0.89.
EXPERIMENTS
This section presents various empirical results that validate the effectiveness of the proposed method, DDiF. We primarily focus on high-resolution image datasets, such ImageNet-Subset (Cazenavette et al., 2022; 2023) with resolutions of 128 and 256, since high-dimensional dataset distillation re-
mains challenging task. In addition, we conduct several experiments verify the applicability of DDiF on diverse modalities. We utilize MiniUCF for video (Wang et al.) 2024 Khurram, 2012). Mini Speech Commands for audio (Kim et al., 2022, Warden] 2018), and ModelNet (Wu et al.] 2015). ShapeNet (Chang al. 22155) for 3D. Specific details regarding the dataset, baselines, con- figuration of DDiF, and experimental settings are in Appendix B
4.1 PERFORMANCE COMPARISON
Main Results Table T shows the overall performance for ImageNer-Subset with resolution 128 and 256 under IPC=1. We utilize trajectory matching (TM) for 128 resolution and distribution matching (DM) for 256 resolution. DDiF achieves the best performances in all experimental set- tings. We highlight that the performance improvement from Vanilla to DDiF is significant, sup- porting the effectiveness of the proposed method (see the A row in Table []. In addition, DDIF exhibits a substantial performance margin compared to the second-best performer: up to 4.1%p at 128 resolution, and from 8.4%p to 19. 19.9%p at 256 resolution. Moreover, when a larger budget is available, DDiF shows consistent improvement and highly competitive performance with baseline, as shown in Table These results demonstrate that neural field-based parameterization significantly enhances the dataset distillation performance, particularly when the budget is very limited. Please refer Appendix C.1 for the additional experimental results such low-resolution datasets.
7

---

Published as a conference paper at ICLR 2025
Table 1: Test accuracies (%) on ImageNet-Subset" for parameterization methods under IPC-1, "IPC" denotes instances per class, which implies the budget constraint. Bold and Underline mean the best and second- best performance of each column, respectively. "f" indicates our implementation results. indicates no reported results. "A" represents the performance improvement over Vanilla, which denotes input-sized parameterization. The full table with standard deviations is in Appendix C.7
Reselation
128
128
256
256
Subset
Nette
Wool
Fruit
Yellow
Meow
Squaw
Nette
Wool
Fruit
Yellow
Meow
Squawk
Input-sized
Vanilla FRePo
51.4"
48.1
29.7"
29.7
28.87
47.51
33.31
41.0"
32.1
20.0
19.5
33.4
21.2
27.6
Static
IDC
61.4
34.5
38.0
56.5
39.5
50.2
53.71
30.2"
33.1"
52.21
34.67
47.0"
FrelD
66.8
38.3
43.7
63.2
43.2
57.0
54.21
31.2
32.57
49.17
34.07
43.17
HaBa
51.9
32.4
34.7
50.4
36.9
41.9
Parameterized
SPEED
66.9
38.0
43.4
62.6
43.6
60.9
57.7
Vanilla+RTP
69.6
38.8
45.2'
66.4
46.5
63.2'
LatenDD
56.1
30.7
363
NSD
68.6
35.2
39.8
61.0
45.2
52.9
DGM Price
H-GLaD
45.4
28.3
23.4
25.6
26.0
29.6
39.7
35.8
Function
DDiF
+20.6
+13.2
42.9
+19.4
+21.5
+14.1
+26.0
67.0
+35.7
+19.6
+23.7
43.2
+29.7
+23.6
44.8
+39.4
Entire dataset
87.4
67.0
63.9
84.4
66.7
87.5
92.5
80.1
70.2
90.5
72.2
93.2
Table Test accuracies (%) on ImageNet-Subset Table (128 x 128) under IPC=10.
Average test accuracies (%) on ImageNet-Subset (128 x 128) across AlexNet, VGG11, ResNet18, and ViT, under IPC=1.
Method
Nette
Wool
Fruit
Yellow
Meow
Squawk
TM
35.8
40.3
52.3
Method
Nette
Woof
Fruit
Yellow
Mcow
Squawk
FRePo
66.5
42.2
TM
22.0
14.8
17.1
22.3
16.2
25.5
IDC
70.8
39.8
46.4
68.7
47.9
65.4
IDC
27.9
19.5
23.9
28.0
19.8
29.9
FreD
72.0
41.3
47.0
69.2
48.6
67.3
FreD
36.2
23.7
23.6
312
37.4
HaBa
64.7
38.6
42.5
63.0
42.9
56.8
GLaD
30.4
17.1
21.1
19.6
28.2
SPEED
72.9
44.1
50,0
70.5
52.0
71.8
H-GLaD
30.8
17.4
20.1
28.8
DDIF
74.6
44.9
49.8
70.5
50.6
72.3
LD3M
32.0
19.9
21.4
22.1
30.4
Entire T
87.4
67.0
63.9
84.4
66.7
87.5
DDiF
59.3
34.1
39.3
51.1
33.8
54.0
Cruss-architecture Generalization. The network structure used for training with the distilled dataset may differ from the one used for distillation. Accordingly, parameterization methods should achieve consistent performance enhancements across various test network architectures. In this study, we employ AlexNet (Krizhevsky et al. 2012). VGGI1 (Simonyan & Zisserman 2014). ResNet18 (He et al. ) and ViT (Dosovitskiy) 2020)) while ConvNetDS utilized in training. Table 3 presents that DDiF consistently outperforms. Notably, DDiF achieves a remarkable per- formance gap over the second-best best performer from 10.4%p to 23.1%p. These results indicate that DDiF effectivlly encodes important task-relevant information regardless of the training network.
Universality to Distillation Loss. Another important evaluation criterion for parameteri- zation is whether it constantly improves the performance across various dataset distillation losses. We examine the performance of DDiF using gradient matching (DC) and distribution matching (DM) to evaluate its steady improve- ments. In Table 4, DDiF achieves the best per- formances in both distillation losses. These re- sults confirm the universality of DDiF to distil- lation loss, representing its wide adaptiveness.
Table 4: Test accuracies (%) on Image Net-Subset (128 x 128) across DC and DM under IPC=1.
3
Method
Nette
Woof
Fruit
Mcow
Squawk
Vanilla
34.2
22.5
21.0
22.0
32.0
IDC
45.4
25.5
26.8
25.3
34.6
FreD
49.1
26.1
30.0
28.7
39.7
DC
GLaD
35.4
22.3
20.7
22.6
33.8
H-GLaD
36.9
24.0
22.4
24.1
35.3
DDiF
61.2
35.2
37.8
39.1
54.3
Vanilla
30.4
20.7
20.4
20.1
26.6
IDC
48.3
27.0
29.9
30.5
38.8
FreD
56.2
31.0
33.4
33.3
42.7
DM
GLaD
32.2
21.2
21.8
22.3
27.6
Various Modality. Conventional parameteri- zation methods have primarily been developed within the image domain, and their applicabil- ity to other modalities has not been sufficiently explored. Since the neural field provides general- ized representation of diverse grid- based data structures, DDiF can be naturally applied to various modalities beyond the image domain. We conduct experiments in the video, audio, and 3D voxel do- mains to evaluate its efficacy. First, Figure 4 shows the test performances on the video domain with regard to the required budget for running each method. SDD (Wang et al. 2024), which focuses
H-GLaD
34.8
23.9
24.4
24.2
29.5
DDiF
69.2
42.0
45.3
45.8
64.6
8

---

Published as a conference paper at ICLR 2025
25
Table Test accuracies (% on Audio domain.
Table 6: Test accuracies (%) on 3D voxel domain under IPC=1.
Test accuracy
Spec. / class
10
20
C.
Method
ModelNet
ShapeNet
15
Random eDM
Random
42.6
57.0
Random
60.9
68.5
Herding
SDD
Herding
56.2
72.9
Vanilla
56.3
48.6
10
K-Center DDIF
DSA
DM
65.0
69.1
77.2
74.0
DC
IDC
FreD
78.7
85.6
88.2
79.9
10
10
10
IDC-I
73.3
83.0
DDiF
87.1
89.6
Total utilized budget
IDC-I+ HaBa
74.5
84.3
Vanilla
75.3
80.4
Figure 4: Test accuracies (%) on Video domain. Each black line denotes the same num-
IDC
82.9
86.6
DM
IDC
85.6
85.3
FreD
87.3
90.6
DDiF
90.5
92.7
DDiF
88.4
93.1
ber of decoded instances per
Entire
T
93.4
Entire T
91.6
98.3
class, and 5, respectively.
the video domain, achieves higher performance than coreset selections and DM; however, still quires a large storage budget. On the other hand, DDiF achieves competitive performance even with budget equivalent to only 1.7% of SDD. In addition, Tables 5 and 6 show the test performances of the audio and 3D voxel domains, respectively. Both tables demonstrate that DDiF achieves the highest performance. These results confirm the efficacy of DDiF across various modality datasets.
Cross-reet resssresolttion Generalization. As
mentioned in Section 3.1 previous studies can only perform posiprocessing to resize the optimized synthetic datasets, resulting in information distortion. On the contrary,
Test accuracy gap (%)
Tist accuracy (9%)
80
-10
Vanila
-15
50
DDiF can easily decode data of various sizes by adjusting the coordinate set, due to the continuous nature of the neural field. We introduce novel experiment in the dataset distillation community that assesses generalization performance when the evaluation data size differs from that used for distillation. We define this exper- iment as cross-resolution generalization. We apply the interpolation techniques, such as nearest, bilinear, and bicubic, for the previous parameterization methods.
-20
FreD
bour
128(Org) 256
256
512
Test resolation
Test resolition
(a) Accuracy
(b) Accuracy degradation
Figure 5: (a) Test accuracies (%) with different image resolutions. The original resolution is 128 128. (b) Test accuracy gap (%) from original resolution. We use bilinear interpolation for previous studies.
Figure Sa shows test accuracies on each resolution when utilizing the corresponding network architecture, ConvNetD6 for 256 and ConvNetD7 for 512. DDiF shows the best performance over all test resolutions. Figure shows the percentage of decrease test accuracy with the resolution change i.e. (ACC ACCttst) ACCr) DDIF shows the least decrease with regard to resolution difference, being evidently robust to resolution change. Its robustness opens a new adaptability of dataset distillation methods to more wide-range situations. Please refer to Appendix C.4 for the experimental results on different resizing techniques and the same test architecture.
4.2 ADDITIONAL ANALYSIS AND ABLATION STUDY
Fixed Number of Decoded Instances. To further investigate the coding efficiency and expressive- ness, we hold the number of decoded synthetic instances constant while varying the budget allocated each instance, and then evaluate performance. Under the IPC=1 setting, FreD, SPEED, and DDiF decode 8, 15, and 51 synthetic instances per class at resolution of 128, respectively. We consider two settings in which the number of decoded synthetic instances determined by either baselines or DDiF. Table 7 shows that DDiF achieves strong performance while using less budget than the baselines given the same number of decoded instances. It is also worth noting that, when using the same budget, DDiF decodes more synthetic instances and achieves significantly higher performance. These results support the high coding efficiency and expressiveness DDiF, demonstrating that its superiority arises from improvements in both quality and diversity.

---

Published as a conference paper at ICLR 2025
English Foxhound
Table
Table 7: Test accuracies (%) ImageNet-Subset (128 x 128) under several DIPCs. "DIPC" indicates the number of decoded synthetic instances per class. Note that the total number of budget parameters for IPC=1 491.52k.
Initialized
DIPC
Method
Unilized budget
Nette
Woof
Fruit
Yellow
Meow
Squawk
Vanilla DDiF
491,52k
51.4
29.7
28.8
47.5
33.3
41.0
Distilled
9.63k
49.1
29.4
27.3
44.8
31.9
39.0
Frel)
491.52k
66.8
38.3
43.7
63.2
43.2
57.0
DDiF
77.04k
67.1
37.8
43.6
61.5
44.5
58.3
SPEED
491.52k
66.9
38.0
43.4
62.6
43.6
60.9
15
DDiF
144,45k
68.3
39.7
45.9
65.8
45.4
61.1
Figure 6: Visualization of the decoded synthetic instances from DDiF on image (left) and 3D voxel (right).
Vanilla
25067.5 52k
73.0
42.8
48.2
69.1
47.2
69.0
DDiF
491.52k
72.0
42.9
48.2
69.0
47.4
67.0
Qualitative Analysis. Figure 6 visualizes the decoded synthetic instances by DDiF on various modalities. As shown in the first row in Figure 6 DDiF effectively encodes high- high-dimensional data even with very small budget, regardless utilizes budget that is 1.96% of the original data size for images and 2.87% for 3D voxel. After the distillation stage, each decoded synthetic instance involves flass-discriminattee with significant budget reductions (see the second row). Notably, since the synthetic neural field continuous function, the quantity changes smoothly as the position changes. Please refer to Appendix C.7 for additional visualizations of images, videos, and 3D voxels.
the modality. For instance, each synthetic neural field
features, even
Ablation Studies. DDiF employs neural field, which a neural net- work, to store the distilled infor- mation. We conduct several abla- tion studies to investigate the effect
(51)
(435)
70
(51)
(826)
Test accuraty (%)
Test socurayy
65
66
60
of different components of the neu-
ral network. Figures 7a and 7b demonstrate that DDiF consistently enhances performance, regardless of changes in the number of layers or network width. As the number of lay- ers and the network width increase, the number of decoded synthetic stances decreases because the num- ber of parameters each synthetic neural field increases. By comparing the layer and width, the width has fewer decoded synthetic instances, but it has a more gradual performance change than the layer. To explain the rationale, as shown in Eq. (4), width d directly affects the number of basis functions kt while layer L only affects other factors. Thus, increasing d leads to modest change, as the increase in expressiveness offsets the reduction in quantity. In contrast, increasing results in relatively larger change, as the expressiveness remains similar while the quantity decreases. Even when the structure of the neural field is changed as FFN (Tancik et al. 2020). DDiF consistently shows performance improvement (see Figure 7ch.
50
50
20
30
40
FFN SIREN
Number layers
Hidden dimension wisth
(a) Layer L
(b) Width d
(c) Structure
Figure 7: Ablation studies on (a) layer L, (b) width d, and (c) structure of neural field. The bottom black dashed line indicates the performance of Vanilla (TM).
CONCLUSION
This paper introduces DDiF, a novel parameterization framework for dataset distillation that en- codes information from the large-scale dataset into synthetic neural fields under constrained stor- age budget. By utilizing neural fields, DDiF efficiently encapsulates distilled information from high-dimensional grid-based data and easily decodes data of various sizes. We theoretically analyze the expressiveness of DDiF by investigating the feasible space of decoded synthetic instances and demonstrate that DDiF possesses greater expressiveness than the previous method. Through exten- sive experiments, we demonstrate that DDiF consistently exhibits performance improvements, high generalization and robustness, and broad adaptability across diverse modality datasets.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by Samsung Electronics Co., Ltd (No. 10231011-07374-000 This work was also supported by the IITP(Instiute of Information & Coummunications Technology Planning & (raluuation-TTR((fooommn
Technology Research Center) grant funded by the Korea govem- ment Menistrry of Science and SOIIIT--002-RSS 20200222-)))3)268 We thank Wonjun Choi (won- jun4. a144.choie @ samsung.com and Seongeun Kim (se91 .kim@ wamsung..mnoooo comm from the Core Algorithm Lab, AT Center, for their warm and keen discussions on our research.
REFERENCES
Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions with formulas, graphs, and mathematical tables, volume 55. US Govermment printing office, 1948.
Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards ai. Large-scale kernel machines, 34(5):1-41, 2007.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33:14879-1448 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Gener- alizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9739-3748, 2023.
Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:
2015.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-
1k with constant memory. In International Conference on Machine Learning, pp. 6565-6590. PMLR, 2023.
George Cybenko. Approximation by superpositions sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable mem-
ories for neural networks. Advances in Neural Information Processing Systems, 35:34391-34400 2022.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929, 2020.
Yuxuan Duan, Jianfu Zhang, and Liqing Zhang. Dataset distillation in latent space. arXiv preprint arXiv:2311. 15544, 2023.
Rakesh Dugad and Narendra Ahuja. fast scheme for image size change the compressed domain. IEEE Transactions Circuits and Systems for Video Technology, 11(4):461 -444, 2001.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models distributions of func- tions. arXiv preprint arXiv:2102.04776, 2021.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-lligned trajectory matching. arXiv preprint 10X0v22310005773, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2025
Jeremy Howard. A smaller subset of 10 easily classified classes from imagenet and a little more french. URL http::/ggituu. comffastai/mmgeeette,, 2019.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0830 2020.
Movlud Kerimovich Kerimov. Studies on the zeros of bessel functions and methods for their com- putation. Computational mathematics and mathematical physics, 54:1337-1388, 2014.
Soomro Khurram. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0 2222.440, 2012.
Jang-Hyun Kim, Jinuk Kim, Scong Joon Oh, Sangdoo Yun, Hwanjun Song. Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning, pp. 11 11102-11111 101811118. PMLR, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https: /prss//arxiv... org/ aatp//bs//1/11 1112. 6980
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowl- edge factorization and sharing. arXiv preprint arXiv:2208. 10494, 2022.
Shiye Lei and Dacheng Tao. A comprehensive survey of dataset distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference Robot Learning, pp. 112-123, PMLR, 2022.
Songhua Liu, Kai Wang, Xingyi Yang. Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. arXiv preprint arX1v:2210.16774, 2022.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural information processing systems, 30, 2017.
Lars Morten Mescheder. Stability and expressiveness of deep generative models. PhD thesis, Uni- versität Tübingen, 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Brian B Moser, Federico Raue, Sebastian Palacio, Stanislay Frolov, and Andreas Dengel. Latent dataset distillation with diffusion models. arXiv preprint arXiv:2403.03880 2024.
Tiago Novello. Understanding sinusoidal neural networks. arXiv preprint arXiv:2212.01833, 2022.
RB Paris. An inequality for the bessel function v(Vx). SIAM journal on mathematical analysis, 15 (1):203-205, 1984.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.
Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Tian Qin, Zhiwei Deng, and David Alvarez-Mehh A label is worth thousand images in dataset distillation. arXiv preprint arXiv:2406. 10485, 2024.
12

---

Published as a conference paper at ICLR 2025
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl- Sohl-Dicksteen. On the pressive power deep neural networks. In international conference on machine learning, pp- 2847-2854. PMLR, 2017.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning. pp. 5301-5310. PMLR, 2019.
Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3674-3683, 2020.
Noveen Sachdeva and Julian McAuley. Data distillation:
survey,
arXiv preprint
arXiv:2301.04272, 2023.
Noveen Sachdeva, Zexue He, Wang-Cheng Kang, Jianmo Ni, Derek Zhiyuan Cheng, and Julian McAuley. Farzi data: Autoregressive data distillation. arXiv preprint arXiv:2310.09983 2023.
Liyue Shen, John Pauly, and Lei Xing. Nerp: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction. IEEE Transactions on Neural Networks and Learning Systems, (5(1))7700-78)) 2022.
Donghyeok Shin, Seungjae Shin, and Il-Chul Moon. Frequency domain-based dataset distillation. Advances in Neural Information Processing Systems, 36, 2024.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances neural information processing systems, 33:7462-74735 2020.
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10753-10764, 2021.
K Soomro. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 2012.
Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D 4: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024a.
Duo Su, Junjie Hou, Guang Li, Ren Togo, Rui Song, Takahiro Ogawa, and Miki Haseyama. Gener- ative dataset distillation based on diffusion model. arXiv preprint arXiv:2408.086. 2024b.
Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In 2021 International Joint Conference Neural Networks (IJCNN), pp. 1-8. IEEE, 2021.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9390-9399, 2024.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Haohan Wang. Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IÉEE/CVF conference computer vision and pattern recognition, pp. 8684-8694, 2020a.
13

---

Published as a conference paper at ICLR 2025
Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You. Dim: Distilling dataset into generative model. arXiv preprint arXiv:2303.047000 2023.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 2018.
Zhihao Wang, Jian Chen, and Steven CH Hoi. Deep learning for image super-resolution: survey.
IEEE transactions on pattern analysis and machine intelligence, 43(10):3365-3383 2020b.
Ziyu Wang, Yue Xu, Cewu Lu, and Yong-Lu Li. Dancing with still images: Video distillation
via static-dynamic disentanglement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6296-6304, 2024.
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03200 2018.
George Neville Watson. A treatise on the theory of Bessel functions, volume 2. The University Press, 1922.
Xing Wei, Anjia Cao, Funing Yang, and Zhiheng Ma. Sparse parameterization for epitomic dataset distillation. Advances in Neural Information Processing Systems, 36, 2024.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Graphics Forum, volume 41, pp. 641-676. Wiley Online Library, 2022.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv: 0ttp:/900..62,,. 2019.
Shaolei Yang, Shen Cheng, Mingbo Hong. Haoqiang Fan. Xing Wei, and Shuaicheng Liu. Neural spectral decomposition for dataset distillation. arXiv preprint arXiv:2408. 162.6 2024.
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Guangming Zang, Ramzi Idoughi, Rui Li, Peter Wonka, and Wolfgang Heidrich. Intratomo: self- supervised learning- based tomography via sinogram synthesis and prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1960-1970, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference Machine Learning, pp. 12674-12685. PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Bo Zhao, Konda Reddy Mopuri. and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.0592 05929, 2020.
Haizhong Zheng, Jiachen Sun, Shutong Wu, Bhavya Kailkhura, Zhuoqing Mao, Chaowei Xiao, and Atul Prakash. Leveraging hierarchical feature sharing for efficient dataset condensation. arXiv preprint ar XxX1:2310.00506, 2023.
Xinhao Zhong, Hao Fang, Bin Chen, Xulin Gu, Tao Dai, Meikang Qiu, and Shu-Tao Xia. Hierar- chical features matter: A deep exploration of gan priors for improved dataset distillation. arXiv preprint arXiv:2406.00 05704, 2024.
14

---

Published as a conference paper at ICLR 2025
A PROOFS AND MORE ANALYSIS
A. PROOF OF PROPOSITION 3.1
Proposition 3.1. Consider two functions 91. 92 where 9i Z; + RDD for 1.2. Also, consider
Two matrix variables Z; = [211.... 2iM_ where their columns 2ij E 2, for = 1,2 and j
1,... M. We denotes (Z,) (912))), (..2=))....(((4aa)) for 1,2. Set 9 (olg: Z + R for i = 1,2. If 91 (1) - (2)) for any 91 E 91 and 92 E G2. then e0. ((gi(Z))) 2
min 742 (2)).
Proof. For i = 1,2, let g, Z = arg min E9. (Z)) Note that gi E (i S
(2(22) for j 1.... M and any 92 G2. It implies that there exists some 92 92, Zu 2
such that 92 (Zz) = gi (21). By the definition of 92, Zz, for any 92 E G2, La 2 ((1)( S
(Z2)) Therefore, ((g2 & ((oi In conclusion, min 299. min 2202. (Z)))
(g (Z1)
The following Corollary 1 states that the relationship between the feasible space of synthetic in- stances from input-sized parameterization and the optimal value of the dataset distillation objective, when the number of synthetic instances is the same.
Corollary A.1. Let two matrix variables X₃ [11 ... Fias] and X2 Im ... F2asl consisting of columns Tij E Xx S. RR for = 1,2 and j 1, ,, M. X1 C. X2. then minx,ex ((X1) N
min X2EX (x
To support Corollary A.1 we investi-
Test accurany (9)
gate the performance changes of input-
50
Test accuracy
sized parameterization while imposing
40
constraints on the feasible space of syn-
30
thetic instances under the fixed number of synthetic instances. We consider two types of constraints: 1) dimension masking and 2) value clipping. As shown in Figure g when the feasible space becomes smaller (i.e., as the restrictions are enforced more strongly), the dataset distillation perfor- mances decrease. These results serve as direct evidence of Corollary A.1
30
20
75
Masking ratio (%)
Clipping valus R
(a) Dimension masking
(b) Value clipping
Figure 8: Test accuracies (%) of input-sized parame- terization. We utilize TM on ImageNette (128 128) under IPC=1.
A.2 DERIVATION OF EQ.
To derive the feasible space of DDiF, begin with the theorem from Novello (002). which state as follows:
Theorem A.2. (Novello 2022) Consider a neural network F R R with two hidden layers and width d. If F utilizes sine activation function, then F& represents fanction which is the of
sines and cosines:
Fx(x) b(2) + Σ
ax cos (Wxx) + 8g sin (waxr)
(6)
kEZ
where Wk (()). Pk. = (k, b0)) + b(t)) as -Ei AR,i sin (Pk.)) and Bk
ΣL Ap, cos (yP..s). Also, Ak, == w( de (W:")) and (e(!!")) = I Jk, (()) where Jk,
denotes Bessel function of the first kind of order ky.
Then, by using trigonometric identity, Eq. 66) is represented by the sum of cosines as follows:
Fs (x) (22)
Σ ke2d
as cos(w)x) 32 sin (x)x)
15

---

Published as a conference paper at ICLR 2025
= = + Σ
i sin((ow,)) cos(w + Σ 1=1
AR. cos((pt.)) sin(w
kEZd
A sin What + ppn.))
kEZd
= b(2)
π A cos WaD + Phs)) where Oh, = ph,i 2
kE2d
A.3 PROOF OF THEOREM 3.2
Lemma A.3. For d E zt, there exists x E IR such that IT= Jk, (x) + O where kj E Z, = 1, d and Jkg denotes the Bessel function of the first kind of order k.-
Proof. For j 1,2, ..., d, denote the zero set of Jn, on the real line by (x
0}. Note that ((x,,) is discrete set, when k; is integer, 8o it is countable set (Watson 1922
Abramowitz Stegun 1948 Kerimov 2014 Then, U (Jk,)) ) is also countable set. Since R is uncountable set, there exists some x E R such that & Uパ= (Jk,)). Equivalently, Ju,(x) + 0 for every j = 1.2,...,d d. Hence, for that choice of I. II;= Ju, (x) +0
Theorem 3.2. Consider the truncation of Eq. (4 over KC = [k|l S 5 i.e. Fø(x)
1122) EkEK, Σi=1 AR,i cos (wix + (h..)- Suppose that FreD and DDiF utilize the same number of parameters, i.e., [M]] and the number of parameters in ib are fixed at a given value B. Bf 2 and ( 2 1 ( exp [(34B-2] -1). then g(x; [) S. Fo(x) for any x E Cn- 2
Proof. Since we consider Fv. : + R. the number of parameters of DDiF is +4 For a given budget constraint B, d Iv3+ B 2-
2] is the maximum width of F. which satisfy d² +4++++ < B. Since B 2 DDiF is able to construct a valid neural network, i.c., 2 To prove g(x; S (2), is sufficient to show that there exist neural network parameters &i = (W()), 160 which satisfy α(x; I') (x) for any I' and x E CN (O...., 1}. First, decompose Eq. 44 into
sum over K. C 2d and the other terms:
Fx(x)
6(2) Σ
Ae. cos (wEx + (pR.s)
kE 1=1
Ak.a cos (WE: (k.s)) +
Σ
Σ
AEs cos (wix (pk.i)
kEA
kER; /K i=1
Let K. {K K2 E u E u} and IN [1,1 11TH E RN is the
vector of size N. Set W(0) I IN. ((0)
x IN. ((1)
JIN, and h(2)
2N
-ENEAAx Σ AK, cos (WEr + :.h.)) Note that the absolute value of the Bessel function of the (4 Paris 1984) for any r > O. Also, by Lemma A.3 w() and W(2) can be configured to satisfy Dí AR.: En W (2) II Js, (W!))) To
first kind has finite upper bound [)p(r)
p
for fixed u. Under these parameters, Fx) is same as g(;; I):
Fo(x) = ((2) +
A cos (wix + Pkk)) + Σ
Σ
ARs cos (wie + s/..)
KEK
MEAi V/C.
Σ NEW
Tu cos
= g(x; T)
2N
16

---

Published as a conference paper at ICLR 2025
Next, we prove that there is no such that (x;; I)
(z) for some and Cx. Note that F
can represent up to (2(+1)==1 (Novello 2022). By recalling that d - LV3 B - 2/, we can show the following equivalence from the inequality assumption of (:
log(2B +1)
log(2B B + 1)
Ç
exp
Iv3+ B 2] 2 -1)
log(2 >
Lv3 + B - 2]
(2C +)1vv+++ - B
2
It implies that DDiF can cover more frequency than FreD. Consequently, it is possible to choose
k and W which satisfy wi (k, W()) * xu for any e U. Due to the orthogonality of N
cosine functions having different frequencies, there is no I' that represents cos (wix + (k,,) terms. It implies (2;; I) cannot express F (x) with some v parameters.
B EXPERIMENTAL DETAILS
B.1 DATASETS
Image Domain. We evaluate DDiF on various benchmark image datasets. 1) ImageNet-Subsee
(Howard) 2019 Cazenavette et al. (022) is dataset consisting of subset of similar characteristics in the ImageNet. the experiment, we consider various types of subsets by following Cazenavette
al. (202); ImageNette (various objects), ImageWoof (dog breeds), ImageFruit (fruit category), ImageMeow (cats), ImageSquawk (birds), Image Yellow (yellowish objects). Each subset has 10 classes and more than 10,000 instances. We utilize two types of resolution: 128 128 and 256 x 256. 2) CIFAR-IO Krizhevsky et al. 2009) consists of 60,000 RGB images in classes. Each image has 32 x 32 size. Each class contains 5,000 images for training and 1,000 images for testing. 3) CIFAR- 100 (Krizhevsky et al. 2009) consists of 60,000 32 x 32 RGB images of 100 categories. Each class is split into 500 for training and 100 for testing.
Video Domain. We utilize MiniUCF (Wang et al. 2024), a subset of UCF101 (Soomro 2012) which includes 50 classes. The videos are sampled to 16 frames, and the frames cropped and resized to 112 x 112. Each data has 16 x 3 x 112 x size.
Audio Domain. We utilize Mini Speech Commands (Kim al. 2022). subset of the original Speech Commands dataset (Warden 2018). We follow the data processing of Kim et al. (2022). The dataset consists of 8 classes, and each class has 875/125 data for training/testinng respectively. Each data is 64 x 64 log-scale magnitude spectrograms by short-time Fourier transform (STFT).
3D Domain. We utilize core version of ModelNet-10 10 (Wu et al., 2015) and ShapeNet (Chang) ct al. 2015), which are widely used in 3D. They include 10 classes and 16 classes, respectively. Each 3D point cloud data is converted into 32 x 32 32 voxel.
B.2 NETWORK ARCHITECTURES.
ConvNet. By following previous studies, we leverage the ConvNetDr as default network archi- tecture for both distillation and evaluation of synthetic datasets. The ConvNetDn is convolutional neural network with duplicate blocks. Each blocks consist of a convolution layer with 3 x 3- shape 128 filters, an instance normalization layer, ReLU, and an average pooling with 2 x kernel size with stride 2. Lastly, it contains linear classifier, which outputs the logits. Depending on the resolution of the real dataset, we utilize different depth n. Specifically, ConvNetD3 for 32 x 32 CIFAR-10 and CIFAR-IO0 ConvNetD4 for 64 x 64 Audio spectrograms, ConvNetD5 for 128 x 128 ImageNet-Subee ConvNetD6 for 256 x 256 ImageNet-Subsee and ConvNetD7 for 512 x 512 ImageNet-Subset.
AlexNet. AlexNet a basic convolutional neural network architecture suggested in (Krizheskyy
et al. (2012). It consists of 5 convolution layers, 3 max-pooling layers, 2 Normalized layers, 2 fully connected layers, and SoftMax layer. In each convolution layer, the ReLU activation function is utilized. We adopt this network to evaluate cross-architecture performance of DDiF.
17

---

Published as a conference paper at ICLR 2025
VGG1L. VGGI1 (Simonyan & Zisserman 2014) is also applied for evaluation, which attributes to weighted layers. It consists of convolution layers and 3 fully connected layers. Its design is straightforward yet powerful, providing a balance between depth and computational efficiency. The number of trainable parameters is around 1 132M, making it larger than earlier models but still suitable for medium-scale tasks. We adopt this network evaluate the cross-architecture performance of DDiF.
ResNet18. ResNet18 (He al. 2016) introduces residual connections, which help mitigate the vanishing gradient problem in deep networks by allowing gradients to bypass certain layers. It consists of 18 layers with residual blocks, each composed of two convolutional layers followed by activation and normalization with around 1 IM trainable parameters. We utilize ResNet18 as one of the architecture for evaluating synthetic datasets.
VIT. Vision Transformer (Dosovitskiy) 2020) utilizes the transformer architecture, initially de- signed for sequence modeling tasks in NIP. For image classification, it divides images into non- overlapping patches and processes them sequence using self-attention mechanisms. ViT has around 10M trainable parameters in its base form and offers a competitive alternative to CNNs, demonstrating the effectiveness of transformers in vision tasks. We selected VIT as the final net- work to evaluate synthetic image datasets.
CDee. For the 3D voxel domain, we utilize Conv3DNet (Shin et al. 2024), a 3D version of ConvNet. Conv3DNet consists of three repeated blocks, each containing 3 x 3 x 3 convolutional layer with 64 filters, 3D instance normalization, ReLU activation, and 3D average pooling with a 2 x 2 2 filter, and stride of 2. Lastly, it contains linear classifier.
B.3 BASELINES.
Since our main focus lies on the parameterization of dataset distillation, we compare DDiF with 1) static decoding, which are IDC KKm et al., 2022) and FreD (Shin et al. 20024); 2) parameterized decoding, which is RTP (Deng & Russakovsky 2022), HaBa (Liu et al. 2022), SPEED (Wei et al.) 2024). LatentDD (Duan et al. (023), and NSD (Yang et al. 2024); and 3) deep generative prior, which include GLaD (Cazenavette et al.] 2023), H-GLaD (Zhong et al. 2024), and LD3M (Moser et al. 2024). We also demonstrate the performance improvement of DDiF compared to input-sized parameterization, denoted as Vanilla.
B.4 IMPLEMENTATION SETTINGS.
Although any loss can be adapted to DDiF, we utilize TM (Cazenavette et al. 2022) for L. as
default unless specified. Following previous studies, we use DSA (Zhao Bilen 2021). ), which con- sists of color jittering, cropping, cutout, flipping, scaling, and rotation. We adopt ZCA whitening on CIFAR-IU 10 (IPC=1, 10) and CIFAR- (IPC=1) with the Kornia (Riba et al. 2020) implementation.
We adopt SIREN [Sizzmann et al. 2020) for synthetic field Fg a default. SIREN is multilayer perceptron with a sinusoidal activation function, and it is widely used in the neural field area due to its simple structure. We use the same width across all layers in synthetic neural field i.e. di d for all Z. We utilize normalized coordinates defined on [-1,1]" for n-dimension data to enhance stability (Sitzmann al. 2020). , rather than using integer coordinates, which have wide range. For cross-resolution experiments, we utilize the coordinate set C, which consists of evenly spaced points within the interval [-1, 1] according the target resolution. We provide the detailed configuration of the synthetic neural field, the resulting size of each neural field, the number of synthetic instances per class, and the total number of neural fields Table g We use Adam optimizer (Kingma & Ba, 2017) for all experiments. We fix the iteration number and learning rate for warm-up initialization of synthetic neural field as 5,000 and 0.0005. Without any description to distillation loss, we gen- erally use matching training trajectory (TM) objective for dataset distillation loss L. Following the previous studies, we utilize two types of default TM hyperparameters same as SPEED (Wei et al, 2024) and FreD (Shin et al. 2024). ). We run 15,000 iterations for TM and 20,000 iterations for DM. We use a mixture of RTX 3090, L40S, and Tesla A100 to run our experiments. We follow the con- ventional evaluation procedure of the previous studies: train 5 randomly initialized networks with an optimized synthetic dataset and evaluate the classification performance. We provide the detailed hyperparameters in Table
18

---

Published as a conference paper at ICLR 2025
Table 8: Configuration of the synthetic neural field. In the case of Video, there is no increment of decoded instances because we experimented with the fixed number of decoded instances.
Increment of decoded instances
Modality
Dataset
IPC
L
d
size (z))
2
3
81
x37
CIFARIO
10
2
2
6
3
81
x37.9
2
3
543
x5.64
2
10
3
173
x
Image
CIFARI00
10
2
15
3
333
x9.2
50
2
2
30
3
1113
X2.76
2
3
20
3
963
x51
ImageNet-Subsel (128 x 128)
10
20
3
963
X51
50
3
40
3
3523
x 13.94
ImageNet-Subset (256 x 256)
3
40
3
3523
x55
3
8483
Video
MiniUCF
3
6
40
3
8483
10
2
3
10
1
261
x 15,6
Audio
Mini Speech Commands
20
10
261
15.6
ModelNet
3
20
1
941
x34
3D
ShapeNet
3
20
941
x34
Table 9: Configuration of hyperparameters for optimization.
(a) Gradient matching (DC)
(b) Distribution matching (DM)
Synthetic batch size
Learning rate (Neural field)
Synthetic batch size
Leaming rate (Neural field)
Dutaset
IPC
Dataset
IPC
ImageNet-Subeee
(128 128)
5 x 10
ImageNet-Subsee (128 x 128)
5 x 10-5
10
64
10-5
Mini Speech Commands
20
64
10-1
Subsett (256 x 256)
10
ModelNet
1
10-4
10
ShapeNet
1
10-1
MiniUCF
5
10
ModelNet
10-4
ShapeNet
10-4
(c) Trajectory matching (TM)
Synthetic steps
Expert epochs
Max start epoch
Synthetic batch size
Learning rate (Neural field)
Learning rate (Step size)
Learning rate
Dataset
IPC
(Teacher)
I
60
10
74
10-3
10-5
10-2
CIFAR-10
10
60
10
256
10-3
10-5
10-2
50
60
2
40
235
10-4
10-5
10-2
60
40
170
10
10-5
10-2
CIFAR-100
10
60
40
230
10
10-5
10-2
50
60
40
276
10-3
10-3
10-2
20
10
102
10-4
10-0
10-2
ImageNat-Subsct
(128 x 128)
10
20
30
10-4
10-5
10-2
50
40
20
30
10-4
10-5
10-2
19

---

Published as a conference paper at ICLR 2025
B.5 ALGORITHM OF DDIF
The main difference between DDiF and conventional parameterization methods is the decoding process for synthetic instances. Basically, the neural field takes coordinates as input and output quantities. To generate single synthetic instance, each coordinate c E C is input into the synthetic neural field Fv. and then the resulting value is assigned to the corresponding coordinate of the de-
coded synthetic instance ( Algorithms and 2 specify training procedure and decoding process we include loop over coordinates for clarity. However, should be noted that in the actual implementation, the coordinate set is input to the neural network in full-batch manner.
of DDiF, respectively. Algorithm 2
Algorithm Training procedure of DDiF
Algorithm 2 Decoding process of Synthetic stances in DDiF
Input: Original real dataset T; Dataset distillation
loss L; Initialized y = { |0]
Learning
Input: Set of parameters of synthetic neural
fields Mi; Coordinate set C
rate 17 Output: Parameterized
Output: Set of decoded synthetic instances Xs
synthetic
dataset
1: Initialize Xs +- 0
{(bb) (1b5.ū,)}}
",) j=1
for for to w
1: Initialize coordinate set C from E T
3: Initialize žn E mR X.xNN.
2: for j 1 to | do
4: for C E do
3: Sample real instance (z, y) T
5: 3. (c,
4: Vj f-
6: end for
5: Optimize thy with Eq. 2
7: Xss f X Uūj
6: end for
8: end for
7: repeat
8: Sample a real mini- baini-bachh By T
9: Bs (Fx(C) v from Ve W
10: + T vB,,
until convergence
C ADDITIONAL EXPERIMENTAL RESULTS
C.I PERFORMANCE COMPARISON ON LOW-DIMENSIONAL DATASETS
To verify the wide applicability of DDiF, we conduct experiments on low-dimensional datasets, such CIFAR-IO and CIFAR-100. In Table IO DDiF exhibits highly competitive performances with previous studies. These results demonstrate that DDiF is also properly applicable to low- dimensional datasets, while DDiF shows significant performance improvement in high- nigh-dimensional datasets.
Table 10: Test accuracies (%) on CIFAR-10 and CIFAR-100. Bold and Underline means best and second-best performance of each column, respectively. " indicates no reported results.
Dataset
CIFAR10
CIFARI00
IPC
10
50
10
50
TM
46.3
65.3
71.620.2
24.3
40.1
47.7.0.2
Input-sized
FRePo
46.8
65.540.4
71.7a0.2
28.7
42.5
44.3
IDC
50.040.4
67.5
74.5
Static
FreD
60.6
70.3
75.8
34.620.4
42.7+0.2
47.8
HaBa
48.3
69.940.4
74.0.1..
47.040.2
RTP
66.4+0.4
71.2++0.
73.640.5
34.0+0.4
42.9
Parameterized
HMN
65.7
73.7+0.1
76.940.2
36.3
45.4+0.2
48.5
SPEED
63.2+0.1
73.540.2
77.740.4
40.4+0.4
45.940.3
49.1
NSD
68.5
73.4+0..
75.240.6
36,540.3
46.1
Function
DDiF
66.5
74.0
77.5
42.1
46.0+..2
49.9
20

---

Published as a conference paper at ICLR 2025
C.2 ADDITIONAL PERFORMANCE COMPARISON ON HIGH-DIMENSIONAL DATASETS
Comparison under Large Budget (IPC=50). In general, high-dimensional dataset distillation under large budget rarely addressed in previous studies due their significant computational cost. To demonstrate the efficacy of DDiF even in a large budget setting, we conducted experiments on ImageNette (128 x 128) under IPC=50. DDiF with TM achieves 75. 7.33% while vanilla with TM achieves 72.8%+0.89 It means that DDiF effectively improves the dataset distillation performance even with larger storage budget for high resolution.
Fixed DIPC under 256 resolution. We further compare with input-sized parameterization on ImageNet-Subset (256 x 256) under the fixed number of decoded synthetic instances. Under the IPC=1 setting, DDiF can decode 55 synthetic instances per class on 256 resolution. Similar to the main paper, we consider two settings: 1) varying the number of decoded synthetic instances from the baselines, and 2) varying the number of decoded synthetic instances from DDiF. Table 11 presents mixed performances on ImageNet- Subsett (256 x 256). However, we emphasize that DDiF utilizes much smaller budget. Moreover, it is worth noting that DDiF' achieves higher performance than Vanilla in some cases. We believe that it is related previous findings that input-sized parameteri- zation includes superfluous or irrelevant information (Lei & Tao 2023 Yu et al. 2023 Sachdeva & McAuley, 2023). These experimental results support that DDiF involves sufficient representational power while using much smaller budget compared to the input-sized parameterization.
Table 11:
Test accuracies (%) on ImageNet-Subset (256 x 256) with input-sized parameterization (Vanilla) and DDiE. "DIPC" denotes the number of decoded instances per class.
DIPC
Methods Utilized Budget
Nette
Woof
Fruit
Ycllow
Mcow
Squawk
Vanilla
1,966.08k
32.1
20.0
19.5
33.4
21.2
27.6
DDiF
79.53k
31.2++0.8
21.2:09
21.3=15
34.8+110
20.0
27.9=25
Vanilla
55 1,966.08k
70.1
37.5
41.3208
64.2
47.1
64.3:112
55
DDiF
1,966.08k
67.8
39.6
43.2=17
63.1 1.08
44.8
67.0=n9
C.3
COMPATIBILITY WITH SOFT LABEL
Recently, various previous literatures have demonstrated that using soft labels instead of one-hot labels leads to significant performance improvements in dataset distillation (Cui et al.] 2023 Qin et al. 2024). As mentioned in the main paper. DDiF can also utilize a soft label technique, but employ fixed one-hot label for a fair comparison with the previous param- eterization methods. Therefore, we conduct ad- ditional experiments to validate the compatibil- ity of DDiF with soft label. We consider RDED
Table 12: Test accuracies (%) obtained by varying the synthetic label under IPC=50 and ConvNet.
ImageNette (128 x 128)
Method
CIFAR-100
RDED
33.5
57.8
Fixed onc-hot label
DATM
50.8
DDiF
49.9
75,2213
RDED
$7.0
83.8
Soft label
DATM
55.0102
DDiF
58.2
86.9 1.0
(Sun et al. 2024) and DATM (Guo et al.) 2023) as our baselines since they employ the soft label and exhibit state-of of-the--rrrr- art performances on CIFAR-10. CIFAR-100, and ImageNette (128 x 128). To apply soft labels to DDiF, we utilize the region-level soft label technique from RDED on the de- coded synthetic instances, which are already obtained through training with TM. Table T2 presents
the results as follows:
RDED and DATM using soft label exhibit significant performance improvements compared to their performance without soft labels, consistently demonstrating the previous findings.
Applying RDED's soft label technique to DDiF results in notable performance improve- ment, consistent with previous studies.
Under the soft label setting, DDiF achieves higher performance with significant gap than RDED and DATM, establishing a new state-of-the-art (SOTA).
In summary, prioritize experiments under the fixed one-hot label setting to ensure fair com- parison with existing parameterization methods. However, our experiments confirm that DDiF can effectively utilize soft label, achieving substantial performance improvements when applied.
21

---

Published as a conference paper at ICLR 2025
C.4 ADDITIONAL RESULTS ON CROSS- RESOLUTION GENERALIZATION
Detailed experímental results. We ap-
ply the spatial- upsampling meth-
ods, such as nearest, bilinear, and bicubic. for the optimized synthetic instances of
Teol accuracy 3tp (%)
Test sccuracy (%)
previous parameterization methods. Par-
-20
Meill
ticularly, FreD can also utilize frequency- domain upsampling since it stores masked frequency coefficients. The most widely used for frequency-domain upsampling is zero-padding the frequency coeffi- cients before inverse frequency transform, which means assigning zeros the high- frequency components (Dugad & Ahuja 2001). For example, in the case DCT. which is the default setting of FreD, the process of upsampling an N N-resolution frequency coefficient F' to an M(> N) resolution image can be described as
40
40
FroD
-50
128(Crg) 256
512
256
512
Test resclution
Test resolutionn
(a) Accuracy
(b) Accuracy degradation
Figure 9: (a) Test accuracies (%) with different image resolutions. The original resolution is 128 128. (b) Test accuracy gap (%) from original resolution. We use bilinear interpolation for previous studies. We uti- lize ConvNetD5, which is the same architecture in the distillation stage.
IDCT (ON
[ OM-N] where
O0M-N OM-N
NN) is the scaling factor. We refer this frequency-
domain upsampling as "zero-padding". For DDiF, we constructed a coordinate set suitable for the test resolution, and then input into the optimized synthetic neural field to generate the upsampled synthetic instance. We refer to this method as "coordinate interpolation".
presents the detailed experimental results. We observe that the previous parameterizations show drastic performance degradation regardless of the interpolation method used. Whereas, DDiF still achieves the highest cross-resolution generalization performance. Also, Figure 9 presents the cross-resolution generalization performance under the same network architecture in the distillation stage. As seen in Figure 5 DDiF achieves the best performance and shows the least performance degradation over all resolutions. These extensive results consistently demonstrate that DDiF is ro- bust to resolution change, and this robustness is largely driven by the continuous nature of the syn- thetic neural field.
Table 13
Comparison with Full dataset downsampling. The most intuitive and straightforward way to reduce the budget of dataset is by downsampling, which reduces the budget size of each instance. means that is possible to downsample the full dataset to a specific resolution for storage and then upsample it to the test resolution. One may doubt that this simple method can show high cross-resolution generalization performance. To verify the superiority of DDiF on cross-resolution generalization, we additionally conduct performance comparison with full dataset downsampling.
Table 14 shows the experiment results where the full dataset was downsampled and then upsampled to the original resolution during the test phase. When the budget is similar (when the downsampled resolution is x 4), DDiF outperforms the full dataset downsampling method. We also experimented when the downsampled resolution was the same as the resolution of the decoded synthetic instance. In this case, the downsampled dataset achieved better performance, as expected. However, this set- ting requires 1,289 times more budget than DDiF since the number of instances not reduced. Such a setup deviates from the core purpose of dataset distillation, which aims to optimize performance under strict budget constraints.
Furthermore, we emphasize that the full dataset downsampling has two limitations on cross- resolution generalization. First, it requires slightly different assumption. Cross-resolution gen- eralization experiment, which we proposed, is modeled to evaluate the dataset distillation ability to generalize to higher resolution settings. It involves training on a low-resolution (128 x 128) synthetic dataset and testing high-resolution (256 256) data. However, full dataset downsam- pling diverges from this cross-resolution setting since assumes the availability of a high-resolution dataset (256 256). Second, the training time during the evaluation stage increases since the number of data points remains the same as the full dataset. While the memory budget may be comparable, this increased time cost undesirable, especially in scenarios where efficiency is critical.
22

---

Published as a conference paper at ICLR 2025
Table 13: Test accuracies (%) with different resolutions and networks. The original resolution is 128 x 128. We denote the difference between ordinary and cross-resolution performance as
Di ACCorg ACCtes and relative ratio Ratio ACCorg ACCrest
ACCorg
Test resolution
Test network
Method
Upsampling
Accuracy (f)
Diff (J)
Ratio (4)
29.5
21.9
0.43
Vanilla
bilincar
31.2+111
20.2
0.39
bicubic
30.7+2.0
20.7
0.40
nearest
$4.811.2
6.6
0.11
IDC
bilinear
$4.012.0
7.4
0.12
bicuhic
55.0+1.6
0.10
ConvNetDS
nearest
58.8+1.4
0.12
SPEED
bilinear
57.7.00.8
0.14
bicubic
58.1+10
8.8
0.13
nearest
55.2+2.2
11.6
0.17
bilinear
55.0+2.6
11.8
0.16
FreD
bicubic
56.411.4
10.4
0.16
zcro-padding
53.8+14
13.0
0.19
DDiF
coord. interpolation
66.3:1.9
5.7
0.08
nearest
44.0+1.7
7.3
0.14
Vanilla
bilinear
43.2+1.11
8.2
0.16
bicubic
43.9+1.8
0.14
nearest
54.7+1.6
0.11
IDC
bilinear
$4.52116
0.11
bicubic
55,411.3
6.0
0.10
ConvNctD6
62,0+1.a
0,07
SPEED
bilincar
61.8+1.8
5.1
0.08
bicubic
62.611.1
4.3
0.06
nearest
60.9
5.9
0.09
bilincar
+0.1
6.7
0.10
FreD
bicuhic
61.4+0.8
0.09
zero-palding
61.811.0
0.07
DDiF
coord. interpolation
70,611.2
0.02
nearest
27.4+1.
24.0
0.47
Vanilla
bilincar
27.1
24.2
0.47
bicubic
27.1a1.
24.3
0.47
nearest
38.6+2.7
22.8
0.37
IDC
bilinear
37.5+2.3
23.9
0.39
bicubic
39.5
21.9
0.36
ConvNetD5
nearest
43.312.2
23.6
0.35
SPEED
bilinear
45.0+1.5
21.9
0.33
bicubic
44.8+5.0
0.33
nearest
42.5+2.5
24.3
0.36
bilinear
41.4+1.5
25.4
0.38
FreD
512
bicubic
41.6+1.
25.2
0.38
zero-padding
42.9+1.5
23.9
0.36
DDiF
coord. interpolation
58.7.11.2
13.3
0.18
nearest
41.2+1.5
10.1
0.20
Vanilla
bilincar
40.6+2.6
10.7
0.21
bicubic
40.4+1.9
10.9
0.21
nearest
51.511.8
9.9
0.16
IDC
bilincar
50.7+2.1
10.7
0.17
bicuhic
51.2+2.8
10.7
0.17
ConvNe1D7
nearest
59.612.0
7.3
0.11
SPEED
bilinear
59.5122
0.11
bicubic
60.1+1.7
6.8
0.10
nearest
+5.1
11.7
0.18
bilinear
53.841.0
13.0
0.19
FreD
bicubic
54.4+0.9
12.4
0.19
zcro-padding
56,3+0.8
10.5
0.16
DDiF
coord. interpolation
69.0
3.0
0.04
23

---

Published as a conference paper at ICLR 2025
Table 14: Performance comparison when the test resolution 256 256. We utilize bicubic inter- polation for full dataset resizing. The relative budget ratio indicates the ratio of full dataset down- sampling over DDiF.
Test network
Method
Original resolution
Downsampled resolution
Relative budget ratio
Accuracy
256 x 256 256 x 256
4x4
1.3
45.2+ 23
Downsample
128 x 128
1,289.4
91.3.1 0.5
ConvNerDS
DDiF
128 x 128
1.0
66.3# 1.9
256 256
4x4
1.3
44.7=
Downsample
ConvNerD6
256 256
128 128
1,289.4
91.2=00
DDiF
128 x 128
1.0
70.6: 1.3
C.5 ROBUSTNESS TO CORRUPTION
We further investigate the robustness against corruption of DDiF in the trained synthetic datasets by evaluating on ImageNet-Subset-e This subset is designed specifically to assess robustness across varying corruption types and severity levels. We report the average test accuracies over 15 corruption types, each evaluated across 5 levels of severity, for each class in ImageNet-Subset-C. Table 15 sum- marizes the performance DDiF and baseline models. The results in Table 15 clearly demonstrate that DDiF has the same substantial robustness to resolution change and corruption.
Table 15: Test accuracies (%) on ImageNet-Subsee Cubbettt- under IPC=1. Accuracy on ImageSquawk-C of TM is not reported in previous works.
Method
ImageNette-C C
W
ImageFruit-C
ImageYellow-C Image:Meow-C
ImageSquawk.CC
TM
38.011.6
23.811.0
22.71111
35.611.7
23.2
IDC
34.520.
18.7104
28.520.9
36.8
22.2+1.2
26.8100.5
FreD
51.2.0.8
31.010.9
32.3
48.2.1.0
30.3:0.3
45.910.6
DDiF
54.510.6
34.010.4
36.610.4
47.2.0.7
30.3
53.8
C.6 TIME COMPLEXITY
As mentioned earlier, the neural field takes coordinates as input and produces quantities as output. This distinet char- acteristic of the neural field offers the advantage of being resolution-invariant but may raise concerns regarding the de- coding process time. We admit that the decoding time of DDiF indeed increases as resolution grows due to the need to forward larger number of coordinates through the neural field. To ad- dress this concern, we conducted an experiment measuring the wall-clock time for decoding single instance.
Table 16: Wall-clock time (ms) of the decoding process for single synthetic instance. "ms" indicates the millisecond.
128
128
256
Vanilla IDC
0.31
0.31
0.40
2.83
FreD
0.46
1.20
Table 16 shows the results with 128 image resolution demon- strate that while the time cost of DDiF larger than methods relying on non-parameterized decoding functions, such as IDC and FreD, remains comparable to methods that use param- eterized decoding functions, such as HaBa and SPEED, and
HaBa
2.81
SPEED
2,20
GLaD
31.33
DDiF
2.49
3.25
exhibits lower time cost than methods that utilize pre-trained generative models, such as GLaD. As the resolution increases to 256, the decoding time of DDiF also increases and it is slightly larger than non-parann oon-parameterized decoding functions. conclusion, although the decoding process time of DDiF increases the resolution increases, does not differ significantly from that of conventional parameterization methods. We attribute this to 1) the use small neural network structure for the synthetic neural field and 2) the full-batch forwarding of the coordinate set in the implementation.
C.7 FULL TABLE WITH STANDARD DEVIATION AND ADDITIONAL VISUALIZATION
For improved layout, we have positioned full tables with standard deviation and additional example figures at the end of the paper. Please refer to Table 17 for ImageNet-Subset (128 x 128) under IPC=1; Table 18 for ImageNet-Subbet (256 x 256) under IPC=1; Table 19 for ImageNet- Subset under (128 128) IPC=10; Table 20 for cross- arssssccchteettre: and Table 21 for robustness to the loss.
In addition, please refer to Figures 10 to 15 for Image domain; Figure [6] for 3D domain; and Figure 17 for video domain.
24

---

Published as a conference paper at ICLR 2025
D DISCUSSIONS
D.1 MORE COMPARISON WITH FRED
As seen in Eqs. 4 and (5). the functions represented by DDiF and FreD are similar, both being the sum of cosine functions. Therefore, we can perform a term-by-tern comparison of both equations.
DDiF enables to change the amplitudes Ag, frrquenccees wk. phases Pk and shift b(O), while FreD only allows the amplitudes. It indicates that DDiF has higher representation ability than FreD.
Although FreD a finite sum of cosine functions with a fixed frequency, DDiF represents infinite sum of cosine functions with tunable frequency. It means that DDiF can cover a wide range of frequencies from low to high by selecting various k. According to the empirical findings in Wang et al. (020a). it has been demonstrated that datasets with larger number of frequencies exhibit improved generalization performance.
DDiF a continuous function, whereas FreD is a discrete function. Due to this charac- teristic, FreD cannot encode information for coordinates that were not provided during the distillation stage. contrast, since DDiF operates over a continuous domain, it inherently stores information for coordinates that were supplied during the distillation stage.
In summary, DDiF has more flexible and expressive function than FreD.
D.2 DISCUSSION ABOUT THEORETICAL ANALYSIS
We provide the theoretical analysis to propose framework for comparing parameterization meth- ods, which have traditionally been evaluated solely based on performance, through expressiveness, i.e. the size of the feasible space. Proposition 3.11 indicates that a larger feasible space for decoded synthetic instances through parameterization methods results in lower dataset distillation loss. The- orem 3.2 claims that DDiF has greater expressiveness than prior work (FreD) when the utilized parameters for a single synthetic instance are the same. We believe that the proposed theoretical analysis framework will serve comerstone for future theoretical comparisons of parameteriza- tion methods in dataset distillation areas.
However, this theoretical amalysis still has room for further improvement. In the theoretical analysis, we compare the expressiveness under the fixed number of decoded instances and/or the same uti- lized parameters for single synthetic instance, not the fixed entire storage budget. We empirically demonstrate the superiority of DDiF in each of the aforementioned cases: 1) the fixed number of decoded instances (see Table 7. 2) the same utilized parameters for single synthetic instance (see Figure B). and 3) the fixed entire storage budget (see Table and others). Even though the proposed theoretical analysis in this paper is experimentally verified through extensive results, this framework has the limitation of not primarily focusing on the fixed entire storage budget scenario, which is the most basic setting dataset distillation.
We believe that constructing a theoretical framework to compare the expressiveness of parameteriza- tion methods under a fixed entire storage budget is necessary, and the proposed theoretical analysis in this paper can serve foundational background for such efforts.
D.3 COMPARISON WITH DIM
As mentioned in Section 3 the synthetic function has several possible forms. DiM WWagg et al.) 2023) employs probability density function as a synthetic function and utilizes deep generative model to parameterize it. Specifically, the decoded synthetic instance of DiM is the sampled output of a deep generative model by inputting random noise:
min L(T where = (2). 2 Z~N(0,1)
DDiF and DiM have in common that they store the distilled information in the synthetic function and only store the parameters of the function without any additional codes. However, there are several structural differences.
25

---

Published as a conference paper at ICLR 2025
The output of DiM still depends on the data dimension. As aforementioned, this type of decoding function requires a more complicated structure and storage budget as the data dimension grows larger. Actually, DiM has not been extensively tested high-resolution datasets. On the contrary, DDiF stores information regardless of data dimension, which indicates broader applicability across various resolutions.
The decoding process of DiM is stochastic. Due the stochasticity. DiM can sample the diverse decoded synthetic instances and save the redeployment cost. However, at the same time, DiM carries the risk of generating less informative synthetic instances. Consequently, it leads to instability in training on downstream tasks. Furthermore, DiM might suffer from redundant sampling due mode collapse, well-known issue of the generative model. Meanwhile, since the decoding process of DDiF is deterministic, DDiF has an advantage in stability.
D.4 LIMITATION
Less efficiency on Low-dimensional datasets. One of the main ideas in parameterization is panding the trade-off curve between quantity and quality: reducing the utilized budget of each syn- thetic instance, while maximally preserving the expressiveness of it. While DDiF has extensive feature coverage theoretically and shows competitive performances with previous studies experi- mentally, it might be less efficient for some low-dimensional datasets due to the structural features of the neural field. This is because, given the low-dimensional instance, it can be difficult to design neural field that is sufficiently expressive with fewer parameters. spite of this issue, we repeat- edly highlight that DDiF has a significant performance improvement on high-dimensional datasets. Furthermore, we speculate that deeper analysis of the neural field structure could be interesting direction for future research in dataset distillation.
Individual Parameterization. Several studies have claimed that storing intra- and inter-class in- formation in shared component is effective. From this perspective, proposed method, which has one-to-one correspondence between synthetic instances and synthetic neural fields, does not have component to store shared information. We believe that it can be extended by adding modu- lation or conditional code, and this paper may serve as a good starting point.
26

---

Published as a conference paper at ICLR 2025
Table 17: Test accuracies (%) ImageNet-Subset (128 128) with regard to various parameteri- zation methods under IPC=1.
Subset
Nette
Woof
Fruit
Yellow
Mcow
Squawk
TM
51,412.3
29.7
28.8
47.5+1.
33.310.7
41.0a1.5
Input-sized
FRePo
48.1
29.7=0.6
IDC
61.41.1.0
34.5
38.011.1
56.5
1..
10.2
Static
FreD
66.8+0.4
38.3=1.5
43.7+1.6
63.2+1.0
43.2+0.8
57.0+0.8
HaBa
51.9
32.4
34.7
50.4
36,910.9
41.9111.4
Parameterized
SPEED
66.9+0.7
38.0=0.9
43,440.6
62.6+1.3
43,6+0.7
60.9
TM+RTP
69.6+uu4
38.8=1.1
45.2+1.7
66.44u..
46.5+1.8
63.2:t1.0
NSD
68.610.8
35.2=0.4
39.8+0.2
61.0+0.5
45.2+00.1
52.9#0.7
GLaD H-GLaD
38.7+1.6
23.4
23.140.4
26.0+1.1
35.8+1.4
DGM Prior
45,4+1.1
28.3=0.2
25.6+0.7
29. 1..
39.7+0.8
Function
DDiF
72.0
42.9 10.7
48.2
69.0
17.
67.0.11.3
Table 18: Test accuracies (%) on ImageNet-Subset (256 x 256) with regard to various parameteri- zaiton methods under IPC=1.
Subset
Nette
Woof
Fruit
Yellow
Meow
Squawk
Input-sized
DM
32.1
20.0
19.5
33.4
21.2
27.6
IDC
53.7+1.2
30.2+1.5
33.1+1.5
52.2+1.4
34.6+1.8
47.0+1.5
Static
FreD
54.2+1.1
31.2+00.
32.5+19
49.1
34.0+1.2
43.1+1.5
Parameterized
SPEED
57.7+0.9
DGM Prior
LatentDM
56.1
28.0
30.7
36.3
47.1
Function
DDiF
67.8+1.0
39.6+1.6
43.2+1.7
63.1
44.8+
67.0+0.8
Table 19: Test accuracies (%) ImageNet-Subset (128 128) with regard to various parameteri- zation methods under IPC=10.
Subset
Nette
Woof
Fruit
Yellow
Mcow
Squawk
TM
63.0=1.3 66.5
1..8
40.3
60.0
40.4
52.3
Input-sized
FRePo
42.2+0.9
IDC
70.8=0.5
39.8+0.9
46,4+1.4
68.7 7.0.8
47.9+1.4
65,441.2
Static
FreD
7..0
41.3
47.0+:.11
69.2
48.6
67.3
HaBa
64.7=1.6
38.6 38.6+1.3
42.5+1.6
63.0+1.6
42.9+0.9
56.8+1.0
Parameterized
SPEED
72.9
44.1+1.4
50,0
70.5
52.0
71.8+1.3
Function
DDiF
74.6=0.7
44.9
49.8
70.5
50.6+11.1
72.3+1.3
27

---

Published as a conference paper at ICLR 2025
Table 20: Test accuracies (%) on cross Architecture networks with ImageNet Subsets (128 128)
under IPC=1
Test Net
Method
Nette
Woof
Fruit
Yellow
Meow
Squawk
TM
13.2+0.6
10.0=0.0
10.0+0.0
11.0+0.2
9.8=0.0
IDC
17.4
16.5
17.9+0.7
20.6+0.9
16.8
20.7+110
AlexNet
FreD
30.7
23.9
13.810.7
19.811.2
14.4.00.5
36.3.10.3
DDiF
60.7
16.4
41.8
56.2
40.3
60.5
TM
17.4+2.1
12.6+1.8
11,8+10
16.9+1.1
13.8+1.3
IDC
19.6
16.0:221
13.841.3
16.813.5
13.1
19.1
VGG11
FreD 21.8+229
17.1
12.6426
18.211.1
13.2+1.9
18.6+22.3
DDiF
1..6
29.9
33.8
44.2
1..
17.
TM
34.9+2.3
20.7+1.0
23.1+1.5
43.4+1.1
22.8+22.2
IDC
43.6+1.3
23.24008
32,9428
44.2+3.5
28.2+0.5
47.8+11.9
ResNet18
FreD
1..8
20.6
34.011.9
49.3
29.0
50.2
DDiF
1..8
11.9
42.0.1.9
55.9
35.8
62.6
TM
22.611.1
15.9
23.3.1..4
18.111.3
18.610.99
IDC
31.010.6
22.4.108
31.110.8
30.3
21.4.10.7
32.2.112
VIT
FreD
38.4
25.4
31.941.4
37.612.0
19.7
44.4
DDiF
59.0+0.4
32.8+10.8
39.4+0.8
47.9
27.0=1.66
14.8
Table 21: Compatibility on different dataset distillation loss with ImageNet Subsets (128 x 128)
under IPC=1
C
Method
Nette
Woof
Fruit
Meow
Squawk
Vanilla
34.2a1.7
22.5+1.0
21.0=0.9
22.0.0.6
32.011.5
IDC
45.440.7
25.5+0.7
26.8=0.4
25.3+0.6
34.6+0.5
FreD
49.140.8
26.141.1
30.0=o.7
28.7+1.0
39.7+0.7
DC
GLaD
35,4+1.2
22.3+1.1
20.7=1.1
22.6#0.8
33,8+0.9
H-GLaD
36,9+0.8
24.0+0.8
22.4=1.1
24.1
35.3+1.0
DDiF
61.2.11.0
35.2
1.1
39.111..3
54.3
Vanilla
30.4+2.7
20.7+1.0
20.4=1.9
20.1+1..
26.6+2.6
IDC
48.3+1.3
27.0+1.0
29.9=0.7
30.5+1.0
38,8+14
FreD
56.241.0
31.0111.
33.4
33.310.6
42.7
DM
GLaD
32.241.7
21.24:1.5
21.8=1.8
22.311.6
27.61119
H-GLaD
34.8+1.0
23.941.9
24.4-2.1
24.2+11.1
29.5
DDiF
69.2+1.0
42.0+0.44
45.3-1.8
45.8
64.6+1.1
28

---

Published as a conference paper at ICLR 2025
(a) Initialization
(b) Synthesized
Figure 10: (a) Warm-up initialized images ImageNette with DDiF, (b) Best- Best-performed synthetic
dataset represented by DDiF. We visualize the first 10 images, while DDiF constructs 51 images per class under the same budget.
(a) Initialization
(b) Synthesized
Figure 11: (a) Warm-up initialized images on Image Woof with DDiF, (b) Best-performed synthetic dataset represented by DDiF. We visualize the first 10 images, while DDiF constructs 51 images per class under the same budget.
29

---

Published as a conference paper at ICLR 2025
(a) Initialization
(b) Synthesized
Figure 12: (a) Warm-up initialized images ImageFruit with DDiF, (b) Bert-performed synthetic dataset represented by DDiF. We visualize the first 10 images, while DDiF constructs 51 images per class under the same budget.
(a) Initialization
(b) Synthesized
Figure 13: (a) Warm-up initialized images on Image Yellow with DDiF, (b) Best-performed synthetic dataset represented by DDiF. We visualize the first 10 images, while DDiF constructs 51 images per class under the same budget.
30

---

Published as a conference paper at ICLR 2025
(a) Initialization
(b) Synthesized
Figure 14: (a) Warm-up initialized images Image with DDiF, (b) Best-performed synthetic dataset represented by DDiF. We visualize the first 10 images, while DDiF constructs 51 images per class under the same budget.
(a) Initialization
(b) Synthesized
Figure 15: (a) Warm-up initialized images on ImageSquawk with DDiF, (b) Best-performed syn- thetic dataset represented by DDiF. We visualize the first 10 images, while DDiF constructs 51 images per class under the same budget.
31

---

Published as a conference paper at ICLR 2025
(a) Initialization
(b) Synthesized
Figure 16: (a) Warm-up initialization images and (b) Synthesized images of ModelNet-10. Start reading labels from the top and continue right: 1) bathtub, 2) bed, 3) chair, 4) desk, 5) dresser, 6) monitor, 7) nightstand, 8) sofa, 9) table, 10) toilet
(a) Initialization
(b) Synthesized
Figure 17: (a) Warm-up initialization images and (b) Synthesized images of MiniUCF. Start reading labels from the top: 1) FrihheeCatch, 2) HammerThrrow, 3) HulaHoop, 4) JumpingJack, 5) Parallel- Bars
32


Reviews:
[{"review_id": "sNXLIk2YPc", "reviewer": "Reviewer_Rr6g", "summary": "This paper proposes a parameterization-based dataset distillation method called DDIF. The authors enhance traditional parameterization-based methods by introducing a neural field representation. According to the authors, their contributions lie in the theoretical analysis of neural fields for parameterized dataset distillation and the performance gains achieved by their proposed method. The experiments conducted on ImageNet subset and cifar10/100 demonstrates the outstanding performance of DDiF as stated in this paper. Besides the performance, the authors claim that the advantage of their DDiF is the coding efficiency introduced by their neural field representation. Additionally, DDiF is well-suited to datasets with varying resolutions due to its resolution-invariant nature.  The proposed DDiF has a good potential to boost the development of dataset distillation. However, there exists some unclear part of the proposed method.", "questions": "Most of my questions are listed in the weakness part. The other main concern is how to ensure a fair comparison across parameterized dataset distillation methods. As the decoder network will not be counted into the distillation budget, the architectures, number of parameters, and the training dataset will heavily influence the performance of parameterized dataset distillation. If we could not constrain the decoder network well, the easiest way to boost performance would be using a larger network, better architecture, and training the network with more datasets to boost its generalization ability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of leveraging neural field for dataset distillation is innovative. \n\n2. The performance of the parameterized dataset distillation is unexpectedly high in the extreme scenario with IPC = 1.\n\n3. The authors conduct a lot of experiments including cross-architecture experiments and abaltion studies on different distillation loss functions. Also, the performance of DDiF is outstranding.", "weaknesses": "I believe this paper has the potential to be a milestone in dataset distillation. However, it is difficult to follow, which raises several concerns about the proposed method.\n\n\n1. The paper is not easy to understand:\n\n(a) The algorithm 1 does not tell the most significant improvement introduced by the DDiF, as it is very close to a standard parameterization-based method. What part makes the DDiF outstanding? The design of the decoder network? or the Eq(2).\n\n(b) I cannot understand the coding efficiency mentioned in line 188. While I recognize the resolution-invariant nature, I see no evidence supporting an advantage in coding efficiency. I hope the authors could provide numerical results here instead of rewriting the paragraph from line 188 to 204. \n\n(c) This paper is 27 pages long, yet the authors should provide more details about their experiments. Important settings are missing, such as the calculation of the distillation budget, the architecture of the decoder network, the number of parameters, the training datasets, and specific training details. Additionally, the evaluation metrics for the reported performance are unclear—how many times were the evaluation experiments repeated?\n\n(d) How should I interpret the sentence in line 370: ‘We utilize trajectory matching (TM) for 128 resolution and distribution matching (DM) for 256 resolution’? Why is the performance at 256 resolution significantly lower than at 128 resolution?\n\n\n2. What is the purpose to repeat Theorem 1(Novello, 2022) and theorem 2 here? I think the reader would like to have an explanation on the outstanding performance of the neural field, instead of a relatively correct theorem and proof. \n\n3. One critical issue with parameterized dataset distillation is the neglect of the decoder network. Only the encoded patterns are included in the distillation budget, while the decoder network is excluded. That explains the greater advantages of parameterized dataset distillation with IPC = 1, due to the marginal information carried by the decoder network. Such advantages will be smaller when ipc increases. \n\n4 Followed by question 3, why did the authors neglect the experiments with IPC=50? Is it due to the performance degradation of parameterized dataset distillation in higher IPC?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LLhNS1fgmx", "reviewer": "Reviewer_Y1Uh", "summary": "- This paper proposed using Neural Fields to parameterize distilled datasets (DDiF)\n- The basic idea is to replace each distilled image by a neural field which implicitly defines the image (by evaluating the field at grid points)\n- The method gets performance gains by reducing the parameter count per images (provided each neural field is small enough), so you can you use more images\n- The method is robust to different dataset distillation loss choices", "questions": "- See weaknesses\n- For the resolution generalization experiments, can you additionally report the performance of downsampling the full dataset down to the reduced distilled dataset resolution (as opposed to upsampling the distilled datasets, as you have presented). If there is better performance by downsampling the original data, then there would not be a convincing argument why one should upsample the distilled dataset\n- Is it not possible to upsample FreD distilled images by just evaluating at more points (like with equation 6)? Did the authors use this method for the upsampling experiments? If not, then this would be very important to make the comparison fair.\n- Can you also report for the performance of vanilla dataset distillation performance, with the same number of distilled images in table 1,2,3? If I understand correctly, right now you only present the performance matching the number of parameters, but it would a useful datapoint to consider what performance hit is given by parameterizing with DDiF as opposed to directly parameterizing the images\n- Is it possible to produce a curve showing the effect of neural field architecture choices on the performance? For example, given a fixed parameter budget, how does the width of the neural field (for example), affect the distillation performance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method is straightforward and is easily compatible with existing DD algorithms.\n- Good experimental results, particularly on non-image domains (with high dimensionality, where this method would have the greatest wins)\n- Interesting results with cross-resolution generalization, which is a new idea in dataset distillation (although I have questions about the evaluation - see the questions section)", "weaknesses": "- I don't think the theoretical analysis is particularly enlightening. The theoretical analysis can be boiled down to \"if you have more degrees of freedom, then you can get lower distillation loss, and DDiF has more degrees of freedom than FreD since DDiF can modify the frequencies as well as their coefficients, whereas FreD can only modify the coefficients\". However, this theoretical analysis is rather meaningless as adding the ability to control the frequencies (and the biases) adds more parameters. This analysis is only useful if you can fix the number of free parameters between FreD and DDiF. I understand this is analysis is difficult and can really only be verified experimentally (which the authors have done), but this caveat with the theoretical analysis needs to be mentioned.\n- There is no mention about the additional cost of evaluating the neural field at the coordinate points. The additional cost of evaluating the field at all datapoints could very significant, especially for high-resolution/higher dimensional data, but is not discussed in the paper. It would be useful to compare the additional runtime requirements, compared to directly parameterizing the images/data themselves, particularly for the high-resolution data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HmljHlO3az", "reviewer": "Reviewer_e7GQ", "summary": "This paper proposes to use a neural field framework to encapsulate and learn a dataset, i.e. dataset distillation. The neural field is prompted by coordinates to generate images, and can produce at different resolutions. The experiments show very promising results.", "questions": "See above. I think the paper is very interesting. If the authors addresses my questions above, I'd like to consider further recommendation of this paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ This paper proposes a very interesting extension of parameterization for dataset distillation, and uses a neural field to compress information.\n+ The experimental results show very promising performance on challenging datasets\n+ The generalization on different resolution capability is intriguing", "weaknesses": "- The paper's theme is on parameterization of image datasets, yet the introduction section rarely talks about the relevant topic and works to distinguish this paper out. I think the authors should specifically discuss and compare with [1, 2, 3, others] principal-wise to help the reader understand how the progress is made in DD parameterization and the delta in this paper.\n\n- In the experiment section, I'd encourage the authors to add a table row showing the number of images (not ipc budget) used for re-training. Reparameterization methods tend to have more images, which I think is fine, but needs to be explicitly mentioned for clarity.\n\n- One critical point in the experiment: is neural field better than simple basis decomposition used in RTP[1]? In the previous literature, the bases-coefficient decomposition seems to outperform HaBa quite a lot on simpler datasets. It should not be too hard to use RTP's decomposition method (bases + coefficient) and a DD loss that this paper is using (e.g., trajectory matching loss), and run experiments on the ImageNet benchmarks to perform a direct comparison. This will provide more support that neural fields can work very well and outperform the previous SOTA parameterization.\n\n- How heavy is the data augmentation?\n\n[1] Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks\n\n[2] Dataset Distillation via Factorization\n\n[3] Farzi Data: Autoregressive Data Distillation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7OWL575QRm", "reviewer": "Reviewer_XRWD", "summary": "A neural field-based parameterisation learned to encode the synthetic dataset for the downstream model training termed Distill Dataset into Neural Field (DDiF), is proposed in this submission. Comparied with the line of research that directly stores the synthetic dataset, a few neural fields parameterised by the neural networks are learned to yield pixel values with feed-in coordinates in the inference time, following the principle from FreD. DDiF is empirically tested on dataset condensation benchmarks especially in the image per category 1 and 10 setting on ImageNet with noticeable improvements.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea is simple to follow and easy to implement. \n2. The experiment results demonstrate the efficiency of neural field embedding but some concerns may be raised that I will point out later. \n3. The writing is clear and easy to follow.", "weaknesses": "1. The theoretical analysis concludes that DDiF has lower synthetic data loss than Fred due to the former feasible space being larger in Theorem2. The main reasons are based on Observations 1 and 2. However, there is no clear clue about the source of the observation. \n\n2. The budget comparison is not given. Since the DDiF architecture details are not clear, there is no direct comparison of the cost of saving the neural field and the synthetic dataset. \n\n3. I'm not sure the algorithm can be generalised to large IPC settings since in most experiments, there are only 1 and 10 IPCs, even for Cifar10 and CIf100 in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)