Distilling Dataset into Neural Field

Authors: Donghyeok Shin, HeeSun Bae, Gyuwon Sim, Wanmo Kang, Il-chul Moon
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Dataset distillation, Dataset condensation, Neural field

Abstract:
Utilizing a large-scale dataset is essential for training high-performance deep learning models, but it also comes with substantial computation and storage costs. To overcome these challenges, dataset distillation has emerged as a promising solution by compressing the large-scale dataset into a smaller synthetic dataset that retains the essential information needed for training. This paper proposes a novel parameterization framework for dataset distillation, coined Distilling Dataset into Neural Field (DDiF), which leverages the neural field to store the necessary information of the large-scale dataset. Due to the unique nature of the neural field, which takes coordinates as input and output quantity, DDiF effectively preserves the information and easily generates various shapes of data. We theoretically confirm that DDiF exhibits greater expressiveness than some previous literature when the utilized budget for a single synthetic instance is the same. Through extensive experiments, we demonstrate that DDiF achieves superior performance on several benchmark datasets, extending beyond the image domain to include video, audio, and 3D voxel. We release the code at \url{https://github.com/aailab-kaist/DDiF}.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730288298157

thread_type:
official_review

<thread_object>
creator: reviewer_xrwd

<content>
summary:
A neural field-based parameterisation learned to encode the synthetic dataset for the downstream model training termed Distill Dataset into Neural Field (DDiF), is proposed in this submission. Comparied with the line of research that directly stores the synthetic dataset, a few neural fields parameterised by the neural networks are learned to yield pixel values with feed-in coordinates in the inference time, following the principle from FreD. DDiF is empirically tested on dataset condensation benchmarks especially in the image per category 1 and 10 setting on ImageNet with noticeable improvements.

strengths:
1. The idea is simple to follow and easy to implement. 
2. The experiment results demonstrate the efficiency of neural field embedding but some concerns may be raised that I will point out later. 
3. The writing is clear and easy to follow.

weaknesses:
1. The theoretical analysis concludes that DDiF has lower synthetic data loss than Fred due to the former feasible space being larger in Theorem2. The main reasons are based on Observations 1 and 2. However, there is no clear clue about the source of the observation. 

2. The budget comparison is not given. Since the DDiF architecture details are not clear, there is no direct comparison of the cost of saving the neural field and the synthetic dataset. 

3. I'm not sure the algorithm can be generalised to large IPC settings since in most experiments, there are only 1 and 10 IPCs, even for Cifar10 and CIf100 in the appendix.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730333546603

thread_type:
official_review

<thread_object>
creator: reviewer_e7gq

<content>
summary:
This paper proposes to use a neural field framework to encapsulate and learn a dataset, i.e. dataset distillation. The neural field is prompted by coordinates to generate images, and can produce at different resolutions. The experiments show very promising results.

strengths:
+ This paper proposes a very interesting extension of parameterization for dataset distillation, and uses a neural field to compress information.
+ The experimental results show very promising performance on challenging datasets
+ The generalization on different resolution capability is intriguing

weaknesses:
- The paper's theme is on parameterization of image datasets, yet the introduction section rarely talks about the relevant topic and works to distinguish this paper out. I think the authors should specifically discuss and compare with [1, 2, 3, others] principal-wise to help the reader understand how the progress is made in DD parameterization and the delta in this paper.

- In the experiment section, I'd encourage the authors to add a table row showing the number of images (not ipc budget) used for re-training. Reparameterization methods tend to have more images, which I think is fine, but needs to be explicitly mentioned for clarity.

- One critical point in the experiment: is neural field better than simple basis decomposition used in RTP[1]? In the previous literature, the bases-coefficient decomposition seems to outperform HaBa quite a lot on simpler datasets. It should not be too hard to use RTP's decomposition method (bases + coefficient) and a DD loss that this paper is using (e.g., trajectory matching loss), and run experiments on the ImageNet benchmarks to perform a direct comparison. This will provide more support that neural fields can work very well and outperform the previous SOTA parameterization.

- How heavy is the data augmentation?

[1] Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks

[2] Dataset Distillation via Factorization

[3] Farzi Data: Autoregressive Data Distillation

questions:
See above. I think the paper is very interesting. If the authors addresses my questions above, I'd like to consider further recommendation of this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660053011

thread_type:
official_review

<thread_object>
creator: reviewer_y1uh

<content>
summary:
- This paper proposed using Neural Fields to parameterize distilled datasets (DDiF)
- The basic idea is to replace each distilled image by a neural field which implicitly defines the image (by evaluating the field at grid points)
- The method gets performance gains by reducing the parameter count per images (provided each neural field is small enough), so you can you use more images
- The method is robust to different dataset distillation loss choices

strengths:
- The method is straightforward and is easily compatible with existing DD algorithms.
- Good experimental results, particularly on non-image domains (with high dimensionality, where this method would have the greatest wins)
- Interesting results with cross-resolution generalization, which is a new idea in dataset distillation (although I have questions about the evaluation - see the questions section)

weaknesses:
- I don't think the theoretical analysis is particularly enlightening. The theoretical analysis can be boiled down to "if you have more degrees of freedom, then you can get lower distillation loss, and DDiF has more degrees of freedom than FreD since DDiF can modify the frequencies as well as their coefficients, whereas FreD can only modify the coefficients". However, this theoretical analysis is rather meaningless as adding the ability to control the frequencies (and the biases) adds more parameters. This analysis is only useful if you can fix the number of free parameters between FreD and DDiF. I understand this is analysis is difficult and can really only be verified experimentally (which the authors have done), but this caveat with the theoretical analysis needs to be mentioned.
- There is no mention about the additional cost of evaluating the neural field at the coordinate points. The additional cost of evaluating the field at all datapoints could very significant, especially for high-resolution/higher dimensional data, but is not discussed in the paper. It would be useful to compare the additional runtime requirements, compared to directly parameterizing the images/data themselves, particularly for the high-resolution data.

questions:
- See weaknesses
- For the resolution generalization experiments, can you additionally report the performance of downsampling the full dataset down to the reduced distilled dataset resolution (as opposed to upsampling the distilled datasets, as you have presented). If there is better performance by downsampling the original data, then there would not be a convincing argument why one should upsample the distilled dataset
- Is it not possible to upsample FreD distilled images by just evaluating at more points (like with equation 6)? Did the authors use this method for the upsampling experiments? If not, then this would be very important to make the comparison fair.
- Can you also report for the performance of vanilla dataset distillation performance, with the same number of distilled images in table 1,2,3? If I understand correctly, right now you only present the performance matching the number of parameters, but it would a useful datapoint to consider what performance hit is given by parameterizing with DDiF as opposed to directly parameterizing the images
- Is it possible to produce a curve showing the effect of neural field architecture choices on the performance? For example, given a fixed parameter budget, how does the width of the neural field (for example), affect the distillation performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693946137

thread_type:
official_review

<thread_object>
creator: reviewer_rr6g

<content>
summary:
This paper proposes a parameterization-based dataset distillation method called DDIF. The authors enhance traditional parameterization-based methods by introducing a neural field representation. According to the authors, their contributions lie in the theoretical analysis of neural fields for parameterized dataset distillation and the performance gains achieved by their proposed method. The experiments conducted on ImageNet subset and cifar10/100 demonstrates the outstanding performance of DDiF as stated in this paper. Besides the performance, the authors claim that the advantage of their DDiF is the coding efficiency introduced by their neural field representation. Additionally, DDiF is well-suited to datasets with varying resolutions due to its resolution-invariant nature.  The proposed DDiF has a good potential to boost the development of dataset distillation. However, there exists some unclear part of the proposed method.

strengths:
1. The idea of leveraging neural field for dataset distillation is innovative. 

2. The performance of the parameterized dataset distillation is unexpectedly high in the extreme scenario with IPC = 1.

3. The authors conduct a lot of experiments including cross-architecture experiments and abaltion studies on different distillation loss functions. Also, the performance of DDiF is outstranding.

weaknesses:
I believe this paper has the potential to be a milestone in dataset distillation. However, it is difficult to follow, which raises several concerns about the proposed method.


1. The paper is not easy to understand:

(a) The algorithm 1 does not tell the most significant improvement introduced by the DDiF, as it is very close to a standard parameterization-based method. What part makes the DDiF outstanding? The design of the decoder network? or the Eq(2).

(b) I cannot understand the coding efficiency mentioned in line 188. While I recognize the resolution-invariant nature, I see no evidence supporting an advantage in coding efficiency. I hope the authors could provide numerical results here instead of rewriting the paragraph from line 188 to 204. 

(c) This paper is 27 pages long, yet the authors should provide more details about their experiments. Important settings are missing, such as the calculation of the distillation budget, the architecture of the decoder network, the number of parameters, the training datasets, and specific training details. Additionally, the evaluation metrics for the reported performance are unclear—how many times were the evaluation experiments repeated?

(d) How should I interpret the sentence in line 370: ‘We utilize trajectory matching (TM) for 128 resolution and distribution matching (DM) for 256 resolution’? Why is the performance at 256 resolution significantly lower than at 128 resolution?


2. What is the purpose to repeat Theorem 1(Novello, 2022) and theorem 2 here? I think the reader would like to have an explanation on the outstanding performance of the neural field, instead of a relatively correct theorem and proof. 

3. One critical issue with parameterized dataset distillation is the neglect of the decoder network. Only the encoded patterns are included in the distillation budget, while the decoder network is excluded. That explains the greater advantages of parameterized dataset distillation with IPC = 1, due to the marginal information carried by the decoder network. Such advantages will be smaller when ipc increases. 

4 Followed by question 3, why did the authors neglect the experiments with IPC=50? Is it due to the performance degradation of parameterized dataset distillation in higher IPC?

questions:
Most of my questions are listed in the weakness part. The other main concern is how to ensure a fair comparison across parameterized dataset distillation methods. As the decoder network will not be counted into the distillation budget, the architectures, number of parameters, and the training dataset will heavily influence the performance of parameterized dataset distillation. If we could not constrain the decoder network well, the easiest way to boost performance would be using a larger network, better architecture, and training the network with more datasets to boost its generalization ability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732301806043

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response to Reviewers

comment:
Dear Reviewers:

We would like to express sincere gratitude for the constructive comments on our work and thank you so much for your time and effort. We provide individual responses to each reviewer's concerns and update the manuscript to reflect the reviewer's comments. We highlight the changes in purple. Please check our response and feel free to post additional comments. We are willing to provide further explanation on any unclear point.

Best regards.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733308115295

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Discussion Period

comment:
Dear AC and Reviewers,

We sincerely appreciate the time, effort, and dedication you have devoted to reviewing our manuscript. The reviewers' comprehensive feedback has guided us in enhancing the clarity and quality of our research. We are grateful for the recognition from the reviewers for the **novelty** (`Rr6g`, `e7GQ`), **simplicity and ease of compatibility** (`Y1Uh`, `XRWD`), **high performance in extensive experiments including non-image domains** (`Rr6g`, `Y1Uh`, `e7GQ`), and **presenting a new experimental design**, cross-resolution generalization (`Y1Uh`, `e7GQ`) in our work.

The reviewers also raised some concerns, and we have attempted to address these concerns as thoroughly as possible. Below, we summarize the raised key concerns and our responses.

* Clarification of budget usage
    * **[Budget calculation]** The reviewers pointed out that the budget calculation is not given (`Rr6g`, `XRWD`). To ensure a fair comparison, we configured the hyperparameters in all experimental settings such that the utilized budget did not exceed that of input-sized parameterization. The formula for budget calculation and the corresponding hyperparameter values were already included in the initial manuscript. During the discussion period, we provided additional explanations and detailed calculation results.
    * **[Number of decoded instances]** The reviewers suggested specifying the number of decoded instances for the proposed method (`e7GQ`, `XRWD`). The proposed method generates more decoded instances within the same budget due to its high coding efficiency. During the discussion period, we specified the exact increment in the number of decoded instances achieved by the proposed method.
* Additional Experiments
    * **[Large IPC setting]** The reviewer inquired whether the proposed method performs well even in large IPC setting (`Rr6g`, `XRWD`). We demonstrated that the proposed method is sufficiently effective even in large IPC setting through performance comparisons across various datasets. Furthermore, we showed that applying the soft label technique, which is used in the state-of-the-art methods (RDED, DATM), allows the proposed method to achieve higher performance than SOTA methods with a significant margin (`XRWD`).
    * **[Fixed number of decoded instances]** To support the high coding efficiency (`Rr6g`) and expressiveness (`Y1Uh`) of the proposed method, and to demonstrate that the performance improvement is not solely attributed to the increase in the number of decoded instances (`XRWD`), we additionally conducted the experiments under the fixed number of decoded instances. In this setting, the proposed method demonstrated highly competitive or even superior performance while using significantly less budget compared to input-sized and previous parameterization methods (FreD, SPEED). 
    * **[Additional baselines in cross-resolution generalization]** The reviewer requested a comparison between downsampling of the full dataset and frequency-domain upsampling in the cross-resolution generalization (`Y1Uh`). We conducted additional experiments and demonstrated that the proposed method still exhibits strong cross-resolution generalization.
    * **[Comparison with bases-coefficient decomposition]** The reviewer requested a performance comparison with the based-coefficient decomposition method (RTP) (`e7GQ`). We have already included the performance comparison for CIFAR-10 and CIFAR-100 in the initial manuscript, and we explained that the proposed method achieves high performance in the ImageNet-Subset (128) experiments we additionally conducted.
* Theoretical analysis
    * **[Additional explanation]** We provided additional explanations regarding the objectives, significance, and contributions of the theoretical analysis. (`Rr6g`, `Y1Uh`)
    * **[Empirical evidence]** We provided additional experimental evidence to support the theoretical analysis. (`Y1Uh`, `XRWD`)
* Further explanation
    * We provided further clarification regarding the experimental setting (`Rr6g`, `e7GQ`, `XRWD`), algorithm (`Rr6g`), computational cost (`Y1Uh`), ablation study (`Y1Uh`), comparison with previous studies (`e7GQ`), and cost of data augmentation (`e7GQ`).

We have incorporated the points raised by the reviewer in the first round into the revised manuscript, which has been uploaded, and the comments mentioned in the second round will be reflected in the final manuscript. We hope that the summary above helps provide a deeper understanding of the proposed method and addresses the reviewers’ concerns.

Once again, we sincerely express our gratitude to the AC and all reviewers.

Best Regards,
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel parameterization-based dataset distillation (DD) approach by leveraging neural fields to factorize data. The proposed method offers a fresh perspective for extending DD beyond image data, effectively generalizing to diverse modalities such as video, audio, and 3D point clouds. The approach demonstrates impressive performance under high compression ratios, achieving notable improvements in various tasks. While its effectiveness under large IPC settings remains a limitation, this does not detract from the contributions the work makes to the dataset distillation community. Therefore, I recommend accepting this paper.