Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper formulates the dataset distillation as a memory addressing process where the synthetic datasets can be generated through an addressing matrix based on a learned memory representation. The proposed method improves the compression rate of distilled data by considering the class similarity and improving the BPTT framework. The proposed method is evaluated on five image classification datasets and applied to continual learning and few-shot learning.
strengths_and_weaknesses: This paper is well-motivated and focuses on a significant problem (the parameterization of the distilled data) in dataset distillation. The proposed method is novel, and the results are encouraging.
- Originality: The proposed method proposes a new way to parameterize the distilled data and opens up a new application for few-shot learning.
- Quality: This paper is technically sound, and most of the arguments are well supported. The evaluation is not perfectly fair, and additional study is needed to understand the method better.
- Clarity: This paper is easy to follow and mostly well-written, with minor flaws.
- Significance: The proposed method achieves state-of-the-art performance when a fixed memory budget is given. The addressable memory parameterization can also be applied to other dataset distillation methods. This paper also shows that dataset distillation can benefit continual learning and few-shot learning.
questions: Table 1
- The comparison to previous methods is not perfectly fair, and "I/C" is misleading. Due to the parameterization, the proposed method can generate more synthetic data than previous methods. Therefore, the proposed method is expected to achieve higher accuracy than previous methods. Thus, the number of addressing matrices (or the exact number of synthesis examples) needs to be shown somewhere in the table. I want to see whether more distilled data can fully explain the improvement or if the distilled images also get better quality. Appendix Figures 4 and 5 show that the recalled images lack diversity. I am surprised that the method achieves such a good performance.

Table 2
- It may be better to clarify the abbreviation (i.e., ds, Aug) in the caption.

Figure 3
- You mention that unrolling the trajectories long enough (e.g., 200 steps) can improve performance (Line 202). Could you show a complete picture in Figure 3 up to 200 steps? Due to the long unroll of the BPTT, I would also like to see a plot of test accuracy versus training time and the memory requirement of the proposed method. How does it compare to other baseline methods?

Figure 4
- I am curious about the trade-off between the number of bases and addressing matrices given a fixed memory budget. What is the general rule to scale these two numbers? I want some plot where the x-axis is the memory budge, and the y-axis is the ratio between these two numbers.

Results 2
- I can see that long unrolls will be very important, but it is unclear why momentum can be so significant? What's the insight behind that?

Line 301
- "the buffer size keeps shrinking when more compressed representation of tasks is stored": does it imply that the later distilled data tend to have a low quality?

Misc
- What would happen if you use a soft label for query (as a data augmentation) during test time? Will the performance get improved?
- It will be interesting to see the visualization of the interpolation between two different classes. Does the "half dog half ship" image make sense?
- I want to see some visualization of the images generated by BPTT. How does it compare to the previous method?
limitations: Due to the long unroll of the BPTT, the training efficiency and memory requirement can be the biggest concern when scaling this approach to complex datasets or larger models.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Based on the observation that classes can share concepts and representations, the authors propose a new Dataset Distillation method that trains a learnable memory shared across classes. The objective is to learn basic concepts that can be shared and then, through a function A, learn to combine these concepts to generate examples given a class y_i. To learn the method, the authors use a bi-level optimization process. The inner loop verified the generalization properties of the current memory, and the outer loop updated the memory and attention function. Additional to the method, the authors propose a modification in the momentum factor during the inner loop that helps improve performance. This method achieves good results in various Dataset Distillation benchmarks and good results in Continual Learning scenarios.
strengths_and_weaknesses: Strengths:
- S.1 The observation that concepts are shared across tasks is very intuitive. This observation, as the authors note, helps compress data so that it does not grow linearly as the number of classes increases, and may also lead to finding better concepts for classes that the model has not considered before.not seen.
- S.2 The paper is well written, placing much emphasis on the different contributions that the authors propose. The different concepts and notations used are well explained.
- S.3 Using the bi-level optimization strategy helps generate weights that can support the problem of continuous labels.
- S.4 The results show that the proposed method achieves outstanding results. The experiments that were carried out demonstrate the contributions proposed by the authors.

Weaknesses:
- W.1 The reason for adding the momentum update is not clear. Nor how it is done. Is there any intuition behind this contribution?
- W.2 Is there a reason why line 12 of Algorithm 1 does not use the opt function? It is used in equation 3, but not in the algorithm, which can cause confusion.
- W.3 It is known that benchmarks of the dataset distillation problem are standard and normally use only small images. However, many of the previous methods have problems when the size of the images increases. It would be interesting to confirm the performance of this method on larger images. Alternatively, add it to the limitations.
- W.4 There is no comparison of the algorithm's time or the number of iterations. Adding more iterations to the inner loop must come at a high cost.
- W.5 Typo: Line 110, the abbreviation BPTT is used before defining
questions: Q.1 In several works, the attention masks are learned independently of the model/memory weights, so that interference between the learning processes is avoided. Does it make sense to learn M and A separately? In some ways, this experiment seems helpful to ensure that the memory acquires basic concepts, and not that function A selects useful random vectors that can achieve a goal, like finding the lottery ticket from a memory.

Q.1.2 Perhaps the previous question is related to what happens in Fig. 2. For some classes, there is an intuitive and testable relationship of similarity. However, this similarity is not seen in other intuitively similar classes (cats and dogs). Is there any explanation for what is happening?

Q.3 One of the problems with using long paths in the inner loop is the amount of memory used to store the entire associated computation graph. Although there are techniques to mitigate this, it is not mentioned. Can this proposal be explained in more detail? Has this problem been encountered?

Q.4 It is unclear how the method is used in the Continual Learning environment. A single-pass context is mentioned. However, how can the proposed method be learned in just one data pass? Did the simple-pass apply only to the test adaptation? Or is the model capable of learning to compress information in a single epoch?
limitations: The authors mention a limitation that is very relevant and I share
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to compress the critical information of a large dataset into compact addressable memories. This method achieves better results than the previous dataset distillation/condensation methods.
strengths_and_weaknesses: Strengths.
1. The results are much better than previous methods.
2. The method is based on a reasonable assumption that there are redundancies in the previous condensed dataset. The method achieves a higher compression rate by reducing redundancies.
3. The paper is well organized and developed.

Weaknesses.
1. The experiments are only on small datasets. The size of the dataset $N$ and the dimension of input $d$ are small. The authors should report results on large datasets. Specifically, in [15], Tiny ImageNet and ImageNet subsets are used in the experiments.
2. The experiments on large models instead of small models are needed. Does the method depend on a specific neural architecture? The authors may investigate other neural networks.
3. Other missing experiments are mentioned in the questions below.
4. The writing is fair. Several typos and errors are listed below.

* Line 45. significanly -> significantly
* Line 50. lead to -> leads to
* Line 63. There has been -> There have been
* Line 64. criterions -> criteria
* Line 66. It emphasize on -> It emphasizes
* Line 70. gaussian processes -> Gaussian processes
* Line 79. dillema -> dilemma
* Lines 92, 100, 151. e.g. -> e.g.,
* Line 99. the number of synthetic data sample -> the number of synthetic data samples
* Line 115. seprately -> separately
* Line 121. outperforms -> outperform
* Line 146. defines -> define
* Line 170. having to enumerating -> having to enumerate
* Equation 3. J(\phi) -> min J(\phi)
* Line 179. discuss in detail about the learning framework -> discuss in detail the learning framework
* Line 212. there are strong evidence -> there is strong evidence
* Line 213. there are information re-using -> there is information re-using
* Line 223. with shape 28 × 28 -> with a shape of 28 × 28
* Line 228. are color image 228 dataset -> are color image 228 datasets
* Line 263. distinct to each other -> distinct from each other
* Line 292. 200 sample -> 200 samples
* Line 308. pervious -> previous
questions: 1. Algorithm 1. Lines 8 - 13 are the detailed optimization algorithm for $\theta$. In line 15, there is the optimization algorithm for $\phi$. What are the details of this optimization algorithm?
2. There is also a research direction that generative models are used to condense a dataset. This direction is closely related to this work and should be discussed in the paper.
3. In previous data distillation works, the number of inner-loop optimization steps is small. The major reason is that the memory consumption and execution time are quite large with a large trajectory. Could the authors discuss how they address this issue?
4. There is a trade-off between time and space. If we prefer a higher compression rate, we have to pay the cost of extra computation (e.g., the computation for decompression). Could the authors discuss this trade-off and make a comparison with previous work?
5. Instead of reporting the performance given a specified compression rate, it is also important that what is the compression rate if we can fully recover the original performance. Namely, what is the size of the smallest dataset that can be trained to achieve the full dataset performance?
6. The authors use the notation $M=${$b_0, ..., b_{K-1}$}. It can be simplified as $M \in R^{K \times d}$.
7. Is it possible to compress the condensed dataset directly? For instance, the previous methods can generate a condensed dataset. Could we apply several lossless compression on them? It is better for the authors to discuss it.
limitations: As a derived method of the dataset distillation framework, the proposed method inherits the limitations of the original method on the large models and large datasets. The authors also mention it in the last section.
review_title: null

Meta-review: metareview: This paper proposes a new dataset distillation method that achieves SotA results on several benchmarks. Authors were very responsive to answer reviewers' questions, and made significant improvements to the manuscript, also adding additional results confirming the benefits of their approach. At the end of the discussion period, there is a clear consensus for acceptance, due to the fact that this approach is original, well motivated and achieves strong results.

I thus recommend to accept the paper, even though some concerns remain regarding the scalability of the algorithm (in terms of memory usage and running time).
confidence: Certain
Decision: Accept