Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new parameterization (HaBa) of image data for dataset condensation. The proposed method decomposes a dataset into two components: data hallucination networks and bases, and considers the relationship between different condensed data points. Experiments show that HaBa achieves a better compression rate and better cross-architecture generalization performance.
strengths_and_weaknesses: This paper is well-motivated and focuses on an essential problem (the parameterization of the condensed data) for current dataset condensation methods. The proposed method is novel, and the results are encouraging.

- Originality: Decomposing the condensed data to bases and hallucinators is novel and interesting. Related work focusing on the parameterization of the condensed data: https://arxiv.org/abs/2205.14959, https://arxiv.org/abs/2206.02916.
- Quality: Most arguments are well-supported, and the experiments demonstrate the effectiveness of the proposed method. Some additional ablation studies and experiments are needed to make the paper more convincing.
- Clarity: This paper is well-written and easy to follow, though the position of some figures and tables can disrupt the reading experience. Some training details can be added to the appendix.
- Significance: The experiments demonstrate the effectiveness of the proposed method and can inspire some future work on the parameterization of the condensed data.
questions: 3.1 Basis and Hallucinator:
- I am curious about the functionality of the encoder of the Hallucinators. Is it necessary? What will happen if you remove the encoder part and only train the decoder and corresponding style vector?
- I wonder if it would be more parameter-efficient to learn multiple style vector pairs ( $\sigma$ , $\mu$ ) for a particular encoder and decoder pair.
- I guess the weights can learn more information than the fixed basis. Why do you only use 1 Conv-ReLu block for Hallucinators? What will happen if you consider more layers in the encoder and decoder structure? Why not consider increasing the capacity of the hallucinator by expanding its width?

3.2 Adversarial Contrastive Constrain
- Why do you want a contrastive objective for the feature extractor and a cosine similarity for the hallucinator? Could you have just one objective (e.g., cosine objective) for both the feature extractor and the hallucinator (the feature extractor maximizes it, and the hallucinator minimizes it)?

4.1 Datasets and Implementing Details
- How long does the training take with the new parameterization compared to baselines? Is the optimization easier or harder with the new parameterization?
- I can imagine that it is possible that some combination of hallucinator and basis may not be appropriate and introduce bias into the downstream classifier. For example, the dog's style may not be suitable for a ship. Suppose you train the downstream classifier on a ship instance with dog style. Would the classifier become confused at test time? What is your opinion on that?
- Because the proposed methods have so many hyperparameters, I suggest adding a table summary of the hyperparameters used for each experiment in the appendix.

4.3 Ablation Studies
- Cross-Architecture Performance: Lack of explanation. Why does it improve the cross-architecture performance? What is the insight behind it? What design choice gives this nice property?
- Cross-Domain Performance: Maybe change the title to "robustness to corruption"? I am unclear why "diverse training data" can give this nice property.
- I would like to see some ablation studies on what will happen if we increase the hallucinator capacity (increase depth or width).

Misc
- Line 63: "Such an adversarial scheme, in turn, enforces the hallucinators to produce diversified images and recover original data distribution thoroughly." The argument that "recover original data distribution" appears many times in the paper. I don't think dataset condensation methods aim to recover the data distribution. They are mining for informative examples instead. How come the condensed data recovers the original data distribution? Visually, the condensed data looks very different from the real data. What is your opinion on this? I would be more comfortable with an argument like "capture the discriminative feature better."
- Line 89: "supervision signals from downstream tasks, e.g., cross-entropy loss for classification, are not rigorous enough for synthetic samples." What do you mean by "rigorous enough"? What do you consider a rigorous supervision signal?
- Line 211: Typo. 10, 100, and 200 -> 10, 100, and 100
- Figure 4 and Table 1 are far away from their text, which disrupts the reading flow.
- Table 4 is confusing since it includes two different comparisons. Splitting into two tables or moving one to the appendix seems better.

I would be happy to increase my score if my questions can be addressed appropriately.
limitations: The authors may want to discuss the limitations of their methods. Here are some of my incomplete thoughts.
- There are so many hyperparameters in the proposed method that may impede its practical usage.
- The parameterization only works for image data but may want to consider other modalities.
- When scaling the method to many bases and hallucinators, there may be some optimization issues.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper investigates different ways to parametrize the synthetic learned/distilled dataset in the Dataset Distillation [1]. In particular, they propose to reparametrize the dataset as $\{G_j(z_i)\}_{ij}$, where $z_i$'s are "basis" (i.e., latents in usual generative modelling terminology), and $G_i$'s are the "hallucinators" (i.e., generators). (I took liberty to use variable $z$ rather than the $\hat{x}$ in paper, for better clarity). 

This reparametrization can be essentially applied with any Dataset Distillation method. In particular, the authors show that combining it with MTT (the SOTA Datasett Distillation work), can learn better distilled datasets **with same total number of parameters but with a much larger dataset size**. In fact, the performance gain mostly come from the increased dataset size (see below).


[1] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.
strengths_and_weaknesses: I will first discuss what I believe are two important issues of the current paper, and then provide a list of strengths and weaknesses.

### **Parameter count and dataset size**

It is important to note that
+ The method essentially is a **compression** of the distilled dataset in terms of #parameters, but **not in** dataset size, the usual quantity of interest in Dataset Distillation. The authors unfortunately do not make this clear in most cases when claiming improvements, which in reality comes at a cost of larger distilled dataset size.
+ It is unclear why #parameters should be a meaningful metric for Dataset Distillation. The paper does not provide arguments for it, or compare with any image compression methods.
+ When comparing with same **distilled dataset size**, the proposed approach often obtains slightly inferior performance (than the original pixel parameterization). This runs directly in contrary of the authors' claim that optimizing pixels directly is difficult for learning relations between samples (lines 29-33,43-44,121-124,329-331).

### **Improper credit assignment in writing**

I believe that it is greatly damages the integrity and openness of the academic community to 
+ Refer to this task as Dataset Condensation (DC) [2] rather than Dataset Distillation (DD) [1] 
+ Write as if the DC paper [2] introduces this task (entire paper, esp. introduction and abstract; DD is only mentioned once in related work and as a row in results table), 

when
+ Both DD [1] and DC [2] talk about the same task, with DD paper predating DC paper for **2 years**, DC paper citing DD paper and acknowledging that they investigate the same task,
+ Neither DD paper nor DC paper explicitly gives a name to the task, but calls their proposed method DD or DC, and seems to refer to the task as DD or DC. 

It is **extremely misleading** to people not familiar with this area, and essentially assigns credit in an obviously wrong way. I sincerely hope the authors did not do this purposefully. Admittedly, some other papers do the same thing, but it is no excuse to keep doing the harm. I strongly urge the authors to revise the paper in this aspect.

### **List of Strengths and Weaknesses**

**Strengths:**
+ The authors show that, by reparametrization with deep generators in Dataset Distillation training, it is possible to further compress a distilled dataset into fewer parameters, without losing much performance. 
+ Therefore, with the same parameter count, the parametrization achieves better Dataset Distillation performance (at the cost of larger distilled datasets). If parameter count turns out a useful metric in future, this can potentially have better use cases, after validating its benefits against other image compression techniques.

**Weaknesses:**
+ ~The stated motivation for proposed reparametrization is that (original) pixel parametrization don't learn relations. This claim is not immediately clear why it is true, and is not verified. In fact, results in paper show that parametrization does not help when the dataset size is kept the same.~
+ ~The motivating example in Section 1 is misleading.~
+ The paper mentions reduced storage cost, but it is unclear why it should matter when the distilled dataset often only contains tens or hundreds of images.
+ ~Writing is somewhat misleading in that whenever performance improvement is mentioned, the increased dataset size is almost never mentioned.~
+ ~No comparison with standard image compression techniques.~
+ Missing comparison with other DD parametrization work.

Please see my questions and suggestions below.

[1] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.

[2] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929, 2020.

---- 
The author's comment addressed most of my concerns (striked through text above). I have adjusted my score accordingly.
questions: 1. **Why should we care about lower parameter count? Comparison with standard image compression techniques?**

   Dataset Distillation (DD)'s scientific inquiry and practical applications both rely on the smaller distilled dataset size. Are there any applications that strongly relies on parameter counts? Indeed, smaller storage cost is always better, but I'm not sure if it matters that much when the distilled dataset often just contain tens or hundreds of images. Suppose that we care, then the authors should compare with regular compression techniques, including lossless and lossy ones, image-specific and generic ones.

   As an example, a simple lossless GIF compression of [a publicly available image from the distilled dataset of CIFAR-10 trained using MTT](https://georgecazenavette.github.io/mtt-distillation/images/cifar10_zca_10/airplane_4.png) is just 1923 bytes (with meta info!), versus the raw 3072 bytes.  It should be easy to compress much further, and essentially get much smaller parameter count.

2. **Given the importance of distilled dataset size in Dataset Distillation research, the paper should also compare methods at the same dataset size.**

   E.g., the proposed method at **BPC(bases-per-class)=10** has **IPC(images-per-class)=50**, but is only compared with other methods run at **IPC=11**. In fact, when fixing the dataset size, the proposed method is often slightly inferior (e.g., BPC=10 vs IPC=50).  

   The current comparison is done at the same parameter count, which is only meaningful if the emphasis on parameter count cannot be sufficiently justified (1st question). In either case, same dataset size (IPC) comparison should be done.

3. **The stated motivation is not justified or verified.** 

   The motivation for the proposed bias-hallucinator parameterization  is that the original pixel parametrization don't learn relations among samples. (lines 29-33)

   1. What exactly is "relations" referring to here? 

   2. The pixel parametrization is the most flexible and can represent any images. Admittedly, different parametrizations lead to different inductive biases / implicit regularizations. Is it correct to understand the claim as a hypothesis that the bias-hallucinator parameterization has better inductive bias to learn better distilled datasets due to sharing the same hallucinator across biases?  

   If so, the hypothesis seems false from the experiment results (see 2nd question above).  In other words, the stated performance gain mostly come from the increased distilled dataset size, rather than from the proposed parametrization.

3. **Motivating example in Section 1 is misleading.**

   It is a comparison among **(1)** proposed parameterization **(2)** original pixel parametrization w/ same #parameters and thus much less \#images **(3)** same as (2), but combining images from different checkpoints to form a larger distilled dataset.

   (3) is a really bizarre and misleading choice because the original parametrization is perfectly capable of learning a larger distilled dataset, and (3) is not really reducing any \#parameter or storage cost either. In fact, the later experiment results show that simply using original parametrization to distill a larger set have similar performance as (1). This makes the choice of (3) quite suspicious.

3. **This is essentially generative modelling with a discrete set of latents. Why use image-shaped "basis" rather than generic vectors?**

   It is not really wrong, but a somewhat weird choice given the common practice of using generic latent vectors. An explanation would be better to have.

4. **Please emphasize somewhere early in the paper that the increased dataset size is needed to achieve good performance.**

   A reader should not have to reach experiment section and to do the arithmetic to realize that the proposed technique comes at a cost.

5. **Please reconsider the choice of using DC to refer to the task and writing the paper as if DC introduced the task (see first section).**

6. The paper have some quite confusing places. Here are a few:


     1. Fig. 3 mentions a $\mathcal{L}_{cts.}$ that is not mentioned anywhere else.

     3.  " Constrain" => " Constraint" in multiple places.
limitations: I do not think that the limitation is sufficiently discussed. As I emphasized above, the authors should make it clear that the claimed improvements come with the cost of much larger distilled dataset size, the one metric that Dataset Distillation research cares about the most.  Furthermore, the added generator components should increase both training time and the training GPU memory cost. The training time effect is not really discussed other than a sentence in the checklist, which is not part of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel algorithm for dataset condensation and introduces a dataset factorization approach - HABA which frame the factorization as a hallucinator-basis problem. This paper also introduces a pair of adversarial contrastive constrains to increase the diversity of generated images and inject more discriminant information into the factorization.
strengths_and_weaknesses: ### Strengths
1. This paper is well written and easy to follow, and the presentation is clear.

2. The overall idea is novel and the experimental results are convincing.

### Weaknesses
I have to be honest that I'm not an expert in this area, but I have a few questions about this paper.

1. Since the author has mentioned that the early dataset compression methods are inspired by the knowledge distillation, I'm wondering if these compressed data can be used for KD methods.

2. Although this paper has performed the experiments across different architectures, however, I'm wondering about the effect of the model size on the final performance. (i.e. the performance gap between the compressed data with the whole data on ResNet18 will be greater or smaller than ResNet-101 ?)
questions: See Weaknesses.
limitations: adequately addressed
review_title: null

Meta-review: metareview: The reviewers originally had concerns but these have been well addressed by the authors in a thorough rebuttal and there is a consensus for acceptance. We encourage the authors to incorporate all the comments from the reviewers in the final version.
confidence: Less certain
Decision: Accept