Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work improves the performance of low-shot learning by expanding the small-scale dataset by utilizing powerful generative models (i.e. decoders). To expand the dataset in a more meaningful way, this work proposes three conditions to ensure that the synthetic samples should be consistent with the real samples in zero-shot prediction perspective and diverse enough to improve the final performance of the models trained on the expanded dataset. Experiments on the proposed methods based on MAE and DALL-E 2 (without upscaling modules) show that this expansion policy indeed improves the classification performance in a low-shot setting. 
strength_and_weaknesses: **Strengths**:

* Expanding the dataset based on a powerful decoder is a very intuitive and interesting direction to few-shot learning. In addition, this work proposes three criteria on how to create effective synthetic samples. And, these have been empirically validated in several experiments.  
* The appendix helps me understand the details of the algorithm to reproduce the results. And, all experiments have been conducted on publicly available checkpoints, so the accessibility of this approach is quite high. 

**Weaknesses**: 

My biggest concern in this work is the low performance of the main experiments, shown in Table 1. Compared to the transfer learning or linear evaluation results from ALIGN and CLIP, the numbers in Table 1 aren’t convincing enough to argue that this kind of data expansion policy is practically useful than the standard protocol, i.e. training an encoder by SSL objectives then fine-tuning the encoder on small-scale datasets. 

I summarize the performance gap between this method and CLIP and ALIGN below: 

| Method | Caltech101 | Cars | Flowers | Pets | 
| ---------- | -------------- | -------------- | -------------- | -------------- | 
| GIF-MAE (proposed one) | 58.4 | 44.5 | 84.4 | 52.4 |
| GIF-DALLE2 (proposed one) | 63.0 | 53.1 | 88.2 | 66.4 | 
| CLIP (ViT-B/32) - linear eval.    | 93.0  | 81.8  | 96.9 | 90.0 |
| CLIP (ViT-L/14, 334px) - linear eval.    | 96.0 | 91.5 | 99.2 | 95.1 |
| ALIGN - transfer learning | - | 96.1 | 99.7 | 96.2 | 

ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision, https://arxiv.org/abs/2102.05918

**More questions**

Q1. In my understanding, the roles of the entropy maximization and diversity promotion seem to be very similar to each other. Could you elaborate a little bit more on the reason why both conditions are required instead of using the diversity promotion? 

Q2. In table 1, GIF-DALLE consistently performs better than GIF-MAE, though the resolution of the synthetic samples based on GIF-MAE is much higher than the ones on GIF-DALLE (224px vs. 64px.), as described in Appendix C.1. It would be helpful to describe the reason why GIF-DALLE performs better than the MAE in the revised manuscript. 
clarity,_quality,_novelty_and_reproducibility: In terms of clarity, this manuscript is generally well-written. It could be better to include some details of the main algorithm, but the appendix helps me a lot enough to implement the approach. 

In terms of reproducibility, all experiments have been conducted based on the publicly available checkpoints. And, the manuscript contains the hyperparameters and detailed experiments setups, so I believe that I wouldn’t be in trouble for reproducing the numbers in the tables. 

In terms of quality and novelty, I have some major issues on the final performance of this method. The numbers in Table 1 are much worse than the linear evaluation and fine-tuning results in CLIP and ALIGN papers. This makes me wonder if this kind of dataset expansion based on a generative model is really better than the standard SSL approaches. 
summary_of_the_review: I’m leaning towards rejection, since I’m not sure that the performance gain achieved by this work is really significant or not, compared to the simple recipes used in many SSL methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a Guided Imagination Framework (GIF) for data augmentation. The original small-set dataset can be expanded by generating new images from powerful generative models such as DALL-E or MAE. Compared with the non-parametric data augmentations, such generative-models-guided ones bring extra new information due to the nature of the models themselves. Diversity and fidelity are the two crucial factors for success. This paper proposed three criteria as guidance: prediction consistency, entropy maximization, and diversity promotion. GIF is verified on multiple natural or medical image datasets with a stable boost compared with the baselines.
strength_and_weaknesses: Pros:
+ The setting of this paper is new and interesting. A similar idea has been applied in the GAN models, which seems not to be working. The latest progress in diffusion models is significant. It has shown a great enhancement in the quality of synthetic images. Applying such generative models to help discriminative tasks is a natural and straightforward idea. To the best of my knowledge, this is a pioneering work in this area.

+ This paper has achieved very promising results over many benchmarks, with a 29.9% accuracy gain on average over six natural image datasets and a 10.4% accuracy gain on average over three medical ones. Such an improvement is significant compared with non-parametric data augmentations.

+ With the help of three criteria as guidance, there is a further accuracy gain due to the increase of new information in generated data. The proposed disturbance is simple and effective.

Cons:
- One of my major concerns is that all the evaluations are conducted over small datasets where MAE or DALL-E should easily cover such domains. I am not very surprised by the improvement in these datasets. I am wondering if this will be helpful for other tasks/domains on a larger scale. 

- There is no sufficient theoretical analysis in this paper. The whole paper is based on empirical studies with solid experiments whose quality can be further improved with more discussion of analysis in the theoretical aspect.

- Despite the expansion efficiency of GIF claimed in Sec. 5.1, I am doubtful about the time and cost efficiency due to the low speed and high computation of the diffusion model in inference.

Questions and Other Concerns:
1. I am wondering what is the open-source plan of this paper since the DALL-E is not an open-sourced model.

2. It would be good if the authors shared the time and computational costs needed for dataset expansion.

3. The images generated by MAE seem to have bad visual quality. How to explain the benefits of such noisy images for augmentation? Is it any possibility that they could be harmful to accuracy if the image quality is pretty bad and why?

clarity,_quality,_novelty_and_reproducibility: The quality, clarity and originality are good.
summary_of_the_review: Despite many concerns about this paper, I am still positive due to its novelty and the promising results in many small datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to tackle a new task of dataset expansion, which is to automatically find new labeled data to augment an existing dataset.
To do so, they propose to generate them using a generative model (DALL-E2) using CLIP guidance.
The idea is that using DALL-E2 would be able to generate variations around existing samples, and CLIP can be used to guide the generated samples to only generate useful data.
In addition to DALL-E2, they show they can also use Masked Auto-Encoders (MAE) as a backbone, with the same CLIP guidance.
They compare their method of expansion against standard data-augmentation techniques on a set of small natural image datasets, as well as medical image datasets.
strength_and_weaknesses: Strengths:
- The paper tackles a very relevant problem of training in a low-data regime. Using generative models for this purpose is an intuitive solution that shows promises but has so far been struggling to obtain significant success. The paper is a step in that direction.
- The diverse set of experiments provides good insights into the method. I found the results on medical images and the corresponding discussion in Appendix E regarding the limitations of GIF-DALLE especially important, and believe there should be more discussions in the main paper on that subject.

Weaknesses:
- It is not clear how GIF-MAE fits in the narrative.
  - In most sections the main focus is GIF-DALLE and the MAE variant is barely mentioned, almost like an afterthought. In the experiments, however, it takes a very significant portion.
The authors might want to fix this imbalance. It could be done by integrating GIF-MAE earlier in the motivation, or instead by treating it even more entirely as a side contribution in the later sections.
  - The sentence in the introduction "Considering that DALL-E2 and MAE (He et al., 2022) have been shown to be powerful in generating and reconstructing images, we explore their use as prior generative models for imagination in this work." seems particularly misleading. Can the authors clarify if they consider MAE a powerful image generator, and if so, in which sense?
- The framing of the results is useful but seems artificial. As such the presented results are insufficient to draw conclusions regarding the usefulness of the method in practice.
  - Comparing their methods against standard data augmentations with a fixed number of samples is fine, but it would be useful to have the performance of the baselines with a virtually infinite number of augmented samples.
Indeed, a key difference between the two approaches is that, in practice, data generation has significant memory and a computation footprint while it should be very easy to add infinite standard data augmentation in the training pipeline with little cost.
  - This comparison might also not be fair as the GIF-DALLE has had access to Laion-400M. Another interesting baseline would have been to simply automatically select and label additional unlabelled data using CLIP embeddings. This could also serve as some sort of ablation study to check if the generator is actually useful, or if directly using the raw dataset with CLIP pseudo-labeling is enough.

Question:
- It is mentioned that GIF-DALLE generates images of size 64x64 (Appendix C.1) while the downstream models use inputs of size 224x224 (Appendix C.3). These numbers seem off considering the shown samples. Can the authors confirm these numbers are used for all datasets?  If so, how exactly are the outputs of GIF-DALLE upsampled and processed, especially with regard to the random cropping mentioned in C.3?
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written. The method and experiments are clearly described.
As mentioned before, the GIF-MAE variant could be better integrated and additional discussions on limitations would be useful in the main paper.

As a related work that also leverages latents of deep generative networks to build augmentations, the authors could mention:

Generative Models as a Data Source for Multiview Representation Learning. ICLR 2022.
Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola
summary_of_the_review: The paper tackles an interesting problem in a novel way, showing promising results. It contains a lot of information, shedding light on the capabilities of current generative models and text embedding to synthesize training data.

My two main concerns are 1) about the integration of the GIF-MAE model in the narrative of the paper and 2) that the current experiments are not sufficient to be convinced about the usefulness of the method in practice. Additionally, I would also like some discussions regarding the limitations of the method in the main paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a Guided Imagination Framework (GIF) for generating more data samples for small datasets. The proposed model generates more image samples using two large-scale models trained with very large datasets (e.g., CLIP ViT-B/32 trained with private 400M image-text pairs and Dall-E2 decoder trained with LAION-400M). To generate new images, GIF first encodes the reference image using the CLIP ViT-B/32 encoder, and decodes a perturbed version of the encoded feature using the Dall-E2 decoder. This paper also proposes three optimization objectives based on CLIP features (both image and label text features) to seek the best perturbations for generating new samples: (1) zero-shot prediction consistency (2) entropy maximization and (3) diversity promotion. The experiments on six small natural image datasets and three small medical image datasets show that the proposed GIF shows improvements in the small dataset regime.
strength_and_weaknesses: ## Strength

- This paper leverages the knowledge from the models trained with large-scale datasets by generating more data samples
- Compared to previous data augmentation methods, the generated images look more "realistic" and seem helpful
- This paper provides enough analysis to show the effectiveness of the proposed method (e.g., various architectures, where to optimize, pixel-wise vs. channel-wise noise, ...)
- The paper is well-written and easy to follow.

## Weakness

### Comparison with CLIP zero-shot / linear-probe results

I think it is the most critical weakness of this paper. Note that the proposed method heavily relies on CLIP ViT-B/32 zero-shot performance and Dall-E2 decoder where both models are trained with 400M image-text pairs (CLIP is trained with private samples and Dall-E2 decoder is trained with LAION-400M, respectively). I checked the zero-shot performance of CLIP ResNet-50 and ViT-B/32 models and found that the proposed method is much worse than CLIP zero-shot performances (from the original CLIP paper [R1] Table 17):

| Methods                         | Caltech | Cars | Flowers | DTD | CIFAR100-S | Pets | Avg. |
|-----------------------------|-----------|------|----------|-------|---------------|-------|------|
| GIF-Dall-E (ResNet-50)   | 63.0       | 53.1 | 88.2.     | 39.5  | 54.5            | 66.4  | 60.8 |
| CLIP RN50 Zero-shot     | 82.1       | 55.8 | 65.9      | 41.7 | 41.6             | 85.4  | 62.1 |
| CLIP ViT-B/32 Zero-shot | 87.9      | 59.4  | 66.7     | 44.5 | 65.1              | 87.0  | 68.4 |
| CLIP ViT-B/32 Linear probe | 93.0 | 74.9  | 96.9     | 76.5  | 80.5 (full data) | 90.0 | 85.3 |

Note that directly comparing CLIP RN50 and the conventional RN50 could be unfair because CLIP ResNet-50 is not exactly the same as the conventional ResNet-50 (e.g., attention pooling, antialias pooling, ...). However, this paper uses ViT-B/32 CLIP for generating new images, and it outperforms the proposed method (except Flowers). Furthermore, if we allow access to the training datasets, then the CLIP model outperforms GIF, CLIP zero-shot with a significantly large gap (85.3 vs. 60.8 in the average accuracy). I attached CLIP ViT-B/32 linear probe results from Table 10 of the original CLIP paper [R1]. Note that the linear probe result for CIFAR100-S is made by the full training dataset, but it is known that the linear probe is less sensitive to the dataset size than the full model fine-tuning. However, I acknowledge that the comparison is not fair for CIFAR100-S (zero-shot results are fair because they are not trained with the training dataset).

Why do we need dataset expansion if CLIP zero-shot performance (or linear probe) is better than dataset expansion? If we just use CLIP zero-shot predictions, then (1) we do not need Dall-E2 or MAE decoders (2) we do not need to generate more image samples (3) we do not need to train models on the generated images. If we allow (3), the CLIP linear probe shows significant results.

I do not have CLIP results for the three medical datasets, but unless CLIP results are very poor for the medical datasets, I do not think the proposed scenario is not practical.

### Too heavy computation resources

Even if we assume that the big models trained with large-scale datasets (e.g., CLIP encoder, Dall-E2, or MAE decoder) are publicly accessible from the web, so that no extra resource is required, the proposed method still needs heavy computations to (1) generate images (2) train with x20 images. Note that the data augmentation methods usually assume the number of the total iterations is not changed, e.g., the vanilla training and Cutout augmentation for the given training dataset needs the same iterations. Following the terminology of this paper, it is common to use x1 for data augmentation. Data augmentation with xN is often called "repeated augmentation" [R2]. Usually, when we use repeated augmentation, it is not recommended to use "epochs" because a model is updated N times more using SGD optimizer, and it is known that more update brings a better performance [R3].

To summarize, the proposed method needs three additional heavy computation resources (1) training big models with large-scale datasets -- it could be ignored if we only consider image datasets, but if we consider other domains, it limits the usage of the method (2) generating images using the CLIP encoder, Dall-E2 or MAE decoder with optimized perturbations (3) the dataset expansion itself amplifies the total computational resource if we do not limit the number of iterations (i.e., x20 needs 20 times more computations than x1).

### References

[R1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, 2021.
[R2] Touvron, Hugo, et al. "Training data-efficient image transformers & distillation through attention." International Conference on Machine Learning. PMLR, 2021.
[R3] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

### Minor comments

- It looks that the bib for the COYO dataset in page 1 is broken ("Minwoo Byeon, 2022").
- It could be interesting if there exists the analysis of dropping the original images, but only using the augmented images (i.e., using only $\mathcal D_S$ in Problem statement (Section 3)
- The details of $S_{con}$ $S_{ent}$ $S_{div}$ could be more well described, e.g., showing the exact formulations using `\equation`
    - The formulation for the entropy minimization is not clear. As far as the reviewer understood, it calculates the entropy of the predicted probability (e.g., in this case, normalized cosine similarity). I cannot understand how encouraging higher prediction entropy (i.e., showing a more "uniform" prediction) helps the optimization. Is there any guess from the authors?
    - The formulation for the $S_{div}$ is not clear. How can we compute KL between features? I presume that it is achievable by optimizing the lower bound of KL divergence (as VAE)
- I wonder if the method works for larger datasets, such as CIFAR-100 (full dataset) or ImageNet-1K. Especially, I wonder about CIFAR-100 results. ImageNet-1K would not be necessary considering the scope of this paper.
- [Not necessary] It could be interesting if there are more comparisons with more advanced image augmentation methods, such as mixed sample data augmentation (e.g., Mixup, CutMix).
- [Minor] I suggest clarifying that the method is designed for "a small dataset" from the title, e.g., "Expanding Datasets With Guided Imagination for a small dataset"
clarity,_quality,_novelty_and_reproducibility: In terms of clarity and quality, this paper is well-written, well-organized, and easy to follow. I really enjoyed reading this paper.

The idea of augmentation by generation is not very new. However, I think the method contribution of this paper is fair. I think this paper discusses related previous work enough.

However, in terms of empirical novelty, this paper is neither significant nor novel. Even CLIP zero-shot outperforms the proposed method although the method heavily relies on the CLIP zero-shot prediction itself. Furthermore, if we allow access to training data, CLIP linear probe outperforms the proposed method with a significantly large gap.

I am a little bit concerned about reproducibility when the big models are not accessible. For example, as far as the reviewer understood, the Dall-E2 and MAE decoders are trained by the authors using LAION-400M. Training with LAION-400M is not always available by many researchers. Even if one can use the dataset (and enough GPUs), the reproducibility of large-scale training itself is often challenging. Except for this big model part, I think this paper is fairly reproducible with the attached pseudo code (C.1).
summary_of_the_review: This paper is well-written, and the method looks fairly novel. However, the performance of the proposed method is much worse than CLIP zero-shot or linear probing, where they even use much fewer computational resources than the proposed method (no need to generate images, no need to tune the full models for x20 data samples). I think the empirical contribution and the practicalness of this paper are not enough to be accepted at the ICLR conference.

---

During the reviewer-author discussion period, the authors provide more information than the original paper. For example, the authors showed that the CLIP zero-shot is worse than the proposed method in some medical domains. However, as my following comments, I don't think the current version of the paper is not enough to be accepted at the ICLR conference. Even if we consider the additional content provided during the discussion period, the expected revision will be very significant (the main message in introduction will be revised, the main table will be revised, following explanations and the analysis will be revised); another review process will be required.

I am also not convinced by the additional experimental results. Especially, I am not positive to the additional experiments on medical domains because CLIP zero-shot shows almost random (10%) accuracy, while the quality check of the proposed generative process is done by the low-performance CLIP zero-shot. I cannot agree with the following comment
> "Although the zero-shot performance of CLIP is not good, its feature space is already relatively discriminative (given that CLIP linear-probing performance is good)" 

The proposed method uses the zero-shot performance (zero-shot prediction consistency) for the "informativeness guidance". If the CLIP model has no power to discriminate the "ground-truth" images, then how can it guide whether the generated images are informative or not? Overall, I think this paper will need additional careful discussions related to my initial concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: ### Summary
This paper proposes the *Guided Imagination Framework* (GIF) that expands the datasets with images generated from strong generative model decoders like DALLE2 and CLIP encoders. This would be potentially be really helpful for small datasets.  The paper uses both MAE and DALLE2 models as the decoder for the classification experiments that it focuses on. 

Below I will list some of the weaknesses and strengths pointed out by the reviewers for this paper:

### Strengths
- A simple and intuitive idea that paves the way for an interesting research direction. 
- The results should be easy to reproduce given the level of details provided in the paper.
- The diverse set of experiments provides good insights into the method. 
- The paper is well-written and easy to follow.

### Weaknesses
- Poor performance of the main experiments. The performance of the method is worse than CLIP and ALIGN methods as pointed out by the reviewers on some of the datasets like Caltech 101, Pets, Flowers and Cars datasets. 
- The datasets MAE and DALLE-2 are trained on my cover some of the datasets the GIF were evaluated on which would be problematic.
- GIF-DALLE's performance is much worse than the CLIP approach.
- Requirement for the heavy computations

**Decision:** The proposed idea is very interesting and the authors did a really good job answering the concerns raised by the reviewers. However, reviewer cRYy and sxmr both had serious concerns about the experiments and the paper. The proposed approach's performance was significantly worse than the approaches like CLIP and ALIGN. The reviewers suggested that the proposed GIF approach is more general in a way that it might provide improvements on tasks or datasets where CLIP/ALIGN may not be applicable, for example on medical image domain, because of the domain gap between the dataset that they were trained on vs evaluated on whereas GIF does not necessarily need to suffer from that. The authors tried to addressed those concerns during the rebuttal. However, the authors need to do substantial amount of changes on the paper to address these concerns. Thus, I think the paper might benefit from another round of reviews. Thus I recommend the authors to resubmit this work to another venue, after addressing the concerns raised by the reviewers.


justification_for_why_not_higher_score: As pointed out by the reviewers cRYy and sxmr, the performance of the GIF approach is poorer than CLIP and ALIGN on some of the datasets that GIF was tested in. The paper proposed their approach as a generally applicable domain. However, that is not necessarily true. This paper needs to go through a substantial amount of changes and thus would benefit from another round of reviews.
justification_for_why_not_lower_score: There is no lower score.
Decision: Reject