Diversity-Enhanced and Classification-Aware Prompt Learning for Few-Shot Learning via Stable Diffusion

Authors: Gaoqin Chang, Jun Shu, Xiang Yuan, Deyu Meng
Primary area: generative models
Keywords: meta-learning, synthetic dataset generation, diffusion model

Abstract:
Recent text-to-image generative models have exhibited an impressive ability to generate fairly realistic images from some text prompts. In this work, we explore to leverage off-the-shelf text-to-image generative models to train non-specific downstream few-shot classification model architectures using synthetic dataset to classify real images. Current approaches use hand-crafted or model-generated text prompts of text-to-image generative models to generated desired synthetic images, however, they have limited capability of generating diversity images. 
Especially, their synthetic datasets has relatively limited relevance to the downstream classification tasks. This makes them 
fairly hard to guarantee training models from synthetic images are efficient in practice. To address this issue, we propose a method capable of adaptively learning proper text prompts for the off-the-shelf diffusion
model to generate diverse and classification-aware synthetic images. Our approach shows notable improvements in various
classification datasets, with results comparable to existing prompt designing methods. 
We find that replacing data generation strategy of existing zero/few-shot methods with proposed method could consistly improves downstream classification performance across different network architectures, demostrating its model-agnostic characteristic for few-shot learning. This makes it possible to train an efficient downstream few-shot learning models from synthetic images generated by proposed method for real problems.

Decision: reject
Reviewer scores: 6, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1729239134413

thread_type:
official_review

<thread_object>
creator: reviewer_9gwj

<content>
summary:
Authors tackle the problem of synthetic data generation for downstream training. They look at this from the perspective of increasing prompt diversity, to increase image diversity. This do this by generating proper prompts through meta learning of the prompt pool. They evaluate this against other prompt-learning methods and on top of SOTA methods.

strengths:
- Well written

- Novel methodology

- Good show of qualitative results

weaknesses:
Weaknesses are given in order of how I perceive their importance (issues that weigh heavier in my review decisions are first):

1) I find the main experimental results comparing to SOTA prompt methods in Table 1 lacking (specifically in terms of datasets), for the following reasons:

1a) The number of datasets evaluated is lacking very far behind other SOTA methods. E.g. Sus-X (cited in this paper) trains on 19 datasets, only three of which are used here (and two subsets). Furthermore, the datasets used in DeCap are quite similar in terms of subject, granularity, modality, etc. This is concerning because the problem statement of generating training images is highly class / dataset dependent. To be rigorously shown, this needs to be evaluated on a diverse variety of datasets. I believe that to support the claims in this paper, this method would need to be evaluated on a minimum of 3 / 4 of the following categories: a FG dataset (e.g. Stanford Cars of FGVC Aircraft), two more 'types' of objects (e.g. Food101, FGVC Aircraft, assuming animals has already been covered by Pets), a different modality (e.g. EuroSAT, ImageNet-Sketch), a scene-based dataset (e.g. SUN397). I would like to reiterate that I believe this is the minimum, and more ideally a few more strategically chosen datasets should be evaluated.

1b) CIFAR and STL are 32x32 and 96x96 resolutions, respectively. SDXL-Turbo generates images at 512x512. Hence, they are too far out-of-distribution for how the diffusion model generates to be strong evidence in themselves. I don't find it problematic in itself to include them, but these datasets being weak further exacerbates the issue described in 1a.

2) Intuitively, DeCap looks expensive. I'm not convinced that the performance gains are enough to justify the cost (unless you correct me on the cost). This would be stronger if the GPU hours were compared between DeCap and other SOTA in table 1 (LE, CiP) in addition to how much it adds for other SOTA methods (FakeIt, SuS-X, CaFo).

3) When combining DeCap with SOTA, significant gains are only seen on FakeIt. Already SuS-X and CaFo (more recent works) show gains of only 1% or less on average. There are two outcomes that I am concerned these results imply at least one of the following:

3a) FakeIt is trained from scratch, while the other two fine-tune SOTA models. It could be that in the context of fine-tuning pre-trained models, DeCap does not affect as much.

3b) FakeIt is older and both SuS-X and CaFo are newer. It could be that the challenges DeCap combats have been already well-addressed by newer works. 

4) The work misses ablations on the design choices. E.g. why not optimize the prompt without meta learning, choice of loss function, choice of meta learning algorithm, etc.

5) The ablations on number of prompts, etc. are done on STL-10 (see 1b)

questions:
1) What is your justification for dataset selection? Does it align with previous work?

2) What is the cost of DeCap? I'm not finding the breakdown, but please point me to it if it is already there.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730550830701

thread_type:
official_review

<thread_object>
creator: reviewer_1eaj

<content>
summary:
The paper proposes a Diversity-Enhanced and Class-Aware Prompt (DeCap) learning strategy to discover appropriate text prompts for downstream few-shot classification tasks, which addresses the issue of the lack of diversity in existing prompt-based methods. The proposed method includes a diversity-enhanced prompt pool construction and a class-aware prompt learning strategy. Empirical experiments are included to demonstrate the effectiveness of this approach.

strengths:
1. This paper presents an innovative strategy that involves constructing more diverse prompts and introduces a prompt-learning technique tailored for downstream classification tasks to achieve optimal prompts. The approach is novel, and its effectiveness is supported by experimental results.
2. The paper systematically compares and summarizes various methods of data curation through prompts in T2I models. This overview is valuable for readers seeking to understand progress in this area.

weaknesses:
1. Although this work proposes an interesting gradient-free method for discovering effective prompts for synthetic data, the main limitation is that “**attempting to approximate the few-shot data domain by altering prompts is inefficient and lacks generalization capability**”.

- **Regarding efficiency**: The meta-learning paradigm requires substantial sampling, and the pipeline includes too many uncontrolled components. For instance, generating images from text involves a high degree of freedom, while even slight variations in the images can significantly alter the classifier’s decision boundary. Thus a discussion of the running time should be included, along with an analysis of the strategy’s robustness with respect to T2I hyperparameters beyond prompts, such as generation using fixed seed to a random seed.
- **Regarding generalization**: The genetic algorithm essentially searches for effective prompts to approximate the domain of the few-shot dataset (assuming Equation 2 uses accuracy on the few-shot data as the fitness function for prompts rather than accuracy on the test set). Consequently, this approach may limit performance on out-of-domain test scenarios. Results on ImageNet domain generalization datasets, such as ImageNet-R, would be beneficial to verify the generalization capability. 

2. Some recent data augmentation studies are not included [1,2,3]. Leveraging the inherent knowledge in few-shot images directly to curate data, bypassing text prompts, may offer a more efficient solution. Adding discussions and comparisons between prompt-based and image-based (e.g., image editing) data curation methods would provide better insight, such Real-Guidance [4] and Da-fusion [1].


3. The impact of varying shot numbers on the prompt-mining strategy is not addressed. Section 2.2 indicates that all experiments used a 10-shot setting. I suspect that the strategy may be ineffective in extremely low-shot cases (e.g., 1-shot). As noted in Weakness #1.2, this could result in synthetic data failing to adequately cover the evaluation dataset. Can you provide the more results across scenarios with varying shot numbers?

[1] DA-fusion: Effective Data Augmentation With Diffusion Models. Brandon Trabucco, et-al.

[2] Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model. Zhicai Wang, et-al.

[3] Diffusemix: Label-preserving data augmentation with diffusion models. Khawar Islam, et-al.

[4] Is synthetic data from generative models ready for image recognition? Ruifei He, et-al

questions:
My questions regarding efficiency and generalization capability are shown in the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730599709025

thread_type:
official_review

<thread_object>
creator: reviewer_w3va

<content>
summary:
The authors propose a prompt learning method to boost classification performance. To overcome the lack of diversity of hand-crafted prompts, they also leverage prompts from language models. Once a prompt pool has been created, genetic algorithm finds out the best working prompt for classification performance.

strengths:
- The method is easy to follow.
- It proposes a total solution for few-shot classification problem.

weaknesses:
- At the first glance of this paper, the though 'commercially or open-sourced LLM or Multi-modal LLM will show much better performance with even not using genetic algorithm' came up in my mind. That means many readers would think just the same I did and the authors are responsible for prove them wrong. I have tried myself requesting various prompts to a commercial LLM and the results seemed much better than the examples in this paper.
- The task is limited to classification problem and datasets used for evaluation are insufficient to make this paper solid. It is hard to tell if this paper worth dropping another paper for this conference.

questions:
Comments in weaknesses must be resolved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720732625

thread_type:
official_review

<thread_object>
creator: reviewer_bqtv

<content>
summary:
The paper presents DeCap, a novel method for enhancing few-shot learning through the generation of diverse and classification-aware synthetic images using text-to-image generative models.
The proposed automated prompt learning method reduces the manual effort required to craft effective prompts and its meta-learning strategy can align synthetic data generation with downstream classification tasks to improve the performance in various network architectures.

strengths:
The innovation of this work lies in the adaptive learning of prompts for synthetic data generation, which is tailored to the specific requirements of few-shot learning tasks. 
This work use a genetic algorithm for the outer-loop optimization to search for the optimal prompt set, which is a discrete search problem. The inner-loop optimization involves training a classification model on synthetic data generated by the selected prompts.
The paper demonstrates that DeCap can improve the performance of existing zero/few-shot learning methods in different network architectures

weaknesses:
A significant concern with DeCap's approach is the adequacy of the prompt pool in capturing the full spectrum of variations necessary for diverse few-shot learning tasks. The paper does not provide a clear methodology for ensuring that the prompt pool is comprehensive enough, or that the most contributive prompts are consistently selected across different optimization iterations and datasets. This lack of clarity raises questions about the reliability and robustness of the prompt selection process.

The paper primarily relies on accuracy as the performance metric for evaluating the few-shot learning models. However, in the context of few-shot learning, a more nuanced evaluation that includes "n-shot n-way" scenarios and a suite of metrics such as precision, recall, and F1 score is essential. The current approach may not fully capture the model's performance across different classification tasks, especially when the distribution of samples per class varies significantly. This limitation restricts the comprehensive assessment of the model's robustness and generalization capabilities in few-shot settings.

questions:
Could you provide a few examples of prompts that were either consistently selected or commonly rejected across different datasets? Additionally, would it be possible to include visualizations that demonstrate the prompt selection process over the course of your optimization? Such examples and visualizations would greatly aid in understanding the behavior of your DeCap method and the rationale of the prompts it selects for synthetic image generation？

A critical aspect of DeCap's approach is the ability to generate diverse and classification-aware synthetic images, which hinges on the richness and detail of the prompt pool. My concern is whether the current prompt pool contains a sufficient level of fine-grained detail to effectively capture the diversity required for various few-shot learning tasks. Could you elaborate on how you ensure that the prompts in your pool are not only comprehensive but also detailed enough to represent the nuances of different classification tasks? Additionally, how do you assess whether the prompts selected across different optimization iterations and datasets are indeed the most contributive ones in terms of fine-grained detail?

Could you consider expanding the evaluation framework to include a range of "n-shot n-way" scenarios? The paper would benefit from a more holistic evaluation that includes the F1 score, precision, and recall in your analysis. Specifically, how would these metrics help in understanding the model's generalization performance in terms of false positives and false negatives, which are critical considerations in few-shot learning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732539436390

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
We thank all reviewers for their valuable feedback on our paper. We are happy to see that: 
	
- Proposed adaptive prompt learning strategy tailored for downstream few-shot classification tasks was received as **innovative, novel, effective, valuable** by the reviewers;
	
- The paper is **well written and easy to follow**;
	
- The paper **systematically compares and summarizes** various methods of data curation through prompts in T2I models, and present a  **good show of qualitative results** to demonstrate the effectiveness of proposed approach;
	
convinced the reviewers as strengths.
	
While we acknowledge that our manuscript could be further improved. We have addressed all reviewer comments, including:
	
- requests for additional experiments on a diverse variety of datasets (Reviewer 9gWJ,Reviewer W3vA) - displayed in the PDF;
	
- questions about our methodology, e.g., the comprehensiveness of propmt pool and the visualizations of the prompt selection process (Reviewer Bqtv), generalization and efficiency (Reviewer 1Eaj);
	
- questions about our experiment setup, e.g., evaluation framework and metric (Reviewer Bqtv)
	
We hope we have addressed all concerns and look forward to the discussion phase.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to enhance the performance of few shot learning by generating diverse and useful synthetic data from text-to-image models. The paper was reviewed by four knowledgeable referees who acknowledged that the paper is well written and easy to follow (W3vA, 9gWJ), the presented methodology is novel (1Eaj, 9gWJ), and the qualitative results are sufficient (9gWJ). The main concerns raised by the reviewers were:
1. Unconvincing experimental validation of the approach: missing metrics and datasets (Bqtv, 1Eaj, 9gWJ), limited scope (W3vA), missing comparisons with baselines / prior art (W3vA, 1Eaj, 9gW), missing ablations (9gWJ), and debatable significance/robustness of the presented results (Bqtv, 9gWJ)
2. Missing cost vs. improvements analyses (9gWJ)
3. Unclear efficiency and generalizability of the method (1Eaj)
4. Unclear adequacy of the prompt pool in capturing sufficient diversity (Bqtv)

The authors partially addressed the reviewers concerns during rebuttal and discussion. In particular, the authors included additional metrics (e.g. precision, recall, F1-score) and datasets, discussed the comprehensiveness of the prompt pool and selection process, and shared additional experimental evidence on OOD generalization, extreme few shot scenarios, and comparisons with other synthetic datasets. After rebuttal / discussions, reviewers remained unconvinced and kept raising concerns about the evaluation of the proposed approach. In particular, the improvements / advantages of the proposed approach are not convincing. The breadth of datasets considered for quantitative evaluation appeared to be narrower than the ones typically used in the literature, the differences appeared overall marginal making in challenging to assess the significance of the work, MLLM baselines would strengthen the contribution, and systematic ablations would help justify the proposed approach. Therefore, the result of the reviewers discussion period leans towards rejection. The MR agrees with the reviewers' concerns and therefore recommends to reject. The MR encourages the authors to consider the feedback received to improve future iterations of their work.