GeNIe: Generative Hard Negative Images Through Diffusion

Authors: Soroush Abbasi Koohpayegani, Anuj Singh, Navaneet K L, Hamed Pirsiavash, Hadi Jamali-Rad
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: data augmentation;diffusion models; computer vision; few-shot learning; long-tail classification;

Abstract:
Data augmentation is crucial in training deep models, preventing them from overfitting to limited data. Recent advances in generative AI, e.g., diffusion models, have enabled more sophisticated augmentation techniques that produce data resembling natural images. We introduce GeNIe a novel augmentation method which leverages a latent diffusion model conditioned on a text prompt to combine two contrasting data points (an image from the source category and a text prompt from the target category) to generate challenging augmentations. To achieve this, we adjust the noise level (equivalently, number of diffusion iterations) to ensure the generated image retains low-level and background features from the source image while representing the target category, resulting in a hard negative sample for the source category. We further automate and enhance GeNIe by adaptively adjusting the noise level selection on a per image basis (coined as GeNIe-Ada), leading to further performance improvements. Our extensive experiments, in both few-shot and long-tail distribution settings, demonstrate the effectiveness of our novel augmentation method and its superior performance over the prior art.

Decision: reject
Reviewer scores: 5, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730645994208

thread_type:
official_review

<thread_object>
creator: reviewer_ny5s

<content>
summary:
The paper leverages Stable Diffusion to generate hard negatives as data augmentation for image classification. The hard negatives are generated by feeding a source image into the model's encoder but conditioning the reverse diffusion process with a text prompt corresponding to the target class. 

The proposed approach includes an automatic noise-level adjustment per sample, which controls the degree of deviation from the source image, thereby aligning more with the target class prompt. The authors show that the choice of the noise level is critical, as only a restricted range of values allow for a good balance between source and target classes. The noise is chosen by generating multiple images with different noise levels and choosing the level where the semantic shift between the source and target images is the highest.

The proposed augmentation is evaluated on image classification tasks, in few-shot and long-tail distribution settings.

strengths:
- The proposed approach allows incorporating diversity into the image generation process, which is a crucial aspect in diffusion-based image generation.
- The paper effectively illustrates the importance of selecting an appropriate noise level in text-conditioned image generation.
- The proposed augmentation is evaluated with extensive experiments on image classification in few-shot and long-tail settings.

weaknesses:
**Method section**
My main concern regards the automatic noise level selection process. It involves generating multiple images with various noise levels and selecting the one where the semantic shift is the highest. This seems highly inefficient given the time required to generate these images.

**Data augmentation.** 
I am confused about the *weak* and *strong* augmentation policies GeNIe is compared to. Are strong augmentations (random color jitter, random grayscale, Gaussian blur) applied in addition to weak augmentations (random resize crop, random flipping)? Otherwise, why are the weak augmentations solely geometric, and the strong ones solely photometric? 
A balanced policy combining both geometric and photometric transformations could be a better contender. For example, it could consist of random resize crop (with e.g., a scale of 0.4), random flipping, color jittering, and maybe MixUp or CutMix on top.

**Visualizations.** 
The PCA representations of DINOv2 in Figure 5 could be improved and expanded. DINOv2's representations are excellent, and their PCA is usually very insightful.

questions:
**Visualizations.** Improve visualization of the PCA over DINOv2's embeddings as suggested in Weaknesses section. Also, increasing the resolution of images across all figures would help fully appreciating the variations in image generation. 

**Data augmentation.** Run experiments with data augmentation policies combining both geometric and photometric transformations, as suggested in Weaknesses section.

**Datasets.**
In the supplementary material Table A1, the augmentation is evaluated on fine-grained classification by training a SVM on DINOv2's pretrained **ViT-g**. This evaluation is quite insightful and would deserve being integrated in the main paper. To make space, consider relocating some few-shot table contents to the supplementary. It would also be very interesting to see the same evaluation in the non-few-shot setting (*i.e.* training with the full training set), which could be directly compared to the linear evaluation results reported by DINOv2. Note that for Aircraft, DINOv2 evaluates on the **variants** categories. 

**Minor remarks on formatting.**
- Consider using *multicol* to improve table readability.
- Please write **ViT-g** instead of **ViT-G** for DINOv2's model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660521781

thread_type:
official_review

<thread_object>
creator: reviewer_eenj

<content>
summary:
In this paper, the authors try to illustrate their idea by proposing these points:
1. The authors introduce GeNIe, a novel yet elegantly simple diffusion-based augmentation method to create challenging augmentations in the manifold of natural images. For the first time, to our best knowledge, GeNIe achieves this by combining two sources of information (a source image, and a contradictory target prompt) through a noise-level adjustment mechanism.
2. They further extend GeNIe by automating the noise-level adjustment strategy on a per-sample basis
(called GeNIe-Ada), to enable generating hard negative samples in the context of image classification, leading also to further performance enhancement.
3. To substantiate the impact of GeNIe, authors present a suit of quantitative and qualitative results including extensive experimentation on two challenging tasks: few-shot and long tail distribution settings corroborating that GeNIe (and its extension GeNIe-Ada) significantly improve the downstream classification performance.

strengths:
1. GeNIe leverages a latent diffusion model to create challenging augmentation images by blending features from two classes (source and target). By adjusting the noise level, GeNIe maintains low-level details from the source image while incorporating semantic features of the target class, creating what the authors term "hard negative" samples. This process potentially aids in classifier training by providing near-boundary examples.

2.Adaptive Mechanism (GeNIe-Ada):
The GeNIe-Ada extension adds significant value by adaptively selecting noise levels for each sample, fine-tuning the augmentation to be more effective. This adaptive approach is well-aligned with real-world scenarios where data variability and distributional shifts are common, making it a practical addition.

weaknesses:
1. They didn't give more contribute to the model itself. It's just an application to generate images.
2. Limited number of downstream cases.

questions:
Could you add more experiments on some other mainstream tasks? Do you have comparisons with other models for data augmentation? Such as GAN?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682119693

thread_type:
official_review

<thread_object>
creator: reviewer_ddir

<content>
summary:
This paper proposes a solution to generate augmentations using diffusion models to improve the performance of classifiers in few-shot and long-tailed settings, which are characterized by low-data scenarios. To that end, the authors propose using a latent diffusion model that combines a source image (with a known category, e.g., dog) with a target category prompt (e.g., cat), producing challenging samples that retain background features from the source category but resemble the target category. This approach aims to push model learning near decision boundaries, providing more informative augmentations than traditional methods.

GeNIe, inspired by image editing through diffusion models, works by adjusting noise levels in the diffusion process to control the balance between source and target features in the generated image. The authors demonstrate that lower noise levels retain more of the source image's visual characteristics, while higher levels focus on the target category. To automate the noise level per sample, the authors further introduce GeNIe-Ada. This variant stems from the observation that there is a sudden change in appearance when the noise levels are uniformly sampled. The authors leverage this observation along with the assumption that they also have access to another target sample. They then propose to compare the similarities of this sample and the generated sample in the latent space of a classifier.

To validate the utility of these augmentations, the authors conduct experiments on few-shot and long-tail classification tasks and demonstrate significant improvements on standard benchmarks.

strengths:
The paper is well-written and easy to follow. The motivation is sound, and using diffusion models to generate augmented views, even though it is a well-explored area, the authors focus on generating hard negative images. This approach forces the classifiers to learn the true characteristics of the category rather than rely on spurious correlations, such as background. In many ways, the motivation is similar to foreground and background editing ideas. Their method also addresses the general problem of CNNs, which tend to rely more on textures than on the shapes of objects while classifying. Furthermore, the results are impressive across different benchmarks.

weaknesses:
I have a few concerns and questions regarding GenIE. While generative models are indeed a powerful approach, they can be restrictive in the following sense: their applicability in general, and GenIE in particular, requires that the generative model already knows the concepts of the target category, which is a strong assumption to make. In that sense, generative model-based augmentations should be viewed from two perspectives: (I) those that change the background and (ii) those that change the category. In the first setting, the use of generative models makes a lot of sense because it aligns with the paper's motivation but is concept agnostic. The second category is harder to argue because it requires the generative model to have complete knowledge of all target categories, which would severely limit its applicability. Category 1 also has the added benefit of generalizing to newer domains. For example, in SiSTA: Target-Aware Generative Augmentations for Single-Shot Adaptation (ICML 2023), the authors do not view this as an image editing problem but rather adapt a generative model like StyleGAN to the target domain using a single image and then sample from the model to generate augmented variants. These variants are then used to train classifiers with source-free unsupervised domain adaptation. In this context, GenIE requires both source samples, and its general treatment of this problem as image editing is limiting.

I would like to see comparisons or discussions regarding the strong assumptions that GenIE makes in this context. This is an important discussion because even simpler methods like mixup do not require explicit knowledge of target categories, which GenIE's reliance on diffusion models necessitates.

Similarly, the diffusion models used in the paper are large-scale foundation models trained on LAION. This raises questions about the improvements: it is unclear whether the gains are due to the proposed method or the large-scale pretraining of the diffusion model. It would also be important to discuss relevant work, such as (Tian, Yonglong, et al. "Stablerep: Synthetic images from text-to-image models make strong visual representation learners." Advances in Neural Information Processing Systems 36 (2024).), which uses Stable Diffusion to generate synthetic positive samples for contrastive self-supervised learning training.

Lastly, it would be interesting to see if synthetic image augmentations can be used alongside general mixup, similar to (Graikos, Alexandros, Srikar Yellapragada, and Dimitris Samaras. "Conditional Generation from Pre-Trained Diffusion Models using Denoiser Representations." BMVC. 2023.), which has shown that synthetic images from a conditional diffusion model can provide complementary benefits to mixup.

questions:
Please see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692879687

thread_type:
official_review

<thread_object>
creator: reviewer_x612

<content>
summary:
This paper introduces a novel data augmentation method based on a latent diffusion model. It combines target and source categories in the latent noise-level to generate hard-negative samples for the source category. The impact of the proposed GeNIe is evaluated on two famous data-scarce problems.

strengths:
- This paper proposes a realistic solution for practical problems of data augmentation using recent text-to-image generative models.
- The proposed method, GeNIe, contains an effective and adaptive strategy for data augmentation.

weaknesses:
- The exact method for GeNIe is not detailed in the main script, although the key idea is presented, which can cause confusion.
- The proposed GeNIe is a simple data augmentation method on the latent space of Stable Diffusion, which lacks the novelty of the method except for modifying noise levels for better augmentation.
- GeNIe-Ada requires an additional classifier to search the decision boundary between two categories.

questions:
- Is the classifier used for GeNIe-Ada the same as the one trained for the target task? How does the performance of this classifier affect the effectiveness of GeNIe-Ada?
- Is there no information leakage by using stable diffusion in few-shot classification for mini-ImageNet and tiered-ImageNet?
- Does using images from all other classes as the source image in line 352 mean that source images are selected from the support set of the other 4 classes, or all other 63 classes (for mini-ImageNet)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732312840223

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to ALL reviewers

comment:
We do appreciate reviewer's constructive feedback which helped to further improve the quality and clarity of the paper. We are please by the positive feedback from reviewer **X612** for finding our proposed method "effective" and "a realistic solution for practical problems", from reviewer **dDiR** for finding our paper "well-written and easy to follow", and for finding our motivation behind GeNIe "sound". We appricate reviewer **eEnJ**'s comment on GeNIe-Ada as an extension that "adds significant value", and reviewer **ny5s**'s comment on our "extensive" experimental evaluation. 

After perusing reviewer's remarks and recommendations, we have put in tremendous effort to provide further evidence (new experimentation and qualitative demonstrations) to corroborate the efficacy of GeNIe as summarized below:
- In response to reviewer **X612** and **dDiR**, regarding information leakage between LAION and mini/tieredImagenet, we present a statistical results on image embeddings ensuring discrepancy between the generated augmentations and the test set; 
- In response to reviewer **X612**, we present new few-shot classification results with a different backbone for selecting optimal hard-negatives in the case of GeNIe-Ada.
- In response to reviewer **dDiR**, we demonstrate the robustness of GeNIe in cases where the target class is unknown to the diffusion model by parameter-efficient fine-tuning the base diffusion model on few images of the target class.
- In response to reviewer **eEnJ**, We include additional experiments to further demonstrate the effectiveness of GeNIe in scenarios where category labels are unavailable and compare its performance with the DAFusion baseline.
- In response to reviewer **dDiR**, **ny5s**, we demonstrate new experimental results with: (i.) a Mixed augmentation strategy, (ii.) GeNIe alongside the MixUp augmentation strategy and (iii.) Combinations of Mixed+CutMix and Mixed+MixUp augmentation strategies.

**Remark:** Note that the additional experimental results and explanations have been added to the revised manuscript in text color **blue**.

We do hope this addresses reviewer's concerns and questions, and look forward to engaging further during reviewer-author discussion period.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733148337646

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to reviewers dDiR and X612 - Part 1

comment:
### General Response to reviewers **dDiR** and **X612**:

We do appreciate your constructive feedback which helped to further improve the quality and clarity of our paper. We notice a common theme in your final remarks around (i). GeNIe's potential limited applicability to tasks with SDv1.5 "known" classes, and (ii). Potential data overlap between LAION and few-shot test sets. To answer these two, we have presented new experimental results and further clarifications around our method: 

1. GeNIe generates optimal hard-negative samples using two sources of information - i. a source image, and ii. a target class. In a challenging scenario where the target classes are "unknown" to the diffusion model, GeNIe can still utilize other _distractor_ classes that do not belong to the set of the unknown target clasess. These distractor classes can be selected from a set of top-k other categorizes different from the source class yet similar to it with respect to their CLIP text-embeddings. As an extension to our work, we could also utilize an LLM such as GPT-4 to query a target class that might be confused with the source image's class, to be used as the text-prompt for the diffusion model.

2. To demonstrate that GeNIe is still applicable in cases where the target class is not known to the diffusion model, we test its performance in a scenario where we _do not_ input target classes as text-prompts into the diffusion model. Instead, we now use textual inversion to fine-tune the diffusion model (embeddings) on a few images belonging the unknown target classes. This fine-tuning allows the diffusion model to learn embeddings specific to the target class, enabling the generation of the desired hard-negative examples. Note that once the diffusion model is fine-tuned, the procedure to generate hard-negatives using partial noising and a contradictory text-prompt remains the same. As can be seen in Table A3 (also presented below), GeNIe+TxtInv performs significantly better than the DAFusion baseline by offering gains in few-shot accuracy of up to 9%. We would like to re-iterate that in this case, we _do not utilize any information about the target category labels_. DAFusion also employs textual-inversion-based fine-tuning; however, it does so without generating hard-negative samples. GeNIe's superior performance indicates that it is effective even in scenarios where the diffusion model is _unaware of the target-class_.



### ResNet-18

| Augmentation       | Method          | Pre-training | 1-shot         | 5-shot         |
|--------------------|-----------------|--------------|----------------|----------------|
| DAFusion (2024)    | UniSiam (2023)  | unsup.       | 64.3 ± 1.8     | 82.0 ± 1.4     |
| **GeNIe+TxtInv**   | UniSiam (2023)  | unsup.       | **73.9 ± 0.8** | **84.6 ± 0.9** |

### ResNet-50

| Augmentation       | Method          | Pre-training | 1-shot         | 5-shot         |
|--------------------|-----------------|--------------|----------------|----------------|
| DAFusion (2024)    | UniSiam (2023)  | unsup.       | 65.7 ± 1.8     | 83.9 ± 1.2     |
| **GeNIe+TxtInv**   | UniSiam (2023)  | unsup.       | **76.2 ± 1.2** | **86.2 ± 0.9** |
</content>
</thread_object>

</thread>
<thread>
creation date:
1733175937481

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the discussion period

comment:
Dear reviewers and the AC,
 
We appreciate your constructive feedback which helped us further improve the quality and clarity of the paper. We are pleased by the positive feedback from the reviewers for finding our proposed method "effective", "novel yet elegantly simple", "a realistic solution for practical problems", and for finding our paper "well-written and easy to follow", and for finding our motivation behind GeNIe "sound".
 
As the discussion period comes to an end, we would like to summarize the tremendous efforts we have put in during the rebuttal period (new sets of experimental results and qualitative demonstrations) to address the reviewers’ remarks as follows:
 
- In response to reviewer **X612** and **dDiR**, regarding a potential distribution overlap between LAION and mini/tieredImagenet, we presented statistical and qualitative results on image embeddings ensuring discrepancy between the generated augmentations and the test set. These have been added to Fig. A3 in Section A2 of the revised draft.
- In response to reviewer **X612**, we presented new few-shot classification results with two different backbones - DEit-B and VAE of SDv1.5, for selecting optimal hard-negatives in the case of GeNIe-Ada. These have been added to Table A2 in Section A4 of the revised draft. Additionally, we have also added a detailed pytorch-style pseudocode of GeNIe in Alg.2 Section A5 of the revised draft.
- In response to reviewer **dDiR**, we demonstrated the robustness of GeNIe in cases where the target class is unknown to the diffusion model by parameter-efficient (textual-inversion) fine-tuning the base diffusion model on a few images of the target class and demonstrate superior performance than the DAFusion baseline, on-par performance with GeNIe, GeNIe-Ada. These have been added to Table A3 in Section A6 of the revised draft.
- In response to reviewer **dDiR**, **ny5s**, we shared new experimental results with: (i.) a Mixed augmentation strategy, (ii.) GeNIe alongside the MixUp augmentation strategy and (iii.) Combinations of Mixed+CutMix and Mixed+MixUp augmentation strategies. These are now included in Table A1, Section A3 of our revised draft.
- In follow-up responses to reviewer **ny5s**, we demonstrated a new qualitative analysis of 10 different semantic trajectories of GeNIe with the same source and target classes. We observe that each trajectory has a separate rate/speed of semantic shift from source to target, while following the same path. These are included in Fig A5, Section A12 of the revised draft. Additionally, we provide further insight into GeNIe's category transitions by illustrating generated images across different $r$'s in which GeNIe was applied twice to transform an image between a source image and a target category. These have now been added to Fig. A6, Section A14 of the revised draft.
 
Our responses detailed above have led to an increase in reviewer **ny5s**'s score to 8 and reviewer **dDiR**'s score to 6.
 
**Remark:** Note that the additional experimental results and explanations have been added to the revised manuscript in $\textcolor{blue}{blue}$.
 
We do hope our additional experiments and responses have addressed all the reviewers' concerns and questions. If so, we would greatly appreciate it if you please kindly consider re-evaluating/increasing our paper's final score.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes to leverage generative models to obtain data augmentations that improve downstream performance of few shot and long tail datasets. Augmentations are produced by combining a source image with a target category prompt.

The paper was reviewed by four knowledgeable reviewers who acknowledged that the paper is well written and easy to follow (dDiR), the motivation sound (dDiR), and the solution relatively effective (X612). The reviewers raised concerns about:

1. Missing information on the method (X612)
2. Limited novelty (eEnJ), strong assumptions made (X612, dDiR)
3. Potential information leakage (from pre-trained generative models and about ImageNet) (X612)
4. Missing some related work discussion/comparisons (eEnJ, dDiR)
5. Possible inefficiency of the approach (ny5S)
6. Unclear significance of results (e.g. unclear whether improvements come from the method or from the pre-trained diffusion model) (dDiR).

During rebuttal/discussion, the authors discussed the questions related to SDv1.5 known classes, and potential data overlap between pre-training datasets and few-shot test set. They provided missing details on the method and experimental setup, argued for the novelty of their approach, shared additional experiments analyzing the effect of the feature extractor, discussed the surfaced leakage issue, added some comparisons suggested by the reviewers, as well as combining the proposed approach with standard data augmentation techniques, and promised results on additional datasets. After rebuttal, one reviewer signals clear accept, one leans towards rejection, and the remaining two appear to position themselves as borderline. However, as pointed by the authors, some reviewers remain unresponsive during discussion time. Given this, the AC took the time to go over the paper, the reviews and discussions. Although the AC agrees with the reviewers that the proposed method may hold potential, the AC remains unconvinced by the experimental evidence provided. In particular, although prior art is discussed in the related work, important comparisons are missing in the experimental section (e.g. StableRep, feedback guidance, and Fill Up). Compared to Feedback Guidance and Fill Up, the improvements achieved by Genie on the Few split of the ImageNet-LT dataset, appear less impressive. It is also unclear what the effect of the number of synthetic images is on the downstream performance, or whether using a more recent text-to-image model would result in different findings. For these reasons, and the remaining concerns of the reviewers, the AC cannot recommend acceptance, despite the merits outlined by some reviewers. The AC encourages the authors to consider the feedback received to improve future iterations of their work.