GeNIe: Generative Hard Negative Images Through Diffusion

## Abstract

Abstract Data augmentation is crucial in training deep models, preventing them from overfitting to limited data. Recent advances in generative AI, e.g., diffusion models, have enabled more sophisticated augmentation techniques that produce data resembling natural images. We introduce GeNIe a novel augmentation method which leverages a latent diffusion model conditioned on a text prompt to combine two contrasting data points (an image from the source category and a text prompt from the target category) to generate challenging augmentations. To achieve this, we adjust the noise level (equivalently, number of diffusion iterations) to ensure the generated image retains low-level and background features from the source image while representing the target category, resulting in a hard negative sample for the source category. We further automate and enhance GeNIe by adaptively adjusting the noise level selection on a per image basis (coined as GeNIe-Ada ), leading to further performance improvements. Our extensive experiments, in both few-shot and long-tail distribution settings, demonstrate the effectiveness of our novel augmentation method and its superior performance over the prior art. Our code is available at: https://github.com/UCDvision/GeNIe .

## 1 Introduction

Augmentation has become an integral part of training deep learning models, particularly when faced with limited training data. For instance, when it comes to image classification with limited number of samples per class, model generalization ability can be significantly hindered.
Simple transformations like rotation, cropping, and adjustments in brightness artificially diversify the training set, offering the model a more comprehensive grasp of potential data variations.
Hence, augmentation can serve as a practical strategy to boost the model’s learning capacity, minimizing the risk of overfitting and facilitating effective knowledge transfer from limited labelled data to real-world scenarios. Various image augmentation methods, encompassing standard transformations, and learning-based approaches have been proposed [; ; ; ; ]. Some augmentation strategies combine two images possibly from two different categories to generate a new sample image. The simplest ones in this category are MixUp [] and CutMix [] where two images are combined in the pixel space.
However, the resulting augmentations often do not lie within the manifold of natural images and act as out-of-distribution samples that will not be encountered during testing.

Recently, leveraging generative models for data augmentation has gained an upsurge of attention [; ; ; ]. These interesting studies, either based on fine-tuning or prompt engineering of diffusion models, are mostly focused on generating *generic augmentations* without considering the impact of other classes and incorporating that information into the generative process for a classification context. We take a different approach to generate challenging augmentations near the decision boundaries of a downstream classifier. Inspired by diffusion-based image editing methods [; ] some of which are previously used for data augmentation, we propose to use conditional latent diffusion models [] for generating *hard negative* images. Our core idea (coined as GeNIe) is to sample source images from various categories and prompt the diffusion model with a contradictory text corresponding to a different target category. We demonstrate that the choice of noise level (or equivalently number of iterations) for the diffusion process plays a pivotal role in generating images that semantically belong to the target category while retaining low-level features from the source image. We argue that these generated samples serve as *hard negatives* [; ] for the source category (or from a dual perspective hard positives for the target category). To further enhance GeNIe, we propose an adaptive noise level selection strategy (dubbed as GeNIe-Ada) enabling it to adjust noise levels automatically per sample.

Figure: Figure 1: Generative Hard Negative Images Through Diffusion (GeNIe): generates hard negative images that belong to the target category but are similar to the source image from low-level feature and contextual perspectives. GeNIe starts from a source image passing it through a partial noise addition process, and conditioning it on a different target category. By controlling the amount of noise, the reverse latent diffusion process generates images that serve as *hard negatives* for the source category.
Refer to caption: extracted/5983662/figures/genie_teaser4.jpg

To establish the impact of GeNIe, we focus on two challenging scenarios: *long-tail* and *few-shot* settings. In real-world applications, data often follows a long-tail distribution, where common scenarios dominate and rare occurrences are underrepresented. For instance, a person jaywalking a highway causes models to struggle with such unusual scenarios. Combating such a bias or lack of sufficient data samples during model training is essential in building robust models for self-driving cars or surveillance systems, to name a few. Same challenge arises in few-shot learning settings where the model has to learn from only a handful of samples. Our extensive quantitative and qualitative experimentation, on a suite of few-shot and long-tail distribution settings, corroborate the effectiveness of the proposed novel augmentation method (GeNIe, GeNIe-Ada) in generating hard negatives, corroborating its significant impact on categories with a limited number of samples. A high-level sketch of GeNIe is illustrated in Fig. 1. Our main contributions are summarized below:

- -
We introduce GeNIe, a novel yet elegantly simple diffusion-based augmentation method to create challenging augmentations in the manifold of natural images. For the first time, to our best knowledge, GeNIe achieves this by combining two sources of information (a source image, and a contradictory target prompt) through a noise-level adjustment mechanism.
- -
We further extend GeNIe by automating the noise-level adjustment strategy on a per-sample basis (called GeNIe-Ada), to enable generating hard negative samples in the context of image classification, leading also to further performance enhancement.
- -
To substantiate the impact of GeNIe, we present a suit of quantitative and qualitative results including extensive experimentation on two challenging tasks: few-shot and long tail distribution settings corroborating that GeNIe (and its extension GeNIe-Ada) significantly improve the downstream classification performance.

## 2 Related Work

Data Augmentations. Simple flipping, cropping, colour jittering, and blurring are some forms of image augmentations []. These augmentations are commonly adopted in training deep learning models. However, using these data augmentations is not trivial in some domains. For example, using blurring might remove important low-level information from medical images. More advanced approaches, such as MixUp [] and CutMix [], mix images and their labels accordingly [; ; ; ]. However, the resulting augmentations are not natural images anymore, and thus, act as out-of-distribution samples that will not be seen at test time. Another strand of research tailors the augmentation strategy through a learning process to fit the training data [; ; ]. Unlike the above methods, we propose to utilize pre-trained latent diffusion models to generate hard negatives (in contrast to generic augmentations) through a noise adaptation strategy discussed in Section 3.

Data Augmentation with Generative Models. Using synthesized images from generative models to augment training data has been studied before in many domains [; ], including domain adaptation [], visual alignment [], and mitigation of dataset bias [; ; ]. For example, [] introduces a methodology aimed at enhancing test set evaluation through augmentation. While previous methods predominantly relied on GANs [; ; ] as the generative model, more recent studies promote using diffusion models to augment the data [; ; ; ; ; ; ; ; ; ; ]. More specifically, [; ; ; ] study the effectiveness of text-to-image diffusion models in data augmentation by diversification of each class with synthetic images. [] leverages a text-to-image diffusion model and fine-tunes it on the downstream dataset using textual-inversion [] to increase the diversity of existing samples. [] also utilizes a text-to-image diffusion model, but with a BLIP [] model to generate meaningful captions from the existing images. [] utilizes diffusion models for augmentation to correct model mistakes. [] uses CLIP [] to filter generated images. [] utilizes text-based diffusion and a large language model (LLM) to diversify the training data. [] uses an LLM to generate text descriptions of failure modes associated with spurious correlations, which are then used to generate synthetic data through generative models. The challenge here is that the LLM has little understanding of such failure scenarios and contexts.

We take a completely different approach here, without replying on any extra source of information (e.g., through an LLM). Inspired by image editing approaches such as Boomerang [] and SDEdit [], we propose to adaptively guide a latent diffusion model to generate *hard negatives* images [; ] on a per-sample basis per category. In a nutshell, the aforementioned studies focus on improving the diversity of each class with effective prompts and diffusion models, however, we focus on generating effective *hard negative* samples for each class by combining two sources of contradicting information (images from the source category and text prompt from the target category).

Language Guided Recognition Models. Vision-Language foundation models (VLMs) [; ; ; ; ; ] utilize human language to guide the generation of images or to extract features from images that are aligned with human language.
For example, CLIP [] shows decent zero-shot performance on many downstream tasks by matching images to their text descriptions. Some recent works improve the utilization of human language in the prompt [; ], and others use a diffusion model directly as a classifier []. Similar to the above, we use a foundation model (Stable Diffusion 1.5 []) to improve the downstream task. Concretely, we utilize category names of the downstream tasks to augment their associate training data with hard negative samples.

Few-Shot Learning. In Few-shot Learning (FSL), we pre-train a model with abundant data to learn a rich representation, then fine-tune it on new tasks with only a few available samples. In supervised FSL [; ; ; ; ; ; ; ; ], pretraining is done on a labeled dataset, whereas in unsupervised FSL [; ; ; ; ; ; ; ; ] the pretraining has to be conducted on an unlabeled dataset posing an extra challenge in the learning paradigm and neighboring these methods closer to the realm of self-supervised learning. Even though FSL is not of primal interest in this work, we assess the impact of GeNIe on a number of few-shot scenarios and state-of-the-art baselines by accentuating on its impact on the few-shot inference stage.

## 3 Proposed Method: GeNIe

Given a source image $X_{S}$ from category S = $\small{<}\texttt{source category}\small{>}$, we are interested in generating a target image $X_{r}$ from category $T=\small{<}\texttt{target category}\small{>}$. In doing so, we intend to ensure the low-level visual features or background context of the source image are preserved, so that we generate samples that would serve as *hard negatives* for the *source* image. To this aim, we adopt a conditional latent diffusion model (such as Stable Diffusion, []) conditioned on a text prompt of the following format “A photo of a $T=\small{<}\texttt{target category}\small{>}$”.

Key Idea. GeNIe in its basic form is a simple yet effective augmentation sample generator for improving a classifier $f_{\theta}(.)$ with the following two key aspects: (i) inspired by [; ] instead of adding the full amount of noise $\sigma_{max}$ and going through all $N_{max}$ (being typically $50$) steps of denoising, we use less amount of noise ($r\sigma_{max}$, with $r\in(0,1)$) and consequently fewer number of denoising iterations ($\lfloor rN_{max}\rfloor$); (ii) we prompt the diffusion model with a $P$ mandating a target category $T$ different than the source $S$. Hence, we denote the conditional diffusion process as $X_{r}=$ STDiff$(X_{S},P,r)$. In such a construct, the proximity of the final decoded image $X_{r}$ to the source image $X_{S}$ or the target category defined through the text prompt $P$ depends on $r$. Hence, by controlling the amount of noise, we can generate images that blend characteristics of both the text prompt $P$ and the source image $X_{S}$. If we do not provide much of visual details in the text prompt (e.g., desired background, etc.), we expect the decoded image $X_{r}$ to follow the details of $X_{S}$ while reflecting the semantics of the text prompt $P$. We argue, and demonstrate later, that the newly generated samples can serve as *hard negative* examples for the source category $S$ since they share the low-level features of $X_{S}$ while representing the semantics of the target category, $T$. Notably, the source category $S$ can be randomly sampled or be carefully extracted from the confusion matrix of $f_{\theta}(.)$ based on real training data. The latter might result in even *harder negative* samples being now cognizant of model confusions. Finally, we will append our initial dataset with the newly generated hard negative samples through GeNIe and (re)train the classifier model.

Figure: Figure 2: Effect of noise ratio, $r$, in GeNIe: we employ GeNIe to generate augmentations for the target classes (motorcycle and cat) with varying $r$. Smaller $r$ yields images closely resembling the source semantics, creating an inconsistency with the intended target label. By tracing $r$ from 0 0 to $1$, augmentations gradually transition from source image characteristics to the target category. However, a distinct shift from the source to the target occurs at a specific $r$ that may vary for different source images or target categories. For more examples, please refer to Fig. A5.
Refer to caption: extracted/5983662/figures/noise_effect_vis5.jpg

Enhancing GeNIe: GeNIe-Ada. One of the remarkable aspects of GeNIe lies in its simple application, requiring only $X_{S}$, $P$, and $r$. However, selecting the appropriate value for $r$ poses a challenge as it profoundly influences the outcome. When $r$ is small, the resulting $X_{r}$ tends to closely resemble $X_{S}$, and conversely, when $r$ is large (closer to $1$), it tends to resemble the semantics of the target category. This phenomenon arises because a smaller noise level restricts the capacity of the diffusion model to deviate from the semantics of the input $X_{S}$. Thus, a critical question emerges: how can we select $r$ for a particular source image to generate samples that preserve the low-level semantics of the source category $S$ in $X_{S}$ while effectively representing the semantics of the target category $T$? We propose a method to determine an ideal value for $r$.

Our intuition suggests that by varying the noise ratio $r$ from 0 0 to $1$, $X_{r}$ will progressively resemble category $S$ in the beginning and category $T$ towards the end. However, somewhere between 0 0 and $1$, $X_{r}$ will undergo a rapid transition from category $S$ to $T$. This phenomenon is empirically observed in our experiments with varying $r$, as depicted in Fig. 2. Although the exact reason for this rapid change remains uncertain, one possible explanation is that the intermediate points between two categories reside far from the natural image manifold, thus, challenging the diffusion model’s capability to generate them. Ideally, we should select $r$ corresponding to just after this rapid semantic transition, as at this point, $X_{r}$ exhibits the highest similarity to the source image while belonging to the target category.

We propose to trace the semantic trajectory between $X_{S}$ and $X_{T}$ through the lens of the classifier $f_{\theta}(.)$. As shown in Algorithm 1, assuming access to the classifier backbone $f_{\theta}(.)$ and at least one example $X_{T}$ from the target category, we convert both $X_{S}$ and $X_{T}$ into their respective latent vectors $Z_{S}$ and $Z_{T}$ by passing them through $f_{\theta}(.)$. Then, we sample $M$ values for $r$ uniformly distributed $\in(0,1)$, generating their corresponding $X_{r}$ and their latent vectors $Z_{r}$ for all those $r$. Subsequently, we calculate $d_{r}=\frac{(Z_{r}-Z_{S})^{T}(Z_{T}-Z_{S})}{||Z_{T}-Z_{S}||_{2}}$ as the distance between $Z_{r}$ and $Z_{S}$ projected onto the vector connecting $Z_{S}$ and $Z_{T}$.
Our hypothesis posits that the rapid semantic transition corresponds to a sharp change in this projected distance. Therefore, we sample $n$ values for $r$ uniformly distributed between 0 0 and $1$, and analyze the variations in $d_{r}$. We identify the largest gap in $d_{r}$ and select the $r$ value just after the gap when increasing $r$, as detailed in Algorithm 1 and illustrated in Fig. 3.

Figure: Algorithm 1 GeNIe-Ada
Refer to caption: extracted/5983662/figures/geniepp_teaser3.jpg

## 4 Experiments

Since the impact of augmentation is more pronounced when the training data is limited, we evaluate the impact of GeNIe on Few-Shot classification in Section 4.1, Long-Tailed classification in Section 4.2, and fine-grained classification in Section 4.3. For GeNIe-Ada in all scenarios, we utilize GeNIe to generate augmentations from the noise level set $\{0.5,0.6,0.7,0.8,0.9\}$. The selection of the appropriate noise level per source image and target is adaptive, achieved through Algorithm 1.

Baselines. We use Stable Diffusion 1.5 [] as our base diffusion model. In all settings, we use the same prompt format to generate images for the target class: i.e., “A photo of a $\small{<}\texttt{target category}\small{>}$”, where we replace the target category with the target category label. We generate $512\times 512$ images for all methods. For fairness in comparison, we generate the same number of new images for each class. We use a single NVIDIA RTX $3090$ for image generation. We consider $4$ diffusion-based baselines and a suite of traditional data augmentation baselines.

Img2Img [; ]:
We sample an image from a target class, add noise to its latent representation and then pass it along with a prompt for the target category through reverse diffusion. The focus here is on a target class for which we generate extra positive samples. Adding large amount of noise leads to generating an image less similar to the original image. We use two different noise magnitudes for this baseline: $r=0.3$ and $r=0.7$ and denote them by Img2Img^L and Img2Img^H, respectively.

Txt2Img [; ]: For this baseline, we omit the forward diffusion process and only use the reverse process starting from a text prompt for the target class of interest. This is similar to the base text-to-image generation strategy adopted in [; ; ; ; ]. Fig. 4
illustrates a set of generated augmentation examples for Txt2Img, Img2Img, and GeNIe.

DAFusion []: In this method, an embedding is optimized with a set of images for each class to correspond to the classes in the dataset. This approach is introduced in Textual Inversion []. We optimize an embedding for 5000 iterations for each class in the dataset, followed by augmentation similar as the DAFusion method.

Cap2Aug[]: It is a recent diffusion-based data augmentation strategy that uses image captions as text prompts for an image-to-image diffusion model.

Traditional Data Augmentation: We consider both weak and strong traditional augmentations. More specifically, for weak augmentation we use random resize crop with scaling $\in[0.2,1.0]$ and horizontal flipping. For strong augmentation, we consider random color jitter, random grayscale, and Gaussian blur. For the sake of completeness, we also compare against data augmentations such as CutMix [] and MixUp [] that combine two images together.

Figure: Figure 4: Visualization of Generative Samples: We compare GeNIe with two baselines: Img2Img^L augmentation: both image and text prompt are from the same category. Adding noise does not change the image much, so they are not hard examples. Txt2Img augmentation: We simply use the text prompt only to generate an image for the desired category (e.g., using a text2image method). Such images may be far from the domain of our task since the generation is not informed by any visual data from our task. GeNIe augmentation: We use the target category name in the text prompt only along with the source image.
Refer to caption: extracted/5983662/figures/genie_vis_small.jpg

### 4.1 Few-shot Classification

We assess the impact of GeNIe compared to other augmentations in a number of few-shot classification (FSL) scenarios, where the model has to learn only from the samples contained in the ($N$-way, $K$-shot) support set and infer on the query set. Note that this corresponds to an inference-only FSL setting where a pretraining stage on an abundant dataset is discarded. The goal is to assess how well the model can benefit from the augmentations while keeping the original $N\times K$ samples intact.

Datasets. We conduct our few-shot experiments on two most commonly adopted few-shot classification datasets: mini-Imagenet [] and tiered-Imagenet []. mini-Imagenet is a subset of ImageNet [] for few-shot classification. It contains $100$ classes with $600$ samples each. We follow the predominantly adopted settings of [; ] where we split the entire dataset into $64$ classes for training, $16$ for validation and $20$ for testing. tiered-Imagenet is a larger subset of ImageNet with $608$ classes and a total of $779,165$ images, which are grouped into $34$ higher-level nodes in the ImageNet human-curated hierarchy. This set of nodes is partitioned into $20$, $6$, and $8$ disjoint sets of training, validation, and testing nodes, and the corresponding classes form the respective meta-sets.

Evaluation. To quantify the impact of different augmentation methods, we evaluate the test-set accuracies of a state-of-the-art unsupervised few-shot learning method with GeNIe and compare them against the accuracies obtained using other augmentation methods. Specifically, we use UniSiam [] pre-trained with ResNet-18, ResNet-34 and ResNet-50 backbones and follow its evaluation strategy of fine-tuning a logistic regressor to perform ($N$-way, $K$-shot) classification on the test sets of mini- and tiered-Imagenet. Following [], an episode consists of a labeled support-set and an unlabelled query-set. The support-set contains $N$ randomly sampled classes where each class contains $K$ samples, whereas the query-set contains $Q$ randomly sampled unlabeled images per class. We conduct our experiments on the two most commonly adopted settings: ($5$-way, $1$-shot) and ($5$-way, $5$-shot) classification settings. Following the literature, we sample $16$-shots per class for the query set in both settings. We report the test accuracies along with the $95\%$ confidence interval over $600$ and $1000$ episodes for mini-ImageNet and tiered-ImageNet, respectively.

**Table 1: mini-ImageNet: We use our augmentations on ($5$-way, $1$-shot) and ($5$-way, $5$-shot) few-shot settings of mini-Imagenet dataset with 3 different backbones (ResNet-18, 34, and 50). We compare with various baselines and show that our augmentations with UniSiam outperform all the baselines including Txt2Img and DAFusion augmentation. The number of generated images per class is 4 for 1-shot and 20 for 5-shot settings.**
| ResNet-18 |  |  |  |  |
| --- | --- | --- | --- | --- |
| Augmentation | Method | Pre-training | 1-shot | 5-shot |
| - | iDeMe-Net [] | sup. | 59.1$\pm$0.9 | 74.6$\pm$0.7 |
| - | Robust + dist [] | sup. | 63.7$\pm$0.6 | 81.2$\pm$0.4 |
| - | AFHN [] | sup. | 62.4$\pm$0.7 | 78.2$\pm$0.6 |
| Weak | ProtoNet+SSL [] | sup.+ssl | - | 76.6 |
| Weak | Neg-Cosine [] | sup. | 62.3$\pm$0.8 | 80.9$\pm$0.6 |
| - | Centroid Align[] | sup. | 59.9$\pm$0.7 | 80.4$\pm$0.7 |
| - | Baseline [] | sup. | 59.6$\pm$0.8 | 77.3$\pm$0.6 |
| - | Baseline++ [] | sup. | 59.0$\pm$0.8 | 76.7$\pm$0.6 |
| Weak | PSST [] | sup.+ssl | 59.5$\pm$0.5 | 77.4$\pm$0.5 |
| Weak | UMTRA [] | unsup. | 43.1$\pm$0.4 | 53.4$\pm$0.3 |
| Weak | ProtoCLR [] | unsup. | 50.9$\pm$0.4 | 71.6$\pm$0.3 |
| Weak | SimCLR [] | unsup. | 62.6$\pm$0.4 | 79.7$\pm$0.3 |
| Weak | SimSiam [] | unsup. | 62.8$\pm$0.4 | 79.9$\pm$0.3 |
| Weak | UniSiam+dist [] | unsup. | 64.1$\pm$0.4 | 82.3$\pm$0.3 |
| Weak | UniSiam [] | unsup. | 63.1$\pm$0.8 | 81.4$\pm$0.5 |
| Strong | UniSiam [] | unsup. | 62.8$\pm$0.8 | 81.2$\pm$0.6 |
| CutMix [] | UniSiam [] | unsup. | 62.7$\pm$0.8 | 80.6$\pm$0.6 |
| MixUp [] | UniSiam [] | unsup. | 62.1$\pm$0.8 | 80.7$\pm$0.6 |
| Img2Img^L[] | UniSiam [] | unsup. | 63.9$\pm$0.8 | 82.1$\pm$0.5 |
| Img2Img^H[] | UniSiam [] | unsup. | 69.1$\pm$0.7 | 84.0$\pm$0.5 |
| Txt2Img[; ] | UniSiam [] | unsup. | 74.1$\pm$0.6 | 84.6$\pm$0.5 |
| DAFusion [] | UniSiam [] | unsup. | 64.3$\pm$1.8 | 82.0$\pm$1.4 |
| GeNIe (Ours) | UniSiam [] | unsup. | 75.5$\pm$0.6 | 85.4$\pm$0.4 |
| GeNIe-Ada (Ours) | UniSiam [] | unsup. | 76.8$\pm$0.6 | 85.9$\pm$0.4 |

Implementation Details: GeNIe generates augmented images for each class using images from all other classes as the source image. We use $r=0.8$ in our experiments. We generate $4$ samples per class as augmentations in the $5$-way, $1$-shot setting and $20$ samples per class as augmentations in the $5$-way, $5$-shot setting. For the sake of a fair comparison, we ensure that the total number of labelled samples in the support set after augmentation remains the same across all different traditional and generative augmentation methodologies. Due to the expensive training of embeddings for each class in each episode, we only evaluated the DA-Fusion baseline on the first 100 episodes.

Results: The results on mini-Imagenet and tiered-Imagenet for both ($5$-way, $1$ and $5$-shot) settings are summarized in Table 1 and Table 3, respectively. Regardless of the choice of backbone, we observe that GeNIe helps consistently improve UniSiam’s performance and outperform other supervised and unsupervised few-shot classification methods as well as other diffusion-based [; ; ; ] and classical [; ] data augmentation techniques on both datasets, across both ($5$-way, $1$ and $5$-shot) settings. Our noise adaptive method of selecting optimal augmentations per source image (GeNIe-Ada) further improves GeNIe’s performance across all three backbones, both few-shot settings, and both datasets (mini and tiered-Imagenet). Few-shot accuracies for ResNet-34 computed on tieredImagenet are reported in Section A.2 of the appendix. Note that employing CutMix and MixUp seems to lead to performance degradation compared to weak augmentations, probably due to overfitting since these methods can only choose from $4$ other classes to mix.

### 4.2 Long-Tailed Classification

We evaluate our method on long-tailed data, where the number of instances per class is unbalanced, with most categories having limited samples (tail). Our goal is to mitigate this bias by augmenting the tail of the distribution with generated samples. We evaluate GeNIe using two different backbones and methods: the ViT architecture with LViT [], and ResNet50 with VL-LTR [].

Following LViT [], we first train an MAE [] and ViT on the unbalanced dataset without any augmentation. Next, we train the Balanced Fine-Tuning stage of LViT by incorporating the augmentation data generated using GeNIe or other baselines. For ResNet50, we use VL-LTR code to fine-tune the CLIP [] ResNet50 pretrained backbone with generated augmentations by GeNIe.

Dataset: We perform experiments on ImageNet-LT []. It contains $115.8$K images from $1,000$ categories. The number of images per class varies from $1280$ to $5$. Imagenet-LT classes can be divided into $3$ groups: “Few” with less than $20$ images, “Med” with $20-100$ images, and “Many” with more than $100$ images. Imagenet-LT uses the same validation set as ImageNet. We augment “Few” categories only and limit the number of generated images to $50$ samples per class. For GeNIe, instead of randomly sampling the source images from other classes, we use a confusion matrix on the training data to find the top-$4$ most confused classes and only consider those classes for random sampling of the source image. The source category may be from “Many”, “Med”, or “Few sets”.

Results: Augmenting training data with GeNIe-Ada improves accuracy on the “Few” set by $11.7\%$ and $4.4\%$ compared with LViT only and LViT with Txt2Img augmentation baselines respectively.
In ResNet50, GeNIe-Ada outperforms Cap2Aug baseline in “Few” categories by $7.6\%$. The results are summarized in Table 3. Please refer to Section  A.3 for implementation details.

**Table 2: tiered-ImageNet: Accuracies (% $\pm$ std) for $5$-way, $1$-shot and $5$-way, $5$-shot classification settings on the test-set. We compare against various SOTA supervised and unsupervised few-shot classification baselines as well as other augmentation methods, with UniSiam [] pre-trained ResNet-18,50 backbones.**
| ResNet-18 |  |  |  |  |
| --- | --- | --- | --- | --- |
| Augmentation | Method | Pre-training | 1-shot | 5-shot |
| Weak | SimCLR[] | unsup. | 63.4$\pm$0.4 | 79.2$\pm$0.3 |
| Weak | SimSiam [] | unsup. | 64.1$\pm$0.4 | 81.4$\pm$0.3 |
| Weak | UniSiam [] | unsup. | 63.1$\pm$0.7 | 81.0$\pm$0.5 |
| Strong | UniSiam [] | unsup. | 62.8$\pm$0.7 | 80.9$\pm$0.5 |
| CutMix [] | UniSiam [] | unsup. | 62.1$\pm$0.7 | 78.9$\pm$0.6 |
| MixUp [] | UniSiam [] | unsup. | 62.1$\pm$0.7 | 78.4$\pm$0.6 |
| Img2Img^L[] | UniSiam [] | unsup. | 63.9$\pm$0.7 | 81.8$\pm$0.5 |
| Img2Img^H[] | UniSiam [] | unsup. | 68.7$\pm$0.7 | 83.5$\pm$0.5 |
| Txt2Img[] | UniSiam [] | unsup. | 72.9$\pm$0.6 | 84.2$\pm$0.5 |
| DAFusion [] | UniSiam [] | unsup. | 62.6$\pm$2.1 | 81.0$\pm$1.5 |
| GeNIe(Ours) | UniSiam [] | unsup. | 73.6$\pm$0.6 | 85.0$\pm$0.4 |
| GeNIe-Ada(Ours) | UniSiam [] | unsup. | 75.1$\pm$0.6 | 85.5$\pm$0.5 |
| ResNet-50 |  |  |  |  |
| Weak | PDA+Net [] | unsup. | 69.0$\pm$0.9 | 84.2$\pm$0.7 |
| Weak | Meta-DM [] | unsup. | 69.6$\pm$0.4 | 86.5$\pm$0.3 |
| Weak | UniSiam + dist [] | unsup. | 69.6$\pm$0.4 | 86.5$\pm$0.3 |
| Weak | UniSiam [] | unsup. | 66.8$\pm$0.7 | 84.7$\pm$0.5 |
| Strong | UniSiam [] | unsup. | 66.5$\pm$0.7 | 84.5$\pm$0.5 |
| CutMix [] | UniSiam [] | unsup. | 66.0$\pm$0.7 | 83.3$\pm$0.5 |
| MixUp [] | UniSiam [] | unsup. | 66.1$\pm$0.5 | 84.1$\pm$0.8 |
| Img2Img^L[] | UniSiam [] | unsup. | 67.8$\pm$0.7 | 85.3$\pm$0.5 |
| Img2Img^H[] | UniSiam [] | unsup. | 72.4$\pm$0.7 | 86.7$\pm$0.4 |
| Txt2Img[] | UniSiam [] | unsup. | 77.1$\pm$0.6 | 87.3$\pm$0.4 |
| DAFusion [] | UniSiam [] | unsup. | 66.5$\pm$2.2 | 84.8$\pm$1.4 |
| GeNIe (Ours) | UniSiam [] | unsup. | 78.0$\pm$0.6 | 88.0$\pm$0.4 |
| GeNIe-Ada (Ours) | UniSiam [] | unsup. | 78.8$\pm$0.6 | 88.6$\pm$0.6 |

### 4.3 Fine-grained Few-shot Classification

To further investigate the impact of the proposed method, we compare GeNIe with other text-based data augmentation techniques across four distinct fine-grained datasets in a $20$-way, $1$-shot classification setting. We employ the pre-trained DINOV2 ViT-G [] backbone as a feature extractor to derive features from training images. Subsequently, an SVM classifier is trained on these features, and we report the Top-$1$ accuracy of the model on the test set.

Datasets:
We assess our method on several datasets: Food101 [] with $101$ classes of foods, CUB200 [] with $200$ bird species classes, Cars196 [] with $196$ car model classes, and FGVC-Aircraft [] with $41$ aircraft manufacturer classes. We provide detailed information around fine-grained datasets in Table A1. The reported metric is the average Top-$1$ accuracy over $100$ episodes. Each episode involves sampling $20$ classes and $1$-shot from the training set, with the final model evaluated on the respective test set.

Implementation Details:
We enhance the basic prompt by incorporating the superclass name for the fine-grained dataset: “A photo of a $\small{<}\texttt{target class}\small{>}$, a type of $\small{<}\texttt{superclass}\small{>}$". For instance, in the food dataset and the burger class, our prompt reads: “A photo of a burger, a type of food." No additional augmentation is used for generative methods in this context. We generate $19$ samples for both cases of our method and also the baseline with weak augmentation.

**Table 4: Few-shot Learning on Fine-grained dataset: We utilize an SVM classifier trained atop the DINOV2 ViT-G pretrained backbone, reporting Top-1 accuracy for the test set of each dataset. The baseline is an SVM trained on the same backbone using weak augmentation. Across all datasets, GeNIe surpasses this baseline.**
| Method | Birds | Cars | Foods | Aircraft |
| --- | --- | --- | --- | --- |
|  | CUB200 [] | Cars196 [] | Food101 [] | Aircraft [] |
| Baseline | 90.3 | 49.8 | 82.9 | 29.2 |
| Img2Img^L[] | 90.7 | 50.4 | 87.4 | 31.0 |
| Img2Img^H[] | 91.3 | 56.4 | 91.7 | 34.7 |
| Txt2Img[] | 92.0 | 81.3 | 93.0 | 41.7 |
| GeNIe (r=$0.5$) | 92.0 | 84.6 | 91.5 | 39.8 |
| GeNIe (r=$0.6$) | 92.2 | 87.1 | 92.5 | 45.0 |
| GeNIe (r=$0.7$) | 92.5 | 87.9 | 92.9 | 47.0 |
| GeNIe (r=$0.8$) | 92.5 | 87.7 | 93.1 | 46.5 |
| GeNIe (r=$0.9$) | 92.4 | 87.1 | 93.1 | 45.7 |
| GeNIe-Ada | 92.6 | 87.9 | 93.1 | 46.9 |

Results: Table 4 summarizes the results. GeNIe outperforms all other baselines, including Txt2Img, by margins upto $0.5\%$ on CUB200 [], $6.6\%$ on Cars196 [], $0.1\%$ on Food101 [] and $5.3\%$ on FGVC-Aircraft []. Notably, GeNIe exhibits great effectiveness in more challenging datasets, outperforming the baseline with traditional augmentation by about $38\%$ for the Cars dataset and by roughly $17\%$ for the Aircraft dataset. It can be observed here that GeNIe-Ada performs on-par with GeNIe with a fixed noise level, eliminating the necessity for noise level search in GeNIe.

### 4.4 Ablation and Further Analysis

Semantic Shift from Source to Target Class. The core motivation behind GeNIe-Ada is that by varying the noise ratio $r$ from 0 0 to $1$, augmented sample $X_{r}$ will progressively shift its semantic category from source ($S$) in the beginning to target category ($T$) towards the end. However, somewhere between 0 0 and $1$, $X_{r}$ will undergo a rapid transition from $S$ to $T$. To demonstrate this hypothesis empirically, in Figs. 5 and A3, we visualize pairs of source images and target categories with their respective GeNIe generated augmentations for different noise ratios $r$, along with their corresponding PCA-projected embedding scatter plots (on the far left). We extract embeddings for all the images using a DINOv2 ViT-G pretrained backbone, which we assume as an oracle model in identifying the right category. We observe that as $r$ increases from $0.3$ to $0.8$, the images transition to embody more of the target category’s semantics while preserving the contextual features of the source image. This transition of semantics can also be observed in the embedding plots (on the left) where they consistently shift from the proximity of the source image (blue star) to the target class’s centroid (red cross) as the noise ratio $r$ increases.
The sparse distribution of points within $r=[0.4,0.6]$ for the first image and $r=[0.2,0.4]$ for the second image aligns with our intuition of a rapid transition from category $S$ to $T$, thus empirically affirming our motivation behind GeNIe-Ada.

To further establish this, in Fig. 6, we demonstrate the efficacy of GeNIe in generating hard negatives at the decision boundaries of an SVM classifier, which is trained on the labelled support set of the few-shot tasks of mini-Imagenet, without any augmentations. We then plot source and target class probabilities ($P(Y_{S}|X_{r})$ and $P(Y_{T}|X_{r})$, respectively) of the generated augmentation samples $X_{r}$. For both $r=0.6$ and $0.7$, there is significant overlap between $P(Y_{S}|X_{r})$ and $P(Y_{T}|X_{r})$, making it difficult for the classifier to decide the correct class. On the right-hand-side, GeNIe-Ada automatically selects the best $r$ resulting in the most overlap between the two distributions, thus offering the hardest negative sample among the considered $r$ values (for more details see A.1). Note that a large overlap between distributions is not sufficient to call the generated samples hard negatives because they should also belong to the target category. This is, however, confirmed by the high Oracle accuracy in Table 5 (elaborated in detail in the following paragraph) which verifies that majority of the generated augmentation samples do belong to the target category.

Label consistency of the generated samples.
The choice of noise ratio $r$ is important in producing hard negative examples. In Table 5, we present the accuracy of the GeNIe model across various noise ratios, alongside the oracle accuracy, which is an ImageNet pre-trained DeiT-Base [] classifier. We observe a decline in the label consistency of generated data (quantified by the performance of the oracle model) when decreasing the noise level. Reducing $r$ also results in a degradation in the performance of the final few-shot model ($87.2\%\rightarrow 77.6\%$) corroborating that an appropriate choice of $r$ plays a crucial role in our design strategy. We investigate this further in the following paragraph.

Figure: Figure 5: Embedding visualizations of generative augmentations: We pass all generative augmentations through DINOv2 ViT-G (serving as an oracle) to extract their corresponding embeddings and visualize them with PCA. As shown, the extent of semantic shifts varies based on both the source image and the target class.
Refer to caption: extracted/5983662/figures/genie_vis_small2.jpg

Figure: Figure 6: Why GeNIe augmentations are challenging? While deciding which class the generated augmentations ($X_{r}$) belong to is already difficult within $r=[0.6,0.7]$ (due to high overlap between $P(Y_{S}|X_{r})$ and $P(Y_{T}|X_{r})$), GeNIe-Ada selects the best noise threshold ($r^{*}$) offering the hardest negative sample.
Refer to caption: extracted/5983662/figures/genie-nada-p.png

Effect of Noise in GeNIe.
We examine the impact of noise on the performance of the few-shot model in Table 5. Noise levels $r\in[0.7,0.8]$ yield the best performance. Conversely, utilizing noise levels below $0.7$ diminishes performance due to label inconsistency, as is demonstrated in Table 5 and Fig 5. As such, determining the appropriate noise level is pivotal for the performance of GeNIe to be able to generate challenging hard negatives while maintaining label consistency. An alternative approach to finding the optimal noise level involves using GeNIe-Ada to adaptively select the noise level for each source image and target class. As demonstrated in Tables 5 and 4, GeNIe-Ada achieves performance that is comparable to or surpasses that of GeNIe with fixed noise levels.

Effect of Diffusion Models in GeNIe.
We have tried experimenting with both smaller as well as more recent diffusion models. More specifically, we have used Stable Diffusion XL-Turbo to generate hard-negatives through GeNIe and GeNIe-Ada. Few-shot classification results on miniImagenet with these augmentations are shown in Table 5. The accuracies follow a similar trend to that of Table 1, where Stable Diffusion 1.5 was used to generate augmentations. GeNIe-Ada improves UniSiam’s few-shot performance the most as compared to GeNIe with different noise ratios $r$, and even when compared to Txt2Img. This empirically indicates the robustness of GeNIe and GeNIe-Ada to different diffusion engines. Note that, Stable Diffusion XL-Turbo by default uses $4$ steps for the sake of optimization, and to ensure we can have the right granularity for the choice of $r$ we have set the number of steps to $10$. That is already 5 times faster than the standard Stable Diffusion v1.5 with $50$ steps. Our experiments with Stable Diffusion v3 (which is a totally different model with a Transformers backbone) also in Table 5 also convey the same message. As such, we believe our approach is generalizable across different diffusion models.

**Table 5: Effect of Noise and Diffusion Models in GeNIe: We use the same setting as in Table 1 to study the effect of the amount of noise. As expected (also shown in Fig 5), small noise results in worse accuracy since some generated images may be from the source category rather than the target one. For $r=0.5$ only $73\%$ of the generated data is from the target category. This behaviour is also shown in Fig. 2. Notably, reducing the noise level below $0.7$ is associated with a decline in oracle accuracy and subsequent degradation in the performance of the final few-shot model. Note that the high oracle accuracy of GeNIe-Ada demonstrates its capability to adaptively select the noise level per source and target, ensuring semantic consistency with the intended target. To further demonstrate GeNIe’s ability to generalize across different diffusion models, we replace the diffusion model with SD3 and SDXL-Turbo. The resulting accuracies follow a similar trend to those in Table 1, confirming GeNIe’s advantage over Txt2Img across various diffusion models.**
| Method | Generative | Noise | ResNet-18 | ResNet-34 | ResNet-50 | Oracle |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Model | r= | 1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot | Acc |
| Txt2Img | SD 1.5 | - | 74.1$\pm$0.6 | 84.6$\pm$0.5 | 75.4$\pm$0.6 | 85.5$\pm$0.5 | 76.4$\pm$0.6 | 86.5$\pm$0.4 | - |
| GeNIe | SD 1.5 | 0.5 | 60.4$\pm$0.8 | 74.1$\pm$0.6 | 62.0$\pm$0.8 | 75.8$\pm$0.6 | 63.7$\pm$0.9 | 77.6$\pm$0.6 | 73.4$\pm$0.5 |
| GeNIe | SD 1.5 | 0.6 | 69.7$\pm$0.7 | 80.7$\pm$0.5 | 71.1$\pm$0.7 | 82.2$\pm$0.5 | 72.1$\pm$0.7 | 82.8$\pm$0.5 | 85.8$\pm$0.4 |
| GeNIe | SD 1.5 | 0.7 | 74.5$\pm$0.6 | 83.3$\pm$0.5 | 76.4$\pm$0.6 | 84.4$\pm$0.5 | 77.1$\pm$0.6 | 85.0$\pm$0.4 | 94.5$\pm$0.2 |
| GeNIe | SD 1.5 | 0.8 | 75.5$\pm$0.6 | 85.4$\pm$0.4 | 77.1$\pm$0.6 | 86.3$\pm$0.4 | 77.3$\pm$0.6 | 87.2$\pm$0.4 | 98.2$\pm$0.1 |
| GeNIe | SD 1.5 | 0.9 | 75.0$\pm$0.6 | 85.3$\pm$0.4 | 77.6$\pm$0.6 | 86.2$\pm$0.4 | 77.7$\pm$0.6 | 87.0$\pm$0.4 | 99.3$\pm$0.1 |
| GeNIe-Ada | SD 1.5 | Adaptive | 76.8$\pm$0.6 | 85.9$\pm$0.4 | 78.5$\pm$0.6 | 86.6$\pm$0.4 | 78.6$\pm$0.6 | 87.9$\pm$0.4 | 98.9$\pm$0.2 |
| Txt2Img | SDXL-Turbo | - | 72.5$\pm$0.3 | 82.1$\pm$0.6 | 76.2$\pm$0.2 | 84.4$\pm$0.3 | 76.7$\pm$0.6 | 85.9$\pm$0.5 | - |
| GeNIe | SDXL-Turbo | 0.5 | 61.2$\pm$0.5 | 73.5$\pm$0.2 | 61.5$\pm$0.2 | 74.9$\pm$0.3 | 63.1$\pm$0.2 | 76.5$\pm$0.6 | - |
| GeNIe | SDXL-Turbo | 0.6 | 70.2$\pm$0.2 | 79.3$\pm$0.4 | 71.2$\pm$0.7 | 81.4$\pm$0.6 | 73.2$\pm$0.2 | 82.4$\pm$0.5 | - |
| GeNIe | SDXL-Turbo | 0.7 | 73.1$\pm$0.3 | 83.5$\pm$0.5 | 76.1$\pm$0.6 | 85.3$\pm$0.4 | 77.2$\pm$0.6 | 84.2$\pm$0.4 | - |
| GeNIe | SDXL-Turbo | 0.8 | 74.2$\pm$0.3 | 85.1$\pm$0.3 | 76.9$\pm$0.4 | 85.5$\pm$0.5 | 78.7$\pm$0.6 | 87.7$\pm$0.4 | - |
| GeNIe | SDXL-Turbo | 0.9 | 73.9$\pm$0.4 | 84.9$\pm$0.7 | 76.6$\pm$0.7 | 84.2$\pm$0.6 | 78.1$\pm$0.5 | 87.0$\pm$0.4 | - |
| GeNIe-Ada | SDXL-Turbo | Adaptive | 75.1$\pm$0.3 | 87.1$\pm$0.8 | 78.9$\pm$0.5 | 85.2$\pm$0.5 | 79.0$\pm$0.6 | 88.6$\pm$0.2 | - |
| Txt2Img | SD 3 | - | 73.6$\pm$1.7 | 82.9$\pm$1.2 | 76.7$\pm$1.5 | 85.5$\pm$1.3 | 77.2$\pm$1.9 | 85.0$\pm$1.2 | - |
| GeNIe | SD 3 | 0.5 | 62.0$\pm$1.2 | 72.9$\pm$1.1 | 62.5$\pm$0.9 | 73.9$\pm$1.0 | 64.1$\pm$0.5 | 76.1$\pm$1.9 | - |
| GeNIe | SD 3 | 0.6 | 70.8$\pm$1.5 | 79.1$\pm$1.9 | 71.8$\pm$1.2 | 82.1$\pm$1.3 | 74.1$\pm$1.5 | 83.4$\pm$1.8 | - |
| GeNIe | SD 3 | 0.7 | 74.6$\pm$0.8 | 84.5$\pm$1.2 | 76.5$\pm$1.9 | 86.2$\pm$1.6 | 78.5$\pm$1.9 | 84.0$\pm$1.1 | - |
| GeNIe | SD 3 | 0.8 | 75.9$\pm$1.2 | 86.3$\pm$1.7 | 77.8$\pm$1.9 | 85.5$\pm$1.9 | 79.2$\pm$1.7 | 88.3$\pm$1.9 | - |
| GeNIe | SD 3 | 0.9 | 75.1$\pm$0.5 | 85.2$\pm$1.2 | 78.1$\pm$1.3 | 86.2$\pm$1.2 | 77.1$\pm$1.9 | 88.9$\pm$0.8 | - |
| GeNIe-Ada | SD 3 | Adaptive | 76.8$\pm$1.3 | 87.5$\pm$1.5 | 78.9$\pm$1.3 | 87.7$\pm$1.5 | 79.1$\pm$1.4 | 89.5$\pm$1.0 | - |

## 5 Concluding Remarks

GeNIe, for the first time to our knowledge, combines contradictory sources of information (a source image, and a different target category prompt) through a noise adjustment strategy into a conditional latent diffusion model to generate challenging augmentations, which can serve as hard negatives.

Limitation.
The required time to create augmentations through GeNIe is on par with any typical diffusion-based competitors [; ]; however, this is naturally slower than traditional augmentation techniques [; ]. This is not a bottleneck in offline augmentation strategies, but can be considered a limiting factor in real-time scenarios. Recent studies are already mitigating this through advancements in diffusion model efficiency [; ; ]. Another challenge present in any generative AI-based augmentation technique is the domain shift between the distribution of training data and the downstream context they might be used for augmentation. A possible remedy is to fine-tune the diffusion backbone on a rather small dataset from the downstream task.

Broader Impact. We believe ideas from GeNIe can have a significant impact when it comes to generating hard augmentations challenging and thus enhancing downstream tasks beyond classification. At the same time, just like any other generative model, GeNIe can also introduce inherent biases stemming from the training data used to build its diffusion backbone, which can reflect and amplify societal prejudices or inaccuracies. Therefore, it is crucial to carefully mitigate potential biases in generative models such as GeNIe to ensure a fair and ethical deployment of deep learning systems.

## Appendix A Appendix

### A.1 Analyzing GeNIe , GeNIe-Ada ’s Class-Probabilities

The core aim of GeNIe and GeNIe-Ada is to address the failure modes of a classifier by generating *challenging* samples located near the decision boundary of each class pair, which facilitates the learning process in effectively enhancing the decision boundary between classes. As summarized in Table 5 and illustrated in Fig. 5, we have empirically corroborated that GeNIe and GeNIe-Ada can respectively produce samples $X_{r},X_{r^{*}}$ that are negative with respect to the source image $X_{S}$, while semantically belonging to the class $T$.
To further analyze the effectiveness of GeNIe and GeNIe-Ada, we compare the source class-probabilities $P(Y_{S}|X_{r})$ and target-class probabilities $P(Y_{S}|X_{r})$ of augmented samples $X_{r}$.

Figure: Figure A1: Significant overlap between $P(Y_{S}|X_{r^{*}})$ and $P(Y_{T}|X_{r^{*}})$ indicates high class-confusion for augmented samples generated by GeNIe-Ada.
Refer to caption: extracted/5983662/figures/genie-ada-p.png

To compute these class probabilities, we first fit an SVM classifier (as followed in UniSiam []) only on the labelled support set embeddings of each episode in the miniImagenet test dataset. Then, we perform inference using each episode’s SVM classifier on its respective $X_{r}$’s and extract its class probabilities of belonging to its source class $S$ and target class $T$. These per augmentation-sample source and target class probabilities are then averaged for each episode for each $r\in\{0.5,0.6,0.7,0.8,0.9\}$ in the case of GeNIe and for the optimal $r=r^{*}$ per sample in the case of GeNIe-Ada, plotted as density plots in Fig. 6, Fig. A1, respectively. Fig. 6 illustrates that $P(Y_{S}|X_{r})$ and $P(Y_{T}|X_{r})$ have significant overlap in the case of $r\in\{0.6,0.7\}$ indicating class-confusion for $X_{r}$.

Furthermore, Fig. A1 illustrates that when using the optimal $r=r^{*}$ found by GeNIe-Ada per sample, $P(Y_{S}|X_{r})$ and $P(Y_{T}|X_{r})$ significantly overlap around probability scores of $0.2-0.45$, indicating class confusion for GeNIe-Ada augmentations. This corroborates with our analysis in Section 4.4, Table 5 and additionally empirically proves that the augmented samples generated by GeNIe for $r\in\{0.6,0.7\}$ and GeNIe-Ada for $r=r^{*}$ are actually located near the decision boundary of each class pair.

**Table A1: Train and test split details of the fine-grained datasets. We use the provided train set for few-shot task generation, and the provided test sets for our evaluation. Aircraft dataset uses the manufacturer hierarchy.**
| Dataset | Classes | Train | Test |
| --- | --- | --- | --- |
|  |  | samples | samples |
| CUB200 [] | 200 | 5994 | 5794 |
| Food101 [] | 101 | 75750 | 25250 |
| Cars [] | 196 | 8144 | 8041 |
| Aircraft [] | 41 | 6,667 | 3333 |

### A.2 Few-shot Classification with ResNet-34 on tiered Imagenet

**Table A2: tiered-ImageNet: Accuracies (% $\pm$ std) for $5$-way, $1$-shot and $5$-way, $5$-shot classification settings on the test-set. We compare against various SOTA supervised and unsupervised few-shot classification baselines as well as other augmentation methods, with UniSiam [] pre-trained ResNet-34 backbone.**
| ResNet-34 |  |  |  |  |
| --- | --- | --- | --- | --- |
| Augmentation | Method | Pre-training | 1-shot | 5-shot |
| Weak | MAML + dist [] | sup. | 51.7$\pm$1.8 | 70.3$\pm$1.7 |
| Weak | ProtoNet [] | sup. | 52.0$\pm$1.2 | 72.1$\pm$1.5 |
| Weak | UniSiam + dist [] | unsup. | 68.7$\pm$0.4 | 85.7$\pm$0.3 |
| Weak | UniSiam [] | unsup. | 65.0$\pm$0.7 | 82.5$\pm$0.5 |
| Strong | UniSiam [] | unsup. | 64.8$\pm$0.7 | 82.4$\pm$0.5 |
| CutMix [] | UniSiam [] | unsup. | 63.8$\pm$0.7 | 80.3$\pm$0.6 |
| MixUp [] | UniSiam [] | unsup. | 64.1$\pm$0.7 | 80.0$\pm$0.6 |
| Img2Img^L[] | UniSiam [] | unsup. | 66.1$\pm$0.7 | 83.1$\pm$0.5 |
| Img2Img^H[] | UniSiam [] | unsup. | 70.4$\pm$0.7 | 84.7$\pm$0.5 |
| Txt2Img[] | UniSiam [] | unsup. | 75.0$\pm$0.6 | 85.4$\pm$0.4 |
| DAFusion [] | UniSiam [] | unsup. | 64.1$\pm$2.1 | 82.8$\pm$1.4 |
| GeNIe (Ours) | UniSiam [] | unsup. | 75.7$\pm$0.6 | 86.0$\pm$0.4 |
| GeNIe-Ada (Ours) | UniSiam [] | unsup. | 76.9$\pm$0.6 | 86.3$\pm$0.2 |

We follow the same evaluation protocol here as mentioned in section 4.1. As summarized in Table A2, GeNIe and GeNIe-Ada outperform all other data augmentation techniques.

### A.3 Additional details of Long-Tail experiments

We present a comprehensive version of Table 3 to benchmark the performance with different backbone architectures (e.g., ResNet50) and to compare against previous long-tail baselines; this is detailed in Table A4.

Implementation Details of LViT: We download the pre-trained ViT-B of LViT [] and finetune it with Bal-BCE loss proposed therein on the augmented dataset. Training takes 2 hours on four NVIDIA RTX 3090 GPUs. We use the same hyperparameters as in [] for finetuning: $100$ epochs, $lr=0.008$, batch size of $1024$, CutMix and MixUp for the data augmentation.

Implementation Details of VL-LTR: We use the official code of VL-LTR [] for our experiments. We use a pre-trained CLIP ResNet-50 backbone. We followed the hyperparameters reported in VL-LTR []. We augment only “Few” category and train the backbone with the VL-LTR [] method. Training takes 4 hours on $8$ NVIDIA RTX 3090 GPUs.

**Table A3: Few-shot classification comparison of GeNIe-Ada with Txt2Img on miniImagenet.**
| Method | ResNet-18 | ResNet-34 | ResNet-50 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | 1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot |
| Txt2Img | 76.9$\pm$1.0 | 86.5$\pm$0.9 | 77.1$\pm$0.8 | 86.7$\pm$1.0 | 77.2$\pm$1.3 | 86.8$\pm$0.9 |
| GeNIe-Ada | 77.7$\pm$0.8 | 87.4$\pm$1.0 | 78.3$\pm$0.9 | 87.8$\pm$0.9 | 79.1$\pm$1.1 | 88.4$\pm$1.2 |

**Table A4: Long-Tailed ImageNet-LT: We compare different augmentation methods on ImageNet-LT and report Top-1 accuracy for “Few”, “Medium”, and “Many” sets. $\dagger$ indicates results with ResNeXt50. $*$: indicates training with 384 resolution so is not directly comparable with other methods with 224 resolution. On the “Few” set and LiVT method, our augmentations improve the accuracy by $11.7$ points compared to LiVT original augmentation and $4.4$ points compared to Txt2Img.**
| ResNet-50 |  |  |  |  |
| --- | --- | --- | --- | --- |
| Method | Many | Med. | Few | Overall Acc |
| CE [] | 64.0 | 33.8 | 5.8 | 41.6 |
| LDAM [] | 60.4 | 46.9 | 30.7 | 49.8 |
| c-RT [] | 61.8 | 46.2 | 27.3 | 49.6 |
| $\tau$-Norm [] | 59.1 | 46.9 | 30.7 | 49.4 |
| Causal [] | 62.7 | 48.8 | 31.6 | 51.8 |
| Logit Adj. [] | 61.1 | 47.5 | 27.6 | 50.1 |
| RIDE(4E)$\dagger$ [] | 68.3 | 53.5 | 35.9 | 56.8 |
| MiSLAS [] | 62.9 | 50.7 | 34.3 | 52.7 |
| DisAlign [] | 61.3 | 52.2 | 31.4 | 52.9 |
| ACE$\dagger$ [] | 71.7 | 54.6 | 23.5 | 56.6 |
| PaCo$\dagger$ [] | 68.0 | 56.4 | 37.2 | 58.2 |
| TADE$\dagger$ [] | 66.5 | 57.0 | 43.5 | 58.8 |
| TSC [] | 63.5 | 49.7 | 30.4 | 52.4 |
| GCL [] | 63.0 | 52.7 | 37.1 | 54.5 |
| TLC [] | 68.9 | 55.7 | 40.8 | 55.1 |
| BCL$\dagger$ [] | 67.6 | 54.6 | 36.6 | 57.2 |
| NCL [] | 67.3 | 55.4 | 39.0 | 57.7 |
| SAFA [] | 63.8 | 49.9 | 33.4 | 53.1 |
| DOC [] | 65.1 | 52.8 | 34.2 | 55.0 |
| DLSA [] | 67.8 | 54.5 | 38.8 | 57.5 |
| ResLT [] | 63.3 | 53.3 | 40.3 | 55.1 |
| PaCo [] | 68.2 | 58.7 | 41.0 | 60.0 |
| LWS [] | 62.2 | 48.6 | 31.8 | 51.5 |
| Zero-shot CLIP [] | 60.8 | 59.3 | 58.6 | 59.8 |
| DRO-LT [] | 64.0 | 49.8 | 33.1 | 53.5 |
| VL-LTR [] | 77.8 | 67.0 | 50.8 | 70.1 |
| Cap2Aug [] | 78.5 | 67.7 | 51.9 | 70.9 |
| GeNIe-Ada | 79.2 | 64.6 | 59.5 | 71.5 |
| ViT-B |  |  |  |  |
| LiVT* [] | 76.4 | 59.7 | 42.7 | 63.8 |
| ViT [] | 50.5 | 23.5 | 6.9 | 31.6 |
| MAE [] | 74.7 | 48.2 | 19.4 | 54.5 |
| DeiT [] | 70.4 | 40.9 | 12.8 | 48.4 |
| LiVT [] | 73.6 | 56.4 | 41.0 | 60.9 |
| LiVT + Img2Img^L | 74.3 | 56.4 | 34.3 | 60.5 |
| LiVT + Img2Img^H | 73.8 | 56.4 | 45.3 | 61.6 |
| LiVT + Txt2Img | 74.9 | 55.6 | 48.3 | 62.2 |
| LiVT + GeNIe (r=$0.8$) | 74.5 | 56.7 | 50.9 | 62.8 |
| LiVT + GeNIe-Ada | 74.0 | 56.9 | 52.7 | 63.1 |

Figure: Figure A2: Key components of GeNIe: (i) careful choice of $r$ and (ii) contradictory prompt are two key idea behind GeNIe
Refer to caption: extracted/5983662/figures/rebut_vis7.jpg

### A.4 Extra Computation of GeNIe-Ada

Given that GeNIe-Ada searches for the best hard-negative between multiple noise-ratios $r$’s, it naturally requires a higher compute budget than txt2Img that only uses $r=1$. For this experiment, we use GeNIe-Ada with $r\in\{0.6,0.7,0.8\}$ to compare with Txt2Img. Based on this, we only have $3$ paths, with steps of $0.1$), and for each of which we go through partial reverse diffusion process. E.g. for $r=0.6$ we do $30$ steps instead of standard $50$ steps of Stable Diffusion. This practically breaks down the total run-time of GeNIe-Ada to approximately $2$ times that of the standard reverse diffusion (GeNIe-Ada: total $r=0.6+0.7+0.8=2.1$ vs Txt2Img total $r=1$). Thus, to be fair, we generate twice as many Txt2Img augmentations as compared to GeNIe-Ada to keep a constant compute budget across the methods, following your suggestion. The results are shown in Table A3. As can be seen, even in this new setting, GeNIe-Ada offers a performance improvement of $0.8\%$ to $1.9\%$ across different backbones.

### A.5 How does GeNIe control which features are retained or changed?

We instruct the diffusion model to generate an image by combining the latent noise of the source image with the textual prompt of the target category. This combination is controlled by the amount of added noise and the number of reverse diffusion iterations. This approach aims to produce an image that aligns closely with the semantics of the target category while preserving the background and features from the source image that are unrelated to the target.

To demonstrate this, in Figure A2, We are progessivley moving towards the two key components of GeNIe: (i) careful choice of $r$ and (ii) contradictory prompt. The input image is a bird in a cage. The top row shows a Stable Diffusion model, unprompted. As can be seen, such a model can generate anything (irrespective of the input image) with a large $r$. Now prompting the same model with “a photo of a bird” allows the model to preserve low-level and contextual features of the input image (up to $r=0.7$ and $0.8$), until for a large $r\geq 0.9$ it returns a bird but the context has nothing to do with the source input. This illustrates how a careful choice of $r$ can help preserve such low-level features, and is a key idea behind GeNIe. However, we also need a semantic switch to a different target class as shown in the last row where a hardly seen image of a dog in a cage is generated by a combination of a careful choice of $r$ and the contradictory prompt - leading to the full mechanics of GeNIe. This sample now serves as hard negative for the source image (bird class).

### A.6 More Visualizations

Additional qualitative results resembling the style presented in Fig. 4 are presented in Fig. A4, and more visuals akin to Fig. 2 can be found in Fig. A5. Moreover, we also present more visualization similar to the style in Fig. 5 in Fig. A3.

Figure: Figure A3: Effect of noise in GeNIe: Similar to Fig. 5, we pass all the generated augmentations through the DinoV2 ViT-G model, which acts as our oracle model, to obtain their associated embeddings. Subsequently, we employ PCA for visualization purposes. The visualization reveals that the magnitude of semantic transformations is contingent upon both the source image and the specified target category.
Refer to caption: extracted/5983662/figures/traj0_supp.jpg

Figure: Figure A4: Visualization of Generative Samples: More visualization akin to Fig. 4. We compare GeNIe with two baselines: Img2Img^L augmentation uses both image and text prompt from the same category, resulting in less challenging examples. Txt2Img augmentation generates images based solely on a text prompt, potentially deviating from the task’s visual domain. GeNIe augmentation incorporates the target category name in the text prompt along with the source image, producing desired images with an optimal amount of noise, and balancing the impact of the source image and text prompt.
Refer to caption: extracted/5983662/figures/genie_vis_supp_long.jpg

Figure: Figure A5: Effect of noise in GeNIe: Akin to Fig. 2, we use GeNIe to create augmentations with varying noise levels. As is illustrated in the examples above, a reduced amount of noise leads to images closely mirroring the semantics of the source images, causing a misalignment with the intended target label.
Refer to caption: extracted/5983662/figures/noise_ab_sup.jpg

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Is the classifier used for GeNIe-Ada the same as the one trained for the target task? How does the performance of this classifier affect the effectiveness of GeNIe-Ada?
- Is there no information leakage by using stable diffusion in few-shot classification for mini-ImageNet and tiered-ImageNet?
- Does using images from all other classes as the source image in line 352 mean that source images are selected from the support set of the other 4 classes, or all other 63 classes (for mini-ImageNet)?
soundness: 3
strengths: - This paper proposes a realistic solution for practical problems of data augmentation using recent text-to-image generative models.
- The proposed method, GeNIe, contains an effective and adaptive strategy for data augmentation.
summary: This paper introduces a novel data augmentation method based on a latent diffusion model. It combines target and source categories in the latent noise-level to generate hard-negative samples for the source category. The impact of the proposed GeNIe is evaluated on two famous data-scarce problems.
weaknesses: - The exact method for GeNIe is not detailed in the main script, although the key idea is presented, which can cause confusion.
- The proposed GeNIe is a simple data augmentation method on the latent space of Stable Diffusion, which lacks the novelty of the method except for modifying noise levels for better augmentation.
- GeNIe-Ada requires an additional classifier to search the decision boundary between two categories.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see weaknesses
soundness: 3
strengths: The paper is well-written and easy to follow. The motivation is sound, and using diffusion models to generate augmented views, even though it is a well-explored area, the authors focus on generating hard negative images. This approach forces the classifiers to learn the true characteristics of the category rather than rely on spurious correlations, such as background. In many ways, the motivation is similar to foreground and background editing ideas. Their method also addresses the general problem of CNNs, which tend to rely more on textures than on the shapes of objects while classifying. Furthermore, the results are impressive across different benchmarks.
summary: This paper proposes a solution to generate augmentations using diffusion models to improve the performance of classifiers in few-shot and long-tailed settings, which are characterized by low-data scenarios. To that end, the authors propose using a latent diffusion model that combines a source image (with a known category, e.g., dog) with a target category prompt (e.g., cat), producing challenging samples that retain background features from the source category but resemble the target category. This approach aims to push model learning near decision boundaries, providing more informative augmentations than traditional methods.

GeNIe, inspired by image editing through diffusion models, works by adjusting noise levels in the diffusion process to control the balance between source and target features in the generated image. The authors demonstrate that lower noise levels retain more of the source image's visual characteristics, while higher levels focus on the target category. To automate the noise level per sample, the authors further introduce GeNIe-Ada. This variant stems from the observation that there is a sudden change in appearance when the noise levels are uniformly sampled. The authors leverage this observation along with the assumption that they also have access to another target sample. They then propose to compare the similarities of this sample and the generated sample in the latent space of a classifier.

To validate the utility of these augmentations, the authors conduct experiments on few-shot and long-tail classification tasks and demonstrate significant improvements on standard benchmarks.
weaknesses: I have a few concerns and questions regarding GenIE. While generative models are indeed a powerful approach, they can be restrictive in the following sense: their applicability in general, and GenIE in particular, requires that the generative model already knows the concepts of the target category, which is a strong assumption to make. In that sense, generative model-based augmentations should be viewed from two perspectives: (I) those that change the background and (ii) those that change the category. In the first setting, the use of generative models makes a lot of sense because it aligns with the paper's motivation but is concept agnostic. The second category is harder to argue because it requires the generative model to have complete knowledge of all target categories, which would severely limit its applicability. Category 1 also has the added benefit of generalizing to newer domains. For example, in SiSTA: Target-Aware Generative Augmentations for Single-Shot Adaptation (ICML 2023), the authors do not view this as an image editing problem but rather adapt a generative model like StyleGAN to the target domain using a single image and then sample from the model to generate augmented variants. These variants are then used to train classifiers with source-free unsupervised domain adaptation. In this context, GenIE requires both source samples, and its general treatment of this problem as image editing is limiting.

I would like to see comparisons or discussions regarding the strong assumptions that GenIE makes in this context. This is an important discussion because even simpler methods like mixup do not require explicit knowledge of target categories, which GenIE's reliance on diffusion models necessitates.

Similarly, the diffusion models used in the paper are large-scale foundation models trained on LAION. This raises questions about the improvements: it is unclear whether the gains are due to the proposed method or the large-scale pretraining of the diffusion model. It would also be important to discuss relevant work, such as (Tian, Yonglong, et al. "Stablerep: Synthetic images from text-to-image models make strong visual representation learners." Advances in Neural Information Processing Systems 36 (2024).), which uses Stable Diffusion to generate synthetic positive samples for contrastive self-supervised learning training.

Lastly, it would be interesting to see if synthetic image augmentations can be used alongside general mixup, similar to (Graikos, Alexandros, Srikar Yellapragada, and Dimitris Samaras. "Conditional Generation from Pre-Trained Diffusion Models using Denoiser Representations." BMVC. 2023.), which has shown that synthetic images from a conditional diffusion model can provide complementary benefits to mixup.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Could you add more experiments on some other mainstream tasks? Do you have comparisons with other models for data augmentation? Such as GAN?
soundness: 2
strengths: 1. GeNIe leverages a latent diffusion model to create challenging augmentation images by blending features from two classes (source and target). By adjusting the noise level, GeNIe maintains low-level details from the source image while incorporating semantic features of the target class, creating what the authors term "hard negative" samples. This process potentially aids in classifier training by providing near-boundary examples.

2.Adaptive Mechanism (GeNIe-Ada):
The GeNIe-Ada extension adds significant value by adaptively selecting noise levels for each sample, fine-tuning the augmentation to be more effective. This adaptive approach is well-aligned with real-world scenarios where data variability and distributional shifts are common, making it a practical addition.
summary: In this paper, the authors try to illustrate their idea by proposing these points:
1. The authors introduce GeNIe, a novel yet elegantly simple diffusion-based augmentation method to create challenging augmentations in the manifold of natural images. For the first time, to our best knowledge, GeNIe achieves this by combining two sources of information (a source image, and a contradictory target prompt) through a noise-level adjustment mechanism.
2. They further extend GeNIe by automating the noise-level adjustment strategy on a per-sample basis
(called GeNIe-Ada), to enable generating hard negative samples in the context of image classification, leading also to further performance enhancement.
3. To substantiate the impact of GeNIe, authors present a suit of quantitative and qualitative results including extensive experimentation on two challenging tasks: few-shot and long tail distribution settings corroborating that GeNIe (and its extension GeNIe-Ada) significantly improve the downstream classification performance.
weaknesses: 1. They didn't give more contribute to the model itself. It's just an application to generate images.
2. Limited number of downstream cases.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: **Visualizations.** Improve visualization of the PCA over DINOv2's embeddings as suggested in Weaknesses section. Also, increasing the resolution of images across all figures would help fully appreciating the variations in image generation. 

**Data augmentation.** Run experiments with data augmentation policies combining both geometric and photometric transformations, as suggested in Weaknesses section.

**Datasets.**
In the supplementary material Table A1, the augmentation is evaluated on fine-grained classification by training a SVM on DINOv2's pretrained **ViT-g**. This evaluation is quite insightful and would deserve being integrated in the main paper. To make space, consider relocating some few-shot table contents to the supplementary. It would also be very interesting to see the same evaluation in the non-few-shot setting (*i.e.* training with the full training set), which could be directly compared to the linear evaluation results reported by DINOv2. Note that for Aircraft, DINOv2 evaluates on the **variants** categories. 

**Minor remarks on formatting.**
- Consider using *multicol* to improve table readability.
- Please write **ViT-g** instead of **ViT-G** for DINOv2's model.
soundness: 3
strengths: - The proposed approach allows incorporating diversity into the image generation process, which is a crucial aspect in diffusion-based image generation.
- The paper effectively illustrates the importance of selecting an appropriate noise level in text-conditioned image generation.
- The proposed augmentation is evaluated with extensive experiments on image classification in few-shot and long-tail settings.
summary: The paper leverages Stable Diffusion to generate hard negatives as data augmentation for image classification. The hard negatives are generated by feeding a source image into the model's encoder but conditioning the reverse diffusion process with a text prompt corresponding to the target class. 

The proposed approach includes an automatic noise-level adjustment per sample, which controls the degree of deviation from the source image, thereby aligning more with the target class prompt. The authors show that the choice of the noise level is critical, as only a restricted range of values allow for a good balance between source and target classes. The noise is chosen by generating multiple images with different noise levels and choosing the level where the semantic shift between the source and target images is the highest.

The proposed augmentation is evaluated on image classification tasks, in few-shot and long-tail distribution settings.
weaknesses: **Method section**
My main concern regards the automatic noise level selection process. It involves generating multiple images with various noise levels and selecting the one where the semantic shift is the highest. This seems highly inefficient given the time required to generate these images.

**Data augmentation.** 
I am confused about the *weak* and *strong* augmentation policies GeNIe is compared to. Are strong augmentations (random color jitter, random grayscale, Gaussian blur) applied in addition to weak augmentations (random resize crop, random flipping)? Otherwise, why are the weak augmentations solely geometric, and the strong ones solely photometric? 
A balanced policy combining both geometric and photometric transformations could be a better contender. For example, it could consist of random resize crop (with e.g., a scale of 0.4), random flipping, color jittering, and maybe MixUp or CutMix on top.

**Visualizations.** 
The PCA representations of DINOv2 in Figure 5 could be improved and expanded. DINOv2's representations are excellent, and their PCA is usually very insightful.

Meta-review: This paper proposes to leverage generative models to obtain data augmentations that improve downstream performance of few shot and long tail datasets. Augmentations are produced by combining a source image with a target category prompt.

The paper was reviewed by four knowledgeable reviewers who acknowledged that the paper is well written and easy to follow (dDiR), the motivation sound (dDiR), and the solution relatively effective (X612). The reviewers raised concerns about:

1. Missing information on the method (X612)
2. Limited novelty (eEnJ), strong assumptions made (X612, dDiR)
3. Potential information leakage (from pre-trained generative models and about ImageNet) (X612)
4. Missing some related work discussion/comparisons (eEnJ, dDiR)
5. Possible inefficiency of the approach (ny5S)
6. Unclear significance of results (e.g. unclear whether improvements come from the method or from the pre-trained diffusion model) (dDiR).

During rebuttal/discussion, the authors discussed the questions related to SDv1.5 known classes, and potential data overlap between pre-training datasets and few-shot test set. They provided missing details on the method and experimental setup, argued for the novelty of their approach, shared additional experiments analyzing the effect of the feature extractor, discussed the surfaced leakage issue, added some comparisons suggested by the reviewers, as well as combining the proposed approach with standard data augmentation techniques, and promised results on additional datasets. After rebuttal, one reviewer signals clear accept, one leans towards rejection, and the remaining two appear to position themselves as borderline. However, as pointed by the authors, some reviewers remain unresponsive during discussion time. Given this, the AC took the time to go over the paper, the reviews and discussions. Although the AC agrees with the reviewers that the proposed method may hold potential, the AC remains unconvinced by the experimental evidence provided. In particular, although prior art is discussed in the related work, important comparisons are missing in the experimental section (e.g. StableRep, feedback guidance, and Fill Up). Compared to Feedback Guidance and Fill Up, the improvements achieved by Genie on the Few split of the ImageNet-LT dataset, appear less impressive. It is also unclear what the effect of the number of synthetic images is on the downstream performance, or whether using a more recent text-to-image model would result in different findings. For these reasons, and the remaining concerns of the reviewers, the AC cannot recommend acceptance, despite the merits outlined by some reviewers. The AC encourages the authors to consider the feedback received to improve future iterations of their work.

Decision: Reject