Title: GeNIe: Generative Hard Negative Images Through Diffusion
Venue: NeurIPS 2024
Accepted: no
Keywords: Diffusion Models

Data Augmentation

Few-shot

Long-Tail
Abstract: Data augmentation is crucial in training deep models, preventing them from overfitting to limited data. Recent advances in generative AI, e.g., diffusion models, have enabled more sophisticated augmentation techniques that produce data resembling natural images. We introduce GeNIe a novel augmentation method which leverages a latent diffusion model conditioned on a text prompt to combine two contrasting data points (an image from the source category and a text prompt from the target category) to generate challenging augmentations. To achieve this, we adjust the noise level (equivalently, number of diffusion iterations) to ensure the generated image retains low-level and background features from the source image while representing the target category, resulting in a hard negative sample for the source category. We further automate and enhance GeNIe by adaptively adjusting the noise level selection on a per image basis (coined as GeNIe-Ada), leading to further performance improvements. Our extensive experiments, in both few-shot and long-tail distribution settings, demonstrate the effectiveness of our novel augmentation method and its superior performance over the prior art.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to employ text-to-image latent diffusion models to augment images through a controlled modification such that the resultant class is different from the source class. Such augmented images are referred to as hard negative images. Building upon SDEdit style image modification, the paper controls the extent of modification by adaptively determining the appropriate noise-scale for each image separately. The benefits of this type of augmentation have been demonstrated on few-shot and long-tailed imagenet classification tasks.\n\nSTRENGTHS:\n- The paper is very well written presenting the core idea of generating hard-negative images by modifying an image with a caption of another class. This idea is simple, intuitive and interesting. \n- Furthermore, the algorithm to determine the optimal noise-level for each image adaptively is not only simple and intuitive but also effective in eliminating dependence on hyperparameters. I feel that a connection can be made to the recent work [1] on phase-transition in diffusion models since this algorithm is attempting to find the diffusion-time when phase-transition occurs. \n- The evaluation is comprehensive considering a variety of diffusion-augmentation baselines as well as traditional augmentations.\n- The paper illustrates the effectiveness of the adaptive search procedure through separate experiments with DINO-v2 and visualisations.\n- In many cases, synthetic data generation with a diffusion model may be replaced by a simpler retrieval baseline [2]. However, the goal of this work is to use a diffusion model to search for and generate hard negatives, which is an interesting deviation from some of the previous synthetic data augmentation approaches. \n\n[1] Sclocchi, Antonio, Alessandro Favero, and Matthieu Wyart. \"A phase transition in diffusion models reveals the hierarchical nature of data.\" arXiv preprint arXiv:2402.16991 (2024).\n\nWEAKNESSES:\n- From the various results in the paper, it seems that the Text2Image, GeNIe, and GeNIe-Ada achieve comparable performance with respect to each other on average. This seems to suggest that the majority of the gains can be attributed to the increased number of _distinct_ examples --- as compared to regular augmentations which simply apply different transformations to the same image --- for each class rather than the hard-negatives in GeNIe/GeNIe-Ada. \n- Additionally, it seems that beyond some threshold, any value of $r$ that changes the source-image to the target image yields comparable performance indicating that it may be sufficient to generate an augmentation that is similar to source-image and it need not specifically be a _hard-negative_.  It may be useful to consider some other applications where images lying in the boundary of the classifier may be informative: for example, see recent work on generating outliers [1] for OOD detection. \n- GeNIe-Ada algorithm is compute-intensive as compared to a simple Text2Image augmentation since it requires generating several augmentations for each source image before selecting one optimal augmentation that lies on the decision boundary. Given how close the text2image and genie-ada performances are in some cases, it may be possible that we could generate more augmentations using text2image in the same compute budget and improve over GeNIe. \n- (minor) GeNIe is applicable to the fine-tuning stage rather than the pretraining stage.\n\n[1] Du, X., Sun, Y., Zhu, J. and Li, Y. Dream the impossible: Outlier imagination with diffusion models. NeurIPS 2024.\n\nQUESTIONS:\nSee weaknesses. \n- How can we apply GeNIe for classification involving a larger number of classes? Could you please elaborate on this statement in the paper: _For GeNIe, instead of randomly sampling the source images from other classes, we use a confusion matrix on the training data to find the top-4 most confused classes and only consider those classes for random sampling of the source image. The source category may be from “Many”, “Med”, or “Few sets”._ If the training converges, how can the confusion matrix on train data be informative? \n- If the class of the source image and target class are not semantically compatible, how does the modification look like? \n- Do you have any examples of failure cases of algorithm 1?\n\nLIMITATIONS:\nYes, limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GeNIe, a data augmentation method for training vision models using synthetic images. GeNIe generates images by combining a source category image with a target category text prompt, selecting those that feature source characteristics but belong to the target category as negative samples. Experimental results show that GeNIe improves performance in both few-shot and long-tail distribution settings.\n\nSTRENGTHS:\n* The proposed GeNIe improves the performance in few-shot and long-tail distribution settings.\n* The paper provides extensive experiments to support the claims,  including the selection of noise levels.\n* The paper is well-written and easy to follow.\n\nWEAKNESSES:\nThe key idea of GeNIe is to use image editing to combine features from two categories. Here are several questions:\n\n* Regarding controllable image augmentation\n  * Line 9 mentions that GeNIe \"retains low-level and background features from the source image.\" How does GeNIe control which features are retained or changed?\n  * To combine features from different categories, how about adding the attribute from the target category to the prompt? For example, a \"[dog] with [wings]\".  This method does not require carefully selection of denoise steps. \n  * Other image editing methods, such as those in [1] and [2], efficiently control image changes using prompts or user instructions.  For example, they can transform a car into a motorcycle in Figure 2, while keeping the background unchanged for more challenging negative samples. What advantages does GeNIe offer over these methods?\n\n\n\n* GeNIe generates images \"using images from all other classes as the source image\" (line 227). Will all (source image, target prompt) pairs lead to effective image generation? Which types of pairs contribute the most to the final accuracy?\n\n     [1] Prompt-to-Prompt Image Editing with Cross-Attention Control\n\n     [2] InstructPix2Pix: Learning to Follow Image Editing Instructions, CVPR 2023\n\nQUESTIONS:\nPlease refer to the above weaknesses.\n\nLIMITATIONS:\nThe paper has discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the idea is to generate data for data augmentation by utilizing a pre-trained diffusion model. The method employs different text prompts and an adjusted noise scheduler to generate hard negative samples for the source distribution. \"GeNIe\" creates new augmentations using diffusion by leveraging source images and contradictory target prompts. \"GeNIe-Ada\" adjusts noise levels on a per-sample basis, using the classifier as the condition boundary to select the right threshold.\n\nSTRENGTHS:\n- The method offers infinite possibilities to separate the source from the target.\n- The idea is simple, original, and convincing.\n- The ablation studies and experiments demonstrate strong performance.\n\nWEAKNESSES:\n- The method is slow, particularly GeNIe-Ada, as it requires generating an image through multiple forward passes of a diffusion model and using a classifier to select the appropriate threshold $r$.\n\n- The number of steps required to retain low-level features is crucial for optimizing the method's performance.\n\n- The method relies on access to a foundational text-to-image model trained on billions of images.\n\nQUESTIONS:\n- Stable diffusion utilizes data scraped from the web (LAION), and there is a high probability that the image from validation set of ImageNet is included in the LAION training set. Moreover, stable diffusion tends to replicate the training set [1]. How do you ensure that the augmented images used are not also present in our testing set?\n\n- How does the method perform when using a diffusion model trained exclusively on ImageNet or other diffusion models besides Stable Diffusion?\n\n- How do you ensure that the generated images still resemble natural images? Some prompts could diverge significantly from the source distribution, resulting in images that may be far from the original distribution.\n\n[1] Nicholas Carlini et al.: \"Extracting Training Data from Diffusion Models\"\n\nLIMITATIONS:\n/", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper introduces a novel augmentation method based on diffusion models. A latent diffusion model conditioned on a text prompt generates hard negatives, by adjusting the noise level. The hard negatives can be used as challenging augmentations. The authors demonstrate the effectiveness of their approach on long-tail and few-shot settings.

STRENGTHS:
- Well-written paper with clear contributions and presentation.
- Extensive experiments and evaluation.
- Interesting and useful idea.
- Code included in the supplementary.

WEAKNESSES:
I am generally happy with the paper, experiments, and presentation. A weakness seems to be the selection of the noise ratio r.  The authors propose an algorithm for this. However, I am concerned how sensitive it is for different datasets or classification settings. This might affect performance in other settings or in real-world scenarios. If this is true, it might degrade the overall method's usefulness.

QUESTIONS:
Can the authors comment on the above weakness, regarding the selection of r and its sensitivity to the dataset or setting? 

Also, have the authors considered a different latent diffusion model? Would a smaller diffusion model and/or trained on a smaller amount of data give similar results/benefit?

LIMITATIONS:
The authors have added a section for limitations and a section for broader impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13