Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentation

Authors: eyal  michaeli
ohad  fried
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:40:18

Abstract: Fine-grained visual classification (FGVC) involves classifying closely related subcategories. This task is inherently difficult due to the subtle differences between classes and the high intra-class variance. Moreover, FGVC datasets are typically small and challenging to gather, thus highlighting a significant need for effective data augmentation.
Recent advancements in text-to-image diffusion models have introduced new possibilities for data augmentation in image classification. While these models have been used to generate training data for classification tasks, their effectiveness in full-dataset training of FGVC models remains under-explored. Recent techniques that rely on text-to-image generation or Img2Img methods, such as SDEdit, often struggle to generate images that accurately represent the class while modifying them to a degree that significantly increases the dataset's diversity. To address these challenges, we present SaSPA: Structure and Subject Preserving Augmentation. Contrary to recent methods, our method does not use real images as guidance, thereby increasing generation flexibility and promoting greater diversity. To ensure accurate class representation, we employ conditioning mechanisms, specifically by conditioning on image edges and subject representation.
We conduct extensive experiments and benchmark SaSPA against both traditional and generative data augmentation techniques. SaSPA consistently outperforms all established baselines across multiple settings, including full dataset training and contextual bias. Additionally, our results reveal interesting patterns in using synthetic data for FGVC models; for instance, we find a relationship between the amount of real data used and the optimal proportion of synthetic data.

Main paper: 
 introduction: 
 Deep learning's remarkable success across various applications relies heavily on large-scale annotated datasets, such as ImageNet [12] , which provide the foundational data necessary for training effective models. However, in fine-grained visual classification (FGVC), the datasets are typically smaller and less diverse, presenting unique challenges in training robust models. Data augmentation emerges as a natural solution to artificially enhance dataset size and variability. However, traditional data augmentation methods are limited in the amount of diversity they introduce [15] .Text-to-image diffusion models have opened new avenues for generative image augmentation. Within the realm of classification, diffusion models have shown promise on standard image recognition datasets such as ImageNet [2, 50, 3] . However, their application in FGVC remains under-explored.Generating synthetic data for FGVC presents unique challenges, as preserving class fidelity is (1) more crucial than with common object datasets due to the similarity between classes and the reliance of the models on subtle details to differentiate between classes and (2) challenging to achieve because the training data for text-to-image models often lacks a substantial representation of these distinct objects [26] . For instance, there might be enough data to accurately represent "An airplane", but not "A Boeing 767-200 airplane".Recent generative methods evaluated for FGVC augmentation typically use real images as guidance in an Img2Img manner [15, 54, 21, 60] . While this helps maintain visual similarity to the target domain, it limits the degree of diversity that can be introduced, resulting in a trade-off between class fidelity and diversity [16] (see Figure 1 ). We aim to free the generative process from this constraint of adhering to specific source images. To this end, we propose SaSPA: Structure and Subject Preserving Augmentation, a method that conditions the generation on more abstract representations rather than direct image inputs. Specifically, we leverage structural conditioning in the diffusion model via edge maps extracted from source images. This allows the generated samples to respect the broad shape and composition of objects in the target domain. Crucially, the lack of specific image conditioning enables greater flexibility in rendering surface details. To further ensure the preservation of fine-grained class characteristics, we integrate subject representation conditioning. By combining edge-based structural conditioning with category-level conditioning, SaSPA can generate highly diverse, class-consistent synthetic images without being overly influenced by any specific real data sample. Furthermore, to enrich the diversity and applicability of our generated images, we generate prompts with an LLM according to the dataset meta-class (a class encompassing all sub-classes). These prompts are designed to guide the diffusion model in producing variations that are not only diverse but also class-consistent and relevant to the target domain. Additionally, to maintain the quality and relevance of the generated images, we implement a robust filtering strategy that eliminates any samples that fail to meet predefined quality thresholds by utilizing a dataset-trained model and CLIP.We summarize our contributions as follows: (1) We propose SaSPA, a generative augmentation pipeline for fine-grained visual classification that generates diverse, class-consistent synthetic images without relying on specific real images for conditioning. (2) We conduct extensive experiments and benchmark SaSPA against both traditional and recent generative data augmentation methods. SaSPA consistently outperforms all established baselines across multiple settings, including the challenging and less-explored full dataset training, as well as in scenarios of contextual bias and few-shot classification. (3) Our analysis provides insights on effectively leveraging synthetic data to improve the performance of fine-grained classification models. For instance, we find that as the amount of real data decreases, we should increase the proportion of synthetic data used. 
related work: 
 Data Augmentation with Generative Models. Synthesizing training samples using generative models is an active and challenging area of research. Initial efforts in this field [70, 4, 49, 33] leveraged Generative Adversarial Networks (GANs) to create labeled training samples. Recently, the emergence of powerful text-to-image diffusion models such as Stable Diffusion [46] has created exciting opportunities for advancing generative image augmentation. These models have been employed across a range of applications, including semantic segmentation [19, 61, 62, 36] , object detection [8, 7, 59] , and classification [34, 2, 50, 3] , demonstrating their versatility and effectiveness For image classification tasks, diffusion models have demonstrated promising results on standard Real Image Edge Detector Meta-class "Airplane"(1)"A {sub-class} airplane flying over a desert" ...Edges Map Subject Reference Images (1) (2)Filter Augmentations(2) image recognition datasets such as ImageNet [2, 50, 3] . However, their application in FGVC has typically been limited to particular settings such as few-shot learning [21, 54, 52, 26] where data scarcity significantly enhances the impact of data augmentation, contextual bias, and domain generalization [15] , settings that are more straightforward to enhance as targeted augmentations can directly address and balance the skewed distributions. Our goal is to tackle the more challenging task of training on full FGVC datasets. Moreover, recent generative augmentation methods often use Img2Img techniques like SDEdit to maintain class fidelity, though this comes at the cost of reduced diversity. Some methods involve fine-tuning the network or its components, which can be expensive and may still struggle to balance class fidelity with the added diversity necessary for effective FGVC augmentation. Our goal is to avoid the decrease in diversity associated with using real images as guidance and to avoid the complexity and expense of fine-tuning the generation model.Text-to-Image Diffusion Models. Diffusion models [23] have achieved unprecedented success in generating photo-realistic images [13] . Models like Stable Diffusion [46] , DALL-E 2 [43] , and others [37, 48] exemplify this capability. These models have also driven advancements in other generative areas. For instance, SDEdit [32] integrates real images partway through the reverse diffusion process for image editing. Techniques like ControlNet [68] and T2I-Adapter [35] condition image generation on inputs beyond text such as edges and world normals, while methods such as Textual Inversion [18] and DreamBooth [47] can generate specific subjects from just a few example images. More recently, BLIP-diffusion [27] , which is based on Stable Diffusion and BLIP-2 [28] , has demonstrated impressive zero-shot subject-driven generation using only one example image. Our method benefits directly from these advancements, employing ControlNet and BLIP-diffusion.Traditional Data Augmentation. Traditional data augmentation methods typically include operations such as random cropping, flipping, and color-space changes to generate new variations [10] . Recent strategies, like mixup-based methods, aim to enhance diversity by blending patches from two input images [66] or using convex combinations [67] . Weakly Supervised Data Augmentation Network (WS-DAN), used in recent FGVC works such as CAL [44] , aims to improve FGVC by generating attention maps to highlight discriminative object parts and guiding augmentation with attention cropping and dropping. However, these methods introduce limited diversity [15] , as they do not alter the semantic features present in the image. 
method: 
 Our goal is to augment a labeled training dataset for FGVC to increase its diversity while faithfully representing the sub-classes. The key insight of our method is to minimize reliance on any particular source image during generation and instead condition the generation on more abstract representations, thereby increasing diversity while accurately representing the designated class (see Figure 1 ). To achieve this, we employ abstract conditions such as edges, which capture the object's structure, and subject representation, which aims to preserve fine-grained class characteristics. The process, illustrated in Figure 2 , unfolds in five steps, outlined below:We aim to use a generative text-to-image model, which requires input prompts to guide the image synthesis process. To ensure that the prompts generated align broadly with the primary category of each dataset, our method begins by identifying the meta-class for each dataset, such as "Airplane" for the Aircraft [30] dataset and "Car" for the Stanford Cars [24] dataset.Prompt Generation via GPT-4 [1] . We input the meta-class to GPT-4 with the instruction to produce 100 unique, relevant, and diverse prompts, each inherently containing the term of the meta-class. This strategy ensures that the generated images stay true to the fundamental aspects of the meta-class while hopefully containing relevant and diverse scenarios. To increase the specificity and relevance of these prompts, we integrate the relevant sub-class into each prompt whenever it is used. Unlike a recent work [15] , which also uses GPT-4 to create prompts, we do not require image captions of the dataset. The exact instructions for GPT-4 and more example prompts are in Appendix E.1.To ensure that the generated images maintain the overall structure and shape of objects belonging to their respective classes, we condition the synthesis process on edge representations extracted from real images in the dataset. Concretely, for a dataset of N images, we extract one-channel edge representations using the Canny edge detector [6] . This yields a set of N edge-based visual priors that capture the structural characteristics of each sub-class. By conditioning the generative model on these edge maps, we can preserve object shape and layout during synthesis while allowing flexibility in rendering other surface-level details. In Table 17 , we additionally explore the use of HED [63] as an alternative technique for edge extraction and structural conditioning.Diversity in synthetic data is crucial for effective training [55, 45, 31] . Unlike most recent approaches [21, 15, 54] , our method focuses on edges and subject representation as a prior rather than the source image. We show in Figure 1 that this approach maintains class fidelity, and as a result of not using the source image, affords the model greater flexibility to introduce novel semantic features such as weather, lighting, or even new elements both within and outside the object's confines. This strategy might be particularly beneficial in FGVC tasks, where the subtle differences between classes are crucial, and hence, maintaining class fidelity while introducing diversity is of paramount importance.Conditioning on edge maps of a real image ensures that generated images align closely with real structural features. Interestingly, we notice that structural conditioning cues the generation model to accurately represent the target sub-class, enhancing not only the correct representation of structural features but also the correct representation of non-structural attributes like color and texture. This structure-guided synthesis approach effectively enhances the model's ability to maintain class fidelity across varied image generations.Conditioning on subject representation further enhances the generation model's ability to produce images with accurate sub-class representation. This ensures the correct representation of the sub-class on levels beyond structure, such as texture, color, and other visual features.Using these two mechanisms together ensures correct sub-class representation across datasets, whether the primary distinctions between sub-classes lie in structural features, texture, color, or any combination of them.Due to its impressive results and widespread use, we employ ControlNet [68] conditioned on Canny edges [6] for edge map conditioning. For subject representation conditioning and as a base model for ControlNet, we utilize BLIP-diffusion [27] , a model built upon Stable Diffusion [46] and BLIP-2 [28] that emphasizes subject representation and supports zero-shot subject-driven generation using one reference image. We chose BLIP-diffusion for its zero-shot capabilities and open-source availability.Using BLIP-diffusion with ControlNet requires a prompt, an edge map, and a reference image of the target subject. To maintain sub-class accuracy while introducing diversity at the sub-class level, the reference image is selected from the same sub-class but differs from the real image used to extract the edge map (we experiment with BLIP-diffusion inputs in Table 4 ). Specifically, we generate M = 2 augmentations for each real image in the training set: we extract an edge map for each real image, and for each edge map, we randomly select M prompts and M subject reference images from the same sub-class. These inputs are then fed into the generation model of ControlNet with BLIP-diffusion as the base model to produce M augmentations of the real image. Example augmentations are visualized at Figure 3 . DTD [9] has only one prompt per real image, because we utilize image captions as prompts for it, as explained in Appendix D.We aim to remove low-quality augmentations, which appear in two forms: (1) meta-class misrepresentation and (2) sub-class misrepresentation.Semantic Filtering. To alleviate meta-class misrepresentation, we utilize semantic filtering as described in ALIA [15] . Using CLIP [41] , this process evaluates the relevance of generated images to the specific task at hand. For example, in a car dataset, each generated image is assessed against a variety of prompts such as "a photo of a car", "a photo of an object", "a photo of a scene", "a photo", and "a black photo". Images that CLIP does not recognize as "a photo of a car" are excluded to ensure that the augmented dataset closely aligns with the target domain.Predictive Confidence Filtering. To ensure each augmentation faithfully represents its designated sub-class, we implement a predictive confidence filtering strategy inspired by recent work [21] strategy CLIP Filtering. This method employs CLIP [41] to filter out images that do not strongly correlate with the textual labels of their class among all classes in the dataset. However, the limitation of using CLIP in this context is its insufficient granularity in understanding fine-grained concepts.For our method, we discard any augmented images for which the true label does not rank within the top-k predictions of a baseline model trained on the original dataset. This approach helps to exclude images that likely misrepresent the source sub-class, thus maintaining a high-quality dataset for model training. In our implementation, we use k = 10. Further details about this method, the baseline model used, and other filtering techniques are discussed in Appendix E.2.We train the downstream classification model using the filtered, generated samples. Let α denote the augmentation ratio, representing the probability that a real training sample will be replaced with a generated synthetic sample during each epoch. This replacement process is repeated for every sample in each epoch, allowing each real sample to be either retained or replaced by an augmented version. We employ this replacement strategy instead of simply adding the augmented data to the original dataset, as doing so would unnecessarily increase the number of iterations per epoch. By that, we ensure fair comparisons across training sessions. 4 ExperimentsOur objective is to explore the extent to which synthetic data, particularly through our approach, contributes to various FGVC tasks. We aim to understand the significance of each component of our method and identify optimal strategies for leveraging synthetic data in FGVC.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental setup: 
 For generation, we employ BLIP-diffusion for SaSPA and Stable Diffusion v1.5 [46] for all other diffusion-based augmentation methods.For training, we follow the implementation strategy outlined in the CAL study [44] , tailored for FGVC. We use ResNet50 [20] as the primary architecture within the CAL framework unless specified otherwise. Each dataset is fine-tuned using pre-trained ImageNet weights. More data generation and training details can be found in Appendix D.Comparison Methods. We benchmark our method, SaSPA, against established traditional and generative data augmentation techniques. In the traditional category, our comparisons include: CAL-Aug [44] : utilizes random flipping, cropping, and color-space variations. RandAugment [11] : applies a series of random image transformations such as rotation, shearing, and color variations to training images. CutMix [66] : generates mixed samples by randomly cutting and pasting patches between training images to encourage the model to learn more localized and discriminative features.Combined Methods: Tests the synergistic effects of CAL-Aug with CutMix and RandAug with CutMix. In the generative category, we compare with: Real-Guidance [21] : applies Img2Img with a low translation strength (s = 0.15) to maintain high fidelity to the original images. ALIA [15] : Uses real image captions and GPT-generated domain descriptions based on these captions as prompts for Img2Img translations. Detailed descriptions of these baseline methods are in Appendix D.3.Datasets. We evaluate on five FGVC datasets, using the full datasets for training. We use Aircraft [30] ,Stanford Cars [24] , CUB [58] , DTD [9] , and CompCars [64] . For datasets lacking a predefined validation split, we establish one. For CompCars, we utilize the exterior car parts split, focusing exclusively on classifying images of car components: head light, tail light, fog light, and front into the correct car type. Further details on the exact splits are provided in Appendix C.Results. We present the test accuracy of various augmentation methods in Table 1 . For each dataset, the most effective traditional augmentation method (marked by an underline) is identified using its validation set and consistently combined with all generative approaches to optimize performance for that dataset. This approach is grounded in findings that standalone generative methods generally perform better when integrated with traditional augmentations [60] , a trend also evident in Table 7 . We can see that for all datasets and shots, SaSPA outperforms all other augmentation methods.Our key findings: (1) SaSPA achieves consistent improvements across all datasets, with or without BLIP-diffusion integration, and it consistently outperforms traditional and generative augmentation methods by a significant margin. ( 2) The benefits of BLIP-diffusion vary depending on dataset characteristics; while it improves performance in datasets where texture and style play a crucial role in differentiation, such as DTD, CUB, and CompCars, it is not optimal for the Aircraft dataset, and has no significant impact on the Cars dataset, where structural features are more important for classification. We attribute this to the fact that using BLIP-diffusion confines the augmentations to be similar to other subjects within the same sub-class, which can limit diversity. (3) Both generative baselines fail to achieve consistent improvements and sometimes even reduce performance.Experimental Setting. This section investigates the efficacy of various augmentation strategies in few-shot fine-grained classification scenarios, focusing on how synthetic data affects performance with increasing numbers of training examples ("shots"). We conduct evaluations using three datasets: Aircraft [30] , Cars [24] , and DTD [9] , assessing performance at 4, 8, 12, and 16 shots.The training and data generation approaches remain consistent with those described in Appendix D, with two modifications: we use 100 epochs (down from 140), and we do not employ predictive confidence filtering for shots 4 and 8. The latter adjustment is due to the reduced reliability of the model's predictions, resulting from the limited training data. Additionally, we increase the augmentation ratio to α = 0.6, as identified to be better for scenarios with limited data in Table 5 .Results. The results in Figure 4 show that SaSPA consistently outperforms all other augmentation methods across all datasets and various shot counts. As seen in other works [54, 21] , the benefit of augmentation diminishes as the number of shots increases, a trend most noticeable in the Cars dataset. Contrary to prior work, the gains provided by SaSPA remain substantial even at higher shot counts; notably, in the Cars dataset at 16 shots, SaSPA achieves an accuracy of 91.0%, surpassing the second-best performance of 88.3% by RG [21] . Interestingly, SaSPA sometimes matches or exceeds the performance enhancement achieved by increasing the dataset size. For example, in the DTD dataset, utilizing SaSPA with 8 shots results in an accuracy of 54.8%, slightly surpassing the 54.6% obtained when adding 50% more real data (a total of 12 shots) when relying solely on real data and the best traditional augmentation.Experimental Setup. To evaluate the effectiveness of our method in mitigating real-world contextual biases, we use the contextual bias split of the Aircraft [30] dataset constructed by Dunlap et al. [15] . The split uses two visually similar classes: Boeing-767 and Airbus-322. Each image in this split is categorized as "sky", "grass", or "road" depending on its background, with ambiguous examples filtered out. The bias in the dataset is introduced by training on 400 samples where Airbus aircraft are exclusively associated with road backgrounds and Boeing aircraft with grass backgrounds, although both types may appear against sky backgrounds. The exact split breakdowns are in Table 19 .Table 2 : Classification performance on the contextually biased Aircraft dataset [30] , detailing overall, in-domain (ID) and out-of-domain (OOD) accuracies for each augmentation method. We follow the same training and generation implementation settings as for the FGVC setting (Appendix D), and we compare against the same generative methods. We also compare against the optimal traditional augmentation for the Aircraft dataset (CAL-Aug), as defined in Section 4.2.Results. The results in Table 2 show that SaSPA outperforms all other methods in overall and out-ofdomain (OOD) accuracy, demonstrating its effectiveness in mitigating contextual bias. However, it falls short in in-domain (ID) accuracy. A distinct inverse relationship is observed between ID and OOD accuracy: methods that induce more significant changes from the original image-such as ALIA, which uses stronger translations than Real-Guidance (RG)-tend to achieve higher OOD accuracy but lower ID accuracy. This trend suggests that greater modifications can help reduce over-fitting to in-domain characteristics, enhancing the model's ability to generalize effectively to new, unseen conditions. As depicted in Figure 1 , even a higher translation strength (s = 0.5/0.75 ) yields limited diversity compared to our method. Consequently, the alterations produced by RG and ALIA are insufficient to significantly mitigate the contextual bias present in the dataset, as effective background variation is crucial for addressing such biases.In this section, we compare our method with diff-mix, a generative augmentation approach proposed concurrently by Wang et al. [60] . diff-mix was also evaluated on full FGVC datasets and demonstrated impressive results. This method enriches datasets through image translations between classes, utilizing personalization techniques such as textual inversion [18] and DreamBooth [47] to fine-tune the generative model for each sub-class. This fine-tuning enhances the model's ability to capture and represent class-specific nuances. In contrast, our method does not involve fine-tuning, aiming to simplify the process and minimize computational costs.Experimental Setup. In this analysis, we evaluate the performance of our SaSPA augmentation method using the diff-mix training setup, as detailed in their work. By using their open-source implementation, we further assess the robustness of our method with a different training setup.To ensure fairness, We use the same number of augmentations (M) as diff-mix did. More details regarding training setup are in Appendix B.3.Our results, detailed in Table 3 , highlight where SaSPA performs well and identify areas for potential improvement. The findings can be summarized as follows: (1) While diff-mix employs computationally intensive fine-tuning techniques to enhance class representation, we prioritize simplicity and lower computational demands in our approach. Despite this, SaSPA consistently outperforms diff-mix on both the Aircraft and Cars datasets across all architectures, whether combined with CutMix or used alone, demonstrating its robustness across various augmentation contexts. This 1 , it does not perform as well here. We hypothesize that the use of higher resolution emphasizes finer details in each class, which may be overwhelming for SaSPA on some datasets but less so for diff-mix, likely due to the heavy fine-tuning process integrated into their method. These results indicate that some form of fine-tuning might be advantageous for complex datasets like CUB to achieve better performance. The full table, including comparisons to more augmentation methods, can be found in Appendix B.3.In Table 4 , we conduct an extensive ablation study to evaluate the effectiveness of our proposed generation strategies. Specifically, we examine the integration of edge guidance, Img2Img as an alternative for edge guidance with strength = 0.5 and in combination with edge guidance with strength = 0.85, and subject representation. We also investigate the effect of using the same image for both edges extraction and the subject reference image ("Edges=Ref."). Additionally, we test the impact of appending half of the prompts with artistic styles (column 'Art.') as described in Appendix B.8.The results demonstrate the importance of combining structural and subject-level conditioning while enabling diverse generations through separate input sources, yielding the best performance across most datasets. Key observations include: (1) Edge Guidance alone improves performance significantly compared to Text-to-Image or SDEdit [32] (Img2Img), highlighting its important role in providing structural guidance. (2) Subject representation alone does not enhance performance, indicating additional structural conditioning is necessary. (3) Using different source images for edges and subject reference images adds beneficial diversity. (4) Surprisingly, text-to-image generation (first row) outperforms SDEdit, likely due to its increased diversity and despite the lower fidelity, which our filtering mechanism can handle as it filters out low-fidelity images. (5) Incorporating artistic prompts has inconsistent effects, usually boosting performance with Edge Guidance but often degrading it when combined with subject representation. This inconsistency may stem from the fact that subject representation uses BLIP-diffusion [27] , which is a different base model than Stable Diffusion [46] , as Stable Diffusion is fine-tuned. Additionally, in CUB, artistic prompts offer no improvement even when using Edge Guidance without subject representation, likely due to the dataset's heavy reliance on color as a primary discriminator between bird types, potentially disrupted by artistic prompts. 
limitations and future directions: 
 Limitations. Although we demonstrated that SaSPA could generate images with high class fidelity through conditions such as edge maps and subject representation, it still remains dependent on the underlying generation models. For instance, we found that applying SaSPA to the CUB dataset at a higher resolution does not improve performance. Additionally, SaSPA relies on large language models (LLMs) to generate relevant and diverse prompts given the meta-class. While this is usually effective, it may not produce optimal prompts if the LLM lacks knowledge of the meta-class.Future Directions. Several avenues exist to enhance the flexibility and performance of our method in future research. Firstly, we hope our work inspires the use of additional methods to condition the synthesis process beyond using real images, as we have shown to be effective. Another promising avenue is to apply SaSPA to additional tasks such as classification of common objects, object detection, and semantic segmentation. Additionally, maintaining temporal consistency in settings that use consecutive frames, such as autonomous driving, remains a significant challenge. Addressing this issue could expand the applicability of SaSPA to a broader range of use cases. Moreover, ongoing advancements in generative models are likely to bring further improvements to our pipeline. Finally, effectively generating and using synthetic data remains an active research area, and identifying optimal strategies for both the generation process and the training integration remains an important future direction. 
conclusion: 
 We propose SaSPA, a generative augmentation method specifically designed for FGVC. Our method generates diverse, class-consistent synthetic images through conditioning on edge maps and subject representation. SaSPA consistently outperforms both traditional and recent generative data augmentation methods. It demonstrates superior performance across multiple settings, including multiple setups of the challenging and less-explored full dataset training, as well as in scenarios of contextual bias and few-shot classification. Limitations and future directions are discussed in Section 5.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release any new data or models that would require such safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: Yes, all used libraries, such as PyTorch [39] and diffusers [57] , are properly credited, and their licenses and terms of use are respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We attached the code to the supplemental, and we will release the code publicly.The code is documented. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Generative data augmentation can benefit many fields by creating more robust models while protecting privacy. By reducing the reliance on real data, it addresses privacy concerns and lowers the costs and time needed for data collection and annotation, thereby enhancing the accessibility of advanced machine learning techniques. It is also important to note that synthetic data can inherit biases from the generative models, potentially leading to biased training outcomes.In Figure 5 , we evaluate various augmentation ratios (the probability of a real image being replaced by a synthetic one in a given mini-batch). We find that for most datasets, excluding CUB, the optimal range for α lies between 0.2 and 0.5, with marginal differences within this range. Consequently, we selected α = 0.4 as the default augmentation ratio. However, for the CUB dataset, the default choice is α = 0.1. Considering the relatively lower improvement on CUB in Table 1 and the underperformance on the diff-mix benchmark (Table 3 ), both of which are likely attributed to lower class-fidelity, it seems that lower class fidelity necessitates a lower augmentation ratio. A possible explanation is that higher augmentation ratios are more likely to introduce bias during training when class fidelity is lower.In Table 5 , we examine the interaction between α and the percentage of real data used. We use two α values for each dataset: the default value used throughout the paper and a higher value (α high = α+0.2). We observe that for all amounts of real data, SaSPA achieves notable improvements with diminishing returns, similar to the trends observed in Section 4.3. An interesting pattern emerges: as the amount of real data decreases, the optimal value of α tends to increase. This trend is consistent across all datasets. For instance, in the Cars dataset, when all real data is used (Frac. 1.0), α = 0.6 performs worse than α = 0.4. However, for smaller percentages of real data (e.g., 10%, 30% or 50%), using α = 0.6 yields better performance. This pattern is similarly observed in the Aircraft and CUB datasets, indicating that higher values of α are more beneficial when the amount of real data is limited.diff-mix [60] compared its method to more augmentation techniques. In this section, we present the complete results, including comparisons to those other methods. [20] with a resolution of 448 2 and ViT-B/16 [14] with a resolution of 384 2 , both of which are higher than the 224 2 resolution we use across the paper. We incorporate the integration of ControlNet and BLIP-diffusion for Cars and CUB datasets. We do not use BLIP-diffusion for the Aircraft dataset as it proved to be a better option, as evidenced in Table 4 . The accuracies of diff-mix and other methods, as reported in Table 1 of the diff-mix paper [60] , establish the benchmarks for our comparative analysis.Comparison Methods. The compared methods, implemented by diff-mix, include (1) Real-Filtering (RF) and (2) Real-Guidance (RG), both proposed by He et al. [21] . RG is described in Appendix D.3, which they implement with a lower translation strength (s = 0.1). RF is a variation of Real-Guidance that generates images from scratch and filters out low-quality images by using CLIP [41] features from real samples to exclude synthetic images that resemble those from other classes. (3) DA-Fusion [54] solely fine-tunes the identifier using textual inversion [18] to personalize each sub-class and employs randomized strength strategy (s ∈ {0.25, 0.5, 0.75, 1.0}), and non-generative augmentation methods (4) CutMix [66] and (5) Mixup [67] .Results. Results show that SaSPA outperforms all methods across both architectures when evaluated on Aircraft and Cars, despite diff-mix using heavy fine-tuning. Continuing the discussion on CUB evaluation in Section 4.5, CutMix outperforms all methods when using the ViT-B/16 architecture, while diff-mix leads on ResNet50. Notably, SaSPA outperforms all other generative baselines on CUB except diff-mix using both architectures.Across our experiments, we combined SaSPA with the best traditional augmentation method, as described in Section 4.1. To test how SaSPA behaves without augmentation and whether it depends on the best traditional augmentation, we evaluated its interaction with CAL-Aug [44] (the default traditional augmentation used in CAL), the best traditional augmentation, and no traditional augmentation at all. Note that for 3 out of 5 datasets, CAL-Aug is the best traditional augmentation, so we did not provide separate results for SaSPA with the best traditional augmentation for these datasets (as they are the same). From the results in Table 7 , we observe that using no traditional augmentation significantly under-performs compared to using CAL-Aug or the best traditional augmentation. Additionally, CAL-Aug proved to be a robust choice, yielding similar accuracy across all datasets, with only a slight decrease in performance for CompCars.Our primary experiments utilized ResNet50 as the backbone architecture for CAL. To further evaluate how deeper backbones or other network architectures might benefit from our augmentation method, we analyzed results across three FGVC datasets as detailed in Table 8 . The performance of both ViT [14] and CAL with ResNet101 as a backbone is presented. Results indicate that both deeper networks, such as ResNet101, and the ViT architecture benefit from our augmentation method. Together with our comparison with diff-mix [60] using their training setup, this demonstrates that our method is robust across a variety of architectures and training setups.In this study, we start with a base training set, utilizing 50% of the available real data. We examine the impact of varying the number of SaSPA augmentations, M , from 0 to 5 on the validation accuracy for Aircraft and Cars datasets. Additionally, we compare it to the effect of increasing the dataset size by adding an additional 25% of the real data without SaSPA augmentations.Results presented in Figure 6 demonstrate a consistent increase in validation accuracy for both datasets as the number of augmentations increases. Notably, the Cars dataset shows robust performance improvements, even surpassing the results achieved by adding 25% of real data when M = 4, indicating the effectiveness of the SaSPA augmentations in this context. For the Aircraft dataset, the accuracy nearly reaches the levels achieved by adding 25% more real data when M = 5.In response to reviewer feedback, we expanded our evaluation to include two additional FGVC datasets: Stanford Dogs and the Oxford-IIIT Pet Dataset, to further assess the robustness of our proposed method, SaSPA. Stanford Dogs comprises 20,580 images from 120 dog breeds, while the Oxford-IIIT Pet Dataset includes 7,349 images from 37 breeds (25 dog and 12 cat breeds).We compared with CAL-Aug, Real Guidance, and ALIA. Results, presented in Table 9 , indicate that SaSPA improves performance on both datasets, thereby strengthening the findings regarding its efficacy as a generative augmentation method. Combined with earlier DTD and CUB datasets results, this evaluation confirms that SaSPA effectively handles non-rigid objects. To assess the effectiveness of our proposed prompt generation, we evaluated various prompt strategies, and the results are detailed in Table 10 . To accelerate the experimentation process, these experiments were conducted using only ControlNet with SD XL Turbo on 50% of the data. Five main strategies were compared: (1) Captions: Direct use of captions as prompts, leveraging BLIP-2 [28] for captioning, as demonstrated to be effective in prior work [25] . ( 2) LE (Language Enhancement) [21] and (3) ALIA [15] are described in Appendix D.3. (4-5) Our Method with and without appending artistic styles. The artistic style augmentation involves appending half of the prompts with the phrases ", a painting of <artist>", where <artist> refers to renowned artists such as van Gogh, Monet, or Picasso. This approach aims to diversify textures and colors, prompting an increase in the model's robustness.The results show that our prompt generation method, either with or without incorporating artistic styles, consistently outperforms other approaches. Caption-based prompts yield the least effective performance, while the ALIA and LE methods fall somewhere in between.To evaluate whether increasing the number of prompts would enhance our method's performance, we compared generating 200 prompts to generating 100 prompts using our method. The results in Table 11 show no significant difference when using 200 prompts, indicating that 100 prompts are sufficient.A common metric to evaluate the quality of a generative model is the Fréchet Inception Distance (FID) [22] , a metric that measures the similarity between the distribution of generated images and real images. However, does it accurately measure how effective an augmentation method is? In Table 12 , we report the FID values, calculated using augmentations alongside their respective real datasets, as well as the corresponding accuracy achieved with each augmentation method. We observe that generative baselines such as Real Guidance and ALIA achieve lower FID scores, which suggest a higher similarity to the real data distribution. We suspect that this is the result of generating images that closely mimic the original dataset. In contrast, our method, SaSPA, is designed to create diverse augmentations that substantially differ from the real images, leading to higher FID scores. Despite these higher FID values, as shown in Table 12 , SaSPA demonstrates superior performance enhancements in accuracy across datasets. This highlights the importance of evaluating generative augmentation methods not only based on realism and similarity to real images, as measured by FID, but primarily on their actual impact on model performance. In the next section, we further provide an alternative metric for generative data augmentation.LPIPS [69] measures the perceptual difference between two images. By calculating the average LPIPS distance between original images and their respective augmentations, we can quantify the diversity introduced by an augmentation method. We argue that this metric, combined with qualitative evidence of class fidelity, provides a robust measure for evaluating generative data augmentation. Note that this metric will apply only for augmentations that are derived from real images. Generation from scratch will require a different metric, probably a dataset-level diversity metric. Table 13 demonstrates that SaSPA achieves significantly higher LPIPS scores compared to Real Guidance (RG) and ALIA, indicating that SaSPA introduces much greater diversity in the generated augmentations. This substantial increase in diversity is crucial for enhancing model robustness and performance [31] . Qualitative evidence can be found in Figure 7 .B.12 Investigating the Potential of Newer Base Models Recently, text-to-image diffusion models have made incredible progress, particularly those based on Stable Diffusion (SD) [46] . Notable advancements include SD XL [40] and SD XL Turbo [51] . In this section, we aim to explore the compatibility of Edge Guidance with other base models. Unfortunately, as these models are relatively new, BLIP-diffusion [27] has not yet released versions built upon them, preventing us from utilizing subject representation. However, as shown in Table 4 , our full pipeline without subject representation still achieves impressive results. Additionally, Table 4 indicates that when subject representation is not used, it is slightly beneficial for most datasets, except CUB, to append half the prompts with artistic styles. Therefore, we adopt this strategy. In Table 14 , we experiment with SD v1.5, SD XL, and SD XL Turbo. Note that ControlNet versions for SD XL and SD XL Turbo are still experimental, and will require reevaluation as the models mature.The results indicate that integrating Edge Guidance generally has a positive impact across base models, except on the CUB dataset, which aligns with our earlier findings in Table 4 . Additionally, SD XL and SD XL Turbo typically outperform SD v1.5, suggesting that more advanced base models may lead to further improvements in performance. A common technique when training with synthetic data is to first train using the synthetic data, then fine-tune on the real data [56, 38, 17] . Inspired by this, we investigate stopping the data augmentation at earlier training epochs. Results are presented in Table 15 . For these ablation experiments on evaluating early augmentation stoppage, we used 50% of the real Aircraft dataset for faster experimentation. We find no benefit from early augmentation stoppage. There is a downward trend in accuracy when stopping early, with the worst results at the earliest epoch stopped (20 out of 140). We believe that the high diversity introduced by our augmentations reduces over-fitting, mitigating the need for an explicit domain adaptation strategy. As a result, the model continues to benefit from the augmented data throughout the training process. Additional experiments were conducted using a 448x448 resolution on the CompCars, DTD, and CUB datasets, employing SaSPA and the best augmentation methods identified in Table 1 . The experiments, replicated with two different seeds, are detailed in Table 16 .Combined with our earlier diff-mix comparisons at both 448x448 and 384x384 resolutions (Table 3 ), these results complete our high-resolution evaluation across all datasets. Notably, we observed consistent performance improvements in all datasets except CUB. We hypothesize that CUB's finegrained details such as feather patterns and colors present significant challenges at higher resolutions, impacting the efficacy of generative methods. In conclusion, SaSPA demonstrates promising results for most datasets, affirming its overall benefits. Using Canny edge maps [6] as a condition has proven effective for generating images with high diversity and class fidelity. Here, we experiment with a different kind of edges: Holistically-Nested Edge Detection (HED) edges [63] . Canny edges are more focused on detecting the intensity gradients of the image, often capturing finer details, whereas HED edges provide a more structured representation by capturing object boundaries in a holistic manner. We experiment with both types in Table 17 , using the default generation parameters without using BLIP-diffusion, and on 50% of the data for faster experimentation. Using Canny edges resulted in slightly higher validation accuracy on the Aircraft dataset.We provide the number of samples for each dataset split used in our experiments in Table 18 . Additionally, we include the number of images for each background class (sky, grass, road) used to create the contextually biased training set, as shown in Table 19 . We utilize the dataset test split for the reported test. For datasets lacking a validation split (Cars, CUB, CompCars), we generate one by using 33% of the training set. Note that in the diff-mix training setup (Section 4.5), the training datasets for CUB and Cars consist of the original splits, as they did not use a separate validation split.Unless stated otherwise, the following experimental setup applies to all experiments in the paper. Data Generation. All generative methods use the Diffusers library [57] . We employ BLIPdiffusion [27] and ControlNet [68] for SaSPA. Besides the prompt, an edge map for ControlNet, and a reference image for BLIP-diffusion as inputs for our generation, BLIP-diffusion requires source subject text and target subject text as inputs. We simply use the meta-class (e.g., "Airplane" for Aircraft dataset, "Bird" for CUB) of the dataset for both source and target subject texts. For all other diffusion-based augmentation methods, we use Stable Diffusion v1.5 [46] . For all diffusion-based models, including SaSPA, we use the DDIM sampler [53] with 30 inference steps and a guidance scale of 7.5. Images are resized to ensure the shortest side is 512 pixels before processing with Img2Img or ControlNet. We set the ControlNet conditioning scale to 0.75. For text-to-image generation, images are generated at a resolution of 512x512. We generate M = 2 augmentations per original image for each experiment, and we use augmentation ratio α = 0.4 for all datasets except CUB [58] , for which we use α = 0.1 as evidenced to be better in Figure 5 . We use k = 10 in the top-k Confidence filtering. We use four NVIDIA GeForce RTX 3090 GPUs for image generation and training.Training. We follow the implementation strategy outlined in the CAL study [44] , tailored for FGVC. We use ResNet50 [20] as the primary architecture within the CAL framework unless specified otherwise. Each dataset is fine-tuned using pre-trained ImageNet [12] weights. Optimization is performed with an SGD optimizer, with a momentum of 0.9 and a weight decay of 10 -5 , over 140 epochs. We adjust the learning rate and batch size during hyper-parameter tuning to achieve the highest validation accuracy. Training images are resized to 224x224 pixels. Results are averaged across three seeds. Specific values of hyper-parameters are in Table 20 .Specifics on DTD [9] dataset. The DTD dataset is a collection of images categorized by various textures, such as Marbled, Waffled, and Banded. We found that this dataset differs from other fine-grained datasets as it is not fine-grained at the same level. Classes like "Marbled" and "Waffled" have significant differences from each other. Therefore, feeding the meta-class ("Texture") to an LLM will provide prompts that are not suitable for all sub-classes in the dataset. Hence, we did not use our prompt generation method. This could be addressed in the future by feeding the LLM with each sub-class. Instead, we simply used image captions.Hyper-parameters To select the hyper-parameters for each dataset, we train CAL [44] with learning rates of [0.00001, 0.0001, 0.001, 0.01, 0.1] and batch size [4, 8, 16, 32] , selecting the configuration that results in the highest validation accuracy. These parameters, shown in Table 20 , are then used across all methods.In this section, we outline the computational resources required for our primary experiments. We utilize four NVIDIA GeForce RTX 3090 GPUs for image generation and training purposes, but we report running times for a single GPU. Training with ResNet50 necessitates up to 5.5 GB of GPU RAM. The duration of our experiments varies depending on the dataset, with the longest running being approximately three hours. As no fine-tuning is performed, our generation process includes [64] 0.001 8 10 -5 140 SGD 0.9 Cars [24] 0.001 8 10 -5 140 SGD 0.9 CUB [58] 0.001 16 10 -5 140 SGD 0.9 DTD [9] 0.001 16 10 -5 140 SGD 0.9 Airbus vs. Boeing [15] 0.001 4 10 -5 140 SGD 0.9 only I/O and a forward pass through the generation model. We report here only the generation times and do not include I/O times, as these can vary heavily based on system configuration and server load. For image augmentation using ControlNet with BLIP-diffusion as the base model, generating each image takes 2.96 seconds and requires up to 10 GB of GPU memory. Therefore, creating two augmentations for the Aircraft dataset's training set would take approximately five and a half hours. When switching to SD XL Turbo as the base model, with two inference steps (the default for this base model), the augmentation time is reduced to 0.52 seconds, and the GPU memory requirement increases to up to 16 GB. In this configuration, generating two augmentations for the Aircraft dataset's training set would take less than one hour.In this section, we provide additional details on the generative baselines we compared against.Real-Guidance (RG): This method achieves impressive few-shot classification performance. For prompt generation, an off-the-shelf word-to-sentence T5 model, pre-trained on the "Colossal Clean Crawled Corpus" [42] and fine-tuned on the CommonGen dataset [29] , is utilized to diversify language prompts. The model is used in order to generate a total of 200 prompts based on the meta-class. For image generation, SDEdit with a low translation strength (s = 0.15) is used. Filtering is performed using CLIP filtering, which is described in Section 3.4.ALIA [15] : This method showed impressive results in addressing contextual bias and domain generalization. For prompt generation, GPT-4 [1] is employed to summarize image captions of the training dataset into a concise list of fewer than 10 domains, which are then used inside prompts. Image generation is carried out using either SDEdit with medium strength (around 0.5) or InstructPix2Pix [5] . For filtering, they use a confidence-based filtering approach where a model f is trained, and a confidence threshold t y for each class y is established by averaging the softmax scores of the correct labels from the training set. An edited image x ′ with a predicted label ŷ is excluded if confidence(f (x ′ ), ŷ) ≥ t ŷ . This thresholding ensures that images for which the predicted label ŷ matches the true label y are removed due to redundancy. Additionally, images where ŷ ̸ = y with high confidence are also filtered out because they likely represent a significant alteration, making them resemble another class more closely. For our pipeline, we observe that this approach tends to overly filter augmentations where ŷ = y, as augmentations that could be redundant due to similarity to the original real image do not occur in our method, as we do not use real images as guidance for augmentation.In this section, we provide more details on how we used GPT-4 to create prompts. After identifying the meta-class of the FGVC dataset, we input it into the following instruction:"Generate 100 prompts for the class [meta-class] to use in a text-to-image model. Each prompt should:• Include the word [meta-class] to ensure the image focuses on this object.• Ensure diversity in each prompt by varying environmental settings, such as weather and time of day. You can include subtle enhancements like vegetation or small objects to add depth to the scene, ensuring these elements do not narrowly define the [meta-class] beyond its broad classification.• The prompts should meet the specified quantity requirement."No quality control is used over the generated prompts.This section elaborates on our filtering mechanisms that remove lower-quality augmentations that do not correctly represent the sub-class or the meta-class. Additionally, we compare the effectiveness of alternative filtering methods in Table 21 .Predictive Confidence Filtering utilizes the baseline model's confidence to filter out augmentations whose true label does not rank within the model's top-k predictions (k = 10). This baseline model is selected based on optimal performance outcomes from a hyperparameter sweep, as described in Appendix D.1. The choice of k can affect the results: using too low of a k can result in excessive filtering, limiting augmentations to those the baseline model already handles well, whereas too high of a k results in insufficient filtering, allowing low-quality augmentations to pass through. Therefore, we ablate on k as well to find the optimal value. We show a visualization of this filtering method in Figure 8 .We also evaluate other filtering methods, including CLIP filtering [21] , Semantic Filtering, and ALIA confidence filtering [15] , as described in Section 3.4 and Appendix D.3.Note that for certain datasets, such as Cars [24] and Aircraft [30] , the augmentations remain so consistent that only minimal filtering is required. For instance, out of the total augmentations produced for the Cars dataset and using our filtering method, only 0.1% were filtered out. However, this percentage is more significant for other datasets like CompCars [64] , where 4.5% of augmentations were filtered. Example augmentations of Real Guidance, ALIA, and our method are visualized in Figure 7 .Examples of augmentations that were and were not filtered for three FGVC datasets are in Figure 8 . Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The paper introduces a generative augmentation method for fine-grained visual categorization (FGVC) that creates diverse, class-consistent synthetic images using ControlNet conditioned on edge maps and BLIP diffusion for zero-shot image generation. The method shows promising results on datasets like Aircraft, Stanford Cars, CUB, DTD, and Compcars. 

All the reviewers are satisfied with the authors' concerns. The authors need to fix the minors concerns, including the need for a limitations section addressing issues like less satisfying performance on high-resolution datasets (e.g., CUB) and demonstrating broader applicability by testing on more diverse datasets beyond FGVC.

Reviews: review: paper_summary: paper_summary: This paper presents SaSPA, a generative augmentation method specifically designed for FGVC. This method generates diverse, class-consistent synthetic images through conditioning on edge maps and subject representation. They use ControlNet condoned on edge maps and uses blip diffusion for its ability to zero-shot image generation. Their method is really smart and novel. They show results on Aircraft, Stanford Cars, CUB, DTD and Compcars for fine-grained evaluation.
strength_weakness: strength_weakness: 1) Datasets are of small scale on which the experiments have been done. I would expect this method to perform well on fine-grained evaluation of imagenet dogs setup as well. Showing experiments on 200 classes of dogs on ImageNet will make the paper even stronger. 
2) How much is the performance improvement over methods like CLIP in few shot settings? Even if there is not much ( which is fair since here the aim is not image text training), it would be a good idea to show how behind this would be.
3) To be really honest, I believe the results are pretty incremental. In Table 3 the difference between Diff-Mix & SaSPA is 0.3%, which is quite small and is probably in the noise range. The method is really smart and novel and I think just using this method on small-scale experiments on fine-grained datasets is kind of underselling the paper. I believe the method has a lot of potential and can be used to do large-scale training. Especially using edge-based control net and blip diffusion is a really good idea.
4) I will urge the authors to rethink the experimental results and encourage them to show these results on larger datasets with more convincing results.
5) The Fig4 caption is really small and should be a bit more detailed.
questions: questions: I think the paper needs more experimental results, but the image generation is really good.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes a data-augmentation technique tailored to fine-grained image classification.
The goal is to increase class fidelity while maintaining high variance in the images, something current diffusion-based data augmentation techniques are struggling with, especially in the fine-grained domain.
 
The main idea is to preserve both the scene-structure (by conditioning on edges with controlnet) and subjects (by using blip diffusion). Rather than augmenting a single sample, the edge map, class subject reference image and prompt are sampled independently. An image filtering strategy is proposed to increase the data quality. The method doesn’t require any finetuning of the generative model.
 
The method is shown to outperform baselines on 5 datasets and different scenarios such as full dataset training and few-shot classification are explored. Better generalization to unseen backgrounds compared to classifiers trained on the data augmentation baselines is also shown.
strength_weakness: strength_weakness: 1. The baseline classifier architecture is a standard image classifier. Meanwhile, there are architectures tailored to the fine-grained classification domain such as the MetaFormer [1, 2], it would be preferrable to also evaluate on state-of-the-art classifiers from the FGVC domain.
2. It is not clear how the method would perform on a dataset of objects from a less common domain (not so well represented in the training data of the diffusion models)
See Questions for more details.
questions: questions: ## General remarks
1. Code release is only mentioned in section 3.5, which is easy to miss, I would place it in abstract/introduction.
2. Even though it is proposed for FGVC, the method is in principle applicable elsewhere - have you tried applying it to standard image classification datasets? Is there a reason why you think the method would not work well?

## Add weakness 2
3. The evaluation datasets consist of very common objects/animals which are well-represented in the training data of diffusion models such as cars and birds. It would help understanding the strenghts and weaknesses of the method to see its performance on a dataset with less common meta-classes, for example the iNaturalist.


**References**:

[1] Diao, Qishuai, et al. "Metaformer: A unified meta framework for fine-grained recognition." arXiv preprint arXiv:2203.02751 (2022).

[2] He, Ju, et al. "Transfg: A transformer architecture for fine-grained recognition." Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 1. 2022.
limitations: limitations: Limitations are adequately addressed in the appendix.
paper_summary: paper_summary: The paper proposes SaSPA, which uses several large, pre-trained generative, language, and vision-language models to produce high-quality synthetic images. Specifically they focus on images for FGVC, where the high inter-class similarities make it challenging to synthesize data that is not only diverse (capturing the intra-class variability) but also faithful to the intended class label. They combined GPT-4 with BLIP ControlNet diffusion, with edge map conditioning and post-generation filtering (via top-10 ranking with CLIP) to achieve performance gains compared to baselines, with only 2 augmentations per image (notably less than the 10x once thought to be a rule of thumb for training with synthetic data).
strength_weakness: strength_weakness: The results do not deliver on the promise of the paper's motivation. Principally,

[W1] It is unclear if this approach can contribute to some state-of-the-art result. In some areas (e.g. text-to-image generation itself), asking SOTA of a research people might represent an unreasonable burden. However, in an area like FGVC, such evaluations are both practical and necessary. The authors should show that the performance gains on the ResNet50 (Table 1) don't vanish when applied to stronger methods (and I don't really count ResNet101 here, it's barely better than R50), particularly for CUB.

Speaking of CUB, it seems from the teaser that this method may struggle to capture the most fine-grained differences, such as when it recolors the engine. Perhaps this is why the method does not fare so well with the birds dataset, or in other words,

[W2] The method's inferiority to CutMix on CUB for 448x448 images is very concerning (Table 3). The higher resolution is the standard for FGVC, and a method that only works well for low resolution is quite limited in impact. Additionally, it calls into question whether the method is good mainly for images of rigid objects, or if it could work well for other datasets of living things (Stanford Dogs, Fungi, NABirds). 

On a more minor note,

[W3] Some information seems misplaced. Image resolution is critical context in FGVC, especially if the paper uses smaller-than-normal images. The comparison in Table 3 should also compare M for diff-mix and the SaSPA.

and

[W4] Using CLIP for the filtering step seems strange. A model trained on the dataset (or even an ensemble) would likely filter the data much more reliably. Would this not be a much stronger approach?
questions: questions: My weaknesses correspond to questions.

[W1] - If this data augmentation is applied to a SOTA FGVC model, is the impact orthogonal, e.g., do I get a new SOTA? Or is this just another way to get the model to learn the same information as the existing tricks? 

[W2] - Does this method work well beyond Cars/Aircraft/DTD? Will it break for other datasets, similar to how it seems to break for CUB?

[W4] - How was CLIP selected for the filtering step?

I am currently leaning reject, mainly due to W1 and W2. For clarity's sake, I have similar concerns about diff-mix and would not necessarily have voted to accept, had I reviewed that paper. I say this just because W1 applies to diff-mix as well, and replying to my review by pointing this out will not assuage my concern.
limitations: limitations: Yes.

Comments: title: Concerns Resolved
comment: We are happy that most of your concerns are resolved, and we thank you for your positive feedback.
Thank you for the response and your effort!

My concerns have been mostly addressed and I am keeping my positive score. The only remaining one is that I am not convinced the DTD dataset results are enough to show the generalization of the method to metaclasses less frequently represented in the training data of diff. models.

 As mentioned in the original review, I think this is a well-written paper with solid contributions. To make the contributions stronger, I am missing positive results on more diverse datasets, for example, showing the broader applicability of the method outside of FGVC, or more datasets with less common objects, to increase my score.
We are happy that all your concerns are resolved. We will incorporate the clarifications, and add a clearly marked limitation section as you suggested.
All my weaknesses have been properly addressed. My apologies for the confusion on your Predictive Confidence Filtering (and corresponding F.2).

As far as diff-mix goes, I realize now that I wasn't clear in my review, but I realize that it is concurrent work (that's why I mentioned it as minor, on both strengths and weaknesses). Presenting it isn't mandatory, but given that it has presented, preferably it would be presented properly (and it seems the main issue was corrected). As far as my evaluation of this work, and my rating, the comparison to diff-mix is only a positive, insofar as it reflects this work's thoroughness.

I would suggest potentially adding a clearly marked limitations section to the final manuscript to address lingering issues such as less satisfying performance on high resolution CUB.