Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentation
Authors: Eyal Michaeli
Ohad Fried
Venue: NeurIPS 2024 poster (2024)
Keywords: Fine-grained Visual Classification
Data Augmentation
Synthetic Data
Diffusion Models
Image Classification
Primary area: machine_vision
TL;DR: Our paper introduces SaSPA, a novel data augmentation (DA) method for fine-grained visual classification that surpasses existing DA methods by generating synthetic data that is both more diverse and accurately represents fine-grained classes.

Abstract: Fine-grained visual classification (FGVC) involves classifying closely related subcategories. This task is inherently difficult due to the subtle differences between classes and the high intra-class variance. Moreover, FGVC datasets are typically small and challenging to gather, thus highlighting a significant need for effective data augmentation.
Recent advancements in text-to-image diffusion models have introduced new possibilities for data augmentation in image classification. While these models have been used to generate training data for classification tasks, their effectiveness in full-dataset training of FGVC models remains under-explored. Recent techniques that rely on text-to-image generation or Img2Img methods, such as SDEdit, often struggle to generate images that accurately represent the class while modifying them to a degree that significantly increases the dataset's diversity. To address these challenges, we present SaSPA: Structure and Subject Preserving Augmentation. Contrary to recent methods, our method does not use real images as guidance, thereby increasing generation flexibility and promoting greater diversity. To ensure accurate class representation, we employ conditioning mechanisms, specifically by conditioning on image edges and subject representation.
We conduct extensive experiments and benchmark SaSPA against both traditional and generative data augmentation techniques. SaSPA consistently outperforms all established baselines across multiple settings, including full dataset training and contextual bias. Additionally, our results reveal interesting patterns in using synthetic data for FGVC models; for instance, we find a relationship between the amount of real data used and the optimal proportion of synthetic data.

Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentation
Eyal Michaeli
Ohad Fried Department of Computer Science Reichman University
Department of Computer Science Reichman University
eyal.mi ctl.mmiihael@ppstt runi.ac.il ac
ofriedCruni. ac. il
Texi2Image
Img2lmg (5.0.5)
Img2lmg (5#00.55
SaSPA (Ours)
Original
Asgndet
SoSPA
SUPII
Augmenting FGVC
Trode-off
Diverity
Boeing 367-200 airplane flying high above froven fundra, icy fandscapss"
Figure 1: Various generative augmentation methods applied on Aircraft (30 Text-to-image often compromises class fidelity, visible by the unrealistic aircraft design (i.e., tail at both ends). Img2lmg trades off fidelity and diversity: lower strength (e-g., 0.5) introduces minimal semantic changes, resulting in higher fidelity but limited diversity, whereas higher strength (e.g., 0.75) introduces diversity but also inacouracies such as the incorrectly added engine. In contrast, SaSPA achieves high fidelity and diversity, critical for Fine-Grained Visual Classification tasks. D Diversity, F Fiidelitty
Abstract
Fine-grained visual classification (FGVC) involves classifying closely related sub-classes. This task is difficult due to the subtle differences between classes and the high intra-class variance. Moreover, FGVC datasets are typically small and challenging to gather, thus highlighting significant need for effective data augmentation. Recent advancements in text-to-image diffusion models offer new possibilities for augmenting classification datasets. While these models have been used to generate training data for classification tasks, their effectiveness in full- dataset training of FGVC models remains under-explored. Recent techniques that rely on Text2Image generation or Img2lmg methods, often struggle to generate images that accurately represent the class while modifying them to degree that significantly increases the dataset's diversity. To address these challenges, we present SaSPA: Structure and Subject Preserving Augmentation. Contrary recent methods, our method does not use real images as guidance, thereby increasing generation flexibility and promoting greater diversity. To ensure accurate class representation, we employ conditioning mechanisms, specifically by conditioning on image edges and subject representation. We conduct extensive experiments and benchmark SaSPA against both traditional and recent generative data augmentation methods. SaSPA consistently outperforms all established baselines across multiple settings, including full dataset training, contextual bias, and few-shot classification. Additionally, our results reveal interesting patterns using synthetic data for FGVC models; for instance, we find relationship between the amount of real data used and the optimal proportion of synthetic data. We release our source code.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Deep learning's remarkable success across various applications relies heavily on large-scale annotated datasets, such as ImageNet [1]], which provide the foundational data necessary for training effective models. However, in fine-grained visual classification (FGVC), the datasets are typically smaller and less diverse, presenting unique challenges in training robust models. Data augmentation emerges as a natural solution to artificially enhance dataset size and variability. However, traditional data augmentation methods are limited in the amount of diversity they introduce [[5)]
Text-to-image diffusion models have opened new avenues for generative image augmentation. Within the realm of classification, diffusion models have shown promise on standard image recognition However, their application in FGVC remains under-explored.
datasets such as ImageNet 22 50 3
Generating synthetic data for FGVC presents unique challenges, as preserving class fidelity is (I) more crucial than with common object datasets due to the similarity between classes and the reliance of the models on subtle details to differentiate between classes and (2) challenging to achieve because the training data for rext-to-image models often lacks a substantial representation of these distinct objects [26]. For instance, there might be enough data to accurately represent "An airplane", but not "A Boeing 767-200 airplane".
Recent generative methods evaluated for FGVC augmentation typically use real images as guidance in Img2Img manner [15 [54] 21 60) While this helps maintain visual similarity to the target domain, it limits the degree of diversity that can be introduced, resulting in a trade-off between class fidelity and diversity (16] (see Figure[]. We aim to free the generative process from this constraint of adhering to specific source images. To this end, we propose SaSPA: Structure and Subject Preserving Augmentation, a method that conditions the generation on more abstract representations rather than direct image inputs. Specifically, we leverage structural conditioning in the diffusion model via edge maps extracted from source images. This allows the generated samples to respect the broad shape and composition of objects in the target domain. Crucially, the lack of specific image conditioning enables greater flexibility in rendering surface details. To further ensure the preservation of fine- meneggrinedd class characteristics, we integrate subject representation conditioning. By combining edge-based structural conditioning with category-level conditioning, SaSPA can generate highly diverse, class-consistent synthetic images without being overly influenced by any specific real data sample.
Furthermore, to enrich the diversity and applicability of our generated images, we generate prompts with an LLM according to the dataset meta-class (a class encompassing all sub-classes). These prompts are designed to guide the diffusion model in producing variations that are not only diverse but also dlasscconsistent and relevant to the target domain. Additionally, to maintain the quality and relevance of the generated images, we implement a robust filtering strategy that eliminates any samples that fail to meet predefined quality thresholds by utilizing Mataset-traine model and CLIP.
We summarize contributions follows: (1) We propose SaSPA, generative augmentation pipeline for fine-grained visual classification that generates diverse, class caass-oonsistett synthetic images without relying on specific real images for conditioning.
(2) We conduct extensive experiments and benchmark SaSPA against both traditional and recent generative data augmentation methods. SaSPA consistently outperforms all established baselines across multiple settings, including the challenging and less-explored full dataset training, as well as in scenarios of contextual bias and few-shot classification. (3) Our analysis provides insights on effectively leveraging synthetic data to improve the performance of tine-graing classification models. For instance, we find that as the amount of real data decreases, we should increase the proportion of synthetic data used.
2 Related Work
Data Augmentation with Generative Models. Synthesizing training samples using generative models is an active and challenging area of research. Initial efforts in this field 770 4
49 33
leveraged Generative Adversarial Networks (GANs) to create labeled training samples. Recently, the emergence of powerful text-to-imape diffusion models such as Stable Diffusion [46] has created
exciting opportunities for advancing generative image augmentation. These models have been
employed across a range of applications. including semantic segmentation [19 61
62 36 object
detection R 7 (9). and classification <4 2 50 B]. demonstrating their versatility and effectiveness
For image classification tasks, diffusion models have demonstrated promising results on standard

---

Meta-class "Aiplann""
GPT4
Prompts
Augmentations
Filter
loob alooe] sirpone Bijng dobett"
ControlNet + BLIP
Subject Reference Images
X
Diffusion
(1)
(2)
Real Image
Edges Map
(20
D x
Edge Detector
Figure 2: SaSPA Pipeline: For a given FGVC dataset, we generate prompts via GPT-4 based on the meta-class. Each real image undergoes edge detection to provide structural outlines. These edges are used M times, each time with different prompt and different subject reference image from the same sub-class, as inputs to a ControlNet with BLIP-Diffusion as the base model. The generated images are then filtered using dataset-trained model and CLIP ensure relevance and quality.
image recognition datasets such ImageNet 50 3
However, their application in FGVC
has typically been limited to particular settings such as few-shot learning [21, [4| 52 26) where data scarcity significantly enhances the impact of data augmentation, contextual bias, and domain generalization [15], settings that are more straightforward to enhance as targeted augmentations can directly address and balance the skewed distributions. Our goal is tackle the more challenging task of training on full FGVC datasets. Moreover, recent generative augmentation methods often use Img2Img techniques like SDEdit to maintain class fidelity, though this comes at the cost of reduced diversity. Some methods involve fine-tuning the network or its components, which can be expensive and may still struggle to balance class fidelity with the added diversity necessary for effective FGVC augmentation. Our goal is to avoid the decrease in diversity associated with using real images as guidance and avoid the complexity and expense fine-tuning the generation model.
text-to-Image Diffusson Models. Diffusion models 23 have achieved unprecedented success in generating photo-realistic images 133 Models like Stable Diffusion [46]. DALL-E 2 43], and others 33 48] exemplify this capability. These models have also driven advancements in other generative areas. For instance, SDEdit [32) integrates real images partway through the reverse diffusion process for image editing. Techniques like ControlNet [68] and T2I-Adapter [35] condition image generation on inputs beyond text such as edges and world normals, while methods such as Textual Inversion 18 and DreamBooth [47] can generate specific subjects from just few example images. More recently, dlff-diifusion 227] which based on Stable Diffusion and BLIP-2 (28) has demonstrated impressive zero-shot subject-driven generation using only one example image. Our method benefits directly from these advancements, employing ControlNet and BLIP- BLIf-diffusion.
Traditional Data Augmentation. Traditional data augmentation methods typically include operations such random cropping, flipping, and color-space changes to generate new variations [10]. Recent strategies, like mixup-based methods, aim to enhance diversity by blending patches from two input images [66] or using convex combinations [67. Weakly Supervised Data Augmentation Network used in recent FGVC works such as CAL [44], aims to improve FGVC by generating attention maps to highlight discriminative object parts and guiding augmentation with attention cropping and dropping. However, these methods introduce limited diversity [15], as they do not alter the semantic features present in the image.
3 Method
Our goal is to augment labeled training dataset for FGVC to increase its diversity while faithfully representing the sub-classes. The key insight our method is to minimize reliance on any particular source image during generation and instead condition the generation on more abstract representations, thereby increasing diversity while accurately representing the designated class (see Figure TT. To achieve this, we employ abstract conditions such edges, which capture the object's structure,
3

---

and subject representation. which aims to preserve fine-gnh grained class characteristics. The process, illustrated in Figure 2 unfolds five steps, outlined below:
3.1 Construction of Prompts
We aim to use generative text-to-image model, which requires input prompts to guide the image synthesis process. To ensure that the prompts generated align broadly with the primary category of each dataset, method begins by identifying the meta-class for each dataset, such "Airplane" for the Aircraft [30] dataset and "Car" for the Stanford Cars (24] dataset.
Prompt Generation via 4 11 We input the meta-class to GPT-4 with the instruction to produce 100 unique, relevant, and diverse prompts, each inherently containing the term of the meta-class. This strategy ensures that the generated images stay true to the fundamental aspects of the meta-class while hopefully containing relevant and diverse scenarios. To increase the specificity and relevance of these prompts, we integrate the relevant sub-class into each prompt whenever it is used. Unlike recent work [13, which also uses GPT-4 to create prompts, we do not require image captions of the dataset. The exact instructions for GPT-4 and more example prompts are in Appendix E.1
3.2 Visual Prior Extraction
To ensure that the generated images maintain the overall structure and shape of objects belonging to their respective classes, condition the synthesis process edge representations extracted from real images in the dataset. Concretely, for a dataset of N images, we extract one-channel edge representations using the Canny edge detector [6. This yields set of N edge-based visual priors that capture the structural characteristics of each sub-class. By conditioning the generative model on these edge maps, we can preserve object shape and layout during synthesis while allowing flexibility in rendering other surface-level details. In Table 17 we additionally explore the use of HED an alternative technique for edge extraction and structural conditioning.
66) as
3.3 Image Generation
Unlike most recent approaches [21 our method focuses on edges and subject representation as a prior rather than the source image. We show in Figure I that this approach maintains class fidelity, and result of not using the source image, affords the model greater flexibility to introduce novel semantic features such as weather, lighting, or even elements both within and outside the object's confines. This strategy might be particularly beneficial in FGVC tasks, where the subtle differences between classes are crucial, and hence, maintaining class fidelity while introducing diversity is of paramount importance.
Diversity in synthetic data is crucial for effective training [55 45 31
Conditioning on edge maps of real image ensures that generated images align closely with real structural features. Interestingly, we notice that structural conditioning cues the generation model to accurately represent the target sub-class, enhancing not only the correct representation of structural features but also the correct representation of non-structural attributes like color and texture. This structure-guided synthesis approach effectively enhances the model's ability to maintain class fidelity across varied image generations.
Conditioning on subject representation further enhances the generation model's ability to produce images with accurate sub-class representation. This ensures the correct representation of the sub-class on levels beyond structure, such as texture, color, and other visual features.
Using these two mechanisms together ensures correct sub-class representation across datasets, whether the primary distinctions between sub-classes lie in structural features, texture, color, or any combination of them.
Due to its impressive results and widespread use, we employ ControlNet [68 conditioned on Canny edges 66) for edge map conditioning. For subject representation conditioning and as a base model for ControlNet, we utilize BLIP-diffusion 277 model built upon Stable Diffusion [46 and BLIP-2 2 28 that emphasizes subject representation and supports zero-shot subject-driven generation using one reference image. We chose Bifi-diffusion for its zero-shot capabilities and open-source availability.
Using BI.f--iifussonn with ControlNet requires a prompt, an edge map, and reference image the target subject. To maintain sub-class accuracy while introducing diversity at the sub-class level, the

---

up of Headllghtt of "graphite 13 car snowy urban setting.
"A coblat blue (} car winter
"A aquamarine {
"pastel blue i car in charming old european town"
car in rain- drenched tropical
wonderland,
Original
streetlights glowing"
Original
covered trees"
foresi"
CompCary
Cars
*A bird on sandy beach with the Background"
(A bird on brunch colored leaves"
close up knitted bag with O testure"
Original
Original
CUB
DTI
Figure 3: Example augmentations using our method (SaSPA). The {} placeholder represents the specific sub-class.
reference image is selected from the same sub-class but differs from the real image used to extract the edge map (we experiment with BLIP-diffusion inputs T. Specifically, we generate M augmentations for each real image in the training set: extract edge map for each real image, and for each edge map, we randomly select M prompts and M subject reference images from the same sub-class. These inputs are then fed into the generation model of ControlNet with BLIP-diffusse as the base model to produce M augmentations
the real image. Example augmentations are visualized at Figure 3 DTD [9 has only prompt per real image, because utilize image captions as prompts for it, as explained in Appendix
3.4 Filtering
We aim to remove luw-quality augmentations, which appear in two forms: (1) meta-class misrepre- sentation and (2) sub-class mistepresentation.
Semantic Filtering. To alleviate meta-class misrepresentation, we utilize semantic filtering as described in ALIA [15]. Using CLIP [11, this process evaluates the relevance of generated images the specific task at hand. For example,
car dataset, each generated image assessed against variety of prompts such as "a photo of car", "a photo of an object", "a photo of a scene", "a photo", and black photo". Images that CLIP does not recognize as "a photo of a car" are excluded to ensure that the augmented dataset closely aligns with the target domain.
Predictive Confidence Filtering. To ensure each augmentation faithfully represents its designated sub-class, we implement predictive confidence filtering strategy inspired by recent work [2I] strategy CLIP Filtering. This method employs CLIP [4] filter out images that do not strongly correlate with the textual labels of their class among all classes in the dataset. However, the limitation of using CLIP in this context its insufficient granularity understanding fine-graine For our method, we discard any augmented images for which the true label does not rank within the top-k predictions of baseline model trained on the original dataset. This approach helps to exclude images that likely misrepresent the source sub-class, thus maintaining high-quality dataset for model training. In our implementation, we use 10. Further details about this method, the baseline model used, and other filtering techniques are discussed in Appendix F.2
concepts.
3.5 Training Downstream Model
We train the downstream classification model using the filtered, generated samples. Let denote the augmentation ratio, representing the probability that real training sample will be replaced with generated synthetic sample during each epoch. This replacement process is repeated for every sample each epoch, allowing each real sample to be either retained or replaced by an augmented version. We employ this replacement strategy instead of simply adding the augmented data to the original dataset, as doing so would unnecessarily increase the number iterations per epoch. By that, we ensure fair comparisons across training sessions.
5

---

Table 1: Results on full FGVC Datasets. This table presents the test accuracy of various augmen- tation strategies across five FGVC datasets. The highest values for each dataset are shown in bold, while the highest validation accuracies achieved by traditional augmentation methods are underlined.
Type
Augmentation Method
Aireraft
CompCars
Cars
CUB
DTD
No Aug
81.4
67.0
91.8
81.5
68.5
CAL-Aug
84.9
70.5
92.4
82.5
69.7
RandAug
83.7
72.5
92.6
81.5
69.3
Traditional
CutMix
81.8
66.9
91.7
81.8
69.2
CAL-Aug + CutMix
84.5
70.2
92.7
82.4
69.7
RandAug CutMix
84.0
72.6
92.7
81.2
69.2
Real Guidance
84.8
73.1
92.9
82.8
68.5
Generative
ALIA
83.1
72.9
92.6
82.0
69.1
SaSPA w/o BLIP-diffusion
87.4
74.8
93.7
83.0
Ours
SaSPA
86.6
76.2
93.8
83.2
71.9
4
Experiments
Our objective is to explore the extent to which synthetic data, particularly through our approach, contributes various FGVC tasks. We aim to understand the significance of each component of our method and identify optimal strategies for leveraging synthetic data in FGVC.
4.1 Experimental Setup
For generation, we employ BLIP-diffuse for SaSPA and Stable Diffusion v1.5 44 for all other diffusion-based augmentation methods.
For training, we follow the implementation strategy outlined in the CAL study tailored for FGVC. We ResNet50 [20] the primary architecture within the CAL framework unless specified otherwise. Each dataset fine-tuned using pre- trained ImageNet weights. More data generation and training details be found in Appendix D
Comparison Methods. We benchmark our method, SaSPA, against established traditional and generative data augmentation techniques. In the traditional category, comparisons include: CAL-Aug [44]: utilizes random flipping, cropping, and color-space variations. RandAugment [II]: applies series of random image transformations such as rotation, shearing, and color variations to training images. CutMix [66]: generates mixed samples by randomly cutting and pasting patches between training images to encourage the model learn more localized and discriminative features. Combined Methods: Tests the synergistic effects of CAL-Aug with CutMix and RandAug with CutMix. In the generative category, low translation strength (s = 0.15) to maintain high fidelity to the original images. ALIA [15] Uses real image captions and GPT-generated domain descriptions based these captions prompts for Img2Img translations. Detailed descriptions of these baseline methods are in Appendix D.3
compare with: Real-Guidance [21]: applies Img2lmg with
4.2 Fine-grained Visual Classification
Datasets. We evaluate on five FGVC datasets, using the full datasets for training. We use Aircraft [30, Stanford Cars (24). CUB (58), DTD [9, and CompCars [64). For datasets lacking predefined validation split, we establish one. For CompCars, we utilize the exterior car parts split, focusing
exclusively classifying images of car components: head light, tail light, fog light, and front into the correct car type. Further details on the exact splits are provided in Appendix C
We present the test accuracy of various augmentation methods in Table I For each dataset, the most effective traditional augmentation method (marked by an underline) identified using its validation set and consistently combined with all generative approaches to optimize performance for that dataset. This approach grounded findings that standalone generative methods generally perform better when integrated with traditional augmentations [60), trend also evident in Table
Results.

---

Aircraft
Cars
DTD
Accuracy (%6
Ancuracy (%0
80
Aocuracy (%G
55
80
70-
50
50
40
50
45
12 Number Shots
8
16
8
12
16
4
12 Number Shots
8
16
Number Shots
Best Trad Aug
Real Guidance
ALIA
SaSPA
Figure 4: Figure 4: Few-shot test accuracy across three FGVC datasets: Aireraft, Cars, and DTD, using different augmentation methods. The number of few-shots tested includes 4, 8, 12, and 16. We can see that for all datasets and shots, SaSPA outperforms all other augmentation methods.
Our key findings: (1) SaSPA achieves consistent improvements across all datasets, with or without BLIP-diffusion integration, and it consistently outperforms traditional and generative augmentation methods by significant margin. (2) The benefits of BLIP-diffusion vary depending on dataset characteristics; while it improves performance in datasets where texture and style play a crucial role differentiation, such as DTD, CUB, and CompCars, is not optimal for the Aircraft dataset, and has no significant impact on the Cars dataset, where structural features are more important for classification. We attribute this the fact that using diff-uiifusion confines the augmentations be similar to other subjects within the same sub-class, which can limit diversity. (3) Both generative baselines fail to achieve consistent improvements and sometimes even reduce performance.
4.3 Few-shot Learning
Experimental Setting. This section investigates the efficacy of various augmentation strategies in few-shot fine-grained classification scenarios, focusing on how synthetic data affects performance with increasing numbers of training examples ("shots")). We conduct evaluations using three datasets:
Aircraft [30], Cars [24]. and DTD [9, assessing performance at 4, 8, 12, and 16 shots.
The training and data generation approaches remain consistent with those described in Appendix D with two modifications: we use 100 epochs (down from 140), and we do not employ predictive confidence filtering for shots 4 and 8. The latter adjustment is due to the reduced reliability of the model's 's predictions, resulting from the limited training data. Additionally, we increase the augmentation ratio to 0.6, identified to be better for scenarios with limited data in Table
Results. The results in Figure 4 show that SaSPA consistently outperforms all other augmentation methods across all datasets and various shot counts. As seen in other works [54] 21. the benefit of augmentation diminishes as the number of shots increases, a trend most noticeable in the Cars dataset. Contrary to prior work, the gains provided by SaSPA remain substantial even at higher shot counts; notably, in the Cars dataset at 16 shots, SaSPA achieves an accuracy of 91.0%, surpassing the second-best performance of 88.3% by RG (21] Interestingly, SaSPA sometimes matches or exceeds the performance enhancement achieved by increasing the dataset size. For example, in the DTD dataset, utilizing SaSPA with 8 shots results in accuracy of 54.8%. slightly surpassing the 54.6% obtained when adding 50% more real data (a total of 12 shots) when relying solely on real data and the best traditional augmentation.
4.4 Mitigating Contextual Bias (Airbus vs. Boeing)
Experimental Setup. To evaluate the effectiveness of our method in mitigating real-world contextual biases, we use the contextual bias split of the Aircraft [30] dataset constructed by Dunlap et al. (I5). The split uses two visually similar classes: Boeing- 767 and Airbus-322, Each image in this split is categorized as "sky", "grass", or "road" depending on its background, with ambiguous examples filtered out. The bias in the dataset is introduced by training on 400 samples where Airbus aircraft are exclusively associated with road backgrounds and Boeing aircraft with grass backgrounds, although both types may appear against sky backgrounds. The exact split breakdowns are in Table 19
7

---

Table 2: Classification performance on the contextually biased Aircraft dataset [30], detailing overall, in-domain (ID) and out- domain (OOD) accuracies for each augmentation method.
Augmentation Method
Acc.
ID Acc.
OOD Acc.
Best Trad Aug (CAL-Aug)
71.0
88.2
10.2
Real Guidance 21
71.7
86.9
17.7
6S SaSPA w/o BLIP-diffusion
AL.A
71.8
84.9
25.1
73.0
81.9
41.5
Table 3: Comparison to concurrent work diff-mix [60. Test accuracy on FGVC datasets. indicates values taken from the diff-mix paper. 71 Textual Inversion, DB DreamBooth, X- No fine fone-tuning
ResNet50@ 448
ViT-B/16@ 384
Aug. Method
FT Strategy
Aircraft
Car
CUB
Aircraft
Car
CUB
CutMix
89.44
94.73
87.23
83.50
94.83
90.52
Diff-Mix i
TI+DB
90.25
95.12
87.16
84.33
95.09
90.05
Diff-Mix + CutMixt
TI+DB
90.01
95.21
87.56
85.12
95.26
90.35
SaSPA (Ours)
90.59
95.29
86.92
85.48
95.12
89.70
SaSPA (Ours) + CutMix
X
90.79
95.34
87.14
85.72
95.37
89.92
We follow the same training and generation implementation settings as for the FGVC setting (Ap- pendix DD) and we compare against the same generative methods. We also compare against the optimal Iraditional augmentation for the Aircraft dataset (CAL-Aug), as defined in Section 4.2
Results. The results in Table 2 show that SaSPA outperforms all other methods in overall and out-of- domain (OOD) accuracy, demonstrating its effectivenees in mitigating contextual bias. However, it falls short in-domain (ID) accuracy. A distinct inverse relationship is observed between ID and OOD accuracy: methods that induce more significant changes from the original image-Such as ALIA, which uses stronger translations than Real-Guidance (RG)-tend tend to achieve higher OOD accuracy but lower ID accuracy. This trend suggests that greater modifications can help reduce over-fitting to in-domain characteristics, enhancing the model's ability to generalize effectively to new, unseen conditions. As depicted in Figure [ even a higher translation strength (s = 0.5/0.75 yields limited diversity compared to our method. Consequently, the alterations produced by RG and ALIA are insufficient to significantly mitigate the contextual bias present in the dataset, as effective background variation crucial for addressing such biases.
4.5 Comparing SaSPA with Concurrent Work diff-mix
In this section, we compare our method with diff-mix, generative augmentation approach proposed concurrently by Wang et al. (60). diff-mix was also evaluated on full FGVC datasets and demonstrated impressive results. This method enriches datasets through image translations between classes, utilizing personalization techniques such textual inversion [18] and DreamBooth 477 to fine-tune the generative model for each sub-class. This fin-ttning enhances the model's 's ability to capture and represent class-specific nuances. In contrast, method does not involve fine-tuning, aiming to simplify the process and minimize computational costs.
Experimental Setup. In this analysis, we evaluate the performance of our SaSPA augmentation method using the diff-mix training setup, as detailed in their work. By using their open-source implementation, we further assess the robustness of our method with different training setup, To ensure fairness, We use the same number of augmentations (M) as diff-mix did. More details regarding training setup are in Appendix B.3
Results. Our results, detailed in Table highlight where SaSPA performs well and identify areas for potential improvement. The findings can be summarized as follows: (1) While diff-mix employs computationally intensive fine-tuning techniques to enhance class representation, we prioritize simplicity and lower computational demands in our approach. Despite this, SaSPA consistently outperforms diff-mix on both the Aircraft and Cars datasets across all architectures, whether combined with CutMix or used alone, demonstrating its robustness across various augmentation contexts. This

---

Table 4: Ablation Study: Effects of different generation strategies on various FGVC Datasets. 'Subj. means subject representation is used. "Edges=Subj." indicates that the real image used to extract the edges the same as the subject reference image. 'Art.' indicates that half the prompts are appended with artistic styles. For each dataset, bold indicates the highest validation accuracy, and underline indicates the second highest. Ticks under each column mean the component is used.
Method
Edge Guidance
Img21mg
Subj.
Inputs
Art.
Aircraft
Cars
CUB
DTD
Best trad aug
84.3
92.7
81.4
67.9
Ours
83.3
92.9
82.1
67.8
83.0
92.8
80.7
66.0
81.5
91.6
81.1
68.1
85.7
93.4
81.8
68.4
86.2
93.8
81.6
68.6
84.9
93.0
81.3
67.8
Euges=Subj.
85.2
93.1
81.3
68.7
Euges
85.5
93.7
82.6
69.2
Edges//Subj.
85.4
93.9
83.0
69.9
also shows that conditioning the generation on more abstract representations, as we do for correct class representation, can overcome the absence of extensive fine-tuning. (2) The CUB dataset posed unique challenges, with diff-mix outperforming SaSPA using ResNet50 and both diff-mix mix and SaSPA under-pen aader-performing relative CutMix using ViT-B/16. Notably, despite SaSPA outperforming all methods, including CutMix in Table [ it does not perform as well here. We hypothesize that the use of higher resolution emphasizes finer details in each class, which may overwhelming for SaSPA on some datasets but less so for diff-mix, likely due to the heavy fine-tuning process integrated into their method. These results indicate that some form of fine-tuning might advantageous for complex datasets like CUB to achieve better performance. The full table, including comparisons to more augmentation methods, can be found in Appendix
4.6 Effect of Different Generation Strategies on Performance
In Table we conduct an extensive ablation study to evaluate the effectiveness of our proposed generation strategies. Specifically, we examine the integration of edge guidance, Img2Img as an alternative for edge guidance with strength = 0.5 and in combination with edge guidance with strength 0.85, and subject representation. We also investigate the effect of using the same image for both edges extraction and the subject reference image ("Edges=Ref."), Additionally, we test the impact of appending half of the prompts with artistic styles (column 'Art.' as described Appendix B.8
The results demonstrate the importance of combining structural and subject-level conditioning while enabling diverse generations through separate input sources, yielding the best performance across most datasets. Key observations include: (1) Edge Guidance alone improves performance significantly compared to Text-to-Image or SDEdit 32 (Img21mg), highlighting its important role in providing structural guidance. (2) Subject representation alone does not enhance performance, indicating additional structural conditioning is necessary. (3) Using different source images for edges and subject reference images adds beneficial diversity. (4) Surprisingly, text-to-image generation (first row) outperforms SDEdit, likely due its increased diversity and despite the lower fidelity, which our filtering mechanism can handle as it filters out low- ow-fiielity images. (5) Incorporating artistic prompts has inconsistent effects, usually boosting performance with Edge Guidance but often degrading when combined with subject representation. This inconsistency may stem from the fact that subject representation uses -ifffuifo (27), which is different base model than Stable Diffusion [46), as Stable Diffusion is fine-tuned. Additionally, in CUB, artistic prompts offer no improvement even when using Edge Guidance without subject representation, likely due to the dataset's heavy reliance on color primary discriminator between bird types, potentially disrupted by artistic prompts.

---

Limitations and Future Directions
Limitations. Although we demonstrated that SaSPA could generate images with high class fidelity through conditions such as edge maps and subject representation, it still remains dependent on the underlying generation models. For instance, we found that applying SaSPA the CUB dataset at higher resolution does not improve performance. Additionally, SaSPA relies on large language models (LLMs) to generate relevant and diverse prompts given the meta-class. While this usually effective, it may not produce optimal prompts if the LLM lacks knowledge of the meta-class.
Future Directions. Several avenues exist enhance the flexibility and performance of method in future research. Firstly, we hope our work inspires the use of additional methods to condition the synthesis process beyond using real images, as have shown be effective. Another promising avenue is to apply SaSPA to additional tasks such as classification of common objects, object detection, and semantic segmentation. Additionally, maintaining temporal consistency in settings that use consecutive frames, such as autonomous driving, remains significant challenge. Addressing this issue could expand the applicability of SaSPA broader range of use cases. Moreover, ongoing advancements in generative models are likely to bring further improvements to our pipeline. Finally, effectively generating and using synthetic data remains an active research area, and identifying optimal strategies for both the generation process and the training integration remains important future direction.
Conclusion
We propose SaSPA, generative augmentation method specifically designed for FGVC. Our method generates diverse, class-consistent synthetic images through conditioning on edge maps and subject representation. SaSPA consistently outperforms both traditional and recent generative data augmen- tation methods. It demonstrates superior performance across multiple settings, including multiple setups of the challenging and less-explored full dataset training, as well in scenarios of contextual bias and few-shot classification. Limitations and future directions are discussed in Section 5
7 Acknowledgments
This work was supported in part by the Israel Science Foundation (grant No. 1574/21).
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
URL htt.appss//opeeeeeeee.eeee//////
DIRsoxjyPm
[3] Hritik Bansal and Aditya Grover. Leaving reality to imagination: Robust classification via generated datasets. In International Conference on Learning Representations (ICLR), 2023.
[4] Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: training models from generated images. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2020.
[5] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[6] John Canny. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, (AM1-866))679-6988 1986.
[7] Kai Chen, Enze Xie, Zhe Chen, Lanqing Hong. Zhenguo Li, and Dit-Yan Yeung. Integrating geometric control into text-to-image diffusion models for high-quality detection data generation via text prompt. arXiv preprint arXiv:2306.04000 2023.
[8) Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19830-19843, October 2023.
[9] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014.
[10] Shorten Connor and Taghi Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[11] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition workshops, pages 702-703, 2020.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pullern recognition, pages 248-255. leee, 2009.
[13] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL https://openrevies: wwwneeeffouuu www.mmiddddiiiibidddiiemma
[15] Lisa Dunlap. Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph Gonzalez, and Trevor Dar- rell. Diversify your vision datasets with automatic diffusion-based augmentation. In Proceedings of the 37th Conference Neurul Information Processing Systems (NeurIPS 2023), 2023. URL
filersffies///eeeeee /iles//apeer/223/ http://90rp222a//aaccc1771111111 Cot.raact-Cooffeencece.om html

---

[16] Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now, arXiv preprint arXiv:2312.04567, 2023.
[17] Sharon Fogel, Hadar Averbuch-Elue Sarel Cohen, Shai Mazor, and Roee Litman. Scrabblegan: Semi-supervised varying length handwritten text generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4324-4333, 2020.
[18] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference Learning Representations, 2023.
URL httpp::/ nwt/ http:/orrrmn/////////////OCOOT./
[19] Yunhao Ge, Jiashu Xu, Brian Nlong Zhao, Neel Joshi. Laurent Itti, and Vibhav Vineet. Beyond generation: Harnessing text image models for object detection and segmentation. arXiv preprint arXiv:2309.05956, 2023.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[21] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023. URL /ttpp:///openrevie httppp/nreeeeee.eeeoorrrrrrr/ree het//forum/idd-nnnnZZZRBa
[22] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time- scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 6885140-681,, 2020.
[24] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE International Conference Computer Vision Workshops, pages 554-561, 2013.
[25] Shiye Lei, Hao Chen, Sen Zhang, Bo Zhao, and Dacheng Tao. Image captions are natural prompts for text-to-image models. arXiv preprint vrXiv:2307.08526, 2023.
[26] Bo Li, Haotian Liu, Liangyu Chen, Yong Jae Lee, Chunyuan Li, and Ziwei Liu. Benchmarking and analyzing generative data for visual recognition. arXiv preprint arXiv:2307.13697, 2023.
[27] Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems, 36, 2024.
(28) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[29] Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. arXiv preprint arXiv:
2019.
[30] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew B Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv: 13X.vv3306.5/5// 2013.
[31] David Marwood, Shumeet Baluja, and Yair Alon. Diversity and diffusion: Observations on synthetic image distributions with stable diffusion. arXiv preprint arXiv:231..000 2023.
[32] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wa, Jun- Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
12

---

[33] Francisco J Moreno-Baree José M Jerez, and Leonardo Franco. Improving classification accuracy using data augmentation on small data sets, Expert Systems with Applications, 161: 1 113696, 2020.
[34] Francisco J Moreno-Baree José M Jerez, and Leonardo Franco. Improving classification accuracy using data augmentation on small data sets, Expert Systems with Applications, 161: 1 113696, 2020.
[35] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296 4304, 2024.
[36] Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion- synthetic data generation for pixel-level semantic segmentation. Advances in Neural Information Processing Systems, 36, 2024.
[37] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ily Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1074. 2021.
[38] Farzan Erlik Nowruzi, Prince Kapoor, Dhanvin Kolhatkar, Fahed Al Hassanat, Robert Laganiere, and Julien Rebut. How much real data do we actually need: Analyzing object detection performance using synthetic and real data. arXiv preprint urXiiv/907.0000
2019.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
[40] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high nigh-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
URL htttps://operrewe/ www.sn.e/ffeee.... www.:neennrrrn//. iatrrummmddddi522R8xxxx
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[42] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67 2020.
[43] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2004001125, 1(2):3, 2022.
[44] Yongming Rao. Guangyi Chen, Jiwen Lu, and Jie Zhou. Counterfactual attention learning for fine-gras graingrinedd visual categorization and re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1025-1034, 2021.
[45] Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. Advances in neural information processing systems, 32, 2019.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[47] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-225 10, 2023.
13

---

[48] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494 2022.
[49] Vignesh Sampath, Iñaki Maurtua, Juan Jose Aguilar Martin, and Aitor Gutierrez. A survey on generative adversarial networks for imbalance problems in computer vision tasks. Journal of big Data, 8:1-59, 2021.
[50] Mert Bülent Samyyiliz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake till you make it: Learning transferable representations from synthetic imagenet clones. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8011-8021, 2023.
[51] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:23/1.17042 2023.
[52] Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes. Diversity is definitely needed: Improving model-agnostic zero-shot classification via stable diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 769-778, 2023.
[53] Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL htt.fformmmidSSttiiaaaa
[54] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024. URL https: http:::://ppeervveeee www. net/forum/id=2302 07944
[55] Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 969-979997 969-977, 2018.
[56] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 109-117, 2017.
[57] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Yehao Li, Mishig Davaakhuu, Aedan S. Culotta, and Camilo Rodrigues. Diffusers: State-of-the-art diffusion models. https:/ hts://iitae tithub. www..o/gggggnacee/eooeoooeooeeee 05-10.
2022. Accessed: 2023-
[58] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
[59] Yibo Wang, Ruiyuan Gao, Kai Chen, Kaiqiang Zhou, Yingjie Cai, Lanqing Hong, Zhenguo Li, Lihui Jiang, Dit- Yan Yeung, Qiang Xu, et al. Detdiffusion: Synergizing generative and percep- tive models for enhanced data generation and perception. arXiv preprint 2403:2303.33. 2024.
[60] Zhicai Wang, Longhui Wei, Tan Wang, Heyu Chen, Yanbin Hao, Xiang Wang, Xiangnan He, and Qi Tian. Enhance image classification via inter-class image mixup with diffusion model. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL https::///pppp thecvf http://tuuaa//2//pspseeee
[61] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1206-1217. 2023.
14

---

[62] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[63] Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision, pages
2015.
[64] Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine- grained categorization and verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3973-3981, 2015.
[65] Zhuoran Yu, Chenchen Zhu, Sean Culatana, Raghuraman Krishnamoorthi, Fanyi Xiao, and Yong Jae Lee. Diversify, don't 't fine-tune: Scaling up visual recognition training with synthetic images. arXiv preprint arXiv:2312.02255. 2023.
[66] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pages 6023-6032, 2019.
[67] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.0942,, 2017.
[68] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[69] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[70] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145-10155, 2021.
15

---

Supplementary Material
Table of Contents
A Broader Impact
16
B
More Experiments
16
B.1
Effect of Augmentation Ratio on Performance]
B.2
Effect of Augmentation Ratio Performance Different Amounts Real Data
B.3
Comparing SaSPA with More Augmentation Methods
17
B.4
Effect of Traditional Augmentations with SaSPA
18
B.5
Performance Across Network Architectures
19
B.6
Effect of Scaling the Number of Augmentations (M)
................................
19
Extended Evaluation Additional Datasets)
................................
20
B.8
Evaluating Different Prompt Strategies
20
B..9
Will More Prompts Improve Performance?
20
10
Assessing the Relevance of FID in Generative Data Augmentation
21
8.
Evaluating Augmentation Diversity with LPIPS)
21
Investigating Potential Newer Base Models
22
B. 13 Does Stopping Augmentation Early Epochs Help
22
B.
14
Performance at Higher Resolutions
23
B.15 Choice of Conditioning Type
23
Dataset details
23
D
Implementation details
23
D.1
Experimental Setup
23
D.2
Compute Requirements
24
D.3
More details on Generative Baselines
25
E
More Methodology details
25
E.1
Prompt Generation via GPT-4
25
E.2
Filtering Strategies
26
F
More Visualizations
26
F.1
Qualitative Comparison with Generative Augmentation Methods
26
2
Contidence Filtering Visualization
26
A Broader Impact
Generative data augmentation can benefit many fields by creating more robust models while protecting privacy. By reducing the reliance on real data, it addresses privacy concerns and lowers the costs and time needed for data collection and annotation, thereby enhancing the accessibility of advanced machine learning techniques. It is also important to note that synthetic data can inherit biases from the generative models, potentially leading to biased training outcomes.
B
More Experiments
B.1 Effeet of Augmentation Ratio on Performance
In Figure 5 we evaluate various augmentation ratios (the probability of real image being replaced by synthetic one in given mini-batch). We find that for most datasets, excluding CUB, the optimal
16

---

Aircraft
Cars
DTD
CUB
Accuracy (%)
0.1
0.2
0.3
0.4
0.5
0.6
0.75
Aug Ratio (a)
Figure 5: Line plots of Augmentation Ratio (a) validation accuracy for Aircraft, Cars, DTD, and CUB datasets.
Table 5: Effect of amount of real data used (as fraction of the complete dataset) and a values validation accuracy when augmenting with SaSPA
Aircraft
Cars
CUB
Real Data Frac.
Best
SaSPA
SaSPA
Best
SaSPA
SaSPA
Best Trad Aug
SaSPA
SaSPA
Trad Aug
(a)
(Arigh)
Trad Aug
(a)
(Ahigh)
(a)
(Atiggh
0.1
26.9
40.8
41.0
29.3
50.5
51.6
32.7
38.4
41.4
0.3
59.7
69.8
70.0
70.8
83.9
84.3
61.3
66.2
68.3
0.5
73.5
78.7
77.9
84.7
89.3
89.5
72.0
74.8
76.0
0.75
80.6
82.9
82.5
90.7
92.6
92.3
77.6
80.3
80.7
1.0
84.3
85.4
84.0
92.7
93.9
93.6
81.4
83.0
82.0
range for lies between 0.2 and 0.5, with marginal differences within this range. Consequently, we selected 0.4 as the default augmentation ratio. However, for the CUB dataset, the default choice a 0.1. Considering the relatively lower improvement on CUB in Table and the underperformance the diff-mix benchmark (Table 3), both of which are likely attributed to lower class-fidell class-fidelity, seems that lower class fidelity necessitates a lower augmentation ratio. A possible explanation is that higher augmentation ratios more likely to introduce bias during training when class fidelity lower.
B.2 Effect of Augmentation Ratio on Performance with Different Amounts of Real Data
In Table 5 examine the interaction between and the percentage of real data used. We use two a values for each dataset: the default value used throughout the paper and a higher value (Ohigh 0.22). We observe that for all amounts of real data, SaSPA achieves notable improvements with diminishing returns, similar to the trends observed Section 4.3 An interesting pattern emerges: the amount of real data decreases, the optimal value tends to increase. This trend is consistent across all datasets. For instance, in the Cars dataset, when all real data is used (Frac. 1.0), a = 0.6 performs worse than 50%), using = 0.6 yields better performance. This pattern is similarly observed in the Aircraft and CUB datasets, indicating that higher values of a are more beneficial when the amount real data limited.
0.4. However, for smaller percentages of real data (e.g., 10%. 30% or
B.3 Comparing SaSPA with More Augmentation Methods
diff-mix (60] compared method more augmentation techniques. this section, we present the complete results, including comparisons to those other methods.
17

---

Table 6: Comparison to concurrent work diff-mix [60]. Test aceuracy on different datasets. in- dicates values taken from the diff-mix paper. TI- Textual Inversion, DB DreamBooth, X- No fine-tuning.
ResNeL500448
VIT-B/16@388
Aug. Method
FT Strategy
Aircraft
Car
CUB
Aircraft
Car
CUB
89.09
94.54
86.64
83,50
94.21
89.37
CutMix Mixup
89.44
94.73
87.23
83.50
94.83
90.52
89.41
94.49
86.68
84.31
94.98
90.32
Real-filtering f Real-guidance
X
88.54
94.59
85.60
83.07
94.66
89.49
X
89.07
94.55
86.71
83.17
94.65
89.54
DA-fusion f
TT
87.64
94.69
86.30
81.88
94.53
89.40
Diff-Mix ?
TI+DB
90.25
95.12
87.16
84.33
95.09
90.05
Diff-Mix CutMixt
TI+DB
90.01
95.21
87.56
85.12
95.26
90.35
SaSPA (Ours)
X
90.59
95.29
86.92
85,48
95.12
89.70
SaSPA (Ours) + CutMix
X
90.79
95.34
87.14
85.72
95.37
89.92
Table 7: Test performance of SaSPA combined with different traditional data augmentation methods.
Type
Aireraft Augmentation Method
CompCars
Cars
CUB
DTD
No Aug
81.4
67.0
91.8
81.5
68.5
Traditional
CAL-Aug
84.9
70.5
92.4
82.5
69.7
Best Trad Aug
72.6
92.7
SaSPA w/o Trad Aug
84.1
74.1
92.8
81.7
69.7
Ours
SaSPA w/ CAL-Aug
86.6
75.8
93.8
83.2
71.9
SaSPA w/ Best Aug
76.2
93.8
Experimental Setup. As noted in Section 4.5 we use diff-mix training setup. This setup employs ResNet50 [20) with resolution of 448 and ViT-B/16 [14] with resolution of 3842, both which are higher than the 224 resolution we use across the paper. We incorporate the integration of ControlNet and BLIP-diffusion for Cars and CUB datasets. We do not use BLIP-diffusion for the Aircraft dataset as it proved to be a better option, as evidenced in Table 4 The accuracies of diff-mix and other methods, as reported in Table of the diff-mix paper [60, establish the benchmarks for our comparative analysis.
Comparison Methods. The compared methods, implemented by diff-mix, include (1) Real-Filtering (RF) and (2) Real-Guidance (RG), both proposed by He et al. [21]. RG is described in Appendix [D.3] which they implement with lower translation strength (s 0.1). RF is variation of Real-Guidance
that generates images from scratch and filters out low-quality images by using CLIP 41 features from real samples exclude synthetic images that resemble those from other classes. (3) DA-Fusion (54)
solely fine-tunes the identifier using textual inversion (18) to personalize each sub-class and employs randomized strength strategy (s (0.25, 0.5, 0.75, 1.0)), and non-generative augmentation methods (4) CutMix [66] and (5) Mixup 67].
Results. Results show that SaSPA outperforms all methods across both architectures when evaluated on Aircraft and Cars, despite diff-mix using heavy fine-tuning. Continuing the discussion on CUB evaluation in Section 4.5 CutMix outperforms all methods when using the ViT-B/16 architecture, while diff-mix leads on ResNet50. Notably, SaSPA outperforms all other generative baselines on CUB except diff-mix using both architectures.
B.4 Effect of Traditional Augmentations with SaSPA
Across our experiments, we combined SaSPA with the best traditional augmentation method, as described in Section 4.1 To test how SaSPA behaves without augmentation and whether it depends on the best traditional augmentation, we evaluated its interaction with CAL-Aug 44 (the default traditional augmentation used in CAL), the best traditional augmentation, and no traditional augmen- tation at all. Note that for 3 out of 5 datasets, CAL-Aug is the best traditional augmentation,
18

---

Table 8: Results on the test set of three FGVC datasets for VIT and ResNet101 architectures
Aug Method
ViT
ResNet101
Aircraft
Cars
DTD
Aircraft
Cars
DTD
Best Trad Aug
82.3
91.2
74.9
85.5
93.2
69.3
Real Guidance
82.2
90,9
75.4
85.1
93.0
70.3
ALJAA
82.0
91.0
75.1
85.0
92.9
69.4
SaSPA
86.1
91.5
76.3
87.1
94.2
72.0
95
Aircraft
Cars
Accuracy (%)
85
75
70
0
Number of Augmentations (M)
Figure 6: Effect of the number of SaSPA augmentations (M) on validation accuracy for Aircraft and Cars datasets. Horizontal lines represent the use 75% real data without SaSPA augmentations.
not provide separate results for SaSPA with the best traditional augmentation for these datasets (as they are the same). From the results in Table 7, we observe that using no traditional augmentation significantly mnder-performs compared to using CAL-Aug or the best traditional augmentation. Addi- tionally, CAL-Aug proved to be robust choice, yielding similar accuracy across all datasets, with only a slight decrease in performance for CompCars.
B.5
Performance Across Network Architectures
Our primary experiments utilized ResNet50 as the backbone architecture for CAL. To further evaluate how deeper backbones other network architectures might benefit from augmentation method, we analyzed results across three FGVC datasets as detailed in Table g The performance of both ViT 14] and CAL with ResNet101 a backbone is presented. Results indicate that both deeper networks, such as ResNet101, and the VIT architecture benefit from our augmentation method. Together with comparison with diff-mix (60) using their training setup, this demonstrates that method is robust across a variety of architectures and training setups.
B.6 Effect of Scaling the Number of Augmentations (M)
In this study, we start with a base training set, utilizing 50% of the available real data. We examine the impact of varying the number of SaSPA augmentations, M, from to 5 on the validation accuracy for Aircraft and Cars datasets. Additionally, we compare it to the effect of increasing the dataset size by adding additional 25% of the real data without SaSPA augmentations.
Results presented in [demonssemoostraall a consistent increase in validation accuracy for both datasets the number of augmentations increases. Notably, the Cars dataset shows robust performance
19

---

Table Additional datasets. We report test accuracy on two additional FGVC datasets: Stanford
Dogs and The Oxford- IIIT Pet Dataset. The highest values for each dataset are shown in bold.
Augmentation Method
Pet
Dogs
CAL-Aug
92.9
83.9
Real Guidance
92.9
83.5
ALIA
92.7
82.8
SaSPA
93.6
84.3
improvements, even surpassing the results achieved by adding 25% of real data when M 4, indicating the effectiveness of the SaSPA augmentations in this context. For the Aircraft dataset, the accuracy nearly reaches the levels achieved by adding 25% more real data when M ==5, 5.
B.7 Extended Evaluation on Additional Datasets
In response to reviewer feedback, we expanded our evaluation to include two additional FGVC datasets: Stanford Dogs and the Oxford-IIIT Pet Dataset, further assess the robustness of proposed method, SaSPA. Stanford Dogs comprises 20,580 images from 120 dog breeds, while the Oxford-IIIT Pet Dataset includes 7,349 images from 37 breeds (25 dog and 12 cat breeds).
We compared with CAL-Aug, Real Guidance, and ALIA. Results, presented in Table 2 indicate that SaSPA improves performance on both datasets, thereby strengthening the findings regarding its efficacy as a generative augmentation method. Combined with earlier DTD and CUB datasets results, this evaluation confirms that SaSPA effectively handles non-rigid objects.
B.8 Evaluating Different Prompt Strategies
Table 10: Comparison of prompt strategies across two FGVC datasets. The highest values are highlighted in bold, while the second highest underlined.
Prompt Strategy
Aircraft
Cars
Captions
76.8
87.4
LE
78.3
87.9
ALIA (GPT)
78.2
88.1
Ours (GPT)
78.3
88.7
Ours (GPT) Art
78.6
88.9
To assess the effectiveness of our proposed prompt generation, evaluated various prompt strategies, and the results are detailed in Table TT] To accelerate the experimentation process, these experiments were conducted using only ControlNet with SD XL Turbo on 50% of the data. Five main strategies were compared: (1) Captions: Direct use of captions prompts, leveraging BLIP-2 2 [28] for captioning, as demonstrated be effective prior work (25] (2) LE (Language Enhancement)
[21] and (3) ALIA [15] are described in Appendix D3 (4-5) Our Method with and without appending artistic styles. The artistic style augmentation involves appending half of the prompts with the phrases ", a painting of <artist>", where <artist> refers to renowned artists such as van Gogh, Monet, Picasso. This approach aims diversify textures and colors, prompting an increase in the model's 's robustness.
The results show that prompt generation method, either with without incorporating artistic styles, consistently outperforms other approaches. Caption-based prompts yield the least effective performance, while the ALIA and LE methods fall somewhere in between.
B.9
Will More Prompts Improve Performance?
To evaluate whether increasing the number of prompts would enhance our method's performance, we compared generating 200 prompts generating 100 prompts using our method.
20

---

Table 11: Validation accuracy on the Aircraft dataset using 100 and 200 prompts generated by our method.
Prompt Count
Accuracy
100
78.9
200
79.0
The results in Table 11 show no significant difference when using 200 prompts. indicating that 100 prompts are sufficient.
B.10 Assessing the Relevance of FID in Generative Data Augmentation
A common metric evaluate the quality of generative model is the Fréchet Inception Distance (FID) [22], metric that measures the similarity between the distribution of generated images and real images. However, does it accurately measure how effective augmentation method is?
Table 12: Combined FID and accuracy results for various generative augmentation methods across four FGVC datasets.
Aircraft
CompCars
Cars
CUB
Aug Method
FID
Acc.
FID
Acc.
FID
Acc.
FID
Acc.
Real Guidance
3.39
84.8
8.74
73.1
9.08
92.9
5.93
82.8
ALIA SaSPA
4.59
83.1
13.96
72.9
9.68
92.6
10.88
82.0
7.89
86.6
21.22
76.2
18.21
93.8
13.44
83.2
In Table 12 report the FID values, calculated using augmentations alongside their respective real datasets, as well the corresponding accuracy achieved with each augmentation method. We observe that generative baselines such as Real Guidance and ALIA achieve lower FID scores, which suggest higher similarity to the real data distribution. We suspect that this is the result of generating images that closely mimic the original dataset. In contrast, our method, SaSPA, designed to create diverse augmentations that substantially differ from the real images, leading to higher FID scores. Despite these higher FID values, as shown in Table 12] SaSPA demonstrates superior performance enhancements in accuracy across datasets. This highlights the importance of evaluating generative augmentation methods not only based on realism and similarity to real images, as measured by FID, but primarily their actual impact model performance. In the next section, we further provide an alternative metric for generative data augmentation.
B.11
Evaluating Augmentation Diversity with LPIPS
measures the perceptual difference between two images. By calculating the average LPIPS distance between original images and their respective augmentations, we can quantify the diversity introduced by augmentation method. We argue that this metrie, combined with qualitative evidence of class fidelity, provides robust measure for evaluating generative data augmentation. Note that this metric will apply only for augmentations that are derived from real images. Generation from scratch will require a different metric, probably a dataset-level diversity metric.
LPIPS
69
Table 13: Combined diversity score and accuracy results for various generative augmentation methods across five FGVC datasets.
Aircraft
CompCars
Cars
CUB
DTD
Aug Method
Diversity Acc.
Diversity Acc.
Diversity Acc.
Diversity Acc. Diversity
Acc.
Real Guidance
0.11
84.8
0.10
73.1
0.10
92.9
0.15
82.8
0.18
68.5
ALIA
0.24
83.1
0.29
72.9
0.26
92.6
0.33
82.0
0.37
69.1
SaSPA
0.55
86.6
0.53
76.2
0.57
93.8
0.66
83.2
0.58
71.9
21

---

Table 13 demonstrates that SaSPA achieves significantly higher LPIPS scores compared to Real Guidance (RG) and ALIA, indicating that SaSPA introduces much greater diversity in the generated augmentations. This substantial increase in diversity is crucial for enhancing model robustness and performance 31 Qualitative evidence can be found Figure
B.12 Investigating the Potential of Newer Base Models
Table 14: Validation accuracy of our method with different base models. Generations do not include BLIP-diffusion.
Base Model
Edge Guidance
Aireraft
CompCars
Cars
CUB
Best Trad Aug
84.3
62.5
92.7
81.4
83.3
63.1
92.8
82.1
SD v1.5
86.2
64.7
93.8
81.8
83.5
62.8
93.5
82.2
SD XL Turbo
86.4
65.0
93.7
82.3
83.8
62.7
93.4
82.6
SD XL
86.7
64.6
93.7
82.3
Recently, text-to-image diffusion models have made incredible progress, particularly those based on Stable Diffusion (SD) [46]. Notable advancements include SD XL [40] and SD XL Turbo [5T]. In this section, aim to explore the compatibility of Edge Guidance with other base models. Unfortunately, as these models are relatively new, BIIP-diffusion (27 has not yet released versions built upon them, preventing from utilizing subject representation. However, shown in Table 4 our full pipeline without subject representation still achieves impressive results. Additionally, Table 4 indicates that when subject representation is not used, it slightly beneficial for most datasets, except CUB, to append half the prompts with artistic styles. Therefore, we adopt this strategy. In Table [4] we experiment with SD v1.5, SD XL, and SD XL Turbo. Note that ControlNet versions for SD XL and SD XL Turbo are still experimental, and will require reevaluation as the models mature.
The results indicate that integrating Edge Guidance generally has positive impact across base models, except on the CUB dataset, which aligns with our carlier findings in Table Additionally, SD XL and SD XL. Turbo typically outperform SD v1.5, suggesting that more advanced base models may lead to further improvements in performance.
B.13 Does Stopping Augmentation at Early Epochs Help?
Table 15: Impact of stopping SaSPA augmentation at different training epochs on validation accuracy of the Aircraft 30 dataset.
Epoch Stop
Accuracy
0 (No Aug)
73.5
20
75.8
40
77.2
60
77.1
77.3
100
77.4
120
77.7
140 (Full Aug)
78.7
common technique when training with synthetic data is first train using the synthetic data, then fine-tune on the real data [56] 38 [7]. Inspired by this, we investigate stopping the data augmentation at earlier training epochs. Results are presented in Table 15 For these ablation experiments on evaluating early augmentation stoppage, we used 50% of the real Aircraft dataset for faster experimentation. We find benefit from early augmentation stoppage. There is downward trend in accuracy when stopping early, with the worst results at the earliest epoch stopped (20 out
22

---

of 140). We believe that the high diversity introduced by our augmentations reduces over-fitting, mitigating the need for an explicit domain adaptation strategy. As a result, the model continues to benefit from the augmented data throughout the training process.
B.14 Performance at Higher Resolutions
Table 16: Higher resolution results. Comparison of our method (SaSPA) with the best augmentation method per dataset. All results are using 448x448 resolution, and reported on the test set of each dataset.
Method
CompCars
DTD CUB
Best Aug
75.1
69.6 86.7
SaSPA
77.6
72.0 86.7
Additional experiments were conducted using 448x448 resolution on the CompCars, DTD, and CUB datasets, employing SaSPA and the best augmentation methods identified in Table [ The experiments, replicated with two different seeds, are detailed in Table 16
Combined with our earlier diff-mix comparisons at both 448x448 and 384x384 resolutions (Table 3 these results complete our high-resolution evaluation across all datasets. Notably, we observed consistent performance improvements in all datasets except CUB. We hypothesize that CUB's ' fine- grained details such feather patterns and colors present significant challenges higher resolutions, impacting the efficacy of generative methods. In conclusion, SaSPA demonstrates promising results for most datasets, affirming its overall benefits.
B.15 Choice of Conditioning Type
Table 17: Validation accuracy on the Aircraft dataset using different conditioning types Con.rolNet.
Condition Type
Accuracy
Canny Edges
78.9
HED Edges
78.6
Using Canny edge maps as a condition has proven effective for generating images with high diversity and class fidelity. Here, we experiment with different kind of edges: Holistically- Nested Edge Detection (HED) edges (63] Canny edges are more focused on detecting the intensity gradients of the image, often capturing finer details, whereas HED edges provide more structured representation by capturing object boundaries in a holistic manner. We experiment with both types in Table 17 using the default generation parameters without using BLIP-diffusion, and on 50% of the data faster experimentation. Using Canny edges resulted slightly higher validation accuracy on the Aircraft dataset.
C Dataset details
provide the number of samples for each dataset split used in our experiments in Table [8] Additionally, we include the number of images for each background class (sky, grass, road) used to create the contextually biased training set, as shown Table 19 We utilize the dataset test split for the reported test. For datasets lacking validation split (Cars, CUB, CompCars), we generate one by using 33% of the training set. Note that in the diff-mix training setup (Section 4.5). the training datasets for CUB and Cars consist of the original splits, they did not use separate validation split.
D Implementation details
D.1 Experimental Setup
Unless stated otherwise, the following experimental setup applies to all experiments in the paper.
23

---

Table 18: Dataset Split Sizes.
Dataset
Training
Validation
Testing
Aircrfft
3,334
3,333
3,333
CompCars
3,733
1,838
4,683
Cars
5,457
2,687
8,041
CUB
4,016
1,978
5,794
DTD
1,880
1,880
1,880
Airbus VS Boeing
409
358
707
Table 19: Dataset Statistics for Contextually Biased Planes
Airhus
Boeing
sky
grass
mad
sky
grass
med
Train
98
0
70
129
112
0
Val
21
21
137
44
Test
175
51
51
222
104
104
Data Generation. All generative methods use the Diffusers library [57 We employ BLIP- diffusion [27] and ControlNet [68] for SaSPA. Besides the prompt, an edge map for ControlNet, and reference image for Biff-diffusion as inputs for our generation, BLIP-diffusion requires source subject text and target subject text as inputs. We simply use the meia-class (e.g., "Airplane" for Aircraft dataset, "Bird" for CUB) of the dataset for both source and target subject texts. For other diffusion-based augmentation methods, we use Stable Diffusion v1.5 [46.. For all diffusion- based models, including SaSPA, we use the DDIM sampler (53] with 30 inference steps and guidance scale of Images resized to ensure the shortest side is 512 pixels before processing with Img2Img ControlNet. We set the ControlNet conditioning scale to 0.75. For texl-to-image generation, images generated resolution of 5 12x512.. We generate M = 2 augmentations per original image for each experiment, and we use augmentation ratio a = 0.4 for all datasets except CUB [58), for which we filtering. We use four NVIDIA GeForce RTX 3090 GPUs for image generation and training.
0.1 evidenced to be better in Figure We use k = 10 in the top-k Confidence
Training. We follow the implementation strategy outlined in the CAL study [4], tailored for FGVC. We use ResNet50 (20 as the primary architecture within the CAL framework unless specified otherwise. Each dataset is fine-tuned using pre-trained ImageNet [12) weights. Optimization is performed with an SGD optimizer, with a momentum of 0.9 and a weight decay of 10 over 140 epochs. We adjust the learning rate and batch size during hyper-parameter tuning to achieve the highest validation accuracy. Training images are resized to 224x224 pixels. Results are averaged across three seeds. Specific values of
are in Table 20
Specifies on DTD [9] dataset. The DTD dataset is collection of images categorized by various textures, such as Marbled, Waffled, and Banded. We found that this dataset differs from other fine-grained datasets as it not fine-grained the same level. Classes like "Marbled" and "Waffled" have significant differences from each other. Therefore, feeding the meia-class ("Texture")) to an LLM will provide prompts that are not suitable for all sub-classes in the dataset. Hence, we did not use our prompt generation method. This could be addressed in the future by feeding the LLM with each sub-class. Instead, we simply used image captions.
To select the hyper- parameters for each dataset, we train CAL with learning rates of [0.00001, 0.0001, 0.001, 0.01, 0.1] and batch size [4. 8, 16, 32]. selecting the configuration that results in the highest validation accuracy. These parameters, shown in Table 20 are then used across all methods.
Slyper-prraeete
D.2 Compute Requirements
In this section, we outline the computational resources required for our primary experiments. We utilize four NVIDIA GeForce RTX 3090 GPUs for image generation and training purposes, but we report running times for single GPU. Training with ResNet50 necessitates up to 5.5 GB of GPU RAM. The duration of our experiments varies depending on the dataset, with the longest running being approximately three hours. As no fine-tuning is performed, our generation process includes
24

---

Table 20: Hyperparameters
Dataset
Learning Rate
Batch Size
Weight Decay
Epochs
Optimizer
Momentum
Aircraft 80
0.001
4
10
140
SGD
0.9
CompCars 64
0.001
8
10
140
SGD
0,9
Cars
0,001
8
10
140
SGD
0.9
CUB
58
0.001
10
140
SGD
0.9
DTD
0,001
16
10ad
140
SGD
0.9
Airbus vs. Boeing
0.001
4
10
140
SGD
0.9
only I/O and forward pass through the generation model. We report here only the generation times and do not include I/O times, as these can vary heavily based on system configuration and server load. For image augmentation using ControlNet with BLIP-diffusion the base model, generating each image takes 2.96 seconds and requires to 10 GB of GPU memory. Therefore, creating two augmentations for the Aircraft dataset's training set would take approximately five and half hours. When switching to SD XL Turbo as the base model, with two inference steps (the default for this base model), the augmentation time is reduced to 0.52 seconds, and the GPU memory requirement increases to to 16 GB. In this configuration, generating two augmentations for the Aircraft dataset's training set would take less than one hour.
D.3 More details on Generative Baselines
In this section, we provide additional details on the generative baselines compared against.
Real-Guidance (RG): This method achieves impressive few-shot classification performance. For prompt generation, an off-the-shelf word-to-sentence T5 model, pre-trained on the "Colossal Clean Crawled Corpus" " [2] and fine-tuned on the CommonGen dataset [29]. is utilized to diversify language prompts. The model is used in order to generate a total of 200 prompts based on the meta-class. For image generation, SDEdit with a low translation strength (s 0.15) is used. Filtering performed using CLIP filtering, which is described in Section 3.4
This method showed impressive results addressing contextual bias and domain gener- alization. For prompt generation, GPT-4 [ is employed to summarize image captions of the training dataset into a concise list of fewer than 10 domains, which are then used inside prompts. Image gener- ation is carried out using either SDEdit with medium strength (around 0.5) or InstructPix2Pix (5) For filtering, they use a confidence-basee filtering approach where a model f is trained, and a confidence threshold ty for each class is established by averaging the softmax scores of the correct labels from the training set. An edited image x' with a predicted label ý is excluded if confidence (f 9 2 tg- This thresholding ensures that images for which the predicted label matches the true label y removed due to redundancy. Additionally, images where ý + y with high confidence are also filtered out because they likely represent significant alteration, making them resemble another class more closely. For our pipeline, we observe that this approach tends to overly filter augmentations where y. as augmentations that could be redundant due to similarity to the original real image do not occur in our method, as we do not use real images as guidance for augmentation.
ALIA 15
E
More Methodology details
E.1
Prompt Generation via GPT-4
In this section, we provide more details on how we used GPT-4 to create prompts. After identifying the meta-class of the FGVC dataset, we input it into the following instruction:
"Generate 100 prompts for the class [meta-class] to use in a text-to-image model. Each prompt should:
Include the word [meta-class] ensure the image focuses on this object.
Ensure diversity each prompt by varying environmental settings, such as weather and time of day. You can include subtle enhancements like vegetation or small objects to add depth to the scene, ensuring these elements do not narrowly define the [meta-class] beyond its broad classification.
25

---

The prompts should meet the specified quantity requirement."
No quality control is used over the generated prompts.
E.2
Filtering Strategies
This section elaborates filtering mechanisms that remove lower-quality augmentations that do not correctly represent the sub-class or the meta-class. Additionally, we compare the effectiveness of alternative filtering methods in Table 21
Predictive Confidence Filtering utilizes the baseline model's confidence to filter out augmentations whose true label does not rank within the model's s top-k predictions (k = 10). This baseline model selected based on optimal performance outcomes from a hyperparameter sweep, as described in Appendix D.I The choice of k can affect the results: using too low of & can result in excessive filtering, limiting augmentations to those the baseline model already handles well, whereas too high of k results in insufficient filtering, allowing low-quality augmentations pass through. Therefore, we ablate on k as well to find the optimal value. We show a visualization of this filtering method in Figure
also evaluate other filtering methods, including CLIP filtering 21 Semantic Filtering, and ALIA confidence filtering [55. described in Section 3.4 and Appendix D33
Note that for certain datasets, such Cars and Aircraft [30]. the augmentations remain so consistent that only minimal filtering is required. For instance, out of the total augmentations produced for the Cars dataset and using our filtering method, only 0.1% were filtered out. However, this percentage is more significant for other datasets like CompCars [64] where 4.5% of augmentations were filtered.
Table 21: Performance of different filtering methods on the CompCars validation dataset, highlighting effectiveness of combined and individual strategies.
Prompt Strategy
Accuracy
No filter
49.4
CLIP Filtering
48.1
Semantic Filtering
49.6
ALIA Confidence Filtering
49.6
ALIA Confidence Filtering Semantic Filtering
49.8
Top-1 Confidence Filtering
47.4
Top-5 Confidence Filtering
49.6
Top-10 Confidence Filtering
49,8
Top-20 Confidence Filtering
49.4
Top- 10 Confidence filtering + Semantic filtering
50.1
Results from employing the various filters are presented in Table 21 Note that for faster experimenta- tion, we used 50% of the data (hence the low accuracy). Observations include: (1) CLIP filtering leads to poorer performance than using no filter, likely because CLIP struggles with fine-graid eneigraieed concepts such specific car model tail lights. (2) Our confidence filtering method achieves the best results at = 10. (3) Combining our confidence filtering with semantic filtering surpasses all other methods.
F
More Visualizations
F1
Qualitative Comparison with Generative Augmentation Methods
Example augmentations of Real Guidance, ALIA, and our method are visualized in Figure
E2
Confidence Filtering Visualization
Examples of augmentations that were and were not filtered for three FGVC datasets are in Figure
26

---

Original
Real-Guidance
ALIA
Ours (SaSPA)
Aireraft
CompCars
Cars
GMC Yukcn Denaii
GMC Yukin Denali
CACC Valten Panel
GMC "ukion Donall
CUB
DTD
Figure 7: Qualitative results of different generative augmentation methods: Real-Guidance, ALIA, and SaSPA on five FGVC datasets. Real Guidance produces very subtle variations from the original image due to the low translation strength they used. ALIA generates visible variations, but they are considerably less diverse compared to the augmentations produced by SaSPA.
Original
Augmetation
Original
Augmetation
Original
Augmetation
Filtered
Unfiltrrdd
Figure 8: Randomly selected augmentations of SaSPA that were and were not filtered for Aircraft, CompCars, and CUB.
27

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim to outperform previous traditional and generative data augmentation methods, clearly visible in Table [
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Discussed in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We thoroughly outline implementation details throughout the paper for each experiment. For example, in Appendix and Table 18
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification: All datasets used in this paper are public. The code is attached in the supple- mental and will be released publicly well.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
CooeeSumiiii ssi mnylicyy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe implementation details, including data splits, hyper- pareeeearameeess, and more details in Appendix C and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We opted to report error bars because (1) the results are stable and consistent
across runs, reducing the necessity for error bars, and (2) to improve readability. We use seeds for each run and test in various contexts and scenarios, making our evaluation robust and reliable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The training experiments we run are quite common, using model based on ResNet50 and generation based on Stable Diffusion, both of which are heavily researched. Moreover, detailed information on the type and amount of GPUs used, memory requirements well other relevant resources is provided in Appendix D.2
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://p//.aarsss.ccoooooo///eea
Answer: [Yes]
Justification: yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any data models that would require such safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Yes, all used libraries, such PyTorch (39 and diffusers [57 properly credited, and their licenses and terms of use are respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We attached the code the supplemental, and we will release the code publicly. The code is documented.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
33

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "148LQ5wCBi", "reviewer": "Reviewer_yjvN", "summary": "This paper presents SaSPA, a generative augmentation method specifically designed for FGVC. This method generates diverse, class-consistent synthetic images through conditioning on edge maps and subject representation. They use ControlNet condoned on edge maps and uses blip diffusion for its ability to zero-shot image generation. Their method is really smart and novel. They show results on Aircraft, Stanford Cars, CUB, DTD and Compcars for fine-grained evaluation.", "questions": "I think the paper needs more experimental results, but the image generation is really good.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) I found the methodology of this paper to be highly impressive. The approach is both intelligent and relatively straightforward, making it intuitively appealing. \n2) Utilizing GPT-4 to generate captions, followed by employing ControlNet on edges to create images via BLIP Diffusion, is a particularly smart strategy. \n3) Additionally, the analysis on mitigating contextual bias is excellent, providing insightful and valuable results.", "weaknesses": "1) Datasets are of small scale on which the experiments have been done. I would expect this method to perform well on fine-grained evaluation of imagenet dogs setup as well. Showing experiments on 200 classes of dogs on ImageNet will make the paper even stronger. \n2) How much is the performance improvement over methods like CLIP in few shot settings? Even if there is not much ( which is fair since here the aim is not image text training), it would be a good idea to show how behind this would be.\n3) To be really honest, I believe the results are pretty incremental. In Table 3 the difference between Diff-Mix & SaSPA is 0.3%, which is quite small and is probably in the noise range. The method is really smart and novel and I think just using this method on small-scale experiments on fine-grained datasets is kind of underselling the paper. I believe the method has a lot of potential and can be used to do large-scale training. Especially using edge-based control net and blip diffusion is a really good idea.\n4) I will urge the authors to rethink the experimental results and encourage them to show these results on larger datasets with more convincing results.\n5) The Fig4 caption is really small and should be a bit more detailed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your recognition of the novelty of SaSPA. We're glad you found our strategy and analysis impressive and valuable.\n\n\n**“Datasets are of small scale on which the experiments have been done. I would expect this method to perform well on fine-grained evaluation of imagenet dogs setup as well. Showing experiments on 200 classes of dogs on ImageNet will make the paper even stronger.”**\n* Small-scale datasets are common in FGVC due to the infrequent appearance of specific objects in the real world and the subtle differences between sub-categories, which complicate labeling efforts [1]. This limited data availability is precisely what makes FGVC an interesting topic, as it highlights the challenge and importance of generating effective training data.\n* The Stanford Dogs dataset (also referred to as ImageNet Dogs) contains approximately 20,000 images across 120 classes. This is comparable in scale to the Stanford Cars dataset, which contains about 16,000 images across 196 classes. Thus, the scale of our current datasets is consistent with standard practices in FGVC research.\n* Nevertheless, to further address the concern and increase confidence in our method, we conducted an experiment on the Stanford Dogs dataset, as requested. The results are presented in Table 2 of the rebuttal PDF. Additionally, we included the Oxford-IIIT Pet Dataset in our evaluation. Both datasets showed improved results with SaSPA.\n\n\n  \n  \n\n**“How much is the performance improvement over methods like CLIP in few shot settings? Even if there is not much ( which is fair since here the aim is not image text training), it would be a good idea to show how behind this would be.”**\n  \nThe zero-shot accuracy of CLIP is included in Figure 2 of the rebuttal PDF. As shown, SaSPA consistently outperforms CLIP in all shots, including 4-shots, whereas other augmentation methods fall short for Cars and DTD.  \n  \n\n\n**“To be really honest, I believe the results are pretty incremental. In Table 3 the difference between Diff-Mix & SaSPA is 0.3%, which is quite small and is probably in the noise range. ”**\n  \n* Our main results (Table 1, Table 2, Figure 4) show significant improvements over recent SoTA augmentation methods, both generative and non-generative, across various setups.\n* As for Table 3, diff-mix is a **concurrent paper**. ​​According to the official guidelines, authors are not expected to compare to such works. Nevertheless, we chose to add this comparison for completeness. The methods are quite different, and we believe both are valuable for the research community. Unlike Diff-Mix, which uses fine-tuning for its generative model, our method does not rely on such heavy fine-tuning. We have made it clearer in the paper that diff-mix is a concurrent work, as this point may have been missed.\n\n  \n  \n**“The method is really smart and novel and I think just using this method on small-scale experiments on fine-grained datasets is kind of underselling the paper. I believe the method has a lot of potential and can be used to do large-scale training. Especially using edge-based control net and blip diffusion is a really good idea”**\n  \nThanks for acknowledging the novelty of our method!\n\n  \n  \n**“I will urge the authors to rethink the experimental results and encourage them to show these results on larger datasets with more convincing results.”**\n  \nSee response to W1 and W3 above. We added new results.\n\n\n**“The Fig4 caption is really small and should be a bit more detailed.”**\n  \nThank you for the feedback. We have updated the caption for Figure 4 to be more detailed and easier to read.\n  \n  \n[1] Dunlap, Lisa, et al. \"Diversify your vision datasets with automatic diffusion-based augmentation.\" Advances in neural information processing systems 36 (2023): 79024-79034.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6QaLMSmzwM", "reviewer": "Reviewer_JRLS", "summary": "The paper proposes a data-augmentation technique tailored to fine-grained image classification.\nThe goal is to increase class fidelity while maintaining high variance in the images, something current diffusion-based data augmentation techniques are struggling with, especially in the fine-grained domain.\n \nThe main idea is to preserve both the scene-structure (by conditioning on edges with controlnet) and subjects (by using blip diffusion). Rather than augmenting a single sample, the edge map, class subject reference image and prompt are sampled independently. An image filtering strategy is proposed to increase the data quality. The method doesn’t require any finetuning of the generative model.\n \nThe method is shown to outperform baselines on 5 datasets and different scenarios such as full dataset training and few-shot classification are explored. Better generalization to unseen backgrounds compared to classifiers trained on the data augmentation baselines is also shown.", "questions": "## General remarks\n1. Code release is only mentioned in section 3.5, which is easy to miss, I would place it in abstract/introduction.\n2. Even though it is proposed for FGVC, the method is in principle applicable elsewhere - have you tried applying it to standard image classification datasets? Is there a reason why you think the method would not work well?\n\n## Add weakness 2\n3. The evaluation datasets consist of very common objects/animals which are well-represented in the training data of diffusion models such as cars and birds. It would help understanding the strenghts and weaknesses of the method to see its performance on a dataset with less common meta-classes, for example the iNaturalist.\n\n\n**References**:\n\n[1] Diao, Qishuai, et al. \"Metaformer: A unified meta framework for fine-grained recognition.\" arXiv preprint arXiv:2203.02751 (2022).\n\n[2] He, Ju, et al. \"Transfg: A transformer architecture for fine-grained recognition.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 1. 2022.", "limitations": "Limitations are adequately addressed in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The method builds on existing techniques, combining them in a novel way, which may lead to followup works\n2. The method is well-contextualized within related work\n3. The method is simple (easy to udnerstand and implement), novel and addresses an important problem of training deep learning models on domains with limited amount of training data, in particular for fine-grained visual classification\n4. The paper is well written, easy to follow, with well-designed figures explaining the methodology\n5. Extensive evaluation including many ablation expeirments justifying the design choices\n6. Experiments are well described with all the necessary details such as information on hyper-parameter tuning, code will also be released for reproducibility", "weaknesses": "1. The baseline classifier architecture is a standard image classifier. Meanwhile, there are architectures tailored to the fine-grained classification domain such as the MetaFormer [1, 2], it would be preferrable to also evaluate on state-of-the-art classifiers from the FGVC domain.\n2. It is not clear how the method would perform on a dataset of objects from a less common domain (not so well represented in the training data of the diffusion models)\nSee Questions for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our method's novelty and effectiveness in addressing training with limited data. We're glad you found the paper well-written and the evaluations helpful. Our code has already been released for reproducibility.\n\n**“The baseline classifier architecture is a standard image classifier. Meanwhile, there are architectures tailored to the fine-grained classification domain such as the MetaFormer [1, 2], it would be preferable to also evaluate on state-of-the-art classifiers from the FGVC domain.”**\n- We'd like to clarify that our baseline architecture is not simply a standard ResNet, but rather the CAL [1] architecture, which is tailored for FGVC and was SoTA at the time of its publication. ResNet is used as a backbone in this architecture. \n- We have updated the paper to make this distinction clearer, preventing any potential confusion for future readers.  \n- While CAL may not be the current SOTA for some datasets, it remains a strong and relevant baseline for FGVC tasks. For instance, MetaFormer shows mixed results when compared to CAL (MetaFormer is better than CAL on CUB but performs worse on Cars, with similar results on aircraft). \n- Nevertheless, we acknowledge the value of evaluating our method on more architectures like MetaFormer, and we will have results for another architecture such as the MetaFormer for the camera-ready version of our paper.\n\n\n\n**“Code release is only mentioned in section 3.5, which is easy to miss, I would place it in abstract/introduction.”**. \n  \nThank you for the suggestion. We have moved the code release information from section 3.5 to the abstract/introduction to increase visibility.\n\n**“Even though it is proposed for FGVC, the method is in principle applicable elsewhere - have you tried applying it to standard image classification datasets? Is there a reason why you think the method would not work well?”**\n- Thank you for considering the broader applicability of our method. There is no inherent reason our method should not work well on standard image classification datasets. \n- We focused on FGVC after identifying a gap in the existing literature concerning the application of diffusion models as generative augmentation within this domain. FGVC poses unique challenges and opportunities due to the subtle distinctions between classes, the struggles of current generative models to create images with accurate subclass fidelity, and the scarcity of data.\n- We believe our method has the potential to improve standard image classification as well, given its strong performance in generating diverse and accurate images for fine-grained categories. While our current focus was on FGVC, we are excited about the possibility of extending our approach to more general datasets in future work. Hence, we have updated our future work section.\n\n**“The evaluation datasets consist of very common objects/animals which are well-represented in the training data of diffusion models such as cars and birds. It would help understanding the strenghts and weaknesses of the method to see its performance on a dataset with less common meta-classes, for example the iNaturalist.”**\n- We need to differentiate between two cases:\n  - **Common Meta Classes with Uncommon Subclasses:** While the meta classes like airplanes and birds are common, the subclasses are not. For instance, generating in a text-to-image manner a Boeing 737-300 airplane can be challenging (see Figure 1 in the rebuttal PDF).\n  - **Uncommon Meta Classes:** Some datasets we chose have uncommon meta classes. For example:\n    - DTD (Describable Textures Dataset): Contains specific textures such as Blotchy, Honeycombed, and Laced, which are not very common.\n    - CompCars: Includes images of car parts such as headlights and taillights, which are not commonly associated with the correct caption in datasets like LAION, that were used to train the generation models.\n  \nWe realize that evaluating on even more uncommon object datasets could further demonstrate the robustness of our method, and if the reviewer finds that it will help the exposition, we will have results for a uncommon dataset for the camera-ready version of our paper.\n  \n[1] Rao, Yongming, et al. \"Counterfactual attention learning for fine-grained visual categorization and re-identification.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mgi7UQ6BDD", "reviewer": "Reviewer_Z7K4", "summary": "The paper proposes SaSPA, which uses several large, pre-trained generative, language, and vision-language models to produce high-quality synthetic images. Specifically they focus on images for FGVC, where the high inter-class similarities make it challenging to synthesize data that is not only diverse (capturing the intra-class variability) but also faithful to the intended class label. They combined GPT-4 with BLIP ControlNet diffusion, with edge map conditioning and post-generation filtering (via top-10 ranking with CLIP) to achieve performance gains compared to baselines, with only 2 augmentations per image (notably less than the 10x once thought to be a rule of thumb for training with synthetic data).", "questions": "My weaknesses correspond to questions.\n\n[W1] - If this data augmentation is applied to a SOTA FGVC model, is the impact orthogonal, e.g., do I get a new SOTA? Or is this just another way to get the model to learn the same information as the existing tricks? \n\n[W2] - Does this method work well beyond Cars/Aircraft/DTD? Will it break for other datasets, similar to how it seems to break for CUB?\n\n[W4] - How was CLIP selected for the filtering step?\n\nI am currently leaning reject, mainly due to W1 and W2. For clarity's sake, I have similar concerns about diff-mix and would not necessarily have voted to accept, had I reviewed that paper. I say this just because W1 applies to diff-mix as well, and replying to my review by pointing this out will not assuage my concern.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] SaSPA seems well-motivated, rather than haphazardly throwing large language and generative models at the task, the paper presents a carefully crafted pipeline that addresses the fidelity issue with both strong conditions (such as the edge maps) and also filtering. \n\n[S2] The paper is mostly well-presented, easy to follow, with motivation, method, and results that flow well both narratively and logically.\n\n[S3] The experiments are, for the most part, very thorough. (that's not to say nothing is missing, see W1)\n\nThis does not matter so much, but\n\n[S4] The differences between the proposed method and diff-mix are well-articulated, and SaSPA has some clear advantages to mitigate its disadvantages (such as not requiring additional training of the generative pipeline).", "weaknesses": "The results do not deliver on the promise of the paper's motivation. Principally,\n\n[W1] It is unclear if this approach can contribute to some state-of-the-art result. In some areas (e.g. text-to-image generation itself), asking SOTA of a research people might represent an unreasonable burden. However, in an area like FGVC, such evaluations are both practical and necessary. The authors should show that the performance gains on the ResNet50 (Table 1) don't vanish when applied to stronger methods (and I don't really count ResNet101 here, it's barely better than R50), particularly for CUB.\n\nSpeaking of CUB, it seems from the teaser that this method may struggle to capture the most fine-grained differences, such as when it recolors the engine. Perhaps this is why the method does not fare so well with the birds dataset, or in other words,\n\n[W2] The method's inferiority to CutMix on CUB for 448x448 images is very concerning (Table 3). The higher resolution is the standard for FGVC, and a method that only works well for low resolution is quite limited in impact. Additionally, it calls into question whether the method is good mainly for images of rigid objects, or if it could work well for other datasets of living things (Stanford Dogs, Fungi, NABirds). \n\nOn a more minor note,\n\n[W3] Some information seems misplaced. Image resolution is critical context in FGVC, especially if the paper uses smaller-than-normal images. The comparison in Table 3 should also compare M for diff-mix and the SaSPA.\n\nand\n\n[W4] Using CLIP for the filtering step seems strange. A model trained on the dataset (or even an ensemble) would likely filter the data much more reliably. Would this not be a much stronger approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and insightful feedback. We appreciate your recognition of SaSPA's well-motivated and carefully crafted pipeline, as well as your positive comments on the presentation and thoroughness of our experiments. Your constructive points will help us improve our work further.\n  \n**“to achieve performance gains compared to baselines, with only 2 augmentations per image (notably less than the 10x once thought to be a rule of thumb for training with synthetic data).”**  \nThank you for highlighting this! We have now included this point in the paper.\n\n**“[W1] It is unclear if this approach can contribute to some state-of-the-art result. In some areas (e.g. text-to-image generation itself), asking SOTA of a research people might represent an unreasonable burden. However, in an area like FGVC, such evaluations are both practical and necessary. The authors should show that the performance gains on the ResNet50 (Table 1) don't vanish when applied to stronger methods (and I don't really count ResNet101 here, it's barely better than R50), particularly for CUB.”**  \n- We'd like to clarify that our baseline architecture is not simply a standard ResNet, but rather the CAL [1] architecture, which is tailored for FGVC and was SoTA at the time of its publication. ResNet is used as a backbone in this architecture.  \n- We have updated the paper to make this distinction clearer, preventing any potential confusion for future readers.  \n- While CAL may not be the current SoTA for some datasets, it remains a strong and relevant baseline for FGVC tasks. For instance, MetaFormer [2], suggested by reviewer JRLS, shows mixed results when compared to CAL (MetaFormer is better than CAL on CUB but performs worse on Cars, with similar results on aircraft).  \n- Nevertheless, we acknowledge the value of evaluating our method on more architectures, and we will have results for a more recent architecture for the camera-ready version of our paper.\n\n**“The method's inferiority to CutMix on CUB for 448x448 images is very concerning (Table 3). The higher resolution is the standard for FGVC, and a method that only works well for low resolution is quite limited in impact.”**  \nPlease refer to the global rebuttal, we have added experiments on high-resolution images.\n\n**“Additionally, it calls into question whether the method is good mainly for images of rigid objects, or if it could work well for other datasets of living things (Stanford Dogs, Fungi, NABirds).”**  \nPlease refer to the global rebuttal, we have added experiments on two new datasets: Dogs and Pet.\n\n**“[W3] Some information seems misplaced. Image resolution is critical context in FGVC, especially if the paper uses smaller-than-normal images. The comparison in Table 3 should also compare M for diff-mix and the SaSPA.”**  \n- Regarding image resolution, we have updated our manuscript to make the resolution clearer.  \n- Please also refer to the global rebuttal, we have added results for higher-resolution images as well.  \n- Regarding the comparison in Table 3, generally speaking, we have two training schemes in our paper: (1) our training scheme and (2) the diff-mix training scheme. We opted to use the diff-mix training scheme when comparing to it because this provides a stronger comparison, favoring diff-mix as they trained and validated using their own training scheme and parameters. Hence, we cited diff-mix results from their paper rather than training using our pipeline, including for the few-shot scenarios.  \n- We have used the same M for SaSPA as diff-mix used to ensure an apples-to-apples comparison. We updated the manuscript with this information.  \n- It is important to note that diff-mix is a **concurrent paper**. According to the official guidelines, authors are not expected to compare to such works. Nevertheless, we chose to add this comparison for completeness. The methods are quite different, and we believe both are valuable for the research community. Unlike Diff-Mix, which uses fine-tuning for its generative model, our method does not rely on such heavy fine-tuning. We have made it clearer in the paper that diff-mix is a concurrent work, as this point may have been missed.\n\n**“[W4] Using CLIP for the filtering step seems strange. A model trained on the dataset (or even an ensemble) would likely filter the data much more reliably. Would this not be a much stronger approach?”**  \n- Our methods uses 2 filtering methods:\n  - (1) Semantic filtering, proposed by a recent work ALIA [3] to alleviate meta-class misrepresentation. Using **CLIP,** this process evaluates the relevance of generated images to the specific task at hand. For example, in a car dataset, each generated image is assessed against a variety of prompts such as “a photo of a car”, “a photo of an object”, “a photo of a scene”, “a photo”, and “a black photo”. Images that CLIP does not recognize as “a photo of a car” are excluded to ensure that the augmented dataset closely aligns with the target domain.\n- (2) Our top-10 filtering, which uses a **model trained on the dataset**, as you suggested. We made it clearer in the paper.\n- Note: Your intuition is correct! As we show in the paper (appendix F.2 Filtering Strategies), top-10 filtering is more important than semantic filtering, as evident in Table 19.\n\n    \n  \n  \n[1] Rao, Yongming, et al. \"Counterfactual attention learning for fine-grained visual categorization and re-identification.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.    \n[2] Diao, Qishuai, et al. \"Metaformer: A unified meta framework for fine-grained recognition.\" arXiv preprint arXiv:2203.02751 (2022).  \n[3] Dunlap, Lisa, et al. \"Diversify your vision datasets with automatic diffusion-based augmentation.\" Advances in neural information processing systems 36 (2023): 79024-79034.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' time, thoughtful comments, valuable suggestions, and their recognition of the potential positive impact of our method. Below, we address their common questions and concerns, in addition to the individual response per-review. In response to the feedback received, we have made several modifications to the paper and added new results, as requested. These changes have increased the clarity of our work and strengthened our findings even more.


## More Datasets

As requested by some reviewers, we added evaluations on two new datasets: the Stanford Dogs dataset and the Oxford-IIIT Pet Dataset.  
	
* The Stanford Dogs dataset contains 20,580 images of 120 dog breeds from around the world, built using images and labels from ImageNet for fine-grained visual classification. We used 50% of this dataset due to time constraints.  
  
* The Oxford-IIIT Pet Dataset contains 7,349 images of 37 breeds (25 dog breeds and 12 cat breeds) and is also used for fine-grained visual classification. We used 100% of this dataset.  
  
* In Table 2 of the rebuttal PDF, we compare SaSPA with CAL-Aug, which is typically the strongest traditional augmentation in our experiments.  
* We observed that SaSPA results in an improvement for both datasets, further strengthening the findings in our paper regarding SaSPA as a generative augmentation method.
* Combined with the results on DTD and CUB, we now have four datasets that are not of rigid objects, demonstrating that our method is effective beyond rigid objects (a concern raised by reviewer Z7K4)

## Will SaSPA Work on Higher Resolutions?
We have conducted additional experiments using 448x448 resolution on CompCars, DTD, and CUB datasets, employing SaSPA and the best augmentation method for each dataset (as per Table 1 in our paper). Each experiment was repeated with two seeds.
  
Table 1 in the rebuttal PDF presents the test accuracy for these high-resolution runs. Combined with our previous diff-mix comparison which used both 448x448 and 384x384 resolutions (Table 3 in the paper), we now have higher resolution results for all datasets.

Key findings:
* We observed consistent improvements across all datasets except CUB.
* For CUB, we hypothesize that the very fine-grained details such as feather patterns and colors present a significant challenge for generative methods, making it difficult to generate accurate representations at higher resolutions. 
  
In conclusion, SaSPA demonstrates promising results for most datasets, affirming its overall benefits.

Meta-review: The paper introduces a generative augmentation method for fine-grained visual categorization (FGVC) that creates diverse, class-consistent synthetic images using ControlNet conditioned on edge maps and BLIP diffusion for zero-shot image generation. The method shows promising results on datasets like Aircraft, Stanford Cars, CUB, DTD, and Compcars. 

All the reviewers are satisfied with the authors' concerns. The authors need to fix the minors concerns, including the need for a limitations section addressing issues like less satisfying performance on high-resolution datasets (e.g., CUB) and demonstrating broader applicability by testing on more diverse datasets beyond FGVC.
Decision: Accept (poster)