Decoupled Data Augmentation for Improving Image Classification

## Abstract

Abstract Recent advancements in image mixing and generative data augmentation have shown promise in enhancing image classification. However, these techniques face the challenge of balancing semantic fidelity with diversity. Specifically, image mixing involves interpolating two images to create a new one, but this pixel-level interpolation can compromise fidelity. Generative augmentation uses text-to-image generative models to synthesize or modify images, often limiting diversity to avoid generating out-of-distribution data that potentially affects accuracy. We propose that this fidelity-diversity dilemma partially stems from the whole-image paradigm of existing methods. Since an image comprises the class-dependent part (CDP) and the class-independent part (CIP), where each part has fundamentally different impacts on the image’s fidelity, treating different parts uniformly can therefore be misleading. To address this fidelity-diversity dilemma, we introduce Decoupled Data Augmentation (De-DA), which resolves the dilemma by separating images into CDPs and CIPs and handling them adaptively. To maintain fidelity, we use generative models to modify real CDPs under controlled conditions, preserving semantic consistency. To enhance diversity, we replace the image’s CIP with inter-class variants, creating diverse CDP-CIP combinations. Additionally, we implement an online randomized combination strategy during training to generate numerous distinct CDP-CIP combinations cost-effectively. Comprehensive empirical evaluations validate the effectiveness of our method.

## 1 Introduction

Data augmentation is extensively employed to enhance neural network performance. Traditional data augmentation, such as random shifting, cropping, and rotation, are widely used due to their simplicity and effectiveness, becoming standard practice in nearly all training algorithms. Recently, two innovative types of data augmentation have shown potential for improving image classification:

- •
Image-Mixing Data Augmentation. Generate augmented images by integrating two or more randomly picked natural images at the pixel or feature level, creating virtual data between classes.
The online combination paradigm allows for the efficient production of many images with extensive pixel-level variations at a low cost, yet the images often look unrealistic and face fidelity problems, as noted by (Kang & Kim, ; Islam et al., ).
- •
Generative Data Augmentation. This method leverages generative models to create images using prompts generated manually or via textual inversion to align with class labels. However, as noted by (Islam et al., ), this method is not yet mature for data-rich learning scenarios. Crafting prompts that ensure model-generated images match the actual data distribution is difficult, requiring expert knowledge to describe class objects and challenges in capturing the dataset’s style. Additionally, textual inversion often leads to limited image diversity due to information loss, reducing the diversity of the generated images, as mentioned by (Wang et al., ). Both forms of prompt guidance encounter issues of misalignment or limited variation, resulting in limited performance improvements.

Readers can refer to Figure 1 for examples of various data augmentation methods. It is evident that a trade-off exists between semantic fidelity and diversity in these methods. Naturally, the question arises: *’How can semantic fidelity be preserved while simultaneously enhancing diversity?’*

The prevailing practice of treating images as indivisible units in existing data augmentation methods presents a fundamental obstacle to achieving both fidelity and diversity. This whole-image paradigm, while enriching diversity, often results in excessive and detrimental variations to class-dependent objects, severely compromising fidelity. In contrast, viewing images from a disentangled perspective could alleviate this challenge by applying distinct strategies to class-dependent parts and class-independent parts: a conservative strategy on CDPs to maintain fidelity and an aggressive strategy on CIPs to enhance diversity.

Based on this insight, we propose a novel data augmentation framework, Decoupled Data Augmentation (De-DA), which addresses the fidelity-diversity dilemma through a decoupling strategy. Specifically, we first separate images into class-dependent parts (CDPs) and class-independent parts (CIPs) using SAM (Kirillov et al., ), and then tailor our adaptive strategies for respective parts according to their distinct characteristics. To preserve semantic fidelity, we use class identifiers derived from intra-class CDPs as conditions to edit real CDPs with controlled strength, elaborately varying them while preserving their semantic consistency. To encourage diversity, we replace the original CIP of the images with a random CIP sampled from an inter-class image. Furthermore, we adopt an online randomized combination strategy, pairing one CDP (real or synthetic) with one CIP (cross-class real CIPs) at random positions and transformations to provide the model with various combinations during the training stage, further enhancing diversity. In summary, both conservatively translated CDPs and real CIPs align with the actual data, ensuring that the generated images maintain fidelity, while the semantic edits on CDPs and diverse CDP-CIP combinations significantly enrich variety.

Compared to previous image-mixing methods, De-DA fuses CDPs and CIPs at the semantic level rather than the pixel level, thereby enhancing fidelity. De-DA also distinguishes itself from other generative methods via applying textual inversion (Gal et al., ) and SDEdit (Meng et al., ) to isolated CDPs instead of the entire image, thus avoiding the negative effects of noisy information in the image. Furthermore, De-DA’s decouple-and-combine paradigm enables the production of more images at a lower cost than prior generative methods. Our contributions include:

- •
De-DA shows a solution to the fidelity-diversity dilemma in previous data augmentation methods by decoupling images into class-dependent parts and class-independent parts and managing these parts adaptively.
- •
To our knowledge, we are the first to apply textual inversion and SDEdit to isolated CDPs instead of entire images in the field of data augmentation, which minimizes the negative impact from the noisy information in the images. Additionally, we propose truncated-timestep textual inversion to reduce the computational burden, enhancing practicability.
- •
Extensive experiments on domain-specific classification, multi-label classification, and data-scarce learning scenarios comprehensively validate the effectiveness of De-DA.

Table: Table 1: Comparing data augmentation methods on fidelity and diversity.

Figure: Figure 2: Illustration of the mechanisms of different data augmentation methods. Row 1: Image-mixing methods, such as Mixup (Zhang et al., ) and CutMix (Yun et al., ) create mixed images through pixel-level interpolation. DiffuseMix (Islam et al., ) uses style prompts (e.g., ”Sunset”) to transform input images, generating varied-style images which are then concatenated to form a hybrid image. DA-Fusion (Trabucco et al., ) uses the intra-class identifier ${\color[rgb]{0.0,0.42,0.24}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.0,0.42,0.24}\textbf{V1}_{\text{image}}}$, while Diff-Mix (Wang et al., ) employs an another class’s identifier ${\color[rgb]{0.0,0.42,0.24}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.0,0.42,0.24}\textbf{V2}_{\text{image}}}$ to translate natural images with SDEdit, but these methods face issues of limited variety or constrained fidelity. Row 2: Our proposed De-DA maintains fidelity by editing CDPs conditioned with ${\color[rgb]{0.8,0.33,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.8,0.33,0.0}\textbf{V1}_{\text{CDP}}}$ through a transparency image-to-image diffusion pipeline which is specifically designed for handling transparent images. It also enhances diversity by replacing CIPs and applying random transformations to CDPs, resulting in faithful and diverse images.
Refer to caption: extracted/5933288/fig/fig2-241001.png

## 2 Related Work

Image-mixing and generative data augmentation methods are two approaches akin to De-DA. Table 1 offers an overview of prominent image-mixing and generative data augmentation methods, with Figure 2 depicting their mechanisms.

#### Image-Mixing Data Augmentation.

Image mixing is a non-generative data augmentation technique used during training to provide classifiers with numerous mixed images, which helps smooth decision boundaries and enhance image classification (Zhang et al., ). Early methods, such as Mixup (Zhang et al., ) and CutMix (Yun et al., ), create new images by linearly combining two images at the pixel or patch level. However, this mixing can compromise the semantic integrity of class-specific objects. To address this, advanced approaches like SaliencyMix (Uddin et al., ), SnapMix (Huang et al., ), PuzzleMix (Kim et al., ), CoMixup (Kim et al., ), and GuidedMixup (Kang & Kim, ) use saliency maps to ensure important regions are preserved. Despite this guidance, class-specific objects may still be distorted, producing virtual images that deviate significantly from the actual data distribution, resulting in limited semantic fidelity. In contrast, De-DA addresses this issue by combining CDPs and CIPs at the semantic level, rather than at the pixel level, to preserve fidelity.

#### Generative Data Augmentation.

Generative data augmentation leverages advanced text-to-image models to create new images. Initial research (He et al., ) demonstrates that text-to-image diffusion can generate synthetic data that effectively enhances classification performance in data-scarce scenarios, particularly when conditioned on detailed class descriptions. Azizi et al. () improved generation quality by fine-tuning diffusion models on ImageNet, leading to better classification performance. Bansal & Grover (); Yuan et al. () utilized style prompts to synthesize images for improving domain adaptation. Subsequent studies explored various prompt enhancement techniques. For instance, Li et al. () added image captions to prompts for better distribution alignment, while Yu et al. () employed large language models to create diverse and detailed prompts at scale. However, these methods often struggle to produce in-distribution images due to the black-box nature of generative models and challenges in accurately describing the abstract characteristics of datasets. Consequently, they do not significantly enhance domain-specific classification tasks, where similar data distributions between different classes increase the likelihood of generating images with ambiguous labels. To address this issue, researchers leverage textual inversion (Gal et al., ) to learn class identifiers from real images, bypassing the difficulties of prompt design. Using these learned class identifiers, DA-Fusion (Meng et al., ) employs SDEdit to modify real samples into new ones with controlled generation strength, ensuring semantic consistency but limited variation. Diff-Mix (Wang et al., ) augments images with inter-class personalized identifiers to vary the images’ backgrounds while maintaining a faithful foreground. However, there is a non-negligible probability of producing unexpected images with unfaithful foregrounds and unchanged backgrounds, necessitating the use of CLIP as a filter to remove problematic samples. DiffuseMix (Islam et al., ) uses manually crafted style prompts, such as ”Sunset,” to vary input images and concatenate two varied-style images into a hybrid image. However, the diversity in DiffuseMix is primarily reflected images’ style, with limited variation in images’ semantic content. We conclude that existing generative data augmentation methods often sacrifice either diversity or fidelity. De-DA mitigates this issue through a decoupling strategy, effectively controlling CIP to diversify and CDP to maintain fidelity.

Figure: Figure 3: The pipeline of De-DA. Left: (1) Images are decoupled into CDPs and CIPs. Missing regions in each CIP are inpainted, creating a pool of inter-class inpainted CIPs. (2) Truncated-Timestep Textual Inversion (TTTI) are applied to the real CDPs to efficiently learn the class-specific identifiers ${\color[rgb]{0.8,0.33,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.8,0.33,0.0}\textbf{V}_{1}},{\color[rgb]{0.8,0.33,0.0}\definecolor[named]{ pgfstrokecolor}{rgb}{0.8,0.33,0.0}\textbf{V}_{2}},\dots$ for each class. These identifiers are then used to semantically modify real CDPs into new synthetic CDPs. Right: (3) CDPs and CIPs are combined by pairing a real or synthetic CDP (with probability $1-p_{\text{syn}}$ or $p_{\text{syn}}$) with a randomly selected CIP to create a new image. With probability $p_{mix}$, an inter-class CDP is added to generate a mixed-CDPs image.
Refer to caption: extracted/5933288/fig/pipeline-end_2.png

## 3 Decoupled Data Augmentation

De-DA is a framework designed to address the fidelity-diversity trade-off through a decoupling strategy. As illustrated in Figure 3, it initially separates class-dependent parts (CDPs) and class-independent parts (CIPs) using SAM, which forms the foundation of De-DA. De-DA employs class identifiers derived from intra-class CDPs to conditionally edit real CDPs, then pairs a real or synthetic CDP with randomly selected CIPs to create new images.

#### Decoupling Images into CDPs and CIPs.

The initial phase of De-DA involves separating training samples into class-dependent parts and class-independent parts, as the basement of our De-DA. Practically, we utilize Lang-SAM (Kirillov et al., )(^1^11Lang-SAM ([https://github.com/luca-medeiros/lang-segment-anything](https://github.com/luca-medeiros/lang-segment-anything)) is a prompt-guided segmentation tool based on GroundingDINO (Zuwei Long, ) and SAM (Kirillov et al., ).), an off-the-shelf, prompt-based segmentation tool, to obtain segmentation masks for class-dependent parts using domain or class name prompts (e.g., ”bird” for CUB-200-2011). If multiple CDP masks are generated in one image, they are aggregated into a single mask to ensure complete coverage of the CDP. Masked regions are labeled as CDPs, while remaining image portions are classified as CIPs. We apply alpha pyramid image blending to fill the missing areas in the segmented CIPs.

#### Conservative Generation of CDP.

Following prior research (Trabucco et al., ; Zhou et al., ), we use textual inversion to derive identifiers for each class and employ SDEdit to transform natural images conditioned on these prompts. Unlike previous methods, our approach applies textual inversion and SDEdit solely to the class-dependent parts (CDPs) rather than the entire image. This strategy addresses two critical issues: (1) Learning class prompts from CDPs ensures that the derived concept accurately corresponds to class-specific objects. (2) Applying SDEdit to CDPs prevents interference from class-independent parts, enhancing SDEdit’s performance. This contrast is shown in Figure 4. However, applying textual inversion and SDEdit to CDPs is challenging, as traditional methods are only designed for RGB images. To accommodate transparent CDPs, we employ LayerDiffuse (Zhang & Agrawala, ), which equips diffusion models with a dedicated transparency encoder and decoder capable of encoding the alpha channel into latents and decoding latents into RGBA images. Specifically, our transparency image-to-image pipeline operates as follows: we first add noise $\epsilon\sim\mathcal{N}(0,1)$ to real CDPs (indicated by $x_{0}^{\text{ref}}$) at timestep $\lfloor T_{s}\rfloor$, where $s\in[0,1]$ indicates the generation strength ($s=0$ refers no editing and $s=1.0$ indicates generation from scratch), followed by denoising:

$$ $x_{\lfloor S_{T_{s}}\rfloor}=\sqrt{\tilde{\alpha}_{\lfloor S_{T_{s}}\rfloor}}x _{0}^{\text{ref}}+\sqrt{1-\tilde{\alpha}_{\lfloor S_{T_{s}}\rfloor}}\epsilon\\$ (1) $$

Denoise $x_{\lfloor S_{T_{s}}\rfloor}$ using LayerDiffuse reverse diffusion conditioned on the learned identifier $\textbf{V}_{\text{CDP}}$ (we will discuss later), starting from the timestep $\lfloor T_{s}\rfloor$ to 0 0, yielding the final edited CDP $x_{0}$.

$$ $x_{t-1}=x_{t}-\epsilon_{\theta}\left(x_{t},t,\textbf{V}_{\text{CDP}}\right), \quad t=\lfloor S_{T_{s}}\rfloor,\dots,1$ (2) $$

Here, $\textbf{V}_{\text{CDP}}$ is the class identifier derived from each class’s real CDPs using textual inversion. To alleviate the computational cost of textual inversion, we apply truncated-timestep textual inversion tailored for SDEdit. In this method, the prompts are trained only on the timestep from 0 0 to $\lfloor S_{T_{s}}\rfloor$ instead of all timesteps, promoting quicker convergence. Formally, truncated-timestep textual inversion learns $\textbf{V}_{\text{CDP}}$ by

$$ $\textbf{V}_{\text{CDP}}=\operatorname*{arg\,min}_{c}\mathbb{E}_{t\in[0,\; \lfloor S_{T_{s}}\rfloor]}\left[\|\epsilon-\epsilon_{\theta}(x_{t},c,t)\|\right]$ (3) $$

Figure: Figure 4: Examples illustrate the differences between applying SDEdit to the entire images and the pure CDPs. We observe that in generative methods, the background can negatively affect the performance of SDEdit. Row 1: The generative model misinterprets a person in red clothing in the background as the bird’s crest. Row 2: A person in the background is mistakenly integrated into the bird during SDEdit, reducing the fidelity of the translated image. Row 3: Ice in the background is misrepresented as birds. In contrast, the right three columns showcase images generated by De-DA. De-DA involves applying textual inversion and SDEdit to isolated CDPs, facilitating modifications to avian features like feathers, eyes, and legs without altering their labels. By focusing on isolated CDPs, De-DA effectively mitigates the influence of background noise on the translation process.
Refer to caption: extracted/5933288/fig/fig4-240930.png

#### Inter-class Random Sampling of Class-Independent Parts.

Our approach to handling Class-Independent Parts (CIPs) derives from observations of real datasets. Specifically, real datasets exhibit significant intra-class uniformity but restricted cross-class diversity. For example, in the CUB-200-2011 dataset (Wah et al., ), 90% of Common Yellowthroat images feature branches in the background. Albatross images often show water surfaces, while Jaeger images typically capture them in flight. Based on these observations, we propose an intra-class CIP sampling strategy rather than generating new CIPs. This method sufficiently enhances the CIP diversity of synthetic images and has proven effective in our experiments. The inter-class CIP replacement strategy offers two main advantages: (1) it is computationally efficient and avoids producing out-of-distribution CIPs, which can occur with generative data augmentation; and (2) CIP replacement can generate numerous images with the same CDPs but different CIPs, thereby enabling the trained model to better focus on critical regions of the images, which is validated in experiments 4.2.

#### Online Randomized Combination.

During the training phase, we combine CDPs and CIPs by selecting a CDP and a CIP at random. The CDP is randomly resized and pasted onto a random position on the CIP, with specific random transformations applied to generate a new image. This random placement enables the model to learn position-independent features effectively. Additionally, two different CDPs are occasionally mixed with one CIP to create new multi-label samples, resulting in semantic interpolations between two classes. The weight of each label is proportional to the pixel area of the respective CDPs, following the implementation of CutMix (Yun et al., ). Compared to interpolations created by image-mixing data augmentation, our semantic interpolations appear more realistic and maintain semantic fidelity. Experiments (Section 4.3) confirm that the randomized combination and CDP mixing strategies could lead to performance improvements. Besides superior generation quality, the decouple-and-combine prototype of De-DA can produce a substantially larger number of images with high efficiency compared to generative methods. Specifically, suppose each class consists of $M$ training samples. If we generate $K$ synthetic CDPs for each real CDP, we can obtain a total of $CM(1+K)$ different CDP-CIP combinations ($C$ is the number of classes) for that image, where $1+K$ is the total number of real and synthetic CDPs derived from that real CDP and $CM$ is the total number of different CIPs.

## 4 Experiments

In this section, we comprehensively analyze De-DA by answering the following questions:

Q1: Can De-DA outperform other methods in conventional classification tasks?

Q2: Can De-DA still surpass other methods in various settings such as data-scarce scenarios?

Q3: How do the modules in our approach and the hyperparameters affect our method’s performance?

To answer Q1, in Section 4.1, we compare De-DA to peer methods across different domain-specific datasets. In Section 4.2, we address Q2 by examining its performance in data-scarce scenarios, multi-label classification, and a replaced-background dataset, demonstrating the performance gains of De-DA in various contexts. For Q3, we conduct extensive ablation studies on each module and hyperparameter to assess their impacts and explain our chosen settings in Section 4.3.

### 4.1 Comparison on Conventional Classification

#### Experimental Setting.

We tested data augmentation methods on three classical domain-specific datasets: CUB-200-2011 (Wah et al., ), Aircraft (Maji et al., ), and Stanford Cars (Krause et al., ), following the experimental settings of DiffuseMix (Wang et al., ) and Diff-Mix (Wang et al., ). Experiments are conducted using three smaller models—ResNet-18 (He et al., ), ResNet-50 (He et al., ), and DenseNet121 (Huang et al., )—as well as a large pretrained model, ViT-B/16 (Dosovitskiy et al., ). To ensure fairness, we adhere to prior work for our training implementations. For the small models, we followed the GuidedMix implementation (Kang & Kim, ), training from scratch with cross-entropy loss.(^2^22Our results might differ from Diff-Mix for small models (ResNet-18, ResNet-50) because we trained from scratch, whereas Diff-Mix began with a pretrained model.) For ViT-B/16, we follow Diff-Mix, fine-tuning the ViT model with label smoothing loss. Unless otherwise specified, in De-DA, we set the expansion multiplier for each real CDP to 3. The generation strength for textual inversion and SDEdit is fixed at $s=0.4$. During training with De-DA, nautral images are replaced with augmented data with a probability $p_{\text{aug}}=0.5$. For CDP-CIP combinations, the probability of using mixed CDP $p_{mix}$ is 0.5, and the synthetic CDP is used with a probability $p_{syn}=0.25$.

#### Peer Methods.

We compare De-DA with ten peer methods, including six image-mixing and four generative approaches. The image-mixing methods include: (1) Mixup (Zhang et al., ), which linearly combines pairs of images and their labels; (2) CutMix (Yun et al., ), which replaces a portion of one image with a patch from another; (3) SaliencyMix (Uddin et al., ); (4) Co-Mixup (Kim et al., ); and (5) Guided-AP and (6) Guided-SR (Kang & Kim, ), which use saliency maps to guide the mixing process, alleviating the issue of corrupted class-specific objects. The generative methods include: (1) Real-Guidance (He et al., ), which augments the dataset using label-name guidance at a fixed low strength $s=0.1$; (2) DiffuseMix (Islam et al., ), which creates hybrid images from different conditional prompts using fractal blending; (3) DA-Fusion (Trabucco et al., ), which augments images with identifiers learned from intra-class images at random strengths $s\in\{0.25,0.5,0.75,1.0\}$. (4) Diff-Mix (Wang et al., ), which augments images with the identifiers learned from other classes’ images at random strengths $s\in\{0.5,0.7,0.9\}$. Both Real-Guidance and Diff-Mix additionally make use of the large vision-language model CLIP Radford et al. () to evaluate label confidence, aiding in the filtering of distorted images. We set the expansion multiplier for all augmented methods and adjust the augmentation probability $p_{\text{aug}}$ according to each method’s recommendation.

Table: Table 2: (Training from scratch) Conventional classification on domain-specific dataset. We bolden the highest and underline the second highest.

Table: Table 3: Accuracy of finetuning on Vit-B/16.

Table: Table 5: Classification result on out-of-distribution dataset Waterbird Sagawa* et al. (), each image is crafted by combine CUB-200-2011’s foregrounds with the background from Places Zhou et al. (). Higher accuracy represents higher background robustness.

Figure: (a)
Refer to caption: x1.png

#### Comparison on Conventional Classification.

Table 2 presents the test accuracy of various augmentation strategies across three domain-specific datasets. All methods use an input resolution of 448 and are trained from scratch. The results indicate the following: (1) De-DA consistently outperforms other data augmentation methods on CUB-200-2011 and Aircraft by a significant margin. Specifically, De-DA achieves test accuracies of 80.07% and 82.06%, exceeding the second-highest accuracy by notable margins of 2.3% and 4.41%, respectively, with ResNet-18. (2) De-DA does not surpass Diff-Mix in test accuracy on Stanford Cars. We hypothesize this is due to the class-specific object occupying most of the image area in Stanford Cars, making the CIP replacement strategy less effective for performance gains in this dataset. (3) Generative augmentation methods like DA-Fusion and Real-Guidance do not enhance accuracy, suggesting that the standard use of diffusion for image editing is limited in its ability to improve accuracy due to restricted diversity.

#### Comparison on Fine-Tuning on Large Models.

Table 4 reports the accuracy of different methods on large model fine-tuning with an input resolution of 384. The results indicate: (1) De-DA consistently demonstrates superior performance compared to other methods. (2) The improvement of De-DA over the second-highest method is less pronounced than when training from scratch.

### 4.2 Comparisons on Various Tasks

To evaluate performance in data-scarce scenarios, we create a version of the CUB-200-2011 dataset by randomly selecting 10 images per class, following the settings of DiffuseMix (Islam et al., ). To assess how data augmentation aids in learning background-robust features, we test accuracy on the Waterbird dataset, which combines bird foregrounds from CUB-200-2011 with backgrounds from the Places dataset (Zhou et al., ). Here, (Waterbird, Water) indicates (the type of bird, the type of background). We further compare De-DA to other methods on the multi-label classification dataset Pascal (Everingham et al., ) to validate its performance in improving multi-label classification. Additionally, we demonstrate that De-DA is compatible with other data augmentation techniques, such as RandAugment(Cubuk et al., ).

#### Comparison in Data-Scarce Scenarios.

The results on the data-scarce CUB-200-2011 dataset are shown in Table 4. We observe that De-DA significantly outperforms all other methods, achieving an accuracy of 54.52% on ResNet-18, which is 8.77% higher than the second-best method. This remarkable improvement is due to the substantially larger number of augmented images generated by De-DA’s online combination strategy, which compensates for the lack of data. Furthermore, compared to image-mixing methods that also produce a large number of mixed images, De-DA shows significant improvement, demonstrating that the images generated by De-DA are much more effective due to their high diversity and fidelity.

#### Comparison on Background Robustness.

Table 5 presents the experimental results on the Waterbird dataset, which evaluates the model’s robustness against background replacement, rather than relying on the background. De-DA clearly outperforms other methods in all categories, achieving an average improvement of 5.98%, which is 3.70% higher than the second-best method, Diff-Mix. This validates our earlier statement that De-DA helps the model learn CIP-independent features, enabling the model to focus on the class-specific object for classification.

Figure: Figure 6: Visual examples of different generative data augmentation methods.
Refer to caption: extracted/5933288/fig/Comparison241001-2.png

#### Comparison on Multi-Label Classification.

Figure 5(a) compares different methods on a multi-label classification. De-DA achieves 23.02% on ResNet-18 and 22.05% on ResNet-50, surpassing all other methods by a non-trivial margin, demonstrating that De-DA can effectively improve multi-label classification through mixed-CDPs augmented samples.

#### Comparing Diversity of Generative Data Augmentation.

Figure 5(b) quantitatively compares De-DA to other generative methods using Peak Signal-to-Noise Ratio (PSNR). A lower PSNR value indicates higher diversity. The results show that De-DA achieves greater diversity than the other methods. Figure 6 presents the examples of different generative data augmentation methods, validating our aforementioned statement. Specifically, we observe that (1) DiffuseMix diversifies images from a stylistic perspective rather than a semantic level. While it shows robustness to adversarial noise, it is less effective in improving test accuracy. (2) Real-Guidance slightly modifies images using SDEdit at a low strength. Although it maintains semantic consistency, it struggles with background invariance. (3) Da-Fusion has the same issue as Real-Guidance. (4) Diff-Mix uses identifiers from other classes to transform the input image, aiming to vary the background while preserving the semantic fidelity of the foreground. However, it often significantly alters the foreground greatly without effectively diversifying the background.

Figure: Figure 7: Compatibility of De-DA with RandAugment.
Refer to caption: x3.png

#### Compatibility with Traditional Data Augmentation.

We evaluate the compatibility of De-DA with RandAugment (Cubuk et al., ), as shown in Figure 7. The results indicate that combining De-DA with RandAugment outperforms using De-DA alone, indicating that De-DA and traditional data augmentation RandAugment are two mutually reinforcing mechanisms. The result validates the extensibility of De-DA.

### 4.3 Ablation Studies

#### Experimental Setting.

We first evaluate the impact of each hyperparameter of De-DA on CUB-200-2011 with ResNet-18. Then, to evaluate the contribution of each component of our approach, an ablation study is conducted by incrementally adding each component. We focused on components including synthetic CDP, CIP replacement, the randomized combination and the CDP mixing technique. The experimental baseline model is ResNet-18 and the dataset is Aircraft.

Figure: Figure 8: (a) Impact of the probability of replacing real data with augmented data $p_{\text{aug}}$; (b) Impact of the probability of using edited CDPs $p_{\text{syn}}$; (c) Impact of the probability of using CDP mixing $p_{\text{mix}}$; (d) Impact of the generation strength $s$.
Refer to caption: x4.png

#### Ablation on Hyperparameters.

The impact of the De-DA’s hyperparameters is shown in Figure 8. The experimental results lead to the following conclusions: (1) The performance of De-DA improves as $p_{\text{aug}}$ increases from 0.00 to 0.50, peaking at 0.5, indicating that a balanced approach of using both generated and original data is optimal. (2) The highest performance for three datasets is observed at $P_{\text{syn}}=0.25$. Using either no synthetic CDP or only synthetic CDPs results in a performance decline. (3) We observe that CDP mixing leads to significant improvement on CUB-200-2011 at $p_{\text{mix}}=0.25$, demonstrating that CDP mixing effectively boosts classification. (4) A generation strength of $s=0.4$ consistently yields improvement across the three datasets. Peak performance occurs at different strengths: Aircraft and Standford Car peak at $s=0.4$, while CUB-200-2011 peaks at $s=0.2$, possibly because the inter-class images in the CUB-200-2011 are more similar than in the other two datasets. However, too high a strength can result in performance decline, e.g. $s=0.8$ achieves a lower accuracy than $s=0.0$ on CUB-200-2011. These ablation studies explain our hyperparameter choices. The results indicate that De-DA is relatively robust to hyperparameter settings. The impact of the expansion multiplier is discussed in the appendix.

Figure: Figure 9: Ablation study on each module.
Refer to caption: x5.png

#### Ablation on Each Module of De-DA.

Figure 9 shows the module ablation results. (a) represents vanilla training without data augmentation. (b) involves replacing the CDP of the original sample with a same-size synthetic CDP at the same position, with a probability $p_{syn}=0.25$. This improves upon the baseline, validating the effectiveness of semantically edited CDPs in image classification. (c) replaces the CDP with a synthetic one using a random combination strategy that varies the position and size of the CDPs, further enhancing (b) by 1.34 %, indicating that random combination strategy do compile with CDP editing. (d) employs only the CIP replacement strategy without CDP editing or random combination, effectively boosting accuracy, which highlights the importance of CIP diversity. (e) utilizes an inter-class CIP strategy with random combination, further improving performance, indicating that CIP replacement and random combination are two mutually reinforcing mechanisms. (f) incorporates the strategies of CDP editing, inter-class CIP replacement, and random combination, achieving an accuracy of 80.15%, which surpasses the accuracies of all other peer methods. (g) is the complete version of De-DA, incorporating the CDP-mixing strategy. This integration further enhances the performance.

## 5 Conclusions and Future Works

In this work, we propose an innovative approach to address the fidelity-diversity dilemma through decoupled data augmentation (De-DA). We decouple images into class-dependent and class-independent parts, with CDP maintaining semantic fidelity and CIP enhancing diversity. The decouple-and-combine strategy of De-DA enables the production of faithful and diverse images at scale, with lower computational costs compared to generative methods. Experiments validate that De-DA effectively improves conventional classification, data-scarce classification, and multi-label classification. De-DA also helps models learn background-independent features. Future work could explore several directions: (1) decoupling images in a more fine-grained manner to improve performance in fine-grained retrieval tasks; (2) developing adaptive strategies for designing generation strength based on dataset characteristics; (3) exploring new CDP-CIP combination approach to further boost diversity.

## Appendix A Appendix

**Table 6: General hyperparameters of De-DA.**
| Hyperparameter Name | Value |
| --- | --- |
| Segmentation prompt | CUB: ”Bird”; Aircraft: ”Aircraft”; Standford Car: ”Car”; Pascal: label name |
| Probability of using augmented data ($p_{\text{aug}}$) | 0.5 |
| Probability of using synthetic CDP ($p_{\text{syn}}$) | 0.25 |
| Probability of applying CDP mixing ($p_{\text{mix}}$) | 0.5 |
| Generation strength ($s$) | 0.4 |
| Textual inversion token initialization | CUB: ”Bird”; Aircraft: ”Aircraft”; Standford Car: ”Car”; Pascal: label name |
| Textual inversion batch size | 32 |
| Textual inversion learning rate | 1e-4 |
| Textual inversion training steps | 400 |
| SDEdit prompt | “a photo of a <class name>” |
| Stable diffusion guidance scale | 7.0 |
| Stable diffusion resolution (pixels) | 512 |
| Stable diffusion denoising steps | 25 |

**Table 7: Training hyperparameters of De-DA.**
| Architecture | Resnet-18 | Resnet-50 | DenseNet-121 | ViT-B/16 |
| --- | --- | --- | --- | --- |
| Learning rate | 0.01 | 0.01 | 0.01 | 0.001 |
| Epochs | 300 | 300 | 300 | 120 |
| Batch size | 16 | 16 | 8 | 32 |

### A.1 Implementation Details

All of our experiments are conducted on a system equipped with 96 CPU cores (Platinum 8255C @ 2.50GHz) and 8 GPU Tesla V100 cards. For doupling the images into CDPs and CIPs, we employ LangSAM (^3^33[https://github.com/luca-medeiros/lang-segment-anything/](https://github.com/luca-medeiros/lang-segment-anything/)) with prompt guided. For inpainting the missing part of
For the training implementations of ResNet-18, ResNet-50, and DenseNet-121, we adhere to the official training script from GuidedMix (Kang & Kim, ) (^4^44[https://github.com/3neutronstar/GuidedMixup/blob/main/FGVC/main.py](https://github.com/3neutronstar/GuidedMixup/blob/main/FGVC/main.py)). For the training of Vit-B/16, we followed the official implementation provided by Diff-Mix (Wang et al., ) (^5^55[https://github.com/Zhicaiwww/Diff-Mix/blob/master/downstream_tasks/train_hub.py](https://github.com/Zhicaiwww/Diff-Mix/blob/master/downstream_tasks/train_hub.py)). Additionally, we introduce the hyperparameters related to decoupling, truncated-timestep textual inversion and SDEdit. Specific values for these hyperparameters are provided in Table 6 and Table 7.

**Table 8: Ablation on the expansion multiplier for each real CDP.**
| \rowcolormygray<br>Number | CUB | Aircraft | Car |
| --- | --- | --- | --- |
| $\times 1$ | 79.68 | 81.50 | 91.91 |
| $\times 3$ | 80.07 | 82.06 | 92.23 |
| $\times 6$ | 80.18 | 81.52 | 91.92 |
| $\times 9$ | 79.89 | 80.96 | 90.85 |

### A.2 Ablation Study

#### Impact of Expansion Multiplier.

The impact of the expansion multiplier is presented in Table 8, which shows the accuarcy at different expansion multipliers $\times 1,\times 3,\times 6,\times 10$. De-DA achieves optimal accuracy at a multiplier of $\times 3$ for both Aircraft and Standford Car datasets, while Cub-200-2011 peaks at $\times 6$. The difference likely stems from the distinct data distributions inherent to each dataset. Notably, increasing the expansion multiplier does not necessarily improve performance, a phenomenon also observed in (Trabucco et al., ; Wang et al., ). This suggests that excessive data augmentation may bias the model towards the generated data, hindering its ability to generalize effectively to real data.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * Is the issue that models struggle with distinguishing background and foreground only present in scenarios with relatively limited data, or is this a problem for larger datasets? 
* Can this method be regarded as distilling the knowledge from SAM and diffusion models, which have been trained on large-scale datasets?
* What is the computational complexity of the method when processing images, and is it feasible to apply it to larger datasets?
soundness: 3
strengths: * The method of dividing images into CDP and CIP regions sounds reasonable and the experiment results show its effectiveness.
* The paper demonstrates the effectiveness of the method on different networks and datasets.
summary: This paper introduces a method to decouple an image into its class-dependent part (CDP) and class-independent part (CIP), and subsequently processes these two components individually for data augmentation. Specifically, the method first uses SAM to segment out CDP and CIP. Then, it appropriately augments the CDP part using a diffusion model. After that, it randomly combines the CDP with CIP from different images to increase image diversity.
weaknesses: * The paper only conducted experiments on datasets with relatively limited data. It did not illustrate the effectiveness on larger and more general datasets. Can this method work on larger datasets like ImageNet, similar to how mixup or cutmix do? 
* In the experiments comparing with RandAugment, only the results of “DE-DA” and “DE-DA + RandAugment” are provided. I believe that further results for “RandAugment only” should be included. Because it would be better to demonstrate that the “DE-DA + RandAugment” method outperforms “RandAugment only”.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I am uncertain about the appropriateness of using PSNR to evaluate diversity in Figure 5b. PSNR measures pixel-level deviation, meaning that a simple Mixup operation could also result in low PSNR, but it would be difficult to argue that Mixup inherently promotes high diversity. I would appreciate further clarification from the authors on this point.
soundness: 3
strengths: 1. This paper introduces a simple yet effective method that balances fidelity and diversity in synthetic data. The core idea is to decouple CIPs and CDPs using an off-the-shelf segmentor, augmenting them separately and then combining them to create new samples. This approach presents a methodological innovation compared to prior generative data augmentation techniques. 
2. De-DA utilizes layer-based composition to generate synthetic samples, making the synthesis process highly efficient. This is beneficial in resource-constrained settings,
3. In Table 1 and Figure 2, the authors provide a comprehensive overview of various data augmentation methods from the perspectives of diversity and fidelity, which offers valuable insights for readers aiming to understand advancements in this area.
summary: The paper introduces a data augmentation framework called Decoupled Data Augmentation (De-DA) to tackle the fidelity-diversity dilemma. This approach involves a decoupling strategy that separates images into class-dependent parts (CDPs) and class-independent parts (CIPs) using SAM. The method then applies text inversion and SDEdit to the CDP, and subsequently combines it with randomly selected CIPs to generate new images. This process aims to maintain semantic consistency while enhancing diversity
weaknesses: 1.Although the Online Randomized Combination method is efficient, it may lead to semantically unnatural compositions. For instance, birds in the synthetic samples do not always appear naturally perched on branches (see the last row of birds in Figure 6, where proper positioning on branches is rare). While semantic naturalness may not always be critical for classification, this could reduce the generalizability of synthetic data.

2.The authors should include more visualizations of De-DA results, especially showcasing foreground variations in CIP and the inpainting results for CDP. Additionally, it would be valuable to discuss some lower-quality samples and include a discussion on the limitations of the current approach.

3.Some experimental settings are overly brief. For instance, details regarding the multi-label classification implementation are missing, and the authors provide insufficient explanation on how samples are constructed in this scenario.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: De-DA generates augmented images using SAM and diffusion models. How does it compare with other baselines in terms of efficiency? 

The proposed method is evaluated only on fine-grained classification tasks. Some similar mixing methods are evaluated on object detection and instance segmentation tasks. For example, Copy-Paste augmentation [2] is a low-cost augmentation method that copies and pastes a random object into another image. How does De-DA compare with this baseline in instance segmentation tasks?
soundness: 2
strengths: - The paper is well-written and is easy to understand

- The experimental results are promising
summary: The paper proposes De-DA, which is a framework for sample-mixing and generative data augmentation. Specifically, it decouples the frontal object and background of an image using SAM. During training, De-DA applies generative diffusion models to transform the object and paste it to an extracted background from another image. The method helps to reduce the background noise when transforming the object and shows strong experimental results on fine-grained classification tasks.
weaknesses: Limited Novelty. The approach of using background changes to create augmented data has been studied before. For example, InSP[1] swaps the saliency part of two images from the same class and is tested on CUB, Stanford Car, and FGVC-Aircraft datasets. Copy-paste augmentation [2] is a low-cost augmentation method that copies and pastes a random object into another image, for instance segmentation. Applying textual inversion and SDEdit to transform objects was suggested in DA-Fusion. The SDEdit and SAM models are also off-the-shelf methods proposed in previous works. The proposed idea of transforming extracted objects and pasting them into the backgrounds from other images is somewhat incremental to existing works. 

De-DA proposes to extract class-dependent parts of an image using SAM. If the SAM model is not pre-trained on the target domain, it may fail to capture the class-dependent part, such as medical images. In addition, the class-dependent parts of an image are not limited to the background. For example, in gesture classification, the gesture is also a class-dependent part; in scene classification, the whole image should be considered. It seems that the proposed method cannot capture non-background class-dependent features and is limited to fine-grained object classification tasks.

Limited evaluation of the tested datasets and tasks. It appears that all three tested datasets are fine-grained classification datasets. The backgrounds in fine-grained datasets are relatively easy to transfer as the objects are mostly the same. However, this may not be the case if the object classes are semantically different. For example, swapping the background of a marine animal and a desert animal may not be appropriate. Therefore, I suggest the authors test their method on large-scale datasets with more diverse objects, e.g., the ImageNet dataset. 

[1] Zhang, Lianbo, Shaoli Huang, and Wei Liu. "Intra-class part swapping for fine-grained image classification." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.

[2] Ghiasi, Golnaz, et al. "Simple copy-paste is a strong data augmentation method for instance segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Long-tailed image classification is a natural data-scarce scenario to test the effectiveness of data augmentation methods. I am curious about the paper's performance on long-tailed datasets like classical Places-LT& ImageNet-LT, and CUB-LT & Flower-LT used in Diff-Mix.

2. Without SAM, could other instance segmentation methods achieve similar foreground-background separation? How significant is SAM's impact? I believe this is crucial for demonstrating the robustness of the proposed method. If it relies heavily on SAM, its effectiveness could be significantly diminished, as previous methods did not use additional tools like SAM for foreground-background separation. I am curious to see if the method is still effective under poor separation conditions.

I believe the current version does not meet the standards for acceptance. However, I acknowledge the authors' motivation and their effective use of pre-trained models to address the data augmentation problem. I am willing to increase my score if the authors can adequately address these weaknesses and questions during the rebuttal.
soundness: 3
strengths: 1. The motivation is sound. The paper proposes separating foreground and background before targeted augmentation, aiming to maintain fidelity while increasing diversity. This approach is reasonable and effective, contrasting with previous methods that targeted the entire image.
2. The paper effectively leverages existing technologies like SAM and LayerDiffuse to propose a new data augmentation framework, yielding valid experimental results.
3. Extensive experimental results and open-source code suggest the method is reproducible.
summary: This paper proposes a decoupled data augmentation framework for vision classification tasks, which separately augments the class-dependent part (CDP) and the class-independent part (CIP). Specifically, by employing SAM to initially separate CDP and CIP, the paper then designs a transparency img2img pipeline to augment CDP, inspired by SDEdit and LayerDiffuse, and finally generates synthetic samples by combining the generated CDPs with randomly sampled CIPs. Experiments on various datasets demonstrate the effectiveness of the proposed method.
weaknesses: **Major:**

1. Limited technical contribution. The paper utilizes existing techniques such as SAM, LayerDiffuse, and text inversion. While these technologies are well-utilized to produce seemingly credible results, there is a lack of novel technical contributions and design. Additionally, performance improvements gained by using SAM are unsurprising and come with increased computational costs. Would the method still work without SAM? Are there alternatives to SAM?
2. The writing needs further polish. There are confusing descriptions, particularly in the Method section's Conservative Generation of CDP. For instance, in Eq. 3, the variable $ c $ seems to refer to a text prompt, but its definition is missing. Moreover, as a critical part of augmentation, there is no experimental discussion on it. Also, if Line 233’s $\lfloor T_{s}\rfloor$ indicates a timestep, what does $\lfloor S_{T_{s}}\rfloor$ in Eq. 1 mean? $ S $ is also undefined—perhaps it means total steps? Numerous other issues are noted in the Minor weaknesses.
3. I recommend adding a Background section to provide a brief review of SDEdit and the diffusion model used, along with clear symbol definitions.
4. Experimental results are confusing. For example, the source of the experimental results is unclear—are they reproduced by the authors or cited from another paper? In Table 2, the authors label “training from scratch,” yet the vanilla results for DiffuseMix on ResNet-50@448 are 65.50, 80.29, and 85.52, compared to this paper’s 72.54, 71.53, and 91.32, which are significantly higher than those reported in DiffuseMix’s Table 14. The authors need to clarify the experimental setup and result sources. Additionally, in Table 5, results for Vanilla, CutMix, DA-Fusion, and Diff-Mix are cited from Diff-Mix, but the source for the remaining results is missing.

**Minor:**

1. Inconsistent use of \citet{} and \citep{}. Please verify correct usage in Line 058, Line 061, Line 068, and Line 346.
2. Incorrect citation. The citation for DiffuseMix in Line 315 seems wrong.
3. Incorrect Y-axis label. Why is the metric for multi-label classification in Figure 5(a) PSNR?
4. Line 731 seems incomplete. "For inpainting the missing part of For the training ..." What does this mean?????