IPMix: Label-Preserving Data Augmentation Method for Training Robust Classifiers
Authors: Zhenglin Huang
Xiaoan Bao
Na Zhang
Qingqi Zhang
Xiao mei Tu
Biao Wu
Xi Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: data augmentation
robustness
safety

Abstract: Data augmentation has been proven effective for training high-accuracy convolutional neural network classifiers by preventing overfitting. However, building deep neural networks in real-world scenarios requires not only high accuracy on clean data but also robustness when data distributions shift. While prior methods have proposed that there is a trade-off between accuracy and robustness, we propose IPMix, a simple data augmentation approach to improve robustness without hurting clean accuracy. IPMix integrates three levels of data augmentation (image-level, patch-level, and pixel-level) into a coherent and label-preserving technique to increase the diversity of training data with limited computational overhead. To further improve the robustness, IPMix introduces structural complexity at different levels to generate more diverse images and adopts the random mixing method for multi-scale information fusion. Experiments demonstrate that IPMix outperforms state-of-the-art corruption robustness on CIFAR-C and ImageNet-C. In addition, we show that IPMix also significantly improves the other safety measures, including robustness to adversarial perturbations, calibration, prediction consistency, and anomaly detection, achieving state-of-the-art or comparable results on several benchmarks, including ImageNet-R, ImageNet-A, and ImageNet-O.

IPMix: Label-Preserving Data Augmentation Method for Training Robust Classifiers
Zhenglin Huang', Xianan Bao', Na Zhang': Qingqi Zhang?, Xiaomei Tu Biao Wu', and Xi Yang*
School of Artificial Intellienge, Zhejiang Sei-Tech University Yamaguchi University 2ZGUC University of Science and Technology of China, Hefei, China
Abstract
Data augmentation has been proven effective for training high-accuracy convolu- tional neural network classifiers by preventing overfitting. However, building deep neural networks in real-world scenarios requires not only high accuracy clean data but also robustness when data distributions shift. While prior methods have proposed that there is trade-off between accuracy and robustness, propose IPMíx, a simple data augmentation approach to improve robustness without hurting clean accuracy. IPMix integrates three levels of data augmentation (image-level, patch-level, and pixel-level) into coherent and label-preserving technique to increase the diversity of training data with limited computational overhead. To further improve the robustness, IPMix introduces structural complexity at different levels generate more diverse images and adopts the random mixing method for multi-scale information fusion. Experiments demonstrate that IPMix outperforms state-of-the the-art corruption robustness on CIFAR-C and ImageNet-CC In addition, we show that IPMix also significantly improves the other safety measures, in- cluding robustness to adversarial perturbations, calibration, prediction consistency, and anomaly detection, achieving state-of-the-art or comparable results several benchmarks. Code is available t/gitb. com/ httm/h/ssaberrIMMM.
Introduction
Deep neural network models have recently achieved remarkable performance various computer vision tasks, such as zero-shot image classification [[H3), 3D object detection [4H6], and face recognition [78 In real-world scenarios, models can achieve impressive accuracy when training and test distributions are identical, but challenges appear when confronted with out-of-distributie examples 11) such natural corruptions [12]. adversarial perturbations [ and anomaly patterns 14, necessitating robustness across distribution shifts. Data augmentation has been proposed partially alleviate this issue, which applies diverse transformations
clean images to generate new training examples [5] [6] Furthermore, a high diversity of augmented images enables neural networks to resist data distribution shifts and improve robustness [7] Data augmentation approaches generally fall into three subgroups: image-level, patch-level, and pixel-level augmentations.
Image-level augmentation techniques [1-200 apply transformations the whole image, such as brightness, sharpness, and solarization, to increase the total amount training data. Patch-level aug- mentation techniques [2] [2 typically mask replace region of an image, compelling classifiers to focus on less discriminative portions. Meanwhile, pixel-level augmentation techniques [23 24
mix images using pixel-wise weighted averages to increase diversity within the training dataset.
Conresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

CutOut
Mixup
CutMix
AugMMxx
PixMix
IPMix
Figure I:
Visual comparison of various data augmentation methods. IPMix utilizes the structural complexity of fractals and multi-scale information generate diverse examples.
Previous studies have focused on either pixel-level patch-level information to improve model performance. However, most these techniques are label-variant, which may lead to manifold intru- sion 25 [6] and decrease performance
unseen data. Simultaneously, a limitation of image-level data augmentation techniques is the computationally expensive search for an optimal augmentation policy, often exceeding the training process's ' complexity [18] [9]. Given these considerations and the potential for enhancing data augmentation strategies, we mainly discuss one question in this paper: How take advantage of the strengths of the three methods while avoiding their drawbacks?
Our contributions are as follows:
We propose IPMix, label-preserving data augmentation approach, which integrates three levels of data augmentation into single framework with limited computational overhead, demonstrating that these approaches are complementary and that a unification among them is necessary to achieve robustness.
To further enhance model performance, IPMix incorporates structural complexity from synthetic data at various levels to produce more diverse images. Additionally, we employ random mixing methods and scar-like image patches for multi-scale information fusion.
Extensive experiments demonstrate that IPMix achieves state-of off-the-rt art corruption robustness and improves numerous safety metrics compared with other data augmentation approaches.
IPMix integrates the three data augmentation techniques in label-preserving fashion, effec- tively circumventing potential manifold intru- sion and maintaining label consistency (27). Fur- thermore, inspired by prior work, IPMix climi- nates the need search for an optimal data aug- mentation policy, thus reducing computational costs. By addressing these challenges, IPMix has achieved significant improvements, as de- picted in Figure 2] In comparison to other meth- ods that focus on leveraging one of these cate- gories for enhancement, IPMix achieves state- of-the-art results in accuracy and robustness.
IPM
PieMis
Aughta
Trhailugment
SallencyMix
PuziaMi
AusssAugmoni
Randkugment
Palkev
Pech-ive
mage Ine
CurCui
Cuthis
Geraiin
00.75
01
75
12:00
02:50
Figure 2: The performance of different levels of data augmentation methods on CIFAR- 100. Com- pared other approaches which focus utilizing only one category. IPMix achieves state-of-the-an accuracy and robustness.
Since IPMix involves different levels of data augmentation techniques, it naturally motivates us to design a novel mixing method for bet- ter information fusion. Previous research has demonstrated that enhancing training data diver- sity [23 29 and image structural complexity [30 31 is crucial for improving model robustness. The structural complexity of synthetic data, such as fractals and statistical information, can bolster model performance through pre-training (32] or blending with clean images 24). For better data integration, IPMix mixes clean images with synthetic pictures at different scales by random mixing to improve structural complexity, which can generate more diverse images to improve robustness.
Building on the enhancement of corruption robustness, we further extend IPMix's capabilities to enhance various safety metrics to fulfill the demands of constructing secure and reliable systems in real-world situations [I]. We demonstrate that IPMix improves numerous safety metries, including corruption robustness, calibrated uncertainty estimates, adversarial robustness, anomaly detection, and prediction consistency. On CIFAR- -O-AR--O-CCAR and CIFAR-100-C. IPMix achieves the best results across different architectures. On ImageNet, IPMix outperforms previous methods and gains a substantial improvement on various safety measure benehmarks, achieving state-of-the-a or comparable results on ImageNet-R, ImageNet-A. and ImageNet-O B3 34]

---

Addidion
Muttipiicatian atlin
Parch Stape112
Pixe-bbe
W
posterize
translate-t
equallee
1-m1
Figure 3: Top: Sample fractals from IPMix set. Bottom: An example of IPMix applied dog image, k- 2, = We randomly select P (pixel and patch) data augmentation methods and image- level data augmentation methods to generate highly diverse set of augmented images. We sample We (k: = 2, in this case) from Dirichlet distribution and use skip connection (m sample from a Beta distribution) maintain semantic consistency.
2 Related Works
2.1 Data Augmentation
Data augmentation is crucial to the success of modern neural networks, contributing significantly to the improvement model generalization performance. The presented data augmentation approaches can be classified into three high-level categories: image-level, pixel-level, and patch-level augmentations.
Image-level data augmentation. Image-level data augmentation methods are commonly label- preserving, applying transformations on the whole image to improve data diversity. AutoAug- ment [19 utilizes reinforcement learning to automatically search optimal compositions of transfor- mations. Adversarial Auto AutoAugment 335 generates adversarial images to extend data and produces dynamic policy during training. TrivialAugment (36) randomly selects an operation and the magni- tude to reduce search space and improve performance. AugMix (37] uses multiple transformations to create high diversity of augmented images, achieving state-of-the-ar results on corruption robustness and calibration. AugMax (29] unifies diversity and hardness to search for the worst-case mixing strategy. PRIME uses max-entropy image transformations boost model corruption robustness.
Pixel-level data augmentation. Pixel-level data augmentation methods mix images using pixel-wise weighted averages. MixUp [23] generates augmented images by linearly interpolating between two randomly selected images and their corresponding labels. Manifold MixUp (39] encourages neural networks to learn smooth interpolations between data points the hidden layers, improving accuracy by comparison with MixUp. PixMix (24) utilizes structural complexity synthetic pictures, such as fractals and feature visualizations, to improve model performance. Our work shared similarities with PixMix, but we use multi-scale information and better information fusion methods to train robust models by leveraging more diverse examples.
Patch-level data augmentation. Patch-level data augmentation methods mask or replace parts of the original image with different information. CutOut (28) randomly masks out regions of a clean image to learn less discriminative portions, thereby improving accuracy. CutMix [40) replaces patch of an original image with another randomly picked image to improve performance. Patch Gaussian [41]. which inputs patch of Gaussian noise into the clean picture, combines the improved accuracy of CutOut with the noise robustness of Gaussian. SaliencyMix 442 based on the maximum intensity pixel local in the saliency map, replaces square patch of the original image with salient information from another image. TokenMix (43) improves the performance of vision transformers by partitioning the mixing region into multiple separated parts and mixing two images at the token level. AutoMix (4 optimizes both the mixed sample generation task and the mixup classification task in momentum training pipeline with corresponding sub-networks in bi-level optimization framework.

---

2.2 Safety Measures
When deploying network models real-world scenarios, it is crucial to consider comprehensive security measures beyond standard accuracy. Implementing unsafe machine learning systems in high-stakes environments [45 can lead to incalculable losses. With the rise of multimodal large language models (MLLMs) [88 (88, safety issues are receiving increasing attention because their superior performance still makes mistakes. For example, GPT-4 [49] may be confidently wrong in its predictions and disturbed by adversarial questions. Previous research has proposed various safety measures, including but not limited to robustness and calibration.
Robustness. Corruption robustness considers how to improve the model resistance to unseen natural perturbations under data distribution shifts. As a variant the original ImageNet, ImageNet-C (51) consists of 15 diverse commonplace corruptions belonging to different categories with five levels of severity, regarded as a general benchmark for corruption robustness. In addition to natural corruption, Hendrycks et al. [33] demonstrate that models should measure generalization to various abstract visual renditions. The robustness of adversarial attacks focuses on defending against imperceptible perturbations to images [52). Prior works have proposed that there is trade-off'
between the robustness of adversarial perturbations and clean image accuracy [53] 54. ImageNet-O and ImageNet-A [34], widely regarded as benchmarks for evaluating image classifier performance under shifts in both input data and label distributions, are utilized for anomaly detection.
Calibrated prediction confidences, which indicate whether model's output should be trusted, are valuable for classification models real-world settings. Bayesian approaches (55] are widely used to deal with uncertainty estimation. Kuleshov et al. (56) utilize recalibration methods to solve the miscalibration of credible intervals. Ovadia al. [57] provide a benchmark for evaluating the accuracy and uncertainty of models under data distributional shifts.
Calibration.
2.3 Training with Synthetic Data
Previous works have proved that training with synthetic data can improve performance real datasets. Dehidatta al. [58] discover that combining synthetic annotated datasets with real data can significantly improve the performance of instance detection. Baradad et al. [31] generate synthetic data by utilizing various procedural noise models. In addition, they find that naturalism and diversity two important properties for synthetic data to achieve comparable results with real datasets.
Kataoka et al. [59 60) propose a suite of datasets generated by formula-driven supervised learning.
3 An Attempt to Integrate Existing Approaches
Some prior studies 24 40 have
Table 1: The combination of different levels of data augmen- tation. M, C and A are abbreviations for MixUp, CutMix, and AugMix, respectively.
suggested that combining different data augmentation techniques with existing methods can improve accu- racy on standard datasets. However, these works merely employed sim- ple combinations without consider- ing the compatibility between meth- ods at different levels. Simultane- ously, these studies chose the clean accuracy as the sole evaluation met- ric and have not taken the model's safety performance into account. In this section, select MixUp 223] CutMix [40], and Aug- Mix [37] as representative data augmentation approaches for pixel-level, patch-level, and image-level, respectively, to conduct combination experiments of these approaches on CIFAR-100. Please refer to Appendix F for more details about the combination experiments.
Classification Error({)
Robustness mCE(L)
Calibration RMS(L)
Vanilla
21.3
50
14.6
+M
20.5 (-0.8)
45.9(-4.1)
10.5(-4.1)
+M+C
20.2 (-1.1)
46.1(-3.9)
22.7(+8.1)
+M+C+A
23.4 (+2.1)
50.1(+0.1)
25.6(+11)
Results on Table [ demonstrate that simply combining different data augmentation methods may significantly impair model performance. This could be attributed to the excessive perturbation of training data caused by the combination of these methods, making the newly generated samples more challenging identify and impacting the model's ability to learn useful features, leading to performance degradation. When multiple label-variant methods are combined, manifold intrusion

---

P augmentation operation
Image augmentation operation
Original image
Mixed image
Chaln-Mixed
Linear Míx
Mixed-input
Figure 4: Different mixing framework of IPMix. augmentation operation represents pixel-level and patch-level augmentation operations. ! Utilizing P operations and image-level operations in different chains and mixing the results. @ A clean image randomly carried out by P operations or image-level operations in linear combinations to generate IPMix image. 3 leveraging the mixed image as new input.
issues may more likely to arise. One possible solution for better information integration to incor- porate approaches (e.g., MixUp) into rearch-based data augmentation techniques (2) (3).. However, searching the space for an optimal DA policy will bring expensive computation. Furthermore, this approach aims improving clean accuracy and does not consider the overall safety performance.
IPMix: A Simple Method for Training Robust Classifiers
In this section, propose IPMix, which integrates three levels of data augmentation methods into a label-preserving approach, comprehensively improving safety metries without sacrificing clean accuracy. We first demonstrate how to merge various techniques into coherent framework and then propose novel approaches to achieve superior information fusion.
4.1 Integrates Different Levels into Coherent Approach
Pixel-level & Patch-level. As label-preserving data augmentation approach, IPMix uses the equation below to mix two input images:
B o 2 + + B)
(1)
Where Vi the input image and 22 represents an unlabeled synthetic image (e-g., fractals, spectrum, or auto- sutt-geenerated contours). B is a mask matrix suitable for both patch-level and pixel-level data augmentation methods, and is a binary mask filled with ones, having the same dimensions as B. c represents the element-wise product. When performing mixing operations at the patch level, choose a patch of random size and position from B, with a value of À (sample from Beta distribution) in this range and value of in other areas, which ensures that except for the mixing patch, the rest of the generated image comes from T1. When performing mixing operations at the pixel level, we treat the entire image patch, with value of X. To make it efficient, adopt fractals representatives of synthetic data. However, IPMix is insensitive to mixing sets change, as shown Table g
Fractals are geometric shapes with structural complexities and natural geometries. While previous works [32 [6]] merely use iterated function systems (IFS) to create fractal data, we employ the Escape-time Algorithm for generating "orbit trap" complex fractals to enhance dataset complexity and diversity. Please refer to Appendix for details about generating fractal images.
The above-described method provides two key advantages: (1) We utilize a simple approach to combine operations of two levels, facilitating better information fusion. (2) Our method is label- preserving, ensuring it is not affected by manifold intrusion while eliminating the need for label smoothing 622 In the following sections. we refer the method used in Eq. II as P-level data augmentation, signifying the employment of both patch-level and pixel-level methods.
Image-level. IPMix leverages various augmentation techniques and compositions to create image that does not deviate significantly from the original. Drawing inspiration from previous works (36] 37). randomly sample operations from PIL (e.g., brightness, sharpness) and randomly sample strengths to enhance the diversity of training data without expensive searching. Notably, these operations are disjoint from ImageNet-C corruptions, ensuring the robustness test's validity.

---

The IPMix framework. To deter- mine the most effective methods for combining P-level and image- level, we conducted experiments using different mixing structures to generate diverse set of IPMix images, illustrated in the Fig- ure and Table While Linear Mix achieves excellent results in clean accuracy and corruption ro- bustness, it performs poorly in cal- ibrated prediction confidence. Mixed Input performs better calibration but is inferior in accuracy and corruption robustness compared to Chain-Mixed. Consequently, chose Chain-Mixed as the default framework for IPMix. Furthermore, the experimental results highlight the potential of establishing general framework for integrating various data augmentation methods.
Table Results are reported on CIFAR-100 and CIFAR-100- with ResNeXt-299 The Chain-Mixed achieves the most balanced result on these metries. Bold best.
Classification Error({)
Robustness mCE(()
Calibration RMS(L)
Chain-Mixed Linear Mix
18.3
28.1
3.8
18.2
27.4
13.5
Mixed Input 19.8
29.6
3.6
4.2 Multi-scale Information Fusion
IPMix can enhance the diversity and the structural complexity of training data to improve model performance. However, we found that simple mixing methods restrict the model's capabilities. To overcome this issue, we use random mixing and scar-like image patches for achieving more effective information fusion.
Random mixing, In previous data augmentation works, is typical either linearly mix two images or extract specific image features, such as saliency [22 42 which requires addi- tional computations, for image mixing. IPMix incorporates various levels of operations. its objective is to enhance the mixing of images, ultimately increasing data diversity. To accomplish this objective, IPMix employs four mixing oper- ations: addition, multiplication, random pixel mixing, and random element mixing (63). Random pixel mixing creates binary mask of size x W that operates on each chan- nel sequentially, while random element mixing generates binary mask size x W x 3 (RGB) that applies to channels simultancously. An example is shown in Figure 5 The experiments in Appendix B. show that both operations beneficial better information mixing between images and fractals.
Random Pixels Mising
Random Elements Mixing
Figure 5: Top: mixing operations. Bottom: Ex- amples of IPMix-Seat mixing and IPMix IPMix-Square mixing.
Examples of random
Scar-like image patches. IPMix-Scar employs a long, thin rectangular box filled with an image patch enhance dataset diversity, which has proven effective for anomaly detection [64). An example of patch mixing is illustrated in Figure First, IPMix randomly selects point and a sear or square of the previously chosen size from the current image. Next, IPMix crops corresponding portions of the current image and the fractal picture and combine them.
Finally, we obtain IPMix, which employs various levels of data augmentation create diverse transformations with image structural complexity and data diversity. Figure 3 displays an example of IPMix, where denotes the number of augmented chains, and represents the maximum number of times an image can be augmented. The algorithm of IPMix is summarized in Appendix D.
5 Experiments
In this section, we showcase the significant performance improvements brought by IPMix on clean datasets in multiple settings. We present the evaluation results of IPMix for image classification on three datasets CIFAR-IIAOO 10, CIFAR- 100 [65], and ImageNet [66] -6rossc across various models. Besides clean Classification, we assess IPMix on diverse safety tasks, including adversarial attack robustness, corruption robustness, prediction consistency, calibration, and anomaly detection. Please refer to Appendix C for details about the evaluation metrics. Lastly, we evaluate the properties of IPMix in thorough ablation studies and compare approach with different levels of methods.

---

Table 3: Clean Error for IPMix on CIFAR-10 and CIFAR-100, lower is better. Top CIFAR-10.
Bottom CIFAR- 100. Mean and standard derivation over three random seeds is shown for each experiment. Bold is best.
Vanilla
MixUp
CutOut
CutMix
AugMix
PixMix
IPMix
WideResNet4O-4
4.4
3.8
3.6
4.0
4.3
4. (240000
4.0g (=0.00)
WidcResNet28-10
3.8 (.0
3.6(40.08)
3.4
3.4
3.4 3.4000.07)
3.8 18.
3,3 (=0.08)
ResNeXt-299
4.3 (2000)
3.8 11111
4, 4.2(20008)
3.8 3,811 02)
4.2 4.2
3,8 (10.00)
3.8 (10000)
ResNet-18
4.400)
4.2,10.04)
4.1.10.03)
4.001)
4.5(10.03)
(.0(00)05)
4.2
Mean
4.2
3.9
3.8
3,8
4.1
4.0
3,8
WideResNer40-4
(10.11
10.5
11.1
20.3(11.15))
20.6
20.47
19.4
WideResNer28-10
19.0
(2.4
(80
=8.0
(9.4
(8.3
(7.4(+0.25)
ResNeXt-29
20.4 (2011)
(20.11)
190
19.5
(0.4
20.1 20.1
18.3 (8.3(20022)
ResNet-18
23.7
21.0(=0.07)
22.0()
20.8, (=00
(3.0(+0.14)
21.6(=0.15)
21.6(+0.28)
Mean
21.1
20.0
20.1
19.7
20.8
20.1
19.2
Table 4: Mean Corruption Error (mCE) for IPMix across architectures on CIFAR-10-C and CIFAR- 100-C, lower is better. Top: CIFAR- 10-C. Bottom CIFAR-100-C. Bold is best.
Vanilla
MixUp
CutOut
CutMix
AugMix
PixMix
IPMix
WideResNNer4044
26.4(10.14)
20,10.15)
25.9(10.1))
26
10(10.10
9.5(10.14)
8.6 14)
WideResNer28-11
10.2
(9.2(=0.17)
23.5(20.17)
9.1 9.1
8.7(40.4)
7.5
ResNeXI-29
27.5
23.6(20.18)
(2.3
28.5
11.3
9,2(20.12)
8.6
ResNet-18
25(20.00)
20(=0.15)
(4.1
(4.7(#0.19)
10.4(#0.1))
(=0.11)
8.4(+0.17)
Mean
25.8
20.9
25.2
26
10
9.1
8.2
WideResNer40-4
10.10.15)
(209
(1.5(20.17)
50(10.19)
(30
(1.1
28.6(=0.155)
WedcResNet28-10
(8.5(+0021)
(a.18)
(8.2
31.5 (+05() (8.6(+0)21)
(+.3
(-0.29)
ResNeXt-29
(1.4(20.1))
47.9 (20.1)
(1
(2.4(+0
34. (+0.24)
30.6(20.28)
28. (=0.31)
ResNet-18
10(20.18)
45.5 (20022)
(2.2
(2...4))
(500.2))
31.4(20.29)
29.9 (9.9 (=2.29)
Mcan
50
45.9
50.2
50.5
33.4
30.3
28.3
We evaluate IPMix on CIFAR-IO-C, CIFAR-100-C, and ImageNet-C to measure its resistance to cor- ruption data shifts. We test IPMix on CIFAR- 10-P, CIFAR-100-PP and ImageNet-PP to measure network prediction stability against minor perturbations. To thoroughly demonstrate our method's 's capabili- ties, we assess on supplementary datasets, including ImageNet-R, ImageNet-O, and ImageNet-A, Experiments on these datasets validate our approach's 's robustness under real-world distribution shifts.
5.1 Evaluation on CIFAR
We experiment with different backbone architectures on CIFAR-IO and CIFAR- including 40-4 Wide ResNet [67]. 28-10 10 Wide ResNet, ResNeXt-29 66), and Resnet- 18 [69]. We compare IPMix with various data augmentation methods, including CutOut, Mix CutMix, AugMix, and PixMix. Please refer to Appendix A for more details about the training configurations.
Accuracy. Table 3 we demonstrate that IPMix improves standard accuracy across architectures. In comparison with other approaches, IPMix achieves the best or comparable accuracy, showing the improvement of safety measures is not at the cost of hurting clean accuracy.
Corruption robustness. Results show that IPMix substantially improves corruption robustness across architectures. Compared AugMix on CIFAR-I00-C, IPMix achieves 4.7% (40-4) and 4.9% (28- 10) improvement on WideResNet, 6% on ResNeXt, and 5.1 % on ResNet. Table demonstrates that IPMix achieves state-of-the-- results on both CIFAR-10-C and CIFAR-100-C
Calibration, We utilize RMS calibration error 70 to evaluate the empirical frequency of correctness. As depicted in Figure 6. IPMix surpasses other methods, achieving state-of-the-an results.
Prediction consistency. We leverage the mean flip rate (mFR) evaluate prediction consistency on CIFAR- 10-P and CIFAR-I00- [51). IPMix achieves the lowest mFR, as shown in Figure 7
7

---

Vanilla
MixUp
CutOut
CutMixx
AugMix
PixMix
IPMix
Acouracy
RME--168
RMS-1055%
RMS-1147%
RM4S-4 3%%
10MS-6.3
RMS-777%
RMS-2.8%
Confidence
Figure 6: The results of calibration CIFAR-100. IPMix achieves the lowest RMS error in all data augmentation methods, improving 11.8% by comparing with Vanilla.
Prediction Consisteney
Adversarial Robustness
Adversarial error rate(
Mean Flip Rate (%)
I6TT
10
96
3.6
36
13
81.4
Baseline MixUp CatOut CutMix AueMix PixMix JPMix
Baseline MixUp CutOat CutMix AugMix
PixMis IPMix
Figure 7: Left:
prediction consistency. Right: adversarial robustness. IPMix achieves the best
results both metries, demonstrating its ability improve overall security performance.
Adversarial robustness. This measure evaluates the resistance of adversarially perturbed by projected gradient descent. We utilize PGD 7 to verify the adversarial robustness of image classifiers. The results in Figure show that IPMix achieves the lowest error.
Table 5: The results of IPMix on ImageNet. For Anomaly Detection, we test the accuracy on ImageNet-A and AUPR on ImageNet-O. higher is better. IPMix achieves round improvement over various data augmentation methods. Bold is best, and underline is second.
Classification
Robustness
Consistency
Calibration
Anomaly Detection
Clean Emoe(_)
ImageNet-C mCD(()
ImageNet-R [moo(()
ImageNet-P ml/(())
R
A RMS())
ImageNet-A Clussificatio(()
ImageNet-O AUPR(1)
RMS(J)
RMS())
Vanilla
23.9
78.6
64
57.7
19.9
47
2.2
16.2
MixUpp2 23
22.7
76.5
62.4
54.6
9.3
41.7
49.3
5,2
16.1
CurDut 28
CurMix 40
22.9
77.2
66.5
58.1
9.6
44.2
48
7.2
16.5
AngMix
22.6
3.8
AugMax 29
22.9
67.4
62.1
54.6
12.1
44.7
3.9
17.1
PixMix 24
22.4
65.4
59.8
50.8
12.3
44
5.9
17.3
[PMix
22.2
57.4
48.5
5.2 Evaluation on ImageNet
For ImageNet experiments, we compare different data augmentation methods, including MixUp. CutOut, CutMix, AugMix, AugMax [29], and PixMix. We utilize SGD optimizer with an initial learning rate of 0.01 to train ResNet-50 for 180 epochs following cosine decay schedule. Please refer to Appendix A for more details about the training configurations.
IPMix achieves state-of-the-art or comparable performances on a broad range of safety measures, as shown in T Compared with other methods, IPMix improves the resistance of out-of-distribution shifts without reducing clean accuracy. On corruption robustness, IPMix outperforms Vanilla by 15.6% and AugMix by 5.5%, achieving state-of-the -fe-o---thhttt art results. On ImageNet-Re R, IPMix demonstrates the ability to improve rendition robustness, increasing by 6.6% by comparison with Vanilla. ImageNet-P, IPMix improves mFR by 9.2% over Vanilla and 2.3% over PixMix. On calibration tests, IPMix surpasses all methods on ImageNet-C, ImageNet-R, and ImageNet-A, improving RMS by 0.1 % 5.1 %, and 13.5% by comparison with the second-best approach. Furthermore, IPMix achieves convincing results on I and ImageNet-O, demonstrating exceptional ability in anomaly detection. The results demonstrate that IPMix roundly improve safety metrics.

---

Table 6: Ablation results of different components of IPMix on ImageNet with ResNet-50.
Classifiationn
Rohustness
Consistency
Calibration
Anomaly Detection
ImageNNet-C mCE())
ImageNat-R Error(])
ImageNet-P mFR(()
R RMS())
ImaggeNct-A Classilication()
ImageNet-O AUPR(T)
Error(])
RMS(()
RMS(.)
weo patch
22.8 22.3)
65, (esuuo
240.11
49. (9.au)
7.
2.400008)
31. (==00))
61e0.01
17.7
pixe!
(2n.55)
(10.00
8.2
w/o
23.5
66 (6.2
59.5
49.6)eau
8.rea.
33.5(-015))
17.8 170.00
5.3 Ablation Study
In this paragraph, we evaluate the properties of our approach by abla- tion experiments. We first study the influence of different parts of IPMix on performance and then assess the stability of IPMix under various mix- ing sources. Please refer to more ab- lation experiments in Appendix B.I.
Table 7: Ablation results different components IPMix on CIFAR-IO0 Mean and standard derivation over three random seeds shown for each experiment. Bold is the best.
Classification
Robustness
Calibration
IPMix
19.4
28.6
2.8
w/o patch
19.7 (=0.13)
30 (#0.21)
4.6 (+0.07)
Components IPMix. In this sec- tion, we evaluate the influence of dif- ferent IPMix components on perfor-
w/o pixel
19.6 (10009)
33 (10.35)
8.2 (10011)
w/o image
20.1 (+0.27)
34 (#0.65)
8.6 (+0.21)
mance. We execute ablation experiments on the three primary IPMix constituents: image-level, patch-level, and pixel-level. The results show the indispensable contribution of each component to enhancing model performance, demonstrating that these approaches are complementary and that a unification among them is necessary to achieve robustness. The ablation experiment results are shown in Table and Table 7 Please refer to thorough analysis in Appendix J.
Mixing sources. The excellent per- formance of IPMix is partly due to the structural complexity of fractal pictures. In this part, we examine the sensitivity of IPMix to different fractal sources on CIFAR- 100. We report clean accuracy, corruption ro- bustness, and calibration from differ- ent sources with WRN40-4. Fractal + FVis is the default setting of PixMix, which consists of fractals and fea- ture visualization. FractalDB [59) consists of fractal images generated by Iterated Function System (IFS).
Table 8: Ablation results on IPMix across different mixing sets. The results show that IPMix is insensitive to mixing sets change.
Classification Error(L)
Robustness mCE(L)
Calibration RMS(()
Mixing sets
Fractal + FVis
19.4
28.8
3.3
FractalDB
20
29
5.4
RCDB
19.5
28.4
3.2
Dead Leaves
19.4
29.1
3.1
Spectrum
19.8
29.2
4
fractals(ours)
19.4
28.6
2.8
RCDB [60] consists of auto-generated contours. Dead Leaves and Spectrum generated from genera- tive image models (31). The full results show in Table
5.4 The Comparison with Different Levels of Method
In this section, perform extensive performance comparison between IPMix and range of existing methods using multiple metrics. We consider AutoAugment, RandAugment, and Triv- ialAugment (36] representative image-level techniques, while SaliencyMix, PuzzleMix (72). and Co-Mixup
22 serve as typical patch-level techniques. For pixel-level methods, Manifold Mixup stands representative choice. IPMix does not require searching for the optimal DA policy like lmaee-level techniques. In contrast to patch-level approaches, IPMix eliminates the need for saliency computations. The results in Table 9 show that IPMix outperformed all other methods all metrics.
Analysis of IPMix
IPMix combines three levels of data augmentation into unified, preee-pren technique to improve model performance. We believe that IPMix's 's superior performance is due to the increased

---

Baseline
A AugMix
Pix
CutMix
IPMix
Image
Baseline
A ugMix
PixMix
CutMix
IPMix
Figure 8: Top: t-SNE visualization. The features from the penultimate layer of a WRN40-4 trained on CIFARIO. Compared with other approaches, IPMix has distinct boundaries between different category clusters and generates diverse samples cover boundary areas, thereby improving the generalization ability. Bottom: The Grad-CAM visualization, with input images sourced from ImageNet-A, demonstrates that IPMix excels in identifying objects within complex scenarios.
Table 9: Results of different augmentation methods CIFAR-100 and CIFAR-100-C with 28-10
Wide ResNet. Bold best.
Classification Error(()
Robustness mCE(L)
Adversaries Error(()
Consistency mFR(L)
Calibration RMS(U)
Methods
AutoAugment
19
17.7 (#0.11) (20.(1)111)
38.4 38.4(40.13)
97.8(=0.22)
8(400.6)
(.9
RandAugment
20
17.8 17.8(10.14)
(1.5
96.6 (50.25))
8.6(20.10)
7.9 (=0.04)
Augment
36
17.9 (+0.13))
96.3(40.21)
35.4 (5.4
(.3.0))))
8.7 (=0.01)
SaliencyMix
[42
18.3
(8.3(1).24)
(6.7(11.21)
10.8(11.07))
7.1
PuzzleMix
722
18.1 (=0.11)
(7.9(1
96. (=0.23))
10.5 (+:.04)
7.5 (+0.88)
Co-Mixup [22
18.0 (8.0(21..1))
35.6(10.25)
95.6(10.21)
10.1 (4..00)))5)
7.7 (.7.00)
Manifold Mixup (39
18.3 8 (20.21)
51.3
93.4(=0.17)
29.9 (10.28)
10.2
IPMix
17.4 (0.25))
26.6(10:29)
91.3 (20.21)
(.0.11)
6.4(20007)
data diversity and enhanced regularization effect. For more intuitive demonstration of these effects, we utilize t-SNE and Class Activation Mapping (CAM) 73 for visualizations, as shown in Figure R
Increasing diversity. IPMix increases the diversity of training data by mixing data multiple levels, enabling the model to learn a greater variety feature combinations and patterns. Furthermore, the integration of synthetic data from distinct distributions (e.g., fractals), further amplifies this diversity.
Enhanced regularization effeett The approach of mixing data also serves potent regularization technique. By randomly mixing samples, the model is compelled to learn more robust features rather than overly relying on specific sample or class characteristics, which reduces the risk of overfitting and enhances the model's performance in different environments.
Conclusion
We propose IPMix, which leverages different levels of augmentation techniques and image structural complexity to improve model performance. By employing random mixing methods, we facilitate more effective information fusion. The experimental results indicate that IPMix can significantly improve various safety metrics. We hope our work will attract attention to joining different methods into coherent and synergetic approaches to improve robustness and other safety measures. This adaptation is crucial given the growing importance of safety requirements in systems design.
Acknowledgement
This work under the help of Xi Yang. We thank him for his selfless help and valuable suggestions.
10

---

References
[1] X. Wang, Chen Chen, Yuhu Cheng, and 7. Jane Wang. Zero-shot image classification based deep feature extraction, IEEE Transactions on Cognitive and Developmental Systems, 10:432 444, 2018.
[2] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18102 181128122222 2021.
[3] Xiuye Gu. Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. International Conference on Learning Representations, 2021.
[4] Daniel Maturana and Sebastian A. Scherer. Voxnet: A convolutional neural network for real-time object recognition, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922-928, 2015.
[5] Binh Yang. Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7652-7660) 7660, 2018
[6] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3144-3153, 2021.
[7] Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. Proceedings of the 25th ACM international conference on Multimedia, 2017.
[8] H. Wang. Yitong Wang, Zheng Zhou, Xing Ji, Zhifeng Li, Dihong Gong, Jin Zhou, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5265-5274, 2018
[9] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, 2019.
[10] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag. Edoardo Debenedetti, Edoardo Debenedetti, Mung Chiang. Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. ArXiv, abs/2010.09670, 2020.
[11] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. ArXiv, abs/2109.13916. 2021.
[12] Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and corruptions in natural corruption robustness. ArXiv, abs/210..112733 2021.
[13] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. ArXiv, abs/1702.04267. 2017.
[14] Mahmoud Said Elsayed, Nhien-An Kh-Ka Soumyabrata Dev, and Anca Delia Jurcut. Network anomaly detection using Istm based autoencoder. Proceedings of 16th ACM Symposium on Qo5 and Security for Wireless and Mobile Networks, 2020.
[15] Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Tiu. Keepaugment: A simple information- pelormation-preservingg data augmentation approach. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1055-1064, 2020.
[16] Dan Andrei Calian, Florian Stimberg, Olivia Wiles, Sylvestre-Alviss Rebuffi, Andr'as as Gyorgy, Timothy Mann, and Sven Gowal. Defending against image corruptions through adversarial augmentations. ArXiv, 2021.
[17] Adarsh Subbaswany, Roy J. Adams, and Suchi Saria. Evaluating model robustness and stability to dataset shift. In International Conference Artificial Intelligence and Statistics, 2021.
[18] Sungbin Lim, Ildoo Kim, Taesup Kim, Chihcon Kim, and Sungwoong Kim. Fast autoaugment. Neural Information Processing Systems, 2019.
[19] Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation strategies from data. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 113 113-123, 13-112,,2111 2019.

---

[20] Ekin Dogus Cubuk. Barret Zoph, Jonathon Shiens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced search space. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1008-3017, 2019.
[21] Ryo Takahashi, Takashi Matsuhara, and Kuniaki Uchara. Ricap: Random image cropping and patching data augmentation for deep cnns. In Asian Conference on Machine Learning. 2018.
[22] Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. ArXiv, abs/2102.03065, 2021.
[23] Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ArXiv, abs/ 2017.
[24] Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang. Bo Li. Dawn Xiaodong Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16762-16771, 2021.
[25] Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regularization. In AAAI Conference on Artificial Intelligence, 2018.
[26] Raphael Baena, Lucas Drumetz, and Vincent Gripon. Preventing manifold intrusion with locality: Local mixup. ArXiv, abs/2201.04368, 2022.
[27] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying com-suupreiseed learning with consistency and confidence. ArXiv, abs/2001 007655, 2020.
[28] Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout, ArXiv, abs/1708 005520 2017.
[29] Haotao Wang. Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. In Neural Information Processing Systems, 2021.
[30] Raphael Gontijo Lopes, Sylvia J. Smullin, Ekin Dogus Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. ArXiv, abs/2002.08973, 2020.
[31] Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking noise. In Neural Information Processing Systems, 2021.
[32] Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images. International Journal of Computer Vision, 130:990 1007, 2021.
[33] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli. Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5320-8339, 2020.
[34] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Xiaodong Song. Natural adversarial examples. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15257-15266, 2019.
[35] Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhaobai Zhong. Adversarial autoaugment. ArXiv, abs/1912. 11188, 2019.
[36] Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 754-762, 2021.
[37] Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ArXiv, abs/1912.02781, 2019.
[38] Apostolos Modas, Rahul Rade, Guillermo Ortiz- Jiménez, Seyed- Frossard. Prime: A few primitives can boost robustness to common corruptions. In European Conference on Computer Vision, 2021.
Seyed-Mohsen Mohsen Moosavi-Dezfode and Pascal
[39] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, 2018
12

---

[40] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6022-6031, 2019.
[41] Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin Dogus Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. ArXiv, abs/ 1ed.192060261... 2019.
[42] A. F. M. Shahab Uddin, Mst Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bac. Salieneymix: A saliency guided data augmentation strategy for better regularization. ArXiv, abs/2006.01791, 2020.
[43] Jihao Liu, B. Liu, Hang Zhou, Hongsheng Li, and Yu Liu. Tokenmix: Rethinking image mixing for data augmentation in vision transformers, In European Conference on Computer Vision, 2022.
[44] Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Jianzhu Guo, and Stan Z. Li. Automix: Unveiling the power of mixup for stronger classifiers. In European Conference on Computer Vision, 2021.
[45] Samuel G. Finlayson, Isaac S. Kohane, and Andrew Beam. Adversarial attacks against medical deep learning systems. ArXiv, 1bs/00004, 2018.
[46] Nicolas Papernot, Patrick Mcdaniel, Arunesh Sinha, and Michael Wellman. Sok: Security and privacy in machine learning. 2018 IEFE European Symposium on Security and Privacy (EuroS&P) pages 399-414, 2018.
[47] Ludmila Skaf, Elvira Buonocore, Stefano Dumontet, Roberto Capone, and Pier Paolo Franzese. Applying network analysis to explore the global scientific literature on food security. Ecol. Informatics, 56: 101062, 2020.
[48] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorek, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. ArXiv, abs/2302. 110223222.4444, 14045, 2023.
[49] OpenAL. Gpt-4 technical report. ArXiv, /tbb///33.00774,, 2023.
[50] Yongliang Shen, Kaitao Song, Xu Tan, Dong Sheng Li, Weiming Lu, and Yue Ting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and friends huggingface. ArXiv, abs/2303.11580 2023.
[51] Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. ArXiv, abs/1903.1221 12261, 2018.
[52] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9185-9193. 2017.
[53] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Loddon Yuille. and Quoc V. Le. Adversarial examples improve image recognition. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 316-825, 2019.
[54] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Loddon Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 501-509, 2018
[55] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ArXiv, 0bs/1999.44599 2017.
[56] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. ArXiv, abs/1807.00266 2018
[57] Yaniv Ovadia, Emily Fertig, Jie Jessic Ren, Zachary Nado, D. Sculley, Scbastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's 's uncertainty? evaluating predictive uncertainty under datasct shift. ArXiv, abs/1906.02553. 2019.
[58] Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance detection. 2017 IEEF International Conference on Computer Vision (ICCV), pages 1310-1319, 2017.
[59] Hirokatsu Kataoka, Asato Matsumoto, Ryosuke Yamada. Yutaka Satoh, Eisuke Yamagata, and Nakamasa Inoue. Formula-drise driven supervised learning with recursive tiling patterns. 2021 IEEE/CVF International Conference Computer Vision Workshops (ICCVW), pages
2021.
13

---

[60] Hirokatsu Kataoka, Ryo Hayamizu. Ryosuke Yamada, Kodai Nakashima, Sora Takashima, Xinyu Zhang. Edgar Josafat Martinez-Noriega, Nakamasa Inoue, and Rio Yokota. Replacing labeled real-image datasets with auto-generated contours. 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 21200-21209 2022.
[61] Connor Anderson and Ryan Farrell. Improving fractal pre-training. 2022 JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2412-2421, 2021.
[62] Rafacl Müller, Simon Korbblith, and Geoffrey Hinton. When does label smoothing help? In Neural Information Processing Systems, 2019.
[63] Cecilia Summers and Michael Dinneen. Improved mixed-example data augmentation. 2019 JEEE Winter Conference on Applications of Computer Vision (WACV), pages 1262-1270, 2018.
[64] Chun-Liap Li, Kihyuk Sohn, Jinsung Yoon, Tomas Pfister. Cutpaste: Sule-supe learning for anomaly detection and localization. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2021.
[65] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[66] Deng, Dong. Richard Socher, Li-Jia Li. K. and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Puttern Recognition, pages 248-255, 2009.
[67] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv, abs/1605.077 071500714,,, 2016.
[68] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transfor- mations for deep neural networks. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 5987-5995, 2016.
[69] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015.
[70] Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. ArXiv, abs/1 112./181,04466, 2018.
[71] Aleksander Madry, Alcksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ArXiv, abs/1706.0600// 7606666.33 2017.
[72] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. ArXiv, abs/2009.06962, 2020.
[73] Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basse based localization. International Journal of Computer Vision, 128:336-359, 2016.
14


Reviews:
[{"review_id": "XE4LdX2lrk", "reviewer": "Reviewer_FKxz", "summary": "This paper presents IPMix, a new data augmentation strategy that enhances robustness without compromising clean accuracy. The proposed approach diversifies the training data with limited computational overhead. IPMix introduces structural complexity at different levels to generate a wider variety of images and employs a random mixing method for multi-scale information fusion to bolster robustness.", "questions": "- There seems to be no discussion on Table 1, which feels disconnected from the rest of the paper. Could the authors elaborate on this?\n- In Section 4.1 \"The IPMix framework\", the terms \"Chain-Mixed\", \"linear mix\", and \"mixed input\" are not clearly defined. Could the authors clarify these and possibly include the corresponding framework name in Figure 4 alongside the number?\n- In Section 3, integration experiments are described. It is suggested that combining previous approaches may result in diminished performance, as the optimal parameter for using each data augmentation independently may not be optimal when used in combination. Could decreasing the augmentation strength of each method yield better performance?\n- Table 6 suggests that image-level data augmentation is the most crucial part of CIFAR100. Is this observation consistent with ImageNet as well?\n- How can the advantages of the three methods be leveraged while avoiding their drawbacks? Could the authors provide more specific information about the drawbacks of each method?\n", "limitations": "The authors discuss the limitation and societal impact in Appendix.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses augmentation from both a clean accuracy and safety perspective, which is commendable.\n- The paper is generally well-written and easy to follow.\n- The figures provided in the paper are informative and well-polished and enhance the understanding of the proposed strategy.\n- The paper includes many implementation details and ablation studies in Appendix.", "weaknesses": "- The paper could benefit from including error bars to account for variance across different seeds, particularly when the performance of each method is comparable.\n- The paper could provide a more in-depth analysis of why the proposed method improves each safety metric. While the explanations regarding diversity and regularization effects are reasonable, it would be beneficial to understand which part of the pipeline is most crucial for a specific safety measure (e.g., robustness, calibration). The significance of increasing structural complexity using \"fractals\" could be discussed more.\n- The title emphasizes the \"label-preserving\" property, yet it is unclear how vital this is. The authors could discuss the potential outcomes of mixing two images with different labels and linearly interpolating the label.\n- The experimental focus is primarily on ResNet variants. Including results on the performance of other architectures (e.g., Transformer) could strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed reviews and insightful comments! \n\nW1: Your suggestion is incorporated in the new version of the paper. Thank you.\n\nW2-1: \"which component is vital for a specific safety measure.\"\n\nThank you for your concerns. We will detailed analyze the impact of each part on different safety metrics through ablation experiment on CIFAR-100. \n|     | Classification | Robustness | Adversaries | Consistency | Calibration |\n|---|------|----|----|--|--|\n| IPMix | **19.4 $\\pm $0.17**| **28.6 $\\pm $0.2**  | 89.4 $\\pm $0.18   | **4.3 $\\pm $0.09**                | **2.8 $\\pm $0.07** |\n| w/o Patch| 19.7 $ \\pm $0.13  | 30 $ \\pm $0.21| 91.7 $ \\pm $0.15 | 4.7   $ \\pm $0.02             | 4.6 $ \\pm $0.07|\n| w/o Pixel|19.6 $ \\pm $0.09| 33 $ \\pm $0.35 | 92.6 $ \\pm $0.20 | 5.2 $ \\pm $0.05               | 8.2 $ \\pm $0.12|\n| w/o Image |20.1 $ \\pm $0.27    | 34 $ \\pm $0.65 | **87.8 $ \\pm $0.22**  | 5.5 $ \\pm $0.11               | 8.6 $ \\pm $0.21|\n\n**Accuracy**: The **image-level** augmentation has the most substantial effect on accuracy, aligning with current findings[1][2] that image-level methods are commonly used to boost accuracy.\n\n**Robustness**: Both **pixel-level** and **image-level** augmentations improve robustness. Since pixel-level introduces fine-grained variations for pattern recognition, while image-level increases dataset diversity, preventing the model from merely memorizing fixed augmentations.\n\n**Calibration & Consistency**: The **Image-level** part significantly influences calibration and consistency, which increases diversity to improve the prediction calibration across scenarios and ensures consistency in responses to minor perturbations.\n\n**Adversarial Attacks**: Without the **image-level** component, adversarial performance improves, implying diverse data might weaken defense against attacks. Conversely, removing **pixel-level** methods will degrade adversarial robustness, given their inherent resistance to perturbations.\n\nW2-2: The significance of using \"fractals.\" \n\nIn recent visual research, given the challenges and cost of acquiring labeled data, there's a growing emphasis on synthetic data generation as an alternative to real data. In this field, the majority of research focuses on pre-training with synthetic data and then fine-tuning for downstream tasks to reduce reliance on costly labeled data. Notably, [3] emphasize that both diversity and **structural complexity** are pivotal for better performance. Moreover, as previously indicated in research [4], these two properties are also vital to enhancing model robustness. Building on this, we want to explore if synthetic data of high structural complexity can be employed for **data augmentation** at different levels to improve model robustness. Significant experimental results with IPMix validate our ideas. \n\nBased on this, we believe that what is truly important is using **synthetic data** to enhance the **structural complexity** of the data. We selected fractals for their unique properties: natural complexity and simple mathematical foundations for image generation. However, our experiments confirm that the performance of IPMix remains relatively stable under the change of synthetic data. Please refer to Appendix B.1 for more details.\n\nW3: \"The importance of label-preserving\"\n\nMixing images with distinct labels and linearly interpolating between them will impose certain “local linearity” constraints on the model’s input space beyond the data manifold, which may lead to **\"manifold intrusion\"**. Consider one experiment on MNIST. If we use MixUp to linearly mix two numbers, such as \"1\" and \"5\", the generated image will show the characteristics of \"8\". When the generated \"8\" **collides with** a real \"8\" in the data manifold, there will be a problem of manifold intrusion. Since the two samples have similar characteristics, one is the real label and the other is a soft label (\"1\" and \"5\"). This will interfere with its ability to understand and classify categories and degrade model performance. \n\nBuilding on these finds, we introduced IPMix, which mixes the original image with **unlabeled** synthetic data. This approach increases data diversity and image structural complexity without altering the label, thus enhancing robustness against various data shifts. We believe that this **label-preserving** characteristic is vital to IPMix's outstanding performance.\n\nW4: We have added experiments on the transformer architecture, please see results in the \"global\" response.\n\nQ1: Thanks for your questions. We discussed Table 1 between L162-L170, where we addressed the phenomenon observed and its potential reasons. The table illustrates that mixing different levels (image, patch, and pixel) sometimes yields worse performance than standalone methods. This could be attributed to the added complexity in the training data, possibly affecting the models' ability to extract meaningful features. For a deeper dive, please refer to Appendix F, which elaborates on this with experimental design. \n\nQ2: The terms are defined in Figure 4's caption. We've updated the figure based on your feedback. Thank you.\n\nQ3:  We have designed experiments based on your inspiration, please view results and analysis in the \"global\" response PDF, Table 1.\n\nQ4:  Thank you for your question. Regarding ImageNet, we haven't conducted an ablation study due to time and computational constraints. However, based on our CIFAR experiments, we believe image-level augmentation remains crucial for ImageNet. We will add ImageNet results in the new version. \n\nQ5: Please see the detailed answer in the \"global\" response.\n\nDue to the word limit in the rebuttal, we couldn't address all concerns exhaustively. Please let us know if you have any further questions.\n\nPlease refer to relevant references in the \"global\" response."}, {"review_id": "BQT64G6QOi", "reviewer": "Reviewer_Bz96", "summary": "This paper proposes a new data augmentation method, IPMix, for vision models that integrates three levels of augmentation techniques (image level, patch level, and pixel level). The approach is label preserving and enforces extra structural complexity to enhance the robustness and other safety measures. Experimental results on multiple open datasets and backbone models are provided to support the research claims in the paper.", "questions": "I think the main question that should be answered is the main technical contribution made in this paper compared to the existing literature, and what main challenges have been solved.", "limitations": "Yes, the limitations are discussed in the Appendix of the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. In addition to clean accuracy, the authors seek to optimize both the model robustness and various safety measures in the paper.\n\n2. The provided evaluations are extensive and improvements are observed in multiple performance perspectives compared to the baselines.\n\n3. The paper is generally easy to understand and follow.", "weaknesses": "1. Throughout the whole paper, it is kind of hard to understand what is the main technical contribution made in the paper that is original and novel. Although multiple components have been introduced, they seem to be trivial applications and combinations of previously published approaches, since the design of each component of inspired by previous works. Besides, those components are not connected and introduced through a coherent idea but are more like random pieces that are very loosely connected. What main technical challenges have been solved in this paper? If we consider the combination pipeline of different augmentations as the main contribution, it is still trivial because the authors just experimented with different intuitive combination methods in Table 2 and selected the overall best one.\n\n2. From the paper, I don't understand exactly why label-variant augmentations lead to manifold intrusion and lower robustness against unseen data. If the label-preserving property is important, can we enhance the CutMix and MixUp by mixing original images with fractal images and achieving a similar performance as IPMIx?\n\n3. In the evaluation section, the authors only focus on showing the results but do not provide any analysis of the results, which results in the fact that the only piece of information expressed there is IPMix is better than the baselines in various settings, but not about why IPMix works better.\n\n4. The ablation study is not designed in the correct way. Its current form only conveys the message that all three levels of augmentations are needed while removing any one of them will lead to degradations in the performance. In addition to that, the authors should use experiments to show the performance degradation without proper combinations, fractals (label variance), and the pieces in Section 4.2.\n\n5. The authors mentioned the computation efficiency improvement caused by the approach, but no evaluations are performed to test the computation efficiency between IPMix and conventional image-level augmentations.\n\n6. Here are some minor comments:\n\n    (1) Since the paper depends on image-specific intuitions, it would be good to be precise on the scope in the paper title. \n\n    (2) In line 183, the \\lambda is directly used without introducing its purpose and functionality.\n\n    (3) The specific design scope and definition of IPMix is unclear. The title of Section 4.1 is about IPMix, while Section 4.2 further introduces more design components. Do they belong to part of IPMix? If so, the organization of the paper should be fixed.\n\n    (4) The ticks in Figure 6 are too small and hard to recognize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and detailed reviews.\n\n Below we will address your concerns about novelty and what problem we solved.\n\nW1-1: \"novelty and technical contributions.\"\n* **Diversity & Complexity**: While data diversity and structural complexity are keys to model robustness, most previous methods to improve robustness focus on either generating diverse samples or increasing complexity. We are the **first** to propose a data augmentation method that mixes fractals with the original image at different levels and combines the strengths of these two methods into a unified and coherent whole, to enhance the overall performance.\n* **Comprehensive Safety Metrics**: Most methods target only accuracy or data shifts from corruption. In contrast, we aim to elevate performance across multiple safety metrics, which is more meaningful from a practical perspective.\n*  **Unique Paradigm**: Although some components of our approach have been proposed before, we believe that the paradigm of our entire method is **unprecedented**. For instance, while paper [5] introduced Random Pixels and Random Elements with notably lower accuracy, we found that mixing original images with fractals via these methods enhances robustness without compromising accuracy. Our motivation is to combine these components to showcase potential performance. We posit that this task is non-trivial, but rather holds significant value and can benefit future research.\n\nW1-2: 'what main challenges have been solved.\"\n\n**Diverse Distribution Shifts:** We improved the model's ability to resist distribution shifts at various levels.\n\n**Holistic Safety Performance:** Our approach offers a comprehensive improvement across multiple safety metrics without sacrificing accuracy.\n\n**Research Direction:** We pioneered a direction for exploration wherein by mixing original images with synthetic images in a label-preserving way, we've amplified data's structural complexity, thus elevating overall model performance.\n\nBased on the aforementioned statements, we firmly believe that IPMix is both novel and significant.\n\nW2-1: \"The drawbacks of label-variant methods. \"\n\nPlease see the detailed answer in the \"global\" response.\n\nW2-2: \"What if mixing MixUp and CutMix with fractals in a label-preserving way?\"\n\nPlease note that MixUp regularizes the neural network to favor simple linear behavior in-between training examples, and CutMix aims to recognize two training objects from their respective partial views. If we directly mix them with fractals outside the dataset, it may violate the original intention of these methods. Nevertheless, we still considered two experimental scenarios: 1. Mixing the original image with fractals first, then leveraging MixUp/CutMix based on the generated image in a **label-variant** way. 2. Directly mix the original image with fractals in a **label-preserving** way. Due to limited space, we only show the second solution the result of.\n\n**Experiment on CIFAR-100 with WRN40-4:**\n\n||Classification|Robustness|Adversaries|Consistency|Calibration|\n|-|-|-|-|-|-|\n|MixUp|20.5 $\\pm $0.11|45.9 $\\pm $0.12|96.9 $\\pm $0.14|8.9 $\\pm $0.04|10.5 $\\pm $0.02 \n|MixUp + fractal|20.6 $\\pm $0.19|46.5 $\\pm $0.23|97.1 $\\pm $0.21|9.9 $\\pm $0.09|11.3 $\\pm $0.04 \n|CutMix|20.3 $\\pm $0.08|50 $\\pm $0.09|96.9 $\\pm $0.12|12 $\\pm $0.05|10.5 $\\pm $0.05 \n|CutMix + fractal|20.8 $\\pm $0.13|50.3 $\\pm $0.21|97.0 $\\pm $0.16|10.7 $\\pm $0.11|14.4 $\\pm $0.08 \n|IPMix|**19.4 $\\pm $0.17**|**28.6 $\\pm $0.2**|**89.4 $\\pm $0.18**|**4.3 $\\pm $0.09**|**2.8$\\pm $0.07** \n \nThe experiment shows that directly mixing the original image with fractals in a label-preserving way will degrade model performance. This may stem from the added structural complexity failing to offset the loss of inductive bias.\n\nW3: We provide an analysis of why IPMix outperforms the other augmentation methods in Section 6. We apologize for the misleading in the evaluation section and will modify relevant content in the revised manuscript. \n\nW4:  Thank you for the valuable feedback. \n\nWe would like to kindly point out that these concerns have been addressed in Appendix B. Due to space constraints, we placed the ablation experiments in the Appendix and apologize for any misleading.\n\nW5: \nIPMix offers significant computational advantages by requiring only training once. In contrast, conventional image-level methods demand an extensive search for the optimal augmentation policy, as noted in [1,2]. The table below further illustrates the search overhead difference between IPMix and image-level methods. The data are from the original paper and vary depending on the dataset used (CIFAR, SVHN, and ImageNet).\n\n|Method| Serach Overhead |\n|-|-|\n|AutoAugment|40-800x\n|RandAugment|4-80x\n|Fast AutoAugment|1x\n|IPMix|**0x**  \n\nW6(1): Your suggestion is incorporated in the new version of the paper. Thank you.\n\nW6(2):  Thank you for pointing that out. In the equation provided, \\lambda represents the mixing ratio, derived from the Beta distribution ($\\alpha=1$ by default), indicating the proportion of the content taken from the input image x1 and the synthetic image x2 when performing mixing operations. We ensure clear representation in the revised manuscript.\n\nW6(3): In our manuscript, Section 4.1 presents the core framework of IPMix, while Section 4.2 introduces two methods designed for more effective information fusion. These two parts contribute to the excellent performance of IPMix. To provide better clarity, we plan to adjust the section titles as follows:\n\nSection 4: IPMix \n\nSection 4.1: Integrating Different Levels into a Coherent Approach \n\nSection 4.2: Multi-Scale Information Fusion\n\nWe appreciate your feedback once again.\n\nW6(4): Please refer to the \"global\" response PDF for the revised Figure 6.\n\nDue to the word limit in the rebuttal, we couldn't address all concerns exhaustively. Please let us know if you have any further questions.\n\nPlease refer to relevant references in the \"global\" response."}, {"review_id": "xpY5gqoS05", "reviewer": "Reviewer_jXDs", "summary": "The authors propose IPMix -- a new data augmentation technique applied during training that improves model robustness across a variety of test-time tasks (e.g., distributional shift, adversarial data augmentation) without compromising accuracy on in distribution tasks (when compared to a model trained in a traditional manner). The acronym IPMix indicates that the technique utilizes augmentations at the Image (I) and Pixel/Patch (P) level and mixes these together with each other and the original image. IPMix augments samples via a convex combination of an augmented and original image, where the novelty in the augmentation operations applied to the image (compared to existing augmentation techniques e.g. Augmix) come from operations between an image and unlabeled synthetic images (e.g., fractals). IPMix research is motivated to produce safe ML systems (e.g., improve model robustness and calibration).\n\n**Main Contributions:**\n\n* IPMix: A label preserving data augmentation technique that achieves most balanced performance across a variety of tasks desired for safe ML systems (accuracy, robustness, calibration, and anomaly detection).\n* A thorough empirical study of IPMix using multiple datasets, architectures, and tasks compared to large set of existing data augmentation schemes.\n\n**Post-rebuttal update**\nThe authors addressed all of my weaknesses and questions and I have raised my score from a weak accept to accept.", "questions": "1.\t**Motivation**: What is the motivation for unlabeled synthetic images (e.g., fractals) used in IPMix set? I suspect this is very clear to the authors, but it is not well motivated or explained in the main body. \n2.\t**Experiments**: It is unclear from the paper and appendix if multiple replicates are trained for each architecture/dataset/augmentation configuration (Tables 3, 4, and Figure 6). I am under the impression that a single model was trained for each configuration. If that is the case, it is unclear how much deviation (in accuracy, robustness, calibration) the models trained with varying data augmentation schemes have. I would typically expect some result on this for smaller datasets (e.g., CIFAR-10). Is this something you considered that I missed?\n3.\t**Overhead/cost**: Is there an observable overhead/cost of using IPMix during training (when compared to vanilla training or other data augmentation techniques)? In my experience, Augmix increases training time due to augmentation scheme so I am curious if IPMix is more/less costly than Augmix (as it produces better models than Augmix).\n4.\t**Robustness**: How do models trained using IPMix compare to robustbench leaderboard Common corruptions (i.e., CIFAR-10-C, ImageNet-C)? I suspect they are comparable, but it would potentially strengthen the work by comparing to SOTA models from robustbench. The architectures used in the paper are commonly used on robustbench so it would likely be an easy/straightforward comparison. *In this direction: Did you consider utilizing the Jensen-Shannon divergence (JSD) loss utilized in the original Augmix paper in experiments with IPMix? I suspect this may offer additional improvements to models trained with IPMix so it may be worthwhile to consider if comparing to robustbench models.*\n5. **Figure 6**: Is this calibration error for a single architecture trained using each data augmentation technique evaluated on CIFAR-100 (e.g. ResNet-18)? The caption does not specify.\n\nMinor suggestions:\n* Table 1 is provided but never referenced or commented on in the text of the paper. I would typically expect some minor commentary on tables/figures.\n* Table 1: “Accuracy Error” seems like odd/confusing phrasing to me. Wouldn’t “Classification Error” be clearer?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Experiments**: Experimental results with IPMix compared to existing data augmentation methods (in terms of model accuracy, robustness, and calibration) are impressive and are largely consistent across many datasets and architectures.\n* **Data augmentation analysis**: Some analysis of models trained with varying data augmentation schemes highlights strengths of IPMix (Figure 8) in comparison to existing data augmentation schemes.", "weaknesses": "-\t**Motivation**: The use of unlabeled synthetic images in the Pixel/Patch mixing did not seem well motivated to me. I agree that IPMix seems to work well based on the experimental results and understand that the choice of unlabeled synthetic images avoids mislabeled augmented training images. However, I feel that some motivation (practical or theoretical) is needed to help clarify and better motivate the design of IPMix and why it would improve performance beyond existing methods (e.g., Augmix) that do not produce augmented images with incorrect labels.\n-\t**Paper organization**: A lot of the paper is relegated to the appendix (Appendix F (combination experiments), Appendix E (Fractal generation), Appendix B.1 (IPMix insensitive to mixing set change), Appendix C (IPMix algorithm). I understand space constraints and submission limitations, but I would have liked to see more of the details of IPMix and experiments informing its design in the main body. Perhaps the related work section could have been in the appendix to make space for this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time in reviewing our paper.\n\nW1-1: \"The motivation of using synthetic images. \"\n\nDue to the difficulty in acquiring labeled data, as well as human bias and data privacy issues, more and more research focuses on obtaining synthetic data that can be generated without labeling. Moreover, most studies concentrate on pre-training on synthetic data and then transferring to downstream tasks to alleviate the model's dependence on labeled data. Among these studies, [3] pointed out that **diversity** and **structural complexity** of synthetic data are keys to excellent performance. Meanwhile, as previously indicated in research [4], these two properties are also vital to enhancing model robustness. Since synthetic data are easy to obtain and mixing with them do not encounter \"manifold intrusion\", which leads us to question whether synthetic data can be applied to data augmentation at different scales, improving the model's robustness by increasing the image's structural complexity and data diversity. Furthermore, our thoughts are inspired by and tally with **human visual principles**. We continually develop our ability to distinguish various subjects by being exposed to a wide array of objects (diversity) and encountering intricately structured targets (structural complexity). Finally, the significant results of IPMix validate our ideas.\n\nBased on this, we believe the motivation for utilizing unlabeled synthetic data is sufficient and necessary. \n\nW1-2: \"Comparison with other label-preserving methods\"\n\nBelow we will address your concerns by comparing IPMix with AugMix from different views to show why our approach improves performance.\n\n*  **Motivation**: AugMix mixes an image with its versions that have been processed with image-level methods (e.g., rotation) to improve robustness. However, AugMix has lower accuracy compared to other patch-level methods, as shown in Table 1. Our motivation is to combine the strengths of different levels of methods into a unified and coherent whole, to enhance the overall performance.\n\n*  **Objective**: The purpose of AugMix is to tackle data shifts caused by corruption, but the actual situations encountered in real-world scenarios are much more complex, as mentioned in the paper [6]. Our objective is to improve performance across multiple safety metrics simultaneously, which is more meaningful from a **practical perspective**.\n\n*  **Method**: Except for the image-level augmentation, IPMix mixes clean data with fractals at different levels to improve **structural complexity** and uses **random pixels mixing** and **random elements mixing** for better data integration. While AugMix applies augmentation transformations only at the image level, its performance in classifying partially occluded objects is sub-optimal.\n\nWe firmly believe that the above reasons are the reasons why IPMix performs well.\n\nW2: We genuinely appreciate your suggestion. \n\nWe value the thorough acknowledgment of related work as it respects prior research. Meanwhile, we recognize and agree with your suggestion to detail more of IPMix in the main paper. We will strive to incorporate more details while adhering to page limits in the revised manuscript.\n\nQ1: Please refer to W1 for detailed response.\n\nQ2: We apologize for the oversight. While multiple replicates were conducted, only the ablation study in Table 6 explicitly mentions this. Due to space constraints, we only present the repeated results of PixMix, AugMix, and IPMix on CIFAR-10 here, and we will ensure that the revised manuscript includes all the results.\n\n**Experiments on CIFAR-10 with WRN28-10:**\n\n||Classification (Error)↓|Robustness (mCE)↓ |Adversaries (Error)↓|Consistency (mFR)↓|Calibration (RMS)↓|\n|-|-|-|-|-|-|\n|AugMix|3.4 $ \\pm $0.16|9.1 $ \\pm $0.21|81.9 $ \\pm $0.21|1.2 $ \\pm $0.11|3.8 $ \\pm $0.13|\n|PixMix|3.8 $ \\pm $0.20|8.7 $ \\pm $0.21|92.9 $ \\pm $0.25|1.6 $ \\pm $0.09|2.9 $ \\pm $0.21|\n|IPMix|**3.3 $ \\pm $0.17**|**7.5 $ \\pm $0.23**|**76.4 $ \\pm $0.23**|**1.1 $ \\pm $0.10**|**1.9 $ \\pm $0.16**|\n\nQ3: Thank you for your question. We have addressed the overhead of using IPMix during training in Appendix G. Please refer to it for a detailed response.\n\nQ4-1: It's worth noting that direct comparisons with other methods might be subject to discrepancies due to factors such as epochs, architecture, and the use of extra data. Based on robustbench, for CIFAR-10-C, the Winning Hand achieved a Robustness accuracy of **92.78%** on WRN28-4. In contrast, IPMix achieved **91.4%** and **92.5%** on WRN40-4 and WRN28-10, respectively, and please note that we only trained for 100 epochs. For CIFAR-100 -C, while Winning Hand reached **71.08%** accuracy on WRN28-4, our method surpassed this with **71.4%** and **73.4%** on WRN40-4 and WRN28-10, respectively, which significantly outperforms the current best approach. As for ImageNet-C, given that our ResNet18 model is considerably smaller than ResNet50, we haven't made a comparison. We aim to explore this direction in future experiments and believe that the existing results are sufficient to prove the significance of IPMix. Please note that our goal is to improve overall safety metrics.\n\nQ4-2: \"JSD loss\"\n\nWe did experiments with the JSD-loss and found it can improve performance. However, using JSD loss requires at least three times the memory per batch, which can significantly extend the training time. Hence, we consider treating JSD as a hyperparameter and add analysis and experiments to the revised manuscript.\n\nQ5: The calibration error was evaluated on WRN40-4. Please refer to the \"global\" response PDF for the revised Figure 6.\n\nM1: We will add data analysis in the revised manuscript. Thanks.\n\nM2: Your suggestion is incorporated in the new version of the paper. Thank you.\n\nDue to the word limit in the rebuttal, we couldn't address all concerns exhaustively. Please let us know if you have any further questions.\n\nPlease refer to relevant references in the \"global\" response.\n"}, {"review_id": "ee9unjX1ui", "reviewer": "Reviewer_kzv7", "summary": "This work proposes a data augmentation method, named IPMix, which is designed to improve robustness without hurting clean accuracy. Specifically, IPMix utilizes different levels of data augmentation (image-level, patch-level, and pixel-level) to increase the diversity of training data and introduces structural complexity to generate more diverse images. The authors have evaluated the robustness of IPMix to adversarial perturbations, calibration, prediction consistency, and anomaly detection.", "questions": "Apart from the points in Weaknesses, the reviewer also has some other questions:\n\n1. Validation across different architectures. The authors have only validated IPMix on ResNet which not be enough to prove the generalization ability of IPMix.\n2. Revision on Fig. 4. It is advised to add the names of the three methods from Tab. 2 to Fig. 4 which will be easier for reading.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The evaluation of robustness and ablation study is extensive. The authors have comprehensively validated the robustness of IPMix, including adversarial perturbations, calibration, prediction consistency, and anomaly detection. Moreover, the ablation of design choices is in detail.\n2. The results on small-scale datasets are promising. Overall, IPMix achieves promising accuracy on CIFAR datasets, as well as safety metrics.", "weaknesses": "1. Comparisons with RandAugment [1]. The whole design of IPMix seems to be a complicated combination of multiple existing data augmentation methods and the image-level transformations are important components of RandAugment. According to Tab. 6, these image-level transformations are the most effective modules in the whole design and it is suggested to compare IPMix with RandAugment, which is one of the most popular choices in current training frameworks.\n2. Generalization ability to competing methods. Current training frameworks of object recognition [2][3] often utilize multiple data augmentation methods during training, the reviewer wonders whether IPMix is compatible with these methods and could further improve the performance.\n3. Results on ImageNet. The accuracy of IPMix on imageNet is not very convincing and the whole results on ImageNet seem strange, as Mixup, CutMix both lead to inferior results compared to the baseline. This phenomenon is contradictory to the results of their papers and the explanation of this part is highly appreciated.\n\n[1] Cubuk E D, Zoph B, Shlens J, et al. Randaugment: Practical automated data augmentation with a reduced search space[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 2020: 702-703.\n[2] Liu Z, Mao H, Wu C Y, et al. A convnet for the 2020s[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 11976-11986.\n[3] Li K, Wang Y, Gao P, et al. Uniformer: Unified transformer for efficient spatiotemporal representation learning[J]. arXiv preprint arXiv:2201.04676, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time in reviewing our paper.\n\nW1: Thank you for your valuable suggestion. \n\nWe agree that RandAugment is a popular image-level method and believe that comparing with it can strengthen our work. While space constraints in our main paper limited a direct comparison with RandAugment, we did feature it as a representative image-level method in Figure 2. Additionally, in Appendix B.2, we contrasted IPMix with other prevalent image-level techniques, such as AutoAugment, RandAugment, and TrivialAugment, underscoring the advantages of our method. For a more comprehensive review, we've summarized the content from Appendix B.2, which evaluates Accuracy, Robustness, Consistency, Calibration, and Adversaries on CIFAR100 with WRN-28x10. Mean and standard derivation over three random seeds is shown for each experiment.\n\n||Classification (Error)↓|Robustness (mCE)↓ |Adversaries (Error)↓|Consistency (mFR)↓|Calibration (RMS)↓|\n|-|-|-|-|-|-|\n|AutoAugment|17.7 $ \\pm $0.11|38.4 $ \\pm $0.15|97.8 $ \\pm $0.22|8 $ \\pm $0.06|7.9 $ \\pm $0.06|\n|RandAugment|17.8 $ \\pm $0.14|41.5 $ \\pm $0.13|96.6 $ \\pm $0.25|8.6 $ \\pm $0.10|7.9 $ \\pm $0.04|\n|TrivialAugment|18$ \\pm $0.13|35.4 $ \\pm $0.21|96.3 $ \\pm $0.23|7.3 $ \\pm $0.07|8.7 $ \\pm $0.04|\n|IPMix|**17.4 $ \\pm $0.17**|**26.6 $ \\pm $0.19**|**91.3 $ \\pm $0.21**|**4.2 $ \\pm $0.11**|**6.4 $ \\pm $0.07**|\n\nW2: Thank you for your questions about the generalization ability of IPMix. To address this, we conducted additional experiments on UniFormer and ConvNeXt, as you mentioned. Due to time and resource constraints, we conducted our experiments on smaller versions of the corresponding models. Specially, we trained UniFormer-XXS (224x224) from scratch for 300 epochs and fine-tuned ConvNeXt-Tiny for 90 epochs. Below, we have provided the relevant experimental results. These results demonstrate the generality and effectiveness of IPMix.\n\n||Classification (Accuracy)↑|ImageNet-C (Error)↓ |ImageNet-R (Error)↓|Calibration (RMS)↓\n|-|-|-|-|-|\n|UniFormer|80.2|52|57.2|6.1\n|UniFormer+IPMix|**80.3 (↑0.1)**|**43.8 (↓8.2)**|**56.5 (↓0.7)**|**5.9 (↓0.2)**\n|ConvNeXt|81.5|43|52.5|7.2\n|ConvNeXt+IPMix|**81.8 (↑0.3)**|**36.2 (↓6.8)**|**50.8 (↓1.7)**|**7.1 (↓0.1)**\n\nW3: Thank you for your feedback.\n\nFirstly, we'd like to point out that the experimental settings in our study differ from the original papers. In the original works, Mixup trained on ResNet-50 and ResNet-101 for 200 epochs, whereas CutMix trained for 300 epochs. In contrast, our experiments were conducted on ResNet-18, with only 90 epochs due to computational constraints and for easier tuning. This variation in settings can lead to different outcomes, as the **data complexity** and **model capacity** play vital roles in determining performance.\n\nDiving deeper from a theoretical perspective, the VC dimension of a model, denoted as $D$, sets the boundary for the maximum dataset size it can \"perfectly\" classify.  For the generalization error on test data to be close to its training error, we typically require the number of training samples, $n$, to satisfy: $ n = O_{\\gamma, \\delta}(D) $, where $D$ is the VC dimension of the model, and $γ$ is the desired error boundary. However, data augmentation techniques like Mixup and CutMix increase the effective training sample count, $n$, by diversifying and complicating the data.  It's pivotal to recognize that this doesn't inherently increase the model's capacity in terms of its VC dimension.  *Consequently, while our training size, $n$, has expanded, the model's capability may not match this increase, potentially leading to **underfitting***. This relationship emphasizes that the augmented number of training samples, $n$, must align with the model's VC dimension $D$ to ensure adequate performance.\n\n Moreover, data augmentation techniques, such as Mixup and CutMix, enhance the complexity of training data, expanding and complicating the data's feature space. However, these augmentation methods may introduce a **\"manifold intrusion\"** problem during data mixing. *In essence, generating new samples with soft labels may conflict with the original ones across data manifolds*. Please refer to the \"global\" response for more details.\n\nTherefore, we face a dual challenge: on one hand, the augmentation methods increase data complexity, requiring a model with a higher VC dimension for a perfect fit; on the other hand, due to manifold intrusion, we may now be training on a less authentic data distribution. Thus, when a model's complexity isn't sufficient to capture this enhanced data complexity, such as ResNet-18, it might lead to **underfitting**.\n\nIt's important to note that simply increasing the training epochs won't fix this underfitting. For instance, even if MixUp and CutMix are applied for 300 epochs on ResNet-18, their performance might still not surpass Vanilla, as shown in the table follows:\n\n|Backbones|Beta|ResNet18 |\n|-|-|-|\n|Epochs|$\\alpha$|300 epochs\n|Vanilla|-|71.83\n|MixUp|0.2|71.72\n|CutMix|1|71.01\n\n**Note: You can find these results from https://github.com/Westlake-AI/openmixup/tree/main/configs/classification/imagenet/mixups/**\n\nHence, our experimental results are in line with your intuition. When we employ data augmentation techniques, we must consider multiple factors such as the **model's size**, **dataset size**, and **hyperparameter settings**. Methods such as CutMix and MixUp may only demonstrate their efficacy on ResNet-50 or **larger models**.\n\nQ1: We have added experiments on the transformer architecture, please see results in the \"global\" response.\n\nQ2: We have modified it according to your suggestion, please see the revised Figure 4 in the \"global\" response PDF.\n\nDue to the word limit in the rebuttal, we couldn't address all concerns exhaustively. Please let us know if you have any further questions.\n"}, {"review_id": "kj0rg9oFab", "reviewer": "Reviewer_jACH", "summary": "This paper proposes IPMix, a data augmentation strategy to improve model robustness (and other safety metrics like calibration). The idea builds upon PixMix, a recent data augmentation strategy generating diversity via synthetic pictures. IPMix performs this at multiple scales (pixel, patch, image levels) and mixes the resulting transformed images. The evaluations are performed on CIFAR-10, CIFAR-100, ImageNet and their shifted versions against several data augmentation baselines.", "questions": "Please see weaknesses.\n", "limitations": "Please see weaknesses.\n\nAfter rebuttal, I am increasing my rating to borderline accept.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is very well written, it is quite easy to understand. The method is clear and reasonable, given that distribution shifts may happen at different levels it makes sense to address them at different levels. The comparisons and ablation studies are well thought. Additional experimental results in supplementary are also helpful.\n", "weaknesses": "My main concerns are about ImageNet results which suggest minor improvements over PixMix, e.g. ImageNet-C results are 78.1 vs 77.8. The results are close for other benchmarks too like ImageNet-R and ImageNet-P (very close to AugMix). Same goes for anomaly detection results. This makes me wonder: Does using synthetic data and mixing have inherent limitations in terms of generating diversity? For example, what happens if you increase your diversity using recent diffusion models to generate data, e.g. as in [A]?\n\nMy other questions and comments:\n- I would be interested in seeing comparisons to DeepAugment as well, as it has been shown to improve robustness notably when combined with AugMix.\n- How does IPMix perform against natural distribution shifts like ImageNet-V2 and ObjectNet? What about more recent ImageNet benchmarks like ImageNet-3DCC and ImageNet-E? I believe having these evaluations could also provide some insights into the limitations I mentioned above.\n- When I was reading L201 I thought about the work of Mintun et al. ([15] in supmat) and I was happy to see that you added evaluations on CIFAR-C-bar in the supplementary, which again seems to suggest minor improvements over PixMix. What would be the results for ImageNet-C-bar? \n- It is nice that you added training overhead in supmat too, but I think it should be better linked in the main paper as I was going to ask about it and it wasn’t pointed out in the main paper.\n- I understand this is a CNN based study but given the landscape and domination of ViTs I believe evaluations on ViT backbones are also necessary to truly understand the performance of IPMix.\n\n[A] Sariyildiz et al. Fake It Till You Make It: Learning Transferable Representations From Synthetic ImageNet Clones, CVPR 2023.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed concerns.\n\nW1: First and foremost, I'd like to emphasize that the improvements we achieved are not minor. Our approach consistently shows improvement across various safety metrics by comparison with other methods, which is more meaningful from a practical perspective.\n\nWe value your suggestion. While employing diffusion to generate new samples on ImageNet indeed introduces diversity[A], it inevitably alters the labels during mixing with origin images (Since generated dogs are still dogs). Mixing directly with these generated images may violate our original intention of label-preserving.\n\nInspired by your idea, we designed a feasible experiment that applied the diffusion model to our IPMix set. Specifically, we use stable diffusion to generate an additional 10000 images, leading to a combined dataset of 20000 images. Due to time constraints and limited computational resources during the rebuttal period, our experiments were only conducted on CIFAR-10/100 using the WRN28-10. Mean and standard derivation over three random seeds is shown for each experiment.\n\n**Experiments on CIFAR-100 with WRN28-10:**\n\n||Classification (Error)↓|Robustness (mCE)↓ |Adversaries (Error)↓|Consistency (mFR)↓|Calibration (RMS)↓|\n|-|-|-|-|-|-|\n|IPMix|17.4 $ \\pm $0.17|26.6 $ \\pm $0.19|91.3 $ \\pm $0.21|4.2 $ \\pm $0.11|**6.4 $ \\pm $0.07**|\n|IPMix+diffusion|**17.3 $ \\pm $0.15**|**26 $ \\pm $0.25**|**91.2 $ \\pm $0.17**|**4.1 $ \\pm $0.13**|6.5 $ \\pm $0.09|\n\n**Experiments on CIFAR-10 with WRN28-10:**\n\n||Classification (Error)↓|Robustness (mCE)↓ |Adversaries (Error)↓|Consistency (mFR)↓|Calibration (RMS)↓|\n|-|-|-|-|-|-|\n|IPMix|3.3 $ \\pm $0.17|7.5 $ \\pm $0.23|**76.4 $ \\pm $0.23**|1.1 $ \\pm $0.10|1.9 $ \\pm $0.16|\n|IPMix+diffusion|**3.3 $ \\pm $0.14**|**7.2 $ \\pm $0.21**|76.5 $ \\pm $0.19|**1.0 $ \\pm $0.08**|**1.8 $ \\pm $0.11**|\n\nFrom the experimental results, after using the diffusion model to generate new samples, the performance of IPMix has a minor improvement.\n\nFrom the current research and our experimental results, it seems premature to provide a definitive answer to \"whether there are inherent limitations of synthetic data to generate diversity\". Mixing data requires a thorough evaluation of the **original data distribution** and the **nature of synthetic data**. With the wide range of image generation methods available today, such as diffusion-based, statistical, and procedural image models, it's challenging to pinpoint *which type of synthetic data maximizes diversity enhancement for specific data distribution*. A potential solution might be to adopt an AutoAugment approach, using different synthetic data for various datasets, and conducting an extensive search through the augmentation policy space to identify the optimal synthetic data suited for a particular dataset. Nevertheless, this approach entails a significant computational cost, often exceeding the expense of training. Interestingly, our experiments suggest that **combining multiple types of synthetic data** (e.g., diffusion model and fractal data) seems to yield better results. This inspires us that there may be a way to mix different types of synthetic data to **maximize the diversity** of different data sets. We believe this could be a promising research direction and plan to explore it further in our future work.\n\nQ1: Thank you for your question. \n\nDue to time and computational constraints during the rebuttal, we were unable to train a model from scratch on ImageNet with DeepAugment+IPMix. As an alternative, we conducted our experiments on Tiny-ImageNet. Specifically, Tiny-ImageNet was chosen because it retains a data distribution similar to ImageNet but has a significantly reduced dataset size, making our experiments more manageable. On Tiny-ImageNet, we trained both AugMix, IPMix, DeepAugment+AugMix and DeepAugment+IPMix for 300 epochs on ResNet-18 and evaluated metrics on Tiny-ImageNet-C. Below are our experimental results:\n\n|Method|Accuracy(↑)|mCE(↓)\n|-|-|-|\n|Vanilla| 61.5|100|\n|AugMix| 61.8|83.2|\n|IPMix| 62.9|80.1|\n|DeepAugment+AugMix| 59.6|78.4|\n|DeepAugment+IPMix| **60.6**(+1)|**75.7**(-2.7)|\n\nFrom the experimental results, combining DeepAugment with IPMix significantly improves robustness. However, this improvement comes at the cost of loss of accuracy, demonstrating a trade-off between accuracy and robustness. It's possible that DeepAugment introduces distorted images that improve robustness, which might be misaligned with the origin data distribution, thereby compromising clean accuracy.\n\nQ2&Q3: Thanks for your questions.\n\nWe have conducted experiments on different data sets. The experimental results are shown in the table below:\n\n**Experiments on ImageNet with ResNet-18:**\n\n|Method|ObjectNet|Imagenet-sketch|Imagenet-style|Imagenetv2|ImageNet-E|ImageNet-3DCC|ImageNet-C-bar|\n|-|-|-|-|-|-|-|-|\n|AugMix| 9.7|22.6|9.5|55.8|70.5|24.1|38.4|\n|PixMix| 10.8|22.1|**10.8**|54.9|69.7|**27.4**| 40.5|\n|IPMix| **11.1**|**22.8**|10.4|**56.5**|**71.5**|27.2| **41.1**|\n\nFrom the experimental results, overall, IPMix has achieved better results against different data shifts. However, it only achieved suboptimal performance on ImageNet-style and ImageNet-3DCC. We will explore the reasons and optimize our method in future work.\n\nQ4: Due to the space limitation of the main paper, we have to put part of the content in the appendix. Meanwhile, as NeurIPS requires separate submissions for the main paper and appendix, direct links within the main text were not feasible. We apologize for any inconvenience caused.\n\nQ5: We have added experiments on the transformer architecture, please see results in the \"global\" response.\n\nDue to the word limit in the rebuttal, we couldn't address all concerns exhaustively. Please let us know if you have any further questions.\n\n[A] Sariyildiz et al. Fake It Till You Make It: Learning Transferable Representations From Synthetic ImageNet Clones, CVPR 2023."}]

Author rebuttal:
First, we thank all reviewers for your help and support. Due to the limitation of input characters on OpenReview, we have to put some questions that the reviewers are generally concerned about here and answer them together.

**1. Experiments on Transformer:**

Given the limited time and computational resources available during the rebuttal phase, we strategically conducted three experiments:
* We trained a small ViT for 300 epochs on CIFAR-100 and CIFAR-10. This step aimed to confirm IPMix's potential on smaller datasets using Transformer architectures.
* We initialized DeiT-tiny with IPMix and trained it from scratch for 300 epochs.
* We fine-tuned Swin-Tiny using IPMix for an additional 90 epochs.

These experiments were crafted to showcase the versatility of IPMix across diverse scenarios while highlighting its consistent performance and adaptability within various Transformer architectures.

**Experiments on CIFAR-10:**

||Classification|Robustness|Adversaries|Consistency|Calibration|
|-|-|-|-|-|-|
|Vanilla|19.5 $\pm $0.07|27.7 $\pm $0.14|91.3 $\pm $0.13|5.9 $\pm $0.02|10
|MixUp|19 $\pm $0.11|34.7 $\pm $0.21|89.3 $\pm $0.21|6 $\pm $0.05|9.9 $\pm $0.03 
|CutMix|19.3 $\pm $0.08|34.3 $\pm $0.19|89.1 $\pm $0.14|5.5 $\pm $0.05|7.5 $\pm $0.02 
|PixMix|28.4 $\pm $0.14|33.9 $\pm $0.24|91 $\pm $0.12|6.5 $\pm $0.11|4.4 $\pm $0.07 
|AugMix|20.3 $\pm $0.14|25.6 $\pm $0.2|80.3 $\pm $0.16|5.1 $\pm $0.09|6$\pm $0.08
|IPMix|**19.2 $\pm $0.12**|**23.7 $\pm $0.2**|**75.8 $\pm $0.13**|**3.7 $\pm $0.07**|**5.3$\pm $0.07** 

**Experiments on CIFAR-100:**

||Classification|Robustness|Adversaries|Consistency|Calibration|
|-|-|-|-|-|-|
|Vanilla|40.1 $\pm $0.12|56.3 $\pm $0.1|96.2 $\pm $0.14|12.4$\pm $0.04|14.8$\pm $0.02
|MixUp|40 $\pm $0.14|56 $\pm $0.18|92.5 $\pm $0.18|9.8$\pm $0.03|9.5 $\pm$0.02 
|CutMix|39.5 $\pm $0.11|56.3 $\pm $0.15|96.2 $\pm $0.17|10$\pm $0.03|9.8 $\pm$0.03 
|PixMix|48.7 $\pm $0.14|54.3 $\pm $0.21|93.2 $\pm $0.14|10.9$\pm $0.07|4.9 $\pm$0.04 
|AugMix|35.3 $\pm $0.17|42.4 $\pm $0.21|84.6 $\pm $0.16|6.9$\pm $0.03|6.4$\pm$0.07
|IPMix|**32.6 $\pm $0.11**|**39.6 $\pm $0.23**|**83.2 $\pm $0.15**|**6.3$\pm $0.04**|**5.3$\pm $0.05** 

**Experiments on DeiT-tiny:**

|Method|Classification (Accuracy)↑|ImageNet-C (Error)↓|ImageNet-R (Error)↓|Calibration (RMS)↓
|-|-|-|-|-|
|DeiT-T|72|57|66.8|13.3|
|+IPMix|**72.6(+0.6)**|**50.4(-6.6)**|**64.1(-2.7)**|**10.5(-2.8)**|

**Experiments on Swin-tiny:**

|Method|Classification (Accuracy)↑|ImageNet-C (Error)↓|ImageNet-R (Error)↓|Calibration (RMS)↓
|-|-|-|-|-|
|Swin-T|80.9|47.1|58.3|7.3|
|+IPMix|**81.2(+0.3)**|**41.3(-5.8)**|**56.5(-1.7)**|**6.7(-0.6)**|

**2. The drawbacks of three methods:**

**The drawbacks of label variant methods:**

* **Pixel-level:** Mixing images with distinct labels and linearly interpolating between them will impose certain “local linearity” constraints on the model’s input space beyond the data manifold, which may lead to **"manifold intrusion"**. Consider one experiment on MNIST. If we use MixUp to linearly mix two numbers, such as "1" and "5", the generated image will show the characteristics of "8". When the generated "8" **collides with** a real "8" in the data manifold, there will be a problem of manifold intrusion. Since the two samples have similar characteristics, one is the real label and the other is a soft label ("1" and "5"). This will interfere with its ability to understand and classify categories and degrade model performance. 

* **Patch-level:** The problem of manifold intrusion also occurs in the patch-level method, termed **"label mismatch."** This occurs when the chosen source patch doesn't accurately represent the source object, leading the interpolated label misleads the model to learn unexpected feature representation. For example, using CutMix to mix images of a cat and a dog. CutMix might select 20% of the background area from the cat image without information about the object (cat). However, their interpolated labels encourage the model to learn both objects’ features (dog and cat) from that training image and degrade model performance.

**The drawbacks of image-level methods:**

Image-level data augmentation increases data diversity by applying label-preserving transformations to the whole image. Notable among these are search-based methods like AutoAugment, RandAugment, and FastAugment. While they improve performance effectively, the **computationally expensive search** for an optimal augmentation policy often exceeds the training process’s complexity. Thus, efforts to minimize the **search space**, **optimize search parameters**, and **uncover potential universal pipelines** are central to the effectiveness of these methods.

In conclusion, we solve these questions by:

* Incorporate **structural complexity** from synthetic data at various levels to produce more diverse images. Our method is **label-preserving**, ensuring it is not affected by manifold intrusion.

* Randomly sample operations from PIL (e.g., brightness, sharpness) and randomly sample strengths to enhance the diversity of training data **without expensive searching**.

* Integrate three levels of data augmentation into a single framework with limited computational overhead, demonstrating that these approaches are complementary and that a unification among them is necessary to achieve robustness.

Due to space constraints, we provide abbreviated references here for clarity and convenience.

[1] Cubuk, Ekin Dogus et al. “AutoAugment: Learning Augmentation Strategies From Data.” 

[2] Cubuk, Ekin Dogus et al. “Randaugment: Practical automated data augmentation with a reduced search space.”

[3] Baradad, Manel et al. “Learning to See by Looking at Noise.” 

[4] Hendrycks, Dan et al. “PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures.” 

[5] Summers, Cecilia and Michael J. Dinneen. “Improved Mixed-Example Data Augmentation.” 

[6] Hendrycks, Dan et al. “Unsolved Problems in ML Safety.” 


Meta-review: The paper proposes IPMix that mixes images at three different levels - image, patch and pixel. This helps the model generalize better of corrupted images. The review ratings are 1 accept, 2 weak accepts, 1 borderline accept and 1 borderline reject. The reviewer who gave borderline reject mentioned that they would like to change it to borderline accept in the reviewer-ac discussion stage. The reviewers agree that the method is easy to follow and the experiment results are strong for both clean and corrupt accuracies. The AC is pleased to recommend acceptance of the paper. The authors should include the comparisons with RandAug and others in the rebuttal to the main paper as one reviewer suggested.
Decision: Accept (poster)