Title: IPMix: Label-Preserving Data Augmentation Method for Training Robust Classifiers
Venue: NeurIPS 2023
Accepted: yes
Keywords: data augmentation

robustness

safety
Abstract: Data augmentation has been proven effective for training high-accuracy convolutional neural network classifiers by preventing overfitting. However, building deep neural networks in real-world scenarios requires not only high accuracy on clean data but also robustness when data distributions shift. While prior methods have proposed that there is a trade-off between accuracy and robustness, we propose IPMix, a simple data augmentation approach to improve robustness without hurting clean accuracy. IPMix integrates three levels of data augmentation (image-level, patch-level, and pixel-level) into a coherent and label-preserving technique to increase the diversity of training data with limited computational overhead. To further improve the robustness, IPMix introduces structural complexity at different levels to generate more diverse images and adopts the random mixing method for multi-scale information fusion. Experiments demonstrate that IPMix outperforms state-of-the-art corruption robustness on CIFAR-C and ImageNet-C. In addition, we show that IPMix also significantly improves the other safety measures, including robustness to adversarial perturbations, calibration, prediction consistency, and anomaly detection, achieving state-of-the-art or comparable results on several benchmarks, including ImageNet-R, ImageNet-A, and ImageNet-O.

IPMix: Label-Preserving Data Augmentation Method for Training Robust Classifiers

Abstract

Data augmentation has been proven effective for training high-accuracy convolutional neural network classifiers by preventing overfitting. However, building deep neural networks in real-world scenarios requires not only high accuracy on clean data but also robustness when data distributions shift. While prior methods have proposed that there is a trade-off between accuracy and robustness, we propose IPMix, a simple data augmentation approach to improve robustness without hurting clean accuracy. IPMix integrates three levels of data augmentation (image-level, patch-level, and pixel-level) into a coherent and label-preserving technique to increase the diversity of training data with limited computational overhead. To further improve the robustness, IPMix introduces structural complexity at different levels to generate more diverse images and adopts the random mixing method for multi-scale information fusion. Experiments demonstrate that IPMix outperforms state-of-the-art corruption robustness on CIFAR-C and ImageNet-C. In addition, we show that IPMix also significantly improves the other safety measures, including robustness to adversarial perturbations, calibration, prediction consistency, and anomaly detection, achieving state-of-the-art or comparable results on several benchmarks. Code is available at https://github.com/hzlsaber/IPMix .

1 Introduction

Deep neural network models have recently achieved remarkable performance on various computer vision tasks, such as zero-shot image classification [1-3], 3D object detection [4-6], and face recognition [7, 8]. In real-world scenarios, models can achieve impressive accuracy when training and test distributions are identical, but challenges appear when confronted with out-of-distribution examples [9-11], such as natural corruptions [12], adversarial perturbations [13], and anomaly patterns [14], necessitating robustness across distribution shifts. Data augmentation has been proposed to partially alleviate this issue, which applies diverse transformations on clean images to generate new training examples [15, 16]. Furthermore, a high diversity of augmented images enables neural networks to resist data distribution shifts and improve robustness [17]. Data augmentation approaches generally fall into three subgroups: image-level, patch-level, and pixel-level augmentations.

Image-level augmentation techniques [18-20] apply transformations on the whole image, such as brightness, sharpness, and solarization, to increase the total amount of training data. Patch-level augmentation techniques [21, 22] typically mask or replace a region of an image, compelling classifiers to focus on less discriminative portions. Meanwhile, pixel-level augmentation techniques [23, 24] mix images using pixel-wise weighted averages to increase diversity within the training dataset.

CutMix

CutOut

Mixup

AugMix

PixMix

IPMix

Previous studies have focused on either pixel-level or patch-level information to improve model performance. However, most of these techniques are label-variant, which may lead to manifold intrusion [25, 26] and decrease performance on unseen data. Simultaneously, a limitation of image-level data augmentation techniques is the computationally expensive search for an optimal augmentation policy, often exceeding the training process's complexity [18, 19]. Given these considerations and the potential for enhancing data augmentation strategies, we mainly discuss one question in this paper: How to take advantage of the strengths of the three methods while avoiding their drawbacks?

Our contributions are as follows:

We propose IPMix , a label-preserving data augmentation approach, which integrates three levels of data augmentation into a single framework with limited computational overhead, demonstrating that these approaches are complementary and that a unification among them is necessary to achieve robustness.

To further enhance model performance, IPMix incorporates structural complexity from synthetic data at various levels to produce more diverse images. Additionally, we employ random mixing methods and scar-like image patches for multi-scale information fusion.

Extensive experiments demonstrate that IPMix achieves state-of-the-art corruption robustness and improves numerous safety metrics compared with other data augmentation approaches.

IPMix integrates the three data augmentation techniques in a label-preserving fashion, effectively circumventing potential manifold intrusion and maintaining label consistency[27]. Furthermore, inspired by prior work, IPMix eliminates the need to search for an optimal data augmentation policy, thus reducing computational costs. By addressing these challenges, IPMix has achieved significant improvements, as depicted in Figure 2. In comparison to other methods that focus on leveraging one of these categories for enhancement, IPMix achieves stateof-the-art results in accuracy and robustness.

Since IPMix involves different levels of data augmentation techniques, it naturally motivates us to design a novel mixing method for better information fusion. Previous research has demonstrated that enhancing training data diver-

IPMix

PixMix

70

AugMix

Robustness(mCE)

65

Co-MixUp

TrivalAugment

60

SaliencyMix

PuzzleMix

口

AutoAugment

RandAugment

Pixel-level

55

MixUp

Patch-level

Image-level

CutOut

CutMix

x

★

IPMix

Baseline

80.50

80.75

81.00

81.25

81.50

81.75

82.00

82.25

82.50

Accuracy(%)

sity [23, 28, 29] and image structural complexity [30, 31] is crucial for improving model robustness. The structural complexity of synthetic data, such as fractals and statistical information, can bolster model performance through pre-training [32] or blending with clean images [24]. For better data integration, IPMix mixes clean images with synthetic pictures at different scales by random mixing to improve structural complexity, which can generate more diverse images to improve robustness.

Building on the enhancement of corruption robustness, we further extend IPMix's capabilities to enhance various safety metrics to fulfill the demands of constructing secure and reliable systems in real-world situations [11]. We demonstrate that IPMix improves numerous safety metrics, including corruption robustness, calibrated uncertainty estimates, adversarial robustness, anomaly detection, and prediction consistency. On CIFAR-10-C and CIFAR-100-C, IPMix achieves the best results across different architectures. On ImageNet, IPMix outperforms previous methods and gains a substantial improvement on various safety measure benchmarks, achieving state-of-the-art or comparable results on ImageNet-R, ImageNet-A, and ImageNet-O [33, 34].

m

1-m

equalize

translate-x

posterize

00

Multiplication

Pixel-level

Addition

PatchSize=12

evel

lon

2 Related Works

2.1 Data Augmentation

Data augmentation is crucial to the success of modern neural networks, contributing significantly to the improvement of model generalization performance. The presented data augmentation approaches can be classified into three high-level categories: image-level, pixel-level, and patch-level augmentations.

Image-level data augmentation. Image-level data augmentation methods are commonly labelpreserving, applying transformations on the whole image to improve data diversity. AutoAugment [19] utilizes reinforcement learning to automatically search optimal compositions of transformations. Adversarial AutoAugment [35] generates adversarial images to extend data and produces a dynamic policy during training. TrivialAugment [36] randomly selects an operation and the magnitude to reduce search space and improve performance. AugMix [37] uses multiple transformations to create high diversity of augmented images, achieving state-of-the-art results on corruption robustness and calibration. AugMax [29] unifies diversity and hardness to search for the worst-case mixing strategy. PRIME [38] uses max-entropy image transformations to boost model corruption robustness.

Pixel-level data augmentation. Pixel-level data augmentation methods mix images using pixel-wise weighted averages. MixUp [23] generates augmented images by linearly interpolating between two randomly selected images and their corresponding labels. Manifold MixUp [39] encourages neural networks to learn smooth interpolations between data points in the hidden layers, improving accuracy by comparison with MixUp. PixMix [24] utilizes structural complexity synthetic pictures, such as fractals and feature visualizations, to improve model performance. Our work shared similarities with PixMix, but we use multi-scale information and better information fusion methods to train robust models by leveraging more diverse examples.

Patch-level data augmentation. Patch-level data augmentation methods mask or replace parts of the original image with different information. CutOut [28] randomly masks out regions of a clean image to learn less discriminative portions, thereby improving accuracy. CutMix [40] replaces a patch of an original image with another randomly picked image to improve performance. Patch Gaussian [41], which inputs a patch of Gaussian noise into the clean picture, combines the improved accuracy of CutOut with the noise robustness of Gaussian. SaliencyMix [42], based on the maximum intensity pixel local in the saliency map, replaces a square patch of the original image with salient information from another image. TokenMix [43] improves the performance of vision transformers by partitioning the mixing region into multiple separated parts and mixing two images at the token level. AutoMix [44] optimizes both the mixed sample generation task and the mixup classification task in a momentum training pipeline with corresponding sub-networks in a bi-level optimization framework.

2.2 Safety Measures

When deploying network models in real-world scenarios, it is crucial to consider comprehensive security measures beyond standard accuracy. Implementing unsafe machine learning systems in high-stakes environments [45-47] can lead to incalculable losses. With the rise of multimodal large language models (MLLMs) [48-50], safety issues are receiving increasing attention because their superior performance still makes mistakes. For example, GPT-4 [49] may be confidently wrong in its predictions and disturbed by adversarial questions. Previous research has proposed various safety measures, including but not limited to robustness and calibration.

Robustness. Corruption robustness considers how to improve the model resistance to unseen natural perturbations under data distribution shifts. As a variant of the original ImageNet, ImageNet-C [51] consists of 15 diverse commonplace corruptions belonging to different categories with five levels of severity, regarded as a general benchmark for corruption robustness. In addition to natural corruption, Hendrycks et al. [33] demonstrate that models should measure generalization to various abstract visual renditions. The robustness of adversarial attacks focuses on defending against imperceptible perturbations to images [52]. Prior works have proposed that there is a trade-off between the robustness of adversarial perturbations and clean image accuracy [53, 54]. ImageNet-O and ImageNet-A [34], widely regarded as benchmarks for evaluating image classifier performance under shifts in both input data and label distributions, are utilized for anomaly detection.

Calibration. Calibrated prediction confidences, which indicate whether a model's output should be trusted, are valuable for classification models in real-world settings. Bayesian approaches [55] are widely used to deal with uncertainty estimation. Kuleshov et al. [56] utilize recalibration methods to solve the miscalibration of credible intervals. Ovadia et al. [57] provide a benchmark for evaluating the accuracy and uncertainty of models under data distributional shifts.

2.3 Training with Synthetic Data

Previous works have proved that training with synthetic data can improve performance on real datasets. Debidatta et al. [58] discover that combining synthetic annotated datasets with real data can significantly improve the performance of instance detection. Baradad et al. [31] generate synthetic data by utilizing various procedural noise models. In addition, they find that naturalism and diversity are two important properties for synthetic data to achieve comparable results with real datasets. Kataoka et al. [59, 60] propose a suite of datasets generated by formula-driven supervised learning.

3 An Attempt to Integrate Existing Approaches

Some prior studies [24, 40] have suggested that combining different data augmentation techniques with existing methods can improve accuracy on standard datasets. However, these works merely employed simple combinations without considering the compatibility between methods at different levels. Simultaneously, these studies chose the clean accuracy as the sole evaluation metric and have not taken the model's

safety performance into account. In this section, we select MixUp [23], CutMix [40], and AugMix [37] as representative data augmentation approaches for pixel-level, patch-level, and image-level, respectively, to conduct combination experiments of these approaches on CIFAR-100. Please refer to Appendix F for more details about the combination experiments.

Results on Table 1 demonstrate that simply combining different data augmentation methods may significantly impair model performance. This could be attributed to the excessive perturbation of training data caused by the combination of these methods, making the newly generated samples more challenging to identify and impacting the model's ability to learn useful features, leading to performance degradation. When multiple label-variant methods are combined, manifold intrusion

①

②

③

: P augmentation operation

: Image augmentation operation

: Original image

: Mixed image

Chain-Mixed

Mixed-input

issues may be more likely to arise. One possible solution for better information integration is to incorporate approaches ( e . g ., MixUp) into search-based data augmentation techniques [20, 36]. However, searching the space for an optimal DA policy will bring expensive computation. Furthermore, this approach aims at improving clean accuracy and does not consider the overall safety performance.

4 IPMix: A Simple Method for Training Robust Classifiers

In this section, we propose IPMix, which integrates three levels of data augmentation methods into a label-preserving approach, comprehensively improving safety metrics without sacrificing clean accuracy. We first demonstrate how to merge various techniques into a coherent framework and then propose novel approaches to achieve superior information fusion.

4.1 Integrates Different Levels into A Coherent Approach

Pixel-level & Patch-level. As a label-preserving data augmentation approach, IPMix uses the equation below to mix two input images:

Where x 1 is the input image and x 2 represents an unlabeled synthetic image ( e . g ., fractals, spectrum, or auto-generated contours). B is a mask matrix suitable for both patch-level and pixel-level data augmentation methods, and I is a binary mask filled with ones, having the same dimensions as B . ⊙ represents the element-wise product. When performing mixing operations at the patch level, we choose a patch of random size and position from B , with a value of λ (sample from Beta distribution) in this range and a value of 1 in other areas, which ensures that except for the mixing patch, the rest of the generated image comes from x 1 . When performing mixing operations at the pixel level, we treat the entire image as a patch, with a value of λ . To make it efficient, we adopt fractals as representatives of synthetic data. However, IPMix is insensitive to mixing sets change, as shown in Table 8.

Fractals are geometric shapes with structural complexities and natural geometries. While previous works [32, 61] merely use iterated function systems (IFS) to create fractal data, we employ the Escape-time Algorithm for generating "orbit trap" complex fractals to enhance dataset complexity and diversity. Please refer to Appendix E for details about generating fractal images.

The above-described method provides two key advantages: (1) We utilize a simple approach to combine operations of two levels, facilitating better information fusion. (2) Our method is labelpreserving, ensuring it is not affected by manifold intrusion while eliminating the need for label smoothing [62]. In the following sections, we refer to the method used in Eq. (1) as P-level data augmentation, signifying the employment of both patch-level and pixel-level methods.

Image-level. IPMix leverages various augmentation techniques and compositions to create a new image that does not deviate significantly from the original. Drawing inspiration from previous works [36, 37], we randomly sample operations from PIL ( e . g ., brightness, sharpness) and randomly sample strengths to enhance the diversity of training data without expensive searching. Notably, these operations are disjoint from ImageNet-C corruptions, ensuring the robustness test's validity.

The IPMix framework. To determine the most effective methods for combining P-level and imagelevel, we conducted experiments using different mixing structures to generate a diverse set of IPMix images, as illustrated in the Figure 4 and Table 2. While Linear Mix achieves excellent results in clean accuracy and corruption robustness, it performs poorly in cal-

ibrated prediction confidence. Mixed Input performs better in calibration but is inferior in accuracy and corruption robustness compared to Chain-Mixed. Consequently, we chose Chain-Mixed as the default framework for IPMix. Furthermore, the experimental results highlight the potential of establishing a general framework for integrating various data augmentation methods.

4.2 Multi-scale Information Fusion

IPMix can enhance the diversity and the structural complexity of training data to improve model performance. However, we found that simple mixing methods restrict the model's capabilities.

To overcome this issue, we use random mixing and scar-like image patches for achieving more effective information fusion.

Random mixing. In previous data augmentation works, it is typical to either linearly mix two images or extract specific image features, such as saliency [22, 42], which requires additional computations, for image mixing. As IPMix incorporates various levels of operations, its objective is to enhance the mixing of images, ultimately increasing data diversity. To accomplish this objective, IPMix employs four mixing operations: addition, multiplication, random pixel mixing, and random element mixing [63]. Random pixel mixing creates a binary mask of size H × W × 1 that operates on each channel sequentially, while random element mixing generates a binary mask of size H × W × 3 (RGB) that applies to all channels simultaneously. An example is shown in Figure 5. The experiments in Appendix B.1 show that both operations are beneficial to better information mixing between images and fractals.

Scar-like image patches. IPMix-Scar employs a long, thin rectangular box filled with an image patch to enhance dataset diversity, which has proven effective for anomaly detection [64]. An example of patch mixing is illustrated in Figure 5. First, IPMix randomly selects a point and a scar or square of the previously chosen size from the current image. Next, IPMix crops corresponding portions of the current image and the fractal picture and combine them.

Finally, we obtain IPMix, which employs various levels of data augmentation to create diverse transformations with image structural complexity and data diversity. Figure 3 displays an example of IPMix, where k denotes the number of augmented chains, and t represents the maximum number of times an image can be augmented. The algorithm of IPMix is summarized in Appendix D.

5 Experiments

In this section, we showcase the significant performance improvements brought by IPMix on clean datasets in multiple settings. We present the evaluation results of IPMix for image classification on three datasets-CIFAR-10, CIFAR-100 [65], and ImageNet [66]-across various models. Besides clean Classification, we assess IPMix on diverse safety tasks, including adversarial attack robustness, corruption robustness, prediction consistency, calibration, and anomaly detection. Please refer to Appendix C for details about the evaluation metrics. Lastly, we evaluate the properties of IPMix in thorough ablation studies and compare our approach with different levels of methods.

Random Pixels Mixing

Random Elements Mixing

We evaluate IPMix on CIFAR-10-C, CIFAR-100-C, and ImageNet-C to measure its resistance to corruption data shifts. We test IPMix on CIFAR-10-P, CIFAR-100-P, and ImageNet-P to measure network prediction stability against minor perturbations. To thoroughly demonstrate our method's capabilities, we assess it on supplementary datasets, including ImageNet-R, ImageNet-O, and ImageNet-A. Experiments on these datasets validate our approach's robustness under real-world distribution shifts.

5.1 Evaluation on CIFAR

We experiment with different backbone architectures on CIFAR-10 and CIFAR-100, including 40-4 Wide ResNet [67], 28-10 Wide ResNet, ResNeXt-29 [68], and Resnet-18 [69]. We compare IPMix with various data augmentation methods, including CutOut, MixUp, CutMix, AugMix, and PixMix. Please refer to Appendix A for more details about the training configurations.

Accuracy. In Table 3, we demonstrate that IPMix improves standard accuracy across architectures. In comparison with other approaches, IPMix achieves the best or comparable accuracy, showing the improvement of safety measures is not at the cost of hurting clean accuracy.

Corruption robustness. Results show that IPMix substantially improves corruption robustness across architectures. Compared to AugMix on CIFAR-100-C, IPMix achieves 4.7% (40-4) and 4.9% (28-10) improvement on WideResNet, 6% on ResNeXt, and 5.1% on ResNet. Table 4 demonstrates that IPMix achieves state-of-the-art results on both CIFAR-10-C and CIFAR-100-C.

Calibration. We utilize RMS calibration error [70] to evaluate the empirical frequency of correctness. As depicted in Figure 6, IPMix surpasses other methods, achieving state-of-the-art results.

Prediction consistency. We leverage the mean flip rate (mFR) to evaluate prediction consistency on CIFAR-10-P and CIFAR-100-P [51]. IPMix achieves the lowest mFR, as shown in Figure 7.

MixUp

RMS=14.6%

RMS=10.5%

RMS=11.4%

RMS=9.3%

RMS=6.3%

RMS=7.7%

RMS=2.8%

Vanilla

CutOut

CutMix

AugMix

PixMix

IPMix

Confidence

10.7

8.9

11.9

12

5.6

5.6

4.3

0

2

4

6

8

10

12

14

CutOut CutMix AugMix PixMix

IPMix

Prediction Consistency

96.8

96.9

98.5

97.4

95.4

91.3

89.4

84

86

88

90

92

94

96

98

100

CutOut CutMix AugMix PixMix

IPMix

Adversarial error rate(%)

Adversarial Robustness

Adversarial robustness. This measure evaluates the resistance of adversarially perturbed by projected gradient descent. We utilize PGD [71] to verify the adversarial robustness of image classifiers. The results in Figure 7 show that IPMix achieves the lowest error.

5.2 Evaluation on ImageNet

For ImageNet experiments, we compare different data augmentation methods, including MixUp, CutOut, CutMix, AugMix, AugMax [29], and PixMix. We utilize SGD optimizer with an initial learning rate of 0.01 to train ResNet-50 for 180 epochs following a cosine decay schedule. Please refer to Appendix A for more details about the training configurations.

IPMix achieves state-of-the-art or comparable performances on a broad range of safety measures, as shown in Table 5. Compared with other methods, IPMix improves the resistance of out-of-distribution shifts without reducing clean accuracy. On corruption robustness, IPMix outperforms Vanilla by 15.6% and AugMix by 5.5% , achieving state-of-the-art results. On ImageNet-R, IPMix demonstrates the ability to improve rendition robustness, increasing by 6.6% by comparison with Vanilla. On ImageNet-P, IPMix improves mFR by 9.2% over Vanilla and 2.3% over PixMix. On calibration tests, IPMix surpasses all methods on ImageNet-C, ImageNet-R, and ImageNet-A, improving RMS by 0.1% , 5.1% , and 13.5% by comparison with the second-best approach. Furthermore, IPMix achieves convincing results on ImageNet-A and ImageNet-O, demonstrating its exceptional ability in anomaly detection. The results demonstrate that IPMix can roundly improve safety metrics.

5.3 Ablation Study

In this paragraph, we evaluate the properties of our approach by ablation experiments. We first study the influence of different parts of IPMix on performance and then assess the stability of IPMix under various mixing sources. Please refer to more ablation experiments in Appendix B.1.

Components of IPMix. In this section, we evaluate the influence of different IPMix components on perfor-

Table 7: Ablation results of different components of IPMix on CIFAR-100. Mean and standard derivation over three random seeds is shown for each experiment. Bold is the best.

mance. We execute ablation experiments on the three primary IPMix constituents: image-level, patch-level, and pixel-level. The results show the indispensable contribution of each component to enhancing model performance, demonstrating that these approaches are complementary and that a unification among them is necessary to achieve robustness. The ablation experiment results are shown in Table 6 and Table 7. Please refer to thorough analysis in Appendix J.

Mixing sources. The excellent performance of IPMix is partly due to the structural complexity of fractal pictures. In this part, we examine the sensitivity of IPMix to different fractal sources on CIFAR-100. We report clean accuracy, corruption robustness, and calibration from different sources with WRN40-4. Fractal + FVis is the default setting of PixMix, which consists of fractals and feature visualization. FractalDB [59] consists of fractal images generated by Iterated Function System (IFS).

Table 8: Ablation results on IPMix across different mixing sets.The results show that IPMix is insensitive to mixing sets change.

RCDB [60] consists of auto-generated contours. Dead Leaves and Spectrum generated from generative image models [31]. The full results show in Table 8.

5.4 The Comparison with Different Levels of Method

In this section, we perform an extensive performance comparison between IPMix and a range of existing methods using multiple metrics. We consider AutoAugment, RandAugment, and TrivialAugment [36] as representative image-level techniques, while SaliencyMix, PuzzleMix [72], and Co-Mixup [22] serve as typical patch-level techniques. For pixel-level methods, Manifold Mixup stands as our representative choice. IPMix does not require searching for the optimal DA policy like image-level techniques. In contrast to patch-level approaches, IPMix eliminates the need for saliency computations. The results in Table 9 show that IPMix outperformed all other methods on all metrics.

6 Analysis of IPMix

IPMix combines three levels of data augmentation into a unified, label-preserving technique to improve model performance. We believe that IPMix's superior performance is due to the increased

Image

Baseline

AugMix

PixMix

CutMix

IPMix

Baseline

AugMix

PixMix

CutMix

IPMix

data diversity and enhanced regularization effect. For a more intuitive demonstration of these effects, we utilize t-SNE and Class Activation Mapping (CAM) [73] for visualizations, as shown in Figure 8.

Increasing diversity. IPMix increases the diversity of training data by mixing data at multiple levels, enabling the model to learn a greater variety of feature combinations and patterns. Furthermore, the integration of synthetic data from distinct distributions ( e . g ., fractals), further amplifies this diversity.

Enhanced regularization effect. The approach of mixing data also serves as a potent regularization technique. By randomly mixing samples, the model is compelled to learn more robust features rather than overly relying on specific sample or class characteristics, which reduces the risk of overfitting and enhances the model's performance in different environments.

7 Conclusion

We propose IPMix, which leverages different levels of augmentation techniques and image structural complexity to improve model performance. By employing random mixing methods, we facilitate more effective information fusion. The experimental results indicate that IPMix can significantly improve various safety metrics. We hope our work will attract attention to joining different methods into coherent and synergetic approaches to improve robustness and other safety measures. This adaptation is crucial given the growing importance of safety requirements in systems design.

References

X. Wang, Chen Chen, Yuhu Cheng, and Z. Jane Wang. Zero-shot image classification based on deep feature extraction. IEEE Transactions on Cognitive and Developmental Systems , 10:432-444, 2018.

Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18102-18112, 2021.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations , 2021.

Daniel Maturana and Sebastian A. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 922-928, 2015.

Binh Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7652-7660, 2018.

Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3144-3153, 2021.

Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. Proceedings of the 25th ACM international conference on Multimedia , 2017.

H. Wang, Yitong Wang, Zheng Zhou, Xing Ji, Zhifeng Li, Dihong Gong, Jin Zhou, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5265-5274, 2018.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning , 2019.

Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Edoardo Debenedetti, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. ArXiv , abs/2010.09670, 2020.

Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. ArXiv , abs/2109.13916, 2021.

Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and corruptions in natural corruption robustness. ArXiv , abs/2102.11273, 2021.

Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. ArXiv , abs/1702.04267, 2017.

Mahmoud Said Elsayed, Nhien-An Le-Khac, Soumyabrata Dev, and Anca Delia Jurcut. Network anomaly detection using lstm based autoencoder. Proceedings of the 16th ACM Symposium on QoS and Security for Wireless and Mobile Networks , 2020.

Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Keepaugment: A simple information-preserving data augmentation approach. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1055-1064, 2020.

Dan Andrei Calian, Florian Stimberg, Olivia Wiles, Sylvestre-Alvise Rebuffi, Andr'as Gyorgy, Timothy A. Mann, and Sven Gowal. Defending against image corruptions through adversarial augmentations. ArXiv , abs/2104.01086, 2021.

Adarsh Subbaswamy, Roy J. Adams, and Suchi Saria. Evaluating model robustness and stability to dataset shift. In International Conference on Artificial Intelligence and Statistics , 2021.

Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Neural Information Processing Systems , 2019.

Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 113-123, 2019.

Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 3008-3017, 2019.

Ryo Takahashi, Takashi Matsubara, and Kuniaki Uehara. Ricap: Random image cropping and patching data augmentation for deep cnns. In Asian Conference on Machine Learning , 2018.

Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. ArXiv , abs/2102.03065, 2021.

Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ArXiv , abs/1710.09412, 2017.

Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Xiaodong Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16762-16771, 2021.

Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regularization. In AAAI Conference on Artificial Intelligence , 2018.

Raphael Baena, Lucas Drumetz, and Vincent Gripon. Preventing manifold intrusion with locality: Local mixup. ArXiv , abs/2201.04368, 2022.

Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. ArXiv , abs/2001.07685, 2020.

Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. ArXiv , abs/1708.04552, 2017.

Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. In Neural Information Processing Systems , 2021.

Raphael Gontijo Lopes, Sylvia J. Smullin, Ekin Dogus Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. ArXiv , abs/2002.08973, 2020.

Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. In Neural Information Processing Systems , 2021.

Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images. International Journal of Computer Vision , 130:990 - 1007, 2021.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8320-8329, 2020.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Xiaodong Song. Natural adversarial examples. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15257-15266, 2019.

Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhaobai Zhong. Adversarial autoaugment. ArXiv , abs/1912.11188, 2019.

Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 754-762, 2021.

Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ArXiv , abs/1912.02781, 2019.

Apostolos Modas, Rahul Rade, Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Prime: A few primitives can boost robustness to common corruptions. In European Conference on Computer Vision , 2021.

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning , 2018.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6022-6031, 2019.

Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin Dogus Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. ArXiv , abs/1906.02611, 2019.

A. F. M. Shahab Uddin, Mst Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bae. Saliencymix: A saliency guided data augmentation strategy for better regularization. ArXiv , abs/2006.01791, 2020.

Jihao Liu, B. Liu, Hang Zhou, Hongsheng Li, and Yu Liu. Tokenmix: Rethinking image mixing for data augmentation in vision transformers. In European Conference on Computer Vision , 2022.

Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Jianzhu Guo, and Stan Z. Li. Automix: Unveiling the power of mixup for stronger classifiers. In European Conference on Computer Vision , 2021.

Samuel G. Finlayson, Isaac S. Kohane, and Andrew Beam. Adversarial attacks against medical deep learning systems. ArXiv , abs/1804.05296, 2018.

Nicolas Papernot, Patrick Mcdaniel, Arunesh Sinha, and Michael P. Wellman. Sok: Security and privacy in machine learning. 2018 IEEE European Symposium on Security and Privacy (EuroS&P) , pages 399-414, 2018.

Ludmila Skaf, Elvira Buonocore, Stefano Dumontet, Roberto Capone, and Pier Paolo Franzese. Applying network analysis to explore the global scientific literature on food security. Ecol. Informatics , 56:101062, 2020.

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. ArXiv , abs/2302.14045, 2023.

OpenAI. Gpt-4 technical report. ArXiv , abs/2303.08774, 2023.

Yongliang Shen, Kaitao Song, Xu Tan, Dong Sheng Li, Weiming Lu, and Yue Ting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. ArXiv , abs/2303.17580, 2023.

Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ArXiv , abs/1903.12261, 2018.

Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9185-9193, 2017.

Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Loddon Yuille, and Quoc V. Le. Adversarial examples improve image recognition. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 816-825, 2019.

Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Loddon Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 501-509, 2018.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ArXiv , abs/1706.04599, 2017.

Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. ArXiv , abs/1807.00263, 2018.

Yaniv Ovadia, Emily Fertig, Jie Jessie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. ArXiv , abs/1906.02530, 2019.

Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance detection. 2017 IEEE International Conference on Computer Vision (ICCV) , pages 1310-1319, 2017.

Hirokatsu Kataoka, Asato Matsumoto, Ryosuke Yamada, Yutaka Satoh, Eisuke Yamagata, and Nakamasa Inoue. Formula-driven supervised learning with recursive tiling patterns. 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) , pages 4081-4088, 2021.

Hirokatsu Kataoka, Ryo Hayamizu, Ryosuke Yamada, Kodai Nakashima, Sora Takashima, Xinyu Zhang, Edgar Josafat Martinez-Noriega, Nakamasa Inoue, and Rio Yokota. Replacing labeled real-image datasets with auto-generated contours. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 21200-21209, 2022.

Connor Anderson and Ryan Farrell. Improving fractal pre-training. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2412-2421, 2021.

Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Neural Information Processing Systems , 2019.

Cecilia Summers and Michael J. Dinneen. Improved mixed-example data augmentation. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 1262-1270, 2018.

Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9659-9669, 2021.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv , abs/1605.07146, 2016.

Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5987-5995, 2016.

Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2015.

Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. ArXiv , abs/1812.04606, 2018.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ArXiv , abs/1706.06083, 2017.

Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. ArXiv , abs/2009.06962, 2020.

Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision , 128:336-359, 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes IPMix, a data augmentation strategy to improve model robustness (and other safety metrics like calibration). The idea builds upon PixMix, a recent data augmentation strategy generating diversity via synthetic pictures. IPMix performs this at multiple scales (pixel, patch, image levels) and mixes the resulting transformed images. The evaluations are performed on CIFAR-10, CIFAR-100, ImageNet and their shifted versions against several data augmentation baselines.\n\nSTRENGTHS:\nThe paper is very well written, it is quite easy to understand. The method is clear and reasonable, given that distribution shifts may happen at different levels it makes sense to address them at different levels. The comparisons and ablation studies are well thought. Additional experimental results in supplementary are also helpful.\n\nWEAKNESSES:\nMy main concerns are about ImageNet results which suggest minor improvements over PixMix, e.g. ImageNet-C results are 78.1 vs 77.8. The results are close for other benchmarks too like ImageNet-R and ImageNet-P (very close to AugMix). Same goes for anomaly detection results. This makes me wonder: Does using synthetic data and mixing have inherent limitations in terms of generating diversity? For example, what happens if you increase your diversity using recent diffusion models to generate data, e.g. as in [A]?\n\nMy other questions and comments:\n- I would be interested in seeing comparisons to DeepAugment as well, as it has been shown to improve robustness notably when combined with AugMix.\n- How does IPMix perform against natural distribution shifts like ImageNet-V2 and ObjectNet? What about more recent ImageNet benchmarks like ImageNet-3DCC and ImageNet-E? I believe having these evaluations could also provide some insights into the limitations I mentioned above.\n- When I was reading L201 I thought about the work of Mintun et al. ([15] in supmat) and I was happy to see that you added evaluations on CIFAR-C-bar in the supplementary, which again seems to suggest minor improvements over PixMix. What would be the results for ImageNet-C-bar? \n- It is nice that you added training overhead in supmat too, but I think it should be better linked in the main paper as I was going to ask about it and it wasn’t pointed out in the main paper.\n- I understand this is a CNN based study but given the landscape and domination of ViTs I believe evaluations on ViT backbones are also necessary to truly understand the performance of IPMix.\n\n[A] Sariyildiz et al. Fake It Till You Make It: Learning Transferable Representations From Synthetic ImageNet Clones, CVPR 2023.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nPlease see weaknesses.\n\nAfter rebuttal, I am increasing my rating to borderline accept.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a data augmentation method, named IPMix, which is designed to improve robustness without hurting clean accuracy. Specifically, IPMix utilizes different levels of data augmentation (image-level, patch-level, and pixel-level) to increase the diversity of training data and introduces structural complexity to generate more diverse images. The authors have evaluated the robustness of IPMix to adversarial perturbations, calibration, prediction consistency, and anomaly detection.\n\nSTRENGTHS:\n1. The evaluation of robustness and ablation study is extensive. The authors have comprehensively validated the robustness of IPMix, including adversarial perturbations, calibration, prediction consistency, and anomaly detection. Moreover, the ablation of design choices is in detail.\n2. The results on small-scale datasets are promising. Overall, IPMix achieves promising accuracy on CIFAR datasets, as well as safety metrics.\n\nWEAKNESSES:\n1. Comparisons with RandAugment [1]. The whole design of IPMix seems to be a complicated combination of multiple existing data augmentation methods and the image-level transformations are important components of RandAugment. According to Tab. 6, these image-level transformations are the most effective modules in the whole design and it is suggested to compare IPMix with RandAugment, which is one of the most popular choices in current training frameworks.\n2. Generalization ability to competing methods. Current training frameworks of object recognition [2][3] often utilize multiple data augmentation methods during training, the reviewer wonders whether IPMix is compatible with these methods and could further improve the performance.\n3. Results on ImageNet. The accuracy of IPMix on imageNet is not very convincing and the whole results on ImageNet seem strange, as Mixup, CutMix both lead to inferior results compared to the baseline. This phenomenon is contradictory to the results of their papers and the explanation of this part is highly appreciated.\n\n[1] Cubuk E D, Zoph B, Shlens J, et al. Randaugment: Practical automated data augmentation with a reduced search space[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 2020: 702-703.\n[2] Liu Z, Mao H, Wu C Y, et al. A convnet for the 2020s[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 11976-11986.\n[3] Li K, Wang Y, Gao P, et al. Uniformer: Unified transformer for efficient spatiotemporal representation learning[J]. arXiv preprint arXiv:2201.04676, 2022.\n\nQUESTIONS:\nApart from the points in Weaknesses, the reviewer also has some other questions:\n\n1. Validation across different architectures. The authors have only validated IPMix on ResNet which not be enough to prove the generalization ability of IPMix.\n2. Revision on Fig. 4. It is advised to add the names of the three methods from Tab. 2 to Fig. 4 which will be easier for reading.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose IPMix -- a new data augmentation technique applied during training that improves model robustness across a variety of test-time tasks (e.g., distributional shift, adversarial data augmentation) without compromising accuracy on in distribution tasks (when compared to a model trained in a traditional manner). The acronym IPMix indicates that the technique utilizes augmentations at the Image (I) and Pixel/Patch (P) level and mixes these together with each other and the original image. IPMix augments samples via a convex combination of an augmented and original image, where the novelty in the augmentation operations applied to the image (compared to existing augmentation techniques e.g. Augmix) come from operations between an image and unlabeled synthetic images (e.g., fractals). IPMix research is motivated to produce safe ML systems (e.g., improve model robustness and calibration).\n\n**Main Contributions:**\n\n* IPMix: A label preserving data augmentation technique that achieves most balanced performance across a variety of tasks desired for safe ML systems (accuracy, robustness, calibration, and anomaly detection).\n* A thorough empirical study of IPMix using multiple datasets, architectures, and tasks compared to large set of existing data augmentation schemes.\n\n**Post-rebuttal update**\nThe authors addressed all of my weaknesses and questions and I have raised my score from a weak accept to accept.\n\nSTRENGTHS:\n* **Experiments**: Experimental results with IPMix compared to existing data augmentation methods (in terms of model accuracy, robustness, and calibration) are impressive and are largely consistent across many datasets and architectures.\n* **Data augmentation analysis**: Some analysis of models trained with varying data augmentation schemes highlights strengths of IPMix (Figure 8) in comparison to existing data augmentation schemes.\n\nWEAKNESSES:\n-\t**Motivation**: The use of unlabeled synthetic images in the Pixel/Patch mixing did not seem well motivated to me. I agree that IPMix seems to work well based on the experimental results and understand that the choice of unlabeled synthetic images avoids mislabeled augmented training images. However, I feel that some motivation (practical or theoretical) is needed to help clarify and better motivate the design of IPMix and why it would improve performance beyond existing methods (e.g., Augmix) that do not produce augmented images with incorrect labels.\n-\t**Paper organization**: A lot of the paper is relegated to the appendix (Appendix F (combination experiments), Appendix E (Fractal generation), Appendix B.1 (IPMix insensitive to mixing set change), Appendix C (IPMix algorithm). I understand space constraints and submission limitations, but I would have liked to see more of the details of IPMix and experiments informing its design in the main body. Perhaps the related work section could have been in the appendix to make space for this.\n\nQUESTIONS:\n1.\t**Motivation**: What is the motivation for unlabeled synthetic images (e.g., fractals) used in IPMix set? I suspect this is very clear to the authors, but it is not well motivated or explained in the main body. \n2.\t**Experiments**: It is unclear from the paper and appendix if multiple replicates are trained for each architecture/dataset/augmentation configuration (Tables 3, 4, and Figure 6). I am under the impression that a single model was trained for each configuration. If that is the case, it is unclear how much deviation (in accuracy, robustness, calibration) the models trained with varying data augmentation schemes have. I would typically expect some result on this for smaller datasets (e.g., CIFAR-10). Is this something you considered that I missed?\n3.\t**Overhead/cost**: Is there an observable overhead/cost of using IPMix during training (when compared to vanilla training or other data augmentation techniques)? In my experience, Augmix increases training time due to augmentation scheme so I am curious if IPMix is more/less costly than Augmix (as it produces better models than Augmix).\n4.\t**Robustness**: How do models trained using IPMix compare to robustbench leaderboard Common corruptions (i.e., CIFAR-10-C, ImageNet-C)? I suspect they are comparable, but it would potentially strengthen the work by comparing to SOTA models from robustbench. The architectures used in the paper are commonly used on robustbench so it would likely be an easy/straightforward comparison. *In this direction: Did you consider utilizing the Jensen-Shannon divergence (JSD) loss utilized in the original Augmix paper in experiments with IPMix? I suspect this may offer additional improvements to models trained with IPMix so it may be worthwhile to consider if comparing to robustbench models.*\n5. **Figure 6**: Is this calibration error for a single architecture trained using each data augmentation technique evaluated on CIFAR-100 (e.g. ResNet-18)? The caption does not specify.\n\nMinor suggestions:\n* Table 1 is provided but never referenced or commented on in the text of the paper. I would typically expect some minor commentary on tables/figures.\n* Table 1: “Accuracy Error” seems like odd/confusing phrasing to me. Wouldn’t “Classification Error” be clearer?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new data augmentation method, IPMix, for vision models that integrates three levels of augmentation techniques (image level, patch level, and pixel level). The approach is label preserving and enforces extra structural complexity to enhance the robustness and other safety measures. Experimental results on multiple open datasets and backbone models are provided to support the research claims in the paper.\n\nSTRENGTHS:\n1. In addition to clean accuracy, the authors seek to optimize both the model robustness and various safety measures in the paper.\n\n2. The provided evaluations are extensive and improvements are observed in multiple performance perspectives compared to the baselines.\n\n3. The paper is generally easy to understand and follow.\n\nWEAKNESSES:\n1. Throughout the whole paper, it is kind of hard to understand what is the main technical contribution made in the paper that is original and novel. Although multiple components have been introduced, they seem to be trivial applications and combinations of previously published approaches, since the design of each component of inspired by previous works. Besides, those components are not connected and introduced through a coherent idea but are more like random pieces that are very loosely connected. What main technical challenges have been solved in this paper? If we consider the combination pipeline of different augmentations as the main contribution, it is still trivial because the authors just experimented with different intuitive combination methods in Table 2 and selected the overall best one.\n\n2. From the paper, I don't understand exactly why label-variant augmentations lead to manifold intrusion and lower robustness against unseen data. If the label-preserving property is important, can we enhance the CutMix and MixUp by mixing original images with fractal images and achieving a similar performance as IPMIx?\n\n3. In the evaluation section, the authors only focus on showing the results but do not provide any analysis of the results, which results in the fact that the only piece of information expressed there is IPMix is better than the baselines in various settings, but not about why IPMix works better.\n\n4. The ablation study is not designed in the correct way. Its current form only conveys the message that all three levels of augmentations are needed while removing any one of them will lead to degradations in the performance. In addition to that, the authors should use experiments to show the performance degradation without proper combinations, fractals (label variance), and the pieces in Section 4.2.\n\n5. The authors mentioned the computation efficiency improvement caused by the approach, but no evaluations are performed to test the computation efficiency between IPMix and conventional image-level augmentations.\n\n6. Here are some minor comments:\n\n    (1) Since the paper depends on image-specific intuitions, it would be good to be precise on the scope in the paper title. \n\n    (2) In line 183, the \\lambda is directly used without introducing its purpose and functionality.\n\n    (3) The specific design scope and definition of IPMix is unclear. The title of Section 4.1 is about IPMix, while Section 4.2 further introduces more design components. Do they belong to part of IPMix? If so, the organization of the paper should be fixed.\n\n    (4) The ticks in Figure 6 are too small and hard to recognize.\n\nQUESTIONS:\nI think the main question that should be answered is the main technical contribution made in this paper compared to the existing literature, and what main challenges have been solved.\n\nLIMITATIONS:\nYes, the limitations are discussed in the Appendix of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents IPMix, a new data augmentation strategy that enhances robustness without compromising clean accuracy. The proposed approach diversifies the training data with limited computational overhead. IPMix introduces structural complexity at different levels to generate a wider variety of images and employs a random mixing method for multi-scale information fusion to bolster robustness.\n\nSTRENGTHS:\n- The paper addresses augmentation from both a clean accuracy and safety perspective, which is commendable.\n- The paper is generally well-written and easy to follow.\n- The figures provided in the paper are informative and well-polished and enhance the understanding of the proposed strategy.\n- The paper includes many implementation details and ablation studies in Appendix.\n\nWEAKNESSES:\n- The paper could benefit from including error bars to account for variance across different seeds, particularly when the performance of each method is comparable.\n- The paper could provide a more in-depth analysis of why the proposed method improves each safety metric. While the explanations regarding diversity and regularization effects are reasonable, it would be beneficial to understand which part of the pipeline is most crucial for a specific safety measure (e.g., robustness, calibration). The significance of increasing structural complexity using \"fractals\" could be discussed more.\n- The title emphasizes the \"label-preserving\" property, yet it is unclear how vital this is. The authors could discuss the potential outcomes of mixing two images with different labels and linearly interpolating the label.\n- The experimental focus is primarily on ResNet variants. Including results on the performance of other architectures (e.g., Transformer) could strengthen the paper.\n\nQUESTIONS:\n- There seems to be no discussion on Table 1, which feels disconnected from the rest of the paper. Could the authors elaborate on this?\n- In Section 4.1 \"The IPMix framework\", the terms \"Chain-Mixed\", \"linear mix\", and \"mixed input\" are not clearly defined. Could the authors clarify these and possibly include the corresponding framework name in Figure 4 alongside the number?\n- In Section 3, integration experiments are described. It is suggested that combining previous approaches may result in diminished performance, as the optimal parameter for using each data augmentation independently may not be optimal when used in combination. Could decreasing the augmentation strength of each method yield better performance?\n- Table 6 suggests that image-level data augmentation is the most crucial part of CIFAR100. Is this observation consistent with ImageNet as well?\n- How can the advantages of the three methods be leveraged while avoiding their drawbacks? Could the authors provide more specific information about the drawbacks of each method?\n\nLIMITATIONS:\nThe authors discuss the limitation and societal impact in Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-17"}