SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenglin Huang $^{1}$ , Xianan Bao $^{1}$ , Na Zhang $^{1*}$ , Qingqi Zhang $^{2}$ , Xiaomei Tu $^{3}$ , Biao Wu $^{1}$ , and Xi Yang $^{4}$
$^{1}$ School of Artificial Intelligence, Zhejiang Sci-Tech University  $^{2}$ Yamaguchi University  $^{3}$ ZGUC
<sup>4</sup>University of Science and Technology of China, Hefei, China
Data augmentation has been proven effective for training high-accuracy convolutional neural network classifiers by preventing overfitting. However, building deep neural networks in real-world scenarios requires not only high accuracy on clean data but also robustness when data distributions shift. While prior methods have proposed that there is a trade-off between accuracy and robustness, we propose IPMix, a simple data augmentation approach to improve robustness without hurting clean accuracy. IPMix integrates three levels of data augmentation (image-level, patch-level, and pixel-level) into a coherent and label-preserving technique to increase the diversity of training data with limited computational overhead. To further improve the robustness, IPMix introduces structural complexity at different levels to generate more diverse images and adopts the random mixing method for multi-scale information fusion. Experiments demonstrate that IPMix outperforms state-of-the-art corruption robustness on CIFAR-C and ImageNet-C. In addition, we show that IPMix also significantly improves the other safety measures, including robustness to adversarial perturbations, calibration, prediction consistency, and anomaly detection, achieving state-of-the-art or comparable results on several benchmarks. Code is available at https://github.com/hzlsaber/IPMix.
Deep neural network models have recently achieved remarkable performance on various computer vision tasks, such as zero-shot image classification [1-3], 3D object detection [4-6], and face recognition [7, 8]. In real-world scenarios, models can achieve impressive accuracy when training and test distributions are identical, but challenges appear when confronted with out-of-distribution examples [9-11], such as natural corruptions [12], adversarial perturbations [13], and anomaly patterns [14], necessitating robustness across distribution shifts. Data augmentation has been proposed to partially alleviate this issue, which applies diverse transformations on clean images to generate new training examples [15, 16]. Furthermore, a high diversity of augmented images enables neural networks to resist data distribution shifts and improve robustness [17]. Data augmentation approaches generally fall into three subgroups: image-level, patch-level, and pixel-level augmentations.
Image-level augmentation techniques [18-20] apply transformations on the whole image, such as brightness, sharpness, and solarization, to increase the total amount of training data. Patch-level augmentation techniques [21, 22] typically mask or replace a region of an image, compelling classifiers to focus on less discriminative portions. Meanwhile, pixel-level augmentation techniques [23, 24] mix images using pixel-wise weighted averages to increase diversity within the training dataset.
Figure 1: Visual comparison of various data augmentation methods. IPMix utilizes the structural complexity of fractals and multi-scale information to generate more diverse examples.
Previous studies have focused on either pixel-level or patch-level information to improve model performance. However, most of these techniques are label-variant, which may lead to manifold intrusion [25, 26] and decrease performance on unseen data. Simultaneously, a limitation of image-level data augmentation techniques is the computationally expensive search for an optimal augmentation policy, often exceeding the training process's complexity [18, 19]. Given these considerations and the potential for enhancing data augmentation strategies, we mainly discuss one question in this paper: How to take advantage of the strengths of the three methods while avoiding their drawbacks?
- We propose IPMix, a label-preserving data augmentation approach, which integrates three levels of data augmentation into a single framework with limited computational overhead, demonstrating that these approaches are complementary and that a unification among them is necessary to achieve robustness.
- To further enhance model performance, IPMix incorporates structural complexity from synthetic data at various levels to produce more diverse images. Additionally, we employ random mixing methods and scar-like image patches for multi-scale information fusion.
- Extensive experiments demonstrate that IPMix achieves state-of-the-art corruption robustness and improves numerous safety metrics compared with other data augmentation approaches.
IPMix integrates the three data augmentation techniques in a label-preserving fashion, effectively circumventing potential manifold intrusion and maintaining label consistency[27]. Furthermore, inspired by prior work, IPMix eliminates the need to search for an optimal data augmentation policy, thus reducing computational costs. By addressing these challenges, IPMix has achieved significant improvements, as depicted in Figure 2. In comparison to other methods that focus on leveraging one of these categories for enhancement, IPMix achieves state-of-the-art results in accuracy and robustness.
Since IPMix involves different levels of data augmentation techniques, it naturally motivates us to design a novel mixing method for better information fusion. Previous research has demonstrated that enhancing training data diver
sity [23, 28, 29] and image structural complexity [30, 31] is crucial for improving model robustness. The structural complexity of synthetic data, such as fractals and statistical information, can bolster model performance through pre-training [32] or blending with clean images [24]. For better data integration, IPMix mixes clean images with synthetic pictures at different scales by random mixing to improve structural complexity, which can generate more diverse images to improve robustness.
Building on the enhancement of corruption robustness, we further extend IPMix's capabilities to enhance various safety metrics to fulfill the demands of constructing secure and reliable systems in real-world situations [11]. We demonstrate that IPMix improves numerous safety metrics, including corruption robustness, calibrated uncertainty estimates, adversarial robustness, anomaly detection, and prediction consistency. On CIFAR-10-C and CIFAR-100-C, IPMix achieves the best results across different architectures. On ImageNet, IPMix outperforms previous methods and gains a substantial improvement on various safety measure benchmarks, achieving state-of-the-art or comparable results on ImageNet-R, ImageNet-A, and ImageNet-O [33, 34].
Figure 2: The performance of different levels of data augmentation methods on CIFAR-100. Compared to other approaches which focus on utilizing only one category, IPMix achieves state-of-the-art accuracy and robustness.
[30, 31] is crucial for improving model robustness. As fractals and statistical information, can bolster or blending with clean images [24]. For better data-semantic pictures at different scales by random mixing generate more diverse images to improve robustness.
Figure 3: Top: Sample fractals from IPMix set. Bottom: An example of IPMix applied on a dog image,  $k = 2$ ,  $t = 3$ . We randomly select P (pixel and patch) data augmentation methods and image-level data augmentation methods to generate a highly diverse set of augmented images. We sample  $w_{k}$  ( $k = 2$ , in this case) from Dirichlet distribution and use skip connection ( $m$  sample from a Beta distribution) to maintain semantic consistency.
Data augmentation is crucial to the success of modern neural networks, contributing significantly to the improvement of model generalization performance. The presented data augmentation approaches can be classified into three high-level categories: image-level, pixel-level, and patch-level augmentations.
Image-level data augmentation. Image-level data augmentation methods are commonly label-preserving, applying transformations on the whole image to improve data diversity. AutoAugment [19] utilizes reinforcement learning to automatically search optimal compositions of transformations. Adversarial AutoAugment [35] generates adversarial images to extend data and produces a dynamic policy during training. TrivialAugment [36] randomly selects an operation and the magnitude to reduce search space and improve performance. AugMix [37] uses multiple transformations to create high diversity of augmented images, achieving state-of-the-art results on corruption robustness and calibration. AugMax [29] unifies diversity and hardness to search for the worst-case mixing strategy. PRIME [38] uses max-entropy image transformations to boost model corruption robustness.
Pixel-level data augmentation. Pixel-level data augmentation methods mix images using pixel-wise weighted averages. MixUp [23] generates augmented images by linearly interpolating between two randomly selected images and their corresponding labels. Manifold MixUp [39] encourages neural networks to learn smooth interpolations between data points in the hidden layers, improving accuracy by comparison with MixUp. PixMix [24] utilizes structural complexity synthetic pictures, such as fractals and feature visualizations, to improve model performance. Our work shared similarities with PixMix, but we use multi-scale information and better information fusion methods to train robust models by leveraging more diverse examples.
Patch-level data augmentation. Patch-level data augmentation methods mask or replace parts of the original image with different information. CutOut [28] randomly masks out regions of a clean image to learn less discriminative portions, thereby improving accuracy. CutMix [40] replaces a patch of an original image with another randomly picked image to improve performance. Patch Gaussian [41], which inputs a patch of Gaussian noise into the clean picture, combines the improved accuracy of CutOut with the noise robustness of Gaussian. SaliencyMix [42], based on the maximum intensity pixel local in the saliency map, replaces a square patch of the original image with salient information from another image. TokenMix [43] improves the performance of vision transformers by partitioning the mixing region into multiple separated parts and mixing two images at the token level. AutoMix [44] optimizes both the mixed sample generation task and the mixup classification task in a momentum training pipeline with corresponding sub-networks in a bi-level optimization framework.
2.2 Safety Measures
When deploying network models in real-world scenarios, it is crucial to consider comprehensive security measures beyond standard accuracy. Implementing unsafe machine learning systems in high-stakes environments [45-47] can lead to incalculable losses. With the rise of multimodal large language models (MLLMs) [48-50], safety issues are receiving increasing attention because their superior performance still makes mistakes. For example, GPT-4 [49] may be confidently wrong in its predictions and disturbed by adversarial questions. Previous research has proposed various safety measures, including but not limited to robustness and calibration.
Robustness. Corruption robustness considers how to improve the model resistance to unseen natural perturbations under data distribution shifts. As a variant of the original ImageNet, ImageNet-C [51] consists of 15 diverse commonplace corruptions belonging to different categories with five levels of severity, regarded as a general benchmark for corruption robustness. In addition to natural corruption, Hendrycks et al. [33] demonstrate that models should measure generalization to various abstract visual renditions. The robustness of adversarial attacks focuses on defending against imperceptible perturbations to images [52]. Prior works have proposed that there is a trade-off between the robustness of adversarial perturbations and clean image accuracy [53, 54]. ImageNet-O and ImageNet-A [34], widely regarded as benchmarks for evaluating image classifier performance under shifts in both input data and label distributions, are utilized for anomaly detection.
Calibration. Calibrated prediction confidence, which indicate whether a model's output should be trusted, are valuable for classification models in real-world settings. Bayesian approaches [55] are widely used to deal with uncertainty estimation. Kuleshov et al. [56] utilize recalibration methods to solve the miscalibration of credible intervals. Ovadia et al. [57] provide a benchmark for evaluating the accuracy and uncertainty of models under data distributional shifts.
2.3 Training with Synthetic Data
Previous works have proved that training with synthetic data can improve performance on real datasets. Debidatta et al. [58] discover that combining synthetic annotated datasets with real data can significantly improve the performance of instance detection. Baradad et al. [31] generate synthetic data by utilizing various procedural noise models. In addition, they find that naturalism and diversity are two important properties for synthetic data to achieve comparable results with real datasets. Kataoka et al. [59, 60] propose a suite of datasets generated by formula-driven supervised learning.
3 An Attempt to Integrate Existing Approaches
Some prior studies [24, 40] have suggested that combining different data augmentation techniques with existing methods can improve accuracy on standard datasets. However, these works merely employed simple combinations without considering the compatibility between methods at different levels. Simultaneously, these studies chose the clean accuracy as the sole evaluation metric and have not taken the model's
Table 1: The combination of different levels of data augmentation. M, C and A are abbreviations for MixUp, CutMix, and AugMix, respectively.
Classification Error(↓)	Robustness mCE(↓)	Calibration RMS(↓)
Vanilla	21.3	50	14.6
+M	20.5 (-0.8)	45.9(-4.1)	10.5(-4.1)
+M+C	20.2 (-1.1)	46.1(-3.9)	22.7(+8.1)
+M+C+A	23.4 (+2.1)	50.1(+0.1)	25.6(+11)
safety performance into account. In this section, we select MixUp [23], CutMix [40], and AugMix [37] as representative data augmentation approaches for pixel-level, patch-level, and image-level, respectively, to conduct combination experiments of these approaches on CIFAR-100. Please refer to Appendix F for more details about the combination experiments.
Results on Table 1 demonstrate that simply combining different data augmentation methods may significantly impair model performance. This could be attributed to the excessive perturbation of training data caused by the combination of these methods, making the newly generated samples more challenging to identify and impacting the model's ability to learn useful features, leading to performance degradation. When multiple label-variant methods are combined, manifold intrusion
Figure 4: Different mixing framework of IPMix. P augmentation operation represents pixel-level and patch-level augmentation operations. ① Utilizing P operations and image-level operations in different chains and mixing the results. ② A clean image is randomly carried out by P operations or image-level operations in linear combinations to generate an IPMix image. ③ leveraging the mixed image as a new input.
issues may be more likely to arise. One possible solution for better information integration is to incorporate approaches (e.g., MixUp) into search-based data augmentation techniques [20, 36]. However, searching the space for an optimal DA policy will bring expensive computation. Furthermore, this approach aims at improving clean accuracy and does not consider the overall safety performance.
4 IPMix: A Simple Method for Training Robust Classifiers
In this section, we propose IPMix, which integrates three levels of data augmentation methods into a label-preserving approach, comprehensively improving safety metrics without sacrificing clean accuracy. We first demonstrate how to merge various techniques into a coherent framework and then propose novel approaches to achieve superior information fusion.
4.1 Integrates Different Levels into A Coherent Approach
Pixel-level & Patch-level. As a label-preserving data augmentation approach, IPMix uses the equation below to mix two input images:
$$
\tilde {x} = B \odot x _ {1} + (I - B) \odot x _ {2} \tag {1}
$$
Where  $x_{1}$  is the input image and  $x_{2}$  represents an unlabeled synthetic image (e.g., fractals, spectrum, or auto-generated contours).  $B$  is a mask matrix suitable for both patch-level and pixel-level data augmentation methods, and  $I$  is a binary mask filled with ones, having the same dimensions as  $B$ .  $\odot$  represents the element-wise product. When performing mixing operations at the patch level, we choose a patch of random size and position from  $B$ , with a value of  $\lambda$  (sample from Beta distribution) in this range and a value of 1 in other areas, which ensures that except for the mixing patch, the rest of the generated image comes from  $x_{1}$ . When performing mixing operations at the pixel level, we treat the entire image as a patch, with a value of  $\lambda$ . To make it efficient, we adopt fractals as representatives of synthetic data. However, IPMix is insensitive to mixing sets change, as shown in Table 8.
Fractals are geometric shapes with structural complexities and natural geometries. While previous works [32, 61] merely use iterated function systems (IFS) to create fractal data, we employ the Escape-time Algorithm for generating "orbit trap" complex fractals to enhance dataset complexity and diversity. Please refer to Appendix E for details about generating fractal images.
The above-described method provides two key advantages: (1) We utilize a simple approach to combine operations of two levels, facilitating better information fusion. (2) Our method is label-preserving, ensuring it is not affected by manifold intrusion while eliminating the need for label smoothing [62]. In the following sections, we refer to the method used in Eq. (1) as P-level data augmentation, signifying the employment of both patch-level and pixel-level methods.
Image-level. IPMix leverages various augmentation techniques and compositions to create a new image that does not deviate significantly from the original. Drawing inspiration from previous works [36, 37], we randomly sample operations from PIL (e.g., brightness, sharpness) and randomly sample strengths to enhance the diversity of training data without expensive searching. Notably, these operations are disjoint from ImageNet-C corruptions, ensuring the robustness test's validity.
The IPMix framework. To determine the most effective methods for combining P-level and image-level, we conducted experiments using different mixing structures to generate a diverse set of IPMix images, as illustrated in the Figure 4 and Table 2. While Linear Mix achieves excellent results in clean accuracy and corruption robustness, it performs poorly in cal
Table 2: Results are reported on CIFAR-100 and CIFAR-100-C with ResNeXt-29. The Chain-Mixed achieves the most balanced result on these metrics. Bold is best.
Classification Error(↓)	Robustness mCE(↓)	Calibration RMS(↓)
Chain-Mixed	18.3	28.1	3.8
Linear Mix	18.2	27.4	13.5
Mixed Input	19.8	29.6	3.6
ibrated prediction confidence. Mixed Input performs better in calibration but is inferior in accuracy and corruption robustness compared to Chain-Mixed. Consequently, we chose Chain-Mixed as the default framework for IPMix. Furthermore, the experimental results highlight the potential of establishing a general framework for integrating various data augmentation methods.
4.2 Multi-scale Information Fusion
IPMix can enhance the diversity and the structural complexity of training data to improve model performance. However, we found that simple mixing methods restrict the model's capabilities.
To overcome this issue, we use random mixing and scar-like image patches for achieving more effective information fusion.
Random mixing. In previous data augmentation works, it is typical to either linearly mix two images or extract specific image features, such as saliency [22, 42], which requires additional computations, for image mixing. As IPMix incorporates various levels of operations, its objective is to enhance the mixing of images, ultimately increasing data diversity. To accomplish this objective, IPMix employs four mixing operations: addition, multiplication, random pixel mixing, and random element mixing [63]. Random pixel mixing creates a binary mask of size  $H \times W \times 1$  that operates on each channel sequentially, while random element mixing generates a binary mask of size  $H \times W \times 3$  (RGB) that applies to all channels simultaneously. An example is shown in Figure 5. The experiments in Appendix B.1 show that both operations are beneficial to better information mixing between images and fractals.
Figure 5: Top: Examples of random mixing operations. Bottom: Examples of IPMix-Scar mixing and IPMix-Square mixing.
Scar-like image patches. IPMix-Scar employs a long, thin
rectangular box filled with an image patch to enhance dataset diversity, which has proven effective for anomaly detection [64]. An example of patch mixing is illustrated in Figure 5. First, IPMix randomly selects a point and a scar or square of the previously chosen size from the current image. Next, IPMix crops corresponding portions of the current image and the fractal picture and combine them.
Finally, we obtain IPMix, which employs various levels of data augmentation to create diverse transformations with image structural complexity and data diversity. Figure 3 displays an example of IPMix, where  $k$  denotes the number of augmented chains, and  $t$  represents the maximum number of times an image can be augmented. The algorithm of IPMix is summarized in Appendix D.
In this section, we showcase the significant performance improvements brought by IPMix on clean datasets in multiple settings. We present the evaluation results of IPMix for image classification on three datasets—CIFAR-10, CIFAR-100 [65], and ImageNet [66]—across various models. Besides clean Classification, we assess IPMix on diverse safety tasks, including adversarial attack robustness, corruption robustness, prediction consistency, calibration, and anomaly detection. Please refer to Appendix C for details about the evaluation metrics. Lastly, we evaluate the properties of IPMix in thorough ablation studies and compare our approach with different levels of methods.
Table 3: Clean Error for IPMix on CIFAR-10 and CIFAR-100, lower is better. Top : CIFAR-10. Bottom : CIFAR-100. Mean and standard derivation over three random seeds is shown for each experiment. Bold is best.
Vanilla	MixUp	CutOut	CutMix	AugMix	PixMix	IPMix
WideResNet40-4	4.4(±0.05)	3.8(±0.06)	3.6(±0.05)	4.0(±0.04)	4.3(±0.08)	4.1(±0.08)	4.0(±0.06)
WideResNet28-10	3.8(±0.07)	3.6(±0.08)	3.4(±0.06)	3.4(±0.05)	3.4(±0.07)	3.8(±0.13)	3.3(±0.08)
ResNeXt-29	4.3(±0.04)	3.8(±0.11)	4.2(±0.08)	3.8(±0.02)	4.2(±0.05)	3.8(±0.09)	3.8(±0.07)
ResNet-18	4.4(±0.05)	4.2(±0.04)	4.1(±0.05)	4.0(±0.04)	4.5(±0.03)	4.4(±0.05)	4.2(±0.07)
Mean	4.2	3.9	3.8	3.8	4.1	4.0	3.8
WideResNet40-4	21.3(±0.11)	20.5(±0.13)	19.9(±0.11)	20.3(±0.15)	20.6(±0.15)	20.4(±0.17)	19.4(±0.14)
WideResNet28-10	19.0(±0.13)	18.4(±0.12)	18.8(±0.15)	18.0(±0.11)	19.4(±0.11)	18.3(±0.13)	17.4(±0.25)
ResNeXt-29	20.4(±0.11)	20.3(±0.12)	19.6(±0.13)	19.5(±0.13)	20.4(±0.13)	20.1(±0.11)	18.3(±0.22)
ResNet-18	23.7(±0.09)	21.0(±0.07)	22.0(±0.11)	20.8(±0.12)	23.0(±0.14)	21.6(±0.15)	21.6(±0.23)
Mean	21.1	20.0	20.1	19.7	20.8	20.1	19.2