SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Mean Corruption Error (mCE) for IPMix across architectures on CIFAR-10-C and CIFAR-100-C, lower is better. Top : CIFAR-10-C. Bottom : CIFAR-100-C. Bold is best.
Vanilla	MixUp	CutOut	CutMix	AugMix	PixMix	IPMix
WideResNet40-4	26.4(±0.14)	21(±0.15)	25.9(±0.13)	26(±0.13)	10(±0.12)	9.5(±0.14)	8.6(±0.14)
WideResNet28-10	24.2(±0.15)	19.2(±0.17)	23.5(±0.17)	25.1(±0.13)	9.1(±0.14)	8.7(±0.14)	7.5(±0.17)
ResNeXt-29	27.5(±0.11)	23.6(±0.18)	27.3(±0.18)	28.5(±0.18)	11.3(±0.15)	9.2(±0.12)	8.6(±0.19)
ResNet-18	25(±0.09)	20(±0.15)	24.1(±0.13)	24.7(±0.19)	10.4(±0.13)	9(±0.11)	8.4(±0.17)
Mean	25.8	20.9	25.2	26	10	9.1	8.2
WideResNet40-4	50(±0.15)	45.9(±0.19)	51.5(±0.17)	50(±0.19)	33.3(±0.22)	31.1(±0.19)	28.6(±0.15)
WideResNet28-10	48.5(±0.21)	44.2(±0.18)	48.2(±0.15)	48.6(±0.21)	31.5(±0.21)	28.3(±0.21)	26.6(±0.29)
ResNeXt-29	51.4(±0.19)	47.9(±0.21)	51(±0.17)	52.4(±0.22)	34.1(±0.24)	30.6(±0.23)	28.1(±0.31)
ResNet-18	50(±0.18)	45.5(±0.21)	50.2(±0.19)	50.8(±0.24)	35(±0.25)	31.4(±0.28)	29.9(±0.29)
Mean	50	45.9	50.2	50.5	33.4	30.3	28.3
We evaluate IPMix on CIFAR-10-C, CIFAR-100-C, and ImageNet-C to measure its resistance to corruption data shifts. We test IPMix on CIFAR-10-P, CIFAR-100-P, and ImageNet-P to measure network prediction stability against minor perturbations. To thoroughly demonstrate our method's capabilities, we assess it on supplementary datasets, including ImageNet-R, ImageNet-O, and ImageNet-A. Experiments on these datasets validate our approach's robustness under real-world distribution shifts.
5.1 Evaluation on CIFAR
We experiment with different backbone architectures on CIFAR-10 and CIFAR-100, including 40-4 Wide ResNet [67], 28-10 Wide ResNet, ResNeXt-29 [68], and Resnet-18 [69]. We compare IPMix with various data augmentation methods, including CutOut, MixUp, CutMix, AugMix, and PixMix. Please refer to Appendix A for more details about the training configurations.
Accuracy. In Table 3, we demonstrate that IPMix improves standard accuracy across architectures. In comparison with other approaches, IPMix achieves the best or comparable accuracy, showing the improvement of safety measures is not at the cost of hurting clean accuracy.
Corruption robustness. Results show that IPMix substantially improves corruption robustness across architectures. Compared to AugMix on CIFAR-100-C, IPMix achieves  $4.7\%$  (40-4) and  $4.9\%$  (28-10) improvement on WideResNet,  $6\%$  on ResNeXt, and  $5.1\%$  on ResNet. Table 4 demonstrates that IPMix achieves state-of-the-art results on both CIFAR-10-C and CIFAR-100-C.
Calibration. We utilize RMS calibration error [70] to evaluate the empirical frequency of correctness. As depicted in Figure 6, IPMix surpasses other methods, achieving state-of-the-art results.
Prediction consistency. We leverage the mean flip rate (mFR) to evaluate prediction consistency on CIFAR-10-P and CIFAR-100-P [51]. IPMix achieves the lowest mFR, as shown in Figure 7.
Figure 6: The results of calibration on CIFAR-100. IPMix achieves the lowest RMS error in all data augmentation methods, improving  $11.8\%$  by comparing with Vanilla.
Prediction Consistency
Adversarial Robustness
Figure 7: Left: prediction consistency. Right: adversarial robustness. IPMix achieves the best results on both metrics, demonstrating its ability to improve overall security performance.
Adversarial robustness. This measure evaluates the resistance of adversarially perturbed by projected gradient descent. We utilize PGD [71] to verify the adversarial robustness of image classifiers. The results in Figure 7 show that IPMix achieves the lowest error.
Table 5: The results of IPMix on ImageNet. For Anomaly Detection, we test the accuracy on ImageNet-A and AUPR on ImageNet-O, higher is better. IPMix achieves round improvement over various data augmentation methods. Bold is best, and underline is second.
Classification	Robustness	Consistency	Calibration	Anomaly Detection
Clean Error(↓)	ImageNet-C mCE(↓)	ImageNet-R Error(↓)	ImageNet-P mFR(↓)	C RMS(↓)	R RMS(↓)	A RMS(↓)	ImageNet-A Classification(↑)	ImageNet-O AUPR(↑)
Vanilla	23.9	78.6	64	57.7	12	19.9	47	2.2	16.2
MixUp[23]	22.7	76.5	62.4	54.6	9.3	41.7	49.3	5.2	16.1
CutOut[28]	22.6	73.1	64.6	57.9	11.3	19.7	46.3	4.7	15.9
CutMix[40]	22.9	77.2	66.5	58.1	9.6	44.2	48	7.2	16.5
AugMix[37]	22.6	68.5	61.8	52.3	8.1	13.1	43.5	3.8	17.4
AugMax[29]	22.9	67.4	62.1	54.6	8.8	12.1	44.7	3.9	17.1
PixMix[24]	22.4	65.4	59.8	50.8	7.2	12.3	44	5.9	17.3
IPMix	22.2	63	57.4	48.5	7.1	7	30	6.6	18.2
5.2 Evaluation on ImageNet
For ImageNet experiments, we compare different data augmentation methods, including MixUp, CutOut, CutMix, AugMix, AugMax [29], and PixMix. We utilize SGD optimizer with an initial learning rate of 0.01 to train ResNet-50 for 180 epochs following a cosine decay schedule. Please refer to Appendix A for more details about the training configurations.
IPMix achieves state-of-the-art or comparable performances on a broad range of safety measures, as shown in Table 5. Compared with other methods, IPMix improves the resistance of out-of-distribution shifts without reducing clean accuracy. On corruption robustness, IPMix outperforms Vanilla by  $15.6\%$  and AugMix by  $5.5\%$ , achieving state-of-the-art results. On ImageNet-R, IPMix demonstrates the ability to improve rendition robustness, increasing by  $6.6\%$  by comparison with Vanilla. On ImageNet-P, IPMix improves mFR by  $9.2\%$  over Vanilla and  $2.3\%$  over PixMix. On calibration tests, IPMix surpasses all methods on ImageNet-C, ImageNet-R, and ImageNet-A, improving RMS by  $0.1\%$ ,  $5.1\%$ , and  $13.5\%$  by comparison with the second-best approach. Furthermore, IPMix achieves convincing results on ImageNet-A and ImageNet-O, demonstrating its exceptional ability in anomaly detection. The results demonstrate that IPMix can roundly improve safety metrics.
Table 6: Ablation results of different components of IPMix on ImageNet with ResNet-50.
Classification	Robustness	Consistency	Calibration	Anomaly Detection
Clean Error(↓)	ImageNet-C mCE(↓)	ImageNet-R Error(↓)	ImageNet-P mFR(↓)	C RMS(↓)	R RMS(↓)	A RMS(↓)	ImageNet-A Classification(↑)	ImageNet-O AUPR(↑)
IPMix	22.2	63	57.4	48.5	7.1	7	30	6.6	18.2
w/o patch	22.8(±0.11)	65.1(±0.16)	58.8(±0.11)	49.1(±0.15)	7.8(±0.11)	7.4(±0.08)	31.1(±0.01)	6(±0.01)	17.7(±0.02)
w/o pixel	23.1(±0.15)	65.6(±0.19)	59.3(±0.13)	49.5(±0.17)	8.2(±0.13)	7.4(±0.09)	32.4(±0.11)	5.6(±0.01)	17.2(±0.03)
w/o image	23.5(±0.16)	66.2(±0.21)	59.5(±0.17)	49.6(±0.14)	8.8(±0.13)	8.1(±0.13)	33.5(±0.13)	6.5(±0.02)	17.8(±0.03)
In this paragraph, we evaluate the properties of our approach by ablation experiments. We first study the influence of different parts of IPMix on performance and then assess the stability of IPMix under various mixing sources. Please refer to more ablation experiments in Appendix B.1.
Components of IPMix. In this section, we evaluate the influence of different IPMix components on perfor
Table 7: Ablation results of different components of IPMix on CIFAR-100. Mean and standard derivation over three random seeds is shown for each experiment. Bold is the best.
Classification	Robustness	Calibration
IPMix	19.4	28.6	2.8
w/o patch	19.7(±0.13)	30 (±0.21)	4.6 (±0.07)
w/o pixel	19.6 (±0.09)	33 (±0.35)	8.2 (±0.12)
w/o image	20.1 (±0.27)	34 (±0.65)	8.6 (±0.21)
mance. We execute ablation experiments on the three primary IPMix constituents: image-level, patch-level, and pixel-level. The results show the indispensable contribution of each component to enhancing model performance, demonstrating that these approaches are complementary and that a unification among them is necessary to achieve robustness. The ablation experiment results are shown in Table 6 and Table 7. Please refer to thorough analysis in Appendix J.
Mixing sources. The excellent performance of IPMix is partly due to the structural complexity of fractal pictures. In this part, we examine the sensitivity of IPMix to different fractal sources on CIFAR-100. We report clean accuracy, corruption robustness, and calibration from different sources with WRN40-4. Fractal + FVis is the default setting of PixMix, which consists of fractals and feature visualization. FractalDB [59] consists of fractal images generated by Iterated Function System (IFS).
Table 8: Ablation results on IPMix across different mixing sets. The results show that IPMix is insensitive to mixing sets change.
Mixing sets	Classification Error(↓)	Robustness mCE(↓)	Calibration RMS(↓)
Fractal + FVis	19.4	28.8	3.3
FractalDB	20	29	5.4
RCDB	19.5	28.4	3.2
Dead Leaves	19.4	29.1	3.1
Spectrum	19.8	29.2	4
fractals(ours)	19.4	28.6	2.8
RCDB [60] consists of auto-generated contours. Dead Leaves and Spectrum generated from generative image models [31]. The full results show in Table 8.
5.4 The Comparison with Different Levels of Method
In this section, we perform an extensive performance comparison between IPMix and a range of existing methods using multiple metrics. We consider AutoAugment, RandAugment, and TrivialAugment [36] as representative image-level techniques, while SaliencyMix, PuzzleMix [72], and Co-Mixup [22] serve as typical patch-level techniques. For pixel-level methods, Manifold Mixup stands as our representative choice. IPMix does not require searching for the optimal DA policy like image-level techniques. In contrast to patch-level approaches, IPMix eliminates the need for saliency computations. The results in Table 9 show that IPMix outperformed all other methods on all metrics.
6 Analysis of IPMix
IPMix combines three levels of data augmentation into a unified, label-preserving technique to improve model performance. We believe that IPMix's superior performance is due to the increased
AugMix
PixMix
CutMix
IPMix
PixMix
Figure 8: Top: t-SNE visualization. The features are from the penultimate layer of a WRN40-4 trained on CIFAR10. Compared with other approaches, IPMix has distinct boundaries between different category clusters and generates diverse samples to cover boundary areas, thereby improving the generalization ability. Bottom: The Grad-CAM visualization, with input images sourced from ImageNet-A, demonstrates that IPMix excels in identifying objects within complex scenarios.
Table 9: Results of different augmentation methods on CIFAR-100 and CIFAR-100-C with 28-10 Wide ResNet. Bold is best.
Methods	Classification Error(↓)	Robustness mCE(↓)	Adversaries Error(↓)	Consistency mFR(↓)	Calibration RMS(↓)
AutoAugment [19]	17.7(±0.11)	38.4(±0.15)	97.8(±0.22)	8(±0.06)	7.9(±0.06)
RandAugment [20]	17.8(±0.14)	41.5(±0.13)	96.6(±0.25)	8.6(±0.10)	7.9(±0.04)
TrivialAugment [36]	17.9(±0.13)	96.3(±0.21)	35.4(±0.23)	7.3(±0.07)	8.7(±0.04)
SaliencyMix [42]	18.3(±0.14)	38.3(±0.24)	96.7(±0.21)	10.8(±0.07)	7.1(±0.07)
PuzzleMix [72]	18.1(±0.11)	37.9(±0.21)	96.1(±0.23)	10.5(±0.04)	7.5(±0.08)
Co-Mixup [22]	18.0(±0.19)	35.6(±0.25)	95.6(±0.21)	10.1(±0.05)	7.7(±0.04)
Manifold Mixup [39]	18.8(±0.21)	51.3(±0.23)	93.4(±0.17)	29.9(±0.28)	10.2(±0.09)
IPMix	17.4(±0.25)	26.6(±0.29)	91.3(±0.21)	4.2(±0.11)	6.4(±0.07)
data diversity and enhanced regularization effect. For a more intuitive demonstration of these effects, we utilize t-SNE and Class Activation Mapping (CAM) [73] for visualizations, as shown in Figure 8.
Increasing diversity. IPMix increases the diversity of training data by mixing data at multiple levels, enabling the model to learn a greater variety of feature combinations and patterns. Furthermore, the integration of synthetic data from distinct distributions (e.g., fractals), further amplifies this diversity.
Enhanced regularization effect. The approach of mixing data also serves as a potent regularization technique. By randomly mixing samples, the model is compelled to learn more robust features rather than overly relying on specific sample or class characteristics, which reduces the risk of overfitting and enhances the model's performance in different environments.
We propose IPMix, which leverages different levels of augmentation techniques and image structural complexity to improve model performance. By employing random mixing methods, we facilitate more effective information fusion. The experimental results indicate that IPMix can significantly improve various safety metrics. We hope our work will attract attention to joining different methods into coherent and synergetic approaches to improve robustness and other safety measures. This adaptation is crucial given the growing importance of safety requirements in systems design.
This work was under the help of Xi Yang. We thank him for his selfless help and valuable suggestions.
[60] Hirokatsu Kataoka, Ryo Hayamizu, Ryosuke Yamada, Kodai Nakashima, Sora Takashima, Xinyu Zhang, Edgar Josafat Martinez-Noriega, Nakamasa Inoue, and Rio Yokota. Replacing labeled real-image datasets with auto-generated contours. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21200-21209, 2022.
[61] Connor Anderson and Ryan Farrell. Improving fractal pre-training. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2412-2421, 2021.
[62] Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Neural Information Processing Systems, 2019.
[63] Cecilia Summers and Michael J. Dinneen. Improved mixed-example data augmentation. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1262-1270, 2018.
[64] Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9659-9669, 2021.
[65] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[66] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
[67] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv, abs/1605.07146, 2016.
[68] Saining Xie, Ross B. Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987-5995, 2016.
[69] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015.
[70] Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. ArXiv, abs/1812.04606, 2018.
[71] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ArXiv, abs/1706.06083, 2017.
[72] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. ArXiv, abs/2009.06962, 2020.
[73] Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128:336-359, 2016.