SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table D13: Relative Rank AUCROC of auto-selected pipelines by ADGym.
na	SOTA	RS	SS	GT	DL-single	DL-ensemble	ML-single	ML-ensemble
2-stage	end2end	2-stage	end2end	XGB	CatB	XGB	CatB
5	0.289	0.275	0.231	0.986	0.598	0.551	0.609	0.573	0.582	0.562	0.605	0.633
10	0.243	0.216	0.202	0.982	0.594	0.530	0.635	0.588	0.610	0.588	0.659	0.645
20	0.225	0.215	0.244	0.978	0.601	0.530	0.621	0.581	0.586	0.587	0.640	0.685
Table D14: Relative Rank AUCPR of auto-selected pipelines by ADGym.
na	SOTA	RS	SS	GT	DL-single	DL-ensemble	ML-single	ML-ensemble
2-stage	end2end	2-stage	end2end	XGB	CatB	XGB	CatB
5	0.353	0.284	0.276	0.984	0.572	0.545	0.600	0.552	0.557	0.566	0.608	0.596
10	0.275	0.297	0.270	0.982	0.563	0.527	0.599	0.569	0.547	0.569	0.628	0.668
20	0.260	0.319	0.313	0.973	0.549	0.514	0.614	0.539	0.536	0.600	0.617	0.659
D.5 Additional Results of Evaluations under Real-world Issues
In real-world applications, AD tasks frequently encounter complex scenarios characterized by noisy and corrupted input data. In this section, we assess the performance and robustness of various AD design choices under three prevalent but imperfect real-world conditions. All experiments are conducted within a weakly-supervised framework, wherein only a sparse set of ten anomaly samples are labeled.
- Duplicated Anomalies. Abnormal data are likely to recur multiple times due to various factors. These repeated anomalies often impact many density-based AD algorithms. To simulate this scenario, we duplicated each abnormal samples three times in all datasets for our experiments.
- Irrelevant Features. In real-world tabular data, there often exist irrelevant features (columns). Even when feature selection techniques such as Random Forests or Logistic Regression are employed, their efficacy in AD applications is not guaranteed. Therefore, assessing the robustness of AD design choices in the presence of such irrelevant features is crucial. To examine this, we add an additional  $30\%$  irrelevant features to all datasets by generating uniform noise.
- Annotation Errors. Annotation errors are among the most common forms of noise encountered in real-world scenarios, especially in AD tasks where the data is extremely imbalanced. To assess the robustness of the design choices against label noise, we inverted  $10\%$  of the labels in each dataset, limiting this modification to the training sets.
We present the experimental results in Figure D16 and Figure D17 employing both AUCROC and AUCPR metrics. It is evident from the figures that the performance of various design choices is significantly impacted by Irrelevant Features and Annotation Errors, manifesting as a noticeable decline in both metrics. The conclusion variances among different design choices are small under the influence of Irrelevant Features, trending towards convergence. Surprisingly, we find that when Duplicated Anomalies are present, the performance of all design choices tends to improve, and the variance also increases, possibly mitigating the issue of data imbalance. What is particularly encouraging in our experiments is that the ResNet architecture not only shows a substantial and stable advantage in scenarios with Duplicated Anomalies but also exhibits less impact in the other two settings. We also conduct experiments with an Irrelevant Features noisy ratio of 0.1 and an Annotation Errors ratio of 0.05, where only a slight mitigation in the impact on design choices, and the conclusions remained consistent with our current findings.
(b) Network architecture
(d) Loss function
Figure D16: AUCROC performance of design choices under real-world issues, excluding GAN-based methods due to high computational time.
(c) Activation function
(e) Optimizer
(b) Network architecture
(d) Loss function
Figure D17: AUCPR performance of design choices under real-world issues.