SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minqi Jiang $^{1,\ast}$ , Chaochuan Hou $^{1,\ast}$ , Ao Zheng $^{1,\ast}$ , Songqiao Han $^{1\dagger}$ , Hailiang Huang $^{1,2\dagger}$ , Qingsong Wen $^{3}$ , Xiyang Hu $^{4\dagger}$ , Yue Zhao $^{4\dagger}$
$^{1}$ AI Lab, Shanghai University of Finance and Economics
$^{2}$ MoE Key Laboratory of Interdisciplinary Research of Computation and Economics
$^{3}$ DAMO Academy, Alibaba Group  $^{4}$ Carnegie Mellon University
jiangmq95@163.com, houchaochuan@foxmail.com, zheng-ao@outlook.com,
{han.songqiao,hlhuang}@shufe.edu.cn,qingsongedu@gmail.com,
{xiyanghu,zhaoy}@cmu.edu
Deep learning (DL) techniques have recently found success in anomaly detection (AD) across various fields such as finance, medical services, and cloud computing. However, most of the current research tends to view deep AD algorithms as a whole, without dissecting the contributions of individual design choices like loss functions and network architectures. This view tends to diminish the value of preliminary steps like data preprocessing, as more attention is given to newly designed loss functions, network architectures, and learning paradigms. In this paper, we aim to bridge this gap by asking two key questions: (i) Which design choices in deep AD methods are crucial for detecting anomalies? (ii) How can we automatically select the optimal design choices for a given AD dataset, instead of relying on generic, pre-existing solutions? To address these questions, we introduce ADGym, a platform specifically crafted for comprehensive evaluation and automatic selection of AD design elements in deep methods. Our extensive experiments reveal that relying solely on existing leading methods is not sufficient. In contrast, models developed using ADGym significantly surpass current state-of-the-art techniques.
Anomaly detection (AD) aims to identify data objects that significantly deviate from the majority of samples, with numerous successful applications in intrusion detection [36, 44], fault detection [22, 85], medical diagnosis [13, 38], fraud detection [2, 8, 9], social media analysis [81, 86], etc. Recently, deep neural networks have become the primary techniques in AD due to their powerful representation learning capacity [57]. Among all, weakly-supervised AD (WSAD) methods [56, 58, 59, 69, 83, 90], which leverage imperfect ground truth anomaly labels (e.g., those that are incomplete, inaccurate, or inexact) in deep neural networks for AD, have gained attention in this new frontier [29]. A comprehensive study by [26] showcases WSAD's superiority over unsupervised AD techniques, especially for real-world conditions where the ground truth labels are never complete and accurate. Thus, our work introduces ADGym, a platform for understanding and designing deep WSAD methods<sup>2</sup>, with the potential to be extended to unsupervised and supervised deep AD methods.
Goal I: Understanding Design Choices of Deep AD. Many WSAD methods attribute their improvements to novel network architectures or loss functions, based on the authors' understanding of anomalies. However, these choices represent only a fraction of the design considerations, as there are many other factors to consider, such as data preprocessing and model training techniques. In Table
*Contribute equally. † Corresponding authors.
2For brevity, AD methods in this work refer to deep WSAD methods.
Table 1: ADGym supports a comprehensive list of design choices for deep AD methods.
Pipeline	Design Dimensions	Design Choices
Data Handling	Data Augmentation	[Oversampling, SMOTE, Mixup, GAN]
Data Preprocessing	[MinMax, Normalization]
Network Construction	Network Architecture	[MLP, AutoEncoder, ResNet, FTTransformer]
Hidden Layers	[[20], [100, 20], [100, 50, 20]]
Activation	[Tanh, ReLU, LeakyReLU]
Dropout	[0.0, 0.1, 0.3]
Initialization	[default, Xavier (normal), Kaiming (normal)]
Network Training	Loss Function	[BCE, Focal, Minus, Inverse, Hinge, Deviation, Ordinal]
Optimizer	[SGD, Adam, RMSprop]
Epochs	[20, 50, 100]
Batch Size	[16, 64, 256]
Learning Rate	[1e-2, 1e-3]
Weight Decay	[1e-2, 1e-4]
Note: Bolded design dimensions are those of greater impact on the AD task (discussed in detail in §4).
1, we list specific design choices for deep AD models and group them as design dimensions. Our experiments reveal that some of these dimensions (in bold) significantly impact the performance.
Many questions remain unanswered in current AD studies, such as the interaction between components within an AD method, the relative importance of each component in performance, and the potential of new data-centric techniques to improve AD model performance. Thus, in the first part of the study (§3.2), we have evaluated various design combinations on large benchmark datasets. Interestingly, the optimal model, comprising different design choices by combination, varies by datasets and notably outperforms existing state-of-the-art (SOTA) AD models. This raises a key question: how can we automatically design AD models for datasets other than using existing models?
Goal II: Constructing AD Algorithms Automatically via ADGym. Indeed, our prior research [88, 53, 79] also shows there is no one-size-fits-all AD model for every dataset—we must select AD models based on the underlying dataset. Our prior work focuses on model selection from a pre-defined list, mainly for non-neural-network AD methods with limited design choices [88, 89]. In today's deep learning era, a pre-defined list is not sufficient, given the large number of design choices in Table 1 and §3.2). There could be infinite deep AD models with different design combinations.
In the second part of this study, we develop a design-choice-level selection approach that utilizes meta-learning, called ADGym (see §3.3). In a nutshell, our approach leverages the supervision signals from historical datasets to guide selecting the best design choices for constructing AD models tailored to a new dataset. We aim to enhance the existing AD model selection process, ensuring the best combination of design choices for a given application or dataset. What sets ADGym apart from prior works is our focus on granular design choice selection tailored for deep AD, which has a much larger space than a pre-defined model list for existing methods only. See Fig. 1 for a comparison.
What Do We Learn from the Experiments (\$4)? ADGym helps us better understand and design deep AD methods, with key observations from our experiments: (1) No single design choice consistently outperforms others across all datasets, justifying the need for an automated approach to collectively select the most effective design choices. (2) Employing a meta-predictor to automate design choice selection yields notable improvements over static SOTA methods. (3) We can make meta-predictors better by using ensemble techniques or increasing the range of design choices.
To sum up, our work makes the following technical contributions:
1. Understanding AD Design Choices via Benchmarking. We present the first benchmark that breaks down and compares diverse deep AD design choices across 29 real-world datasets and 4 groups of synthetic datasets<sup>1</sup>, which leads to a few interesting observations.
2. Design Choice Automation. We introduce ADGym, the first automated framework for selecting design choices for weakly-supervised AD, which significantly outperforms SOTA AD methods.
3. Accessibility and Extensibility. We have made ADGym publicly available<sup>2</sup> so that practitioners can build better-than-SOTA AD methods. It is also easy to include new design choices.
<sup>1</sup>348 datasets generated with different seeds to simulate 4 types of anomalies. See Appx. D.2 for details.
2The code is available at https://github.com/Minqi824/ADGym
2.1 Weakly-supervised Anomaly Detection (WSAD)
Due to the cost and difficulties in data annotation, previous studies [42, 43, 47, 68, 91] mainly focus on developing unsupervised AD methods with different assumptions of data distribution [1], while they are shown to have no statistically significant difference from each other in a recent benchmark [26]. In practice, however, there could exist at least a handful of labeled instances that are either identified by domain experts or the bad cases occurred during the deployment of AD methods. Therefore, recent studies [56, 58, 59, 69, 90] propose weakly-supervised AD methods to effectively leverage the valuable knowledge in labeled data and thus facilitate anomaly identification. Nevertheless, existing WSAD methods are often used as it is, without detailed evaluation of each design choice like network architectures and loss functions. A more granular analysis of specific components of WSAD methods could be helpful for a deeper understanding of the AD methods.
2.2 Benchmarks for Anomaly Detection
Anomaly detection benchmarks mainly perform large comparisons and evaluations on the detection performance of different AD methods under unified and controlled conditions. While previous studies [12, 17, 23, 71, 72] focus on benchmarking classical machine learning AD methods, there has been an increasing trend towards benchmarking deep learning AD methods as well. [67] reviews both classic and deep AD models and points to the connections between classic shallow algorithms and deep AD models. Our previous work [26] performs the largest-scale AD benchmark so far, evaluating 30 shallow and deep AD algorithms across 57 benchmark datasets under different levels of supervision. Besides, some benchmark works focus on AD tasks with different data modalities, including time-series [46, 60, 35], graph [49, 48], CV [78] and NLP [63]. All of these benchmarks focus on discovering which AD algorithms (as a whole) are more effective. Differently, we focus on understanding the effectiveness of design choices in deep AD methods, complementing these works.
2.3 Automatic Model Selection for AD
2.3.1 Unsupervised Anomaly Model Selection
Developing automatic model selection schemes for unsupervised AD algorithms faces a major challenge in lacking evaluation criteria. A recent survey [53] provides a comprehensive survey of using internal model evaluation strategies that solely rely on input features and outlier scores for model selection. Their extensive experiment shows that none of the existing and adapted evaluation strategies would be practically useful for unsupervised AD model selection.
Another existing work stream leverages meta-learning, based on the similarity between the new task and historical datasets where model performance is available. Some notable work [88, 89, 87] assume that model performance can be generalized across similar datasets, thus selecting the model for a new task based on similar historical tasks. Specifically, [89, 87] introduce the idea of building a meta-predictor (which is a regressor) to predict the model performance on a new dataset by training it on historical model performances. In ADGym, we take the idea of meta-learning to train a specialized meta-predictor for deep AD methods, where existing works [88, 89] only focus on non-neural-network-based AD models with a relatively small model pool, where we extend to more complex deep AD design spaces. See §3.3 for details.
2.3.2 Supervised Anomaly Model Selection, HP Optimization, and Neural Arch. Search
Supervised AD model selection and HP optimization (HPO) aim to train a set of models, evaluate their performance using ground truth labels, and ultimately select the best model. However, as previously mentioned, ground truth labels in AD are scarce, which prevents supervised selection from being widely explored. PyODDS [41] optimizes an AD pipeline, while TODS [34] creates a selection system for time series data. Both, however, focus primarily on shallow models, excluding most deep models. Recently, AutoOD [40] conducts neural architecture search (NAS) for deep AD using labels. However, it only focuses on AutoEncoder models over image data. AutoPatch[31] and [73] are another two NAS works for AD, but they still do not consider networks beyond CNNs.
Figure 1: The framework of ADGym. Existing AD model selection focuses on selecting the best model from a small, fixed pool. Our proposed method is more flexible to build a good model by choosing different parts, including data handling, network construction, and network training.
Here, we highlight the difference between ADGym and the above works. First, ADGym aims to select from a large, comprehensive design pool, as opposed to existing model selection, which is often restricted to a small pre-defined model pool; meanwhile, ADGym also brings more granularity than general HPO and NAS, which only focus on a small set of HPs (given neural architectures can also be considered as HPs). Second, ADGym supports more scenarios other than focusing on a single family of AD methods, e.g., autoencoders, or just network architectures. Third, different from the SOTA method AutoOD, ADGym is a zero-shot algorithm that does not require any model building for a new dataset, thereby significantly reducing the online time; see §3.3 for details.
3 ADGym: Benchmarking and Automating Design Choices in Deep AD
3.1 Problem Definition
In this paper, we focus on the common weakly supervised AD scenario, where given a training dataset  $\mathcal{D} = \{\pmb{x}_1^u,\dots,\pmb{x}_k^u,\left(\pmb{x}_{k + 1}^a,y_{k + 1}^a\right),\dots,\left(\pmb{x}_{k + m}^a,y_{k + m}^a\right)\}$  contains both unlabeled samples  $\mathcal{D}_u = \{\pmb{x}_i^u\}_{i = 1}^k$  and a handful of labeled anomalies  $\mathcal{D}_a = \{\left(\pmb{x}_j^a,y_j^a\right)\}_{j = 1}^m$ , where  $\pmb {x}\in \mathbb{R}^{d}$  represents the input feature and  $y_{j}^{a}$  is the label of identified anomalies. Usually, we have  $m\ll k$ , since only limited prior knowledge of anomalies is available. Such data assumption is more practical for AD problems, and has been studied in recent deep AD methods [56, 58, 59, 69, 90]. The goal of a weakly-supervised AD model  $M$  is to assign higher anomaly scores to anomalies.
3.2 Goal I: Understanding Design Choices of Deep AD
The first primary goal of this study is to investigate the large design space of deep AD solutions, which should cover as many design choices as possible, e.g., different data augmentation methods, network architectures, etc. Following the taxonomy of the previous study [80] and practical experiences in industrial applications [2, 22, 78], we decouple the standard process of deep AD methods, starting from the input data and ending with model training. The Pipeline includes: data handling  $\rightarrow$  network construction  $\rightarrow$  network training, as shown in Table 1. For each step of the pipeline, we further categorize it as different Design Dimensions, where diverse Design Choices can be specified to instantiate an AD method. More details are shown in Appx. C.1
With the comprehensive design space above, we not only evaluate the possible designs of weakly-supervised AD methods, but also investigate several interesting design dimensions often overlooked in previous AD research. For example, previous AD studies often perform model training on the raw input data (instead of augmented data for mitigating class imbalance problems), and simple network architectures like multi-layer perceptron (MLP) (instead of more recent network architectures like ResNet [27] and Transformer [75]), where the other cutting-edge techniques have already been widely used in other domains like NLP and CV. To sum up, we pair the combination of comprehensive design choices in Table 1 with benchmark AD datasets to unlock new insights. See results in §4.2.
3.3 Goal II: Constructing AD Algorithms Automatically via ADGym
From Model Selection to Pipeline Selection. With the large AD design space illustrated in the last section, we investigate how to construct effective AD methods given the downstream applications
automatically. Given a pre-defined model set  $\mathcal{M} = \{M_1, \dots, M_m\}$  that includes  $m$  combinations of applicable design choices, model selection picks a model  $M \in \mathcal{M}$  to train on a dataset  $\mathcal{D}_{test}$  and output the anomaly score  $O := M(\mathbf{X}_{test})$  to achieve the highest detection performance. In our case, we construct a pipeline  $P$  from our design space to generate each model  $M$ . As shown in Fig. 1, our design (right) brings more flexibility by choosing from details than existing model selection works that only choose from a fixed, small set of AD models.
Meta-learning for Pipeline Selection. Meta learning has been recently used in unsupervised AD model selection [88, 89], where the core idea is to transfer knowledge from model performance information on historical datasets to the selection on a new dataset. Intuitively, if two datasets are similar, their best models should also resemble. Under the meta-learning framework, we assume there are  $n$  historical AD datasets (i.e., detection tasks)  $\mathcal{D}_{\mathrm{train}} = \{\mathcal{D}_1,\dots ,\mathcal{D}_n\}$ ; each meta-train dataset is accompanied with ground truth labels for performance evaluation. To leverage prior knowledge for pipeline selection on a new dataset, we first conduct experiments on  $n$  historical datasets to evaluate and collect the performance of  $m$  possible designed pipelines (as the combination of all applicable design choices), by two widely used metrics: AUCROC (Area Under Receiver Operating Characteristic Curve) and AUCPR (Area Under Precision-Recall Curve).
For each metric, we acquire the corresponding performance matrix  $\pmb{P} \in \mathbb{R}^{n \times m}$ , where  $P_{i,j}$  corresponds to the  $j$ -th constructed AD model's performance on the  $i$ -th historical dataset. Meanwhile, historical datasets vary in task difficulty, resulting in variations in the numerical range of the constructed AD models' performance. Therefore, we convert the value of the performance into their relative/normalized ranking, where  $P_{i,j} = \text{rank}(P_{i,j}) / m \in [0,1]$ . Smaller ranking values indicate better performance on the corresponding dataset.
To predict the performance of a given pipeline on a new dataset, we propose to train a meta-predictor as a regression problem: the input of meta-predictor is  $\{\mathbf{E}_i^{meta},\mathbf{E}_j^{comp}\}$ , corresponding to the meta-feature [88] (i.e., the unified representations of a dataset) of  $i$ -th dataset and the embedding of  $j$ -th AD component (i.e., the representation of a pipeline/components). We defer the specific details into Appx. C.2. Given the meta-predictor  $f(\cdot)$ , we train it to map dataset and pipeline characteristics to their corresponding performance ranking across all historical datasets, as shown in Eq. (1). Refer to our open-sourced code for details of embeddings and the choice of regression models.
$$
f: \underbrace {\mathbf {E} _ {i} ^ {m e t a}} _ {\text {m e t a f e a t u r e s}}, \underbrace {\mathbf {E} _ {j} ^ {c o m p}} _ {\text {c o m p o n e n t e m b e d .}} \mapsto P _ {i, j}, i \in \{1, \dots , n \}, j \in \{1, \dots , m \} \tag {1}
$$
For a newcoming dataset (i.e., test dataset  $\mathbf{X}_{\mathrm{test}}$ ), we acquire the predicted relative ranking of different AD components using the trained  $f(\cdot)$ , and select top-1 ( $k$ ) to construct AD model(s). Note this procedure is zero-shot without needing any neural network training on  $\mathbf{X}_{\mathrm{test}}$  but only extracting meta-features and pipeline embeddings. We show the effectiveness of the meta-predictor in §4.3.
4.1 Experiment Settings
AD Datasets and Baselines. In ADGym, we gather datasets from the previous AD studies and benchmarks [12, 19, 26, 57, 64] for evaluating existing AD models and different AD designs. Datasets with sample sizes smaller than 1000, as well as those with problematic model results, are removed, resulting in a total of 29 remaining datasets. These datasets cover many diverse application domains, such as healthcare, audio and language processing, image processing, finance, etc. Furthermore, based on our previous work[26], we categorize anomalies into four types, local, global, cluster and dependency anomalies, and generate synthetic datasets for each individual anomaly type. For each dataset,  $70\%$  data is split as the training set and the remaining  $30\%$  is used as the test set. We use stratified sampling to keep the anomaly ratio consistent. For each experiment, we repeat 3 times and report the average. Further details of datasets and baselines are presented in Appx. A and B.
Meta-predictor in ADGym. We introduce two types of meta-predictors in ADGym, namely DL-based and ML-based. The DL-based meta-predictor is instantiated as a two-layer MLP and trained for 100 epochs with early stopping. The training process utilizes the Adam optimizer [32] with a
<sup>1</sup>We analyze the results across domains and find no single set of best design choices. See Appx.D.3.
learning rate of 0.001 and batch size of 512. For the ML-based meta-predictor, We instantiate the meta-predictor with XGBoost and CatBoost and their default hyperparameter settings. Considering the expensive computational cost of training meta-predictors on the entire design space, we randomly sample 1,000 design choices for each experiment illustrated below. See details in Appx. C.2.
Evaluation Metrics. We perform evaluations with two widely used metrics: AUCROC (Area Under Receiver Operating Characteristic Curve), AUCPR (Area Under Precision-Recall Curve) value, and the relative rankings corresponding to these two metrics<sup>1</sup>.
4.2 Large Evaluation on AD Design Choices
In this work, we perform large evaluations on the decoupled pipelines according to the standard procedure of AD methods. Such analysis is often overlooked in previous AD studies, and we investigate each design dimension of decoupled pipelines by fixing its corresponding design choice (e.g., Focal loss), and randomly sampling other dimensional design choices to construct AD models, e.g.,  $\{DateAugmentation: Mixup, Network Architecture: FTT, LossFunction: Focal, \ldots\}$ , ...,  $\{DateAugmentation: GAN, Network Architecture: MLP, LossFunction: Focal, \ldots\}$ . In other words, we investigate each design dimension by controlling other variables. Different design choices are compared w.r.t.  $n_a = 5, 10, 20$  and demonstrated with box plots, where the number of comparisons in each box is ensured to be the same. In the following subsections, we analyze the benchmark results on each of the design dimensions, namely data handling, network construction, and network training.
4.2.1 Data Handling
For the design dimension of data augmentation methods (e.g., SMOTE $^2$  and Mixup methods) shown in Figure 2, we find that almost no method has brought significant performance improvements. This could be explained by the fact that, unlike time series [76], NLP [70], or CV tasks [20], data augmentation is rarely incorporated into the design of existing AD methods tailored for tabular data [57]. Besides, our results indicate that GAN-based augmentation method is even worse than simpler methods like oversampling, probably due to the difficulty of modeling and generating realistic anomalies for the tabular data. The same trend is observed in the synthetic datasets with individual anomalies. In the vast majority of cases, the results from data augmentation are inferior to those from the original data. (See Appx.D.2.) For data preprocessing methods, We do not observe a significant difference between the minmax scaling and normalizing methods.
(a) Data augmentation
Figure 2: AUCROC performance of data handling designs. There is no significant difference between different design choices in both data augmentation and preprocessing dimensions.
(b) Data preprocessing
4.2.2 Network Construction