SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We investigate various design dimensions of network construction, as is shown in Figure 3. For network architectures, MLP is still a competitive baseline in AD tasks, and even outperforms other more complex architecture designs like ResNet and FTTransformer w.r.t.  $n_a = 5$ , i.e., only a handful of labeled anomalies are available in the training stage. Moreover, we suggest that the ResNet model could also serve as both an effective and stable network architecture, especially when more labeled anomalies can be acquired (e.g.,  $n_a = 10$  and  $n_a = 20$ ). However, we do not find significant advantages of the FTTransformer, where its AUCROC performance is generally worse than other network architectures. This may be due to the reason that compared to the supervised tasks for tabular
<sup>1</sup>Due to page limit, we report the AUCPR and the relative ranking results in the Appx. D.1.2. We find similar conclusions can be drawn between AUCROC and AUCPR metrics.
For  $n_a = 5$ , SMOTE raises an error since the number of labeled anomalies is smaller than the neighbors.
data, very limited labeled data (corresponding to the noises in the unlabeled data) can be leveraged to guide the training process of such a complicated model. Furthermore, compared to our earlier work on ADBench[26] where the FTTransformer shows competitive performance, it can be reasonably inferred that the FTTransformer has limited robustness to hyperparameter tuning, especially after we dissected its parameters. We also identify the advantage of the AutoEncoder architecture on synthetic datasets with individual anomaly types, particularly in the local and global anomaly datasets (see Appx. D). A plausible explanation is that the normal data in these synthesized datasets follows a specific distribution that can be more effectively reconstructed.
For the activation function, Tanh and LeakyReLU appear to be more effective than the ReLU function in tabular AD tasks, where this conclusion holds true for different  $n_a$ . Besides, we do not observe significant differences in the design dimensions of hidden layers, dropout, and network initialization.
(a) Network architecture
(b) Hidden layers
(d) Dropout
Figure 3: AUCROC performance of network construction designs. MLP is still an effective architecture for AD tasks, where Tanh and LeakyReLU activation functions are generally better than ReLU.
(c) Activation function
(e) Network initialization
4.2.3 Network Training
In this subsection, we evaluate different design dimensions in the network training step, as shown in Fig. 4. Based on the results, we surprisingly observe that the classical BCE loss could be a competitive baseline when batch resampling method is applied in the training process. Moreover, our results show the advantages of hinge loss [56, 59], which achieves better AUCROC performance w.r.t. different  $n_a$ , and deviation loss [59] in all synthetic datasets with a significant similarity across the anomalies. (details in Appx. D). For model optimization, we observe that Adam and RMSprop optimizers are better than the classical SGD, where large training epochs (e.g., epochs=100) lead to overfitting on limited labeled data, resulting in a relatively inferior performance. For the design dimensions of batch size, learning rate, and weight decay, they do not show obvious differences in terms of the AUCROC metric.
4.3 Automatic Component Construction via ADGym
In this subsection, we verify the effectiveness of the proposed meta-predictors by answering the questions illustrated below. Beyond the fixed SOTA AD methods, we provide three model construction plans for a thorough baseline comparison. The random selection plan (RS) randomly generates a pipeline from all design choices. The supervised selection plan (SS) selects the pipeline according
<sup>1</sup>Batch resampling method [59] randomly samples a training batch with half of the samples from the unlabeled data and another half from the labeled anomalies. We applied this method for each loss function in order to mitigate the impact of the class imbalance problem, except for the inverse loss, as it significantly affects the gradient update of the neural network. More descriptions of these methods can be found in the Appx. C.
(a) Loss function
Figure 4: AUCROC performance of network training designs. Hinge loss is generally a better AD loss function. Both Adam and RMSprop optimizers outperform classical SGD. Smaller epochs could prevent overfitting on limited labeled data and thus achieve better performance.
(c) Epochs
(e) Learning rate
(b) Optimizer
(d) Batch size
(f) Weight decay
to the evaluation of validation sets using labels. The ground truth plan (GT) is the best-performing pipeline in a sample of 1000. We iteratively leave one dataset as the testing dataset, and leverage all the remaining datasets to train the meta-predictor(s). The central experimental results of ADGym's performance are provided in Table 3. We report the average AUCROC performance across real-world datasets and demonstrate the complete experimental results in the Appx. D.
In the subsequent paragraphs, we present a comparison between the AD models constructed by the meta-predictor and the SOTA AD methods. Also, we discuss the impact of meta-predictors' design on their performance, including meta-features, the loss function of the predictors, and more.
1. Is the model constructed by meta-predictor better than existing SOTA methods?
Table 2: Performance of the baseline SOTA AD methods.
na	GANomaly	REPEN	DeepSAD	DevNet	FEAWAD	ResNet	FTTransformer
5	0.605	0.784	0.718	0.761	0.745	0.617	0.782
10	0.616	0.787	0.750	0.792	0.789	0.673	0.816
20	0.615	0.801	0.780	0.828	0.829	0.735	0.843
Table 3: Performance of auto-selected pipelines by ADGym. RS, SS, and GT refer to the random selection, supervised selection, and the best performance among all sampled 1000 pipelines, respectively. DL or ML corresponds to the meta-predictor that is either instantiated with MLP or tree-based methods like XGBoost. The suffix -ensemble refers to ensemble multiple predictions of meta-predictors. "2-stage" and "end2end" correspond to the two-stage and end-to-end for extracting meta-features.
na	RS	SS	GT	DL-single	DL-ensemble	ML-single	ML-ensemble
2-stage	end2end	2-stage	end2end	XGB	CatB	XGB	CatB
5	0.735	0.613	0.902	0.800	0.811	0.813	0.813	0.802	0.804	0.815	0.821
10	0.757	0.696	0.912	0.836	0.833	0.843	0.837	0.836	0.839	0.849	0.847
20	0.777	0.747	0.921	0.859	0.850	0.865	0.857	0.857	0.861	0.869	0.872
Our results show that the automatic construction of AD models is indeed capable of surpassing those SOTA methods, where they achieve better AUCROC performance in all settings, and this conclusion holds true under various numbers of labeled anomalies  $n_a$ . Specifically, the average/best AUCROC performance of meta-predictors shows a relative improvement over the best SOTA models REPEN 2.6%/3.7% w.r.t.  $n_a = 5$ , FTTransformer 2.4%/3.3% w.r.t.  $n_a = 10$  and 1.8%/2.9% w.r.t.  $n_a = 20$ .
Moreover, the average AUCROC performance of meta-predictors is  $9.4\%$ ,  $9.4\%$ , and  $8.5\%$  higher than that of SOTA models w.r.t.  $n_a = 5$ , 10, and 20, respectively.
Moreover, the clear advantage of GT to the current SOTA methods indicates that existing AD solutions could be further improved through exploring the design space or automatic selection techniques, which also validates the value of ADGym. Besides, compared to random selection (RS) and supervised selection (SS) which select design choices based on the performance of limited labeled data, automatic selection via meta-predictor(s) is significantly more effective. However, we find that the SS method is even inferior to the RS method. A reasonable explanation is that the scarcity of labeled data makes it difficult for the selected design choices to generalize well on newcoming data.
2. Is it more helpful for meta-predictors to transfer knowledge across different datasets by end-to-end trained meta-features or extraction-based meta-features?
Here, we explore the impact of different methods for extracting meta-features on meta-predictor performance, as shown in Table 3. This includes: (i) Two-stage method, where the meta-features of a specific dataset are first extracted by the method proposed in MetaOD [88], i.e., extracting both statistical features like min, max, variance, skewness, covariance, etc., and landmarker features that depend on the unsupervised AD algorithms to capture outlying characteristics of a dataset. The extracted meta-features  $E^{meta}$  are then concatenated with  $n_a$  and  $E^{comp}$  as inputs to the metapredictor. (ii) End-to-end method, where the meta-features are directly extracted by the meta-predictor [30] and optimized with the learning process of the downstream task.
Generally, we observe close performances between the two-stage and end-to-end methods of extracting meta-features in the meta-predictors. This conclusion holds valid not only for the default MSE loss, but also for the Ranknet loss [11] used in the meta-predictors.
3. Is the tree-based ensemble meta-predictor effective in tabular AD? Can meta-predictors further benefit from model ensembling?
Consistent with the findings in the previous studies [24, 25, 26], we still find that the tree-based ensemble model(s) are better solutions for tabular-based AD tasks, where the performance of meta-predictors are improved when the MLP trainer used in meta-predictor is replaced by the ensemble models like XGBoost and CatBoost. Moreover, DL-based meta-predictor can also benefit from the model ensembling strategy, as we observe that both two-stage and end-to-end meta-predictors improve when we ensemble the anomaly scores of predicted top- $k$  combinations of design choices.
4. Do advanced loss functions bring performance gains to the meta-predictor?
In addition to the default MSE loss function, we also investigate several other losses (as shown in Table 4), including (i) Weighted MSE imposes a stronger penalty on errors in predicting the top and bottom design choices. (ii) Pearson correlation between the predictions and ground-truth targets. (iii) Ranknet [11] learns to rank different design choices, which is widely used in recommendation systems. However, compared to the results of MSE loss shown in Table 3, we do not find significant improvement when more complicated loss functions are implemented to train the meta-predictors.
Table 4: AUCROC performance of meta-predictors trained on other loss functions. Different loss functions do not yield significant performance improvement for the meta-predictor.
na	Weighted MSE	Pearson	Ranknet
DL-single	DL-ensemble	DL-single	DL-ensemble	DL-single	DL-ensemble
2-stage	end2end	2-stage	end2end	2-stage	end2end	2-stage	end2end	2-stage	end2end	2-stage	end2end
5	0.801	0.748	0.811	0.774	0.778	0.808	0.804	0.815	0.800	0.809	0.821	0.813
10	0.830	0.770	0.842	0.786	0.825	0.833	0.834	0.843	0.835	0.836	0.841	0.844
20	0.844	0.786	0.862	0.801	0.839	0.840	0.852	0.854	0.853	0.853	0.862	0.859
5. Does larger AD design space bring performance gains to the meta-predictors?
In order to explore whether the meta-predictors can uncover more promising AD design choices beyond those bolded design dimensions highlighted in Table 1, we include more possible design dimensions like hidden layers, dropout, and initialization methods in network construction, and epochs, batch size, and weight decay in network training, while maintaining the scale of design space (i.e., the number of Cartesian products of design choices) at 1,000. Table 5 shows that the metapredictors generally benefit from a larger design space, where the AUCROC performances of both DL- and ML-based meta-predictors achieve improvements compared to that of small design space
shown in Table 3. It's worth mentioning that DL meta-predictors seem to surpass ML meta-predictors in a larger space. This may be due to a raised complexity and lower probability of over-fitting in a larger selection pool, both benefiting DL meta-predictors. However, this performance gap is still trivial (0.002-0.006 in absolute value). Considering a better efficiency, we still present the ML meta-predictors as the formal recommendation.
Table 5: AUCROC performance of meta-predictors trained on large scale design space. Larger design space provides potentially more effective design choices for newcoming datasets.
na	RS	SS	GT	DL-single	DL-ensemble	ML-single	ML-ensemble
2-stage	end2end	2-stage	end2end	XGB	CatB	XGB	CatB
5	0.738	0.657	0.904	0.824	0.808	0.829	0.826	0.814	0.814	0.825	0.825
10	0.767	0.731	0.912	0.842	0.830	0.853	0.850	0.843	0.846	0.850	0.851
20	0.791	0.750	0.922	0.863	0.846	0.876	0.859	0.860	0.863	0.870	0.874
6. Does refining AD design space bring performance gains to the meta-predictor?
We further refined the design space in ADGym and excluded the design choices whose average performances on the training datasets are below the median, resulting in better yet fewer design choices that can be utilized for training meta-predictors. The refined results are reported in Table 6, compared to Table 3. We have not found that refining the design space brings any gains to the meta-predictor, possibly because such approach loses many potential design choices that could perform well on newcoming dataset, or loses (half of) the training data of meta-predictors.
Table 6: AUCROC performance of meta-predictors trained on refined design space. No significant improvement of meta-predictor is observed since not only potentially better design choices are discarded, but also results in a reduction of training data in meta-predictors.
na	RS	SS	GT	DL-single	DL-ensemble	ML-single	ML-ensemble
2-stage	end2end	2-stage	end2end	XGB	CatB	XGB	CatB
5	0.766	0.653	0.902	0.797	0.809	0.812	0.817	0.798	0.807	0.816	0.819
10	0.803	0.700	0.912	0.824	0.833	0.838	0.835	0.838	0.830	0.848	0.842
20	0.833	0.754	0.921	0.851	0.830	0.867	0.848	0.874	0.867	0.877	0.872
5 Conclusions, Limitations, and Future Directions
In this paper, we introduce ADGym, a novel platform designed for benchmarking and automating AD design choices. We break down AD algorithms into granular components and systematically assess each one's effectiveness through extensive experiments. Furthermore, we develop an automated method for selecting optimal design choices, enabling the creation of AD models that surpass current SOTA algorithms. Our results highlight the crucial role design choices play and offer a structured approach to optimizing them in model development. With the broader AD community in mind, we have made ADGym openly available. We believe its comprehensive design choice evaluation capabilities will significantly contribute to future advancements in automated AD model generation.
Looking ahead, we see several opportunities to enhance ADGym and broaden its application. Firstly, we plan to incorporate time-series AD tasks that handle data with temporal variations. By automating the design choice selection, AD models will be better equipped to adapt to these changing distributions, thereby improving anomaly detection in dynamic time-series contexts. Currently, ADGym is geared towards weakly supervised AD. In the future, we aim to include unsupervised neural network-based AD algorithms as well. Additionally, it is worth noting that non-neural-network-based techniques, such as ensemble-tree methods, have shown great promise in practical scenarios. Therefore, exploring automatic pipeline formulation in these areas is an exciting and valuable direction for future research.
We thank anonymous reviewers for their insightful feedback and comments. M.J., C.H., A.Z., S.H., and H.H. are supported by the National Natural Science Foundation of China under Grant No. 72271151, and the National Key Research and Development Program of China under Grant No. 2022YFC3303301. M.J., C.H., A.Z., S.H., and H.H. thank the financial support provided by FlagInfo-SHUFE Joint Laboratory. X.H. and Y.Z. are independently supported by CMU.
[89] Y. Zhao, S. Zhang, and L. Akoglu. Toward unsupervised outlier model selection. In IEEE International Conference on Data Mining (ICDM). IEEE, 2022.
[90] Y. Zhou, X. Song, Y. Zhang, F. Liu, C. Zhu, and L. Liu. Feature encoding with autoencoders for weakly supervised anomaly detection. TNNLS, 2021.
[91] B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho, and H. Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018.
We provide more details of the evaluated 29 datasets ( $\S$ A), compared baselines ( $\S$ B), the proposed ADGym ( $\S$ C), and additional experimental results ( $\S$ D).
A Dataset List
Most of the datasets used for model evaluation are derived from our previous work [26], and we drop those datasets smaller than 1,000, and use the subsets of 3,000 for those datasets greater than 3,000 due to the computational cost. We also remove those datasets that cause errors for the compared baseline models. This results in a total of 29 datasets, as is described in Table A1. These datasets cover many application domains, including healthcare (e.g., disease diagnosis), audio and language processing (e.g., speech recognition), image processing (e.g., object identification), finance (e.g., financial fraud detection), and more, where we show this information in the last column.
Table A1: Data description of the 29 datasets included in ADGym.
Data	# Samples	# Features	# Anomaly	% Anomaly	Category	Reference
ALOI	49534	27	1508	3.04	Image	[19]
annthyroid	7200	6	534	7.42	Healthcare	[61]
backdoor	95329	196	2329	2.44	Network	[55]
campaign	41188	62	4640	11.27	Finance	[59]
Cardiotocography	2114	21	466	22.04	Healthcare	[7]
celeba	202599	39	4547	2.24	Image	[59]
census	299285	500	18568	6.20	Sociology	[59]
donors	619326	10	36710	5.93	Sociology	[59]
fault	1941	27	673	34.67	Physical	[19]
landsat	6435	36	1333	20.71	Astronautics	[19]
letter	1600	32	100	6.25	Image	[21]
magic.gamma	19020	10	6688	35.16	Physical	[19]
mammography	11183	6	260	2.32	Healthcare	[77]
mnist	7603	100	700	9.21	Image	[37]
musk	3062	166	97	3.17	Chemistry	[16]
optdigits	5216	64	150	2.88	Image	[5]
PageBlocks	5393	10	510	9.46	Document	[54]
pendigits	6870	16	156	2.27	Image	[4]
satellite	6435	36	2036	31.64	Astronautics	[64]
satimage-2	5803	36	71	1.22	Astronautics	[64]
shuttle	49097	9	3511	7.15	Astronautics	[64]
skin	245057	3	50859	20.75	Image	[19]
SpamBase	4207	57	1679	39.91	Document	[12]
speech	3686	400	61	1.65	Linguistics	[10]
thyroid	3772	6	93	2.47	Healthcare	[62]
vowels	1456	12	50	3.43	Linguistics	[33]
Waveform	3443	21	100	2.90	Physics	[50]
Wilt	4819	5	257	5.33	Botany	[12]
yeast	1484	8	507	34.16	Biology	[28]
local	synthesis	[26]
global	synthesis	[26]
cluster	synthesis	[26]
dependency	synthesis	[26]
B Compared Baselines
We compare the automatically selected AD models via ADGym with the following weakly- and fully-supervised baselines, which are considered as effective AD solutions in the previous work [26].
1. Semi-Supervised Anomaly Detection via Adversarial Training (GANomaly) [3]. A GAN-based method defines the reconstruction error of the input data as the anomaly score. We replace the convolutional layer in its original version with the dense layer for the tabular AD task, where the hidden size of the encoder-decoder-encoder structure of GANomaly is set to half of the input dimension. We train the GANomaly for 50 epochs with 64 batch size, where the SGD [66] optimizer with 0.01 learning rate and 0.7 momentum is applied for both the generator and discriminator.
2. REpresentations for a random nEarest Neighbor distance-based method (REPEN) [56]. A neural network-based model that leverages transformed low-dimensional embedding for random distance-based detectors. The hidden size of REPEN is set to 20, and the margin of triplet loss is set to 1000. REPEN is trained for 1000 epochs with early stopping. The batch size is set to 256, where the total number of steps (batches of samples) is set to 50. Adadelta [82] optimizer with 0.001 learning rate and  $0.95\rho$  is applied to update network parameters.
3. Deep Semi-supervised Anomaly Detection (DeepSAD) [69]. DeepSAD is a deep one-class method that improves its unsupervised version DeepSVDD [68] by penalizing the inverse distances of anomaly representation such that anomalies must be mapped further away from the hypersphere center. The hyperparameter  $\eta$  in the loss function is set to 1.0, where DeepSAD is trained for 50 epochs with 128 batch size. Adam optimizer with 0.001 learning rate and  $10^{-6}$  weight decay is applied for updating the network parameters. DeepSAD additionally employs an autoencoder for calculating the initial center of the hypersphere, where the autoencoder is trained for 100 epochs with 128 batch size, and optimized by Adam optimizer with learning rate 0.001 and  $10^{-6}$  weight decay.
4. Deviation Networks (DevNet) [59]. A neural network-based model uses a prior Gaussian distribution to enforce a statistical deviation score of input instances. The margin hyperparameter  $a$  in the deviation loss is set to 5. DevNet is trained for 50 epochs with 512 batch size, where the total number of steps is set to 20. RMSprop [66] optimizer with 0.001 learning rate and  $0.95\rho$  is applied to update network parameters.
5. Feature Encoding With Autoencoders for Weakly Supervised Anomaly Detection (FEAWAD) [90]. A neural network-based model that integrates the network architecture of DAGMM [91] with the deviation loss in DevNet [59]. FEAWAD is trained for 30 epochs with 512 batch size, where the total number of steps is set to 20. Adam optimizer with 0.0001 learning rate is applied to update network parameters.
6. Residual Nets (ResNet) [24]. This method introduces a ResNet-like architecture [27] for tabular data. ResNet is trained for 100 epochs with 64 batch size. AdamW [51] optimizer with 0.001 learning rate is applied to update network parameters.
7. Feature Tokenizer + Transformer (FTTransformer) [24]. FTTransformer is an effective adaptation of the Transformer architecture [75] for tabular data. FTTransformer is trained for 100 epochs with 64 batch size. AdamW optimizer with 0.0001 learning rate and  $10^{-5}$  weight decay is applied to update network parameters.
C Details of ADGym
In this section, we provide more details of the design space in ADGym, including the pipelines of Data Handling, Network Construction, and Network Training. Besides, we also introduce detailed descriptions of the extracted meta-features and the trained meta-predictors.
C.1 Design Choices Specification
Data Handling
Data augmentation aims to enhance the quality of training data by generating synthetic anomalies, mitigating the class-imbalance problem in AD tasks. For a dataset  $\mathcal{D} = \{\pmb{x}_1^u,\dots,\pmb{x}_k^u,(\pmb{x}_{k + 1}^a,y_{k + 1}^a),\dots,(\pmb{x}_{k + m}^a,y_{k + m}^a)\}$  defined in the main paper, the output after a specific data augmentation method is  $D^{\prime} = \{\pmb{x}_{1}^{u},\dots,\pmb{x}_{k}^{u},(\pmb{x}_{k + 1}^{a},y_{k + 1}^{a}),\dots,(\pmb{x}_{k + m}^{a},y_{k + m}^{a}),(\pmb{x}_{k + m + 1}^{a},y_{k + m + 1}^{a}),\dots,(\pmb{x}_{k + m + j}^{a},y_{k + m + j}^{a})\}$ , where  $m\ll k$  and  $m + j\approx k$ . This will exhibit a more balanced distribution between the normal and abnormal ones. Except for the default setting (maintaining the original class distribution of the dataset), we present four additional choices in this dimension: Oversampling, SMOTE [14], Mixup [39, 74, 84], and GAN [6].
Data preprocessing includes two commonly used methods, MinMaxScaler and Normalization. Given the input data  $D \in \mathbb{R}^{n \times d}$ , for each feature vector  $F \in \mathbb{R}^n$ , MinMax scales  $F$  to the range  $[a,b]$ :  $F' = (F - F_{min}) * (b - a) / (F_{max} - F_{min}) + a$ , where  $F_{min}$  and  $F_{max}$  represent the minimum and maximum of  $F$ . For each sample vector  $S \in \mathbb{R}^d$ , Normalization computes its L2 norm and scale  $S$  to  $S' = S / ||S||^2$  if  $||S|| \neq 0$ .
Network Construction