SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Network Construction is often regarded as an important part in the CV or NLP domain, while it is often neglected in the AD problem. In fact, some design choices like AutoEncoder [90] and Transformer [24] could facilitate the detection performance of downstream tabular AD tasks. We comprehensively present the entire network construction process from various aspects, such as network architecture, hidden layers, activation functions, dropout rate, and parameter initialization, and provide numerous design choices.
Network Training
Many researchers regard the loss function as the key component of AD models. Binary cross entropy (BCE) loss is the conventionally accepted choice for classification tasks but is often considered inappropriate for AD. However, its value as a robust baseline may be overlooked by the AD community, as we have pointed out in the
main part §4.2.3. In ADGym, we explore the potential of diverse loss functions under different combinations of design choices.
As an overview of these loss functions, Focal loss [45] increases the weighting of abnormal data in cross-entropy loss, enabling the model to focus more on the classification of anomalies. Minus loss [18] offers divergent update directions for anomaly scores amidst normal and abnormal data, and circumvents the issue of loss explosion with an upper bound. Resembling minus loss in approach, Inverse loss [69] is inherently self-limiting. Hinge loss [56, 59] leverages a pre-established hyperparameter,  $M$ , to maintain an anomaly score margin of a minimum of  $M$ . Deviation loss [59] mandates that the anomaly scores of normal data align with a standard Gaussian distribution, simultaneously ensuring a minimal  $M$  margin for anomaly scores. Other dimensions here are consistent with common DL.
C.2 Meta-features and Meta-predictors
Details and the selected list of meta-features
Data characterization measures which represent task information are fundamental for meta-learning [65]. We extract multiple categories of them as meta-features in the reference of [88], including: (1) Statistical features depict the dataset from multiple statistical indicators, including but not limited to characteristics such as the minimum value, maximum value, mean, and variance, as well as combinations of these features. These characteristics are widely used in the previous AutoML studies, demonstrating their feasibility for application. (2) Landmarker features are constructed by a series of AD models, including iForest, HBOS, LODA, and PCA (anomalies are reconstructed harder), which represent more AD task-specific information. For each of the four different unsupervised AD models trained by a given dataset, we extract their characteristics to engineer a series of meta-features (See Table C3 for complete landmaker features). Overall, the experiment on the proposed two-stage method has proven that we can construct similar models for datasets with similar landmarker meta-features.
Table C2: Selected meta-features for characterizing an arbitrary dataset.
Name	Formula	Rationale	Variants
Nr instances	n	Speed, Scalability	Pn, log(n), log(n/p)
Nr features	p	Curse of dimensionality	log(p), % categorical
Sample mean	μ	Concentration
Sample median	X̂	Concentration
Sample var	σ2	Dispersion
Sample min	maxX	Data range
Sample max	minX	Data range
Sample std	σ	Dispersion
Percentile	Pi	Dispersion	q1, q25, q75, q99
Interquartile Range (IQR)	q75 - q25	Dispersion
Normalized mean	μmaxX	Data range
Normalized median	X̂maxX	Data range
Sample range	maxX - minX	Data range
Sample Gini	Dispersion
Median absolute deviation	median(X - X̂)	Variability and dispersion
Average absolute deviation	avg(X - X̂)	Variability and dispersion
Quantile Coefficient Dispersion	(q75 - q25)/(q75 + q25)	Dispersion
Coefficient of variance	Dispersion
Outlier outside 1 & 99	% samples outside 1% or 99%	Basic outlying patterns
Outlier 3 STD	% samples outside 3σ	Basic outlying patterns
Normal test	If a sample differs from a normal dist.	Feature normality	5th to 10th moments
kth moments
Skewness	Feature skewness	Feature normality	min, max, μ, σ, skewness, kurtosis
Kurtosis	μ4σ4	Feature normality	min, max, μ, σ, skewness, kurtosis
Correlation	ρ	Feature interdependence	min, max, μ, σ, skewness, kurtosis
Covariance	Cov	Feature interdependence	min, max, μ, σ, skewness, kurtosis
Sparsity	#Unique values/n	Degree of discreteness	min, max, μ, σ, skewness, kurtosis
ANOVA p-value	pANOVA	Feature redundancy	min, max, μ, σ, skewness, kurtosis
Coeff of variation	σx/μx	Dispersion
Norm. entropy	H(X)/log2n	Feature informativeness	min,max, σ, μ
Landmarker (HBOS)	See Table C3	Outlying patterns	Histogram density
Landmarker (LODA)	See Table C3	Outlying patterns	Histogram density
Landmarker (PCA)	See Table C3	Outlying patterns	Explained variance ratio, singular values
Landmarker (iForest)	See See Table C3	Outlying patterns	# of leaves, tree depth, feature importance
Detailed and Complete List of Meta-predictors We represent each design choice with LabelEncoder class from the scikit-learn library. These encoded data, together with the meta-features and the number of labeled anomalies  $n_{a}$ , are considered as the input data of the following three kinds of meta-predictors.
Two-stage meta-predictor. This meta-predictor uses the meta-features described above. It firstly extracts meta-features  $E_{i}^{meta}$  from the given training datasets as an essential input, and then combines these meta-features with design choice embeddings and  $n_a$ , learning to predict the relative performance rank.
Table C3: Landmarker features. An example of feature is the minimization of tree depth from the model iForest.
Model	Perspective	Variants
iForest	Tree depth
Number of leaves	min,max,mean,std, skewness,kurtosis
Mean of base tree feature importance
Max of base tess feature importance
HBOS	Mean of each histogram(per feature)	min,max,mean,std, skewness,kurtosis
Max of each histogram(per feature)
LODA	Mean of each random projection(per feature)	min,max,mean,std, skewness,kurtosis
Max of each random projection(per feature)
Mean of each histogram(per feature)
Max of each histogram(per feature)
PCA	Explained variance ratio on the first three principal components	The percentage of variance it captures for the top 3 principal components
Singular values	The top 3 singular values generated during SVD process
End-to-end meta-predictor. This meta-predictor learns meta-features implicitly through an end-to-end fashion that is inspired by Dataset2Vec[30].
Ensembled version of the above two types of meta-predictors. It has been proved that ensemble models, such as XGBoost and CatBoost, perform well in AD tasks, which motivates us to explore whether the ensemble meta-predictor exhibits superior performance in experiments. From another perspective, this is also a validation of whether our proposed meta-predictor possesses high robustness. Specifically speaking, for each AD task (dataset), we select the top-  $k$  (default  $k = 5$  in our experiments) best-performance design pipelines predicted by meta-predictor, and utilize the average voting strategy to output the anomaly scores.
Details on training strategies of meta-predictors
We experimented with different loss functions to better guide the meta-predictor in selecting the top- $k$  design pipelines. During the training process of meta-predictors, weighted MSE loss assigns greater weights for those design choices with both higher real and predicted performance. The former aims to help the model focus on accurately predicting high-performing design pipelines, while the latter avoids mistakenly predicting poor-performing pipelines as part of the top- $k$ . Pearson loss emphasizes the linear correlation between the predicted performances of different design choices and their actual ones. Ranknet loss [11], a commonly used pairwise algorithm in Learning to Rank, aims to model the probability that one design pipeline is superior to another in a probabilistic manner.
D.1 Additional Results of Large Evaluations on AD Design Choices
D.1.1 Using AUCPR as Metrics
For AUCPR performances, we evaluate different design dimensions via the box plots, as shown in Figure D1, D2, and D3, respectively. Overall, we find that the conclusions drawn based on either AUCROC (shown in the main paper) or AUCPR are consistent.
Figure D1: AUCPR performance of data handling designs. There is no significant difference between different design choices in both data augmentation and preprocessing dimensions.
Figure D2: AUCPR performance of network construction designs. MLP is still an effective architecture for AD tasks, where Tanh and LeakyReLU activation functions are generally better than ReLU.
(c) Activation function
(e) Network initialization
(e) Learning rate
(c) Epochs
Figure D3: AUCPR performance of network training designs. Hinge loss is generally a better AD loss function. Both Adam and RMSprop optimizers outperform classical SGD. Smaller epochs could prevent overfitting on limited labeled data and thus achieve better performance.
(f) Weight decay
(d) Batch size
D.1.2 Using Relative Rank as Metrics
Considering the varying degrees of difficulty across different datasets, we also demonstrate the relative rankings of both AUCROC (as shown in figure D4~D6) and AUCPR (as shown in figure D7~D9) to measure the performance of different design choices. Specifically, we rank 1000 sampled design choices on each dataset. For a more intuitive comparison with our previous results, we calculate  $1 - normalized(rank/design choice))$ , i.e., the inverse of the normalized rank value, as the metric, where higher values indicate better performance. We found that such a metric further amplifies the differences between various design choices and generally aligns with the previous experimental findings. Notably, under the metric of relative ranking, we observe a significant decline in the performance of ResNet, which is inferior to AE. Drawing from our previous experiments, it becomes evident that ResNet's stability across various design pipelines did not manifest under this metric, which, however, is a crucial advantage in practical scenarios.
Figure D4: AUCROC relative rank performance of data handling designs.
Figure D5: AUCROC relative rank performance of network construction designs.
Figure D6: AUCROC relative rank performance of network training designs. Hinge loss is generally a better AD loss function.
Figure D7: AUCPR relative rank performance of data handling designs.
Figure D8: AUCPR relative rank performance of network construction designs.
Figure D9: AUCPR relative rank performance of network training designs.
D.1.3 Unsupervised Network Pre-training
In addition to the existing network initialization methods introduced in Table 1, we also investigate whether deep learning based AD models can benefit from unsupervised pre-training tasks. For tabular AD problems, we follow [68, 69] to first train an AutoEncoder that has an encoder with the same architecture as the corresponding network, e.g., constructing encoder-decoder transformer block in FTTransformer, and then utilize the reconstruction loss (mean squared error between the input data and its reconstructed ones) to perform model training. After that, we initialize corresponding models with the converged parameters of the encoder and fine-tune them on the downstream tasks, which contain only limited labeled data.
Pre-training has been widely used for NLP [15] and CV [52] tasks and verified to significantly enhance the performance of downstream tasks. However, for tabular AD problems, we did not observe a significant advantage of pre-training over other network initialization methods, as shown in Figure D10. This may be due to the reason that compared to the NLP and CV tasks that contain rich textual semantics and visual patterns, the inherent structure of tabular data is more rigid and constrained, resulting in the model struggling to learn general features without label guidance.
Figure D10: AUCROC and AUCPR performance of network pre-training.
D.2 Additional Results for Different Anomalies of Synthetic Datasets
Based on 29 real-world scenario datasets, we generated synthetic datasets using three random seeds with four specific anomaly types: local, global, dependency, and cluster. These types were defined in our previous work [26]. After excluding datasets that could not be generated properly, we conducted two experiments on remaining synthetic datasets (totaling over 300): a large benchmark for design choices and a performance experiment for ADGym using global anomaly types as an example.
D.2.1 Large Evaluation on Design Choices over Synthetic Anomalies
This evaluation is conducted to expose the correlation between the effectiveness of different design choices and the characteristics of the datasets. To more intuitively display our experimental results, we selected three
representative design dimensions: data augmentation, network architecture, and loss function. As shown in figure D11, data augmentation techniques tend to have counterproductive effects on single-anomaly datasets in the vast majority of cases. Figure D12 shows the performance over different network architectures, where ResNet consistently excels over other architectures when dealing with dependency anomalies. However, its performance diminishes with datasets that have a global anomalies context and fewer anomalies. Compared to the two dimensions mentioned above, the variance between different loss functions is significantly greater, making the selection of an appropriate loss function crucial, as shown in figure D13. The deviation loss consistently demonstrates competitive results across various anomaly types and numbers of anomalies, and the performance of hinge loss also improves rapidly as the number of anomalies increases.
(a)  $1a = 5$
(b)  $\mathrm{{la}} = {10}$
Figure D11: Performance of data augmentation designs across various anomalies. Each point indicates the average performance across three random seeds. Anomalies: global (G), local (L), cluster (C), and dependency (D). The x-axis and y-axis show improvement ratios in AUC-ROC and AUC-PR, respectively
(c)  $\mathrm{{la}} = {20}$
(b)  $\mathrm{{la}} = {10}$
(c)  $\mathrm{{la}} = {20}$
Figure D12: performance of network architecture designs over different type of anomalies.
(a)  $\mathrm{{la}} = 5$
(b)  $\mathrm{{la}} = {10}$
Figure D13: performance of loss function designs over different type of anomalies.
D.2.2 Additional Results of ADGym over datasets with global anomalies
In this section, we investigate whether meta-predictor can select the best design choices based on dataset meta-features. We chose the meta-predictor trained with two shallow model methods and focused on datasets with global anomalies. These datasets have previously been shown to favor competitive design choices, such as the AE architecture and deviation loss function. Figures D14 and D15 respectively display our results concerning the network architecture and loss function design dimensions, both of which have been shown to significantly impact model performance in earlier experiments. We represent the results using scatter plots, where all results are averaged over three seeds and the number of anomalies; the closer to the top-right corner indicates better performance of the design choice. Empirical evidence shows that the meta-predictor can effectively select the best design choice (AE and deviation loss) in both dimensions, with the prediction results for network architecture almost perfectly aligning with the actual results.
(a) CatBoost
(b) XGBoost
Figure D14: performance of meta-predictors on network architecture over global anomalies. Figures (a) and (b) show meta-predictors output trained by CatBoost and XGBoost, and (c) shows result of previous benchmark experiments on global anomalies (as ground truth baseline). For a more intuitive comparison, we subtract the output from 1 to ensure that the best design choices are positioned closer to the top-right corner.
(c) Ground Truth
(a) CatBoost
Figure D15: performance of meta-predictors on loss function over global anomalies.
(b) XGBoost
D.3 Additional Results for Different Domains of Datasets
In order to investigate the performance of the AD design choices across different domains, we categorize 29 datasets into 14 distinct domains and aggregate the experimental results based on the domain of each dataset. In Table D4 and D5, each data point represents the average performance rank of the design choices (as indicated by the rows) on the datasets within a specific domain (as indicated by the columns). We observe that no design choices are universally superior or inferior. A case in point is the GAN-based data augmentation method, which generally underperforms on most datasets but shows a clear advantage in the financial domain. A similar situation also occurs with the performance of FTTransformer on datasets in the Sociology domain. Meanwhile, some designs that have performed well in past experiments, such as the deviation loss function, rank last on datasets in the financial domain. These results underscore the necessity of automated selection of AD designs and align with our previous conclusions.
Table D4: AUCROC performance rank of design choices across different domains. First column show different design dimensions, where aug, na, af, ls are abbreviations for data augmentation, network architecture, activate function, and loss function, respectively
design	method	Astronautics	Biology	Botany	Chemistry	Document	Finance	Healthcare	Image	Linguistics	Network	Physical	Physics	Sociology	Web
aug	GAN	5	5	5	5	5	2	5	5	5	5	5	5	4	5
aug	Mixup	2	2	1	3	4	4	4	2	1	4	3	4	2	2
aug	Origin	4	3	3	4	3	5	3	4	2	3	2	2	5	3
aug	Oversampling	3	4	2	2	2	3	2	3	3	2	4	3	3	4
aug	SMOTE	1	1	4	1	1	1	1	1	4	1	1	1	1	1
na	AE	2	3	2	2	1	2	2	3	4	2	2	2	3	3
na	FTT	4	4	4	4	4	4	4	4	1	4	4	4	1	4
na	MLP	1	1	3	1	2	1	1	1	3	1	1	1	2	2
na	ResNet	3	2	1	3	3	3	3	2	2	3	3	3	4	1
af	LeakyReLU	1	2	1	1	2	1	2	2	3	1	2	1	3	3
af	ReLU	3	3	2	3	3	3	3	3	1	3	3	3	2	2
af	Tanh	2	1	3	2	1	2	1	1	2	2	1	2	1	1
ls	bce	3	4	3	1	2	4	4	1	4	1	1	2	1	3
ls	deviation	4	3	2	2	4	6	3	3	2	3	3	4	4	1
ls	focal	5	6	5	4	6	2	5	5	6	5	6	5	5	5
ls	hinge	2	1	1	5	3	3	1	2	1	2	4	1	2	2
ls	inverse	6	5	6	6	5	5	6	6	5	6	5	6	6	6
ls	minus	1	2	4	3	1	1	2	4	3	4	2	3	3	4
Table D5: AUCPR performance rank of design choices across different domains.
design	method	Astronautics	Biology	Botany	Chemistry	Document	Finance	Healthcare	Image	Linguistics	Network	Physical	Physics	Sociology	Web
aug	GAN	4	4	4	4	4	1	4	4	4	4	4	4	4	4
aug	Mixup	1	1	1	2	3	3	3	1	1	1	2	1	1	1
aug	Origin	3	2	2	3	2	4	2	3	3	3	1	2	3	2
aug	Oversampling	2	3	3	1	1	2	1	2	2	2	3	3	2	3
aug	SMOTE	5	5	5	5	5	5	5	5	5	5	5	5	5	5
na	AE	3	3	2	2	1	2	2	3	4	3	2	3	4	3
na	FTT	4	4	4	4	4	4	4	4	2	4	4	4	1	4
na	MLP	1	2	3	1	2	1	1	1	3	1	1	1	2	2
na	ResNet	2	1	1	3	3	3	3	2	1	2	3	2	3	1
af	LeakyReLU	2	3	2	2	2	1	2	2	3	2	2	1	2	2
af	ReLU	3	2	1	3	3	3	3	3	1	3	3	3	1	3
af	Tanh	1	1	3	1	1	2	1	1	2	1	1	2	3	1
lf	bce	4	4	3	3	2	5	4	3	3	5	1	2	3	3
lf	deviation	1	3	2	1	4	6	2	1	1	3	2	3	2	2
lf	focal	5	6	5	5	6	2	5	5	6	4	6	5	5	5
lf	hinge	2	1	1	4	3	3	1	2	2	2	3	1	1	1
lf	inverse	6	5	6	6	5	4	6	6	5	6	5	6	6	6
lf	minus	3	2	4	2	1	1	3	4	4	1	4	4	4	4
D.4 Additional Results of Automatic Component Construction via ADGym
D.4.1 Using AUCPR as Metrics
In this subsection, we show the AUCPR performance of compared baseline AD methods, as well as different versions of meta-predictors proposed in our ADGym, as is shown in Table D6~D10. We still observe similar results to those illustrated in the main paper, where: (1) the AUCPR performances of automatically constructed AD methods via meta-predictors are better than current state-of-the-art AD solutions. (2) Meta-predictors benefit from either the instantiation of the ensembled tree-based model or an assembling strategy. (3) Large design space generally improves the performance of meta-predictors.
Table D6: Baseline of SOTA AD methods.
na	GANomaly	REPEN	DeepSAD	DevNet	FEAWAD	ResNet	FTTransformer
5	0.235	0.480	0.333	0.484	0.470	0.363	0.489
10	0.240	0.476	0.417	0.530	0.530	0.433	0.561
20	0.250	0.517	0.491	0.584	0.579	0.528	0.623
Table D7: Performance of auto-selected pipelines by ADGym. RS, SS, and GT refer to the random selection, supervised selection, and the best performance among all design choices, respectively. DL or ML corresponds to the meta-predictor that is either instantiated with MLP or tree-based method like XGBoost. The suffix -ensemble refers to ensemble multiple predictions of meta-predictors. "2-stage" and "end2end" corresponds to the two-stage and end-to-end for extracting meta-features.
na	RS	SS	GT	DL-single	DL-ensemble	ML-single	ML-ensemble
2-stage	end2end	2-stage	end2end	XGB	CatB	XGB	CatB
5	0.447	0.365	0.706	0.531	0.533	0.538	0.534	0.532	0.539	0.553	0.547
10	0.478	0.459	0.737	0.589	0.592	0.600	0.597	0.600	0.603	0.617	0.614
20	0.509	0.550	0.759	0.622	0.616	0.639	0.623	0.642	0.657	0.662	0.669
Table D8: AUCPR performance of meta-predictors trained on other loss functions. Different loss functions do not yield significant performance improvement for the meta-predictor.
na	Weighted MSE	Pearson	Ranknet
DL-single	DL-ensemble	DL-single	DL-ensemble	DL-single	DL-ensemble
2-stage	end2end	2-stage	end2end	2-stage	end2end	2-stage	end2end	2-stage	end2end	2-stage	end2end
5	0.495	0.500	0.502	0.501	0.524	0.525	0.527	0.533	0.525	0.527	0.529	0.553
10	0.558	0.533	0.571	0.543	0.585	0.581	0.603	0.599	0.583	0.581	0.592	0.599
20	0.592	0.573	0.614	0.560	0.631	0.602	0.640	0.609	0.635	0.624	0.642	0.623
Table D9: AUCPR performance of meta-predictors trained on large scale design space. Larger design space provides potentially more effective design choices for newcomers datasets.
na	RS	SS	GT	DL-single	DL-ensemble	ML-single	ML-ensemble
2-stage	end2end	2-stage	end2end	XGB	CatB	XGB	CatB
5	0.449	0.373	0.716	0.539	0.537	0.547	0.556	0.554	0.544	0.558	0.551
10	0.491	0.455	0.742	0.590	0.583	0.613	0.604	0.593	0.588	0.609	0.622
20	0.528	0.557	0.762	0.641	0.617	0.656	0.630	0.624	0.641	0.660	0.662
Table D10: AUCPR performance of meta-predictors trained on refined design space. No significant improvement of meta-predictor is observed since not only potentially better design choices are discarded, but also results in a reduction of training data in meta-predictors.
na	RS	SS	GT	DL-single	DL-ensemble	ML-single	ML-ensemble
2-stage	end2end	2-stage	end2end	XGB	CatB	XGB	CatB
5	0.467	0.370	0.706	0.528	0.509	0.531	0.523	0.522	0.536	0.541	0.540
10	0.532	0.458	0.737	0.573	0.550	0.593	0.563	0.597	0.612	0.607	0.620
20	0.564	0.540	0.759	0.614	0.580	0.656	0.582	0.635	0.647	0.659	0.671
D.4.2 Using Relative Rank as Metrics
In this subsection, we present results using the relative rankings based on AUCROC and AUCPR as metrics to mitigate the impact of variations in dataset difficulty. Specifically, we compute the relative rankings for each datasets across all models, including design pipelines generated by various versions of ADGym meta-predictor. We then subtract these normalized results from 1, ensuring that larger values indicate better performance. Table D11 and Table D12 depict the relative rankings of various SOTA models for AUCROC and AUCPR, respectively, while Table D13 and Table D14 compare the performance of different meta-predictor versions. It's important to note that performance cannot be compared across tables using this relative ranking metric, so we include the average result of the SOTA models as a baseline for ranking purposes. Our findings are consistent with previous conclusions, indicating that the performance of design pipelines from the meta predictor surpasses that of the existing SOTA models.
Table D11: Baseline of SOTA AD methods with AUCROC relative rank.
na	GANomaly	REPEN	DeepSAD	DevNet	FEAWAD	ResNet	FTTransformer
5	0.383	0.738	0.557	0.665	0.609	0.349	0.663
10	0.331	0.662	0.575	0.685	0.641	0.375	0.699
20	0.301	0.612	0.577	0.672	0.676	0.394	0.736
Table D12: Baseline of SOTA AD methods with AUCPR relative rank.
na	GANomaly	REPEN	DeepSAD	DevNet	FEAWAD	ResNet	FTTransformer
5	0.370	0.672	0.450	0.690	0.661	0.474	0.652
10	0.310	0.605	0.490	0.710	0.686	0.484	0.686
20	0.271	0.586	0.503	0.691	0.698	0.503	0.719