Hyperparameter Sensitivity in Deep Outlier Detection: Analysis and a Scalable Hyper-Ensemble Solution
Authors: Xueying Ding
Lingxiao Zhao
Leman Akoglu
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep outlier detection
unsupervised model selection
ensemble learning

Abstract: Outlier detection (OD) literature exhibits numerous algorithms as it applies to diverse domains. However, given a new detection task, it is unclear how to choose an algorithm to use, nor how to set its hyperparameter(s) (HPs) in unsupervised settings. HP tuning is an ever-growing problem with the arrival of many new detectors based on deep learning, which usually come with a long list of HPs. Surprisingly, the issue of model selection in the outlier mining literature has been “the elephant in the room”; a significant factor in unlocking the utmost potential of deep methods, yet little said or done to systematically tackle the issue. In the first part of this paper, we conduct the first large-scale analysis on the HP sensitivity of deep OD methods, and through more than 35,000 trained models, quantitatively demonstrate that model selection is inevitable. Next, we design a HP-robust and scalable deep hyper-ensemble model called ROBOD that assembles models with varying HP configurations, bypassing the choice paralysis. Importantly, we introduce novel strategies to speed up ensemble training, such as parameter sharing, batch/simultaneous training, and data subsampling, that allow us to train fewer models with fewer parameters. Extensive experiments on both image and tabular datasets show that ROBOD achieves and retains robust, state-of-the-art detection performance as compared to its modern counterparts, while taking only 2-10% of the time by the naïve hyper-ensemble with independent training.

Hyperparameter Sensitivity in Deep Outlier Detection Analysis and Scalable Hyper-Ensemble Solution
Xueying Ding Carnegie Mellon University xding28cs. cmu. edu
Lingxiao Zhao Carnegie Mellon University lingxiao@cmi
Leman Akoglu
Carnegie Mellon University
lakoglućcs cmu edu
Abstract
Outlier detection (OD) literature exhibits numerous algorithms it applies diverse domains. However, given a new detection task, it is unclear how to choose an algorithm to use, nor how to set its hyperparameter(s) (HPs) in unsupervised settings. HP tuning is an ever-growing problem with the arrival of many new detectors based on deep learning, which usually come with long list of HPs. Surprisingly, the issue of model selection in the outlier mining literature has been "the elephant in the room"; a significant factor in unlocking the utmost potential of deep methods, yet little said or done to systematically tackle the issue. In the first part of this paper, we conduct the first large-scale analysis on the HP sensitivity of deep OD methods, and through more than 35,000 trained models, quantitatively demonstrate that model selection is inevitable. Next, we design HP-robust and scalable deep hyper-ensemble model called ROBOD that assembles models with varying HP configurations, bypassing the choice paralysis. Importantly, we introduce novel strategies to speed up ensemble training, such parameter sharing, batch/simultaneous training, and data subsampling, that allow us to train fewer models with fewer parameters. Extensive experiments both image and tabular datasets show that ROBOD achieves and retains robust, state-of of the-art detection performance compared to its modern counterparts, while taking only 2-10% of the time by the naive hyper-ensemble with independent training.
Introduction
Outlier detection (OD) finds numerous real-world applications in finance, security, healthcare, to name a few. Thanks to this popularity, the literature has grown to offer a large catalog of detection algorithms [ With the recent advances in deep learning, the literature has been booming with the addition of many more OD models based on deep neural networks (NNs). (See surveys
[ 36
While there is no shortage of OD methods today, given new task, it is unclear how to choose which algorithm or model to use, nor how configure its hyperparameter(s) (HPs) in unsupervised settings. That is, the fundamental problem of outlier model selection remains vastly understudied. Several evaluation studies have illustrated the sensitivity to HPs for traditional (i.e. non-deep) OD methods 2 g 17 Most surprisingly, however, the issue of HP tuning/model selection for the newly burgeoning deep OD models has been "the elephant the room"; well-known problem that no one seems to want to bring up, which is exactly the focus of this paper.
Deep OD models are promising thanks to appealing properties such as task-driven representation learning and end-to-end optimization. On the other hand, while their traditional counterparts had only 1-2 HP deep OD models come with long list of HPs: (i) architecture HPs (e.g. depth, width), (ii) regularization HPs (e.g. dropout, weight decay rates), and (iii) optimization HPs (e.g.
c.g.. k in nearest neighbor based LOF 7. in OCSVM 39 sample size e and #trces in IF 28
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

learning rate, epochs). These are inherited ones from regular deep NNs while most also exhibit their coodee--ppccfficsspeeaaizzz HPs. It would not be a freak occurrence to assume that their performance is heavily dependent on these HP settings. However, our closer analysis of the experiment testbeds in recent literature on deep OD models falls far from systematically addressing the issue. (See Appx. A. Table X for preview.) We find hardly any discussion on model selection, with only a few work empirically studying sensitivity but to model-specific HPs only, Majority of work report results for a single "recommended" (how, unclear) configuration used for all datasets, or tune only a subset of the HPs on labeled validation and sometimes even test (!) data. To the best of our knowledge, there is no existing work that attempts (unsupervised) model selection for deep OD models.
In this work, our research goals are two-fold. First, through extensive experiments, we quantitatively demonstrate that deep OD models from various families are all sensitive to their HP settings. Our analysis shows that model selection is inevitable and is key to truly unlock the utmost potential of deep OD models. Second, motivated by our analysis, we propose a scalable deep hyper-ensemble called ROBOD that obviates HP selection through assembly of deep autoencoders with varying HP configurations. To speed up ensemble training, we introduce novel architectural and training strategies, and train fewer models, with fewer parameters, on smaller subsamples of data; by leveraging parameter sharing and joint/imullaneous training. The main contributions of our work are as follows.
First large-scale study on HP sensitivity of deep OD models: We build large testbed to systematically measure the performance variability of deep OD models under varying HP settings. Our study involves models from four different families, on both image and vector data, under both "clean" (i.e. inlier only) as well as "polluted"' (i.e. outlir-cootammnated) training data, over 3 random initializations. 80-800+ different HP configurations per model across 4-8 unique HPs. Overall, our analysis involves more than 35,000 runs, (Sec. E ROBOD, new deep hyper-ensemble OD model: Motivated by our empirical study, we propose a ensemble model called ROBOD which combines scores from a collection of models, each trained with different HP configuration. Rather than trying choose,
ROBOD fully bypasses the choice of and hence sensitivity to HP settings, and achieves robust (i.e. stable) performance across different initializations. (Sec. 4.T)
Design strategies to speed up hyper-ensemble training: We propose speed-up strategies to efficiently hyper-ensemble model depth and width. We use an autoencoder (AE) with skip-connections to simultaneously train multiple AEs with different depths. In addition, we employ batch training of multiple models and use zero-masking on shared parameters to get different widths. Together, these provide a 90-98% savings in running time. (Sec. 4.2)
Extensive experiments: Besides our sarge-scale measurement study, we also perform experiments additional benchmark datasets, comparing ROBOD to bascline deep OD models as well as a traditional tree-ensemble. ROBOD achieves competitive or often better performance which, importantly, exhibits low variance by random initialization. (Sec. 5 We expect that our work will increase awareness and help shift the community's 's focus (at least to some extent) from building the next yet-another deep OD model toward the fundamental issue of unsupervised model selection and hyperparameter-robuss model design. To foster future research, we open source all code and datasets at /tttts://ggiiee //gptttttt..t.../.g com/
ROBOD
2 Related Work
Unsupervised Outlier Detection (OD). There exists a large pool of what-is-now-called traditional, ie. not deep learning based, OD methods [0]. These methods work with the original features or subspaces thereof, and typically exhibit just one or two hyperparameters (HPs) _ With recent advances in deep learning, there has been a boom in deep OD models, as those can learn new task-dependent
feature representations and directly optimize an OD objective. Despite their short history, multiple
surveys have been published that aim to cover this fast-growing literature O 31 3d While deep OD models have been shown to outperform their traditional counterparts, they exhibit a much longer list of (typically 4-8) HPs (e.g. depth, width, dropout, weight decay, learning rate, epochs, etc. besides other speelfspe HPs) that makes them very challenging tune in unsupervised settings.
Model Selection in Unsupervised OD: Prior (Black) Art. At large, unsupervised outlier model selection remains to be vastly understudied, yet extremely important area. Various evaluation studies have reported traditional detectors to be quite sensitive to their HP choices ② 17,, raising concern for the fair evaluation and comparison of different models. Earlier work on automatically selecting

---

HP's are limited to one-class models 14 40 41 unsupervised) model evaluation heuristics have been proposed [1 input data (without labels) and the output (i.e., outlier scores). MetaOD 47
More recently, general-purpose internal (i.e.,
30 which solely rely on the employs meta-learning to transfer information from similar historical tasks to new task for model selection, which has only been tested on traditional OD models. Different from those that aim to select single model, ensemble models have also been employed for OD B1. including those that combine models from the same family [26] as well as heterogeneous detectors from different families
29
33
Regarding deep OD methods, we have surveyed a large collection of recent papers and their experi- mental testbed and HP settings, summary of which is given in Appx. A. Table g To our surprise, we found hardly any discussion on model selection, with only a few work presenting sensitivity analysis with respect to not all but some, model-specific HPs. While some work reserve labeled validation/hold-outt data to tune a subset of the HPs
231, majority of them fix the HP values A9I. Moreover, a non-negligibe number of existing work choose some critical HPs empirically on test data (!) to yield optimum last column). Some work that builds on previous models (e.g., deep 35 vs. multi-sphere extension [3 transformation-basee methods I6 for
6
18
and call them "recommended"/defff
settings 5
11
37
38
46
results 4 35 4 (See Table g
SVDD-based methods
images vs. their extension to vector data [5] AnoGAN
37 and the follow-up EGBAD (46] use the
same architecture and HP settings as the prior work for consistent/fair" comparison. However is unlikely that the same HP values would work comparably for different models.
Admittedly, it challenging to tune (a long list of) HPs in the absence of labels, yet, the opacity the deep OD literature warrants careful investigation on the stability of model performance under varying HP settings, and ultimately on the fair comparison between these and traditional OD methods.
Deep Model Ensembles. Recently, deep NN predictions have been found to be often poorly calibrated 20) As Bayesian learning does not offer straightforward training, deep ensemble models have been proposed as simple alternative 25 to improve predictive uncertainty, as well as efficient In this work, we leverage ensemble modeling toward different goal: to improve the stability and robustness of unsupervised OD models to HP settings, combining predictions from models with different HPss into an OD eyper-ensemble. The closest to our work is Wenzel et al. 's deep syper-ensemble [3) which, different from ours, considers supervised problems, to further foster diversity in the ensemble and thereby achieve better uncertainty estimation.
ways of training deep NN ensembles MV 2
3 Hyperparameter-Sennitivvyy Analysis of Deep OD
3.1 Testbed Setup
Models. We study HP sensitivity of five deep OD methods of four different types: a basic deep
autoencoder VanillaAE trained with reconstruction loss, robust deep autoencoder RDA 48 one--laas classification based DeepSVDD 35), adversarial training based GANomaly [4], and an (AF) ensemble model RandNet [1I] These exhibit 4 to HPs, listed in Table (See Appx. A.2 for descriptions.) (Note that RandNet not a hyper-ensembe
members use the same HP configs except for NN sparsity.) We detine a grid 2-3 different values for each HP, including the author-recommended values when available (See details in Appx. Table () and train each deep OD method with all combinations; yielding 81-864 different models. (Note that 192 RandNet models each consists of an ensemble of 50 or 200 AEs.) We repeat each experiment 3 times with different initializations.
Table 1: Deep OD models used for studying hyperparameter sensitivity. We give the number (in parenthesis) and the list of HPs for each method, along with the total number of models trained for evaluation. (See Appx. A.2 for HP descriptions and Appx. Table D for list of grid values per HP.)
Method
List of hyperparameters (HPs)
#models
VanillaAE
(4) n layers layer_decay LR iter
RDA 48
(6) A n_layers layer_decay -LR inner_iter iter
324
DeepSVDD :5
(8) conv_diin
fc_dim- Relu pluusllpe-... pretr iter pretr_LR- iter LR wgght
864
GANomaly 4
(6) Wade = 1 W2om Wene 7_dim I.R iter
162
RandNet
n_ayerss layer_decay- sample_r r ens_size pretr pretr_iter= iter= 100 iter I.R wight_d-= do=
192
Train/ Testt settings. In their original papers, DeepSVDD and GANomaly are trained on what we refer to as Clean (inlier only) data, and tested on a disjoint test dataset. In contrast, RDA and RandNet consider the transductive setting where the train data is the same as the test data, containing inliers as
3

---

0.90
0.95-
0.9
0.65
0.85
0,90
0,6
0.80
0.55
0.73-
0.80
0,7
0.4r
0.55
0.4
0.70
0.75
0.6
0.65
0.6
0.21
VanillaeE
DeepSVDD
GANomally
DeesSVOD(N-3)
DespSVDD(C:1)
RandNeedhyy
RandNettCan
GNomaly
Mean
Min Loss
HP-ensemble
Mcan
MPPensamble
Figure 1: (left) AUROC performance of deep OD methods with different HP configurations (circles) on MNIST-4 showcase notable variation (i.e., sensitivity). (See footnote 2 below.) (right) Similar results on additional datasets; DeepSVDD on MNIST-5 and CIFAR 10aauto, RandNet Thyroid and Cardio, and GANomaly on MNIST-4out. Reported values (dashed lines) are overestimates of Mean (D). Hyper-ensemble (X)) improves notably over Mean.
well as outliers, which we refer to as Polluted. It is often understood to be more challenging than the Clean setting for unsupervised OD. For completeness, we evaluate these methods under both settings.
Datasets. For evaluation we consider both image point datasets. As in the original papers [4] 355, use MNIST and CIFAR10 construct OD tasks, except for RandNet which uses fully-connected rather than convolutional layers and originally tested only on point-cloud benchmarks.
Metrics. Across all HP settings of deep OD method, we report the mean AUROC, standard deviation (stdev), as well as minimum and maximum. Mean corresponds to the expected performance when HP config is randomly picked from our (multi-dimensional) grid. We also report the average mean and its stdev over repeated runs. For datasets that are directly comparable to those in the original papers, contrast the mean model performance that obtain to that reported value for cettorrreccmmentee
HPs. We compare to a simple model (i.c. HP) selection heuristic, which selects the (one) model with the lowest loss/objective value. In addition, instead of selecting one, we average the scores from all configurations and report the AUROC of this hyper-ensemble.
Over five deep OD methods, tens to hundreds of HP configs, multiple initializations, Clean and Polluted settings, and various datasets, we have trained a total more than 35,000 models. As such, our study constitutes the first large-scale HP sensitivity analysis in the deep outlier mining literature.
3.2 Results and Observations
Fig. [eftt) provides the AUROC performances for VanillaAE, RDA, DeepSVDD and GANomaly across all HP configurations (circles), for the Polluted setting (See Appx. A. Fig. 0 for Clean setting.) on MNIST-4 dataset where digit '4' images are designated as inliers and the rest nine classes down-sampled at 10% as outliers, in the DeepSVDD paper (35]. Horizontal bars mark the 1st, 2nd (i.e. median), and 3rd quartiles, square the mean AUROC across HPs, triangle the selected model by lowest loss, and star the AUROC of the hyper-ensemble. (Corresponding plots over 3 per method are in Appx. A.3 Fig. Table 2 provides summary statistics as well as the hyper-ensemble performance for comparison for both the (left) Clean and (right) Polluted settings.
First, in Fig. we observe that all methods exhibit notable variability in performance across HPs, with many thane-than--vvragee configurations; e.g. for GANomaly, as well as many better-than-averann models; e.g. for RDA. Mean performance is considerably lower than the best model's, lss, illustrating the opportunity room for improvement. As one would expect, the performances in Table D are lower in the Polluted setting as compared to Clean (less so for the robust RDA), while sensitivity (i.e. stdev) comparable only slightly higher; showing that HP choice is critical under both settings.
We also find that selecting a model by the value of loss (V), despite requiring to train and choose from all models, often is much worse than the mean, i.c. random picking (single) model (L); proving this simple heuristic ineffective. On the other hand, the hyper-ensemble (x²) outperforms the mean in all cases, with quite small standard deviation across runs (i.e. low sensitivity to initialization).
We remark that although the methods are presented side-by-side the goal here is not to compare them "head-te to head" but rather, to analyze each one's performance variability by varying HPs on individual datasets.

---

Table 2: Basic stats of AUROC (%) distribution over varying HPs on MNIST-4 under (left) Clean and (right) Polluted settings. There is significant gap between the best and worst HP settings, with notable stdev around the mean. Polluted results have lower mean, and comparable or slightly higher variance. Hyper-ensemble outperforms random choice (i.e. mean), with low variability by initialization.
Method
Min&Max
Mean&Sid.
Mean lavg. runs]
Hyper-ons. Mean (avg. runs)
Max
Mtdn&
Mean
Hyper-ens. Mean (avg. runs)
(avg. runs
Vanilla AE
87.41
98.19
93.12929
93,12+0.008
98, 98.30+0.03
76.34
91.20
85.73-2.95
85.76+00.08
90.46+00112 86.82=0.04
RDA
81.79
95.96
88.9442.45 43
88.95=0.02
89.39
62.29
93.35
36.30-3368
80.24-0.06
DespSVDD
75.75
96.58
92.39.1.02
92.37.0.02
95.9410.05
67.54
91.71
81,65=4.15
81.6610.100
93.07.0.70
GANomaly
17.49
95.73
78.90166.3
78.26+00.51
8687740.31
29.06
86.60
67.20+ 10.30
67.45-40.32
73.87:0.522
The reported performance (0.949 = 0.008) of DeepSVDD on MNIST-4 (under Clean) B51 is similar yet somewhat optimistic over the mean value obtain. MNIST-4 is easy task for DeepSVDD since mean AUROC is already around 0.924. Thus, we set up MNIST-5 (with digit '5' as inliers) with reported AUROC 0.885 + 0.009, as well CIFAR a0-t (with class 'automobile' as inliers) (in both cases rest of the classes are subsampled at 10% each as outliers) with reported AUROC 0.659 + 0.021. We run DeepSVDD on both datasets under all 864 configurations. As shown in Fig. | (right), our mean AUROCs are 0.857 + 0.037 for MNIST-5, and 0.605 + 0.024 for CIFARIO 10-auto. (See Appx. A.3 Fig. 8 for all 3 runs.) The reported performances for the "recommended" HP in DeepSVDD (dashed red lines) appear to be optimistic over the mean, i.e. what one would expect by random choice (in the absence of any labels or other strategies).
The optimistic reporting trend holds for GANomaly and RandNet as well. Following their original paper [4) and different from 35 we up MNIST-4out dataset contain digit '4' images this time as the outliers and the rest nine classes as inliers. GANomaly performances across 162 HP configs are shown in Fig. | (right), where the reported result (red line) in 4 of AUROC 0.795 appears notably
higher than the mean 0.668 + 0.105 that we obtain. (See Appx. A.3 Fig. D for all 3 runs.)
For RandNet, we first find and verify third-party implementation, as the authors could not pub- licly share theirs, by replicating similar performances to those reported in [I]] using the author- recommended HP settings on all 8 datasets. Then for Cardio and Thyroid datasets, train 192 RandNet ensembles with varying HPs under Polluted as in the original paper. Results from one run shown in Fig. (right). (See Appx. A.3 Fig. 10 for all 3 runs.) On Cardio, AUROC mean is 0.894 + 0.045 vs. 0.929 (reported), and Thyroid mean is 0.822 + 0.026 vs. 0.904 (reported).
In summary, the take-aways from our analysis are as follows. First, it is clear that deep OD models are sensitive to their HP settings, showing that model/HP selection inevitable. The mean/expected value (of random choice) can be quite away from the best model, motivating this line of research. Second, the recommended settings in recent deep OD papers are arguably optimistic; otherwise more transparency into their selection mechanism is warranted. Finally, hyper-ensemble performance is superior to the mean, possibly owing to different HPs implicitly imposing diversity among constituent models which helps improve detection. Besides performance improvement, hyper-ensembling obviates model selection by joining all models rather than choosing one and is not much sensitive to initialization setting the stage for our proposed HP-robust OI) method ROBOD
4
ROBOD: A Deep Hyper-ensemble for Hyperparamettr-Robuus OD
4.1 Motivation and Overview
The main research question we consider is RQ1) how to design an unsupervised deep OD model that is robust to its HPs, i.e. an OD method that has stable, low-variance predictions under varying HPs. Motivated by our sensitivity analysis in Sec. B we propose a deep autoencoder (AE) hyper-ensemble model that combines scores from AE models with different HP configurations.
Definition 4.1 (Hyper-ensemble) Given model family M with H HPs, let X E RR denote specific setting of the HPs. A hyper-ensemble averages the output (outlier scores) from a finite number of base models with m different config.s, i.e. A._Miss X:) for point x.
Hyper-ensembles exhibit multiple advantages. First, and to our end goal, they ease the model/HP selection burden, bypassing the "choice paralysis". They are less sensitive to random initializations,
2 Besides fixed values for various HPs, Ruff et al. (35] recommend HPs that differ by dataset; e.g. on MNIST they use 2 CNN modules w/ size and 4 filters, on CIFARIO they use 3 modules w/ size 32, 64, and 128 filters.
5

---

Hyper-ens. width through zero-masking
Hyper-en Hyper-ens. ens. depth w/ AE-S
OLADuR
fx
Output
Colpu
W
AE-2
We
AE-2
AE-2
AE-4
AE-6
W
h
h
W
Inputt
Imput
Irpui
W
W
W
W
Sk
St
zeros"
shared weights
zero-mask
Joint/batch training w/ shared param.s
Figure 2: Main design elements in ROBOD: (1) AE with Skip links (AE-S): Each AE-S, denoted f. hyper-ensembles multiple AE models with various depths (in the figure, AF-2, 4 and 6), with shared parameters; e.g. for f.. layer 2 weights W (in dashed circle) is shared among different-depth AEs (i.e. AE-4 and AE-6). (2) Batch ensemble (BE) training with zero-masks: Multiple AE-S models with shared parameters are trained simultaneously, having various widths thanks to zero-masking; e.g. at layer 2, W is shared by all three different-widee AE-S models in the batch (denoted fj. fk. ft), where hidden sizes follow [hyl > hhs > hi| through varying size zero-masks on 8& and 81. (sr" depicts an outer product, and denotes Hadamard/element wise matrix multiplication.)
with lower variance performance. Moreover, they can even boost detection performance thanks to the diversity offered by different HPs and ensemble prediction.
The caveat is that deep ensembles computationally expensive to train. As such, the second research task we tackle is RQ2) how to speed up hyper-ensemble training. To that end, we propose novel architectural and training strategies. In nutshell, these strategies involve three main ideas: (1) we design a multi-layer AE architecture with skip connections, denoted AE-S, that helps hyper-ensemble, under single model, varying-depth AEs with shared parameters; (2) employ batch ensemble training 42 of multiple AE-S models using varying size zero-masking that helps hyper-ensemble varying-win width AES all trained simultaneously with shared parameters: and (3) we train each AE-S on a subsample and use out-of-sample scoring. In effect, these strategies allow us to build fewer models, with fewer total number of parameters, on less training data taking only 2-10% of the time that the naïve ensemble training would take where each model is trained independently.
Fig. 2 illustrates the main design elements of ROBOD pictorially. We present details of the architecture and the training as follows.
4.2 Design Strategies for Speeding up Ensemble Training
4.2.1
Hyper-ensembling depth: One model for multiple depths
Skip or shortcut connections are applied in NNs for various purposes: e.g., ResNet 21 for solving the depth degradation problem, in JK-Net 3 for representation learning with varying localities, in
U-Net 32 and stochastic depth NNs 22 for training models with adaptive depth, and so on.
Due to the "hourglass" structure of the AE, skipping one layer in an AE would cause dimensionality mismatch for the next layer. Instead, we create long shortcuts by skipping the middle layers while keeping the outer encoder-decoder pairs symmetrically, and refer to this architecture as AE-S.
An illustration of hyper-ensembling depth be found in Fig. 2 see e.g. the AE-S denoted by fy- Given a 2L-layer AÉ-S (L encoder and decoder layers), there are 2L trainable weight matrices. For input D, we generate L outputs by allowing the input to pass through the outermost encoder-decodes pair (which we denote as AE-2), the two outermost encoder-decoder pairs (AE-4), and so on, until the full structure (A F-2L) is traversed by the input. In effect, this trains different AEs under one AE-S model. Then, the overall loss LAE-S of the depth-tyyprreessmbbingg AE-S is the summation of the member AEs' reconstruction errors (denoted A Frr). Eq. |I gives example of the loss function for a 6-layer AE-S as shown in Fig. 2
LAE-S = AErr (D; W2)) + ErT [ :..) + AETTAE (D; WW:WW.) (I) AE-S is an implicit hyper-ensemble which allows simultaneous/joint training of AEs with various depths. It is computationally efficient thanks to parameter sharing, the outer layers' weights are

---

reused and tuned among different members. Each ensemble member can also play a regularization effect and prevent an AE from producing low scores for outliers due to overfitting. (See Appx. A.4).
4.2.2 Hyper-ensembling width: Zero-masked joint training
BatchEnsemble [42 (BE) is a state-of-the-arr parameter-efficien deep ensemble training approach. However, off-the-shelee it does not allow for hyper-ensebling, that is, all models in the batch trained with the same HP configuration, and share equal size parameters. Our approach builds on BE, adapting it to simultaneously train varying-width AE-S with shared parameters.
First, we briefly review the BE architecture with K AEs. For a specific layer BE, weight W E Rm is shared across all K individual members, while each member i maintains two trainable rank-1 vectors: S, E I and r, E R". The outer product (+) of r, and s creates a "mask" (a matrix in IR onto the shared weight W and generates individual weight W, For the input mini-batch X, E R the forward propagation computation is
① (X,W., 4 (X: (W (s, r)))))= = 4 ((X, os;:
(2)
where & is the activation function, is the element-wise product, and r; and s; are broadcasted row-wise or column-wise depending on the shapes at play. To vectorize the calculation for all K members in the ensemble, two matrices S E IR [gKxm, xm and R E Rxxr are constructed with rows of s; and r;, respectively. Thus, for the input X E Rkmxn, the BE layer computes the next layer as
cb ((X S)W R)
Since X is composed of K mini-batching X, 's tiling up, each member can utilize a different mini- batch of input. The K members are training simultaneously in parallel using one forward pass, thus relieving the memory and computational burden of traditional ensemble methods.
While BE itself is not a hyper-ensemble, we can leverage this architecture to aggregate networks of different widths, also in an efficient way. The widths of neural network layer correspond to the rows in the weight matrix. For a specific layer, instead of directly operating on W, we instead initialize a zero-one masking vector ag E {0, 1}". a; is to element- multiply with the r; E IR", such that the operation W os; (r? o will create a masking matrix of size (m X r) and sparsify the individual weight W, by allowing zero's 'ddouut rows. In vectorized notation, the matrix A E {0, 13 xr is composed of K distinct masking vectors. Then, the masked BE forward propagation becomes
& (X (R A))
(4)
Sparsifying neural networks has been shown decrease storage and improve training efficiency, with many algorithms built to wisely prune the neural network (22). While zero-masker BE is similar to creating neural networks of varying density, our main goal is to obtain models with different widths (i.e. hidden sizes), such that when trained under BE, creates a width-llyper-ensembbe Specifically, we construct the mero-masked BE layer with the maximum width and specify the zeros in respective a vectors, such that the masked-out individual weights correspond to varying-width models.
The zero-masked BE is efficient as A is fixed throughout training. The element- wise product of the masking incurs little extra time during forward and backward propagation, while all the (rank-1) vectors are cheap to store compared to separate weight matrices in traditional ensemble training.
4.2.3 ROBOD: The overall hyper-ensemble
Let H/ denote the total number of HPs for an AE-S, where we define a grid of values for each HP; e.g. depth -|4, 6, 8], width- -(64, 128, 256), Irn_rate =[1e-3,5e 51), drop_out = 0.2). etc.
For the two HPs, depth and width, we specify the AE-S with the largest depth value (following Sec. 4.2.1 say 2L) in the grid and leverage the skip connections to obtain the smaller-depth AEs. Similarly, we specify each AE-S in the ceroed BE with the largest width value (following Sec.
4.2.2 say K) in the grid and leverage the mero-masking to obtain other, smaller-width AEs.
Then, mazrr-mmsked BE trains in parallel K varying-width AE-S models, each being an ensemble of varying-depth AEs. Outlier score for a point x is averaged across all KL AE models as
1 K KL
I.
s(x)
Σ s((x),
where sx(x)
Σ x
((AB-20)
(x; A)112,
(5)
d=1
7

---

where Sa is the i'th AE-S member, /((AE22d) denotes the AE associated with depth 2d within La and X is a vector depieting a specific configuration all the remaining (H 22) HPs;; c.g. [irn_rate =le -3 drop_out put =0.2, etc.]. Denoting the total number such configurations by B, ROBOD averages scores, i.c. the sxx values, from B different zero craseddsed BEs as the final outlier score of x.
4.2.4 Further speed up by subsampling
As shown in Eq. BE allows mini-batching, where each data point can be used by one or several different ensemble members. This makes BE a natural fit to subsampling, which further expedites the training procedure. To this end, we create (X X kK splits of the training data, where for each AE-S member i, we divide the training data into X and X X./X;" and solely train on s(x) Kz Σ 1(x E (xx))s;(x) where K" = = 1(x E X²ut).
from X,". We then compute the out-of-sample"" outlier score of point x as
Experiments
5.1 Experimental Setup
Baselines. We compare to SOTA deep OD methods VanillaAE, RDA
43
DeepSVDD <5 and
RandNet [Л the last of which is deep AE ensemble. We also include the tree-ensemble Isolation Forest 28 (IF) which stands as the SOTA among traditional detectors 12]
Besides ROBOD without subsampling, we experiment with two subsampling versions, denoted ROBOD-6, for sampling rates 5 E (0.1. We also compare to the naïve ROBOD with independent training, denoted i-ROBOD.
Configurations. The baselines exhibit 2-8 HPs, per which we define small grid of values (See Appx. Table TTp. We report the expected AUROC performance, i.e., averaged across all configurations in the grid, along with the standard deviation. For ROBOD (and variants) we set L-6, and K 88 the other HP config.s are listed in Appx. Table I To measure sensitivity to random initialization, we average performance across 3 runs. All models are trained on a NVIDIA RTX A6000 GPUs server.
Datasets. We conduct experiments on image datasets from MNIST and CI,AR1O, as well as 3 tabular datasets from the ODDS repos-
Table 3: Dataset statistics.
itory 5 MNIST and CIFAR1O are multi-class, where we pick one class as the inliers and subsample the rest at 10% each to constitute outliers. Datasets have varying outlier and size, with images having high dimensionality. (See Table MNIST-4, -5, depict respective digits as inliers. CIFARIO-O and -1 refer to airplane and auto class as inliers, respectively.) Details on dataset description and preparation can be found in Appx. Appx.A...2 The experiments are all conducted under the Polluted (i.c., transductive) setting.
Name
# pts.
dim. outl.%
MNIST-4
6426
x
10.0
MNIST-5
6426 1x28x28
10.0
MNIST-8
6426 1x28x28
10.0
CIFARI0-0
5500 3x32x32
10.0
1IF
5500 3x32x32
10.0
Thyroid
3772
6
2.5
Cardio
1831
21
Lympho
148
5.2 Results
With main results shown in Table we want to answer the following questions: Q1. How does ROBOD compare to state-of-the-art (SOTA) OD methods? ROBOD achieves superior perfor- mance to all deep OD baselines on MNIST and tabular datasets, and competitive performance on CIFARIO datasets against the overall runner-up RandNet. Notably, ROBOD performs similarly or even better than i-ROBOD, the latter potentially owing to the guarding effect of parameter sharing against overfitting. Moreover, similar performance train each hyper-ensemble member with 50% or even 10% of the data.
be retained under subsampling, where
Q2. How much does ROBOD'S performance vary by initialization? The standard deviation (stdev) of the deep OD baselines is notably large; it is slightly smaller for the ensemble model RandNet, whereas ROBOD has significantly smaller stdev, with near-zero sensitivity to random initialization. The sensitivity deep baselines suggests that with an arbitrary choice of HPs in the absence of any other guidance, one may acquire much less satisfactory outcomes than the average AUROC.
Appliccable to transductive OD only; for inductive OD, a point is scored by all ensemble members. http://odds.c wttnybrook.eeduu
8

---

Table 4: AUROC (%) performance of OD methods. Baselines (top) avg. ed across HP config.s, ROBOD and variants (bottom) avg. ed over runs w/ random init.s.; = one stdev. Entries in green depict the method w/ least variance. Highlighted in bold and underline are the best and runner-up.
MNIST-4
MNIST:5
MNIST-8
CIFAR10-auto
Cardio
Thymid
Lympho
VanillaAE
814=9.4
73.6=10.1
83,3:44.6
60.342.0
59.244.7
87.1
81.1 =85
89.4=11.7
RDA
79.7411.2
61.9-11.4
82.6+10.2
53.+888
54.14889
78.4+12.1
80.9-5.3
78.0-12.2
DeepSVDD
81.9=4.3
75.74:3.8
85,3:199
55.613.1
59.0+3.0
54.6:7..
67.5
66.5=15.6
RandNet
3.1
79.3++3.8
85.4+222
59.1+2.6
59.5+33.
89.2445.3
81.5-5.8
92.34-8.1
IF
84.2=0088
10.1
7..5+:1.2
42.920.5
62.7.1.88
94.1:1099
97.9=0.4
99.5110.3
ROBOD
88.0-0.0
81,4+0.11
87.8+0.0
59.4+000
59,240.11
93.54F0.1
86.1-0.6
98.7++0,1
RoaOD-0.5
86.7=0.1
78.5+:0.1
88.110.11
59.610.5
58.4:10.3
92.141.33
87.9=1.3
98.74:0.1
ROBOD-0.1
86,7=0.0
78.540.1
88.2:10.1
59.31000
58.3+002
91840.3
88.4=0.5
99.0+0.1
i-RoyOD 84.5=0.0
74.610,1
87.0:10.0
62.5:10.0
61.7:0.1
87.110.1
93.0=000
98.910.0
An interesting observation is regarding the traditional IF baseline, which excels on (low-dim.) tabular datasets, and remains competitive on MNIST-4 and CIFAR10-auto It also shows relatively small variance w.r.t. its (two) HPs. However, is significantly inferior on other high-dim. image OD tasks, which may be attributed to its lack of representation learning. Nevertheless, the competitiveness of this simple baseline suggests that traditional OD methods cannot be ignored in the 'horse-race' of developing new deep OD methods, not only for their competitiveness but also for their robustness to only-a-few HPs that makes them easy to employ by practitioners.
In Fig. we show the running time (in sec.s) versus the performance for all OD methods (colored symbols) on one im- age and one tabular dataset for brevity. (See Appx. A.6 Fig. T2] for remaining datasets.) ROBOD achieves higher perfor- mance, with lower, near-zero variability compared to deep baselines. Traditional IF, based on randomized trees, is very fast, yet often underperforms on high dimen- sional tasks such as with images.
09
AUROC
AUROC
08-
98.3% saving
0.7
91.3% saving
0.6
large variance by HP
large variance by HPs
12
1P
107
10
10'
Running time (log scale)
Running time (log scale)
VerilaAd
SDA
DeesSVED
Sandler:
10000
nosco
konosos no1co-21
Figure 3: Running time (in log scale) vs. AUROC performance of OD methods (symbols) on (left) MNIST- 8 and (right) Cardio. Vertical bars depict one (1) stdev across HP config.s for the baselines.
Q3. What ning time compared to naïve hyper- ensemble w/ independent training? We study the savings that ROBOD offers compared to the naïve hyper-ensembling. As shown in Table 5 it runs 3-10x faster across datasets. With subsampling 10%, ROBOD-0.1 runs in about 1.6-10% of what it takes by i-ROBOD, Fig. g (on MNIST-5) shows that while i-ROBOD time increases by the number of models, i.c. larger and K (fixing other HPs). ROBOD takes near-constant time thanks to the batch/simultaneous training of varying-depthe and -width members.
the savings in
Table 5: Running time (sec.) of the naïve ensemble vs. ROBOD. ROBOD-0.1 that trains with 10% subsampling offers 90-98% relative savings in training time.
1omingg (sec)
1:3
ROBOD IROBOD
1030
500
MT4
MTS5
MT-
air
CIF-auto
Cardio
Thy.
Lym.
iRowOD
24192
24656
23165
38830
39562
8205
14313
1804
K: AE-S
RonOb>
4630
44%
10721
10728
3049
Figure 4: Runtime comparison of Ro- BOD to i-ROOD w/ varying L and K.
Ro
2111
Savings (%)
90.74
91.32
9138
9220
92.41
98.37
97.94
97.56
Q4. How sensitive hyper-ensembles are to their own HPs, which include (1) the selected HP ranges, and (2) the number of sub-models? The two HP terms directly related, since expanding/shrinking the HP ranges result in more/fewer sub-models constituting the ensemble. We first answer how HP value ranges affect the accuracy of hyper-ensembles. With MNIST-4, we expand, shrink or shift the HP ranges for the VanillaAE, which are consitituents of i-ROBOD. The experimental details are provided in Appx. A.7.I. As shown in Table 6 i-ROOOD achieves more stable results across all settings. Moreover, ROBOD produces lower variance with respect to its own HPs than that of the individual model results.
We also study how number of sub-models impact our model. For the same MNIST-4, DeepSVDD and VanillaAE are selected as the base model for hyper-ensembles, with details in Appx. A.7.2 Fig.

---

Table 6: Mean and stdev of AUROCs for i-RoBOD vs. VanillaAE. i-RoBOD's performance is evaluated with different HP ranges in Table T2 Mean and stdev for VanillaAE are calculated over all constituent sub-models from the same table. The hyper-ensemble i-ROBOD has considerably lower performance variation w.r.t. its HPs, than individually traind VanillaAE.
Mtan&Std. (i-ROBOD) Mean&Std. (VanillaAE)
83.0++2.8
78.9=99.8
shows the AUROC corresponding to different number of sub-models (for DeepSVDD have both Polluted (left) and Clean (middle) settings, and for VanillaAE we have Polluted (right) setting only). Observe that AUROC quickly stabilizes and the variance shrinks, the number of sub-model, grows beyond a certain size. The larger number of sub-models is, the more stable is the performance.
Finally, we conduct the sensitivity analysis to both HP value ranges and the number of sub-models (since they are correlated) for our proposed ROBOD. We evaluate on the Cardio dataset and provide the details in Appx. A.7.3 Table shows that ROBOD yields smaller variance to its own HPs and provides more competitive results than other benchmarked models. To summarize, our experiments show that larger number of sub-models and expanded HP ranges under fine grids can relax hyper-ensemble model's i dependency on its own HPs.
DeepSVDD (Polluted)
DeepSVDD (Clean)
VanillaAE (Polluted)
0.96
0.85
std
0,90
AUROC
AUROC
AUROC
0.94
0.80
0.92
0.75
0.80
Meon
Mcan
std
std
0.70
0.90
200
400
200 400
200
400
Number of Submodels Figure 5: Number of sub-models vs. AUROC of ensembled-OD methods on MNIST-4. Shaded areas
Number of Submodels
Number Submodels
depict one (1) stdev around the mean performance of the ensemble. For all cases the performance quickly improves to stable state after 20 sub-models, with notably small standard deviation.
Table 7: For Cardio dataset, provide mean and stdev AUROCs of ROBOD and benchmarks from HP ranges in Tables [5] (RoBOD) and I0 (benchmarks). The hyper-ensemble ROBOD has considerably lower performance variation w.r.t. its HPs.
ROBOD
VanillaAE
RDA
DeepSVDD
RandNet
IF
93.6=00.4
87.7+7.7
78.4
54.6+7..9
89.2+5..3 94.1 0.9
Conclusion
In this work, we have provided a thorough analysis on the hyperparameter (HP) sensitivitiy of several state-of-the-art deep outlier detection (OD) methods. Our findings quantitatively confirm that model selection is vital and advocate efforts in this line of research. To this end, introduce ROBOD, a scalable hyper-ensemble OD method that remedies the "choice paralysis" by assembling various autoencoder models of different HP configurations, hence obviating HP/model selection. We speed up ensemble training through novel strategies that simultancously train varying depth and width models under parameter sharing. Extensive experiments the competitiveness of ROBOD compared to existing OD baselines, while providing consistent results across random initializations. We hope that work increases awareness to the unsupervised model selection challenge for the newly booming deep OD literature and motivates future work on hyperparameter--ohuu model design.
image and point-cloud datasets show
Acknowledgments
This work is sponsored by NSF CAREER 1452425. We also thank PwC Risk and Regulatory Services Innovation Center at Carnegie Mellon University. Any conclusions expressed in this material are those of the author and do not necessarily reflect the views, expressed implied, of the funding parties.
10

---

References
[1] C. C. Aggarwal. Ouilier Analysis. Springer Publishing Company, Incorporated, 2nd edition, 2016. ISBN 3319475770.
[2] C. C. Aggarwal and S. Sathe. Theoretical foundations and algorithms for outlier ensembles. SIGKDD Explor, 17(1):24 47, 2015.
[3] C. C. Aggarwal and S. Sathe. Outlier ensembles: An introduction. Springer, 2017.
[4] S. Akcay, A. Atapour-barghoone and T. P. Breckon. Ganomaly: Semi-supervised anomaly detection via adversarial training. In Asian conference on computer vision, pages 622-637. Springer, 2018.
[5] L. Bergman and Y. Hoshen. Classificattion based anomaly detection for general data. In JCLR, 2020. URL 1BtvS
https: hatpps://openrevv..
[6] A. Boyd, Bamler, Mandt, and Smyth. Neural transformation learning for deep anomaly detection beyond images. In 34th Conference on Neural Information Processing Systems, 2020.
M. Breunig, H.-P. Kriegel, R. Ng, and J. Sander. LOF: Identifying density- based local outliers. In SIGMOD Conference. pages 93-104. ACM, 2000. URL http:: : //áblp uni trreer de/ conf/ si sigmod/a simdd//00mm00000mmmmmm html4Breem gKNSOO SIGMOD Record 29(2), June 2000.
G. O. Campos, A. Zimek, Sander, G. Campello, B. Micenková, E. Schubert, I. Assent, and E. Houle. On evaluation of unsupervised outlier detection; measures, datasets, and an empirical doi:
study. Data Mín. Knowl. Discov. 30(4):891 -27., 2016.
00177//100610015-4444
URL
https://doi. ooooor///// 1007 11111100111110000000101000000000
[9] R. Chalapathy and Chawla. Deep learning for anomaly detection: A survey. 2019. URL http
//arxiv..o wwwpvioorg/bb///// /1011. 03407 cite arxiv:1901.0344 1934077
[10] V. Chandola, A. Banerjee, and V. Kumar. Anomaly detection: A survey, ACM Computing Surveys, 41())
15:1- 15:588 July 2009. ISSN 0360-0300. doi: 10. (0.1145/1541880.1544 1541882. URLhttpp:/dos acm. org/
10. 10.114//154188 1541882
[11] J. Chen, S. Sathe, C. C. Aggarwal, and D. S. Turaga. Outlier detection with autoencoder ensembles. In
N. V. Chawla and W. Wang, editors, SDM, pages 90-98. SIAM, 2017. ISBN 978-1-61197-497-3. URL
http://dblp.. uni www.trerrrerrre wwtpd////db.//deeeeeee conf adm/ sdm2017 htt.1#hhhhhAA111
[12] A. Emmout, S. Das, T. Dietterich, A. Fern, and W.-K. Wong. A meta-analysis of the anomaly detection problem. arXiv preprint a5X 2015.
[13] 7. Ghafoori and C. Leckie. Deep multi-sphere support vector data description. In SDM, pages 109-117. SIAM, 2020.
[14] Z. Ghafoori, S. M. Erfani, S. Rajasegarar, J. Bezdek, S. Karunasekera, and C. Leckie. Efficient unsupervised parameter estimation for one-class support vector machines. IEEE transactions on neural networks and learning systems, 29(10):0077-5700 2018.
[15] Goix. How to evaluate the quality of unsupervised anomaly detection algorithms? CoRR. Abs/1607.01152. 2016. URL htttp://bbl..u http://llppliiiiii... trier.rrrrrrr wtwierrreerrarreeeeee
.ttml
Goix16
[16] Golan and Elniv.. Deep anomaly detection using geometric transfornations. In S. Bengio. M. Wallsch, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamettt editors, NeurlPS, pages 9781-9791, 2018. URL htt:ttp:://aae http:://blp... uni trier wwrrer.de/dlle de/db/ conf coonf/nippi wwtiiis/npps2000 htmllGol anE18
[17] Goldstein and Uchida. A comparative evaluation unsupervised anomaly detection algorithms for multivariate data, PLOS, 144), 2016.
[18] S. Goyal, A. Raghunathan, M. Jain, H. V. Vimhaddri, and P. Jain. Droce: Deep robust one-class classification. In International Conference Machine Learning, pages 3711-3721. PMLR, 2020.
[19] H. Guan, L. K. Mokadam, X. Shen, S.-H. Lim, and R. M. Patton. Fleet: Flexible efficient ensemble training for heterogeneous deep neural networks. In I. S. Dhillon, D. S. Papailiopoulos, and V. Sze. editors, MLSys. misys.org, 2020. URL http:://blpp...... uni-tries htt.d//b//coo//ll//////aa/////// html #GGanMSLPPM
[20] C. Guo, G. Pleiss, Sun, and K. Q. Weinherger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330, 2017.
[21] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778. IEBE Computer Society, 2016. ISBN 978- 178-1-4673-8851-1 URL httb:: tritr http://oon///pp///p////0//
html #He2RS16
[22] T. Hoetler, D. Alistarh, T. Ben-Nun. N. Dryden, and A. Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research. 22 (241):1-124 2021.
11

---

[23] W. Hu, M. Wang. Q. Qin, J. Ma, and B. Liu. Hrn: A holistic approach to one class learning. Advances in Neural Information Processing Systems, 33:19111-19124 2020.
[24] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In B. Leibe, Matas, N. Sebe, and M. Welling, editors, ECCV (4), volume 9908 of Lecture Notes in Computer Science, pages 646-661. Springer, 2016. ISBN 978-3-319-4649--3. URL http:://tp/// uni-trier
[25] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. von Luxburg. S. Bengio. H. M. Wallach. R. Fergus, V. N. Vishwanathan, and R. Garnett, editors, NIPS, pages 6402-6413, 2017. URL http:://b// uni uni- trierr de/ab/conf / http//ipp/////p///// nwpp2017
[26] A. Lazarevic and V. Kumar. Feature bagging for outlier detection. R. Grossman, R. J. Bayardo. and K. P. Bennett, editors, KDD, pages 157-166, ACM, 2005. ISBN 1-59593-135-X. URL http:
//dblp. uni trier. de/ hb/db/ conf kdd/kkd22005.... http::0055.01111aaaaaaa httpahaaaaaaeeeeeiiiiiie
[27] Y. Lecun, L.. Bottou, Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 11:2778-22222, 1998. doi: 10.1 109/5.76691.
[28] F. T. Liu. K. M. Ting, and Z. Zhou. Isolation forest. ICDM, pages 413-422, IEEE Computer Soci- ety. 2008. ISBN 978-0-7695-3502-49 URL /RLt:tt:://bblp..t uni-trirr de/ htt://b/cooo http://onf//iccc icdm2008 html #Liu1208
[29] H. O. Marques, R. G. B. Campello, A. Zimek, and J. Sander. On the internal evaluation of unsupervised outlier detection. In SSDBM, pages 7:1-7:12. ACM, 2015. URL http:://blp.... wwwuumiiiiii trrierrirrr. de/
conf /asdbm/ htt///asdmm/ssddn2111s 201bm2015 httml#Marqesc2211
[30] V. Nguyen, T. Nguyen, and U. Nguyen. An evaluation method for unsupervised anomaly detection algorithms. Journal of Computer Science and Cybernetics, 32(3):259 272, 2017. ISSN 96339663, doi: 10. 1011555/1813-96666 0a0///883-99668 936633/3////////5555555 URL. URI.btt:: httt:://vjs.et www./i/iiidee ht//ndexx.n wwt.deppppppi..e.eeeepeeep jec/ aataaaattileeeee wwtcceeeeeeeeee...1
[31] G. Pang, C. Shen, L. Cao, and A. V. D. Hengel. Deep learning for anomaly detection: A review. ACM Computing Surveys (CSUR), 54(2):1-38. 2021.
[32] Y. Qi, Y. Wang. X. Zheng, and Z. Wu. Robust feature learning by stacked autoencoder with maximum correntropy criterion. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 67 6716-6720, 2014. doi: 10. SWW.:IAAAASP2214..6554880
[33] S. Rayana and L. Akoglu. Less is more: Building selective anomaly ensembles. ACM Trans, Knowl, Dis- cov. Data, 10(4):42:1-44 14::3333442333, 2016. http:///:///bpppp umi trier /tt.eer/de/bb//ooraaas/////kk htt.///aaaaas/ddddddddddtoo
html #tmaaaaaa
[34] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In N. Navab, J. Hornegger, W. M. Wells, and A. Frangi. editors, Medicul Image Computing and Computer- Assisted Intervention MICCAI 2015, pages 234-241, Cham, 2015. Springer International Publishing. ISBN 478-3-319-22554444
[35] L. Ruff, N. Gömitz, L. Deccke, S. A. Siddiqui, R. A. Vandermeulen. A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In J. G. Dy and A. Krause, editors, ICML, volume 80, pages 1390-4399, PMIRR, 2018. URL Jttp
mlr. pttpss/v80//rff11a/a/aa//aa
[36] L. Ruff, J. R. Kauffimann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5): 756-795, May 2021. ISSN 1558-2256. doi: 10. 10.10000/0O.. 33052444....
[37] T. Schlegl, P. Sceböck, S. M. Waldstein, U. Schmidt- Schmidt-Erfurthh and G. Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In M. Niethammer, M. Styner, S. R. Aylward, H. Zhu, I. Oguz, P. Yap, and D. Shen, editors, IPMI, volume 10265 of Lecture Notes in Computer Science, pages 146-157. Springer. 2017. ISBN 378-3-319-59050-9 URL http://bblp uni-triee.de//en
httppmmmmmmmm11/111//// htw.hhhmmsss........ htm_:4Schoog1$$LL11
[38] T. Schlegl, P. Sechöck, S. M. Waldstein, G. Langs, and U. Schmidt- f-anogan: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Anal., 54:30-44, 2019. URL. http://ablp..
de/ wwwpa///oooooaaaoaa s/mi a a/mia54 httm1#Schleg15WWSSr
[39] B. Schölkopf. R. C. Williamson, A. J. Smola. J. Shawe- novelty detection. In NIPS, pages 582-588. 588. The MIT Press, 1999. URL http: uni twre-triercoom de/
and J. C. Platt. Support vector method for
#Scho db/ /t/.///niis///////////oo////ss// https///iipii99999 hhw.ehhsttsscooooooooooaooao kooffflsppsS
[40] D. M. Tax and R. P. Duin. Outliers and data descriptions. In Proceedings of the 7th Annual Conference of the Advanced School for Computing and Imaging, pages 234-241, 2001.
[41] S. Wang, Q Liu, E. Zhu, F. Porikli, and J. Yin. Hyperparameter selection of one-class support vector machine by self ada-tivee data shifting. Pattern Recognition, 74:198 141111 2018
12

---

[42] Y. Wen, D. Tran, and J. Ba. BatchEnsemble: an alternative approachh to efficient ensemble and life- long learning. In ICLR. OpenReview.net, 2020. URL httt:////d uni-triier.deddbcooom /icrr/
iclr2020 html #M4
[43] F. Wenzel, J. Snoek, D. Tran, and R. Jenatton. Hyperparameter ensembles quantification, arXiv preprint arXiv:2006. 13570. 2020.
ensembes for robustness and uncertainty
[44] Xie, L.. Xu, and F. Chen, Image denoising and inpainting with deep neural networks. Proceedings of the 25th International Conference on Newral Information Processing Systems- Volume J, NIPS'12 12, page 341-349 Red Hook. NY. USA, 2012. Curran Associates Inc.
[45] K. Xu, Li, Y. Tian, T. Sonobe, K.-i. Kawarabayashi, and S. Jegelka. Representation Learning on Graphs with Jumping Knowledge Networks. volume 80 of Proceedings of Machine Learning Research, pages 5453 5462, Stockholmsmåssan, Stockholm Sweden, 2018. PMLR. URL http://procceedings mlr press/v00/uul 11.800/xx1.... html
[46] H. Zenati, C. S. Foo, B. Lecouat, G. Manek, and R. Chandrasckhar. Efficient gan-based anomaly detection. arXiv preprint
2018.
[47] Y. Zhao, Rossi, and L. Akoglu. Automatic unsupervised outlier model selection. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[48] C. 7hou and R. C. Paffemroth. Anomaly detection with robust deep autoencoders. In KDD, pages 665-674 ACM. 2017. ISBN 978- 478-1-4503-4888-4 URL http://dblp.... uni-trier wwtpb//ddd.d..eeeee kdd/kdd2017
html #2hou17
[49] B. Zong, Q. Song, M. R. Min. W. Cheng, C. Lumezanu, D. ki Cho, and H. Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR. OpenReview.net, 2018. URL http://dblp.. uni- trier. dw/db/ http:////oo.ne/alll/////aaaaaaa.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Our solution to tackle hyper- parameter (HP) sensitivity for deep OD is an ensemble model. We have emphasized throughout the paper that ensemble training is costlier than training a single model. In fact, one of our key contributions is to speed up ensemble training through novel
design strategies, yet, further improvements are left to future research.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We open-source our source code (including datasets, our method as well as SOTA methods imple- mentations) at https: /7 github. , com/ myvivion/ ROBOD to which we point the Introduction section.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Yes. We include the hyperparameters used in sensitivity analysis in Table Z in Sec. Additional HP information is provided in Appx. A.2
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report empirical AUROC values across multiple hyperparameter configurations and random initializations and report the mean as well as the standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We report the computational resources in Sec. 5
13

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasngg new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We utilize standard image datasets (MNIST and CIFARIO) along with 3 labular dataset. We reference the public repository from which the labular data are collected.
(b) Did you mention the license of the assets? [No] All the datasets we experimented with are open-source with a public license.
(c) Did you include any new assets either in the supplemental material or as URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "af-ylfAQgpk", "reviewer": "Reviewer_9hEy", "summary": "The papers provide 2 main contributions:\n\n1. The paper performs a large-scale study of how different model hyperparameters (e.g., number of layers, number of training iterations, learning rate, etc.) affect the performance of several existing deep learning-based outlier detection models (vanilla autoencoder, RDA, DeepSVDD, GANomaly, and RandNet). The results show that these models are quite sensitive to the hyperparameter choices, which are often not explored in the original papers. Furthermore, ensembling the outputs of these models (across different hyperparameter runs) generally yields higher outlier-detection performance than methods for choosing a single hyperparameter.\n\n2. The paper proposes an \"ensemble\" of autoencoders of varying depths and widths called RobOD, which is used for outlier detection. I put \"ensemble\" in quotes, because RobOD is not an ensemble of independently trained models, but rather an \"ensemble\" in a similar sense to using Monte-Carlo dropout to generate a large set of models from a single trained model. Empirical results demonstrate how RobOD is competitive at outlier detection against other models, while being significantly more robust to the hyperparameter choices. (An alternative interpretation is that RobOD eliminates having to choose between depth and width hyperparameters, so there are simply fewer hyperparameters to deal with.)", "questions": "**Q1: What exactly is the \"polluted\" setting?** \\\nMy understanding is that in the \"polluted\" setting, you have a training set that includes both inliers and outliers, but you don't actually have the labels that separate the inliers from the outliers. Is my understanding correct?\n\n**Q2: How well does the RobOD style of ensembling across different model depths and widths apply to autoencoder tasks in general, beyond outlier detection?** \\\nEssentially, I'm struggling a bit to understand whether the authors think that managing hyperparameters is unique to (or particularly difficult with) outlier-detection, which I don't think is the case. I think the authors could have a much stronger paper if they demonstrate how their multi-depth / multi-width autoencoder setup improves the performance of autoencoders in general. For example, did this ensembling reduce reconstruction loss?", "limitations": "The authors briefly discuss that their method is costlier than training a single model, and they do not list any negative societal impacts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n**S1. Large-scale experiments**\\\nThrough a large set of experiments, the authors conclusively demonstrate the importance and influence of hyperparameters on current deep learning-based outlier detection models.\n\n**S2: RobOD demonstrates a method for creating an \"ensemble\"-like autoencoder model\"** \\\nBy enforcing weight-sharing and adding reconstruction losses at varying depths, RobOD only requires a single autoencoder model that can \"simulate\" multiple autoencoders of varying depths and widths.\n\n## Weaknesses\n\n**W1. Missing details on how autoencoder is used for OD**\\\nPerhaps this comes from my ignorance about OD models in general, but in the paper I didn't see any mention of how an autoencoder model is actually used for outlier detection. Suppose I have trained an autoencoder in an unsupervised manner by minimizing reconstruction loss. Then what? How is it used for outlier detection? For example, I can envision several ways of using a trained autoencoder for OD. If the autoencoder is a variational autoencoder (VAE), then I can calculate an estimated density of a given example under the training data distribution. If the autoencoder is deterministic, then I could threshold the reconstruction loss: if an example has high reconstruction loss, then I deem it an outlier. Are you using either of these methods? Or something totally different?\n\n**W2. Missing analysis of RobOD ensembling procedure**\\\nOne thing I'd like to see is whether the multi-depth / multi-width autoencoder setup improves the performance of autoencoders in general. For example, did this ensembling reduce the autoencoder's reconstruction loss?\n\n**W3. Does not separate variation in hyperparameters from variation in initialization**\\\nMaybe I missed this point somewhere in the text, but I struggled to separate the two main sources of variation in model output. First, there are many hyperparameter choices. Second, there are many random initializations. How are the authors certain that the benefit from \"ensembling\" was due to ensembling across hyperparameters, instead of ensembling across many random initializations? If you took a vanilla autoencoder with the same width and depth as the widest/deepest RobOD constituent, then used many random initializations, how does that compare against RobOD?\n\n**W4. Missing contextualization with key related works**\\\nTo me, the multi-width ensembling adopted in RobOD is quite similar to Monte-Carlo dropout, except that the dropping-out is not random but rather follows a particular deterministic pattern. If my understanding of RobOD is incorrect, please correct me! However, if my understanding is correct, then I think it's worth addressing this similarity. Likewise, the multi-depth ensembling in RobOD is quite similar to stochastic-depth models, except again the dropping-out of layers is not random but follows a particular deterministic pattern. Furthermore, the RobOD autoencoder also somewhat resembles a \"U-net,\" except that the loss is reconstruction instead of segmentation. Making these connections to existing literature would be helpful to understanding RobOD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8jiCOSgp1N6", "reviewer": "Reviewer_9Tjo", "summary": "This paper studies hyperparameter (HP) sensitivity for the outlier detection (OD) task, which is a fundamental challenge in the unsupervised setting. The authors conducted an extensive empirical study of several deep OD models' performance with various HP choices and demonstrated these SOTA OD models are very sensitive to HP selection. A deep auto-encoder ensemble model is proposed to reduce the dependency on HP and achieve better OD performance with the help of the ensemble. In order to prevent long running time for training each individual model in the ensemble, model parameters are shared among multiple models through three techniques: 1) skip link for depth ensembling; 2) batch ensemble for width ensembling; and 3) sub-sampling. Several experiments have been performed to demonstrate the effectiveness of the proposed ensemble method and the efficiency of the model training.", "questions": "1. The reviewer wonders if the authors can provide the analysis of the HP robustness of the ensemble methods. For example, try multiple different settings in Table 9, resulting in various numbers of sub-models, to demonstrate the HP robustness of the ensemble methods.\n1. Table 2 gives the **Hyper-ens. Mean** and **Mean**. Although the performance seems improved here, the stddev for the **Hyper-ens. Mean** is generally higher than its counterpart. Does this mean that the ensemble model here is also sensitive to HP?\n1. Although the parameter sharing over multiple deep AEs helps with the training speed, the reviewer guesses that such a strategy will degrade the outlier performance for each sub-model (the losses for sub-models are competing with each other) resulting in overall lower performance for the entire ensemble, compared to training each model individually. However, in Table 4, it seems ROBOD outperforms i-ROBOD in multiple tasks. Any reasoning would help the reviewer understand how parameter sharing helps with OD accuracy performance.\n1. It seems IF presents good results over vector-based data, and even for CIFAR10-auto. Any comparison/analysis between deep models vs. traditional OD methods would help readers understand the difference.\n1. For image task, does ROBOD use CNN architecture, or FC?", "limitations": "The main limitation of this paper is mentioned as the second weakness previously. If the major issue (HP robustness of the ensemble ) is not answered, the use of the ensemble method cannot be justified, which also weakens the training speed-up claim for the ensemble method.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths\n  1. This paper raises an essential question for the outlier detection task. Under an unsupervised setting, the model performance is agnostic before actual prediction/testing due to the lack of ground-truth label information, resulting in no clear HP selection strategy, compared to the supervised counterparts (validation).\n  1. The reviewer appreciates the time and efforts put into the experimental study of the existing OD models in Section 3. This study clearly points out that most deep OD models are sensitive to HP selection, and the final performance of models may suffer from potentially-large variation.\n  1. The parameter-sharing AE model and the three related ensemble techniques proposed in this paper may help reduce the training time for large deep ensembles.\n* Weakness\n  1. Although rarely used in deep OD models (majorly due to long training time), the ensemble ideas are widely used in OD tasks. For example, the Isolation Forest (IF) is an ensemble tree method. The ensemble ideas inside this paper simply aggregate the outlier scores from models inside the ensemble, which is the common practice.\n  1. The paper doesn't actually answer **RQ1) how to design an unsupervised deep OD model that is robust to its HPs**. Note that **RQ1** is different from **Q1** and **Q2** given in Section 5. Although the ensemble method reduces the HP dependency for each individual sub-model inside the ensemble, the HP for the entire ensemble is not studied. For each sub-model, the HP may be model depth, layer width, learning rate, etc. The HP for the entire ensemble may be the number of sub-models, sub-model HP selection range, etc. The first paragraph in Section 4.2.3 gives an example of the sub-model HP selection range. The current ensemble presented in the paper (Table 9 in the appendix) can only be considered as a single HP setting for the entire ensemble. It is unclear how the method performs if different values are given for Table 9. Therefore, we cannot tell if the proposed ensemble is robust to its HPs or not (RQ1). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1tQeVZCOQGk", "reviewer": "Reviewer_FAet", "summary": "This paper studies the influence of hyperparameter for deep OD, as well as the strategy that alleviates the hyperparameter tuning issue during the training process. First, the authors soundly identifies the problem of hyperparameter selection in deep OD by extensive experiments with various AE-based deep OD methods and benchmarks. Then, with a general idea to avoid hyperparameter tuning via hyper-ensemble, the authors propose to leverage skip connection and batch ensemble technique to speed up the hyper-ensemble calculation. The results justify the effectiveness of the proposed strategy.", "questions": "1. The range of hyperparameters seems to be small in many experiments. Many hyperparameters only have two or three candidate values, which is not the case in practice. What will happen if the hyperparameter is selected from a larger range?\n\n2. Is it possible to extend the strategy of ROBOD to other transformation based deep OD methods like [1]?\n\n3. It will be better the conduct more extensive experiments on dataset like MNIST, not just the cases where 4/5/8 are used as inliers in Table 4. I am not sure why the authors pick those three cases to conduct the experiments. Can authors explain the reason?\n\nRef:\n\n[1] Deep anomaly detection using geometric transformations. NeurIPS 2018.", "limitations": "Despite that this is a good work in general, authors should add more discussion on the limit of their work. No negative social impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This is a well-motivated paper that tries to address a real problem that is often neglected. Hyperparameter selection is an important issue for almost all machine learning tasks, but it is especially challenging for unsupervised tasks like deep OD, as the outliers are often not accessible for validation. I appreciate the authors' effort to bring this topic up. Besides, the authors provides an extensive review on existing deep OD methods and a thorough analysis on them, which makes the discussion more convincing.\n\n2. The presentation of the paper is good. I do enjoy reading this paper, as it follows a very clear pipeline: identifying a valuable problem-giving a possible solution-tailoring the solution for improvement, which is a good example that exhibits a complete process of research. Besides, all concepts are clarified and used consistently across the entire paper. \n\n3. The proposed ROBOD seems to be simple and easy-to-implement, but enjoys both good effectiveness and efficiency against other AE-based baselines.\n\nWeaknesses:\n\n1.  The largest limit of this work is that the discussion seems to be limited to AE-based deep OD methods. I do understand that AE plays a center role in deep OD, but its performance is also shown by numerous works to be less satisfactory, especially on relatively complex datasets. By contrast, recent self-supervised deep OD methods (e.g. transformation based methods, CSI) achieve significantly better performance in many cases, and the authors did not discuss them as examples, which is a clear weakness to me.\n\n2. The extendibility of the proposed ROBOD is a problem. For me, it is not straightforward to extend the skip connection based strategy to non-AE based deep OD methods that do not use a symmetric DNN architecture. In the meantime, ROBOD only considers the acceleration of two hyperparameters (though they are two important ones). For now, it is hard to call ROBOD a generally applicable framework for deep OD. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept