Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies hyperparameter (HP) sensitivity for the outlier detection (OD) task, which is a fundamental challenge in the unsupervised setting. The authors conducted an extensive empirical study of several deep OD models' performance with various HP choices and demonstrated these SOTA OD models are very sensitive to HP selection. A deep auto-encoder ensemble model is proposed to reduce the dependency on HP and achieve better OD performance with the help of the ensemble. In order to prevent long running time for training each individual model in the ensemble, model parameters are shared among multiple models through three techniques: 1) skip link for depth ensembling; 2) batch ensemble for width ensembling; and 3) sub-sampling. Several experiments have been performed to demonstrate the effectiveness of the proposed ensemble method and the efficiency of the model training.
strengths_and_weaknesses: * Strengths
  1. This paper raises an essential question for the outlier detection task. Under an unsupervised setting, the model performance is agnostic before actual prediction/testing due to the lack of ground-truth label information, resulting in no clear HP selection strategy, compared to the supervised counterparts (validation).
  1. The reviewer appreciates the time and efforts put into the experimental study of the existing OD models in Section 3. This study clearly points out that most deep OD models are sensitive to HP selection, and the final performance of models may suffer from potentially-large variation.
  1. The parameter-sharing AE model and the three related ensemble techniques proposed in this paper may help reduce the training time for large deep ensembles.
* Weakness
  1. Although rarely used in deep OD models (majorly due to long training time), the ensemble ideas are widely used in OD tasks. For example, the Isolation Forest (IF) is an ensemble tree method. The ensemble ideas inside this paper simply aggregate the outlier scores from models inside the ensemble, which is the common practice.
  1. The paper doesn't actually answer **RQ1) how to design an unsupervised deep OD model that is robust to its HPs**. Note that **RQ1** is different from **Q1** and **Q2** given in Section 5. Although the ensemble method reduces the HP dependency for each individual sub-model inside the ensemble, the HP for the entire ensemble is not studied. For each sub-model, the HP may be model depth, layer width, learning rate, etc. The HP for the entire ensemble may be the number of sub-models, sub-model HP selection range, etc. The first paragraph in Section 4.2.3 gives an example of the sub-model HP selection range. The current ensemble presented in the paper (Table 9 in the appendix) can only be considered as a single HP setting for the entire ensemble. It is unclear how the method performs if different values are given for Table 9. Therefore, we cannot tell if the proposed ensemble is robust to its HPs or not (RQ1). 
questions: 1. The reviewer wonders if the authors can provide the analysis of the HP robustness of the ensemble methods. For example, try multiple different settings in Table 9, resulting in various numbers of sub-models, to demonstrate the HP robustness of the ensemble methods.
1. Table 2 gives the **Hyper-ens. Mean** and **Mean**. Although the performance seems improved here, the stddev for the **Hyper-ens. Mean** is generally higher than its counterpart. Does this mean that the ensemble model here is also sensitive to HP?
1. Although the parameter sharing over multiple deep AEs helps with the training speed, the reviewer guesses that such a strategy will degrade the outlier performance for each sub-model (the losses for sub-models are competing with each other) resulting in overall lower performance for the entire ensemble, compared to training each model individually. However, in Table 4, it seems ROBOD outperforms i-ROBOD in multiple tasks. Any reasoning would help the reviewer understand how parameter sharing helps with OD accuracy performance.
1. It seems IF presents good results over vector-based data, and even for CIFAR10-auto. Any comparison/analysis between deep models vs. traditional OD methods would help readers understand the difference.
1. For image task, does ROBOD use CNN architecture, or FC?
limitations: The main limitation of this paper is mentioned as the second weakness previously. If the major issue (HP robustness of the ensemble ) is not answered, the use of the ensemble method cannot be justified, which also weakens the training speed-up claim for the ensemble method.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The papers provide 2 main contributions:

1. The paper performs a large-scale study of how different model hyperparameters (e.g., number of layers, number of training iterations, learning rate, etc.) affect the performance of several existing deep learning-based outlier detection models (vanilla autoencoder, RDA, DeepSVDD, GANomaly, and RandNet). The results show that these models are quite sensitive to the hyperparameter choices, which are often not explored in the original papers. Furthermore, ensembling the outputs of these models (across different hyperparameter runs) generally yields higher outlier-detection performance than methods for choosing a single hyperparameter.

2. The paper proposes an "ensemble" of autoencoders of varying depths and widths called RobOD, which is used for outlier detection. I put "ensemble" in quotes, because RobOD is not an ensemble of independently trained models, but rather an "ensemble" in a similar sense to using Monte-Carlo dropout to generate a large set of models from a single trained model. Empirical results demonstrate how RobOD is competitive at outlier detection against other models, while being significantly more robust to the hyperparameter choices. (An alternative interpretation is that RobOD eliminates having to choose between depth and width hyperparameters, so there are simply fewer hyperparameters to deal with.)
strengths_and_weaknesses: ## Strengths

**S1. Large-scale experiments**\
Through a large set of experiments, the authors conclusively demonstrate the importance and influence of hyperparameters on current deep learning-based outlier detection models.

**S2: RobOD demonstrates a method for creating an "ensemble"-like autoencoder model"** \
By enforcing weight-sharing and adding reconstruction losses at varying depths, RobOD only requires a single autoencoder model that can "simulate" multiple autoencoders of varying depths and widths.

## Weaknesses

**W1. Missing details on how autoencoder is used for OD**\
Perhaps this comes from my ignorance about OD models in general, but in the paper I didn't see any mention of how an autoencoder model is actually used for outlier detection. Suppose I have trained an autoencoder in an unsupervised manner by minimizing reconstruction loss. Then what? How is it used for outlier detection? For example, I can envision several ways of using a trained autoencoder for OD. If the autoencoder is a variational autoencoder (VAE), then I can calculate an estimated density of a given example under the training data distribution. If the autoencoder is deterministic, then I could threshold the reconstruction loss: if an example has high reconstruction loss, then I deem it an outlier. Are you using either of these methods? Or something totally different?

**W2. Missing analysis of RobOD ensembling procedure**\
One thing I'd like to see is whether the multi-depth / multi-width autoencoder setup improves the performance of autoencoders in general. For example, did this ensembling reduce the autoencoder's reconstruction loss?

**W3. Does not separate variation in hyperparameters from variation in initialization**\
Maybe I missed this point somewhere in the text, but I struggled to separate the two main sources of variation in model output. First, there are many hyperparameter choices. Second, there are many random initializations. How are the authors certain that the benefit from "ensembling" was due to ensembling across hyperparameters, instead of ensembling across many random initializations? If you took a vanilla autoencoder with the same width and depth as the widest/deepest RobOD constituent, then used many random initializations, how does that compare against RobOD?

**W4. Missing contextualization with key related works**\
To me, the multi-width ensembling adopted in RobOD is quite similar to Monte-Carlo dropout, except that the dropping-out is not random but rather follows a particular deterministic pattern. If my understanding of RobOD is incorrect, please correct me! However, if my understanding is correct, then I think it's worth addressing this similarity. Likewise, the multi-depth ensembling in RobOD is quite similar to stochastic-depth models, except again the dropping-out of layers is not random but follows a particular deterministic pattern. Furthermore, the RobOD autoencoder also somewhat resembles a "U-net," except that the loss is reconstruction instead of segmentation. Making these connections to existing literature would be helpful to understanding RobOD.
questions: **Q1: What exactly is the "polluted" setting?** \
My understanding is that in the "polluted" setting, you have a training set that includes both inliers and outliers, but you don't actually have the labels that separate the inliers from the outliers. Is my understanding correct?

**Q2: How well does the RobOD style of ensembling across different model depths and widths apply to autoencoder tasks in general, beyond outlier detection?** \
Essentially, I'm struggling a bit to understand whether the authors think that managing hyperparameters is unique to (or particularly difficult with) outlier-detection, which I don't think is the case. I think the authors could have a much stronger paper if they demonstrate how their multi-depth / multi-width autoencoder setup improves the performance of autoencoders in general. For example, did this ensembling reduce reconstruction loss?
limitations: The authors briefly discuss that their method is costlier than training a single model, and they do not list any negative societal impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the influence of hyperparameter for deep OD, as well as the strategy that alleviates the hyperparameter tuning issue during the training process. First, the authors soundly identifies the problem of hyperparameter selection in deep OD by extensive experiments with various AE-based deep OD methods and benchmarks. Then, with a general idea to avoid hyperparameter tuning via hyper-ensemble, the authors propose to leverage skip connection and batch ensemble technique to speed up the hyper-ensemble calculation. The results justify the effectiveness of the proposed strategy.
strengths_and_weaknesses: Strengths:

1. This is a well-motivated paper that tries to address a real problem that is often neglected. Hyperparameter selection is an important issue for almost all machine learning tasks, but it is especially challenging for unsupervised tasks like deep OD, as the outliers are often not accessible for validation. I appreciate the authors' effort to bring this topic up. Besides, the authors provides an extensive review on existing deep OD methods and a thorough analysis on them, which makes the discussion more convincing.

2. The presentation of the paper is good. I do enjoy reading this paper, as it follows a very clear pipeline: identifying a valuable problem-giving a possible solution-tailoring the solution for improvement, which is a good example that exhibits a complete process of research. Besides, all concepts are clarified and used consistently across the entire paper. 

3. The proposed ROBOD seems to be simple and easy-to-implement, but enjoys both good effectiveness and efficiency against other AE-based baselines.

Weaknesses:

1.  The largest limit of this work is that the discussion seems to be limited to AE-based deep OD methods. I do understand that AE plays a center role in deep OD, but its performance is also shown by numerous works to be less satisfactory, especially on relatively complex datasets. By contrast, recent self-supervised deep OD methods (e.g. transformation based methods, CSI) achieve significantly better performance in many cases, and the authors did not discuss them as examples, which is a clear weakness to me.

2. The extendibility of the proposed ROBOD is a problem. For me, it is not straightforward to extend the skip connection based strategy to non-AE based deep OD methods that do not use a symmetric DNN architecture. In the meantime, ROBOD only considers the acceleration of two hyperparameters (though they are two important ones). For now, it is hard to call ROBOD a generally applicable framework for deep OD. 
questions: 1. The range of hyperparameters seems to be small in many experiments. Many hyperparameters only have two or three candidate values, which is not the case in practice. What will happen if the hyperparameter is selected from a larger range?

2. Is it possible to extend the strategy of ROBOD to other transformation based deep OD methods like [1]?

3. It will be better the conduct more extensive experiments on dataset like MNIST, not just the cases where 4/5/8 are used as inliers in Table 4. I am not sure why the authors pick those three cases to conduct the experiments. Can authors explain the reason?

Ref:

[1] Deep anomaly detection using geometric transformations. NeurIPS 2018.
limitations: Despite that this is a good work in general, authors should add more discussion on the limit of their work. No negative social impact.
review_title: null

Meta-review: metareview: This paper empirically demonstrates the sensitivity of unsupervised OD methods to hyperparameters and proposes ensembles of models with differing hyperparameters along with training techniques based on weight sharing to do so efficiently. 

The authors provided additional experiments to answer some reviewer's major concerns regarding the (meta-)HP robustness of the ensemble methods. While there are natural fluctuations with respect to the (meta) hyperparameters, a larger number of hyperparameters included usually resulted in a close to optimal AUROC.

Another concern for two reviewers was the extendability of the efficient ensemble training techniques to other models such as GANs etc. As the authors replied, even though skip connections might not be adaptable to other techniques, e.g. for ensembling various widths, zero-masked layers with BatchEnsemble would also be broadly applicable to ensemble-learn other unsupervised representation learning models. Perhaps in the final version, the authors further discuss with a short experiment how the AE-specific scaling techniques add to the performance.

A further concern was the lack of theoretical underpinning about the (meta)HP-robustness. Given that the reviewers agree that this paper provides ample empirical evidence and praise the experimental value, we think theoretical work (providing HP sensitivity results is highly nontrivial in general, let alone for neural networks) can be part of future work but the lack of it in the current manuscript should not prevent the publication of this work.
confidence: Certain
Decision: Accept