Unsupervised Anomaly Detection by Robust Collaborative Autoencoders
Authors: Boyang Liu
Ding Wang
Kaixiang Lin
Pang-Ning Tan
Jiayu Zhou
Venue:  (2021)
Keywords: Anomaly Detection
Robustness

Abstract: Unsupervised anomaly detection plays a crucial role in many critical applications. Driven by the success of deep learning, recent years have witnessed growing interests in applying deep neural networks (DNNs) to anomaly detection problems. A common approach is to use autoencoders to learn a feature representation for the normal (non-anomalous) observations in the data. The reconstruction error of the autoencoder is then used as outlier scores to detect anomalies. However, due to the high complexity brought upon by over-parameterization of DNNs, the reconstruction error of the anomalies could also be small, which hampers the effectiveness of these methods. To alleviate this problem, we propose a robust framework using collaborative autoencoders to jointly identify normal observations from the data while learning its feature representation. We investigate the theoretical properties of the framework and empirically show its outstanding performance as compared to other DNN-based methods. Our experimental results also show the resiliency of the framework to missing values compared to other baseline methods.

Under review as a conference paper at ICLR 2021
UNSUPERVISED ANOMALY DETECTION BY ROBUST COLLABORATIVE AUTOENCODERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Unsupervised anomaly detection plays critical role in many real-world applica- tions, from computer security healtheare. A common approach based on deep learning is to apply autoencoders to learn a feature representation of the normal (non-anomalous) observations and use the reconstruction error of each observa- tion to detect anomalies present in the data. However, due to the high complex- ity brought upon by perr-parameterization of the deep neural networks (DNNs), the anomalies themselves may have small reconstruction errors, which degrades the performance of these methods. To address this problem, we present a robust framework for detecting anomalies using collaborative autoencoders. Unlike pre- vious methods, our framework does not require supervised label information nor access clean (uncorrupted) examples during training. We investigate the the- oretical properties of our framework and perform extensive experiments to com- pare its performance against other DNN-based methods. Our experimental results show the superior performance of the proposed framework as well as its robust- ness noise due to missing value imputation compared to the baseline methods.
INTRODUCTION
Anomaly detection (AD) is the task of identifying abnormal observations in the data. It has been successfully applied to many applications, from malware detection to medical diagnosis (Chandola cet al. 2009). Driven by the success of deep learning, AD methods based on deep neural networks
(DNNs) (Zhou & Paffenroth| 2017 Aggarwal & Sathe 2017|; Ruff et al. 2018 Zong et al. 2018 Hendrycks et al. (018) have attracted increasing attention recently. Unfortunately. DNN meth- ods have several known drawbacks when applied to AD problems. First, since many of them are al., 2018). this requires labeled exam- ples of anomalies, which are often expensive to acquire and may not be representative enough in non-stationary environments. Supervised AD methods are also susceptible to the class imbalance problem as anomalies rare compared to normal observations. Some DNN methods rely having access to clean data to ensure that the feature representation learning is not contaminated by anoma- lies during training (Zong al. 2018 Ruff al. 2018 Pidhorskyi et al., 2018: Fan et al.) 2020). This limits their applicability as acquiring a representative clean data itself is a tricky problem. Due these limitations, there have been concerted efforts develop robust unsupervised DNN meth- ods that do not assume the availability of supervised labels nor clean training data (Chandola et al.) 2009 Liu et al.,220 2019). Deep autoencoders are perhaps one of the most widely used unsupervised AD methods (Sakurada & Yair 2014 Vincent et al.) 2010). An autoencoder compresses the origi- nal data by learning latent representation that minimizes the reconstruction loss. is based on the working assumption that normal observations are easier to compress than anomalies. Unfortunately, such an assumption may not hold in practice since DNNs are often over-parameterized and have the capability to overfit the anomalies (Zhang et al. 2016). thus degrading their overall performance.
based on the supervised learning approach (Hendrycks
To improve their performance, the unsupervised DNN methods must consider the trade-off between model capacity and overfitting to the anomalies. One way to control the model capacity is through regularization. Many regularization methods for deep networks have been developed to control model capacity, e.g., by constraining the norms of the model parameters explicitly perturbing the training process (Srivastava et al. 20144. However, these approaches do not prevent the networks from being able to perfectly fit random data (Zhang et al. 2016). As consequence, the regulariza-

---

Under review as a conference paper at ICLR 2021
tion approaches cannot prevent the anomalies from being memorized, especially in an unsupervised learning setting.
Our work is motivated by recent advances in supervised learning on the robustness of DNNs for noisy labeled data by learning the weights of the training examples (Jiang et al., 2017 Han et al, 2018). Unlike previous studies, our goal is to learn the weights in an unsupervised learning fash- lon so that normal observations are assigned higher weights than the anomalies when calculating reconstruction error. The weights help to reduce the influence of anomalies when learning a feature representation of the data. Since existing approaches for weight learning are supervised, they are inapplicable to unsupervised AD. Instead, we propose an unsupervised robust collaborative autoen- coders (RCA) method that trains a pair of autoencoders in a collaborative fashion and jointly learns their model parameters and sample weights. Each autoencoder selects a subset of samples with lowest reconstruction errors from mini-batch to learn their feature representation. By discarding samples with high reconstruction errors, the algorithm is biased towards learning the representation for clean data, thereby reducing its risk of memorizing anomalies. However, by selecting only easy- to-fit samples in each iteration, this may lead to premature convergence of the algorithm without sufficient exploration of the loss surface. Thus, instead of selecting the samples to update its own model parameters, each autoencoder will shuffle its selected samples to the other autoencoder, who will use the samples update their model parameters. The sample selection procedure is illustrated in Figure [ During the testing phase, we apply the dropout mechanism used in training to produce multiple output predictions for each test point by repeating the forward pass multiple times. These ensemble of outputs are then aggregated to obtain a more robust estimate of the anomaly score.
The main contributions of this paper are as follows. First, we present a novel framework for unsuper- vised AD using robust collaborative autoencoders (RCA). Second, we provide rigorous theoretical analysis to understand the mechanism behind RCA. We also describe the convergence of RCA the solution obtained if it was trained on clean data only. We show that the worst-case scenario for RCA better than conventional autoencoders and analyze the conditions under which RCA guaranteed to find the anomalies. Finally, we empirically demonstrate that RCA outperforms state-of-the-ar unsupervised AD methods for the majority of the datasets used in this study, even in the presence of noise due to missing value imputation.
2 RELATED WORK
There are numerous methods developed for anomaly detection, survey of which can be found in [Chandola et al. 2009). Reconstruction-base methods, such as principal component analysis popular approaches, whereby the input data is projected to lower-
(PCA) and autoencoders, dimensional space before it was transformed back to its original feature space. The distance between the input and reconstructed data is used determine the anomaly scores of the data points. More advanced unsupervised AD methods have been developed recently. (Zoo & Paffenroth (2017) com- bined robust PCA with an autoencoder to decompose the data into mixture of normal and anomaly parts. Zong et al.) (2018) jointly learned a low dimensional embedding and density of the data, us- ing the density of each point as its anomaly score while Ruff et al. (2018) extended the traditional one-class SVM approach to a deep learning setting. Wang et al. (2019) applied an end-to-end self- supervised learning approach to the unsupervised AD problem. However, their approach is designed for image data, requiring operations such as rotation and patch reshuffling.
Despite the recent progress on deep unsupervised AD, current methods do not explicitly prevent the neural network from incorporating anomalies into their learned representation, thereby degrading the model performance. One way to address the issue is by assigning a weight to each data point, giving higher weights to the normal data to make the model more robust against anomalies. The idea of learning a weight for each data point is not new in supervised learning. A classic example is boosting assigned higher weights encourage the
(Freund et al. 1996), where hard to classify examples model to classify them more accurately. An opposite strategy is used in self-paced learning (Kumar al. 2010). where the algorithm assigns higher weights to masier-to-classify examples and lower weights to harder ones. This strategy was also used by other methods for learning from noisy labeled data, including Jann et al.] (2017) and Han al. (2018). Furthermore, there are many studies providing theoretical analysis on the benefits of choosing samples with smaller loss to drive the optimization algorithm (Shen & Sanghavi 2018 Shah et al. 2020).
2

---

Under review as a conference paper at ICLR 2021
samppl sntaccan
subsample
A1
A1
x1
Aampling
Data
shufting
see pemplh
subsample
A2
x2
A2
Figure I: An illustration of the training phase for the proposed RCA framework. METHODOLOGY
This section introduces the proposed robust collaborative autoencoder (RCA) framework and ana- lyze its properties. Let X E Rnxd denote the input data, where n is the number of observations and d is the number of features. Our goal is to classify each data point z; X as anomaly normal observation. Let O C X denote the set of true anomalies in the data. We assume the anomaly ratio, |0/n, is giver or can be approximately estimated.
The RCA framework trains a pair of autoencoders, Ai and A2, with different initializations. In each iteration during training, the autoencoders will each apply forward pass on a mini-batch randomly sampled from the training data and compute the reconstruction eπror of each data point in the mini- batch. The data points in the mini-batch are then sorted according to their reconstruction errors and each mini-batch selects the points with lowest errors to be exchanged with the other autoencoder. Each autoencoder subsequently performs a back-propage paek-propagttion step to update its model parameters using the samples it receives from the other autoencoder. Upon convergence, the averaged recon- struction error of each data point used to determine the anomaly score. A pseudocode of the training phase for RCA is given in Algorithm while the testing phase is given in Algorithm 2
Algorithm 1: Robust Collaborative Autoencoders (Training Phase)
input: training data Xera. test data X.,t, reconstruction loss function anomaly e, dropout epochs: maz epach:
training
return mained antonnodders, Ai and A
Initialize sutoencoders Ai and Ag; sample selection rate = and best loss 5"' +
while spock S. max epock do for minibatch S X.r.:
S, (orward( S, drapout = 0), $, forwardi, S, dropout O);
sample Senppeeselection((((S, S), ez sample [($2, 8);
+ Aorward(A1, S(cz), drapout = r). 82 + A2,ward(,A,, S(c,) dropout r); end
A₁ backpropts S(ea], dropout = r), Au + backprop(S,. S Sj,,), dropout = r):
Xeve1 +- [Ar,ard(A1, X.st, dropout 0), 8 (A2, Xeve, dropont m 0):
Eicut 2(X X.st) + L(X Xta:;;
if Eeene < E" then E" = Eenas A = As. Az = A2 end:
B max(ß-
max cpo.c
end
Algorithm 2: Robust Collaborative Autoencoders (Testing Phase)
input:
test data Xtats trained autoencoders A dropout rate r 0, size ensemble v;
return anomafy scone:
Initialize an empty set of reconstraction errors: & = (). for
f forward( Ai X dropout r). [f=oowandd(A), Xtre, dropout = r):
{ E (5: E2)/2
end
anomaly, w.w.illyy.oorre. averuge(()):
RCA differs from conventional autoencoders in several ways. First, its autoencoders are trained using only selected data points with small reconstruction errors. The selected points are then ex- changed between the autoencoders to avoid premature convergence. Furthermore, in the testing phase, each autoencoder applies dropout mechanism to generate multiple predicted outputs. The averaged ensemble output is used as the final anomaly score. Details of these steps are given next.
In practice, users would typically specify the top-k anomalies to be examined and verified, where & ne.
3

---

Under review as a conference paper at ICLR 2021
3.1 SAMPLE SELECTION
Given a mini-batch, Xm C C, our sample selection procedure chooses a subset of points as "clean" samples to update the parameters of an autoencoder by minimizing the following objective function:
mill,.e Σx EX. cx((x; w), s.t. Vi:: G {0, 13, cf ,r,
where fx, w) denotes the reconstruction loss of a data point x for an autoencoder with parameter w. Although o; is binary-valued, as will be shown later, the probability that data point is selected to update an autoencoder depends on the probability it is chosen to be part of the mini-batch and the probability it has among the lowest reconstruction errors within the mini-batch.
We use an alternating minimization approach to solve the objective function. By fixing c and op- timizing w, this reduces to solving the standard autoencoder problem using the selected "clean" samples (i.e., those with = 1) by applying an optimizer such as Adam (Kingma & Ba 2014). When w is fixed, the objective function reduces to a linear programming problem, which admits simple solution, where the data points are sorted based on their reconstruction loss, fx,, w). We set e 1 to (5 100)% of the points in the mini-batch with lowest reconstruction loss. This procedure is denoted by the to 3 when selecting the "clean" samples. In the early stages training, all the samples the mini- batch are selected to update the model, with gradually fewer points selected as training progresses. The rationale for this approach is that we should not drop too many data points early, especially in the first few epochs, when the autoencoders have not properly learn the feature representation. The autoencoders start to overfit the anomalies when the number of training epochs is sufficiently large. A linear decay function from / 1 to 3 1 c (see the second last line in Algorithm I was found to work well in practice, so we use this setting in our experiments.
function in Algorithm [ A weight decay approach is applied
Next, we analyze the convergence properties of our sample selection procedure. Let & be the mini- batch size and w be the current parameter of the autoencoder. Our algorithm selects (3 x 100)% of the data points with lowest errors in the mini-batch for updating the autoencoder. Let x() X denote the data point with 7th smallest reconstruction loss among all n points and p:(w) be the probability that X(a) is chosen by the sample selection procedure to update the parameters of the autoencoder. Assuming sampling without replacement, we consider two cases: S 3kk and i > Sk. In the first case (when i < Bk), X(a) is used to update the autoencoder as long as it is selected to be part of the mini-batch In the second case (when i > Bk), X(t) is chosen only if it is part of the and has among the (Bk)-th lowest errors among all data points in the mini-batch. Thus:
if S Sk,
(.(w)= =
(2)
otherwise.
The corresponding probability pi(w) for sampling with replacement is also provided in the Ap- pendix section. The objective function for our sample selection procedure (Equation [ can be stated as F(w)- = Ex Ex())xP((W)) f w). Let ((w"r) be the set of stationary points of F(w.. Furthermore, let 12(w*) be the set of stationary points for the loss on clean data only, i.e., F(w)- Ex fi(w), while ?(wwta) be the set of stationary points for the loss on the entire data, i... Σ Exexx f.w.. For brevity, we have used f((w) denote f(x,, w). Furthermore, we denote (,(w") as the set of stationary points for the individual loss, f((w). Our analysis on the convergence properties of our sample selection approach is based on the following assumptions:
Assumption (Gradient Regularity) max,,w [V S G.
Assumption (Bounded Clean Objective) Let F(w) = Exieo Ji(w), There exists constant B > 0 such that the following inequality holds: maxs,; [(F(w:) F(w;))l S B.
Assumption 3 (Individual L-smooth) For every individual loss fe(w), the following inequality holds: Vi.j [[V - f((w))) S L.ll::ww
Assumption (Equal Minima) The minimum value of every individual loss is the same, i.e. Vi.j minw fc(w)= min [w).
4

---

Under review as a conference paper at ICLR 2021
Assumption 5 (Individual Strong Convexity) For every individual loss fi(w), the following in- equality holds: Vi,j : VVV[(( VVt((w)) psl W; -Wyl-
To simplify the notation, we define Tmmax = max. Timin = min;(T.), Mmax = max;( and Mmin (n4) Since F(w) the sum over the loss for clean data, is easy to see that Assumption 3 implies F(w) is n11 - E) Lmax smoothness, while Assumption iimplies that F F(w) n11 C)/min convex. We thus define M n11 c) and m n11
5
Mminn
Note that Assumptions 13 are common for non-convex optimization. Assumption 4 reasonable in an over-vvrammae over-parameterized DNN setting (Zhang et al. al.] 2016). Although Assumption 5lis the strongest as- sumption, it is only used to discuss correctness of our algorithm in Theorem [
but not for Theorems A similar assumption has been used in [hhah et al, (2020) in their proof of correctness.
and 2
We define the constants 8 > O and Φ 2 as follows:
Vx; $ O, Vxj EO:: max (w))))((()))))))) |/v y|l 5 & min (w*) (w)(()))))))))))) ||z y/1,
Vx, E
(w") | -yll S 500
(3)
If the loss is convex, then f2(w*) (wi} and (2(w") = {w"). The above equation can be Φô, Vx, & O, Vx, E EO. The constants and ø
simplified to: |w - w'll 8 S [w; - w thus provide bounds on the distance between w) of an anomaly point and w" for clean data.
S -w
We first consider a non-convex setting. Our goal is to determine whether the parameters learned from the samples chosen by our procedure, which optimizes F(w), converges to w", the solution obtained by minimizing the loss for clean data, F F(w) Ex.eo fi(w).
Theorem Let F(w) = Ex.eo f((w) be a twice-differentiabl function.
Consider the sequence { (w(t)} generated by optimizing F(w) (x)((W)((()) w). iee, w(++1) Let max [[VF(w(()) - E, P. (w) = C.
w(t) 11()) E,
Based on Assumptions min [[F(((())
3 if 17(0) satisfies Ei=1 (1 co and Ei (10 S 00, then
Cwhen! T 00.
The preceding theorem shows the convergence property of optimizing F(w) to a C- approximated stationary point of the loss function for clean data, where depends on the sample selection approach. For example, if p(() = 1, vx, & O and pi(w) = 0, vx, E O, then C = 0. In this ideal situation, the solution for optimizing I't (w) reduces to that for vanilla SGD clean data.
Remark
Next, we compare the stationary point of F(w) against the stationary point of the loss function for the entire data (without sample selection).
Theorem Let F(w) = Ex.4o fi(w) be a twice-differentiable function and C is defined The- orem 1 Consider the sequence {wer) generated by optimizing F'w) = E. (f w), and the sequence {Wne] generated by stan- dard SGD on the entire data, w(t+1) = (!)) - (xx:). Based on Assumptions
i.e. w(++) wit) rf( (.)(((
7 and
C' < (min(neG, 2, (() satisfies Eis () large enough T and w E (((",,) such that min;=0.1....1 [VF )1 [IVF(())
and Eie 100 S co, then there exists
This theorem analogous to the result given in Shah et al. (2020), which has a convex assumption on their loss function, whereas our theorem is applicable even for the non-convex case. Although the theorem is for worst case analysis, our experiments show that, on average, our method easily outperforms other DNN methods that use all the data.
Remark
Theorem 2 suggests that, long as C is smaller than threshold. sample selection gives better convergence to the stationary points for clean data, compared to using all the data. As the anomaly ratio increases or the distance nearest outlier increases, sample selection will improve the conver- gence to stationary points for clean data in the worst case scenario compared to no sample selection.
Below give a sufficient condition for guaranteeing correctness when the objective is restricted to a convex setting. We assume that Yx; & O fww) = 0 and Vx, E O (w() > 0. Assuming f(w) is convex and its gradient is upper bounded, there exists ball of radius > around w/"
5

---

The proof is given in the Appendix section. The convergence guarantee depends on having a small enough value of C. which related to 6, the distance between the nearest anomalies and the normal points, and the landscape of the loss surface K. A small i suggests that the loss surface is very sharp Under review as a conference paper at ICLR 2021
defined as follows:
B, (w") = {w [i(w) < (ww) Vx, O0,x, E O, || - w" S r).
The above definition describes ball around the optimal point, in which the normal observations have a smaller loss than the anomalies. Based on this definition, the following theorem describes the sufficient condition for our algorithm to converge to a solution within the ball.
Ex f((w) be a twice- wrice-iiferentiabbe function and Liaa maxx, go (Li) be the maximum Lipschitz smoothness for the clean data and Pmin minxx be the minimum convexity for anomalies. Consider the sequence {werl generated by w(++1) fxx,) and max w(t) ([VF((())) -, (
Theorem Let F(w)
w(!) (1((
C. Define
Limaxx and suppose Assumptions I hold. n satisfy Et=1 r/()) = co, Etsns(( S 00,
Pmmn
and C C (1+Tx)))) [1+*)) = -0 (2) then there exists r > such that war E B, (w*).
for anomalies (large minn)) but flat for normal data (small Lmaz). In this case, most areas of the loss surface will have smaller loss for normal observations but larger loss for anomalies (assuming equal minima among all the points). Due to their larger loss, the anomalies have smaller probability to be selected as "clean" sample by the proposed RCA algorithm.
The analysis above shows that sample selection benefits convergence of our method to the stationary points for clean data. However, ultimate goal is to improve generalization performance, not just converging to good stationary points of the training data. When sample selection is applied to just one autoencoder, the algorithm may converge too quickly we use only samples with low reconstruction loss to compute the gradient, making it susceptible to overfitting (Zhang et al. 2016). Thus, instead of using only the self-selected samples for model update, we train two autoencoders collaboratively and shuffle the selected samples between them to avoid overfitting. Similar ideas have been found be effective in supervised learning for data with noisy labels (Han al. 2018).
3.2 ENSEMBLE EVALUATION
Unsupervised anomaly detection using an ensemble of model outputs have been shown to be highly effective in previous studies (Liu et al., 2008 Zhao et al.) 2019 Emmott et al.) 2015 Aggarwal & Sathe 2017). However, incorporating ensemble method to deep learning is a challenging problem as it is expensive to train a large number of DNNs. In this paper, use the dropout mechanism (Srvaattvaa et al. 2014) to emulate the ensemble process. Dropouts are typically used during the training phase only. In RCA, we employ the dropout mechanism during testing well. Specifically, we use the networks of perturbed structures to perform multiple forward passes over the data in order to obtain set of reconstruction losses for each test point. The final anomaly score is computed by averaging the reconstruction losses. Although dropout may increase the overall reconstruction loss, we expect a more robust estimation of the anomaly score using this procedure.
EXPERIMENTS
We have performed extensive experiments on both synthetic and real-world data to compare the performance of RCA against other baseline methods and to investigate its robustness to noise due to missing value imputation. The code is attached in the supplementary materials in submission.
4.1 RESULTS ON SYNTHETIC DATA
To better understand how RCA overcomes the limitations of conventional autoencoders (AE) on datasets with anomalies, we experimented with a synthetic 2-dimensional dataset. The dataset con- tains a pair of crescent-shaped moons with Gaussian noise (Pedregosa et al. 2011) representing the normal observations and anomalies generated from a 2-dimensional uniform distribution. In this experiment, we vary the proportion of anomalies from 10% to 40% while fixing the sample size to be 10,000. Samples with the top-[(1 e)no highest anomaly scores are classified as anomalies, where is the anomaly ratio.

---

Under review as a conference paper at ICLR 2021
Ground Truth
AE
RCA
%Anomalies Dropped
RCA
e
RCA
Figure 2: The first and second rows are results for 10% and 40% anomaly ratio, respectively. The last column shows the fraction of points with highest reconstruction loss that are true anomalies. We show the results for 10% (top row) and 40% (bottom row) anomaly ratio in Figure Observe that the performance of both methods decreases with increasing anomaly ratio. However, results for RCA (third column) are more robust than AE (second column). In particular, when the anomaly ratio is 40%, AE fails to capture the true manifold of the normal data, unlike RCA. The result is consistent with Theorem 2 which states that, when anomaly ratio increases, using the subset of data selected by our algorithm better than using all the data.
4.2 RESULTS ON REAL-WORLD. DATA
For performance comparison, we use 18 benchmark datasets obtained from the Stony Brook ODDS library (Rayanal 2016). A summary description of the data is given in Table 2 in appendix. We reserve 60% of the data for training and the remaining 40% for testing. The performance of the competing methods are evaluated based on their Area under ROC curve (AUC) scores. We also performed experiments on the CIFARIO dataset, for which the results are given in the Appendix.
We compared RCA against the following baseline methods: SVDD (deep one-class SVM) (Ruff (ct al. 2018). VAE (Variational autoencoder) (An & Cho 2015 Kingma & Welling 2013), DAGMM (deep gaussian mixture model) (Zong et al. 2018), SO-GAAL (Single-Objective Generative Ad- versarial Active Learning) (Liu et al. 2019). J OCSVM (one-class SVM) (Chen et al. 2001), and IF (isolation forest) (Liu et al. 2008). Note that SVDD and DAGMM are two recent deep unsupervised AD methods while OCSVM and IF are two state-of-the-art AD methods. In addition, we also per- form an ablation study to compare RCA against its four variants: AE (standard autoencoders without collaborative networks) and RCA-E (RCA without ensemble evaluation), and RCA-SS (RCA with- out sample selection). Since the methods are unsupervised, to ensure fair comparison, we maintain similar hyperparameter settings for all the competing DNN-based approaches to the best that we can (details can be found in the supplementary materials). Experimental results are reported based on their average AUC scores across 10 random initializations.
Figure 3a summarize the results of our experiments. The full table can be found in the Appendix. Note that RCA outperforms the deep unsupervised AD methods (SO-GAAL. DAGMM, SVDD) in 17 out of 18 datasets. These results suggest that the strategies employed by RCA are more effective at detecting anomalies compared to the ones used by the baseline deep unsupervised AD methods. Surprisingly, some of the more complex DNN basclines such as SO-GAAL, DAGMM, and SVDD perform poorly the datasets. Their poor performance can be explained follows. First, most of these baseline methods assume the availability of clean training data, whereas in our experiments, the training data was contaminated with anomalies to reflect more realistic settings. Second, the same network architecture on every datasets for all the methods (including RCA), since there is guidance for tuning the network structure given that it is an unsupervised AD task. Finally, as will be discussed in Section 4.3) the performance of conventional unsupervised AD methods such as OCSVM and IF degrade significantly as the amount of missing values in the data increases, unlike the proposed RCA framework.
4.3 RESULTS FOR ABLATION STUDY AND ANOMALY DETECTION WITH MISSING VALUES
As real-world datasets are often imperfect, we compare the performance of RCA and other baseline methods in terms of their robustness to missing values. Mean imputation common approach to deal with missing values. In this experiment, we add missing values randomly in the features of
Mooe results for 20% and 30% can he found in the appendix

---

Under review as a conference paper at ICLR 2021
-HA 10.
Wind
peoight
brosgrew
0.4
AS
trenithere
Inowtor
RCA---VA VAE -.. 50_GAAL AE -- ----AAGMM---M SVDD
(a) AUC comparison of RCA against DNN base- lines. Details are given in Table in Appendix.
(b) Results of RCA for overestimating e by 0.05, 0.1,
0.2. Details are given in Table D Appendix.
Figure 3: Experimental results on 18 benchmark datasets from ODDS repository.
Missing Ratio
RCA-E
RCA-S8
VAB
SO-GAAL
AE
DAGMM
SVDD
OCSVM
IF
0.0
11:2.
16.022
14:04
1740
15.0.3
18:00)
18:0
10.1-7
10.88
0.1
12-1-5
14-1-3
16-1-1
16-0-2
14-04
17-0-1
18-0-9
13-14
12-0-6
022
11:1-6
13:3-2
14-2-2
1740-
1340-5
18400
1840
15403
940-9
0.3
9-3-
13-1-4
15-0-3
17-1-0
13-0-5
18-0-0
18-0-
1640-
14-1-3
0.4
1040-8
12-24
14-04
15-0-3
12-0-6
1740-
1840-
16-022
15-0-3
0.3
8:377
10-177
11-16
1440-4
9-0-9
15403
174-1
14.13
13-0-5
Table I: Comparison of RCA against various competing methods in terms of (dwin--ddaww-+oos)) on 18 benchmark datasets with different missing ratios. RCA-E (no ensemble), RCA-SS (no sample selection), AE (no ensemble and no sample selection) are used for ablation study of our method.
each benchmark dataset and apply mean imputation replace the missing values. Such imputation process will likely introduce noise into the data. We vary the percentage of missing values from 10% to 50% and compare the average AUC scores of the competing methods. The results are summarized in Table [ which shows the number of wins, draws, and losses of RCA compared to each baseline method on the 18 benchmark datasets. We also include results from ablation study to investigate the effectiveness of using sample selection and ensemble evaluation. Specifically, we compare RCA against its variants, RCA-E, RCA-SS, and AE, The results show that our framework better than the baselines on the majority of the datasets in almost all settings. In particular, RCA consistently outperforms both DAGMM and SVDD by more than 80%, demonstrating the robustness our algorithm compared to other deep unsupervised AD methods when training data is contaminated. Additionally, as the missing ratio increases to more than 30%, outperforms IF and OCSVM by more than 70% on the datasets. On the other hand, the advantage of RCA over its variants, AE, RCA- SS, and RCA-E, is significant when the missing ratio is less than 40%, but becomes less significant higher missing ratios. Finally, since the true anomaly ratio is often unknown in practice, conducted experiments to evaluate the robustness of RCA when is overestimated by 5%, 10%, or 20% from their true values on all the datasets. Fig. 3b shows the AUC scores for RCA do not change significantly even when was highly overestimated on most of the datasets.
5 CONCLUSION
This paper introduces RCA, a robust collaborative autoencoder framework for unsupervised AD. The framework is designed to overcome limitations of existing deep unsupervised AD methods due to wer-parameterization of the DNNs, which hampers their effectiveness. We theoretically show the effectiveness of our algorithm to prevent model overfitting due to anomalies. In addition, empirically show that RCA outperforms various state-of-the-ar unsupervised AD algorithms in most experimental settings. We also found RCA to more robust to noise introduced by missing value imputation compared to other baseline methods. In the future, aim to extend the proposed framework to incorporate more than two autoencoders. We will also investigate whether is possible to relax some of the assumptions behind our theoretical bounds to realistic scenarios.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Charu C Aggarwal and Saket Sathe. Outlier ensembles: An introduction. Springer, 2017.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. Special Lecture IE, 2(1):1-18, 2015.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: survey, ACM com- puting surveys (CSUR), (1):111558 2009.
Yunqiang Chen, Xiang Sean Zhou, and Thomas S Huang. One-class svm for learning in im- age retrieval. In Proceedings 2001 International Conference on Image Processing (Cat. No. 01CH37205), volume 1, pp. 34-37. IEEE, 2001.
Andrew Emmott, Shubhomoy Das, Thomas Dietterich, Alan Fern, and Weng-Keen Wong. A meta- analysis of the anomaly detection problem. arXiv preprint 1rXiv:1503.01 2015.
Jinan Fan, Qianru Zhang, Jialei Zhu, Meng Zhang, Zhou Yang, and Hanxiang Cao. Robust deep
auto-encoding gaussian process regression for unsupervised anomaly detection. Neurocomputing, 376:180-190, 2020.
Yoav Freund, Robert E Schapire, al. Experiments with boosting algorithm. In ieml, volume 96, pp. 148-156. Citeseer, 1996.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 181 04606, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv:1712.05055, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1rXiv:1312.6114,0 2013.
arXiv preprint
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances Neural Information Processing Systems, pp. 1189-1197 2010.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 Eighth IEEE Interna- tional Conference on Data Mining, pp. 413-422. IEEE, 2008
Yezheng Liu, Zhe Li, Chong Zhou, Yuanchun Jiang, Jianshan Sun, Meng Wang, and Xiangnan He. Generative adversarial active learning for unsupervised outlier detection. IEEE Transactions on Knowledge and Data Engineering, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830. 2011.
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. Advances neural information processing systems, pp. 6822-68223 68333, 2018.
Shebuti Rayana. ODDS library. Stony Brook University, Department Computer Sciences, http: //odds. www.sstoowoooooo. edul 2016.

---

Under review as a conference paper at ICLR 2021
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pp. 1393-4402, 2018.
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimen- sionality reduction. In Proceedings of the MLSDA 2014 2nd Workshop Machine Learning for Sensory Data Analysis, pp. 4-11, 2014.
Vatsal Shah, Xiaoxia Wu, and Sujay Sanghavi. Choosing the sample with lowest loss makes sgd robust. arXiv preprint arXiv:2001.033/1 2020.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss mini- mization. XrXiv preprint arXiv: 1810: 1874, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research, 15(1):1929 1958, 2014.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. Journal of machine learning research, 1((eec):3371-3000 2010.
Siqi Wang, Yijie Zeng, Xinwang Liu, En Zhu, Jianping Yin, Chuanfu Xu, and Marius Kloft. Effec- tive End-to-end Unsupervised Outlier Detection via Inlier Priority of Discriminative Network. In Advances in Neural Information Processing Systems, pp. 1960-5973, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: 16/1::661..333 1.05530 2016.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. arXiv preprint 1901.0/99 2019.
Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders. In Pro- ceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 665-674, 2017.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. 2018.
A APPENDIX
A.
DATA STATISTICS
The data statisties are in table 2
A.2
FULL TABLE RESULTS ON ODDS
See table
A.3 DISCUSSION ABOUT SVDD AND DAGMM
The reason why SVDD and DAGMM performs bad in ODDS might have several reasons. First, the results reported in the SVDD paper assume training data has no contamination. For the DAGMM paper, it has only datasets that overlaps with our experiments (thyroid, arrhythmia) for which the results reported are also for clean data. In contrast, the training data used our ex- periments are contaminated with anomalies. Also, the DAGMM paper acknowledges that their performance degrades when the amount of contamination increases. Furthermore, unlike our exper- iments, the results reported in the DAGMM paper use different network structure for each dataset to obtain good performance, this is because that they have clean dataset, which makes it possible to tuning the network structure and hyperparameters. However, choosing the right network structure is
10

---

Under review as a conference paper at ICLR 2021
impractical for unsupervised anomaly detection without ground truth labels available. In exper- iments, all baselines and our method use the same network structure across different datasets. Since there is official code of DAGMM from the authors,
implementation of DAGMM are highly depends on these two open source implementations Fl Another question people may ask is that why complexed deep methods such DAGMM, SVDI, SO-GAAL cannot beat shallow methods such as OCSVM and Isolation Forest. According to our best knowledge, there is no evidence SVDD, SO-GAAL and DAGMM performed better than OCSVM and IF datasets beyond the benchmark image data (CIFAR-10) In fact, the SO-GAAL, OCSVM and IF results reported in our paper for ODDS dataset are consistent with the numbers reported in PyOD [ Python toolkit for outlier de- tection. Also, Reference [8] compared DAGMM, SVDD (denoted E2E), against OCSVM. The results shown in Table of the paper are similar to ours. In order to make sure that our algorithm better than SVDD or there is nothing wrong in our implementation about SVDD, we also conduct experiments on CIFARIO. Since our methods are not specifically designed for the image data, we process the CIFARIO results in following way:
We use pytorch official implmentation of vgg19 (pretrained on ImageNet) to extract 4096 dimen- sional feature representation of CIFARIO to perform the anomaly detection for each class (10 sub- datasets). Each sub-datasets consists 5000 normal class and 5% anomalies, which are random sam- pled from other class. The training, testing data consists 80% and 20% of data (i.e. training data has (5000 + 250) 0.8 = 4200 samples, testing data has 1050 samples). All results are averaged over random seeds. The results are in the figure For RCA, DAGMM, SVDD, we use the same network structure in our paper. For SVDD.Original, we directly borrow the number from original SVDD paper (Ruff et al. 2018
A.4 PROOF OF THEOREM 1
denote M = n11 - Lima to be the smoothness of function Eieo fi(w), denote the up- date rule (++1) (tt) 1 Σ. of For the normal stochastic gradient descent, by smoothness, we have:
M
F(wlt+1)) F(w(() (VV())) ((++))
(()) -w)) +
12+
2
S - (VE((() (9), Vf((w(())) + (0) M
|Vf(((())]
(4)
2
hhtgss//gtttbb.c//ddaittaaaaaal com//amicctand77/dggm.. httpss//jtttbb..cmmmmnna///AAAAA We use the pyod implmentation http://giiubccom//yaaaooo/////// of SO-GAAL, VAE, IF OCSVM Table 2: Summary of benchmark data, where N sample size and is number of features.
Dataset
NV
d
Anomaly ratio
vowels
1456
12
3.4%
pima
768
35%
optdigits
5216
64
3%
sensor
58509
48
9.1%
Tetter
1600
32
6.25%
cardio
1831
21
9.6%
arrhythmia
452
274
15%
breastw
683
9
35%
musk
3062
166
3.2%
mnist
7603
100
9.2%
satimage-2
5803
36
1.2%
satellite
6435
36
32%
mammography
11183
2.32%
thyroid
3772
6
2.5%
annthyroid
7200
6
7.42%
ionosphere
33
36%
pendigits
6870
16
2.27%
shuttle
49097
7%
11

---

Under review as a conference paper at ICLR 2021
Table 3: Performance comparison of RCA against various baseline methods in terms of average
AUC scores and its standard deviation across 10 random seeds.
Dataset
RCA
VAE
SO-GAAL
AE DAGMM SVDD
OCSVM IF
vowels
0.857+0..03
0.09 0.400.00.0.000000000 10000 0.7.5..04..665.000 0.760.04007660001
pima
0.996:10.01
letter
cardio
breastw
musk
mnist
satellite
0.69+F0.01
thyroid
ionosphere
shuttle
glass
Table Sensitivity Analysis about e, the first number averaged auc score and the second number is the standard deviation. All experiments are repeated for 10 random seeds.
Dataset
6 0.05
6 0.1
6 0.2
vowels
pima
0.70040001007155:00000111 00000000000000000000000000000000
opidigits
0.914+0.02
sensor
0.996+0.11100000000. 11000000000000000000000000000000
letter
cardio
0.946410.01
arrhythmia
0.807+0.05
breastw
0.98610001
musk
[000+500000000000000000000000110
mnist
0222:000079000000000000000011060
0.827+0.02
satimage-2
satellite
0.69110.01
11811000000000000000000000000000
mammography
0.85910001
0:100000000000001111000000101110
thyroid
0.941:0.01
annthyroid
ionosphere
0.796:000100794444222
0.796:10.01
0.794400110770000000000000000000
pendigits
0.903-10.02 00A.-00000000-00000000-000000000
shuttle
01110000110000000000000000000000
12

---

Under review as a conference paper at ICLR 2021
0.8
0.75
0./
0.65
0.5
0.55
0.5
0,45
Aifflne
AutoMotlle
Bird
Cat
Deer
Dog
Trog
Horee
SHip
Truck
200
Terind
Figure 4: Results on CIFARIO for SVDD, DAGMM, and our method. x-axis are the normal class while y-axis is the auc score. SVDD are our implementation on contaminated data described above. The SVDD-Original is the number reported in SVDD original paper (where they use clean training data). We could see that even use contaminated data, our model still outperforms deep svdd in most settings, and SVDD performs bad when training data has contamination. (i.e. SVDD performs much worse than SVDD Original)
Take expectation V "[((w (w())) by our sampling probability and applying triangle inequality on the last term with inequality Σ p < Epi. Pi, we have
EF(w*+!)- - -F(w(1)) S -1 (V)((()) [n(w())(((())) + (n((w()) 7 M
2
S <F En(wO)) -VF()) - |V]()]| + (0)2 MG
+, MG²
2
complete the square and let F(w(1)) = -
(1), we have
(9)
|VF (1)) + (t) [VF(w()) - vf( - |V((((0)) 1/() MG2
2
2
(9) -
vf(w(00)11 ()
2 IVF( n MG²
2
Move the gradient norm to the left, and take total expectation, we have
(E][VF(((())] < (EF(((() - (F((((+)))) + ()) v (0' MG²
Sum it from t = t = T, we have:
T E (W 2 2 EF (
Σ " 0 (
1000 7(00) MG²
T
T
< 2B + Σ 10 10 Σ 10? MG²
1=0
1
+M E
(02 [t(
min 1.1........
V/ )1² E (0)]
2B
C
(5)
Σn(tj
By using the assumption of learning rate En = x0, En (t)) < oc), the first two term can be ignored when 7 goes to infinity. We can get the convergence in theorem (The convergence
13

---

Under review as a conference paper at ICLR 2021
rate lTg(T) is get by assume learning rate is 17(*) 1/t, which satisfy the above learning rate assumption).
We also provided a better convergence rate compared to submitted manuscript with a stricter learning rate setting. Start from equation 4 we have:
[pi(w(o)Vii(w + Epi(w(() 17/199
M
(F(w*tl)- - F(( S -n (*) FVF(WW) (w)
V (1)
2
< <- En(( VF(-)) - |[((]] T, MG
2
complete square in different way and let ((w(1)) = Σ [2((w())/(()) we have
1 2
// )1 + 17(1)
1
- -F(m())-V((((")))
n00 MC2
-
2
1 1/19
1
+ 5/00 TG
MG²
[VF(
(1)
+ 2
[F((((())
2
2
[IIFF((()) - (/( () (0)2 1/10" (M (M+ + 1)G
2
Move the gradient norm the left, and take total expectation, we have
T VFF 11 & ()) - (F(w((+))) + 2 [BIVF((W(O) - - + 1/000 (M + 1)G²
2
Sum it from to T, we have:
E VF (F( = (F(w(")) -
1/ (M + L)G²
Bl BIVFF((() (1) v +
2
TC
T
n000
(M + 1)G²
B
+
2
2
1=0
B
(M+1) Σn
TC
min 12,11.2.....
V F (w
)1
En
Σn
Σn
By assume the learning rate is constant, we write the RHS function of learning rate
B
(M + 1)G2n
n
TC
f(n) =
+
+
Tn
2
Tn C
B
(M + 1)G² n
+
+
Tn
2
7
Let (M
+ 1)G² = H, we have
Tn + Hπ
B
C
2 7
We study the minima of function:
f(x) + +br
x x² (a/b)), f(x") =
14

---

Under review as a conference paper at ICLR 2021
2B 2C
Thus, to minimize the bound in RHS, have the optimal learning rate 17"
V
HT +
by
B letting T
H
+ +
Then, the optimal value RHS is:
2
f(n') f(1')= - 2(1
(T
B
(
H
- 2BH +2CH $ V
2BH
+ V2CH =0
-+)+ O
/
T
VT
BI 0 1 VT +
Thus we can conclude that m.
This is even a better results compared theorem 1 in our paper, since we could achieve the con-
vergence rate of V V compared to log(T)), while the error term is dependent on vCC instead of C. However, to achieve this rate, we need stricter condition on learning rate.
A.5 PROOF OF THEOREM 2
We analysis the stationary point of using all data. Let w denotes the stationary point by using the entire data, w denotes the stationary point by using the clean data, by the stationary condition of Was we have
Σ fi(wee) - Σ (wii))))
JEO
4
Σ Vf(( m1)l
Vfz ii)ll
go
JEO
Upper bound for LHS
P
P
Σ V.(( [L =
f((w iss
- (w(w"))
Σ
(V(((t)) V.((")))
- >>
La dollw"s, wl
:g0
IÉO
ig0
= -1 e)nLmas max
I Rs:
will
< M6 (ō is defined in equation 3 in the submitted manuscript)
Another upper bound for LHS from RHS
P
9
(w nEG
100
JEO
Thus, we have
IVF )1 min(neG, M6)
2B +
20
From theorem 1, by setting n¹"
V HT
H it is trivial to get
2BH
min 0.1........
V 12
+ 2CH
T
Or by assumption of mn 00, (1 < from equation 5 get
Σn n
min t=0.1.2.....
E (WVF(((()) < EF 11 < 0
Σn
0
Σ
+ C
We would like to study the worst case, when the upper bound of our algorithm is better than the upper bound without sample selection.
Thus, we want the following holds when goes infinity
2BH
+ 2CH < min(neG, M8)²
T
15

---

Under review as a conference paper at ICLR 2021
When goes to infinity, by the assumption have for learning rate, have:
vC (min(nxG M optimal Ir)
Or similarly, in deminishing learning rate setting, we have
C v (min(neG, M6))² En = Σ En 2
Thus both LHS and RHS the upper bound, thus could get the conclusion that in worst cases, our solution is better than algorithm without sample selection, which gets our conclusion of exis- tence.
A.6 PROOF OF THEOREM 3
Now, we try to prove the correctness of the algorithm. We assume w satisfy fi(w") < f,("), & X, E O.
Without loss of generality, we could define 6, の 2 1 as below:
8 S W
S 66, Vj E
Now, we try to answer the first question, under what conditions, our solution is perfect. We define some neighbors around the optimal point. According to our anomaly detection setting, we with those anomalies should have higher loss compared to the normal data point:
= {
w www)(w) < f((w), vi & O,jE0,
}
B, (ww))
(8)
|| w
We knew that such ball with radius must be existed since the loss function above is both strongly smooth.
In order to better describe B, (w"), would like to analysis the boundary of the ball.
Denote the set of intersection between the loss surface of the normal data and the loss surface of the abnormal data as:
Sw = (Weyl = fx))
(9)
Then, can write the boundary point of the ball as:
W B = arg min |/ - w'll
(10)
wE
At w B. by using smoothness and convexity, we have
Li [B w*12
f((B)) & fa(w") + (W - w", VV((("))) f5(WB) 2 / (W))) + (WB wf. Vfz (
2
[ w 11
By the definition of w and equal minimum assumption, without loss of generality, we could assume the minimum is O, which does not affect the results. Then, we have
L; 2
fi(WB) <
W 11
-f)(WB)
Hí
wa w ||
2
Adding two inequality, we have:
IWB w1
>
M T,
WB w)1l
By triangle inequality, we have
[WB w
+
|
w || 2
|
w
16

---

Under review as a conference paper at ICLR 2021
Combining above two inequalities, we have
Mí L
[wB w 2
(Ilw
w"l
l|wB
w1
Pi 8 Li
(1 +
l|WB w 2
Li
L'max
let K
V
Mnin
WB
will
82
5
L
1
+
P5
where Liaz denotes the maximum lipschitz smoothness in clean data and pmm denotes the mini- mum convexity in anomalies.
Define F(w)= = Exiço Ji(w), we have that function F(w) satisfy m = n11 - Umnn convexity. Similarly, we know that F ((w) also satisfy M = n11 Lmaz smoothness. Then, we can have:
| will S m VF(( VF(w")Il
V (w II
[ w V
Now, our goal is trying to upper bound the tern optimal learning rate, we have
IVF I||- According to theorem 1 for fixed
2BH + 2CH
min 101,2.....
V
()
Now, we have:
/BH
|w - 2 S m² V
2CH
T
[WB w 21 + k
8)2
l| Thus, we could get the sufficient condition for w²1 1|2 < WB w2
m² 2 2BH +CH S
1 8)2
1 K
by using the optimal fixed learning rate and assume 7' is sufficiently large, rearrange the term, we have:
8 )2 2
VC
)2
V2H
(1 + k)m
C 0 -
8
Similarly, from the theorem for the diminishing learning rate condition Σn -0, En S co.
we have
C (1+xmm - 0 -
=
We can conclude that as long as the above inequality holds, we can guarantee that our algorithm returns the correct answer.
17

---

Under review as a conference paper at ICLR 2021
Clear date
Anomaly Eangerous
Asciraly dats
Oangorous zonc
Figure 5: Illustration of the effect of theorem 3. The left figure means we have small L and large Menin. which leads large According to our bound, large k is not good in terms of correctness guarantee. As we shall see the left figure, in this case, the dangerous zone is very large compared to the right figure, where we have large I small Memn: and small In the right figure, we could see that the orange curve will be dropped since the probability of the orange curve is sampled is very small in our method.
B PROBABILITY OF SAMPLING WITH REPLACEMENT
In submitted manuscript, we only show the probability of sampling without replacement due limit of the space. In here, we show the probability of sampling with replacement:
if S Bk; otherwise
(w(w)=
(11)
Ey=0
C SUPPLEMENTARY EXPERIMENTAL RESULTS
In this section, we show the supplementary experiment results, which are omitted in submitted manuscript due to the limit of space.
C. NETWORK HYPERPARAMETER FOR TWO-MOON DATA
The network structure for both autoencoders and our method is the same for a fair comparison. The network has one layer encoder and two layer decoder. In the hidden layer, we have 0.5 dropout ratio, the number of hidden nodes for all layers are set to be 128. All activation function is chosen to the tanh function. The maximum training epochs are 200. The stopping criterion the loss of testing data. The ensemble number is set to be 1000 for our method. Training ratio is 60% and testing ratio is 40%. We use Adam optimizer for both method and the initial learning rate is set to be 3e-4. Batchsize for both methods are 128.
C.2 NETWORK HYPERPARAMETER FOR BENCHMARK DATA
Specifically, we use 6-layer fully connected autoencoder with 128 hidden nodes in every layer except for the bottleneck layer, which has 10 hidden nodes. We also set the dropout rate to 0.5 for every hidden layer. The deep neural networks are trained using ADAM, with learning rate initialized to 3e-4 and a batchsize of 128. The maximum epochs is set to be 100 with a stopping criterion determined from the minimum reconstruction loss of the test data. The reconstruction loss function for the opt-digits dataset cross entropy loss since the feature of this data are all discrete. The rest data reconstruction loss are all mean square error loss. The activation function is chose to be LeakyReLU with a = 0.1. For the SVDD, we pretrain the autoencoder for 50 epochs, and use the encoder the initialization of SVDD except the last layer.
18

---

Under review as a conference paper at ICLR 2021
Ground Truth
AE
RCA
%Anomalies Dropped
RCA
AE
86
REA
2 &
E.
14
05-
04
RCA
0.
AE
Figure 6: The rows are the results for 10%, 20%.00%.30% and 40% anomaly ratio from top to bottom, respectively. The last column shows the fraction of points with highest reconstruction loss that true anomalies.
C.3 RESULTS FOR Two MOON
In submitted manuscript, due to the limit of the space, we only show the results for 10% and 40% anomaly ratio. In this section, provide the results for 10%, 20%, 30%, and 40% in Figure
19


Reviews:
[{"review_id": "Zz6F3fUwfRK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis manuscript proposes a novel learning method to improve the robustness of unsupervised anomaly detection called robust collaborative autoencoders (RCA). This combines two autoencoders which exchange samples from heterogeneous batches according to the rankings that each individual model assigned.\n\nWhile the manuscript contains a number of interesting theoretical motivations, its experimental section contains some weakness, and I would hope for it to include a larger set of competitors, as well as a more flexible model class with which RCA is paired.\n\n### Strengths\n\nThe paper follows a standard structure and is logically organized. Theoretical results are provided that underpin the sample selection criteria that is used in RCA, in particular. The experimental section compares RCA against some other (however mostly simple) anomaly detection methods, and the authors propose an interesting idea meant to enhance robustness, a particularly desirable property when dealing with anomaly detection.\n\nThe authors provide code, which is always a plus. Experimental results are computed from ten random seeds, with standard deviations included. A methodological description is included in Section 3.\n\n### Weaknesses\n\nMy main criticism revolves around the extent of the experimental section. Given the generality of AE architectures and their wide applicability to all types of AD (on images, text, etc.), it would have been interesting to learn how RCA fares in different scenarios:  for instance, RCA incorporated with more recent convolutional autoencoding setups is missing in the evaluation. Unfortunately, RCA is evaluated on mostly low-dimensional data, and against simple competitor models/AE architectures.\n\nTo counter any doubts around the feasibility of RCA to scale to more complex AE setups and datasets, it would be interesting to see RCA evaluated on more standard real-world benchmark data (such as CIFAR-10, which is widely used in the standard anomaly detection literature), or with more complex autoencoders, e.g. those proposed in Huang et al. (2019). An additional question that remains unexplored is how well RCA can scale to more than two sub-AE modules, and whether this would be a practical thing to do. Experiments (or some discussion) in this direction would be very interesting!\n\n### Additional remarks\n\n* Axes in Figure 2 are illegible.\n* The table formatting is sub-optimal, c.f. the instructions for submission.\n* The formatting of Alg. 1 and 2 can be improved.\n* Table 3 is hard to read, this might be better in a figure.\n* Why not move Assumption 5 in the vicinity of Theorem 3, since this is only required there? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q9s2S1Cdnoh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a Robust Collaborative Autoencoder (RCA) for unsupervised anomaly detection. The authors focused on the overparameterization of existing NN-based unsupervised anomaly detection methods, and the proposed method aims to overcome the overparameterization problem. The main contibutinos of the proposed method are that (1) it uses two autoencoders, each of which is trained using only selected data points and (2) monte carlo (MC) dropout was used for inference.\n\nAlthough this paper has an interesting idea, i have doubt about the contributions. My comments are as below.\n\n1) First of all, to me it was very difficult to read this paper. The notations are very confusing.\n\n2) In Introduction section, it is confusing what the main focus of this paper is. They mentioned like \"unlike previous studies, our goal is tho learn the weights in an unsupervised learning fashion\". But because it seems the topic of this paper belongs to \"unsupervised anomaly detection\" (the labels indicating whether anomaly or not are assumed available in the training data), the point that your method is in an unsupervised learning fashion is pretty obvious.  You don't need to discuss about \"supervised approachs\" throughout the paper, but please clearly mention that at the beginning of the introduction section, and only discuss your method and other \"unsupervised\" anomaly detection methods.\n\n3) If the overparameterization is the problem when we build a NN for unsupervised anomaly segmentation (e.g. autoencoder-AE), we can simply think about various well-known NN regulaization techniques for the AE as remedy. I also think the two parts of the proposed method (corresponding to the contrbutions (1) and (2) )work as regularization for the AE. I'm curious if there's any reason to prefer the proposed method to other regularization techniques?\n\n4) The proposed RCA method involves an ensemble prediction by using MC-dropout (described in section 3.2). The authors metioned this is one of their research contribution, but the use of MC-dropout is quite general in neural network research. Also, while the proposed method definitely benifits from the use of MC-dropout, other unsupervised anomaly detections based on neural networks (e.g. AE, VAE,...) can also improve by employing MC-dropout. The ablation study in Table 1 showed that RCA significantly outperforms RCA-E (RCA without ensembling).  The authors can implement MC-dropout-based ensemble versions of AE, VAE, and other nn-based methods like Deep SVDD, and check whether the proposed only benefits from MC-dropout or RCA just outperforms others regardless of MC-dropout.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3mKqL-Afy5M", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe submission tackles unsupervised anomaly detection, specifically in a scenario where supervision labels are not available, only information about the ratio of anomalous examples in the data set. They suggest an architecture consisting of two auto-encoders collaboratively determining anomalous samples and updating their weights based on data that is deemed normal. The authors provide a theoretical analysis of the selection process, and validate anomaly detection performance on a range of experiments.\n\n# Pros\n\n**Interesting challenge**\n\nTackling the problem of potentially contaminated data heads on instead of side-stepping it with assumptions like guaranteed normalcy of the training data set is an interesting challenge.\n\n**Relative simplicity of the approach**\n\nThe suggested algorithm is a remarkably simple extension (or self-regularization?) to vanilla auto-encoding. The changes are fairly minimal, losses and AE architectures remain the same. At the same time, the suggested duplication of AEs and selection of data directly address contaminated anomaly detection data sets. This allows for potentially widespread applicability of the idea to other kinds of data, problems or architectures. \n\n**Theoretical underpinning of the algorithm.**\n\nI applaud the author's effort in examining and motivating the suggested changes not just by experimental results, but by a more rigorous theoretical analysis. This is a big plus in a typically very evaluation- and application driven field. This especially holds true given how small the architectural changes are: proving their legitimacy both theoretically and experimentally can make for a strong contribution. \n\n# Cons\n\n**The motivation and context could be stated more clearly.**\n\nThe setting that the authors assume and the contributions to that setting are muddied throughout the paper. Abstract and introduction discuss various problems of DNN-based AD methods, for instance overparameterization. At the core of their setting, however, is contaminated data without any label information beyond (an estimate of) the ratio of anomalies. \n\nI believe the authors should emphasize the setting much clearer, motivate their choice of setting compared to more common approaches in the literature like unsupervised learning on \"guaranteed\" normal data.\n\n**The theoretical analysis is relatively overemphasized.**\n\nAs stated above, I believe the theoretical analysis is a strength of the paper. Spending four pages on the methods section, and two of those an the assumptions, theorems, and remarks, compared to a total of two pages of evaluation, the authors clearly emphasize this aspect of their contribution. From this perspective, I believe the theoretical analysis takes up too much space, especially considering the very application-driven nature of anomaly detection methods. Put bluntly: The theorems are certainly interesting and worth having which is why I consider them a pro; but they are not strong enough to justify the amount of space compared to, e.g., evaluation.\n\nI believe the main text should stick to the theorems, and spend less time on technical details and more time contextualizing the results: How strong are the results? How valid are the assumptions? After all, the proof largely hinges on the assumptions, so they deserve more scrutiny than they currently get. Are the theoretical results reflected in the experimental evaluation? If not, why?\n\nThe bit about the integer program to determine the selected data points, taking up half a page, seems like a retrospective justification for the perfectly valid design decision to pick the fraction of examples with lowest reconstruction error, but otherwise does not add much.\n\n**The evaluation is too coarse.**\n\nThe previous point dovetails with my main criticism: the evaluation. This should have been a much stronger focus of the paper, given that anomaly detection is very application-driven.\n\nThe synthetic data set is nice to examine qualitative results. However, the evaluation is purely qualitative, where quantitative metrics would also be in order. The right-most column of figure 2 is the only hint at performance. We see that a lot of anomalies are not selected in both rows. I might be misunderstanding something, but this hints towards a massive amount of false positives and negatives? \n\nThe real-world experiments are lacking a lot of evaluations in my opinion. The analysis is reduced to \"winning\" the AUC score against the baselines on as many data sets as possible. While that's desirable, it's not helpful in understanding the pros and cons of certain algorithms. This is particularly true given that the experimental setup of contaminated data violates the assumptions of many of the baselines.\n\nOverestimating the share of anomalies is studied, although the given results are very hard to parse. This begs the question: What happens if I underestimate the contamination? This should lead to more anomalies being part of the data used for backprop. Given that the algorithm is based around that ratio, or an estimate thereof, I would have liked to see a stronger focus on it.\n\nAs you hypothesize, your selection method biases the representation learning. This should be examined in an experimental evaluation. This is particularly true given the venue.\n\n**The presentation can be improved.**\n\nThis is not a decisive point, but I believe potential readers would greatly benefit from improvements in structure, writing, and layout. I have gathered a number of suggestions further down.\n\n# Recommendation\n\nGenerally, I believe the suggested architecture and algorithm are worth pursuing and eventually publishing. This may be in contrast with the length of the positive feedback vs. the negative feedback, but in this case the opposite is true: the core idea is intriguing, but the paper on it can be improved.\n\nI believe the paper needs to be more precise and nuanced in answering a potential user's question: When and why should I consider this algorithm? In my view, the paper can and should be improved on two fronts:\n\n1. The experimental evaluation needs to be more thorough, and in particular less focused on \"winning\" over baselines, but on understanding and showcasing defining properties of the suggested algorithm.\n2. The presentation can be made much approachable.\n\nOverall, I believe the paper as is should be rejected. I nevertheless strongly encourage the authors to improve their evaluation and manuscript and resubmit!\n\n# Questions\n\nGenerally, the authors aim at fairness by fixing auto-encoder structures. Does this mean that RCAs generally have a multiple of learnable parameters compared to, e.g., the AE baseline? Further, how did you determine the hyperparameters? I would argue an HPS is due when comparing such different models.\n\nGiven that VAEs seem to be the baseline that compares most favorably according to your result, it would seem fairly obvious to try \"RCVAEs\", where everything is the same except reconstruction losses are replaced either by the likelihood term of VAEs or the ELBO directly. Have you considered this, and if so, why haven't you tried it?\n\nCould you elaborate more on how a user with entirely unlabeled data would go about making a good guess for the ratio of anomalies, so as to be able to use your algorithm?\n\n# Further Feedback\n\nI believe the presentation of the method and results could be improved in several places. Take these as suggestions for a revision---I don't see a particular need to address these points in a rebuttal.\n\nThe introduction is very long. It contains a substantial amount of related work, and a fairly detailed description of the proposed method. I would encourage the authors to move those bits to the respective dedicated sections and give the reader a more precise problem formulation, and particularly what part of the problems are tackled by the contributions of this submission.\n\nFigure 1 is not particularly illustrative: One can see that subsampling and shuffling is going on, otherwise one has to have a pretty good idea of the method to understand the illustration. On a side-note, I would encourage the authors to investigate tikz or similar alternatives for a cleaner style that is more integrated with the notation of the paper.\n\nFigure 2 cuts the lower part of the top row including a (redundant) legend.\n\nAlgorithms 1 and 2 could also be clearer. What is S, \\hat{S}_1 etc? First it's a minibatch, then it's the output of an auto-encoder? In algorithm 2, the notation of the forward step changes. \\xi is a set, then the last step of the for loop does both set and arithmetic operations on \\xi_1 and \\xi_2. In this light, I would argue that a more mathematical notation in favor of a programmatic notation would help the reader. This would also shorten the lines and make them more readable.\n\nGenerally, the authors aim at fairness by fixing auto-encoder structures. Does this mean that Generally, notation could be a little clearer. \\mathcal O is the set of anomalies, which is usually big-O notation, for which you use \\mathbb O. Sets sometimes have upper-case Greek letters, sometimes lower-case Greek letters. The probability probability p_i(w) is not actually the probability of w, but the probability of i as a function of w. Such small inaccuracies amount to an unnecessary increased mental burden for the reader.\n\nThe results of section 4.2 could be presented in a much clearer format. The figures are illegible at 100% zoom. Putting this aside, both figures 3a and 3b are unnecessarily difficult to process. Consider 3b: The interesting aspect of the figure is the disparity between the lines along certain circle segments. 90% of the graph are uninformative space, and the nuances are lost.\n\nI have a strong, possibly subjective or biased opinion about the way of presentation of table 1. There is something to be said about compressing large tables like e.g. tables 3 and 4 in the appendix into digestible formats. That said, I think wins, draws, and losses are the wrong mind set to approach baselines to begin with. In this particular case it oversimplifies the matter by quite a margin.\n\nThe appendix leaves room for improvement, in particular w.r.t. layout and typesetting. Equations should be broken to stay within the margins. Punctuation should conclude equation blocks, and within the environment instead of the next line. Equations should be indented. Delimiters like parentheses should have appropriate height, for instance by making more liberal use of \\left and \\right for parentheses for better readability. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "134RZfNhEQQ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed approach differs from autoencoder based anomaly detection approach in the following ways\n(a) Autoencoders are trained using only selected data points with small reconstruction errors. These are selected using a sampling scheme with theoretical guarantees on convergence.  The selected points are then shuffled between two autoencoders. \n\n(b) During the testing phase, each autoencoder applies dropout to generate multiple predictions. The averaged ensemble output is used as the final anomaly score.\n\nSome of the issues with this paper\n\n(a) One key issue is why just two autoencoders (which the authors delegate for future work). However, it is key to understanding utility of such an ensemble based shuffling framework.\n\n(b) Poor presentation of results\n    1) Figure 2 legend issue\n    2) Figure 3 (a) is better presented as a table (Table 3 in Appendix should be here instead). Very hard to interpret it in the current form. Similar comments for Figure 3(b) and Table 1. Also for anomaly detection benchmarking AUC is not sufficient and the authors have to present AUPR or F-1 scores also. I suggest looking at these recent papers for presentation of experimental results \n\nhttp://proceedings.mlr.press/v108/kim20c/kim20c.pdf\n\n\nhttps://proceedings.icml.cc/paper/2020/file/0d59701b3474225fca5563e015965886-Paper.pdf (Goyal et al. ICML 2020)\n\n(c) Theorem 3 might have a connection with the notion of r-robustness presented in https://arxiv.org/abs/2007.07365\n      so authors would want to make it clear how they differ.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes an autoencoder-based approach to anomaly detection.  The main weakness—not untypical for papers in this application area—is the experimental section.  The problem itself may be not well-defined, and of course that makes practical comparison difficult. Perhaps different measures—e.g., remaining life—may be better to compare on, and give better data sets.
Decision: Reject