Restricted Generative Projection for One-Class Classification and Anomaly detection
Authors: Feng Xiao
Ruoyu Sun
Jicong Fan
Venue: Submitted to ICLR 2023 (2023)

Abstract: We present a novel framework for one-class classification and anomaly detection. The core idea is to learn a mapping to transform the unknown distribution of training (normal) data to a known distribution that is supposed to be different from the transformed distribution of unknown abnormal data. Crucially, the target distribution of training data should be sufficiently simple, compact, and informative. The simplicity is to ensure that we can sample from the distribution easily, the compactness is to ensure that the decision boundary between normal data and abnormal data is clear and reliable, and the informativeness is to ensure that the transformed data preserve the important information of the original data. Therefore, we propose to use truncated Gaussian, uniform in hyperball, uniform on hypersphere, or uniform between hyperspheres, as the target distribution.  We then minimize the distance between the transformed data distribution and the target distribution while keeping the reconstruction error for the original data small enough. Our model is simple and easy to train especially compared with those based on generative models. Comparative studies on a few benchmark datasets verify the effectiveness of our method in comparison to baselines.

Under review as a conference paper at ICLR 2023
RESTRICTED GENERATIVE PROJECTION FOR ONE- CLASS CLASSIFICATION AND ANOMALY DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
We present a novel framework for one-class classification and anomaly detec- tion. The core idea is to learn a mapping to transform the unknown distribution of training (normal) data to known distribution that is supposed to be differ- ent from the transformed distribution of unknown abnormal data. Crucially, the target distribution of training data should be sufficiently simple, compact, and in- formative. The simplicity is to ensure that we can sample from the distribution easily, the compactness is to ensure that the decision boundary between normal data and abnormal data is clear and reliable, and the informativeness is to ensure that the transformed data preserve the important information of the original data. Therefore, propose use truncated Gaussian, uniform in hyperball, uniform on hypersphere, or uniform between hyperspheres, as the target distribution. We then minimize the distance between the transformed data distribution and the get distribution while keeping the reconstruction error for the original data small enough. Our model simple and easy to train especially compared with those based on generative models. Comparative studies on few benchmark datasets verify the effectiveness of our method in comparison to baselines.
INTRODUCTION
Anomaly detection (AD) aims to distinguish normal data and abnormal data using a model trained
on only normal data without using any information of abnormal data (Chandola et al. 22099] Pang
al. 2021 Ruff al. 2021). AD useful in numerous real problems such as intrusion detection for video surveillance. fraud detection in finance, and fault detection for sensors. Many AD methods have been proposed in the past decades (Schölkopf et 1999 2001 Tax & Duin 2004 Liu al., 2008). For instance, [Schölkopf et al. (2001) proposed the one-class support vector machine (OC- SVM) that finds, in a high-dimensional kernel feature space, a hyperplane yielding a large distance between the normal training data and the origin. Tax & Duin (2004) presented the support vector data description (SVDD), which obtains spherically shaped boundary (with minimum volume) around the normal training data to identify abnormal samples. There
also many deep learning
based AD methods (Erfani et al.) 2016] Ruff et al.) (218) Golan & El-Yamm Yi-iviiv]] [2018] H Hendrycks et al.
2018
Abati al. 2019; Pidhorskyi al. 2018 Zong et al. 20188] Wang al. 2019 Liznerski
et al.
2020
Qiu et al. 2021 Raghuram et al. 2021 Wang et al. (021).
Deep learning based AD methods may be organized into three categories. The first category is based on compression and reconstruction. These methods usually use autoencoder (Hinton & Salakhutdi- nov 2006 Kingma& & Welling, 2013) to learn low-dimensional representation to reconstruct the high-dimensional data (Vincent et al., 2008 Wang et al. 202T). It is expected that the learned au- toencoder on the normal training data a much higher reconstruction error on unknown abnormal data than on normal data. The second category is based on the combination of classical onc-class classification (Tax Duin 2004; Golan & El-ian 2018) and deep learning (Ruff et al. 2018 2019 2020 Perera & Patel 2019 |Bhattachary et al. [021] Shenkar & Wolf [022] Chen et al., 2022). p. For instance, Ruff et al. (2018) proposed a method called deep one-class SVDD. The main idea is to use deep learning to construct minimum-radin hypersphere to include all the training data, while the unknown abnormal data are expected to fall outside. The last category is based on generative learning adversarial learning (Malhotra et al. 2016 Deecke et al. 2018 Pidhorskyi iet al., 20188] Nguyen et al., 2019 Perera et 2019) Goyal et al., 2020; Raghuram et al., [022]] Yan al. 2021). For example, Perera al. (209)) proposed to the generative adversarial net-

---

Under review as a conference paper at ICLR 2023
work (GAN) (Goodfellow et al.] 2014) with constrained latent representation to detect anomalies for image data. Goyal et al. (2020) presented a method called deep robust one-class classification (DROCC). The method aims to find a owwdimensiooal manifold to accommodate the normal data
via an adversarial optimization approach.
Although deep learning AD methods have shown promising performance various datasets, they still have limitations. For instance, the one-class classification methods such as Deep SVDD (Ruff al. 2018) only ensure that the normal data could be included by a hypersphere but cannot guar- antee that the normal data are distributed evenly in the hypersphere, which may lead to large empty regions in the hypersphere and hence yield incorrect decision boundary. The adversarial learning methods such as (Nguyen et al. [0199] Perera et al., [219 Goyal et al. 2020) may suffer from high computational cost and instability in optimization.
In this work, we present restricted generative projection framework for one-class classification and anomaly detection. The model of the framework efficient to train and able to provide reliable decision boundaries for precise anomaly detection. Our main idea is to train deep neural network to convert the distribution normal training data to a target distribution that simple, compact, and informative, which will provide reliable decision boundary to identify abnormal data from normal data. There are many choices for the target distribution, such as truncated Gaussian and uniform on hypersphere. Our contributions three-fold.
We present novel framework for one-class classification and anomaly detection. It aims to transform the data distribution to some target distributions that are easy to be violated by unknown abnormal data.
We present few simple, compact, and informative target distributions and propose min- imize the distances between the converted data distribution and these target distributions via minimizing the maximum mean discrepancy.
We conduct extensive experiments compare the performance of different target distribu- tions and compare our method with state-of-the-art competitors. The numerical results on five benchmark datasets verify the effectiveness of our methods.
2 METHODOLOGY
Suppose we have a set of m-dimensional training data X = {[x1, 22,,,X2.... X2...., X.,) drawn from un- known bounded distribution Dx and any samples drawn from Dx are normal data. We want to train model M on X to determine whether test data Xnew is drawn from Dx or not. One may con- sider to estimate the density function (denoted by Px) of Dx using some techniques such as kernel density estimation (Rosenblatt) 1956). Suppose the estimation Px is good enough, then can determine whether Xnew is normal or not according to the value of (Xnew): if (Xnew) is zero or close to zero, Xnew is abnormal data point; otherwise, Xnew is normal data point [1] However, the dimensionality of the data is often high and hence is very difficult to obtain a good estimation px-
We propose to learn a mapping T R - Rd transform the unknown bounded distribution Dx to a known distribution D, while there still exists a mapping T Rd + IR' that can recover Dx from Dz approximately. Let ps be the density function of D,, Then we can determine whether Xnew is normal or not according to the value of Pz P2(T((x (T(Xo)))) To be more precisely, we want to solve the following problem
minimize M (T((2)), + (T Dx), T,7"
(1)
where M(:,) .) denotes some distance metric between two distributions and À is a trade-off parameter for the two terms. Note that if X 0, T may convert any distribution to Dz and lose the ability of distinguishing normal data and abnormal data. Based on the universal approximation theorems (Pinkus) 1999 Lu et al.) 2017) and substantial success of neural networks, use deep neural networks (DNN) to model 7 and T' respectively. Let So and gs be two DNNs with parameters and é respectively. We solve the following problem
Herr we assume that the distributions of normal data and abnormal data do not overlap. Otherwise, it is difficult determine whether single point is normal or not.
2

---

Under review as a conference paper at ICLR 2023
minimize M (Dio(x), Da) + AM (Poolto()); Dx):
(2)
0,0
where So and De serve encoder and decoder respectively. However, problem is intractable
because Dx is unknown and D sx(()) (xs(f(())) cannot computed analytically. Note that the samples of Dx and D (x(lf((x)) are given and paired. Then the second term in the objective of Q
can be replaced by sample reconstruction error such as n Ei= ||x; (x(((() On the other hand, we can also sample from (x)) and D2 easily but their samples are not paired. Hence, the metric M in the first term the objective of (2) should be able to measure the distance between two distributions using their finite samples. To this end, we propose to use the kernel maximum mean discrepancy (MMD) (rreton et al. 2012) to measure the distance between D)(()) and Dz. In statisties, MMD is often used for -wo-Sample test and its principle is to find function that assumes different expectations on two different distributions:
MMD = sup (Ep]f(x)] Eylf(y))). If
where p. are probability distributions, F is class of functions f : X + R and H. denotes reproducing kernel Hilbert space. Its empirical estimate can be given by
T
m
n
MMD*|F, X,
xx)
+ n(n 1)
kyy.,,))
m mm
1)
(4)
2
k k(x,y,);
773
where = {xI...... Xm) and = (yI...., yn) are samples consisting of i.id observations drawn from and q, respectively. :)) denotes kernel function, e.g., k(x,y) exp(->l|x yl Gaussian kernel.
Based the above analysis, reformulate 2
minimize MMD (Zo, ZT) 8,
x; (o(
(5)
where Zg = (fo(x)). fx ) and ZT {z, 2; ~ Dz, 1.....n). The first of the objective function in (5) makes fo learn the mapping from data distribution Dx to target distribution D2 and the second term ensures that So can preserve the main information of observations provided that À is sufficiently large. Figure I shows the paradigm of our method.
D
X
fo
N MMD ZT)
g
X
Za
2 minimize
n
Figure The training stage of Restricted Generative Projection.
Now we introduce four examples of simple and compact D2 for 5 The four distributions are Gaussian in Hypersphere (GiHS), Uniform in Hypersphere (UiHS), Uniform between Hyperspheres (UbHS), and Uniform on Hypersphere (UoHS). Their 2-dimensional examples are visualized in Figure

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
Figure 2:
Samples (in orange) from 2-D target (bounded) distributions. Plots (a), (b), (c), (d) denote GiHS, UiHS, UbHS, UoHS respectively.
GiHS (Figure a is actually a truncated Gaussian. Suppose we want to draw n samples from GiHS. A simple approach is drawing (1 + p)n samples from a standard d-dimensional Gaussian and discarding the pn samples with larger & norms. The maximum & norm of the remaining n points the radius of the hypersphere. One may also use inverse transform method (Marsaglia (963). We have the following proposition (a simple proof can be found in the appendix).
Proposition 2.1. Suppose Z1. Z2..... Zn are sampled from (0, Ia) independently. Then for any r v d/2 the following inequality holds
22
1(2 d)
IP [11zl
r)
exp
j I....,n.
(6)
2
In the proposition, using union bound, we have
(_1² 72
/d(222 - d)
max 15J5n
S r
2 1 - cxp
2
means a hypersphere of radius r can include all the n samples with probability at least 1 - np, where p = exp(-(r² + /d 2))/2)) On the other hand, according to (6). if we sample n/p samples from (0,, La), the expected number of samples falling into hypersphere of radius is at least n. Thus, suppose we sample n'' samples and remove the - n samples with larger le norms, the expected radius of the smallest hypersphere encasing the remaining n samples is at most ro. where is the solution of the function log(n' /n) r² + /d(2r2 d w.r.r.
UiHS (Figure b)b is hyperball in which all the samples are distributed uniformly. To sample from UiHS, first need to sample from (f(-1, 1)d, Then discard all the data points outsides the radius-1 hyperball centered at the origin. The following proposition shows some probability result of sampling from d-dimensional uniform distribution.
Proposition 2.2. Suppose 21, Z ,, Zn sampled from UI (1-1,111 1) 1)d independently. Then for any r > 0, the following inequality holds
4d
2 22) 5(3r2 ddz2
j 1, -=.....,
Using union bound for
8
obtain
4dn
PP
||, 5r Isi5n
max
>
(9)
5(3r
d)2
It means hypersphere of radius r can include all the samples with probability at least 1 np, where p (3)2 d)2. On the other hand, (8) indicates that if we draw n/p samples from 1) the expected number of samples falling into a hypersphere of radius is at least n. Actually, sampling from UiHS is closely related to the Curse Dimensionality and we need to sample large number of points from U(-1, 1)d if d is large because only small volume of the hypercube is inside the hyperball. To be more precisely, letting Vnypereube be the volume a hypercube with length 2r and Vhyperball be the volume of a hyperball with radius have
Vnyperball
n 4/2
d2d-1T(d/2) n
(10)
V Mhypercube
4

---

Under review as a conference paper at ICLR 2023
where I is the gamma function. Therefore, need to draw n/n samples from (-,, 1)d to ensure that the expected number of samples included the hyperball is n, where r is small is large.
UbHS (Figure c) can be obtained via UiHS. We first sample from UiHS and then remove all samples included by a smaller hypersphere. Since the volume ratio two hyperballs with radius r1 and r2 is (72 )d need to draw n/(1 (r2/r1)") samples from UiHS to ensure that the expected
number of samples between the two hyperspheres is
UoHS (Figure d) can be easily obtained via sampling from (0, Ia). Specifically, for every z; drawn from .V (O Ia), normalize it z, rz:/||zil|, where is the predefined radius of the hypersphere.
In testing stage, only use the trained fø to calculate anomaly scores. For given test sample Xnew, we define anomaly score for each target distribution by
f [Xnow)]l ||2 r/r
for UoHS for GiHB or UiHS for UbHS
s( ((Nnew)
=
f (xow) 22, (Il.f (Xeew) 122 r1) (xxex))))l r2),
(11)
We call our method Restricted Generative Projection (RGP), which has four variants, denoted by RGP-GiHS, RGP-UiHS, RGP-UbHS, and RGP-UoHS respectively, though any target bounded distribution applies.
3 CONNECTION WITH PREVIOUS WORK
Our method has a connection with the variational autoencoder (VAE) (Kingma & Welling 2013). Both methods are autoencoders. The latent distribution VAE often Gaussian and not bounded while the latent distribution in our method is more general and bounded. The optimizations of VAE and our method are also different: VAE involves Kl.-divergence while our method involves MMD.
Our method is closely related to Deep SVDD (Ruff et al. 2018). Both our method and Deep SVDD aim to project the normal training data into some space such that decision boundary between normal data and unknown abnormal data can be found easily. Deep SVDD minimizes the sum of the squared distances between the projected data and a predefined center to find hypersphere to include the normal training data, which cannot ensure that the data in the hypersphere are evenly distributed or close to Gaussian and hence may lead to an ineffective decision boundary.
Compared with the autoencoder based anomaly detection method NAE Yoon et al. 2021 that uses reconstruction error to normalize autoencoder, our method pays more attention to learning a mapping that transform the unknown data distribution into simple and compact target distribution.
Similar to our method, Perera et al. (2019) also considered bounded latent distribution in autoen- coder for anomaly detection. They proposed to train denoising autoencoder with a hyper-cube (multi-dimensional uniform) supported latent space, via adversarial training. Obviously, the latent distribution and optimization of our method are different from theirs. In addition, the latent distri- butions of our method are more compact than the multi-dimensional uniform latent distribution of their method.
EXPERIMENTS
4.1 DATASETS AND BASELINES
In this section, we compare the proposed method with several state-of-the-art anomaly detection methods on three tabular datasets and two widely used image datasets for one-class classification. The datasets are detailed as follows.
Abalone (Dua 2017) is dataset of physical measurements of abalone to predict the age. It contains 1,920 instances with 8 attributes.
Arrhythmia Rayana 2016) is an ECG dataset. It was used to identify arrhythmic samples in five classes and contains 452 instances with 279 attributes.
5

---

Under review as a conference paper at ICLR 2023
Thyroid (Rayana
2016) is hypothyroid disease dataset that contains 3,772 instances with
6 attributes.
Mashion-MNIST Xiao al. 2017
contains 70,000 grey-scale images with 10 classes.
CIFAR-10 Krizhevsky et al. 2009) is a widely-used benchmark for image anomaly de- tection. It contains 60,000 color images with 10 classes.
We compare method with three classic shallow models, four deep autoencoder based methods, three deep generative model based methods, and some latest anomaly detection methods.
Classic shallow models: local outlier factor( [reunng ct al. 2000 one-class support
vector (OChin))DC-SVM) (Schölkopf al. 2001 isolation forest (IF)
Liu al.
2008
Deep autoencoder based methods: denoising auto-encoder (DAE)
Vincent et al.
2008
DCAE (Seeböck et al. 2016 E2E-AE and DAGMM (Zong al.
2018). DCN
Caron
et al.
2018).
Deep generative model based methods: Ano et al. 2017
ADGAN Deecke
et al.
2018). OCGAN (Perera et al. 2019).
Some latest anomaly detection methods: DeepSVDD(Ruff et al.
2018
GOAD SCADN
(Bergman & Hoshen 2020), DROCC (Goyal et al. 2020), HRN (Hu et al.
2020
Yaan et 2021). NeuTraL AD (Qiu et al. 2021)) GOCC (henkarr & Wolf 2022)
4.2 IMPLEMENTATION DETAILS AND EVALUATION METRICS
In this section, introduce the implementation details of the proposed method RGP and describe experimental settings for image and tabular datasets. Note that all the compared methods do not utilize any pre-trained feature extractors.
For the three tabular datasets (Abalone, Arrhythmia, and Thyroid), in our method, fo, Ir are both MLPs. We follow the dataset preparation of (Zong et al. 2018) preprocess the three tabular datasets for one-class classification task. The hyper-parameter À is set to 1.0 for the three datasets. For the two image datasets (FaShion-MNIST and CIFAR-IO), in our method, Sa. Jo both CNNs. Since both image datasets contain 10 different classes, we conduct 10 independent one-class clas- sification tasks on both datasets. In each task on CIFAR-I0, there are 5,000 training samples and 10,000 testing samples. In each task on Fashion-MNIST, there are 6,000 training samples and 10,000 testing samples. The hyper-paramesn À is chosen from (0.05, 0.2, 0.1, 1.0) and varies for different classes.
In our method, regarding the restricted target distributions GiHS, we first generate large number (denoted by N) of samples from Gaussian or uniform, sort the samples according to their f2 norms, and set r to be the pN-th smallest & norm, where p 0.9. each iteration (mini-batch) of the optimization, to GiHS and hence omitted here for simplicity. For UoHS, we draw samples from Gaussian and normalize them to have unit lg norm, then they lie unit hypersphere uniformly. The procedure is repeated in each iteration mini--batch) of the optimization. We use Adam (Kingma & Ba 2014)) as the optimizer in our method. For Fashion-MNIST, CIFAR- 10, and Arrhythmia, the learning rate is set to 0. 0.0001. For Abalone and Thyroid, the learning rate is set to 0.001. The details of our network settings are provided in the supplementary material. All experiments were run on AMD EPYC CPU with 64 cores and with NVIDIA Tesla A GPU, CUDA 1 1....
resample Zr according to The sampling for UiHS and UbHS similar
To evaluate the performance of all methods, we follow the previous works such as (Ruff 2018) and (Zong et al., 2018) to use AUC (Area Under the ROC curve) for image datasets and F1 -ccore for tabular datasets.
4.3 RESULTS IMAGE DATASETS
Table I and Table 2 compare RGP under four different settings with classic shallow methods, deep autoencoder based methods, generative model based methods, and some latest state-of- of-e-e--rr-trr---- meth- ods. We have the following observations.

---

Under review as a conference paper at ICLR 2023
Firstly, in contrast to classic shallow methods such as DC-SVM((chölkopp al. 2001 and IF (Liu et al. 2008). our RGP has significantly higher AUC scores on all classes of CIFAR-I0 and most classes of Fashion-MNIST. An interesting phenomenon is that most learning based methods including ours have inferior performance compared to IF (Liu ctal._2000 on class 'Sandal" of Fashion-MNIST.
Our methods outperformed the deep autoencoder based methods and generative model based methods in most cases and have competitive performance compared to the state- of-the-art in all cases.
RGP has superior performance on most classes of Fashion-MNIST and CIFAR-10 datasets under the setting of UoHS(uniform distribution on hypersphere), Furthermore, our four settings have relatively close performance on Fashion-MNIST. On CIFAR-10, UoHS out- performed other settings consistently.
Table I: Average AUC(%) of one-class anomaly detection on Fashion-MNIST. For the competitive methods we only report their mean performance due to the space limit, while we further report the standard deviation for the proposed methods. denotes we run the official released code to obtain the results, and the best two results are marked in bold.
Ankle boot
Normal Class
Tishirt
Tronser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
OC.SVM (Scholkopf et al. 2001
86.10
93,90
85.60
85,90
84.60
81.30
78.60
97.60
79.50
97,80
(Liu 2008
91.00
97.80
87.20
93.20
90.50
93.00
80.20
98.20
88.70
95.40
DAE Vileent et 2008
86.70
97.80
80.80
91.40
86.50
92.10
73.80
97.70
78.20
96.30
DACMM Zung el al
2018
42.10
55.10
50.40
57.00
26.90
70.50
48.30
83.50
49.90
34,00
ADGAN Disekk el
2013
89.90
81,90
87.60
91.20
86.50
89.60
74.30
97.20
89.00
97.10
OCGAN (epera et al. 2019
85.50
93.40
85.00
88.10
85,80
88,50
77.50
93.90
82.70
97,80
DeepSVDiD [Kolf
2018
79.10
94.00
83.00
82,90
87.00
80,30
74.90
94.20
79,10
93.20
DROCC" [Gogatet cl.
2720
88.32
97.94
87.31
87.89
86.53
91. 80
77. 64
95.37
81.35
94,75
HRN (Hu 20XU)
92.70
98.50
88,50
93.10
92.10
91.30
80
99.00
94.60
98.80
RGP CGPP-GITS(CC (Ours)
93.37
97.37
89.03
92.63
92.41
87.37
79.42
98.09
91.36
96.86
(0.70)
(0.20)
(0.02)
(0.31)
(0.11)
(2.38)
(0.86)
(0.54)
(0.16)
(0.03)
RGP-UiHS (Ours)
90.49
97.19
90.88
91.26
91.26
89.37
79.97
97.55
92.02
97.82
(0.09)
(0.29)
(0.18)
(0.72)
(0.10)
(0.12)
(0.15)
(0.06)
(1.11)
(0.36)
RGP-UbHS (Ours)
91.10
97.20
89,90
92.29
91.35
82.33
78.59
96.98
91.23
96.55
(0.28)
(0.15)
(0.21)
(0.15)
(0.31)
(5.27)
(1.53)
(0.62)
(1.02)
(1.12)
RGP-UoIIS (Ouns)
97.23
90.73
85.09
80.02
98.54
91.14
97.99
(0.41)
(0.15)
(0.32)
(0.35)
(0.40)
(1.15)
(1.05)
(0.18)
(0.26)
(0.29)
Table 2: Average AUC(%) of one-class anomaly detection on 110 For the competitive meth-
ods we only report their mean performance due to the space limit, while we further report the stan- dard deviation for the proposed method. denotes we run the official released code to obtain the results, and the best two results are marked in bold.
Normal Class
Airplane
Auta- mobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Trunk
OC-SVM (Scholkopf et al. 2001
61.10
63.80
50.00
55.90
66.00
62.40
74,70
62.60
74.90
75.90
[Lu
2008
66.10
43.70
74.30
52.30
70.70
$3.00
53.20
DCAE Seeback et al 2016
59.10
57.40
48.90
58.40
54.00
62,20
51.20
58.60
76,80
67.30
DAE Vingenl a. 2008
41.10
47.80
61.60
56.20
72.80
51.30
68.80
49.70
48.70
37.80
DAGMMT
2018
41.40
57.10
53.80
51,20
52.20
49.30
64.90
55.30
51,90
54,20
AmoGAN- Schlogi et st 2013
67.10
54.70
$2.90
$4.50
65.10
60.30
58.50
62.50
75.80
66.50
ADGAN Deecke el
63.20
52,90
60.70
65.90
61.10
63.00
74.40
64,20
OCGAN (orera
2019
75.70
53.10
64.00
62.00
72.30
62,00
72.30
57.50
82.00
55.40
DeepSVDD [Kull
2018
61.70
65.90
50.80
59.10
60.90
65.70
67.70
67.30
75.90
73.10
DROCC* Gogutet
2020
80.10
73.41
68.78
63.36
70.81
65.01
68.83
71.13
63.81
75.49
HRN (Hu et al. 4020
77,30
69.90
60.60
64.40
71.50
67.40
77.40
64.90
82.50
77.30
71.88
63.69
$7.06
$9.12
67.73
61.29
74.52
63.61
76.40
73.36
RGP-GiHS (Ours)
(0.24)
(0.65)
(0.36)
(1.86)
(1.69)
(1.19)
(0.60)
(0.82)
(0.41)
(0.58)
69.83
64.17
63.69
59.55
67.41
67.90
77.34
64.01
78.73
75.08
RGP-UiHS (Ours)
(0.47)
(1.17)
(2.03)
(1.49)
(0.90)
(1.06)
(0.98)
(1.13)
(0.11)
(1.36)
RGP-UbHS (Ours)
66.96
66.27
61.00
$7.34
68.47
67.29
76.33
63.87
77.13
72.63
(4.14)
(0.75)
(0.09)
(2.25)
(1.23)
(0.06)
(0.74)
(0.02)
(0.29)
(0.12)
RGP-UoHS (Ours)
79.30
68.05
62.93
64.64
73.85
68.32
80.08
66.29
82.23
77.56
(0.50)
(0.02)
(0.96)
(0.05)
(0.09)
(0.20)
(0.32)
(0.19)
(0.21)
(0.74)
4.4 RESULTS ON TABULAR DATASETS
In Table 3 we report the Fl-scores of our methods in comparison to ten baselines on Arbythhia,
Thyroid, and Abalone. Our four methods significantly outperform all baseline methods in all cases.
7

---

Under review as a conference paper at ICLR 2023
Particularly, RGP-UoHS has 18.73%, 8.70% and 16.96% improvements the three datasets in terms of Fi-score compared to the runner-up, respectively. Compared with the performance im- provement of RGPs on Fashion-MNIST and CIFAR- 10, RGPs the three tabular datasets are more significant.
In addition the quantitative results, choose Thyroid (with 6 attributes) transform the data distribution to 2-dimensional target distributions, which are visualized in Figure We that RGPs are effective transform the data distribution the restricted target distributions. Therefore, RGPs can learn clear boundaries for normal data and hence have satisfactory performance in anomaly detection.
example and
Table 3: Average F1- F1-Scores(%) (6) with standard deviation three tabular datasets." denotes we run the official released code NeuTral AD to obtain the result of Abalone, and the results of Arrhyth- mia and Thyroid are from the original paper (Qiu et al. 202 The best two results are marked in
bold.
Methods
Abalone
Anhythhma
Thyroid
OC-SVM (Schölkopf et al.
2001)
48.00 + 0.00
46.00 +-0.00
39.00
+1.00
LOF (Breunig al. 2000)
33.00 +1.00
51.00 +1.00
54.00
+1.00
DCN (Caron et al. 2018)
40.00 #1.00
38.00 #3.00
33,00 =3.00
E2E-AE Zong el al. 2018)
33.00 #3.00
#3.00
13.00 #4.00
DAGMM Zong et al. 2018
20.00 +3.00
49.00 #3.00
49.00 =4.00
DeepSVDD (Ruff al.) 2018)
62.00 #1.00
54.00
73.00 =0.00
GoAD (Bergman & Hoshen 2020
61.00 #2.00
51.00 +2.00
72.00 1.00
DROCC Goyal et al. 2020
68.00 2.00
69.00 2.00
78.00 3.00
NeuTral AD Qruet al.
2021
62.07 2.81
60.30 1.10
76.80 = 1.90
GOCC Shenkar & Wolf
2022
61.80 + 1.80
76.80 = 1.20
RGP-GiHS (Ours)
84.79 + 0.36
78.63 + 0.55
92.60 + 0.37
RGP-UiHS (Ours)
84.79 = 0.65
76.98 0.72
91.83 + 0.39
RGP-UbHS (Ours)
83.79 = 0.81
74.22 + 0.78
89.53 + 0.00
RGP-UoHS (Ours)
86.73 = 1.10
77.70 + 0.40
94.96 0.28
-5
0.5
0.0-
-15
-25-
-4
-1.5
-5.0
-2.5
0.0
2.5
5.0
-20
-10
-30
-2.
0n
(a)
(c)
0.5
-0.5
-10
-15
-10
(b)
(d)
Figure 3: Visualization of mapping data distribution to 2-dimensional target distribution on Thyroid datasets. Plots (a), (b), (c), (d) refer to GiHS, UiHS, UbHS, UoHS, respectively. The blue points, orange points, green points, red points denote samples from target distribution, samples from train- ing data, normal samples from testing set and abnormal samples from testing respectively.
4.5 ABLATION STUDY
There is one hyperparameter in our method, namely À in problem (5). Now we show the influence of X the performance of our method. Figure shows F1-scores of our methods with À varying
8

---

Under review as a conference paper at ICLR 2023
from 0 to 100, the three tabular datasets. Too small or too large can lower the performance of RGP. When À is very tiny, the reconstruction term of (5) makes less impact on the training target and fo can easily transform the training data to the target distribution but ignores the importance of original data distribution (see Figure 5). On the other hand, when is very large, the MMD term of optimization objective becomes trivial for the whole training target and fo under the constraint of reconstruction term more concentrates on the original data distribution yet can not learn a good mapping from data distribution to target distribution. Figure 5 illlustras the influence of hyper- parameter À on training set of Thyroid dataset. It can be clearly observed that fo transform training data to target distribution better with the decrease of the X.
Thyrid
Thynic
Brysd
Thyed
F1-score(%)
12
F1 F1-scare(%%
83-
Artryhin
F1 F1-score (%6
66-
90-
60
10
0.1
05.1
5 10
179
83
03
5
100
6.1
6.5
5.10
100
05.
5.30
100
(a)
(c)
(d)
Figure 4: The ablation study of hyper-paaamette À on testing set of three tabular datasets under four different restrictions. Plots (a), (b), (c), (d) correspond to GiHS, UiHS, UbHS, UoHIB, respectively. is chosen from {0, 0.1, 0.5, 1, 5, 10, 100}
2 = 0.1
À = 0.01
À = 0.001
0.1
À 0.01
A 0.001
2
2
1
-2
2
-
1
-1 0
-1
1
a
(c)
1 0.1
À 0.01
À 0.001
A = 0.1
A = 0.01
À = 0.001
0
0
0
1-
1
1
1
(d)
Figure 5: The ablation study of hyper-parameter À on training set of Thyroid dataset under four different restrictions. Plots (a), (b), (c), (d) correspond to GiHS, UiHS. UbHS, UoHS, respectively.
CONCLUSION
We have presented a novel and simple framework for one-class classification and anomaly detection. Our method RGP aims to convert the data distribution to a simple, compact and informative target distribution that can be easily violated by abnormal data. We presented four target distributions and the numerical results showed that uniform hypersphere is more effective than other distributions in anomaly detection. The reason is that the uniform on hypersphere is much more compact than other distributions and hence provides smaller room for abnormal data points to fall into.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space autoregression for novelty detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 481-490, 2019.
Liron Bergman and Yedid Hoshen. Classification-basee anomaly detection for general data. In Proceedings of the International Conference on Learning Representations, 2020.
Arindam Bhattacharya, Sumanth Varambally, Amitabha Bagchi, and Srikanta Bedathur. Fast one- class classification using class boundary-preserving random projections. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 66-74, 2021.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, pp. 93-104, 2000.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision, pp. 132-149, 2018.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey, ACM Com- puling Surveys (CSUR), 41 2009.
Yuanhong Chen, Yu Tian, Guansong Pang, and Gustavo Carneiro. Deep one-class classification via interpolated gaussian descriptor. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In Joint European Conference Ma- chine Learning and Knowledge Discovery in Databases, pp. 3-17. Springer, 2018
Graff C. Dua, D. Uci machine learning repository, 2017. URL http::/ ics.icc... uci. ledu/mm
Sarah M Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and large-scale anomaly detection using linear one-class svm with deep learning. Pattern Recognition, 58:121-134, 2016.
Izhak Golan and Ran El- Yaniy. Deep anomaly detection using geometric transformations. Advances in neural information processing systems, 31, 2018.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- fion Processing Systems, 27, 2014.
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateck Jain. Droce: Deep robust one-class classification. In Proceedings of the International Conference Machine Learning, pp. 3711 1-3721. PMLR, 2020.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research. 13(1):723-773, 2012.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 1812. 2.04606, 2018.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(57866:504-557, 2006.
Wenpeng Hu, Mengyu Wang. Qi Qin, Jinwen Ma, and Bing Liu. Hm:: A holistic approach to one class learning. Advances Neural Information Processing Systems, 33:19111-19124, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 22820,99, 2014.
10

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. arXiv113/2.114,222 2013.
Auto-encoding variational bayes. arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selec- tion. Annals of Statistics, pp. 1302-1338, 2000.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In Proceedings of the IEEE International Conference on Data Mining, pp. 413-422. IEEE, 2008.
Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus- Robert Müller. Explainable deep one-class classification. arXiv preprint arXiv:2007.00 0600, 2020.
Zhou Lu, Hongming Pu, Feicheng Wang. Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 6232-6240, 2017.
Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. Lstm-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00144 2016.
George Marsaglia. Generating variable from the tail of the normal distribution. Technical report, BOEING SCIENTIFIC RESEARCH LABS SEATTLE WA, 1963.
Duc Tam Nguyen, Zhongyu Lou, Michael Klar, and Thomas Brox. Anomaly detection with multiple-hypotheses predictions. In Proceedings ohe the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4800 PMLR, 09- 15 Jun 2019.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Computing Surveys (CSUR), 54(2):1-38, 2021.
Pramuditha Perera and Vishal M Patel. Learning deep features for one-class classification. IEEE
Transactions on Image Processing, 28(11):5450-546) 2019.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 2898-2906, 2019.
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. Advances Neural Information Processing Systems, 31, 2018.
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143 195, 1999.
Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. Neural transforma- tion learning for deep anomaly detection beyond images. In Proceedings of the International Conference on Machine Learning, pp. 8703-8711 PMLR, 2021.
Jayaram Raghuram, Varun Chandrasekaran, Somesh Jha, and Suman Banerjee. A general frame- work for detecting anomalous inputs to dnn classifiers. In Proceedings of the International Con- ference Machine Learning, pp. 8764 8775. PMLR, 2021.
Rayana. Odds library,, 2016. URL httl:tt/////de cs. stonybrook.ed..
Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. The annals of mathematical statistics, pp. 832-837, 1956.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In Proceedings of the International Conference Machine Learning, pp. 4393 4402. PMLR, 2018.
11

---

Under review as a conference paper at ICLR 2023
Lukas Ruff, Robert A Vandermeulen, Nico Görmitz, Alexander Binder, Emmanuel Müller, Klaus- Robert Müller, and Marius Kloft. Deep supervise anomaly detection. In Proceedings of the International Conference Learning Representations. 2019.
Lukas Ruff, Robert Vandermeulen, Billy Joe Franks, Klaus-Rober Müller, and Marius Kloft. Rethinking assumptions in deep anomaly detection. arXiv preprint vrxiv:2006.00339 2020.
Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus- Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 2021.
Thomas Schlegl, Philipp Seehöck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference information processing in medical imaging, pp. 146- 157. Springer, 2017.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe-Taylor and John Platt. Support vector method for novelty detection. Advances in Neural Information Processing Systems, 12, 1999.
Bernhard Schölkopf. John C Platt, John Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of a high-dimensional distribution. Neural Ccomputation, 13(7))1443-
2001.
Philipp Seeböck, Sebastian Waldstein, Sophie Klimscha, Bianca S Gerendas, René Donner, Thomas Schlegl, Ursula Schmidt-Erfurth, and Georg Langs. Identifying and categorizing anomalies in retinal imaging data. arXiv preprint arXiv: 1ol:: 00686, 2016.
Tom Shenkar and Lior Wolf. Anomaly detection for tabular data with internal contrastive learning- In Proceedings of the International Conference on Learning Representations, 2022.
David MJ Tax and Robert PW Duin. Support vector data description. Machine Learning, 54(1): 45 66, 2004.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, pp. 1096-1103, 2008.
Jingjing Wang, Sun Sun, and Yaoliang Yu. Multivariate triangular quantile maps for novelty detec- tion. Advances in Neural Information Processing Systems, 32:5060-5071, 2019.
Shaoyu Wang, Xinyu Wang, Liangpei Zhang, and Yanfci Zhong. Auto-ad: Autonomous hyperspec- tral anomaly detection network based fully convolutional autoencoder. IEEE Transactions Geoscience and Remote Sensing, 60:1-14, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Xudong Yan, Huaidong Zhang, Xuemiao Xu, Xiaowei Hu, and Pheng-Ann Heng. Learning semantic context from normal samples for unsupervised anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3110-3118, 2021.
Sangwoong Yoon, Yung- Noh, and Frank Park. Autoencoding under normalization constraints.
In International Conference on Machine Learning, pp. 12087-12097. PMLR, 2021.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Pro- ceedings of the International Conference on Learning Representations, 2018.
12

---

Under review as a conference paper at ICLR 2023
A MORE NUMERICAL RESULTS
In addition to the results of Tables, we here report the average AUC of all classes in F We see that on Fashion-MNIST, our methods have competitive performance as IF and HRN. On CIFAR-10. our RGP-UoHS outperformed all other methods.
Average AUC(%)
Average AUC(%)
50-
40
20
OCSO4N
DAE
Datana
ADGAN
OCTAN
Dapavou
Daarc
NoTass
RoPonn
SC-S0M
DAE
ADRAH
occaal
Depsyon
SCADN
niRuse
RoPoon
RGC
Fashion-MNIST
CIFAR-10
Figure 6: The Average AUC(%) score of Fashion-MNIST and CIFAR-10 10 on all classes.
We also conduct one-class anomaly detection MNIST, and report experimental results in Table
Table 4: Average AUC(%) of one-class anomaly detection on MNIST.
Normal Class
3
5
7
9
Avg
OC-SVM (Shookkopf et al. 2001]
98.60
99.50
82.50
88. 18.10
94.90
77.10
96.50
93.70
88.90
93. 10
91.28
(Luu
98.00
97.30
88.60
89.90
92.70
85.50
95.60
92.00
93.50
92.29
DCAE Sectock el al. 2016
97.60
98.30
85.40
86.70
86.50
78.20
94.60
92.30
86.50
90:40
89.64
ANOGAN Deeckee
96.60
99.20
85.00
85.70
85.30
93.50
91.27
Deep SVDD Kull et al 2015
98.00
99.70
91.70
91.90
94,90
88.50
98.30
94.60
93.90
96.50
94.79
RGP-UOHS (Ours)
99.50
98.90
94.81
94.89
95.55
92.05
97.41
96.39
90.95
96.25
95.66
(0.07)
(0.27)
(0.59)
(0.28)
(0.32)
(1.75)
(0.14)
(0.21)
(0.35)
(0.24)
In addition to comparison of evaluation metrics, we also measure the time cost of different types of anomaly detection methods on training phase with GUP acceleration or not. We run the experiments of one-class classification on MNIST, Fashion-MNISTT, CIFATI0 and report the time cost in Table 6 7 Table 5 compares training time of Deep SVDD (Ruff et al. 2018). DROCC (Goyal et al., 2020). OCGAN (Perera et al.] 2219)) RGP on MNIST with the same batch size and training epochs, and without GPU acceleration. Table and Table 7 show the training time of Deep SVDD (Ruff et al. 2018). DROCC (Goyal et al. 2020), RGP on Fashion-MNIST.. CIFAR10 with accelerating of NVIDIA RTX2080 GPU(1x). Table 6 and magnitude on time cost with GPU acceleration and our method has few advantage
that the three methods have the same
both Fashion- MNIST and CIFARIO when ensuring the convergence of model. The training hyperparameters of Deep SVDD (Ruff et al. 2018) and DROCC (Goyal et al. 2020) in table 6 and table 7 are both from the official settings and listed in Table [8] We do not obtain the time cost OCGAN (Perera et al. 2019) with GPU acceleration since the official code occurs system error when using GPU.
13

---

Under review as a conference paper at ICLR 2023
Table 5: Training time(seconds) of one-class anomaly detection MNIST without GPU accelera-
[ion.
Nommal Class
0
Avg
Daag SVDD Rull al
2018
1187.16 16
1254.45
1083.25
1150,95
1076.51
1021.4
1113.29
1152.54
1109.05 05
1097.22
1120.55
OCGAN Poorte
18050. 79
20400109
17851.42
18579. 87
17757.39
16850.2
17748. 48
19553.55
17759.54
17751.16
18259 24
DROCC Mappler 10207
21269 80
21134.64
22030.75
21157 20
22269. 17
21159.06
21270. 14
21014.52
20953.55
21:36644
21359. 59
RGP-GHFT (Ours)
13665 78
15 69
13227.82
12006. 53
1:697.85
11334.88
11881 15
12789 74
13086.9
11646.02
12642.8
RGP-LAIS (Oers)
14986 82
16808.55 84
12177.22
12136.28
12322.70 76
11020 81
12345 14
11726. 78
11858.33
12115.30 30
12759. 73
RGP UbJIS (Ouns
11766.77 71
13301.99
12132.35
12261.36
11678.56
10825.82
13657.7
15064.25
13865.64
14083.29 29
12863.60
RGP Un]18 (Ouns)
14020.15 54
16066.29
14028.68
18276.65
13153.98
12812.56
14189.72
15797.74
13891994
14330.59
14416.83
Table 6: Training time(seconds) of one-class anomaly detection on Fashion-MNIST with GPU ac- celeration.
Normal Class
Tishirt
Trouser
Pullover
Dress
Cout
Sandal
Shirt
Sneaker
Bag
boot
Avg
SVDD (Ruff 2018
563.46
561.48
566.92
561.12
568.27
557.55
563.02
559.02
564.22
560..4
562.24
DROCC
Goyal et 2020
510.17
507.88
507.76
506.62
504.71
503.79
503.99
502.18
505.85
504.14
505.70
RGP-GiNS GiNS (Ours)
499.40
499.77
499.53
499.10
498.65
500.20
498.58
499.98
500.78
496.23
199.22
(Ours)
499.91
499.31
499
499.94
RGP-UbHS (Ours)
499.64
499.69
499.95
500.08
499 66
499.35
500 11
499.7
59
500.39
499.91
RGP-1ioHS (Ours)
484.26
487.74
490.22
489.46
486.55
485.00
484.80
484.79
485.71
485.45
486.39
Table 7: Training time(seconds) of one-class anomaly detection CIFARIO with GPU accelera- tion.
Normal Class
Airplane
Auto-
Bird
Cat
Deer
Frog
Horse
Trunk
Avg
Deep SVDD Ruff al 2018
943.97
946.16
941.57
932.88
944.01
943.51
942.04
942.93
941.69
941.69
942.04
DROCC (Goyar 20200
970.90
976.35
974.08
971.07
972.03
971.09
974.67
971.60
972.54
972.63
RGP-GIHS (Ours)
747.48
748.08
747.83
748.57
748. 34
747.33
747.33
746.74
747.48
746.03
747.52
RGP-UHHS (Oun)
746.95
747.17
747.51
746.31
347.72
747,80
747.58
747.78
747.80
747.36
747.39
RGP-UbHS (Ours)
746.12
746.11
745.21
747.21
746.97
746.08
745.06
746.01
746.49
745.45
746.07
RGP-UoHS (Ours)
713.25
712.59
712.70
712.61
712.97
712,86
712.59
713.02
712.72
712.77
712.80
Table 8:
Training settings of one-class anomaly detection on Fashion-MNIST, CIFAR10 and
MNIST.
Methods
Dataset
batch size
training
Deep SVDD (Ruff 2018)
MNIST
200
100(epochs)
DROCC Goyal et 2020)
MNIST
200
100(epochs)
OCGAN Perera el al.
2019
MNIST
200
100(epochs)
RGP-GiHS (Ours)
MNIST
200
100(epochs)
RGP-UiHS (Ours)
MNIST
200
100(epochs)
RGP-UbHS (Ours)
MNIST
200
100(epochs)
RGP-UoHS (Ours)
MNIST
200
100(epochs)
Deep SVDD (Ruff 2018
Fashion-MNISS MNIST
50
150(epochs)
DROCC (Goyal et al. 2020
Fashion -NNIST
256
100(epochs)
RGP-GiHS (Ur-Gs
200
100(epochs)
RGP-UiHS (Ours)
Fashion-MNISS
200
100(epochs)
RGP-UbHS (Ours)
Fashion-MNIST
200
100(epochs)
RGP-UoHS (Ours)
Pashion-MNIST
200
100(epochs)
Deep SVDD (Ruff
2018
CIFARIO
50
150(epochs)
DROCC (Goyal et al. 2020)
CIFARIO
256
100(epochs)
RGP-GiHS (Ours)
CIFARI0
200
200(epochs)
RGP-UiHS (Ours)
CIFARIO
200
200(epochs)
RGP-UbHS (Ours)
CIFARIO
200
200(epochs)
RGP-UoHS (Ours)
CIFARIO
200
200(epochs)
B PROOF FOR PROPOSITION 2.1
Proof. Letting 21. 2........ Zn be i..d Gaussian variables with mean 0 and variance 1. According
to Lemma of (Laurent Massart 2000) (in which letting ai ap 1), the following
14

---

Under review as a conference paper at ICLR 2023
inequality holds for any
4 2 d 2v 21 S exp((-t).
(12)
Letting (21, 22,.... =a) and p2 = d + 2 + 21, we have
12 v d( d)
[[|z] 2 r) S exp
(13)
where d < 2r2, This finished the proof.
C PROOF FOR PROPOSITION 2.2
Proof. Letting 21, 22,...; Zd be i.i.d variables of (1-1,1), 1). Then E(2,) and Var(z;)
4
i -1,2.... n. Using Chebyshev's 's Inequality, for any t >> 0, we obtain
d
4d
P
2 t
(14)
3
45
follows that
4d
P
2
+
(15)
4512
Letting ,2 = + t. we have
P
2)) 5
j = 1,2,
(16)
5(3r - d)2
Note that one may obtain tighter tail bound than [66
15

---

Under review as a conference paper at ICLR 2023
D DETAILED NEURAL NETWORKS ARCHITECTURE
For the two image datasets (Fashion-MNIST and CIFAR-10), in our method, fo. Io are both CNNs. The detailed neural network architecture showed in Table g
fa in.channels=1, mid mid_dimm=228, if Fashion- in.channels=3, mid__imm= 1024, if CIFAR-10.
Basic-Block(in brbockii.cchannlls. channelllss))))
serii-Bloocki..ccanneeesses out_channels=128)
Bea-BBloccincccannel----l out_channel==256
paaiicBloockin.chaneeee out.chameels=512) AveragePool2d
Linear(mid dim, bias=False)
96
Basic- Block((nnchannelly out_channels=512)
Upsample(xcalc_ factor-2, moooe='neares/
Diae-BBoockin.ccaaees---- ou._chaneels=256)
Upsample(scale faceorr2,. modee'nearest' )
Basic-Block(in channels=256, out channels=128)
Upsample(scale factor=2, mode='nearsst))
Basie-Block(in.ccaannee out_channell=32)
out_channels=644
Basic Biwockkkiinncchhnee---44444
Conv2d 32a. out.channcls=11. kemeL padding-1, bias=False)
BatchNorm22(16), LeakyReLU
Conv2d Con channels=32, wwt.channelsinn.channee kernel padding=1,
paddiingg bias=False)
Bwl.Bllicccccanneeee out_channels)
Conv2d(in channels, out channels, kernel size=3, padding=l, bias=False)
Baachoorm2dooutcchanneee LeakyReI.U
Conv2d(out_channeee out_channels, kemnl- padding=1, bias=False) BatchNorm2d(out channels))
Table 9: Architecture of the CNN-based neural network for CIFAR-I0 and Fashion-MNIST.
For the three tabular datasets (Abalone, Arrhythmia, Thyroid), in our method, fe, 9o are both MLPs. The detailed neural network architecture showed in Table 10
So
96 Linear( 128, input dim, bias=False)
Linear(input dim, 64, bias=False), LeakyyRlL Linear(64, 128, bias=False)
Table 10: Architecture of the ML.--aased neural network for tabular dataset.
16


Reviews:
[{"review_id": "v7HIlCed1R", "reviewer": "Reviewer_sjwv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a method for anomaly detection problems. To this end, the original data samples are transformed into a new space that has the desired target data distribution in the transformed space. The resulting loss function attempts to minimize both the learned distribution and the target distribution as well as the reconstruction errors of the original data samples. MLPs (multi-layer perceptron) are used to learn the transformation. The Gaussian in Hypersphere, Uniform hypersphere, Uniform between hyperspheres, and Uniform on a hypersphere models are used target distributions. The proposed method bears similarity to  auto-encoder networks (since exact architectures are used for learning) and Deep SVDD method since compact hypersphere are used for modeling target distributions. Therefore, the novelty is limited. The method is tested on several easy datasets and better accuracies are reported. However, the method is not compared to recent state-of-the-art methods. Therefore, comparisons are not fair.", "summary_of_the_review": "The paper contribution is weak since there exists similar methods and there is no experimental evidence to demonstrate the superiority of the proposed method. More precisely, the results are inferior to the Deep SVDD variants that use compact hypersphere models for approximating normal data distributions as in the proposed method. Also, motivation for using other target distributions is not given. Therefore, my recommendation will be rejection. ", "strength_and_weaknesses": "The main strengths of the paper can be summarized as follows:\ni) In general, the paper is written well despite some minor errors.\nii) The authors propose some minor novelties over the existing methods in the literature.\niii) Better accuracies are obtained compared to some baseline methods.\n\nThe main weaknesses of the paper can be summarized as follows:\ni) The novelty is very limited since the proposed method has close ties to the existing methods. More formally, it is very similar to auto-encoder networks in terms of architecture and loss function. Furthermore, compact hyperspheres and related distributions are used for target distribution, and these models are already used in Deep SVDD and its variants.\nii) Compact hypersphere models make sense for target distribution and it is largely used for anomaly detection. However, what is the motivation for using other target distributions such as Uniform between hyperspheres. What are the advantages?\niii) Experimental evaluation is quite weak and biased. The authors must compare their methods to the recent ones. Especially, Deep SVDD variants yield much higher accuracies. Please see the new references below. For example, the current SOTA AUC score is 96% on Cifar-10, and 93% on fashion Mnist datasets. The authors’ reported accuracies are much lower. Also, why do not the authors conduct tests on Mnist dataset (used in the most of the anomaly detection papers) and KDD and KDDNew datasets (used in Qiu, et al., Neural transformation learning for deep anomaly detection beyond images. In Proceedings of the International\nConference on Machine Learning, 2021)?\nMinor Issues:\n1) Please use inscribe or include instead of encase.\n2) It seems there is typo in the first contribution. “… that are easy to be violated” is negative thing. It should be changed as “that are not easy to be evaluated”.\n3) There is a mismatch between the equations (1) and (2). D_z should be changed as T(D_x)) in the first equation or the second equation must be corrected accordingly.\n\nReferences:\n[1] D. Hendrycks, M. Mazeika, T. Dietterich, Deep anomaly detection with outlier exposure, in: International Conference on Learning and Recognition (ICLR), 2019.\n[2] P. Liznerski, L. Ruff, R. A. Vandermeulen, B. J. Franks, M. Kloft, K.-R. Muller, Explainable deep one-class classification, in: International Conference on Learning and Recognition (ICLR), 2021.\n[3] L. Ruff, R. A. Vandermeulen, B. J. Franks, K.-R. Muller, M. Kloft, Rethinking assumptions in deep anomaly detection, in: International Conference on Machine Learning Workshops, 2021.\n[4] L. Ruff, R. A. Vandermeulen, N. Gornitz, A. Binder, E. Muller, K.-R. Muller, M. Kloft, Deep semi-supervised anomaly detection, in: International Conference on Learning and Recognition (ICLR), 2020. \n[5] I. Golan, R. El-Yaniv, Deep anomaly detection using geometric transformations, in: NeurIPS, 2018.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "bTyxS1Fklz", "reviewer": "Reviewer_23uq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an outlier detection algorithm that is based on bidirectional mapping between the data space and the latent space. The proposed algorithm maps the normal data to a restricted region in the latent space, letting the outliers map to the outside of the region.", "summary_of_the_review": "The paper is generally clear and sound. The presented results seem reproducible. However, I do have a concern regarding its novelty, where I expect some comment from the authors.", "strength_and_weaknesses": "# Strength\n\n* The paper is overall very clearly written. The mathematical notations are easy to follow. The key concepts, including the definition of outlier and the definition of Maximum Mean Discrepancy, are explained in detail.\n* The problem this paper addresses is relevant to the machine learning community.\n* The proposed algorithm shows promising empirical results.\n\n# Weaknesses\n\n* My biggest concern is that the proposed method looks significantly similar to Adversarial Autoencoder (AAE; Makhzani et al., 2015 ; https://arxiv.org/abs/1511.05644 ). AAE also uses MMD loss to encourage the mapped normal data to follow a specific distribution. Although the original AAE paper did not explore the outlier detection application, there are other papers that applied AAE to outlier detection tasks, such as [this](https://arxiv.org/abs/1902.06924).  Could you elaborate on the differences between the proposed method and AAE?\n\n* How can we ensure that an outlier sample is always mapped to the outside of the designated region and does not fall into the inlier region in the latent space? Since the data space is high-dimensional and we often use a smaller dimensional latent space, we often observe \"feature collapse\" (for example, see [this paper](https://proceedings.neurips.cc/paper/2020/hash/543e83748234f7cbab21aa0ade66565f-Abstract.html) or [this paper](https://proceedings.mlr.press/v119/van-amersfoort20a.html)), meaning that an inlier A and an outlier B are mapped to the same latent space point Z.\n\n* Having a good empirical performance on Table 1 and Table 2 are good, but the benchmark could have been made more challenging. An outlier detection task (and also a one-class classification problem as well, if you allow me to use the two terms interchangeably) tends to be more difficult as the variability within the inlier distribution becomes larger. For example, an outlier detection task where the inlier distribution consists of multiple clusters. This setting can be simulated by using nine out of ten classes as inliers and the remaining single class as outliers. [This paper (Yoon et al., 2021)](https://proceedings.mlr.press/v139/yoon21c.html) showed that autoencoder-based methods are particularly weak at this setting. The paper also introduced an autoencoder-based outlier detection method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8j922hpMWm", "reviewer": "Reviewer_wCV4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel method for one-class classification and anomaly detection, by integrating restricted generative projection using 4 different variants of volume restrictions", "summary_of_the_review": "The proposed methods is interesting. However, it seems that it is taking some ideas from different works and combining them, such as the used of MMD, as opposed to KL-divergence in VAE, and the use of hyperspheres with several variants, as opposed to a hypercube as proposed by Perera et al. (2019).\n\nIt is said that other methods may suffer from high computational cost and instability in optimization. However, this is also the case of the proposed method. The authors did not provide any analysis of the computational complexity of the proposed method. Moreover, the conducted experiments did not provide any measure of this computational complexity, such as the time requested for training, or computing the FLOPS.\n\nThere are many spelling and grammatical errors. We give here some of them:\nThe first category are, the second category are, the last category are\nHypershpere\nall the samples distribute uniformly\ndiscard all data points outsides\nhypersphre\nIt was used to identity\nSince both image datasets contains\nwe both set learning rate to\nThe details of the our network\nunder four different restriction", "strength_and_weaknesses": "See summary of review for more details", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6yxn0hjmlI", "reviewer": "Reviewer_ByPd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the abnormal detection problem by introducing a one-class AutoEncoder-based model, leveraging kernel maximum mean discrepancy (MMD) in the latent space. The model is expected to learn a bounded latent distribution for each class, requiring one AE model to be trained for each class. The model is trained with the MMD loss and an L2 reconstruction loss. The method is technically sound and good results are achieved on three tabular datasets and two image datasets.", "summary_of_the_review": "Overall, the paper introduces a new AE-based method for abnormal detection by introducing distribution prior to the features in the latent space. The idea is interesting and generally effective, though not achieving state-of-the-art in many cases. The significance of technical contribution and results appears to be not strong enough for acceptance as a regular paper in ICLR, but perhaps a workshop.", "strength_and_weaknesses": "Strength\n- The idea is simple and well-motivated.\n- The paper is generally easy to follow.\n\nWeaknesses\n- In a nutshell, the method appears simply adding a regularization term for training an Auto-Encoder, though with a well-motivated distribution prior.\n- Performance does not exceed the state-of-the-art in many cases (e.g., Tab 1 & 2). \n- One model is needed for each class, making it less practical in a real deployment, for which a model that can handle all normal classes is preferred. Would it be possible to encode multiple classes in a single model for AD?\n- How could the hyperparameter be determined without accessing the actual AD data?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject