On unsupervised-supervised risk and one-class neural networks
Authors: Christophe Cerisara
Venue:  (2020)
Keywords: unsupervised training
one-class models

Abstract: Most unsupervised neural networks training methods concern generative models, deep clustering, pretraining or some form of representation learning. We rather deal in this work with unsupervised training of the final classification stage of a standard deep learning stack, with a focus on two types of methods: unsupervised-supervised risk approximations and one-class models. We derive a new analytical solution for the former and identify and analyze its similarity with the latter.
We apply and validate the proposed approach on multiple experimental conditions, in particular on four challenging recent Natural Language Processing tasks as well as on an anomaly detection task, where it improves over state-of-the-art models.

Under review as a conference paper at ICLR 2020
ON UNSUPERVISED-SSUPERIISS RISK AND CLASS NEURAL NETWORKS
ONE-
Anonymous authors Paper under double-blind review
ABSTRACT
Most unsupervised neural networks training methods concern generative models, deep clustering, pretraining or some form of representation learning. We rather deal in this work with unsupervised training of the final classification stage of standard deep learning stack, with a focus on two types of methods: unsupervised- supervised risk approximations and one-class models. We derive new analytical solution for the former and identify and analyze its similarity with the latter. We apply and validate the proposed approach multiple experimental conditions, in particular on four challenging recent Natural Language Processing tasks as well on an anomaly detection task, where it improves over state-of-the-an models.
INTRODUCTION
Machine learning systems often share the same architecture composed of two stages: the first stage computes representations of the input observations, while the second stage performs classification based on these representations. Most unsupervised training methods focus on the first stage: rep- resentation learning. This includes for instance generative models (VAE, GAN...), clustering tech- niques and, in the Natural Language Processing (NLP) domain, all recent contextual words embed- dings (RoBERTa, XLNet, GPT-2...).
This work rather deals with the final classification step, more precisely how to train neural classifiers in unsupervised way. In contrast to unsupervised training of the first stage that aims at learning representations, unsupervised training of the final stage may rather pursue one of the following objectives, among others:
Training one-class models for anomaly detection
Exploiting unsupervised approximations of the classifier risk train model from priori knowledge and unlabeled data instead of labeled samples
The former is a special type of binary classification task, where the positive class represents "nor- mal" observations and the objective is identify unknown and often observations that can be considered as anomalies and form the negative class.
The latter deals with training standard discriminative classifiers without labels, i.e., when assuming that the precise target classification task is not defined explicitly with sample labels, but implicitly with a priori knowledge. We review in Section 2 the family of one-class models as well an unsupervised approximation of the risk, and explore their relation in Section 3.3 hence bridging the gap between both unsupervised discriminative classification approaches.
The main original contributions of this work are:
We derive an exact and analytical solution (Eq to the risk approximation proposed Balasubramanian al. (201T)
We analyze the properties this solution, which lead to the following new results:
We extend this solution into end-to-end differentiable loss that can be easily inte- grated into any modern deep learning toolkit (
We propose an unsupervised training algorithm based on this analysis (Alg 1

---

Under review as a conference paper at ICLR 2020
We propose posterior regularization term to improve this approach (Eq
We identify and study the similarity of this approximation with the one-class neural net- work anomaly detection method (Section 3.3
We validate experimentally the unsupervised model on several datasets and tasks, including a comparison with State-o--the-an one-class neural networks (Section
2 RELATED WORK
We focus in this literature review two unsupervised training methods for discriminative classifiers that do not aim at computing representations of the input space, but that rather exploit such repre- sentations perform a final classification task. The first such method is unsupervised-superiiss (we have adopted the terminology of the original paper) approximation has been proposed by [Balasubramanian al. (2011) and that is detailed in Section [2.] The sec- ond class of methods is the family of one-class models for anomaly detection, which is reviewed in Section 2.2
the classifier risk that
2.1
RISK APPROXIMATION
Let be given a binary linear classifier with parameters / that computes scalar score f(x) E IR for observation æ. The classifier outputs class ÿ = 0 iff f(z) <= 0, and ÿ = iff f(x) > 0. The risk
of this classifier with hinge loss is (Balasubramanian al. 2011
R(0) Fx((a)) [11 - f(x) (2y 1])+]
R(0) = Ply=0) = p(f(z) aly = 0) dd Ply p(f(x) aly = a)
(2)
Balasubramanian et (2011) prove that this risk can be optimized in an unsupervised way, as the labels are not required to compute Eq when assuming that:
The class-marginal prior P(y) is known;
The class-conditional distribution of the scores [(f(()]y) is Gaussian, which is supported
by the central limit theorem please refer to Balasubramanian et al. (2011) for further details.
The training algorithm proposed by the authors consists in the combination of (i) a gradient descent to optimize the linear classifier parameters 0; and (ii) the Expectation-Maxxiizattio (EM) algorithm, to compute the Gaussian parameters.
We derive a new formulation of this risk and study it in Section 3.1
2.2 ONE-CLASS MODELS
One-class models are based on the assumption that all observations belong to single positive, "normal" class, except for (a few) outliers associated with the negative class. Given that there is no label to identify which observations are outliers, the problem can be cast as unsupervised training problem. This class of models are typically used in anomaly detection applications.
The model at the origin of this research domain is the One-Class SVM (Schölkopf et al. 2001 (OC- SVM). This model projects positive observations into feature space, and computes an hyper-plane in this feature space that separates most of these points from the region close to the origin, where outliers (noise) are assumed be. The objective function of this model
min + ||w1 -
1 N
Σ
N
Ei
under the constraints (e; slack variables): et 2 0 and -t (bx:)) 2 Ci.
2

---

Under review as a conference paper at ICLR 2020
corresponds to the linear classifier weights and Φ is the non-linear SVM projection. wT ((x) the signed distance between any of the samples and the decision hyperplane.
A powerful extension of the one-class SVM is the Support Vector Data Description (SVDD) model (Tax & Duin 2004). This model exploits an hypersphere with radius R and center c instead of hyperplane to separate the positive and negative classes:
N
min R,c.e
R²
Σ
Ei
V.N
under the constraints that e; 2 0 and (()()))) c|
R
+
The SVDD model has been enriched by Ruff et (2018) learn representation @w (2)) computed with deep neural network with parameters W, which gives the Deep SVDD model:
N
min R.W
R
+
max (0, (1:)(()) - cl1 R²)
Σ |[Wil?
(3)
VN
2
This model is trained by alternating Stochastic Gradient Descent (SGD) step on W and computing the optimum R.
Finally, the original OC-SVM has also been extended as a deep learning model with the One-Clas Neural Network (OC-NN) (Chalapathy al. 2018). In this model, the final linear layer in stack of deep neural network layers interpreted as defining the decision hyperplane:
N
1 111
min wn, V.r
|
+
EN Σ
max (0,r wnt.:=//VVVa g(Vx.)) -r
(4)
with V the previous layers that compute representation of the input and g() the previous activation. This model is trained by alternating a SGD step to update (V, w) and computing the optimal r.
Another model of this family has recently been published: the One-class Convolutional Neural Network (Oza & Patel 2018), but the training objective of this model departs from the previous unsupervised training objectives, this model is trained with the standard cross-entropy loss with negative samples that are artificially generated from Gaussian distribution centered at the origin.
3 UNSUPERVISED SUPERVISED RISK
3.1 EXACT RISK DERIVATION
Starting from Eq we derive closed-form solution compute the risk from the two Gaussian means and variances E that model the distribution of the score f(x) (we note P(y = 0) = Po):
R(, E)
PD
no)
(1-
( 14)
-1
0ov2 2 ))
(5)
2
1- po (1 m1))
+ erf
1 111
σ 0Iv2 ))
+
Poo (-1: Mo, (o. σo) (1
Po) po) N1:
with
N(a:11,0)= 22
Balasubramanian et al. (201]) proposed to optimize the risk with finite differences. We rather pro- pose to use the analytical solution derived in Eq which has the following advantages:
The risk value is exact and not approximated;
See full derivation in Appendix.
3

---

Under review as a conference paper at ICLR 2020
Computation of the risk is much faster using Eq
5 than with numerical approximations;
This equation is differentiable with respect to the Gaussian parameters. We derive next another function that relates the Gaussian parameters to the model parameters Hence, the full risk can be directly integrated as loss function in deep learning toolkits;
The analytical equation can be analyzed, which leads to novel insights as shown next.
Let us plot Equation 5 as a function of (no, (1)) in Figure (left), for Po 0.1 and dg 01 11
20
kgd+20
0.0
0.6
014
10.0 3.0
2.5 20.0
103
Figure 1: Risk as function of both (po, μ1) (left), and only pi (right) for #1 2, 01 and ao E {0.1,1.3)
When we fix Pi, can see in Figure I (right) that the risk function of no can be well approxi- mated by a scaled and translated rectified linear function, as long as the variances are small enough. Furthermore, the lower dg (and σΙ) is, the better the risk is. Varying μ1 and σ1 only translates this curve vertically, above the horizontal axis. So, assuming that the risk has first been minimized with respect to fl1 then the global minimum of the risk may be obtained by decreasing linearly No- Conversely, lower risks are obtained when ll is increasing. Although we have not exploited this piece-wise linear approximation of the risk in implementation, is interesting to compare it to the max(0, ..- term in Equation 4
Let us now make another assumption: that both modes (pa, an) and (μr, a1 of the score distribution are well separated. This is reasonable assumption when we are not too far away from the global optimum, because the previous analysis has already shown that getting close to the optimum implies that po is small, is large and that do and are small. Then, good approximation of po and can be computed by splitting all the scores f(x according to the Po-quantile Tpo defined as
ExEx {())</(a) N
Po arg min
Po
(6)
where the set of all observations X is of size N. Let us call X the subset of size N of all data points that are on the left side of the po -u-quaniie::
X = fx E X s.t. f(x) < f(Ipo)}
and similarly for the other side:
X fx s.t. 2 2 [(
We can now approximate the Gaussian parameters deterministically:
no ~ N
1
Σ
f(x)
1 N+
Σ f(x)
(7)
xEX
xeX+
2
2 s(o)
Σ
f(0)2
Σ f(z)
of
Σ f())3
N+
x-X-
#EX-
+
4

---

Under review as a conference paper at ICLR 2020
Intuitively, decreasing the risk may be achieved by decreasing Pu. du, and increasing μl. Plug- ging these equations into equation S gives a differentiable loss with respect to the network parame- ters, which can be used in every modern deep learning toolkit.
3.2 GEOMETRIC INTERPRETATION
Following Chalapathy et al. (2018). we can consider a deep neural network that computes some rep- resentation of ipputs. These representations are then passed final binary linear classification layer with a single scalar output. This final layer, and optionally the previous layers, may be trained by minimizing our unsupervised risk Eq [ with Stochastic Gradient Descent. As discussed in Section 2.2 this final layer actually defines an hyperplane that separates both positive and negative instances, and its output is the signed distance between each observation and this hyperplane, Mo the average of these signed distances for all points that are one side of the hyperplane (X' and μ1 for points on the other side X+, decreasing Hi and increasing μr can be interpreted moving away all samples in X and in X as far possible from the hyperplane, show in Figure (
An important constraint is that the proportion of points on both sides of the hyperplane should be equal (or close) to Po, otherwise, an easy way to decrease the risk with unbalanced classes to translate the hyper-plane along the vector infinitely, moving all samples into the most frequent class, The constraint is thus
f(x < 0 Vx X and ((x Vor X
This constraint can be fulfilled by adding another term to the risk, which becomes:
(0) R(0) + f(xpp))
While Balasubramanian et al. (201T) have used the class marginal only prior information, the additional term in Equation 8 can be seen posterior regularization term, which forces the poste- rior distribution Ply = 0|X) ) to match Po-
Algorithm summarizes the training procedure.
Algorithm End-to-end unsupervised training
Initialization:
Let consider a binary classification task, for which we assume that the proportion of class-O elements Po known approximately;
Let be given a corpus of observations {at without labels;
Let be given a deep neural network (z) with parameters á that computes a vectorial representation of an input x, which is fed to a final linear classification layer fe(g with parameters 0; á and 0 may be initialized randomly or pretrained.
Iterate:
Run forward pass on the dataset {2,)I with the current parameters 6,0. A. Compute classifier scores {s+ = fo fo(go(;;))
over the full corpus N, or over a batch of observations that is large enough to assume that the distribution classes in the batch is representative of the distribution in the whole corpus.
Sort the list of scores (sx) (s)snn to compute the Po Po-quantile = following Equation 6
Compute the Gaussian parameters μ (Ho. 11). (oa. σι) with Equations
Compute the risk (Eq 8 with these Gaussian parameters.
Apply automatic differentiation to compute V R(H, E), and optionally V. R(u, E);
Run step of SGD to update 6, and optionally Φ.
3.3 RELATION WITH ONE-CLASS NEURAL NETWORKS
The One-Class Neural Network Chalapathy et al. (018)) similarly splits the set of observations with an hyper-plane defined by the last layer of deep neural network stack, but while Equation splits

---

Under review as a conference paper at ICLR 2020
the samples according to Pu. the OC-NN splits them according to the v--uuntile of the points sorted by their signed distance to the hyper-plane, where y controls the number data points that are allowed to be the negative side of the hyper-plane. This hyper-parament plays the same role as our Po- By rewriting their distance with our notation f(x), their loss (Eq(4) becomes:
min
L+ VN
Σ (max(0,, r- r- f(xx)))
where T. is term that does not depend on zi. Chalapathy et al. (2018) compute the optimal r as "the v-quantile" the scores, so we can rewrite r [Troo) and their sum as our /lg:
1N N
(f(#p)) - Ho) = min (L f(zpo) -po)
min L
Their objective thus aims at maximizing po. i.e. making all negative samples close possible to the hyperplane, as shown in Figure 2 (left). This equation strongly resembles the linear approxima- tion of the risk that we have depicted in Figure except that the OC-NN takes into account only the negative part of the embeddings space, while our risk includes both negative and positive parts, and that the gradients are in opposite directions, as summarized in Figure
negative samples
samples
Unsup gradiers
posttive
OC low gadien
positive
samples
samples
Gool lpter
Boal Laye
Iyperiias
hypeplase
Figure 2: Comparative illustration of OC-NN and unsupervised risk approximation: observations are represented in the embeddings space just before the final linear layer; the hyperplane is defined by the parameters 0, and fo(x) is the signed distance of the samples to the hyperplane. During training. the OC-NN tends reduce the distance between the negative samples and the hyperplane, while our unsupervised loss tends to increase this distance for both negative and positive samples.
EXPERIMENTAL VALIDATION
The proposed unsupervised risk is coded pytorch Paszke al. 2017 al.
and is freely distributed 2 is evaluated in various tasks: (i) on a synthetic toy classification dataset; (ii) on the Wisconsin Breast Cancer benchmark; (iii) on four NLP tasks and (iv) on a standard anomaly detection task. Following the related works, the standard unsupervised accuracy metric (Xie et al. 2016) is used for the first three cases, while the Area Under Curve (AUC) metric is used for anomaly detection.
4.1 SYNTHETIC DATASET
We first validate our approach on a synthetic dataset, which contains 10,000 4-dimensional instances
sampled from bi- bi-Gaussian distribution (po = 0.6, = [1, 1, 1, 1] = [1. 1. 1 17:pp P1 =
/1 = [-2, 22 -2, - . = [1, 1, We train two simple models with 1,000 training epochs: one with single layer, and another one with two layers and 2 hidden neurons (half of the input size). The accuracy per training epoch is shown on the left curve in Figure 3
"The code is given in the supplementary material and is further distributed with open-source license on public git repository.

---

Under review as a conference paper at ICLR 2020
0.55
1layer
21aym
0.55
3.9
0125
0.50
3.8
0.8
0.75
2.7
0.55
0.55
0
200
200
300
430
500
800
700
800
800
2000
130
200
800
400
530
600
700
800
800
1000
Figure 3: Accuracy function of the number of unsupervised training epochs on the synthetic dataset (left) and on the Wisconsin Breast Cancer dataset (right).
We further study the sensitivity of our algorithm to initial conditions by retraining the model times with random initial parameters: the standard deviation of the accuracy is smaller than 2%. This first experiment validates that the unsupervised training algorithm is able to quickly and reliably converge towards the expected solution when the feature space explicitly encodes the class information.
4.2 WISCONSIN BREAST CANCER DATASET
We validate next our unsupervised approach on standard machine learning benchmark for binary classification: the Wisconsin Breast Cancer dataset (Dua & Karra Taniskidou, (017), composed of 569 instances with 30 dimensions each. The right curve in Figure shows the accuracy of both our and 2-layers models. 15 hidden neurons (half of the input dimension) are used for the 2-layers model.
The convergence of our method is also fast and stable this more realistic dataset. The state-of- the-art for supervised learning on this dataset is 99.1% of accuracy (Osman 2017). With 91 % of accuracy, our approach performs relatively well given that it is purely unsupervised. As a com- parison, we have run K-Means clustering algorithm the same dataset, which gives 85% of accuracy.
4.3 SENTEVAL TASKS
We validate next our unsupervised approach on four recent and more difficult Natural Language Processing (NLP) binary classification datasets:
Movie Review (MR): classification of positive vs. negative movie reviews;
Product Review (CR): classification of positive vs. negative product reviews;
Subjectivity status (SUBJ): classification subjective vs. objective movie reviews;
Opinion polarity (MPQA): classification of positive vs. negative movie reviews.
These datasets as well as the experimental evaluation protocol that we have used are described in details in Conneau & Kiela (2018). This protocol first computes sentence representation with the state-of-the-aar method InferSent (Conneau et al. 2017), and then passes these sentence embed- dings into simple feed-forward network that is trained each dataset.
We have adopted the same experimental protocol and the same not train the final feed-forward network with supervised labels and the cross-entropy loss, but rather train it without any label and with our proposed unsupervised loss. Table I summarizes the accuracy of the state-of-the-ar the supervised models trained on the full corpus (InferSent sup.) and on only 100 instances (InferSent 100-ex), as well as the accuracy of the proposed unsupervised model (Unsup risk). Results in italic taken from Conneau et al. (201)), other results computed.
except that we do

---

Under review as a conference paper at ICLR 2020
Table 1: Unsupervised accuracy four binary NLP tasks
System
CR
SUBJ
MPQA
MR
InferSent sup.
86.3
92.4
90.2
81.1
InferSent 100-ex
63.8
62.5
70.1
53.9
Unsup risk
66.8
83.0
70.9
59.7
We can observe that the proposed purely unsupervised method always gives at least as good results as the state-of-the-art transfer learning model trained on 100 reviews, with notable improvement of +20% absolute for the subjectivity classification task.
4.4 ANOMALY DETECTION
We finally validate our approach on an anomaly detection task. We adopt the same dataset and experimental protocol than Ruff et al. (2018) ChaCaaapy et al. (2018) for comparison. The tasks consists in detecting outliers in digits mages, where the "normal class" is the positive class and is composed of images corresponding to a single target digit, and the outliers are randomly sampled from the other digits images. Our model is composed of a single additional weardforward layer on top of the Ruff et al. (2018) model. This final classification layer is initialized from the Ruff et al. (2018) parameters, and is then trained in unsupervised way with the loss in Equation 8 We tune the hyper-parameter (number of epochs and learning rate) on a development corpus obtained by keeping the same positive instances from the training corpus, but adding different negative training samples. We rerun every experiment 10 times with different seeds to compute the standard deviation. For the DeeD the authors code, which may differ because of slightly varying conditions. The DeepS outputs on the right are the ones that our own model is based on, and with which should be compared
report both the figures from the original paper, and the results obtained with
Table 2: Results (AUC) on anomaly detection (": from original papers)
OC-NN"
DeepSVDD*
DeepSVDD
Eq
0
97.6 + 1.7
98.0 0.7
98,
98.3 +
99.5 + 0.0
99.7 0.1
9
99.5
2
2.1
=1.7=0.8
1
93.1
86.5
91.9 1.5
90.5 1
9.9
4
93.3 2.4
94.9 = 0.8
94.0 1.3
94.8
86.5 + 3.3
88.5 = 0.9
86.3 1.3
2.3
6
97.1-14
98.3 = 0.5
98.0 + 0.6
97.6
2.1
94.6 0.9
9..7
95.0 1.7
8
8.77
93.9 1.6
92.7 - 0.9
93.6 1.4
9
93.5 + 3.3
96.5 = 0.3
96.0 - 0.7
+6.4
We can note that our proposed unsupervised method always improve compared to the One-Class neural network, and is also generally better than the DeepSVDD model run the same platform. Compared to the one-class models, our approach exploits information from all instances instead of only the negative samples (see Figure 2). Furthermore, under reasonable assumptions, our loss converges towards the theoretical optimum of the classifier risk (See Eq [T).
CONCLUSION
We have shown that both wwwpueerviiedsuuprvisssee classifier risk approximation and one-class neural networks lead to similar training procedures, although they optimize slightly different objective. One of the main difference is that the former exploits all training samples, positive and negative, which should lead to better parameter estimates. This seems to be confirmed by experimental vali- dation. Based on the similarity between both types of methods, we have also shown experimentally wewpeerviieddsuperiisssse classifier risk approximation is valuable method to be included in the set of approaches dedicated to anomaly detection. future works, we plan to extend this approach for multi-class classification and few-shot learning.
and through analysis that the
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Krishnakumar Balasubramanian, Pinar Donmez, and Guy Lehanon. Unsupervised supervised learn- ing II: Margin-hass hased classification without labels. Journal of Machine Learning Research, 12: 3119-3145, 2011.
Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla. Anomaly detection using one-class neural networks. (rXivv/802.06660) 2018.
Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence represen- tations. /803.
2018.
Alexis Conneau, Douwe Kicla, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised
learning of universal sentence representations from natural language inference data. In Proc. of the Conference on Empirical Methods in Natural Language Processing, pp. 670-680, 2017.
Dheeru Dua and Efi Karra Taniskidou. //archive. ics wei. edu/ml
UCI machine learning repository, 2017. URL http:
Ahmed Hamza Osman. An enhanced breast cancer diagnosis scheme based on two-step-svm tech- nique. Int. J. Adv. Comput. Sci. Appl, 8:158-165, 2017.
Poojan Oza and Vishal Patel. One-class convolutional neural network. IEEE Signal Processing Letters, 26(2):277-28 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Lukas Ruff, Nico Görnitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Robert Vandermeulen, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International Conference on Machine Learning, pp. 4390-4399, 2018.
Bernhard Schölkopf, John C Platt, John Shawe-Taylor Alex J Smola, and Robert C Williamson. Estimating the support of high-dimensional distribution. Neural computation, 13(7)11443-1411, 2001.
David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54(1): 45-66, 2004.
J. Xie, R. Girshick, and A. Farhadi. Unsupervised deep embedding for clustering analysis. In Proc. ICML, pp. 478-487, New York, 2016.
A APPENDIX
A.1 DERIVATION OF THE UNSUPERVISED RISK
Let given binary linear classifier with parameters 0 that computes a scalar score f(x) = DR 0,0r E IR for observation æ IR". The classifier outputs class ğ 0 iff f(x) S 0, and y iff [(x) 0. The true/gold class label noted y E (0,1), 1}. The risk of this classifier with hinge loss is (Balasubramanian al. 2011):
R(O) Ep(x.)) [[1 f(x) (2y 1))_]
P(y P(y= p(f(x) = aly = 0) aa+
P p(f(x)) aly 1) da+
We assume the conditional distributions follow normal distribution: ( 0) N(po
9

---

Under review as a conference paper at ICLR 2020
p(f(z)ly=1) = ~ (1)
where Ni, o) is the standard normal distribution with mean and variance o². We can then rewrite approximation of this risk under the previous assumption and the additional assumption that the class priors P(y) are known:
R = P(y 0) / N(as 00)(1 + a + P(y P(y=1) / N(a; -
Removing the non-linearity:
P(y=1) R= P(y /_ Ho); + adda P(y - 1) [ N(a; aadda
Distributing:
P(y 0) [ +00
No + P(y=0) [ (o,
R
p(y = 1) [[ p1 P(y 1) [ a
We know that the cumulative distribution function a (1i o) normal
1 2
x 1
F(x) 1 + erf
ov2
So the integral of Gaussian is:
[ N(2;,,,)ddr F(b) F(a) =2
1 ( erf
b 11
a (a- 11
erf
[o22/
0v2
We know that erf( (000) = -1, erf((()) and erf(+00) = 1,
PΦy = 0)
-1--0 MO)
)) + P 00 +P [
1 erf
aN(a: No, oo)
2
0ov2
P(y 1)
1- H1
P(y 1)) I aNaaa
+ erf
2
01v2
Integration by parts give:
xN od) bF(b) - aFF(a) - 1( F(x)dr
We also know that
/ erf(x)dr rerf(x) + x=
VT +C
So
J F(x)dz= b-a l' erf
x 1
dr
2 0
We use integration by substitution:
[" erf
I dr =0/2 1 erf
x xz
dr
((b)
0/2 erf(u)du
2 o jula)
with u(x)
2-4
ov2
So
(()) erf(u)du u(b)erf(u(b)) u(a)erf(u(a)) 1 (n()) -e- u(())
Vx
u(()
10

---

Under review as a conference paper at ICLR 2020
(l(b)
erf(u)du= b- erf
b- 66 11
1
16 11 u)
erf
+
(e
20
20
a()) o oo2
o
ov2
VT
erf 8- M co/2
b (b-1)) 11
a a-
o
10-p12
(a-p)2
dr (b w)erf
(a
pteef
+
(e
20
20
(ov2)
ov2
V
F(x)dr
b-
b # erf
b- 1
M erf
a a0 μ
+
(c
10-212
20
(a-212
20
ov2
2t
Plugging into the former equation:
f. xN (N) =
a
p
b- 11
a
p
a -
14
o
[6 2)²
La m)
erf
+
erf
20
20
2
2
[o22
2
2n
[. xN (N
b
b- 1
a
a erf
a - H
+
erf
+
2
2
ov2
2
2
v 2
a
erf
b - 11
+
erf
+
2
2
2
0v2
0/2
a
a
" erf
a- 1
(6- w)²
(a-p) 2a
erf
(e
2n
e
2
ov2
2
o
2x
Simplifying
I- x:N (x; p1, o)d
a H
erf
erf
0/2
0o2
(b-p)²
(a-p) 20
20
2m
Note: another way to obtain this result is to use the following known formula:
J IN IN, a f."" N(;; o² (N(2;
So
Je IN IN(x:,,o)dr
11 erf 6-1 11 erf 0 )) - o² -(a.m)))))( 2
0v22/
When + +00:
NN(x; = 4 11-erf ( )) + ((2((a))))))
2
And when + -o:
"(1 erf
b (b-m) ) Nb:
IN(
1
_
Our risk is:
P(y= ===0 0)
-1 -1 - 11 )) PP =0) [" +
R
[ 1-erf erf
(a oo)da
2
G
P(y 1)
1 1- H1
02 )) - P(y = 1) J- (1)
1 crf
2
11

---

Under review as a conference paper at ICLR 2020
So
Ply 0 (1-eff erf (-1-11) 110 ))
P(y 0)p0
(1- (1-off erf ( (-1-/ 110 )
R
+
0ov2
2
P(y = N-1 To) +
P(y = 1+
1 - ))
PV 1)p1
(1+
(1-
0Iv2 )
2
P(y (1;
And finally, the risk as a function of the bi-Gaussian parameters is:
P 0)
- Ho
))
R
(1 + mo)
1 erf
+ P(y = --)0 +
2
0ov2
P = 1)
1- H1
(1 m1)
1 erf
P(y ())=)))((;))
2
01v2
12


Reviews:
[{"review_id": "SkxQrbQiYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE:\nI acknowledge that I’ve read the author responses as well as other reviews. \nAfter reading, I would keep my rating at 3 (Weak Reject), since the key reasons for my rating still hold.\n\n####################\n\nThis work makes a connection between recently introduced one-class neural networks [8, 4] and the unsupervised approximation of the binary classifier risk under the hinge loss [1]. An explicit expression of this risk approximation is derived for the case that the prior class probabilities are known and that the class-conditional distributions of classification scores are Gaussian. This solution is then used to formulate an end-to-end differentiable loss for unsupervised binary classification which is combined with a posterior class probability regularizer to avoid trivial solutions. Finally, the paper presents an experimental evaluation on synthetic data, the Wisconsin Breast Cancer dataset, four NLP tasks, as well as on the anomaly detection task on MNIST where the proposed method slightly outperforms the two existing one-class networks [8, 4].\n\nI think this paper makes an interesting, original connection between unsupervised-supervised risk estimation and one-class neural networks which provides a principled motivation for existing methods [8, 4] and also hints to potential flaws in their formulations, namely that OC-NN [4] and soft-boundary Deep SVDD [8] make no use of positive samples during learning (as illustrated in Figure 2). The paper is not yet ready for acceptance in my opinion, however, due to the following key reasons: \n(i) The experimental evaluation is not convincing and not sufficient to assess the significance of results; \n(ii) Though making this connection is interesting, the theoretical derivations presented in the paper are rather straightforward.\n\n(i) I think the experimental evaluation is the weakest part of the paper at the moment which I find not convincing due to the lack of competitors, the use of rather simple datasets, and missing experimental details. The synthetic experiment only serves as a sanity check not giving any additional insights. As the proposed unsupervised method approximates the risk of a supervised binary classifier, I agree that it makes sense to compare to the supervised “gold standard” on binary classification tasks (Wisconsin and NLP sentiment tasks) to infer the unsupervised-supervised performance gap. However, there is no comparison to other unsupervised competitors (OC-SVM, GMM, Deep SVDD, OC-NN, etc.) to put the performance of the proposed method in these experiments into perspective (only K-Means does not establish a strong baseline). Those classification tasks further are not really a convincing use case in my mind since labels here are usually available. In contrast, I find anomaly detection to be an important application of this method, but an evaluation solely on MNIST that also lacks recent deep competitors [6] is not sufficient to assess the significance of the presented results. Moreover, from the text it seems that only the hyperparameters of the proposed method are tuned on some validation set that includes positive as well as negative samples which would be an unfair advantage and might explain the slight edge in performance. Finally, many experimental details are not reported: (ia) Are the networks used randomly initialized or pretrained? (ib) How are prior class probabilities set? (ic) What are the batch sizes (relevant for quantile estimation) (id) What score are the hyperparameters tuned on? (ie) Are negative samples in the validation set from all the anomaly classes?\n\n(ii) The technical derivations in the paper (and the appendix) are correct but rather straightforward. The theoretical heavy-lifting is from Balasubramanian et al. [1] and this paper presents an explicit solution for the risk approximation under the assumption that the prior class probabilities are known and the class-conditional score distributions are Gaussian. I do not want to discount that making this connection and loss derivation may lead to significant results (which is left to be demonstrated experimentally), but I find the current theoretical contribution on its own not sufficient. Parts of the theoretical Section 3 could also be greatly cut in my opinion (no need to define a quantile or sample mean and variance etc.). Finally, one key property of the loss expressed in the paper is its differentiability and use with autograd, but this does not hold after adding the posterior regularization term which is based on the empirical p0-quantile, correct? (the gradient of the quantile is zero almost everywhere due to the argmin)\n\nApart from the two key points above, the presentation of the paper is unpolished (nested lists in the main text, etc.) and major deep anomaly detection related work [10, 6, 5, 7, 3] is missing.\n\n\n####################\n*Additional Feedback*\n\n*Positive Highlights*\n1. The paper makes an interesting connection between unsupervised-supervised risk approximation [1] and recently introduced one-class neural networks [8, 4] that provides a principled motivation and points to potential flaws in existing formulations.\n2. I think the question how to learn neural classifiers in an unsupervised manner is original and interesting.\n3. The technical derivations in the paper are rigorous and correct.\n\n*Ideas for Improvement*\n4. Make a comparison to state-of-the-art unsupervised deep anomaly detection (AD) methods.\n5. Run AD experiments on more complex datasets like Fashion-MNIST, CIFAR-10, and the recently introduced MVTec [2].\n6. Include major deep AD works [10, 6, 5, 7, 3] into the related work.\n7. Compress Sections 1–3 (fewer lists; no need to give definitions of a quantile, sample mean and variance; etc.)\n8. Motivate the non-AD experiments. Currently these appear rather constructed artificially.\n9. In the NLP tasks, make a comparison to text-specific one-class classifiers [9].\n10. Add a sensitivity analysis w.r.t. the prior class probability p0 to infer robustness to this parameter that seems crucial.\n11. Provide guidance how to select p0 in a particular application.\n12. Consistently report performance metrics with standard deviations in your tables to allow to infer statistical\nsignificance.\n13. What to do if there are no negative, but only normal samples as in fully unsupervised AD? Nevertheless make an assumption on the class prior?\n\n*Minor comments*\n14. Unordered lists should be used sparsely in a main text, stylistically speaking. Avoid nesting as in the introduction.\n15. Section 2.1, first sentence (and elsewhere): “Let be given ...” is grammatically wrong. Correct would be either “Let f be a binary linear classifier ...” or “Given a binary linear classifier f ...”.\n16. In Section 2.1, index the classifier $f$ with parameter $\\theta$, i.e. $f_\\theta$. Otherwise the risk optimization parameter $\\theta$ does not even appear on the right hand side of the risk Eq. (1).\n17. Combine Eqs. (1) and (2) into one equation.\n18. Consistently enumerate equations throughout the paper or do not enumerate at all.\n19. In Eq. (3) index $i$ misses in the sum.\n20. A subsection title following a section title directly is bad style. A new major section should at least be introduced with a few sentences on what this section is about.\n21. Mention that $erf$ is the Gaussian error function.\n22. Center the equation in Section 3.2.\n23. Plots is Figure 3 are rather poorly formatted: use thicker lines and more distinctive colors; place the legend legibly.\n24. Show the average with confidence intervals over the 10 runs in Figure 3.\n25. Put results, as in Section 4.2 on the Wisconsin Breast Cancer dataset rather in a table.\n\n\n####################\n*References*\n[1] K. Balasubramanian, P. Donmez, and G. Lebanon. Unsupervised supervised learning ii: Margin-based classification without labels. Journal of Machine Learning Research, 12(Nov):3119–3145, 2011.\n[2] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.\n[3] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019.\n[4] R. Chalapathy, A. K. Menon, and S. Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360, 2018.\n[5] H. Choi, E. Jang, and A. A. Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392, 2018.\n[6] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.\n[7] D. Hendrycks, M. Mazeika, and T. G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.\n[8] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.\n[9] L. Ruff, Y. Zemlyanskiy, R. Vandermeulen, T. Schnake, and M. Kloft. Self-attentive, multi-context one-class classification for unsupervised anomaly detection on text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4061–4071, 2019.\n[10] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Proceedings International Conference on Information Processing in Medical Imaging, pages 146–157. Springer, 2017.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HygubeqpFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The anonymous authors consider the problem of training of classifiers in an unsupervised way. They propose an extension to a one-class based approach that can do anomaly detection in an unsupervised fashion.\n\nThe main contribution is a modification of the target function for the training of one-class NN. The experiments are not convincing and the modification doesn't seem to provide much inside into representation learning and anomaly detection area. \n\n1. Figure 3: no axis labels\n2. ROC AUC is not the best quality to measure the quality of imbalanced classification problems or anomaly detection, PR AUC (average precision) is better\n3. In Table 2 and other experiments, there is a comparison to only one existed method e.g. authors don't reproduce results for OC-NN\n4. Table 1: why compare a supervised method to an unsupervised one and don't compare to other methods?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lq7qqAKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThis paper proposes an algorithm to train a binary classifier without supervision, simply relying on (i) class prior, (ii) the hypothesis that class conditional classifier scores are Gaussian distributed. Experiments over sentiment classification and anomaly detection highlight the effectiveness of the approach.\n\nReview Summary\n\nThe paper reads well and is technically correct. It proposes a simple algorithm for unsupervised training of binary classifier. Experiments are appropriate but lack baseline comparison with generative models and a thorough description of the unsupervised validation procedure. Overall, the approach is simple and it would be a good paper with the addition of baselines (mixture) and the clarification of the validation procedure.\n\nDetailed review:\n\nThe paper is divided into two parts: a closed form solution to the problem introduced by Balasubramanian et al 2011 and an algorithm leveraging class prior for training. Both parts are clear. Since the algorithm only requires the derivative of 5 wrt model parameters, could you write this derivative (it should yield a simpler expression without erf, no?).\n\nThe experimental study could discuss the robustness wrt to the choice of p_0 and report a grid of experiments over a training set with varying true and assumed p_0. In unsupervised classification, the problem of parameter validation always occurs, e.g. for senteval, how did you select p_0 and the other parameter of the model? If you used a labeled validation set, could you report its size? Could you report the performance of a supervised system trained on a set of that size? Could you report the performance of your method when fine tuned with the label of that validation set?\n\nFor the senteval experiments, it would be interesting to report the number of labels for which the supervised and unsupervised accuracy are equal. It would be more informative than simply reporting accuracy with 100 labels. When reporting the number of labels, you need to report the size of the training and validation set combined.\n\nAs far as baseline are concerned, it would be necessary to consider generative models such as mixture models and possibly mixture model with constraints on the mixing weights (Chauveau 2013).  Adding one class SVM (RBF, polynomial kernel) and one class neural nets (Ruff, Chalapati) for SentEval would be necessary to show the advantage of your approach. A curve number of training points accuracy for all models is a must have.\n\nYou could also cite related work in adaptation to new class prior, class prior estimation, see below:\n\nDidier Chauveau, David Hunter. ECM and MM algorithms for normal mixtures with constrained parameters. 2013. ffhal-00625285v2f\n\nAdjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure\nM Saerens, P Latinne, C Decaestecker - Neural computation, 2002 - MIT Press\n\nSemi-Supervised Learning of Class Balance under Class-Prior Change by Distribution Matching\nMarthinus Christoffel du Plessis, Masashi Sugiyama, ICML12\n\nClass proportion estimation with application to multiclass anomaly rejection\nT Sanderson, C Scott - Artificial Intelligence and Statistics, 2014\n\nFinally, I feel that the assumption that model scores are Gaussian distributed would be strongly justified if you could plot the distribution/run a statistical test on whether it is the case for a supervised model with the same architecture as yours.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes a connection between one-class neural networks and the unsupervised approximation of the binary classifier risk under the hinge loss. An important contribution of the paper is the algorithm to train a binary classifier without supervision by using class prior and the hypothesis that class conditional classifier scores have normal distribution. The technical contribution of the paper is novel and brings an increased understanding into one-class neural networks. The equations and the modeling present in the paper are sound and the paper is well-written.

However, in its current form, as pointed out by the reviewers, the experimental section is rather weak and can be substantially improved by adding extra experiments as suggested by reviewers #1, #2. Since its submission the paper has not yet been updated to incorporate these comments. Thus, for now, I recommend rejection of this paper, however on improvements I'm sure it can be a good contribution in other conferences.
Decision: Reject