Anomaly detection with semi-supervised classification based on risk estimators
Authors: Le Thi Khanh Hien
Sukanya Patra
Souhaib Ben Taieb
Venue: Accepted by TMLR ()

Abstract: A significant limitation of one-class classification anomaly detection methods is their reliance on the assumption that unlabeled training data only contains normal instances. To overcome this impractical assumption, we propose two novel classification-based anomaly detection methods. Firstly, we introduce a semi-supervised shallow anomaly detection method based on an unbiased risk estimator. Secondly, we present a semi-supervised deep anomaly detection method utilizing a nonnegative (biased) risk estimator. We establish estimation error bounds and excess risk bounds for both risk minimizers. Additionally, we propose techniques to select appropriate regularization parameters that ensure the nonnegativity of the empirical risk in the shallow model under specific loss functions. Our extensive experiments provide evidence of the effectiveness of the risk-based anomaly detection methods.

Published in Transactions on Machine Learning Research (March/20P)
Anomaly detection with semi-supervise classification based on risk estimators*
Le Thi Khanh Hien
Khonhhiennt&@gmaal.com com
Deportment of Mathematics and Operational Reseorch University of Mons, Belgium
Sukanya Patra
sukamya. Sumonss.c.b
Department of Computer Science, University of Mons, Belgium
Souhaib Ben Taieb
souhaib. wwnntaiieb-uooo......
Department of Computer Science, Universily of Mons, Belgium
Reviewed OpenReview: pttps://openreviiees net/ ftrro? id= BBuu
Abstract
A significant limitation of one-class classification anomaly detection methods is their reliance on the assumption that unlabeled training data only contains normal instances. To overcome this impractical assumption, we propose two novel classification--ess basedd anomaly detection methods. Firstly, we introduce a semi-supervised shallow anomaly detection method based on unbiased risk estimator. Secondly, we present semi-supervised deep anomaly detection method utilizing nonnegative (biased) risk estimator. We establish estimation error bounds and excess risk bounds for both risk minimizers. Additionally, propose techniques to select appropriate regularization parameters that ensure the nonnegativity of the empirical risk in the shallow model under specific loss functions. Our extensive experiments provide evidence of the effectiveness of the risk-based anomaly detection methods.
1 Introduction
Anomaly Detection (AD) can be defined as the task of identifying instances that deviates significantly from the majority of the data instances, see e.g., (Chandola et al.] 2009 Pang al.] 2020 Rulf al. 2021 for comprchensive surveys on AD. One important approach for AD one-class classification Khan & Madden (2014); Tax Duin (1999). It can be viewed a specialized binary classification problem aimed at learning model that distinguishes between positive (normal) and negative (anomalous) classes. This approach assumes that the unlabeled dataset primarily consists of data from the normal class. By utilizing sufficient amount of normal data, one-class classification AD (OC-AD) methods identify a decision boundary that encompasses all the normal points. For example, the decision boundaries of shallow OC-AD methods include a hyperplane with maximum margin (Schölkopf et al. (2001), a compact spherical boundary Tax & Duin (1999) 2004).
an clliptical boundary [Roussceuw & Van Dricssen, 1999 Roussceuw 1985), a pair of subspaces Wang & Cherian 22019) even a collection of multiple spheres Görnitz et al. (2018). To enhance their applicability in
high-dimensional settings, these shallow methods have been extended into deep methods Erfani et al. (016)
Ruff et al. (0118).
Unsupervised learning, where only unlabeled data is available, represents the most common setting in AD. Unsupervised AD methods typically assume that the training data consists solely normal instances Hodge
& Austin (2004); PPimentel et al. (2014); Zimek et al. (2012). However, in real-world scenarios, labeled samples may be available alongside the unlabeled dataset, leading the development of semi-supervised
*This work is supported by the FLARACC research project (Federated Learning and Augmented Reality Advanced Control Centers), funded by the Wallonia region in Belgium.

---

Published in Transactions on Machine Learning Research (March/200) /2024)
AD methods, including semi-supervised OC-AD methods Görnitz et al. (2009): ; Munoz-Mari al. (2010)
shallow /deep one-class anomaly detection methods do not explicitly handle mixed unlabeled data. This is because they typically assume that there are no anomalous instances prcsent in the unlabeled datasct, which is impractical in real-world sccnarios.
Ruff et al. (2020). It is important to note that aussupervise//smiisupprvvriii
Classification methods that handle mixed unlabeled data have been extensively studied in the field of learning with positive and unlabeled examples (LPUE or PU learning). In this context, we have access information on positive and unlabeled data, but negative data is unavailable. PU learning methods have also been utilized semi-supervised AD methods Bekker & Davis (2020): Blanchard et al. (2010); [Chandola et al. (2009) Ju et al.
2020)) Tt is widely recognized that incorporating labeled anomalies even if only a few instances, can greatly enhance the AD performance Görnitz et al. (2013); ; Kiran et al. (0188). Semi-supervised AD methods
that consider the availability of negalive data have demonstrated highly promising AD performance Han
et al.
p Ruff et al. [2021 2020).
To overcome the impractical assumption of OC-AD methods, we adopt the key concept of risk-based PU learning methods du Plessis et al. (2014: 2015); Kiryo et al. (2017); Sakai et al. (2017). These methods propose empirical estimators for the risk associated with the learning problem. In order to improve anomaly detection performance, we focus on the semi-supervised setting where a negative dataset is also available. It is noteworthy that the estimation of risk in anomaly detection is a relatively unexplored subject, distinguished by specific characteristics, especially in terms of error bounds, which are not commonly found in current. anomaly detection approaches.
Contributions
Our main contributions are summarized follows.
Considering AD as a semi cemi-supervised binary classification problem, where we have access to a positive dataset, a negative dataset, and an unlabeled dataset that may contain anomalous examples, we introduce two risk-based AD methods. These methods include a shallow AD approach developed using an unbiased risk estimator and a deep AD method based on a nonnegative risk estimator.
We develop methods to select suitable regularization that ensures the nonnegativity of the empirical risk in the proposed shallow AD method. This is crucial as negative empirical risk can lead to significant overfitting issues Kiryo et al. (2017).
We additionally establish estimation error bounds and excess risk bounds for the two risk minimizers, building upon the theoretical findings presented in Kiryo et al. 2017]; Niu et al. (2016).
We conduct extensive experiments on benchmark AD datasets obtained from Adbench Han et al. (022) compare the performance of our proposed risk-based AD (rAD) methods against various bascline methods.
Organization We discuss related work in Section 2 and provide a brief background on risk estimators in Section g We then introduce the two risk estimators that form the basis of our risk-based AD methods in Section Additionally, we present theoretical analysis Section 5 present experimental results Section 6 highlight limitations in Section 7 and conclude the paper in Section All proofs and additional experiments can be found in the supplementary material.
2 Related work
We direct readers to Ruff et al. (2021); Roth et al. (2022) for a comprehensive review of recent advancements in anomaly detection Lechniques (with particular focus on industrial anomaly detection). We remark that our primary contribution lies in delving deeper into the approach of risk estimation, technique that remains relatively unexplored within the context of anomaly detection. In the following, we provide a brief overview of the most relevant works related to our proposed risk-based anomaly detection methods.
AD methods Outlier detection (Hawkins 1980 Hodge & Austin 2004), novelty detection (aalu 1988
Pimentel et al., 2014), and AD are closcly related topics. In fact, these terms have been used interchangeably, and solutions to outlier detection and novelty detection are often used for AD and vice versa. AD methods can be generally classified into three types (i) density-based methods, which estimate the probability distribution
2

---

Published in Transactions on Machine Learning Research March/20244
of normal instances Lecun et al. methods, which learn a model that fails to reconstruct anomalous instances but succeeds to reconstruct
[006]] Li ct al. 2019 Parzen (9662)
Pincus 1995 (ti) reconstruction-basee
normal instances Dhillon et al. (2004); Hawkins (1974); ; Hawkins et al. (2002); Huang et al. (2006); Yan et al.
(2021), and (iit) one-class classification methods. We refer the readers to Ruff et al. 2021 for a comprchensive review of the three types of AD methods.
PU learning methods Regarding PU learning methods, they can be classified into three categories: biased learning, two-step techniques, and class-prior incorporation. Similarly to onc-class classification AD methods, biased PU learning methods make impractical assumption: they assune /label all unlabeled instances negative, sec c.g.: Lee & Liu (2003); Liu et al. (2003). Although the PU learning methods using two-step techniques do not have such assumption, they are heuristics since they first identify "reliable' negative examples and then apply (semi-) )supervised learning techniques to the positive labeled instances and the reliable negative instances, see e.g., Li Liu| (2003); Chaudhari & Shevade (2012). To have some theoretical guarantee, the fass-prior incorporation methods need assume that the class priors are known, see e.g.: du Plessis et. al. (2014); Elkan & Noto (2008); Hsieh el. al. (2019). We refer the readers to Bekker &
Davis (2020) and the references therein for more details the three types of PU learning methods. Methods Sakai et al. (0017) belong to
that rely on risk estimators du Plessis et al. 20114 2015 Kiryo et al. 2017
the third category.
3 Background on risk estimators
Let x and y E {+1,-1} be random variables with joint density p(x, y). The class-conditional densities are (z(x) P(zly = +1) and pa(z) = P(zly -1). Let Tp p(v = +1) and nn = p(y -1) be the class-prior probabilities for the positive and negative classes. We have TP nn = 1. Suppose the positive (P), " negative (A) and unlabeled (2) data are sampled independently as (P) = (=/) pp(x), (N) = {
Ph p(x), where pa( (L) = {
p(x) = <p)p(x) + (x).
Given (P), (N) and (2), let us consider binary classification problem from r to y. Suppose g Rd R is a decision function that needs to be trained from (P), (N) and (4), and f R (+1, -1 - R is a loss function that imposes a cost l(t, y) if the predicted output is t and the expected output is y. Under loss E, let us denote
R (0) E,,,, (()(())))
Ex (l)(())) (106)
E ((g()))))
R₂ (g) (x
-1)], [1) R, Ex-p, (m) [(g(()) -1)], R,), (g) - (( -1]]
Given l and assuming that T. is known (in practice, Tp can be effectively estimated from (P), (W) and (4)
du Plessis & Sugiyama 2013): Sacrens et al. (2022) ) our goal is to find g that minimizes the risk of g, which is defined by
R(g)= = ((x,)))p((m)) (( [<((()) v)]
R, (g) + + R, (g).
(2)
In ordinary classification, the optimal classifier minimizes the expected misclassification rate that corresponds
to using zero-one loss in (2) (t. v) = 0 if ty > 0 and (o-1(t,v) v) = 1 otherwise. We denote [(g) =
PN risk estimator In supervised learning when we have fully labeled data, R(g) can be approximated by PN risk estimator Rg) = RR (g) + n. R2 (g), where
1 np
R+(( =
(g(of))+1) +1), (T) nh (l((o)) -1).
(3)
PU risk estimator In PU learning when (A) is unavailable, du Plessis el al. 2014 2015 propose methods to approximate R(g) from (P) and (ZT). From we have n,, R, (g) R, (g) T R2 (g) which implies that
Kiryo el al. 2017
R(g) (R R2 ) + Ry (g)-
(4)
3

---

Published in Transactions on Machine Learning Research Mlarch/2024/
When satisfies the symmetric condition E(t, +1) E(t, -1) then we have R(g) 2Tp (g) Tp + R2 (g), which can be approxinated by
pu (g) = 2 2 R2 (g) - Ty + R-
(5)
where Rt(()) is defined in 33 and (g) (g) nl -i (g(xx))) -1), see du Plessis et al. (2014). When
satisfies the lincar-odd condition l(t, +1) [(t.
-1) - then Rgg) be approximated by
1
R R (g) = up np i=1
g( RT (0),
(6)
soc du Plessis ct al. (0155) The authors in Kiryo et al.
2017 propose non-negatioe PU risk estimator
R pa (g) R2 (g) max RS (g) RT (g)) (a)),
where (0) np E (g((x))-11) Note that R.pi (g) is a biased estimator.
NU risk estimator Similarly, considering NU learning when (P) is unavailable, see Sakai et al. (2017) ,
NU risk estimators can be formulated by combining the equation xp R (g) = RR (g) Tn R (g) (which is
derived from and 2 to obtain
R(g) = Tn (R - R,, + (g).
(8)
With a loss satisfying the symmetric condition, we have a nonconvex NU risk estimator
(1)) = 2n, Rr (g) - Tn + (g).
(9)
where (g) is defined in (3) and (g) = 1 The (g(x"") +1)- And with a loss satisfying the linear-odd condition, we get a convex NU risk estimator
1 Ru
R (g)
nn Thn
Σ o(x;)) (g).
(10)
i=1
Finally, Sakai et al. (2177] proposes to use linear combination between the PN, the NU, and the PU risk ofdu Plessis et al. 2014 4 2015
4 The proposed semi-supervised anomaly detection methods
In the previous section, we presented risk estimators for the PU learning problem where (W) is unavailable. Let us consider the setting where we have access to (P), (W) as well as (4)). We perceive semi-supervised AD as a binary classification problem from x to E (+1,+111 -1], where +1 represents the normal class and -1 represents the anomalous class. Our goal is to propose risk estimators for the risk in (2). Specifically, we propose two risk estimators for semi-supervised AD that lead to two risk-based AD methods.
If we take convex combination of ( and
obtain
(R - R. (g) (1 ()TRR (0) + n., R, (g)) R(g) al a(-Tm, +
aRt (g) (1 - a)T, (g) R, (g) ar,, R,t.
(11)
where a E 10,
The empirical version of I1 yields the following linear combination of PN and NU risk estimators:
,,, + (g) at, R (g).
Rt aRt
,, + (1)+++++)+++
(12)
While R²(2(() was also considered in Sakai al. (2017) they only focused on the set of linear classifiers with two specifie losses the (scaled) ramp loss and the truncated (scaled) squared loss (see (Skaii et. al. 2017
4

---

Published in Transactions on Machine Learning Research Mlarch/2024/
Section 4.1)). We consider a more general setting for Ra and also propose methods to choose appropriate regularization for R! to avoid negative empirical risks. In fact, R! may take negative values when & is unbounded due to the negative term -an, R, (g). Theorem summarizes the conditions that guarantee a nonnegative objective.
Inspired by R R.0l (g) in 77. we also propose the following nonnegative risk estimator:
(1) = n. R, (g) + pp R (g) + max (0, (t n. R (g))
(13)
where the max term is introduced since R - n, R (g) = RT must be nonnegative. Note that Rpp (g) was designed for the PU learning problem while we propose R!! for the AD problem which often assumes anomalies are rare. In other words, we put more emphasis on R (g) rather than R2 (g).
In Section we will establish the theoretical estimation error bounds and excess risk bounds for the minimizers of both minges R( (g) and minges R. (g). where 9 is some class function. We now present the practical optimization problems involved when using R (g) and R. R..
Optimization problems Suppose g is paramcterized by w, which needs to be learned from (P), (N) and (T). When R( in 13 is used, the corresponding optimization problem for AD is
a) aT
min
(g(x;)), -1) +
(g((x)) +1)
ylya
np
(14)
The [g((x)) +1)
nn (g(x)))+1)} AR(w)).
max {0, nu
Tly
where R some regularizer, and A 2 regularization parameter. And when R! in 12 used, the corresponding optimization problem is
(g(a"), +1) + (1 a)Tp
min
((g(x))++1)
Thu
(15)
TT
{g((x)) -1) aT
(g((z)), +1) + AR(w)
Unfortunately, the objective of 15 is not guaranteed to be nonnegative due to the negalive term amna Ein ((g(z)),+1), As pointed out by Kiryo et al. (2017), this can lead to serious overfitting problems. The following thcorem provides methods choose the regularizatin parameters such that the nonnegativity of the objective of 15p guaranteed.
Theorem Suppose there erist positive constants by, b2 and by such that
f(t, (tt--)) f(t. +1) bb/ and E(t, -(t,-1) 2 b2(bs - [t]).
(16)
(In Table we give examples of loss functions that satisfy 16 see their proofs in the supp. material.)
(i) We have
((gax))-11)
an
e(g(x)),+1) (1 a)mbbbb - ((1 - a)bb + +abi)
|x(x)))
Thy
(8) If choose X and R such that
nn
AR(w) 2 ((1 aa)b2 + abı)
Fn Σ nn
[g(2;")
b2bs
(17)
5

---

Published in Transactions on Machine Learning Research (March/200)
Table 1: Examples of loss functions satisfying 166
Name
E(t, = {(2) with ty
Bounded
(b), bs)
Hinge loss
max (0, 1 2)
X
(2,
Double hinge loss Squared loss
max (0. (1 - 2) -z}
X
(1,1/2,1)
2 - 1)²
(2,1/2,1/2)
max 1 =]2 if -1
Modified Huber loss
(4,1,1/2
-42
otherwise
Logistic loss
In(1 exp(-2))
(1,1,
Sigmoid loss
1/(1 + exp(z))
(1,1/2,1)
Ramp loss
max(0, min/1, (1 2]/
(1,1/2,1)
then the objective of always nonnegative.
(tii) Consider the specific case g(x) ((x)) where Rd - I82 is a feature map transformation. The following choices of X and R satisfy 17
R(w) = 1| and 2 ta--))bztbj)> F
where max([lo(x)]le 1..... nn, (note that, in practice,
1(1-a)b2bg
scole the data to have
R(w)= || and X Goo ((1 a b223 + ab, ) where Coo max :i I,..., nn (in practice scole the data to have Coo 1).
We consider both shallow and deep implementation of the rAD method. In the following, Th and x. = T
will denote estimates of the real class- class-prior probabilities Tp and respectively.
A shallow rAD method We plug in g(x) = (w, (())) in 15 (the empirical version of 12 where o: Rd RR4 is a feature map transformation, and choose the regularization method proposed Theorem (iii.. Specifically, we solve the following minimization problem:
Elw 6(x)), +1)+ (1 a) By
min Σ
a
E(w E( (xf),+11
np
(18)
Bu
The
n
Elw TruThle (b(x))=1))
an
lww ((xx), +1) AR(w).
nn
nn
A deep rAD method We plug in g(x) o(;; W) in 14 (the empirical version of 13
where W is a
set of weights of a deep neural network. Specifically, we train a deep neural network by solving the following optimization problem:
11) (1 a)
min W
lbbaf +1)
1=1
(19)
+ max (0, Bu
l'bax; W), +1)
(lo(o); + +R(W)).
where R can be any regularizer. Note that we focus on these specific implementations but it is also possible to consider a deep model with 115) or a shallow model with (14p. It is noteworthy to mention that in our initial numerical findings, we have observed that the shallow model in (18) frequently yiclds better results compared to the shallow model with (14. while the deep model in outperforms the deep model with 15

---

Published in Transactions on Machine Learning Research Alarch/2024)
5
Risk bounds
In this section, we establish the cstimation error bound and the excess risk bound for g' and g² which are the empirical risk minimizers obtained by minges (g) and minges R² (0), where G is function class.
Let 0² be the true risk minimizer, that is, g" arg mingeg R(g). Throughout this section, we assume that (i) g -= {g lo & C for some constant Cg. and (ii) there exists C2 > 0 such that sup|iis, max, E(t,y) I Cz. It worth noting that the set of linear classifiers with bounded norms and feature maps special case of Condition
g= [g(x)) (mt, ((x)) ||ollle S Cue [lx(x)) S Co), where H is a Hilbert space, @ a feature map, and Cw and Co are positive constants.
(20)
Given y, (g) is an unbiased estimator of R(g) but R! is a biased estimator. The following proposition
estimates the bias of R!!)) (see Incquality 21 and shows that, for fixed g, Re (g) and R!?) (g) converge
to Rg)) with the rate O In
Jp
vn. (see Inequality (22 and 23
Vin
Tup
Proposition 1 Consider a classifier g. Suppose there erists Pg such that R2 (g) 2 Pg and denole
"g = aam.C exp (- (-C3(/nnttt//m))
200,02
Then the bias of Ri! (g) salisfies
0 E (g)) R(g) S Cg.
(21)
Morcover, for any 8 5>0 O, we have the following inequalities hold with probability at least 1 - 8
(1 +1+ a)an
(1 - O)
(2) - (R)) < CcvIn(2/88)2
+
+
(22)
Vun
//p
Vnnu"
and
(1 + a) (1+a)m
(1 (1-a)Tp
S Cry (2200/2/)/2
+ Eg.
(23)
Vin
Up
VTn
Estimation error bound The Rademacher complexity of for a sample of size n drawn from some distribution q (see e.g., Mohri et al. (2018) is defined by R9) == E Ez [E, (sup R E (Z,)))
where ,,...,24 Zm are i.i.d random variables following distribution 4, 7 (Z,,.., ) o Jn are
independent random variables uniformly chosen from {-1,1), and o (01.... .o...... on). Similarly to KKryyo et al. 2017 Theorem 4), we can establish the following estimation error bound for g².
(+) Theorem (Estimation error bound for g¹) We assume that (1) there erists > 0 such that R (g) 2
for all e 9, (ti) if G then -g E G, and (iit) t 1(t, and t ++ l(t, -1) are Lipiscchitz continuous over {t: |d| s C.}. Denote e = an, Ct exp (- 00///attt/ For any 8 > 0, the following inequality hold with probabilily at least 1 - 6
2x000"
R(0¹) R(g") 8(1 + a) L.FnLe
Pp (9) + Rn. (1 (1-a) (1 + aam +
+ 2Cce/mm2//8/22
vmp + a ) + 2c.
(24)
Vnn
By using basic uniform deviation bound Mohri et al. (2018), the McDiarmid's inequality McDiarmid (1989), and Talagrand's contraction lemma Ledoux & Talagrand 1991), we can prove the following estimation error bound for g².
Theorem (Estimation error bound for g²) Assume that t E(t, 1) and + e(t, -1) are Le Le-Lpscchite continuous over (1: |0| S Cg). For any small 8 > 0, the following inequality hold with probabilily at least 1 8
R(g²) R(g") I 4(1 o) p (g) + 4(a + L (9) + Rn.. (G)+
(1-a)m (1 + a)
(25)
+ (Cc//n(6//6/7
7
nu
7

---

Published in Transactions on Machine Learning Research (larch/2024)
Note that Theorem 3 explicitly states the error bound for 92 with any loss function that satisfies the Lipschitz continuity assumption. The (scaled) ramp loss and the truncated (scaled) squared loss considered in Sakai et al. 2017) have Le = 1/2.
Excess risk bound The excess risk focuses on the error due to the use of surrogates for the 0-1 loss function. Denote 1* infgez T(g) and R inf R(g), where F the set of all measurable functions. By using (Bartlett et al.) 2006] Theorem 1) (see (22) in the supp. material), Theorem 2 and Theorem 3 we can derive the following excess risk bound for g' and g².
loss (see Definilion in the supp. material), then there exists convez, invertible, and nondecreasing transformation the with tor (0) and the following inequalities hold with probability at least 1 -
Corollary 1 I is a classifenniion
1(g') 1" S +E (BI - RR), [(9") 1' S (B2 -
- R*),
where B1 and B2 are the right hand side of 24 and 25 respectiucly.
6 Experiments
A. Experiments with shallow rAD
Baseline methods and implementation
We compare rAD with OC-SVM Schölkopf et al. (2001). ECOD Li ct al. 20022, COPOD Li ct al. 2020 semi-supervised OC-SVM Munoz-Mari et al. (2010), and
the PU methods using the risk estimator (g) given in (4 Note that Rg)(g) = Rpu (Ø) given in d5) if f (2)
satisfies the symmetric condition, and (g) Rpu (g) given in (6) if satisfies the linear-odd condition. We implement rAD and PU methods with 3 losses: squared loss, hinge loss, and modified Huber loss. For
rAD, we use l2 regularization and take ((x) in (18 i.e. no kernel is used. We set = 0.1 and Th = 0.8
(Th = 0.2) as default values for both the shallow rAD and the PU methods. Note that the real Tr of the datasets can be different.
Datasets We test the algorithms on 26 classical anomaly detection benchmark datasets from Han et al. (2022), whose Ta ranges from 0.02 0.4. The real nn of the datasets are given in the first column of Table 2 We randomly split each dataset 30 times into train and test data with ratio of 7:3, i.e. we have 30 trials for each dataset. Then, for cach trial, we randomly select 5% of the train data to make the labeled data and keep the remaining 95% as unlabeled data.
Experimental results In Table 2 we report the mean and standard error (SE) of the AUC (area under the ROC curve) over 30 trials of the 26 benchmark datasets. We obscrve that, on average, rAD outperforms the PU methods, OC-SVM methods, ECOD, and COPOD. The difference between the AUC of rAD and that of PU is large on the datasets with n. S 0.2 but is small when Tn is larger. We also notice that rAD with modified Huber loss often gives better results than rAD with square loss and hinge loss.
Sensitivity analysis for ア5 With 0.1, we run shallow rAD on the 30 trials for σ5 E (1-Tn. 0.7, 0.6] ( when -.- 1- no approximation is made). The results are reported in Table 3 for 9 benchmark datasets and the results of the 17 remaining datasets are given in Table 5 in the supp. material. From Table 2 5 we can see that we can obtain good results even if T is different from Tp. In fact, with a = 0.1, we get worse AUG means when n is close to Tp. The combination (a, xT) = (0.1, or (a, (Q,=5))= = (0.1, seem to be good choices across the datasets. Compared to the other two losses, we found the modified Huber loss to be robust to the values of Tp.
Sensitivity analysis for We run shallow rAD (with fixed N = 0.8) on the 30 trials of each datasct for a E (0.3, 0.7, 0.9). The results are reported in Table / for the 9 benchmark datasets and the results of the 17 rernaining datasets are given in Table D in in the supp. material. From Table 2 and we can see that the AUC means do not decrease significantly when we increase a (except for the dataset InternetAds). Hence, shallow rAD with o 0.8 is also robust to different values of a.
B. Experiments with deep rAD
8

---

Published in Transactions on Machine Learning Research Alarch/2024]
Table 2: Mean (and SE x 100)) of the AUC over 30 trials. The best means are highlighted in bold. d, n, and u.. denole the feature dimension, the sample size of the dataset, and the ratio of negative samples in the dataset.
datases (1.)))))
zAD
PU
OC-SVM
ECOD
COPOD
soml- OC-SV
squaze
binge
m-Huber
aquare
binge
m-Huber
pendighte
0.98( 0.22)
0.98
0.98( 0.22)
0.78( 4.79)
0. 0.78(4.83)
0.78(1.77)
0.86(0.31)
0.92(0.16)
0.90(0.17)
0.82(2.48)
(16, 6870, 0.02)
mammmographyy
0.91( (1:29)
0.91 (.91())29)
0 0.29)
0.87( 1.19)
0.87(1.499
(.87(1.188)
0.91(0.30) (.77()
0.91(0.29)
0.61(2.97)
(G. 11 180, 0.02)
cpndigits (64, 5216, 0.00)
1.00 0.07)
1.00().07)
1.00 0.06)
0.76( 2.93)
0. 0.76(3.02)
0.77(2.80)
(.46().53)
0.68(0.35) 0.00(0.40)
0.83(1.82)
Stampe 340, 0.09)
0.90( 1-44)
(.90(1.24)
0.900 1.16)
0.76( 3.37)
0. 037763
0.71(1.21)
0. 0.88(0.64) 0.65(1.74)
0.93(0.11)
0. 0.69(3.85)
cardio (21, 0.300
0.92( 2.03)
0.89(2.12)
0.93( 1.99)
0.83( 2.09)
0.81(1.93) (.81(2.31)
0.93(0.21) 0.94(0.23) (.87()))32)
0. (.79().)2)
InteractAds (1553, 1965, 0.19)
0.73( 3.00)
(.87())49)
0.76! 0.92)
0.64( 3:45)
(.77(0.66) 0.77(0.37)
(.00().54)
0.05(0:46) (.68(0
0. 0.64(00977)
Cardottrography (21, 2114, 0.22)
0.86( 1.32)
0.81(1.68)
0.88( 1.10)
0.81( 1.86)
0. 0.79(2.01)
0.72(0.41) (.82(1.75)
0.66(0.40) (.78().33)
0. 0.81(0.80)
usagie gamma (10, 020, 0.35)
0.78( 0.47)
(0.49))
0.78( 0.15)
0.78( 0.69)
0.77(0.71)
(.78()
(.56(().18)
0.68(0.11)) 0.64(..12)
0.54(0.32)
SpazaBase (87, 4207, 0.40)
0.94( 0.16)
(.94().11)
0.94( 0.16)
0. 0.93( 0.20)
0.98(0.19) 19)
(.93().21)
0.5-(00228
0. 0.09(0.21) 0.08().21)
0. 0.64((0.5))
satimaage 2 (36, 3803, 0.013
0.99( 0.17)
0.99(0.16)
0.99; 0.17)
0.80( 4.40)
0.77(4:29)
0.82(1.47)
1.00(0.00)
0.97(0.31) (.98().37)
0.51(4.52)
thyrold (6. 3772, 0.02)
1.00 0.05)
1.00(0.04)
1.00( 0.05)
0.86( 2.95)
(.87(2.95)
0.86(2.89)
(.93(0.31)
0.91(0.155 (.98().07)
0. 0.70(2.25)
vowels (12, 1466, 0.08)
0.85( 1.45)
0.82(1.39)
0.86 1.42)
0.663( 2.59)
00. (.62(2245)
0.64(2.66)
0.72(1-40)
0.58(1.20)
0.49(1.12)
0.69(2.55)
Wavedorm (21, 3443, 0.00)
0.83( 1.49)
0.81(1.80)
0.84( 1:33)
0. 0.60( 2.67)
0. 0.00(2.68)
(.67().64)
(.67(0.70)
0.61(0.71)
0.74(0.53)
0. 0.78(1.11)
CIRAR10-1 (512, 5263, 0.0G)
0.77( 0.84)
(.77(0.84)
0.77( 0.86)
0.39( 1.70)
0.59(1.83)
0.59(1.63)
0.61(0.50)
(.53(0.455
0.19(0.45)
0. 0.74(0.73)
SVIIN-1 1
0.83( (.46)
0.83(0.45)
0.84 0.47)
0.69( 1.42)
(.60(1.37)
(.69().333)
(.66())27)
0.63(0.33) (.63
0. 0.71
(G12, 10 000, 0.05)
20mews-1
0.64( 1.56)
0.61(1.14)
0.68( 1.70)
0.51( 1.57)
0.52(1.21)
(.53
0.52(0.71)
0 0.48(0.76)
0.48(0.71)
0.65(1.54)
(708, 2514, 0.06)
agnimee11
0.97( (.27)
0.93(0.69)
0.98( 0.18)
0.79( 1.28)
0.74(1.53)
0.81(1.12)
0.76(0.25)
0.75(0.24)
0.76(0.24)
0.89(0.10)
(788, 10000, 0.05)
azsazon
D.80( (.76))
0.76(0.87)
0.82( 0.69)
0.633( 0.98)
0.60(1.06)
(.63())))5)
0.51(0.39) 0.54(31.40)
0.48(0.39)
0. d.78(0.56)
(768, 10000, 0.05)
Imdd (708, 20000, 0.06)
0.82( 0.73)
0.77(1.00)
0.83( 0.63)
0.63( 1.30)
0.61(1.22)
0.63(1.35)
0.50(0.-43)
0.19(0.42)
0. 0.50(0.12)
0. 0.78(0.60)
yelp (768, 10000, 0.06)
0.89( (.85)
(.83(1.36)
0.90 0.73)
0.70 1.63)
0.67(1.67)
0.71(1.55)
0.61(0.31)
0.52(0.333 0.53(0.32)
0. 0.82(0.63)
muiet (100, 7603, 0.08)
0.96( 0.14)
(.96())
0.96 0.14)
(1.92( 0.59)
0.92(0.60) (.92().37)
0.80(0.24)
0.78(0.22) 0.75(..23)
0) 0.85(0.55)
campaign (82, 168, 0.11)
0.85( 0.16)
(0.1)))
0.85( 0.16)
0.84( 0.30)
0.84(..30)
0.84(0.30)
0.68(0.12)
0.78(0.08) 0.77(0.08)
0.77(0.41)
veriabead 240, 0.18)
0.72( 2.57)
(.75()
0.74 2.58)
0.39| 2.60)
0.58(2.68)
0.60(2.65)
0.48(2.18)
(.35(1.88) 0.43(1.38)
0. 0.68(2.59)
landaar (30, 6185, 0.21)
0.73( (.200)
0.73(0.21)
0.79( 0.19)
0.70( 0.52)
0. 0.70(0.51)
(0.1()
(.35())28)
0.42()).24) (.36)))25)
0.76(0.60)
(38, 6135, 0.30))
0.80( 0.22)
(0.02)
0.80 0.22)
0.80( 0.26)
(.55() (0.27) (0.5)
0.64(0.23) (.50
0. 0.67(0.72)
fanlt (27, 1941, 0.55)
0.64( (187))
(6)
0.84[ 1.
0.50(1.29) 0.58(1.27) (
( (.52()
0.47(0.47)
0.46(0.49)
0. 0.37(0.99)

---

Published in Transactions on Machine Learning Research (March /2024)
Table 3: AUC means of shallow rAD over 30 trials for different g.. highlighted in bold.
The significant changes the AUC means are
squarry pp
hinge/ xx
m Huber mp
Dataset
0.9
0.7
0.6
1- R
0.9
0.7
0.6
0.9
0.7
0.6
pendigits
0.96
0.98
0.98
0.98
0.94
0.98
0.98
0.98
0.97
0.98
0.98
0.98
mammography
0.90
0.91
0.91
0.91
0.90
0.91
0.91
0.91
0.90
0.91
0.91
0.91
optdigits
0.96
0.99
0.997
0.997
0.93
0.99
0.997
0.997
0.98
0.996
0.988
0.998
Stamps
0.80
0.80
0.82
0.82
0.81
0.81
0.81
0.80
0.80
0.80
0.80
0.80
cardio
0.91
0.91
0.92
0.92
0.87
0.88
0.88
0.89
0.92
0.93
0.91
Internet.Ads
0.77
0.77
0.70
0.60
0.85
0.86
0.87
0.87
0.86
Cardiotocography
0.58
0.89
0.89
0.87
0.85
0.57
0.87
0.90
0.90
0.90
magic.gamma
0.78
0.77
0.78
0.78
0.78
0.77
0.78
0.78
0.78
0.78
0.78
0.78
SpamBase
0.94
0.94
0.94
0.94
0.94
0.93
0.94
0.94
0,94
0.94
0.94
0.94
Table 4: AUC means of shallow TAD over 30 trials for different a. highlighted in bold.
The significant changes in the AUC means are
square/ /a
hinge /a
m- m-Hluber/a
Datasel
0.3
0.7
0.9
0.3
0.7
0.9
0.3
0.7
0.9
pendigits
0.98
0.98
0.98
0.98
0.98
0.98
0.98
0.98
0.98
mammography
0.91
0.91
0.91
0.91
0.91
0.91
0.91
0.91
0.91
optdigits
0.997
0.995
0.99
0.996
0.995
0.99
0.997
0.996
0.99
Stamps
0.83
0.82
0.82
0.81
0.82
0.81
0.80
0.81
0.81
cardio
0.92
0.91
0.91
0.88
0.87
0.85
0.93
0.93
0.93
InternetAds
0.79
0.87
0.85
0.71
0.65
Cardiotocography
0.87
0.87
0.87
0.86
0.85
0.83
0.90
0.89
0.88
magic.gamma
0.78
0.78
0.78
0.78
0.77
0.78
0.78
0.78
SpamBase Base
0.94
0.94
0.93
0.94
0.94
0.93
0.94
0.94
0.94
10

---

Published in Transactions on Machine Learning Research (March/2024]
Baseline methods and implementation We compare deep rAD with the Latent Outlier Exposure method (T.OF) Qiu et al. (2022), the deep semi-supervised AD method (deep SAD) Ruff et al. the PU learning method with nonnegative risk estimator and sigmoid loss (unPU) Kiryo et al. 2017)) For deep SAD and nnPU, we use default hyperparameter settings and network architectures as in their original implementation by the authors. We use the same network architectures as deep SAD for experiments on FashionMMNIST and MNIST datasets. For experiments on CIFAR-10, the network architecture from nnPU is used. In deep rAD, the optimization problem in (19) is solved using ADAM. We implement 4 losses for deep rAD: squared loss, sigmoid loss, logistic loss, and modified Huber loss. We set = 0.1 and Tp = 0.8 (thus " = 0.2) as default values for deep rAD.
(2020) and
Datasets We test the algorithms on 3 benchmnrkk k-classes-out datasets: MNIST, Bashion-MNIST and CIFAR-10 10 (all have 10 classes). We use AD setups following previous works Chalapathy et al. (0018); Ruff ct al. (2020): for each nn E (0.01, 0.05, 0.1, 0.2), we set one of the ten classes to be a positive class, letting the remaining nine classes be anomalics and maintaining the ratio between normal instances and anomaly instances such that, the setup has the required Th (so we have 10 selups corresponding to 10 classes). We note that the anomalous data in our generation process can originate from more than one of the nine classes (unlike in the sctup of deep SAD where the anomaly is only from onc of the nine classes). For cach Nn: we repeat this generation process 2 times to get 20 AD setups (or 20 trials). Then, in each trial, we randomly choose "y (with "/ E (0.05, 0.1, 0.2]) portion of the train data to be labeled and keep the remaining (1 m)) portion as unlabeled data. Note that we make the labeled data for nnPU only from normal instances. To make labeled data for deep rAD and deep SAD, (1 - Tn) portion taken from the nnPU labeled data (which contain only normal instances), and the remaining H, portion is taken from the anomalous instances. Hence, the number of labeled anomalous instances for deep rAD and deep SAD is about (7x x Ta) portion of the train data.
Experiment results In Figure D we report the mean and standard deviation (std) of the AUC over 20 trials on the datasets with inercasing pollution ratio T. and default 7 = 0.05. The results for n E (0.1, and 3 show that, on CIFAR-10, LOE performs the best and deep rAD methods on average provide better AUC than deep SAD and nnPU; deep rAD and deep SAD have similar performance when x, = 0.01 but their AUC difference is significant when nr is increased. On FMNIST, deep TAD methods, on average, are better than the others when Tn is increased but the AUC improvement is small. On MNIST, deep SAD is best; and when either nn or n is increased, deep rAD catch up with deep SAD while LOÉ gives worse AUC than the others. Deep rAD with quadratic loss underperforms the other TAD methods on MNIST and FMNIST. On average, deep rAD with logistic loss performs best among the rAD methods. Tt is also interesting to note that the presence of anomalies from multiple classes, the performance of deep SAD degrades over the performance reported in Ruff et al. (2020). The degradation is more severe for CIFAR-10.
are given in Figures 2
and 3 Figures 1 2
To observe the impact of the amount of labeled data, report the results for the datasets with т. = 0.1 and E (0.05, 0.1, 0.2) in Figure 5 We observe that all the semi-supervised methods improve when we increase 7 From 7 = 0.05 to 7 = 0.1 (i.e., 5% more labeled data), deep rAD methods show a significant. improvement in performance.
Sensitivity analysis for TH. We run deep rAD with a = 0.1 on the 20 trials of cach datasct for Ty e {1-Th, Tn, 0.9, )a} (when Th = nay it an exact estimation of Tp, and when Th. - Ta, we can say Tp is bad estimation of xp)- We report the result in Table 7 in the supp. material. Again, we see that oe is not necessarily precise estimation of upi and TF) = (O.I, 0.8) or ()) (0.1,0.7) are good settings. These results are consistent with the results of shallow rAD.
Sensitivity analysis for a We fix के = 0.8 and run deep rAD with additional values of a E (0.5, 0.9] (a = 0.1 is the default setting). We report the results for the datasets with म. = 0.1 and 7 = 0.05 in Figure 4 The results for the datasets with other values of n. and n are given in the supp. material. We observe that CIFAR-10, AUC decreases when is increased; however, the difference is not significant. On FMNIST and MNIST, deep rAD with Tf 0.8 is quite robust to the change of a.
11

---

Published in Transactions on Machine Learning Research (March/200)
Besetines: rAD(ours):
DespSAD
rnPu
Lof-Soft
LOE Hard
Baselines:
DeepSAD
LOE Logistic
Squared
Sigmoid
rAD(ours):
Squared loss
Sigmood
Hubber loss
MINIST
MNIST
1.0
22
0.6
0.6
rMNIST
1.0
0.8
AW
0.6
CIFAR-10
CIFAR-10
1.0
20
A0C
0.8
1
0.01
0.05
0.1
0.2
0.01
0.05
0.1
0.2
Pullution rebio the training datanat
Pollution rakio in the training dalasal
Figure 1: AUC mean and std over 20 trials with various Figure 2: AUC mean and SE over 20 trials with various Ta and = 0.1
and 72- 0.05
Baselines:
DecoSAD
0 nnPU
LOE Soft
LOE Hard
MNIST
FMNIST
CIFAR-10
rAD(ours):
Squared loss
Slamoid loss
Logistk loss
N-Hubber loss
rÅD with Square Loss
rAD with Sigmoid Loss
MNIST
1.0
1.0
10
0.9
0.8
FMNIST
0.7
0.8
rAD with Logistic Loss
rAD with M-Hubber Loss
CIFAR-10
AUC
T731
0.8
0.1
Pollution rato Rt the training cataset
Figure 3: AUC mean and std over 20 trials with various Figure 4: AUC mean and std over 20 trials at various Wy and default 7 = 0.2
a for the datasets with " = 0.05 and n, = 0.1
7 Limitations
On the implementation side, although the experiments have shown that our rAD methods are quite robust to the changes of the parameters and TH, we still have to lune them to obtain the best AD performance. Furthermore, solving the optimization problem in (19 is challenging for very large-scale dataset since the max operator does not allow parallel computations. On the thcoretical side, although the risk bounds are we still need the assumption that Tp and _n are
established for the proposed risk minimizers in Section known in advance, which is limitation.
12

---

Published in Transactions on Machine Learning Research March/2024
Baselines:
DespSAD
nnPU
rAD(ours):
Squared loss
Sigmoid loss
Logistic lces
M-Hubber loss
MNIST
0.90
FMNIST
1.0
0.9
CIFAR-10
0.8
i
0.8
T
0.05
0.1 Ratio data in training dalaset
0.2
Figure 5: AUC mean and std over 20 trials at various for fixing nn 0.1
8 Conclusion
With semi-supervised classification based on risk estimators, we have introduced a shallow AD method equipped with suitable regularization as well a deep AD method. Theoretically, we have established the estimation error bounds and the excess risk bounds for the risk minimizers. Empirically, the shallow AD methods show significant improvement over the bascline methods while the deep AD methods compete lavorably with the baselines. Let conclude the paper by giving some possible future research directions that address the limitation given in Section 7 One possible research direction is to develop method that can learn the best combination of (a, mT) from the available data. On the other hand, our experiments have shown that using a = 0.1, precise estimation Tp and nn are not necessarily needed to obtain good accuracy in terms of AUC. Hence, another possible research direction would be to study the theoretical bounds of the risk minimizers with मp and T. replaced by some estimates. Finally, investigating effective optimization techniques to tackle the nonconvex Problem (19) is also important research direction aimed at overcoming the difficulties associated with handling exceedingly large-scale datasets.
We express our sincere appreciation to the reviewers and the action editor for their comments, which greatly helped improve the paper.
Acknowledgement
13

---

Published in Transactions on Machine Learning Research Mlarch/2024/
References
Peter I. Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-155 2006. doi: 00.1188/01614455500000000000
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109: 719 760, 2020.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. Journal of Machine Learning Research, 11(99)22973 3009, 2010. URL
htmll
Raghavendra Chalapathy, Aditya Menon, and Sanjay Chawla. Anomaly delection using one-class neural networks. 02 2018. http:/ wtwp://voorr/bb//122...00
Varun Chandola, Arindam Banerjec, and Vipin Kumar. Anomaly detection. ACM Computing Surveys, 41(3): 1 1-88, 7 2009. ISSN 0360-0300. doi: 0114451554180015511 URL https://dl http://lllaac....ommmmmmmmm. btt..orgg//o//111114/0.
1541880 1541888
Sncha Chaudhari and Shirish Shevade. Learning from positive and unlabelled examples using maximum margin clustering. In Proceedings of the 19th International Conference on Neural Information Processing Volume Part III, ICONIP'12, pp. 465 055-473, Berlin, Heidelberg, 2012. Springer-Verlag, ISBN 9783642314862. doi: 10.1007/978-3-6422
56. URL /ttt::://o01..0r/11000100//////1
Inderjit S. Dhillon, Yuqiang Guan, and Brian Kulis. Kernel k-mcans: Spectral clustering and normalized cuts. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '04, pp. 551-556, New York, NY, USA, 2004. Association for Computing Machinery, ISBN 1581 138881.
doi: 10. 10.1145 001144//001005201411118 URL https: 10tg:/0..11//01102211111100 1014118
Marthinus du Plessis and Masashi Sugiyama. Semi-supervised learning of class balance under class- prior change by distribution matching. Newral networks the official journal of the International Neural Network Society, 50C:110 119, 11 2013. doi:
Marthinus C du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https :
Marthinus C du Plessis, Gang Niu, and Masashi Sugiyama. Couvex formulation for learning from positive and unlabeled data. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1386-13833 Lille, Francc, 07 09 Jul 2015. PMLR. URL
mlr. wwwpprressvve. v37 wttp:///ppss............. html
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08 pp. 213 220, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605581934. doi: 10.1 01114551141190.14000
URL htttps://do..org/1... 1145/14411800 1401920
Sarah M. Frfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High-dimensional and large-scale anomaly detection using a lincar onc-class SVM with deep lcarning. Pattern Recognition, 58:121 134, 10 2016. ISSN 0031-3203. doi: :.11016/.PATTOO..2211....0
Nico Görnitz, Marius Kloft, and Ulf Brefeld. Active and semi-supervised data domain description. Lect Notes Artif Intell, volume 5781, pp. 407-422, 09 2009. ISBN 978-3-644--41199-2 doi: 10.1007/ 978-3-642-04180-5 44.
Nico Görnitz, Marius Kloft, Konrad Rieck, and Ulf Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research, 46:235-266 262, 2013.
14

---

Published in Transactions on Machine Learning Research Mlarch/20244
Nico Görnitz, Luiz Alberto Lima, Klaus-Robert Miller, Marius Kloft, and Shinichi Nakajima. Support vector data descriptions and k -means clustering: One class? TEEE Transactions on Neural Networks and Learning Systems, 29(9):3994 4006, 2018. doi:
Songqiao Han, Xiyang Hu, Hailiang Huang, Mingqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. In Neural Information Processing Systems (NeurIPS), 2022.
D. M. Hawkins. Identification of outliers. Monographs on applied probability and statistics. Chapman and Hall, London fu.a.], 1980. ISBN 041221900X. URL /ttpp://goo.gbt www.:/bbvv..ddDB/./e 2./DB HTTCC///AT-CCCHEEAECCCC 1002:/540000e100ppp0222277772772
Douglas M. Hawkins. The detection of errors in multivariate data using principal components. Journal of the American Statistical Association, (3344)) 344, 1974. ISSN 01621459.
Simon Hawkins, Hongxing He, Graham Williams, and Rohan Baxter. Outlier detection using replicator neural networks. In Yahiko Kambayashi, Werner Winiwarter, and Masatoshi Arikawa (ods.), Data Warehousing and Knowledge Discovery, pp. 170 180, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg. ISBN 978-3-540-46145-6
Victoria Hodge and Jim Austin. A survey of outlier detection methodologics. Artificial Intelligence Review, 22:85-126, 10 2004. doi: 10.1023 10.1023/B:ATRR.OOO 2EATRF.0000045502.11 10941..
Yu-Guan Hsich, Gang Niu, and Masashi Sugiyama. Classification from positive, unlabeled and biased negative data. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseorch, pp. 2820-2822 PMLR, 09- 15 Jun 2019.
Ling Huang, XuanLong Nguyen, Minos Garofalakis, Michael Jordan, Anthony Joseph, and Nina Taft. In- network pca and anomaly detection. In Schölkopf, Platt, and T. Hoffman (eds.), Advonces in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https://proceeediggs www.eeedddnnnnnnnnss.e.eo cc/ .ddf
Hyunjun Ju, Dongha Lee, Junyoung Hwang, Junghyun Namkung. and Hwanjo Yu. Pumad: Pu met- ric learning for anomaly detection. Information Sciences, 523:167-183, 2020. ISSN 0020-0255. doi: http:////ddooorgr/.011//////.// pi1/800000255523322222
Shehroz S. Khan and Michael G. Madden. One-class classificatin:: taxonomy of study and review of techniques. The Knowledge Engineering Review, 378))344-374, 2014. doi: 101077/50668881100443x..
Bangalore Kiran, Dilip Thomas, and Ranjith Parakkal. An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. Journal of Imaging, 4, 01 2018. doi: 10.3390/jimaging44020
Ryuichi Kiryo, Gang Niu, Marthinus D du Plessis, and Masashi Sugiyama. Posilive-unlabeled learning with non-negative risk estimator. In 1. Guyon, U. Von Luxburg, S. Bengio, Н. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL https://proceedinggs www.ips...cooo sew.rips.cc/paper/0001//// 7CC55c80007742277777777777222222 Paper.pdf
Yann Lecun, Sumit Chopra, and Raia Hadsell. A tutorial on energy- based learning. MIT Press, 01 2006.
Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: Isoperimetry and Processes. Springer Berlin Heidelberg, Berlin, Heidelberg, 1991. ISBN 978-3-444-20212-44 doi: 10.10077997-----2222222- 6. URL
Wee Sun Loe and Bing Liu. Learning with positive and unlabeled examples using weighted logistic regression. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML/03, pp. 448 455.. AAAI Press, 2003. ISBN 1577351894.
15

---

Published in Transactions on Machine Learning Research Mlarch/20244
Dan Li, Dacheng Chen, Lei Shi, Baihong Jin, Jonathan Goh, and See-Kiong Ng. Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In International Conference on Artificial Neural Networks, 2019.
Xiaoli Li and Bing Liu. Learning classify texts using positive and unlabeled data. In Proceedings of Eighteenth International Joint Conference Artificial Intelligence (IJCAI-03): 2003; Acapulco, Merico, pp. 587-594, 01 2003.
Zheng Li, Yue Zhao, Nicola Botta, Cezar lonescu, and Xiyang Hu. COPOD: copula- based outlier detection. In IEFF International Conference on Data Mining (ICDM). TEFE, 2020.
Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Tonescu, and George Chen. Eood: Unsupervised outlier detection using empirical cumulative distribution functions. IEEE Transactions on Knowledge and Data Engineering, pp. 1 1, 2022. doi: 10.1109 10.1109/TKDE. 302
B. Liu, Y. DDai, X. Li, W.S. Lee, and P.S. Yu. Building text classifiers using positive and unlabeled examples. In Third IEEE International Conference Data Mining, pp. 179-186, 2003. doi: MLLLLO/IIIMM.00000 1250918...
Colin McDiarmid. On the method of bounded differences. In Surveys in Combinatorics, 1989.
Mchryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2 edition, 2018. ISBN 978-0-262-03940--6
Jordi Munoz-Mant Mari, Francesca Bovolo, Luis Gomez-Chova, Lorenzo Bruzzone, and Gustavo Camp-Vall Valls. Semisu- pervised one-class support vector machines for classification of remote sensing data. IEFE Transactions on Geoscience and Remote Sensing, 48(8):3188 3197, 2010. doi: 1EL109/TGRS, 2T000/TTTGSS.2200245555566
Gang Niu, Marthinus C. du Plessis, Tomoya Sakai, Yao Ma, and Masashi Sugiyama. Theoretical comparisons of positive-unlablled learning against positive-negative learning. In NIPS'16, pp. 1207-1215, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton van den Hengel. Deep Learning for Anomaly Detection: A Review. ACM Computing Surveys, 54(2), 7 2020. doi: 10.1145/3430950. //arxiv.orge wwgg/ba/////////// 0tt.0000220000tt////aoo0 doi. org/10. 1145/3439955
URL http:
Emanuel Parzen. On Estimation of Probability Density Function and Mode on JSTOR. The annals of mathematical statistics, 33):1106 1076, 1962.
Marco A.F. Pimentel, David A. Clifton, Lei Clifton, and Lionel Tarassenko. A review of novelty detection. w0t.siiprro2211.......
Signal Processing, 99:215 249, 2014. ISSN 0165-1684, doi: URL
R. Pincus. Barnett, and lewis t.: Outliers in statistical data. 3rd edition. wiley & sons 1994, xvii. 582 pp., f.49.55. Biometrical Journal, 37(2):256-256, 1995. doi: httpss//doi..org110 /00.1000//mmj4447177770000 URL
Chen Qiu, Aodong Li, Marius Kloft, Maja Rudolph, and Stephan Mandt. Latent outlier exposure for anomaly detection with contaminated data, In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (cds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18153-18167. PMLR, 17-23 Jul 2022. URL https: hatee////occeddneee wariigg.m.r.. www.m.r...ssss.e. www.press/vvve htteess/vi2//ii2222 html
Karsten Roth, Latha Pemula, Joaquin
Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards Total Recall in Industrial Anomaly Detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14318-14328, 2022. doi: (005500/axxi..106.0666 URL
https://arxiv..rrt http:///axxvv..og/bbbss wwt..r////bb//10..0222222 08265v2
Peter J Rousscouw. Multivariate estimation with high breakdown point. Mathematical statistics and applications, 8(37):283 297, 1985.
16

---

Published in Transactions on Machine Learning Research Mlarch/2024/
Peter J. Roussceuw and Katrien Van Driessen. A fast algorithm for the minimum covariance determinant estimator. Technometrics, (3):2211 223-22 1999. ISSN 15372723. doi: 10. 10.1050/0040170000 001.070619999114866777 10485670.
URL https://www htop:::/www.ttanffolliiiom
tandfonling ine. http://wwwwooooooooooooottttttra
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep onc-class classification. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learaing, volune 80 of Proceedings of Machine Learning Research, pp. 4393 4402. PMLR, 10-15 Jul 2018.
Lukas Ruff, Robert A. Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. In International Conference Learning Represenlations, 2020. URL
Lukas Ruff, Jacob Kauffmann, Robert Vandermeulen, Gregoire Montavon, Wojeiech Samck, Marius Kloft, Thomas Dietterich, and Klbes-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, PP:1-40, 02 2021. doi: 10.1099/PRR0C.22223300222
Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs a classifier to new a priori probabilities: simple procedure. Neural computation, 14:21-41, 02 2002. doi: 10.1162/ 089976602253254446
Tomoya Sakai, Marthimus Christoffel du Plessis, Gang Niu, and Masashi Sugiyama. Semi-supervised classification based on classification from positive and unlabeled data. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2998 3006. PMLR, 06 11 Aug 2017. URL https://proceeddngs. mll cress/v70/sakai172 html
Ychuda Salu. Models of neural novelty detectors, with similaritics to cerebral cortex. Biosystems, (2):99 113, 1988. ISSN 0303-2647. doi:
URL https
/ www..cciencedirett.com ort.a//ccw...oocccccc//////////a
Bernhard Schölkopf, John Platt, John Shawe-Taylor Alexander Smola, and Robert Williamson. Estimating support of a high-dimensional distribution. Neural Computation, 13:1443-1477 07 2001. doi: 10.1162/ 0899766011502661955
David M.J. Tax and Robert P.W. Duin. Support vector domain description. Pattern Recognition Letters, 20 (11-13):1191-1199) 11 1999. ISSN 0167-8655. doi: 111001650117-85555000
David M.J. Tax and Robert P.W. Duin. Support vector data description. Machine Learning, 54:45 66, 2004.
doi: 10.1023 B..1023/B:MACHE BBBB/BBAAAHHOOOO 0000008084.608110
Jue Wang and Anoop Cherian. Gods: Generalized one-class discriminative subspaces for anomaly detection. Tn 2019 TEEE/CVF International Conference on Computer Vision (ICCV), pp. 8200-8210, 8210, 2019. doi: 10.1109/1CCO
Xudong Yan, Huaidong Zhang, Xuemiao Xu, Xiaowei Hu, and Pheng-Amn Heng. Learning Semantic Context. from Normal Samples for Unsupervised Anomaly Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3):3110 3118, 5 2021. URL https://ojs maaii.og/imge herg/index... http://adex..pp/AAAA/arrrcc.. ptpp/AAA//article/ view/16420
Arthur Zimek, Frich Schubert, and Peer Kröger. A survey on unsupervised outlier detection in high-dimensional numerical data. Statistical Analysis and Data Mining, 5:363 387, 10 2012. doi: 10.1002/ssa /sam. 11161.
17


Reviews:
[{"review_id": "TCrCb2W2jY", "reviewer": "Reviewer_5ape", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n-\tThis paper is well-written.\n-\tExtensive experiments and detailed theoretical analysis are included.\n-\tThe proposed method is reasonable.\n\n### Weaknesses\n\n1.\tExperiment setting.\n\nDifferent from the previous methods, this paper also assumes that we can access the negative set, which is usually hard to label. They should present the number of known negative samples. This information is helpful for us to estimate the labeling overload.\n\n2.\tNecessity of the theoretical analysis.\n\nIn theorem 1, the authors prove that the risk defined in Eq (15) is non-negative. I appreciate the proof and theoretical analysis. However, since they use Eq (15) as a loss function, does it have to be nonnegative? The necessity of Theorem 1 should be clarified. \nIn my opinion, this prove ensures that we can justify how well is the model trained based on the training loss, which requires the training loss curve.\n\n3.\tTwo types of methods.\n\nAs the authors state, this paper presents two types of methods. It is important to analyze more about their differences and relationship.\n\n-\tWhy do the authors only apply the Eq (15) to the shallow method and the Eq (14) to the deep one? Although the authors state that the cross application is also doable (last sentence in Section 3), it is still necessary to check the final performance.\n\n-\tIs the Eq (15) inherently better than Eq (14)? If we come across the anomaly detection task, how to choose the method in practice?\n\n4.\tNotations in Section 2.\nThe upper notation for risk is just pure digits, making it hard to read. A more accessible notation is expected.\n\nAfter rebuttal, I have changed my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the semi-supervised anomaly detection. Different from prior works, this paper assumes that a positive set, a negative set and an unlabeled set are attained during training. Directly derived from the risk estimators, the authors propose two types of methods: a shallow one with unbiased risk estimator and a deep model with biased risk estimator. Complete theoretical analysis and extensive experiments are provided to verify the model effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors need to clarify the experiment setting, necessity of theoretical analysis and the relationship between two types of methods. The writing in Section 2 can also be improved.", "broader_impact_concerns": "This is an theoritical paper, which can insipre some works in anomaly detection area.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VC4IcUDfb4", "reviewer": "Reviewer_C1v7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. **New Methods:** The paper introduces two new approaches for finding anomalies using semi-supervised learning. \n\n2. **Solid Theoretical Background:** The authors provide a detailed mathematical explanation for their methods. This includes calculations that show how well the methods should work and their limitations.\n\n3. **Helpful Techniques for Choosing Parameters:** The paper offers practical ways to choose certain settings (regularization parameters) for these methods, making them easier to use effectively.\n\n### Weaknesses\n\n1. **Handling Large Data:** The paper could talk more about how these methods work with very big sets of data.\n\n2. **Limited Application Scope:** Showing how these methods can be used in specific real-life situations would help understand their practical use.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper makes several significant contributions to the field of anomaly detection, particularly in the context of semi-supervised learning. Here is a brief overview of these contributions:\n\n1. **Introduction of Novel Semi-Supervised Anomaly Detection Methods:** The paper presents two new methods for anomaly detection, one based on shallow learning and the other on deep learning. These methods are designed to operate in a semi-supervised framework, addressing the common issue in anomaly detection where it is impractical to assume that all unlabeled training data are normal instances.\n\n2. **Theoretical Foundations:** The authors establish a strong theoretical basis for these methods by deriving estimation error bounds and excess risk bounds. This contribution is significant as it provides a mathematical understanding of the performance and limitations of the proposed methods.\n\n3. **Techniques for Parameter Selection:** The paper proposes techniques for selecting appropriate regularization parameters in the context of these semi-supervised anomaly detection methods. This is a practical contribution that aids in the effective application of these methods.\n\n4. **Extensive Experimental Validation:** The authors conduct a comprehensive set of experiments to demonstrate the effectiveness of the proposed methods. These experiments not only validate the theoretical claims but also show the practical applicability of the methods in various scenarios.\n\n5. **Advancing Semi-Supervised Anomaly Detection:** By addressing the limitations of one-class classification in anomaly detection, the paper contributes to the advancement of semi-supervised learning approaches in this field.\n\nThese contributions collectively represent a significant step forward in the anomaly detection domain, particularly in scenarios where labeled data is scarce or imbalanced. The combination of theoretical rigor and practical applicability makes this work a valuable addition to the literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please discuss more the large data and application scope.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oARvL6pXMy", "reviewer": "Reviewer_hBqs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is easy to follow.\n\n2. The proposed method is novel.\n\n3. The motivation is clear.\n\nWeaknesses:\n\n1. I know the authors discussed several related works in the Introduction. However, it is better to add a subsection to Section 2 to provide more background on the related works. \n\n2.  The baseline methods are very limited. For example, the baselines in Section 6 A are too old. OC-SVM was proposed in 2001. The semi-supervised OC-SVM was proposed in 2010. \n\n3. It seems like most of the experimental results are in supp. materials. I suggest moving them into the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The authors approach anomaly detection as a semi-supervised binary classification problem, involving positive, negative, and unlabeled datasets (potentially with anomalies). They introduce two risk-based AD techniques: a shallow AD method utilizing an unbiased risk estimator, and a deep AD strategy using a nonnegative risk estimator.\n\n2. For the shallow AD method, the authors develop techniques to select appropriate regularization, ensuring nonnegative empirical risk.\n\n3. The authors derive estimation error bounds and excess risk bounds for both risk minimizes\n\n4. Through comprehensive experiments on benchmark AD datasets, the authors evaluate the efficacy of our risk-based AD (rAD) methods in comparison to various standard methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suggest adding more compared methods (baselines) to demonstrate the effectiveness of the proposed method.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes two semi-supervised anomaly detection methods based on risk estimators - a shallow method using an unbiased risk estimator and a deep method using a biased risk estimator.

Reviewers found the paper to be well-written and easy to follow. The theoretical foundations and analysis were seen as major strengths. However, some concerns were raised about the experimental baselines being dated and the application scope being limited.

In response, the authors acknowledged some limitations and made several changes. They provided more details on the experimental setup and negative samples. The necessity of theoretical proofs was clarified. Discussion was added on handling large data and optimization challenges. Notations were improved and experiments moved to the main text.

While one reviewer still felt the baselines could be updated, others felt the paper addressed their initial feedback well. On balance, most reviewers felt the paper demonstrated novel contributions and recommended acceptance, pending minor revisions. 

In summary, through discussion and revision the paper was strengthened. While some open questions remain, the novel methods, solid foundations and positive evaluation overall suggest the work represents a solid step forward for semi-supervised anomaly detection. I recommend "accept with minor revision". 

To be clear, the revisions required for acceptance include the addition of recent baselines, both shallow and deep ones, for comparison. It is important to note that the request is not for state-of-the-art performance, but rather to evaluate and understand how the method performs against recent and robust baselines.

Actually I'm not convinced by the response:
> In terms of shallow anomaly detection (AD) methods, despite the introduction of OC-SVM in 2001, it remains among the top unsupervised shallow approaches. Notably, Han et al. (2022) highlighted it as one of the three representative shallow methods alongside Isolation Forest (IForest) (Liu et al., 2008) and Empirical-Cumulative-distribution-based Outlier Detection (ECOD) (Li et al., 2023). According to Table D4 and Section 4.2 of Han et al. (2022), there is no statistically significant superiority among these three methods. Given our paper’s focus on overcoming the impractical assumption of one-class classification AD methods, we opt for OC- SVM as a baseline unsupervised method in our experiments. It’s worth noting that, to the best of our knowledge, even though semi-supervised OC-SVM was introduced in 2010, it, along with PU methods, stands as a state-of-the-art choice for semi-supervised shallow AD methods, as discussed in (Ruff et al., 2021).


- Just because a method is described as "representative" does not necessarily mean it is effective or of good accuracy. Being representative could simply indicate that the method was one of the earliest or most widely referenced in its category.
- I skimmed (Ruff et al., 2021) and didn't find the statement that "semi-supervised OC-SVM ... along with PU methods, stands as a state-of-the-art choice for semi-supervised shallow AD methods." Therefore, I suggest a more thorough justification of the chosen baselines.
 

> Concerning deep AD methods, we acknowledge the reviewer’s suggestion to incorporate additional baseline methods in our experiments. However, as our main contribution revolves around a more in-depth exploration of risk estimation, an approach that has been relatively underexplored in the realm of AD, we *believe* that utilizing deep SAD and the deep PU learning methods as baselines is pertinent for illustrating the effectiveness of our proposed deep risk-based AD method. 

I don't understand the rationale or the process behind forming this belief.
Decision: Accept (poster)