Revisiting Non-separable Binary Classification and its Applications in Anomaly Detection
Authors: Matthew Lau
ISMAILA SECK
Athanasios P Meliopoulos
Wenke Lee
Eugene Ndiaye
Venue: Accepted by TMLR ()

Abstract: The inability to linearly classify $\texttt{XOR}$ has motivated much of deep learning. We revisit this age-old problem and show that $\textit{linear}$ classification of $\texttt{XOR}$ is indeed possible. Instead of separating data between halfspaces, we propose a slightly different paradigm, $\texttt{equality separation}$, that adapts the SVM objective to distinguish data within or outside the margin. Our classifier can then be integrated into neural network pipelines with a smooth approximation. From its properties, we intuit that equality separation is suitable for anomaly detection. To formalize this notion, we introduce $\textit{closing numbers}$, a quantitative measure on the capacity for classifiers to form closed decision regions for anomaly detection. Springboarding from this theoretical connection between binary classification and anomaly detection, we test our hypothesis on supervised anomaly detection experiments, showing that equality separation can detect both seen and unseen anomalies.

Published in Transactions on Machine Learning Research (06/2024)
Revisiting Non-separable Binary Classification and its Applications in Anomaly Detection
Matthew Lau
mlau,00gatechh. edu
School of Cybersecurity and Privacy, Georgia Institute of Technology
Ismaila Seck
ismaila Camaila&leego...o at
Lengo AI
Athanasios P Meliopoulos
sokis, wwwnaegttcccc.mm edu
School of Electricol and Computer Engineering, Georgia Institute of Technology
Wenke Lee
wenker@ce.gatecce edu
School of Cybersecurity and Privacy, Georgia Institute of Technology
Eugene Ndiaye
ndiaye com
Apple
Reviewed on OpenReview: https: // openreview. net/ forum? id= =0J846BXh1
//
Abstract
The inability to linearly classify XOR has motivated much of deep learning. We revisit this age- old problem and show that linear classification of XOR is indeed possible. Instead of separating data between halfspaces, we propose slightly different paradigm, equality separation, that adapts the SVM objective to distinguish data within or outside the margin. Our classifier can then be integrated into neural network pipelines with a smooth approximation. From its properties, we intuit that equality separation is suitable for anomaly detection. To formalize this notion, we introduce closing numbers, a quantitative mcasure the capacity for classifiers to form closed decision regions for anomaly detection. Springboarding from this theoretical connection between binary classification and anomaly detection, we test our hypothesis on supervised anomaly detection experiments, showing that equality separation can dctect both seen and unseon anomalics.
1 Introduction
Linear classification of non-linearly separable dataset A common belief in machine learning is that linear models (like the perceptron (Minsky & Papert 1969)) cannot model some simple functions, shown by the example XOR (exclusive OR) in classic tooks (e.g. Goodfellow et al. (2016)). This failure
justifies using non-linear transforms such as neural networks (NNs). We reexamine this counter-example
XOR can be learnt by linear models: single neuron without hidden layers feature representations.
Consider learning logical functions AND, OR, and XOR shown in Figure 1 For AND and OR, there is a separating line where half the space contains only points with label 0 and the other half only points with label 1. However, there no scparating line for XOR. The most popular
non-linearly transform the input via kernels (Seki et al. 2008) or non-linear activation functions with hidden layers (Singh & Pandey 2016) (e.g. Figure 1d Conversely, Flake (1993) modified the oulput representation non-linearly to solve XOR.

---

Published in Transactions on Machine Learning Research (06/2024)
AND
OR
XOR
Halfspace Separator
Halfspace Separato:
Equnaity Sepertoo
Equality Separator
Equality Sepecator
Halespace Separator (NA)
(a) AND is possible.
(b) OR possible.
(c) XOR only with equality sep.
Two Genesien deseifien by equdity seperatioon
Equalisy separation for Ensarly non-eeprabble data
Xon with Nous Linnut Resoosentation lay Its
Equility Separttor
Multi-Layer Pesceptton
e-error margin
c-esror margin
(d) XOR with MLP is possible.
(c) Halfspace separable data.
(f) Halfspace non-separable data.
Figure 1: Linear classification by halfspace separators and equality separators for logical functions (Figures la Ic and with hidden layer with hidden units h1, (Figure Id]. In general, equality scparation can classily linearly separable (Figure Tel and non-separable (Figure data.
However, resorting to non-lineann non-linearities is not necessary. We propose using the line not to separate the input space, but to tell if a class is either on the line or not. This linear classifier can perfectly model the three logical functions above, including XOR (Figure D. We name this paradigm as equality separalion. fact, it can solve linearly separable and non-separable problems, demonstrating its flexibility (e.g. Figure Te From the asymmetry of its classification rule, we posit its utility for anomaly detection (AD).
Anomaly detection is paramount for detecting system failure. There may be some limited examples of failures (anomalics), but we usually do not have access to all possible types of anomalies. For instance, cyber defenders want to detect known/seen and unknown/ unknown/unseen /unseen ('zero-day') attacks. The most direct way to train an AD model is via binary classification with empirical risk minimization (FRM), with normal data and anomalies as the two classes. However, classic binary classification principles break when unseen anomalies do not share features with both classes of normal and seen anomalics (Ruff et al., (020)) Rather than conventionally using the hyperplane as separator, we posit that using it to model the manifold of normal data is more suitable: points oloos to the line normal while points far from the line anomalies. To formalize this idea of "suitability", we introduce closing numbers to understand a binary classifier's 's ability to detect anomalies. Based on their suitability, we can integrate them into NNs for AD via standard ERM.
Contributions
We propose the equality separation paradigm. Equality separators use the distance from a datum to a learnt hyperplane as the classification metrie. We summarize our contributions:
As a linear classifier, they have twice the VC dimension of halfspace separators. consequence its ability to linearly classify XOR, which has never been done before. Showing the connection with the smooth bump activation, equality separation can be used in NNs and trained with gradient descent.
We introduce closing numbers, a quantitative metric of the capacity of hypothesis classes and activation functions to form closed decision regions. We corroborate with this theory and empirically observe that equality separation can achieve a balance in detecting both known and unknown anomalies in one-class settings.
2

---

Published in Transactions on Machine Learning Research (06/2024)
2 Equality separator: binary classification revisited
A standard classification problem is as follows. Let X' S IR" be the feature space, : {0, 1} be the label space and H be a hypothesis class. Given the true function c: X' + y (i.e. the concept), we aim to find hypothesis H. that reduces the generalization error L(h) == (P((x)) + c(x)) for x E X.
Lincar predictors (Shalev-Swwarzz & Ben-Davvd] 2014, Chapter 9) are the simplest hypothesis classes we can consider. In R", the class of linear predictors comprises a set of affine functions
An {x w.:. R
and a labeling function Φ: IR + y. The linear predictor class is
6 A. = {x + 6(hw,o(x)) lwwbd E An}.
Halfspace separators as in the perceptron use the sign function as e to split the space R into two regions with a hyperplane: sign(z) = 2 2 O, and sign(=) = 0 otherwise. Equivalently, sign(z) = 1(= 2 0) = 1x-upp (o) for indicator function 1. A critical issue in classification is that halfspace separators cannot achieve zero generalization error when the input space X' is linearly inseparable (Minsky & Papert 1969). To resolve this, we propose to edit the set for the indicator function of Φ.
The core change is as follows: we change the labeling from ((2) = sign(2) 1(z 2 0) to 6(2) = 1(z 0)
or ((2) = I(z close to 0). We introduce two notions of equality separators: the strict equality seperator and the generalized, robust c-error separator. We define the first type as follows:
Definition 2.1. A strict equality separator a docision rule where the hypothosis class is of the form
H = (x (x+ 1(liw.(x)) = 0) low E
1(1ww, + 0): : /t..b E An
(1)
Geometrically, the strict equality separator checks if a datum is on or off some hyperplane i.c. Φ 1 [() or Ø 13y00-- In halfspace separation, labels arbitrary, and capture this with the union of classes in (1)
However, hyperplane has zero volume and sampled points may be noisy. To increase its robustness, can allow some error for a datun within distance from hyperplane. To incorporate this e-error margin, we swap mapping & from 1l (o) to ss for the set S. = {= E TR. - S 2 < for 0:
Definition 2.2. The e-error separator is a decision rule where the hypothesis class is of the form
H. = fx + 1s. (hw.((x))) lwwbb E An) U fx - (hw,b())))))) hw.b E Ax} for c > 0.
The c-error separator modifies the SVM objective. In classification, SVMs produce the optimal separating hyperplane such that training examples are in the e-sized margin (up some slack) (Cortes & Vapnik 1995). Meanwhile, the margin in e-error separators is reserved for one class, while the region outside is for the other. With this striking example, we proceed to analyze its expressivity.
2.1 VC dimension
Vapnik Chnnik-hhervonenkis (VC) dimension Vapnik & Chervonenkis 1971 measures the complexity of a hypothesis class to give bounds on its generalization error. The VC dimension of halfspace separator in R2 is + (Shale--Shwartz & BeviD 2014 Theorem 9.2). We claim that equality separators are not merely an XOR solver, but more expressive than halfspace separators in general:
Theorem 2.3. For hypothesis class of strict equality separators H in Def. 2.1 VCdim(H) = 2n + 1.
We defer the proof to Appendix B and build intuition in R.2. We draw a line through the class with the fewest. points - the union of indicator function classes allows us to. This is always possible with non-colinear points: the maximum number of points that the class with the minimum points has is 2, which perfectly determines line. points is not possible: a labeling with 3 points for each class admits an overdetermined system which is unsolvable in general. Extrapolating to higher dimensions, the VC dimension is more than 2n.
3

---

Published in Transactions on Machine Learning Research (06/2024)
Corollary 2.4. For e-crror separators He in Def. 2.2 2n + 1 VCdim(H.. 2 2n +
The lower-bound relics on Theorem 2.3 while the upper bound is constructed by the union of hypothosis classes. Details are in Appendix B We explore the significance of more expressivity in Section 3.4
2.2 Related works from classification
Flake
1993
designs hyper-hill classifiers that are NNs with bump act activations, such as the Gaussian function
1 14 27 σ
O() exp
2/
(2)
2
where: 2 is the input into the activation function and J4, are paramcters which can be learnt.
Our contributions are two-fold here. First, given knowledge of the class in the margin, we explicitly derive that equality separators are the linear, discrete representation of the bump activation. Discrete classifiers are ubiquitously obtained by learning a continuous (activation) function and then thresholding the output convert the continuous score a classification decision (e.g. modeling halfspace separation with sigmoid activation). This way, we can model equality separation for one class using bump activations. Second, we give the generalization of the linear model which allows label flipping.
Label flipping is perspective from which we can understand the doubled VC dimension. Switching labels does not fundamentally change the optimization objective in halfspace separation due to its symmetry. The asymmetry of equality separators yields a different optimization objective labels are swapped the goal is to obtain the margin of another class instead. This increases expressivity, making label flipping non-vacuous. For instance, without label flipping, it is impossible to classify the function OR with the strict equality separator Flake 1993). However, knowing that OR induces more points from the positive class, we can choose to instead model the negative class.
Another intuition of the doubled VC dimension is that the equality separator is the intersection between 2 halfspacos w'x++ + b c and w + b 2 -c. In contrast, modcling a halfspace separator w x + b with an equality separator requires nou-linearity like ReLU: (max(0, w x + b) + O).
From a hypothesis testing perspective, equality separators yield a more robust separating hyperplane compared to SVMs in binary classification. We link the c-crror separator to the most powerful test, the likelihood ratio test, via Twin SVMs (Jayadeva et al. 2007). In binary classification, Twin SVM learns an é-error separator for each class, w x + bj = 0 and w + b2 = 0. For a test datum x', we can test its membership to a class
based on its distance to class E (1,2), 6 = |w x +b:1 to resemble likelihood ratio test
Tt is natural to use the ratio between the distances l|w
W + +bel |WTx x + bil W 2 20).
22)==1 =1
E1
[W2]
By constraining the Twin SVM to (1) parallel hyperplancs (w1 = w2), (2) different biases (b1 * b2) and (3) equal importance to classify both classes (1 = 1), we obtain a separating hyperplane 2w wof"xb+++00 = 0 between the 2 classes (Appendix C). This separating hyperplane lies halfway between the hyperplanes for each class, where 1ba-bal resembles the margin in SVMs. Without loss of generality, for bi > b2, the region in between the Twilla
2 hyperplanes {x E -62 < wf'x < }bi} acts as the soft margin. The separating hyperplane derives from the learnt e-error separators for each class, which is learnt using the mass of each class [ This is unlike SVMs, which use support vectors to learn the separating hyperplane (Cortes & Vapnik 1995). Support vectors may be outliers far from high density regions for each class, so using them may not be as robust as two e-error separators.
3 Anomaly detection
We focus supervised AD, where different types of anomalies are not fully represented in the overall anomaly (negative) class during training. Distinguishing between the normal and anomaly class is difficult due to (a)
Linear discriminant analysis is also similar by learning the hyperplane via the class masses (see Hastie al. 2004
4

---

Published in Transactions on Machine Learning Research (06/2024)
class imbalance, (b) lack of common features among anomalics and (c) no guarantees to detect unseen anomalies Ruff et al. 2020). We aim to find the manifold that normal data (positive class) live on where anomalies do not. This more general non non-linear formulation of equality separation shows the utility of our paradigm.
3.1 Background
Unsupervised AD has been the main focus of AD literature, where training data is unlabelled and assumed to be mostly normal (i.e. non-anomalous). Since the anomaly class is not well represented in training, most methods train models on auxiliary tasks using normal data. This assumes that the model is unlikely to perform well at the known task during inference if the datum is unseen/ /anomaoos (Chandola
ct al. 2009). Popular tasks include regression via autoencoding and forecasting (Kim et al., 2020) Hoffmann
2007 density estimation (Zong et al. 2018 Breunig et al., 2000) and contrastive learning (Qiu et al. 2021
Shenkar & Wolf )022).
Classification methods have generally not focused on supervised AD either. Classification tasks that optimize directly on the AD objective (which is connected to Neyman-Pcarson classification Davenport et al. 2006)) artificially generate anomalies (Steinwart et al.) 2005 Sipple 2020 Goyal et al., 2020 Davenport et al., 2006 ). However, synthetic anomalies are hard to sample in high dimensions (Ruff et al. 2021) and may be
unrepresentative of real anomalies, so covariate shift of the anomaly class during test time may affect AD per- formance. The lack of common features among anomalies hinders models from generalizing to detecting unseen anomalies. Nevertheless, directly optimizing over the AD objective is closer to the lask and has benefits such as more effective explainable artificial intelligence results (see Sipple (2020)). Equality separation shifts the per- spective of classification from a probabilistic to a geometric lens, aiming to reduce the dependency on sampling.
Limited labeled anomalies is a less explored problem, mostly explored from a semi-supervised approach. Methodologicallly, this is the same as supervised AD; a small portion of the dataset is labeled while the unlabeled portion is assumed to be (mostly) normal. [aag et al. (2019) proposes deviation networks to aid direct binary classification. They stay within halfspace separation and focus on anomalies limited in number rather than lype. Rulf et al. (2020) proposes Deep Semi-Supervised Anomaly Detection (SAD), which performs an auxiliary task of learning data representations such that normal data have representations within a hypersphere while anomalics fall outside (this is idcologically the same as having an RBF activation in the output layer). Their baseline comparison on directly solving AD with ERM on seen anomalies does not yield competitive results because "supervised classification learning principles... ... [are] ill-defined for AD" (Ruff et al. 2020
Out-of-distribution (OOD) detection is the field that motivates our application of equality separation in AD. To reject OOD inputs and reduce approximation error, classifiers can be designed to induce less open decision regions in imput space (G.C. Vasconcelos 1995), also known as open space risk minimization (Scheirer et al. 2013). Decision regions where the decision boundary encloses points belonging to a single class are considered closed. Forming closed decision regions has a strong connection to density level set istimttion under the concentration assumption: for some class with density function p, the density level set for significance level a (and corresponding threshold Ta) can be formulated as small, non-empty set
Co = (x X p(x) > Ta)
arg min ( 2 1 1-a)
measurable
while anomalies of the class belong to X'CC, (Ruff et al. 2021). Especially when feature space X' is unbounded, see connection between classifiers that induce closed decision regions and classifiers that induce smaller, finito-volumed density level sets Ca. Smaller sets are 'more conservative', reducing the open space risk (i.c. region of wrongly classifying an anomaly as normal/false negatives). To model the normal class in (one-class) AD, we can estimate the density level set. From a simple classification framework, we estimate the level sets from data by forming decision boundary that encloses points within the class, similar to SAD. Figure 2a] is an example where the true decision boundary encloses a region in the input space such that all points in those regions correspond to high density regions of particular class.
5

---

Published in Transactions on Machine Learning Research (06/2024)
Table I: Difference between perceptron, equality separator and RBF network (inputs from R").
Decision Rule
Perceptron
Equality separator
RBF network
Bias in Input Space
Halfspace Sigmoid, ReLU Global (n + 1)
Hyperplane
Point /Maet
Activation Function
Bump
RBF
Properties (Closing Number)
Semi-local (n)
Local (1)
One-vs-set machines (Scheirer et al. 2013) and its successors (e.g. Cevikalp 2017] use the idea of 2 parallel hyperplanes for open set recognition, but the optimization procedure is not flexible enough to integrate directly into NNs as we do through bump activations. We extend this idea to the one-class scenario, where the negative data are not other meaningful classes, but (scen) anomalics. We view our contributions as the union of these various ideas applied to AD and the insights to why they work in unison. Based on the concentration assumption in AD, we seek to generalize one-vs-set machines to neural networks, which results in hyper-hill networks. However, it is not straightforward to characterize if we can still obtain closed decision regions after passing data through a NN. We proceed to analyze locality properties of activation functions, which leads to our proposed closing rumbers which helps characterize closed decision regions across the NN.
3.2 Locality: precursor to closed decision regions
Using NNs as classifiers can help to find these complex decision boundaries, but it is not immediately clear if the decision regions can be closed and, consequently, if the density level set can be well estimated. In NNs, classifiers are modeled by their smooth versions (activation functions) and have largely been split. into 3 categories: global, semi-local and local (Flake) 1993 Hartman & Keeler 1991). Local mappings, e.g. radial basis function (RBF) networks and activation functions, "activate" around center point f.c. have outputs beyond threshold. In contrast, global mappings (e.g. perceptron, sigmoid, ReLU) activate on a whole halfspace. Semi-local mappings straddle in between. In particular, Flake (1993) proposes bump activations, hyper-ridge and hyper-hill classification which are locally activated along one dimension and globally activated along others. Since equality separators are the discrete version of bump activations (Section 2.2 we view equality separators as semi-local too. We summarize these observations Table D
The semi-locality of equality separators and bump activations achieve the benefits of global mappings like sigmoids in perceptrons and local mappings like RBF activations in RBF networks (Flake) 1993). Local
activations can converge faster (Hartman & Keeler 1991] and provide reliability in adversarial settings [Goodfellow et al. 2015 G.C. Vasconcelos 1995). while global mappings work well in high dimensions with exponentially fewer neurons (Hartman & Keeler 1991) and are less sensitive to initializations (Flake) 1993),
making them easier to train (Goodfellow et al. 2015). Consider our XOR example with a single neuron: sigmoids in perceptrons are too global, while one RBF unit cannot cover 2 points from either class. Equality separation and bump activations limit the globality of the perceptron but retain a higher globality than RBF activations, which allows them to linearly classify XOR. But what does "globality" " or "locality" precisely mean, and can we quantitatively measure this? Perhaps more importantly, can enclose decision regions?
3.3 Locality and closing numbers for closed decision regions
The locality of classifier not only has implications on classification, but also on AD. Specifically, it is intuitive to see a connection between the locality of classifiers and their ability to form closed decision regions, which in turn is useful for density level set estimation in AD. To quantify the learnability of clased decision regions, we propose one metric to formalize this notion of locality:
Definition 3.1. The closing number of a hypothesis class is the minimum number of hypotheses from the class such that their intersection produces a closed (positive-volumed" decision region.
Significance of closing numbers Closing numbers is not only an important formalization of a big picture idea of locality but also has two other implications. First, the output of a neuron is a function (e.g. scaled
2 Meaningguul analysis requires constraints that decision regions (in raw or latent space) has non-zero volume.

---

Published in Transactions on Machine Learning Research (06/2024)
One-Class Classification Input Spacee
Intersection of Equality Separators
Intersection of Hallpaace Separators
Closs 20 Geussiani
Class A (og. 20 Coussonn Not Class (énomaliees bquality Separotors required
Class A (c.g. 20 Coussonn
Class (tncmalles
Not Class A Hénona
Data (from Class
Ha Espace Separstors required
(a) Normal data is in blue circle.
(b) Equality separators: rectangle.
(c) Halfspace separators: triangle.
Normal data occupies a space with non-zero, finite volume. Classifiers can form closed decision regions to capture this.
Figure 2:
weighted average for sigmoid of neuron activations in the previous layer. Thus, the closing number represents the minimum number of neurons in a layer to induce a closed decision region in the previous layer when using an activation of corresponding locality. This information crucial in hyperparameter selection for the number of neurons we design our neural network to have each layer if we want to enclose decision regions. By extension, also suggests the (minimum) dimension of the space that contains false negatives (undetoctod anomalies), since any projection of unseen (anomalous) data the normal manifold will cause false negalives. Second, the number (or, more specifically, the measure) of configurations of learnable parameters that do indeed enclose the decision region suggests how much guarantee we have to form closed decision regions. In summary, closing numbers tell users the amount of risk for false negatives they take when choosing fewer neurons than the closing number and the analysis of the configuration suggests the residue risk even when users put the corresponding number of neurons. We proceed to state the closing numbers for respective classifiers.
Theorem 3.2. Closing number of RBF, equality and halfspace seporators is 1, n and n + respectively in R Closed decision regions are formed with the intersection of the corresponding number of classifiers everywhere for RBF separalors and almost everywhere for equality separalors, bul normal veclors need to form n-simplez volume for halfspace separators.
The proof is in Appendix D In 2D, these closed decision regions would correspond to circle (e.g. the circle in Figure 2a). parallelogram (Figure 2b) and triangle (Figure 2ch respectively.
The implication of closing numbers and the conditions needed for its realization in the continnous version (i.e. activation functions) across adjacent layers in feod-forward NNs is as follows. Although equality scparation has relatively large closing number compared regions almost everywhere in the space of n learnable n-sized normal vectors (which is matrix in NN). Meanwhile, we have reason to expect closed decision regions from halfspace separation even with + 1 (or more) neurons in the following layer. Closing mumbers provide thcoretical framework to understand the corroborale with semi-locality of bump functions FFlkee) 1993), where it is easier to learn than RBFs but more robust to adversarial settings than halfspace separators (G.C. Vasconcelos 1995). Additionally, Goodfellow et al. (2015) argued for the decrease robustness of sigmoid and ReLU to adversarial attacks due to the linearity in individual activations. Closing numbers complement this view, providing global perspective of activations working together to induce closed decision regions. This gives insight. on the capacity and learnability of closed decision regions.
RBF, we are still guaranteed to have closed decision weight
3.4 Controlling generalization error with inductive bias in AD
For binary classification, we allow label flipping because it is a priori unclear which class should model to be in the margin. This changes for AD. To enclose the decision region, we impose the constraint that the normal (and not the anomaly) class should lie within the margin of equality separators. Using domain knowledge to assign normal data to the positive class, we aim to find the normal data manifold rather than discriminate between normal and (seen) anomalies. better AD objective model lowers the approximation
3Parameters can be normalized while negations represent. the complement of region.
7

---

Published in Transactions on Machine Learning Research (06/2024)
error (i.e. the error between the Bayes classifier and the best classifier in our hypothesis class) compared to halfspace separators. Consequently, prohibiting label flipping reduces to hyper-hill classification, which has VC dimension of n +1 1 like halfspace separation (Flake, 1993), resulting in the same generalization bound as halfspace separators for the estimation error. Compared to halfspacc separators, we obtain the same bound on the estimation error while improving the approximation error in AD. Therefore, we locus on AD applications.
4 Applications
Optimization scheme To obtain a classifier, we perform ERM on data 96) In this paper, we consider mean-squared error (MSE) and logistic loss (LL) as loss functions. Optimizing with BFGS with LL and MSE on the affine class Ag produces a solution for XOR, corroborating with the results in Flakee (1993).
For the bump function in (2) we fix f 0. A smaller a initializes stricter decision boundary but limits gradient flow. We defer further analysis of the role of a to Appendix F.T
Experiments For binary classification, we observe that equality separators can classify some linearly non-separable data (c.g. Figure If.. They are also competitive with SVMs in classifying lincarly separable data (e.g. Figure le where equality separation is more conservative in classifying the positive brown class). Since our focus is to evaluate in supervised AD, we defer results and further discussion of each experiment to Appendix F.2 For supervised AD, we explore 2 settings. The first with synthetic data, where we can understand the differences between classifiers more clearly with visualizations and ablation studies. The second is with real-world data from cyber-security, medical and manufacturing. We evaluate the ability of models to separate normal and anomaly classes with area under the precision-recall curve (AUPR), also known as average precision. For evaluation, we seek to answer two research questions (RQs): RQ1. Is equality separation better than conventional approaches of halfspace separation or RBF separation? RQ2. Can pure ERM work for supervised AD? For RQI, we compare across different locality approaches (equality separation, halfspace separation or RBFs) for a fixed method. For RQ2, we compare standard ERM with other methods on seen and unseen anomalies with other supervised methods for fixed locality approach.
4.1 Supervised anomaly detection: toy model
We explore NNs as embedding functions into a linear feature space for non-linear supervised AD. With a simple binary classification objective, we test how well different NNs can form one-class decision boundaries. Other deep methods have been used for unsupervised (Zong et al., 2018), semi-supervised [Ruff et al., 2020) and supervised (Yamanaka et al., 2019) AD. We only consider directly optimizing over the traditional binary classification objective. To show how local activations are more suitable for AD as suggested by closing numbers, we mitigate their limitations of vanishing gradients by experimenting with low dimensionality.
Setup To demonstrate AD with seen and unseen anomalies, we generate synthetic 2D data by adapting the problem in Sicradzki et al. (2022). The normal (positivd") class is uniformly sampled from a circle of radius 1, and the anomaly (negative) class during training and testing is uniformly sampled from circles of radius and 0.5 respectively, with all centred the origin. This task is not trivial because it is easy to overfit the normal class to the ball rather than just its boundary. To challenge each method, we limit this experiment to 100 data samples. More details can be found in Appendix F.3
Models We make apples-to-apples comparisons with classical binary classifiers: tree-based methods decision trees (DT), random forest (RF), extreme gradient boosted trees (XGB), and halfspace methods logistic regression (I.R) with RBF kernel and SVM with RBF kernel. To illustrate the difficulty of this AD task, we also use standard AD baselines: One-Class SVM (OCSVM) with RBF kernel, Local Outlier Factor (LOF) and Isolation Forest (IsoF) [
We compare these shallow models with equality separators with an RBF kernel. We also compare NNs directly trained for binary classification, We modify NNs with 2 hidden layers
We note that AD is hard because any validation set has information about unseen anomallcs, its utility as proxy of test-time generalization is limited. Thus, generalization hcavily relics on the design of a model.
TThis is according to neuron activation and differs from AD literature. More comments in Appendix F.3 Note that these are unsupervised methods. We use them only to benchmark and our main comparisons.
8

---

Published in Transactions on Machine Learning Research (06/2024)
Table Test AUPR for syuthetic data comparing shallow models and neural networks.
(a) Shallow models.
(b) For NNs, modify the output layer (halfspace separation (HS), equality separation (ES) and RBF separation (RS) in the rows and activation functions in the columns. Random classifier AUPR is 0.75,
Shallow Models
AUPR
DT
0.87++0.03
RF
0.88+t0001
NN Activatio
Leaky ReLU
Bump
RBF
XGB
0.55::0.01
LR
0.54+0.00
HS
0.62:0.11
0.92+t0.13
0.8310.18
SVM
0.54++0.00
ES (ours)
0.83+0.15
0.99+0000
1.00+0.00
OCSVM
0.83+0.000
RS
0.97+10.05
0.91
0.97+00006
IsoF
0.86+
LOF
1.00#+.000
ES (ours)
1.00+0.00
(a) RF
(b) IsoF
(c) LOF
(d) ES (RBF)
(e) HS-b
(I) ES-F
(g) RS-r
(l)) Color map: brighter colors ++ normal class.
Figure 3: Sample heatmap predictions by different models. The middle circle the positive class, while the outer and inner circle is the negative class during training and testing respectively. Figures Bel 3f and 3g are deep models, with hidden layer activations representated by the suffix 'b' for bump and 'r' for RBF. Shallow equality separator (Figure 3d) has one-class decision boundary closest to ground truth, followed by equality separator neural networks with RBF activation (Figure 3i
have (a) halfspace separation, RBF separation or equality scparation the output layer, and (b) Ieaky ReLU, RBF or bump activations in hidden layers. We provide quantitative (Table 2 and qualitative (Figure 3 analyses that suggest either accurate one-class classification overfitting to seen anomalies.
Shallow models From Table 2 we see classical binary classifiers XGB, LR and SVM perform worse than random and are inappropriate for supervised AD. DT, RF and LOF perform better than random but overfit the training data; oven though LOF has a perfect tost AUPR, it has a train AUPR of 0.84, suggesting a bad one-class decision boundary (Figure 3e while DT and RF overfit with an foo norm-like decision boundary (e.g. Figure 3a). IsoF has a high AUPR (Figure 3b). but our ES with RBF kernel has the highest and perfect AUPR with decision boundary close to ground truth (Figure 3d.. Overall, observe that equality separation is better than halfspace separation (RQ1) and ERM works only with equality separation (RQ2).
Neural networks Shallow methods cannot always be used (c.g. large amounts of non-tabular high-diimensional data), so we also experiment on NNs with hidden layers. Though straightforward, our equality separator and bump activation achieves better NN performance. Answering RQ1 and RQ2, we observe that equality separators and RBF separators with hidden RBF activations trained with vanilla ERM consistently obtain perfect AUPR (Table 2), and we validate that closing numbers are useful in quantifying the ability to induce closed decision boundaries. Looking at individual models, about half of the equality

---

Published in Transactions on Machine Learning Research (06/2024)
separators achieve high separation between normal and anomaly classes, with anomalies 8 times further from the hyperplane in the penultimate layer than normal data. In contrast, only 1 out of 20 RBF separatons with RBF activations attain such high separation - most models produced outputs similar to Figure [g The semi-locality of equality separators retain locality for AD while enhancing separation between normal and anomaly classes of NNs with RBF activations.
Geometry of decision boundary More local activation functions that have a reduced region being activated correspond to regions in space that the model has seen before. Through our inductive bias of the geometry formalized through closing numbers, we can use more local activation functions to sidestep the assumptions and difficulty of probabilisticaly reducing the open space risk via ERM by sampling anomalies, such as specifying a distribution to sample in the high-dimensional imput space (Sipple 2020] or latent space TTo et al. 2023). This intuition of the geometry is also present in how the margin width changes across layers. Like how deep learning has been obscrved to extract lower-level fcatures in carlier layers and higher-level features in later layers, we expect margin width to be larger in earlier layers to allow more information through and smaller in later layers to be more discriminatory. In Appendix F.3 we corroborate with this intuition. Through ablations, we also show that the equality scparation's 's good results remain regardless of number of hidden layers and loss function.
4.2 Supervised anomaly detection: real-world data
Cyber-attacks We use the NSL-KDD network intrusion dataset (Tavallace et al.) 2009) for real-world cyber-attack experiments. To simulate unseen attacks (anomalies), training data only has normal data and denial of service attacks (DoS), and test data has DoS and unseen attacks, probe, privilege escalation and remote access attacks. For simplicity, we consider NNs with only type of activation function in terms of locality as vanilla binary classifiers rather than switching between activations between hidden and output layers like in the previous experiments. Baselines we compare against are SVMs and one-vs-set machines (OVS) with RBF kernels for binary classifiers, and OCSVM, IsoF and LOF for unsupervised AD. Since the original OVS returns a binary decision, so we also include an edited version from the perspective of equality separation which we denote as OVS-ES, where we center a bump activation in the center of the two hyperplancs used for the original OVS. The state-of-thc-art (SOTA) deep AD methods we compare against make use of labels: (1) negative sampling (NS), where binary classifiers are trained with generated anomalies (Sipple) 2020) and (2) SAD, where AD is based on hypersphere membership (Ruff et al.) 2020). As note, SAD is effectively an RBF scparator on a pro-trained autoencoder. We report results in Table 3 comparing methods within their categories.
Shallow methods Due to their similarity, we expect some generalization of DoS to probe attacks, while little is expected for privilege escalation or remote access attacks. Our SVM baseline validates this, achieving a much higher increase performance from random for probe altacks than privilege and access alacks.
The RBF kernel seems to help encode bias of similarity through locality we observe a drop in AUPR across all attacks when using a linear kernel (Table 15 in Appendix F.4j. For OVS, we see significant drop in performance on the seen DoS attack and probe attack, while unseen performance attacks are high. Such a trade-off in performance likely suggests that these unseen attacks and some normal data are around the region where the SVM prediction is very high. Our OVS-ES mitigates large trade-offs, maintaining seen anomaly AUPR while slightly increasing the unseen anomaly AUPR. Overall, there is some gencralization from seen to unseen anomalies, but it is not obvious how to improve these shallow methods. Among unsupervised benchmarks, we also see a trade-off between detecting known and unknown anomalies: OCSVM is best al detecting privilege and access atlacks, while IsoF is best at detecting DoS and probe attacks. Notably, unsupervised methods are not competitive with the supervised SVM on seen (DoS) attacks.
privilege and access
NN Across the NN methods, we notice that equality separators/ bump activations are the most competitive at detecting both seen and unseen anomalies, often obtaining the largest AUPR in their category (binary classifiers, NS and SAD). In particular, equality separation with SAD and standard binary classification seem to be among the best, being competitive with the SVM bascline. we observe that equality separation in NNs is better than halfspace separation and RBFs (RQ1) and vanilla ERM with equality separation is
10

---

Published in Transactions on Machine Learning Research (06/2024)
Table 3: Test AUPR (mean + standard deviation) on NSL-KDD dataset, grouped according to different methods. The different methods are random classifiers (calculatod in expectation), shallow supervised (Sup.) and unsupervised (Unsup.) methods, binary classifier NNs, Negative Sampling (-NS) and Deep Semi-Supervised Anomaly Detection (-SAD). We compare using different kinds of activations in NNs: global (HS), semi-local (ES) and local (RS). AUPR is reported vis-a-vis each attack and all attacks (overall). DoS attacks are seen during training, while the rest are not.
Model Attack
DoS
Probe
Privilege
Access
Overall
Random
0.435
0,200
0.007
0.220
0.569
SVM
0.009-10.000
0.78710.000
0.037-100000
0.524+0.000
0.94810.000
Sup.
OVS
0.717=0.000
0.600=0.000
0.503=00000
0.610=0.000
0.785#0.000
OVS-ES (ours)
0.956=000000
0.785=
0.038=100000
0.535
0.949=0.000
Unsup.
OCSVM
0.779110000
0.827=00000
0.405=10000
0.760=0.000
0.897=0.000
IsoF
0.765=20.073
0.850=0.066
0.089=#00044
0.392=0.029
0.865#0.031
LOF
0.405
0.567+0.000
0.039-+00000
0.455
0.718+0.000
HS
0.944+000016
0.739
0.016+000066
0.180+0.033
0.877+0.010
ERM
ES (ours)
0.97410.001
0.717
0.045:1.014
0.510
0.94110.011
RS
0.356++0.002
0.213++0002
0.010++0.000
0.228
0.406++0.001
HS-NS
0.936+0.000
0.642+0.030
0.006
0.139+0.001
0.845+000006
NS
ES-NS (ours)
0.945:10.009
0.659#0.0033
0.023:10.011
0.206
0.881+0.007
RS-NS
0.350#0.002
0.207#0.002
0.009
0.223+0.002
0.401
HS-SAD
0.955#00003
0.766=0.011
0.100:10.000
0.447
0.935
SAD
ES-SAD (ours)
0.06010.002
0.795=10004
0.047+0.002
0.50910.009
0.95210.002
RS-SAD
0.905
0.678=00000
0.022+00000
0.142
0.846=0.000
competitive with SAD without any pre-training (or sampling, as in NS), as well as a good alternative to other supervised methods which achieve good AUPR on the seen DoS attacks (RQ2).
Medical dataset We use the thyroid dataset (Quinlan 1987p for medical anomalies. Using hyperfunction as the seen anomaly and subnormal as unseen, we compare ERM methods in Table Aa] Even though the thyroid dataset has the fewest dimensions of all datasets (21D), we RBF separation already performing very poorly despile its stellar performance in our 2D synthelic dataset. Nevertheless, equality separation still detects seen and unseen anomalies better than halfspace separation (RQ1).
Low data, high-dimensional image dataset. MVTTc dataset (Bergmann et al. 2019 has images of normal and defective objects. We select one anomaly class to be seen during training for supervised AD, while the other anomaly classes are only soen during test time. There are on the order of 300 training samples, while feature embeddings (from DINOv2 (Oquab et al. 2023 are 1024-dimensional.
We compare AUPR of binary classifiers across 6 objects in Table 4b zooming in on individual defect detection performance for pills (which has the most number of defect groups) in Table 4c Since DINOv2 acts as a kernel to embed features into high dimensional space, consider halfspace separation our baseline, while RBF scparators act like SAD in this case for comparison. During training, we observe that RBF scparators consistently have high training error and low train AUPR (below 0.55), suggesting underfitting. Here, too much locality impedes learning, especially in high dimensions. Meanwhile, we see equality separators detecting unseen anomalies the best for 5/6 of the objects. Furthermore, equality separators are consistently the best across different anomaly types (Table 4ch. Honce, cquality separators can be integrated with foundation models to ensure higher fidelity to the AD objective (RQ1, RQ2).
In general, we see that more local activations perform better in the low-dimension synthetic dataset, while more global activations perform better in the higher dimensional datasets. Along with
Limitations/Exttesiion
11

---

Published in Transactions on Machine Learning Research (06/2024)
Table 4: Test AUPR (mean + standard deviation) on thyroid and MVTec dataset across various anomaly types.
(a) Thyroid AUPR for hyperfunction (b) MVTec overall AUPR for 6 objects. SAD MVTec AUPR for pills object by defect type.
(seen) and subnormal (unseen) anomalies.
is implemented with RBF.
Object
HS
RS (SAD)
ES [ours]
Defort
I1S
ES (ouns)
Diagnosi
RS
(ours)
Capaule
0.518+0.002
0.894-0.000
0.947-00.003
Combine
0.913#0.010
0.943=0.016
Hyper.
0.007-0.013
0.090-0.002
0.93660.000
Haseluat
0.932:00.006
0.783=0.000
0.932=0.034
Contam.
0.768+0.029
0.802-0.114
Subnorm.
0.459L0.000
0.082L0.002
0.596L0.007
Leather
1.00010.000
0.818_0.000 0.00
0.996:20.000
Crack
0.652#0.009
0.730=0.044
Overall
0.61520.052
0,10510:001
0.72610.004
0.91740.005
0.908=0.000
0.833
Impriat
0.470:0.035
0.573=0.091
Wood
0.584+00004
0.806-0.000
0.986-0.0007
Type
0.885.10.071
0.910
Zipper
0.908+0.000
0.870=0.000
1.000=0.0000
Scratch
0.655:0.021
0.690=0.058
observations on training performance, we notice a trade-off between the ability to learn from the empirical risk (i.e. converging to a low loss) and the robustness to reject unseen samples, which we quantify with closing numbers and observe through our empirical results. This observation complements the understanding from Goodfellow et al. (2015) on the robustness of more local activations from the lens of enclosing decision regions across consecutive layers in neural networks (i.e. closing numbers). Comparing the results of NNs to the SVM shallow baseline for the NSL-KDD dataset, we suspect that more work needs to be explored on how to learn well while being able to reject anomalies in NNs.
In addition, we note that equality separation generally has good results regardless of dimensionality. However, we corroborate with Flake (1993) that bump activations in equality separation can be sensitive to initializations, evidenced by some variance in test AUPR. To mitigate this, we can set a to be large to mitigate this. Ensembling equality separators can also help suppress inaccurate outputs from bad initializations. We discuss more in Appendix F.3 We also note that NNs can struggle at generalizing to unseen data, as OCSVM achieves better AUPR than NNs for the unseen attacks in NSL-KDD. We posit that understanding how the activation function, loss function and optimization scheme work together in equality separation to form closed decision regions is the next step towards designing better NNs for AD. From our experiments in varying dimensions, we suspect that the key to balancing learning and enclosing decision regions is mitigating vanishing gradients for the regions classified as anomalous. For unseen anomalies, although equality separation is not competitive with OCSVM, equality separation still outperforms other supervised methods as a whole, highlighting the suitability of equality scparation for AD.
5 Conclusion
We revisit the belief that XOR is not linearly separable, which prompts us to propose a new linear classification paradigm, equality separation. Equality scparators use the distance of a datum from a hyperplanc as their classification metric. They have twice the VC dimension of halfspace separators (e.g. it can linearly solve some linearly non-separable problems like XOR) and enjoy the benefits of globality and locality, which we formalize and quantify with closing numbers. Closing numbers guide hyperparameter choice of the number of neurons in each hidden layer of a neural network and suggest how often we can expect closed decision regions under different activation functions. Using the bump activation in hyper-hill classifiers to model equality separation, we interpret hyper-hill classifiers as equality separators across each neural network layer which encloses decision regions almost everywhere. This property is especially useful in supervised anomaly detection when limited supervision is present for anomalies and for representation learning of the normal class when we require the normal class to be within the margin. We support these claims empirically, quantitatively and qualitatively showing that equality separation can produce better one-class decision boundaries. Equality separat ion can be used with kernels, neural networks and foundation models, striking a good balance detecting seen and unseen anomalies in both low and high dimensions.
Broader Impact Statement
Not applicable.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Acknowledgments
The authors thank the anonymous reviewers for their constructive feedback during the review process. The authors also thank Tri Kien Tran and Tanmay Kenjale for sharing code for MVTee dataset preprocessing. This work was supported in part by the Department of Energy (DoE) project GR00001113.
13

---

Published in Transactions on Machine Learning Research (06/2024)
References
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mytec ad - comprehensive real-world dataset for unsupervised anomaly detection. In 2019 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9584-9988 25929, 2019. doi:
Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. LOF: Local Outliers, pp. 93 104. Association for Computing Machinery, New York, NY, USA, 2000. ISBN 1581132174. URL
Identifying Densily- Rensity-Based
11w.00420113449.... 335388
Hakan Cevikalp. Best fitting hyperplanes for classification. IEEE Transactions on Pallern Analysis ond Machine Intelligence, 39(6):106--1088) 1088, 2017. doi: 10.1109 20A000TTAA...211.555787700
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM Comput. Sure., 41),,, jul 2009. ISSN 0360-0300. doi: 10. 00.114615511880.54411888 URL https://doi.org/// or.110..11//1
0141880.1541888
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20(3):273-29) Sep 1995.
ISSN 1573-0565. doi: 10.1007 /F009940044 URL
Mark A. Davenport, Richard G. Baraniuk, and Clayton D. Scott. Learning minimum volume sets with support vector machincs. In 2006 16th JEEE Signal Processing Society Workshop Machine Learning for Signal Processing, pp. 301-306, 2006. doi: D0.1109/MLSP.2006..
Gary William Flake. Nonmonotonic Activation Functions in Multilayer Perceptrons. PhD thesis, The University of Maryland, 1993.
D.L. Bisset G.C. Vasconcelos, M.C. Fairhurst. Investigating feedforward neural networks with respect to the rejection of spurious patterns. Pottern Recognition Letters, 16:207-212, 1995.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. Adaptive computation and ma- chine learning. MIT Press, 2016. ISBN 9780262035613. URL https://books google. in/books7id= MP9SDQAAQBAJ
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. Yoshma Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, Son Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URI. http:
7/axxiv.or org. wwt.a////.....5 1412.
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. DROCC: Deep robust one-class classification. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3711 3721. PMLR, 13 18 Jul 2020. URL http:://////o mlr...mm.....mm http:/ccedaigg.......rres///////
Eric Hartman and James D. Kocler. Predicting the future: Advantages semilocal units. Neural Computation, 3(4):566 578, 1991. doi: 10.1162/necn htt./ecc...99991 1991.3.4.55
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning, data mining, inference, and prodiction. Springer Series in Statistics, 23(3):528 529, January 2004. doi: 10.1002/sim.166
URL httpp:://dol 1rg/ 1616
Heiko Hoffmann. Kernel pea for novelty detection. Pattern Recognition, 40(3) 2007. ISSN 0031-3203. doi: BTt/cle/ps//903113/666631111
2006.000000000000000 URL http://wwww.cccieeeeaeeeecccoces
Jayadeva, R. Khemchandani, and Suresh Chandra. Twin support vector machines for pattern classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(5):905-910, 2007. doi: 10.1109/
1006
14

---

Published in Transactions on Machine Learning Research (06/2024)
Young- geun Kim, Yongchan Kwon, Hyunwoong Chang, and Myunghee Cho Paik. Lipschitz continuous autoencoders in application to anomaly detection. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 2507-2517. PMLR, 26-28 Aug 2020. URL https: mamm20cchmm...
//proceedings...
wwwpiiiiin.mmm.....mm...mmm www.mml.rrre... pelss.rressv108,
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 Fighth IEFE International Conference on Data Mining, pp. 413 422, 2008. doi: 1..109//CDM.2008.177..
Marvin Minsky and Seymour A. Papert. Perceptrons: An Introduction to Computational Geometry. The MIT Press, 1969. doi: httsllmmmmppsss11///0 001.00011
URL
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Guansong Pang. Chunhua Shen, and Anton van den Hengel. Deep anomaly detection with deviation networks. Tn Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Móning, KDD '19, pp. 353 362, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362016. doi: 10.1145/3292500... 022955003330871. URL http:::/a...org/11..115553.. 1r1/1000.111//995500 3330871
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, 1999. URL htt.htss///iissssrr...iee psu.
wtt...s...........dddd.d........ edu/doc
Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. Neural transformation learning for deep anomaly detection beyond images. In International Conference on Machine Learning, pp. 8703 8714. PMLR, 2021.
Ross Quinlan. Thyroid Discase. 10tp:/000224//50D00
UCI
Machine
Learning Repository, 1987.
DOL:
Lukas Rulf, Robert A. Vandermeulen, Nico Görnilz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. In International Conference on Learning Representations, 2020. URL
Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756-795, 2021.
doi: 10.1109/JPROC:20221 .3052449.
Walter Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Transactions on Pallern Analysis and Machine Intelligence, 35(7):1757 1772, 2013. doi:
Hirosato Seki, Fuhito Mizuguchi, Satoshi Watanabe, Hiroaki Ishii, and Masaharu Mizumoto. Sirms connected fuzzy inference method using kernel method. In 2008 IEEE International Conference on Systems, Mon and Cybernetics, pp. 1776 1781, 2008. doi: 10.1 10.1109/ICSMCC 200.S....000.811110
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorillms. Cambridge University Press, Cambridge, England, July 2014.
Tom Shenkar and Lior Wolf. Anomaly detection for tabular data with internal contrastive learning. In International Conference on Learning Representations, 2022. URL httts::/ .ot.
hsz2bt46bT
15

---

Published in Transactions on Machine Learning Research (06/2024)
Yuval Sieradzki, Nitzan Hodos, Gal Yehuda, and Assaf Schuster. Coin flipping neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, 1CML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 Proceedings of Machine Learning Research, pp. 20195 20214. PMLR, 2022. URL attpps//prroceeding wwwigggsmmr.... mwr.mlrrresee pttpee//vi1/sierraaaarrtaata
Vaibhav Kant Singh and Shweta Pandey. Minimum configuration mlp for solving xor problem. In 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), pp. 174-179, 2016.
John Sipple. Interpretable, multidimensional, multimodal anomaly detection with negative sampling for detection of device failure. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp-
6016-9025, PMLR, 2020. URL
Ingo Steinwart, Don Hush, and Clint Scovel. A classification framework for anomaly detection. Journal of Machine Learning Research, 6(8):211 232, 2005. URL http:p///mna...or/aapee///////ts html
Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametrrc outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL
JHKlpEZqduQ
Mahbod Tavallace, Ebrahim Bagheri, Wei Lu, and Ali A. Ghorbani. A detailed analysis the kdd cup 99 data set. In 2009 TEFE Symposium on Compulational Intelligence for Security and Defense Applications, SCA.//CISAA...0.333666
pp. 6, 2009. doi: V. N. Vapnik and A. Ya. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilitics. Theory of Probability & Its Applications, (6(2):264-280) 1971. doi: 10.1137/1116025. URL
kttpps//dois org/ 10. 1137/1116025
Yuki Yamanaka, Tomoharu Iwata, Hiroshi Takahashi, Masanori Yamada, and Sckitoshi Kanai. Autocncoding binary classifiers for supervised anomaly detection. In Abhaya C. Nayak and Alok Sharma (eds.), PRICAI 2010: Trends in Artificial Intelligence, pp. 647-659, Cham, 2019. Springer International Publishing. ISBN
978-3-030-299111
Bo Zong, Qi Song, Martin Rengiang Min, Wei Cheng, Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018. URL https:///peeeeee.....tttteeeee http:///ww.....ffoomm/rreaadaab.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix contents
This appendix is ordered as follows.
Appendix
A
mentions connections of the equality separator with other concepts in the literature.
2. Appendix B proves the VC dimension of equality separators, including a proof in 2D for intuition.
Appendix shows the connection between SVMs and equality separation through formulation of likelihood ratio test.
Appendix D proves the closing numbers of equality separators and halfspace separators, showing why oquality separators can induce closed dccision regions better than halfspace separators.
5. Appendix E discusses the different choices of bump activation functions and loss functions, which bias the hypothesis to model different things. In summary, the choosing different bump activation functions may model hyperplane compared to a margin, while mean-squared error loss may be more suitable than logistic loss when there label noise, such as through anomaly sampling.
6. Appendix F lays out all experimental details for reproducibilty, ablations and visualizations. For instance, we show that the margin width increases earlier layors while decrcasing in later layers, which corresponds to the intuition deeper layers learning more specialized information and being more selective activate on fewer inputs (Appendix F.3) Code used can be accessed
17

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix A
More connections of equality separators
We explore more connections of equality separators with other concepts in machine learning.
A.1 Connection to hyperrectangle classifiers
Since the halfspace separator is tightly connected to our decision rule, our decision rule is also connected to other classifiers that are related to halfspace separators. To illustrate this, we use one example of the hypothesis class being the set of all hyperrectangles in R". Just as how using hyperrectangles as classifiers in R" can be seen as using the intersection of 2n halfspaces, we can also see its relation to our c-crror separator.
For each dimension E [m], define a weight vector W, E RR, a bias term b, E R and a width parameter E, E Rn that will control the width of the hyperrectangle in the 7th dimension. Hyperrectangle classifiers can be produced by the interscction of these c-crror separators. In fact, can also generalize this to hyperparalleeogranss too, by removing the constraint of the orthogonality of the weight vector.
A.2 Connection with linear and support vector regression
Let us view equality separation from regression perspective and observe the similarities and differences between equality separation and regression. Due to the equality form of w + b = O, the strict equality separator is similar to linear regression model for one class. Mcanwhile, our c-crror scparator can be seen as a support vector regression problem for one class, with e defining the error/noise tolerated. In support vector regression (SVR) (llatt 1999), points within the margin are not penalized for being a small distance away from the hyperplane.
However, our approach differs from regression in two ways. First, equality separation handles anomalies differently in supervised anomaly detection (AD). Our optimization scheme accounts for information provided by the other class, which can act a form of regularization. In vanilla linear regression problem, the negative class may not matter so much during training, and we can discard the negative class during training and perform linear regression with only the positive class. However, in a learning setting with a non-linear embedding function (like a neural network), the negative class can help with updating the embedding function such that the negative class will not lic on the hyperplane (and can help prevent degenerate solutions, such as projecting all points onto a single point). SVR can account for anomalies during training, but anomalies are identified in an unsupervised fashion and cannot utilize labeled information. For classification task, the distance from the hyperplane determines how similar data point is to the class defined by the hyperplane, and we can also leverage label information to learn this. This inductive bias is especially useful for one-class classification.
Second, equality separation is agnostic to the exact position on the regression manifold. The aim of regression is to transform datum into its corresponding response variable. In equality separation, we wish to classify the datum, so we are only concerned if the transformed datum lies somewhere on the regression manifold (of some class), indicating that it belongs to the corresponding class.
For sake of concreteness, consider the homogenous linear case. The output of the equality separator would tell us if the input was a valid input by checking membership to the margin. On the other hand, the SVR model would tell us what the response of the input is given that the input is valid (b can be broken down into the sum of a learnt translation and the response variable for the corresponding input during training). More generally, given that there is some (unknown) regression task with a non-lincar relationship, neural networks with equality separators can be viewed as first embedding the input manifold onto hyperplane (with margin) and checking the validity of the input by verifying if the embedded input lies the hyperplane. If a regression task were specified, then we would instead want the output of the hyperplanc. For equality separators implemented in neural networks, there are many possible embeddings that will regress inputs to different parts of the hyperplane, but as long as the input is embedded onto the hyperplane, our task of classification is done.
Such a perspective also suggests that equality separation similar but idcologically different from classification: traditional classification seeks to make a statement about datum being more likely than not, while equality
18

---

Published in Transactions on Machine Learning Research (06/2024)
separation is more concerned about manifold membership This is probabilistic in nature, and its geometric implication is that data are tipped over decision boundaries. Ideologically, there are no 'optimal' data, only more likely ones. In this way, equality separation is closer to regression in the sense that there is an optimal point to reach zero loss, which is exactly on the hyperplanc.
A.3 More details on hyper-ridge, hyper-hill classification and une-vs-set machine
We provide more details on the models most related to equality separation in the broader literature, which are hyper-ridge classification, hyper-hill elassification and one-vs-set machines. In summary, hyper-ridge classifiers, hyper-hill classifiers and one-vs-set machines all do not allow label flipping and specify that the positive class should be within the margin, which is the constrained version of equality separation AD applications.
The hyper-ridge network
F Flake 1993
uses the ridge activation function, with the classification rule being
sign (w" b)3).
is a smooth approximation of the hyper-ridge network, inducing "nommonotonic, 'bump-like', activation function" such as the Gaussian
The hyper-hill network Flake 1993
exp (- (w [w""x+0)
(-(w"xx b)²
x2
in 2 These two networks are proposed for regular binary classification.
On the other hand, one-vs-set machine is proposed for multi-class classification while reducing open space risk for open set recognition. It was first proposed in [Scheirer et al. (2013), but has also extended in [Ceiiklpp (2017). Tt trains a linear SVM to first find a linear decision boundary between classes, then produces a second parallel hyperplane to classify the positive class within the margin.
These classification approaches are fundamentally the same, finding the margin where the positive class lies in. They merely differ in how this margin is found: one-vs-set machine does this linearly while hyper-ridge and hyper-hill classifiers do this with many non-linear layers.
However, these 3 approaches do not explicitly state the linear condition for classification while we do, which is the generalization of the & labeling function to the indicator function. This linear observation leads to a more generalized version, which emphasizes the asymmetry between the two classes where one class falls in the margin while the other falls outside, so we instead refrain from specifying which class falls the margin without prior knowledge. This classifier (equality separator) is linear and admits twice the VC dimension of halfspace separation, while these 3 other classifiers admit the same VC dimension of halfspace separation Flake 1993 As a concrete example, even though these 3 other classifiers can classify XOR, they camnot classify both OR and AND. However, equality separation allows label flipping from the observation that labels in regular binary classification are arbitrary. This insight is intuitive but (1) immediately makes our classifier have twice the expressivity and (2) suggests that the previous models are helpful when domain knowledge informs us of the class belonging to the margin.
In particular, equality separation reduces these 3 classification approaches in AD applications, because assign the normal class to be the positive class (i.e. the class within the margin). In AD, assert prior knowledge of the specific labels of the classes, where the labels not arbitrary and have specific meaning (normal versus anomalies i.c. the complement of normal data).
We are claiming that classification tasks are about manifold membership. Rather, we are suggesting that more natural view manifold membership from equality separation perspective.
19

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix B VC dimension proofs and comments
To layer the intuition, we will first prove the VC dimension in R² of the strict equality separator for binary variables, then for the whole space of R² and finally, the general version of R". The former two will provide good intuitions for the generalization to RR.. We note that we will assume that n 2 2 in this section since the case for n = 1 is trivial. We denote H. as the hypothesis class of strict equality separators, as defined in function class F.
Definition 2.1 Let VCdim(FF denote the VC-dimension
B.1 Binary (boolean) variables in I=
We first through a concrete example to illustrate how we can find hypotheses to classify all points using label flipping.
In the special case where x is binary (ie. has entries which are binary variables) in R2, we can observe that (C)im(H) 4, which is the maximum number of points in the binary feature space in IR?.
To how always shatter the 4 points of (0, 0), (0, 1), (1,0) and (1,
use the following rule:
If there a class that docs not have any point that belongs to it, then a line passing through none of the points sulfices as a classifier.
2. If there is class that has only point that belongs to it, then line passing through that point that does not pass through any other point suffices classifier. Examples given in the AND and OR cases.
3. there is a class that has only 2 points that belong to it, then a line passing through those 2 points as a classifier. Example is given the XOR cases.
Other cases are equivalent to label flipping, which can be captured by the hypothesis class H. Hence, the set. of 4 points can be shattered by H.
B.2 VC dimension in 2
In fact, analyse a more general sctting where the elements of x live in R² rather than being contrained to binary variables. In this more general setting, obtain the following theorem. +
Theorem B.1. Let H (1("2xx + = 0)) ( and R2. Then, VCdim(H) 5.
The VC-dimension of H IR.2 is an increase from the VC-dimension of the half-space separator. The increase in due to the assignment rule as proposed in Appendix B.1
To prove the VC-dimension, will construct 2-sided inequality with lemmas
B.2 and B.3.
Lemma B.2. Let H. (1 (1(wTx b 0)) U (1(wTxx b b 0)} and x e IR, Then, VCdim(H) 2 5.
Proof. We find a set of 5 distinct points that are shattered by H. Let these 5 points be arranged on a unit circle centred at the origin, equally spaced (in the complex plane, the points will be the 5th roots of unity). Following the assignment rule in Appendix B.I we can always draw line that classifies the points. Therefore, these 5 points are shattered by H.
x + b- Lemma B.3. Let H. = (1(wTxx x
x 0)} and x E IR? Then, VCdim S
Proof. We will prove that any 6 distinct points cannot be shattered by H Consider 2 exhaustive cases: (i) when there does not exist 3 colinear points and (ii) when there exists 3 points that colinear (ie. lie on the same line).
8The where points not distinct trivial show the inability shatter, we will explicitly mention this fact in future analyses.
20

---

Published in Transactions on Machine Learning Research (06/2024)
In case (i), any labelling that has 3 points from class 0 and the remaining 3 points from class 1 will not produce a perfect classifier. Since 3 points belong to each class, then if there exists a perfect classifier from H, then the line must pass through all 3 points from either class. Since no 3 points are colinear, then there is no line that passes through points from either class. Therefore, any 6 points that do not have 3 colincar points are not shattered by H.
In case (ii), it not possible shatter the 6 points. Denote the concept (ie. the true function) as c: X' + V. Consider 3 colinear points a, a2, ag R Denote the other points as d,, d2, e E R Let c(ar) c(d1) c(d2) = 0 and c(ag) c(as) = c(e) = 1. The linear classifier from H. has to be a line that lies on all the classes of 1 class. However, there is no line that lies on all the points from class 1, because that line will either
contain a point from class 0 (e.g. the line passing through ag and ag also passes through ar), or
2. not contain all the points from class (because line that does not pass through az and as will not have a2 or az).
Hence, the line needs to pass through all 3 points from class 0. If such a line does not exist, then the proof is done. If such a line exists, then there exists some points d1, d2 E R that are colinear to a but not a2, ag or e. Then, define new labelling function é' that switches the labels of d2 and e. In other words, d(ai) = d'(d1) d' c'(e) 0 and d((a)) c' (as) = ('{2) 1. Then, we face a similar difficulty in classifying the colincar points a, d and c, because no line contains all class 0 points. Since there exists a labelling where no hypothesis from H. can classify these 6 points, then any 6 points that have 3 colinear points are not shattered by H.
Thus, there does not exist any set of 6 points that can be shattered by H.
B.3 Proof of Theorem 2.3 VC dimension in R"
Before proving Theorem 2.3 we first restale it:
Theorem B.4. Let E R" for E (2,,,--), and the hypothesis class be the strich equality seporator H [1(w"x*+3 [1 + + Then, VCdim(H) = 22 + 1.
The classical halfspace separator gives VC dimension of 1. An intuitive way to interpret the factor of 2 in the n-term is that we allow 2 kinds of hypotheses one that asserts equality and one that does not. In essence, our hypothesis class allows for label flipping, which gives it the flexibility of expression. Hence, the equality separator hypothesis class is on the order of twice as expressive as the halfspace separator. However, allowing label flipping for the halfspace separator does not increase its expressivity. The asymmetry of the equality separator gives rise to this property that label flipping is indeed useful, doubling its expressivity.
We break Theorem 2.3 into a 2-sided inequality with lemmas B.5 and B.6
Lemma B.5. Let H = [1( x + b bb= 0)) U (1(w"xx x + b 0)} and x E R=. Then,
VCdim (H) 2 2n + 1.
Proof. Consider set of 2n + distinct points such that for any hyperplane R", only a maximum of n points lie on the hyperplane. Then, for any labelling of the 2n 1 points, let the positive class be the class that has the fewest data points that belong to it. (We can do this because our hypothesis class allows for label flipping.) Let the number of data points belonging to this positive class be k. Notice that A < 1 2n+1 n. Denote the non-empty set of all hyperplanes that contain all of these at most & points as H.
If T n, then | = 1 (ie. the hyperplane is unique). Note that this unique hyperplane does not contain any other points, by construction of the data points, so the hyperplane is a perfect classifier on this labelling.
Otherwise, le n, then [H| oo (ie. there are infinitely many hyperplanes). With induction, we prove that there is a hyperplane that passes through h points that does not pass through the other (2n + 1) k points.
21

---

Published in Transactions on Machine Learning Research (06/2024)
Base case: k 1. For any set of k points, let H be the set of all hyperplanes that pass through these k points, and let H' C H be the set of all hyperplanes that pass through the & points but also pass through at. least 1 of the other (2n + 1) - k points. Then, ||| = oo while [ = (2n + - k = n n 2 because the other n n+2 2 points (ic. the points from the negative class) restrict us from using those hyperplancs for classification. Since H| > [H'] and H' S H, then H' is a proper subset of H, and there exists a hyperplane in H that is not in H' By definition, this hyperplane contains all points and none of the other (2n + 1) - k points.
Inductive step: Assume that there exists hyperplane that passes through any k + points and does not pass through the other (2n + +1 (k + 1) points, for 0 0 n < Then, we prove that there exists hyperplane that passes through any k points. Consider any set of k + points (and denote this as P) and consider a hyperplane h that passes through the k + 1 points but not the other 2n - k points. Let the set of the other 2n - k points be Q (ie. P and Q form a partition of the set of all 2n +1 1 points). Let p E P be any point in P (since P is non-empty). Then, there an added free variable (ie. added degree of freedom)
define a hyperplane that passes through P)(p). To avoid passing through points in QU {p), this free variable is unable to take on a finite number of values to avoid the (2n 1) k points. Since the variable can take on infinite number of values except a finite number of values, there is a value that this variable can lake on. Using this value, we can find hyperplane that passes through all k points in P while not passing through the (2n + 1) - k points in Q J {p).
By induction, we have shown that for all E f0, 1,..,, n - 1}, that there exists a hyperplane that passes through & points but not the other (2n + 1) - k points. Hence, there always exists a hyperplane to classify the k points from the positive class such that none of the (2n + 1) - points from the negative class are on this hyperplane.
Since a perfect classifier from H always exists for the aforementioned dataset for any labelling, then there exists set of 2n + 1 points in R that H can shatter. Thus, VCdim(H) 2 2m + 1.
Lemma B.6. Let H (1(w"x x + b= 0)) U (1(w" x + b+ and x E RK. Then,
VCdim (H) S 11
The proof only requires basic lincar algebra and the idea is as follows. We desire to find a labeling on 2n + 2 points such that there is no strict equality separator for that labeling. For some labeling, we consider (translated) subspaces of the 2 classes defined by the points from that class and relabel points until we find a labeling that satisfies the above condition. In the arguments, we often consider 2 classes with n points each, and leave the remaining 2 points to help us produce the desired labeling by labeling the points with the corresponding labels and sometimes swapping the labeling of points already with labels. We include the high level overview of the proof in Figure
Proof- Consider any set of distinct 2n + 2 points in RR, and denote this set as X. We aim to show that there exists a labelling on these points such that no h E H can classify the points perfectly (i.e. H cannot shatter any set of 2n + 2 points). We will exhaustively consider cases and show that in all cases, there exists a labelling such that no classifier in H can perfectly classify all the points on such a labelling.
Note that if X lies on a hyperplane, then there exists a labelling half of X as positive and the other half as negative that would fail to produce a perfect classifier. Hence, we consider the fact that X does not lie on one hyperplane.
Let (Pi, Q} be partition of X, where PP
IP21 and |Q| 2. Let M A + Bax|A| be the function that outputs a matrix where each column is an element of A. Formally, for non-empty, finite set. A == (al,.., a } S X, define M as function that generates the matrix ( = [a.,..., a|A]] from A.
Case If there exists a partition of X such that M(P)) and M(2) are not full rank, then consider points b P1 and E P2 such that rank(M(PP)))) ((b)) = rank(M(P)) and (M = rank()((P)).. Such points exist based on a counting argument, given that ((P)) and (P2) are not full rank. Then, consider the labelling such that Pt M(b), c belong to class one and P2 [c), b, Q belong to class two, A perfect classifier
22

---

Published in Transactions on Machine Learning Research (06/2024)
P.. P2. partition size n, respective there partition where M(P)). are full rank?
Case Both are rank deficient. Consider "non- contributing" P,.c Pa.
Case 2: Only (M(P2)
Case 3: Both
is rank deficient. Consider "non- contributing" p Ps. Does a p such that p is not on the hyperplane of P
full rank.
Let Q = (p.4). Do p. lie on P or P, hyperplanes?
Label Class 1: c
Neither.
One.
Both.
Class 2: b.
Label Class 1: Pt
the Pa
Do p,, q lie an same hyperplane?
Yes
No
hyperplane, p
on neither. Is P
Class 2: P
hyperplane on Pj
No
Yes
Label
No
Is "non- contributing" unique?
hyperplane?
Class 2: P. Ap.
Class P
Consider s E P2 that is not on P, hyperplane.
No
Let p be on the hyperplane of P,, q the hyperplane Pz.
9 the hyperplane of Pt- Are they also on the hyperplane of P,?
Yes
Yes
Find x E Pi such that
Label
Label
Px Up generates the same
Class 1: P, Class 2: 122.x.9.
Class 1: P
Class 2: P,.
Yes
No
hyperplane Pj.
Find such that P, \x q generates the same hyperplane P,
Note: "Non-cantributing point" refers to point in the set such that remaving will decrease the dimension of the set.
Label Class 1: Y Class P, V,,x
Label Class 1:
Class 2: P.
Figure 4: Proof overview flow chart of Lemma B.6
23

---

Published in Transactions on Machine Learning Research (06/2024)
from H does not exist on this labelling: using hyperplane generated by P to label class one the positive class will classify b wrongly because b É negative although it is from the positive class. Likewise, using a hyperplane generated by P2Mc) to label class two as the positive class will classify с wrongly because с E (Pa(M)))))))).
(by assumption) will be classified as
Note that using hyperplanes other than the ones described will also fail to produce perfect classifiers, because points in Pt (when class one is the positive class) or P22 (when class two is the positive class) will not be classified correctly. We omit this comment in further analyses.
Otherwise, consider the case where there exists a partition of X such that either (P1) or ((2) is full rank. Without loss of generality, let ((2)) be rank deficient, while ((P) is full rank. By the same rank(M (Ps)..
Case 2
counting argument, there exists non-empty set K E P22 such that (P3
If there is point p K for any K such that p & (o)() then consider the labelling such that Pi.p belong to class one and (2))), Q belong to class two. A perfect classifier from H. does not exist on this labelling; using the unique hyperplane generated by P1 to label class one as the positive class will classify p wrongly, because p & (P)))) On the other hand, using a hyperplane generated by P2)(p) to label class two as the positive class will classify p wrongly because p E Pol(M(E) (P2
Thus, consider that there is no such p that exists (ic. for any K that fulfills the constraints, E K will belong to the column space of M(P)). Then, pick K such that K is the largest possible set. that fulfills the constraints (ie. ())) = (ank(M((2))) Note that [K] 1.
If [K| > 1, then consider 2 distinct points p.q E K and the labelling such that Pi,p.Q P. Q belong to class one and P2 {p) belong to class two. A perfect classifier from H does not exist on this labelling: using the unique hyperplane generated by Pi to label class one as the positive class will classify q wrongly, because q E (P)). On the other hand, using a hyperplanc generated by PM to label class two as the positive class will classify P wrongly because p E
Otherwise, if |K| L, then consider x E P1 such that Pi PiMx)UU UK generates unique hyperplane which is the same hyperplanee as the unique hyperplane generated by Pt. Such point exists because p E K is a linear combination of points in PT. Hence, such a point x exists TM Then, consider the labelling where P1 Mxx,, K belong to class one and labelling: using the unique hyperplane generated by P PIX(x UK to label class one as the positive class will classify x wrongly, because x E (P)() On the other hand, using a hyperplane generated by PAKK to label class two as the positive class will classify p E K wrongly because p E (Ps)))))
x, Q belong to class two. A perfect classifier from H does not exist on this
Case 3 Lastly, we consider the case such that there is no such partition such that both M(Pi) and ((p) are not rank deficient. In other words, the 2 hyperplanes generated by P1 and P2 respectively are both unique. Consider Q := q).
If both p and q do not lie in the column space of either M(P) or M(22), then consider the labelling where PI.D belong to class one and P2. q belong to class two. A perfect classifier from H does not exist on this labelling: using the unique hyperplane generated by P1 to label class one as the positive class will classify q wrongly, because p & (P))). On the other hand, using the unique hyperplane generated by P2 to label class two as the positive class will classify q wrongly bocause q & (P2))
Otherwise, there exists a point in Q that lies in the column space of either M(P)) or M(P)). Consider the case where one point does not lie in the column space of either M (P1) or (P2). Without loss of generality, let p be the point that does not lie in the column space of either (P:) or M(22), and let q E col(M (P2)). If col(M( C (P)(), then consider the labelling where Pı,q belong to class one and P2. P belong to class two. A perfect classifier from H. does not exist on this labelling: using the unique hyperplane generated by P1 to label class one as the positive class will classify some point x E P2 wrongly, because x E (P)) S (PI) is classified as the positive class although it belongs to the negative class. Meanwhile, using the unique hyperplane generated by P2 to label class two as the positive class will classify
9 hyperplane such that all points in Pb) lie on We repeat similar argument future analyses, and omit this comment in the future for brevity.
24

---

Published in Transactions on Machine Learning Research (06/2024)
P wrongly because p E col (P2() Else, (P2)) g (P))) so there exists a point 8 E P3 such that s E (o.() Then, M(P (s) U {q}) is full rank because is a linear combination of all points in Pz. Hence, the unique hyperplane generated by Pa U {q} is the same hyperplane as the unique hyperplane generated by P2. Then consider the labelling where S1,8 belong to class one and P2 (=) q. p belong to class two. A perfect classifier from H does not exist on this labelling: using the unique hyperplane generated by P1 to label class one as the positive class will classify wrongly, because s & (ol)). On the other hand, using the unique hyperplane generated by Ps U {q} to label class two the positive class will classify p wrongly because p & co2(
Hence, consider the case where both p and q lic on the column space of ((P)) M(22). If they lic on different column spaces, without loss of generality, let p E (PI)) and q E (P2)) Note that p & (P2) and q & col(M by our assumption. Then consider the labelling where Pı.q belong to class one and P2. p belong to class two. A perfect classifier from H does not exist on this labelling: using the unique hyperplane generated by P1 to label class one as the positive class will classify q wrongly, because q E (P))) On the other hand, using the unique hyperplane generated by P2 to label class two as the positive class will classify P wrongly because p & cP2(
Thus, we consider the case where and lie on the same column space. Without loss of generality, let P.qE (P.))) If there is a point that does not lie in the column space of M(P3), then without loss of generality, let this point be p. Consider a point x E P1 the unique hyperplane generated by PMX[ U {q} is the same hyperplane as the unique hyperplane generated by Pt, because q is a linear combination of points in P1. Then consider the labelling where Pi Mx)). q belong to class one and P2, p, belong to class two. A perfect classifier from H does not exist this labelling: using the unique hyperplane generated by P1 to label class one as the positive class will classify x wrongly, because x E (ol(M will be labelled as positive although it is from the negative class. On the other hand, using the unique hyperplane generated by P2 to label class two as the positive class will classify wrongly because & co)(M
Otherwise, consider the case where both p, q E (P1( and p, q E col() Then consider the labelling where Pi,q belong to class one and P2, P belong to class two. A perfect classifier from H does not exist on this labelling: using the unique hyperplane generated by P1 to label class one as the positive class will classify p wrongly, because p E (P.) will be labelled as positive although it is from the negative class. On the other hand, using the unique hyperplane generated by P2 to label class two as the positive class will classify q wrongly because col(M(P:) will be labelled as positive although it is from the negative class.
We have exhaustively considered cases and showed that, for any set of 2n + 2 distinct points, H is unable to shatter it. Thus, VCdim(H) S 2n +
B.4 Proof of Corollary 2.4 VC dimension of c-error separator
Before proving Corollary 2.4 we first restate it:
Corollary B.7. For e-error separators He in Def. 2.2 2n + S VCdim(H.) S 2n +
The lower bound of 2n + 1 follows from Theorem 2.3 since the margin of the e-error separator can be made arbitrarily small.
The upper bound is from the union property of hypothesis classcs found in Excrcisc 6.11 in Chapter 6.8 of & Ben- (2014). The VC dimension of hyper-hill (equivalent to hyper-ridge) classifiers are n + Flake 1993), so we have an upper bound of (n + 1) (n + +) + = 2n + 3.
Work can be done to show that the VC dimension in I² is 2n + 2. We conjecture that this is true for all n > 2 as well, but leave this to future work. Note that for n = 1, the VC dimension is 2n + 1 = 3 the reduction in expressivity seems to arise from the distance of a given datum to be a given point rather than a line, plane or hyperplane (which extends to infinity, unlike a point).
25

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix C Proof for separating hyperplane in pseudo likelihood ratio test
We provide the proof in Section 2.2 that, when we use 2 parallel equality separators (wN -22) and equal class weighting (1 = 1, ie. a false positive is as bad as a false negative) for binary classification, our pseudo likelihood ratio test.
I 2t) =1
W + b2l
1 |will
wf + b1 22
|
decomposes to
-1) x + bi + 22)
which corresponds to the separating hyperplane
2w x + + b2.
Proof. Instead of dealing with absolute values, we take the square on both sides (which does not change the inequality because both sides are positive). With parallel hyperplane and equal class weighting, we are able to cancel out the square terms to obtain separating hyperplane. The exact steps are detailed as follows:
1
62 21) 9
21==1
11
2 12 C
(e) P22
1 ((w3'x x + 22) 2 (wfx
1 ((w3x + b2)² - f 20)
((WIx b2) (w]x+b))² 0)
1 1=1
1 ((wIx+ b2)² (w((x+)))20
Wi = W2
1 (2wīx((b2 - b1) + (62 - 6f) 20
expand the square
((b2 (2w x (b2 b.)) 2))
((b2 by (2w b, bb) 0)
If by # b2, then we have
1
62 2t E1
((b2 61) +b +1
+ b2) 2 0)
1 (2w7 x + b, + b2 2 20.
We note that in the case of XOR, b, = by and the hyperplanc to classify cach class is the same. Hence, there is no separating hyperplane between the classes. This is another way we can view the linear non-separability of XOR.
26

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix D Closing Numbers
Formally, the closing number of hypothesis class is defined as
k
CN(H) == min (k: 3h1 h E H, 0 < Vol KEN
n h4
< oo)
where Vol is the standard Lebesgue measure and Ni- by 1x We [k], (x)x= 13 is the set of all points that is labeled positive by all hypotheses. Note that although possible in reality, we disallow zero- volumed decision regions due to degenerate cases in analysis (e.g. the intersection between two non-intersecting parallel halfspaces is not particularly interesting for classifying data).
Closing numbers are a dimension- number, similar to VC dimension, Instead of quantifying the expressivity of a hypothesis class, quantifies the capacity to reduce open space risk induced by a hypothesis class. While classical measures refer to the open space risk of a particular hypothesis (similar to the empirical risk of particular hypothesis), closing numbers refer to the hypothesis class (like VC dimension). Such measure of hypothesis class is useful in choosing which hypothesis classes to use in practice the generalization scen examples (i.e. normal data and soen anomalics) is captured in the VC dimension, while ability to reject unseen examples (i.e. unseen anomalies) is quantified through closing numbers I
We first note that whenever we refer to a closed decision region in this section, we refer to a region with positive (i.c. non-zero) volume unless otherwise stated.
To prove that the closing number is k, it is helpful to show the existence of closed decision region using the intersection of k hypotheses and the impossibility using fewer (i.e. intersection of k - hypotheses).
We proceed to prove Theorem 3.2 To recall, we restate Theorem 3.2
Theorem D.1. Closing number of RBF, equality and halfspace separators is 1, and n + 1 respectively in R". Closed decision regions are formed with the intersection of the corresponding number of classifiers everywhere for RBF separators and almost everywhere for equality separators, but normal vectors need to form an n-simplex volume for halfspuce separulors.
We break the theorem into lemmas:
Lemma D.2. Closing number of RBF seporalors in R". Closed decision regions are formed everywhere. Lemma D.3. Closing number of equality seperators n in RR.. Closed decision vegions are formed with the intersection of n equality separators almost everywhere.
Lemma D.4. Closing number of halfspace seporators is n n IR. Closed decision regions are formed with the intersection of n + halfspace separators where normal vectors form an n-simplex volume.
Note that RBF separators have a closing number of 1 by construction, so Lemma D.2 is trivial. We proceed to prove the closing numbers of equality separators (Lemma D.3 and halfspace separators (Lemma D.4.
D.1 Proof of Lemma D.3: closing number of equality separators in E
To show that the closing number is most in R", we observe that hyperrectangle can be formed through the intersection of equality separators with orthogonal normal (ie. weight) vectors where the volume in the margin is considered the positive class. Since the volume of hyperrectangle is finite, so is this intersection.
Proof. If we have the intersection of & < equality separators, observe a decision region with infinite volume or no volume. The first case is when there exists an equality separator that assigns the positive class to the region outside of the margin. This induces an infinite-volume decision region for non-parallel normal vectored equality separators. Otherwise, the decision region either has no volume (when the intersection of parallel equality separators is a pair of hyperplanes, a hyperplane or the empty set) or infinite volume (in
Cloosing numbers describe the finite-ness of the enclosed decision region, there no explicit relation on minimizing the volume of this enclosed space of the chasen hypothesis.
27

---

Published in Transactions on Machine Learning Research (06/2024)
the more general case). The other case is where all equality separators assign positive classification to the region within the margin. Without loss of generality, let the bias terms of all k equality separators be 0 and consider the subspace S spanned by the k normal vectors. Since & < n (or, similarly, in the case where the set of weight vectors of equality separators do not form a basis for RR), then there exists a vector v that is perpendicular to subspace S. The volume of the intersection between these equality separators can then be formulated as
Vol(decision region of l equality separators)
=Vol((x 1})
3v STS S
Σ EE
E [
because the intersections (hyperplanes) have zero measure. When none of the equality separators are strict oquality separators, we obtain an infinite volume since Vol(fx E I: I v S +(+1)) > 0. By definition of the Lebesgue measure, the intersection has (at least) infinite volume by moving along the v direction. Otherwise, Vol([x E 1: v"x S i 13) 0, so we have zero volume. Hence, having fewer than n equality scparators cannot induce closed decision region.
Conditions for closed decision regions Consider equality separators, which be represented as collection of their normal vectors in an X matrix A R=x" The first condition for the fonito-volume argument to hold (by observing the formation of a hyperparallelogram) is that these normal vectors form a set of linearly independent vectors (more precisely, that they form basis for R"). Furthermore, we notice that the set of all rank deficient matrices has zero measure. An intuition for this is that, we have any rauk deficient matrix A, we can always perturb its eigenvalues to obtain a full rank matrix (i.e. non-zero eigenvalues) A + AI for some scalar X, where À can be arbitrarily small. Hence, this set has no volume.
The second condition for equality separators to enclose decision regions is when they hyper-hill classifiers (i.e. normal data belongs within the margin). This is how our insight is birthed that can simply use hyper-hill classification in AD - we are using more expressive hypothesis class with equality separation but technically have the VC dimension of hyper-hills (or hyper-ridge classifiers), which is half of equality separators.
D.2 Proof of Lemma D.4 closing number of halfspace separators in R
To show that the closing number is at most n n 1, we observe that a hypertetrahedron (convex hull of n + points or a bounded convex polytope with non-empty interior) can be formed with the intersection n + hyperplanes, where each the facets of the convex hull correspond to each the n + 1 hyperplanes.
Now, we want show that form bounded convex polytope with non-empty interior, assign a one-to-one correspondence between hyperplanc and facet of the convex polytope. In fact, by considering dual polytope of the convex polytope, there is a further correspondence to each hyperplane and vertex of the dual polytope. With non-empty interior, the dual polytope is re-iimnnsional, so has at least n + vertices. Hence, we need at least n + facets. Thus, to form a bounded convex polytope with non-empty interior with the intersection of hyperplanes, we need at least n + 1 hyperplanes.
is minimal. Since we are considering the intersection of hyperplanes
28

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix E
Design choices: bump activations and loss functions
Using different activations and loss functions to model equality separation introduces different implicit biases in the solutions obtained. We introduce possible activations and loss functions, exploring an intuition of the implicit bias induced by these differences (and hence, what applications fit better with these implicit biases).
E.1 Other possible bump activations
Other activation functions can be used as bump in addition to the Gaussian bump activation that we used in this paper. For instance, we can use the inverso-squared of the hyperbolic tangent, tanh(e/=²), where e is a hyperparameter controlling the flatness the bump function near = (Figure 5). Using the plateau can be used to drastically reduce the penalization when the input is within some distance from the hyperplane, which is similar the idea of hinge loss in SVM. In other words, the Gaussian bump encodes hyperplane membership, while the inverse-squared of the hyperbolic tangent encodes margin membership (which has more forgiving penalty for data off the hyperplane).
Output Activation against Input for == ===
Derivattves of Bumps with respect to Input tor V-
1.0
Gaussian Derivative
0.75
Tanh Dorivative
0.8
0.30
Donvative with respact
Output Activation
0.6
0.4
-0.25
0.2
-0.50
Gaussian Bump: exp(-j()?
Tanh Bump: tanhlk))
0.0
-3
Input 2
Imput 2
(a) Output of the activation functions.
(b) Derivatives of the activation functions.
Figure 5: Two different bump activations: the Gaussian bump which we used in blue, and the hyperbolic tangent bump in orange.
As note, the Gaussian bump is of the form (exp (w; z. 2:)l E Rd for input and weights z, Rd respectively. This is different from the Gaussian RBF, which is of the form exp (||z - cll2) E R for input and center z, c E IRd respectively.
E.2
Loss functions
For input into the activation function in the output layer, we provide visualizations of how the mean-squared error (MSE) and logistic loss (LL) compare for the negative class (Figure 6a] and positive class (Figure 6bb. LL penalizes more on wrong classifications for both classes. Such a scheme good for learning a stricter decision boundary, but potentially harder learn when the distributions of the positive and negative class are close and harder when they overlap. On the other hand, MSE has smaller penalization for wrong classifications, which leads to more tolerance for noise but may lead to learning sub-optimal solutions. Such difference hints at different applications, where LL is generally preferred unless the distributions of the positive and negative class are very close (or overlapping, like in negative sampling), in which case the more conservative MSE would probably be more suitable.
29

---

Published in Transactions on Machine Learning Research (06/2024)
Output Activation Negative Class against Input for
Output Activation of Positive Class against Input for
Output Activation of Negative Class
Output Activation Poskivve Class
-2
-4-
-2
-
-3
-6-
MSE: (0 exp(-1ap
MSE: (1 2X)
Log Loss: logil exp(-5(1
Log Loss: loglexpt-51)
-3
(a) Output of mean-squared error and logistic loss against the input to the bump activation function when the datum is from the negative class.
(b) Output of mean-squared error and logistic loss against the input to the bump activation function when the datum is from the positive class.
Figure 6: Comparison of mean-squared error and logistic loss (a = 1 in the bump activation).
Appendix F
Experimental details
In this section, provide details of our experiments. Unless otherwise stated, training and testing both have m 100 data samples.
F.1 Role of in the bump activation
The gradient of the log- log-concave bump activation is
Oz B(,,u,o)
1l g² exp
1
1
which vanishes to 0 the imput tends towards negative or positive infinity. To prevent vanishing gradients, initialize to be decently large to decrease the sensitivity to weight initialization. Experiments in Appendix F.2.3 will shed further light on how the initialization of a affects performance (including the effect of loss functions). For simplicity, we set = 0 but give the option for a to be learnable across neurons in the same hiddon layer in our experiments. Setting a to be learnable allows the model to adapt to its ability to represent the data, such as decreasing o² to increase the separation between the two classes, Otherwise, the separation is controlled implicitly by w through |/w/|2, where the margin is proportional wila- Unless otherwise stated, a is assumed to be a fixed hyperparameter at 10. In equality separator NNs, we fix o = 0.5 in the output layer for stability during training.
F.2 Linear classification for anomaly detection
To understand the usability of equality separation in NNs, we first look at lincar classification for AD with (1) linearly inseparable data and (2) linearly separable data. We select key results to show here and include more details in the following subsections.
Linear problem 1: living on the plane, linearly inseparable We generale data on hyperplanes as the positive class while the negative class follows a uniform distribution in the ambient space. The positive to negative class imbalance (P:N) of 0.9:0.1. Weights and bias of the hyperplanc are drawn from standard multivariate normal, and data are perturbed with Gaussian noise centered zero with varying standard deviations. We repeat experiments 20 times. Results reported in Table 5
30

---

Published in Transactions on Machine Learning Research (06/2024)
Table AUPR for linear AD of linearly inseparable data with varying standard deviation (Std) of noise and data dimension (Dim). No noise denoted by 0.0 std. Random classifier has 0.90 AUPR.
Std S
5
15
25
30
35
0.0
0.99
1.00=00.00
0.99:10.06
1.00=+.000
1.00:10.00
0.99:10.06
1.00=10.00
0.5
0.98=0.06
1.00=#0.00
0.99#10006
1.00=#0.00
1.00=00000
0.99#10006
1.00==0.00
1.0
0.98++0.05
1.00++0.01
0.98++0.05
1.00++0.00
0.99+-0.04
0.99-1:.06
1.00++0.00
1.5
0.97=00055
1.00=#.01
0.98+0.05
0.99
0.99
0.99
1.00
2.0
0.97=#0.05
0.98+t0.02
0.99
0.99
1.00 0.98+
1.00=:0.00
Table 6: AUPR for 2D Gaussian Separation by SVM and our equality separator (ES), with varying noise multiplier (NM). ES competitive with SVM.
NM
1.0
4.0
7.0
10.0
13.0
16.0
19.0
SVM
1.00=0.00
1.00=#0.00
0.99+0.01
0.98
0.97+0.03
0.97+0.01
0.97=0.03
ES (ours)
1.00
1.00
0.99
0.98
0.97+0002
0.97
0.97=0.01
Especially in higher dimensions, our equality separator is robust even when noise is sampled with higher variance than the weights. The inductive bias hyperplane learning helps our equality separator to be less sensitive to the signal-to-noise ratio. Although simple linear regression with only positive examples may more suitable this case, we demonstrate that using negative examples as regularization can still achieve good performance, suggesting its usability in deep learning.
Linear problem 2: competing with halfspace separators linear separability We compare our equality separator with robust halfspace separation, linear-kernll SVM with weighted class loss of the P:N of 0.9:0.1. To vary the overlap of distributions, we multiply the covariance matrix of each Gaussian by varying scalars, denoted as a noise multiplier. We repeat experiments 20 times and report results in Table B
Unlike the SVM trained, our equality separator does not explicitly account for class imbalance and does not incoporate the robustness of hinge loss. Yet, equality separator performs as well the dlass-weighted SVM and also reduces the open space risk with respect to the positive class (i.c. the class with more data available). Once again, the bias of hyperplane learning encoded in equality separators makes it robust to class imbalance.
F.2.1 Linear problem 1: linearly inseparable data
We report the details of the experiments done experiments linearly non-separable data. As mentioned, we model the positive class to live on some hyperplane, while the negative class lives uniformly in the ambient space, where we vary the dimension of the ambient space d across experiments to be 5, 10, 15, 20, 25, 30 and 35. The normal vector w E Rd the hyperplane is drawn from a standard multivariato normal
W~ MVN(O, Ta)
while the bias b E R is drawn from a standard normal
b b0,,1.
A datum from the positive class xp is generated where the first d dimensions are drawn from uniform distribution
Uniform( (10, , {1,..., d 1}
and the last th dimension calculated by
d-1 i= w
d p
Wd
31

---

Published in Transactions on Machine Learning Research (06/2024)
Table 7: AUPR for linear AD of linearly inseparable data with varying standard deviation (Std) of noise and data dimension (Dim), optimized with mean-squared error. No noise is denoted by 0.0 std. Random classifier has 0.90 AUPR. Number of data samples is 20 times the number of dimensions.
Std Dim
5
10
15
20
25
30
35
0.0
0.99:10.06
1.00:+0.00
1.00++0.00
1.00:+0.00
1.00:+0.00
0.99-10.05
1.00:+0.00
0.5
0.98
1.00:10.00
0.98
1.00:10.00
1.00=00000
0.98::0.05
1.00=00000
1.0
0.98+0.05
0.99
0.99+0.00
0.99:00011
0.99
0.99+0.01 0.98:0.05
1.5
0.97#0.05
0.99
0.98+0.05
0.9940.01
0.99
0.97#0.05
0.99
2.0
0.97++0.05
0.99++0.0
0.98++0.01
0.98+0001
0.98++0.01
0.97++0.05
0.98++0.01
where W, E R is the 7th dimension of normal vector w E R. On the other hand, data from the negative class are obtained by sampling each dimension from a uniform distribution between -10 and 10. There is 0.9:0.1 positive to negative class imbalance, and a total of 100 data samples.
After all data are generated, random noise is added to the data. The random noise follows a Gaussian distribution with zero mean and standard deviation of 0.5, 1.0 and 1.5 (we abuse notation and refer to no noise being added as 0.0 standard deviation) across the different experiments.
We optimize over mcan-squarod error (MSE) where the hypothesis class is the set of affine prodictors Aa and 0 and a = 10 are fixed. The gradient-based optimizer used is Broyden--Fletcher
the lyperparameters Coldfarb-Shannol (BFGS), the loss function used in Section F.2 is MSE, and the parameters are initialized to a standard multivariate normal distribution. We use the minimize function in the oottimie packagd where the stopping criteria is when the step size is 0.
Each trial repeated a total of 20 times by generating new dataset by sampling a different weight and bias vector (that defines a different hyperplane) and running optimization again, and we report the mean and standard deviation of the results. As previously reported, results are in Table 5
Results
The results of our equality separator reported are somewhat interesting. On one hand, as expected, increased level of noise leads to decrcased mcan AUPR. With more noisc, the overlap between the positive and negalive class increases, so the separation among the classes (and hence the mean AUPR) decreases. On the other hand, the performance in higher dimensions generally increases, with mean AUPR increasing and AUPR variance decreasing. This phenomenon is probably because there is a greater density of noise in lower dimensions for constant amount of data, leading to a greater overlap of the 2 classes.
More data for increased dimensionality Furthermore, a linear increase in the data does not seem to affect the performance significantly too. We performed the same experiment, but instead of a constant 100 data samples for all experiments, the number of data samples was adjusted be 20d for inpul dimension d. Inereasing the number of data samples will increase the amount of data from the negative class, and hence increase the overlap between the positive and negative class. However, results reported in Table 7 suggest that this increase in overlap does not affect the mean and variance of AUPR of the equality separator. The inductive bias of linear regression helps in hyperplane learning, even when the overlap between the positive and negative class increases.
As seen in both Table [ and Table 7 there are instances where results are reported with high variance (particularly in 5D and 30D data). These instances of high variability of results suggest that the equality soparator could be scnsitive to initialization. Such variability would be docreased in a regular least-squares linear regression because gradient descent can be run on the convex mean-squared error loss and without. the need for including regularization of the negative examples. However, the regularization in the equality
minimsee-imiiii
32

---

Published in Transactions on Machine Learning Research (06/2024)
Table 8: Logistic loss optimization with constant amount of data for linear AD of linearly inseparable data with varying standard deviation (Std) of noise and data dimension (Dim). No noise is denoted by 0.0 std. Constant 100 data samples. AUPR is reported.
Dim Std
5
10
15
20
25
30
35
0.0
0.95-+0.06
0.98:+0.01
0.98++0.01
0.99:L0.0
0.99-1:0001
0.98:10.05
1.00-1:.000
0.5
0.96:0.05
0.98+0.01
0.9810.01
0.99:1.01 0.99
1.00=1.000 0.98
1.0
0.95#00.05
0.98+0.02
0.98#0.01
0.99
0.994:001
1.00=:0.00 0.98+0.00
1.5
0.94+0.05
0.97#0.02
0.97#0.01
0.98
0.9910.01
0.98+0..05
1.00=
2.0
0.93++0.05
0.96++0.03
0.97++0.01
0.98++0.01
0.98++0.01
0.97++0.05
0.99+0000
Table 9: Logistic loss optimization with number of samples 20 times the number of dimensions, done for linear AD of linearly inseparable data with varying standard deviation (Std) of noise and data dimension (Dim). No noise is denoted by 0.0 std. Number of data samples 20 times the number of dimensions. AUPR is reported.
Std Dim
5
15
25
30
35
0.0
0.95++0.06
0.96++0.01
0.96++0.01
0.96++0.01
0.96++0.01
0.96++0.01
0.96++0.01
0.5
0.96:10.05
0.96:10.02
0.9510.01
0.96:10.01
0.96:0.01
0.95
0.96
1.0
0.95=10055
0.96+0.02
0.95+:001
0.96+00.22
0.96+0.01
0.96+0.01
0.95
1.5
0.94+0..0
0.95++0.02
0.95++0.22
0.95++0.01
0.95++0.01
0.95++0.01
0.95++0.02
2.0
0.93++0.05
0.95++0.02
0.94+0.02
0.95+t0.01
0.95+
0.94
0.94
separator to have negative examples further away from the hyperplane means that the equality separator is able to use negative examples and is useful in deep methods for supervised anomaly detection.
Logistic loss To compare our results with MSE, also use LL and report results in Table for constant 100 data samples across experiments and Table g for the same linear increase in amount of data explored in the MSE case. For constant data, we also observe that LL produces performance that improves (higher mcan and smallor variance in AUPR) as the number dimensions increases. On the other hand, the same linear increase in data as the MSE case does not increase the mean AUPR for equality separators optimized with LLI., but only decreases the variance of the AUPR. This simple comparison hints the different usages of different loss functions.
As suggested in Appendix overlap between the positive and negative class increases (ie. as the noise increases). Since noise is sparser in higher dimension for constant data settings, there is less overlap and LL performs better as the dimensions increase (a sample visualization is shown in Figure 7 Since the hyperplane is overdetermined with the data, finding the hyperplane is not much a problem. However, when the amount of data increases, the amount of noise also increasos. Since LI. is more scnsitive than MSE the overlap between the positive and negative classes, the higher density of noise causes LL to degrade in performance.
E.2 LL is more sensitive wrong classifications, so it performs worse the
F.2.2 Linearly separable data
We report the details of the experiments done for linearly separable data. As mentioned, the cach of the two classes are sampled from a 2D multivariate Gaussians. The positive class follows
(H)
0.2 0.1
Xpositive ~ MYN
0.1
0.2
33

---

Published in Transactions on Machine Learning Research (06/2024)
AUPR of Linearly Nonseparable Data with Log Loss for Dim
AUPR of Linearly Nonseparable Data with MSE Loss for Dim 35
1010
100 samples
1.00
samples
1.005
1.000
AUPR
0.990
0.985
0.980
0.94
samples 20d samples
0.970
0.93
0.00
0.25
0.50
0.75
1.00
1.25
1.50
175
2.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Noise
Noise
(a) Logistic Loss
(b) MSE Loss
Figure 7: AUPR of equality separators on 35D linearly nonseparable data across varying noise levels, optimized either with logistic or MSE loss. Each plot shows the AUPR for the cases keeping the sample size constant or lincarly growing with the dimension. Logistic loss is the most affected by overlap between the positive and negative class, which increases with noise and decreases with dimension (note the different y-axis scales).
while the negative class follows
A Xnegative MVN (H+)
0.2 0.1
0.1
0.2
where & is the noise multiplier, scalar that determines the spread of each class (and hence, the overlap between both classes). There is 0.9:0.1 P:N imbalance and a total of 100 data samples for both training and testing datasets.
To implement the SVM, we use the SVM.. support vector classifier class the leikit-learn package, We specify linear kernel and balanced class weight in its argument, and the prediction probability is based an internal 5-fold cross validation 13
SVM
Equality Separator We optimize over mean-squared error as in Equation 2 where the hypothesis class is the set of affine predictors A₂ and the hyperparameters f O and o 10 are fixed. The gradient-based optimizer used is BFGS, and the parameters are initialized to a standard multivariate normal distribution. We use the minimize function the scipy.ope optimize package where the stopping criteria when the step size is 0.
Each trial is repeated total of 20 times by generating dataset by re-sampling from the same multivariate Gaussians and running optimization again, and report the mean and standard deviation of the results.
F.2.3 Study on sensitivity of
We repeat experiments done in Section F.2 on linear separable data, but vary the value of a the bump
activation (Equation 2 to 0.1, 0.5.5 and 10. We report results in Table 10
Equality separators with 0.5 have competitive performance with SVMs, and results are mostly within standard deviation away from each other across the equality separators with values of 0.5, 5 and 10 (Figure Baj. Equality separators with small at 0.1 have slightly worse performance in the low noise settings, with a lower mean and higher variance for its AUPR, but quickly become competitive with SVMs a noise
hhttps: /tts///iiiitterrrre hwwp:///www....//aeooooomaeeeeee
34

---

Published in Transactions on Machine Learning Research (06/2024)
AUPR against Noise Multiplier
AUPR against Noise Multiplier
1.00
0.99
1.00
0.98
AUPR
0.97
0.96
0.96
S/M
0.94
0.95
Equality a 0.5
SVM Equality a 0.1
Equality aw 5.0
0.94
Equality o- 10.0
Equality a 0.5
0.90
10.0
12.5
15.0
17.5
10.0
12.5
15.0
17.5
20.0
Nosse Multiplier
Muttipller
(a) Mean-squared error optimization. = 0.1 omitted for clearer visualizations.
(b) Logistic loss optimization. o == 5.0, 10.0 omitted for clearer visualizations.
Figure 8: AUPR for 2D Gaussian separation by SVM and equality separator under increasing noise and in its bump activation. Mcan-squared error and logistic loss optimization shown.
multiplier of 2.0 onwards. These good results for large a suggest that large values decrease the sensitivity to randomness, such as the initialization of the weights in the equality separator. These empirical results corroborate with our comment in Appendix F.1 about vanishing gradients with small a values impeding effective gradient-based optimization. However, larger values widen the bump of the Gaussian, which increases the margin for error and may cause predictions to be less informative.
Additionally, for all values of a, equality separation is still quite robust at high noise levels, obtaining slightly lower AUPR variance while maintaining the same mean AUPR SVM. Lower variance in AUPR suggests that equality separators can potentially be less sensitive to noise than SVMs. The decreased sensitivity could be due to the equality separator modelling the hyperplane of the normal class (positive class) rather than the separating hyperplane, capitalizing on the class with more data. Hence, equality separation in class-imbalanced settings can produce competitive results with SVMs, especially when the overlap between the positive and negative class increases.
Logistic loss We perform the same experiments on linearly separable data, but change the loss function LL. We report results in Table II and provide visualization in Figure Sb
Training LL is generally slightly less competitive than training with MSE, although not significantly less. With an increase in overlap between the positive and negative classes, there is generally slightly faster decrease in mean AUPR, with a higher AUPR variance. As explained in Appendix E.2 LL does not handle overlap bet ween classes well, leading to a significant decrease in performance. Interestingly enough, smaller o values seem to achieve better performance with LL. Smaller o values produce stricter decision boundaries, similar to LL compared MSE, which suggest how loss functions and values may pair well together.
Varying dimensionality of linearly separable data To understand the effect increasing dimensionality, we also vary the dimensionality of the multivariate Gaussian. For simplicity, we generate data from zero mean isotropie Gaussians with covariance 0.1k la for noise multiplier k. We test d= 5, 10, 15, 20, 25, 30. All other details remain the same. As informed by our experiments varying for different loss functions, we test on MSE optimization with = 10 and LII. optimization with = 0.5.
In general, equality separators are not siguificantly worse than SVMs, maintaining close to perfect AUPR above 0.99 mean AUPR. As we have observed previously, MSE optimization produces better results (higher mcan AUPR with lower variance) in lower dimensions while L.L optimization produces better results higher dimensions. Figure 9 is sample visualization of the comparison between the performance of equality separators and SVMs at d 30.
35

---

Published in Transactions on Machine Learning Research (06/2024)
Table 10: AUPR for 2D Gaussian separation by SVM and our equality separator (ES), with varying noise multiplier (NM) optimized with mean-squared error. The value of a in the bump activation is varied in the ES to illustrate its effect.
NM
SVM
ES (a = 0.1)
ES (a = 0.5)
ES (a = 5.0)
ES (a = 10.0)
1.0
1.00+0.00
0.96++0.08
1.00++0.00
1.00=+0000
1.00=#0000
1.5
1.00+0.00
0.99:10.04
1.00::0.00
1.00:10.00
1.00=#0.00
2.0
1.00=0.00
1.00
1.00
1.00=+0000
1.00=0.00
2.5
1.00+0.00
1.00+0.00
1.00++0000
1.00++0.00
1.00++0.00
3.0
1.00+0.00
1.00
1.00
1.00=+0000
1.00
3.5
1.00+0.00
1.00
1.00
1.00+L0.01
1.00+0.01
4.0
1.00
1.00
1.00
1.00
1.00
4.5
0.99
0.99+00000
0.90
0.90
0.99
5.0
0.99+0.0
0.99+0.00
0.99
0.99
0.99
5.5
0.99+0.00
0.99+0.01
0.99+0.00
0.99+0.01
0.99+0.01
6.0
0.99-10.01
0.99
0.9
0.99
0.99
6.5
0.99
0.99##.00
0.99
0.99
0.99
7.0
0.99++000
0.99+00001
0.99
0.99
0.99++0.01
7.5
0.90+0001
0.98
0.98
0.98+0.01
0.98+
8.0
0.99
0.98
0.98
0.98
0.98+10.01
8.5
0.99+0.01
0.98
0.98
0.98+0.02
0.98
9.0
0.90+
0.98+0.02
0.98
0.98+0.02
0.98+0.02
9.5
0.98
0.98100001
0.98
0.9810001
0.98
10.0
0.98+
0.98+0.011
0.98
0.98+0.001
0.98+10.01
10.5
0.98
0.98100011
0.98
0.9810001
0.98
11.0
0.98+0.01
0.98+0.01
0.98
0.98+0.01
0.98
11.5
0.98
0.98+0.01
0.98
0.98
0.98
12.0
0.98+0.01
0.98+0.02
0.98+0.02
0.9810.02
0.98
12.5
0.98+0.01
0.97+0.02
0.97:00022
0.97+0002
0.97
13.0
0.97+0.03
0.97+0.02
0.97
0.97+0.02
0.97
13.5
0.97++0.3
0.97+0.00
0.97+0.002
0.97+0.02
0.97+0.02
14.0
0.97
0.97:00.2
0.97:0002
0.97
0.97
14.5
0.98+
0.97:40.01
0.97:40.01
0.9710.01
0.97+00.11
15.0
0.98+0.01
0.97
0.97#00011
0.97
0.97
15.5
0.97+0.03
0.97+0.01
0.97++0.01
0.97+00001
0.97+0.01
16.0
0.97+0.01
0.97+00011
0.97+L0.01
0.9710.01
0.97+0.01
16.5
0.97
0.97+0.02
0.97
0.97+0.02
0.97
17.0
0.9740.03
0.9710.01
0.97+0.01
0.9710.01
0.97+0.01
17.5
0.97+0.03
0.97+00001
0.97+00.01
0.97+0.01
0.97+0.01
18.0
0.90+
0.97+L0.01
0.97+40.01
0.97+0.01
0.97+0001
18.5
0.9740.03
0.97+0.01
0.97+0.01
0.9710.01
0.97+0.01
19.0
0.97+00.3
0.97+0.01
0.97+0.001
0.97+0.011
0.97+0.01
19.5
0.97+0.03
0.97+0.01
0.97+0.01
0.97+0.01
0.97+0.01
20.0
0.97+00.3
0.97+0.01
0.97+0.00
0.97++0.01
0.97+0.01
36

---

Published in Transactions on Machine Learning Research (06/2024)
Table 11: AUPR for 2D Gaussian separation by SVM and our equality separator (ES), with varying noise multiplier (NM) optimized with logistic loss. The value of a in the bump activation is varied in the ES to illustrate its effect.
NM
SVM
ES (a = 0.1)
ES (a = 0.5)
ES (a = 5.0)
ES (a = 10.0)
1.0
1.00=00000
0.99
1.00++0.00
1.00=+0000
1.00=#0000
1.5
1.00
0.98
1.00::0.00
1.00:10.00
0.99
2.0
1.00=
1.00
1.00
1.00=00000
0.99=
2.5
1.00++.000
0.99+0.01
1.00++0.00
0.99++0.02
1.00++0.01
3.0
1.00
0.99
1.00
0.99+00002
0.99
3.5
1.00
0.99
1.00
0.98+
0.98+
4.0
1.00
0.98
0.99
0.98
0.97
4.5
0.99+0.00
0.99
0.98++0.02
0.98+0.02
0.97+0.04
5.0
0.99+0.00
0.99
0.97
0.97
0.97
5.5
0.90+0.00
0.98
0.97+0003
0.97+00.03
0.97+0.03
6.0
0.99-10.01
0.98
0.99
0.96110.03
0.96
6.5
0.99
0.99
0.97++0.03
0.96
0.93
7.0
0.99++0.0
0.98+
0.96++0.05
0.97+0.03
0.95+00.33
0.90+0001
0.98++0.02
0.98+00.2
0.97
0.95
8.0
0.99+:0011
0.99
0.97
0.97
0.95
8.5
0.99+0.01
0.97
0.97=:0.02
0.96+0.03
0.94#0.04
9.0
0.90+0001
0.98+0.02
0.98+0.01
0.97++0.03
0.95+00.04
9.5
0.98
0.98100.01
0.97
0.96110.04
0.95
10.0
0.98+
0.97
0.98+
0.96+
0.95++0.03
10.5
0.98
0.96
0.97
0.95
0.95
11.0
0.98+0.01
0.97
0.97
0.95
0.93
11.5
0.98
0.97
0.97
0.95
0.95+0.03
12.0
0.98+0.01
0.98+0.02
0.98
0.95
0.94+0.03
12.5
0.98+0.01
0.97+0.02
0.96:L0.02
0.9510.03
0.94+0.03
13.0
0.97+0.03
0.96+0.03
0.95
0.95+0.08
0.93
13.5
0.97++0.3
0.97+00.02
0.96++0.03
0.95+0.03
0.95++0.03
14.0
0.97
0.97
0.96:10.03
0.95+00003
0.95+0..03
14.5
0.98+
0.97:L0.02
0.97:40.02
0.95+10.33
0.95+10.03
15.0
0.98+0.01
0.97+0.02
0.97+0.02
0.95+0.03
0.95+0.03
15.5
0.97+00.3
0.97+00002
0.97++0.02
0.95+00.03
0.94
16.0
0.97+0.01
0.97+0002
0.96:L0.03
0.9410004
0.94+0.03
16.5
0.97
0.96+0.03
0.95
0.94+0.08
0.93
17.0
0.9740.03
0.9610.03
0.95
0.9510.03
0.93+0.03
17.5
0.97+0.03
0.96+00.03
0.97++0.2
0.95+0.003
0.93
18.0
0.90+
0.96+40.03
0.95-+0.04
0.95++0.03
0.94+00.3
18.5
0.9740.03
0.9640.02
0.97+0.02
0.95+0.03
0.9440.03
19.0
0.97+00.3
0.96+0.03
0.96+00.03
0.95+0.03
0.95+0.03
19.5
0.97+0.03
0.96+0.02
0.95+40.04
0.95+0.03
0.94+0.03
20.0
0.97+00.3
0.95+0.04
0.95++0.04
0.95++0.03
0.94+00.3
37

---

Published in Transactions on Machine Learning Research (06/2024)
AUPR against Noise Multiplier
AUPR against Noise Multiplier
100050
SVM
SVM Equality
Equality
1000255
1.000
1.00000
0.99975
AUPR
0.99950
0.996
0.99925
0.99900
0.922
25
5.0
10.0
12.5
15.0
17.5
20.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Noise Multiplier
Noise Multiplier
(a) Mean-squared error optimization with
10.
(b) Logistic loss optimization with 0.5.
Figure 9: AUPR of equality separators and SVMs on two lincarly separable 30D multivariate Gaussians.
F.3 Non-linear anomaly detection: synthetic data
We report additional details of the experiments done in Section 41 re-initializing models and training them on the dataset. By using the same dataset but re-initializing the models, we can observe each model's sensitivity to initializations and optimization methods. We perform experiments with a relatively low positive to negative class imbalance of 0. 0.75:0.25 during train and test time to more prominently compare the difference between models.
Experiments repeated 20 times by
As a note, we refer the normal class as the positive class in the text because corresponds to a neuronal activation. However, when calculating results, we follow the literature and refer to anomalies the positive class. Since focus comparing between the separation between normal data and anomalics across methods, these details do not affect the analysis.
Shallow models Shallow models are trained using the sklearn library and the boosted trees from the XGBoost library Hyperparameters are picked using 10-fold stratified cross validation under logistic loss on the validation dataset. We list the hyperparameters that were chosen. If hyperpameter is not listed, we used the default ones from the respective librarics.
Decision trees: Split with random split and half of the features, maximum depth of the tree at 8, no maximum leaf nodes, minimum number of samples required to split an internal node and to be at a leaf node at 2 and 1 respectively, and no class weight.
Split with half of the features, 50 trees, no maximum depth of a tree, no maximum leaf nodes, minimum number of samples required to split an internal node and to be at a leaf node at 2 and 1 respectively, and no class weight.
Random forests:
XGBoost (extreme gradient boosted trees): 50 trees, logistic loss reduction required for further splits, 0.1 for é regularization, X 0.01 for 62 regularization, balanced class weight and subsampling ratio of 0.6.
Logistic regression with RBF kernel: C= 1/A = 0.01 regularization parameter with lz penalty and no class weight.
SVM with RBF kernel: C= 1/2 0.1 regularization parameter with l penalty and no class weight.
OCSVM with RBF kernel: The upper bound set on the fraction of training errors is 0.0001, the kernel cocfficient is the inverse of the product of the number of features (2) and the variance of the data.
14 14htpps:/ http:////hhhhoote.aoleec///////h
hti.
38

---

Published in Transactions on Machine Learning Research (06/2024)
Table 12: Test AUPR for equality separator with RBF kernel for non-linear anomaly detection. We vary the loss function (mean-squared error or logistic loss) and the value of a (0.1, 0.5, 5,
Loss 0.1
0.5
10
MSE
0.98
1.0010.01 1.00=0.00
1.00::0.00 1.00=+0.00
1.00=10.00
LL
0.99
1.00=#0.01
Isolation forest: 50 trees that use all samples and all features and an automatic detection of the contamination ratio based the original paper (Liu al. 2008).
Local outlier factor: 5 neighbours used with a leaf size of 30 and an automatic detection of the contamination ratio based in the original paper [Breunig et al. 2000).
For unsupervised AD, there are two possible ways to use the labels: we can either feed all the data into the algorithm (and include information of the ratio of anomalics to normal data when possible, such for OCSVM), or can just feed in normal data into the algorithm and remove the anomalies (because unsupervised methods assume that most/all all data fed in is normal). We noticed that the first method gives us better results, and we report these.
For equality separators, we report the results of varying the loss function (LL or MSE) and the value of a (0.1, 0.5, 5, 10) in Table 12 Equality separators optimized with LL performed better with lower values of o, while those optimized with MSE performed better with greater values of When separators had tendency of underfitting, scoring an lower average AUPR on train data than that of test data. The underfitting phenomenon at this low value corroborates with our observation of the sensitivity of initialization with smaller values to prevent vanishing gradients.
0.1, the equality
Neural networks NNs used are fully connected feed-forward NNs. Below are the details of the NNs:
NNs are trained using TensorFlow All parameters are assumed to be default unless stated below.
2. We use 5 neurons per hidden layer, limiting the chance of overfitting in this low data regime. Not decreasing the number of neurons is also in line with our analysis of closing numbers.
NNs with leaky ReLU activations in their hidden layers have the leaky ReLU with parameter 0.01. Leaky ReLU chosen over the conventional ReLU because mappings that more local (such the bump activation) are more sensitive to initializations (Flake 1993). and backpropagation with leaky ReLU will not suffer from dead neurons like ReLU would. Since we are creating relatively small NNs, the impact of dead neurons may potentially be large. The parameter is chosen to be relatively small at 0.01 to simulate ReLU more closely.
4. NNs that have bump activations in their hidden layers are initialized with 0.5, which is not trainable by default.
5. NNs with RBF activations in their hidden layers or output layer have a fixed weight of 1.0 with learnable centers that are initialized with Glorot uniform initializer. Fixed weights with learnable centers make fair comparison so that NNs will have the same number of parameters
6. Equality separators have output of bump activation with 0.5 that is fixed at 0.5 by default and have a default loss function of binary cross entropy (ie. logistic loss in this binary
1httpps:/wwel htttpps//www..enssffoo...om www.teenssoowwooo org/
Note that each neuron from regular dense layer will have total + parameters in the affine scaling w a b for w,z Rd but the distance from the hyperplanc is determined by the degrees of freedom, which will still be d. Mcanwhile, the center parameter
RBF neuron is in the same space as the input 2, dedimensional. Hence, we view the number of parameters in single RBF layer and single fully connected layer be the same.
39

---

Published in Transactions on Machine Learning Research (06/2024)
classification problem) that can also be switched out for mean-squared error, while halfspace separators have an sigmoid output activation and have a loss function of binary cross entropy.
7. Weights are initialized with the default Glorot uniform initializer.
8. NNs are trained with the Adam optimizer under a constant learning rate of 0.001.
Training is done for 1000 epochs with early stopping. Early stopping monitors validation loss with a patience of 10 and best weights are restored. Validation split 0.1.
Results In addition to the results presented in the main paper, we include results for equality separators optimized with MSE and equality separators with bump activations in their hidden layers with learnable o. AUPRs of the test datasel are reported in Table [3 From our results, equality separators with RBF activations perform the best with a perfect AUPR during train and test times, regardless whether they were optimized under LL or MSE.
Separation between normal and anomalous data Furthermore, as previously discussed, we observe a larger separation between the positive class and the negative test class for these equality separators with RBF activations optimized under LL. Looking at individual models, about half of these equality separators achieved high separation between the normal and anomaly class, where the difference between the mean output of normal data and anomalies during testing is at least 0.30 (Figure 3f.. In the penultimate layer ("logit space"), this separation corresponds to normal data being distance within about 0.05 away from the hyperplane, while anomalies are a distance of about 0.42 to 0.60 away. Even though the output of anomalies is still relatively high (above 0.5), from the equality separation perspective, these anomalies are more than 8 times further than normal data, so sctting reliable thresholds is more straightforward.
The difference between the means of the positive class and negative class during test time was 0.27 + 0.11 for equality separators with RBF activations optimized under LL, compared to 0.16 + 0.11 for those optimized under MSE. In the penultimate layer of the NNs, the average difference corresponds to a distance of the negative class from the hyperplane of 0.40 for LL optimization and 0.30 for MSE optimization (an output prediction of 0.5 corresponds to a distance 0.59 away from the hyperplane). Since this experiment had no overlap between the positive and negative class, LL optimization was more appropriate, obtaining perfect AUPR like MSE optimization but a greater separation between the positive and negative class. A greater separation suggests one-class decision boundaries that are better.
The next best models achieving close to perfect AUPR are the equality separators with bump activations and RBF separators with leaky ReLU or RBF activations. NNs with bump activations in their hidden layers consistently achieved mean AUPRs of above 0.90, suggesting the versality of the bump activation as a seri-lloca activation function.
On a whole, it seems that a mix of globality and locality in activation functions helps to increase one-class classification performance. While equality separation with the local RBF activation was the most effective, equality separators with bump activations (purely semi-local) and RBF separators with leaky ReLU (local paired with global) also have competitive AUPR. When using leaky ReLU in the hidden layers, increasing the locality of the output activation increases performance, while the less global bump and RBF activations also increase performance of halfspace separators. Mcanwhile, NNs with bump activations in their hidden layers consistently achieved over 0.90 mean AUPR. Our experiments corroborate with our intuition in Section [3. that more local mappings (such as the equality separator and its smooth version, the bump activation) can implicitly encode one-class classification objective. The semi-local property of equality separators allows it to retain locality for AD, while improving the separation between the normal and anomaly class of NNs with only RBF activations.
Ablation: 3 hidden layers We also investigate NNs with 3 hidden layers, which is 1 more hidden layer than our original experiments. We report test time AUPRs in Table In general, NNs with 3 hidden layers achieved worse performance than their counterparts with 2 hidden layers. Coupled with train AUPRs of above 0.97, we can see that NNs with 3 hidden layers generally overfit to the negative class during training. Equality separator NNs with RBF activations with 3 hidden layers optimized under LL maintained perfect
40

---

Published in Transactions on Machine Learning Research (06/2024)
Table 13:
Test time AUPR for supervised AD using neural networks with 2 hidden layers. For NNs, we modify the output layer (halfspace separation (HS), equality separation (ES) and RBF separation (RS)) the rows and activation functions in the columns. For equality separators, we include information on the loss optimized on (logistic loss LL or mean-squared error MSE). If a is learnable in the bump activations in the hidden layers, we include added "s" suffix. (This table includes results from updated experiments done in the main paper.)
NN(Activation
Leaky ReLU
Bump
RBF
HS
0.62:+0.12
0.92+t0.13
0.83
RS
0.97
0.91
0.97=0.06
ES (T.L.)
0.83++0.15
0.99++0.04
1.00=0.00
ES (MSE)
0.77+:0118
0.98:00009
1.00=0.00
ES-s (LL)
0.98
ES-s (MSR)
0.95++0.11
Table 14:
Test time AUPR for supervised AD using neural networks with 3 hidden layers. For NNs, we modily the output layer (halfspace separation (HS), equality separation (ES) and RBF separation (RS)) in the rows and activation functions in the columns. For equality separators, we include information on the loss optimized on (logistic loss LL or mcan-squared error MSE). If a is learnable in the bump activations in the hidden layers, we include an added "s" suffix.
Activatio
Leaky ReLU
Bump
RBF
HS
0.69+0.16
0.92++0.14
0.63
RS
0.96:00.06
0.89
0.94
ES (LL)
0.8310.11
0.95
1.00=0.00
ES (MSE)
0.84+0.16
0.90++0.13
0.99
ES-s (LL)
0.98
ES-s (MSE)
0.90++0.16
AUPR and achieved a separation between the positive and negative class during test time of 0.22 + 0.13, which has a mean that is only slightly lower than their counterpart with 2 hidden layers. Those optimized under MSE also maintained a close to perfect AUPR with a separation of 0.16 = 0.14, which is similar to their counterparts with hidden layers.
In our experiments, NNs with bump activations in their hidden layers were consistently resistant to severe overfitting while maintaining relatively good test performance of mean AUPR least 0.89, with mean test AUPR always within 1 standard deviation of their counterparts with 2 hidden layers. On the other hand, NNs with RBF activations in their hidden layers maintained a high test AUPR except for halfspace separators. Meanwhile, NNs with leaky ReLU generally maintained performance within standard deviation. Tn general, our results suggest that bump activations are more robust to overfitting while being able to achieve good performance for anomaly detection. Future work to more thoroughly investigate this overfitting phenomenon can be done to understand how to further mitigate overfitting, especially without prior exhaustive knowledge of all possible negative examples.
Learnable a in bump activations Having learnable a parameter in the hidden layers of the bump activation generally do not affect AUPR significantly for equality separator NNs optimized under LL or MSE. Moreover, a pattern emerges for both 2- and 3-hidden layered NNs for individual NNs that performed well, the a value the first hidden layer tends to increase while the a value in the latter hidden layer tends decrease, with the final value of usually being higher at the first hidden layer than the latter layers after convergence (Figure 10p. A larger in the earlier layer signifies allowing more information through the layer
41

---

Published in Transactions on Machine Learning Research (06/2024)
Sigma against Epoch
Sigma against Epoch
0.650
bump1 bump2
0.70
0.65
0.600
0.60
0.575
bump1
0.50
tump2
bump3
0.50
0475
0.45
0450
0.40
400
1000
400
800
1000
Epoch number
number
(a) Neural network with 2 hidden layers.
(b) Neural network with 3 hidden layers.
Figure 10: Example behaviour of a in the hidden layers of an equality separator NN with bump activations and learnable in its hidden layers. The numbered suffix of the label of the curves refers the index of the hidden layer (starting from 1).
because there is a larger margin, while lower in the latter layers suggests greater selectivity in inputs for neuron activation. Intuitively, the first hidden layer increases the dimensionality of the input for feature extraction (from 2D inputs to 5D hidden layer), while the latter hidden layers perform variable selection based on the increased dimensionality.
Such behavior of insightful how gradient-baset optimizers learn equality separation (even in hidden layers and with fixed o). Since backpropagation through the latter hidden layers decreases the value of a and decreases the receptive field of the bump activation, we can infer that backpropagation is also likely to decrease the receptive field of latter layers with fixed in their bump activation. This behavior corroborates with the observation in G.C. Vasconcelos (1995) that bump activations lead to more closed decision regions. Such behavior is especially useful in applications like anomaly detection, where equality separation and bump activations seems to be inclined towards encoding the inductive bias one-class classification.
Model averaging for increased reliability NNs with bump activations generally achieved high mean AUPR with some variance. Particularly, equality scparators with bump activations in their hidden layers achieved the highest mean AUPR (close to perfect), for both 2- and 3-hidden layered NNs. High mean AUPR with a high variance suggest that individual NNs are able to achieve good separation of the positive and negative classes during test time. Therefore, NNs may be slightly sensitive to initializations. To mitigate potentially bad initializations, we can view these NNs weak Iearners for anomaly detection. Then, an ensemble model should hopefully produce a better model for anomaly detection.
The simplest approach of averaging the predictions of the 20 models we trained achieves a perfect test AUPR for both the 2-hidden layered NN (Figure 11ab and 3-hidden layered NN (Figure 11]. By averaging the prediction of the numerous trained models, the bad initializations were cancelled out and the ensemble model performs more reliably.
Other methods can be applied to ensemble the models, such as bootstrap aggregation and boosting. Future work can also look into how to initialize and optimize these NNs with bump activations well to mitigate bad initializations. Other work to directly mitigate overfitting during training of equality separators can explored to increase the separation of normal data from anomalies to move towards separation like the shallow equality separator with RBF kernel, such as what initializations pair well with NN optimization.
F.4 Supervised anomaly detection with NSL-KDD dataset
We report additional details of experiments done in Scction 4.2 for NSL-KDD dataset. Since the dataset size is significantly larger than our synthetic dataset. in Section we opt to test NNs with homogenous
42

---

Published in Transactions on Machine Learning Research (06/2024)
(a) hidden layers.
(b) 3 hidden layers.
Figure 11: The heat map of the average of the output prediction of 20 equality separator neural networks with bump activations and fixed 0.5.
type of activation function (i.c. global, semi-local or local) and repeat experiments 3 times. We report more results in Table 15
Supervised methods We run cross validation for SVM to obtain optimal hyperparameters as per the last section. The one-vs-set machine (OVS) (Scheirer et al. 2013) uses a linear kernel SVM define the first hyperplane then finds the second parallel hyperplane where normal data are within both hyperplanes. However, OVS gives 0-1 classification decísion, not contimnous-valued anomaly score, AUPR may not be the best metric here. To produce continuous-valued score, we introduce the equality separation paradigm by setting the hyperplane in the middle of the 2 parallel hyperplanes as the hyperplane used for equality separation. We denote this as OVS-ES. In our experiments, we tried both linear and RBF kernels for each of these 3 mothods (SVM, OVS and OVS-ES).
There no variance in scores because SVM solves a convex optimization problem, and all methods are SVM-based. Thresholded one-vs-set RBF (OVS-RBF), the original thresholded one-vs-set linear (OVS-Lin) and one-vs-sct linear ES (OVS-Lin-ES) have the results and AUPR train results low (0.703), which also reflects on the low AUPR for DoS (seen) atlacks, so ignore their results in the main text.
Unsupervised methods For unsupervisexl AD methods, we test on the 2 options of dealing with labels like before: feed all the data into the algorithm, or just feed in normal data into the algorithm and remove the anomalies (because unsupervised methods assume that most/an all data fed in normal). In the main paper, we report results of only the former. Here, we report the former with a suffix "-A" " and the latter with suffix "_N" in Table 15 We report the best result in bold, but also report results one standard deviation away in bold to account. for variability across experiments with different seeds.
We OCSVM significantly outperforming all the other methods the unseen anomalies for privilege escalation and remote access attacks. However, OCSVMs are unable to capitalize label information, and it is expected that the supervised SVM comfortably outperforms OCSVM as well. Once again, we notice a trade-off of poorer detection seen anomalies with the former and poorer detection of unseen anomalies for the latter. When all data fed into an unsupervised method, intuitive that the model may view anomalous data as normal and misclassify them, leading poorer detection of seen anomalies. However, when only normal data is fed into the unsupervised method, it is interesting to note that the resultant model is more
43

---

Published in Transactions on Machine Learning Research (06/2024)
conservative about the unseen anomalies. Particularly, notice IsoF obtaining the best AUPR for DoS and probe attacks.
As an aside, we note that LOF is a distance-based algorithm, which is a likely reason for its failure in this high-dimensional setting. We also note that LOF and methods that require kernels (OCSVM) take a significantly longer time train in these high-dimensional settings too, which we observe our experiments as well. Additionally, OCSVM and LOF have no randomness there no variance in the AUPR.
Binary classifiers
We details the hyperparameters for training our binary classifier NNs below.
We train NNs for 500 epochs with an early stopping patience of 10 epochs, with best weights restored. NNs were far from reaching the upper limit of 500 epochs.
2. We used the Adam optimizer with learning rate of 3 x 10-4
3. We use 1 hidden layer with 60 neurons, about half of the input dimension. A smaller width in the hidden layer mimicks the need for dimensionality reduction in real-world datasets. This is challenging because the decision region formed will not be closed with respect to more than half of the dimensions for equality separators and halfspace separators.
4. RBF separators and halfspace separators are trained with LL while equality separators are trained with MSE. In this appendix, we experiment with both LL and MSE and report results, noting that there not much effect of the respective classifiers strengths and weaknesses. We note that equality separators trained with MSE also achieve similar performance those trained with LL.
Some results were slightly sensitive, so we repeated experiments 5 times to obtain more reliable result.
6. We use batch normalization before the activation function is applied (less RBF, where the only option is to apply it after).
7. Other hyperparameters are kept the same as our synthetic experiment (e.g. leaky ReLU for global activations, Glorot uniform initializer for weights).
Our choice was partially guided by the observation that it is casy to overfit the training data (i.c. training loss keeps decreasing while validation loss does not). This observation corroborates with learning theory of how the generalization error bound is looser with more expressive hypothesis classes. In particular, we experimented with 1 hidden layer with 60 neurons, 1 hidden layer with 200 neurons, 2 hidden layers with 60 neurons each and 2 hidden layers with 200 neurons cach.
We also notice that using logistic loss or MSE does not affect the results too much for standard ERM binary classifiers.
NS For NS, we maintain the same training regime but detail / modily the following hyperparametters.
The number of anomalies sampled was times the size of the training dataset.
We trained our equality separators with MSE. Since anomalies can be anywhere, including close the normal data, we use MSE be more conservative about wrong classifications, given the potential label noise when artificially sampling anomalies.
SAD For SAD, we pre-train an autoencoder on our training data, then use the encoder as an intialization train it for supervised Al as in Ruff al. (2020). Training details are as stated:
1. We use encoder dimensions of 90-60 (i.e. a hidden layer of 90 neurons between the latent layer of 60 neurons and the input/outping We employ this for all architectures because we observe underfitting without the intermediate hidden layer. We also validate performance by observing training error converge to when the autoencoder only has hidden layer, where the hidden layer has the same number of neurons the input/output.
44

---

Published in Transactions on Machine Learning Research (06/2024)
2. Depending the type of separator test, the autoencoder employs either leaky ReLU (global), bump (semi-local) or RBF (local) activations homogenously.
3. Autoencoder and classification head trained as per Ruff et al. (2020) with hyperparameters as previously stated.
4. l2 regularization for classification (Ruff et al. 2020 was tried with 10 and 10 weights and found to be insensitive. We reported results using a weight of 10-!
We note that do not imply that RBFs inappropriate for AD. In fact, its locality property makes it well-suiled. However, RBFs are tricky to train (Goodfellow et al. 2015) and may require many neurons to work (Flake, 1993), especially in higher dimensional settings (which are mostly the case in reality). For instance, using RBFs on binary classifiers trained with MSE or on SAD have 0.000 standard deviation in the AUPR. We noticed that learning often converged at a loss a few orders of magnitude higher than the other models, which corroborates with our intuition that NNs with RBFs are hard train and have probably converged to highly sub-optimal minimum. Equality separators achieve a good balance between its ability to form closed decision regions and its capacity to learn efficiently.
F.5 Supervised anomaly detection with thyroid dataset
We report additional details of experiments done in Section 4.2 for the thyroid dataset. All details are the same Section F.4 unless otherwise stated.
The thyroid dataset is medical dataset on proper functioning of the thyroid (a gland in the neck), with 21 features and 3 classes, 1 of which is normal while the anomalous classes are hyperfunction and subnormal. For Thyroid dataset, we train on hyperfunction anomalies and test on both hyperfunction and subnormal anomalics.
1. We choose 1 hidden layer with 200 neurons with batch normalization. Using a validation split of 0.15, 0.15, we confirm that we do not overfit to the seen data during training.
We initialize 7 to increase the margin width so initializations are not as sensitive. (We also note that o 5, 10 produce similar results.)
3. We employ learning rate exponential decay with Adam optimizer under LL to speed up convergence, with an initial learning rate of 0.01 and decay of 0.96.
4. We train NNs for 500 epochs with early stopping patience of 50 epochs, with best weights restored.
We report more detailed results in Table 16 which includes a logistic regression baseline. We see that equality soparation the bost detecting both soen and unscon anomalics.
F.6 Supervised anomaly detection with MVTec dataset
We report additional details of experiments done in Section 4.2 for MVTec defect dataset. All details are the same as Section F.4 unless otherwise stated.
Each object has about 300 training samples, so we use (frozen) DINOv2 (Oquab et al. 2023 embed raw image data into features rather than training our own computer vision model from scratch. DINOv2 is a foundation Vision Transformer (ViT) model pre-trained representations obtained from the model are useful, or at least more useful than a model trained from scratch. Specifically, we use the classification CLS token embedding, since it is not patch-dependent and is widely used for downstream classification tasks.
many images, so assume that the feature
There are far fewer data samples than the ambient dimensionality of the data, so we opt for simple methods Our halfspace, equality and RBF separators are all lutput-layered (i.e. no hidden layers) and act as linear probes, except the RBF separator which acts like SAD where DINOv2 plays the role of the encoder (of a
45

---

Published in Transactions on Machine Learning Research (06/2024)
Test AUPR (mean + standard deviation) on NSL.-KDD dataset, grouped according to different methods. The different methods are random classifiers (calculated in expectation), shallow supervised (Sup.) and unsupervised (Unsup.) methods, binary classifier NNs, Negative Sampling (-NS) and Deep Semi-Supervised Anomaly Detection (-SAD). For shallow supervised methods, we test linear (Lin) and RBF kernels. We compare using different kinds of activations in NNs: global (HS), semi-local (ES) and local (RS). AUPR is reported vis-a-vis each atlack and all attacks (overall).. Sullix for unsupervised AD methods determines if only normal data is used (-N) or all data is used (-A). DoS attacks are seen during training, while the rest are not.
Table 15:
Model Attack
Attack
DoS
Probe
Privilege
Access
Overall
Random
0.435
0.200
0.007
0.220
0.569
SVM-RBF
0.959++0.000
0.787+0.000
0.037+00000
0.524+00000
0.940+00000
SVM-Lin
0.876:10.000
0,494=00000
0.021#0.000
0.151=0.000
0.823=1.000
Sup.
OVS-RBF-ES (ours)
0.956=#0.000
0.785=+0.000
0.038=00000
0.535=0.000
0.949=0.000
OVS-RBF
0.717:10.000
0.600+10.000
0.503
0.610=0.000
0.785=10.000
OVS-Lin-ES (ours)
0.717=10000
0.600
0.503=0.000
0.610=0.000
0.785=000000
OVS-Lin
0.717=10.000
0.600:00.000
0.503
0.610=0.000
0.785+10.000
OCSVM-N
0.835+10.000
0.849:10.000
0.382
0.700
0.920+100000
OCSVM-A
0.779110000
0.827=0.0000
0.405=0.000
0.760=0.000
0.89710.000
Unsup.
IsoF-N
0.964
0.960#0.003
0.039+0.007
0.438=0.015
0.957=0.002
IsoF-A
0.765+00073
0.850+0.006
0.08910.044
0.392:100229
0.86510.003
LOF-N
0.759+0.000
0.501+0.000
0.046+00000
0.451+0.000
0.824+00000
LOF-A
0.495
0.567+0.000
0.03910.000
0.455=0.000
0.718
HS
0.94410.016
0.73910.011
0.0161100006
0.180=1.033
0.877
ERM
ES (ours)
0.970=0.006
0.726+0.099
0.047=0.017
0.446=0.124
0.939=0.014
RS
0.356++0.002
0.213+0.002
0.010+00000
0.228++0.002
0.406+00.001
HS, MSE
0.921+0.015
0.682+0.017
0.010=0.006
0.133+0.008
0.846+0.008
ERM
ES, MSE (ours)
0.974+0.001
0.717=0.107
0.045#0.014
0.510=0.118
0.941#0.014
RS, MSE
0.356+0.000
0.215++0.000
0.010+0.000
0.231+0.000
0.406++0.000
HS-NS
0.936+0.001
0.642+0.030
0.006
0.139=0.001
0.845=000006
NS
ES-NS (ours)
0.945
0.659
0.023110011
0.206-10.013
0.88110.007
RS-NS
0.350+#00002
0.207
0.009
0.223=0.002
0.401#0.002
HS-SAD
0.955110.003
0.766=10.01
0.10010.000
0.447=0.000
0.935
SAD
ES-SAD (ours)
0.960-10.002
0.795
0.047
0.509
0.902-0.002
RS-SAD
0.935+10.000
0.678#0.000
0.022+0.000
0.142=0.000
0.846=0.000
Table 16: AUPR for thyroid by diagnosis (i.e. anomaly type).
Diagnosis
Logistic Regression
HS
ES (ours)
RS
Hyperfunction (Seen)
0.889+0.000 0.340=10.001 0.530=0.001
0.907+0.013
0.934+0.004
0.030+00.002
Subnormal (Unseen)
0.596+00.007 0.459
0.082-10002
Overall
0.615+0.032
0.72640.004
0.103=0.001
46

---

Published in Transactions on Machine Learning Research (06/2024)
pre-trained autoencoder) in Ruff et al. (020 to extract meaningful features from the data. Hence, the halfspace separator acts as our baseline. In high dimensions, sampling to cover the whole space is much harder and computationally heavier, so we exclude negative sampling (NS).
Each anomaly class has very few anomalies (usually around 10), so we use all anomalies from 1 anomaly type during training (we pick the first type of anomaly in alphabetical order). In other words, we do not test. on seen anomalies and only test on unseen anomalies. As the best proxy (or at least an upper bound) for performance seen anomalies, we monitor the training loss and AUPR well. For instance, that RBF separators do not converge and have low AUPR, which suggests that test AUPR on seen anomalies is likely to be worse. Mcanwhile, the two other models (halfspace and equality separators) had converging losses and good train AUPR. Out of these models, equality separators had better test AUPR (i.e. better separation between normal and unseen anomalies) on a whole.
observe
47


Reviews:
[{"review_id": "ttDcJEA56U", "reviewer": "Reviewer_FxLN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strenghts***\n\n1. The proof of the VC dimension equality and inequalities appears correct from my reading, and I have no reason to believe it is not novel. \n2. The experiments are extremely extensive and cover more rarely explored datasets. The method makes some sense and the results appear strong.\n3. The introduction is well written\n\n\n***Weaknesses*** \n\n1. The paper is **not** particularly **well organized**. In particular, the distinction between the linear version of the model and the non linear version is not made very clear.  \n\n2. The notion of closing numbers isn't exploited in much depth. I am still having trouble understanding their *significance*, though I do agree with the proof of the result stated and proved in Section D.1.\n\n3. The experiments appear to be \n\n\n\n\n\n\n***Detailed comments regarding weakness 1***\nThe beginning of Section 4.1 should include equations, and the description of those models should be included in the introduction and the discussion of the main results. In general, there are too many words and not enough equations. In addition, claims and their proofs are not well differentiated, with the whole paper written more like a continuous train of thought, which makes it at times difficult to follow. It is not as bad as many submissions but still an issue. \nThis is true even in the proofs of the VC dimension inequalities, which, are sometimes confusing despite being seemingly correct. \n\nFor instance, Appendix B.1 considers the specific case where there are four points with binary coordinates in sample space, and it is not completely clear to me how that helps the proof of the other statements.  Also, it seems like the inductive step on page 20 could be substantially shortened with a high probability/general position argument.\n\nThe sentence structure makes the proofs occasioinally hard to read. For instance, at the bottom of page 21, it is written that (A) \"$M(P_2\\setminus \\\\{s\\\\}\\cup \\\\{p\\\\})$ is full rank\" because (B)\"$q is a linear combination of the points in $P_2$\" and on the next page (page 22), it is then stated that (C)\"the unique hyperplane generated by $P_2\\setminus \\\\{s\\\\}\\cup \\\\{p\\\\}$ is the same hyperplane as the unique hyperplane generated by $P_2$\".  In fact, fact (B) is what can explain fact C, not fact A. There are similar small issues like this in other places, which exacerbates the fact that the proof is very technical and difficult to read in the first place due to the need to split into many different cases. \n\nSection B.5 is not directly linked to any part of the main paper and I am not sure what it is actually talking about. \nSimilarly (but less seriously), Section D.1 doesn't refer to a specific part of the main paper that supposedly contains the claim to prove.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores classification models defined by equality or near **equality separators**: a function $f$ of the data is learned, and the positive class classification region can then be defined as either $\\{x\\in\\mathcal{X}: |f(x)|\\leq \\epsilon\\}$ or its complement, where $\\epsilon$ can be 0 in the case of strict equality separators. In the linear case, $f$ is an affine function of the form $f(x)=w^t x+b$. The contributions are two fold: \n\nOn the **theoretical side**, the authors study the VC dimension and expressivity of the function class associated to equality separators in the linear case. The conclusion is that the VC dimension of strict equality separators is $2n+1$ whilst the VC dimension of approximate equality separators, the dimension is between $2n+1$ and $2n+3$. These results are intuitively understandable because equality separators can be seen as the combination of two hyperplane separators.  A notion of \"closing numbers\" is also introduced to quantify the expressivity of equality separators from an anomaly detection point of view: the closing number of a function class is the number of elements of the class required to create an intersection of finite non zero volume. It is then shown (Section D.1.) that the closing number of equality separators in $\\mathbb{R}^n$ is $n$. In addition, the beginning of the article and introduction justify the concept of equality separators as a solution to the inability of linear classifiers to learn the XOR function, which is a vaguely curious allegory. \n\nOn the **experimental side**, the methods are non linear (see description at the beginning of Section) 4.1) and are applied to the various anomaly detection problems/datasets. The experiments include both the logistic loss and the Mean Square Error  to learn the separators smoothly.  The methods are then compared to many baselines and demonstrate generally superior performance on three main datasets: a thyroid disease detection dataset, a cyber attack detection dataset and a the MVTec industrial dataset. The experiments are very thorough and distinguish the performance on several anomaly types, with the general conclusion that the proposed method is especially successful at correctly identifying anomalies of unknown type. This is also true in synthetic datasets: for instance, one of the datasets considers the normal class to lie on a sphere, and the method is better at identifying anomalies which are inside the sphere, with hyperplane classifiers and even other methods such as LOF struggling to achieve the same performance there.  In addition, many experiments are run on various parameter configurations, number of layers, choice of loss function, etc.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In decreasing order of importance: \n\n1. Could you reorganize the paper so that the statements are made clearer, the lemmas and claims are clearly separated from their proofs? It would be helpful if each of the statements in the appendix were clearly associated with their analogue in the main text. \n\n2. I am having trouble understanding the difference between the proposed model in the experiments setting and the exact model of [1]. As far as I understand, that model includes a learnable function $f$ (a trained neural network) which then identifies anomalies as those datapoints with large $|f(x)|$. How is this different from the proposed approach? \n\n2.1 I know that [1] also needs to remove biases to avoid trivial results. Did you also have to use this trick? \n\n3. In the experiments, the tables are often split several ways based on methods (see e.g. table 3). This allows you to make your method appear in bold despite it not being the best method when considering all of them. What is the rationale behind the separation? \n\n4. Could you explain your comments regarding the \"delta rule\" and the bump functions more mathematically? Are the bump functions used as activation functions throughout the network?\n\n5. Some of the experimental results are presented without much comment/ insight. For instance, the tables on pages 30 and 31. Is there any way you can draw some graphs to compare the approaches with both loss functions there? \n\n***Reference***\n\n[1] Deep One-Class Classification.  Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, Marius Kloft.  ICML 2018.", "broader_impact_concerns": "There are no broader impact concerns to consider.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ucuZF6ltkG", "reviewer": "Reviewer_GCZc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem of anomaly detection and its connection with binary classification is an important and widely-studied problem. The authors re-invest it it a quite original manner, solving a hyperplane fitting optimisation problem in a different manner than existing methods.\n- The paper is generally very well written and the concepts well introduced. The illustrations are also very helpful. The proofs (in the appendices) are clear and well detailed.\n- The experimental section is well-thought and organised.\n\nWeaknesses:\n- The contributions of the paper are slightly incremental to me. \n    - Firstly, unless I am proven wrong, I think using the distances to hyperplanes for binary classification and supervised anomaly detection is not a completely novel idea. For instance, linear discriminant analysis and robust principal component analysis use similar paradigms although the optimisation problem may be different. I think the paper lacks a bit of clarity and exhaustivity on the relationship with other linear binary classifications methods. The authors should better position their paper with respect to the literature and may consider including a literature review section.\n    - Secondly, I am not completely convinced by the connection between the “equality separation paradigm” presented in Section 2 and the “smooth approximation” as a “bump” activation presented in Section 3.2. It is a bit too quick shortcut to me, and also it is hard to understand in Section 3.2 where the activation function is used, for the hidden layers or the output layer? From the experimental section, I think I understood it could be both.\n\n- There are a few statements throughout the paper that are either imprecise or are not well-enough supported. Here are the ones I have noted:\n    - Section 2.2: “Equality separators yield a more robust separating hyperplane compared to SVMs in binary classification” The notion of robustness should be explained here. For instance, in a high-dimensional setting where the dimension is greater than the sample size, this may not be true.\n    - Section 3: “AD where anomaly classes are not fully represented in the training data”  I kind of understand what it means from the experimental section, but I think I this point it makes things a bit confusing since until this point there was only one class for the anomalies.\n    - I don’t understand the 3rd paragraph of A.2. What do the authors mean by “In some sense, equality separation is agnostic to the exact position of the regression manifold.” Also the following sentence seems to state 2 different things: “equality separation is similar to classification in general” and “equality separation is closer to regression”.\n    - Section 3.4: “controlling generalisation error” then “we control the approximation error”. Which error is it here and how is it controlled?\n\n- My understanding is that using the hyperplane not as a separator like in SVM but as a manifold for normal data may be a better strategy for anomaly detection. I think the paper should better clarify in which contexts this applies. For instance, it seems from Section 3.1. that the focus is on settings where the normal data is on a bounded space. What about if the data manifold is not a compact space? Or if the normal test data density is slightly shifted?  I think there should be a longer discussion on the robustness of the equality separator and half-space separator approaches.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a paradigm for binary classification with linear classifiers, that separate points according to their distance (small or large) from a fitted hyperplane. They call this paradigm “equality separation” and apply it to anomaly detection. Secondly the authors introduce a metric called “closing numbers” that quantify the capacity of a hypothesis class to form closed decision regions, which may be useful for separating the normal points from outliers. Finally they propose a “bump” activation function for neural networks, as a smooth approximation of the equality separation paradigm. This activation function is tested on multiple synthetic and real data sets and notably compared to radial basis functions and half-space separators.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes:\n- Better position the contributions of the paper with respect to the literature. Consider including a literature review section in the main text, for instance merging 2.2 and Appendix A.\n- Increase the discussion in the last paragraph of Section 4 on the advantages and disadvantages of equality separation for AD compared to other methods using hyperplane fitting.\n- I think the numerical experiments in Section 4 should include a comparison to the other linear methods mentioned in appendix A.3\n- Clarify the link between the equality separator and using the Gaussian activation function in Neural Networks. Besides, what is the difference between the Gaussian activation function and the RBF?\n\n\nMinor remarks:\n- I have not found the reference in Hastie et al. regarding modelling XOR\n- I don’t understand the following sentence and its relevance here: “With labeled normal anomaly data, can AD be set up as a classic binary classification objective?” \n- I do not understand the main point of the paragraph on “classification methods” in Section 3.1. It seems to justify that supervised AD should be better than unsupervised AD methods that generate synthetic anomalies \n- An extra “and” in “We see a connection between classifiers that induce closed and decision regions and classifiers that induce smaller, finite-volumed density level sets” p.5 \n- I do not understand where the 2nd equality in the proof in Appendix D1 p.25 comes from.\n- The legends in Tables 3 and 4 are not explicit enough.\n- I think one word is missing in the last sentence on page 17", "broader_impact_concerns": "I have not concern on broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bm3CDHSw8Z", "reviewer": "Reviewer_CXYD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I'm not sure what to make of this paper.\n\nOn the whole, I'm worried the paper drastically oversells its contributions. I'm fairly certain most people are aware that if you were to use something like an equality constraint you could use a \"linear\" perceptron to solve the XOR problem.  Further, I'm fairly certain that to most people \"linear perceptron\" as a concept is intimately tied with the idea of a threshold activation function, so one of the primary claims of the paper, that it \"solves\" the XOR problem with a linear perceptron really comes off as a sort of overly technical \"gotcha\".\n\nNot to sound harsh, but I'm not sure why I should care.  The paper says the VC Dimension for an equality classifier is 2n+1 versus the n+1 of an ordinary (thresholded) linear perceptron, but higher VC dimension doesn't mean it is better in any practical sense.  A sine activation function has infinite VC dimension, that doesn't mean we should use it.\n\nSimilarly, the definition of a *closing number*: the minimum number of hypotheses in a hypothesis class such that their intersection produces a closed decision region, I'm not sure why this is useful. In the context of the paper its meant to contribute to a discussion about the difference between global and local features and how semi-local features might be useful, but you could just say that, I think its clear that an epsilon thresholded absolute value activation is semi-local without having to define a new term. It comes across as an unnecessary formalization.\n\nFor many of the neural network experiments, the paper switches over to a gaussian nonlinearity, which has been explored a lot in the past, and feels like a departure from the original observations of the paper, but I think it is done because an equality or even an epsilon thresholded absolute value nonlinearity has vanishing gradient nearly everywhere so should prove difficult to learn, a comment I was hoping to see in the text.  But, abandoning that, the paper ends up looking a lot like an activation paper purposing a gaussian activation function which is a bit dissapointing.\n\nIn general, that is my biggest issue with the paper.  The initial observation feels like its a cute trick and there is useful broader discussion about the role and differences of local, global and semi local features and the use of bounded activation functions. \n\nHowever I feel like, given the gravity the current draft assigns to the claims it makes is not supported by a similar strength in evidence.  The experimental evidence from the paper centers on toy models and NSL-KDD, a weak demonstration. Why do so many rows in Table 3 have 0.000 for their reported errors? What are the errors being reported? How many times were the experiments run?  I find a reported error value of 0.000 very suspicious and at the same time can't seem to find the details of what it represents in the text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper suggests the use of *equality separation*, an equality nonlinearity, or an epsilon threshold variant for use in neural networks, in particular perceptrons.\n\nThe paper establishes the VC dimension and defines and derives a new measure, the *closing number* to argue that equality separation is better than ordinary threshold perceptrons, they argue in particular for anomaly detection.\n\nThe paper claims that equality separation can solve the XOR problem with a linear perceptron.\n\nThe paper demonstrates the use of equality separation on some toy datasets and KDD Anomaly detection datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I'm not really sure what can be done to turn me around here.  Honestly it feels like it would take a fairly substantial edit to the text of the paper before I would feel as though I were comfortable with it.\n\nFeels like either the claims of the paper would have to be essentially all dropped to the level of a speculative discussion with some suggestive toy experiments.\n\nI also have some issues with audience, in the sense that I'm not sure who the target audience is for this paper.  I'm struggling a bit to see who would benefit from the observations of the paper.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: One of the reviewers deemed the contributions of modest interest and questioned the relevance of the work. Given that TMLR puts an emphasis on technical correctness rather than on subjective significance, I recommend acceptance. The detailed arguments provided by the authors in the global rebuttal clearly outlines why this work is worth publishing.
Decision: Accept (poster)