Deep Orthogonal Hypersphere Compression for Anomaly Detection
Authors: Yunhe Zhang
Yan Sun
Jinyu Cai
Jicong Fan
Venue: ICLR 2024 spotlight (2024)
Keywords: Anomaly Detection
Deep Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Many well-known and effective anomaly detection methods assume that a reasonable decision boundary has a hypersphere shape, which however is difficult to obtain in practice and is not sufficiently compact, especially when the data are in high-dimensional spaces. In this paper, we first propose a novel deep anomaly detection model that improves the original hypersphere learning through an orthogonal projection layer, which ensures that the training data distribution is consistent with the hypersphere hypothesis, thereby increasing the true positive rate and decreasing the false negative rate. Moreover, we propose a bi-hypersphere compression method to obtain a hyperspherical shell that yields a more compact decision region than a hyperball, which is demonstrated theoretically and numerically.  The proposed methods are not confined to common datasets such as image and tabular data, but are also extended to a more challenging but promising scenario, graph-level anomaly detection, which learns graph representation with maximum mutual information between the substructure and global structure features while exploring orthogonal single- or bi-hypersphere anomaly decision boundaries. The numerical and visualization results on benchmark datasets demonstrate the superiority of our methods in comparison to many baselines and state-of-the-art methods.

Published as a conference paper at ICLR 2024
DEEP ORTHOGONAL HYPERSPHERE COMPRESSION FOR ANOMALY DETECTION
Yunhe Zhang!. Yan Sun Jinyu Cai Jicong Fan
SScooo of Data Science. The Chinese University of Hong Kong, Shenzhen, China Shenzhen Research Institute of Big Data, Shenzhen, China
School of Computing, National University of Singapore, Singapore
*Institute of Data Science, National University of Singapore, Singapore yansunêcomp nus. edu. sg fanj camgicoon@cuhh edu. cn
zhangyhannie gma 1. com
iyyca i 9958 gma 1 com
ABSTRACT
Many well-known and effective anomaly detection methods assume that a rea- sonable decision boundary has a hypersphere shape, which however is difficult to obtain in practice and is not sufficiently compact, especially when the data are high-dimensional spaces. In this paper, we first propose a novel deep anomaly de- tection model that improves the original hypersphere learning through an orthog- onal projection layer, which ensures that the training data distribution is consistent with the hypersphere hypothesis, thereby increasing the true positive rate and de- creasing the false negative rate. Moreover, we propose bi-hypersphere compres- sion method to obtain a hyperspherical shell that yields a more compact decision region than hyperball, which is demonstrated theoretically and numerically. The proposed methods are not confined to common datasets such as image and tabular data, but are also extended to more challenging but promising scenario, graph- level anomaly detection, which learns graph representation with maximum mutual information between the substructure and global structure features while exploring orthogonal single- or bi-hypersphere anomaly decision boundaries. The numeri- cal and visualization results on benchmark datasets demonstrate the superiority of our methods in comparison to many baselines and state-of-th offth-of-hhe-rt methods.
INTRODUCTION
Anomaly detection plays crucial role in variety of applications, including fraud detection in fi-
nance, fault detection in chemical engineering (Fan & Wang. 2014), medical diagnosis, and the iden-
tification of sudden natural disasters (Aggarwal] 2017). Significant research has been conducted on
anomaly detection using both tabular and image data (Ruff et al., 2018 Fan & Chow 2020 Goyal al., 2020; Chen et 2022 Liznerski al. 2021 Sohn al.] [021] Liznerski et al. (022)) A
common setting is to train a model solely on normal data to distinguish unusual patterns from abnor-
mal ones, which is usually referred to as one-class classification (Schölkopf et al. 1999 Tax & Duin
2004 Ruff et al. 2018 Pang et al. 2021 /eelyy et al.) (2021). For example, the support vector data description (SVDD) proposed by (Tax Duin 2004) obtains spherically shaped boundary around dataset, where data points falling outside the hypersphere will be detected anomalous data. The deep SVDD proposed by (Ruff et al. 2018) trains neural network to transform the input data into space in which normal data are distributed in hyperspherical decision region. Regarding the con- cern that finite training normal data generating distribution may be incomplete or draw from many sets of categories, Kirchheim al. (2022) proposed supervised multi-class hypersphere anomaly detection method. Han et al. (2022) provided a review and comparison of many anomaly detection methods. Compared with common anomaly detection, there relatively little work on graph- data, despite the fact that graph anomaly detection has application scenarios in various problems, such identifying abnormal communities social networks, discriminating whether human- brain networks are healthy [Lanciano et al. 2020). or detecting unusual protein structures in biological
"Corresponding author.

---

Published as a conference paper at ICLR 2024
Peprrsrtation Midale
Trango Torp
Orihugeea Projector
Optrzzation (4)
Tatelerlbo
Laient
Standardized
Reproanaiin
Represctation
Graph Data
Itput
Optimizaiion
Figure I: Architecture of the proposed models (right top: DOHSC; right bottom: DO2HSC), Herein, 2-D visualizations show the trends of training data when applying two optimizations and 3-D visu- alizations illustrate the detection results obtained by them, respectively.
experiments. The target of graph-level anomaly detection is to explore a regular group pattern and distinguish the abnormal manifestations of the group. However, graph data inherently complex and rich in structural and relational information. This characteristic facilitates the learning of pow- erful graph-level representations with discriminative patterns many supervised tasks (c.g., graph classification) but brings many obstacles to unsupervised learning. Graph kernels (Kriege et al., 2020) are useful for both supervised and unsupervised graph learning problems. For graph-level anomaly detection, graph kernels can be combined with one-class SVM (Schölkopf et al. 1999)
or SVDD (Tax & Duin, 2004). This two-stage approach that cannot ensure that implicit fea- tures sufficiently expressive for learning normal data patterns. Recently, researchers proposed several end-to- cod graph-level anomaly detection methods (Ma et al. 2022 Zhao & Akoglu 2021 Qiu al. 2022). For example, Ma et al. (2022) proposed global and local knowledge distilla- tion method for graph-level anomaly detection. Zhao & Akoglu (202T) combined the deep SVDD objective function and graph isomorphism network to learn hypersphere of normal samples.
Although the hypersphere assumption is reasonable and practical, and has led to many successful algorithms (Tax & Duin 2004 Ruff al. 2018 2020 Kirchheim et al., 2022 Zhao & Akoglu 2021 for anomaly detection, still exhibits the following three limitations:
First, minimizing the sum of squares of the difference between each data point and the cen-
ter cannot guarantee that the learned decision boundary is standard hypersphere. Instead, one may obtain a hyperellipsoid (see Figure 2 or other shapes that are inconsistent with the assumption, which will lower the detection accuracy.
The second is that in high-dimensional space the normal data enclosed by hypersphere all far away from the center (see Figure 3 and Proposition with high probability. means that there is normal data around the center of the hypersphere; hence, the normality in the region not supported, whereas anomalous data can still fall into the region. It's related the soap-bubble phenomenon of high-dimensional statistics (Vershynin) 2018).
Last but not least, in nigh-dimensional space, one hypersphere not sufficiently compact. In other words, the distribution of normal data in the hypersphere extremely sparse be- cause of the high dimensionality and limited training data. A high sparsity increases the risk of detecting anomalous data as normal.
To address these issues, we propose two anomaly detection methods. The first one, Deep Orthogonal Hypersphere Contraction (DOHSC), utilizes an orthogonal projection layer to render the deci- sion region more hyperspherical and compact to reduce evaluation errors. The second one, Deep Orthogonal Bi- Bi-HHyersphere Compression (DO2HSC), aims to solve the problem of the soap-bubble phenomenon and incompactness. From 2-dimensional view, DO2HSC limits the decision area (of normal data) to an interval enclosed by two co-centered hyperspheres, and similarly learss the tathooonalityyprjjccoo. representation. Accordingly, a new detection metric is proposed for DO2HSC. The framework of the methods mentioned above is shown in Figure T In addition,
2

---

Published as a conference paper at ICLR 2024
graph-level extensions of DOHSC and DO2HSC are conducted to explore more challenging task, ie., graph-level anomaly detection. In summary, our contributions are three-fold.
First, we present a hypersphere contraction algorithm for anomaly detection tasks with an orthogonal projection layer promote training data distribution close to the standard hy- persphere, thus avoiding inconsistencies between assessment criteria and actual conditions.
Second, we propose the deep orthogonal bi-hypersphere compression model to construct decision region enclosed by two co-centered hyperspheres, which has theoretical sup- ports and solves the problem of hoap-bubble phenomenon and incompactness of the single- hypersphere assumption.
Finally, we extend our methods to graph-level anomaly detection and conduct abundant experiments to show the superiority of our methods over the state-of-the-art.
2 DEEP ORTHOGONAL HYPERSPHERE COMPRESSION
2.1 VANILLA MODEL
Denote a data matrix by X E Raxd with n instances and d features, we first construct an auto-encoder and utilize the latent representation Z J (X) to initialize a decision region's center according Deep SVDD (Ruff al., 2018). i.e, denotes the transpose of the i-th row of X and Jw module with parameters W = (W,, by With this center, we expect to optimize the learned representation of normal data to be distributed as close it as possible, so that the unexpected anomalous data falling out of this hypersphere would be detected. The Hyper- sphere Contraction optimization problem for anomaly detection is first formulated as follows:
- l (x:), where x; an L-layer representation learning
L Σ
1
min
Σ (x,) cll² + W n2
Wil. (I)
2
where the regularization is to reduce over-fitting.
2.2 ORTHOGONAL PROJECTION LAYER
Toy example of decision bound- aries with and without the orthogonal pro- jection layer. Blue circle: assumed deci- sion boundary; black ellipse: actual decision boundary; purple points: normal data; red points: abnormal data.
Figure 2:
Although the goal of Optimization I is learn a hypersphere as the decision boundary, we find that it usually yields a hyperellipsoid or even more irreg- ular shapes (please refer to Section I in the supple- mentary material). This phenomenon would lead to inaccuracies in the testing stage, because the evalu- ation was based on the hypersphere assumption. Figure 2 illustrates an intuitive example. In the left plot, the learned decision boundary (black ellipse) does not match the assumption (blue circle), which decreases the true-positive (TP) rate and increases the false-positive (FP) rate. Thus the de- TP1
tection precision, calculated as TP decreases compared to the right plot. The inconsistency
between the assumption and the actual solution stems from the following two points: 1) the learned features have different variances and the learned features are correlated. Clearly, these two issues cannot be avoided by solely solving Optimization (
To solve these issues, as shown in the right plot of Figure 2 we append an orthogonal projection
layer to the feature layer, i.e., the output of fone Note that we pursue orthogonal features of latent representation rather than computing the projection onto the column or row space of Z E R which is equivalent to performing Principal Component Analysis (PCA) (Wold et al. 1987) and us- ing standardized principal components. Our experiments also justify the necessity of this projection step and the standardization process, which will be discussed further Appendix K Specifically, the projection layer is formulated
Ż Proje ZW", subject to ŽTž L
(2)
3

---

Published as a conference paper at ICLR 2024
where θ {W' R } is the set of projection parameters, I denotes identity matrix, and k' is the projected dimension. To achieve 2 one may consider adding a regularization term 2 [ 7 I 2 with large enough a to the objective, which is not very effective and will lead to one more tuning hyperparameter. Instead, propose achieve (2) via singular value decomposition:
UAV = Z, W == VW
(3)
Assume that there are b samples in one batch, A diag(E1, ..... nal matrix with singular values and right-singular matrix of Z. respectively. It is noteworthy that VE := [vi,..., V denotes the first k' right singular vectors, and A == diag(pi, ... P..,)))). In each forward propagation epoch, the original weight parameter is substituted into a new matrix W in the subsequent loss computations.
P2, Pb) and V are the diago-
2.3 ANOMALY DETECTION
Attaching with an orthogonal projection layer, the improved initialization of the center is rewritten in the following form č Σ 2, which will be fixed until optimization is completed. The final objective function for anomaly detection tasks in a mini-batch would become
min- M,W.
Σ
& 2|
Σ |W]]
(4)
2
WEW
After the training stage, the decision boundary & will be fixed, which is calculated based on the - percentile of the training data distance distribution:
f = arg min P(D rr)
(5)
where D = {d follows a sampled distribution P, and d = - - -oll Accordingly, the anomalous of i-th instance is defined follows:
(6)
where s = (51, ..,............................. 8..) is evident that when the score is positive, the instance is identified as abnormal, and the opposite is considered normal.
The detailed procedures are summarized in Algorithm 1 (see Appendix A), which is termed as DOHSC. DOHSC is easy implement and can ensure that the actual decision boundary is close to a hypersphere. Our numerical results in Section will show the effectiveness.
DEEP ORTHOGONAL BI-HYPERSPHERE COMPRESSION
3.1 MOTIVATION AND THEBRETICAL ANALYSIS
As mentioned in the third paragraph of Section I hoap-bubble phenomenon and incompactness. They can be succinetly summarized as
the hypersphere assumption may encounter the
High-dimensional data enclosed by hypersphere are far from the center naturally, which means the normality within a wide range of distance is not supported.
In high-dimensional space, the data distribution within hypersphere is highly sparse, which leads to an incompact decision region and, hence, heightened risk of detecting abnormal data as normal.
In this section, we present a detailed analysis. Let the anomaly score be determined using ||z - cll where denotes the centroid. The original evaluation of anomaly detection compares the score with a threshold determined by certain quantile (e.g.. 0.95). Specifically, if ||z - ell 2 A. abnormal. This target promoted the location of most samples near the origin. However, empirical exploration has found that most samples are far away from their origin in figh--iimnsionall space. Taking Gaussian distributions example, the distributions would look like bubble which

---

Published as a conference paper at ICLR 2024
means the nigh-dimensional normal data may be more likely to locate in the interval region of bi- hypersphere instead of a simple hypersphere. /eershyii (2018) stated that the typical set, where data has information closest to the expected entropy of the population, of a Gaussian is the thin shell within a distance from the origin, just like the circumstances shown in Figure 3 The higher the dimensionality of the data, the more sampled instances are from the center. We also supplement the anomaly detection simulation of high-dimensional Gaussian data in Appendix [C] to show the significant meaning of bi bi-hypersphere learning. This is formally proven by the following proposition (derived from Lemma of (Laurent & Massart, 2000)):
Proposition 1. Suppose 21 Z2.'... Zn are from N(0, Ia) independently. Then, for any 2; and all t O, the following inequality holds.
P
2
Vd 2v 2 21-
The proposition shows that when the dimension is high, each z, is outside the hypersphere of radius r'= Vd - 2 di with a probability of at least - - When r' is closer to i (refer to equation normal data are more likely to be away from the center (see Figure
Note that, in anomaly detection, 2; (e.g., the learned latent representation) is not necessarily an isotropic Gaussian. However, we obtain the following result.
Let Z; = 2n i :.:=...... I....,N and let f R R be an n--ipschitz function such that s = f(z) are isotropic Gaussian N (c, }a). Let c be a predefined center of {z and suppose l|e - f(c)ll & Then for any z, and all t 2 0, the following inequality holds:
Proposition 2.
Escidee
Figure 3: Soap-bubble phenomenon showed by the histogram of distances from the cen- ter of 10 samples drawn from N (0, La). In high- high-dimensional space, almost all data are far from the center.
| - cll2211 21- (VK-2VEL -V-2 -5) 222211 21-
The proposition (proved in Appendix D indicates that most data (N') satisfy Iz cl r'
7 (VB- 2/kt-c) with a probability of approximately (1) - e C-(NNNN where r' is close to F. This means there is almost no normal training data within the range r'). i.e. normality within the range is not supported by the normal training data. An intuitive example is:
Example 1. Assume the hypersphere is centered at the origin. Consider a data point with all features very close or even equal to zero. This point is very different from the normal training data and should be abnormal data. However, according to the meiric 1 ē||, this point is still in the hypersphere and is finally detected as normal data.
Given the implications of Proposition we recognize that in ligh-dimensional spaces, traditional distance-to-can -oteterr based anomaly scores (equation 6) may lose their reliability due to the concentra- tion of measure phenomenon. Figure 4 a real example of abnormal data falling into a region close to the center of the hypersphere. In addition to the coap-bubble phenomenon, we claim that the data distribution a high-dimensional sphere very sparse when the number n the training data is limited. This means that when is not sufficiently large, there could be large empty holes or regions in which normality is not supported because of the randomness. treat data that fall into holes regions as normal data. Intuitively, for example, the distribution of random points in a 3-D sphere of radius r is much sparser than that in a 2-D circle of radius r. More formally, in hypersphere of radius in k-dimensional space, the expected number of data points n1'(k/2+1) where I is Euler's gamma function. When r not too small, per unit volume is @ = - es increases rapidly as & decreases. See below.
not sufficient to
Example 2. Suppose n = 1000, r= 5. Then @2 = 12.7, 05 2 0.06, and P10 < 0.0001.
We hope to construct a more compact decision region, one with a much larger @k. without changing the feature dimensions.
5

---

Published as a conference paper at ICLR 2024
62
(+) Training Data
(b) Testing Data
Figure 4: Illustration of inevitable flaws in DOHSC on both the training and testing data of COX2. Left: the l no-morm distribution of 4-dimensional distances learned from the real dataset; Right: the pseudo-layout in two-dimensional space sketched by reference to the empirical distribution.
3.2 ARCHITECTURE OF DO2HSC
To solve the issues discussed in the previous section, we propose improved approach, DO2HSC, which sets the decision boundary as an interval region between two co-centered hyperspheres. This can narrow the scope of the decision area induce normal data fill as much of the entire interval area as possible.
After the same representation learning stage, first utilize the DOHSC model for few epochs to initialize the large radius T'ax and small radius Tmin of the interval area according to the 1 percentile and of the sample distance distribution, respectively. The aforementioned descriptions can be mathematically denoted as follows:
Maxx arg min P(D S r) 2
Timi = argmin P(D & <r) 2
After fixing Tmax and Tmin. the objective function of DO2HSC is formulated as follows:
A Σ |W 2
(max (di, [mmx - min (ni,dd, (min + min b
(8)
WEW
This decision loss has the lowest bound 7'mx Tmin- In addition, the evaluation standard of the test data must also be changed based on this interval structure. Specifically, all instances located in the inner hypersphere and outside the outer hypersphere should be identified anomalous individuals; only those located in the interval area should he regarded as normal data. We reset a new score function award the positive samples beyond [Tmin: F'max] while punishing the negative samples within this range. Accordingly, the distinctive scores are calculated by
8; = (d: - Tmax) (d, - rmin),
where i (1....,n) n). In this manner, we can also effectively identify a sample's 's abnormality based on its score. In general, an improved deep anomaly detection algorithm changes the decision bound- ary and makes the normal area more compact. Furthermore, a new practical evaluation was proposed to adapt to the improved detection method. Finally, we summarize the detailed optimization proce- dures in Algorithm (see Appendix A).
The following proposition justifies the superiority of bi-hypersphere compression over single- hypersphere contraction from another perspective:
Proposition 3. Suppose the number of normal training data is the radius of the hypersphere given by DOHSC is Tmars and the radii of the hyperspheres given by DO2HSC are Tmax and Tmin respectively. Without loss of generality, assume that all the training data are included in the learned decision regions. The ratio between the support densities of the decision regions given by DO2HSC and DOHSC is 1-(rson/Ton)
In the proposition (proved in Appendix D density is defined as the number of normal data in unit volume. A higher density indicates a higher confidence in treating data point falling into the deci- sion region as normal data, or treating a data point falling outside the decision region as anomalous 1. the DO2HSC provides advantage of the DO2HSC over the DOHSC is more significant when & is smaller or Tiin is closer Tmax- Here are some examples.
data. Because
reliable decision region than the DOHSC. The

---

Published as a conference paper at ICLR 2024
Example 3. Suppose lo = 50. When Timin max 0.9, K 2 1.01. When T rminn/ / /mmx 0.99, K ~ 2.5.
Suppose & = 10. When Tmin/ T'mx 0.9, R. 25 1.5. When T'in/Tmax Tmax 0.99, K = 10.5.
3.3 GENERALIZATION TO GRAPH-LEVEL ANOMALY DETECTION
Given a set of graphs {Gi, Gr with N samples, the proposed model aims to learn k-dimensional representation and then set a soft boundary accordingly. In this paper, the Graph Isomorphism Network (GIN) (Xu et al. 2019) is employed obtain the graph representation in three stages: first, input the graph data and integrate neighbors of the current node (AGGREGATE); second, combine neighbor and current node features (CONCAT); and finally, all node information (READOUT) is integrated into one global representation. Mathematically. the i-th node features of 1-th layer and the global features of its affiliated j j-th graph are denoted by
Zip CONCAT((2) ( (G,) READOUT 24 1111)
(10)
where 2/ 1 and Zo (G,) RI**, To integrate the contained information and enhance the differentiation between node- and global-level representations, append additional fully connected layers denoted by the forms Mr (-) and Tv respectively, where and are the parameters of the added layers. So the integrated node-level and graph-level representations are
h == MT Ho.v (G,) To (Z,)(G)))
(11)
To better capture the local information, we utilize the batch optimization property of neural networks maximize the mutual information (MI) between local and global representations in each batch
G C G, which is defined by (Sun et al. 2020) as follows:
V r arg max T0,V, (ho Ho)x(G)).
(12)
4.v.T
Specifically, the mutual information estimator Ia,v.r follows the Sensen-Shannon MI estimator (Nowozin et al.] 2016) with positive-negative sampling method, as follows:
To, (ho,7. Ho. (G)) =
Σ
To.0r (G,,,()),HH,,
Ho,
GgI
EGEG
uEG,
(13)
=E GGEG
B( -h (x)) x Ho v (x)) (hū,r(x)) x H xx))()
G3
wEG;
where (2) log(1 + e²). For x as an input sample graph, we calculate the expected mutual information using its positive samples x and negative samples which are generated from the distribution across all graphs in a subset. Given that G (Va, Eg) and the node set VG = { LG| the positive and negative samples are divided in this manner: x X;j if U; G, otherwise, x* = O. Additionally, x produces the opposite result for each of the above conditions. Thus, a data-enclosing decision boundary is required for our anomaly detection task. Let Ho,v,e(G) = Proje (Ho,v (G)), the center of this decision boundary should be initialized through
1
Σ H. H...)
(14)
Collectively, the weight parameters of ④, V and T are 2 = + U U Y, and let R(Q)
=
Eqeo 1.. we formulate the objective function of the graph-level DOHSC
H..v.s(G.) - - - x Σ L herr He H. + * R()), (15)
min 0,0,V,T
G
GEG
where |G| denotes the number of graphs in batch G and À is trade-off factor, the third term a network weight decay regularizer with the hyperparameter μ.. Correspondingly, the objective function of graph-level DO2HSC
H ((Q)). Σ v. (De.r. H He..H.H..o + Ge
min 9,4.
(max (di, Fmax] min (di, (min))
G
(16)

---

Published as a conference paper at ICLR 2024
4 NUMERICAL RESULTS
4.1 EXPERIMENTS ON IMAGE DATA
Datasets: Two image datasets (Fashion-MNIST) CIFAR-I0) are chosen to conduct this experiment. Please refer to the detailed statistic descriptions in Appendix F
Baselines: We followed the settings in (Ruff et al. 2018) and utilized the Area Under Operating Characteris-
Table 1: Average AUCs (%) in one-class anomaly detection on CIFAR-IO. denotes we run the official released code to
tic Curve (AUC) of several state-
obtain the results, and the top two results are marked in bold.
of-the-art anomaly detection algo-
Class
Airplane
Mobile
Dog
Frog
Horee
Ship
rithms, including Deep SVDD (Ruff)
Deep SVDD
65.9
65.7
67.7
67.3
ict al. 2018). OCGAN (Perera et al.
72.3
57.5
72.8
66.5
08.6
67.5
79.3
2019
HRN-L2 and HRN (Hu al.
HRN-L2
48.2
57.4
73.3
61.0
74.1
55.5
71.6
HRN
77.3
69.9
60.6
71.5
67.
77.4
64.9
77.3
2020
PLAD (Cai & Fan 2022)
PLAD
$2.5
30.8
68.8
68.2
71.6
71.2
76.4
73.5
30.5
$0.5
and DROCC (Goyal al.
2020
DOHSS
(0.0)
$1.0
(0.0)
(1.91
(1.8)
(0.0)
73.1
(2.1)
72.4
83.1
(0.0)
(0.4)
(0.7)
(0.7)
All SOTAS' results are given accord- ing their officially reported results or are reproduced by official codes.
DO2HSC
(0.2)
$1.3
(0.3)
82.7
(0.4)
71.3
71.2
(1.3)
(2.3)
72.9
(0.2)
72.8
10.6]
83.0
(0.4)
75.5
$4.4
(0.5)
(0.91
82.0
Considering that there not much room for performance improvement on Fashion-MNIST, we only reproduced the results of recent or most relative algorithms, which contains Deep SVDD (Ruff et al., 2018), and DROCC (Goyal et al. 2020p. The network architecture of Deep SVDD set the same as ours for faireess.
The experimental results are listed in Table On CIFAR-10, both DOHSC and DO2HSC surpassed SOTAs, especially for Dog and Frog. Second, DO2HSC obtained better results compared with DOHSC, which further verifies the effectiveness of bi- perpyeersphere anomaly detection and fully demonstrates its applicability image data. It is also worth mentioning that Deep SVDD plays an important bascline role relative to DOHSC, and DOHSC outperforms it by a large margin in all classes. This illustrates the significant meaning of the proposed orthogonal projection method is constructive. The result of MaSSi is in Appendix G
Results:
4.2 EXPERIMENTS ON TABULAR DATA
Datasets:
Here, we use two tabular datasets (Thyroid, Arrhythmia). and we followed the data split settings in Zong et al.) (2018).
Table 2: Average Fi-scores with the standard devi- ation in one-class anomaly detection two tabular datasets. The best two results are marked in bold.
Results:
The Fl Fs-cccess of our methods and
Thyroid
Arrhtthmia
six baselines are reported in Table signif- icant margin was observed between the base- lines and ours, especially the results of Thy- roid. Despite the challenge posed by the small sample size of the Arrhythmia data, DO2HSC still outperforms PLAD by a mar- gin of 3%. Similarly, the orthogonal projec- tion of DOHSC successfully standardized the results of Deep SVDD.
OCSVM (Schiikkop et al. 1999
0.56 4 0.01
0.64 :: 0.01
Deep SVDD [Rui et ai. 2018
0.73 + 0.00
0.54 1: 0.01
LOF (Breutng el al. 2000
0.54
0.51 0.01
GOAD Bergman Hoshen 2020
0.75
0.52 :: 0.02
DROCE Luyya sl. 2020
0.78 4: 0.03
0.69 0.02
PLAD
0.01
0.71 0.02
DOHSC DO2HSC
0.92
0.70 + 0.03
0.98 + 0.59
0.74 0.02
4.3 EXPERIMENTS ON GRAPH DATA
Datasets:
We further evaluate our models on six real-world graph dataset (COLLAB, COX2, ER_MD, MUTAG, DD and IMDB-Binary). Our experiments followed the standard one-class set- tings and data-split method in a previous work (Zhao & Akoglu 2021 Qiu et al. 2022).
Baselines: We compare our methods with the following methods, including four graph kernels
combined with OCSVM and four state-of-the-ar baselines: RW (Gärtner et al. 2003 Kashima
et al. 2003), SP (Borgwardt & Kriegel, 2005), WL (Shervashidze et al. 2011 and NH (Hido &

---

Published as a conference paper at ICLR 2024
06
(a) DOUSC
(b) DO2ISSC
Training Result
(b) Testing Result
Figure 5: Distance Histograms on ER MD.
Figure 6: 3-D plots of DO2HSC on MUTAG.
Table 3: Average AUCs with standard deviation (10 trials) of different graph-level anomaly detection algorithms. 'DSVDD' stands for 'Deep SVDD', We assess models by regarding every data class as normal data, respectively. The best two results are highlighted bold and '-' means out of memory.
COLLAB
MUTAG
ER.MD
SP
0.000
0,4092 0,000
ALCOCSVM
0.5122 0.0000
0.8054
0.996
0.0000
0.8509
2060
0.000
0,4571
0.0000
0.3262
NH-0CSVM
0.5976 + 0.0000
0.8054
0.0000
0.614 + 0.0000
0.799
0.0274
0.16
0.002
0.5155 + 0.0000
0.3648
0.0000
RW+OCSVM
0.8698 1 0.0000
0.1501
0.0000
0.4820 1 0.0000
0.3484 1 0,0000
OCGIN
0.4217 + 0.0666
0.7565
0.3035
0.1906
0.0857
0,8491
0.0444
0.7466
0.0168
0.5645 + 0.0323
0.4358
0.0338
int:CrnphDDVDDD
0.5663 1 0.0597
0,7926 0.0886
0,4062
0.0978
0.8805
0.0448
0.6166
0.2055
0,5312 1 0.1565
0,5082
0,0704
0.43884 0.0003
0.8004
0.3052
0.2641
0.5781 0.9090
0.7154
0.6504
0,4029
0.0317
DOHSC
0.9185
0.0250
118822
0.0432
0.6630
0.0300
DO2ESC
0.9390 0.0025
0,9836
0.0115
0.8835
0.0118
0,9089
0.0609
0.5250 0.0790
0.6867
0.0226
0.7351 0.0159
Kashima 2009), OCGIN (Zhao & Akoglu 2021 ) infoGraph+Deep SVDD (Sun et al.
2020 Ruff
al.
2018). GLoccllD (Ma et al. 2022) and OCGTL (Qiu et al. 2022).
Results: Table 3 shows the comparable results of graph-level anomaly detection. 1) The proposed methods achieved the best AUC values compared to the other algorithms on all datasets. Both out- perform the other state-of-the-art baselines. 2) DO2HSC is obviously more effective than DOHSC. especially since we observed that there exists large improvement (exceeding 20%) in Class of ER_MD between DOHSC and DO2HSC. A distance distribution visualization is provided show their differences in Figure 5 Owing to length limitations, please refer to Appendix HH for the re- maining results. 3) The anomaly detection visualization results of DO2HSC displayed in Figure 6 also demonstrate excellent performance. We drew them by setting the projection dimension to 3, and please refer to Appendix[[ for the results of different perspectives.
4.4 MORE RESULTS AND ANALYSIS
We provide the time and space complexity analysis in Appendix B Also, the ablation study (including orthogonal projection, mutual information maximization, etc.), parameter sensitivity (e.g., different percentile settings), robustness analysis, and more visualization results are shown in Appendices J [ respectively.
5 CONCLUSION
This paper proposes two novel end-to-end AD methods, DOHSC and DO2HSC, that mitigate the possible shortcomings of hypersphere boundary learning by applying an orthogonal projection for global representation. Furthermore, DO2HSC projects normal data between the interval areas of two co-centered hyperspheres to significantly alleviate the soap-bubble issue and the incompactness of single hypersphere. We also extended DOHSC and DO2HSC to graph-level anomaly detection, which combines the effectiveness of mutual information between the node level and global features to learn graph representation and the power of hypersphere compression. The comprehensive exper- imental results strongly demonstrate the superiority of the DOHSC and DO2HSC on multifarious datasets. One limitation of this work is that did not consider cases in which the training data consisted of multiple classes of normal data, which is beyond the scope of this study. Our source code is available at atthtps: //g thub. com http://wwwwncc3333//oo------0000

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work supported by the National Natural Science Foundation of China under Grant No. 62376236, the General Program SYY1202103241020000000 the research funding T00120210002 of Shenzhen Research Institute of Big Data, and the funding UDF01001770 of The Chinese University of Hong Kong, Shenzhen.
of Shenzhen Fundamental Research,
REFERENCES
Charu C Aggarwal. An introduction to outlier analysis. 2017.
Liron Bergman and Yedid Hoshen. Classification- based anomaly detection for general data. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Karsten M Borgwardt and Hans-Peter Kriegel. Shortest- path kernels on graphs. In Proceedings of the Fifth IEEE International Conference on Data Mining, pp. 8 ppp 2005.
Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. LOF: identifying density-based local outliers. In Proceedings of the ACM SIGMOD International Conference Management of Data, pp. 93-104, 2000.
Jinyu Cai and Jicong Fan. Perturbation learning based anomaly detection. In Advances in Neural Information Processing Systems, 2022.
Yuanhong Chen, Yu Tian, Guansong Pang, and Gustavo Carneiro. Deep one-class classification via interpolated gaussian descriptor. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 383-392, 2022.
Jicong Fan and Tommy W. Chow. Exactly robust kernel principal component analysis. JEEE Transactions Neural Networks and Leurning Systems, 31(3):749-761, 2020.
Jicong Fan and Youqing Wang. Fault detection and diagnosis of non-linear non gaussiasssn dynamic processes using kernel dynamic independent component analysis. Information Sciences, 259: 369-379, 2014.
Thomas Gärtner, Peter Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. Learning Theory and Kernel Machines, pp. 129-143. 2003.
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. DROCC: deep robust one-class classification. In Proceedings of the 37th International Con- ference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 3711-3721, 2020.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs.
Advances in neural information processing systems, 30, 2017.
Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. Advances in Neural Information Processing Systems, 35:32142-32159, 2022.
Shohei Hido and Hisashi Kashima. A linear-time graph kernel. In Ninth IEEE International Con- ference on Data Mining, pp. 179-188, 2009.
Wenpeng Hu, Mengyu Wang, Qi Qin, Jinwen Ma, and Bing Liu. Hm:: A holistic approach one class learning. Advances in Newral Information Processing Systems, 33:19111-19124, 2020.
Hisashi Kashima, Koji Tsuda, and Akihiro Inokuchi. Marginalized kemels between labeled graphs. In Proceedings of the 20th International Conference on Machine Learning, pp. 321-328, 2003.
Konstantin Kirchheim, Marco Filax, and Frank Ortmeier. Multi-class hypersphere anomaly detec- tion. In Proceedings of the 26th International Conference on Pattern Recognition, pp. 2636-2642. IEEE, 2022.
10

---

Published as a conference paper at ICLR 2024
Nils M Kriege, Fredrik D Johansson, and Christopher Morris, A survey graph kernels. Applied Network Science, 5(1):1-42, 2020.
Tommaso Lanciano, Francesco Bonchi, and A. Gionis. Explainable classification of brain networks via contrast subgraphs. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
Beatrice Laurent and Pascal Massart.
Adaptive estimation of a quadratic functional by model selec- tion. Annals of Statistics, pp. 1302-1338, 2000.
Philipp Liznerski, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus- Robert Müller. Explainable deep one-class classification. Proceedings of the 9th International Conference on Learning Representutions, 2021.
Rongrong Ma, Guansong Pang, Ling Chen, and Anton van den Hengel. Deep graph-level anomaly detection by glocal knowledge distillation. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. pp. 704-7122 2022.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural sam- plers using variational divergence minimization. In Advances in Neural Information Processing Systems, volume 2016.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Computing Surveys (CSUR), 54(2):1-38, 2021.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2019.
Chen Qiu, Marius Kloft, Stephan Mandt, and Maja Rudolph. Raising the bar in graph-level anomaly detection. In Lue De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, pp. 2196-2203, 2022.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International Conference on Machine Learning, pp. 4393-4402, 2018.
Lukas Ruff, Robert A. Vandermeulen, Nico Gömiizz, Alexander Binder, Emmanuel Müller, Klaus- Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. In International Conference on Learning Representations, 2020.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe-Taylor, and John Platt. Support vector method for novelty detection. Advances in neural information processing systems, 12, 1999.
Bernhard Schölkopf, Robert C. Williamson, Alexander J. Smola, John Shawe-Taylon Taylor, and John C. Platt. Support vector method for novelty detection. In Advances in Neural Information Processing Systems, pp. 582-588, 1999.
Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert Williamson. Estimating the support of high-dimensional distribution. Neural Computation, 13(7):1443- 1471, 2001.
Naeem Seliya, Azadeh Abdollah Zadeh, and Taghi M Khoshgoftaar. A literature review on one-class classification and its potential applications in big data. Journal of Big Data, 8(1):1-31, 2021.
Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borg- wardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9), 201
11

---

Published as a conference paper at ICLR 2024
Giannis Siglidis, Giannis Nikolentzos, Stratis Limnios, Christos Giatsidis, Konstantinos Skianis, and Michalis Vazirgiannis. Grakel: A graph kernel library in Python. Journal of Machine Learning Research, 21(54):1-5, 2020.
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evalu- ating representations for deep one-class classification. In Proceedings of the 9th International Conference on Learning Representations, 2021.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. In Proceed- ings of the 8th International Conference on Learning Representations, 2020.
Ziheng Sun, Chris Ding, and Jicong Fan. Lovász principle for unsupervised graph representation learning. Thirty- Conference on Neural Information Processing Systems, 2023.
David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54:45-66, 2004.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Max Welling and Thomas N Kipf. Semi-supervised classification with graph convolutional net- works. In Proceedings of the 5th International Conference on Learning Representations, 2016.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3))37-55 1987.
Zhihao Wu, Zhao Zhang, and Jicong Fan. Graph convolutional kernel machine versus graph con- volutional networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful are graph neural networks? In Proceedings of the 7th International Conference on Learning Representations, 2019.
Lingxiao Zhao and Leman Akoglu. On using classification datasets to evaluate graph outlier detec- tion: Peculiar observations and new insights. Big Data, 2021.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian I Lumezanu, Dae-ki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Pro- ceedings of the 6th International Conference on Learning Representations, 2018.
A SUPPLEMENTED ALGORITHM PROCEDURES
Here, we present the detailed procedures for DOHSC and DO2HSC in Algorithms 1 and respec- tively. It begins with a representation learning module and promotes the training data to approximate the center of hypersphere while adding an orthogonal projection layer. In addition, DO2HSC recapped in Algorithm 2 and begins with the same representation learning. In contrast. DOHSC utilizes few epochs to initialize the decision boundaries, after which improved optimization is applied. A graph-level extension is presented in Algorithm 3 The main difference is that graph rep- resentation learning with maximization of the mutual information constraint applied to substitute the common representation learning module. Similarly, the graph-level DO2HSC is the combination of the representation learning part in the graph-level DOHSC and the anomaly detection part in the common DO2HSC.
B TIME AND SPACE COMPLEXITY
The models of DOHSC and DO2HSC can be trained by mini-batch optimization. Suppose the batch size is b, the maximum width of the hidden layers of the L-layer neural network is Wmaxs and the dimension of the input data is d, then the time complexities of the proposed methods are
12

---

Published as a conference paper at ICLR 2024
Algorithm Deep Orthogonal Hypersphere Contraction (DOHSC)
Input:
The input data X E Rexd, dimensions of the latent representation and orthogonal pro- jection layer k', trade-off parameter À and the coefficient of regularization term μa, pretraining epoch T. learning rate
Output: The anomaly detection scores s.
1: Initialize the auto-encoder network parameters W = (Wi, bi and the orthogonal projection layer parameter O;
2: for t 7 do
3: for each batch do
4:
Obtain the latent representation Z = f (X);
Pretraining Stage
5: Update the orthogonal parameter θ of orthogonal projection layer by Eq. (3);
6: Project the latent representation via Eq. (2);
7: Calculate reconstruction loss via A Σ llw (Proje ( (x:))) xil1²:
8: prok-proe the network, update W and θ respectively;
9: end for
end for
11: Initialize the center of hypersphere by 1 E Jw (x;);
repeat
13: for each batch do
14: Calculate anomaly detection loss via Optimization
D Training Stage
15: Repeat steps 6
16:
Back-propagate the encoder network and update (W) 1= and θ, respectively; 17: end for
18: until convergence
19: Compute decision boundary r by Eq. 55;
20: Calculate the anomaly detection scores through Eq. (6
21: return The anomaly detection scores s.
most (bdu LT), where T is the total number of iterations. The space complexities are at most O(bd dwmaax + (L D)w We see that the complexities are linear with the number of samples, which means the proposed methods are scalable to large datasets. Particularly, for high-dimensional data (very large d), we can use small Wmax to improve the efficiency.
C RELATED PROOF OF BI-HYPERSPHERE LEARNING MOTIVATION
The traditional idea of detecting outliers is to inspect the distribution tails under the ideal assumption that the normal data are Gaussian. Following this assumption, one may argue that an anomalous sample can be distinguished by its large Euclidean distance from the data center (&2 norm ||z cll- where с denotes the centroid). Accordingly, the abnormal dataset is {z |z cll > r} for a decision boundary bubble which means the normal data are more likely to be located in a bi-hypersphere (Vershynin, 2018), such (z Tiin < |zz ccl Tmax): ). To better understand this counterintuitive behavior, generate normal samples X ~ (O0, Ia), where dl is the data dimension in {1, 10, 50, 100, 200, 500). As shown in Figure of Section 2.2.1, it indicates that only the univariate Gaussian has near-zero mode, whereas other eigh-dimensional Gaussian distributions leave many off-center spaces in the blank. The soap-bubble problem high- -ith-imenniioall distributions is well demonstrated in Table the higher the dimension, the greater the quantity of data further away from the center, especially for a 0.01 -qaantieee distance. Thus, we cannot make the sanguine assumption that all of the normal data are located within the radius of hypersphere (i.e., {z || cl r)). Using Lemma of (Laurent Massart 2000), we can prove that Proposition 1, which matches the values in Table that when the dimension larger, normal data more likely to lie away from the center.
However, in high dimensional space, Gaussian distributions look like soap-
We also simulated a possible case of outlier detection, in which data were all sampled from a 16- dimensional Gaussian with orthogonal covariance: 10,000 normal samples follow (0, I), the first group of 1,000 outliers is from ((1 11 I), the second group of 500 outliers are from N (P22, I),
13

---

Published as a conference paper at ICLR 2024
Algorithm 2
Deep Orthogonal Bi-Hyperspherre Compression (DO2HSC)
Input:
The input data X E Raxd dimensions of the latent representation and orthogonal pro- jection layer k', trade-off parameter À and the coefficient of regularization term μ., pretraining epoch 71, iterations of initializing decision boundaries T2, learning rate
Output: The anomaly detection scores s.
Initialize the auto-encoder network parameters W (Wi, bi) and the orthogonal projection layer parameter O;
2: for t Ti do for each batch do
4: Repeat steps 48 of DOHSC; end for
Pretraining Stage
6: end for
Update the orthogonal parameter of orthogonal projection layer by Eq.
8: Obtain the global orthogonal latent representation by Eq. (2)
Initialize the center of hypersphere by Ei=1 for To do
cenc
W (x,
Repeat steps end for
13 17 of DOHSC;
Pretraining Stage
Compute decision boundary of DOHSC by Eq. 5
14:
Initialize decision boundaries Tmax and T'mn via Eq.
repeat
16:
for each batch do
Obtain the latent representation Z I (X);
Trrining Stage
18:
Update the orthogonal parameter O of orthogonal projection layer by Eq. 33;
Project the latent representation via Eq. ( Calculate the improved total loss via Optimization
20:
8
4 Back-propagate the network, update {W} and O, respectively; end for
22:
until convergence
24:
Calculate the anomaly detection scores through Eq. return The anomaly detection scores s.
9
Count
300
Anomalous
100
10
32
14
Euclidean norm from the center
Figure Histogram distances (Euclidean norm) from the center of normal samples under 16-dimensional Gaussian distributions NNO,,), I). Three groups of anomalous data are also 16- dimensional and respectively sampled from A (p11 i I), (H,,, and A (p2s, 5I), where the pop- ulation means μ1, H2, Ms are randomized within [0. for each dimension.
and the last group of 2,000 outliers from N (us, 5I). Figure 7 shows that abnormal data from other distributions (group-1 outliers) could fall a small distance away from the center the normal samples.
14

---

Published as a conference paper at ICLR 2024
Algorithm 3 Graph-Level Deep Orthogonal Hypersphere Contraction
Input:
The input graph set G, dimensions of GIN hidden layers k and orthogonal projection layer k, trade-off parameter À and the coefficient of regularization term pretraining epoch T, learning rate 7).
Output: The anomaly detection scores
Initialize the network parameters 4, V, and the orthogonal projection layer parameter θ
for ( + T do
3:
for each batch G do
Obtain the global graph representation H& (G);
Update the orthogonal parameter O of orthogonal projection layer by Eq. 3
6:
Project the global graph representation via H (G) = Proje (G)) Calculate v.r herr. Ho, (G) via Eq. (13
Back-propagate GIN, update @ V, O and Y, respectively; 9: end for end for
Initialize the center of hypersphere by Eq. repeat
14
for each batch G do
Repeat steps 6
15:
Calculate total loss via Optimization 15
Back-propagate GIN and update 4 V. T and θ, respectively; end for
18:
until convergence
Compute decision boundary by Eq. 55:
Calculate the anomaly detection scores through Eq. 66: return The anomaly detection scores
21:
Table 4: Offcenter distance under multivariate Gaussian at different dimensions and quantiles.
Quantile (correspond mmin)
dim=1
dim=10
dim=50
dim=100
dim=200
dim=500
0.01
0.0127
1.5957
5.5035
8.3817
12.5117
20.6978
0.25
0.3115
2.5829
6,5380
9.4908
13.6247
21.8542
0.50
0.6671
3,0504
7.0141
9.9662
14.1054
22.3337
0.75
1.1471
3,5399
7.5032
10.4386
14.5949
22,8200
0.99
2.5921
4.8265
8.7723
11.6049
15.7913
24.0245
D PROOF FOR PROPOSITION
Proof. Since f makes obey (C, I,), according to Proposition 1, we have
|s al 2 Vx 2v
2 1 1t
Since f is -ipiicchh,,,, we have
Is f(c)l [[f(z) f(c)ll S 7/|z - cll-
It follows that
||z - cll 29 -|s- f(c)
21 ([|s - 1 - éc - f(c)l)
21 (Iss el ce)
Now we have
- +)] 2 21-6"
P
z c 2 -1
11-2VVkt
This finished the proof.
15

---

Published as a conference paper at ICLR 2024
E PROOF FOR PROPOSITION 3
Proof. The volume of a hyperball of radius r in k-dimension space Vi((r)
x//2
where
r( +1)
Euler's gamma function. Then, volume of the hypersphere given by DOHSC V.ux x²/² r and the volume of the smaller hypersphere given by DO2HSC is Vinin r r(#+1)
6/2 r
+1
min"
Then, the volume of the decision region given by DO2HSC is Vmax Vinn- The density of the decision region is defined as the number of normal data in unit volume. Therefore, the ratio between the densities of DO2HSC and DOHSC can be computed as
m/(Vmmax minn)
(17)
n n/Vmax
xk/2
r
xk/2
E
r +1
min
Zinin
This finished the proof.
In the case of Tmin = Timxx the volume of DO2HSC is close to an infinitely thin shell, essentially transforming into the surface of hypersphere. In this scenario, the data density of DO2HSC sig- nificantly higher compared with that of DOHSC. However, it is important to note that this situation quite rare, particularly in high-dimensional space.
F EXPERIMENT CONFIGURATION
In this section, experimental settings are presented for reproduction. First, each graph dataset was divided into two parts: the training and testing sets. We randomly sampled 80 percent the normal graph as the training set and the remaining normal graph, together with the randomly sampled ab- normal data in a one-to-one ratio to form the testing set. Regarding the image and tabular datasets, the data splits are already provided in the paper. The detailed statistical information of all tested datasets is given in Tables and
Table 5: Description for non-graph datasets.
Dataset Name
Type
Instances
# Dimension
Thyroid Arrhythmia Fashion-MNISS -NIIST CIFAR-10
Tabular
3772
Tabular
452
274
Image
70000
28 x8
Image
60000
32 x 32 x
Table 6: Description for six graph datasets.
Avg. Nodes
Avg. Edges
Datasets
# Graphs
Classes
# Graph Labels
COLLAB
5000
74.49
2457.78
3
2600 775 / 1625
COX2
467
42.43
44.54
2
365 102
ER_MD
446
21.33
234.85
2
265 181
MUTAG
188
17.93
19.79
2
63 / 125
DD
1178
284.32
715.66
2
691 487
IMDB-Binary
1000
19.77
96.53
2
500 / 500
In the image and tabular experiments, our backbone was consistent with the Deep SVDD, and the preprocessing conformed to the same splits as DROCC. All results originated from the corresponding papers
reproduced according to the official code. Regarding DOHSC model, we set 10 epuchs the pretraining stage to initialize the center of the deci- sion boundary and then train the model in 200 epochs. The percentile v of was selected from
16

---

Published as a conference paper at ICLR 2024
{0.001, 0.003, 0.005, 0.008, 0.01, 0.03, 0.3. The improved method DO2HSC also sets 10- epoch pretraining stage and trains DOHSC for 50 epochs to initialize a suitable center and decision boundaries max and Tmin- where the percentile v of max is the same DOHSC. The main training epoch was set to 200.
In the graph experiment, (Schölkopf et al.) 2001), to compare graph-kernel baselines and used 10-fold cross-validation to make a fair comparison. All graph kernels extract kernel matrix via GraKel (Siglidis et al. 2020) and apply the OCSVM in scikit-learn (Pedregosa et al. 201T). Specifically, we selected Floyd Warshall as the SP kernel's algorithm and set lambda as 0.01 for the RW kernel. The WL kernel algorithm is sensitive to the number of iterations; therefore, we tested four different iterations {2, 5, 8, 10} and reported the best result for each experiment. The outputs were normalized for all the graph kernels. For infoGraph+DDege SVDD, the first stage runs for 20 epochs, and the second stage pretrains for 50 epochs and trains for 100 epochs. In OCGIN, GLocalKD, and OCGTL, the default or reported parameter settings were adopted to reproduce the experimental results.
adopted the classical AD method, One-Class SVM (OCSVM)
Ostectrr fundton oatur
Oogeciin sovetion vagua
ORNecti Lundtton Valle
Number
19
66
Number
DOHSC
(b) DO2HSC
Figure Convergence curves of the proposed models on the MUTAG dataset.
For the graph-level DOHSC, we first set one epoch in the pre-training stage to initialize the center of the decision boundary and then train the model in 500 epochs. The convergence curves are shown in Figure to indicate that the final optimized results were adopted. The improved method DO2HSC is also set as I-epoch pre-training stage and trains DOHSC for five epochs, where the percentile v of Fmax is selected as 0.01. After initialization, the model was trained for 500 epochs. For both proposed approaches, the trade-off factor X set to 10 to ensure decision loss as the main optimization objective. The dimensions of the GIN hidden and orthogonal projection layers fixed at 16 and 8, respectively. For the backbone network, 4-layer GIN and 3-layer fully connected neural network were adopted.
Finally, the averages and standard deviations of the Area Under the ROC curve (AUC) and F1-score were used to support the comparable experiments by repeating each algorithm ten times. A higher metric value indicates better performance.
G SUPPLEMENTED RESULTS ON FASHION-MNIST DATASET
The complete experimental results for the Fashion-MNIST image dataset are given in Table 7 A detailed standard deviation can demonstrate fluctuations performance. The proposed methods are relatively stable, especially for DOHSC.
H SUPPLEMENTARY RESULTS OF GRAPH-LEVEL ANOMALY DETECTION
Here, we give the results of retained 3 graph datasets (COX2, DD, and IMDB-Binary) for graph- level extension in 8 The proposed two models are superior on all datasets and behave much more effectively compared with other SOTAs, which also supports our motivations for graph-level anomaly detections.
I SUPPLEMENTED VISUALIZATION
This section presents the related supplemental visualization results of the anomaly detection task. Figure g shows the distance distributions of the two-stage method, proposed model DOHSC, and
17

---

Published as a conference paper at ICLR 2024
Table 7: Average AUCs in one-class anomaly detection Fashion-MNIST. (The best results are marked in bold.)
Deep SVDD
DROCC
Normal Class
DOHSC
DO2HSC
[Ruff et al. 2018
Goyya et al. 2020
T-shirt
0.8263 = 0.0342
0.8931 + 0.0072
0.9153 + 0.0082
0.9196 = 0.0064
Trouser
0.9632 = 0.0072
0.9835 0.0054
0.9817 = 0.0060
0.9839 0.0020
Pullover
0.7885 = 0.0398
0.8656 + 0.0140
0.8007 + 0.0204
0.8768 = 0.0122
Dress
0.8607 = 0.0124
0.8776 + 0.0269
0.9178 0.0230
0.9171 = 0.0084
Coat
0.8417 = 0.0366
0,8453 = 0.0143
0.8805 + 0.0258
0.9038 = 0.0140
Sandal
0.8902 = 0.0281
0.9336 0.0123
0.8932 + 0.0287
0.9308 0.0070
Shirt
0.7507 = 0.0158
0.7789 + 0.0188
0.8177 + 0.0124
0.8022 = 0.0045
Sneaker
0.9676 = 0.0062
0.9624 + 0.0059
0.9678 0.0050
0.9677 0.0075
Bag
0.9039 = 0.0355
0.7797 + 0.0749
0.9122 + 0.0258
0.9090 = 0.0105
Ankle Boot
0.9488 0.0207
0.9589 + 0.0207
0.9756 + 0.0127
0.9785 0.0038
Table 8: Average AUCs with standard deviation (10 trials) of different graph-level anomaly detection algorithms. 'DSVDD' stands for DDepp SVDD'. We assess models by regarding every data class as normal data, respectively. The best two results are highlighted in bold and means out of memory.
COX2
DD
IMDR-Bine Binary
SP+OCSVM
0.5408 0.0000
D.5760
0.0000
0.6856
0.0000
0,4474
0,0000
0.4592
0.0000
0.4716
0.0000
WL+OCSVM
0.5990 0.0000
0.5057
0.0000
0.7397
0.0000
0.494
+ 0.0000
0.5157 0.0000
0.4607
0,0000
NH-OCSVM
0.4841 0.0000
0.4717 0.0000 0.0000
0.7424
0.0000
0.3684 + 0.0000
0.5521
0.0000
0.4652
0.0000
RW+OCSVM
0.5243 + 0.0000
0.0000 0.6553
0.0000 0.4951
0.0000 0.531
OCGIN
0.5964 0.0578
0.5683
0.0768
0.6659
0.0444
0,6003 + 0.0534
0.4571 = 0.1879
0.3736
0.9816
infoGraph-DSVDD
0.4825 0.0624
0,5029
0,0700
0,3942
0.0436
0.6484 = 0.0236
0,6353 = 0.0277
0,5836
0.0995
KDal
0.3861 0.0131
0.0383 0.3143
0.1952
0.0000
0,5383
0.4812
0.0101
OCGTL
0.5541 1 0.0320
0.4862
0.0224
0.6990
0.0260
0.6767
0.0280
0.6510
0.0180
0.6412
0.0127
DOUIC
0.6263
0.6805
0.0168
0.0600
0.7705
0.0045
DOZHSC
0.6329 1 0.0292
0.6923
0.0433
0.7320 = 0.0194
0.7651 + 0.0317
0.7547
0.0390
0.7737
0.0503
Cile
82
14
Diatance
332
Dirance
(a) infooOohhDDapppppp SVDD
(b) DOHSC
(c) DOZHSC
Figure 9: Distance distributions were obtained by infoGraph+Deep SVDD, the proposed model, and the improved proposed model on COX2. The first row represents the distance distribution of the training samples in relation to the decision boundary. The last row indicates the distance distribution of the test data with respect to the decision boundary.
18

---

Published as a conference paper at ICLR 2024
improved DO2HSC. Here, distance is defined the distance between each sample and the center of the decision hypersphere. Distance distribution denotes the sample proportion at this distance interval relative the corresponding total samples. It can be intuitively observed that most of the distances of the instances were close to the decision boundary because of the fixed learned representation. As mentioned earlier, the jointly trained algorithm mitigated this situation, and the obtained representation caused many instances to have smaller distances from the center of the sphere. Moreover, mentioned in Section 2.2, anomalous data may occur in regions with less training data, particularly in the region close to the center, which is also confirmed by (a) and (b) of Figure 2 In contrast, DO2HSC effectively shrinks the decision area, and we find that the number of outliers is obviously reduced owing to more compact distribution of the training data.
The 3D visualization results of the training and testing stages are also presented the difference be- tween them in Figures TO and
To further support the aforementioned statements, as shown in Figure 12 the anomalous samples are located in the decision region and are closer to the center than other normal samples. On the contrary, the result of DO2HSC effectively prevents this phenomenon.
(a) Trininn Result
(b) Testing Result
Figure 10: Visualization results of the DOHSC with MUTAG in different perspectives.
J PARAMETER SENSITIVITY AND ROBUSTNESS
To confirm the stability of our models, we analyzed the parameter sensitivity and robustness of DOHSC and DO2HSC, respectively. Consider that the projection dimension varies in {4, 8, 16, 32, 64, 128). whereas the hidden layer dimension of the GIN module ranges from 4 to 128. In Figure 13 the DO2HSC model has less volatile performance than DOHSC, especially when the training dataset is sampled from COX2 class O, as shown in Subfigure (d). Noticeably, higher dimension of the GIN hidden layer usually displays better AUC result because the quality of the learned graph representations improves when the embedding space is sufficiently large.
In addition, we assessed different aspects of model robustness. More specifically, the AUC results about two "ratios" are displayed: 1) Different sampling ratios for the training set; 2) Different ratios Subfigures (c) and (f), the purple bars regard
of noise disturbance for the learned representation. normal data as class O, whereas green bars treat normal data as class 1. Note that most AUC results are elevated along with a higher ratio of authentic data in the training stage, demonstrating the potential of our models in the unsupervised setting. On the other hand, when more noise is blended into the training dataset, the AUC performances of the yellow line and blue line always remain stable high level. This outcome verifies the robustness of our model in response to alien data.
19

---

Published as a conference paper at ICLR 2024
14
41
Trainnng Result
02
(b) Testing Result
Figure 11: Visualization results of the DO2HSC with MUTAG in different perspectives.
2-
1-4
2-
(a) DOHSC
DO2HSC
Figure 12: Anomaly detection comparison between DOHSC and DO2HSC on MUTAG.
The percentile parameter sensitivity is presented in this section. It is worth mentioning that we tested DOHSC with varying percentiles in f0.01, O.1...,, 0.8} and tested DO2HSC only in {0.01, 0.05, 0.1) because the two radii of DO2HSC are obtained by the percentiles and 1 v. The two radii are equal when v = 0.5 and the interval between the two co-centered hyperspheres disappears. From the table, the performance would decrease when larger percentile is set obvi- ously. For example, on the MUTAG dataset, setting the percentile as 0.01 is more beneficial for DOHSC than setting DO2HSC due to the change of the interval area.
0......,008)
0.8. and setting the percentile as 0.01 better than setting it as 0.1 for
Table 9: Parameter sensitivity of DOHSC with different percentiles (all normal data is set to Class 0.)
Percentile
Dataset
0.005
0.01
0.1
0.5
0.8
COX2
0.5446
(0.0854)
0.6263 (0.0333)
0.6022 (0.0789)
0.5232 (0.0494)
0.5523 (0.0572)
ER_MD
0.6265 (0.1442)
0.6620 (0.0308)
0.7497 (0.0411)
0.6265 (0.1442)
0.5141 (0.0398)
MUTAG
0,8185 (0.0543)
0.8822 (0.0432)
0.8540 (0.0694)
0.7790 (0.0912)
0.8675 (0.1287)
DD IMDB-Binary
0.6349 (0.0380)
0.7083 (0.0188)
0.6597 (0.0270)
0.6545 (0.0268)
0.6327 (0.0206)
0.7232 (0.0314)
0.7160 (0.0600)
0.7217 (0.0418)
0.7073 (0.0274)
0.6773 (0.0566)
20

---

Published as a conference paper at ICLR 2024
144
Proaecton
Motton Loby
100%
(a)
(b)
(c)
Pogecion
Pogection
Deneroon
Denaraion of GM
Derenson
Dinwraino GNN
(c)
(e)
(f)
Figure 13: Parameter sensitivity and robustness of the proposed models. (a)-(b) Parameter sensitivity of DOHSC with different hidden layer dimensions of GIN and projection dimensions on COX2 with Class O and Class respectively. (d)-(e) Parameter sensitivity of DO2HSC with the same settings. (c) and (f) shows the performance impacts with different ratios of the training set on the IMDB- Binary dataset and added noise disturbances on the MUTAG dataset while training DOHSC and DO2HSC, respectively.
Table 10: Parameter sensitivity of DO2HSC with different percentiles (all normal data is set to Class 0.)
Percentile
Dataset
0.005
0.01
0.05
0.1
COX2
(0.0355))
0.6329 (0.0292)
0.6149 (0.0187)
0.5830 0.583000.07133 (0.0713)
BR_MD
0.6136 (0.0769)
0.6226 (0.0890)
0.6867 (0.0226)
0.6331 (0. (0.1748)
MUTAG
0.7278 (0.0478)
0.9089 (0.0609)
0.8041 (0.1006)
0.6769 (0.1207)
DD
0.7103 (0.0098)
0.7320 (0.0194)
0.6909 (0.0208)
0.6765 (0.0286)
IMDB-Binary
0.6590 (0.0287)
0.6406 (0.0642)
0.5348 (0.0486)
0.5701 (0.0740)
K SUPPLEMENTED RESULTS OF ABLATION STUDY
First, an ablation study of whether orthogonal projection requires standardization was conducted. More precisely, we pursue orthogonal features, that is, finding projection matrix for orthogonal latent representation (with standardization) instead of computing the projection onto the column or row space of the projection matrix (non-standardization),, although they are closely related to each other. This is equivalent to performing PCA and using standardized principal components. Therefore, we compared the DOHSC with and without standardization. From Table [T] 1) it is observed that the performance of DOHSC without standardization is acceptable, and most of its results are better than those of the two-stage baseline, i.e., infoGraph+Deep SVDD. This verifies the superiority of the end-to-end method over two-stage baselines. 2) The standardized model results outperform the non-standardized model results in all cases. 3) DO2HSC surpasses DOHSC no matter with/without the orthogonal projection layer.
Besides, the ablation study using the mutual information maximization loss is shown in Table 12 can be intuitively concluded that mutual information loss does not always have positive effect all data. This also indicates that the designed anomaly detection optimization method and orthogonal projection are effective, instead of entirely, owing the loss of mutual information.
21

---

Published as a conference paper at ICLR 2024
Table 11:
Comparison of the orthogonal projection layer with w/o standardization. "DSVDD"
stands for 'Deep SVDD'. "Non-Std stands for non-Standardization
Class
infoGraph-DDVDD
DOHSC (Non-Std)
DOHSC
DO2HSC (Non-Std)
DO2HSC
MUTAG
0.8805 0.0448
0.8521 + 0.0650
0.8822 + 0.0432
0,9024 0.0207
0,9089
0.0609
0.6166 + 0.2052
0.6918 + 0.1467
0.8115 + 0.0279
0.7624 0.0248
0.8250
0.0790
0
0.4825 1 0.0624
0,5800 1: 0,0473
0.6263 1 0.0333
0.6127 0.0191
0.6329 : 0,0292
COX2
0.0700
0.5029
0.6805
0.0276
0.6923
0.5312 + 0.1545
0,4881 + 0.0626
0.6620
0.0308
0.6148 0.0484
0.6867
0.0226
ER.MD
0.5082 + 0.0704
0,5140 + 0.0356
0.5184 1 0.0793
0.7043 + 0.0011
0.7351 = 0.0159
0.3942 0.0436
0.4029 + 0.0354
0.7083 + 0.0188
0,7308
0.0015
0.7320 = 0.0194
DD
0.6484 0.0236
0.6903 + 0.0215
0.7579 0.0154
0,7000 0.0165
0,7661 0.0317
0
0.6353 + 0.0277
0,5149 + 0,0655
0.6649 + 0.0133
0,6387
0.0578
0.7547 + 0.0399
IMDB-Binary
0.5836 0.0995
0.6505 + 0.0585
0.7705 + 0.0045
0,7032 0.0328
0.7737 = 0.0503
0.5662 0.0597
0.6067 0.1007
0,9185 0.0455
0.7089
0.0335
0.9390
0.0025
COLLAB
0.7926 + 0.0986
0.8958 + 0,0141
0.9755 + 0.0030
0,9033 0.0089
0.9836 + 0,0115
0.4062 0.0978
0.4912 + 0,2000
0.8826 + 0.0250
0.7158 0.1059
0.8835
0.0118
Table 12: Comparison of the loss supervision with or w/o mutual information loss (MIL).
Class
DOHSC (Non MIL)
DOHSC
DO2HSC (Non-MILL
DO2HSC
0.9456 0.0189
0.8822 0.0332
0.8308 + 0.0548
0,9089
0.0609
MUTAG
0.7597 0.0802
0.8115 +0.0279
0.7915 + 0.0274
0.8250
0.0790
0.6349 + 0.0466
0.6263
0.0333
0.6143 + 0.0302
0.6329 + 0.0292
COX2
0.6231 0.0501
0.6805
0.0168
0.6576 + 0.1830
0.6923 + 0.0433
0
0.5837 0.0778
0.6620 0.0308
0.5836 + 0.0909
0.6867 + 0.0226
ER_MD
0.6465 0.0600
0.5184 0.0793
0.7424 + 0.0385
0.7351 + 0.0159
0.4738 0.0412
0.7083
0.0188
0.6882 + 0.0221
0.7320 + 0.0194
DD
0.7197 0.0185
0.7579 0.0154
0.7376 + 0.0244
0.7651 + 0.0317
0
0.5666 + 0.0810
0.6609 + 0.0033
0,6303 + 0.0538
0.7547 : 0.0390
IMDB- BiBB-Bi
0.6827 + 0.0239
0.7705 0.0045
0,6810 + 0.0276
0.7737 + 0.0503
0.9330 0.0539
0.9185
0.0455
0.5415 + 0.0182
0.9390
0.0025
COLLAB
0.9744 0.0017
0.9755
0.0030
0,9293 + 0.0023
0.9836
0.0115
0.8275 0.0765
0.8826
0.0250
0.8452 + 0.0243
0.8835 0.0118
To demonstrate the effectiveness of the orthogonal projection layer (OPL), we conducted ablation studies and compared the comparison of 3-dimensional results produced with and without the OPL. For each model trained on particular dataset class, we show the result without OPL the left side, whereas the result with OPL is displayed on the right. As Figure [4 illustatess the OPL drastically improves the distribution of the embeddings be more spherical rather than elliptical. Similarly, with the help of the OPL, the other embeddings exhibited a more compact and rounded layout.
DOHSC
(b) DO2HSC
Figure 14; Visualizations on the MUTAG dataset Class 0 (left: without OPL: right: with OPL).
L IMBALANCED EXPERIMENTAL RESULTS
We also give the experiment on graph-level datasets with an imbalanced setting of the ratio between anomalies and normal graphs the experimental datasets. Please refer to Table 13, which showcases
22

---

Published as a conference paper at ICLR 2024
DOHSC
(b) DO2HSC
Figure 15: Visualizations on the MUTAG dataset Class (left: without OPL: right: with OPL).
(4) DOHSC
(b) DO2HSC
Figure 16: Visualizations the COX2 dataset Class 0 (left: without OPL; right: with OPL).
our results on imbalanced datasets. The experimental results illustrate that the proposed methods overcome the imbalanced problem better than other algorithms in general. But DO2HSC has more advantageous results.
M RELATED WORK
M.1 SOME SOTA ANOMALY DETECTION METHODS
In this section, first provide overview of some SOTA anomaly detection methods. The method proposed by Perera et al. (2019) involves adversarial training of an auto-encoder and a discrimina- tor, while compelling the latent representation by class of data. Goyal et al. (2020) judged the anomalous data according to the assumption that the normal instances generally lie on a low- dimensional locally linear manifold, and regarded the process of finding the decision boundary in the embedding space as an adversarial manner. Hu et al. (2020) combined holistic regularization with a 2-norm instance-level normalization, thus further proposing an effective one-class learning method. Cai & Fan proposed a perturbation learning based anomaly detection method, which generates negative samples containing the smallest noise as much as possible, to train a detec- tion classifier. The assumption is that, if this classifier can discriminate this type of negative samples and normal data, it should have the ability to distinguish more different anomalous data. All afore- mentioned algorithms are compared in experimental section to support the effectiveness of the proposed improvements.
(a) DOHSC
DO2HSC
Figure 17: Visualizations on the COX2 dataset Class (left: without OPL: right: with OPL).
23

---

Published as a conference paper at ICLR 2024
Table 13: Average AUCs with standard deviation (10 trials) of imbalanced experiments (the ratio of normal data to abnormal data is 10:1). 'DSVDD' stands for 'Deep SVDD'. The best two results are highlighted in bold and "-" means out of memory.
COX2
ER MD
MUTAG
SP+OCSVM
D.4854 + 0.0000
0.7874 = 0.0010
0,2814 0.0000
0,0764 + 0.0000
0.29174 0.0000
0.0266 0.0000
WL+OCSVM
0.4127 + 0.0000
0.8125 = 0.0000
0.5142 = 0.0000
0.1909 = 0.0000
0,7083 = 0.0000
0.0399 0.0000
NH-OCSVM
0.3818 + 0.0385
0.4875
0.0000
0.5774 + 0.0273
0,3215 + 0.0274
0.1910 + 0.0000
0.0573 + 0.0833
RW+OCSVM
0.5220 0.0000
0,2604 0.0000
: 0.0000
0.2800
0.0000
OCGIN
D.6373 1 0,0276
0.5650 1 0.2606
0.6574
0.0487
0,3208 1 0.0779
0.6333 1 0.1261
0.7387 1 0.1990
InfoOraph+DSVDD
1:
0.1594
0,5519 0.1367
0.7653
1 0.2814
0.3787 0.1049
GLocalKD
0.6465 + 0.0066
0.7063
= 0.1391
0,2578 = 0.0000
0.1979
+ 0.0000
0.8958 + 0.0335
0,9719
+ 0.0039
OCGTL
0.5394 1 0.0340
0.6150 0.0903
0,5009
0.0805
0.6972 0.0939
0.6792 1 0.0914
0.9227 0.0116
DOHSC
0.7784 +: 0.0639
0.8600
0.0339
0.7601
0.1000
0.9181
0,0203
0,9583 + 0.0373
0.9653 0.0217
DOZHSC
0.7928 1 0.0327
0.9050 0.0292
0.8443 0.0339
0.9375
0.0669
0.9792 0.0208
0.9800 0.0133
M.2 GRAPH KERNELS AND GRAPH NEURAL NETWORKS
Graph kernels (Kriege et al., 2020 measure the similarity between graphs and are very useful in many tasks involving graphs, such graph classification. A large body of work has emerged in the past years, including kernels based on neighborhood aggregation techniques and walks and paths. (Shervashidze et al. (20TT) introduced the Leisman (WL) algorithm, well-known heuristic for graph isomorphism. In (Hido & Kashima 2009). Neighborhood Hash kernel was in- troduced, where the neighborhood aggregation function is binary arithmetic. The most influential graph kernel for paths-based kernels is the shortest
(SP) kernel (Borgwardt & Kriegel 2005). For walks-based kernels. Gärtner al. (2003) and Kashima al. (2003) simultaneously proposed graph kernels based on random walks, which count the number of label sequences along walks that two graphs have in common. These graph kernel methods have the desirable property that they do not rely on the vector representation of data explicitly but access data only via the Gram matrix.
Another powerful and popular tool for handling graph data is the graph neural network. GNNs play a crucial role in effectively aggregating neighbor information for each node based on the edges in graph data. In the past decade, various improvements and enhancements for GNNs have been pro- posed (Welling & Kipf 2016: Hamilton et al. 2017, Xu et al.] 2019 Sun et al. 2020 Wu et al.) 2023 Sun et al. 2023). GNNs can be applied to both node-level tasks and graph-level tasks. For graph-level tasks, one fundamental problem is graph representation learning. which aims to rep- resent each graph as a vector and often requires a readout or pooling operation. [u et al. (2019) showed that it is more effective graph to a vector, compared to mean and max functions. |W et al. (2023) proposed a framework of graph learning based on kernel functions, which has a comparable or even better performance compared to GNNs.
a sum function convert the representations of nodes of each
M.3 GRAPH-LEVEL ANOMALY DETECTION
There are few studies undertaken in graph-level anomaly detection (GAD). Existing solutions to GAD can be categorized into two families: two-stage and end-to-end. Two-stage GAD methods (Breunig et al. 2000; Schölkopf et al. 1999) first transform graphs into graph embeddings by graph neural networks into similarities between graphs by graph kernels, and then apply off-the-shelf anomaly detectors. The drawbacks mainly include: 1) the graph feature extractor and outlier detec- tor are independent; 2) some graph kernels produce "hand-crafted" features that are deterministic without much space to improve. Whereas, end-to-end approaches overcome these problems by uti- lizing deep graph learning techniques (such as graph convolutional network (Welling and graph isomorphism network (Xu et al., 2219)), which learn an effective graph representation while detecting graph anomaly (Zhao & Akoglu 2021 Qiu et al., 2022 Ma et al. 2022).
2016)
In the past decades, regarding more end-to-end unsupervised graph-level anomaly detections, the graph kernel measures the similarity between graphs. It regards the result as a representation non- strictly implicitly. However, the graph anomaly detection task associated with it usually performs a two-stage process, which cannot maintain the quality of representation learning while learning normal data patterns. Further concerning end-to-end models, Ma et al. (2022) proposed global
24

---

Published as a conference paper at ICLR 2024
and local knowledge distillation method for graph-level anomaly detection, which learns rich global and local normal pattern information by random joint distillation of graph and node representations while needing to train two graph convolutional networks jointly at high time cost. Zhao & Akoglu (2021) combined the Deep Support Vector Data Description (Deep SVDD) objective function and graph isomorphism network to learn hypersphere of normal samples. Qiu al.) (2022) sought
hypersphere decision boundary and optimized the representations learned by k Graph Neural Net- works (GNN) close to the reference GNN while maximizing the differences between GNNs, but did not consider the relationship between the graph-level representation and node features.
25


Reviews:
[{"review_id": "jllxFhL8SI", "reviewer": "Reviewer_wkqC", "summary": "The paper introduces an innovative approach to anomaly detection, enhancing traditional hypersphere learning with an orthogonal projection layer. This improves accuracy and reduces false negatives. The paper also introduces a more compact decision region, a \"hyperspherical shell,\" and extends the methods to graph-level anomaly detection. The experimental results demonstrate the effectiveness of these methods in comparison to existing approaches. The contributions include enhanced anomaly detection techniques, particularly beneficial for high-dimensional and graph-based data.", "questions": "1. The proof of Proposition 2 in section C of supplementary materials should be make more clear.\n\n2. In equation 3, how to guarantee the projected embeddings is orthogonal via  singular value decomposition?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper stands out in the following aspects:\n1.\tOriginality: The paper presents a problem that may lead to suboptimal performance in the field of anomaly detection and provides a solution, offering a novel approach to enhance the performance of anomaly detection algorithms.\n2.\tQuality: The research is of high quality, marked by a well-structured approach, rigorous validation, and superior performance compared to existing methods. The use of benchmark datasets adds to the credibility.\n3.\tClarity: The paper is well written, ensuring clear communication of the research. It offers a logical flow.\n4.\tSignificance: The paper addresses a novel anomaly detection issue, offering potential improvements for high-dimensional and graph-based data. The practical applicability and broad relevance make it highly valuable.", "weaknesses": "1.\tThis article mentions the concept of hyperspheres but doesn't provide a more rigorous theoretical explanation for why standard hyperspheres are superior to boundaries formed by ellipsoids. Adding mathematical proofs or a deeper theoretical foundation would strengthen the paper.\n2.\tHigh-dimensional data and large datasets pose scalability challenges. The paper could address the scalability of the proposed methods and discuss their efficiency and computational requirements in dealing with big data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fURYMKaWtx", "reviewer": "Reviewer_mQBk", "summary": "The paper studies the problem of unsupervised anomaly detection. The authors proposed a deep orthogonal hypersphere compression method, which has two variants. The authors also provided theoretical analysis. The experiments on images, tabular data, and graphs showed that the proposed methods are more effective than the competitors.", "questions": "1. Figure 2 shows that the orthogonal projection improves the performance of anomaly detection. What is the fundamental reason? I suggest the authors provide further analysis as well as some references if possible. \n2. A typo or grammar issue in the first paragraph of Section 2.1.2: ‘cannot be avoided by solving equation 1’, it is not an equation. It is an optimization problem.\n3. Does Proposition 2 mean the distance (to the original or hypersphere center) based anomaly score in high-dimensional space are not reliable? If yes, the authors may add a few words to the last paragraph in Section 2.2.1 to provide a hint or motivation for the new anomaly score defined by (9).\n4. Given Proposition 2, in the high-dimensional space, the normal data are already far away from the origin. Why do we need to further push them to the outer hypersphere, namely, performing the hypersphere compression to reduce the thickness of the shell?\n5. Are $r_{max}$ and $r_{min}$ fixed or adjusted adaptively?\n6. In Proposition 3, when $r_{min}=r_{max}$, $\\kappa$ is infinity. Does this still make sense?\n7. What make graph anomaly detection special compared to image and tabular data anomaly detection? \n8. In Section 3.1, the authors mentioned a comparison method FCDD, but Table 2 doesn’t include the corresponding result.\n9. In Appendix K, the authors showed the results of imbalanced experiments of graph data. Does it mean the results on graph data in the main paper are from balanced experiments? What is the difference between these two settings?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The motivations and technique details of the proposed methods are clearly illustrated. The visualizations (e.g. Figures 1-4, 12 and 13) are very impressive.\n* The ideas especially the two concentric hyperspheres for anomaly detection are novel and interesting. They provide new insights into anomaly detection.\n* The theoretical analysis such as Propositions 1, 2, and 3 make the paper solid.\n* The experiments are extensive. There are image datasets (e.g. CIFAR10), tabular datasets, and six graph datasets.\n* More importantly, in the experiments, the proposed methods significantly outperformed state-of-the-art anomaly detection methods such as DROCC (2020), PLAD (2022), and GLocalKD (2022) and OCGTL (2022).", "weaknesses": "A minor weakness is that some points haven’t been sufficiently explained. Please refer to my questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TSDqshrzLo", "reviewer": "Reviewer_7Ufo", "summary": "This study centers on hypersphere-based anomaly detection challenges, presenting the orthogonal projection layer as an enhancement for deep SVDD. Additionally, the authors introduce the concept of bi-hypersphere anomaly detection. The effectiveness of these proposed modules is rigorously validated through a series of comprehensive experiments and insightful visualizations. Furthermore, the application of the two algorithms is extended to address graph-level anomaly detection, showcasing their versatility and potential impact in various contexts.", "questions": "See Strengths and Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper's content is grounded in sound research, with a particularly innovative contribution in the form of the bi-hypersphere concept.\n\n+ The research is substantiated by a comprehensive and diverse set of experiments, encompassing three distinct data types. The visualizations effectively convey the superiority of the proposed method.\n\n+ The visualization results pertaining to anomaly detection are distinctive and intuitive, enhancing the paper's overall quality. The improvement over the previous baselines is remarkable.", "weaknesses": "- Why can the orthogonal projection layer ensure a standard hypersphere?\n\n- The occurrence of the \"soap bubble phenomenon\" needs further clarification. Does it mean incompletely optimized?\n\n- We know that Deep SVDD compels normal data close to the center of the decision boundary, why do anomalies appear within this decision boundary? What are the main differences and similarities between normal data and these anomaly data?\n\n- Authors claimed that DO2HSC is to control training data to be more compact. I think the complete optimization of DOHSC can also reach this target, so what advantages does DO2HSC have about data compactness?\n\n- Some details need to be double-checked, such as the bolding of three results in Table 13, Class 1 of MUTAG result, while the caption specifies marking only the best two results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5pj9K7Spgh", "reviewer": "Reviewer_5jY6", "summary": "The paper suggests a new technique of hypersphere learning via a particular decision boundary to tackle the problem of Anomaly Detection. The boundary in this case is an orthogonal projection layer and the training data distribution is aligned with this geometry, a fact that encourages the correct prediction. The suggested methods seem to be ubiquitous in the data modalities, with emphasis on the graph data, and this is supported by numerical experiments.", "questions": "- Can the authors offer some details about the averaging of scores (table 1)? Did they conduct repeatedly the same experiment like in tables 2,3?\n- In the tabular data-based experiments, can the authors say why they chose F1 and not AUC ROC again, like in the graph data? \n- For the visualization of the data, the paper pictures it 'by setting the projection dimension to 3'. Thus, was any further processing of the output applied? E.g. a dimensionality reduction technique? If yes, the AUC ROC score after this outcome may have changed from the table 3 score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and the propositions seem sound.\n\n- The paper solves the optimization problem (1) and this choice seems novel. The existing literature presented the following restrictions: (i) the decision surface inferred is not a standard hypersphere but at times a hyperellipsoid, leading to insufficient accuracy, (ii) the 'regular' data are located far from the hypersphere center, thus spoiling the normality of the predicted region and allowing anomalous data to fall into the sphere, (iii) the hypersphere is shows high sparsity leading to misclassification of the anomalous points.\nThe paper suggests (i) DOHC that employs an orthogonal projection layer that limits the evaluation errors, and (ii) DO2HSC that faces the second issue above using two co-centered hyperspheres. \nThe use of a regularization term in the objective function (1) avoids the correlation of the features and the problem of different variances. \n\n- The authors provide extensive experiments in three different cases of datasets. Each case contains several datasets. The comparison to SOTA methods seems superior for the DOHSC and DO2HSC proposed model. \n\n- The proposed architecture is applicable in graph data based on the optimization of (16) function.", "weaknesses": "Minimal weaknesses. \nA weakness that can be pointed out is the use of one class in total for the anomaly detection problem, but this is clarified by the authors.\nGood paper overall.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)