SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yixin Liu $^{1}$ , Kaize Ding $^{2}$ , Qinghua Lu $^{3}$ , Fuyi Li $^{4,5}$ , Leo Yu Zhang $^{6}$ , Shirui Pan $^{6*}$ $^{1}$ Monash University,  $^{2}$ Northwestern University,  $^{3}$ Data61, CSIRO,
<sup>4</sup>Northwest A&F University, <sup>5</sup>The University of Adelaide, <sup>6</sup>Griffith University yixin.liu@monash.edu, kaize.ding@northwestern.edu, qinghua.lu@data61.csiro.au, fuyi.li@nwsuaf.edu.cn, leo.zhang@griffith.edu.au, s.pan@griffth.edu.au
Graph-level anomaly detection (GLAD) aims to identify graphs that exhibit notable dissimilarity compared to the majority in a collection. However, current works primarily focus on evaluating graph-level abnormality while failing to provide meaningful explanations for the predictions, which largely limits their reliability and application scope. In this paper, we investigate a new challenging problem, explainable GLAD, where the learning objective is to predict the abnormality of each graph sample with corresponding explanations, i.e., the vital subgraph that leads to the predictions. To address this challenging problem, we propose a Self-Interpretable Graph aNomaly dETection model (SIGNET for short) that detects anomalous graphs as well as generates informative explanations simultaneously. Specifically, we first introduce the multi-view subgraph information bottleneck (MSIB) framework, serving as the design basis of our self-interpretable GLAD approach. This way SIGNET is able to not only measure the abnormality of each graph based on cross-view mutual information but also provide informative graph rationales by extracting bottleneck subgraphs from the input graph and its dual hypergraph in a self-supervised way. Extensive experiments on 16 datasets demonstrate the anomaly detection capability and self-interpretability of SIGNET.
Graphs are ubiquitous data structures in numerous domains, including chemistry, traffic, and social networks [1, 2, 3]. Among machine learning tasks for graph data, graph-level anomaly detection (GLAD) is a challenge that aims to identify the graphs that exhibit substantial dissimilarity from the majority of graphs in a collection [4]. GLAD presents great potential for various real-world scenarios, such as toxic molecule recognition [5] and pathogenic brain mechanism discovery [6]. Recently, GLAD has drawn increasing research attention, with advanced techniques being applied to this task, e.g., knowledge distillation [4] and one-class classification [7].
Despite their promising performance, existing works [4, 7, 8, 9] mainly aim to answer how to predict abnormal graphs by designing various GLAD architectures; however, they fail to provide explanations for the prediction, i.e., illustrating why these graphs are recognized as anomalies. In real-world applications, it is of great significance to make anomaly detection models explainable [10]. From the perspective of models, valid explainability makes GLAD models trustworthy to meet safety and security requirements [11]. For example, an explainable fraud detection model can pinpoint specific fraudulent behaviors when identifying defrauders, which enhances the reliability of predictions. From the perspective of data, an anomaly detection model with explainability can
Figure 1: A toy example to illustrate (a) GLAD problem and (b) explainable GLAD problem.
help us explicitly understand the anomalous patterns of the dataset, which further supports human experts in data understanding [12]. For instance, an explainable GLAD model for molecules can summarize the functional groups that cause abnormality, enabling researchers to deeply investigate the properties of compounds. Hence, the broad applications of interpreting anomaly detection results motivate us to investigate the problem of Explainable GLAD where the GLAD model is expected to measure the abnormality of each graph sample as well as provide meaningful explanations of the predictions during the inference time. As an example shown in Fig. 1, the GLAD model also extracts a graph rationale [13, 14] corresponding to the predicted anomaly score. Although there are a few studies [10, 15, 16] proposed to explain anomaly detection results for visual or tabular data, explainable GLAD remains underexplored and it is non-trivial to apply those methods to our problem due to the discrete nature of irregular graph-structured data [17].
Towards the goal of designing an explainable GLAD model, two essential challenges need to be solved with careful design: Challenge 1 — how to make the GLAD model self-interpretable? Even though we can leverage existing post-hoc explainers [19, 20] for GNNs to explain the predictions of the GLAD model, such post-hoc explainers are not synergistically learned with the detection models, resulting in the risk of wrong, biased, and sub-optimal explanations [21, 22]. Hence, developing a self-interpretable GLAD model which detects graph-level anomalies with explanations inherently is more desirable and requires urgent research efforts. Challenge 2 — how to learn meaningful graph explanations without using supervision signals? For the problem of GLAD, ground-truth anomalies are usually unavailable during training, raising significant challenges to both detecting anomalies and providing meaningful explanations. Since most of the existing self-interpretable GNNs [13, 21, 22] merely focus on the (semi-)supervised setting, in particular the node/graph classification tasks, how to design a self-interpretable model for the explainable GLAD problem where ground-truth labels are inaccessible remains a challenging task.
To solve the above challenges, in this paper, we develop a novel Self-Interpretable Graph aNomaly dETection model (SIGNET for short). Based on the information bottleneck (IB) principle, we first propose a multi-view subgraph information bottleneck (MSIB) framework, serving as the design basis of our self-interpretable GLAD model. Under the MSIB framework, the instantiated GLAD model is able to predict the abnormality of each graph as well as generate corresponding explanations without relying on ground-truth anomalies simultaneously. To learn the self-interpretable GLAD model without ground-truth anomalies, we introduce the dual hypergraph as a supplemental view of the original graph and employ a unified bottleneck subgraph extractor to extract corresponding graph rationales. By further conducting multi-view learning among the extracted graph rationales, SIGNET is able to learn the feature patterns from both node and edge perspectives in a purely self-supervised manner. During the test phase, we can directly measure the abnormality of each graph sample based on its inter-view agreement (i.e., cross-view mutual information) and derive the corresponding graph rationales for the purpose of explaining the prediction. To sum up, our contribution is three-fold:
- Problem. We propose to investigate the explainable GLAD problem that has broad application prospects. To the best of our knowledge, this is the first attempt to study the explainability problem for graph-level anomaly detection.
- Algorithm. We propose a novel self-interpretable GLAD model termed SIGNET, which infers graph-level anomaly scores and subgraph-level explanations simultaneously with the multi-view subgraph information bottleneck framework.
In this paper, we distinguish the terms "explainability" and "interpretability" following a recent survey paper [18]: "explainable artificial intelligence" is a widespread and high-level concept, hence we define the research problem as "explainable GLAD"; for the model that can provide interpretations of the predictions of itself, we consider it as "interpretable" or "self-interpretable". Detailed definitions are provided in Appendix A.
- Evaluation. We perform extensive experiments to corroborate the anomaly detection performance and self-interpretation ability of SIGNET via thorough comparisons with state-of-the-art methods on 16 benchmark datasets.
In this section, we introduce the preliminaries and briefly review the related works. A more comprehensive literature review can be found in Appendix B.
Notations. Let  $G = (\mathcal{V}, \mathcal{E}, \mathbf{X})$  be a simple graph with  $n$  nodes and  $m$  edges, where  $\mathcal{V}$  is the set of nodes and  $\mathcal{E}$  is the set of edges. The node features are included by feature matrix  $\mathbf{X} \in \mathbb{R}^{n \times d_f}$ , and the connectivity among the nodes is represented by adjacency matrix  $\mathbf{A} \in \mathbb{R}^{n \times n}$ . Unlike simple graphs where each edge only connects two nodes, "hypergraph" is a generalization of a traditional graph structure in which hyperedges connect more than two nodes. We define a hypergraph with  $n^*$  nodes and  $m^*$  hyperedges as  $G^* = (\mathcal{V}^*, \mathcal{E}^*, \mathbf{X}^*)$ , where  $\mathcal{V}^*, \mathcal{E}^*$ , and  $\mathbf{X}^* \in \mathbb{R}^{n^* \times d_f^*}$  are the node set, hyperedge set, and node feature matrix respectively. To indicate the higher-order relations among arbitrary numbers of nodes within a hypergraph, we use an incidence matrix  $\mathbf{M}^* \in \mathbb{R}^{n^* \times m^*}$  to represent the interaction between  $n^*$  nodes and  $m^*$  hyperedges. Alternatively, a simple graph and a hypergraph and be represented by  $G = (\mathbf{A}, \mathbf{X})$  and  $G^* = (\mathbf{M}^*, \mathbf{X}^*)$ , respectively. We denote the Shannon mutual information (MI) of two random variables  $A$  and  $B$  as  $I(A; B)$ .
Graph Neural Networks (GNNs). GNNs are the extension of deep neural networks onto graph data, which have been applied to various graph learning tasks [1, 2, 23, 24, 25, 26, 27, 28]. Mainstream GNNs usually follow the paradigm of message passing [2, 23, 24, 26]. Some studies termed hypergraph neural networks (HGNNs) also apply GNNs to hypergraphs [29, 30, 31]. The formulations of GNN and HGNN are in Appendix C. To make the predictions understandable, some efforts try to uncover the explanation for GNNs [18, 32]. A branch of methods, termed post-hoc GNN explainers, use specialized models to explain the behavior of a trained GNN [19, 20, 33]. Meanwhile, some self-interpretable GNNs can intrinsically provide explanations for predictions using interpretable designs in GNN architectures [13, 21, 22]. While these methods mainly aim at supervised classification scenarios, how to interpret unsupervised anomaly detection models still remains open.
Information Bottleneck (IB). IB is an information theory-based approach for representation learning that trains the encoder by preserving the information that is relevant to label prediction while minimizing the amount of superfluous information [34, 35, 36]. Formally, given the data  $X$  and the label  $Y$ , IB principle aims to find the representation  $Z$  by maximizing the following objective:  $\max_Z I(Z;Y) - \beta I(X;Z)$ , where  $\beta$  is a hyper-parameter to trade off informativeness and compression. To extend IB onto unsupervised learning scenarios, Multi-view Information Bottleneck (MIB) [37] provides an estimizable target for unsupervised multi-view learning, which alleviates the reliance on label  $Y$ . Given two different and distinguishable views  $V_{1}$  and  $V_{2}$  of the same data  $X$ , the objective of MIB is to learn sufficient and compact representations  $Z_{1}$  and  $Z_{2}$  for two views respectively. Taking view  $V_{1}$  as an example, by factorizing the MI between  $V_{1}$  and  $Z_{1}$ , we can identify two components:  $I(V_{1};Z_{1}) = I(V_{1};Z_{1}|V_{2}) + I(V_{2};Z_{1})$ , where the first term is the superfluous information that is expected to be minimized, and the second term is the predictive information that should be maximized. Then,  $Z_{1}$  can be learned using a relaxed Lagrangian objective:
$$
\max  _ {Z _ {1}} I \left(V _ {2}; Z _ {1}\right) - \beta_ {1} I \left(V _ {1}; Z _ {1} \mid V _ {2}\right), \tag {1}
$$
where  $\beta_{1}$  is a trade-off parameter. By optimizing Eq. (1) and its counterpart in view  $V_{2}$ , we can learn informative and compact  $Z_{1}$  and  $Z_{2}$  by extracting the information from each other.
IB principle is also proven to be effective in graph learning tasks, such as graph contrastive learning [38, 39], subgraph recognition [17, 40], graph-based recommendation [41], and robust graph representation learning [22, 42, 43]. Nevertheless, how to leverage the idea of IB on graph anomaly detection tasks is still an open problem.
Graph-level Anomaly Detection (GLAD). GLAD aims to recognize anomalous graphs from a set of graphs by learning an anomaly score for each graph sample to indicate its degree of abnormality [4, 7, 8, 9]. Recent studies try to address the GLAD problem with various advanced techniques, such as knowledge distillation [4], one-class classification [7], transformation learning [8],
and deep graph kernel [44]. However, these methods can only learn the anomaly score but fail to provide explanations, i.e., the graph rationale causing the abnormality, for their predictions.
Problem Formulation. Based on this mainstream unsupervised GLAD paradigm [4, 7, 8], in this paper, we present a novel research problem termed explainable GLAD, where the GLAD model is expected to provide the anomaly score as well as the explanations of such a prediction for each testing graph sample. Formally, the proposed research problem can be formulated by:
Definition 2.1 (Explainable graph-level anomaly detection). Given the training set  $\mathcal{G}_{tr}$  that contains a number of normal graphs, we aim at learning an explainable GLAD model  $f:\mathbb{G}\to (\mathbb{R},\mathbb{G})$  that is able to predict the abnormality of a graph and provide corresponding explanations. In specific, given a graph  $G_{i}$  from the test set  $\mathcal{G}_{te}$  with normal and abnormal graphs, the model can generate an output pair  $f(G_{i}) = (s_{i},G_{i}^{(es)})$ , where  $s_i$  is the anomaly score that indicates the abnormality degree of  $G_{i}$ , and  $G_{i}^{(es)}$  is the subgraph of  $G_{i}$  that explains why  $G_{i}$  is identified as a normal/abnormal sample.
This section details the proposed model SIGNET for explainable GLAD. Firstly, we derive a multi-view subgraph information bottleneck (MSIB) framework (Sec. 3.1) that allows us to identify anomalies with causal interpretations provided. Then, we provide the instantiation of the components in MSIB framework, including view construction (Sec. 3.2), bottle subgraph extraction (Sec. 3.3), and cross-view mutual information (MI) maximization (Sec. 3.4), which compose the SIGNET model. Finally, we introduce the self-interpretable GLAD inference (Sec. 3.5) of SIGNET. The overall learning pipeline of SIGNET is demonstrated in Fig. 2(a).
3.1 MSIB Framework Overview
To achieve the goal of self-interpretable GLAD, an unsupervised learning approach that can jointly predict the abnormality of graphs and yield corresponding explanations is required. Inspired by the concept of information bottleneck (IB) and graph multi-view learning [36, 37, 45], we propose multi-view subgraph information bottleneck (MSIB), a self-interpretable and self-supervised learning framework for GLAD. The learning objective of MSIB is to optimize the "bottleneck subgraphs", the vital substructure on two distinct views of a graph, by maximizing the predictive structural information shared by both graph views while minimizing the superfluous information that is irrelevant to the cross-view agreement. Such an objective can be optimized in a self-supervised manner, without the guidance of ground-truth labels. Due to the observation that latent anomaly patterns of graphs can be effectively captured by multi-view learning [8, 9], we can directly use the cross-view agreement, i.e., the MI between two views, to evaluate the abnormality of a graph sample. Simultaneously, the extracted bottleneck subgraphs provide us with graph rationales to explain the anomaly detection predictions, since they contain the most compact substructure sourced from the original data and the most discriminative knowledge for the predicted abnormality, i.e., the estimated cross-view MI.
Formally, in the proposed MSIB framework, we assume each graph sample  $G$  has two different and distinguishable views  $G^{1}$  and  $G^{2}$ . Then, taking view  $G^{1}$  as an example, the target of MSIB is to learn a bottleneck subgraph  $G^{1(s)}$  for  $G^{1}$  by optimizing the following objective:
$$
\max  _ {G ^ {1 (s)}} I \left(G ^ {2}; G ^ {1 (s)}\right) - \beta_ {1} I \left(G ^ {1}; G ^ {1 (s)} \mid G ^ {2}\right). \tag {2}
$$
Similar to MIB (Eq. (1), the optimization for  $G^{2(s)}$ , the bottleneck subgraph of  $G^2$ , can be written in the same format. Then, by parameterizing bottleneck subgraph extraction and unifying the domain of bottleneck subgraphs, the objective can be transferred to minimize a tractable loss function:
$$
\mathcal {L} _ {M S I B} = - I \left(G ^ {1 (s)}; G ^ {2 (s)}\right) + \beta D _ {S K L} \left(p _ {\theta} \left(G ^ {1 (s)} \mid G ^ {1}\right) \| p _ {\psi} \left(G ^ {2 (s)} \mid G ^ {2}\right)\right), \tag {3}
$$
where  $p_{\theta}(G^{1(s)}|G^{1})$  and  $p_{\psi}(G^{2(s)}|G^{2})$  are the bottleneck subgraph extractors (parameterized by  $\theta$  and  $\psi$ ) for  $G^{1}$  and  $G^{2}$  respectively,  $D_{SKL}(\cdot)$  is the symmetrized Kullback-Leibler (SKL) divergence, and  $\beta$  is a trade-off hyper-parameter. Detailed deductions from Eq. (2) to Eq. (3) are in Appendix D.
MSIB framework can guide us to build a self-interpretable GLAD model. The first term in Eq. (3) tries to maximize the MI between the bottleneck subgraphs from two views, which not only prompts the
(a) The overall pipeline of SIGNET
(b) DHT
Figure 2: (a) The overall pipeline of the proposed model SIGNET, consisting of (i) view construction, (ii) bottleneck subgraph extraction, and (iii) cross-view MI maximization. (b) An illustration of dual hypergraph transformation (DHT), where the nodes  $(\bullet)$  and edges  $(-)$  in the original graph correspond to hyperedges  $(\dots)$  and nodes  $(\square)$  in its dual hypergraph, respectively.
model to capture vital subgraphs but also helps capture the cross-view matching patterns for anomaly detection. The second term in Eq. (3) is a regularization term to align the extractors, which ensures the compactness of bottleneck subgraphs. During inference,  $-I(G^{1(s)};G^{2(s)})$  can be regarded as a measure of abnormality. Meanwhile, the bottleneck subgraphs extracted by  $p_{\theta}$  and  $p_{\psi}$  can serve as explanations. In the following subsections, we take SIGNET as a practical implementation of MSIB framework. We illustrate the instantiations of view construction ( $G^{1}$  and  $G^{2}$ ), subgraph extractors ( $p_{\theta}$  and  $p_{\psi}$ ), MI estimation ( $I(G^{1(s)};G^{2(s)})$ ), and explainable GLAD inference, respectively.
3.2 Dual Hypergraph-based View Construction
To implement MSIB framework for GLAD, the first step is to construct two different and distinguishable views  $G^{1}$  and  $G^{2}$  for each sample  $G$ . In multi-view learning approaches [37, 46, 47, 48], a general strategy is using stochastic perturbation-based data augmentation (e.g., edge modification [46] and feature perturbation [47]) to create multiple views. Despite their success in graph representation learning [46, 47, 48], we claim that perturbation-based view constructions are not appropriate in SIGNET for the following reasons. 1) Low sensitivity to anomalies. Due to the similarity of normal and anomalous graphs in real-world data, perturbations may create anomaly-like data from normal data as the augmented view [49]. In this case, maximizing the cross-view MI would result in reduced sensitivity of the model towards distinguishing between normal and abnormal data, hindering the performance of anomaly detection [50]. 2) Less differentiation. In the principle of MIB, two views should be distinguishable and mutually redundant [37]. However, the views created by graph perturbation from the same sample can be similar to each other, which violates the assumption of our basic framework. 3) Harmful instability. The MI  $I(G^{1(s)};G^{2(s)})$  for abnormality measurement is highly related to the contents of two views. Nevertheless, the view contents generated by stochastic perturbation can be quite unstable due to the randomness, leading to inaccurate estimation of abnormality.
Considering the above limitations, a perturbation-free, distinct, and stable strategy is required for view construction. To this end, we utilize dual hypergraph transformation (DHT) [51] to construct the opposite view of the original graph. Concretely, for a graph sample  $G$ , we define the first view as itself (i.e.,  $G^{1} = G$ ) and the second view as its dual hypergraph  $G^{*}$  (i.e.,  $G^{2} = G^{*}$ ). Based on the hypergraph duality [52, 53], dual hypergraph can be acquired from the original simple graph with DHT: each edge of the original graph is transformed into a node of the dual hypergraph, and each node of the original graph is transformed into a hyperedge of the dual hypergraph [51]. As the example shown in Fig. 2(b), the structural roles of nodes and edges are interchanged by DHT, and the incidence matrix  $\mathbf{M}^{*}$  of the dual hypergraph is the transpose of the incidence matrix of the original graph. To initialize the node features  $\mathbf{X}^{*} \in \mathbb{R}^{m \times d_{f}^{*}}$  of  $G^{*}$ , we can either use the original edge features (if available), or construct edge-level features from the original node features or according to edge-level geometric property.
The DHT-based view construction brings several advantages. Firstly, the dual hypergraph has significantly distinct contents from the original view, which caters to the needs for differentiation in
MIB. Secondly, the dual hypergraph pays more attention to the edge-level information, encouraging the model to capture not only node-level but also edge-level anomaly patterns. Thirdly, DHT is a bijective mapping between two views, avoiding confusion between normal and abnormal samples. Fourthly, DHT is randomness-free, ensuring the stable estimation of MI.
3.3 Bottleneck Subgraph Extraction
In MSIB framework, bottleneck subgraph extraction is a key component that learns to refine the core rationale for abnormality explanations. Following the procedure of MSIB, we need to establish two bottleneck subgraph extractors for the original view  $G$  and dual hypergraph view  $G^{*}$  respectively. To model the discrete subgraph extraction process in a differentiable manner with neural networks, following previous methods [14, 17, 22], we introduce continuous relaxation into the subgraph extractors. Specifically, for the original view  $G$ , we model the subgraph extractor with  $p_{\theta}(G^{(s)}|G) = \prod_{v\in \mathcal{V}}p_{\theta}(v\in \mathcal{V}^{(s)}|G)$ . In practice, the GNN-based extractor takes  $G$  as input and outputs a node probability vector  $\mathbf{p}\in \mathbb{R}^{n\times 1}$ , where each entry indicates the probability that the corresponding node belongs to  $G^{(s)}$ . Similarly, for the dual hypergraph view  $G^{*}$ , an edge-centric HGNN serves as the subgraph extractor  $p_{\psi}$ . It takes  $G^{*}$  as input and outputs an edge probability vector  $\mathbf{p}^{*}\in \mathbb{R}^{m\times 1}$  that indicates if the dual nodes (corresponding to the edges in the original graphs) belong to  $G^{*(s)}$ . Once the probability vectors are calculated, the subgraph extraction can be executed by:
$$
G ^ {(s)} = \left(\mathbf {A}, \mathbf {X} ^ {(s)}\right) = \left(\mathbf {A}, \mathbf {X} \odot \mathbf {p}\right), \quad G ^ {* (s)} = \left(\mathbf {M} ^ {*}, \mathbf {X} ^ {* (s)}\right) = \left(\mathbf {M} ^ {*}, \mathbf {X} ^ {*} \odot \mathbf {p} ^ {*}\right), \tag {4}
$$
where  $\odot$  is the row-wise production. Then, to implement the second term in Eq. (3), we can lift the node probabilities  $\mathbf{p}$  to edge probabilities by  $\mathbf{p}'$  by  $\mathbf{p}_{\mathbb{I}(e_{ij})}' = \mathbf{p}_i\mathbf{p}_j$ , where  $\mathbb{I}(e_{ij})$  is the index of edge connecting node  $v_i$  and  $v_j$ . After re-probabilizing  $\mathbf{p}'$ , the SKL divergence between  $\mathbf{p}'$  and  $\mathbf{p}^*$  can be computed as the regularization term in MSIB framework.