SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hezhe Qiao, Guansong Pang*
School of Computing and Information Systems, Singapore Management University
hezheqiao.2022@phdcs.smu.edu.sg, gspang@smu.edu.sg
We reveal a one-class homophily phenomenon, which is one prevalent property we find empirically in real-world graph anomaly detection (GAD) datasets, i.e., normal nodes tend to have strong connection/affinity with each other, while the homophily in abnormal nodes is significantly weaker than normal nodes. However, this anomaly-discriminative property is ignored by existing GAD methods that are typically built using a conventional anomaly detection objective, such as data reconstruction. In this work, we explore this property to introduce a novel unsupervised anomaly scoring measure for GAD - local node affinity - that assigns a larger anomaly score to nodes that are less affiliated with their neighbors, with the affinity defined as similarity on node attributes/representations. We further propose Truncated Affinity Maximization (TAM) that learns tailored node representations for our anomaly measure by maximizing the local affinity of nodes to their neighbors. Optimizing on the original graph structure can be biased by nonhomophily edges (i.e., edges connecting normal and abnormal nodes). Thus, TAM is instead optimized on truncated graphs where non-homophily edges are removed iteratively to mitigate this bias. The learned representations result in significantly stronger local affinity for normal nodes than abnormal nodes. Extensive empirical results on 10 real-world GAD datasets show that TAM substantially outperforms seven competing models, achieving over  $10\%$  increase in AUROC/AUPRC compared to the best contenders on challenging datasets. Our code is available at https://github.com/mala-lab/TAM-master/.
Graph anomaly detection (GAD) aims to identify abnormal nodes that are different from the majority of the nodes in a graph. It has attracted great research interest in recent years due to its broad real-world applications, e.g., detection of abusive reviews or malicious/fraudulent users [9, 16, 32, 37, 57]. Since graph data is non-Euclidean with diverse graph structure and node attributes, it is challenging to effectively model the underlying normal patterns and detect abnormal nodes in different graphs. To address this challenge, graph neural networks (GNNs) have been widely used for GAD. The GNN-based methods are often built using a data reconstruction [8, 11, 31, 59, 65] or self-supervised learning [15, 17, 28, 54, 63] objective. The data reconstruction approaches focus on learning node representations for GAD by minimizing the errors of reconstructing both node attributes and graph structure, while the self-supervised approaches focus on designing a proxy task that is related to anomaly detection, such as prediction of neighbor hops [17] and prediction of the relationship between a node and a subgraph [28], to learn the node representations for anomaly detection.
These approaches, however, ignore one prevalent anomaly-discriminative property we find empirically in real-world GAD datasets, namely one-class homophily, i.e., normal nodes tend to have strong connection/affinity with each other, while the homophily in abnormal nodes is significantly weaker
*Corresponding author: G. Pang
than normal nodes. This phenomenon can be observed in datasets with synthetic/real anomalies, as shown in Fig. 1(a) (see App. A for results on more datasets). The abnormal nodes do not exhibit homophily relations to each other mainly because abnormal behaviors are unbounded and can be drawn from different distributions.
Motivated by the one-class homophily property, we introduce a novel unsupervised anomaly scoring measure for GAD - local node affinity - that assigns a larger anomaly score to nodes that are less affiliated with their neighbors. Since we do not have access to class labels, we define the affinity in terms of similarity on node attributes/representations to capture the homophily relations within the normal class. Nodes having strong local affinity are the nodes that are connected to many nodes of similar attributes, and those nodes are considered more likely to be normal nodes.
(b)
Figure 1: (a) Homophily and (b) local affinity distributions of normal and abnormal nodes on two popular benchmarks, BlogCatalog [49] and Amazon [10]. The homophily of a given node is calculated using the number of nodes that have the same class label as the given node [13]. The local affinity is calculated on raw attributes (RA) and node representations learned by DGI [51] and TAM, respectively.
One challenge of using this anomaly measure is that some abnormal nodes can also be connected to nodes of similar abnormal behaviors. A straightforward solution to this problem is to apply our anomaly measure in a node representation space learned by off-the-shelf popular representation learning objectives like DGI [51], but the representations of normal and abnormal nodes can become similar due to the presence of non-homophily edges (i.e., edges connecting normal and abnormal nodes) that homogenize the normal and abnormal node representations in GNN message passing. To address this issue, we further propose Truncated Affinity Maximization (TAM) that learns tailored node representations for our anomaly scoring measure by maximizing the local node affinity to their neighbors, with the non-homophily edges being truncated to avoid the over-smooth representation issue. The learned representations result in significantly stronger local affinity for normal nodes than abnormal nodes. As shown in Fig. 1(b), it is difficult for using the local node affinity to distinguish normal and abnormal nodes on raw node attributes and DGI-based node representations, whereas the TAM-based node representation space offers well-separable local affinity results between normal and abnormal nodes. In summary, this work makes the following main contributions:
- We, for the first time, empirically reveal the one-class homophily phenomenon that provides an anomaly-discriminative property for GAD. Motivated by this property, we introduce a novel unsupervised anomaly scoring measure, local node affinity (Sec. 3.2).
- We then introduce Truncated Affinity Maximization (TAM) that learns tailored node representations for the proposed anomaly measure. TAM makes full use of the one-class homophily to learn expressive normal representations by maximizing local node affinity on truncated graphs, offering discriminative local affinity scores for accurate GAD.
- We further introduce two novel components (Sec. 3.3), namely Local Affinity Maximization-based graph neural networks (LAMNet for short) and Normal Structure-preserved Graph Truncation (NSGT), to implement TAM. Empirical results on six real-world GAD datasets show that our TAM model substantially outperforms seven competing models.
Numerous anomaly detection methods have been introduced, including both shallow and deep approaches [5,37], but most of them are focused on non-graph data. There have been many studies on GAD exclusively. Most of previous work use shallow methods [2], such as Radar [22], AMEN [43],
and ANOMALOUS [41]. Matrix decomposition and residual analysis are commonly used in these methods, whose performance is often bottlenecked due to the lack of representation power to capture the rich semantics of the graph data and to handle high-dimensional node attributes and/or sparse graph structures.
GNN-based GAD methods have shown substantially better detection performance in recent years [32]. Although some methods are focused on a supervised setting, such as CARE-GNN [10], PCGNN [27], Fraudre [58], BWGNN [47], and GHRN [12], most of them are unsupervised methods. These unsupervised methods can be generally categorized into two groups: data reconstruction-based and self-supervised-based approach. Below we discuss these most related methods in detail.
Data Reconstruction-based Approach. As one of the most popular methods, graph auto-encoder (GAE) is employed by many researchers to learn the distribution of normal samples for GAD. Ding et al. [8] propose DOMINANT, which reconstructs the graph structure and node attributes by GAE. The anomaly score is defined as the reconstruction error from both node attributes and its structure. It is often difficult for GAE to learn discriminative representations because it can overfit the given graph data. Several mechanisms, including attention, sampling, edge removing, and meta-edge choosing [9, 27, 30, 45], are designed to alleviate this issue during neighborhood aggregation. Some recent variants like ComGA [31] and AnomalyDAE [11] incorporate an attention mechanism to improve the reconstruction. Other methods like SpaceAE [24] and ResGCN [39] aim to differentiate normal and abnormal nodes that do not have significant deviation, e.g., by exploring residual information between nodes. In general, reconstructing the structure of a node based on the similarities to its neighbors is related to but different from local node affinity in TAM (see Sec. 3.2), and GAE and TAM are also learned by a very different objective (see Sec. 3.3). Being GNN-based approaches, both the reconstruction-based approaches and our approach TAM rely on the node's neighborhood information to obtain the anomaly scores, but we explicitly define an anomaly measure from a new perspective, i.e., local node affinity. This offers a fundamentally different approach for GAD.
Self-supervised Approach. Although data reconstruction methods can also be considered as a self-supervised approach, here we focus on non-reconstruction pre-text tasks for GAD. One such popular method is a proxy classification or contrastive learning task [18, 53, 65]. Liu et al. [28] propose CoLA, which combines contrastive learning and sub-graph extraction to perform self-supervised GAD. Based on CoLA, SL-GAD is proposed to develop generative attribute regression and multi-view contrastive learning [63]. There are some methods that leverage some auxiliary information such degree, symmetric and hop to design the self-supervised task [6, 17, 23, 42, 56, 61]. For example, Huang et al. propose the method HCM-A [17] to utilize hop count prediction for GAD. Although some self-supervised methods construct the classification model on the relationship between the node and the contextual subgraph, this group of methods is not related to local node affinity directly, and its performance heavily depends on how the pre-text task is related to anomaly detection. Similar to the self-supervised approaches, the optimization of TAM also relies on an unsupervised objective. However, the self-supervised approaches require the use of some pre-text tasks like surrogate contrastive learning or classification tasks to learn the feature representations for anomaly detection. By contrast, the optimization of TAM is directly driven by a new, plausible anomaly measure, which enables end-to-end optimization of an explicitly defined anomaly measure.
3.1 The Proposed TAM Approach
Problem Statement. We tackle unsupervised anomaly detection on an attributed graph. Specifically, let  $G = (\mathcal{V}, \mathcal{E}, \mathbf{X})$  be an attributed graph, where  $\mathcal{V} = \{v_1, \dots, v_N\}$  denotes its node set,  $\mathcal{E} \subseteq \mathcal{V} \times \mathcal{V}$  with  $e \in \mathcal{E}$  is the edge set,  $e_{ij} = 1$  represents there is a connection between node  $v_i$  and  $v_j$ ,  $\mathbf{X} \in \mathbb{R}^{N \times M}$  denotes the matrix of node attributes and  $\mathbf{x}_i \in \mathbb{R}^M$  is the attribute vector of  $v_i$ , and  $\mathbf{A} \in \{0, 1\}^{N \times N}$  is the adjacency matrix of  $G$  with  $\mathbf{A}_{ij} = 1$  iff  $(v_i, v_j) \in \mathcal{E}$ , then GAD aims to learn an anomaly scoring function  $f: \mathcal{V} \to \mathbb{R}$ , such that  $f(v) < f(v')$ ,  $\forall v \in \mathcal{V}_n, v' \in \mathcal{V}_a$ , where  $\mathcal{V}_n$  and  $\mathcal{V}_a$  denote the set of normal and abnormal nodes, respectively. Per the nature of anomaly, it is typically assumed that  $|\mathcal{V}_n| \gg |\mathcal{V}_a|$ . But in unsupervised GAD we do not have access to the class labels of the nodes during training. In this work, our problem is to learn an unsupervised local node affinity-based anomaly scoring function  $f$ .
Figure 2: Overview of TAM. (a) TAM leverages the observation that normal nodes have stronger affinity relations to their neighbors than anomalies to learn an unsupervised GAD model. It learns a set of affinity maximization GNNs (i.e., LAMNet) on a set of sequentially truncated graphs yielded by our probabilistic graph truncation method NSGT. We build an ensemble of TAM models to make use of the randomness in NSGT for more effective GAD. (b) NSGT iteratively removes edges with a probability proportional to the distance between the connected nodes.
Our Proposed Framework. Motivated by the one-class homophily phenomenon, we introduce a local node affinity-based anomaly score measure. Since it is difficult to capture and quantify the one-class homophily in the raw attribute space and the generic node representation space, we introduce a truncated affinity maximization (TAM) approach to learn a tailored representation space where the local node affinity can well distinguish normal and abnormal nodes based on the one-class homophily property. As shown in Fig. 2, TAM consists of two novel components, namely local affinity maximization-based graph neural networks (LAMNet) and normal structure-preserved graph truncation (NSGT). LAMNet trains a graph neural network using a local affinity maximization objective on an iteratively truncated graph structure yielded by NSGT.
NSGT is designed in a way through which we preserve the homophily edges while eliminating non-homophily edges iteratively. The message passing in LAMNet is then performed using the truncated adjacency matrix rather than the original one. In doing so, we reinforce the strong affinity among nodes with homophily relations to their neighbors (e.g., normal nodes), avoiding the potential bias caused by non-homophily edges (e.g., connections to abnormal nodes). The output node affinity in LAMNet is used to define anomaly score, i.e., the weaker the local affinity is, the more likely the node is an abnormal node. There exists some randomness in our NSGT-based graph truncation. We utilize those randomness for more effective GAD by building a bagging ensemble of TAM.
3.2 Local Node Affinity as Anomaly Measure
As shown in Fig. 1, normal nodes have significantly stronger homophily relations with each other than the abnormal nodes. However, we do not have class label information to calculate the homophily of each node in unsupervised GAD. We instead utilize the local affinity of each node to its neighbors to exploit this one-class homophily property for unsupervised GAD. The local affinity can be defined as an averaged similarity to the neighboring nodes, and the anomaly score  $f$  is opposite to the affinity:
$$
h \left(v _ {i}\right) = \frac {1}{\left| \mathcal {N} \left(v _ {i}\right) \right|} \sum_ {v _ {j} \in \mathcal {N} \left(v _ {i}\right)} \sin \left(\mathbf {x} _ {i}, \mathbf {x} _ {j}\right); f \left(v _ {i}\right) = - h \left(v _ {i}\right), \tag {1}
$$
where  $\mathcal{N}(v_i)$  is the neighbor set of node  $v_{i}$  and  $\mathrm{sim}(\mathbf{x}_i,\mathbf{x}_j) = \frac{\mathbf{x}_i^{\mathrm{T}}\mathbf{x}_j}{\|\mathbf{x}_i\| \|\mathbf{x}_j\|}$  measures the similarity of a node pair  $(v_{i},v_{j})$ . The local node affinity  $h(v_{i})$  is a normal score: the larger the affinity is, the stronger homophily the node has w.r.t. its neighboring nodes based on node attributes, and thus, the more likely the node is a normal node.
The measure in Eq. (1) provides a new perspective to quantify the normality/abnormality of nodes, enabling a much simpler anomaly scoring than existing popular measures such as the reconstruction error  $f(v_{i}) = (1 - \alpha)\| \mathbf{a}_{i} - \widehat{\mathbf{a}}_{i}\|_{2} + \alpha \| \mathbf{x}_{i} - \widehat{\mathbf{x}}_{i}\|_{2}$ , where  $\mathbf{a}_i$  denotes the neighborhood structure of  $v_{i}$ ,  $\widehat{\mathbf{a}}_i$  and  $\widehat{\mathbf{x}}_i$  are the reconstructed structure and attributes for node  $v_{i}$ , and  $\alpha$  is a hyperparameter.
Our anomaly measure also provides a new perspective to learn tailored node representations for unsupervised GAD. Instead of minimizing the commonly used data reconstruction errors, based on the one-class homophily, we can learn anomaly-discriminative node representations by maximizing the local node affinity. Our TAM approach is designed to achieve this goal.
3.3 TAM: Truncated Affinity Maximization
The local node affinity may not work well in the raw node attribute space since (i) there can be many irrelevant attributes in the original data space and (ii) some abnormal nodes can also be connected to nodes of similar attributes. To tackle this issue, TAM is proposed to learn optimal node representations that maximize the local affinity of nodes that have strong homophily relations with their neighbors in terms of node attributes. Due to the overwhelming presence of normal nodes in a graph, the TAM-based learned node representation space is optimized for normal nodes, enabling stronger local affinity for the normal nodes than the abnormal ones. The TAM-based anomaly scoring using local node affinity can be defined as:
$$
f _ {T A M} \left(v _ {i}; \Theta , \mathbf {A}, \mathbf {X}\right) = - \frac {1}{\left| \mathcal {N} \left(v _ {i}\right) \right|} \sum_ {v _ {j} \in \mathcal {N} \left(v _ {i}\right)} \sin \left(\mathbf {h} _ {i}, \mathbf {h} _ {j}\right), \tag {2}
$$
where  $\mathbf{h}_i = \psi(v_i; \Theta, \mathbf{A}, \mathbf{X})$  is a GNN-based node representation of  $v_i$  learned by a mapping function  $\psi$  parameterized by  $\Theta$  in TAM. Below we introduce how we learn the  $\psi$  function via the two components of TAM, LAMNet and NSGT.
Local Affinity Maximization Networks (LAMNet). LAMNet aims to learn a GNN-based mapping function  $\psi$  that maximizes the affinity of nodes with homophily relations to their neighbors, while keeping the affinity of nodes with non-homophily edges are weak. Specifically, the projection from the graph nodes onto new representations using  $\ell$  GNN layers can be generally written as
$$
\mathbf {H} ^ {(\ell)} = \operatorname {G N N} \left(\mathbf {A}, \mathbf {H} ^ {(\ell - 1)}; \mathbf {W} ^ {(\ell)}\right), \tag {3}
$$
where  $\mathbf{H}^{(\ell)}\in \mathbb{R}^{N\times h^{(l)}}$  and  $\mathbf{H}^{(\ell -1)}\in \mathbb{R}^{N\times h^{(l - 1)}}$  are the  $h^{(l)}$ -dimensional and  $h^{(l - 1)}$ -dimensional representations of node  $v_{i}$  in the  $(\ell)$ -th layer and  $(\ell -1)$ -th layer, respectively. In the first GNN layer, i.e., when  $\ell = 1$ , the input  $\mathbf{H}^{(0)}$  is set to the raw attribute matrix  $\mathbf{X}$ .  $\mathbf{W}^{(\ell)}$  are the weight parameters of  $(\ell)$ -th layer. For  $\mathrm{GNN}(\cdot)$ , multiple types of GNNs can be used [50, 55]. In this work, we employ a GCN (graph convolutional network) [20] due to its high efficiency. Then  $\mathbf{H}^{(\ell)}$  can be obtained via
$$
\mathbf {H} ^ {(\ell)} = \phi \left(\mathbf {D} ^ {- \frac {1}{2}} \mathbf {A D} ^ {- \frac {1}{2}} \mathbf {H} ^ {(\ell - 1)} \mathbf {W} ^ {(\ell - 1)}\right), \tag {4}
$$
where  $\mathbf{D} = \mathrm{diag}\left(\mathbf{D}_i\right) = \sum_j\mathbf{A}_{ij}$  is the degree matrix for the graph  $G$ , and  $\phi (\cdot)$  is an activation function. Let  $\mathbf{H}^{(\ell)} = \{\mathbf{h}_1,\mathbf{h}_2,\dots ,\mathbf{h}_N\}$  be the node representations of the last GCN layer, then the mapping function  $\psi$  is a sequential mapping of graph convolutions as in Eq. (4), with  $\Theta = \{\mathbf{W}^1,\mathbf{W}^2,\dots ,\mathbf{W}^{(\ell)}\}$  be the parameter set in our LAMNet. The following objective can then be used to optimize  $\psi$ :
$$
\min  _ {\Theta} \sum_ {v _ {i} \in \mathcal {V}} \left(f _ {T A M} \left(v _ {i}; \Theta , \mathbf {A}, \mathbf {X}\right) + \lambda \frac {1}{| \mathcal {V} \backslash \mathcal {N} \left(v _ {i}\right) |} \sum_ {v _ {k} \in \mathcal {V} \backslash \mathcal {N} \left(v _ {i}\right)} \operatorname {s i m} \left(\mathbf {h} _ {i}, \mathbf {h} _ {k}\right)\right), \tag {5}
$$
where the first term is equivalent to maximizing the local affinity of each node based on the learned node representations, while the second term is a regularization term, and  $\lambda$  is a regularization hyperparameter. The regularization term adds a constraint that the representation of each node should be dissimilar from that of non-adjacent nodes to enforce that the representations of non-local nodes are distinguishable, while maximizing the similarity of the representations of local nodes.
The optimization using Eq. (5) can be largely biased by non-homophily edges. Below we introduce the NSGT component that helps overcome this issue.
Normal Structure-preserved Graph Truncation (NSGT). LAMNet is driven by the one-class homophily property, but its objective and graph convolution operations can be biased by the presence non-homophily edges, i.e., edges that connect normal and abnormal nodes. The NSGT component
is designed to remove these non-homophily edges, yielding a truncated adjacency matrix  $\tilde{\mathbf{A}}$  with homophily edge-based normal graph structure. LAMNet is then performed using the truncated adjacency matrix  $\tilde{\mathbf{A}}$  rather than the original adjacency matrix  $\mathbf{A}$ .
Since homophily edges (i.e., edges connecting normal nodes) connect nodes of similar attributes, the distance between the nodes of homophily edges is often substantially smaller than that of nonhomophily edges. But there can also exist homophily edges that connect dissimilar normal nodes. These can be observed in GAD datasets, as shown in Fig. 3(a-b) for datasets with synthetic/real anomalies. Motivated by this, NSGT takes a probabilistic approach and performs the graph truncation as follows: for a given edge  $e_{ij} = 1$ , it is considered as a non-homophily edge and removed (i.e.  $e_{ij} = 0$ ) if and only if the distance between node  $v_i$  and node  $v_j$  is sufficiently large w.r.t. the neighbor sets of both  $v_i$  and  $v_j$ ,  $\mathcal{N}(v_i)$  and  $\mathcal{N}(v_j)$ . Formally, NSGT truncates the graph by
$$
e _ {i j} \leftarrow 0 \text {i f f} d _ {i j} > r _ {i} \& d _ {i j} > r _ {j}, \forall e _ {i j} = 1, \tag {6}
$$
where  $d_{ij}$  is a Euclidean distance between  $v_{i}$  and  $v_{j}$  based on node attributes,  $r_i$  is a randomly selected value from the range  $[d_{mean}, d_{i,max}]$  where  $d_{mean} = \frac{1}{m} \sum_{(v_i, v_j) \in \varepsilon} d_{ij}$  is the mean distance of graph, where  $m$  is the number of non-zero elements in the adjacent matrix, and  $d_{i,max}$  is the maximum distances in  $\{d_{ik}, v_k \in \mathcal{N}(v_i)\}$ . Similarly,  $r_j$  is randomly sampled from the range  $[d_{mean}, d_{j,max}]$ .
Theoretically, the probability of  $r_i < d_{ij}$  can be defined as
$$
p \left(r _ {i} <   d _ {i j}\right) = \frac {\operatorname* {m a x} \left(d _ {i j} - d _ {\text {m e a n}} , 0\right)}{d _ {i , \max } - d _ {\text {m e a n}}}. \tag {7}
$$
Note that this probability is premised on  $d_{i,\max} > d_{mean}$ , and we set  $p(r_i < d_{ij}) = 0$  if  $d_{i,\max} \leq d_{mean}$ . Then the probability of  $e_{ij}$  being removed during the truncation process is as follows
$$
p \left(\mathcal {E} \backslash e _ {i j}\right) = p \left(r _ {i} <   d _ {i j}\right) p \left(r _ {j} <   d _ {i j}\right), \tag {8}
$$
As shown in Fig. 2(b), if  $e_{ij}$  is a homophily edge, we would have a small  $d_{ij}$ , which results in small  $p(r_i < d_{ij})$  and  $p(r_j < d_{ij})$ , and thus,  $p(\mathcal{E} \setminus e_{ij})$  is also small. By contrast, a non-homophily edge would result in a large  $d_{ij}$ , and ultimately a large  $p(\mathcal{E} \setminus e_{ij})$ . Therefore, NSGT can help eliminate non-homophily edges with a high probability, while preserving the genuine homophily graph structure.