SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Particularly, BlogCatalog is a social blog directory where users can follow each other. Each node represents a user, and each link indicates the following relationships between two users. The attributes of nodes are the tags that describe users and their blogs. ACM is a citation graph dataset where the nodes denote the published papers and the edge denotes the citations relationship between the papers. The attributes of each node are the content of the corresponding paper. BlogCatalog and ACM are popular GAD datasets where the anomalies are injected ones, including structural anomalies and contextual anomalies, which are created following the prior work [28]. Amazon is a graph dataset capturing the relations between users and product reviews. Following [10,60], three different user-user graph datasets are derived from Amazon using different adjacency matrix construction approaches. In this work, we focus on the Amazon-UPU dataset that connects the users who give reviews to at least one same product. The users with less than  $20\%$  are treated as anomalies. Facebook [56] is a social network where users build relationships with others and share their same friends. Reddit is a network of forum posts from the social media Reddit, in which the user who has been banned from the platform is annotated as an anomaly. Their post texts were converted to the vector as their attribute. YelpChi includes hotel and restaurant reviews filtered (spam) and recommended (legitimate) by Yelp. Following [35,44], three different graph datasets derived from Yelp using different connections in user, product review text, and time. In this work, we only use YelpChi-RUR which connects reviews posted by the same user. Note that considering it's difficult to conduct an evaluation on the isolated nodes in the graph, they were removed before modeling.
Amazon-all and YelpChi-all [10] are two datasets by treating the different relations as a single relation following [6]. T-Finance [47] is a transaction network. The nodes represent the unique account and the edges represent there are records between two accounts. The attributes of nodes are the features related to registration days, logging activities, and interaction frequency. The anomalies are fraud, money laundering and online gambling which are annotated manually. OGB-Protein [14] is a biological network where node represents proteins and edge indicates the meaningful association between proteins.
(h)
Figure 5: Homophily distribution of normal nodes and abnormal nodes on the rest of eight datasets
Table 9: Key statistics of the datasets. The real-world datasets with Injected/Real anomalies(I/R).
Data set	Type	R/I	Nodes	Edges	Attributes	Anomalies(Rate)
BlogCatalog	Social Networks	I	5,196	171,743	8,189	300(5.77%)
ACM	Citation Networks	I	16,484	71,980	8,337	597(3.63%)
Amazon	Co-review	R	10244	175,608	25	693(6.66%)
Facebook	Social Networks	R	4,039	88,234	576	27(0.67%)
Reddit	Social Networks	R	10,984	175,608	64	366(3.33%)
YelpChi	Co-review	R	24,741	49,315	32	1,217(4.91%)
Amazon-all	Co-review	R	11,944	4,398,392	25	821(6.87%)
YelpChi-all	Co-review	R	45,941	3,846,979	32	6,674(14.52%)
T-Finance	Transaction Record	R	39,357	21,222,543	10	1,803 (4.58%)
OGB-Protein	Biology Network	I	132,534	39,561,252	8	6000(4.5%)
Figure 6: AUROC and AUPRC results w.r.t. ensemble parameter  $T$
C.1 Hyperparameter Analysis
This section analyzes the sensitivity of TAM w.r.t. two key hyperparameters, including the regularization hyperparameter  $\lambda$  and the ensemble parameters  $T$ . The results on  $\lambda$  and  $T$  are shown in Fig. 6 and Fig. 7, respectively.
Ensemble Hyperparameter  $T$ . As shown in Fig. 6 with increasing  $T$ , TAM generally performs better and becomes stable around  $T \approx 4$ . This is mainly because the use of more ensemble models on truncated graphs reduces the impact of the randomness of truncation and increases the probability of weakening the affinity of abnormal nodes to its neighbors, and this effect would diminish when  $T$  is sufficiently large. The average of local affinity from multiple truncated graph sets is more conducive to anomaly detection.
Regularization Hyperparameter  $\lambda$ . In order to evaluate the effectiveness of  $\lambda$ , we adopt different values to adjust the weight of the regularization. From Fig. 7, we can see that for BlogCatalog and Facebook, adding the regularization improves the effectiveness of the model by a large margin. This is mainly because the use of regularization can prevent all nodes from having identical feature representations. For most real-world datasets with genuine anomalies, the regularization does not significantly improve the effectiveness of the model while decreasing the performance as the increasing of  $\lambda$ . The main reason is that Amazon, Reddit, and YelpChi are real-world datasets with diverse attributes, and the role of regularization term is not reflected during affinity maximization.
Figure 7: AUROC and AUPRC results w.r.t. regularization hyperparameter  $\lambda$
C.2 Complexity Analysis
This subsection analyzes the time complexity of TAM. Specifically, the distance calculation takes  $\mathcal{O}(md_0)$ ,  $m$  is the number of non-zero elements in the adjacent matrix  $\mathbf{A}$ , and  $d_0$  is the dimension of attributes for each node. The graph truncation in TAM takes  $2N\eta$ , where  $N$  is the number of nodes and  $\eta$  is the average degree in the graph. In LAMNet, we build a GCN for each truncated graph, which takes  $\mathcal{O}(md_1h)$  using sparse-dense matrix multiplications, where  $h$  and  $d_1$  denotes summation of all feature maps across different layer and feature dimensions in graph convolution operation, respectively. The construction of a GCN takes  $\mathcal{O}(md_1h)$ . LAMNet also needs to compute all connected pairwise similarities, which takes  $\mathcal{O}(N^2)$ . Thus, the overall complexity of TAM is  $\mathcal{O}(md_0 + (N^2 + md_1h + 2N\eta)KT)$ , where  $K$  is the truncation depth and  $T$  is the ensemble parameter. The complexity is lower than the time complexity in many existing GNN-based graph anomaly detection methods based on the subgraph sampling and hop counting [17, 63].
C.3 Anomaly Scoring
AUROC Results of TAM and Its Variants. We present the AUPRC results of TAM and its two variants, Degree and TAM-T in the main text, where TAM can consistently and significantly outperform both variants. Similar observation can also be found from the AUROC results in Fig. 8(a). Fig. 8(b) shows the AUROC results of anomaly scoring by aggregating the anomaly scores under all truncation scales/departs. Similar to the AUPRC results in the main text, the anomaly scores obtained from different truncation scales/departs can largely improve the detection performance and the results become stable with increasing graph truncation depth  $K$ .
Anomaly Scoring Using Multi-scale Truncation vs Single-scale Truncation. Fig. 9 shows the results of TAM that performs anomaly scoring based on a single-scale graph truncation rather than the default multi-scale graph truncation. As shown in Fig. 9, the increase of  $K$  improves the performance on large datasets such as the Amazon datasets, but it often downgrades the performance on the datasets such as Reddit and YelpChi. The main reason is that the node attributes in these datasets are more similar than the other datasets, restricting the effect of graph truncation. However, the opposite case can occur on the other datasets. To tackle this issue, we define the overall anomaly score as an average score over the anomaly scores obtained from the LAMNets built using truncated graphs under all truncation depths/scales. This resulting multi-scale anomaly score, as shown in Fig. 8(b), performs much more stably than the single-scale anomaly score.
C.4 Shared-weights LAMNet vs. Non-shared-weights LAMNet
In our experiments, the weight parameters in LAMNets are independent from each other by default, i.e., GNNs in LAMNets are independently trained. In this section, we compare TAM with its variant TAM-S where all LAMNets use a single GNN backbone with shared weight parameter.
Figure 8: AUROC results of (a) TAM vs. Degree and TAM-T, and (b) TAM w.r.t. graph truncation depth  $K$ .
Figure 9: AUROC and AUPRC results of TAM using single-truncation scale-based anomaly scores
The results are shown in Tab. 10. It is clear that TAM performs consistently better than, or comparably well to, TAM-S across the six datasets.
C.5 Simple Similarity-based Non-homophily Edge Removal
Our ablation study in the main text has shown the inferior performance of SimilarityCut method compared to our proposed NSGT, where we directly remove  $5\%$  least similar edges. Tab 11 shows the results of removing  $10\%$  and  $30\%$  least similar edges. It is clear that NSGT outperforms these two other cases of SimilarityCut.
C.6 ROC and Precision-Recall Curves
The ROC curve and Precision-Recall curve of TAM on the six datasets are shown in Fig. 10.
Table 10: AUROC and AUPRC results of TAM using shared-weight LAMNets (TAM-S) vs. non-shared-weight LAMNets (TAM).
Metric	Method	Dataset
BlogCatalog	ACM	Amazon	Facebook	Reddit	YelpChi
AUROC	TAM-S	0.8170±0.002	0.8826±0.003	0.7044±0.008	0.9165±0.005	0.6008±0.002	0.5407±0.008
TAM	0.8248±0.003	0.8878±0.024	0.7064±0.010	0.9144±0.008	0.6023±0.004	0.5643±0.007
AUPRC	TAM-S	0.3908±0.002	0.4960±0.001	0.2597±0.002	0.2087±0.006	0.0459±0.003	0.0691±0.002
TAM	0.4182±0.005	0.5124±0.018	0.2634±0.008	0.2233±0.016	0.0446±0.001	0.0778±0.009
Table 11: Performance under different ratio of similarity-based edge removal.  $\theta$  represents the edge-cut ratio based on similarity.
Metric	Method	Dataset
θ=0.05	θ=0.1	θ=0.3	TAM
AUROC	BlogCatalog	0.6650	0.6526	0.6583	0.8210
ACM	0.8668	0.7986	0.6911	0.8878
Amazon	0.5856	0.5827	0.6106	0.7064
Facebook	0.6951	0.7293	0.7934	0.9144
Reddit	0.6007	0.5945	0.5758	0.6028
YelpChi	0.4910	0.4872	0.4754	0.5674
AUPRC	BlogCatalog	0.1621	0.1829	0.1729	0.4152
ACM	0.5109	0.5068	0.4996	0.5124
Amazon	0.0924	0.1092	0.2079	0.2634
Facebook	0.0410	0.1154	0.1374	0.2233
Reddit	0.0467	0.0414	0.0420	0.0446
YelpChi	0.0598	0.0509	0.0519	0.0771
D Description of algorithms
D.1 Competing Methods
- iForest [25] builds multiple trees to isolate the data based on the node's feature. It has been widely used in outlier detection.
- ANOMALOUS [41] proposes a joint network to conduct the selection of attributes in the CUR decomposition and residual analysis. It can avoid the adverse effects brought by noise.
- DOMINANT [8] leverages the auto-encoder for graph anomaly detection. It consists of an encoder layer and a decoder layer which construct the feature and structure of the graph. The reconstruction errors from the feature and structural module are combined as the anomaly score.
- HCM-A [17] constructs an anomaly indicator by estimating hop count based on both global and local contextual information. It also employs Bayesian learning in predicting the shortest path between node pairs.
- CoLA [28] exploits the local information in a contrastive self-supervised framework. They define the positive pair and negative pair for a target node. The anomaly score is defined as the difference value between its negative and positive score.
- SL-GAD [63] constructs two modules including generative attribute regression and multiview contrastive for anomaly detection based on CoLA. The anomaly score is generated from the degree of mismatch between the constructed and original features and the discrimination scores.
- ComGA [31] designs a tailor GCN to learn distinguishable node representations by explicitly capturing community structure.
Their implementation is taken directly from their official web pages or the widely-used PyGOD library [26]. The links to the source code pages are as follows:
- iForest: https://github.com/pygod-team/pygod
- ANOMALOUS: https://github.com/pygod-team/pygod
DOMIANT: https://github.com/kaize0409/GCN_AnomalyDetectionPytorch
Figure 10: The ROC curve and precision-recall curve of our method TAM on all the six datasets used. (a)(b) BlogCatalog; (c)(d) ACM; (e)(f) Amazon; (h)(i) Facebook; (j)(k) Reddit; (l)(m) YelpChi
- HCM-A: https://github.com/TienjinHuang/GraphAnomalyDetection
- CoLA: https://github.com/GRAND-Lab/CoLA:
- SL-GAD: https://github.com/yixinliu233/SL-GAD
ComGA:https://github.com/XuexiongLuoMQ/ComGA
D.2 Pseudo Codes of TAM.
The training algorithms of TAM are summarized in Algorithm 1 and Algorithm 2. Algorithm 1 describes the process of NSGT. Algorithm 2 describes the training process of TAM.
Algorithm 1 NSGT
Input: Attributed Graph,  $G = (\mathcal{V},\mathcal{E})$  , Distance Matrix, M Output: A truncated graph structure  $\tilde{\mathcal{E}}$    
1:/\* Regard the graph as a direct graph \*/   
2: Initialize the directed graph truncation indicator  $e_{(v_i,v_j)} = 1$  iff  $(v_{i},v_{j})\in \mathcal{E}$    
3: Find the mean  $d_{mean}$  from M using  $d_{mean} = \frac{1}{m}\sum_{(v_i,v_j)\in \varepsilon}d_{ij}$    
4: for each  $v$  in V do   
5: Find the maximum  $d_{v,\max}$  from  $\{d(v,v^{\prime}),(v,v^{\prime})\in \mathcal{E}\}$    
6: if  $d_{v,\max} > d_{mean}$  then   
7: Randomly sample  $r$  from  $[d_{mean},d_{v,\max}]$  for node  $v$    
8: for each  $v^{\prime}$  in  $\{v^{\prime},(v,v^{\prime})\in \mathcal{E}\}$  do   
9: if  $d(v,v^{\prime}) > r$  then   
10: Plan to cut the edge  $v$  to  $v^{\prime}$  , i.e.,  $e_{(v,v^{\prime})}\gets 0$    
11: end if   
12: end for   
13: end if   
14: end for   
/\* The edge will be removed only when it is a non-homophily edge from both directions of the connected nodes \*/   
15: for each  $v$  in V do   
16: for each  $v^{\prime}$  in  $\{v^{\prime},(v,v^{\prime})\in \mathcal{E}\}$  do   
17: Cut the edge between  $v$  and  $v^{\prime}$ $\widetilde{\mathcal{E}} = \mathcal{E}\backslash ((v,v^{\prime})\cup (v^{\prime},v))$  ; iff  $e_{(v,v^{\prime})} = e_{(v^{\prime},v)} = 0$    
18: end for   
19: end for   
20: return The truncated graph structure  $\tilde{\mathcal{E}}$
Algorithm 2 TAM
Input: Graph,  $G = (\mathcal{V}, \mathcal{E}, \mathbf{X})$ ,  $N$ : Number of nodes,  $L$ : Number of layers,  $E$ : Training epochs,  $T$ : Ensemble parameter,  $K$ : Truncation depth.
Output: Anomaly scores of all nodes  $s$ .
1: Compute the Euclidean distance  $\mathbf{M} = \{d_{ij}\}$  for each connected node pair  $(v_i, v_j) \in \mathcal{E}$ .
2: Randomly initialize GNN  $(\mathbf{h}_1^{(0)},\mathbf{h}_2^{(0)},\dots,\mathbf{h}_N^{(0)})\gets \mathbf{X},\mathcal{E}^{(1,0)},\dots,\mathcal{E}^{(T,0)}\gets \mathcal{E}$
3: for  $k = 1, \dots, K$  do
4: for  $t = 1, \dots, T$  do
5: /* Graph truncation and update the graph structure */
6:  $\mathcal{E}^{(t,k)} = \mathrm{NSGT}(\mathcal{V},\mathcal{E}^{(t,k - 1)},\mathbf{M})$  /* LAMNet */
7: for epoch  $= 1,\dots ,E$  do
8: for each  $v$  in  $\mathcal{V}$  do
9: for  $l = 1, \dots, L$  do
10:  $\mathbf{h}_{v,l} = \phi (\mathbf{h}_{v,l - 1};\Theta_{t,k})$
11:  $\mathbf{h}_{v,l} = \mathrm{ReLU}\left(\mathrm{AGG}(\{\mathbf{h}_{v',l}:(v,v')\in \mathcal{E}^{(t,k)}\})\right)$
12: end for
13: Calculate  $f_{TAM}(v_i;\Theta_{t,k},\mathbf{A},\mathbf{X})$  by Eq. (2).
14: end for
15: /* Affinity maximization */
16: Minimize  $\sum_{v_i\in \mathcal{V}}\left(f_{TAM}(v_i;\Theta_{t,k},\mathbf{A},\mathbf{X}) + \lambda \frac{1}{|\mathcal{V}\setminus\mathcal{N}(v_i)|}\sum_{v_k\in \mathcal{V}\setminus \mathcal{N}(v_i)}\mathrm{sim}(\mathbf{h}_i,\mathbf{h}_k)\right)$
17: Update  $\Theta_{t,k}$  by using stochastic gradient descent
18: end for
19: end for
20: end for
/* Aggregated anomaly score over  $T$  sets of multi-scale graph truncation depths */
21: return Anomaly Score by  $s(v) = \frac{1}{T \times K} \sum_{k=1}^{K} \sum_{t=1}^{T} f_{TAM}(v_i; \Theta_{t,k}^*, \mathbf{A}, \mathbf{X})$