SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\hat{\mathcal{G}}(v_i)_m = f(\hat{\mathcal{G}}(v_i)_{m-1}; \phi)$  and  $\mathcal{G}'(v_i)_{m+1}^l$  refers to one of the  $L$  target corrupted ego-graphs that GRAPHPATCHER aims to approximate at the  $m$ -th step. This approach allows GRAPHPATCHER to patch the anchor node towards a well-approximated region where its high-degree counterparts should locate, instead of one point randomly sampled from this region.
With  $M - 1$  virtual nodes patched to the ego-graph, we further ask GRAPHPATCHER to generate a last patching node to  $\hat{\mathcal{G}}(v_i)_{M-1}$  and enforce the resulted graph  $\hat{\mathcal{G}}'(v_i)_M$  to match the GNN's prediction on the original ego-graph. The last patching node could be regarded as a slack variable to complement minor differences between the original and the least corrupted ego-graphs, formulated as:
$$
\mathcal {L} _ {\text {r e c o n}} = \sum_ {v _ {i} \in V _ {\mathrm {t r}}} \operatorname {K L - D i v} \left(f _ {g} \left(\mathcal {G} \left(v _ {i}\right); \boldsymbol {\theta} ^ {*}\right) [ v _ {i} ], f _ {g} \left(\hat {\mathcal {G}} \left(v _ {i}\right) _ {M}; \boldsymbol {\theta} ^ {*}\right) [ v _ {i} ]\right), \tag {6}
$$
where  $\hat{\mathcal{G}}(v_i)_M = f(\hat{\mathcal{G}}(v_i)_{M-1};\phi)$ .  $\mathcal{L}_{\mathrm{recon}}$  (Equation (6)) also prevents GRAPHPATCHER from overfitting to the low-degree nodes and enforces GRAPHPATCHER to maintain the target GNN's performance over high-degree nodes, since only marginal distribution modification should be expected with this last patching node. Hence, GRAPHPATCHER is optimized by a linear combination of the above two objectives (i.e.,  $\arg \min_{\phi}\mathcal{L}_{\mathrm{patch}} + \mathcal{L}_{\mathrm{recon}}$ ).
3.2.3 Theoretical Analysis
As shown in Equation (5), one of the important factors that contribute to the success of GRAPH-PATCHER is sampling multiple ego-graphs with the same corruption strength. The following theorem shows that the error is bounded w.r.t. the number of sampled ego-graphs  $L$ .
Theorem 1. Assuming the parameters of GRAPHPATCHER are initialized from the set  $P_{\beta} = \{\phi : ||\phi - \mathcal{N}(\mathbf{0}_{|\phi}; \mathbf{1}_{|\phi|})||_F < \beta\}$  where  $\beta > 0$ , with probability at least  $1 - \delta$  for all  $\phi \in P_{\beta}$ , the error (i.e.,  $\mathbb{E}(\mathcal{L}_{patch}) - \mathcal{L}_{patch}$ ) is bounded by  $\mathcal{O}(\beta \sqrt{\frac{|\phi|}{L}} + \sqrt{\frac{\log(1 / \beta)}{L}})$ .
The proof of Theorem 1 is provided in Appendix C. From the above theorem, we note that without the sampling strategy (i.e.,  $L = 1$ ), the generalization error depends only on the number of parameters (i.e.,  $|\phi|$ ) given the same objective function, which could lead to high variance to the supervision signal and instability to the optimization process. According to this theorem and our empirical observation, an affordable value of  $L$  (e.g.,  $L = 10$ ) delivers stable results across datasets.
Datasets. We conduct comprehensive experiments on seven real-world benchmark datasets that are broadly utilized by the graph community, including Cora, Citeseer, Pubmed, Wiki.cs, Amazon-Photo, Coauthor-CS, ogbn-arxiv, Actor, and Chameleon [44, 28, 12, 33]. This list of datasets covers graphs with distinctive characteristics (i.e., graphs with different domains and dimensions) to fully evaluate the effectiveness of GRAPHPATCHER. The detail of these datasets can be found in Appendix A.
Baselines. We compare GRAPHPATCHER with six state-of-the-art graph learning frameworks from three branches. The first branch specifically aims at enhancing the performance on low-degree nodes, including TAIL-GNN [25], COLBBREW [55], and TUNEUP [13]. The second branch consists of frameworks that focus on handling out-of-distribution scenarios, including EERM [42] and GTRANS [15]. We list this branch of frameworks as baselines because the sub-optimal performance of GNNs over low-degree nodes could be regarded as an out-of-distribution scenario. As GRAPH-PATCHER is a test-time augmentation framework, the last branch of baseline includes DROPEDGE, which is a data augmentation framework employed during training.
Evaluation Protocol. We evaluate all models using the node classification task [22, 38], quantified by the accuracy score. For datasets with publicly available (i.e., ogbn-arxiv, Cora, CiteSeer, and PubMed), we employ their the provided splits for the model training and testing. Whereas for other datasets, we create a random  $10\% / 10\% / 80\%$  split for the training/validation/testing split, to simulate a semi-supervised learning setting. All reported performance is averaged over 10 independent runs with different random seeds. Both mean values and standard deviations for the performances of all models are reported. Besides mitigating the degree bias for supervised GNNs, GRAPHPATCHER is also applicable to self-supervised GNNs. To evaluate the model performance for them, we apply GRAPHPATCHER and TUNEUP to state-of-the-art self-supervised GNNs including DGI [39], GRACE [57], and PARETOGNN [20]. We only compare our proposal with TUNEUP since other frameworks require specific model architectures and hence do not apply to self-supervised GNNs.
Hyper-parameters. We use the optimal settings on all baselines given by the authors for the shared datasets and a simple two-layer GCN [22] as the backbone model architecture for all applicable baselines. Hyper-parameters we tune for GRAPHPATCHER include learning rate, hidden dimension, the augmentation strength at each step, and the total amount of patching steps with details described in Appendix B. Besides, all of our models are trained on a single RTX3090 with 24GB VRAM; additional hardware information can also be found in the appendix.
4.2 Performance Comparison with Baselines
We compare GRAPHPATCHER with six state-of-the-art frameworks that mitigate the degree bias problem and the performances of all models are shown in Table 1. Firstly we notice that the problem of degree bias is quite serious across datasets for GCN. The performances on low-degree nodes are  $\sim 10\%$  lower than those over high-degree nodes. Comparing GCN with COLBBREW, TAIL-GNN, and TUNEUP, we can observe that frameworks that focus specifically on low-degree nodes can usually enhance GNN's performance over the lower percentile (e.g.,  $1.2\%$  accuracy gain on Cora by TUNEUP,  $0.74\%$  on Citeseer by COLBBREW,  $1.38\%$  on Pubmed by TAIL-GNN, etc.). However, these frameworks fall short on the high-degree nodes and sometimes perform worse than the vanilla GCN (e.g.,  $-2.7\%$  accuracy degradation on Cora by TAIL-GNN,  $-11.42\%$  on Wiki.CS by COLBBREW, and  $-2.5\%$  on Amazon Photo by TUNEUP). This phenomenon could result from that they unintentionally create an artificial out-of-distribution scenario, where they only observe low-degree nodes during the training, leading to downgraded performance for high-degree nodes that GNNs originally perform well at. Comparing GCN with GTRANS and EERM, we observe that they deliver similar performances as the vanilla GCN does, indicating that frameworks targeting out-of-distribution scenarios cannot mitigate degree biases. Comparing GRAPHPATCHER with all baselines, we notice that our proposed GRAPHPATCHER consistently improves the low-degree performance with an average improvement gain of 2.23 accuracy score. Besides, unlike other frameworks that have downgraded performance over high-degree nodes, GRAPHPATCHER can maintain GCN's original high-degree superiority, due to our iterative node patching. On average, GRAPHPATCHER improves GCN's overall performance by a 1.4 accuracy score across datasets.
Table 1: Performance (%) of all models over nodes with different degrees. Lower and upper percentile indicate the set of nodes whose degree is ranked in the lower and upper  $33\%$  population respectively. A two-layer GCN is used as the backbone model for all applicable baselines. Bold indicates the best performance and underline indicates the runner-up, with standard deviations as subscripts.
Method	Cora	Citeseer	Pubmed	Wiki_CS	Am.Photo	Co_CS	Arxiv	Chameleon	Actor
ACCURACY ON LOW-DEGREE NODES (LOWER PERCENTILE)
GCN	73.27±0.01	64.86±0.92	76.88±0.40	72.98±0.50	75.59±0.43	84.59±0.45	63.15±0.13	54.05±0.18	27.30±0.52
COLBBREW	73.82±0.98	65.60±0.08	77.72±0.63	73.98±0.52	76.18±0.80	85.56±0.69	63.02±0.21	53.41±0.22	27.88±0.13
TAIL-GNN	71.17±0.80	57.66±0.83	75.38±0.89	74.36±0.18	77.22±0.94	85.13±0.60	OOM	53.48±0.04	27.80±0.62
TUNEUP	74.47±0.34	65.17±0.22	77.18±0.39	72.60±0.75	76.08±0.62	84.68±0.50	63.34±0.32	53.87±0.43	27.94±0.14
EERM	73.40±0.06	64.27±0.33	76.30±0.20	73.12±0.68	75.15±0.59	84.82±0.74	63.20±0.11	54.11±0.32	27.48±0.39
GTRANS	73.16±0.66	64.95±0.83	77.05±1.00	72.15±0.50	75.55±0.55	84.74±0.06	62.88±0.14	54.29±0.14	27.53±0.21
DROPEDGE	73.57±0.97	65.47±0.27	75.68±0.82	73.94±0.20	76.49±0.03	84.31±0.33	61.33±0.33	54.12±0.41	27.39±0.24
GRAPHPATCHER	78.08±0.06	67.27±0.20	78.98±0.21	74.04±0.86	77.84±0.36	86.76±0.84	64.01±0.12	54.48±0.71	29.27±0.57
ACCURACY ON HIGH-DEGREE NODES (UPPER PERCENTILE)
GCN	86.83±0.17	77.25±1.00	80.84±0.76	83.40±0.70	84.07±0.71	90.20±0.37	80.46±0.18	54.11±0.73	27.41±0.29
COLBBREW	84.80±0.04	75.33±0.84	78.66±0.38	71.98±0.95	77.07±0.14	82.16±0.39	70.57±0.36	53.72±0.48	26.67±0.29
TAIL-GNN	84.13±0.48	74.85±0.30	78.74±0.34	78.91±0.97	80.32±0.60	86.75±0.90	OOM	54.53±0.12	27.13±0.44
TUNEUP	87.13±0.67	76.95±0.63	81.74±0.49	83.11±0.53	81.57±0.07	90.65±0.86	80.09±0.51	54.25±0.59	26.64±0.71
EERM	85.89±0.09	76.32±0.23	79.98±0.06	82.98±0.07	84.32±0.96	90.17±0.11	80.37±0.12	54.41±0.71	27.39±0.14
GTRANS	86.32±0.34	76.60±0.44	80.56±0.92	83.42±0.04	83.95±0.99	89.99±0.10	80.77±0.26	54.21±0.19	27.29±0.12
DROPEDGE	86.53±0.99	76.35±0.17	81.44±0.51	83.37±0.43	84.97±0.56	89.28±0.08	80.64±0.36	54.17±0.11	27.38±0.21
GRAPHPATCHER	88.02±0.11	76.65±0.18	83.83±0.79	83.49±0.22	84.17±0.97	90.59±0.46	80.61±0.25	54.20±0.21	27.43±0.62
OVERALL PERFORMANCE
GCN	81.22±0.40	70.51±0.46	79.14±0.31	77.30±0.41	80.38±0.86	88.16±0.66	71.73±0.14	52.83±0.35	27.20±0.57
COLBBREW	80.70±0.86	70.10±0.55	78.66±0.93	73.82±0.69	78.24±0.62	85.80±0.79	63.55±0.48	52.12±0.53	26.75±0.32
TAIL-GNN	79.44±0.64	65.80±0.04	76.14±0.25	74.66±0.18	80.68±0.58	87.02±0.33	OOM	52.46±0.12	27.62±0.47
TUNEUP	82.11±0.39	70.92±0.02	79.91±0.26	76.93±0.81	79.74±0.28	88.46±0.97	71.51±0.30	52.89±0.41	27.32±0.64
EERM	81.47±0.19	70.08±0.19	78.65±0.43	77.29±0.96	79.79±0.61	88.07±0.30	71.70±0.18	52.93±0.24	27.65±0.29
GTRANS	80.79±0.51	69.51±0.93	78.67±0.93	76.39±0.27	80.02±0.80	88.06±0.96	71.77±0.19	52.67±0.35	26.93±0.41
DROPEDGE	81.10±0.31	71.10±0.86	78.90±0.68	77.49±0.78	81.11±0.72	87.56±0.64	71.82±0.33	52.89±0.22	27.28±0.32
GRAPHPATCHER	84.17±0.54	71.65±0.05	81.13±0.68	78.12±0.57	81.23±0.32	89.44±0.79	72.31±0.22	53.21±0.39	28.34±0.24
Table 2: Performance (%) of GRAPHPACKER and TUNEUP for different GNN architectures.
Method	Cora	Citeseer	Pubmed	Wiki.CS	Am.Photo	Co.CS	Arxiv
ACCURACY ON LOW-DEGREE NODES (LOWER PERCENTILE)
GCN	73.27±0.01	64.86±0.92	76.88±0.40	72.98±0.50	75.59±0.43	84.59±0.45	63.15±0.13
+TUNEUP	74.47±0.34	65.17±0.22	77.18±0.39	72.60±0.75	76.08±0.62	84.68±0.50	63.34±0.32
+GRAPHPATCHER	78.08±0.06	67.27±0.20	78.98±0.21	74.04±0.86	77.84±0.36	86.76±0.84	64.01±0.12
G-SAGE	70.57±0.84	67.44±0.11	76.58±0.36	61.83±0.89	76.32±0.33	74.53±0.69	61.64±0.62
+TUNEUP	71.47±0.11	67.44±0.21	77.78±0.92	58.46±0.77	78.48±0.89	74.88±0.76	62.43±0.59
+GRAPHPATCHER	72.33±0.21	67.89±0.33	78.93±0.05	61.46±0.40	78.11±0.13	75.14±0.21	62.92±0.28
GAT	73.27±0.51	69.07±0.11	72.37±0.27	73.72±0.64	79.66±0.58	86.93±0.50	63.54±0.20
+TUNEUP	76.58±0.07	66.67±0.33	72.07±0.81	72.08±0.20	81.08±0.25	86.72±0.84	63.71±0.31
+GRAPHPATCHER	76.88±0.32	70.87±0.78	74.47±0.63	74.26±0.72	80.05±0.19	89.50±0.93	64.12±0.14
ACCURACY ON HIGH-DEGREE NODES (UPPER PERCENTILE)
GCN	86.83±0.17	77.25±1.00	80.84±0.76	83.40±0.70	84.07±0.71	90.20±0.37	80.46±0.18
+TUNEUP	87.13±0.67	76.95±0.63	81.74±0.49	83.11±0.53	81.57±0.07	90.65±0.86	80.09±0.51
+GRAPHPATCHER	88.02±0.11	76.65±0.18	83.83±0.79	83.49±0.22	84.17±0.97	90.59±0.46	80.61±0.25
G-SAGE	82.04±0.01	72.46±0.70	80.24±0.12	60.83±0.18	77.30±0.56	69.24±0.55	78.66±0.08
+TUNEUP	80.84±0.36	73.95±0.36	81.14±0.41	60.90±0.05	79.36±0.89	70.12±0.25	79.26±0.51
+GRAPHPATCHER	82.14±0.48	73.22±0.25	81.66±0.46	61.02±0.44	78.57±0.14	70.53±0.68	79.91±0.31
GAT	85.33±0.36	76.65±0.80	81.14±0.20	82.21±0.44	87.84±0.23	91.33±0.81	81.37±0.16
+TUNEUP	86.23±0.47	76.65±0.33	80.84±0.02	81.73±0.36	89.02±0.78	92.00±0.04	81.44±0.11
+GRAPHPATCHER	86.53±0.37	76.35±0.39	81.14±0.89	82.34±0.08	87.84±0.56	91.61±0.20	81.49±0.15
We further apply GRAPHPATCHER to other GNN architectures (i.e., GraphSAGE [9] and GAT [38]) and compare its performance to TUNEUP. We only compare with TUNEUP since other baselines explore specific model architectures that do not allow a different backbone. From Table 2, we can observe that the issue of degree bias still exists on GAT and GraphSAGE with a performance gap between low- and high-degree nodes around  $\sim 10\%$ . Both TUNEUP and GRAPHPATCHER can improve the performance over low-degree nodes. Specifically, TUNEUP on average improves 0.27 low-degree accuracy for GraphSAGE and 0.40 for GAT across datasets; whereas GRAPHPATCHER improves 1.13 for GraphSAGE and 1.66 for GAT, outperforming TUNEUP by a large margin.
4.3 Performance of GRAPHPATCHER for Self-supervised GNNs
To fully demonstrate the effectiveness of GRAPHPATCHER, we also apply our proposal to self-supervised GNNs, as shown in Table 3. We can observe that self-supervised learning can mitigate degree bias by itself, proved by smaller gaps between low- and high-degree nodes than those of semi-supervised GNNs. Combined with GRAPHPATCHER, the degree biases can be further without sacrificing GNN's original superiority over high-degree nodes. On average, GRAPHPATCHER can enhance the low-degree performance of these three self-supervised GNNs by 1.78, 0.74, and 1.36 accuracy scores respectively.
Table 3: Effectiveness for self-supervised GNNs.
Method	Cora	Pubmed	Wiki.CS
LOW-DEGREE NODES (LOWER PERCENTILE)
DGI	78.47±0.37	75.63±0.82	75.86±0.61
+GRAPHPATCHER	79.95±0.53	78.04±0.97	77.31±0.91
GRACE	77.81±0.73	77.80±0.65	74.31±0.63
+GRAPHPATCHER	78.53±0.82	78.49±0.16	75.12±0.34
PARETOGNN	78.85±0.71	78.32±0.33	74.17±0.18
+GRAPHPATCHER	79.91±0.62	79.11±0.89	76.41±0.22
HIGH-DEGREE NODES (UPPER PERCENTILE)
DGI	86.83±0.82	81.14±0.28	81.09±0.81
+GRAPHPATCHER	86.91±0.10	82.31±0.53	80.95±0.19
GRACE	85.03±0.05	78.74±0.84	83.91±0.56
+GRAPHPATCHER	85.12±0.25	79.58±0.31	84.12±0.22
PARETOGNN	87.03±0.84	80.89±0.84	81.57±0.84
+GRAPHPATCHER	87.32±0.27	80.55±0.32	81.78±0.51
4.4 Effectiveness of GRAPHPatcher for Enhancing SoTA Method
We apply GRAPHPATCHER to GRAND [5], a strong GNN that utilizes a random propagation strategy to perform graph data augmentation and significantly improve the node classification performance. The performance improvement brought by GRAPHPATCHER is shown in Table 4. We observe that GRAPHPATCHER can still consistently improve the node classification for GRAND. Specifically, on low-degree nodes, GRAPHPATCHER can improve 1.40, 2.23, and 4.20 accuracy score on Cora, Citeseer, and Pubmed, respectively. Overall, GRAPHPATCHER further enhances the SoTA perfor
Table 4: Effectiveness for SoTA.
Method	Cora	CiteSeer	Pubmed
LOW-DEGREE NODES (LOWER PERCENTILE)
GRAND	80.18±0.64	70.57±0.68	80.48±0.14
+GRAPHPATCHER	81.58±0.45	72.73±0.29	84.68±0.29
HIGH-DEGREE NODES (UPPER PERCENTILE)
GRAND	88.32±0.75	79.64±0.86	83.53±0.52
+GRAPHPATCHER	88.92±0.18	79.54±0.13	84.43±0.21
OVERALL PERFORMANCE
GRAND	85.22±0.80	74.90±0.77	82.30±0.41
+GRAPHPATCHER	85.90±0.44	76.10±0.38	84.20±0.26
mance on these three datasets, with an outstanding accuracy score of 85.90, 76.10, and 84.20. The significant gain from GRAPHPATCHER indicates that the effectiveness brought by the test-time augmentation is not overlapped with the data augmentation during the training.
4.5 Performance w.r.t. the Number of Patching Nodes
To investigate the necessity of patching multiple nodes, we conduct experiments over the number of patching nodes at the test time. As shown in Figure 3, we notice that the overall performance gradually increments as the number of patching nodes increases, demonstrating that multiple patching nodes are required to remedy the incomplete neighborhood of low-degree nodes. Besides, we discover that the performance of GRAPHPATCHER saturates with around four nodes patched, which aligns with our training procedure, where the length of the ego-graph sequence is at most five. Experiments concerning the number of patching nodes during the optimization and the number of sampled ego-graphs per corruption strength (i.e.,  $M$  and  $L$  in Equation (5)) can be found in Appendix B.
Figure 3: Overall perf. (y-axis) w.r.t. the number of patching nodes (x-axis).
5 Discussion w.r.t. Diffusion Models
Both diffusion models and GRAPHPACHER conduct multiple corruptions to training samples with increasing strengths and generate examples in an iterative fashion. This scheme is conceptually inspired by heat diffusion from physics. However, the motivations behind them are different, where diffusion models focus on the generation quality (i.e., fidelity to the original data distribution) but ours
aims at the results brought by our generated nodes (i.e., the performance improvement). Specifically, diffusion models [10, 32] aim at learning the probability distribution of the data and accordingly generating examples following the learned distribution. Their goal is to generate samples that follow the original data distribution, agnostic of any other factor like the target GNN we have in our scenario. Whereas for GRAPHPACHER, we aim at generating nodes to ego-nets such that the target GNN models deliver better predictions when the node degree is low. We mostly care about performance improvement and the generated node may be very different from the original nodes in the graph.
6 Discussion w.r.t. Generation Methods for Graph
Most graph generation frameworks (including those using diffusion models) explore iterative generation schemes to synthesize real graphs [58, 46, 6, 30, 16, 40]. They improve the generation quality and focus on applications such as molecule design, protein design, and program synthesis. Though GRAPHPACHER also generates patching nodes for ego-graphs, ours is a different research direction than these methods. We do not focus on whether or not the generated patching nodes are faithful to the original data distribution, as long as the low-degree performance is enhanced and the high-degree performance is maintained. Another relevant work named GPT-GNN [14] explores an iterative node generation for pre-training, which also falls under the category of maintaining the original data distribution. In summary, GRAPHPACHER is relevant to these frameworks in the sense that it generates nodes to add to ego-graphs. However, our proposal is motivated by a different reason and we aim at the performance improvement brought by generated nodes in downstream tasks.
We study the problem of degree bias underlying GNNs and accordingly propose a test-time augmentation framework, namely GRAPHPATCHER. GRAPHPATCHER iteratively patches ego-graphs with its generated virtual nodes to remedy the incomplete neighborhood. Through our designated optimization scheme, GRAPHPATCHER not only patches low-degree nodes but also maintains GNN's original superior performance over high-degree nodes. Comprehensive experiments are conducted over seven benchmark datasets and our proposal can consistently enhance GNN's overall performance by up to  $3.6\%$  and low-degree performance by up to  $6.5\%$ , outperforming all baselines by a large margin. Besides, GRAPHPATCHER can also mitigate the degree bias issue for self-supervised GNNs. When applied to graph learning methods with state-of-the-art performance (i.e., GRAND), GRAPHPATCHER can further improve the SoTA performance by a large margin, indicating that the effectiveness brought by the test-time augmentation is not overlapped with existing inductive biases.
Limitation and Broader Impact
One limitation is the additional overhead entailed by generating ego-graphs. To address this limitation, we generate all ego-graphs before the optimization to avoid duplicated computations. This operation takes more hard-disk storage, which is relatively cheap compared with computational resources. Furthermore, we observe no ethical concern entailed by our proposal, but we note that both ethical or unethical applications based on graphs may benefit from the effectiveness of our work. Care should be taken to ensure socially positive and beneficial results of machine learning algorithms.
We appreciate Shifu Hou from University of Notre Dame for valuable discussions and suggestions. We would also like to thank anonymous reviewers for their constructive suggestions and comments (i.e., experiments over heterophilic datasets, connections to diffusion models, and discussion w.r.t. iterative generation models for graphs). This work is partially supported by the NSF under grants IIS-2334193, IIS-2321504, IIS-2203262, IIS-2214376, IIS-2217239, OAC-2218762, CNS-2203261, and CMMI-2146076. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any funding agencies.