SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This paper focuses on developing a novel CL method to improve the generalization ability and robustness of GNN models on learning representations of data samples with dependencies. The proposed method Relational Curriculum Learning (RCL) effectively addresses the unique challenges in designing CL strategy for handling dependencies. First, a self-supervised learning module is developed to select appropriate edges that are expected by the model. Then an optimization model is presented to iteratively increment the edges according to the model training status and a theoretical guarantee of the convergence on the optimization algorithm is given. Finally, an edge reweighting scheme is proposed to steady the numerical process by smoothing the training structure transition. Extensive experiments on synthetic and real-world datasets demonstrate the strength of RCL in improving the generalization ability and robustness.
This work was supported by the National Science Foundation (NSF) Grant No. 1755850, No. 1841520, No. 2007716, No. 2007976, No. 1942594, No. 1907805, a Jeffress Memorial Trust Award, Amazon Research Award, NVIDIA GPU Grant, and Design Knowledge Company (subcontract number: 10827.002.120.04). The authors acknowledge Emory Computer Science department for providing computational resources and technical support that have contributed to the experimental results reported within this paper.
[1] Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning, pages 21–29. PMLR, 2019.
[2] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
[3] Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: Fast learning with graph convolutional networks via importance sampling. In International Conference on Learning Representations, 2018.
[4] Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances in neural information processing systems, 33:19314-19326, 2020.
[5] Guanyi Chu, Xiao Wang, Chuan Shi, and Xunqiang Jiang. Cuco: Graph representation with curriculum contrastive learning. In *IJCAI*, pages 2300–2306, 2021.
[6] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Velickovic. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33:13260-13271, 2020.
[7] Hanjun Dai, Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu, and Le Song. Adversarial attack on graph structured data. In International conference on machine learning, pages 1115-1124. PMLR, 2018.
[8] Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99, 1993.
[9] Negin Entezari, Saba A Al-Sayouri, Amirali Darvishzadeh, and Evangelos E Papalexakis. All you need is low (rank) defending against adversarial attacks on graphs. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 169-177, 2020.
[10] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428, 2019.
[11] Tieliang Gong, Qian Zhao, Deyu Meng, and Zongben Xu. Why curriculum learning & self-paced learning work in big/noisy data: A theoretical perspective. *Big Data & Information Analytics*, 1(1):111, 2016.
[12] Xiaojie Guo, Shiyu Wang, and Liang Zhao. Graph neural networks: Graph transformation. Graph Neural Networks: Foundations, Frontiers, and Applications, pages 251-275, 2022.
[13] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In International Conference on Machine Learning, pages 2535-2544. PMLR, 2019.
[14] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[15] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
[51] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40-48. PMLR, 2016.
[52] Zheng Zhang and Liang Zhao. Representation learning on spatial networks. Advances in Neural Information Processing Systems, 34:2303-2318, 2021.
[53] Zheng Zhang and Liang Zhao. Unsupervised deep subgraph anomaly detection. In 2022 IEEE International Conference on Data Mining (ICDM), pages 753-762. IEEE, 2022.
[54] Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In International Conference on Machine Learning, pages 11458-11468. PMLR, 2020.
[55] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: from clean label detection to noisy label self-correction. In International Conference on Learning Representations, 2020.
[56] Daniel Zügner, Amir Akbarnejad, and Stephan Gunnemann. Adversarial attacks on neural networks for graph data. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2847-2856, 2018.
A Additional Experimental Settings and Results
A.1 Additional Experimental Settings
Figure 4: Visualization of synthetic datasets. Each color represents a class of nodes. Node attributes are sampled from overlapping multi-Gaussian distributions, where the attributes of nodes with close labels are likely to have short distances. Homogeneous edges represent edges that connect nodes of the same class (with the same color). The probability of connecting two nodes of different classes decreases with the distance between the center points of their class distribution. Therefore, the formation probability of a node denotes the edge difficulty, since edges between nodes with close classes are more likely to positively contribute to the prediction under the homogeneous assumption.
Synthetic datasets. To evaluate the effectiveness of our proposed method on datasets with ground-truth difficulty labels on structure, we first follow previous studies [22, 1] to generate a set of synthetic datasets, where the difficulty of edges in generated graphs are indicated by their formation probability. Specifically, as shown in Figure 4, each generated graph is with 5,000 nodes, which are divided into 10 equally sized node classes  $1,2,\ldots ,10$ . The node features are sampled from overlapping multi-Gaussian distributions. Each generated graph is associated with a homophily coefficient (homo) which indicates the likelihood of a node forming a connection to another node with the same label (same color in Figure 4). For example, a generated graph with homo = 0.5 will have on average half of the edges formed between nodes with the same label. For the rest edges that are formed between nodes with different labels (different colors in Figure 4), the probability of forming an edge is inversely proportional to the distances between their labels. Mathematically, the probability of forming an edge between node  $u$  and node  $v$  follows  $p_{u\rightarrow v}\propto e^{-|c_u - c_v|}$ , where the distances between labels  $|c_u - c_v|$  means shortest distance of two classes on a circle. Therefore, the probability of forming an edge in the synthetic graph can reflect how well this edge is expected. Specifically, edges with a higher formation probability, e.g. connecting nodes with the same label or close labels, meaning that there is a higher chance that this connection will positively contribute to the prediction (less chance to be a noisy edge). Conversely, edges with a lower formation probability, e.g., connecting nodes with faraway labels, mean that there is a higher chance that this connection will negatively contribute to the prediction (higher chance to be a noisy edge). We vary the value of homo from 0.1, 0.2, ..., 0.9 to generate nine graphs in total. Similar to previous works [22, 1], we randomly partition each synthetic graph into equal-sized train, validation, and test node splits.
Implementation Details. We use the baseline model (GCN, GIN, GraphSage) as the backbone model for both our RCL method and all comparison methods. For a fair comparison, we require all models follow the same GNN architecture with two convolution layers. For each split, we run each model 10 times to reduce the variance in particular data splits. Test results are according to the best validation results. General training hyperparameters (such as learning rate or the number of training
epochs) are equal for all models. For the pre-trained model to initialize the training structure, we utilize the same model as the backbone model utilized by our method. For example, if we use GCN as the backbone model for RCL, the pre-trained model to initialize is also GCN. All experiments are conducted on a 64-bit machine with four NVIDIA Quadro RTX 8000 GPUs. The proposed method is implemented with Pytorch deep learning framework [32].
The following describes the details of our comparison models.
Graph Neural Networks (GNNs). We first introduce three baseline GNN models as follows.
(i) GCN. Graph Convolutional Networks (GCN) [23] is a commonly used GNN, which introduces a first-order approximation architecture of the Chebyshev spectral convolution operator;
(ii) GIN. Graph Isomorphism Networks (GIN) [50] is a variant of GNN, which has provably powerful discriminating power among the class of 1-order GNNs;
(iii) GraphSage. GraphSage [14] is a GNN method that computes the hidden representation of the root node by aggregating the hidden node representations hierarchically from bottom to top.
Graph structure learning. We then introduce four state-of-the-art methods for jointly learning the optimal graph structure and downstream tasks.
(i) GNNSVD. GNNSVD [9] first apply singular value decomposition (SVD) on the graph adjacency matrix to obtain a low-rank graph structure and apply GNN on the obtained low-rank structure;
(ii) ProGNN. ProGNN [20] is a method to defend against graph adversarial attacks by obtaining a sparse and low-rank graph structure from the input structure;
(iii) NeuralSparse. NeuralSparse [54] is a method to learn robust graph representations by iteratively sampling  $k$ -neighbor subgraphs for each node and sparsing the graph according to the performance on the node classification;
(iv) PTDNet. PTDNet [31] learns a sparsified graph by pruning task-irrelevant edges, where sparsity is controlled by regulating the number of edges.
Curriculum learning on graph data. We introduce a recent curriculum learning work on node classification as follows.
(i) CLNode. CLNode [45] regards nodes as data samples and gradually incorporates more nodes into training according to their difficulty. They apply a heuristic-based strategy to measure the difficulty of nodes, where the nodes that connect neighboring nodes with different classes are considered difficult.
Searching space for hyperparameters.
Number of epochs trained:  $\{\tilde{1} 50,500\}$
Learning rate for model:  $\{1e - 2,5e - 3,1e - 3\}$
Number of GNN layers:  $\{2\}$ ;
Dimension of hidden state:  $\{64\}$
Age parameter  $\lambda : \{1, 2, 3, 4, 5\}$  (A larger value indicates faster pacing for adding edges, where 1 denotes the training structure will converge to the input structure at the final iteration).
A.2 Additional Effectiveness Experiments on Heterophilic Datasets
Dataset	Edge homo ratio	GCN	GCN-RCL	GIN	GIN-RCL
Texas	0.11	0.5645	0.6006	0.5885	0.6156
Cornell	0.30	0.4084	0.5045	0.4234	0.4925
Wisconsin	0.21	0.4923	0.5294	0.5141	0.5599
Actor	0.22	0.2868	0.3186	0.2678	0.3006
Squirrel	0.22	0.2743	0.2999	0.2347	0.2519
Chameleon	0.23	0.3625	0.4385	0.3233	0.4033
Table 4: Node classification results for six real-world heterophilic datasets, where the best performance of each model category in one dataset is highlighted.
In order to further verify the effectiveness of our proposed strategy on heterophilic graph datasets, we have included new experiments on six real-world heterophilic datasets. As shown in Table 4, our method consistently improve performance of backbone GNN models on these heterophilic datasets. Secifically, RCL outperforms the second best method on average by  $5.04\%$ , and  $4.55\%$ , on GCN and GIN backbones, respectively. The results can demonstrate our method is not limited to homophily graphs.
Although the inner product decoder utilized in experiments might imply an underlying homophily assumption, our method can still benefit from leveraging the edge curriculum present within the input datasets. A reasonable explanation is that standard GNN models are usually struggled with the heterophily edges, while our methodology designs a curriculum allowing more focus on homophily edges, which potentially leads to the observed performance boost.
A.3 Additional Effectiveness Experiments on PNA Backbone Model.
Dataset	PNA	PNA-RCL	PNA-linear	PNA-root	GCN	GCN-RCL	GCN-linear	GCN-root
Synthetic-0.3	0.6982	0.7667	0.7463	0.7445	0.6517	0.7398	0.6641	0.6533
Synthetic-0.5	0.8742	0.9016	0.8476	0.8704	0.8715	0.9269	0.8494	0.8854
Synthetic-0.7	0.9658	0.9821	0.9514	0.9766	0.9748	0.9962	0.9712	0.9796
Cora	0.8310	0.8521	0.8145	0.8254	0.8574	0.8715	0.8327	0.8553
CiteSeer	0.7478	0.7652	0.7482	0.7505	0.7893	0.7979	0.7723	0.7814
Computers	0.8989	0.9096	0.8866	0.8975	0.8809	0.9023	0.8713	0.8985
ogbn-arxiv	0.7175	0.7441	0.6980	0.7242	0.7174	0.7408	0.7288	0.7359
Table 5: Node classification results for our method and traditional CL methods using PNA and GCN as backbone. Here 'RCL' denotes our method, while '-linear' and '-root' denotes two traditional CL methods with different pacing functions.
In Table 5, new experiments that adopt modern GNN architecture - PNA model [6] have been added. From the table we can observe that our proposed method improves the performance of PNA backbone by  $2.54\%$  on average, which further verified the effectiveness of our method under different choices of backbone GNN model.
In addition, in Table 5 we further include two traditional CL methods for independent data as additional baselines, following classical works [2, 25]. We employed the supervised training loss of a pretrained GNN model as the difficulty metric, and selected two well-established pacing functions for curriculum design: linear and root pacing, defined as follows:
$$
\mathrm {L i n e a r :} K _ {\mathrm {l i n e a r}} (t) = \frac {t}{T} | V |; \mathrm {R o o t :} K _ {\mathrm {r o o t}} (t) = \sqrt {\frac {t}{T}} | V |,
$$
where  $t$  is the number of current iterations and  $T$  is the number of total iterations, and  $|V|$  is the number of nodes.
We utilized GCN and PNA as backbone architectures, identified by the suffixes '-linear' and '-root'. Across all datasets, the results consistently demonstrate that our proposed method outperforms traditional CL approaches.
A.4 Additional Robustness Experiments on PNA Backbone Model.
We present further robustness test against random noisy edges by using the PNA backbone model. The results are shown in Table 6, which further proves that our curriculum learning approach improves the robustness against edge noise with the advanced PNA model as the backbone.
A.5 Time Complexity Analysis
Here we consider GCN as the backbone. First, the time complexity of an  $L$ -layer GCN is  $O(L|\mathcal{E}|b + L|\mathcal{V}|b^2)$ , where  $b$  is the number of node attributes. Second, the time complexity of measuring the
Dataset	Method	0%	10%	20%	30%	40%	50%	60%	70%	80%	90%
Cora	PNA	0.8310	0.7911	0.7621	0.7402	0.7331	0.7210	0.6894	0.7042	0.6792	0.6617
Cora	PNA-RCL	0.8521	0.8315	0.8162	0.7969	0.7992	0.7951	0.7571	0.7642	0.7457	0.7371
Citeseer	PNA	0.7478	0.7195	0.7184	0.6934	0.6952	0.6920	0.6852	0.6552	0.6481	0.6327
Citeseer	PNA-RCL	0.7652	0.7422	0.7222	0.7254	0.7041	0.7012	0.6953	0.6921	0.6884	0.6794
Table 6: Further robustness test using PNA as backbone model. Here the percentage denotes the ratio of number of added random edges to the original edges.
Synthetic	CiteSeer	Computers	ogbn-arxiv	ogbn-proteins
Vanilla	7.32s	3.90s	16.88s	55.22s	1438.23s
GNNSVD	11.49s	3.82s	35.96s	135.72s	2632.42s
CLNode	6.29s	3.96s	17.02s	58.53s	1545.53s
ProGNN	220.25s	72.42s	1953.23s	(-)	(-)
NeuralSparse	310.02s	88.91s	6553.34s	(-)	(-)
PTDNet	153.43s	48.42s	2942.02s	(-)	(-)
Ours	4.07s	2.42s	14.62s	71.49s	2239.05s
Table 7: Running time of our method and comparison methods. Here (-) denotes an out-of-memory error and Vanilla denotes the standard GNN model.
difficulty levels of edges by reconstruction is  $O(|\mathcal{E}|d)$  where  $d$  is the number of latent embedding dimensions. Third, the time complexity of selecting the edges to add is  $O(|\mathcal{E}|)$ . Therefore, the total time complexity of our algorithm is  $O(|\mathcal{E}|(Lb + d) + L|\mathcal{V}|b^2)$ .
In addition, we compare the total running time of our method and all comparison methods in the Table 7. We can observe that the running time of our proposed method is comparable to that of standard GNN models in all datasets. Notably, our method is even faster than standard GNN models in some datasets. One possible reason is that at the beginning of training, the graphs in our model have much fewer edges than those in standard GNN models. Therefore, the computational cost of the GNN model is also reduced.
A.6 Parameter Sensitivity Analysis
Recall that RCL learns a curriculum to gradually add edges in a given input graph structure to the training process until all edges are included. An interesting question is how the speed of adding edges will affect the performance of the model. Here we conduct experiments to explore the impact of age parameter  $\lambda$  which controls the speed of adding edges to the model performance. Here a larger value of  $\lambda$  means that the training structure will converge to the input structure earlier. For example,  $\lambda = 1$  means that the training structure will probably not converge to the input structure until the last iteration, and  $\lambda = 5$  means that the training structure will converge to the input structure around half of the iterations are complete, and then the model will be trained with the full input structure for the remaining iterations. We present the results on two synthetic datasets (homophily coefficient  $= 0.3$ , 0.6) and two real-world datasets in Figure 5. As can be seen from the figure, the classification results are steady that the average standard deviation is only  $0.41\%$ . It is also worth noting that the peak values for all datasets consistently appear around  $\lambda = 3$ , which indicates that the best performance is when the training structure converges to the full input structure around two-thirds of the iterations are completed.
Figure 5: Parameter sensitivity analysis on four datasets. Here a larger value of  $\lambda$  means the training structure will converge to the original structure at an earlier training stage.
A.7 Visualization of Importance on Smoothing Component
Figure 6: The comparison between our full model and the version without smoothing technique on the training loss trend.
Our experimental results demonstrated the importance of applying our smoothing component in stabilizing the optimization process of training. Figure 6 shows that without the smoothing technique, the training loss spiked that reflects the GNN parameter shifts, which was caused by the number of edges discretely changed. However, after adding the smoothing technique, the training loss can smoothly converge, hence, the smoothing technique plays an important role in stabilizing the training process.
B Mathematical Proof
Theorem 1. We have the following convergence guarantees for Algorithm 1:
- Avoidance of Saddle Points If the second derivatives of  $L(f(\mathbf{X}, \mathbf{A}^{(t)}; \mathbf{w}), \mathbf{y})$  and  $g(\mathbf{S}; \lambda)$  are continuous, then for sufficiently large  $\gamma$ , any bounded sequence  $(\mathbf{w}^{(t)}, \mathbf{S}^{(t)})$  generated by Algorithm 1 with random initializations will not converge to a strict saddle point of  $F$  almost surely.
- Second Order Convergence If the second derivatives of  $L(f(\mathbf{X}, \mathbf{A}^{(t)}; \mathbf{w}), \mathbf{y})$  and  $g(\mathbf{S}; \lambda)$  are continuous, and  $L(f(\mathbf{X}, \mathbf{A}^{(t)}; \mathbf{w}), \mathbf{y})$  and  $g(\mathbf{S}; \lambda)$  satisfy the Kurdyka-Lojasiewicz (KL) property [40], then for sufficiently large  $\gamma$ , any bounded sequence  $(\mathbf{w}^{(t)}, \mathbf{S}^{(t)})$  generated by Algorithm 1 with random initialization will almost surely converge to a second-order stationary point of  $F$ .
Proof. We prove this theorem by Theorem 10 and Corollary 3 from [27].
[Avoidance of Saddle Points] Because the sequence  $(\mathbf{w}^{(t)},\mathbf{S}^{(t)})$  is bounded, and the second derivatives of  $L$  and  $g$  are continuous, then they are bounded. In other words, we have  $\max \{\| \nabla_{\mathbf{w}}^2 L(f(\mathbf{X},\mathbf{A}^{(t)};\mathbf{w}^{(t)}),\mathbf{y})\| ,\| \nabla_{\mathbf{S}}^2 g(S^{(t)};\lambda)\| \} \leq p$ , where  $p > 0$  is a constant. Similarly, it is easy to check that the second derivative of the term  $\sum_{i,j}\mathbf{S}_{ij}\left\| \tilde{\mathbf{A}}_{ij}^{(t)} - \mathbf{A}_{ij}\right\| _2^2$  is bounded, i.e.,  $\max \{\left\| \nabla_{\mathbf{w}}^2\sum_{i,j}\mathbf{S}_{ij}\left\| \tilde{\mathbf{A}}_{ij}^{(t)} - \mathbf{A}_{ij}\right\| _2^2\right\| ,\left\| \nabla_{\mathbf{S}}^2\sum_{i,j}\mathbf{S}_{ij}\left\| \tilde{\mathbf{A}}_{ij}^{(t)} - \mathbf{A}_{ij}\right\| _2^2\right\|\} \leq q$ , where  $q > 0$  is constant and  $\tilde{\mathbf{A}}$  is a function of  $\mathbf{w}$ . Therefore, it means that the objective  $F$  is bi-smooth, i.e.  $\max \{\| \nabla_{\mathbf{w}}^2 F\|\}$ ,  $\| \nabla_{\mathbf{S}}^2 F\|\} \leq p + q$ . In other words,  $F$  satisfies Assumption 4 from [27]. Moreover, the second derivative of  $F$  is continuous. For any  $\gamma >p + q$ , any bounded sequence  $(\mathbf{w}^{(t)},\mathbf{S}^{(t)})$  generated by Algorithm 1 will not converge to a strict saddle of  $F$  almost surely by Theorem 10 from [27].
[Second Order Convergence] From the above proof of avoidance of saddle points, we know that  $F$  satisfies Assumption 4 from [27]. Moreover, because  $L$  and  $g$  satisfy the KL property, and the term  $\sum_{i,j}\mathbf{S}_{ij}\left\| \tilde{\mathbf{A}}_{ij}^{(t)} - \mathbf{A}_{ij}\right\| _2^2$  satisfies the KL property, we conclude that  $F$  satisfy the KL property as
well. From the proof above, we also know that the second derivative of  $F$  is continuous. Because continuous differentiability implies Lipschitz continuity [47], it infers that the first derivative of  $F$  is Lipschitz continuous. As a result,  $F$  satisfies Assumption 1 from [27]. Because  $F$  satisfies Assumptions 1 and 4, then for any  $\gamma > p + q$ , any bounded sequence  $(\mathbf{w}^{(t)}, \mathbf{S}^{(t)})$  generated by Algorithm 1 will almost surely converge to a second-order stationary point of  $F$  by Corollary 3 from [27].
While the convergence of Algorithm 1 entails the second-order optimality conditions of  $f$  and  $g$ , some commonly used  $f$  such as the GNN with sigmoid or tanh activations and some commonly used  $g$  such as the squared  $\ell_2$  norm satisfy the KL property [39, 40], and Algorithm 1 is guaranteed to avoid a strict saddle point and converges to a second-order stationary point.