SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zheng Zhang† Junxiang Wang‡ Liang Zhao†
†Emory University, Atlanta, GA ☑NEC Labs America, Princeton, NJ
{zheng.zhang,liang.zhao}@emory.edu
{junxiang.wang}@alumni.emory.edu
Graph Neural Networks (GNNs) have achieved great success in representing data with dependencies by recursively propagating and aggregating messages along the edges. However, edges in real-world graphs often have varying degrees of difficulty, and some edges may even be noisy to the downstream tasks. Therefore, existing GNNs may lead to suboptimal learned representations because they usually treat every edge in the graph equally. On the other hand, Curriculum Learning (CL), which mimics the human learning principle of learning data samples in a meaningful order, has been shown to be effective in improving the generalization ability and robustness of representation learners by gradually proceeding from easy to more difficult samples during training. Unfortunately, existing CL strategies are designed for independent data samples and cannot trivially generalize to handle data dependencies. To address these issues, we propose a novel CL strategy to gradually incorporate more edges into training according to their difficulty from easy to hard, where the degree of difficulty is measured by how well the edges are expected given the model training status. We demonstrate the strength of our proposed method in improving the generalization ability and robustness of learned representations through extensive experiments on nine synthetic datasets and nine real-world datasets. The code for our proposed method is available at https://github.com/rollingstonezz/Curriculum_learning_for_GNNs.
Inspired by cognitive science studies [8, 33] that humans can benefit from the sequence of learning basic (easy) concepts first and advanced (hard) concepts later, curriculum learning (CL) [2] suggests training a machine learning model with easy data samples first and then gradually introducing more hard samples into the model according to a designed pace, where the difficulty of samples can usually be measured by their training loss [25]. Many previous studies have shown that this easy-to-hard learning strategy can effectively improve the generalization ability of the model [2, 19, 15, 11, 35, 46], and some studies [19, 15, 11] have shown that CL strategies can also increase the robustness of the learned model against noisy training samples. An intuitive explanation is that in CL settings noisy data samples correspond to harder samples, and CL learner spends less time with the harder (noisy) samples to achieve better generalization performance and robustness.
Although CL strategies have achieved great success in many fields such as computer vision and natural language processing, existing methods are designed for independent data (such as images) while designing effective CL methods for data with dependencies has been largely underexplored. For example, in a citation network, two researchers with highly related research topics (e.g. machine learning and data mining) are more likely to collaborate with each other, while the reason behind a collaboration of two researchers with less related research topics (e.g. computer architecture and social science) might be more difficult to understand. Prediction on one sample impacts that of another, forming a graph structure that encompasses all samples connected by their dependencies. There are
many machine learning techniques for such graph-structured data, ranging from traditional models like conditional random field [36], graph kernels [37], to modern deep models like GNNs [29, 30, 52, 38, 49, 12, 53, 42]. However, traditional CL strategies are not designed to handle the curriculum of the dependencies between nodes in graph data, which are insufficient. Handling graph-structured data require not only considering the difficulty in individual samples, but also the difficulty of their dependencies to determine how to gradually composite correlated samples for learning.
As previous CL strategies indicated that an easy-to-hard learning sequence on data samples can improve the generalization and robustness performance, an intuitive question is whether a similar strategy on data dependencies that iteratively involves easy-to-hard edges in learning can also benefit. Unfortunately, there exists no trivial way to directly generalize existing CL strategies on independent data to handle data dependencies due to several unique challenges: (1) Difficulty in quantifying edge selection criteria. Existing CL studies on independent data often use supervised computable metrics (e.g. training loss) to quantify sample difficulty, but how to quantify the difficulties of understanding the dependencies between data samples which has no supervision is challenging. (2) Difficulty in designing an appropriate curriculum to gradually involve edges. Similar to the human learning process, the model should ideally retain a certain degree of freedom to adjust the pacing of including edges according to its own learning status. As existing CL methods for graph data typically use fixed pacing function to involve samples, they can not provide this flexibility. Designing an adaptive pacing function for handling graph data is difficult since it requires joint optimization of both supervised learning tasks on nodes and the number of chosen edges. (3) Difficulty in ensuring convergence and a numerical steady process for CL in graphs. Discrete changes in the number of edges can cause drift in the optimal model parameters between training iterations. How to guarantee a numerically stable learning process for CL on edges is challenging.
In order to address the aforementioned challenges, in this paper, we propose a novel CL algorithm named Relational Curriculum Learning (RCL) to improve the generalization ability and robustness of representation learners on data with dependencies. To address the first challenge, we propose an approach to select the edges by quantifying their corresponding difficulties in a self-supervised learning manner. Specifically, for each training iteration, we choose  $K$  easiest edges whose corresponding relations are most well-expected by the current model. Second, to design an appropriate learning pace for gradually involving more edges in training, we present the learning process as a concise optimization model, which automatically lets the model gradually increase the number  $K$  to involve more edges in training according to its own status. Third, to ensure convergence of optimizing the model, we propose an alternative optimization algorithm with a theoretical convergence guarantee and an edge reweighting scheme to smooth the graph structure transition. Finally, we demonstrate the superior performance of RCL compared to state-of-the-art comparison methods through extensive experiments on both synthetic and real-world datasets.
Curriculum Learning (CL). Bengio et al.[2] pioneered the concept of Curriculum Learning (CL) within the machine learning domain, aiming to improve model performance by gradually including easy to hard samples in training the model. Self-paced learning [25] measures the difficulty of samples by their training loss, which addressed the issue in previous works that difficulties of samples are generated by prior heuristic rules. Therefore, the model can adjust the curriculum of samples according to its own training status. Following works [18, 17, 55] further proposed many supervised measurement metrics for determining curriculums, for example, the diversity of samples [17] or the consistency of model predictions [55]. Meanwhile, many empirical and theoretical studies were proposed to explain why CL could lead to generalization improvement from different perspectives. For example, studies such as MentorNet [19] and Co-teaching [15] empirically found that utilizing CL strategy can achieve better generalization performance when the given training data is noisy. [11] provided theoretical explanations on the denoising mechanism that CL learners waste less time with the noisy samples as they are considered harder samples. Some studies [2, 35, 46, 13, 24] also realized that CL can help accelerate the optimization process of non-convex objectives and improve the speed of convergence in the early stages of training.
Despite great success, most of the existing designed CL strategies are for independent data such as images, and there is little work on generalizing CL strategies to handle samples with dependencies. Few existing attempts on graph-structured data [26, 21, 28], such as [44, 5, 45, 28], simply treat
Figure 1: The overall framework of RCL. (a) The Incremental Edge Selection module first extracts the latent node embedding by the GNN model given the current training structure, then jointly learns the node prediction label  $\mathbf{y}$  and reconstructs the input structure by a decoder. A small residual error on an edge indicates the corresponding dependency is well expected and thus can be added to the refined structure for the next iteration. (b) The iterative learning process of RCL. The model starts with an empty structure and gradually includes more edges until the training structure converges to the input structure.
nodes as independent samples and then apply CL strategies on independent data, which ignore the fundamental and unique dependency information carried by the structure in data, and thus can not well handle the correlation between data samples. Furthermore, these models are mostly based on heuristic-based sample selection strategies [5, 45, 28], which largely limit the generalizability of these methods.
Graph structure learning. Another stream of existing studies that are related to our work is graph structure learning. Recent studies have shown that GNN models are vulnerable to adversarial attacks on graph structure [7, 48]. In order to address this issue, studies in graph structure learning usually aim to jointly learn an optimized graph structure and corresponding graph representations. Existing works [9, 4, 20, 54, 31] typically consider the hypothesis that the intrinsic graph structure should be sparse or low rank from the original input graph by pruning "irrelevant" edges. Thus, they typically use pre-deterministic methods [7, 56, 9] to preprocess graph structure such as singular value decomposition (SVD), or dynamically remove "redundant" edges according to the downstream task performance on the current sparsified structure [4, 20, 31]. However, modifying the graph topology will inevitably lose potentially useful information lying in the removed edges. More importantly, the modified graph structure is usually optimized for maximizing the performance on the training set, which can easily lead to overfitting issues.
Graph neural networks (GNNs) are a class of methods that have shown promising progress in representing structured data in which data samples are correlated with each other. Typically, the data samples are treated as nodes while their dependencies are treated as edges in the constructed graph. Formally, we denote a graph as  $G = (\mathcal{V}, \mathcal{E})$ , where  $\mathcal{V} = \{v_1, v_2, \ldots, v_N\}$  is a set of nodes that  $N = |\mathcal{V}|$  denotes the number of nodes in the graph and  $\mathcal{E} \subseteq \mathcal{V} \times \mathcal{V}$  is the set of edges. We also let  $\mathbf{X} \in \mathbb{R}^{N \times b}$  denote the node attribute matrix and let  $\mathbf{A} \in \mathbb{R}^{N \times N}$  represent the adjacency matrix. Specifically,  $A_{ij} = 1$  denotes that there is an edge connecting nodes  $v_i$  and  $v_j \in \mathcal{V}$ , otherwise  $A_{ij} = 0$ . A GNN model  $f$  maps the node feature matrix  $\mathbf{X}$  associated with the adjacency matrix  $\mathbf{A}$  to the model predictions  $\hat{\mathbf{y}} = f(\mathbf{X}, \mathbf{A})$ , and get the loss  $L_{\mathrm{GNN}} = L(\hat{\mathbf{y}}, \mathbf{y})$ , where  $L$  is the objective function and  $\mathbf{y}$  is the ground-truth label of nodes. The loss on one node  $v_i$  is denoted as  $l_i = L(\hat{y}_i, y_i)$ .
As previous CL methods have shown that an easy-to-hard learning sequence of independent data samples can improve the generalization ability and robustness of the representation learner, the goal of this paper is to develop an effective CL method on data with dependencies, which is extremely difficult due to several unique challenges: (1) Difficulty in designing a feasible principle to select
edges by properly quantifying their difficulties. (2) Difficulty in designing an appropriate pace of curriculum to gradually involve more edges in training based on model status. (3) Difficulty in ensuring convergence and a numerical steady process for optimizing the CL model.
In order to address the above challenges, we propose a novel CL method named Relational Curriculum Learning (RCL). The sequence, which gradually includes edges from easy to hard, is called curriculum and learned in different grown-up stages of training. In order to address the first challenge, we propose a self-supervised module Incremental Edge Selection (IES), which is shown in Figure 1(a), to select the  $K$  easiest edges at each training iteration that are mostly expected by the current model. The details are elaborated in Section 4.1. To address the second challenge, we present a joint optimization framework to automatically increase the number of selected edges  $K$  given its own training status. The framework is elaborated in Figure 1(b) and details can be found in Section 4.2. Finally, to ensure convergence of optimization and steady the numerical process, we propose an EM-style alternative optimization algorithm with a theoretical convergence guarantee in Section 4.2 Algorithm 1 and an edge reweighting scheme to smooth the discrete edge incrementing process in Section 4.3.
4.1 Incremental Edge Selection by Quantifying Difficulties of Sample Dependencies
Here we propose a novel way to select edges by first quantifying their difficulty levels. Existing works on independent data typically use supervised metrics such as training loss of samples to quantify their difficulty level, but there exists no supervised metrics on edges. To address this issue, we propose a self-supervised module Incremental Edge Selection (IES). We first quantify the difficulty of edges by measuring how well the edges are expected from the currently learned embeddings of their connected nodes. Then the most well-expected edges are selected as the easiest edges for the next iteration of training. As shown in Figure 1(a), given the currently selected edges at iteration  $t$ , we first feed them to the GNN model to extract the latent node embeddings. Then we restore the latent node embeddings to the original graph structure through a decoder, which is called the reconstruction of the original graph structure. The residual graph  $\mathbf{R}$ , which is defined as the degree of mismatch between the original adjacency matrix  $\mathbf{A}$  and the reconstructed adjacency matrix  $\tilde{\mathbf{A}}^{(t)}$ , can be considered a strong indicator for describing how well the edges are expected by the current model. Specifically, a smaller residual error indicates a higher probability of being a well-expected edge.
With the developed self-supervised method to measure the difficulties of edges, here we formulate the key learning paradigm of selecting the top  $K$  easiest edges. To obtain the training adjacency matrix  $\mathbf{A}^{(t)}$  that will be fed into the GNN model  $f^{(t)}$ , we introduce a learnable binary mask matrix  $\mathbf{S}$  with each element  $\mathbf{S}_{ij} \in \{0,1\}$ . Thus, the training adjacency matrix at iteration  $t$  can be represented as  $\mathbf{A}^{(t)} = \mathbf{S}^{(t)} \odot \mathbf{A}$ . To filter out the edges with  $K$  smallest residual error, we penalize the summarized residual errors over the selected edges, which can be represented as  $\sum_{i,j} \mathbf{S}_{ij} \mathbf{R}_{ij}$ . Therefore, the learning objective can be presented as follows:
$$
\min  _ {\mathbf {w}} L _ {\mathrm {G N N}} + \beta \sum_ {i, j} \mathbf {S} _ {i j} \mathbf {R} _ {i j}, \tag {1}
$$
$$
s. t. \| \mathbf {S} \| _ {1} \geq K,
$$
where the first term  $L_{\mathrm{GNN}} = L(f(\mathbf{X}, \mathbf{A}^{(t)}; \mathbf{w}), \mathbf{y})$  is the node-level predictive loss, e.g. cross-entropy loss for the node classification task. The second term  $\sum_{i,j} \mathbf{S}_{ij} \mathbf{R}_{ij}$  aims at penalizing the residual errors over the edges selected by the mask matrix  $\mathbf{S}$ .  $\beta$  is a hyperparameter to tune the balance between terms. The constraint is to guarantee only the most  $K$  well-expected edges are selected.
More concretely, the value of a residual edge  $\tilde{\mathbf{A}}_{ij}^{(t)}\in [0,1]$  can be computed by a non-parametric kernel function  $\kappa (\mathbf{z}_i^{(t)},\mathbf{z}_j^{(t)})$ , e.g. the inner product kernel. Then the residual error  $\mathbf{R}_{ij}$  between the input structure and the reconstructed structure can be defined as  $\left\| \tilde{\mathbf{A}}_{ij}^{(t)} - \mathbf{A}_{ij}\right\|$ , where  $\| \cdot \|$  is commonly chosen to be the squared  $\ell_2$ -norm.
4.2 Automatically Control the Pace of Increasing Edges
In order to dynamically include more edges into training, an intuitive way is to iteratively increase the value of  $K$  in Equation 1 to allow more edges to be selected. However, it is difficult to determine an appropriate value of  $K$  with respect to the training status of the model. Besides, directly solving
Algorithm 1 Alternating Minimization Algorithm for Optimizing Equation 2
Input: Node features  $\mathbf{X}$ , adjacency matrix  $\mathbf{A}$ , stepsize  $\mu$  and hyperparameter  $\gamma$
Output: Parameters w of GNN model  $f$
1: Initialize  $\mathbf{w}^{(0)},\mathbf{S}^{(0)},\lambda$
2: while Not converged do
3:  $\mathbf{w}^{(t)} = \arg \min_{\mathbf{w}}L(f(\mathbf{X},\mathbf{A}^{(t - 1)};\mathbf{w}),\mathbf{y}) + \beta \sum_{i,j}\mathbf{S}_{ij}\left\| \tilde{\mathbf{A}}_{ij}^{(t - 1)} - \mathbf{A}_{ij}\right\| +\frac{\gamma}{2}\left\| \mathbf{w} - \mathbf{w}^{(t - 1)}\right\|$
4: Given  $\mathbf{w}^{(t)}$ , extract latent nodes embedding  $\mathbf{Z}^{(t)}$  from GNN model  $f$
5: Calculate reconstructed structure  $\bar{\mathbf{A}}_{ij}^{(t)} = \kappa (\mathbf{z}_i^{(t)},\mathbf{z}_j^{(t)})$  for all pairs of  $i,j$
6:  $\mathbf{S}^{(t)} = \arg \min_{\mathbf{S}}\beta \sum_{i,j}\mathbf{S}_{ij}\left\| \mathbf{A}_{ij} - \tilde{\mathbf{A}}_{ij}^{(t)}\right\| +g(\mathbf{S};\lambda) + \frac{\gamma}{2}\left\| \mathbf{S} - \mathbf{S}^{(t - 1)}\right\|$
7: Compute  $\mathbf{A}^{(t)} = \mathbf{S}^{(t)}\odot \mathbf{A}$
8: if  $\mathbf{A}^{(t)}\neq \mathbf{A}$  then
9: Increase  $\lambda$  by stepsize  $\mu$
0: end if
11: end while
Equation 1 is difficult since  $\mathbf{S}$  is a binary matrix where each element  $\mathbf{S}_{ij} \in \{0,1\}$ , optimizing  $\mathbf{S}$  would require solving a discrete constraint program at each iteration. To address this issue, we first relax the problem into continuous optimization so that each  $\mathbf{S}_{ij}$  can be allowed to take any value in the interval  $[0,1]$ . Note that the inequality  $\| \mathbf{S} \|_1 \geq K$  in Eqn. 1 is equivalent to the equality  $\| \mathbf{S} \|_1 = K$ . This is because the second term in the loss function would always encourage fewer selected edges by the mask matrix  $\mathbf{S}$ , as all values in the residual error matrix  $\mathbf{R}$  and mask matrix  $\mathbf{S}$  are nonnegative. Given this, we can incorporate the equality constraint as a Lagrange multiplier and rewrite the loss function as  $\mathcal{L} = L_{GNN} + \beta \sum_{i,j} \mathbf{S}_{ij} \mathbf{R}_{ij} - \lambda (\| \mathbf{S} \|_1 - K)$ . Considering that  $K$  remains constant, the optimization of the loss function can be equivalently framed by substituting the given constraint with a regularization term denoted as  $g(\mathbf{S}; \lambda)$ . As such, the overall loss function can be reformulated as:
$$
\min  _ {\mathbf {w}, \mathbf {S}} L _ {\mathrm {G N N}} + \beta \sum_ {i, j} \mathbf {S} _ {i j} \mathbf {R} _ {i j} + g (\mathbf {S}; \lambda), \tag {2}
$$
where  $g(\mathbf{S};\lambda) = \lambda \| \mathbf{S} - \mathbf{A}\|$  and  $\| \cdot \|$  is commonly chosen to be the squared  $\ell_2$ -norm. Since the training adjacency matrix  $\mathbf{A}^{(t)} = \mathbf{S}^{(t)}\odot \mathbf{A}$ , as  $\lambda \to \infty$ , more edges in the input structure are included until the training adjacency matrix  $\mathbf{A}^{(t)}$  converges to the input adjacency matrix  $\mathbf{A}$ . Specifically, the regularization term  $g(\mathbf{S};\lambda)$  controls the learning scheme by the age parameter  $\lambda$ , where  $\lambda = \lambda (t)$  grows with the number of iterations. By monotonously increasing the value of  $\lambda$ , the regularization term  $g(\mathbf{S};\lambda)$  will push the mask matrix gradually converge to the input adjacency matrix  $\mathbf{A}$ , resulting in more edges automatically involved in the training structure.
Optimization of learning objective. In optimizing the objective function in Equation 2, we need to jointly optimize parameter  $\mathbf{w}$  for GNN model  $f$  and the mask matrix  $\mathbf{S}$ . To tackle this, we introduce an EM-style optimization scheme (detailed in Algorithm 1) that iteratively updates both. The algorithm uses the node feature matrix  $\mathbf{X}$ , the original adjacency matrix  $\mathbf{A}$ , a step size  $\mu$  to control the age parameter  $\lambda$  increase rate, and a hyperparameter  $\gamma$  for regularization adjustments. Post initialization of  $\mathbf{w}$  and  $\mathbf{S}$ , it alternates between: optimizing GNN model  $f$  (Step 3), extracting latent node embeddings and reconstructing the adjacency matrix (Steps 4 & 5), refining the mask matrix using the reconstructed matrix and regularization, and results in more edges are gradually involved (Step 6), updating the training adjacency matrix (Step 7), and incrementing  $\lambda$  when the training matrix  $\mathbf{A}^{(t)}$  differs from input matrix  $\mathbf{A}$ , incorporating more edges in the next iteration.
Theorem 4.1. We have the following convergence guarantees for Algorithm 1:
- Avoidance of Saddle Points. If the second derivatives of  $L(f(\mathbf{X}, \mathbf{A}^{(t)}; \mathbf{w}), \mathbf{y})$  and  $g(\mathbf{S}; \lambda)$  are continuous, then for sufficiently large  $\gamma$ , any bounded sequence  $(\mathbf{w}^{(t)}, \mathbf{S}^{(t)})$  generated by Algorithm 1 with random initializations will not converge to a strict saddle point of  $F$  almost surely.
- Second Order Convergence. If the second derivatives of  $L(f(\mathbf{X}, \mathbf{A}^{(t)}; \mathbf{w}), \mathbf{y})$  and  $g(\mathbf{S}; \lambda)$  are continuous, and  $L(f(\mathbf{X}, \mathbf{A}^{(t)}; \mathbf{w}), \mathbf{y})$  and  $g(\mathbf{S}; \lambda)$  satisfy the Kurdyka-Lojasiewicz (KL) property [41], then for sufficiently large  $\gamma$ , any bounded sequence  $(\mathbf{w}^{(t)}, \mathbf{S}^{(t)})$  generated by Algorithm 1 with random initialization will almost surely converge to a second-order stationary point of  $F$ .
The detailed proof can be found in Appendix B.
4.3 Smooth Structure Transition by Edge Reweighting