SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zixing Song
New Territories, Hong Kong SAR
zxsong@cse.cuhk.edu.hk
Yifei Zhang
yfzhang@cse.cuhk.edu.hk
Irwin King
king@cse.cuhk.edu.hk
Graph Neural Networks (GNNs) are crucial for machine learning applications with graph-structured data, but their success depends on sufficient labeled data. We present a novel active learning (AL) method for GNNs, extending the Expected Model Change Maximization (EMCM) principle to improve prediction performance on unlabeled data. By presenting a Bayesian interpretation for the node embeddings generated by GNNs under the semi-supervised setting, we efficiently compute the closed-form EMCM acquisition function as the selection criterion for AL without re-training. Our method establishes a direct connection with expected prediction error minimization, offering theoretical guarantees for AL performance. Experiments demonstrate our method's effectiveness compared to existing approaches, in terms of both accuracy and efficiency.
Graph Neural Networks (GNNs) have gained significant recognition in machine learning applications, particularly for graph-structured data [71, 32, 68]. Nevertheless, their efficacy is predominantly contingent on the availability of ample labeled data. The labeling process, often demanding human intervention and domain knowledge, poses high costs in real-world applications. Active Learning (AL) presents a compelling solution to mitigate the issue of limited labeled data [1]. AL boosts the performance of passive learning by iteratively training a model based on the current set of labeled nodes and selecting the query nodes to expand this set based on the different query heuristics or the designed acquisition function. The main idea is to identify the most informative nodes for the oracle to annotate, thereby enhancing test node predictions upon their addition to the training set.
Recently, active learning methods for graph-structured data have seen an upsurge, adapting general active learning techniques to accommodate the non-IID nature of graphs [88, 89, 90]. Predominantly, these methods employ heuristic approaches, selecting nodes based on rudimentary measures such as uncertainty and influence score. While efficient, these strategies lack a direct correlation with expected prediction performance on remaining unlabeled nodes—our primary concern. Hence, We advocate for performance-based active learning methods, selecting nodes that directly optimize anticipated model performance [4]. To balance efficiency, we employ the Expected Model Change Maximization
(EMCM) algorithm [6, 7] from general active learning. EMCM chooses nodes maximizing expected model change in parameters or predictions, serving as a surrogate for expected prediction error.
Implementing the EMCM principle in GNNs is both challenging and meaningful. First, the acquisition function for EMCM calculates the expected model change following the addition of a single labeled node, without the need for re-training. Hence, a Bayesian probabilistic model is preferred in AL [2]. This approach allows for efficient updating of posterior beliefs about the model's parameters or predictions upon introducing a new labeled node, aligning perfectly with performance-based methods like EMCM. However, conventional GNNs lack Bayesian interpretation, while Bayesian GNNs impose significant computational demands, rendering precise approximation of the expected model change without re-training practically impossible. Therefore, a lightweight Bayesian probabilistic learning framework could enable the application of EMCM on graphs. Second, the direct adaptation of EMCM for GNNs, despite its computational advantages over other performance-based methods, remains impractical due to the large number of nodes, which is common in real-world cases. Third, a deep connection can be established between the EMCM method on graphs and the expected prediction error, offering a theoretical guarantee regarding the potentially optimal AL performance.
In light of these insights, we are the first to extend the EMCM principle to GNNs, contributing in the following ways. First, we revisit the training process of the Simplified Graph Convolution (SGC) model [68] for the semi-supervised node classification as an example to obtain an equivalent view of a bi-level optimization problem. This motivates us to propose a regularized single-level optimization learning framework that possesses a clear Bayesian interpretation. Second, we derive the posterior mean and variance of the node embeddings projected onto the truncated graph spectral subspace via the Laplace approximation. This enables the efficient computation of the acquisition function of EMCM for graphs through a closed-form solution. Third, we provide theoretical insights into the proposed method, essentially equating to the selection of the node that directly minimizes the expected prediction errors of the remaining unlabeled nodes, assuming its addition to the training set. This theoretical interpretation aligns with the ultimate goal of AL. Fourth, We perform comprehensive experiments across several datasets, demonstrating the efficacy and efficiency of our proposed method.
We consider a graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  with  $|\mathcal{V}| = n$  nodes. Node feature vectors are represented as  $\boldsymbol{x}_i\in \mathbb{R}^d$ , aggregated into a node feature matrix  $\mathbf{X} = [x_1^{\intercal},\dots ,x_n^{\intercal}]\in \mathbb{R}^{n\times d}$ . The adjacency matrix of  $\mathcal{G}$  is given as  $\mathbf{A}\in \{0,1\}^{n\times n}$  ( $\mathbf{A}\in \mathbb{R}^{n\times n}$  if the graph is weighted), aligned with edge set  $\mathcal{E}$ . Assume there exists a labeling oracle that can map each node  $i$  ( $i\in \mathcal{V} = \{1,\dots ,n\}$ ) to its ground-truth one-hot label vector  $\boldsymbol{y}_i\in \{0,1\}^c$ .  $c$  is the number of classes. The node set  $\mathcal{V}$  is partitioned into the labeled node set  $\mathcal{V}_l(|\mathcal{V}_l| = n_l)$  and the unlabeled node set  $\mathcal{V}_u(|\mathcal{V}_u| = n_u)$ . At first, only the ground-truth labels of labeled nodes  $\{\boldsymbol{y}_i\}_{i\in \mathcal{V}_l}$  are revealed to the learning model.
We investigate the pool-based active learning setting on the aforementioned graph (Appendix (D.1)). The learning model initially undergoes standard training on the labeled set  $\mathcal{V}_l$ ,  $\{\pmb{y}_i\}_{i\in \mathcal{V}_l}$ . Subsequently, it identifies a query node set  $\mathcal{V}_q$  from  $\mathcal{V}_u$  ( $\mathcal{V}_q\subset \mathcal{V}_u$ ) and solicits the labeling oracle for associated labels. Following this, the labeled and unlabeled sets are updated ( $\mathcal{V}_l\gets \mathcal{V}_l\cup \mathcal{V}_q$ ,  $\mathcal{V}_u\gets \mathcal{V}_u\setminus \mathcal{V}_q$ ). This training cycle and query node selection process repeats until the labeling budget  $B$  is depleted ( $|\mathcal{V}_l| = n_l + B$ ). We distinguish between sequential active learning, where one node is selected to query the oracle ( $|\mathcal{V}_q| = 1$ ), and batch active learning, where a batch of  $b$  nodes is chosen ( $|\mathcal{V}_q| = b\in \mathbb{N}^+$ ). The primary objective is to deduce the soft labels of nodes within the unlabeled set  $\{\hat{\pmb{y}}_i\}_{i\in \mathcal{V}_u}$  ( $\hat{\pmb{y}}_i\in [0,1]^c$ ), utilizing the labeling oracle under a total labeling budget  $B$ .
More specifically, in each query node selection step under the transductive setting, we aim to solve Eq. (1) to choose the most informative node(s) and minimize the expected prediction error on the remaining unlabeled nodes. Here,  $\ell \colon \mathbb{R}^c\times \mathbb{R}^c\to \mathbb{R}^+$  signifies the selected loss function.
$$
\underset {\mathcal {V} _ {q}: | \mathcal {V} _ {q} | = b} {\arg \min } \mathbb {E} _ {i \in \mathcal {V} _ {u} \backslash \mathcal {V} _ {q}} [ \ell (\hat {\boldsymbol {y}} _ {i}, \boldsymbol {y} _ {i}) ] \tag {1}
$$
2.2 Graph Neural Networks for Semi-supervised Node Classification
Graph Neural Networks (GNNs) excel in learning representations for graph-structured data [71, 62, 10, 57, 40, 95, 43, 92, 11, 61, 94, 59, 82, 60, 58, 39, 93, 9]. Among them, the Graph Convolutional Networks (GCN) [32] model is the most representative method. Considering the GCN or other GNN models for the node classification task under the semi-supervised setting, there are two main steps during one iteration: (1) Forward pass that fuses both node features  $\mathbf{X}$  and structure information  $\mathbf{A}$  into the low-dimensional representation or node embeddings  $\mathbf{U}(\boldsymbol{\Theta}) \in \mathbb{R}^{n \times c}$  (before the softmax function) with the training parameters  $\boldsymbol{\Theta}$ ; and (2) Backward pass that updates  $\boldsymbol{\Theta}$  through gradient descent according to the training loss function over all labeled nodes as  $\min_{\boldsymbol{\Theta}} \sum_{i \in \mathcal{V}_l} \ell(\mathbf{S}(\mathbf{U}_i(\boldsymbol{\Theta})), \boldsymbol{y}_i)$ . Here,  $\mathbf{U}_i(\boldsymbol{\Theta})$  denotes the  $i$ -th row of  $\mathbf{U}(\boldsymbol{\Theta})$ . Typically,  $\ell(\cdot, \cdot)$  is set as the cross-entropy loss after applying the Softmax function  $\mathbf{S}(\cdot)$  on node embedding  $\mathbf{U}_i(\boldsymbol{\Theta})$ . Different GNNs mainly have different designs in the forward pass, but the backward pass remains the same in general. For a  $K$ -layer GCN, we optimize the training loss such that  $\mathbf{U}(\boldsymbol{\Theta}) = \hat{\tilde{\mathbf{A}}} (\phi (\hat{\tilde{\mathbf{A}}} (\dots \phi (\hat{\tilde{\mathbf{A}}}\mathbf{X}\boldsymbol{\Theta}^{(0)})\dots)\boldsymbol{\Theta}^{(K - 2)}))\boldsymbol{\Theta}^{(K - 1)}$ . Here  $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$  represents the adjacency matrix with the added self-loop and  $\tilde{\mathbf{D}} = \mathbf{D} + \mathbf{I}$  with the diagonal degree matrix  $\mathbf{D} = \mathrm{diag}(d_1,\dots,d_n)$  where  $d_j = \sum_j \mathbf{A}_{i,j}$ . The normalized adjacency matrix is  $\hat{\tilde{\mathbf{A}}} = \tilde{\mathbf{D}}^{-1/2}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-1/2}$ . We denote  $\boldsymbol{\Theta} = \{\boldsymbol{\Theta}^{(i)}\}_{i=0}^{K-1}$ . Note that GCN conducts linear transformation and non-linearity activation  $\phi(\cdot)$  repeatedly. SGC [68] reduces this excess complexity by removing non-linearities and collapsing training parameters  $\boldsymbol{\Theta}$  between consecutive layers. Similarly, a  $K$ -layer SGC also optimizes the training loss such that the node embedding matrix  $\mathbf{U}(\boldsymbol{\Theta})$  is fixed as  $\mathbf{U}(\boldsymbol{\Theta}) = \hat{\tilde{\mathbf{A}}}^K\mathbf{X}\boldsymbol{\Theta}$ .
2.3 General Active Learning and Active Learning on Graphs
Active Learning (AL) [55] selects the most "informative" samples for querying, falling into three categories: uncertainty-based, representativeness-based, and performance-based methods. Uncertainty-based methods target the most uncertain instances [83, 101, 50], using criteria like entropy and margin. Techniques like Query by Disagreement (QBD) and Query by Committee (QBC) [56, 16, 52] also reside here, employing uncertainty sampling to reduce version space. Representativeness-based methods select samples best representing the input distribution, using methods such as density-weighting [14, 46, 27, 28] and clustering [49, 13, 72]. Performance-based methods directly optimize informativeness via Eq. (1) or surrogates, considering the impact of revealing an instance's label on future outcomes. Techniques like Expected Error Reduction (EER) [53], Expected Variance Reduction (EVR) [54], and Expected Model Change Maximization (EMCM) [6] are examples.
There is a recent surging trend to explore AL strategies on graph-structured data. It is unsuitable to directly apply general AL techniques to GNNs since the nodes are not i.i.d. but linked with edges such that connected nodes tend to have the same label. AGE [4] and ANRMAB [18] incorporate node embedding density and PageRank centrality into their node selection. GEEM [51] adapts the Expected Error Reduction (EER) method for graphs, while RIM [89] accounts for noisy oracles in node labeling. Furthermore, ALG [88] optimizes the effectiveness of all nodes influenced by GNNs, and IGP [90] maximizes information gain propagation. Detailed reviews are found in Appendix B.
2.4 Motivations and Challenges
2.4.1 Bayesian Probabilistic Interpretations
Active learning strategies for graphs often directly utilize GCN or other GNN models as their backbone [4, 88, 89, 90, 37], capitalizing on their exemplary representation learning abilities. However, under semi-supervised settings, these models typically lack a Bayesian probabilistic interpretation, a fundamental aspect of AL that quantifies uncertainty for uncertainty-based methods and lays the groundwork for performance-based methods, which incorporate prior knowledge and update the posterior beliefs about the model parameters or predictions, assuming if new labeled data becomes available. Relying on GCN or GNN models without Bayesian interpretations may lead to inefficient sample selection and hinder learning progress. Therefore, integrating a Bayesian probabilistic interpretation within the GNN framework is a desirable enhancement to active learning methodologies. Existing Bayesian GNNs provide well-defined Bayesian interpretations [96, 24], yet they often come with significant computational costs. This renders them impractical for active learning environments,
where model re-training is imperative after each query set selection. To address these challenges, we aim to introduce a lightweight, general Bayesian learning framework for GNN models. This will facilitate subsequent query node set selection by enabling rigorous, informed decision-making [17].
2.4.2 Efficient Computation
While numerous existing graph active learning methods extend uncertainty-based or representativeness-based methods for their simplicity and efficiency (Table 6 in Appendix B.2), these approaches can be sensitive to outliers and largely reliant on prerequisite density estimation functions or clustering algorithms. We instead advocate for performance-based methods, which directly examine the potential impact on the model should an unlabeled node's label be revealed. Well-established methods such as EER [53] and EVR [54] and their graph-based adaptations [51] are notorious for their extremely high computational overhead. As an alternative, we concentrate on the EMCM concept for AL on graphs. This approach selects nodes that yield the maximum expected model change, striking a balance between theoretical interpretations and computational costs. Our proposed general Bayesian learning framework for GNN models enables efficient computation of expected model changes using closed-form solutions. For graphs with an exceptionally large number of nodes, we further employ a spectral approximation technique to curtail computational costs.
2.4.3 Solid Theoretical Guarantees
A significant limitation of most current AL techniques for graphs is the absence of theoretical connections with the expected prediction errors in Eq. (1), the ultimate goal in AL. CSAL [65] offers an in-depth theoretical analysis of label complexity within the context of AL on graphs, but it only verifies the existence of such an algorithm without presenting a concrete implementation. This gap between theory and practice hinders the direct applicability of CSAL. To deploy AL algorithms in real-world situations, it is crucial to devise methods rooted in rigorous theoretical foundations that address this limitation, thereby enabling more efficient learning with fewer labeled examples.
Our starting point is to propose a Bayesian probabilistic learning framework for semi-supervised node classification task based on GNNs. Instead of following the existing literature [88, 90, 37] that use GNNs directly as the backbones for graph AL, our Bayesian probabilistic learning framework naturally supports the dynamic process of AL and allows for seamless updates of the model as new labeled data become available, thus eliminating the need for full model re-training during node selection. This key advantage of our Bayesian probabilistic learning framework paves the way for efficient and interpretable design of our subsequent acquisition function. Following the principle of Expected Model Change Maximization (EMCM) [6] in general AL, we can adjust the likelihood and efficiently update posterior beliefs about the expected change in model parameters, assuming the label of a candidate node from the pool becomes available. More discussions are found in Appendix C.
3.1 A Bayesian Probabilistic Learning Framework for GNNs under Semi-supervised Setting
3.1.1 Optimization View of SGC
We choose the SGC model [68] as an example for derivation and present a Bayesian probabilistic learning framework for the semi-supervised node classification. Recall that in Sec. 2.2, SGC aims to solve the following optimization problem Eq. (2) for semi-supervised node classification.
$$
\min  _ {\boldsymbol {\Theta}} \sum_ {i \in \mathcal {V} _ {l}} \ell (\mathbf {S} (\mathbf {U} _ {i} (\boldsymbol {\Theta})), \boldsymbol {y} _ {i}), \quad \text {s . t .} \quad \mathbf {U} (\boldsymbol {\Theta}) = \hat {\tilde {\mathbf {A}}} ^ {K} \mathbf {X} \boldsymbol {\Theta}. \tag {2}
$$
The objective function in Eq. (2) characterizes the backpropagation process, while the constraint in Eq. (2) describes forward propagation. Different GNNs typically exhibit unique forms of node embeddings  $\mathbf{U}(\Theta)$  in Eq. (2). Notably, several recent studies [42, 102] have focused on unifying the forward pass of GNNs within an optimization framework, but these approaches neglect the backward pass for GNN training. Some research works integrate label information into the unifying framework for GNNs [84, 74], yet none of them provide clear Bayesian interpretations. However, we can still transform the constraint in Eq. (2) into another optimization subproblem [102] based on Theorem 3.1.
Theorem 3.1. The forward pass of a  $K$ -layer SGC,  $\mathbf{U}(\Theta) = \hat{\tilde{\mathbf{A}}}^K\mathbf{X}\Theta$ , optimizes the following problem by performing  $K$  steps of gradient descent:
$$
\hat {\boldsymbol {\Theta}} ^ {*} = \underset {\hat {\boldsymbol {\Theta}}} {\arg \min } \operatorname {T r} (\mathbf {U} ^ {\mathsf {T}} \tilde {\mathbf {L}} \mathbf {U}), \quad s. t. \quad \mathbf {U} = \mathbf {X} \hat {\boldsymbol {\Theta}}.
$$
Here,  $\hat{\Theta}$  is initialized as  $\hat{\Theta}^{(0)} = \Theta$  and  $\tilde{\mathbf{L}} = \mathbf{I} - \hat{\tilde{\mathbf{A}}}$  is the normalized Laplacian matrix. Then,
$$
\mathbf {U} (\boldsymbol {\Theta}) = \hat {\tilde {\mathbf {A}}} ^ {K} \mathbf {X} \boldsymbol {\Theta} = \mathbf {X} \hat {\boldsymbol {\Theta}} ^ {*}
$$
Therefore, Eq. (2) now becomes a bi-level optimization problem as Eq. (3).
$$
\min  _ {\boldsymbol {\Theta}} \sum_ {i \in \mathcal {V} _ {l}} \ell (\mathbf {S} (\mathbf {U} _ {i} (\boldsymbol {\Theta})), \boldsymbol {y} _ {i}), \quad \text {s . t .} \quad \mathbf {U} (\boldsymbol {\Theta}) = \mathbf {X} \hat {\boldsymbol {\Theta}} ^ {*}, \hat {\boldsymbol {\Theta}} ^ {*} = \underset {\hat {\boldsymbol {\Theta}}, \mathbf {U} = \mathbf {X} \hat {\boldsymbol {\Theta}}, \hat {\boldsymbol {\Theta}} ^ {(0)} = \boldsymbol {\Theta}} {\arg \min } \operatorname {T r} (\mathbf {U} ^ {\intercal} \tilde {\mathbf {L}} \mathbf {U}). \quad (3)
$$
3.1.2 From Optimization View to Bayesian Probabilistic View
Although Eq. (3) is an equivalent optimization view of SGC for semi-supervised node classification, it does not possess a clear Bayesian interpretation. The upper-level objective in Eq. (3) minimizes the supervision loss, and the lower-level constraint  $\mathrm{Tr}(\mathbf{U}^{\top}\tilde{\mathbf{L}}\mathbf{U})$  minimizes the graph regularization loss, promoting the homophily assumption that connected nodes share the similar embeddings/labels. Therefore, motivated by [23], we absorb the lower-level constraint into the objective function itself as a relaxed single-level optimization problem in Eq. (4) by minimizing both two terms simultaneously.
$$
\underset {\Theta} {\min } _ {\mathbf {U} (\boldsymbol {\Theta}) = \mathbf {X} \boldsymbol {\Theta}} \mathcal {L} = \sum_ {i \in \mathcal {V} _ {l}} \ell \left(\mathbf {S} \left(\mathbf {U} _ {i} (\boldsymbol {\Theta})\right), \boldsymbol {y} _ {i}\right) + \operatorname {T r} \left(\mathbf {U} (\boldsymbol {\Theta}) ^ {\top} \tilde {\mathbf {L}} _ {\lambda} \mathbf {U} (\boldsymbol {\Theta})\right), \tag {4}
$$
where  $\tilde{\mathbf{L}}_{\lambda} = \tilde{\mathbf{L}} + \lambda \mathbf{I}$  with  $\lambda$  as the balancing factor between two terms. The proposed optimization problem Eq. (4) immediately leads to a Bayesian probabilistic interpretation [67]. For notational simplicity, we focus on the detailed derivation of the binary case in the main body. In the special case of binary node classification,  $\mathbf{U}(\boldsymbol{\Theta}) \in \mathbb{R}^{n \times c}$  can be reduced to  $\mathbf{u}(\boldsymbol{\Theta}) \in \mathbb{R}^n$ , and  $\mathbf{S}(\cdot)$  now becomes the Sigmoid function  $\sigma(\cdot)$  in Eq. (5).
$$
\min  _ {\boldsymbol {\Theta}} \mathcal {L} = \sum_ {i \in \mathcal {V} _ {l}} \ell \left(\sigma \left(u _ {i} (\boldsymbol {\Theta})\right), y _ {i}\right) + \operatorname {T r} \left(\mathbf {u} (\boldsymbol {\Theta}) ^ {\intercal} \tilde {\mathbf {L}} _ {\lambda} \mathbf {u} (\boldsymbol {\Theta})\right), \tag {5}
$$
Note that now  $u_{i}(\Theta) \in \mathbb{R}$  and  $y_{i} \in \{0,1\}$ . Then Theorem 3.2 explicitly reveals the Bayesian probabilistic interpretation of the optimization framework in Eq. (5) [67].
Theorem 3.2. Solving Eq. (5) is equivalent to finding the maximum a posteriori (MAP) estimate of the posterior probability distribution with the density  $\mathbb{P}(\mathbf{u}(\boldsymbol{\Theta})\mid \mathbf{y})$  as
$$
\mathbb {P} (\mathbf {u} (\boldsymbol {\Theta}) \mid \mathbf {y}) \propto \mu (\mathbf {u} (\boldsymbol {\Theta})) \exp (- \Phi_ {\ell} (\mathbf {u} (\boldsymbol {\Theta}))). \tag {6}
$$
Here, the prior  $\mu (\mathbf{u}(\Theta))$  follows a Gaussian prior  $\mathcal{N}(\mathbf{0},\tilde{\mathbf{L}}_{\lambda}^{-1})$  and the likelihood  $\exp (-\Phi_{\ell}(\mathbf{u}(\Theta))$  is defined by the likelihood potential  $\Phi_{\ell}(\mathbf{u}(\Theta))\coloneqq \sum_{i\in \mathcal{V}_l}\ell (\sigma (u_i(\Theta)),y_i)$ $\mathbf{y} = [y_i]_{i\in \mathcal{V}_l}\in \{0,1\}^{n_l}$
The Gaussian prior denotes a prior belief over the distribution of the node embedding  $\mathbf{u}(\Theta)$  governed by the graph structure or captured by the Laplacian matrix in the covariance term. It relates to the second trace term in Eq. (5) or the forward pass of SGC. The likelihood  $(\mathbb{P}(\mathbf{y} \mid \mathbf{u}(\Theta)))$  represents the underlying assumptions about how the observed labels  $y_{i}$  are generated, determined by  $\ell(\cdot, \cdot)$ . It relates to the first supervision loss term in Eq. (5) or the backward pass for training SGC.
Theorem 3.2 provides a clear Bayesian interpretation of SGC for semi-supervised node classification. The forward pass defines a prior over node embeddings parametrized by  $\Theta$ , while the backward pass specifies the likelihood based on the supervision loss function. Together, they yield the posterior, guiding the update of node embeddings after observing limited labels. This insight enables the utilization of the EMCM method [6], which selects nodes leading maximum model change. For efficiency in query node selection, we focus on SGC in this work, despite more complex GNNs like GCN yielding a non-Gaussian prior in Eq. (6). The analysis of more complex GNNs, such as GCN, yields a non-Gaussian prior while keeping all other aspects unchanged in Theorem 3.2. However, for computational efficiency in subsequent query node selection, we adhere to the SGC model for now.
3.2 Expected Model Change Maximization for Graph Active Learning
3.2.1 Challenges for Applying EMCM on Graphs