SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Botao Wang $^{1,2}$ , Jia Li $^{1,2*}$ , Yang Liu $^{1,2}$ , Jiashun Cheng $^{1,2}$ , Yu Rong $^{3}$ , Wenjia Wang $^{1,2}$ , Fugee Tsung $^{1,2}$
$^{1}$ Hong Kong University of Science and Technology, Hong Kong SAR, China  $^{2}$ Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
$^{3}$ Tencent AI Lab, Shenzhen, China
{bwangbk, yliukj, jchengak}@connect.ust.hk
{jialee, wenjiawang season}@ust.hk, yu.rong@hotmail.com
Pseudo labeling (PL) is a wide-applied strategy to enlarge the labeled dataset by self-annotating the potential samples during the training process. Several works have shown that it can improve the graph learning model performance in general. However, we notice that the incorrect labels can be fatal to the graph training process. Inappropriate PL may result in the performance degrading, especially on graph data where the noise can propagate. Surprisingly, the corresponding error is seldom theoretically analyzed in the literature. In this paper, we aim to give deep insights of PL on graph learning models. We first present the error analysis of PL strategy by showing that the error is bounded by the confidence of PL threshold and consistency of multi-view prediction. Then, we theoretically illustrate the effect of PL on convergence property. Based on the analysis, we propose a cautious pseudo labeling methodology in which we pseudo label the samples with highest confidence and multi-view consistency. Finally, extensive experiments demonstrate that the proposed strategy improves graph learning process and outperforms other PL strategies on link prediction and node classification tasks.
Pseudo Labeling (PL) [27, 13] is one of the most popular self-supervised learning approaches and has been widely used to tackle the label sparsity problem. Its core idea is to enlarge the training set by self-labeling. As most self-labeled samples should be consistent with ground truth, the enlarged dataset has a larger sample capacity to improve the model generalization. Plenty of studies have shown the effectiveness of PL [7, 26, 14, 5]. However, there is a trade-off between the benefit of PL and the effect of mislabeled samples. When the benefit of PL outweighs the impact of introduced noise, the performance of the base model (i.e., without PL) can be improved. But for non-i.i.d. condition such as graph data, the introduced noisy labels may transfer among the samples and be amplified, which may degrade the performance of base model. Although several methods have been proposed to alleviate this noisy phenomenon [31, 32, 15], there is still neither a clear explanation nor a quantification of how pseudo labeling affects the graph learning models.
In this study, we attempt to answer questions above. Specifically, we evaluate PL's effect on the prediction error of the base model and the convergence of the empirical loss function. For a graph learning model, the message aggregation process would amplify the noises of incorrect labels introduced by PL. These noises can even accumulate to damage the base model's performance. For
Figure 1: The performance of GAE, PL and proposed CPL strategy on link prediction w.r.t. Actor, WikiCS and Amazon_Photo dataset. Horizontal axis is #PL samples, and vertical axis is AUC.
example, in a two-layer graph neural network (GNN) for node classification, the mislabeled nodes can inappropriately reduce the predicted confidence of their 2-hop neighbors. Moreover, in some circumstances, such as link prediction task, the pseudo labeled sample would not only serve as the label but also as the model inputs in the consequent iterations. This characteristic further aggravates the loss function's convergence due to the noises added to adjacency matrix.
To visualize the side effect of the PL on graph data, we conduct a toy experiment as shown in Figure 1, where the benefit of PL to a popular link prediction model GAE [12] depends on the choice of graph dataset, i.e., PL improves the performance of GAE on Actor, but degrades that on WikiCS. Even worse, due to the incorrect label introduced by PL, PL leads to the model's collapse on Amazon_Photo.
To obtain a stable and consistent result by PL strategy, it is necessary to quantify the impact of introduced noisy labels and theoretically analyze how it affects the graph training procedure compared to the procedure without PL. In this paper, we build a theoretical connection between PL strategy and graph learning models with multi-view augmentations. We prove that the error bound of the PL predictor is jointly bounded by a confidence threshold and the prediction consistency over different augmentations. Moreover, we theoretically analyze that PL strategy affects convergence property by the covariance term in the optimization function. Accordingly, we propose the Cautious Pseudo Labeling (CPL) strategy for the graph learning process that maximizes the confidence threshold by committing the PL samples with high prediction probability. We evaluate the CPL on different graph learning tasks, including link prediction and node classification models, where we observe remarkable and consistent improvements over multiple datasets and base models. As shown in Figure 1, compared with the base model GAE, the average AUC improvement of CPL over three datasets is  $7.79\%$ , which clearly validates the superiority of our model.
In this section, we define the general problem of graph learning, including link prediction and node classification tasks. We give the error bound and convergence property analysis associated with the application of PL strategy. Lastly, we propose cautious PL strategy accordingly.
In graph learning, given graph  $G = (\mathcal{V}, \mathcal{E})$ ,  $\mathcal{V} = \{v_i\}$  is the node set,  $\mathcal{E} = \{(i,j)\}$  is the edge set, and  $|\mathcal{V}| = N$  is the node number. The feature matrix and adjacent matrix are denoted by  $X = [x_{ij}]_{N \times F}$  and  $A = [a_{ij}]_{N \times N}$ , respectively, where  $(i,j) \in \mathcal{E}$  if and only if  $a_{ij} \neq 0$ . A base GNN model  $g$  is trained on the graph  $G$  and the observed labels. It outputs the probability of prediction target. In this work, we adopt the PL scheme which involves a teacher model  $g_{\phi}$  and a student model  $g_{\psi}$ . The teacher model calculates confidence of the unlabeled samples and PL a subset of samples using the strategy  $\mathcal{T}$ . Subsequently, student model utilizes the enlarged set to fine-tune the base model and becomes the teacher model in the next iteration.
Node classification task. The objective of the node classification task is to predict the probabilities of an unlabeled node belonging to different classes  $g: G \to \mathbb{R}^{N \times M}$ , where  $M$  is the number of classes. The original training set comprises the labeled nodes  $\mathcal{V}_o$  and their labels  $Y_o$ . And the model aims to predict the classes of the unlabeled nodes  $\mathcal{V}_u$ . In the PL scheme, the teacher model calculates the confidence of the unlabeled nodes  $\hat{Y}_u$  and assigns pseudo labels to a selected subset of nodes
$\{\mathcal{V}_p,Y_p|\mathcal{V}_p\subset \mathcal{V}_u,Y_p\subset \hat{Y}_u\}$  . Then the student model undergoes fine-tuning on the enlarged label set  $g_{\psi}(G)\to \hat{Y}_o,\hat{Y}_o = Y_o\cup Y_p,\hat{Y}_o$  is the enlarged label set.
Link prediction task. The prediction targets are the probabilities of edges between unlabeled node pairs. In our scheme, the link prediction model  $g(g_{emb}, s)$  consists of an embedding network  $g_{emb}: G \to \mathbb{R}^{N \times D}$  and a score function  $s: \mathbb{R}^D \times \mathbb{R}^D \to \mathbb{R}$  that estimates the probability of the link, where  $D$  is the dimension of the node embeddings. A portion of edges is observed as the training graph  $G = (\mathcal{V}, \mathcal{E}_o), \mathcal{E}_o \subset \mathcal{E}$ . The teacher model predicts the confidence of the unobserved edges  $\mathcal{E}_u$  and enlarge the observed edge set  $\hat{\mathcal{E}}_o = \mathcal{E}_p \cup \mathcal{E}_o$  for fine-tuning  $g_{\psi}: G(\mathcal{V}, \hat{\mathcal{E}}_o) \to \mathbb{R}^{N \times N}$ .
It is worth to mention that in the link prediction task, the node embeddings undergo a change once pseudo labels are added, even before fine-tuning. Because the enlarged edge set is also utilized as input for the GNN.
The optimization target is formulated as minimizing  $\mathcal{L} = \mathrm{CE}(g_{\psi},Y_o) / |Y_o|$ , where  $\mathrm{CE}(\cdot)$  denotes cross entropy,  $Y_{o}$  is the ground truth label of the observed set and represents  $\mathcal{E}_o$  for the link prediction task. The overall performance is evaluated by the 0-1 loss:  $\mathrm{Err}(g) = \mathbb{E}[\arg \max (g_{\psi})\neq Y]$ .
2.2 Pseudo Labeling Error Analysis
We here present the error bound analysis of the graph learning under the PL strategy. To facilitate mathematical treatment, we introduce several assumptions.
2.2.1 Graph perturbation invariant
We assume graph perturbation invariant (GPI) property in GNN, which states that the prediction variance is linearly bounded by the difference between the augmented and original inputs.
Definition 2.1: Given a graph  $G$  and its perturbation  $\hat{G} = G(X\odot M_x,A\odot M_a)$  by the random feature masks  $M_{x}\in \{1,0\}^{N\times F}$  and adjacent matrix mask  $M_{a}\in \{1,0\}^{N\times N}$  satisfying
$$
\frac {1}{N \cdot F} \| \mathbf {1} ^ {N \times F} - M _ {x} \| _ {2} ^ {2} + \frac {1}{N ^ {2}} \| \mathbf {1} ^ {N \times N} - M _ {a} \| _ {2} ^ {2} <   \epsilon , \tag {1}
$$
the GNN  $g(\cdot)$  has GPI property if there exists a constant  $C > 0$  such that the perturbed prediction confidence satisfies  $\| g(\hat{G}) - g(G)\|_2^2 < C\epsilon$ . Here,  $\odot$  is element-wise product, and  $\| \cdot \|_2$  is the 2-norm of the vector or matrix.
The GPI property guarantees the variation of the output confidence is linearly bounded by the degree of graph perturbation, which is similar to the  $C$ -Lipschitz condition applied to GNN.
2.2.2 Additive expansion property
With Definition 2.1, if we have a convex measurable function  $f(y)$  that satisfies the  $C$ -Lipschitz condition, we can find a probability measure on the output prediction  $p_f(y)$  that satisfies the additive expansion property as stated in Proposition 2.2.
Proposition 2.2 Define a local optimal subset as  $U \subset Y$ , whose probability is higher than a threshold  $p_f(y) > 1 - q, y \in U$ , and its perturbation set  $U_{\epsilon} = \{\hat{y} = g(G) : \| \hat{y} - y \|_2 \leq C\epsilon, y \in U\}$ , where  $G \in \{\hat{G}\}$  is the space of the perturbed graph. Then, there exists  $\alpha > 1, \eta > 0$ , s.t. the probability measure  $p_f$  satisfying following additive expansion property:  $p_{\alpha f}(U_{\hat{\epsilon}} \setminus U) \geq p_{\alpha f}(U) + \eta \cdot \alpha$ .
The proposition guarantees the continuity of the measurable function in the neighborhood of the local optimal subset  $U$ . In practice,  $U$  represents the PL samples, which ideally should be close to the ground truth labels in ideal. However, the correctness of the PL samples is discrete where the continuity condition is hard to be satisfied. To address this issue, we can leverage GPI. By applying multi-view augmentations and calculating average confidence, we can reparameterize the probability measure to be continuous. In this condition, the Proposition 2.2 implies that the probability of the neighborhood under the amplified measure is greater than the original local optimum. This observation opens up potential opportunities for optimization. For a detailed proof of Proposition 2.2, please refer to Appendix A.
2.2.3 Prediction error measurement
Then, we use the additive expansion property to calculate the error bound of the multi-view GNN model under PL strategy. According to Theorem B.2 in [25], when we use 0-1 loss to measure the error of predictor, the error bound can be evaluated by the following theorem.
Theorem 2.3 Let  $q > 0$  be a given threshold. For the GNN in the teacher model  $g_{\phi}$ , if its corresponding density measure satisfies additive expansion, the error of the student predictor  $g_{\psi}$  is bounded by
$$
\operatorname {E r r} (g) \leq 2 \left(q + \mathcal {A} \left(g _ {\psi}\right)\right), \tag {2}
$$
where  $\mathcal{A}(g_{\psi}) = \mathbb{E}_{Y_{test}}[I(\exists g_{\psi}(\hat{G})\neq g_{\psi}(G))]$  measures the inconsistency over differently augmented inputs,  $Y_{test}$  is the test set for evaluation.
The brief derivation is provided in Appendix.B. From to Eq.2, we observe that the prediction error (0-1 loss) is bounded by the confidence threshold  $q$  and the expectation of the prediction inconsistency  $\mathcal{A}$ . Intuitively, if the predictions of multi-view augmented inputs tend to be consistent and the threshold is smaller, the error bound will be smaller, which implies a more accurate predictor.
If  $q$  is a small value, the threshold probability of PL  $1 - q$  approaches 1. This cautious approach in self-training leads to a smaller lower bound in the error estimate. When applying random PL, setting the confidence threshold as  $q = 0.5$ , then the maximum theoretical error rate is 1. It means that there is no guarantee on the performance after applying PL.
The inconsistency term  $\mathcal{A}$  is the expectation of the probability that predictions are inconsistent when input graphs are augmented differently. A small value of  $\mathcal{A}$  indicates consistent prediction across different views. In such cases, we have more confidence in the predictions, leading to a smaller error bound. On the other hand, if the predictions from different views are inconsistent with each other, the model lacks robustness, resulting in a larger error bound.
2.3 Convergence Analysis
In this section, we analyze the influence of the PL strategy on the empirical loss to illustrate its convergence property. First, we assume that optimizing teacher model will not influence the convergence property of PL strategy. This assumption stems from the understanding that the teacher model can converge independently without the incorporation of PL.
Assumption 2.4 The loss function defined in Algorithm 1 is non-increasing during the optimization:  $CE(g_{\psi}^{(t)}, \hat{Y}_o^{(t + 1)}) \leq CE(g_{\phi}^{(t)}, \hat{Y}_o^{(t)})$ .
Then, we show that the PL sample selection strategy  $\mathcal{T}$  influences the covariance term derived from the empirical loss, then affects the convergence property.
$$
\mathcal {L} _ {\mathcal {T}} ^ {(t + 1)} \leq \beta \operatorname {C o v} [ \operatorname {c e} (g _ {\psi}, Y), \mathcal {T} ] + \mathcal {L} _ {\mathcal {T}} ^ {(t)} \tag {3}
$$
where  $\beta = |\hat{Y}_u| / (|\hat{Y}_o| + k)$ ,  $ce(\cdot)$  is the element-wise cross entropy. The equality is achieved when the model reaches the global optimal under given training set.
The detailed derivation of the Eq.3 is shown in Appendix.C. From Eq.3, we observe that the effect of PL strategy is decoupled and encapsulated in the covariance term. The covariance sign determines whether the empirical loss will increase or decrease in the next iteration of optimization. For instance, when PL samples are randomly selected,  $\mathcal{T}$  would be independent with  $g_{\psi}$ . The covariance becomes 0, indicating that the PL does not influence the loss function. However, a carefully designed PL strategy can accelerate the optimization of empirical loss and yield improved convergence properties.
2.4 Cautious Pseudo Labeling
According to Theorem 2.3, setting a higher confidence threshold for pseudo labeling (PL) can lead to a lower prediction error. This is reflected in a positive covariance term, which accelerates the optimization of the empirical loss and improves the convergence property. In order to satisfy these requirements, we propose the iterative Cautious Pseudo Labeling (CPL) strategy. CPL involves carefully and iteratively pseudo labeling the most confident samples to maximize these improvements in prediction accuracy and convergence.
Figure 2: A framework illustration of CPL on link prediction: There is a teacher model and a student model that share parameter. The most confident samples in teacher model's prediction are pseudo labeled. Then the student model is fine-tuned on the enlarged dataset and becomes teacher model in the next iteration.
There are a teacher model and a student model in each iteration. First, we calculate multi-view prediction by the teacher model  $g_{\phi}$  as the confidence measure. Although biased, it is a simple and effective measure in many PL strategies. In node classification, the GNN directly outputs the probability as confidence  $g(G)$ . For link prediction, the confidence is determined by the inner product similarity of node embedding  $g(G) = \sigma (E^T E)$ , where  $E = (e_1,\dots,e_N)^T = g_{emb}(G)$  is the node embedding, and  $\sigma (\cdot)$  is the sigmoid function.
Then we select PL samples in unobserved set with the highest confidence, enlarging the training set. We iteratively select top- $k$  confident samples for PL:  $Y_{p}^{(t)} = \mathcal{T}(\hat{Y}_{u}^{t},\overline{g}_{\phi},k)$ , where  $\mathcal{T}\in \{0,1\}^{\left|\hat{Y}_u^{(t)}\right|},\sum \mathcal{T} = k$ .  $\overline{g}_{\phi}^{(t)}$  is the averaged confidence of the multi-view augmentation and is equal to  $g_{\phi}^{(t)}$  for the single input. At the  $t$ -th iteration, the selected PL samples are  $Y_{p}^{(t)}$ .
Then we update the observed and the unobserved set. The student model is fine-tuned by the enlarged set and becomes the new teacher model in the next iteration. We take the link prediction task as the example, whose main scheme is in Fig.2. The complete algorithm of CPL is shown in Algorithm 1.
The confidence threshold  $q(t)$  recorded in Algorithm 1 is the lowest confidence among the PL samples  $Y_{p}^{(t)}$ . At each iteration, we record the lowest confidence in these  $k$  PL samples as  $q^{(t)}$ . We update  $q$  to be  $q^{(t)}$  if  $q^{(t)}$  is smaller. Finally,  $q$  serves as the confidence threshold in Eq.2 for error analysis in Theorem 2.3.
The following theorem states the improvement on convergence property under CPL.
Theorem 2.5 Let  $\mathcal{L}_T^{(t)}$  denote the optimization target at the  $t$ -th iteration. The risk is monotonically decreasing under pseudo labeling strategy, i.e.,
$$
\mathcal {L} _ {\mathcal {T}} ^ {(t + 1)} \leq \beta \operatorname {C o v} \left[ \operatorname {c e} \left(g _ {\psi}, Y\right), \mathcal {T} \right] + \mathcal {L} _ {\mathcal {T}} ^ {(t)} \leq \mathcal {L} _ {\mathcal {T}} ^ {(t)}. \tag {4}
$$
Here  $\mathcal{L}_{\mathcal{T}}^{(t + 1)}$  is calculated by  $1 / |\hat{Y}_o^{(t + 1)}|CE(g_{\psi}^{(t)},\hat{Y}_o^{(t + 1)})$
We have demonstrated the first inequality in Eq.3. The second inequality holds because the covariance between the cross-entropy loss and PL strategy is non-positive. On one hand, each view of  $g_{\psi}$  is positively correlated to the ground-truth label  $Y$  due to the consistency regularization. Then,  $g_{\psi}$  exhibits negative correlation with the cross-entropy loss  $\mathrm{ce}(g_{\psi}, Y)$ . On the other hand, the PL strategy  $\mathcal{T}$  can be seen as a discrete sign function. In CPL, higher confidence samples are more likely to be pseudo-labeled. Therefore,  $\mathcal{T}$  is positively correlated with  $g_{\psi}$ . Consequently,  $\mathcal{T}$  is negatively correlated with the cross-entropy term, resulting in a non-positive covariance term  $\mathrm{Cov}(\cdot)$ . This guarantees the monotone decreasing behavior of the loss function, as stated in Theorem 2.5. Furthermore, since the loss function is lower bounded, Theorem 2.5 ensures the convergence of Algorithm 1.
Algorithm 1: Iterative cautious pseudo labeling.
Input: Graph  $G$  , observed and unobserved label set  $Y_{o},Y_{u}$  , iterative and total pseudo labeling number  $k,K$    
Output: Student model  $g_{\psi}(G)$  , confidence threshold  $q$    
Pre-train teacher model  $g_{\phi}^{(0)}$  on observed set  $\{G,Y_o\}$  .   
Initialize student model:  $g_{\psi}^{(0)} = g_{\phi}^{(0)}$  .   
while  $\left|\hat{Y}_o^{(t)}\right|\leq K$  do Calculate the average confidence of the unobserved set by the multi-view teacher model  $g_{\phi}$  .. Select pseudo labeling subset  $Y_{p}^{(t)}$  with top-  $k$  confidence in  $\hat{Y}_u^{(t)}$  Update set:  $\hat{Y}_o^{(t + 1)} = \hat{Y}_o^{(t)}\cup Y_p^{(t)},\hat{Y}_u^{(t + 1)} = \hat{Y}_u^{(t)}\setminus Y_p^{(t)}$  Update the current confidence threshold  $q$  according to  $q(t)$  Fine-tune the student model  $g_{\psi}^{(t)}$  by minimizing the cross entropy on  $\hat{Y}_o^{(t + 1)}$  Update teacher model  $g_{\phi}^{(t + 1)} = g_{\psi}^{(t)}$  and set  $g_{\psi}^{(t + 1)} = g_{\phi}^{(t + 1)}$ $t = t + 1$  end
Time complexity The computational complexity of CPL depends on the complexity of specific GNN models used. The key operation in CPL is the selection of top- $k$  values. For node classification, it takes  $O(N\log k)$ , while for link prediction, it takes  $O(N^2\log k)$ . However, since GNN models typically compute probabilities for all samples, the overhead introduced by CPL does not increase the complexity of existing methods significantly. For example, the complexity of GAE [12] for link prediction is  $O(|\mathcal{E}_o|D^2 + N^2 D)$  (i.e., the complexity of graph convolution operations and inner-products). Integrating CPL into GAE does not increase the complexity since both involve  $O(N^2)$  operations. In practice, the additional time consumption mainly results from the fine-tuning process.
In this section, we conduct an evaluation to assess the effectiveness of the CPL strategy on both link prediction and node classification tasks. We compare CPL with raw base models as well as other PL strategies. Then, we analyze the impact of CPL capacity, training data ratio, PL strategy, and augmentation methods. Finally, a case study is bridged to the theoretical analysis on convergence and error bound. The implementation is open-sourced at https://github.com/AcEbt/CPL.
3.1 Datasets and Benchmarks
We adopt five public available datasets to evaluate CPL strategy for link prediction, i.e. CiteSeer, Actor, WikiCS, TwitchPT, and Amazon_Photo, and five datasets for node classification, i.e. Cora, CiteSeer, PubMed, Amazon_Photo, and LastFMAxia. Detailed statistics are reported in Table 1.
In link prediction task, as there are few PL-based methods, we apply the CPL strategy on three popular models: GAE [12], node2vec [4], SEAL [29]. To reserve sufficient candidate unobserved samples for PL, the dataset is randomly split into  $10\%$ ,  $40\%$ ,  $50\%$  for training, validation, and testing.
In node classification task, we employ CPL on 4 popular base models: GCN, GraphSAGE, GAT, APPNP. CPL is compared with two other PL strategies, namely M3S[22] and DR-GST[18], using the implementation and parameters provided by the authors  $^{2}$ . We adopt the official split for the citation datasets and  $5\%$ ,  $15\%$ ,  $80\%$  split for other datasets in node classification.
We run experiments with 5 random seeds and report the mean and standard deviations of the metrics.
$^{2}$ https://github.com/BUPT-GAMMA/DR-GST
Table 1: Dataset statistics.
Dataset	Cora	CiteSeer	PubMed	Actor	WikiCS	TwitchPT	Amazon_Photo	LastFMAisa
# Nodes	2,078	3,327	19,717	7,600	11,701	1,912	7,650	7,624
# Links	10,556	9,104	88,648	30,019	216,123	64,510	238,162	55,612
# Features	1433	3,703	500	932	300	128	745	128
# Classes	7	6	3	5	10	2	8	18