SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Performance (AUV%) comparison on link prediction.
Model	CiteSeer	Actor	WikiCS	TwitchPT	Amazon_Photo
AUC(%)	GAE	71.10 ± 0.56	55.34 ± 0.57	90.81 ± 0.69	74.48 ± 3.03	67.92 ± 1.31
GAE+CPL	72.45 ± 0.24	65.58 ± 1.04	95.56 ± 0.24	79.67 ± 3.77	76.30 ± 1.84
node2vec	52.03 ± 0.60	53.30 ± 0.59	88.82 ± 0.28	79.46 ± 0.77	89.32 ± 0.21
node2vec+CPL	55.22 ± 1.63	65.11 ± 2.31	91.99 ± 0.26	84.76 ± 3.52	89.53 ± 0.30
SEAL	63.60 ± 0.01	73.41 ± 0.02	86.01 ± 0.04	87.80 ± 0.01	76.96 ± 0.17
SEAL+CPL	64.33 ± 0.14	73.54 ± 0.01	86.83 ± 0.07	87.87 ± 0.01	78.86 ± 0.01
AP(%)	GAE	72.12 ± 0.63	53.60 ± 1.06	90.58 ± 0.71	69.73 ± 5.06	67.06 ± 0.99
GAE+CPL	73.54 ± 0.20	67.65 ± 1.06	95.58 ± 0.29	79.09 ± 5.48	75.52 ± 4.23
node2vec	52.90 ± 0.36	55.43 ± 0.62	92.54 ± 0.51	83.37 ± 0.52	91.46 ± 0.18
node2vec+CPL	56.19 ± 1.60	68.33 ± 2.85	93.66 ± 0.29	85.87 ± 2.15	91.47 ± 0.21
SEAL	64.38 ± 0.01	73.17 ± 0.12	83.63 ± 0.16	87.69 ± 0.01	73.72 ± 0.56
SEAL+CPL	64.94 ± 0.14	73.44 ± 0.02	86.72 ± 0.12	87.75 ± 0.02	80.36 ± 0.09
3.2 Performance Comparison
3.2.1 Overall performance comparison
For link prediction, Table 2 lists the AUC and AP of raw baselines and ones with CPL employed. We observe that CPL distinctively increases the performance of baseline models in nearly all cases. Note that the CPL strategy can achieve performance gain under the circumstances of both high and low performance (e.g., the AUC of GAE on Actor improves from 55.34 to 65.58 and the AP of SEAL on CiteSeer improves from 64.38 to 64.94).
For node classification, Table 3 shows the improvement on the base models and comparison with other PL strategies. The CPL strategy can consistently improve the performance of the base models. However, other PL strategy may be ineffective or degrade the base model (e.g., DR-GST on Cora with APPNP, M3S for PubMed with GraphSAGE). CPL also consistently outperforms other PL strategies.
The time consumption is also reported in AppendixD.
3.2.2 Impact of pseudo labeling capacity
In the experiment, the number of PL samples in each iteration  $k$  is set from 100 to 1000. We provide a reference interval for the selection of  $k$ . Intuitively, a small  $k$  can lead to a reliable result. But a too small  $k$  will unnecessarily increase the training time. And it does not prominently influence the overall performance as we test on different datasets. In the experiment, the predicted confidence distribution is around 1, as there are usually plenty of potential PL samples.  $k$  is much smaller than the total number of unobserved samples. Then the  $k$ -th highest confidence in the unobserved set will not have much difference, as the cases in Table 4.
3.2.3 Impact of training data ratio
PL enlarges the observed dataset and introduces extra information for the training. This effect has a different degree of contribution on different training data. When the training set is partially applied for the training, the ratio of the observed set ranges from 0.1 to 0.9. The variation of AUC and AP on the Amazon_Photo dataset is shown in Fig.3. The CPL method can consistently improve the performance of the raw model even starting from a small training set. It is also worth to mention that CPL is more likely to have a more significant contribution when the training set is small, as there is more introduced information.
Table 3: Performance (AUV%) comparison on node classification.
Model	Cora	CiteSeer	PubMed	Amazon_Photo	LastFMArea
GCN	Raw	80.74 ± 0.27	69.32 ± 0.44	77.72 ± 0.46	92.62 ± 0.45	78.53 ± 0.60
M3S	80.92 ± 0.74	72.70 ± 0.43	79.36 ± 0.64	93.07 ± 0.25	79.49 ± 1.42
DR-GST	83.54 ± 0.81	72.04 ± 0.53	77.96 ± 0.25	92.89 ± 0.16	79.31 ± 0.55
CPL	83.94 ± 0.42	72.96 ± 0.22	79.98 ± 0.92	93.15 ± 0.24	79.92 ± 0.61
GraphSAGE	Raw	81.12 ± 0.32	69.80 ± 0.19	77.52 ± 0.38	92.46 ± 0.17	80.23 ± 0.28
M3S	83.02 ± 0.49	70.98 ± 2.14	79.12 ± 0.25	92.41 ± 0.14	81.48 ± 0.56
DR-GST	81.02 ± 1.99	72.28 ± 0.35	76.96 ± 0.43	92.58 ± 0.14	81.10 ± 0.30
CPL	84.62 ± 0.19	73.14 ± 0.21	79.72 ± 0.72	92.90 ± 0.20	82.25 ± 0.25
GAT	Raw	81.28 ± 0.87	71.18 ± 0.43	77.34 ± 0.34	93.26 ± 0.31	81.12 ± 0.58
M3S	82.28 ± 0.95	71.70 ± 0.72	79.20 ± 0.21	93.71 ± 0.16	81.82 ± 0.93
DR-GST	83.32 ± 0.31	72.64 ± 0.97	78.28 ± 0.32	93.60 ± 0.13	81.86 ± 0.50
CPL	83.86 ± 0.22	73.02 ± 0.37	79.62 ± 0.31	93.72 ± 0.29	82.89 ± 0.56
APPNP	Raw	82.52 ± 0.69	70.82 ± 0.24	79.96 ± 0.50	93.05 ± 0.29	82.40 ± 0.50
M3S	82.54 ± 0.40	72.58 ± 0.45	79.98 ± 0.14	93.21 ± 0.59	83.55 ± 0.71
DR-GST	82.46 ± 0.87	72.64 ± 0.54	80.00 ± 0.48	93.12 ± 0.32	82.88 ± 0.35
CPL	84.20 ± 0.42	74.22 ± 0.24	80.62 ± 0.24	93.48 ± 0.23	83.56 ± 0.53
Figure 3: The effect of training ratio on Amazon_Photo w.r.t. AUC and AP: the relative improvement is more significant when the training ratio is small (i.e., the observed graph is sparse).
Raw PL
Figure 4: The effect of different data augmentation strategy on Actor w.r.t. AUC and AP: the improvement of multi-view is largest compared to other methods.
Random  
- - - NodeView  
- - - FeatureView  
- - - StructureView  
- Multiview
3.2.4 Impact of noisy pseudo labeling strategy
As the noise introduced by PL has a large effect in the graph, we compare the CPL with the PL strategy. In PL, the samples are selected randomly in the unobserved set, whose labels are estimated by the teacher model. The comparison of AUC's variation on different datasets is shown in Fig.1. We discover that PL has different effects on the baseline models. On Actor, the PL consistently improves the link predictor, but not as much as the CPL. On WikiCS however, PL keeps weakening the AUC, as the introduced noise outweighs. On Amazon_Photo, PL can improve the performance at the first few iterations. But the AUC suddenly drops from  $70\%$  to around  $50\%$  after iterations, which illustrates that introducing too much noise may degrade the model. As for the CPL, although it may drop after reaching its best, i.e. for WikiCS, it can distinctively improve the performance compared to the base model and will not lead to failure.
3.2.5 Impact of multi-view augmentation
During the training, we apply multi-view augmented graphs as the input of the teacher model, so that the continuity condition in Theorem 2.3 is easier to be satisfied. We here conduct the ablation experiments that compares multi-view augmentation with different single-view augmentation methods, including drop node (Node view), feature mask (Feature view), and DropEdge (Structure view). In each experiment, a single augmentation method was applied three times. And "Random" refers to the random selection of samples during PL. As shown in Fig.4, all of the single-view augmentation methods have better performance than the base model and their improvements are almost the same. When we use multi-view augmentation, the AUC and AP are further improved. It illustrates that multi-view augmentation contributes to a more robust graph learning and tends to obtain a consistent result, which echoes with our theoretical analysis.
Table 4: CPL with different  $k$  .
k	CiteSeer
Raw	100	500	2000
AUC(%)	71.10	72.25	72.45	71.98
AP(%)	72.45	73.29	73.54	73.13
Amazon_Photo
k	Raw	5	50	500
AUC(%)	67.92	76.63	76.30	75.95
AP(%)	67.06	75.70	75.52	74.97
Table 5: Case analysis of CPL on LastFMAxia.
GCN	GraphSAGE	GAT	APPNP
Inconsistency A (%)	6.69	4.01	2.96	3.13
Confidence 1-q (%)	77.63	88.35	83.00	86.86
Theoretical Errth (%)	58.12	31.32	39.92	32.54
Experimental Errexp (%)	20.08	17.75	17.11	16.44
PL error (%)	7.78	6.43	8.18	6.02
M3S PL error (%)	65.63	27.00	65.00	27.49
DR-GST PL error (%)	26.31	14.10	13.38	29.09
3.3 Case Study
The case study on node classification is conducted on LastFMAisa on different baselines. The detailed intermediate variables about the error analysis are listed in Table 5. We record the inconsistency  $\mathcal{A}$  and confidence  $1 - q$  during the CPL as Algorithm.1. Then the theoretical error bound  $\mathrm{Err}_{th}(g)$  is calculated by Theorem 2.3. As  $\mathrm{Err}_{th}(g) > \mathrm{Err}_{exp}(g)$ , the experimental error is bounded by the theoretical error, implying that it is an effective bound. We also report the accuracy of PL samples. The CPL has smaller error than other PL strategies and is consistently better than  $\mathrm{Err}_{exp}(g)$ .
The case study on link prediction is conducted based on GAE on WikiCS. The relationship between consistency  $1 - \mathcal{A}(g)$  and the number of PL samples is shown in Fig.5. The result shows that more PL samples help to increase the consistency of the prediction. We also compare the optimization target of PL  $\mathcal{L}_{\mathcal{R}}^{(t)}$  and CPL  $\mathcal{L}_{\mathcal{T}}^{(t)}$  in Fig.5. We discover that the optimization target of CPL converges faster than PL and is consistently smaller. It
implies the covariance term  $\operatorname{Cov}[\mathrm{ce},\mathcal{T}]$  is negative as illustrated in Theorem 2.5. Thus, the CPL strategy can improve the convergence property of graph learning. A detailed illustration of error bound verification and Knowledge discovery is shown in Appendix E.
Figure 5: Case study of the consistency and convergence on WikiCS: Cautious PL (CPL) improves the prediction consistency within error bound, and converges faster than PL.
Pseudo labeling is a popular approach in self-training (ST), aiming to enlarge the training set by self-annotation. Most studies focus on learning a more accurate PL algorithms to avoid noisy samples. Confidence thresholding is a simple yet effective method in ST, where the similarity with ground truth or consistency is used to measure the confidence, ensuring that flawed samples are excluded from the enlarged dataset [9, 20]. [6] uses the perturbation on the hidden states to yield close predictions for similar unlabeled inputs. [7, 16, 26] rely on the consistency of the unsupervised clustering. [30] utilizes adversarial learning to acquire domain-uninformative representations and train a PL classifier. [3, 17] use a contrastive loss to improve the representation learning before PL. Directly modeling noisy labels can also enhance noise tolerance [31], such as using soft pseudo labels [2], or distilling correct information to mitigate overfitting on noise [32].
The confidence measure plays a crucial role in avoiding the overconfident results [11]. [33] constructs confidence regularizers, expecting a smooth prediction with soft-label and preventing infinite entropy minimization. [10] learns a confidence metric based on the generality and unity of its distribution of pseudo loss. [1] uses the distance between the distributions of positive and negative samples as a confidence measure. [24] applies data augmentation on node features, where consistency is used as a confidence measure for classification. The co-training method constructs multi-view classifiers. It adds pseudo labels in each view to provide complementary information for each other, showing better performance than single-view [5].
Some studies on graph data apply the PL strategy to node classification tasks. M3S [21], IFC-GCN[8] utilize the clustering to PL the unlabeled samples. CaGCN-st [23] is a confidence-calibrated model
that utilizes low-confidence but high-accuracy samples. DR-GST [18] employs dropout and edge-drop augmentation to conduct information gain inference for selecting PL samples.
In this study, we provide deep insights into PL strategy by analyzing its impact on prediction error and the convergence properties. We offer theoretical explanations for the effect of PL strategies on graph learning processes, particularly addressing degradation issues. Based on the theoretical analysis, we introduce the CPL strategy, a plug-in and practical technique that can be generally applied to various baseline models. The experiments demonstrate effectiveness and superiority of CPL in link prediction and node classification tasks.
In future work, we plan to explore a more reliable confidence measures as the PL criteria, such as informativeness in the multi-view network and prediction uncertainty.
The research of Tsung was supported in part by the Industrial Informatics and Intelligence (Triple-i) Institute at HKUST(GZ).
The research of Jia Li was supported by NSFC Grant No. 62206067, Tencent AI Lab Rhino-Bird Focused Research Program and Guangzhou-HKUST(GZ) Joint Funding Scheme 2023A03J0673.
[1] Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. “Addressing failure prediction by learning model confidence”. In: NIPS 32 (2019).
[2] Yixiao Ge, Dapeng Chen, and Hongsheng Li. "Mutual mean-teaching: Pseudo label refinery for unsupervised domain adaptation on person re-identification". In: arXiv preprint arXiv:2001.01526 (2020).
[3] Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, et al. "Self-paced contrastive learning with hybrid memory for domain adaptive object re-id". In: NIPS 33 (2020), pp. 11309-11321.
[4] Aditya Grover and Jure Leskovec. "node2vec: Scalable feature learning for networks". In: Proceedings of the 22nd ACM SIGKDD. 2016, pp. 855-864.
[5] Tengda Han, Weidi Xie, and Andrew Zisserman. "Self-supervised co-training for video representation learning". In: NIPS 33 (2020), pp. 5679-5690.
[6] Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. "Revisiting Self-Training for Neural Sequence Generation". In: ICLR. 2020.
[7] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. "Hubert: Self-supervised speech representation learning by masked prediction of hidden units". In: IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), pp. 3451-3460.
[8] Zhihui Hu, Guang Kou, Haoyu Zhang, Na Li, Ke Yang, and Lin Liu. "Rectifying pseudo labels: Iterative feature clustering for graph representation learning". In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021, pp. 720-729.
[9] Zijian Hu, Zhengyu Yang, Xuefeng Hu, and Ram Nevatia. "Simple: Similar pseudo label exploitation for semi-supervised classification". In: CVPR. 2021, pp. 15099-15108.
[10] Kai Huang, Jie Geng, Wen Jiang, Xinyang Deng, and Zhe Xu. “Pseudo-Loss Confidence Metric for Semi-Supervised Few-Shot Learning”. In: ICCV. 2021, pp. 8671–8680.
[11] Susmit Jha, Sunny Raj, Steven Fernandes, Sumit K Jha, Somesh Jha, Brian Jalaian, Gunjan Verma, and Ananthram Swami. “Attribution-based confidence metric for deep neural networks”. In: NIPS 32 (2019).
[12] Thomas N Kipf and Max Welling. "Variational Graph Auto-Encoders". In: NIPS Workshop on Bayesian Deep Learning (2016).
[13] Dong-Hyun Lee. "Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks". In: Workshop on challenges in representation learning, ICML. Vol. 3. 2. 2013, p. 896.
A Proof of Proposition 2.2: additive expansion proposition
We denote the embedding vector of a node  $v_{i}$  by  $e_i \triangleq g_i(\hat{G}) = \mathrm{GNN}(\hat{G})[i]$ . Without loss of generality, we drop the subscript for short. We can also define the density measure  $d_{f}$  as:
$$
p _ {f} (e) \triangleq \frac {\exp (- d (e))}{\int_ {\mathcal {G}} \exp (- d (e)) d g (\mathcal {G})}. \tag {5}
$$
For any subset of the embedding space  $E \subset g(\mathcal{X})$ , the local probability can be measured by  $p_f(E) = \int_E p_f(e)de$ . If we have a local optimal subset  $U \subset E$  with a confidence threshold of  $1 - q$ , and its perturbation  $U_{\epsilon}$ , then the consistency at the boundary of the subset separation problem  $E \to U, E \setminus U$  can be quantified by Cheeger constant. We introduce the continuous Cheeger inequality to elaborate the lower bound of the Cheeger constant under an amplified measure  $\alpha f$ , where  $\alpha > 1$  is a constant.
We first define the restricted Cheeger constant in the link prediction task. Given the function  $f$  and any subset  $E$ , Cheeger constant is calculated by
$$
\mathcal {C} _ {f} (E) \triangleq \lim  _ {\epsilon \rightarrow 0 ^ {+}} \inf  _ {A \subset E} \frac {p _ {f} \left(A _ {\epsilon}\right) - p _ {f} (A)}{\epsilon \min  \left\{p _ {f} (A) , p _ {f} (E \setminus A) \right\}}. \tag {6}
$$
According to the definition, the Cheeger constant is a lower bound of probability density in the neighborhood of the given set. It quantifies the chance of escaping the subset  $A$  under the probability measure  $f$  and reveals the consistency over the set cutting boundary.
Then we prove that for the any subset  $E \subset g(\mathcal{G})$  with its local optimal subset  $U: \{e \in E : p_f(e) > 1 - q\}$ , there exists  $\alpha > 1$  s.t.  $\mathcal{C}_{\alpha f}(E \setminus U) \geq 1$ .
As the measurable function for the link prediction is defined as  $f(e) = -e^{T}e_{a}$ . When  $e^{*} = e_{a}$ ,  $f$  reaches the global minimal. For the embedding vectors outside the local minimal subset  $e_{y}\in g(\mathcal{G})\backslash U$  there exists  $\epsilon >0$  s.t.
$$
f \left(e _ {y}\right) \geq f \left(e ^ {*}\right) + 2 \hat {\epsilon}, \tag {7}
$$
where  $\hat{\epsilon} = C\epsilon$ . If we define  $E_{\epsilon}^{*} = \{e_{\hat{\epsilon}}^{*}\} \cap g(\mathcal{G})$ , where  $e_{\hat{\epsilon}}^{*}$  is the  $\hat{\epsilon}$  neighbor of  $e^{*}$ , according to the Lipchitz condition of  $f$ , for  $e \in E_{\epsilon}^{*}$ , we have:
$$
f \left(e _ {x}\right) \leq f \left(e ^ {*}\right) + \hat {\epsilon} \| e _ {x} - e ^ {*} \| _ {2} \leq f \left(e ^ {*}\right) + \hat {\epsilon}. \tag {8}
$$
Combining Eq.8 and Eq.7 leads to  $f(e_y) - f(e_x) \geq \hat{\epsilon}$ . Thus, for the amplified probability measure  $p_{\alpha f}$ , we have
$$
p _ {\alpha f} \left(e _ {x}\right) / p _ {\alpha f} \left(e _ {y}\right) \geq \exp (\alpha \hat {\epsilon}) \tag {9}
$$
According to the inequality property from [28] (formula 63), we have
$$
\frac {p _ {\alpha f} (U)}{p _ {\alpha f} \left(g (\mathcal {G}) \backslash U\right)} \geq \exp \left(\alpha \hat {\epsilon} - 2 \log \left(2 C ^ {2} / \hat {\epsilon}\right)\right). \tag {10}
$$
As  $p_{\alpha f}(g(\mathcal{G}) \setminus U) + p_{\alpha f}(U) = 1$ . If we select  $\alpha$  large enough s.t. the RHS of Eq.10 is larger than  $1$ ,  $p_{\alpha f}(g(\mathcal{G}) \setminus U) \leq \frac{1}{2}$ . Thus, according to [19] (Theorem 2.6), we have  $\mathcal{C}_{\alpha f} \geq 1$ . It guarantees consistency around the perimeter of the  $U$ . As  $\alpha > 1$  and  $p_f, p_{\alpha f}$  are bounded on any subsets of embedding space. It implies a probability margin  $\eta > 0$  at the neighborhood of the local optimal between two measurable functions  $f, \alpha f$ , where
$$
\eta = \inf  _ {\hat {e} \in U _ {\epsilon} \backslash U, e \in U} \left(p _ {\alpha f} (\hat {e}) - p _ {f} (e)\right). \tag {11}
$$
which, according to [25], implies additive expansion property of the probability measure in the link prediction, as Proposition 2.2.
B Proof of Theorem 2.3: error analysis
In [25],  $\mathcal{M}(g_{\phi})$  is also assumed to satisfy additive-expansion  $(q,\epsilon)$ , where  $\mathcal{M}(g) \triangleq \{y \in Y : g(y) \neq y\}$  is the set of mis-classified samples, and they give the error bound of the trained classifier  $s$  (Theorem B.2):
$$
\operatorname {E r r} (g) \leq 2 (q + \mathcal {A} (g)). \tag {12}
$$
Here in link prediction task,  $\mathcal{M}(g_{\phi})$  is mis-classified samples by the pseudo labeler (teacher model). It can be written by  $\{y_i:Y_p[i]\neq \mathcal{E}_{\mathcal{T}}[i]\}$ , which is intractable during the training. The probability threshold is  $1 - q$  and a local optimal subset  $U$  for PL is constructed accordingly. We aim to let  $\mathcal{M}(g_{\phi})\cap U$  be close to  $\emptyset$ , so that  $g(\mathcal{G})\setminus U$  can cover  $\mathcal{M}(g_{\phi})$  as much as possible. So we define the robust set  $S(g)$  as
$$
\mathcal {S} (g) = \left\{y: g (y) = g (\hat {y}), \hat {y} \in \left\{y _ {\epsilon} \right\} \right\}, \tag {13}
$$
where  $y_{\epsilon}$  is the  $\epsilon$  neighborhood of sample  $y$ . Then, according to Proposition 2.1, we have:
$$
p _ {f} \left(\left\{y \in Y: g _ {\phi} (y) \neq y, y \in \mathcal {S} \left(g _ {\psi}\right) \right\}\right) \leq p _ {\alpha f} \left(g (\mathcal {G}) \backslash U\right) \leq q, \tag {14}
$$
which has similar form with [25] Lemma B.3 for link prediction task. Besides, the analysis of  $p_f(\{y \in Y : g_\phi(y) = y, g_\psi(y) \neq y, y \in S(g_\psi)\})$  and  $p_f(\overline{S(g_\psi)})$  are the same. Thus, the assumption on  $\mathcal{M}(g_\phi)$  is satisfied. Then, we can draw the same conclusion with Eq.12, and the classifier is the student model  $g_\psi$ . The theorem is proved.
C Proof of convergence inequality
The PL strategy  $\mathcal{T}$  for the unlabeled data provides a Bayesian prior, from which we formalize the empirical loss defined in Eq.1 as
$$
\mathcal {L} _ {\mathcal {T}} ^ {(t + 1)} = \frac {1}{\left| \hat {Y} _ {o} ^ {(t)} \right| + k} \left[ \mathrm {C E} \left(g _ {\psi} ^ {(t)}, \hat {Y} _ {o} ^ {(t)}\right) + \mathrm {C E} \left(g _ {\psi} ^ {(t)}, Y _ {p} ^ {(t)}\right) \right]. \tag {15}
$$
We can decompose the cross-entropy loss of the pseudo labeled samples by:
$$
\begin{array}{l} \operatorname {C E} \left(g _ {\psi}, Y _ {p}\right) = \sum_ {\hat {Y} _ {u}} \operatorname {c e} \left(g _ {\psi}, Y\right) \cdot \mathcal {T} \\ = \sum_ {\hat {Y} _ {u}} \left[ \operatorname {c e} \left(g _ {\psi}, Y\right) - Y \left[ \operatorname {c e} \left(g _ {\psi}, Y\right) \right] \right] \cdot [ \mathcal {T} - Y \mathcal {T} ] \tag {16} \\ + Y \mathcal {T} \sum_ {\hat {Y} _ {u}} \operatorname {c e} (g _ {\psi}, Y) + \mathbb {E} [ \operatorname {c e} (g _ {\psi}, Y) ] \sum_ {\hat {Y} _ {u}} \mathcal {T} - \left| \hat {Y} _ {u} \right| Y \mathcal {T} Y [ \operatorname {c e} (g _ {\psi}, Y) ] \\ \end{array}
$$
Thus, Eq.16 can be simplified to:
$$
\begin{array}{l} \operatorname {C E} \left(g _ {\psi}, Y _ {p}\right) = \left| \hat {Y} _ {u} \right| \operatorname {C o v} \left[ \operatorname {c e} \left(g _ {\psi}, Y\right), \mathcal {T} \right] + \mathbb {E} \mathcal {T} \cdot \left| \hat {Y} _ {u} \right| \mathbb {E} \left[ \operatorname {c e} \left(g _ {\psi}, Y\right) \right] \\ + \mathbb {E} [ \operatorname {c e} (g _ {\psi}, Y) ] \cdot \left| \hat {Y} _ {u} \right| \mathbb {E} \mathcal {T} - \left| \hat {Y} _ {u} \right| \mathbb {E} \mathcal {T} \mathbb {E} [ \operatorname {c e} (g _ {\psi}, Y) ] \tag {17} \\ = \left| \hat {Y} _ {u} \right| \operatorname {C o v} [ \operatorname {c e} (g _ {\psi}, Y), \mathcal {T} ] + \left| \hat {Y} _ {u} \right| \mathbb {E T E} [ \operatorname {c e} (g _ {\phi}, Y) ] \\ = \left| \hat {Y} _ {u} \right| \operatorname {C o v} \left[ \operatorname {c e} \left(g _ {\psi}, Y\right), \mathcal {T} \right] + k \mathbb {E} \left[ \operatorname {c e} \left(g _ {\phi}, Y\right) \right] \\ \end{array}
$$
Note that  $\mathcal{T}$  is the indicator-like function, where we have
$$
\mathbb {E} \mathcal {T} = \frac {1}{\left| \hat {Y} _ {u} \right|} \sum_ {\hat {Y} _ {u}} \mathcal {T} = \frac {k}{\left| \hat {Y} _ {u} \right|}. \tag {18}
$$
Based on the Eq.15 and Eq.17, we can rewrite  $\mathcal{L}_{\mathcal{T}}^{(t + 1)}$  as
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {T}} ^ {(t + 1)} = \beta \operatorname {C o v} \left[ \operatorname {c e} \left(g _ {\psi}, Y\right), \mathcal {T} \right] + \frac {1}{\left| \hat {Y} _ {o} ^ {(t)} \right|} \operatorname {C E} \left(g _ {\psi} ^ {(t)}, \hat {Y} _ {o} ^ {(t)}\right) \\ \leq \beta \operatorname {C o v} [ \mathrm {c e} (g _ {\psi}, Y), \mathcal {T} ] + \frac {1}{\left| \hat {Y} _ {o} ^ {(t)} \right|} \mathrm {C E} \left(g _ {\phi} ^ {(t)}, \hat {Y} _ {o} ^ {(t)}\right) \tag {19} \\ = \beta \operatorname {C o v} \left[ \operatorname {c e} \left(g _ {\psi}, Y\right), \mathcal {T} \right] + \mathcal {L} _ {\mathcal {T}} ^ {(t)} \\ \end{array}
$$
where  $\beta = |\hat{Y}_u| / (|\hat{Y}_o| + k)$ . The inequality holds due to the assumption.
Table 6: Time consumption of CPL on link prediction
Base model	CiteSeer	Actor	WikiCS	TwitchPT	AmazonPhoto
GAE	452	6376	7412	3401	4956
node2vec	691	5067	6537	2740	5470
SEAL	2982	14579	17491	11639	24904
Table 7: Time consumption of CPL on node classification
Base model	Cora	CiteSeer	PubMed	AmazonPhoto	LastFMAAsia
GCN	115.8	241	147.6	302.6	104
GAT	251.4	450	388.8	652.2	338.6
SAGE	141.6	237.8	181.2	347	134.6
APPNP	511.2	1005.2	820	843.75	576.2
D Time consumption of CPL