Deep Insights into Noisy Pseudo Labeling on Graph Data
Authors: Botao WANG
Jia Li
Yang Liu
Jiashun Cheng
Yu Rong
Wenjia Wang
Fugee Tsung
Venue: NeurIPS 2023 poster (2023)
Keywords: Pseudo labeling
Graph data
Error analysis
Cautious

Abstract: Pseudo labeling (PL) is a wide-applied strategy to enlarge the labeled dataset by self-annotating the potential samples during the training process. Several works have shown that it can improve the graph learning model performance in general. However, we notice that the incorrect labels can be fatal to the graph training process. Inappropriate PL may result in the performance degrading, especially on graph data where the noise can propagate. Surprisingly, the corresponding error is seldom theoretically analyzed in the literature. In this paper, we aim to give deep insights of PL on graph learning models. We first present the error analysis of PL strategy by showing that the error is bounded by the confidence of PL threshold and consistency of multi-view prediction. Then, we theoretically illustrate the effect of PL on convergence property. Based on the analysis, we propose a cautious pseudo labeling methodology in which we pseudo label the samples with highest confidence and multi-view consistency. Finally, extensive experiments demonstrate that the proposed strategy improves graph learning process and outperforms other PL strategies on link prediction and node classification tasks.

Deep Insights into Noisy Pseudo Labeling on Graph Data
Botao Wang 1., Jia Li Yang Liu 1.2, Jiashun Cheng
Yu Rong", Wenjia Wang Fugee Tsung 1.2
Hong Kong University of Science and Technology. Hong Kong SAR, China
2 Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China TTencent AI Lab, Shenzhen, China
{bwangbk, yliukj, jchengak) jchengakk@cconeett. @connect. ust. hk {jialee, wenjiawang season) @ust. hk, yu. ronggd@oottmll..co com
Abstract
Pseudo labeling (PI.) is wide-applied strategy enlarge the labeled dataset by self-annotating the potential samples during the training process. Several works have shown that can improve the graph learning model performance in general. However, we notice that the incorrect labels can be fatal to the graph training process. Inappropriate
may result in the performance degrading, especially on graph data where the noise can propagate. Surprisingly, the corresponding error is seldom theoretically analyzed in the literature. this paper, we aim give deep insights of PL. graph learning models. We first present the error analysis of PL strategy by showing that the error is bounded by the confidence of PL. threshold and consistency of multi- view prediction. Then, we theoretically illustrate the effect of PI. on convergence property. Based on the analysis, we propose a cautious pscudo labeling methodology in which we pseudo label the samples with highest confidence and multi-view consistency. Finally, extensive experiments demonstrate that the proposed strategy improves graph learning process and outperforms other PL strategies on link prediction and node classification tasks.
Introduction
Pseudo Labeling (PL. [7 13 is one of the most popular self-supervised learning approaches and has been widely used to tackle the label sparsity problem. Its core idea is to enlarge the training set by self-labeling. As most self-labeled samples should be consistent with ground truth, the enlarged dataset has larger sample capacity to improve the model generalization. Plenty of studies have shown the effectiveness of PL 7 26 14 5. However, there a trade-off between the benefit of PL and the effect of mislabeled samples. When the benefit of PL outweighs the impact of introduced the base model (i.e., without PL) can be improved. But for non-i.i.d.
noise, the performance condition such as graph data, the introduced noisy labels may transfer among the samples and be amplified, which may degrade the performance of base model. Although several methods have been proposed to alleviate this noisy phenomenon (31 15), there still neither clear explanation a quantification of how pseudo labeling affects the graph learning models.
In this study, attempt answer questions above. Specifically, evaluate PL's effect the prediction error of the base model and the convergence the empirical loss function. For a graph learning model, the message aggregation process would amplify the noises of incorrect labels introduced by PL. These noises can even accumulate to damage the base model's performance. For
corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

16.0
GAE GAE-PL
AUCC -4
GAE-CPL
AUC
80.0
575
Vic sompptt//cooo
006
924
samples (Amazon,
Figure 1: The performance GAE, PL and proposed CPL strategy on link prediction w.r.t. Actor, WikiCS and Amazon__hoon dataset. Horizontal axis is #PL samples, and vertical axis is AUC.
example, in a two-layer graph neural network (GNN) for node classification, the mislabeled nodes can inappropriately reduce the predicted confidence of their 2-hop neighbors. Moreover, in some circumstances, such as link prediction task, the pseudo labeled sample would not only serve as the label but also the model inputs in the consequent iterations. This characteristic further aggravates the loss function's 's convergence due to the noises added to adjacency matrix.
To visualize the side effect of the PL graph data, conduct toy experiment shown in Figure [ where the benefit of PL. to a popular link prediction model GAE [12] depends on the choice of graph dataset, i.e., PL. improves the performance of GAE Actor, but degrades that WikiCS. Even worse, due to the incorrect label introduced by PL., PL. leads to the model's collapse on Amazon_Phooe
To obtain a stable and consistent result by PL strategy, it is necessary to quantify the impact of introduced noisy labels and theoretically analyze how it affects the graph training procedure compared to the procedure without PL. this paper. we build a theoretical connection between PL strategy and graph learning models with multi-view augmentations. We prove that the error bound of the PL predictor is jointly bounded by a confidence threshold and the prediction consistency over different augmentations. Moreover, we theoretically analyze that PL strategy affects convergence property by the covariance term in the optimization function. Accordingly, we propose the Cautious Pseudo Labeling (CPL) strategy for the graph learning process that maximizes the confidence threshold by committing the PL samples with high prediction probability. We evaluate the CPL on different graph learning tasks, including link prediction and node classification models, where observe remarkable and consistent improvements over multiple datasets and base models. As shown in Figure | compared with the base model GAE, the average AUC improvement of CPL over three datasets is 7.79%, which clearly validates the superiority of our model.
Model
In this section, we define the general problem of graph learning, including link prediction and node classification tasks. We give the error bound and convergence property analysis associated with the application of PL strategy. Lastly, we propose cautious PL strategy accordingly.
2.1 Problem Definition
In graph learning, given graph G (V.E), {we} } is the node set, E {(i,))} is the edge set, and [V] = N is the node number. The feature matrix and adjacent matrix are denoted by X = [x:j]a F and = [a:y]N N. respectively, where (i,j) E E if and only if a. + O. A base GNN model is trained on the graph G and the observed labels. It outputs the probability of prediction target. In this work, we adopt the PL scheme which involves a teacher model ge and student model gv- The teacher model calculates confidence of the unlabeled samples and PI. subset of samples using the strategy T. Subsequently, student model utilizes the enlarged set to fine-tune the base model and becomes the teacher model in the next iteration.
Node classification task. The objective of the node classification task to predict the probabilities of an unlabeled node belonging to different classes g G - R where M is the number of classes. The original training set comprises the labeled nodes V. and their labels Y.- And the model aims to predict the classes of the unlabeled nodes V,, In the PL scheme, the teacher model calculates the confidence of the unlabeled nodes Y,, and assigns pseudo labels to selected subset of nodes

---

Dv. V, C Vu. Yp C }. Then the student model undergoes fine-tuning on the enlarged label set (G) YY,Y YY, Up, Yo is the enlarged label set.
Link prediction task. The prediction targets are the probabilities edges between unlabeled node pairs. In our scheme, the link prediction model ([emb, 8) consists of an embedding network Gemb : G - N and a score function 8 IRD, RD' + R that estimates the probability the link, where D is the dimension of the node embeddings. A portion of edges is observed as the training graph G = (V, ,,), E c E. The teacher model predicts the confidence of the unobserved edges & and enlarge the observed edge set &, & U U for fine-tuning Pv ((VVE.) + RNNNN
It is worth to mention that in the link prediction task, the node embeddings undergo change once pseudo labels are added, even before fine-tuning. Because the enlarged edge set is also utilized as input for the GNN.
The optimization target is formulated as minimizing C CE(gg / Yol. where CE(-) denotes cross entropy, Y, is the ground truth label of the observed set and represents & for the link prediction task. The overall performance is evaluated by the 0-1 loss: Err(g) = El Eljagmax(gw) # Y].
2.2 Pseudo Labeling Error Analysis
We here present the error bound analysis of the graph learning under the PL strategy. To facilitate mathematical treatment, we introduce several assumptions.
2.2.1 Graph perturbation invariant
We assume graph perturbation invariant (GPI) property in GNN, which states that the prediction variance is linearly bounded by the difference between the augmented and original inputs.
Given a graph G and its perturbation G = G((X @ M, Ma) by the random feature masks M2 (1,0)XxP and adjacent matrix mask Ma {1,0) NNN satisfying
Definition 2.1:
1 N
1
NXF
M
NXN_
Ma <E,
F
N2
the GNN g(-) has GPI property if there exists a constans C> such that the perturbed prediction confulence satisfies (lg(G) - (G) < Ce. Here, @ is element-wise product, and ||-112 is the 2-norm of the vector or matrix.
The GPI property guarantees the variation of the output confidence is linearly bounded by the degree of graph perturbation, which is similar to the C-Lipschitz condition applied to GNN.
2.2.2 Additive expansion property
With Definition 2.1, if we have a convex measurable function f(y) that satisfies the C-Lipchitz condition, we can find a probability measure on the output prediction py (y) that satisfies the additive expansion property as stated in Proposition 2.2.
Proposition 2.2 Define local optimal subset U C Y, whose probability is higher than threshold Pr(u) > 1- 9.y E U, and its perturbation set U. = (ū = ( : |1 vll2 S Cc,y U} where
G (G} is the space of the perturbed graph. Then, there exists > 1, 0, s.t. the probability measure satisfying followin additive expansion property: Pas (Ua \ U) 2 Pa)(U) + +1 -n.aa a.
The proposition guarantees the continuity of the measurable function in the neighborhood of the local optimal subset U. In practice, represents the PL samples, which ideally should be close to the ground truth labels in ideal. However, the correctness of the PL samples is discrete where the continuity condition is hard to be satisfied. To address this issue, we can leverage GPL. By applying multi-view augmentations and calculating average confidence, we canreparameterize the probability measure to be continuous. In this condition, the Proposition 2.2 implies that the probability of the neighborhood under the amplified measure is greater than the original local optimum. This observation opens up potential opportunities for optimization. For detailed proof of Proposition 2.2, please refer to Appendix A
3

---

2.2.3 Prediction error measurement
Then, we use the additive expansion property to calculate the error bound of the multi-view GNN model under PL strategy. According to Theorem B.2 (25). when we use 0-1 loss measure the error of predictor, the error bound can be evaluated by the following theorem.
Theorem 2.3 Let q > 0 be a given threshold. For the GNN in the leacher model 9ø- if its corresponding density measure satisfies additive expansion, the error of the student predictor is bounded by
Err(g) s 2 A(ge))
where Algo)= Ey,. 13 (G) + go (G)) measures the inconsistency over differently augmented inputs, Yıeu is the test set for evaluation.
The brief derivation is provided in Appendix
B From to Eq [ we observe that the prediction error (0-1 loss) is bounded by the confidence threshold and the expectation of the prediction inconsistency A. Intuitively, if the predictions of multi-view augmented inputs tend to be consistent and the threshold is smaller, the error bound will be smaller, which implies more accurate predictor.
If q is small value, the threshold probability of PL. 1 approaches 1. This cautious approach in self-training leads to a smaller lower bound in the error estimate. When applying random PL, setting the confidence threshold q 0.5, then the maximum theoretical error rate I. It means that there is no guarantee on the performance after applying PL.
The inconsistency term A is the expectation of the probability that predictions inconsistent when input graphs are augmented differently. small value of A indicates consistent prediction across different views. In such cases, have more confidence in the predictions, leading to smaller error bound. On the other hand, if the predictions from different views are inconsistent with each other, the model lacks robustness, resulting in larger error bound.
2.3 Convergence Analysis
In this section, we analyze the influence the PL strategy on the empirical loss to illustrate its convergence property. First, assume that optimizing teacher model will not influence the convergence property of PL strategy. This assumption stems from the understanding that the teacher model can converge independently without the incorporation of PI..
Assumption 2.4 The loss function defined in Algorithm 7 is con-increasing during the optimization: CE )((())) < CE(9%) (t) (-()))
Then, we show that the PL sample selection strategy T influences the covariance term derived from the empirical loss, then affects the convergence property.
cl+1) S (.v [ce (ov. + cP
(3)
where 3 = 1 + k),, ce(-) is the element- wise cross entropy. The equality is achieved when the model reaches the global optimal under given training set.
The detailed derivation of the Eq is shown in Appendix C From Eq 3 we observe that the effect of PL strategy is decoupled and encapsulated in the covariance term. The covariance sign determines whether the empirical loss will increase or decrease in the next iteration of optimization. For instance, when PL samples are randomly selected, T would be independent with 9v. The covariance becomes O, indicating that the PL does not influence the loss function. However, a carefully designed PL strategy can accelerate the optimization of empirical loss and yield improved convergence properties.
2.4 Cautious Pseudo Labeling
According to Theorem 2.3, setting higher confidence threshold for pseudo labeling (PL) can lead to lower prediction error. This is reflected in a positive covariance term, which accelerates the optimization of the empirical loss and improves the convergence property. In order to satisfy these requirements, we propose the iterative Cautious Pseudo Labeling (CPL) strategy. CPL involves carefully and iteratively pseudo labeling the most confident samples to maximize these improvements in prediction accuracy and convergence.

---

Iteration unit
Adjacent modri
T
Enlarge
SIT
Enlarge Observed Set
Observed
Train
Train
Train
Tencher Link Predictor
Peramater Sharing
Student Link Predictor!
Paremnter shanijs
Student Link Predictor2
Prediction
Prediction
Prediction
Cautious
Caudoous Lsbude
Bu
0.0
0.0
0.8
8.9
0,
0.0
0.8
0.1
0.2
0.7
0.1
Minimization
0u4
0.7
0.8
Minimization
Loss
bu
0.0
0.1
0.9
0:3 0.4
61
Ru
0.
01
Confidence off
Obsereed Inio
Unobseried links
Figure 2: A framework illustration of CPL on link prediction: There is a teacher model and student model that share parameter. The most confident samples in teacher model's prediction are pscudo labeled. Then the student model fine-tuned on the enlarged datasct and becomes teacher model in the iteration.
There are teacher model and student model in each iteration. First, calculate multi-view prediction by the teacher model Iş as the confidence measure. Although biased, is a simple and effective measure in many PI. strategies. In node classification, the GNN directly outputs the probability as confidence (G)) For link prediction, the confidence is determined by the inner product similarity of node embedding g(G) oFF E.,, where E (e1, Jembl (Gon is the node
embedding, and o-)) is the sigmoid function.
Then we select PL samples in unobserved set with the highest confidence, enlarging the train- ing set. We iteratively select top-k confident samples for PL.: y,() T( Io: k), where T E (0.1)
ET = gh the averaged confidence of the multi-viww augmentation (t) for the single input. At the t-th iteration, the selected PL samples are ((
and is equal to 9
Then update the observed and the unobserved set. The student model is fine-tuned by the enlarged set and becomes the new teacher model in the next iteration. We take the link prediction task as the example, whose main scheme in Fig 2 The complete algorithm CPL shown Algorithm [
The confidence threshold qt)) recorded Algorithm is the lowest confidence among the PL samples
y, At each iteration, we record the lowest confidence in these & PL samples as (*)) We update to be (()) if (t) is smaller. Finally, q serves the confidence threshold in Eq 2 for error analysis in Theorem 2.3.
The following theorem states the improvement on convergence property under CPL.
Theorem 2.5 Let ( denote the optimization target at the t-th iteration. The risk is monotonically decreasing under pseudo labeling strategy, i.e.,
c((+1) S 3,30ov |ce (gv, Y), L() S cl)
Here c(+1)) is calculated by 1/ ((+++) (CE(g (((++))).
We have demonstrated the first inequality in Eq The second inequality holds because the covariance between the cross-entropy loss and PL strategy is non-positive. On one hand, each view of is positively correlated the ground-truth label Y due the consistency regularization. Then, Fo exhibits negative correlation with the cross cross-entropy loss ce(gv. Y). On the other hand, the PL strategy T can be seen as discrete sign function. In CPL, higher confidence samples are more likely to be pseudo-labeled negatively correlated with the cross-entropy term, resulting This guarantees the monotone decreasing behavior of the loss function, as stated in Theorem 2.5. Furthermore, since the loss function lower bounded, Theorem 2.5 ensures the convergence of Algorithm!!
Therefore, T is positively correlated with Jø- Consequently, T is
non-positive covariance term Cov(-).
5

---

Algorithm 1: Iterative cautious pseudo labeling.
Input:
Graph G, observed and unobserved label set Yo, Yu. iterative and total pseudo labeling
number k;
Output: Student model gC,()), confidence threshold
Pre-train teacher model g0 on observed set {G,Y,);
Initialize student model: Pg 96
(0) (0);
while { K do
Calculate the average confidence of the unobserved set by the multi-view teacher model Jo-
Select pseudo labeling subset (y(*) with top-k confidence in y:(!)
Update set: (((+1) ( UY y-(++1) y! y
Update the current confidence threshold according to g(();
Fine-tune the student model 9 by minimizing the cross entropy on (+11))
(t) Update teacher model Dg I and set 0g 9 (++1),
(t++1
(++1)
/ = + 1;
end
Time complexity The computational complexity of CPL depends on the complexity of specific GNN models used. The key operation in CPL is the selection of top-k values. For node classification, takes O[Nlog)), while for link prediction, it takes ((Nlogg)) However, since GNN models typically compute probabilities for all samples, the overhead introduced by CPL does not increase the complexity of existing methods significantly. For example, the complexity of GAE 12] for link prediction is O(EE,) D2 + D) (i.e., the complexity of graph convolution operations and inner- products). Integrating CPL into GAE does not increase the complexity since both involve O(N²) operations. In practice, the additional time consumption mainly results from the fine-tuning process.
3 Experiments
In this section, we conduct an evaluation to assess the effectiveness of the CPL strategy on both link prediction and node classification tasks. We compare CPL with raw base models as well other PL strategies. Then, we analyze the impact of CPL capacity, training data ratio, PL strategy, and augmentation methods. Finally, a case study is bridged to the theoretical analysis on convergence and emor bound. The implementation is open-sourced at httpss//gituub.co//A//bCCCCCCC
3.1 Datasets and Benchmarks
We adopt five public available datasets to evaluate CPL strategy for link prediction, i.e. CiteSeer, Actor, WikiCS, TwitchPT, and Amazon Photo, and five datasets for node classification, i.e. Cora, CiteSeer, PubMed, Amazon_Photo, and LastFMAsia. Detailed statistics are reported in Table
In link prediction task, there few PL-based methods, we apply the CPL strategy on three popular models: GAF [12) node2vec (4), SEAL 29 To reserve sufficient candidate unobserved samples for PL, the dataset is randomly split into 10%.40%.50% for training, validation, and testing.
In node classification task, we employ CPL on popular base models: GCN, GraphSAGE, GAT, APPNP. CPL is compared with two other PL strategies, namely M3S[22] and DR-GST 18) using the implementation and parameters provided by the authors We adopt the official split for the citation datasets and 1%,15%%805 split for other datasets in node classification.
run experiments with random seeds and report mean and standard deviations of the metrics.

---

Table Dataset statisties.
Dataset
CiteSeer
PubMed
WikiCS
TwitchPT
Amazon Photo
Lasts
# Nodes
2,078
3.327
19,717
7,600
11,701
1.912
7,650
7,624
# Links
10,556
9,104
88,648
30,019
216,123
64,510
238, 162
55,612
Features
1433
3,703
500
128
a Classes
3
10
18
Table 2: Performance (AUV%) comparison link prediction.
Model
Citeseer
Actor
WikiCS
TwitchPT
Amazon Photo
71.10
55,34 0.57
90.81 = 0.69
74.48 = 3.03
67.92 1.31
GAE+CPL
72.45 + 0.24
65.58 1.04
95.56 0.24
79.67 = 3.77
76.30
node2vec:
52.03 + 0.60
53.30
0.28
79.46 0.77
89.32 0.21
AUC(%)
node2vec+CPL.
55,22
6..11
0.26
84.76 3.52
89.53 0.30
SEAL
63.60 0.01
73.41 0.02
86.01 0.04
87.80 0.01
76,96 0.17
SEAL+CPL
64.33 0.14
73.54 0.01
86.83 + 0.07
87.87 0.01
78.86 0.01
GAE
72.12
53.60 1.06
90.58 0.71
69.73 5.06
67.06 0.99
GAE+CPL
73.54 + 0.20
67,65 1.06
95.58 # 0.29
79.09 + 5.48
75.52 4.23
node2vec
52.90 a: 0.36
55.43 + 0.62
92.54 + 0.51
83.37 0.52
91.46 0.18
AP(%)
Bode2vec+CPL
56.19 +: 1.60
68.33 = 2.85
93.66 0.29
85,87 2.15
91.47 0.21
SEAL
64.38 + 0.01
73.17 0.12
83.63 0,16
87.69 + 0.01
73.72 0.56
SEAL+CPL
64.94 = 0.14
73.44 # 0.02
86.72 + 0.12
87.75 + 0.02
80.36 0.09
3.2
Performance Comparison
3.2.1 Overall performance comparison
For link prediction, Table 2 lists the AUC and AP of raw baselines and ones with CPL employed. We observe that CPL distinctively increases the performance of baseline models in nearly all cases. Note that the CPL strategy can achieve performance gain under the circumstances of both high and low performance (e.g.. the AUC of GAE on Actor improves from 55.34 to 65.58 and the AP of SEAL on CiteSeer improves from 64.38 to 64.94).
For node classification, Table shows the improvement the base models and comparison with other PL strategies. The CPL strategy can consistently improve the performance of the base models. However, other PL strategy may be ineffective or degrade the base model (e.g., DR-GST on Cora with APPNP, M3S for PubMed with GraphSAGE) CPL also consistently outperforms other PL strategies.
The time consumption is also reported in AppendixD
3.2.2 Impact of pseudo labeling capacity
In the experiment, the number of PL samples in each iteration l is set from 100 to 1000. We provide a reference interval for the selection of Intuitively, small k can lead to a reliable result. But too small l will unnecessarily increase the training time. And it does not prominently influence the overall performance as we test on different datasets. In the experiment, the predicted confidence distribution is around 1, as there are usually plenty of potential PL. samples. & is much smaller than the total number unobserved samples. Then the k-th highest confidence in the unobserved set will not have much difference, as the cases in Table
3.2.3 Impact of training data ratio
PL enlarges the observed dataset and introduces extra information for the training. This effect has different degree of contribution on different training data. When the training set partially applied for the training, the ratio of the observed set ranges from 0.1 to 0.9. The variation of AUC and AP on the Amazon Photo dataset shown in Fig 3 The CPL method can consistently improve to mention that CPL more likely to have more significant contribution when the training set is small, as there more introduced information.
performance of the raw model even starting from small training set. It is also worth

---

Table 3: Performance (AUV%) comparison on node classification.
Model
Cora
CiteSeer
PubMed
Amazon
Photo
LasiaMAsia
Raw
80,74 0.27
69.32 0.44
77.72 0.46
92.62 + 0.45
78.53 + 0,60
M3$
80.92 a 0.74
72.70 # 0.43
79.36 a 0.64
93.07 + 0.25
79.49 + 1.42
GCN
DR-GST
83.54 * 0.81
72.04 0.53
77.96 0.25
92.89 + 0.16
79.31 + 0.55
CPL
83,94 = 0.42
72,96 = 0.22
79.98 + 0.92
93.15 + 0.24
79.92 + 0.61
Raw
81,12 0.32
0.32
69.80 + 0.19
77.52 + 0.38
92,46 + 0.17
80.23 + 0.28
M3$
83.02 0.49
70,98 + 2.14
92.41 + 0.14
81.48 +: 0.56
GraphSAGE
DR-GST
81.02 1.99
72.28 0.35
76.96 1.96:0044 0.43
92.58 = 0.14
81.10 + 0.30
CPL
84,62 = 0.19
73.14
79.72 + 0.72
92.90 + 0.20
82.25 + 0.25
Raw
81.28 0.87
71.18 11.18 0.43
77.34 2
93.26 + 0.31
81.12 # 0.58
M3S
82.28 1 0.95
71.70
79.20
9
81.82 + 0.93
GAT
DR-GST
8.33
72.64 0.97
78.28 + 0.32
93.60 0.13
81.86 + 0,50
CPI.
83.86 0.22
73.02 0.37
79.62 a: 0.31
93.72 + 0.29
82.89 + 0.56
Raw
0.69
70.82 0.224
93.05 + 0.29
82.40 + 0.50
M3S
8..54
72.58 0.45
79.98 + 0.14
93.21 + 0.59
83.55 + 0.71
APPNP
DR-GST
82.46 = 0.87
72.64 0.54
80.00 + 0.48
93. + 0.32
82.88 + 0.35
CPL.
84.20 + 0.42
74.22 0.24
80.62 2: 0.24
93.48 + 0.23
83.56 +: 0.53
AM50 210 CEN-AA
The effect of training ratio on Ama- zon_Photo w.r.t. AUC and AP: the relative improve- ment is more significant when the training ratio is small (i.e., the observed graph is sparse).
Figure 3:
Figure 4: The effect different data augmentation strategy on Actor w.r.t AUC and AP: the improvement of multi-view is largest compared other methods.
3.2.4 Impact of noisy pseudo labeling strategy
As the noise introduced by PL has a large effect in the graph, we compare the CPL with the PL strategy. In PL, the samples selected randomly in the unobserved set, whose labels estimated by the teacher model. The comparison of AUC's variation on different datasets is shown in Fig We discover that PL has different effects the baseline models. On Actor, the PL consistently improves the link predictor, but not as much as the CPL. On WikiCS however, PL keeps weakening the AUC, as the introduced noise outweighs. On Amazon__hoor iterations. But the AUC suddenly drops from 70% to around 50% after iterations, which illustrates that introducing too much noise may degrade the model. As for the CPL, although it may drop after reaching its best, i.e. for WikiCS, it can distinctively improve the performance compared to the base model and will not lead to failure.
PL improve the performance the first few
3.2.5 Impact of multi-view augmentation
During the training, we apply multi-view augmented graphs as the input of the teacher model, so that the continuity condition in Theorem 2.3 is casier to be satisfied. We here conduct the ablation experiments that compares multi-view augmentation with different single-view augmentation methods, including drop node (Node view), feature mask (Feature view), and DropEdge (Structure view). In each experiment, a single augmentation method was applied three times. And "Random" refers to the random selection of samples during PL. As shown in Fig all of the single-view augmentation methods have better performance than the base model and their improvements are almost the same. When we use multi-view augmentation, the AUC and AP are further improved. illustrates that multi-view augmentation contributes to a more robust graph learning and tends to obtain a consistent result, which echoes with our theoretical analysis.

---

Table 4: CPL with different k.
Table 5: Case analysis of CPL LastFMAsia.
CiteSeer
GCN
GraphSAGE
GAT
APPNP
Raw
100
500
2000
Inconsistency AA (%)
6.69
4.01
2.96
3.13
AUC(%)
71.10 72.45
72.25
72.45
71.98
Confidence 1
(98)
77.63
88.35
83.00
86.86
AP( %)
73.29
73.54
73.13
Theoretical Erras (%)
58.12
31.32
39.92
32.54
Amazon
Photo
Experimental EtTexp (%)
20.08
17.75
17.11
16.44
k
Raw
5
50
500
PL (%)
7.78
6,43
8.18
6.02
AUC( %))
67.92
76.63
76.30
75.95
M3S PL error (%)
65.63
27.00
65.00
27
AP( %)
67.06
75.70
75.52
74.97
DR-GST PL error (%)
26.31
14.10
13.38
29.09
3.3
Case Study
The case study on node classification conducted on LastFMAisa on different baselines. The detailed intermediate variables about the error analysis listed in Table We record the inconsistency A and confidence 1 - during the CPL as Algorithm 1 Then the theoretical error bound Err, (g) calculated by Theorem 2.3. As Er (g) Errezp (g), the experimental error is bounded by the theoretical error, implying that it is an effective bound. We also report the accuracy of PL samples. The CPL has smaller error than other PL strategies and is consistently better than (gTeap(g).
The case study on link prediction is conducted based on GAE WikiCS. The relationship between consistency 1 A(g) and the number of PL sam- ples is shown in Fig The result shows that more PL samples help increase the consistency of the predic- tion. We also compare the optimiza- tion target of PL L and CPL c in 5 We discover that the optimiza- tion target of CPL converges faster than Pl. and is consistently smaller.
vertll
varilla
cautoou
1.00
argnal
10Cc
EXCSO
Figure 5: Case study of the consistency and convergence WikiCS: Cautious (CPL) improves prediction consistency within error bound, and converges faster than Pl..
implies the covariance term Cov [ce, T] is negative as illustrated in Theorem 2.5. Thus, the CPL strategy can improve the convergence property of graph learning. A detailed illustration of error bound verification and Knowledge discovery is shown in Appendix
Related Works
Pseudo labeling popular approach in self-training (ST), aiming to enlarge the training set by self-annotation. Most studies focus on learning more accurate PL algorithms to avoid noisy samples. Confidence thresholding is a simple yet effective method in ST, where the similarity with ground truth consistency is used to measure the confidence, ensuring that flawed samples excluded from the enlarged dataset [9 20). (6) uses the perturbation on the hidden states to yield close predictions for similar unlabeled inputs.
16 26 rely the consistency of the unsupervised clustering, (30) utilizes adversarial learning to acquire comain-uninformativv representations and train a PL classifier. contrastive loss to improve the representation learning before PL. Directly modeling noisy labels can also enhance noise tolerance (31), such as using soft pseudo labels (2). or distilling correct information mitigate overfitting noise (32
3 17
The confidence measure plays a crucial role in avoiding the overconfident results [T]
33 constructs
confidence regularizers, expecting smooth prediction with soft-label and preventing infinite entropy minimization. (10) learns a confidence metric based on the generality and unity of its distribution of pseudo loss. [ uses the distance between the distributions of positive and negative samples as a confidence measure. 24 applies data augmentation on node features, where consistency used confidence measure for classification. The co-training method constructs multi -iet classifiers. adds pseudo labels in each view to provide complementary information for each other, showing better performance than single-view (5).
Some studies on graph data apply the PL strategy to node classification tasks. M3S (T], IFC-GCN(8
utilize the clustering to PL the unlabeled samples. CaGCN-st (23 is confidence--allibatted model

---

that utilizes low-confidence but -acuurcyy samples. DR-GST [18] employs dropout and edge-drop augmentation to conduct information gain inference for selecting PL samples.
5 Conclusion
In this study, we provide deep insights into PL strategy by analyzing its impact on prediction error and the convergence properties. We offer theoretical explanations for the effect of PL strategies on graph learning processes, particularly addressing degradation issues. Based on the theoretical analysis, we introduce the CPI. strategy, a plug-in and practical technique that can generally applied to various baseline models. The experiments demonstrate effectiveness and superiority of CPL in link prediction and node classification tasks.
In future work, we plan to explore a more reliable confidence measures as the PL criteria, such as informativeness in the multi-view network and prediction uncertainty.
Acknowledgments and Disclosure of Funding
The research of Tsung was supported in part by the Industrial Informatics and Intelligence (Triple-i) Institute at HKUST(GZZ.
The research was supported by NSFC Grant No. 62206067, Tencent Al Lab Rhino-Bird Joint Funding Scheme 2023A03J0673.
Focused Research Program and
References
[1] Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. "Ad- dressing failure prediction by learning model confidence". In: NIPS 32 (2019).
[2] Yixiao Ge, Dapeng Chen, and Hongsheng Li. "Mutual mean-teachings Pseudo label re- finery for unsupervised domain adaptation on person re-identification". In: arXiv preprint arXiv:2001.01523 (2020).
[3] Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, et al. "Self-paced contrastive learning with hybrid memory for domain adaptive object re-id". In: NIPS 33 (2020), pp. 11309-11321.
[4] Aditya Grover and Jure Leskovec. "node2vec: Scalable feature learning for networks". In: Proceedings of the 22nd ACM SIGKDD. 2016, pp. 855-864.
[5] Tengda Han, Weidi Xie, and Andrew Zisserman. "Self-supervised co-training for video repre- sentation learning". In: NIPS 33 (2020), pp. 5679-5690.
[6] Junxian He, Jiatao Gu, Jiajun Shen, and Mare' Aurelio Ranzato. "Revisiting Self Self-Training for Neural Sequence Generation". In: ICLR. 2020.
[7] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdi- nov, and Abdelrahman Mohamed. "Hubert: Self-superviedd speech representation learning by masked prediction of hidden units". In: IEEE/ACM Transactions Audio, Speech, and Language Processing 29 (2021), pp. 3451-3460.
[8] Zhihui Hu, Guang Kou, Haoyu Zhang, Na Li, Ke Yang, and Lin Liu. "Rectifying pseudo labels: Iterative feature clustering for graph representation learning". In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021, pp. 720-729.
[9] Zijian Hu, Zhengyu Yang, Xuefeng Hu, and Ram Nevatia. "Simple: Similar pseudo label exploitation for semi-supervised classification". In: CVPR. 2021, pp- -5099-15108..
[10] Kai Huang, Jie Geng, Wen Jiang, Xinyang Deng, and Zhe Xu. "Pseudo-Loss Confidence Metric for Semi-Supervised Few-Shot Learning.. In: ICCV. 2021, pp. 867 1-8680.
[11] Susmit Jha, Sunny Raj, Steven Fernandes, Sumit K Jha, Somesh Jha, Brian Jalaian, Gunjan Verma, and Ananthram Swami. "Atrrbuttoo--ased confidence metric for deep neural networks".
In: NIPS 32 (2019).
[12] Thomas N Kipf and Max Welling. "Variational Graph Auto-Encoders". In: NIPS Workshop on
Bayesian Deep Learning (2016).
[13] Dong-Hyur Lee. "Pseudo-label: The simple and efficient supi-supe learning method for deep neural networks". In: Workshop on challenges representation learning, ICML. Vol.
2. 2013, p. 896.
10

---

[14] Jia Li, Yongfeng Huang, Heng Chang, and Yu Rong. "Semi- Supervised Hierarchical Graph
Classification". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2022). [15] Jia Li, Jiajin Li, Yang Liu, Jianwei Yu, Yueting Li, and Hong Cheng. "Deconvolutional Networks on Graph Data". In: NeurIPS, 2021.
[16] Jia Li, Yu Rong, Hong Cheng, Helen Meng, Wenbing Huang, and Junzhou Huang. "Semi- supervised graph classification: A hierarchical graph perspective". In: The World Wide Web Conference. 2019, pp. 972-982.
[17] Junnan Li, Caiming Xiong, and Steven CH Hoi. "Comatch: supervipe learning with contrastive graph regularization". In: ICCV. 2021, pp. 9475-9484.
[18) Hongrui Liu, Binbin Hu, Xiao Wang. Chuan Shi, Zhiqiang Zhang, and Jun Zhou. "Confidence may cheat: Self-training on graph neural networks under distribution shift". In: Proceedings of the ACM Web Conference 2022. 2022, pp. 1248-1258.
[19] László Lovász and Miklós Simonovits. "Random walks in a convex body and an improved volume algorithm". In: Random structures & algorithms 4.4 (1993), pp. 359 4122.
[20] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang. Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. "Fixmatch: Simplifying semi- supervised learning with consistency and confidence". In: NIPS 33 (2020), pp. 596-608.
[21] Ke Sun, Zhouchen Lin, and Zhanxing Zhu. "Multi-stage self-supervised learning for graph convolutional networks on graphs with few labeled nodes". In: Proceedings of the AAAI conference on artificial intelligence. Vol. 34. 04. 2020, pp. 5892-5899.
[22] Xiao Wang. Deyu Bo. Chuan Shi. Shaohua Fan, Yanfang Ye, and Yu Philip. "A survey on heterogeneous graph embedding: methods, techniques, applications and sources". In: IEEE
Transactions on Big Data (2022).
[23] Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang. "Be confident! towards trustwor- thy graph neural networks via confidence calibration". In: Advances in Neural Information Processing Systems 34 (2021). pp.
[24] Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, Juncheng Liu, and Bryan Hooi. "Nodeaug: semi-supervised node classification with data augmentation". In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, pp. 207-217.
[25] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. "Theoretical Analysis of Self- Training with Deep Networks on Unlabeled Data". In: ICRL. 2021.
[26] Shaoan Xie. Zibin Zheng. Liang Chen, and Chuan Chen. "Learning semantic representations for unsupervised domain adaptation". In: ICMI.. PMLR. 2018, pp- 5423-5432.
[27] David Yarowsky. "Unsupervised word sense disambiguation rivaling supervised methods". In: 33rd annual meeting of the association for computational linguistics. 1995, pp. 189-196.
[28] Percy Liang Yuchen Zhang and Moses Charikar. "A hitting time analysis of stochastic gradient Langevin dynamics". In: Proceedings of the 2017 Conference on Learning Theory. Vol. 65. Proceedings of Machine Learning Research. PMLR, 2017, pp. 1980-2022.
[29] Muhan Zhang and Yixin Chen. "Link prediction based on graph neural networks". In: NIPS 31 (2018), pp- 5165-5175.
[30] Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. "Collaborative and adversarial network for unsupervised domain adaptation". In: CVPR. 2018, pp. 3801-3809.
[31] Zhilu Zhang and Mert Sabuncu. "Generalized cross entropy loss for training deep neural networks with noisy labels". . In: NIPS 31 (2018).
[32] Zizhao Zhang, Han Zhang, Sercan O Arik, Honglak Lee, and Tomas Pfister. "Distilling effective supervision from severe label noise". In: CVPR. 2020, pp. 9294-9303.
[33] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. "Confidence regularized self-training". In: ICCV. 2019, pp. 5982-5991.
11

---

A Proof of Proposition 2.2: additive expansion proposition
We denote the embedding vector node i by 6, _ g)( = [N Without loss of generality, drop the subscript for short. We can also define the density measure d, as:
exp(-d(e)) Ps(e) Ig (xep(-d(e))dg(())
(5)
For any subset of the embedding space Pr( = JE Py (e)de. If we have a local optimal subset U C E' with a confidence threshold of 1 q. and its perturbation U.. then the consistency at the boundary of the subset separation problem E - U. E \ U can be quantified by Cheeger constant. We introduce the continuous Cheeger inequality to elaborate the lower bound of the Cheeger constant under an amplified measure af. where > 1 is constant.
g(X), the local probability can be measured by
We first define the restricted Cheeger constant in the link prediction task. Given the function / and any subset E, Cheeger constant is calculated by
A lim inf Py(A.) -20))-110 -P) -00 ACE c (min(p;(A),, \ A)}
Cx(F)
According to the definition, the Cheeger constant is lower bound of probability density in the neighborhood of the given set. It quantifies the chance escaping the subset A under the probability measure f and reveals the consistency over the set cutting boundary.
Then we prove that for the any subset E C g()) with its local optimal subset U : E E.E:E pp(e)>> 1 1-g) q). there exists a > 1 s.t. C \ U) 2 1.
As the measurable function for the link prediction is defined as f(e = -ee When car S reaches the global minimal. For the embedding vectors outside the local minimal subset By (G))\U, there exsits > 0 s.t.
f(e,) 2 f(e")(e))))))))))()) )+ +2,
(7)
where ė Ce. If we define E: = } n (()). where et the è neighbor of e*, according to the Lipchitz condition of f, for E E,, we have:
f(e,) 5 + élle, e²112 S f(c")
(8)
Combining and Eq 7 leads to f(ey) [(x)) 2 é. Thus, for the amplified probability measure Paf, we have
Paf (a)(ea)Paa((e))) 2 exp(aε)
(9)
According to the inequality property from 28 (formula 63), we have
(U) 2 exp (aé 2log (2C²/2))
(10)
(af(
As (g(() \ U) + (u)(U)= = 1. If we select a large enough s.t. the RHS of Eq 10 is larger than
1, (g())(())) \ U)= < 2. Thus, according to [19] (Theorem 2.6), we have Caf 2 1. guarantees consistency around the perimeter of the U. As > and Py fo bounded on any subsets of
embedding space. It implies probability margin > 0 at the neighborhood of the local optimal between measurable functions f, af, where
inf EEU, UEEEU (pa/(e) - p)())).
(11)
which, according to 255 implies additive expansion property of the probability measure the link prediction, Proposition 2.2.
12

---

B Proof of Theorem 2.3: error analysis
In [25], M(gs) is also assumed to satisfy additive-expansion (g,e), where M(g) - {y Y gly) 7 y) is the set of mis-classified samples, and they give the error bound of the trained classifier (Theorem B.2):
Err(g) S 2(q + A(g)).
(12)
Here in link prediction task, (gg) mis-classified samples by the pseudo labeler (teacher model). can be written by {yx: [ * { which is intractable during the training. The probability threshold is q and local optimal subset U for PL is constructed accordingly. We aim to let
((s)) n U be close to so that g(g) \ U can cover M(g+) as much as possible. So we define the robust set S(g) as
((g) = (y g (v) = 9 (0).0 / {ve}}:
(13)
where Ye is the neighborhood of sample y. Then, according to Proposition 2.1, have:
Pf (Ly Y go(v) # v.y (())))) Paf (g (9) \ U) 5.
(14)
which has similar form with 25 Lemma B.3 for link prediction task. Besides, the analysis of Go ( * E S(ge))) and py (S(()))) are the same. Thus, the
pr(ty E Y go(y) assumption on M(ga) satisfied. Then, we can draw the same conclusion with Eq 12 and the classifier is the student model Iv- The theorem is proofed.
C Proof of convergence inequality
The PI. strategy T the unlabeled data provides a Bayesian prior, from which we formalize the empirical loss defined in Eq Tas
(1++1)
CE
(1)
CE
(0 g0 (.,,)))) )
(15)
(((1)
+ k
We can decompose the cross-entropy loss of the pseudo labeled samples by:
CE (ge. yp)= = Σ
ce (go, Y). T
P
Σ
Ice (gp, Y)
Ice (ge, Y)Il IT YT
(16)
Y.
ce (gv, Y) + E (go, Y)] Σ
Yra YTY (eeY/ce(A (gv,
Y
V.
Thus, Eq T6 can be simplified
CE (gw, Yp) = Yu Cov iee (gv, r), 7 + ET
Yu E Elce (os. Y)]
+ E Ice (gp, Y)] Y, ET - \
ETE [ce (g Y)]
(17)
Yu Cov iee (g Y) T + [r ETE Ice (go, Y)1
Yu Cov jee (gv, Y). T + kR Ice (ge, Y)]
Note that T the indicator- like function, where we have
k
ET
(18)
Y
13

---

Based on the Eq 15 and Eq 17 we can rewrite ((++1)
cl+
3Cov Ice (g Y), T +
CE
Op Y((0)
3Cov |ce (go, T]
CE
(9) (7(1)
(19)
3Cov Iee (go. Y), 7] CT
where B /.1/((%)++ The inequality holds due to the assumption.
Table Time consumption of CPL on link pre-
Table Time consumption of CPL on node classification
diction
Base model
CiteSeer
Actor
WiaCS
TwitcPT
AmizanPhoto
Rase model
Com
CiteSoer
PuisMed
AmszosPParto
Last
modedvee
GAE
691
452
6376
7412
3401
4956
SAGEE
GCN
GAT
251.4
141.6
115.8
741
450
353.8
147.6
652.2
302.6
338.6
134.6
104
APPNP
1005.2
576.2
D
Time consumption of CPL
The time consumption of CPL on node classification and link prediction are shown in Table and Table respectively.
E
Case study of CPL on link prediction
0.98 for WikiCS. We adopt 5 views of dropout with the augmentation drop rate 0.05. And according to the error bound given by Theorem 2.3, given the confidence threshold, Eq suggests that the higher prediction consistency should lead smaller eтror bound. The final prediction consistency is AAg)) 0.0358, thus, we can calculate еттог bound Err(g) = 0.1116. The AUC and AP are 95.56 + 0.24%. 95.58 + 0.29% which are bounded within Err(g).
Error bound: In the case study, the recorded confidence threshold is
In the 5 random experiments, we add 500 pseudo links in each iteration. Here focus the common PL. links in the first iteration, which considered the most confident samples. We look for the metadata of WikiCS whose node, feature, link and node label represent paper, token, reference relation and topic of the paper respectively. There are These 7 most confident links categorized into 2 groups. We take 3 out of 5 nodes in groupl and the 2 nodes in group2 for analysis, whose detailed information of these nodes is shown in Appendix
Knowledge discovery:
Table Details of Node Information in the Case Study.
Group 1
Group
Node
Node 2702
Node 5688
Node 8906
Node 3489
Node 7680
II Outlinks
17505908
11353631
30138652
23221074
12265137
[6097297]
[6097297]
[244374, 6097297]
[20901]
[]
Title
Ubuntu Hacks
Pungi (software)
LinuxPAE64
Malware Bell
Norton Confidential
Label
Operating systems "ubuntu" "hacks". "ubuntu" "hacks". "tips". "toals". ploring". "using" "tuning". "linux" "book" "tips". "ubuntu" "popular". "linux" "distribu- tion" "book" "pub- lished". "o'rcilly" "media" "june" "2006". "part..
Operating systems "pungi". "software" "pungi". gram". "making" "spins", "fedora" "linux" " "distribution" "release", "7", "up-
Operating systems
Computer security "malware" 'bell".
Computer security "norton". "confi- dential". "norton". "confidential" proo gram", "designed", "encrypt". "pass- words". "online". "detect". "phishing". "sites"
Tokens
"linuxpae64" uxpae64". "port". "linux". "kerne"" "running", "com- patibility". "x86-64",
"lin-
"pro-
"malware"
"bell".
ppro
gram", "made" "tai- wan". "somewhere" "2006". "2007" "malware" "bell". "wies" "install' "au- tomstically" "upon". "visiting". "website". "promoting" "con- taining" "malware"
"mode".
"pm--
wards"
cessor". "kerne".
"capable" "loading"
"1386". "modules"
"device". "drivers".
"supports", '64-bit".
"limux" "appli-
"o "o'reilly,, "hacks", "series"
cations". "user".
"mode"
14

---

For group 1, 3 nodes are connected by the pseudo links, and they are all linked to central node whose degree 321. The metadata information of the nodes are all strongly relevant "Linux" in the "operating systems" topic. Thus, the PI. linked nodes are likely to have common neighbors discovered triangle relationship. In group2, node 3489 has no in/out degree and is pseudo linked to node 7680. Both papers focus on the "malware" p"iisiiig under the topic "Computer security". Although they only have one common token, the CPL strategy successfully discovers the correlation and consistently add it to the training set. The detailed result of the case study is shown in Table 8
15


Reviews:
[{"review_id": "pliwiWZJnd", "reviewer": "Reviewer_nJna", "summary": "The article discusses noisy pseudo labeling (PL) on graph data and proposes a new cautious PL methodology (CPL) to improve the graph learning process. The authors conduct experiments to evaluate CPL strategy for link prediction on various datasets and apply it on popular models in node classification task. The result shows that the proposed strategy outperforms other PL strategies. The paper also provides a theoretical analysis of the impact of noisy labels introduced by PL on the graph training procedure. 1. The error introduced by PL is bounded by the confidence of PL threshold and consistency of multi-view prediction. 2. PL can be designed to contribute to the convergence property.", "questions": "1. Were there any specific assumptions made when applying the proposed approach to the benchmark datasets? It would be helpful to understand the compatibility of the approach with different dataset characteristics, such as class imbalance or noise types. Insights into these considerations would shed light on the generalizability of the approach.\n\n2. Could you provide additional information on the hyperparameters used in the experiments? Specifically, how were the hyperparameters set for the proposed cautious Pseudo Labeling (PL) methodology and the baseline models? Sharing these details would aid in replicating and fine-tuning the approach in future research.\n\n3. In the discussion of results, could you provide further insights into the potential limitations or failure cases of the proposed approach? Understanding the scenarios where the approach may not perform optimally would help in setting realistic expectations and identifying areas for further improvement.", "limitations": "The authors did not mention their limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes a highly original and significant contribution to graph learning. It proposes a new cautious Pseudo Labeling (PL) methodology that addresses limitations in prior works by introducing a confidence threshold and a consistency criterion for selecting high-confidence PL samples. This methodology, combined with a new consistency-based PL (CPL) strategy, improves the convergence property of graph learning and outperforms other PL strategies in link prediction and node classification tasks. The research methodology is rigorous, and the paper is well-structured, clear, and provides practical solutions to the challenges of limited and noisy labeled data.", "weaknesses": "The paper could be strengthened by addressing several weaknesses. Firstly, in Table 2, the authors can also compare with other PL methods in the link prediction task. Secondly, conducting experiments with larger sample sizes or exploring different configurations would provide a more comprehensive evaluation. Thirdly, clarifying the methodology by providing implementation details and explaining data preprocessing steps would enhance replicability and understanding. Finally, considering the applicability of the proposed approach to different domains with highly imbalanced class distributions or datasets with different types of noise would broaden its practical relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are most thankful for your thoughtful assessment, and glad to communicate with you on all your concerns:\n\n> Were there any specific assumptions made when applying the proposed approach to the benchmark datasets? It would be helpful to understand the compatibility of the approach with different dataset characteristics, such as class imbalance or noise types. Insights into these considerations would shed light on the generalizability of the approach.\n\nThe assumptions of the CPL are the Graph Perturbation Invariant (GPI) property and Additive Expansion Property (AEP). GPI guarantees there is no extreme change during the augmentation process. Some scenarios like the adversarial attack and designed noise are excluded. AEP assumes the continuity of the probability measure in the neighborhood of the local optimal set. The sudden variation in probability density may violate this assumption, such as the highly heterophilous graph. There is no evidence showing that the imbalance and noise type influence the efficiency of CPL. However, their effects on the base model is the key point to the overall performance.\n\n> Could you provide additional information on the hyperparameters used in the experiments? Specifically, how were the hyperparameters set for the proposed cautious Pseudo Labeling (PL) methodology and the baseline models? Sharing these details would aid in replicating and fine-tuning the approach in future research.\n\nCPL is characterized by a relatively small number of hyperparameters. Specifically, there two key hyperparameters that need to be set are the capacity of the number of PL per iteration and the corresponding fine-tuning epochs. These hyperparameters can be chosen based on the available computational resources, as discussed in section 3.2.2 of our analysis. In our experiments, we opted to set the fine-tuning epoch to 20% of the pretraining epoch. The iterative PL capacity results are presented in the following tables.\n\nWe have implemented early stop restrictions in our approach. We have defined a lowest acceptable threshold, denoted as Th, and a fine-tuning patience value, denoted as P. In CPL, we employ a top-k strategy. However, if the confidence of a particular sample falls below the threshold Th, it will not be assigned a pseudo-label. The threshold Th can vary depending on the dataset and its specific characteristics. The exact values used for Th are reported in the subsequent section. Furthermore, we have incorporated an early stop mechanism based on the fine-tuning patience P. If the CPL process fails to improve model performance for P consecutive iterations, the training is terminated. In our experiments, we have set P to a value of 10. These early stop restrictions serve to control the quality and effectiveness of PL.\n\nLink prediction:\n\n| Dataset | CiteSeer | Actor | WikiCS | TwitchPT | AmazonPhoto |\n| -- | -- | -- | -- | -- | -- |\n|#PL/itr| 100| 200| 1000 | 300 | 500 |\n| Th | 0.8 | 0.9 | 0.98 | 0.9 | 0.9 |\n\nNode classification:\n| Dataset | Cora | CiteSeer | PubMed | AmazonPhoto | LastFMAsia |\n| -- | -- | -- | -- | -- | -- |\n|#PL/itr| 100| 200| 1000 | 300 | 500 |\n| Th | 0.6 | 0.8 | 0.8 | 0.8 | 0.8 |\n\n> In the discussion of results, could you provide further insights into the potential limitations or failure cases of the proposed approach? Understanding the scenarios where the approach may not perform optimally would help in setting realistic expectations and identifying areas for further improvement.\n\nThe proposed model represents a basic implementation of confidence-based PL for multiview graph learning. While it aligns with theoretical analysis, there is ample room for further enhancement. One potential area for improvement lies in the measurement of confidence. In the current CPL approach, we rely on the average of multiview predicted probabilities, which is a biased estimation of confidence. By incorporating a proper confidence estimation that takes into account uncertainty and factors in downstream tasks, we can potentially enhance the underlying model. Another avenue for advancement is the development of more advanced and adaptive PL strategies. For instance, we could consider incorporating a diversity penalty during the selection of PL candidates. This would help to promote a more diverse and representative set of candidates, thereby enhancing the overall learning process.\n\nOn a different note, providing an explicit condition for the theorem is challenging. However, it is possible to circumvent the assumptions in specific scenarios. As previously mentioned, the AEP assumes the continuity of probability near the local optimum. Yet, situations characterized by sudden variations in probability density, such as those encountered in heterophily graphs or instances of overconfidence in incorrect samples, may violate this assumption. By acknowledging and addressing these specific conditions, we can refine the model's applicability and effectiveness.\n"}, {"review_id": "Z3MfsbDV5q", "reviewer": "Reviewer_iJpS", "summary": "The paper provides an error bound for pseudo labeling on graphs. Moreover, the authors propose a cautious pseudo labeling method and validate it through experiments. ", "questions": "1. Line 86: For \"err(g)\", does it mean the expectation is taken over the test points? Please clarify this point.\n\n2. Line 91: Is the GPI property introduced in this paper, or has it been introduced previously? Please provide the necessary context for understanding this property.\n\n3. Line 104: When stating \"whose probability is higher than a threshold,\" does it mean this condition must hold for each \"y ∈ U\"? Please clarify this point.\n\n4. Line 105: Since \"g(G) ∈ ℝ^(N×M)\", what do you mean by \"ŷ ∈ g(G)\"? Please explain this notation.\n\n5. Line 102: Do you mean we can find \"p_f(⋅)\" or we can find \"α, η\"? Please clarify this statement.\n\n6. Line 108: In \"p_{αf}\", there should be no \"α\" here. Please correct this notation.\n\n7. Line 122: When stating \"the teacher predictor satisfies additive expansion,\" the additive expansion is defined for a probability density, not a classifier. Please provide clarification and ensure consistency in terminology.\n\n8. Line 124: Could you explain what is meant by \"E_{Y_test}\"?\n\n9. Line 145: The first term should change \"t\" to \"t+1\". \n\n10. Line 147: What is the definition of the covariance term in this context? Please provide clarification.\n\n11. Lines 163-164: There seems to be a typo in the sentence, \"we calculate multi-view prediction of the by.\" Please clarify this phrase.\n\n12. Do you have any formal proof of Theorem 2.5? \n\nI am willing to increase my score if these concerns are resolved.", "limitations": "The authors have adequately addressed the limitations ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 1, "contribution": 4, "strengths": "1. The paper presents good experimental results, demonstrating the effectiveness of the proposed cautious pseudo labeling method.\n2. The authors have conducted comprehensive experiments\n3.  The error bound proposed in the paper is deemed useful and practical", "weaknesses": "The paper suffers from unclear notation and contains numerous typos. These issues hinder the reader's understanding and make it challenging to follow the presented ideas. Clarifying the notation and addressing the typos is necessary to improve the clarity of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We sincerely appreciate your efforts in providing us with a detailed review. We have carefully considered all of your insightful suggestions and corrections, and we have incorporated them into the latest version of our draft. We have addressed each comment individually as follows.\n\n> 1. Line 86: For \"err(g)\", does it mean the expectation is taken over the test points?\n\nYes. It measures the performance of the proposed model on the test set. Besides, we also use it to quantify the PL accuracy during the training, where the expectation is taken over the selected PL samples.\n\n> 2. Line 91: Is the GPI property introduced in this paper, or has it been introduced previously? \n\nAnalogous analyses of GPI are frequently encountered, such as \"label-invariant augmentation\" as Definition 4.1 in [1], or the \"augmentation invariant\" in self-supervised learning as Definition 1in [2]. These analyses typically assume the invariance of categories/representation or serve as regularization techniques. In our approach, we relax these constraints by incorporating the Lipschitz regularity, a principle commonly employed in the theoretical analysis of neural networks, as Definition 1in [3] and Definition 1in [4].\n\n[1] Yu, Junchi, Jian Liang, and Ran He. \"Mind the Label Shift of Augmentation-based Graph OOD Generalization.\" IEEE/CVF CVPR. 2023.\n\n[2] Hua, Tianyu, et al. \"On feature decorrelation in self-supervised learning.\" IEEE/CVF ICCV. 2021.\n\n[3] Virmaux, Aladin, and Kevin Scaman. \"Lipschitz regularity of deep neural networks: analysis and efficient estimation.\" NIPS 31 (2018).\n\n[4] Arghal, Raghu, Eric Lei, and Shirin Saeedi Bidokhti. \"Robust graph neural networks via probabilistic Lipschitz constraints.\" Learning for Dynamics and Control Conference. PMLR, 2022.\n\n> 3. Line 104: When stating \"whose probability is higher than a threshold,\" does it mean this condition must hold for each \"y ∈ U\"? \n\nYes, the probability of the elements $y$ in the local optimal set $U$ should be higher than the threshold.\n\n> 4. Line 105: Since \"g(G) ∈ ℝ^(N×M)\", what do you mean by \"ŷ ∈ g(G)\"? \n\nSorry for the notation mistake. g(G) is the output N×M confidence matrix, ŷ is the N×M probability prediction of the augmented graph. It should be ŷ = g(G) here.\n\n> 5. Line 102: Do you mean we can find \"p_f(⋅)\" or we can find \"α, η\"? Please clarify this statement. & 6. Line 108: In \"p_{αf}\", there should be no \"α\" here. \n\nThanks for your judgment. The conclusion should be p_{αf}(U/Uε)>=p_{αf}(U)+αη. This proposition is an assumption on the continuity of the trained GNN-based confidence predictor over the local optimal set. When we apply the graph augmentation, the perturbed local optimal set becomes larger. The increased subset still satisfies the inequality under amplified measure. We aim to find the probability measure for the prediction \"p_f(⋅)\" rather than \"α, η\", which refers to the trained GNN in the experiment. We only wish to illustrate the continuity property. Thus, the proposition only states the existence of the coefficient pair (α,η). Determining the value and analyzing their influence are not the point of our study. We conduct the similar analysis of proposition 1 in [5], they show in detail the bound of coefficient α.\n\n[5] Zhang, Yuchen, Percy Liang, and Moses Charikar. \"A hitting time analysis of stochastic gradient langevin dynamics.\" Conference on Learning Theory. PMLR, 2017.\n\n> 7. Line 122: When stating \"the teacher predictor satisfies additive expansion,\" the additive expansion is defined for a probability density, not a classifier. \n\nSorry for the inaccurate statement. We refer to the confidence predictor g in the teacher model. Its corresponding probability density refers to f in proposition 2.2. It is hard to provide theoretical analysis on the derivative of the neural networks, we can only give the assumption based on the local smoothness of GNN. We could revise the prerequisite of the theorem as \"For the GNN in the teacher model, if its corresponding density measure satisfies additive expansion.\" \n\n> 8. Line 124: Could you explain what is meant by \"E_{Y_test}\"?\n\nIt refers to the expectation taken over the test set. As the reply to your 1st question, the expectation of Err(g) is taken over the test set. Then the inconsistency term A in Theorem 2.3 should be taken from the same range. In the application when the ground truth is unknown, this term could be estimated from the validation set or training set. But the difference between the estimation and the theorem is not the key point in our study.\n\n> 9. Line 145: The first term should change \"t\" to \"t+1\".\n\nYes. According to the definition in Algorithm 1, it should be $CE(g(t)_psi,ŷ_o(t+1))>=CE(g_phi(t),ŷ_o(t))$.\n\n> 10. Line 147: What is the definition of the covariance term in this context?\n\nThere are two variables in the covariance term, the cross-entropy and the PL strategy. In this study, we define the PL strategy T as an indicator function of the N PL candidates. The output is 1 for the selected PL samples, and 0 for the non-PL samples. The cross-entropy term also consists of N elements, each of which is the difference between the predicted confidence of the candidates and their ground truth label. The covariance quantifies the relation between the cross-entropy and PL indicator function over these N candidate samples.\n\n> 11. Lines 163-164: There seems to be a typo in the sentence, \"we calculate multi-view prediction of the by.\" \n\nThanks for pointing out the mistake. It should be  \"we calculate the multi-view prediction by the teacher model.\" \n\n> 12. Do you have any formal proof of Theorem 2.5?\n\nWe have shown all the detailed proof of the first inequality in Appendix. It is hard to conduct theoretical proof of the second inequality, as it would be the symbolized representation of the analysis in Line 184-190."}, {"review_id": "5EhfAtAmC7", "reviewer": "Reviewer_io9b", "summary": "Pseudo labeling is significant for GNN. This paper first theoretically analyzed the effect of pseudo labeling by showing the error bound and the convergence property. Then, accordingly, the paper proposes a cautious pseudo labeling based on confidence and multi-view consistency. The experimental results demonstrate the effectiveness of the proposed cautious pseudo labeling strategy.", "questions": "1) The paper is well written. The analysis of Pseudo labeling for GNN is valuable. However, the analysis or the conclusion is general to all fields instead of highly related to the graph. Besides, the solutions, including high confidence and multi-view consistency, are not that novel. \n\n2) Some contents may contain errors. For example, the results of Figure 1 may contain some errors. When the samples of PL and CPL are equal to/similar to 0, the GAE/GAE+PL/GAE+CPL should have equal/similar performance.\n\n3) More details need to be provided. What are the details of multi-view teachers? Does it refer to multi teachers or one teacher with multi-data-augmentation inputs? If the multi-view is obtained based on the data augmentation, what types of data augmentation are used? What is the data augmentation in Figure 4?\n\n4) More experiments are required. The time complexity should be provided when comparing with previous methods. Performance comparison on link prediction should also contain the previous PL methods.\n\n5) Other questions. In Algorithm 1, the student was fine-tuned based on the teacher model. I wonder whether there exists overfitting since the observed data set has been put into the model again and again. In 3.1, I wonder whether the data from the testing would be selected by CPL to be labeled for training. I wonder whether CPL may reduce the performance of nodes of low-frequently labeled classes.", "limitations": "There is no potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written. The analysis of Pseudo labeling for GNN is valuable. ", "weaknesses": "However, the analysis or the conclusion is general to all fields instead of highly related to the graph. Besides, the solutions, including high confidence and multi-view consistency, are not that novel. \n\nMore experiments are required. The time complexity should be provided when comparing with previous methods. Performance comparison on link prediction should also contain the previous PL methods.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. Please find our response which addresses your concern.\n\n1. \n\nThanks for your agreement that our paper is well written and this is really a good question. Our analysis of the PL strategy focuses on the task of graph learning. We make assumptions regarding the graph properties, i.e., Graph Perturbation Invariant (GPI) property and Additive Expansion Property (AEP). GPI assumes that the representation of the graph does not undergo significant changes when augmentation is applied. AEP assumes the continuity of the probability density in the neighborhood of the local optimal set. It is highly likely that these assumptions hold true for GNNs. The idea of message passing in GNNs smooths out the discrete variations caused by augmentation and ensures continuity. Additionally, GNNs are typically shallow networks, which implies the existence of a measure that prevents extreme fluctuations in values. However, it remains unclear whether these assumptions hold in other scenarios such as tabular data.\n\nThe main contribution of our study lies in providing insights into PL in the context of graph learning. While thresholding and multiview augmentation techniques are not new in the machine learning community, their adoption in our study is a natural solution as indicated by our analysis, rather than trivial combinations in other empirical studies. We aim to quantify the error bound of PL by incorporating these techniques. Our theoretical analysis demonstrates that the performance of PL is influenced by these two factors. Therefore, we combine them to the PL and quantify their impact. Specifically, the noisy PL belong to the augmentation space, and thresholding is used to filter the local optimal set, and its consistency can be considered an explicit measure of the error bound of PL.\n\n2.\n\nThe results depicted in Fig.1 are correct based on our experiments. These experiments were conducted with specific PL capacities, i.g. 2k, 4k, ..., and 100k for the WiKiCS. And we employed a polynomial fitting curve. To avoid overfitting, we refrained from using a high-order polynomial. Thus, there might be two factors contributing to the initial value problem:\n\na) The original training set is relatively small. Consequently, the initial PL samples have a relatively substantial impact and enhance the prediction performance. This also influences the initial value.\n\nb) The fitting result is influenced by the nonlinear relationship between the variables. The nonlinearity can affect the initial fitting value of the function.\n\n3.\n\n\"multi-view teacher model\" refers to a single teacher model with multiple inputs, each of which is augmented separately. We apply various data augmentations to the original graph. These augmented graphs are fed into the teacher model individually. The final prediction is the average predictions generated from these inputs. In the next, we analyze the consistency among these predictions.\n\nIn Fig.4, we employed three different augmentation methods: drop node (Node view), feature mask (Feature view), and DropEdge (Structure view). In each experiment, a single augmentation method was applied three times. As for Multiview, we applied each augmentation method once as a combined augmentation. The \"Random'' refers to the random selection of samples during PL.\n\n 4.\n\nThe overhead of the CPL stems from two main components: the calculation of confidence scores and the fine-tuning process on the enlarged training set. In each iteration, the time required to compute the confidence scores for the PL candidates is approximately the evaluation time of the test set. For the fine-tuning, the total epoch number is roughly twice that of  the pre-training epochs. The total time consumption for fine-tuning is expected to be twice that of the base model.The averaged time consumption (second) :\n\nNode classification:\n\n| Base model | Cora | CiteSeer | PubMed | AmazonPhoto | LastFMAsia |\n| -- | -- | -- | -- | -- | -- |\n| GCN | 115.8 | 241.0 | 147.6 | 302.6 | 104.0 |\n| GAT | 251.4 | 450.0 | 388.8 | 652.2 | 338.6 |\n| SAGE | 141.6 | 237.8 | 181.2 | 347.0 | 134.6 |\n| APPNP | 511.2 | 1005.2 | 820.0 | 843.75 | 576.2 |\n\nLink prediction:\n\n| Base model | CiteSeer | Actor | WikiCS | TwitchPT | AmazonPhoto |\n| -- | -- | -- | -- | -- | -- |\n| GAE | 452 | 6376 | 7412 | 3401| 4956 |\n| node2vec | 691 | 5067 | 6537 | 2740 | 5470 |\n| SEAL | 2982 | 14579 | 17491 | 11639 | 24904 |\n\nWe compare with another study, i.e. EdgeProposal, that proposes a similar approach for link prediction by introducing possible edges during training. The comparison experiment is provided, with metric Hit@20 (%) \n\n| Dataset |GAE | EdgePropsal | CPL | \n| -- | -- | -- | -- |\n| ogb-ddi | 41.4 | 53.4 | 58.9 | \n| ogb-colab | 60.1 | 60.4 | 60.6 | \n\n5.\n\nWe acknowledge that one of the weaknesses of PL-based methods is the risk of overfitting as PL samples often have similar representations to the original training set. However, the error-labeled samples may have a significant influence on the model. To prevent the model from being misled by the error introduced, it is necessary to retain the original training set. Besides, we incorporated a validation set, and the results demonstrated that the CPL does not suffer from overfitting.\n\nThe proposed model is transductive learning. The test set is also the candidate for PL. However, their labels are totally isolated from the training process.\n\nWe did not specifically observe the outcome of imbalanced PL. To explore this further, let us consider an extreme story in which we only pseudo label some classes rather than all classes. This experiment was conducted on CiteSeer with GCN for the node classification task. During CPL, we adjust the imbalance ratio by only pseudo labeling some specific classes. \n\nImbalance CPL:\n\n| # PL class | 0(raw) | 1 | 2 | 3 | 4 | 5  | 6(balanced) |\n| -- | -- | -- | -- | -- | -- | -- | -- |\n| AUC(%) | 69.32 | 69.65 | 70.02 | 70.12 | 70.93 | 72.22 | 72.96 |"}, {"review_id": "vjDCljHnuC", "reviewer": "Reviewer_4kBQ", "summary": "This paper aims to provide in-depth insights into pseudo labeling (PL) in the context of graph learning models. The authors first present an error analysis of the PL strategy, demonstrating that the error is bounded by the confidence threshold of PL and the consistency of multi-view predictions. Furthermore, they theoretically illustrate the impact of PL on convergence properties. Building upon this analysis, they propose a careful pseudo labeling methodology that involves assigning pseudo labels to samples with the highest confidence and multi-view consistency. Finally, extensive experiments demonstrate that the proposed strategy enhances the graph learning process and outperforms alternative PL strategies in link prediction and node classification tasks.", "questions": "How much time does the proposed method consume on the dataset like PubMed?", "limitations": "Some figures could be further polished up, and check the correctness of the theorem.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper theoretically analyzes how the PL strategy affects convergence properties in GNN.\n2. Based on the analysis, this paper proposes a cautious pseudo labeling methodology, and extensive experiments demonstrate the effectiveness of the proposed strategy.", "weaknesses": "1.There might be some mistake in Theorem 2.5. since the covariance between the cross-entropy loss and PL strategy and $\\beta$ is non-negative, the inequality $\\beta Cov + L \\leq L$ holds only if $\\beta Cov = 0$.\n2.It might be better to clearly illustrate $q(t)$ in Figure 2, which demonstrates the framework of the algorithm.\n3.It is suggested to remove the first picture in Figure 1 and add two more pictures to show the performance on node classification.\n4.The word ‘noisy’ in the title is redundant for pseudo labeling. Furthermore, the whole paper rarely mentions ‘noisy PL’.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reading and insightful comments. Following are the responses regarding your concerns. \n\nFor the figures in the rebuttal, please check the PDF file in the global response of the author rebuttal, located at the top of this page.\n\n> 1.There might be some mistake in Theorem 2.5. since the covariance between the cross-entropy loss and PL strategy and  \\beta  is non-negative, the inequality $\\beta Cov + L \\leq L$  holds only if $\\beta Cov = 0$\n\nWe are confident that Theorem 2.5 contains no errors. Our analysis in lines 184-190 demonstrates that the covariance term, denoted as Cov, is never greater than 0. Considering that \\beta is non-negative, the product of \\beta and Cov would be non-positive. Consequently, the loss decreases iteratively, leading to a smaller overall loss.\n\n> 2.It might be better to clearly illustrate  q(t)  in Figure 2, which demonstrates the framework of the algorithm. 3.It is suggested to remove the first picture in Figure 1 and add two more pictures to show the performance on node classification. 4.The word ‘noisy’ in the title is redundant for pseudo labeling. Furthermore, the whole paper rarely mentions ‘noisy PL’.\n\nThanks for your advice. We add the q(t) in Fig.2, as Fig. a in the attachment of our very first response. \n\nWe have conducted experiments to investigate the impact of PL capacity on node classification performance. The results indicate that initially, as the number of PLs increases, there is an improvement in model performance. Then the improvement is followed by a degradation in performance until no nodes remain above the confidence threshold. It is important to note that due to the significantly smaller number of nodes compared to edges, our experiments only capture the first scenario illustrated in Fig. 2, as shown in Fig. b,c of the attachment in our very first response. This is why we specifically selected the PL experiment for the link prediction task in the introduction.\n\nThe primary contribution of our study lies in the theoretical analysis of the error induced by PL, which is a major drawback associated with noisy labels. The entire paper focuses on quantifying and mitigating the influence of noisy PL, addressing this critical issue in label noise research.\n\n> How much time does the proposed method consume on the dataset like PubMed?\n\nThe averaged time consumption (seconds) on PubMed is reported as the following table:\n\n| Base model | GCN | GAT | SAGE | APPNP | \n| -- | -- | -- | -- | -- |\n| Time(s) | 147.6 | 388.8 | 147.6 | 181.2 | 820.0 |\n\n> Some figures could be further polished up, and check the correctness of the theorem.\n\nThanks for your advice. We would update to more clear figures in the latest version and correct the typos in the paper. Please refer to the attachment in our first overall response.\n"}]

Author rebuttal:
The revised main scheme and the required figures of the experiment from  **Reviewer 4kBQ** are shown in the supplementary PDF file.

Meta-review: The paper studied pseudo labeling on graph data, namely how to mitigate the negative effect of incorrect pseudo labels. The authors presented theoretical analysis identifying two important factors, pseudo-label confidence and multi-view consistency, and then proposed a practical method based on the theoretical understandings. The paper has strong motivation and contributions, and the four reviewers were all positive. Thus, we should accept the paper for publication.
Decision: Accept (poster)