PI-GNN: Towards Robust Semi-Supervised Node Classification against Noisy Labels
Authors: Xuefeng Du
Tian Bian
Yu Rong
Bo Han
Tongliang Liu
Tingyang Xu
Wenbing Huang
Junzhou Huang
Venue: ICLR 2022 Submitted (2022)

Abstract: Semi-supervised node classification on graphs is a fundamental problem in graph mining that uses a small set of labeled nodes and many unlabeled nodes for training, so that its performance is quite sensitive to the quality of the node labels. However, it is expensive to maintain the label quality for real-world graph datasets, which presents huge challenges for the learning algorithm to keep a good generalization ability. In this paper, we propose a novel robust learning objective dubbed pairwise interactions (PI) for the model, such as Graph Neural Network (GNN) to combat against noisy labels. Unlike classic robust training approaches that operate on the pointwise interactions between node and class label pairs, PI explicitly forces the embeddings for node pairs that hold a positive PI label to be close to each other, which can be applied to both labeled and unlabeled nodes. We design several instantiations for the PI labels based on the graph structure as well as node class labels, and further propose a new uncertainty-aware training technique to mitigate the negative effect of the sub-optimal PI labels. Extensive experiments on different datasets and GNN architectures demonstrate the effectiveness of PI, which also brings a promising improvement over the state-of-the-art methods.

Under review as a conference paper at ICLR 2022
PI-GNN: TOWARDS ROBUST SEMI-SUPERVISED NODE CLASSIFICATION AGAINST NOISY LABELS
Anonymous authors
Paper under double-blind review
ABSTRACT
remi-supervised node classification, as a fundamental problem in graph learning, leverages unlabeled nodes along with a small portion of labeled nodes for training. Existing methods rely heavily on high-quality labels, which, however, are expensive to obtain in real-world applications since certain noises are inevitably involved during the labeling process. It hence poses an unavoidable challenge for the learning algorithm to generalize well. In this paper, we propose a novel robust learning objective dubbed pairwise interactions (PI) for the model, such Graph Neural Network (GNN) to combat noisy labels. Unlike classic robust training approaches that operate the pointwise interactions between node and class label pairs, PI explicitly forces the embeddings for node pairs that hold a positive PI label be close each other, which can be applied both labeled and unlabeled nodes. We design several instantiations for PI labels based on the graph structure and the node class labels, and further propose a new uncertainty-awer to mitigate the negative effect of the sub-optimal PI labels. Extensive experiments on different datasets and GNN architectures demonstrate the effectiveness of PI, yielding a promising improvement over the stale-of-the-art methods.
training technique
INTRODUCTION
Graphs are ubiquitously used to represent data in different fields, including social networks, bioin- formatics, recommendation systems, and computer network security. Accordingly, graph analysis tasks, such node classification, have significant impact in reality (Lan et al., 2020). 'he success of machine learning models, such as graph neural networks (GNNs) on node classification relies heavily on the collection of large datasets with human-annotated labels (Zhou et al. 2019). However, it is extremely expensive and time-consuming to label millions of nodes with high-quality annota- tions. Therefore, when dealing with large graphs, usually subset of nodes is labeled, and wide spectrum of semi-supervised learning techniques have emerged for improving node classification performance (Zhu et al. 2003 Zhou al. 2003 Kipf & Welling) 2017).
Although achieving promising results, these techniques overlook the existence of noisy node labels. For instance, practitioners often leverage inexpensive alternatives for annotation, such combining human and machine- generated label (Hu et al., 2020), which inevitably yields samples with noisy labels. Since neural networks (including GNNs) are able to memorize any given (random) labels (NT et al. 2019: Zhang et al., 2017). these noisy labels would easily prevent them from generalizing well. Therefore, training robust GNNs for semi-supervised node classification against noisy labels becomes increasingly crucial for safety-critical graph analysis, such as predicting the identity groups of users in social networks or the function of proteins to facilitate wet laboratory experiments, etc.
machine-generated
A natural solution is extending the robust training techniques specifically designed for image data into graphs, such as by estimating noise transition matrix (NT et al., 2019), reighigt training samples (Xia et al.) 2021 Li et al.] 2021 Dong et al., 2020) and modifying model structure (Li) ctt al., 2020a), which achieve promising results. However, there are two limitations. Firstly, some techniques require a clean set of node and label pairs for training, which are not practical to obtain under extremely noisy settings (Li et al. 2021). Secondly, all of them solely rely on the noisy pointwise input-label interactions during training. Although some of them implicitly exploit other attributes (the clean graph structure, etc.) that encodes the pairwise information between nodes, such aggregating features by GNNs the input graphs or launching label propagation to get the pseudo

---

Under review as a conference paper at ICLR 2022
Nodes with clean class labels
Nodes with noisy class labels X5
Mortons
X1 Xg Xg
X4 Kg Ng Xy Xg.
22 X3
X4
Xg
y
Noise ratio-50%
3 1.30
1.25
Nodes with clean P1 labels
Nodes with noisy PT labels
0.75
X1
X3
My
Xa
X,
X2
X2
X4
X5
N
X;
Xn
2
4
4
4
4
0.30
X1
0
0
0
0
X1
0
0
0
0
0 0
(b) Confidence distribution for vanilla GNN
N2
1
0
0
0
32
2
0
0
0
X3
0
0
X4
Noise ratio-25%
Xs
3
0
0
X6
0
X6
3
0
0
0
X6
4
0
0
0
0
37
0
0
0
4
0
X8
(a) Nouse ratio comparisen betweeen node labels and PI labels
(6) The effect of uncertaiinty-aan aawrre training
Figure I: (a) Noise ratio comparison with noisy node labels, which shows the noise ratio of PI labels is much smaller than that of node labels. Red number means noisy label. (b) Density plot for the produced confidence mask with 60% symmetric noise. x-axis denotes the confidence value, where there exist uncertain PI predictions. (c) Test accuracy of the GNN trained by the PI loss from all node pairs (Uniform), the node pairs with larger (Larger Confi.) and smaller confidence (Smaller Confi.), all node pairs weighted by the confidence mask (PI-GNN). Models trained on node pairs with larger confidence perform much better than those trained with lower confidence or without PI labels. PI label is instantiated as the adjacency matrix here.
labels for unlabeled nodes, they still operate on the noisy embeddings for inference, which may lead to limited performance. Thus, designing a robust GNN training technique for node classification against noisy labels is still a challenging problem.
In this paper, we propose a novel framework, which explicitly leverages the Pairwise Interactions in Graph Neural Network (PI-GNN) to perform robust training without auxiliary clean labels. The PI, which is based on the similarity of two nodes, is meaningful for GNN against noisy labels because the noise rate for the PI labels can be much lower than that of the pointwise noisy class labels (Figure a Consider two nodes from the same class have the same noisy labels, their similarity is still high w.r.t the PI, which is helpful for the model learn. Meanwhile, the pointwise information will inject both of these noisy class labels into GNN, which hurts its generalization performance.
To ensure an effective PI learning procedure, two important components are highlighted. The first one is how to design informative PI labels. We propose two instantiations: 1) we use the adjacency matrix as PI labels so that if two nodes are connected, their PI label is positive. 2) We compare the noisy labels, i.c., two nodes that have the same class labels are given a positive PI label. During training, PI-GNN explicitly forces the embedding similarity of two nodes to close to the labels. The second one is how to deal with the sub-optimal PI labels during regularization. It is critical to notice that the two instantiations are not optimal. Firstly, connected nodes do not guarantee to have the same (clean) class label, so their PI label is not always positive. Secondly, the class labels are noisy, and comparing them to obtain the PI labels inevitably introduces noise. To address this problem, introduce uncertainty estimation, which quantifies the reliability of each PI label (b)). Specifically, we propose to exploit a separate GNN only trained with the PI learning objective and calculate confidence mask for each node pair from its PI prediction. Then, we filter out uncertain pairs by weighting the PI loss from another GNN jointly trained with the PI loss and node classification loss. Figure [cc) shows the accuracy of GNNs trained on the node pairs with higher and lower confidence on CiteSeer (Yang et al. 2016) and the former one performs better, which justifies the intuition of PI-GNN. Our main contributions are summarized as follows:
We propose robust GNNs against noisy labels for node classification, which serve as a crucial step towards the reliable deployment of GNNs in complex real-world applications. We introduce promising strategy to explicitly model the pairwise interactions and an uncertainty estimation approach to mitigate the negative effects of sub-optimal PI labels. We demonstrate PI-GNN can be effectively applied on different datasets, GNN architectures and different noise types and rates, e.g., improving the test accuracy by 6.7% on CiteSeer with 80% asymmetric noise.
RELATED WORK
Graph neural networks. Graph neural networks have been widely used to model the graph- structured data with various architectures, such as graph convolutional network (GCN) (Kipf & Welling 2017), graph attention network (GAT) (Velickovie et al. 2018), GraphSAGE (Hamilton
2

---

Under review as a conference paper at ICLR 2022
et al.
2017). Graph Isomorphism Network (GIN) (Xu al. 2019). Simple Graph Convolution
(SGC) (Wu et al. 2019), etc. Common graph analysis tasks, including node classification (Prrk &
Neville 2019 Oono & Suzuki, 2020). link prediction (Back et al. 2020 Zhang & Chen (2018).
graph classification (Bacciu et al. 2018 Hrrica et al. 2020), graph generation (Liao et al. 2019 [Shi
et al. 2020), have been widely studied in literature. However, only few works focused on training robust GNNs against noisy labels, such as by loss correction (NT et al. 2019) for graph classification,
sample re-weighting (Xia al. 2021 Li et al. (02I) for node classification. None of them exploited explicit pairwise interactions, which are compared with our PI-GNN in Section 5.4 Bui et al. (2017);
Stretcu al. 2019) utilized graph structures for cupe-supee learning but with clean labels.
Neural networks with noisy labels. Methods for neural networks against noisy labels can be roughly categorized into three types (Han et al., 2020b), i.e., approaches from the perspective of data (van Rooyen & Williamson, 2017), objective (Reed et al. 2015: MMyato et al. 2019) and optimization (Arpit et (017). Methods based on data mainly built the noise transition matrix to explore the data relationship between clean and noisy label by an adaptation layer (Sukhbaatar et al.
2013), loss correction (Patrini et al. 2017) and prior knowledge (Han et al. (2018)). Methods based on objective modified the learning objective by regularization (Han et al. [2020a), reweightin (Liu & Tao 2016; Wang et al.) (017)) and loss redesign (Thulasidasan et al. 2019). Methods based on optimization mainly changed the optimization policy, such by memorization effect (liang et al.. 2018), self-training (Ren et al. 2018) and co-training (Yu et al. (019). Wu et al. (2020) proposed to use the similarity loss for noisy labels on image data but relied on the noisy transition matrix, which is sensitive to the matrix estimation quality and cannot use the graph structure for regularization. In this paper, we extend several approaches from each category to compare with PI-GNN Section 5.4
PRELIMINARY
Graph Neural Networks. Let G (V,E) be graph with node feature vectors X, for e and edge set E. GNNs use the graph structure and node features X, to learn a representation vector of node hy, or the entire graph hG, which usually follow a neighborhood aggregation strategy and iteratively update the representation of node by aggregating representations of its neighbors. After k iterations of aggregation, a node's 's representation captures the structural information within its k-hop network neighborhood. Formally, the k-th layer of a GNN is
ay AGGREGATE (fth (( :ue N(m))
A COMBINE (al+=",...*))
where k/ is the feature vector of node at the k-th layer. A. X,, (v)) denotes the neighboring nodes of v. The choices of AGGREGATE (.). and COMBINE (+) can be diverse among different GNNs. For example, in GCN, the element-wise mean pooling is used, and the AGGREGATE and
COMBINE steps are integrated as follows:
by (k)
1h ReLU (W- (W-MEAN (k-1) Vu (uue(())) ((v)))))))))))
(2)
where W is learnable matrix. For node classification, each node E V has associated label yv. the node representation (K) of the final layer is used for prediction.
Label-noise representation learning for GNNs. Let X,, be the feature and y, be the label for node v, we deal with a dataset D {D",, D'e which consists of training set D" (..,x. J that drawn from a corrupted distribution D = Y.) where Y denotes the corrupted label. Let Y)ybi be the non-corrupted joint probability distribution of features X and labels and f be the (Bayes) optimal hypothesis from to y. To approximate f", the objective requires a hypothesis space H of hypotheses fo(-) parametrized by 0. A robust algorithm against noisy labels contains the optimization policy to search through H in order to find 0 that corresponds to the optimal function in the hypothesis for D" fe E H. and meanwhile is able to assign correct labels for Dle.
p(A,X, Y)
4 PROPOSED APPROACH
In this section, we introduce our proposed PI-GNN, which mitigates the negative effects of noisy labels for semi-supervised node classification by explicitly exploiting the pairwise interactions in
3

---

Under review as a conference paper at ICLR 2022
== (A
hm
h
m
T
X
Mask Generator fm
Ep (l hms ypI)
Confidence Mask
h
ht x
Epr (nr ht, yP1) lots (ht, 7)
Task Executor ft
Figure 2: The framework of PI-GNN, which has GNNs, i.e., mask generator and task executor for robust semi-supervised node classification. node pair by enforcing the embedding similarity to be close to the PI labels ypy. The mask generator generates a confidence mask, which is applied to the PI loss of the task executor to reduce the uncertainty of its predictions caused the sub-optimal Pl labels. denotes dot product and means element-wist multiplication.
two GNNs fm learn the pairwise interactions between each
GNNs. In what follows, will first provide method overview and then illustrate the learning objective to enhance the pairwise interactions in PI-GNN (Section 4. We introduce the uncertainty- aware robust training approach by using PI for regularization in Section 4.2
Overview. Figure 2 demonstrates the overview of PI-GNN, which is composed of two different GNNs. The first one is only trained with the PI learning objective, whose outputs are used to generate the confidence mask. We denote it as the mask generator fm- The second GNN is trained with both the PI learning objective and the noisy node classification objective, which used to perform the node classification task. We denote it as the task executor f. The PI loss for each node pair in the task executor multiplied by the confidence mask from the mask generator in order to reduce the uncertainty caused by the collected sub-optimal PI labels.
4.1 LEARNING FROM PAIRWISE INTERACTIONS
Let us suppose the semantic class labels for certain nodes in the training set D are corrupted. Since, ultimately, we are interested in finding a GNN model parametrized by θ that minimizes the generalization еπтог clean test set Die natural solution is to exploit additional information for the learning algorithm to find a robust parameter 0 in the hypothesis space H. One straightforward candidate for such information is leveraging the pairwise interactions between two nodes to perform extra regularization, whose learning objective is shown to hold a much smaller noise rate than that with the noisy class labels (Figure [a)).
(A,)) F.))))))) hevv
Construct informative PI labels. In order to perform PI learning, the first step is to construct reliable PI labels yrι E X to alleviate the negative effect of noisy labels for the GNNs. Here [V] is the cardinality of the vertex set the input graph G. While reasonable choice of ypι is by comparing whether two nodes have the same class label and assign those with the same class label a positive PI label, impossible to obtain such PI labels with noisy class labels Therefore, assume i, and A as the "and" operator, we relieve such condition by proposing two relaxed instantiations:
1/ A or 2 ーンAド,
(3)
where the first one models the PI between two nodes by their connectivity in the input adjacency matrix A. The second one firstly performs label propagation (Zhu & Ghahramani 2002) to get the class labels for each node the input graph. Then assigns the PI label of node pair < < i.j to 1 if their noisy class labels are the same and 0 otherwise. Both of them are not optimal because firstly, the fact that two nodes are connected does not mean their clean class labels are the same. Secondly, comparing the noisy class labels also creates noisy PI labels if the same clean class labels are corrupted to two different noisy labels despite the noise rate greatly reduced. We provide an uncertainty aware training approach in Section 4.2 to alleviate this problem.
Learning objective for enhancing pairwise interactions. Given node embeddings h which is calculated by h f(A,X, x,,). 0), let hy h2 be the dot product between two embeddings, and P(h! -hj)
4

---

Under review as a conference paper at ICLR 2022
be the estimated probability of the node pair <ij> has the positive pairwise interaction, the PI learning objective Ep E I2 [[V] is formulated as follows:
Algorithm PI-GNN: Learning Pairwise Interactions for GNNs against noisy labels
Input:
Input graph XV with noisy training data D { (v), F.) vev randomly initialized
GNNs fm and f, with parameter 6n and 8, weight for PI loss B, pretraining epoch K for fm- Total training epoch N.
Output: Robust GNN fi against noisy labels.
for epoch = epehh < N:epoch++ + do
if epoch < K then
Set M = 11, update the parameter 8, of the mask generator fi by Equation Anluationaaand and the parameter
0, of the task executor fi by Equation 7
else
Estimate the confidence mask M by Equation 5 with the mask generator fn- Calculate the PI loss Eyy for the task executor by Equation 6
Update the parameter 6m of the mask generator fm by Equation and the parameter 6 of the task executor by Equation
end
end
return The model fr.
Ep (I:ypr) = Σ
log
P(M h1)++
Σ
-lg(( Phh -h)),,
(4)
<if>etii
<ij>>Bpp
where Bp, Bp denote the node pairs that hold the positive and negative PI labels, respectively.
4.2 UNCERTAINTY-AWARE ROBUST TRAINING
Uncertainty estimation. In order train a robust GNN that not sensitive to the sub-optimal PI labels, we resort to uncertainty estimation and reduce the negative effect of the node pairs that the model is uncertain about during regularization (Equation f Specifically, we measure the uncertainty by calculating the confidence map of the PI predictions follows:
oh? h:), Mii. 1 1 hy),
Fp)(i, =1 (i) =0
(5)
where (i, denotes the PI label between node i and j and a(-) is the sigmoid function. The confidence map M measures the uncertainty by looking at the closeness between the prediction and the given PI label. If the prediction becomes close to the given labels more easily, then the reliability of the PI labels is higher and more attention should paid for such node pairs.
Therefore, we introduce a weiihtttn mechanism for the PI loss Epy = Erj @M where the re- weightet PI loss is obtained by multiplying the confidence mask and original PI loss in an element- way. However, since the GNN is trained with the noisy class labels at the same time, the confidence mask cannot be estimated well.
Decoupling with two branches. In this paper, as shown in Figure 2 we propose decouple the confidence mask estimation and node classification by using two separate GNNs, which are referred mask generator fm and task executor f.- The mask generator generates the confidence mask M by only learning with the PI objective. The task executor uses the mask from the mask generator to re-weight different node pairs in order to combat the noisy PI labels as follows:
E = Epy M,
where Ep denotes the loss for the task executor fi and @ means element-wise multiplication.
The PI learning procedure allows for explicitly exploiting the pairwise interactions between two nodes, resulting in a GNN that is affected less by the noisy class labels. Wu et al. (200)) employed similarity labels ypy for regularization. However, it transforms the noise transition matrix estimated for noisy class labels to correct the similarity labels, which sensitive to the matrix estimation quality. Meanwhile, the pairwise interactions from the input adjacency matrix cannot be explored.
5

---

Under review as a conference paper at ICLR 2022
Overall training. Put them together, we introduce new robust training objective for node classifi- cation against noisy labels on GNNs, leveraging the pairwise interactions in Section 4.1 The key idea is to perform the node classification task by the task executor fi while regularizing Je to produce similar embeddings for nodes that have a closer PI and vice versa. The overall uncertainty-aware training objective for the task executor fi is formulated as:
=
where ß is a hyperparameter to balance the node classification loss f"1, and the PI loss Ey. Besides, the mask generator is trained only by the PI loss by and provides confidence mask following Equation 5 which does not touch noisy class labels for learning. During the inference stage, we discard the mask generator fm and only use the task executor for evaluation, which does not affect the inference speed.
Practically, the learning procedure relies heavily on the quality of the uncertainty estimation by fm- Therefore, we pretrain the mask generator fm for K epochs and re-weight the PI loss of the task executor fyy by Equation Note that the task executor still optimized by the un-reweighted PI loss and the noisy node classification loss during pretraining the mask generator. The training outline is presented in Algorithm [
5 EXPERIMENTS AND RESULTS In this section, we present empirical evidence to validate the effectiveness of PI-GNN on different datasets with different noise types and ratios.
5.1 EXPERIMENTAL. SETTING
Datasets. We used five datasets to evaluate PI-GNN, including Cora, CiteSeer and PubMed with the default dataset split as in KKipf & Welling 2017) and DBLP (Pan et al.) (2066) well as WikiCS dataset (Mernyei& & Cangea 2020j. For the latter two datasets, we used the first 20 nodes from each class for training and the next 20 nodes for validation. The remaining nodes for each class are used as the test set. The statistics of these datasets are summarized in Appendix Section B
Since all datasets are clean, following Patrini et al. (2017). we corrupted these datasets manually by the noise transition matrix 2, = P yj 1) given that noisy y is flipped from clean y. Assume the matrix Q has two representative structures: (1) Symmetry flipping (van Rooyen et al., 2015); (2) Asymmetric pair flipping: a simulation of fine-grained classification with noisy labels, where labelers may make mistakes only within very similar classes. Note the asymmetric case is much harder than the symmetry case. Their precise definition is in Appendix Section A
We tested four different noise rates E E {0.2,0.4, 0.6,4,06,000 0.8, in this paper for two different noise types, which cover lightly and extremely noisy supervision. Note that in the most extreme case, the noise rate 80% for pair flipping means 80% training data have wrong labels that cannot be learned without additional assumptions.
Implementation details. We used three different GNN architectures for evaluation, i.e., GCN, GAT and GraphSAGE, which are implemented following the BorkkkggeEorchhhooooo t (Fey & Lenssen 2019). All of them have two layers. Specifically, the hidden dimension of GCN, GAT and GraphSAGE is to 16, 8 and 64, respectively. GAT has 8 heads for attention calculation in the first layer and head in the second layer. The mean aggregator is used for GraphS AGE. We applied Adam optimizer (Kingma & Ba 2015) with learning rate of 0.01 for GCN and GraphSAGE and 0,005 for GAT. The weight decay is set to 5e-4. We conducted training for 400 epochs on a Tesla P40. The regularization loss weight / set to ////////// 2 M) where |V is the number of nodes and M is the number of edges in the input graph G. The number of pretraining epochs K is set to 50 and the total epoch K is 400. The PI label is constructed based on the adjacency matrix by default. We tuned all the hyperparameters on the validation set and reported the node classification accuracy on the clean test set. Details about the ablation studies on these factors are shown in Section 5.5 Each experiment is repeated for 10 times with random seeds from 1 to 10.
5.2 EFFECTIVENESS ON DIFFERENT DATASETS
We evaluated the effectiveness of PI-GNN five datasets with different noisy labels and noise rates, which is shown in Table [ with GCN as the backbone. Specifically, we are interested to observe 1) whether the introduced pairwise interactions between nodes can improve vanilla GNN against noisy

---

Under review as a conference paper at ICLR 2022
Table I: Test accuracy five different datasets for PI-GNN with GCN as the backbone. Bold numbers superior results. Standard deviation is shown in the bracket.
Noise type
No Noise
Symmetric Noise
Asymaetiic
Noise rabio
022
0.4
0.5
ON
0.4
0.6
0.8
GOO
07722011111
DAI TOLUT
0:2911214)
PI-GNN wa/ ue
0.781101)
0.738/0.02)
0.64(0.05)
0.510/0,04)
0,28710.06)
0.717(0.04)
0.563(0.77)
0.349(0.06
0.232
PI-GNN
0.780K0.01
0.732(0.02)
0.664(0.03)
0.515(0003)
0.296(0.05)
0.72340.03)
0.587(0.07)
0.34770.07)
0.209)0.06)
CheSser
GUN
DAURIAA
DNNNTTT U. U.TRRERS))
PL-GNN wx//
0.6661.03))
06060-03)
0.5260.05)
0.378:0.05)
(.222/0.04)
0.5880002)
0472(0.05)
0.328/0.03
(
PL-GNN
0.684(0.03)
0.642(0.03)
0.591(0.03)
0.432(0.07) 0.245(0.05)
0.628(0.03)
0.531(0.0)
0.353(0.06
(.236))
PubMed
GCN
(.7860101)
D. (
PL-GNN PL-GNN
0.7740.000
0.723(0.03)
0.6280.055
0.458(0.07) 0.374(0.06)
0 0.579(0.777) 0.722(0.03)
0.412/0.05)
0.485(0.03)
0.774(0.00)
0.724(0.03)
0.63800.04
0,470c0.08)
0.370(0.07)
0.723(0003
0.583(0,0))
0.425(0.07)
0.401(0.04)
DBLP
GUN
PI-GNN PI-GNN
0.622(0.085
12) 0.4560.12) 0.565(012)
0.294(0.08)
0.253(..0)
0.399(0))) (.521(0.08
(.334((.09)
(.291(0.12)
0.635(0.04)
0.5640.13)
0,456(0.10)
0.301(0.85) 0.258(0.11)
0.55800.88)
0.453(0.09)
0.3
0.261(0.14)
WIEICS
GCN
USSKUTAAT
(2721013))
PI-ONN wa/ uc
(.676(0.01)
0.552/0.05) 0.22N0.02)
0.396(0.07)
0.197(0.0)
0.607(0.03)
0.483(0.05)
0.303(0.05)
0.125(0.05)
PI-GNN
0.676(0.0))
0.636/0.02)
0.562/0.44) 0.3980007) 0,208(0.07)
0.610KD.0-)
0.479(0.05)
0.300K0.04) 0.135(0.06)
labels and 2) whether the uncertainty estimation to combat the sub-optimal PI labels is beneficial for the test set accuracy. Therefore, compared the accuracy of vanilla GNN, PI-GNN trained without the mask generator for uncertainty estimation (PI-GNN wo/ ue) and PI-GNN.
From Table D we made several observations: Firstly, the GNN trained with the PI learning objective is more robust to noisy labels, where both PI-GNN wo/ ue and PI-GNN perform much better than a vanilla GNN. Secondly, by performing uncertainty estimation by an additional mask generator and weighting the PI loss with the confidence mask, the negative effect of noisy labels is further reduced. For instance, PI-GNN improves the accuracy by 1.1% with the symmetric noise (noise ratio 0.8) on Cora compared to PI-GNN wo/ ue, which justifies the effectiveness of our design. Thirdly, the PI learning objective does not help the GNN with the clean node labels, e.g., 80.4% of a vanilla GCN vs. 78.0% of PI-GNN on Cora, which illustrates the PI-GNN helps to combat noisy supervision rather than inherently improve the node classification with purely clean node labels. Additional results on heterophilous datasets and with lower noise ratios are presented in Appendix Sections D Dand
Table 2: Test accuracy on different graph neural network architectures. Bold numbers are superior results.
Standard deviation is shown in the bracket.
Noise type
No Noise
Symmelric Noise
Asymmeinic Noise
Cora
Noisc ralio
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
GAT
0.741(0:03)
0.647(0077)
0.474((005)
0.273(0D5)
0.71/0011
0.315(0007)
0,255(0305)
0.172((LD5)
PL-GNN wa/
0.78060.01>
0.743(0.02)
0.690X0.05)
0.517(0.07)
0.26110.04
0.73000.03)
0.574(0.07)
0.330(0.05)
0,206(0.05)
PL-GNN
0.79600011
0.7460.031
0.691(0.05)
0.516(0.06)
0.274(0.03)
0.7280.03)
0.569(0.17)
0.32910.05)
0.192(0.05)
Gra
0.80500.01)
0.723.0.22)
0.61100004
0.429/0.06)
0.280:0.07
0.70400.44)
0.29610.05)
0.162/0.01)
PI-GNI
0.77510.01:
0.735(0.03)
0.502/0.06)
0.298/0.07)
0.715(0.03)
0.58110.08
0.368/0.05
0,241(0.06)
PLGGNN
0.786(0.01)
0.756(0.01)
0.721(0.22)
0.584(0.06)
0.308(0.07)
0.755(0.02)
0.64000.8)
0.393/0.09)
0,239(0.06)
CueSeer
GAT
0.6810.011
0.6140.03)
0.542/0.033
0.394(0.05)
0.23400.05
0.588(0.04)
0.41100.06
0.26910.01)
0.175(0.05)
PI-GNN wor/ ue
0.66810.01)
0.618/0.02)
(.440.00))
0.384(0.06)
0,228/0.04)
0.599.0.33
0.325(0.01) 0.75(0.055
0.221(0.04)
PL-GNN
(.6660000)
061660003)
0.9400.04)
(..96(
0.23770.04)
0 0.59700055
0.85(0.05)
0.330(0.04)
0.214(0.04)
GrapIAAGE
(16980L01)
0.24:00.22)
0475005))
(129400144
0.1770194
PI-GINN wo// ue
0.66710.01
0.600(0.02)
0.55(00055
0.411(0.06)
0,226/0.04)
0.6180-08
0.519(0.15)
0.356(0.55)
0,236(0.55)
PI-GNN
0.693(0.01
0.674(0.02)
0.627(0.03
0.503(0.08)
0,256(0.07)
0.66900.02
0.593(0.05)
0.376(0.88
0.237(0.06)
5.3
PERFORMANCE ON DIFFERENT GNN ARCHITECTURES
We evaluated our proposed PI-GNN on different GNN architectures, i.e., GAT and GraphS The experiments are conducted on Cora and CiteSeer dataset, which are shown in Table As can be observed, our proposed approach performs similarly on GAT and GraphSAGE compared to the results on GCN, where the regularization of the pairwise interactions and the uncertainty estimation are both beneficial for model generalization even with extremely noisy supervision. Moreover, the uncertainty estimation is more effective on GraphS improves PI-GNN wo/ ue by 4.2% and 3.1% on average under symmetric noise and asymmetric noise, respectively, which is larger than that for GAT and GCN. It may suggest the mean aggregator in GraphSAGE is more susceptible to the sub-optimal PI labels. We provide significance test on the results of GAT in Appendix Section
For example, in the Cora dataset, PI-GNN
5.4 COMPARISON WITH BASELINES
In order to further demonstrate the competitive performance of PI-GNN, we compared with several powerful baselines for combating noisy labels in literature. For fair comparison, we used the same
7

---

Under review as a conference paper at ICLR 2022
Table Comparative results with baselines. Bold numbers are superior results. LPM-1 means one extra clean label is used for each class. The result on the left and right of each cell is the classification accuracy of the Cora dataset and CiteSeer dataset, respectively.
Noise type Noise ratio
Symmeinic Noise
Asymmelric Noise
0.2
0.4
Test dataset:
Cora CiteSeer
Decoupling GCB
(551(0006))0331100000)
0.4.5000660.3900000
1.69600033)0058810000
0.541(0.05)/ 0.4700000)
(16270007))0,555000000
0.47700.6)/0.383300
0.710(0.04)/ (0.598000000
0.511(0.05) / / 0.468(0.05)
APL
0,6240.06)/0.52200000 0,44600.06
0.707(0.55) / (0.5000000)
0.507(0.06) / 0,456(0.06)
Co-teaching
0.706(0.06) (0.1660.04)
0.457(0.10) 10) / / 0.462(0.00)
LPM-1
(.44200.99)0.46670000
0.447:0.07) 0.395/0008
0.674(0.09) / /0.56300.9)
0.481(0.07) / 0. 0.506(0.08)
T-Revision
(.59660.06)/0551800000
0.4550.066/0.38000
(02500004)/0591000
0.512(0.06) 0.457(0.06)
DivideMix
/
0.463((009) 0..355
0.646(0.01) / (..00 0.428(0.01) / 0.396((.03)
PL-GNN
0.66-00.03) 0.59I(0.03) 251550.0370.4452000 0.7230.03)7 0.628(0.03) 0.587(0.0777 (.53)(
GNN architecture, ie., GCN, and the same overlapping hyperparameters during implementation. The other method--ppecifie hyperparameters are tuned according to the original paper on the validation set. Specifically, compared with noise-transition matrix-based method, T-revision (Xia et al. 2019). robust loss functions, such as Generalized Cross Entropy (GCE) loss (Zhang & Sabuncu 2018 and Active Passive Loss (APL) (Ma al. 2020), optimization-baset approaches, such as Co-teaching (Han et al., 2018b), Decoupling (Malach & Shwllz,S 2017) and DivideMix (Li al., 2020b). We also compared with Label Propagation and Meta learning (LPM) (Xia et al. (0221)
method that is specifically designed for solving label noise for node classification but uses a small set of clean nodes for assistance. We reported the classification accuracy on Cora and CiteSeer with symmetric noise (noise rate E 0.4, 0.6) and asymmetric noise (noise rate E = 0.2, 0.4) in Table B
From 3 PI-GNN outperforms different baselines with a considerable margin, especially under extremely noisy supervisions, e.g.. improving the classification accuracy by 3.7% on CiteSeer under the symmetric noise (noise rate E = 0.6). Moreover, PI-GNN is able to outperform LPM-1, which relieves the strong assumption that auxiliary clean node labels are used for training. We present comparison with traditional graph semi- aomi-superiised learning approaches in Appendix Section C
5.5
ABLATION STUDIES
(6 Cora
(b) CleSeer
Cora
(6) GieeSer
0.44
Symmoscy= 2.
0.53
Agensencc=06
Asymmrorc 0.4
0.0
0.53
B44
0.54
0.52
042
0.34
0:43
0.52
0.45
Figure 3: (a)-(b) Performance of PI-GNN w.x.f different pretraining epochs on Cora and CiteSeer. x axis denotes the value of the pretraining epochs for the mask generator. (c)-(d) Performance PI-GNN w.r.t. the regularization loss weight B. x axis denotes the value of the lass weight and B² is the weight that is aware of the sparsity of the input graph.
Sensitivity to the pretraining epoch of the mask generator. We investigated whether the perfor- mance of PI-GNN is sensitive to the number of pretraining epochs for the mask generator. The experimental results on Cora and CiteSeer with GCN under symmetric and asymmetrie noise (noise rate 8 8 0.6) are shown in Figure (a) and (b). As can be observed, pretraining the mask generator for K epochs is effective for improving the generalization on the clean test set. Given a small K, the confidence mask is not estimated well which is not helpful to apply it on the task executor for regularization. Meanwhile, K should not be too large in order to sufficiently regularize the task executor by the uncertainty-awar training objective. K is set to 50 for all the experiments.
The effect of regularization weight. To observe whether the regularization loss weight B matters to the model performance, we trained PI-GNN with different values of B. i.e., 0.005, 0.01, 0.05, 0.1, 0.2, 0.5 and compared with the value B = v 2 2 which is aware of the sparsity of the graph in Figure 3 (c) and (d). We conducted experiments
Cora and CiteSeer with GCN and showed the results with symmetric and asymmetric noise (noise rate E = 0.4). From the figure, PI-GNN is sensitive to the choice of regularization loss weight B. On both datasets with different noise types, PI-GNN trained with B' achieves the best clean test accuracy, and simultancously avoids heavy tuning procedure the validation set.
8

---

Under review as a conference paper at ICLR 2022
Table 5: Left: Performance of the PI-GNN applied on different label-noise baselines on Cora. Right: Performance of Pl- GNN with different architectures for two branches on CiteSeer.
Noise Type
Sym. Noise
Asym. Noise
Naise Type
Sym. Noise
Asym. Noise
Noise Ratio
0.4
0.4
Noise Ralio
0.8
0.6
MP 2022)
0.624(0008)
0.507(0.6)
GAT only
0.2370004)
0.330K0.04)
APL-PI-GNN
0.656(0.05)
0.549(0.07)
GCN-GAT
0.241(0.05)
0.332(0.03)
T-eeisson Xia et al.
(2019)
0.596(0.06)
0.512(0.06)
GraphSAGE only
0.256(0.07)
0.376(0.08)
FrevissonsPPPUUS
0.615(0.04)
0.536(0.05)
0.266(0.05)
0.383(0.06)
DivideMix Li et al (2020b)
(.628(0.06)
0.428(0.01)
GraphS.AGI only
0.256(0.07)
0.376(0.08)
DivideMii--PEGNN
0.674(0.03)
0.442(0.02)
GAT-GmphSAAGG 0.271(0.04)
(.381(0.06)
The effect of pairwise interaction labels. To demonstrate the importance of informative PI labels for PI-GNN, we tested PI-GNN under different PI labels in addition to those based on adjacency matrix, namely by 1) noisy class label comparison and 2) random labels. We also compared with using the task executor fi to generate the confidence mask without introducing the mask generator. We used GCN as the backbone and tested on two datasets, i.e., Cora and CiteSeer and different noise types. The results are shown in Table 4 From Table 4 using PI labels based on the graph structure obtains the best performance. Directly comparing the similarity of the noisy class labels to get the PI labels or using randomly generated PI labels incurred the worst performance be-
Table 4: Performance of PI-GNN with dif- ferent PI labels.
Noise Type
Symmetiic Nose
Asymmeniic Noise
Nosse Katio
Noiiyy latel compariion
0.550005
Random label
0.46500 05)
0.295/0..0 04)
Task execuiic oaly
(531)003))
0.325/0115)
Adjacency matrix
0.515(0.03)
0.347/0.17)
ClicSeer
Noisy label compartioo
0.3390000)
11275)
cause of the heavy noise during the PI label generation.
Randen lahel
0.35100
0:208(0:14)
Task esocutio only
0.5000.07)
0.540(0-15)
Moreover, removing the mask generator for confidence mask generation decreases the test accuracy because the node embeddings are optimized by the noisy class labels well, which is ineffective for the task executor to generalize even with uncertainty- aware training by itself (32.9% vs. 34.7% for Cora with asymmetric noise and a noise rate of 60%). We also compare with training with the clean PI labels, which can be regarded as an upper bound of our method in Appendix Section
Adjacency matris
0.452/0.07)
0.353/0.06)
F
Application of PI-GNN on label-noise baselines. To observe whether PI-GNN is able to improve the generalization ability for different label-noise baseline models, we extended three representative approaches, i.c., T-revision (Xia et al., 2019). APL (Ma al. 2020 and DivideMix (Li et al.) 2020b) by adding the PI learning objective during training. Specifically, we used the sum of the original loss and the PI loss to optimize the GNN. The weight for PI loss set to B'. We chose GCN as the backbone and reported the test accuracy on Cora with both symmetric noise (noise rate E 0.4) and asymmetric noise (noise rate & = 0.4) in 5 Left. As the result shows, PI-GNN is orthogonal to those robust baseline models, which is potentially useful for improving their performance without bells and whistles. For instance, The test set accuracy is improved by 4.6% on DivideMix under the symmetric noise and thus demonstrates the universality of our proposed PI-GNN.
Different architectures for two branches. PI-GNN allows for a flexible choice of the architectures for the mask generator and the task executor, where a hight-weight mask generator can help large task executor for node classification during the uncertainty-awart aware training. In what follows, we used three different mask generator-task executor pairs, namely GCN-GAT, GCN-GraphSAGBB and GAT-GraphSAAGE. The number of parameters for GCN, GAT, GraphS AGE is 0.02, 0.09 and 0.18 M, respectively. The comparison with using the same architectures are shown in Table 5 Right. From Table 5 Right, using light-weight GNN for the mask generator is able to further improve the clean test accuracy, which is promising for efficient deployment of PI-GNN on real-world graph datasets.
6 CONCLUSION
In this paper, we proposed PI-GNN, a simple but effective learning paradigm for helping the graph neural networks to generalize well with noisy supervision. Our key idea is to leverage the pairwise interactions between nodes to explicitly adjust the similarity of those node embeddings during training. In order to alleviate the negative effect of the collected sub-optimal PI labels, we introduce new aware uncertainty aware training approach to reweight the PI learning objective by its prediction confidence. We conducted extensive experiments to demonstrate that PI-GNN can train GNNs robustly under extremely noisy supervision, which serves as a crucial step towards the reliable deployment of GNNs in complex real-world applications.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper doesn't It raise any ethics concerns. This study doesn't it involve any human subjects, practices data releases, potentially harmful insights, methodologies and applications, potentially conflicts of interest and sponsorship, discrinination/iiafaaanne concerns, privacy and security issues, legal compliance, and research integrity issues.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of experimental results, we will provide source codes of this paper using an anonymous repository link in the discussion phase.
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj. Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste- Julien. A closer look memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICMI. 2017, volume 70, pp. 233-242, 2017.
Davide Bacciu, Federico Errica, and Alessio Micheli. Contextual graph markov model: A deep and generative approach to graph processing. In Proceedings of the 35th International Conference on Machine Learning, JCML. 2018, volume 80, pp. 304-313, 2018
Jinheon Back, Dong Bok Lee, and Sung Ju Hwang. Learning to extrapolate knowledge: Transductive few-shot out-of-graph link prediction. In Advances in Neural Information Processing Systems 33, NeurlPS 2020, 2020.
Thang D. Bui, Sujith Ravi, and Vivek Ramavaijala. Neural graph machines: Learning neural networks using graphs. CoRR, abs/1703 70217003008888 04888, 2017. URL http::::/aa.....//////aaa///////
Hande Dong. Jiawei Chen, Fuli Feng, Xiangnan He, Shuxian Bi, Zhaolin Ding, and Peng Cui. On the equivalence decoupled graph convolution network and label propagation. CoRR, abs/2010. 2020. UtLAttts://aaxttt wwwp://wwwwww.pp//w///////././/. 12408
Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. 8th International Conference on Learning Representations, ICLR 2020, 2020.
Matthias Fey and Jan Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop Representation Learning Graphs and Manifolds, 2019.
William L.. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems 30, pp. 1024-1034, 2017.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor W. Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective noisy supervision. In Advances Neural Information Processing Systems 31, pp. 5841-5857 2018a.
Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems 31, NeurlPS 2018, pp. 8536-8546, 2018b.
Bo Han, Gang Niu, Xingrui Yu, Quamming Yao, Miao Xu, Ivor W. Tsang, and Masashi Sugiyama. SIGUA: forgetting may make learning with noisy labels more robust. In Proceedings of the 37th International Conference on Muchine Learning, ICML 2020, volume 1 119, pp. 4006-4016, 2020a.
Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W. Tsang, James T. Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. CoRR, abs/2011 1,04066, 2020b. URL URL http:/ttt///aaiiiiitata wwtp://wwwrraa......a..o 04406
10

---

Under review as a conference paper at ICLR 2022
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems 33, NeurlPS 2020, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks corrupted labels. In Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, volume 80, pp. 2309-2318, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Reprexentations, ICLR 2017, 2017.
Lin Lan, Pinghui Wang, Xuefeng Du, Kaikai Song, Jing Tao, and Xiaohong Guan. Node classification on graphs with few-shot novel labels via meta transformed network embedding. In Advances in Neural Information Processing Systems 33, NeurlPS 2020, 2020.
Jingling Li, Mozhi Zhang, Keyulu Xu, John P. Dickerson, and Jimmy Ba. Noisy labels can induce good representations. CoRR, abs/2012.12896, 2020a. URL https::/arxiv.org/ /bs/ 2012. 12896
Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels semi- supervised learning. In 8th International Conference on Learning Representations, ICLR 2020, 2020b.
Yayong Li, Jie Yin, and Ling Chen. Unified robust training for graph neuralnetworks against label noise. CoRR, abs/2103.03414 2021. IRttps::t///aaa wttp//wwwwv.......aaaooooooooorr abs/2103, 22411111113331
Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, William L. Hamilton, David Duvenaud, Raquel Urtasun, and Richard S. Zemel. Efficient graph generation with graph recurrent attention networks. In Advances in Neural Information Processing Systems 32, NeurlPS 20/9, pp. 4257 426774267,55 2019.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Trans. Pattern Anal. Mach. Intell., 38(3):447-661, 2016. doi: 10. 10.1109/TPAMM.20152255555 110.0...19991001
URL
2015. 2015.2456899
Qing Lu and Lise Getoor. Link based classification. In Tom Fawcett and Nina Mishra (eds.), Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA, pp. 496-503. AAAI Press, 2003. URL http: pett:://wee aaai. org/ Library/ CMDp://IMMMM//00000 cm -3- 066 php
Xingjun Ma, Hanxun Huang, Yisen Wang. Simone Romano, Sarah M. Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, volume 119, pp. 6643-6555 2020.
Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily necessity for graph neural networks? CoRR, abs/2106.06134, 2021. URL URLstts:://axxi.ccomm
06134
Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In Advances in Neural Information Processing Systems 30, pp. 960-970, 2017.
Péter Mernyei and Catalina Cangea. Wiki-es: A wikipedia-based benchmark for graph neural networks. CoRR, abs/2007.02901, 2020. http::
02901
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 41(8) -9t::999999999,.,.. 2019. doi: 10.1 10.1009/PAM...018 7/doi.org 0. 11 1109 201 8 2858821
URL |
Hoang NT, Choong Jun Jin, and Tsuyoshi Murata. Learning graph neural networks with noisy labels. 905. 01591
CoRR,
2019. http:http:///ttteee
11

---

Under review as a conference paper at ICLR 2022
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Shirui Pan, Jia Wu, Xingquan Zhu, Chengqi Zhang, and Yang Wang. Tri-party deep network representation. In Proceedings of the Twenry-Fifth International Joint Conference Artificial Intelligence, IJCAI 2016, pp. 1895-1901, 2016.
Hogun Park and Jennifer Neville. Exploiting interaction links for node classification with deep graph neural networks. In Proceedings of the eeventy-Eighht International Joint Conference Artificial 2019.
Intelligence, IJCAI 2019, pp.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust label noise: A loss correction approach. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pp. 2233-2241, 2017.
Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping. In 3rd International Conference on Learning Representations ICLR 2015, 2015.
Mengye Ren, Wenyuan Zeng. Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, volume 80, pp. 4331 2018.
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-based autoregressive model for molecular graph generation. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Otilia Stretcu, Krishnamurthy Viswanathan, Dana Movshovitz-Atiia, Emmanouil A. Platanios, Sujith Ravi, and Andrew Tomkins. Graph agreement models for cemi-supervised learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. neurips....... httpp..cccppeee 64f7a Abst
http//pper/2219/haaaeee 9710-8720, 2019. URL Abstrct html
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels, 2015.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff A. Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise deep learning using abstention. In Proceedings of the 36th International Conference on Machine Learning, ICMI. 20/9, volume 97, pp. 6234-6243, 2019.
Brendan van Rooyen and Robert C. Williamson. A theory of learning with compupted labels. J. Mach. Learn. Res., 18:228:1-228::5 2017. URL
html
Brendan van Rooyen, Aditya Krishna Menon, and Robert C. Williamson. Learning with symmetric label noise: The importance of being unhinged. Advances Neural Information Processing Systems 28, pp. (0-18, 2015.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, 2018
Yixin Wang, Alp Kucukelbir, and David M. Blei. Robust probabilistic modeling with bayesian data reweighting. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, volume 70. pp. 3646-366665 2017.
Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 20/9, volume 97, pp. 6861-6871, 2019.
Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang. Haifeng Liu, and Gang Niu. Class2simi: A new perspective on learning with label noise. CoRR, abs/2006.07831, 2020. URL htt.hhtts:///ar.. www./ogg//aa.oo.oooo//o.o/.a.a /206. 07831
12

---

Under review as a conference paper at ICLR 2022
Jun Xia, Haitao Lin, Yongjie Xu, Lirong Wu, Zhangyang Gao, Siyuan Li, and Stan Z. Li. Towards robust graph neural networks against label noise. In OpenReview, 2021.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In Advances in Neural Information Processing Systems 32, NeurlPS 2019, pp. 6835-6846, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? 7th International Conference on Learning Representations, ICLR 2019, 2019.
Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33nd International Conference Machine Learning, ICML 2016, volume 48, pp. 40.48, 2016.
Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor W. Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97, pp. 7164-7173, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 5th International Conference on Learning Representations, ICLR 2017, 2017.
Muhan Zhang and Yixin Chen. Link prediction based graph neural networks. In Advances in Neural Information Processing Systems 31, NeurIPS 2018, pp- 5171-5181, 2018.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems 31, pp. 8792-8800 2018.
Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in Neural Information Processing Systems 16, NIPS 2003, pp. 321-328, 2003.
Fan Zhou, Chengtai Cao, Kunpeng Zhang, Goce Trajcevski, Ting Zhong, and Ji Geng. Meta-gnn: On few-shot node classification in graph meta-learning. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, pp. 2357-2360, 2019.
Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propaga- tion. Technical Report, 2002.
Xiaojin Zhu, Zoubin Ghahramani, and John D. Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the Twentieth International Conference on Machine Learning, ICML 2003, pp. 912-919, 2003.
13

---

Under review as a conference paper at ICLR 2022
Supplementary Material
A DEFINITION OF NOISE
The definition of transition matrix Q is follows. is number of the class. Asymmetric pair flipping:
E
0
0
1 g
0
g
8
g
1 -
Symmetry flipping:
-
8
8
n-1
R-1
n-1
Q =
(9)
R
n-1
8
B DATASET DETAILS
Here we provide the details of graph datasets for node classification.
Table Statistics of the datasets.
Dataset
#Nodes
#Edges
#Classes
Cora
2,485
5,069
CiteSeer
2,110
3,668
PubMed
19,717
44,324
DBLP
17,716
105,734
WikiCS
11,701
216,123
10
C COMPARISON WITH TRADITIONAL GRAPH SEMI-SUPERVISED LEARNING BASED APPROACHES.
For the comparison with the traditional semi- semi-supervised graph embedding methods, follow the same experimental setting and compare with ICA (Lu & Getoor 2003), Planetoid (Yang et al. 2016) and Label Propagation (LP) (Zuu & Ghahramani 2002)) Cora follows. The result shows the advantage of PI-GNN across different noise ratios.
Table 7: Comparison with more baselines Cora Dataset.
type
Symmetric
Asymmeetrcc Noise
Noise ratio
0.0
0.2
U.4
0.6
0.3
0.2
U.4
U.6
0.8
ICA
0.72910.01)
0.609(0.01)
0.523(0.04)
0.394(0.00)
0.15910:00)
0.549(0.00)
0.45(0.00)
0.284(0:0))
0.127(0:01)
0.603(0.00)
0.506(0.02)
0.41((0033)
0.297(0.03)
0.170(0.03)
0.513/0.03)
0.391(0.04)
0.238(0.03)
0.141(0.02)
Planetoid
0.739(0.01)
0.639(0.03)
0.527(0004)
0.379(0.05)
0.265(0.06)
0.627(0.03)
0.441(0.02)
0.271(0.06)
0.210(0.09)
PI-GNN
0.780:0.01)
0.732(0.02)
0.664(0.33
0.515(0.03)
0.296(0.05)
0.723/0033
0.587(0.07)
0.347(0.07)
0.211(0.06)
D EXPERIMENTAL RESULTS ON HETEROPHILOUS DATASETS
We perform extra experiments on heterophilous datasets (a et al. 2021). The results are in the following table. It shows that PI-GNN is still able outperform the vanilla except for
14

---

Under review as a conference paper at ICLR 2022
in Chameleon dataset. Meanwhile, the improvement is somewhat smaller, which implies PI-GNN may be more effective on homophilous datasets.
Table 8: Experimental results heterophilous datasets.
Noise type
Symmetric
Asymmetric
Actor
Noise Ratio
0.4
0.6
0.4
0.6
GCN PI-GNN w/o ue PI-GNN
(.209)0.02)
0.208(0.02)
0.198(0.03)
0.199(0.02)
0.216(0.01)
0.210(0.02)
0.201(0.02)
0.202(0.02)
0.218(0.02)
0.213(0.02)
0.204(0.02)
0.200(0.02)
Chameleon
GCN PI-GNN w/o ue PI-GNN
0.251(0.03)
0.246(0.033
0.245(0.04)
0.228(0.03)
0.264(0.03)
0.249(0.03)
0.242(0.05)
0.229(0.04)
0.269(0.02)
0.251(0.033
0.239(0.05)
0.237(0.04)
E SIGNIFICANCE TEST RESULTS
We perform significance tests to verify whether PI-GNN outperforms the vanilla GNN model significantly using double-sided T-test. We use python package secipysttts.tttest I and report the average results over 10 different runs as follows. PI-GNN is better than GAT because the absolute value of the t-statistic is relatively large and the p-value is small.
Table 9: Statistical significance tests.
Method
Setting
T-statistic
p-value
Cora
Symmetric Noise-0.8
4.78
0.001
GAT vs. PI-GNN
Asymmetric Noisc-0.8
3.42
0.008
CiteSeer
Symmetric Noise-0.8
2.09
0.060
GAT PI-GNN
Asymmetric Noise-0.8
4.63
0.001
F EXPERIMENTAL RESULTS ON USING CLEAN PI LABELS
To observe the node classification results with clean PI labels, we did experiments on Cora, CiteSeer and PubMed with GCN and noise ratio of 0.4 and 0.6. The results are shown in the Table [0 In most cases, clean PI labels can help the PI-GNN to combat noisy labels except for some challenging cases with asymeetric noise. One reason may be the inherent noise exists in clean node labels for Cora, where we cannot obtain perfectly clean PI label.
Table 10: Experimental results on using clean PI labels. Clean PI-GNN means the PI-GNN is trained with the clean PI labels.
Noise type
Symmetric
Asymmetric
Cora
Noise Ratio
0.4
0.6
0.4
0.6
PI-GNN
0.664(0.03)
0.515(0.03)
0.587(0.07)
0.347(0.07)
Clean PI-GNN
0.671(0.03)
0.523(0.03)
(.589)0007)
0.341(0.07)
CiteSeer
PI-GNN
0.591(0.03)
0.432(0.07)
0.531(0.06)
0.353(0.06)
Clean PI-GNN
0.605(0.04)
0.447(0.07)
0.536(0.05)
0.355(0.05)
PubMed
PI-GNN
0.638(0.04)
0.470(0.08)
0.583(0.07)
0.425(0.07)
Clean PI-GNN
0.640(0.02)
0.485(0.07)
0.590(0.07)
0.429(0.07)
15

---

Under review as a conference paper at ICLR 2022
G EXPERIMENTAL RESULTS WITH LOWER NOISE RATIOS
For the PI-GNN under lower noise ratios, we empirically verify its effectiveness on Cora and CiteSeer with the noise ratio of 0.1, which is shown in the following table.
Table 11: Experimental results with lower noise ratios.
Noise type
Symmetric
Asymmetric
Cora
Noise Ratio
0.1
0.1
GCN
0.766(0.03)
0.762(0.04)
PI-GNN wo/ ue PI-GNN
0.769(0.03)
0. 0..63
0.772(0.02)
0.768(0.03)
CiteSeer
Noise Ratio
0.1
0.1
GCN
0.642(0.03)
0.618(0.05)
PI-GNN wo/
0.648(0.04)
(0.03(
PI-GNN
0.659(0.03)
0.658(0.05)
16


Reviews:
[{"review_id": "uni3TJPTpOg", "reviewer": "Reviewer_8yHP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles an important problem in the graph representation learning community, uses a simple but effective approach, and conducts experiments with various settings (especially with/against label-noise baselines). However, it has several weaknesses and should be revised for the following points:\n\nFirst, related work on graph structure learning is not thoroughly surveyed. Using a task to predict graph structure (i.e., adjacency matrix or PI label 1) and label agreement (i.e., PI label 2) is proposed in many existing works. However, this paper does not discuss them and only mentions a few works such as [Stretcu, Otilia, et al.]. I would recommend authors to survey prior works about their methods. Plus, these prior works are not empirically compared by experiments; thus, the technical novelty of the model might be overclaimed.\n\n- Jiang, Bo, et al. \"Semi-supervised learning with graph learning-convolutional networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n- Yu, Donghan, et al. \"Graph-revised convolutional network.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2020.\n- Zheng, Cheng, et al. \"Robust graph representation learning via neural sparsification.\" International Conference on Machine Learning. PMLR, 2020.\n- Jin, Wei, et al. \"Graph structure learning for robust graph neural networks.\" Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.\n- Chen, Yu, Lingfei Wu, and Mohammed Zaki. \"Iterative deep graph learning for graph neural networks: Better and robust node embeddings.\" Advances in Neural Information Processing Systems 33 (2020).\n- Kim, Dongkwan, and Alice Oh. \"How to find your friendly neighborhood: Graph attention design with self-supervision.\" International Conference on Learning Representations. 2020.\n- Fatemi, Bahare, Layla El Asri, and Seyed Mehran Kazemi. \"SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks.\" arXiv preprint arXiv:2102.05034 (2021).\n- Stretcu, Otilia, et al. \"Graph Agreement Models for Semi-Supervised Learning.\" Advances in Neural Information Processing Systems 32 (2019): 8713-8723.\n\nSecond, the complexity and scalability of models are not discussed. The confidence mask requires $|V| \\times |V|$ space and time complexity, and it would be numerically large if we want to deal with a gigantic graph. However, this paper does not analyze how the model parts cost and how large graphs can be used. Here are some quick solutions: (1) conducting experiments on relatively large graphs (e.g., OGB datasets), (2) analyzing complexity and comparing with baselines, and (3) measuring the memory consumption and training/inference wall-clock time.\n\nThird, experimental settings and their demonstration are not consistent. For example, some tables only present a subset of noise ratio settings (e.g., 0.4/0.6, 0.2/0.4 in Table 3, 0.4/0.6 in Table 8). Significant test results are only reported on Cora and CiteSeer (considering that we usually use asterisks, a space-efficient way to represent statistical significance). Is there any specific reason or motivation for this configuration?\n\nLastly, here are some minor concerns and suggestions:\n- The term 'uncertainty estimation' seems far from the well-known meaning in the deep learning (especially in the Bayesian DNN) community. Therefore, it can be misleading to many potential readers. Researchers are now differentiating the softmax output and the model uncertainty (i.e., A model can be uncertain in its predictions even with a high softmax output). For more details, see Gal, Yarin, and Zoubin Ghahramani. \"Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.\" international conference on machine learning. PMLR, 2016.\n- It would be great if the authors used line charts to show the performance against noise ratio of models. Tables are also good, but sometimes it is hard to recognize the difference and see the big picture.\n- It would be great if the authors present the exact value of $\\beta$ in Figure 3.\n", "summary_of_the_paper": "This paper proposes the PI-GNN (Pairwise Interactions in Graph Neural Network) model for semi-supervised tasks for nodes with noisy labels. It employs a new auxiliary task to predict PI (pairwise interactions) labels using a pair of nodes. Since it is costly to get clean PI labels, the authors use (1) two instantiations of PI labels: adjacency matrix and class agreement after label propagation and (2) uncertainty-aware (pre-)training with two model branches. The experiments on five real-world benchmarks demonstrate that PI-GNN outperforms GNN models (GCN, GraphSAGE, and GAT) and methods for noisy label settings (Decoupling, GCE, DivideMix, and so on.) on various noise levels and settings. PI label learning and uncertainty-aware training both contribute to performance improvement.", "summary_of_the_review": "Although this work has several strengths (important problem, simple but effective method, and extensive experiments), it is not ready for publications by following weaknesses:\n- Related work on graph structure learning is not thoroughly surveyed and not compared. Technical novelty on the model might be overclaimed.\n- Scalability is not discussed, even though it has the potential of huge complexity.\n- Experimental settings and their demonstration are not consistent without justifications.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pPsOTq2CNE0", "reviewer": "Reviewer_2qdw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is the new approach is to learn Pairwise Intersection (PI) between graph nodes to enhance the robustness. Ideally, a node pair has positive PI is more likely to have the same labels. Authors further propose strategies to optimize such PI labels.\n\nI think the motivation of this paper is good. One can expect to gain strong intersection between nodes with same label even in presence of noise and such structure upon the graph should have good impact on the robustness. What I am concerned is the experiments, I don not feel the empirical results in this paper achieve significantly accuracy upon vanilla GNN approach (like GCN). In table 1 and 2, some improvements are only from 0.273 to 0.274 or 0.367 to 0.370. Thus, I am not fully convinced the necessarily of using PI-GNN in similar tasks.", "summary_of_the_paper": "This paper proposes a new GNN approach which they call PI-GNN that aims to improve performance in presence of noise or corruption. Empirical results suggest the new approach can slightly improve the accuracy when there is noise.", "summary_of_the_review": "I give a borderline evaluation for this paper. My main complaint is the experiment results are not strong enough. More convincing results or a strong reason why PI-GNN is a good candidate in similar tasks should make me improve the evaluation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oFHARtESNUi", "reviewer": "Reviewer_Lvq5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. From eqn(3), the labels of PI can be the input adjacency matrix A or are generated from label propagation. It is not clear how does label propagation generate PI labels? Is it a semi-supervised way? Are clean labels used? If so, in principle, this setting is incorrect for robustness research. If not, please give a more detailed explanation and training details. Besides, what does this sentence “which does not touch noisy class labels for learning” (on Page 6) mean?\n2. How about the complexity, no analysis is given.\n3. In experiments, although 5 datasets are evaluated in table1, only two small datasets (Cora and Citeseer) are used in all subsequent experiments. Both these two datasets belong to citation datasets, which have similar characteristics. DBLP and WikiCS datasets should be evaluated on ablation study.\n4. From Fig.3 (c)-(d), the performance of PI-GNN always increases with \\beta increasing. More \\beta values should be used to test the trend of accuracy, the most extreme is how the performance when \\beta=1.\n", "summary_of_the_paper": "This paper is mainly dedicated to developing a robust model against noisy labels. Since it is observed from Fig. 1(a) that the noise ratio for the PI labels is lower than that of the point-wise labels, the authors introduce positive and negative sample pairs (Pair Interactions) to enhance model capabilities, which is likely contrastive learning. Moreover, the labels of positive and negative sample pairs may not be confident, e.g., sub-optimal PI labels with noises. To address this problem, the authors introduce another branch (Mask Generator) trained on pair interaction learning objective to generate a confidence map to reweight the loss of PI learning objective in Task Executor. Experimentally, the accuracies of the proposed method are higher than baselines in the case of noisy labels.", "summary_of_the_review": "1. The presentation is overall good and easy to follow, but some descriptions or details are not clear.\n2. The novelty is limited. Essentially, the proposed pair interactions belong to contrastive learning.\n3. The experiments show better performance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B8FS2zQhsjV", "reviewer": "Reviewer_bNFD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The problem is well-motivated.\n\nThis paper focuses on semi-supervised node classification in a setting where noisy node labels exist. As node classification is a fundamental problem in graph machine learning and noisy labels always exist due to some annotation errors, this problem looks well-motivated and interesting to me. \n\n2. The idea of pairwise interactions is intuitive.\n\nTo deal with the problem, the authors propose a new objective called the pairwise interaction. Different from most training objectives which focus on pointwise interactions between nodes and labels, this objective forces similar nodes to have similar node embeddings, where the similarity is measured with both graph structures and node labels. Overall, this objective is intuitive in the context of node classification with noisy labels.\n\n3. The model is simple yet effective.\n\nBased on pairwise interactions, the authors propose PI-GNN, which consists of two GNNs, i.e., a mask generator and a task executor. The mask generator computes a soft mask matrix over node pairs, which is used to reweight node pairs for training. The framework is quite straightforward, and it is able to effectively model the pairwise interactions.\n\nWeaknesses:\n\n1. The novelty of the paper seems limited.\n\nMy major concern is on the novelty of the paper. In terms of problem, node classification with noisy labels has been explored quite extensively in literature. For example, as mentioned in related work, methods based on loss correction and sample reweighting have been applied to GNNs. Given these works, the problem studied in this paper is not very new. \n\nComparing this work with existing works, the authors claim that existing works only consider pointwise interactions while this paper focuses on pairwise interactions. However, the idea of modeling pairwise interactions is also not very new in the field of graph machine learning. For example, [1] uses a GNN to infer whether two linked nodes have the same label, and thus models pairwise interactions. [2] uses Gaussian process with a graph structure-based kernel for node label prediction, which can also achieve modeling pairwise interactions. [3] and [4] use CRFs to model pairwise interactions of node labels. Although all these works leverage pairwise interactions in the clean-label setting, they have very similar ideas to this paper, and they can be extended to deal with noisy labels quite easily.\n\nTherefore, although this paper has some contributions over existing works, I think these contributions are small, and hence the novelty of the paper is limited.\n\n2. Some model designs need more justification.\n\nAlthough PI-GNN is very intuitive, some model designs need more justification.\n\nIn equation (5), why the mask matrix is designed in that way? Is there any example to motivate the design? It would be more convincing if the authors could further elaborate on that or provide some theoretical justification.\n\nIn PI-GNN, two GNNs are leveraged. To me, these two GNNs have quite similar roles in the sense that they both try to learn an embedding for each node. Is there any conceptual difference between the embeddings learned by the two GNNs? What if we use the same GNN encoder for both the mask generator and the task executor?\n\nIn equation (4), a pairwise loss function is used for training. According to the definition, the size of $B_{PI}^+$ is much larger than the size of $B_{PI}^+$, resulting in a problem of label imbalance. How does PI-GNN deal with the problem?\n\n3. It is better to consider some larger datasets.\n\nIn the experiment, all the datasets are very small, and the largest one only has ~10K nodes. If I understand correctly, PI-GNN has higher memory cost than standard GNNs, as PI-GNN needs to compute a soft weight for each pair of nodes. Because of that, it is unclear whether PI-GNN can deal with very large graphs, and it would be more convincing if the authors could run such experiments.\n\n[1] Stretcu, Otilia, Krishnamurthy Viswanathan, Dana Movshovitz-Attias, Emmanouil A. Platanios, Sujith Ravi, and Andrew Tomkins. \"Graph Agreement Models for Semi-Supervised Learning.\" In NeurIPS. 2019.\n\n[2] Ng, Yin Cheng, Nicolò Colombo, and Ricardo Silva. \"Bayesian Semi-supervised Learning with Graph Gaussian Processes.\" In NeurIPS. 2018.\n\n[3] Qu, Meng, Yoshua Bengio, and Jian Tang. \"Gmnn: Graph markov neural networks.\" In ICML. 2019.\n\n[4] Ma, Tengfei, Cao Xiao, Junyuan Shang, and Jimeng Sun. \"CGNF: Conditional Graph Neural Fields.\" (2018).\n", "summary_of_the_paper": "This paper focuses on semi-supervised node classification with noisy node labels. The authors propose a novel learning objective called the pairwise interactions, which encourages node pairs holding positive PI labels to have close node embeddings. Extensive experiments show promising results.", "summary_of_the_review": "This paper focuses on semi-supervised node classification with label noise. Although the paper has some contributions over existing works, I think the contributions are not significant enough, so I lean towards a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the noisy labels problem in semi-supervised node classification and proposes a method that leverages pairwise interactions to explicitly force the embeddings for certain node pairs to be close to each other leading to better robustness.

The reviewers agreed the proposed method is promising. However, the reviewers also had concerns about the novelty, and that certain aspects of the method could be justified better and the experiments should consider larger scale settings to make the paper more convincing. These were the key reasons for rejection.
Decision: Reject