Towards Robust Graph Neural Networks against Label Noise
Authors: Jun Xia
Haitao Lin
Yongjie Xu
Lirong Wu
Zhangyang Gao
Siyuan Li
Stan Z. Li
Venue:  (2021)
Keywords: Graph Neural Networks
Graph Node Classification
Label Noise

Abstract: Massive labeled data have been used in training deep neural networks, thus label noise has become an important issue therein.  Although learning with noisy labels has made great progress on image datasets in recent years, it has not yet been studied in connection with utilizing GNNs to classify graph nodes. In this paper, we proposed a method, named LPM,  to address the problem using Label Propagation (LP) and Meta learning.  Different from previous methods designed for image datasets, our method is based on a special attribute (label smoothness) of graph-structured data, i.e., neighboring nodes in a graph tend to have the same label.  A pseudo label is computed from the neighboring labels for each node in the training set using LP; meta learning is utilized to learn a proper aggregation of the original and pseudo label as the final label.  Experimental results demonstrate that LPM outperforms state-of-the-art methods in graph node classification task with both synthetic and real-world label noise. Source code to reproduce all results will be released.

Under review as a conference paper at ICLR 2021
TOWARDS ROBUST GRAPH NEURAL NETWORKS AGAINST LABEL NOISE
Anonymous authors Paper under double-blind review
ABSTRACT
Massive labeled data have been used in training deep neural networks, thus label noise has become an important issue therein. Although learning with noisy labels has made great progress on image datasets in recent years, it has not yet been studied connection with utilizing GNNs to classify graph nodes. In this paper, we propose method, named LPM, to address the prohlem using Label Propaga- tion (LP) and Meta learning. Different from previous methods designed for image datasets, our method is based on special attribute (label smoothness) of graph- structured data, i.e., neighboring nodes in a graph tend to have the same label. A pseudo label is computed from the neighboring labels for each node in the training set using LP; meta learning is utilized to learn a proper aggregation of the original and pseudo label the final label. Experimental results demonstrate that LPM outperforms state-of-the-art methods in graph node classification task with both synthetic and real-world label noise. Source code reproduce all results will be released.
INTRODUCTION
Deep Neural Networks (DNNs) have achieved great success in various domains, but the necessity of collecting large amount of samples with high-quality labels is both expensive and time-consuming.
To address this problem, cheaper alternatives have emerged. For For example, the onerous labeling pro- cess can be completed we can collect labeled samples from web with search engines and social media. However, all these methods prone to produce noisy labels of low quality. As is shown in recent research (Zhang et al. 2016b), an intractable problem is that DNNs can easily overfit to noisy labels, which dramati- cally degrades the generalization performance. Therefore, necessary and urgent design some valid methods for solving this problem.
some crowdsourced system like Amazon Mechanical Turk W Besides,
Graph Neural Networks (GNNs) have aroused keen research interest recent years, which resulted in rapid progress in graph-structure data analysis (Kipf & Welling 2016 Velickovic et al.) [017]] Xu et al. 2018 Hou et al.) 2019 Wang & Leskovee 2020). Graph node classification is the most- common issue in GNNs. However, almost all the previous works about label noise focus on image classification problem and handling noisy labels the task of graph node elassification with GNNs has not been studied yet. Fortunately, most edges in the graph-structured datasets are intra-class edges (Wang & Leskovec 2020).. indicating that node's label can be estimated by its neighbor nodes' labels. In this paper, utilize this special attribute of graph data to alleviate the damages caused by noisy labels. Moreover, meta learning paradigm serves as a useful tool for us to learn a proper aggregation between origin labels and pseudo labels as the final labels.
The key contributions of this paper are as follows:
To the best of our knowledge, we are the first to focus on the label noise existing in utilizing GNNs to classify graph nodes, which may serve as beginning for future research towards robust GNNs against label noise.
We utilize meta-learning to learn how to aggregate origin labels and pseudo labels properly to get more credible supervision instead of learning to re- re-weight different samples.
hhtps//wwww.mur.cco//

---

Under review as a conference paper at ICLR 2021
We experimentally show that our LPM outperforms state-of-the-ee algorithms in utilizing GNNs to classify graph nodes with both synthetic and real-world label noise.
2 RELATED WORK
2.1 GRAPH NEURAL NETWORKS
To start, X' E R is the input feature matrix, where n denotes the number of nodes in the graph and is the dimension of the input feature vector of each node. We use eu, E & to denote the edge that connects node and v. For each node e E V, its neighbor nodes set can be donated as .N, = {u: Cu,v E E)- For node classification task, the goal of GNNs to learn optimal mapping function f(-) to predict the class label for node v. Generally speaking, GNNs follows a framework including aggregation and combination each layer. Different GNNs have proposed different ways of aggregation and combination. In general, the k-th layer of GNN reads
G EV)E, to denote graph whose nodes set is V and edges set is E, and
(k) Aggregate (a) (fh E (()))) Combine (N a..)),
(1)
where (x) the output for k-th layer of node v, h is the input vector of node v.
2.2 LABEL. PROPAGATION
In Label Propagation (LP), node labels are propagated and aggregated along the edges in the graph (Zhou et al. 2004 Zhu et al. 2005) Wang & Zhang, 2007 Karasuyama & Mamitsuka 2013). There are some works which were designed to improve the performance of label propagation. For example,Goov et al. (016)) proposed novel iterative label propagation algorithm which explicitly optimizes the propagation quality by manipulating the propagation sequence to move from simple to difficult examples: Zhang et al. (2020) introduces triple matrix recovery mechanism to remove noise from the estimated soft labels during propagation. Label propagation has been applied in semi-supervised image classification task. For example, Gong et al. 2017 used weighted K-
nearest neighborhood graph to bridge the datapoints so that the label information can be propagated from the scarce labeled examples to unlabeled examples along the graph edges. Park al. (2020) proposed a novel framwork to propagate the label information of the sampled data (reliable) to adjacent data along similarity based graph. Compared these methods, we utilize the intrinsic graph structure instead of handcrafted graph to propagate clean labels information, which is more reliable for graph-structured data. Besides, GNNs are utilized by us extract features and classify nodes for graph-structured data.
2.3 META-LEARNING BASED METHODS AGAINST NOISY LABELS
Meta-leamnig aims to learn not only neural networks' weights, but also itself, such as hand-designed
parameters, optimizer and on (Andrychowicz al. 2016 Finn al. 2017). Several works have utilized meta-learning paradigm to deal with label noise. For example, Li et al. (2019) has proposed find noise-tolerant model parameters by keeping the consistency between the output of teacher and student networks. and Li et al. (2017b) trains the teacher networks with samples with clean labels and then transfer the knowledge student networks so that the student can learn correctly even if the existence of mislabeled data. Besides, [Ren et al. (2018); Jenni & Favaro) (2018); Shu jet al. (2019) utilize meta-learning paradigm to re-weight samples, i.c., weight samples with clean labels more and weight mislabeled samples less. The weighting factors are optimized by gradient decent or generated by a network to minimizes the loss on a small amount of samples with correct labels. In contrast, meta-learning paradigm is utilized in this paper to leam how aggregate origin labels and pseudo labels properly. We can get more credible supervision by combining the original label information with the label information provided by LP properly.
2

---

Under review as a conference paper at ICLR 2021
3
6
10
(a) Before label propagation
(b) After label propagation
Figure I: Illustration of label propagation in our method. The two types of nodes are distinguished by two colours (blue and green). The nodes surrounded by dotted line are training nodes Dtroin whose label may be incorrect and those surrounded by solid line are clean sets Detean. In Fig- ure [(b] one half every training node is pseudo label predicted by LP and the other half original label. Some nodes' (node 5,7) pseudo labels are the same with their original labels, we select them Deelect to train GNNs and inject them to clean sets for better label propagation. We can get proper labels for the left nodes Dieft (node 6,8,9,10) based on meta learning.
METHODS
3.1 PRELIMINARIES
Given graph data with nodes and their labels D = {(Tg- yo), (I1, (11,81),..., (Fn-1, where By the j-th node and yj E 10 is the label over e classes. Dtrain {x mo), (I1, 31) v1).... (1a-113s--1 are training nodes with noisy labels. Our goal is to en- able the GNNs f(z,; w) trained with noisy sets Derain can also generalize well on test nodes. is the learnable parameters of GNNs. In our method, m nodes with true labels Datean f(I,. ya), (x++1, 1o+1).... ..... (I++m-1: aa--m-------- m-)m-1)) in the graph are provided as the initial clean sets (m < s). GCN (Kipf & Welling 2016) and GAT (Velickovic et al. (017)) are utilized in experi- ments to extract features and classify nodes. Our method includes two main parts: label propagation and label aggregation. We will go into details about these two parts in the following section 3.2 and
section 3.3
3.2 LABEL PROPAGATION
Label Propagation is based on the label smoothness that two connected nodes tend have the same label. Therefore, the weighted average of neighbor nodes" label of a node is similar to this node's true label. An illustration of LP part in our method can be found in Figure. [ The first step of LP is to construct an appropriate neighborhood graph. A common choice is k-nearest graph (Iscen al.) 2019 Liu et al. 2018) but there is an intrinsic graph structure (adjacency matrix A) in graph data, so our similarities matrix W with zero diagonal can be constructed with A, whose elements W are pairwise similarities between node and node j:
As.
W = d(h;, h1) E
(2)
where h2g hy are the feature vectors extracted by GNNs for node i and node j. d(..-) )) a distance measure Eucl..an distance), E is an infinitesimal. Note that we can get W with time complex- ity O( 1) instead On?2) because is a sparse matrix whose edge lists are given. Then we can normalize the similarities matrix W: S= 1/11//2WD 1/22
(3)
where D is diagonal matrix with (i.i)-value be the sum of the i-th row of W. Let (k)
(b(*)__ vn., (r E nxe be the soft label matrix in LP iteration k and the i-th row y! (8)
yn predicted label distribution for node When & O. the initial label matrix ( [b(") y/
is the
10)
3

---

Under review as a conference paper at ICLR 2021
consists of one-hot label vectors for = s 1, ...,, $ m 1(.e., initial clean sets) vectors otherwise. The LP (Zhu et al. 2005) in iteration & can be formulated as:
y(k+ 1) = SY
(4)
y: (++1) = yj vi [s. +ms++m--
(5)
In Eq. (4), every node's label in the (k: + 1) iteration equals the weighted average of its neighbor nodes' labels in k-th iteration. In this way, the clean sets propagate labels to the noisy training nodes according to normalized edge weights. And then in Eq. (5). the labels of clean sets nodes reset to their initial values. The reason is that we can take full advantage of the tiny minority of clean nodes and case that the effect of clean sets fade away.
Co-teaching (Han et al. 2018) and Co-teaching plus (Yu et al. 2019) have been proposed to train DNNs robustly against label noise. There are two DNNs which select samples with small loss from noisy training sets to train each other. Our method is similar to theirs to some extent because LP is utilized by us select true-labeled samples from Drrain for training. However, instead of taking the nodes with small loss as true-labeled nodes, select the nodes Deetect whose original labels are same with pseudo labels for training. Original labels of D Deetect are credible and we also inject them to initial clean sets Delean for better LP in next epoch. This is why method can achieve better performance even few true-labeled nodes are provided.
3.3 META-LEARNING BASED LABEL AGGREGATION
Aggregation net forward
Training loss
Clean loss
Aegrogation net backward
SONN SONNhactoane hackowadd clean
B.Bacloaard bacloward
4.GN forward clean
3.GAN 3,GNN backward
LONN LONN Korward
Dut
Dalam
Gradients descent
Figure 2: Computation graph of meta-learning based label aggregation.
In section 3.2 the selected training nodes (node 5,7 in Figure 1h have been utilized for training and LP but the left training nodes Dieft (node 6,8,9,10 Figure I with abundant information haven't t been fully exploited. In this section, mine the abundant and precious information from Dieft via meta learning. The computation process of label aggregation shown in Figure.
For v(zz, vy) Diefe, can get two loss values:
1 loss (oss(O;.Vi) Vi),
(6)
12 loss(ū;- 07);
(7)
where új is the label predicted by GNNs for training node j and ūj is the pseudo label predicted by LP for node j. We can also get final label y, for node j by aggregating original label and pseudo label ūz:
Dj Auus + (1 A5)ū,- A3 [0.1]
(8)
4

---

Under review as a conference paper at ICLR 2021
where À is the aggregation coefficient. Some previous methods designed weighting function map- ping training loss to sample weights for noisy label problems (Kumar et al. 2010; Ren et al. 2018 Shu et al. 2019). Instead, utilize 3-layer multi- layer perceptron (MLP) the aggregation network g ((5 to map loss values to aggregation coefficient )z:
>, (41 l2; 0) Xy w),
Where 1 || 22 is a 2-dimensional vector which is the concatenation of 1 and 12 and 0 is the weights of aggregation network g. The rationality lies on consensus that samples' loss values affiliated with the credibility of samples' original labels (Kumar et al. 2010 Shu et al. 2019 Yu et al.) 2019). The MLP aggregation networks' input layer are neurons and its output layer one neuron, which can be an approximator to almost any continuous functions. The activation function of the last layer is sigmoid function ensure that output Ag E [0, 1]. We can get the training loss L for node j:
(w, (w, 0) = (oss(ū; (w)) (()),
(10)
Then we can backward on the GNNs:
(e:) = W/
Σ
(11)
Dieft (D)ss)
where a is the learning rate of GNNs. Then get the loss L on clean sets Detean:
L (o))
Σ
loss( (0:)) 9:),
(12)
DDeica
(#..8))
Where f(r;; (o))) is the output of GNNs. Then we can utilize L to update the weights of aggre- gation network:
B+11 = 0, -
(13)
where / is the learning rate of aggregation network. Finally, GNNs' weights can be updated:
w
a
Σ
VuLL 12-1)
(14)
Dieft
To some extent, this part is similar re-weight based methods (Ren et 2018 Shu et al. 2019 However, LPM has two significant advantages. Firstly, re- re-weight based methods can not remove the damages caused by incorrect labels because they assign every noisy training sample positive weight while LPM potentially has the ability to take full advantage of noisy samples positively. Secondly, LPM can generate comparatively credible labels for other usages while re-weight or some other methods can not. Algorithm. T shows all the steps of our algorithm.
3.4 CONVERGENCE OF LPM
Here we show theoretically that the loss functions will converge to critical points under some mild conditions. The detailed proof of the following theorems will be provided in Appendix C
Theorem Suppose the loss function loss is L-Lipschitz smooth, and X(-) is differential with a 8- bounded gradient, twice differential with its Hessian bounded by B with respect to 9. Let the learning rate a₁ = min $ for some k >> O, such that & < 1 and learning rate 3r monotone descent sequence, Br min } for some > 0, such that L < / and E 3r S xo, Ei / <
Do. Then the clean loss of Aggregation Net can achieve (VoL)(1(00))1 S in O(1/²) steps.
More specifically,
(oL < O C OSAST
min
(15)
VT
Theorem Under the conditions of Theorem 1 with the gradient of loss bounded by p. then
lim | m. L (wr, 0e+ 00+1)11 = 0.
(16)
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1: LPM. Line 2-12: label propagation; Line 13-22: label aggregation.
Data: D.DerasnDetean max epochs T. LP iterations in every epoch.A.feature matrix X,GNNs feature extractor f, Aggregation Network y, expanding clean set for IP De
Result: Robust GNNs parameters wr
De DDlean
for t = 0, 2,.1,,.... T do
for V/v E D do h, fa wz);
Aig for (1,j) E (1, nn² 2 do Wi. a(h,,hy)+E
for k 0, 0,1,2,...,
do
( = D 1/2 1/ 1/2 (k+1) = yز (A node j E De) end
Declect -Deeft- Ø: for Vf node E Dirain do
if onehol (u)kk) (k)) y: do Deleet node (i} U UU selecti else do Dieft = node {i} U Dieft:
11
12
end
D. De U Delect
14
Wi + one-step optimization of w, with the selected nodes D Dselect
15
for & node Dieft do
16
új f(rz: wi);
17
4 (oss(0j, y3 ): lg (oss(0).1) Ws):
18
Aj = g(l1 12; 0;);
= loss(ū)( (,)) J, Anys + (1 Az)ūs, A E (0,1); 10,11;:; (w,
19
20
end
21
úi (0:) = w/ Piesi
(aneneDDen VuLy (w, we)
(0,)) = Delean
Σ (Z..i)EDo.. loss( (x:; (xx())) y:):
22
0.41 02 -VooL (())))l
Wr+11 Wi- Diefel
(eg Vo (w; 0w)1)mm
25
end
4
EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
We validate method on six benchmark datasets, namely citation networks (Sen et al. 2008 including Cora, Citeseer and Pubmed. Coauthor-Phy dataset (Shchur et al. 2018 is also utilized in experiments, but the results shown in Appendix [A] due the limited space. Summary of the graph datasets mentioned above are shown in Table. The Clothing IM M (Xiao et al.) 2015) and Webvision (Li et al., 201 dataset are utilized to validate the effectiveness of method in real-world label noise settings. We take ANN graph (k = 5) as the graph structure so that GNNs can be applied in these datasets, which follows previous work (Franceschi al. 2019). More details about our preprocessing on Clothing IM and Webvision datasets can be seen in Appendix B
The experiments conducted with types of label noise: uni noise and flip noise following previous works (Zhang et al.) 2016a Shu et al. 2019). The former means that the label of each sample is independently changed a random class with probability p, and the latter means that the label is independently flipped to similar class with total probability p. The ratio of training, validation, and test nodes are set as 4:4:2. Only nearly 25 nodes with clean labels the validation set provided as the clean set in each dataset and ensure that each class has the same number of samples. For example, we use clean samples per label class for Pubmed. GCN (Kipf & Welling 2016) serves the base classification network model in our experiments and it is trained using Adam (Kingma & Ba 2014) with an initial learning rate 0.01 and a weight decay 5 x 10 10-4, except that the weight decay equals to 0 in Clothing M and Coauthor-Phy datasets.
We compare LPM with multiple baselines using the same network architecture. These baselines are typical and some of them achieve state-of-the-arts performance image datasets, which include:

---

Under review as a conference paper at ICLR 2021
Table 1: Dataset statistics after removing self-loops and duplicate edges (Wang & Leskovec, 2020
Cora
Citeseer
Pubmed
Coauthor-Phy
#nodes
2708
3327
19717
34493
#odges
5278
4552
44324
247962
#features
1433
3703
500
8415
#classes
7
6
3 80.2%
#Intra-class edge rate
81.0%
73.6%
93.1% %
Table 2: Comparison with baselines in test accuracy (%) on Cora and Citeseer with uniform noise ranging from 0% to 80%. Mean accuracy (std) over repetitions are reported. The best and the second best results are highlighted in bold and italic bold respectively.
Dansets
Corn
Citeseer
Methodnoise rate
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Basemedel
87.84 (0.04
$5.92 (0.
82.42 (0.13)
15.77 (0.18)
56, (0.19)
72.67 (0.13)
76.06.0.155)
72:97
67.98 (0.12)
55.26 (0
GCN+FT
88.05 (0.0))
$6.07 (0.13)
82.48 (0.14)
75.88 (0.15)
58.81 (0.23)
77.56 (0.15)
76.24
73.42(0.21)
(0.19)).19)
56.12 (0.28)
L2RW
88.84 (0.19)
65.10 (0.20)
80.67 (0.22)
73.43 (0.42)
50,09 (0.37)
16.73 (0.20)
73.68
(0.99)
62.31 (0.33)
(0.49)
Co-tesding plus
71.68 (0.21)
50.05 (0.31)
36.99 (0.44)
76.28 (0.19)
15.49.0.24)
22.71 (0.13)
(0.41)
MW-Nas
88.33 (8.13:0..16)
$5.93 (0.22)
82.61 (0-45)
75,00 (0:41)
56.37 (0.51)
78.27 (0.12)
76.02 (6.14)
74.25 (0.21)
68.06 (0.25)
(6.330048)
GCElos=FFT
(0.13)
82.69
76.16 (0.15)
60.43
78.01 (0.12)
76.51
74.06 (0118)
58.63
APL-FT
87.68 (0.08)
86.26 (8.05)
82.11 (0.15)
74.49 (0.19)
56, (0.25
76.54 (0.08)
74.32 (0.17)
21.77 (0.15)
66,78 (0.22)
(0.08
Ours
(0.07)
87.46
83995
79.66 (0.22)
63.38 (9.27)
78.12 (0.13)
77.07 (0.06)
75.19 (0.15)
(0.11)
61.7 (0.22)
Base model, referring to the GCN that directly trained on noisy training nodes; Meta-learning based MW-Nets (Shu et al. 2019); Typical and effective method
methods L,RRW (Ren et al.,
2018),
2019); Robust loss function against label noise GCE loss (Zhang & Sabuncu 2018) and APL (Ma et al. 2020); The most recent method based on co-training JoCoR (Wei et al.) 2020). For those baselimes that don't need clean sets (Base model, Co-teaching plus, GCE loss JoCoR and APL), we finetune (denoted by FT in this paper) them on the initial clean sets after the model was trained on training sets for a fair comparison. More experimental details about LPM and all baselines are available in the Appendix B
Co-teaching plus (Yu al.
4.2 RESULTS
Table. 2Jshows the results on Cora and Citeseer with different levels of uniform noise ranging from 0% to 80%. Every experiment are repeated 5 times with different random seeds. Finally, we report the best test accuracy across all epochs averaged over 5 repetitions for each experiment. As can be seen in Table. [ our method gets the best performance across all the datasets and all noise rates, except the second for 0% uniform noise rate. Our method performs even better when the labels are corrupted at high rate. T shows the performance on Cora, Citeseer and Pubmed with different levels of flip noise ranging from 0% to 40%. It can be seen that our method also outperforms state-of-the-arts methods under flip noise across different noise rate, except that the second for 0% flip noise rate. Our method outperforms the corresponding second best method by large margin when the noise rate is 0.4. As can be seen in Table. 4 our method can also perform better than other baselines in datasets with real-world label noise. We also experiment with Graph Attention Networks (Velickovic et al.) 2017) as the feature extractor and classifier, the results shown in Appendix A demonstrate that our method can also perform well with other GNNs.
Table 3: Comparison with baselines in test accuracy (%) on Cora, Citeseer and Pubmed with flip noise ranging from 0% to 40%. Mean accuracy (std) over 5 repetitions are reported. The best and the second best results are highlighted in bold and italic bold respectively.
Datascts
Cora
Citester
Pubmed
Methodinoise rate
0
0.2
0.4
0.2
0.4
0
0.2
0.4
Basemodel
(0.04)
81.64 (0.11)
61.12 (0.24)
77.67 (0.13)
75.91 (0. (0.14)
52.67 (0.35)
86.18 (0.08)
85. (0.21)
74.21 (0.29)
GCN+PT
88.05 (0.06)
82.89 (0.14)
67.39 (0.42)
77.86 (0.15)
75.08 (0.22)
61.41 (0.23)
86.21 (0.09)
85.55 (0.24)
80.88
L2RW
88.84 (8.89)
80.90 (0 (1)
59.00 (0.34)
16.73 (0.20)
71.85 (0.25)
50.04 (0.44)
86.34 (0.14)
(4.54(0.19)
76.97 (0.31]
Costesching PT
(0.1)
81.37 (0.21)
53.00 (0.51)
76.28 (0.19)
74.66.00 (0.21)
60.59 (0.33)
(0.09)
(0.22)
73.99 (0.35)
MW-Ness
88.33 (0.16)
85.33 (0.23)
67.71 (0:43)
78.27 (0.12)
76.84 (0.19)
61.97 (0.33)
86.02 (0.07)
84.74 (0.17)
78.59 (0.28)
GCT
87.87 (0.13)
83.21 (0.13)
67.80 (0.37)
78.01 (0.12)
76.36 (0.20)
63.66 (0.46)
85. (0.11)
85.47 (0.06)
50.03 (0 (0.42)
APL+FT
81.09 (0.14)
70.07 (0.19)
76.54 (0.08)
73.38 (0.
60.51 (0.52)
(0.05)
85.52
70.08
Ours
88.75 (0.07)
86.95 (0.12)
78.97 (0.33)
78.12 (0.13)
76.39 (0.74)
69.71 (0.39)
86.48 (0.05)
83.58 (0.13)
83.15 (0.36)

---

Under review as a conference paper at ICLR 2021
Table 4: Comparison with baselines in test accuracy (%) on Clothing M and Webvision. Mean accuracy (# std) over 5 repetitions are reported. The best is highlighted bold.
Methods
Basemodel
GCN+FT
L2RW
MW-Nets
GCEloss+FT
JoCoR+FT
Ours
IlothingIM
35.83=+0.03
38.05=+0.13
53.5++0.08
54.15+0.23
56.9=0.08
56.3+0.12
57.35++0.11
Webvision
32.43:10.05
34.5810.08
50.12:10.16
52.42:10.25
53.45-00.13
54.1210.22
55.43.10.17
Cheoser
Aromed
Jelered pm
Jibose Amo
Trueltnded enterto
Trunbcheed
Malia
Figure 3: Comparsion of the true-labeled samples rate Dtrain and D Peleect in various datasets.
Table The performance of LPM without label aggregation and LPM with random À in Citeseer.
Noise type
Uniform noise
Flip noise
Method/noise rate
0.2
0.4
0.6
0.8
0.2
0.4
Ours w/o label aggregation
72.07
68.36
65.69
62.16
54.39
69.26
63.14
Ours with random A
76.88
75.08
72.07
68.28
57.40
74.89
68.30
Ours with tuned Ours
77.22
76.31
73.55
69.17
58.54
75.11
68.72
78.12
77.07
75.19
70.05
61.71
76.39
69.71
4.3 ANALYSIS OF THE NECESSITY AND EFFECTIVENESS OF DIFFERENT PARTS
We design five experiments to validate the neces- sity and effectiveness of different components of our algorithm. Firstly, we compare the ratio of true- labeled nodes in Deelect with Dtrain in the last epoch validate the effectiveness of LP. Figure. shows the ratio of true-labeled nodes in Deelect In the last epoch and Dtrain under uniform noise on various datasets. It can be found that nearly all the nodes selected by LP are true-labeled even most training nodes are mislabeled, which demonstrates the great ability of LP to select true-labeled nodes from noisy training nodes. Secondly, we remove the label aggregation in LPM to validate its neces- sity and the result shows that the performance of our method become much worse without label aggrega- tion. is necessary to mine the potential information from the left noisy training nodes after LP selection. Besides. we validate the effectiveness by replacing the learned aggregation coefficients À with random numbers between and 1. It is obvious that the aggregation coefficients optimized by meta learn- ing outperform random X. Also, we assign the percentage of clean nodes of each label class as X (tuned) for comparison. These validate the effectiveness of the meta-learning based label aggrega-
mnbic 0.
noise rate 0.0
0.36
8.0
Figure AA varies during the training stage on Cora with various uniform noise rate.
tion. The results of above two experiments are shown in Table. 5 We denote the average of X of clean nodes and noisy nodes in Dieft as Xetean and Anoise respectively, A Xelean Anoise- We plot the variation of AA during training stage in Figure. 4 It can be observed that Xetean Anoise across the training stage and the margin between Actean and Anoise grows larger with the training process, which suggests that À optimized by method is valid.
8

---

Under review as a conference paper at ICLR 2021
Tesemodel
Basemoda
AFL1FT
60
56
OJrs
2.00
0.00
0.10
0.5
0.25 Nolso rate
0.50 0,20
022
0.40
0.00
003
0.00
0.15
0.20
0.2
0.30
0.29
040
(a) Cora
(b) Citesee
Figure 5: Test accuracy on Cora and Citeseer across various flip noise rate.
4.4 IMPACT OF FINETUNING AND NOISE RATE
We would like investigate how our baselines can perform without finetuning. As can be seen in Figure. the performance of the baselines will degenerate relatively significantly without finetuning across different noise rate. This illustrates that some baselines (without finetuning) that are designed for image datasets may perform relatively poor graph-structured data and this motivates work which trains GNNs robustly utilizing the structure information of graph data. Besides, We also observe that method only drops nearly 9% when the flip noise rate increased from 0% to 40%, whereas the baseline has dropped nearly 20% 30%, which illustrates that our method is more robust, especially high noise rate, At 0% noise, our method only slightly underperforms weights besed methods. This is reasonable because the original labels are all correct but our method will inevitably perturb few clean labels while the re-weights based methods will not.
4.5
SIZE OF THE CLEAN SET
We try to strike balance and understand when finetuning will be effective. As can be seen in Figure. our method can also perform better even if the size of clean set extremely small. The overall test accuracy does not grow much when the size of clean set is large enough. Besides, the test accuracy of basclines with fintuning will increase significantly when the size clean set grows larger. This suggests that finetuning will be valid when the size of clean set grows larger because GNNs can achieve good performance with relatively less samples (Kipf' & Welling, 2016 method also serve as complementary for
Veličković al. (017). From this perspective, finetuning based methods when the size of clean set is large enough.
80
70.0
07.5
$3.0
00.0
37.
sosemodelie
55.0
53.5
OcEoss IPT
30
Quis
10
20
50
cleen
Mumber of cleen nodies
(a) Cora
(b) Citeseer
Figure 6: Test accuracy on Cora and Citeseer across various size of clean set.
5 CONCLUSION AND FUTURE WORK
In this work, proposed robust framwork for GNNs against label noise. This is the first method that specially designed for label noise problem existing in utilizing GNNs to classify graph nodes and it outperforms state-of-the-un arts methods in graph-structured data, which may serve the beginning for future research towards robust GNNs against label noise. As a future work, we may design an inductive robust method. Besides, better methods that don't need clean sets also the goals of

---

Under review as a conference paper at ICLR 2021
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981-3989, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400. 2017.
Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. arXiv preprini 1rX3v 17960, 2019.
Chen Gong, Dacheng Tao, Wei Liu, Liu Liu, and Jie Yang. Label propagation via teaching-to- learn and learning-to-teach. IEEE transactions on neural networks and learning systems, 28(6): 1452-1465, 2016.
Chen Gong, Hengmin Zhang, Jian Yang, and Dacheng Tao. Learning with inadequate and incorrect supervision. In 2017 IEEE International Conference on Data Mining (ICDM). pp. 889-894.
IEEE, 2017.
Edward Grefenstette. Brandon Amos. Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta- learning. arXiv preprint arXiv: 190.01791. 2019.
Bo Han, Quanming Yao. Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018.
Yifan Hou, Jian Zhang, James Cheng, Kaili Ma, Richard TB Ma, Hongzhi Chen, and Ming-Chang Yang. Measuring and improving the use of graph information in graph neural networks. In International Conference Learning Representations, 2019.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5070-5079, 2019.
Simon Jenni and Paolo Favaro. Deep bilevel learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 618-633, 2018
Masayuki Karasuyama and Hiroshi Mamitsuka. Manifold-based similarity adaptation for label prop- agation. In Advances in neural information processing systems, pp. 1547-1555, 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6988 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: wet.::609002977. 2016.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in neural information processing systems, pp. 1 1199-1197, 2010.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy la- beled data. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5051-5059, 2019.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint 1eo.iv:708.028888 2017a.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1910-1918, 2017b.
Yanbin Liu, Juho Lee, Minseop Park. Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv preprint arXiv: 1000.00005.00.... 2018.
10

---

Under review as a conference paper at ICLR 2021
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Nor- malized loss functions for deep learning with noisy labels. arXiv preprint arXiv:2006.13554 2020.
Hyoungseob Park, Minki Jeong, Youngeun Kim, and Changick Kim. Self-training of graph neural networks using similarity reference for robust training with noisy labels. In 2020 IEEE Interna- lional Conference on Image Processing (ICIP), pp. 1951-1955. IEEE, 2020.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for 2018.
robust deep learning. arXiv preprint 18XXx:/0033.0000
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. Al magazine, 29(3):93-93, 2008.
Olcksandr Shchur, Maximilian Mumme, Alcksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: 181188881. 2018.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning an explicit mapping for sample weighting. In Advances Neural Information Processing Systems, pp. 1919-19300 2019.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv: Machine Learning, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 0rXiv:1710.10000 2017.
Fei Wang and Changshui Zhang. Label propagation through linear neighborhoods. IEEE Transac- tions Knowledge and Data Engineering, 20(1):55-67, 2007.
Hongwei Wang and Jure Leskovec. Unifying graph convolutional neural networks and label propa- gation. arXiv preprint arXiv:2002.06755, 2020.
Hongxin Wei. Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. pp. 2691-2699, 2015.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanic Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00822 2018.
Xingrui Yu, Bo Han, Jiangchao Yao. Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? arXiv preprint arXiv:1901.042: 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 2016a.
Chiyuan
Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: 1.03530, 2016b.
Huan Zhang, Zhao Zhang, Mingbo Zhao, Qiaolin Ye, Min Zhang, and Meng Wang. Robust triple- auto-weighted label propagation for classification. IEEE Transactions Neural Networks and Learning Systems, 2020.
celttii-reovery-bbbee
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp.
2018.
Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learn- ing with local and global consistency, In Advances in neural information processing systems, pp. 321-328, 2004.
Xiaojin Zhu, John Lafferty, and Ronald Rosenfeld. Semi-supervised learning with graphs. PhD thesis, Cameegie Mellon University, language technologies institute, school ..... 2005.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX ADDITIONAL EXPERIMENT RESULTS
Table A.6: Comparison with baselines test accuracy (%) on Cora and Pubmed with flip noise ranging from 0% 40% and Graph Attention Networks, The best result are highlighted in bold.
Datasets
Cora
Pubmed
Methods/Noise rate
0
0.2
0.4
0.2
0.4
GAT
89.85
84.13
67.10
85.55
84.57
74.11
GAT+FT
89.85
84.50
73.12
85.55
84.57
80.55
MW-Nets
87.52
84.26
69.99
85.64
84.5
75.82
Co-teaching plus+FT
88.56
85.42
74.94
85,56
84.48
82.40
GC
89.98
84.38
74.23
85.45
84.54
80.65
JoCoR+FT
90.16
85.00
73.74
85,47
84.58
80.95
Ours
89.92
87.20
75.65
85.72
84.62
83.00
Table A.7: Comparison with baselines in test accuracy (%) on Coauthor-Phy with flip noise ranging from 0% to 40%. The best result are highlighted in bold.
Method/Noise rate
0.0
0.1
0.2
0.3
0.4
Basemodel
96.92
96.32
95.57
94.91
86.25
GCN+FT
96.96
96.41
95.54
94.46
92.25
Co-teaching plus+FT
96.45
96.39
96.10
95.27
92.79
MW-Nets
96.56
96.24
95.62
95.56
89.25
GCEloss+FT
96.99
96.58
95.96
94.77
93.64
JoCoR+FT
96.83
96.59
96.07
94.95
94.11
Ours
96.75
96.71
96.49
96.14
95.14
We also take Graph Attention Networks (GAT) as the feature extractor and classifier and the results shown in Table. 6.6 validate that our method can also perform well with various GNNs. Besides, LPM can also perform better than other baselines in larger graph dataset Coauthor-Phy, the results can seen Table. A.7 We also demonstrate confusion matrices of Basemodel and LPM in Figure. A.4 which visually show that our method improve the robustness against label noise of GNNs by a large margin.
B APPENDIX ADDITIONAL DETAILS OUR EXPERIMENTE
Original Clothing M and Webvision datasets are large-scale datasets with real-world label noise. We randomly choose 5000 images in 10 classes from original datasets and every image serves node in the graph, a kNN graph (k=5) is treated as the graph structure so that GNNs can be applied in Clothing IM datasets. This setting similar to some previous works which also aim to apply GNNs in datasets without graph structure. ResNet-50 with ImageNet pretrained weights is utilized by extract feature vectors for all the images.
Table. A.8 shows the different hyper-parametert in LPM experiments for different datasets. In all the experiments, 25 true-labeled nodes are utilized as the initial clean sets or as the samples for
Table A.8: The hyper-paramette of LPM in different datasets.
Cora
Citeseer
Pubmed
Coauthor-Pis
Clothing M
Aggregation Net's learning rale Aggregation Net's mid-dimension Aggregation Net's weight decay
x 10
1 10
1 10
x 10
1 10
64
100
100
64
50
1 x 10
1 10
1x 10
x 10
1 1 10
I.PA iterations
50
50
12

---

Under review as a conference paper at ICLR 2021
Basemodel
Pubmed Basemodel
55.2%
3.4%
1.0%
22.3%
8.%%
3.6
19
5.7%
2.0%
0%
13%
12.1%
The
42.9%
14.10
2.5
0.0%
1.2%
0.
1.1%
0.6%
06.
0.5%
0.5%
60.0%
30
7.00
7.0%
0.4
0.2%
66.7%
72.7%
6.2%
1.0%
12.0%
0.
ml
6.9%
2
(a)
(b)
(C)
pny.
M6.2%
3,4%
9.7%
3.0%
2.8%
14.1%
0.3%
6.%
0.2%
0-
6.7%
23
0.4%
21 5%
3.5%
0.5%
2.1%
The The
28%
02.2%
-04
2:3m
1.2%
0.6%
12%
0.1%
0.6%
14.6%
0.0%
2.9%
31.1%
14.0%
114
24.1%
0.%
1.1%
74.0%
022
2..0% 12.2%
(d)
(e)
Figure A.4: Confusion matrices of Basemodel and LPM various datasets under 40% flip noise.
Figure. 4(a) 4(c) are the results of Basemodel. Figure. 4(d) 4() are the results of LPM.
finetuning and the total epoch of all the experiments is 300. In Co-teaching plus experiment, the initial epoch is 270, the forget rate is 0.1 and 5 epochs for linear drop rate ,the exponent of the forget rate is 1. For MW-Nets, the dimension of the meta net's middle layer is 100 and the learning rate x 10-3 q for GCEloss is 0.1. The combination of Normalized Focal Loss and Mean Absolute Error is utilized in APL experiments, the weight of Normalized Focal Loss is 0.1 and the weight Mean Absolute Error is 10. For JoCoR experiments, the epochs for linear drop rate is and the exponent of the forget rate is 2. The balance coefficient between conventional supervised learning loss and contrastive loss is 0.01. The learning rate and weight decay of Graph Attention Networks are 0.01 and 10-4 The dimension of hidden layer of GAT is 16 and the number of head attentions is 8. The alpha of the leaky relu is 0.2 and the dropout rate is 0.5. Throughout this work we implemented gradient based meta-learning algorithms in PyTorch using the Higher library (Grefenstette et al. 2019).
C APPENDIX CONVERGENCE OF LPM
Our proof of the convergence of LPM mainly follow some previous works (Ren et al. 2018 Shu al.) ) that utilize meta-learning reweight noisy training samples. As illustrated in some previous works (Zhou et al.] 2004: Zhu et al.) 2005), LPA will converge to fixed point. Namely, Duntect and Dieft will converge to fixed sets. our proof, the final Dieff and final D are denoted with n and m for easier illustration. Loss function loss is denoted by / in this proof. Here
we first rewrite the forward and backward equations follows:
új = [(x;;; wx) = Uy (15( w..
(17)
X, o(1()), 0,) [| 0;); 0.) 20; welle.
(18)
1 Σ (()y + (1 - >y)ūū 01)
L tr (wy; 02)
(19)
==1
(0.) wy V I (w: 1...
(20)
13

---

Under review as a conference paper at ICLR 2021
y1 f(z;; ( = yi (w; (r:)las be
(21)
m
1 Σ
[
[.(
(
= m
(yis Ui)
(22)
i=1
i=1
1=1
0u+1 = 0 - (
(23)
W++1 = Wt - aV 1, (w; 0c+1 1
(24)
(ry. M3) is node from the final left training set Diaft:
(Ii, Mi) is node from the final clean Poleani
f is the GCN for classification with its weights w:
g is the Aggregation Net whose input are the nodes from clean set with its weights 0:
L is the loss on clean sets. L is the final training loss.
Iyy, 0) is the loss (such as Cross Entropy) which satisfies linearity given by
1(xy1 + (1 - A)y2. fi) Ml 9) + (1 - 2)1(12.
Derivation of the equation of updating the weights in Aggregation Net
VoLū(w(0))e. 1
0wr(())
a), we)
Ow
9.
00
8.
(25)
According to Equation 20)
(0)(e).. = a we
1()y8) + +(1 +11-11)) -1)))00;
1=1
al(Ayy) + (1 - X1)0 0b)) 18.
0. n
0g
0),
(ii(()
0[Ay[1yj. 0j) + (1 - x3 2,)0 03>1 82
9.
V
n
(w)(()
10.
we (1(y 0,) 1(5) 01))la.
0g
n
(r)
a(l(y) 03) 1(ū5 05))
9.
n
Ow.
Therefore, Equation
25 m
can be written as
VoL(()(0))a.
m
(L
al 01) - -1((i (()),;))
θ)y (0,(0;w)) we)
1nc
8.
Oi
Ow.
00
m
aL (1)
T a(1ys (5) 1(ūj 1)))
A)3 wt)
(w.)
10.
71
m
Ow
80
j=1
a
Σ
G
a (6: we)
10:1
n
m
00
where Gig
DLS (1) T a(ly و - 0,
Owe
Lemma 1. Suppose the loss function is L-Lipschitz smooth, and X(-) is differential with a 5-bounded gradient, twice differential with its Hessian bounded by B with respcet to 8, and the loss function I(:,.) have p-bounded gradients with respect to the parameter w. Then the gradient of w with respect to T((w)) is Lipschitz continuous.
14

---

Under review as a conference paper at ICLR 2021
Proof. The supposition is equivalent to the following inequalities.
|V (1) m
VoL L|le w2ll:
(26)
for any 10,, W23
\Vo\(0; wx)) S pi
(27)
A(0: w1) _ B:
(28)
I (yy 0s (w); =z))) S 6.
(29)
The gradient of 0 with respect to loss clean set reads
V VOL;( 8. a
(1)(w)
a(ly, U5)
(16, (z)) Im
0A, (0; w+)
71
Oro
Ow:
00
j=1
2)1 (0; we):
Gij
10.
00
j=1
Taking the gradient of in both sides of the equation, we have
aG.j
(A,(0; w.)
0 (2:2(0; wt)
ViaL;(0(0)l. -
8.
le. + (Giy)
r.).
n
30
00
802
For the first term in summation,
aGij
Byy (0; we)
8.
l..l
00
00
0
(T)(
<6
T
00
Ow.
(
all 9n) (TTx, 0x))
07,(0;w),
A((yj, 05) -105- 13))
=61 0r
T1
Ow.
00
Ow.
8
(()))
((1 Un) 1(Tk Bk)))
DA (0: we)
a(ly 91)
[(ūJj, 01
=5](
l0. )r.
17
2
Ow.
09
Ow.
82
(1)
all(yR Dn) 1(Tx, 9x))
(A: we)
a(( 9,)
((ū 11))
<oai||
wws |||
9.. I
Ovi2
Ow.
00
Ow:
<4aLpp282.
And for the second term,
82 (0 wxr
OL;(1) dτ ling
a(1(y,, 0,) -1(0;, 1)))
8 80,;(0; wt)
([(),)))
lo.11 =
(T
lo.11 <2BB².
002
Owr
002
Therefore,
((000)))) < Lp282 + 2ap2B.
(30)
Let L. Lp222 + 2ap2B Based on Lagrange mean value theorem, we have
| [V, (oT S T.0110 02
for all 01, 02.
15

---

Under review as a conference paper at ICLR 2021
Theorem 1. Suppose the loss function is L-Lipschitz smooth, and X(-) is differential with 8- bounded gradient, twice differential with its Hessian bounded by B with respect to 9. Let the learning rate a₁ min(1, 4), for some & > 0, such that $ 1 and learning rate Pn monotone descent sequence, Sr m Vr for some e > such that S Vi and Ei Sr S 20, B S 00. Then the loss of Aggregation Net can achieve [ S e in ssteps. More specifically,
| (Voo 0 C USIST
min
(31)
Proof. The iteration for updating the parameter reads
O2+1 0. 3V (v)
In two successive iteration, observe that
(((0)4(00++1) (O)))
(1 - (( +
(1 (L.-1()))) - ( (o)))
(32)
For the first term, given that loss function on clean set Lipschitz smooth, we have
L (()+11))))1 (w)(((+))
s < (L((2((0++1)) (0(+1) (B(+1) > + _ (1 @0((0++)11
According to Equation 20 and
@ (0((0(+1)
n
and thus,
I
((0-1(0))))) ) S ppp² 2
aip²
since the first gradient of loss function bounded by p. By the Lipschitz continuity of L (t)) according to Lemma 1., can be obtained that
L (0x((0)+))) ((.))
L 01+1 8,) + (v),L)((i(()))) -
< ((, +
-/1 110001 - 0.11
2
LB?
V (1),((11
2
LB8
(3, 2 v L.
Therefore, the Equation
32
satisfies
L I - (3:
L88
L L9 L' S a,p²
2
2
T,33 (3, )12
I
< app²
otp² (( + (
2
Summing up above inequalities from to T. we have
T
L
T
L
Σ
(B.
+
Σ (app² + a
2
2
1=1
1=1
L (Br
L
Σ
min Ve, (20, (4
(aup² , afpp')
1=1
16

---

Under review as a conference paper at ICLR 2021
Furthermore,
( ΣI= (asp² + 4ag
min (
Σ (3. -2
Lot
( (01)) + Σ 120 + Lagp')
Σi=1 (2 -188)
()1)) + E/=1 (200p² + La?
Σt= (Bt)
2L 2L-((0((11) + (2
TB.
21) (01) 1
(2
T
Be
Be
2L (1i(
VT
min & ) max(L, VT k }02 L.)
mT
T
k
k2 + L) 2L (01
1
cVT 0
cVT
holds for 1t (Bx) S ΣT (23, I, In conclusion, proves that the algorithm can always achieve millosIST Vo (vx()))) S (())) T steps.
Let (an)i Sns (bn) be two non-negative real sequences such that the series Li; an diverges, the series Ei. an converges, and there exists K 0 such that [|b, b,lll S Kan- Then the segences (b,)ign converges Proof. See the proof of Lemma A.5 in [Stochastic majorizatior-minniizaaoo algorithms for J.
Lemma 2.
Theorem 2.
Suppose the loss function is L-Lipschitz smooth and have p-bounded gradients with respect to training data and clean set, and X(-) is differential with d-bounded gradient twice differ- ential with its Hessian bounded by with respect 6. Let the learning rate a min(1, & }. for some k > 0, such that & < 1 and learning rate 3, a monotone descent sequence, 3, = min +, +
for some > such that L s vi and E Be S co, Ei= 83 oc. Then
lim |V, 1. (," 0z+1))))))) 1000
Proof. is obvious that o, satisfy E ar 0c, 22 a In Eq. of L, we rewrite the update of w as
18
19
20
and the linearity
Wi+1 = - VL' b2t1)
n
W::
ar
W0 (11 + +(1 (1- >, 20(0+11::)) ([T 0z (w.))
n
First, we have the difference of the loss function on training set between two iterations,
("((w+11;044) -
- (we+11; + II" (w141:0041) L (wi; 0441)]-
(33)
17

---

Under review as a conference paper at ICLR 2021
For the first term in Eq 33 by the L-Lipschitz-smooth and p p-bounded gradients of X with respect to training and clean set,
L (1141:::42) L 1 (1++1;00+1))
(11(0)2) - 10(01+1 (1) ((r+1)) + (2y(0++1: (++1))))))+++00 - (0 (11+) 0s (we+1))
02, 0x,(0; (1;1++))
M,11:0241 -00t1)0r))/ )
8
+ -:100-2- 0c+ (K1, Uyys ((wz+1))
00
a (0; (0:w)+1)
5,8
Σ
laets, Be Vo, (())(()))))
VV, ((0r))))))))))))))))) I(y,, (w++1)))
n
For the second term in Eq. 33
L ("141::044) - L ("(r00;00+ 9c+1)
< (Vez L (wri (211) W141 we) + L
"1 we
(aa - Loz V/ L L (wo;; 1111)
2
Therefore, we have
L. (we+1; ((w0+1:0+42) 0c+2) - (wr; 0++1)
1
00,(0; w(+1)
00 la++1, L.
n
65?
+
Vo, ((mr
((we+1)) + 1(85, ((w(+1))) ||| w. Ld (wi; 0.+1)113
La²
(a.
2
Summing up the inequalities in both sides from t to we have
lim
(7,"((e 0c+2) (wi: 02)ll
10000
3
ar wi+1)
|on1l/// (() ((w411))l + [[({,, ((w+11))l2)
30
1=1
j=1
8,32
Vo, I wi (w: ((0,))))) + [1,
2
1=1
La?
Σ
(an
V _
(w,; 0111)12
2
1
18

---

Under review as a conference paper at ICLR 2021
Rearrange the terms of the inequality, obtain
Oit V we L (w0r;(+)))1
i
n
0) w
1ett [1 ((w+11)) + (1 (w++1))
n
09
Lo
V
1 (w) O 0111)11
2
8
V L (to, ((w+1))] + [[ ((w++1))ll))
2
==1
lim || (W1+1; 0142)112 | (w1: 02)ll2
Lot p² |1 (wi; 02)
833 (2Mp²)
lim (1ee+1100))))))) t-900
500.
The inequality next to last holds since our loss function is bounded by M, and the last one holds for Ei of and 2o.1 8 are finite.
In addition, since
D), Wc+1)
la0es (Ll 10) + [1 ((w+1)))
n
00
2=1
<2M pó
3: S 0c,
1=1
we can obtain that
Ore Vw, L (wr 12+1)11 S oc.
(34)
In the other hand, based on the incquality:
([aal + [[][) [[bl]) I ||a blll|a blI,
have
|| L
(W++1 "VVL"(ww)))) (00441100442)) 0142)11 VL' Oz+ 1
+ 72 30
77 1/ B.42) (
(L, [,"" 0241)
22 0(42) (w., 0111
<2 V (w) 02+1), (w,, 0211))112
(wr,
<2/2Lp"B..n
=Car-
For Eq. 34 which reads
Σ Ore vi, L (we; 0c+1) oc,
since Eie Ort 0c, and there exists K
> O, such that [ 0x42) [|VL" (w; 0142)111 S Cas, by Lemma 2., we can conclude that
lim V L (w;; 0++1)l1 = O, 100
which indicates that the gradient loss on training set of our algorithm will finally achieve to zero, and thus the iteration of enables training loss converge.
19


Reviews:
[{"review_id": "yctm2N1LE3b", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*quality*\nThis paper is well-organized. However, the contributions are not clear. Meanwhile, the experimental validation is weak and needs to be improved.\n\n*clarity*\nIt is not difficult to understand the framework of the proposed algorithm.\n\n*originality*\nIn this paper, the proposed algorithm focuses on the label noise existing in graph node classification tasks. The contributions are somewhat limited, since this is not the first work focusing on this problem. Besides, the main motivation is also not new for semi-supervised label propagation. \n\n*significance*\nAlthough the proposed algorithm is limited in its novelty, I feel that the significance of this paper will inspire researchers focusing on this domain.\n\n*pros and cons*\n\nPros:\n1.\tThe topic is practical in realistic machine learning problems, and the proposed method is helpful to reduce the human annotation efforts.\n2.\tThe theoretical study of this paper is meaningful.\n\nCons:\n1.\tThe authors claim that they are the first to focus on the label noise existing in graph node classification tasks. However, to the best of my knowledge, there already exist some prior works, such as ‘Learning with Inadequate and Incorrect Supervision’,‘Self-Training of Graph Neural Networks using Similarity Reference for Robust Training with Noisy Labels’, ‘Noise-robust semisupervised learning by large-scale sparse coding’, ‘Robust semi-supervised classification for noisy labels based on self-paced learning’, and ‘Label Diagnosis through Self Tuning for Web Image Search’. In this sense, the authors cannot claim that they are the first to solve the label noise in the label propagation or node classification task. Besides, the abovementioned works should be cited, discussed, and even compared.\n2.\tSome recent works on label propagation can be cited, such as ‘Label propagation via teaching-to-learn and learning-to-teach’, ‘Robust Triple-Matrix-Recovery-Based Auto-Weighted Label Propagation for Classification’, etc.\n3.\tIn Eq. (8), the authors calculate the final predictions by aggregating the original labels and pseudo labels, in order to fully exploit the abundant information contained in the left training nodes D_left. This is a bit confusing, since both the original label y_j and the pseudo label y^\\tilde_j may not be the correct labels. The motivation behind this aggregation process should be explained appropriately. \n4.\tIn the experiments, it seems that the author do not use the public splitting for Cora, CiteSeer, and PubMed datasets. As a consequence, the results reported in the paper may not be convincing enough. It would be better to follow the public split for fair comparison.\n5.\tThe proposed model outperforms the comparison methods when the level of flip noise is greater than 0, but cannot perform as good as L2RW when the level equals 0. The reason for such inconsistencies should be analyzed in the experiments.\n6.\tAs for me, the selection of clean labels is a critical step. However, the authors do not give enough details about this process. I suggest analyzing the influence of the scale of clean labels on different datasets.\n7.\tI cannot find the hyper-parameter configurations for the baselines, and these information should be provided. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "uyXhiMNwchh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a label propagation based meta learning algorithm to address label noise. Label propagation helps re-label pseudo labels of noisy data and meta learning achieves aggregations. The method is evaluated on several node classification datasets and a custom version of Clothing1M image classification dataset. The comparison to multiple baselines shows better performance.\n\nPros:\n- The combination of graph neural network and meta learning is interesting and novel\n\nCons:\n- The paper is not well written. Some descriptions are unclear. For instance, the usage of $w$ is never defined. The usage of meta learning is not novel, which from a previous method - L2R, but it is never clearly mentioned. However, it tried to use several equations to explain the L2R method, which is not sufficient to be clear but makes unnecessary confusion.\n- Experiments might not be that convincing.\n  - The method is only tested on several graph datasets with synthetic noises, which were uncommon in previous papers. However, most compared methods are only evaluated on common image datasets. The comparison may not be very fair. Moreover, from Table 2 and 3. The proposed method is marginally better than several other methods.\n  - Although the method tries to tackle a real-world dataset Clothing1M, the scalability issue of this algorithm makes it difficult to work so it is only tested on a custom toy version. So the results are not generally useful to compare to other methods which have tested on full Clothing1M.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AlA5s0LC9cr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a robust training algorithm for graph neural networks against label noise. The authors assume the labeled nodes are divided into two parts, clean part without noise and train part with some noise. The proposed method contains two parts. Firstly, it leverages label propagation (LP) trained on the clean nodes to assign pseudo labels on train nodes with noisy labels. Secondly, the authors design a learnable weight \\lambda to learn the label for those noisy nodes where LP does not agree with the original labels. The final graph neural network is trained with clean nodes, high confidence train nodes, and uncertain train nodes with learned labels. The authors conduct experiments on four graph datasets with manual injected noise and one real-world noisy dataset to validate the proposed method.\n\nThe paper studies an important problem of graph neural networks with noisy label. I have several concerns for the paper.\n(1)\tThe idea of using LP (or another algorithm different from GNN) to create pseudo labels for uncertainty nodes is not a new idea (e.g., in [1]). Actually, the LP, original data and GNN are ensembled and the final label comes from the majority vote. When LP agrees with the original data (in noisy part), those labels are retained. \n[1] Li, Qimai, Zhichao Han, and Xiao-Ming Wu. \"Deeper insights into graph convolutional networks for semi-supervised learning.\" arXiv preprint arXiv:1801.07606 (2018).\n(2)\tWhy the joint learning of \\theta with GNN parameters is named meta-learning is not very clear. Algorithm 1 shows both \\theta and \\w_t are fixed to estimate labels for uncertain nodes. Then those parameters are updated with the estimated nodes. Besides, in the experiment part there is no tuned \\lambda compared (random is not good, a wrong \\lambda can hurt the performance). This makes the improvement from the learnable \\lambda less trustful.\n(3)\tThe experiments are less convincing because only one real-world noisy dataset is available. Because the authors have a strong assumption that both noisy and clean labels exist, it would be better if the authors can validate such assumption with some real-world data. The reported numbers show the improvement of proposed method is rather limited, I would suggest run the methods more times and report mean performance with variance. Results with manually tuned \\lambda should be reported. Besides, since GNNs are good for learning on graphs with few labeled nodes (such as the gcn paper actually only use several labeled nodes per class), I would suggest adding some additional experiments for the size of D_clean. How will the rest RL + learnable \\lambda contribute w.r.t the number of clean nodes?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7RHeTJnpnB3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents the one technique using label propagation with meta learning. The label smoothness is used to pseudo label the nodes in the graph. The experimental results look promising in two conditions of label noises. The paper is presented clearly and easy to read. Overall, the quality is good.\n\nThe paper present the idea and experiments clearly.\n\nI checked a few literature and believe this work is original.\n\nPros:\n1. clear presentation\n2. method is simple but seem very effective\n3. the experimental results outperformed the state-of-the-arts in both synthetic label noise and real noise scenarios.\n\nCons:\n1. the contributions can be challenged. If check the GNN with label noise, I do can find some literature published on CVPR and ECCV. And the third one is evaluation result, cannot be categorized as a contribution.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We thank the authors for their detailed answers and for providing an updated version of the paper addressing several of the issues raised by the reviewers, including new experimental results.

The paper is technically correct. The comparison with other methods is thorough and includes ablation studies clarifying the contributions of different aspects of the proposed method. One aspect that has been moderately addressed in the new version is the comparison between the "learned lambda" of the paper with a "tuned lambda" suggested by a reviewer. The authors added results where lambda is set to a particular value, however it would be more interesting relevant to consider a real "tuned lambda", i.e., a scalar parameter, shared by all vertices, that is optimized during training; the goal being to clarifying the benefit (if any) of parameterizing lambda as a function of the node, as opposed to a value shared by all nodes.

The paper is clearly written, particularly the revised version.

The novelty is the weakest aspect of the paper. While the specific problem of learning with noisy labels with GCNN may be new, the field of learning with noisy labels in general, and of using label propagation from clean labels to guide the prediction of uncertain labels, has been proposed before, and mentioned in the reviews. The specific instantiation of this idea to the GCNN framework is novel.

The significance of the work is rather positive. The revised version contains results on two real-world datasets, where the proposed method outperforms several existing ones. As mentioned by a reviewer, this paper may inspire other researchers to explore in more depth the specific problems of learning with noise on graphs with GCNN, and to exploit the knowledge of a limited set of clean labels which may have practical importance to reduce human annotation efforts.

In summary, the paper proposes a novel model and demonstrates its potential to address a possibly important problem.
Although the reviewers did not update their reviews, the authors' responses and updated version correctly addresses several of the initial concerns. The limited conceptual novelty compared to existing work did however not convince us to recommend acceptance, given the high selectivity of the conference.
Decision: Reject