Graph View-Consistent Learning Network
Authors: Zhuolin Liao
Kun Zhan
Venue:  (2021)

Abstract: Recent years, methods based on neural networks have made great achievements in solving large and complex graph problems. However, high efficiency of these methods depends on large training and validation sets, while the acquisition of ground-truth labels is expensive and time-consuming. In this paper, a graph view-consistent learning network (GVCLN) is specially designed for the semi-supervised learning when the number of the labeled samples is very small. We fully exploit the neighborhood aggregation capability of GVCLN and use dual views to obtain different representations. Although the two views have different viewing angles, their observation objects are the same, so their observation representations need to be consistent. For view-consistent representations between two views, two loss functions are designed besides a supervised loss. The supervised loss uses the known labeled set, while a view-consistent loss is applied to the two views to obtain the consistent representation and a pseudo-label loss is designed by using the common high-confidence predictions. GVCLN with these loss functions can obtain the view-consistent representations of the original feature. We also find that preprocessing the node features with specific filter before training is good for subsequent classification tasks. Related experiments have been done on the three citation network datasets of Cora, Citeseer, and PubMed. On several node classification tasks, GVCLN achieves state-of-the-art performance.

Under review as a conference paper at ICLR 2021
GRAPH VIEW-CONSISTENS LEARNING NETWORK
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent years, methods based on neural networks have made great achievements in solving large and complex graph problems. However, high efficiency of these methods depends on large training and validation sets, while the acquisition of ground-truth labels is expensive and time-consuming. In this paper, graph view-consistent learning network (GVCLN) is specially designed for the semi- supervised learning when the number of the labeled samples is very small. We fully exploit the neighborhood aggregation capability of GVCLN and use dual views to obtain different representations. Although the two views have different viewing angles, their observation objects are the same, their observation repre- sentations need to be consistent. For view new-coonisteent representations between two views, two loss functions are designed besides supervised loss. The supervised loss uses the known labeled set, while a niew-consistent loss is applied to the two views to obtain the consistent representation and pseudo-label loss is designed by using the common high-confidence predictions. GVCLN with these loss func- tions can obtain the view-consistent representations of the original feature. We also find that preprocessing the node features with specific filter before training is good for subsequent classification tasks. Related experiments have been done on the three citation network datasets of Cora, Citeseer, and PubMed. On several node classification tasks, GVCLN achieves state-of-the performance.
INTRODUCTION
Convolutional neural networks (CNNs) (Krizhessky et al., (0122) performed outstandingly in solving problems such as image classification (Rawat & Wang 2017). semantic segmentation (Kampffmeyer al., 2016) and machine translation (Cho et al.) 2014) etc. This is because CNNs can effectively reuse the convolution kernel and use the given input to train optimal parameters. The original data mentioned in above problems all have a grid-like data structure, that is, Euclidean spatial data. In reality, there are also lots of -un-Euclidean spatial data, such as social networks, telecommunication networks, biological networks, and brain connection structures, etc. These data are usually repre- sented in the form of graphs, where every node in the graph represents single individual. Graph problems can be roughly divided into there direction: link prediction (Zhang & Chen 2018), graph classification (Zhang et al.) 2018a) and node classification (Kipf & Welling. 2016). In this paper, focus on semi-supervised node classification when the label rate is very low.
Many new methods have been proposed to generalize the convolution operation to process graph structure data on arbitrary graphs for node classification. These methods can be divided into spatial convolution and spectral convolution methods (Zhang al. (2018)). For spatial methods, they di- rectly define graph convolution by designing certain operations on node's 's neighbors. For example, Duvenaud et (2015) propose convolutional neural network that can directly operate on graph data, which can provide an end-to-end feature learning method; Atwood & Towsley (2016) pro- pose a fusion convolutional neural network (DCNNS), which introduces the graph fusion method to incorporate the context information of the nodes in the graph node classification; The Graph At- tention Network (GATs) /Veličković et al. 2017) introduces the attention mechanism into the graph data processing to construct the attention layer for semi-supervised learning. The spectral method generally defines the graph convolution operation on spectral representation of graph. For example, Bruna et al. (2013) propose that graph convolution be defined in the Fourier domain based on the eigenvalue decomposition of the graph Laplacian matrix; Defferrard et al. (2016) propose to use the Chebyshev expansion of the graph Laplacian approximate the spectral domain filtering, which can

---

Under review as a conference paper at ICLR 2021
avoid high computational complexity brought by eigenvalue decomposition; Kipf & Welling (2016) propose a simpler Graph Convolutional Networks (GCNs) for semi-super learning, which can achieve higher classification accuracy by using a simple two-layer networks. However, large train- ing and validation sets are required in these methods to complish effect classification task, while obtaining true labels is time-consumingg laborious and costly. And they send original graph node features directly into networks for training, however, there are lots of redundant information in the original features of the nodes.
In order to train an efficient model with only a few label nodes and even without validation, we put forward our own method: graph view-consistent learning network (GVCLN), which constructs a node classification network based on the consistency between two views. First, we independently train two-view encoders (can be different) to obtain two representations of every node. The func- tion of the viewers is converting high-dimensional node features into low-dimensions. embeddings (Zhu et al. 2020). The clustering hypothesis (Vandenberg & Matthias 1977) show that examples in the same cluster are more likely to have the same label. According cision boundary should try to pass through the place where the data is relatively sparse, so as to avoid dividing the data points in dense clusters both sides of the decision boundary. Although the two views have different viewing angles, their observation objects are the same, so their ob- servation results should be consistent. Therefore, the features encoded by the viewers should make the decision boundary pass through the place where the date is sparse, that is, there should be consistency between the two views. Then, design three loss functions, namely, supervised loss function, consistency loss function, and pscudo-label loss function. The supervised loss uses the known labeled set, while view-consistent loss is applied the two views obtain the consistent representation and pseudo-label loss is designed by using the common high- high-confidence predictions as pseudo label. GVCLN with these loss functions can obtain the view-consistent representation of the original feature. Our contributions are summarized as follows:
this hypothesis, the de-
We propose graph riew-consistent learning framework for semi-supervised node classifi- cation, which fully demonstrates the theoretical structure of graph view-consistency.
We design GVCLN to successfully tackle label insufficiency in semi-supervised learning.
We demonstrate the high efficacy and efficiency of the proposed methods on various semi- supervised node classification tasks.
2 RELATED WORK
2.1 NOTATIONS
A graph contains two parts: node and edge. Each node represents an individual, which can be paper or a person, etc. The edge indicates a connection between two nodes. If the edge connecting two nodes in the graph is directional, it is directed graph, otherwise it is undirected graph. A simple and connected undirected graph can be written as = (V, E), where V is the node set and E is the set of edges. great influence on the graph structure, the graph used in the calculation of the network is generally a self-loop graph, namely G (V. E). which attaches a self-loop to each node in G. A denotes the adjacency matrix and D is the diagonal degree matrix. Therefore, the adjacency matrix and diagonal degree matrix of G are defined as Á = A + I and ń, respectively. / indicates the identity matrix. The node feature matrix is X E gnxd in which, each node is associated with a d-dimensional feature vector x,, The normalized graph Laplacian matrix is defined as L I D-1/2 AD 1/2 which is a symmetric positive semidefinite matrix with eigendecomposition TA where A is a diagonal matrix of the eigenvalues of L, and U Rex is unitary matrix that consists of the eigenvectors of L. The graph convolution operation between signal æ and filter g, (A) = diag (?) is defined as 9, (L) x = Ug, (A) U x, where the parameter 7 E R" corresponds to a vector of spectral filter coefficients.
V represents the number of all nodes in G. Considering that the node itself has
2

---

Under review as a conference paper at ICLR 2021
2.2
FEATURE AGGREGATION
Graph neural Networks (GNNs) follow a neighborhood feature aggregation scheme. Many GNNs different aggregation approaches. Here, introduce most classic feature aggregation ways.
Graph convolutional network (GCN) (Kipf & Welling 2016p has outstanding effects in solving the problem of semi-supervised node classification. Kipf Welling (2016) define the graph convo- lutional layer for feature aggregation in GCN. Symmetrically normalized operation is used on the adjacency matrix of G, i.e., A = -D---11- D-1/2AD 1/2, The graph convolutional layer is defined by.
H a (AH')),
(1)
where H' is the feature of the 1-th layer, θ is the trainable weight matrix of the layer, and o() is nonlinear activation function, e.g.. ReLU(()) = max(0, .).
Veličković et al. (2017) present graph attention network (GAT), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the short- comings of prior methods based on graph convolutions or their approximations. By giving different attention weights Qz to different neighbor nodes, each node-i are able to aggregate their neighbor- hoods features to update their own features. The graph attention layer is defined by,
h
[+1
Σ any whyhh)
JEN;
where ha represents latent embedding feature node i, W is linear transformations weight matrix, and Oij are normalized attention coefficients.
3 MODEL ARCHITECTURE
In this section, we first introduce the overall abstract structure of GVCLN, and then use graph convolutional layer and graph attention layer to build specific GVCLN model to achieve view- consistent learning for node classification.
3.1 MODEL ARCHITECTURE OVERVIEW
general understanding of our GVCLN shown in Fig. [ GVCLN, feature X and the graph A are encoded by two different viewers at the same time, namely Viewer 1 and Viewer 2, the two
viewers aggregate features with three-head graph layers. Viewer 1 uses the graph convolution layer
and Viewer 2 is the graph attention layer. The three-head representations, H(()) H H v E
2), are of concatenation and perform dropout. After dropout, the two-view latent features, 2
and 2 through a same non-linear graph convolutional layer to obtain the prediction, P()) and P(2) A view-consistency loss function is made for the dual-view output consistently.
11
Concat
Output
Dropout
Input
Output
H
Input
H
Viewer
View Consistency
Input
H110
Concat Dropout
Output
Input
Output
A
H
p(2)
H
Viewer 2
Aggregate Feature
Aggregate Feature
Figure 1: GVCLN architecture

---

Under review as a conference paper at ICLR 2021
3.2 VIEWCONNIISTENCY LEARNING
We consider applying view-consistent learning to the view I and view in Figure 1 Inspired by contrastive learning, [Cen et al. (2020) inputs the original sample and the sample after data augmentation (flipping or cropping. etc.) into two same views respectively. Since data augmentation does not change the label information of sample, the output representations of two views are similar. By maximizing the consistency of the two views, better results can be achieved. In this paper, we do not do data augmentation to the data, but used two different viewers. We believe that the same data has the similar output through different viewers and classifiers. In Figure | there is a consistency loss function between the view and the view 2 to realize view-consistent learning of two views. The soft cross entropy function is employed to construct the consistency loss function:
1
(1) (2) Pij In
Evjew
Iipi
π
1=1
where k is the number of class, p(1) viewers for node-i, respectively.
1 and p(2) [pl (2)]
are the predictions of the two
The concept of co-training was proposed by (Blum & Mitchell 1998). co co-training assumes that each data point has two views, and each view sufficient for learning an effective model. Co-training learns separately on two views with labeled data samples to obtain two different learners and then uses the unlabeled data samples in the respective views of the two learners to make predictions. The basic learning process of collaborative training is: Each learner selects the data sample with the top high-confidence predictions in unlabeled data and adds it to the labeled data sample set of another learner for training. This process is repeated until specific stopping condition is met. Co-training needs to meet the following two conditions:
Views are sufficient and redundant, that is, for each view, if enough labeled data is given, each view can learn a learner with good performance. Conditionally independent, that is, the label information of each view is conditionally in- dependent of the label information of another view.
Our view-consistent learning method is similar to co-training, which requires pseudo-labels for train- ing. The selected pseudo-label data cannot be completely clean (the predicted label is exactly the same the real label), but contains noise (the situation where the predicted label is inconsistent with the real label). Due to the powerful representation capabilities of deep neural networks, even noisy labels can be fitted by deep neural networks. Based on (Han et al., 2018). which can train deep neural networks with extremely noisy labels, we propose our own training strategy. We use labeled samples to train two independent learners, namely view 1 and view 2. When the two-view networks are well trained, according to their predictions, the predicted labels of some nodes with the same prediction representations and common high-confidence prediction are selected as pseudo labels. Then, pscudo label set is added to the original labeled set for training. This is repeated until the stop condition is reached.
We believe that pseudo labels have similar importance as real labels, but the number of pseudo-labels taken each time can be controlled by the loss function value and the accuracy of the verification set, and different numbers of pseudo-labels can be given to data sets with different label rates.
GVCLN selects nodes with high confidence and with same prediction results in the two views as pseudo labels. The pseudo labels are voted with high-confidence predictions of the two viewers taken from unlabeled nodes.
The pseudo-label loss has the same effect as consistency loss, that is: making the two views more and more consistent. Meanwhile, supervised loss has the function of fine-tuning and controlling the direction of network optimization.
We choose cross entropy as a loss function to calculate the loss for both the labeled samples and the pseudo labels:
CEa-
Σ
Yij In Pij.
(4)
4

---

Under review as a conference paper at ICLR 2021
where Viy denotes the ground-truth label, i.e., the i-th node belongs to the j-th class, Psj is the network prediction probability of the i-th label.
Thus, the supervised loss function of labeled samples is:
sup
Yij In Pij
(5)
where denotes the number of labeled nodes,
For pseudo labels, we also use cross entropy to calculate the loss function:
l
psscudo
Σ
Viy In
Paj-
(6)
m= ==1
where bly is the pseudo label and denote the number of nodes of the pseudo labels.
3.3 ALGORITHM
Then, the total loss of GCVIN is given by:
Gooaal
sup + e(2) + Blview lpseudo
where 3 is trade-off parameter, which is used to weigh the label loss function and the consistency loss function.
We first pre-train the GVCLN with lview and laup, and second use the total loss function, Linta-- The specific process of GVCLN can be seen in Algorithm
Algorithm The GCVLN algorithm.
1: Input: X,A, A, epoch epoch prez 8
2: Output: p(1) and p(2),
3: Initialize: Model initialization.
4: X= =A'"X
5: for epoch E [1, epochmaxl do
6: if epocch < epoch pre then
f (1) + (2) + flview
7:
8:
Do optimization.
9:
else
httal Coup Clp Blview + Epeoudo
11:
Do optmizaation.
end
13:
end for
In this paper, we use renormalization filter (Li et al. 2019) preprocess features X Åm X. The filter strength is tuned by the filter parameter m. When the label rate is low, m is increased, and when the label rate is high, m reduced. By this operation, label efficient can achieved.
EXPERIMENT
In this section, datasets include three citation network of Cora, Citeseer and PubMed. We conducted experiments with different label rates and without verification set. The division of the experimental dataset is the same as Luan et al. (2019). Classification accuracy the test set is used as metric to evaluate the quality of model.
GVCLN to carry out related experiments and analyze the results in detail. The

---

Under review as a conference paper at ICLR 2021
Table 1: Datasets statistics.
Dataset
Vertices
Edges
Classes
Features
Cora
2708
5429
1433
Citeseer
3327
4732
3703
PubMed
19717
44338
3
500
4.1 DATASETS
We use three citation datasets: Cora, Citeseer and PubMed. which are widely used in graph prob- lems. The specific information of the three data sets can be found in Table
The Cora dataset contains 2708 nodes, which represent 2708 scientific documents and they can be divided into categories. The citation network has 5,429 connection references, so the resulting graph contains 5,429 edges. The node feature is represented by word vector containing 1433 independent and unique words, that is, the node feature is represented by 493-dimensional vector.
Cora:
The Citeseer dataset contains 3327 nodes, which can be divided into categories and each node represents one document. The citation network contains 4732 citation relationships. The feature of each document is represented by a word vector containing 3703 independent and unique words, that is, the feature of a node is represented by 3703-dimensional vector.
Citeseer:
The PubMed dataset contains 19717 scientific papers, which can be divided into three categories. The citation network contains 44,338 citation relationships. The feature of each paper is represented by a word vector containing 500 independent unique words, and the feature of a node is represented by 500-dimensional vector.
PubMed:
4.2 SETTING
GVCLN graph convolutional layer and graph attention layer to build two viewers. The dimen- sion of the hidden layer of Viewer is set to 16, the drop rate is set to 0.2, the first large layer uses three-head graph convolutional layer concatenate together, the second large layer uses only one graph convolutional layer; The dimension of the hidden layer of Viewer 2 is set to 8, the drop rate 0.6, the first large layer uses three-head graph attention layer to concatenate together, and the second large layer uses only one graph convolutional layer. Other specific parameters are given Appendix. Since the quality of the training set sampled each time different, when the sampled training set node is located in the cluster center, the training and testing results are better. When the sampled training set node is located at the classification boundary, will cause the training to be biased and the effect is poor, so the experimental results of this paper are the average results of ten tests. In the experiment, when the label rate is low, the filtering strength is increased and the number of pseudo labels is increased. As the label rate increases, the filtering strength and the number of pseudo labels are gradually reduced. At the same time, when the number of real label increases, the number of pseudo labels should be appropriately reduced.
4.3 RESULTS
In our paper, the framework of view-consistent learning is applied the problem semi-supervised graph node classification and combined with graph feature filtering and pseudo-label learning, so that the GVCLN performs outstandingly when dealing with low label rate problems and without
validation. The division of the experimental dataset is the same as Luan et al. (2019). Classification accuracy of the test set used as metric to evaluate the quality of model. The test results without validation can be seen in Table 2 Then, make specific analysis of the test results of the three datasets.
Our baseline is Stronger Multi-scale Deep Graph Convolutional Networks (Luan et al. 2019), which generalize spectral graph convolution and deep GCN block Krylov subspace forms and devise two architectures, both with the potential to be scaled deeper but each making of the multi-seu scale in- formation differently. Luan et al.] (2019) designed three sub-models: Linear Snowball, Snowball and 'Tuuncaedd Krylov, all of which can efficienlly elassify graph nodes. We used the same experimental

---

Under review as a conference paper at ICLR 2021
setup Luan et al. (2019), and conducted training and testing without validation set and with different label rates. Our method GVCLN can surpass Luan et al. (2019) under different label rates. In the appendix,
give list of parameter settings for GVCLN and Luan et al. (2019). We also compare against other methods, including label propagation using ParWalks (LP) (Wu et al. 2012),
Chebyshev networks (Cheby) (Defferrard et al. (0T6), Co-training (Li al. 2018). Self-training
(Li et al.
2018), Multi-stage self-supervised (M3S) training (Sun et al.) 2019b). graph convolutional
networks (GCN) (Kipf Welling, 2016), GCN with sparse virtual adversarial training (GCN-SVAT)
(Sun et al.
2019a) and GCN with dense virtual adversarial training (GCN-DVAN) (Sun et al. 2019a).
The second, third and fourth column of Table show the accuracy our proposed method GVCLN and other existing methods under different label rates. It can be seen that GVCLN can reach the highest level on three citation datasets. For Citeseer, when the labeling rate 0.5%, the GVCLN test accuracy rate is nearly 6.4% higher than the highest level of Truncated Krylov.
Table 2: Accuracy Without Validation
Dataset
Cora
Citeseer
PubMed
Label rate
0.5% 1%
2% 3%
4%
5%
0.5%
1%
2% 3% 4% 5%
0.03%
0.05%
LP
56.4
62.3
65.4 67.5
69.0
70.2
34.8
40.2
43.6 45.3 46.4
47.3
61.4
66.4
Cheby
38.0
52.0 62.4 70.8
74.1
77.6
31.7
42.8
59.9 66.2 68.3 69.3
40.4
47.3
Co-training
56.6 66.4 73.5 75.9 78.9
80.8
47.3
55.7
62.1 62.5 64.5 65.5
62.2
68.3
Self- Self-training
53.7 66.1 73.8 77.2 79.4
80.0
43.3
58.1
68.2 69.8 70.4 71.0
51.9
58.7
M3S
61.5 67.2 75.6 77.8 78.0
56.1
62.1
66.4 70.3 70.5
59.2
64.4
GCN
42.6 56.9 67.8 74.9 77.6 79.3
33.4
46.5
62.6 66.9 68.7 69.6
46.4
49.7
GCN-SVAT
43.6 53.9 71.4 75.6 78.3 78.5
47.0
52.4
65.8 68.6 69.5 70.7
52.1
56.9
GCN-DVAT
49.0 61.8 71.9 75.9 78.4 78.6
51.5
58.5 67.4 69.2 70.8 71.3
53.3
58.6
Linear Snowball
69.5 74.1 79.4 80.4 81.3 82.2
56.8
65.4 68.8 71.0 72.2 72.2
64.1
69.5
Snowball
67.2 73.5 78.5 80.0 81.5 81.8
56.4
65.0 69.5 71.1 72.3
72.8
62.9
68.3
Truncated Krylov
73.0
75.5
80.3
81.5 82.5
83.4
59.6
66.0
70,2
71.8
72.4
72.2
69.1
71.8
GVCLN
73.5 76.4
80.9
82.7
83.2
84.4
65.9
69.4
71.0
72.3
72.4
72.9
70.5
72.3
L,0
1.
0.3
133
303
280
400
Figure 2: accuracy and loss of Cora 0.5%
Figure 3: all loss of Cora 0.5%
Figure Cora with 5.6%
Figure 5: Citeseer with 3.6%
Figure 6: PubMed with 0.3%

---

Under review as a conference paper at ICLR 2021
Fig. 2 and Fig. show the accuracy curve and loss function curve of the Cora dataset when the label rate is 0.5%. It can be seen that adding a pseudo-label when the epoch of training 200 is helpful for the accuracy and loss function decline.
Fig. is t-SNE visualization diagram the output results when each category contains 20 labeled samples, that is, the label rate is 5.6%. It can be seen from the visualization that the classification effect still relatively good. The seven categories of the Cora data set finally appear to be clustered. Figure 5 and Figure far the t-SNE visualization diagrams of the Citeseer data and PubMed data set, respectively.
The renormalization filter with filter strength m=20 is used in the GVCLN model preprocess the node features, which plays an important role in achieving label efficiency. In the experiment, we performed t-SNE visualization analysis on the original node features and filtered node features respectively, and the visualization results can be seen in Figure and Figure 8 It can be clearly seen that all nodes in the original node feature visualization result are clustered together, and the node features after filtering are scattered and show certain signs of clustering.
Figure 7: t-SNE of Cora raw features
Figure 8: t-SNE of Cora features with filtering
CONCLUSIONS
We propose model, GVCLN, to solve the node classification problem in the case of low label rate. GVCLN adopts a dual-view structure to view-consistent learning. For the two viewers, use the graph convolutional layer and the graph attention layer, respectively, and finally pass through a non-linear graph convolutional layer. Because the graph convolutional layer of viewer is relatively simple, it can quickly learn obtain node representations, and then increases the representation ability of graph attention layer by supervised the consistency loss function in the learning. contrast, the graph attention layer of viewer 2 is more complicated. it can prevent the addition of pseudo-labels from making the GVCLN unstable, and prevent the pseudo-label errors amplify step by step. Thus, We propose a new-consistency learning method, and earry out relevant practices the task of graph node classification. Good results can be obtained the three citation datasets on all label rate and without validation.
We also found that when the label rate is very low, there may be only few labeled nodes. If these few labeled nodes are well represented (can support the entire dataset), the classification effect is particularly significant. On the contrary, when they are not enough to support the entire dataset, the classification effect will be worse. The direction of our future work will shift from a semi-supervised low label rate to unsupervised direction.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
James Atwood and Don Towsley. Diffusiion-convllutionna neural networks. In Advances in neural information processing systems, pp. 1993-2001, 2016.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Pro- ceedings of the eleventh annual conference on Computational learning theory, pp. 92-100, 1998.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6200 2013.
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey F. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: /409. 1259,
2014.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pp. 1884438522 2016.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guri and Ryan Adams. Convolutional networks on graphs for learning molecular 2015.
fingerprints. In Advances in neural information processing systems, pp.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 2527-8532 2018.
Michael Kampffmeyer, Arnt-Borre Salberg, and Robert Jenssen. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition workshops, pp. 1-9, 2016.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv:1609.02000 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, pp. 1097-1105, 2012.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning, arXiv preprint 1001:070000000 2018
Qimai Li, X. Wu, Hongmei Liu, Xiaotong Zhang, and Zhichao Guan. Label efficient semi- supervised learning via graph filtering. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9574-9583, 2019.
Sitao Luan, Mingde Zhao, Xiao-Wer Chang, and Doina Precup. Break the ceiling: Stronger multi- scale deep graph convolutional networks. In Advances in neural information processing systems, pp. 10945-10955, 2019.
Waseem Rawat and Zenghui Wang. Deep convolutional neural networks for image classification: A comprehensive review. Neural computation, 29(9)2352 2449, 2017.
Ke Sun, Zhouchen Lin, Hantao Guo, and Zhanxing Zhu. Virtual adversarial training graph convolutional networks in node classification. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 431-443. Springer, 2019a.
Ke Sun, Zhanxing Zhu, and Zhouchen Lin. Multi-stage self-supervised learning for graph convolu- tional networks. arXiv preprint arXiv: 1038, 2019b.
JM Vandenberg and BT Matthias. Clustering hypothesis of some high tesperrmper suprrconductoo..r..... Science, pp. 194-196, 1977.

---

Under review as a conference paper at ICLR 2021
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10003, 2017.
Xiao-Ming Wu, Zhenguo Li, Anthony M So, John Wright, and Shih-Fu Chang. Learning with partially absorbing random walks. In Advances in neural information processing systems, pp. 3077-3085, 2012.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems, pp. 5165-5175 2018.
Muhan Zhang. Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Thirty-Second AAAI Conference Artificial Intelligence, 2018a.
Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: Algo- rithms, applications and open challenges. In International Conference Computational Social Networks, pp. 79-91. Springer, 2018b.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006. 04131, 2020.
10


Reviews:
[{"review_id": "wXZ7TGmPGLR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to address the limited training data problem by exploiting the consistency between different views of the graph data. However, important details and justification are missing. The major problems are:\n\n(1)\tThe proposed model lacks detailed explanation and justification. How do you generate the view for the graph from its features? How and why do you obtain three head representations using graph convolution or graph attention? Why using convolution and attention for the two views respectively? How do you do dropout? Why using the same graph convolution layer later for the two views? After seven lines of introduction of the model, the paper is focused on training, and leave all the above questions behind.\n(2)\tFor training, how do you judge ˋˋhigh confidence predictions’’ for generating pseudo-labels? What do you mean by ˋˋthe same prediction representations’’, ˋˋverification set’’ and ˋˋlabel rates’’? Moreover, with limited training data, how do you obtain ˋˋwell trained’’ two-view networks? What is the ˋˋstop condition’’, simply max epochs?\n(3)\tThe authors give some settings of parameters and say that the other parameters are specified in the appendix. However, after references there is no appendix.\n\nTo summarize, without justification, the proposed model is not convincing. Without details of the model and implementation, this work is difficult to reproduce.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TiTlgrvK1lN", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Advantage:\n\nThe paper concentrated on an important perspective of graph learning: to utilize a small number of labels for large-scale graph learning. The framework is well demonstrated and the paper is easy to follow. \n\nWeakness:\n\n1. Novelty: My main concern of the paper is about the paper's novelty. There are already some works having similar multi-view contrastive learning frameworks, such as [1]. `Besides, The learning loss of the proposed method is similar to the self-training loss in SimCLR v2 [2].\n\n2. Experiment: The experiment results are not sufficient. The author claimed the proposed method is efficient for graph classification on large training datasets. But the experiment is conducted on three small graph datasets. Results on more larger datasets [3] are expected to support the effectiveness of the proposed method.\n\nReference:\n[1] Contrastive Multi-View Representation Learning on Graphs, ICML 2020\n[2] Big Self-Supervised Models are Strong Semi-Supervised Learners, NeurIPS 2020\n[3] Open Graph Benchmark: Datasets for Machine Learning on Graphs", "extra_scores": {}, "extra_text": {}}, {"review_id": "H0X9-xH8ZG1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nIn this paper, a graph view-consistent learning framework (GVCLN) is proposed. Specifically, two view learners are used to give predictions for the input. Then, a consistency loss is employed to force the two viewers giving the same predictions. Moreover, a co-training scheme is proposed to alleviate the label sparsity problem.\n\n[Pros]\n+ This paper is easy to follow.\n\n[Cons]\n- The novelty of this work is limited. It seems that this work is a simple combination of [1] and [2], with slight modification. Also, the authors are suggested to include more baselines, especially augmentation-based methods, e.g., [3].\n- Three studied datasets are of small scales, which are well know to have unstable results.\n- It is not clear how to select high-confidence pseudo labels. More experiments, e.g., parameter sensitivity analysis wrt the confidence threshold, ablation studies of the two loss, are needed.\n\n[Evaluation]\nOverall this paper presents a simple yet effective framework to semi-supervised node classification. However, the novelty of this work is limited and more experiments are necessary.\n\n[Ref]\n[1] A Simple Framework for Contrastive Learning of Visual Representations, in ICML, 2020\n\n[2] Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes, in AAAI, 2020\n\n[3] NodeAug: Semi-Supervised Node Classification with Data Augmentation, in KDD, 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DvMEEYMXToa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a view-consistent framework to address the issues of expensive labels. In particular, this work first uses graph neural networks and graph attention networks to construct two different latent features of the same data. Then, it uses the same classification neural networks to produce the node classification outcomes. Finally, it uses the classification outcome to construct a so-called \"view loss\". In addition, it uses an incremental strategy to gradually included pseudo labels until some termination conditions are satisfied. \n\nOverall, the paper is easy to understood. However, I think the paper can be improved in each sections:\n[Introduction & Related work]\nThe authors can better organize their presentation on the development and understanding of Graph Neural Networks. At the current stage, these content does not seem to connect to the current development of GNN. \n\n[Model architecture]\n3.1\n1, can the authors explain the reason of using three-head representation? Also, why do the authors use the same non-linear graph convolution layers? Is it because the feature is different already? Can the authors specify the detail settings on this graph convolution layers? I did not find it in other places?\n\n3.2\n2, I can roughly understand the reason of introducing the contrastive learning and co-training. However, maybe the authors should put of the content in the related work part and emphasise the difference of view-consistent algorithm from these two methods. Plus, I did not find experiments that uses the data augmentation method (instead of the view-consistent method). I can see there is contrast learning comparison, however, the other settings of constrast learning may be different?\n3, the inclusion of pseudo labels are not well explained in this section. I was expecting to see more systematic analysis on this procedure, however, the current version can not fully convince me on this procedure. \n4, small issues: view 1 and viewer 1 are both used in the paper and they should be consistent. Eq. (5) and Eq. (6) can be well formatted to save more space for presentation. \n\n[Experiments]\n5, I am expecting to see the implementation code for at least the neural network specification in the main paper or supplementary material. However, they are missing. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6SBGi0kbn94", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper adopts a multi-view learning approach for graph representation learning where some labels are assumed to be available. It uses graph convolution network (GCN) and graph attention network (GAT) to create two different views of the same graph and then define a loss function to force the output due to the two views to be consistent. The low label rate scenario is considered and pseudo labels are created to define an additional loss function to better enforce consistency. Three datasets are used for performance evaluation.\n\nPros:\n- The problem addressed is an important one.\n- The problem formulation is a reasonable one.\n- The paper is clearly presented in general.\n\nCons:\n- The view-consistency idea is good but not particularly new. The two graph “views” are based on existing graph embedding methods. So, I consider the originality of this work is limited.\n- Only incremental improvement in performance is demonstrated in the experimental results.\n- The graphs being evaluated are not particularly large.\n\nComments:\n- In Eq 7, the two terms for supervised loss l_sup^(1) and l_sup^(2) are not clearly defined.\n\nQn:\n- In the conclusion, there is some discussion about how pseudo labels contribute. It will be interesting to see how crucial the pseudo label term contributes to the overall the performance. \n- How will the proposed method be compared with some contrastive learning in terms of performance?\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors consider view-consistency when learning graph neural networks. However, as mentioned by the reviewers, the novelty of the proposed method is limited and the rationality of the implementation is not convincing. More deep discussions about related papers and analytic experiments are required to support this work. Additionally, I have concerns about the scalability of the method --- whether it can deal with more than two views and how it will perform are not studied in this work. I tend to reject it based on its current status. 
Decision: Reject