Graph Neural Networks as Multi-View Learning
Authors: Haoyu Han
Xiaorui Liu
Haitao Mao
MohamadAli Torkamani
Feng Shi
Victor Lee
Jiliang Tang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Neural Networks
Alternating Optimization
Semi-Supervised Learning
Multi-View Learning

Abstract: Graph Neural Networks (GNNs) have demonstrated powerful representation capability in semi-supervised node classification. In this task, there are often three types of information  -- graph structure, node features, and node labels. Existing GNNs usually leverage both node features and graph structure by feature transformation and aggregation, following end-to-end training via node labels. In this paper, we change our perspective by considering these three types of information as three views of nodes. This perspective motivates us to design a new GNN framework as multi-view learning which enables alternating optimization training instead of end-to-end training,  resulting in significantly improved computation and memory efficiency. Extensive experiments with different settings demonstrate the effectiveness and efficiency of the proposed method. 


Under review as a conference paper at ICLR 2023
GRAPH NEURAL NETWORKS AS MULTI-VIEW LEARN- ING
Anonymous authors
Paper under houble-blind review
ABSTRACT
Graph Neural Networks (GNNs) have demonstrated powerful representation ca- pability in semi-supervised node classification. In this task, there are often three types of information graph structure, node features, and node labels. Existing GNNs usually leverage both node features and graph structure by feature trans- formation and aggregation, following end-to-end training via node labels. In this paper, we change our perspective by considering these three types of information as three views of nodes. This perspective motivates us to design new GNN framework as multi-view learning which enables alternating optimization train- ing instead of end-to-end training, resulting in significantly improved computation and memory efficiency. Extensive experiments with different settings demonstrate the effectiveness and efficiency of the proposed method.
INTRODUCTION
Graph is a fundamental data structure that denotes pairwise relationships between entities in a wide variety of domains (Wu all 2019;; Ma & Tang, 202I). Semi-supervised node classification is one of the most crucial tasks on graphs. Given graph structure, node features, and labels on a part of nodes, this task aims predict labels of the remaining nodes. In recent years, Graph Neural Networks (GNNs) have proven to be powerful in semi semi-supervised node classification (Gilmer et all 2017: Kipl Welling 201; Nelcckovicee et 20L7). Existing GNN models provide different archi- tectures to leverage both graph structure and node features. Coupled GNNs, such as GCN (Kipf & Welling 20G) and GAT (Velickovic et all 2017). couple feature transformation and propagation to combine node feature and graph structure in each layer. Decoupled GNNs, such as APPNP (Kliceera cLall M)).S first transform node features and then propagate the transformed features with graph structure for multiple steps. Mcanwhile, there are GNN models such Graph-MLP (Hu et all 2021) that extract graph structure as regularization when integrating with node features. Nevertheless, the majority of aforementioned GNNs utilize node labels via the loss function for end-to-end training.
In essence, existing GNNs have exploited three types of information to facilitate semi- cemi-supervised node classification. This understanding motivates three types of information as three views of nodes. Then we can treat the design of GNN models as multi-view learning. The advantages of this new perspective are multi-fold. First, we can follow key steps in multi-view learning methods to design GNNs by investigating (1) how to capture node information from each view and (2) how fuse information from three views. Such superiority offers us tremendous flexibility to develop new GNN models. Second, multi-view learning has been extensively studied (Xu all 2013) and there a large body of literature that can open new doors for to advance GNN models.
change our perspective by considering these
To demonstrate the potential of this new perspective, following traditional multi-view learning method (Xia et all 2010), we introduce shared latent variable to explore these three views si- multancously in a multi-view learning framework for graph neural networks (MULTIVIE+4GNN) The proposed framework MULTIVIEW4GNN can be conveniently optimized in an altemnatnng way, which remarkably alleviates the computational and memory inefficiency issues of the end-to-end GNNs. Extensive experiments under different settings demonstrate that MULTIVIEW achieve comparable or even better performance than the end-to-end trained GNNs especially when the labeling rate is low, but it has significantly better computation and memory efficiency.
can

---

Under review as a conference paper at ICLR 2023
2 THE PROPOSED FRAMEWORK
We use bold upper-case letters such as X to denote matrices. X denotes its i-th row and X indicates the i-th row and j-th column element. We use bold lower-case letters such to denote vectors. The Frobenius norm and trace of a matrix X are defined as || = / X and
(x(X) = Σ. x Let 9 = (V.&) be a graph, where V is the node set and is the edge set. N denotes the neighborhood node set for node Us. The graph can be represented by an adjacency matrix A E Ruxn, where A.; > O indices that there exists an edge between nodes U; and vj in G, or otherwise A, = 0. Let D diag(ds (d1, .2.....dd, d,)) be the degree matrix, where d; = E, Aij the degree of node v4- The graph Laplacian matrix defined as L D A. We define the normalized adjacency matrix as Á D AD- & and the normalized Laplacian matrix as L I A. Furthermore, suppose that each node is associated with a d-dimensional feature x and we use X = X2.. x,, E Rnxd to denote the feature matrix.
In this work, we focus on the node classification task on graphs. Given a graph G = {A, X) and a partial set of labels 1/2 = fy,, ............. Yi one-hot vector with C classes, our goal is to predict labels of unlabeled nodes. The labels of graph G can also be represented as label matrix Y E g2xc, where Y; y, if o; E V, and Y, 0 if Uy E Vu. The subscript U and L denote the sets of unlabeled and labeled nodes, respectively.
for node set V/ = {v1. .,,,..., W1 where yi E RC is
2.1 MULTI-VIEW LEARNING FOR GNNS
For the node classification task, take new perspective that considers node feature X, graph structure A, and node label Y as three views for nodes, and model graph neural networks as multi- view learning. In particular, we need to jointly model each view and integrate three views. To achieve this goal, we introduce a latent variable F inspired by a traditional multi-view learning method ef all 201O). Then the loss function can be written as:
arg min = x Dx DX, F) + (A(AFF) + DY Fr.,
where F is the introduced latent variable shared by three views, functions to explore node feature, graph structure, and node label, respectively. These functions can contain parameters which we denote them as O Hyper-parameter: x and X2 are introduced to balance the contributions from these three views. One major advantage of the multi-view learning perspective is that enables immense flexibility to design GNN models. Specifically, based on (I), there are numerous designs for Dx (:())), DA((.-) and Dy Examples are shown below:
DA(:.) and Dy (.-) are
Dx is to map node features X to F. In reality, we can first transform X before mapping. Thus, feature transformation methods can be applied including traditional methods such as PCA (Collins el all 2001: Shent 2009) and SVD (Godunov et all 2021), and deep methods such as, MLP and self-attention (Vaswani et all 2017). We also have various choices of the map- ping functions such as Multi-Dimensional Scaling (MDS) (Hout et all 2013) which preserves the pairwise distance between X and F and any distance measurements.
DA aims to impose constraints on the latent variable F with the graph structure. Traditional graph regularization techniques can also be employed. For instance, the Laplacian regulariza- tion (Y' ct all 2016) is to guide a node i's feature F, to be similar to its neighbors; Locally Linear Embedding (LLE) (Roweis Saul, 2010) to force the F, be reconstructed from its neighbors. Morcover, modern deep graph learning methods can be applied, such as graph em- bedding methods (Perozzi et all, 2014: Grover Leskoved, 2016) and Graph Contrastive Learn- ing (Zhu 2020, Hu et all, 2021). which implicitly encodes node similarity and dissimilarity. Dy establishes the connection between the latent variable L and the ground truth node label Yr for labeled nodes. It can be any classification loss function, such as the Mean Square Error and Cross Entropy Loss.
In this work, we set the dimensions of the latent variable F as R which can be considered as a soft pseudo-label matrix. Then the following designs are chosen for these functions: (i) for Dx, we use MLP with parameter θ to encode the features of node as MLP(X,; adopt the Euclidean distance to map F, as [|MLP(X;; θ) - F F,ll:: (ii) for DA. Laplacian smooth- ness is imposed to constrain the distance between one node's pseudo labels F; and its neighbors as
and then
2

---

Under review as a conference paper at ICLR 2023
Des,, [F./vd - F1y/dd and (iii) for Dy, adopt Mean Square Loss |E,
Y:ll to
constraint the pseudo label of labeled node close its ground truth. These designs lead to multi-view learning framework for graph neural networks (MULTIVIEW4GNN)) Its loss function can be written in the matrix form as:
L = A1 ||MPP(X
FI
(r LF +
F
DA
Dy
where the first term maps node features into the label space, the second term indicates the pseudo labels should be smooth over the graph, and the last term constrains that the pseudo labels should be close to the ground-truth labels for labeled nodes.
Remark. There recent works (Zhu el all MO2I; Ma 202T: Yang et al, 202T) that aim provide a unified optimization framework for understanding the message passing mechanism of dif- ferent GNNs and designing new graph filter layers. However, they only focus on the forward process without taking the backward learning process into consideration. and they still follows the existing GNN architecture with end-to-end training. In this work, we do not aim understand the message passing and design new GNN layers based on existing architectures. Instead, MULTIVIEW4GNN is a new graph deep learning framework as multi-view learning.
2.2 AN ALTERNATING OPTIMIZATION METHOD FOR MULTIVIEW4GNN
difficult to find the optimal solution of the loss function (L) for both F and O simultaneously due to the coupling between the latent variable and model parameters θ The alternating opti- mization (Bezdek & Hathaway, 2002) based iterative algorithm can be natural solution for this challenge. Specifically, for each iteration, we first fix the model parameters @ and update the shared latent variable on all three views. Then, we fix F and update the parameters O which effective in exploring the complementary characteristics of the three views. These two steps alternate until convergence. Next, we show the alternating optimization algorithm in detail.
Update F. Fixing MLP, descent method. The gradient of L with respect to F (i.e., Fu and FL)
minimize L with respect to the latent variable F using the gradient
dc
2 A1 MMP((x)))) + (LF), + A2 Yu)).
(3)
FF
OL
OFu = 2 (1 MLP(Xx)) + (LF)
(4)
The gradient descent update of F with step sizes 7Jr. and rju is:
F++ = F - -27 (AL - MLP(XL)) + (LF*)) + A2 Y()) 221 ((AF)) + M)LLP(X,) + X2YY) + (1- 271 + A2 + 1) F)
(5)
Fh+1 11 = F - 2nu (AI - -LP((x)) + (
= =2m (AFF)) + X)MLP(X)) + (1- 2nu (A1 1) F
(6)
According to the smoothness and strong convexity of the problem, we set ?) = NJU to ensure the decrease of loss value L (Nesteroy et all 2018), and the update becomes:
2(1
+01++>=+1)
A2
Fk+1
(AF*)
\1 + de + MLP(XL) + t1
Y
Lz,
(7)
d1 + A2 +
+
Ao
+
1
d
A
k++
(A J
d2 +1+ MLP(Xu)
F*
(8)
Xi + do
A2
Update O Fixing F we can minimize the loss function L with respect to MLP parameters:
arg min MMLP(X;)) 12
3

---

Under review as a conference paper at ICLR 2023
which equals training the MLP with the soft pseudo labels via the Mean Square Error Loss. Besides, we can also apply the Cross-Entropy Loss, and the details are in Appendix A..
Alternating Optimization and Scalability. The multi-view learning perspective allows derive the alternating optimization solution which provides better flexibility in training than the end-to-end training in existing GNNs. The alternating optimization solution is a highly efficient and scalable training strategy, resulting in significantly improved computation and memory efficiency. Specifi- cally, variable F and MLP model parameters θ can be optimized separately. We can update F once and then train MLP multiple times. Meanwhile, there is no gradient backpropagation through the feature aggregation process so the aggregation steps do not need to store the activation and gradient values, which saves a sigmificant amount of memory and computation. Moreover, due to the alternat- ing updating scheme, we can use stochastic optimization to update F and train MLP by sampling the graph structure and node features. This can further improve the memory and computation efficiency as proved theoretically and empirically in stochastic optimization (Lan, 2020). In particular, only first-order neighbors are sampled in each update step of F, which avoids the neighborhood explosion problem in training large-scale GNNs (Hamillon el all 2017: Fey et all 2021).
2.3 UNDERSTANDINGS OF MULTIVIEW4GNN
Another important advantage of alternating optimization is that it provides helpful insights under- stand MULTIVIEW 4GNN. In particular, based on the updating rules of F and θ we can naturally draw the following understandings on MULTIVIEW4GNI
Understanding 1: Updating F is feature-enhanced Label Propagation. Label Propagation (LP) (Zhou /hhoueetall 20103) is a well- graph cemi-supervised learning method based on the label
smoothing assumption that connected nodes are likely to have the same label. LP can be written in
an iteration form: p(++1)- (A) + (1 YYY, where ( = Y. k is the propagation step, and is a pprrmparreeeer. Comparing LP with our update rule for F in Eq. ( and Eq. (8), we can find that MLP(X) is involved in the propagation where MLP(X) can be regarded as labely generated
by node features. other words, our update rule for F takes advantage of node features, graph structure and labels while LP only uses graph structure and labels.
Understanding 2: Updating is a pseudo-labeling approach. Pseudo-labeling (Lec all 20
Arazo et all ( is popular method in semi-supervised learning that uses small set of labeled data along with a large amount of unlabeled data to improve model performance. It usually generates pseudo labels for the unlabeled data and trains the deep models using both the true labels and pseudo labels with different weights. From this perspective, MULTIVIEW4GNN uses the pseudo labels F to train θ such that it can take advantage of both labeled and unlabeled nodes.
Stage 2: Update MLP
Input graph
MLP
Predictions
Diffusion
Propagate Labels with MLP
Stage 1: Update Pseudo-Labels
Figure I: An overview of the proposed MULTIVIEW4GNN method, where grey color represents unlabeled nodes, and each node is associated with input features.
2.4 IMPLEMENTATION DETAILS OF MULTIVIEW4GNN
In this subsection, detail the implementation of MULTIVIEW4GNN, As shown in Figure M. we first preprocess the node feature through a diffusion, then alternatively update pscudo label and MLP while taking into account the weight of pseudo labels and the class balancing problem, and finally get the prediction of unlabeled nodes. Next, we describe each step in detail.
Preprocessing. From Understanding 1, initialized MLP is needed. In real graphs, labeled data are usually scarce so that it is challenging to get good initialization of MLP with a small number of labels. Therefore, first diffuse the
MLP to enhance the label propagation, so good
4

---

Under review as a conference paper at ICLR 2023
original node features with its neighbors to get smoothing and enhanced features. The new features are obtained from X' = =LP(X, at). Then, we train MLP only using the labeled data for a few epochs to get an initialization, similar to pseudo-labeling methods et 2019: Lee at, 2013).
Update We initial F = Y. Then we update F for labeled nodes and unlabeled nodes by Eq. ( and Eq. (L) respectively. Since F acts as pseudo labels when training the MLP,, normalize F to be the distribution of classes by using the softmax function with temperature after the update:
F iy
exp(Fu/T) Σes E exp(Fa/r7)
where is hyperparameter control the smoothness of pseudo labels.
Pseudo-Label certainty and class balancing. Directly using all pseudo labels to train MLP is not appropriate due the following reasons. First, not all pseudo lahels have the same certainty. Second, pseudo-labels may not be balanced over classes, which will impede learning. To address the first issue, we assign confidence weight to each pscudo-label. According information theory, entropy can be used to quantify a distribution's 's uncertainty, so we define the weight for unlabeled nodes as wi 1 ((F,)
ز== F,y log Fij is the entropy of the pseudo label F,, To deal with the class imbalance problem, we adopt simple method that chooses the same number of unlabeled nodes for each class with the highest weight to train MLP.
[C) where w, E [0. 1] and (F,,)
Update MLP We train MLP using both labeled nodes L and high confidence unlabeled nodes set Up. The loss function can be written as follows:
MMPP , = Σ (MMLP(XX O),
O), + Σny, my (M). jEU,
(10)
IEL
where E(MLP(X F,) I|MLP(X; F2l12 is a MSE loss and is the parameters of MLP.
Prediction. The inference of our method is based on the pscudo labels F, and the predicted class for the unlabeled node can be obtained as c; = arg max, Fij. The overall algorithm, implementation details and code of MULTIVIEW4GNN are shown in Appendix R
2.5 COMPLEXITY ANALYSIS
We provide time and memory complexity analyses for MULTIVIEW4GNN and the following rep- resentative GNNs: GCN (Kipf & Welling 2016), SGC (Wu at 2m19a), and APPNP (Kliepeaa al 2018). Suppose that p is the number of propagation layers, n the number of nodes, m is the number of edges, and c is the number of classes. For simplicity, we assume that the hidden feature dimension is fixed for all transformation layers, and we have e < d in most cases; all feature transformations are updated epochs; Besides, the adjacent matrix A is sparse matrix, and both forward and backward propagation have the same cost. Following (Lietall 20221, we only analyze the inherent differences across models by assuming that all models have the same transfor- mation layers, allowing us to disregard the time required for feature transformation and the memory footprint of network parameters. The time and memory complexities are summarized in Table MI
Time complexity. We first analyze the time complex- ity of feature aggregation. The feature aggregation can be implemented as a sparse-dense matrix multiplication with cost (mdd if the feature has d dimensions. There- fore, the time complexity of training p-layer GCN for t epochs is (2tpmdd) with the gradient backpropagation. For SGC, we only need steps of feature propagation, so the time complexity O(pmd). For APPNP, the gradient also needs backpropagate through p lay- ers, but the feature dimension is c, resulting in the time complexity of (22ppme). Regarding MUL- TIVIEW4GNN, as the model are optimized in an alternating way, there is no need to do both feature transformation and aggregation in each epoch. Rather,
Table Comparison of time and mem- ory complexities.
Method
Time (12pmmd)
Memory
GCN
O'nd + pnd)
SGC
O(pmd)
O(nd)
APPNP
((2tpme)
O(nd + pnc)
MULTIVEEWAGNN
(kkme))
O + nc)
can propagate the pseudo labels only for times during the whole training process. As a result, the time complexity of MULTIVIEW4GNN is Ok<pm)). In practice, choosing from [2, 5] can achieve very promising performance, while t needs to be 500 or 1,000 for other models to converge.
Memory complexity. It requires ((dd) memory for storing node features. For the end-to-end train- ing models, we need to store the intermediate state at each layer for gradient calculation. Specifically, for GCN, we need to store the hidden state for p layers, the memory complexity is O((p+ + 1)nd).
5

---

Under review as a conference paper at ICLR 2023
SGC only needs to store the propagated feature (nd) omit the memory of network param- eters. Similarly, APPNP has the memory complexity of Ond + pnc). For MULTIVIEW4GNN, does not need to store the gradients at each propagation layer. Instead, MULTIVIEW4GNN needs to hold the pseudo label F. So the memory complexity of MULTIVIEWAGNN is O + ne).
If we omit the difference in the dimension of the propagation features (d = c), the time and memory of GCN and APPNP are the same, as they require feature propagation in each epoch. We call the methods that need propagation every epoch Persistent propagation methods. Similarly, the methods that only need to propagate once, such as SGC, SIGN (Rossi et all, 2020), and C&S (Huang ata 2020) have the same time and memory complexity, namely One-time propagation methods. MULTIVIEW 4GNN is Lazy propagation method since the features are propagated & times during training with & being small number. Thus, MUL.TIVEW44GN can be seen as balance between these two groups of methods.
3 EXPERIMENT
In this section, we verify the effectiveness of the proposed method, MULTIVIEW-4GN,, through the semi-supervised node classification tasks. In particular, try answer the following questions:
RQ1: How does MULTIVIEW4GNN perform when compared other models?
RQ2: Is MULTIVIEW4GNN more efficient than sfate-of-the-ar GNNs? How do different components affect MULTIVIEW4GNN?
RQ3:
3.1
EXPERIMENTAL SETTINGS
Datasets. For the transductive semi-supervised node classification task, we choose nine common used datasets including three citation datasets, i.e., Cora, Citeseer and Pubmed etal, (008), two coauthors datasets, i.e., CS and Physics, two Amazon datasets, i.e., Computers and Photo (Shehur all 2018), and two OGB datasets, i.e., ogbn-arxiv and ogbn-products (Hu el all M02D). For the inductive node classification task, we use Reddit and Flikcr datasets (Zeng et al, 2019). More details about these datasets are shown in Appendix D
Following (1 (Lu 202I), we use 10 random data splits for the three citation datasets, and we run the experiments 3 times for each split. We report the average performance and standard deviation. Besides, we also test multiple labeling rates, i.c., 5, 10, 20, 60, 30% and 60% labeled nodes per class, to get a comprehensive comparison. For other datasets, we use the fixed split and run 10 times.
Baselines. We compare the proposed MULTIVIEW4GNN with three groups of methods: (i) Per- sistent propagation methods, i.e., GCN (Kıpí & Welling, 2016), GAT (Vehεčovii et all 2017) and APPNP (Klicpera et all, 2018); (ii) One-time propagation methods, i.e., SGC (Wu et all 2019a), SIGN Rossiet (2020), and C&S (Huang al, 202();; and (iii) Non-GNN methods including MI.P and Label Propagation(Zhon all 2013).. We report test accuracy results of all models selected by the highest validation accuracy. Parameter settings for all methods are illustrated in Appendix E
3.2 PERFORMANCE COMPARISON ON BENCHMARK DATASETS
Transductive Node Classification. The transductive node classification results partially shown in Table 21 We leave results on more datasets and methods in Appendix D due to the space limitation. From these results,
make the following observations:
MULTIVIEW4GNN consistently outperforms other models at low label rates on all datasets. For example, in Cora and CiteSeer with label rate 5, our method can gain 1.2 % and 5.6 % relative improvement compared to the best baselines. This because the pseudo labels generated by our framework are helpful for training models when there are few labels available. When the label rate is high, our method is also comparable to the best results. addition, MULTIVIEW 4GNN is alternately optimized but not end-to-end trained, which suggests that end-to-end training could not be necessary for node semi-supervised classification.
MULTIVIEW4GNY performs the best OGB datasets. For example, in egbn-products it ob- tains 7.86% and 2.63% relative improvement compared to APPNP and SIGN, respectively. Compared with the One-time propagation methods, Persistent propagation methods usually per- form better when the labeling rate is low. In addition, the label propagation outperforms MLP in most cases, indicating the rationality of our proposed feature-enhanced label propagation.

---

Under review as a conference paper at ICLR 2023
Table 2: Transductive node classification accuracy (%) benchmark datasets.
Method
Persisioui propagation
michods
propagation
Dataser
Label
GCN
GAT
APPNP
SGC
SIGN
C&S
MUSTIEEW4GNN
Cora
70.68
2.17
12.97
2.23
75.86
2.34
70.06
:: 2.56
76.30
1.42
78.03
1.17
80.29 1: 1.00
76.28 1 1.22
76.25 1: 1.26
71.04 = 3.30
80.66 1.92
20
79.41
1.30
81.39
1.41
82.34
= 0.67
80.30 + 1.72
79.71 + 1.11
77.96
2213
82.66 +: 0.98
84.30
85.11
1.10
+
30%
86.87 = 1.35
87.24
1.19
87.77 = 1.13
86.97 - 0.90
87.17 + 1.28
87.60
1.12
87.70 1.19
88.49
88.21
CheNeer
61.27 3.85
62.60
5.34
63.92 + 1.39
60.21 + 3.48
57.44 + 3.71
50.39
4.70
67.48 2,90
66.28
2.14
66.81
2.10
67.57 1: 2.05
65.23 1: 2.36
63.87 1: 3.09
58.96
1.75
69.39 2.59
69.00
1.67
69.66
60
72.52
1.74
73.10
1.20
73.50 1: 1.54
71.43 4: 1.26
72.63 1: 1.39
1.79 71.21
72.84 1.65
30%
75.20
0.85
75.01
0.99 76.70
75.71
0.71
75.09
75.09
60%
76.88
1.78
1.81
77.42
1.47
76.66 + 1.59
76.41 + 1.96
76.34
1.37
77.00
1.67
Pubmed
69.76
6.46
70,42
5,36
72.68 1: 5.68
68.55 1 6.88
66.51 1 6.15
65.3
73.51 4,80
72.79
3.58
73.35 + 3.83
75.53 + 3.85
72.80
3.55
71.32 - 3.70
72.51
3.75
75.55
20
71.43
1.93
77.43
2,66
2.11 78.93
76.48 1: 2.84
76.39 1: 2.65
2.49 75.34
79.16 2.26
30%
88.07
0.29
86.51
+ 0.41
87.56 0.39
86
0.43
89.09 + 0.33
88.44
0.40
88.24 0.36
60%
88.48
0.46
86.52
0.56
87.56
0.52
86.63
0.38
89.55
0.56
88.53
88.83
0.55
ogbn-arxiv
54%
71.91
= 0.13
7192
71.61
0.30
68.74
0.12
7133 0.11
71.03
0.15
72.76
0.17
ogbn-proddcts
8%
75.70
0.19
OOM
76.62 22 0.13
74.29 3 0.12
80.5210.16
77,11 0.06
82.641
0.21
The standard deviation of all models is not small across different data splits, especially when the label rate is very low. It demonstrates that splits can significantly affect a model's performance. A similar finding is also observed in the PyTorch-Geomenee paper (Fey & Lenssen, 2019).
Inductive Node Classification. For inductive node classification, only training nodes be ob- served in the graph during training, and all nodes can be used during the inference (Zeng et alg 2019). For MULTIVIEW4GNN inference for the unlabeled node using the feature-enhancee label propagation in Eq (α) and Eq (8) As shown in Table the COLTIVIIEW4GN outperforms other baselines on the inductive node clas- sification task. The only difference between MULTIVIEW 4GNN and MLP is the Seature-enhance label propagation, and the performance improvement can demonstrate its superiority.
first train MLP with the training nodes' features and then do
Table 3: Inductive node classification accuracy (%).
Method
MLP
GCN
APPNP
SGC
C&S
MULTIVIEWHGNN
Reddit
62.84
93.30
94.11
93.85
95.30
95.74
Flickr
37.87
49.20
49.40
50.58
51.46
52.29
3.3 EFFICIENCY COMPARISON
In this subsection, we compare the efficiency of our MULTIVIEW4GNN with other baselines, based on two large datasets, i.c.. ogbn-arxiv and ogbn-products. To make fair comparison, we choose the identical feature transformation layers for each method as there are learnable parameters in feature propagation layers. Besides, we train model parameters with the same iterations in each method, i.e., 500 epochs for ogbn-arxiv and 1,000 epochs for ogbn-products. All the experiments are conducted on the same machine with a NVIDIA RTX A6000 GPU (48 GB memory).
For MULTIVIEW4GNY we can update F with different frequency in training, i.e., 1, 2, 3, 4. 5, and "Full". "Full" means we update both F and MLP in each epoch. For MULTIVIEW 44NN- we only update the F for & times during the training procedure. The overall results shown in Table HL
Table Efficiency comparison of different methods.
Dataset
ogbn-arxiv
ogbn ogbn-products
Method
ACC(%)
Time (s)
Memory (GB)
ACC (%)
Time (s)
Memory (GB)
MLP
55.68
12,01
2,68
61.17
214
21.18
SGC
66.92
12.06
2.71
74.29
215
21.85
SIGN
71.95
24.89
4.67
80.52
492
43.17
GCN
71.91
24.71
3.33
75.70
1,284
38.36
APPNP
71.61
33.70
3.20
76.62
1,913
29.15
MULTIVIEWAGNN-FF Full
72.76
22.28
2,81
81.83
901
24.49
MULTIVIEWAGNN-I
70.09
12.86
2.81
80.03
218
24.49
MULTIVIEWAGNN-2
72.32
12.89
2.81
80.34
219
24.49
MULTIVIEW4GNN-3
72.60
12.92
2.81
81.00
220
24.49
MULTIVIEW4GNN-
72.71
12,95
81.89
221
24.49
MULTIVIEWAGNN-S
72.70
12.98
2.81
82.64
222
24.49
7

---

Under review as a conference paper at ICLR 2023
Training Time. For the Persistent propagation methods including GCN, APPNP and MULTI- VIEW4GNN-Full, the training time is longer than other methods that do not need to propagate every epoch. Both APPNP and MULTIVIEA4GNN-FU need to propagate ten layers every epoch and GCN needs to propagate three layers. However, the training time of MULTIVIEW4GNN- is nearly half of APPNP and still less than GCN, which matches our time complexity analysis in Section R as there is no gradient backpropagate through propagation layers. Compared with the One-time propagation methods like SGC, MULTIVIEW4GNT with only a few update steps, such as AUUTIVIEW4GNN- can achieve better accuracy with a minor increase in training time. For example, the whole training time of MULTIVIEW but it has 8.63% and 11 11224 % relative performance improvements in ogbn-arxiv and ogbn-produets datasets, respectively. Meanwhile, we can observe that MULTIVIEW formance with MUL.TIVIEW4NN--. train the model simultancously for each epoch. This also suggests that the end-to-end training with propagation might not be necessary.
only 0.92s and 7s longer than SGC,
has very similar per- which suggests that there is no need to do propagation and
Memory Cost. Compared with the Persistent propagation methods, MULTIVIEWAGNN requires less memory with no requirement store the hidden states in the propagation layers. Thus, MUL- TIVIEW4GNN can keep a constant memory even with more propagation layers. Compared with MLP and SGC, MULTIVEEW4GN only slightly increases memory as it needs to store the pseudo label matrix as analyzed in Section P In addition, MULTTIIEEAGNN is suitable for large-scale datasets that cannot fit in the GPU memory. First, easy to propagate the pseudo labels using CPUs as the label dimension is often lower than that of features. Then, MULTIVIEW4GNN is amenable to sampling training with mini-batch, which would significantly reduce the memory cost as discussed in Section 2.21 The only additional cost beyond One-time propagation methods is that we need to transfer the result of MLP from GPU to CPU do feature enhanced label propagation. Due to the small dimension of labels and the limited number of propagations during training, the cost negligible.
3.4 ABLATION STUDY
In this subsection, we conduct ablation studies to gain a better understanding of how each component of our method works which correspondingly answers the third question.
Feature Diffusion. is expected that feature diffusion can improve the ac- curacy of the MI.P in the pretrain- ing procedure when the label rate is low and thus improve the quality of pseudo labels F during its following update steps. To validate this, we re- move the feature diffusion step and also use the pseudo labels to train our method, which is called MULTVIEWAGNN-wwoddttiii Experiments are conducted both Cora and CiteSeer datasets. From Figure D1 we can that low label rates, MULTIVIEW that feature diffusion can boost the model's performance on low label rate setting. As the labeling rate increases, the performance gap becomes small, especially in the CiteSeer dataset. This shows that feature diffusion is not the key component our method when the label rate not very low.
50%
Cora
(b) CiteSeer
Figure 2: Performance of MULTIVIEWA4GN variants.
is better than WAw.IViIEwWGNN-Wooooiiiofei
which means
Pseudo Labels. One of the most important advantages of MUNN is that we leverage pseudo labels to better train MLP. To study the contribution of pseudo labels in MULTIVIEW4GNN, we test the model variant WNC.TIVIEWAGNNN--ooooo pseudo which only uses labeled data Cora and CiteSeer datasets. Compared with MULTTVIEW4GNN, Figure shows that pseudo labels have a large impact model performance on both datasets, especially when the label rate is low.
Moreover, we choose the top K confidence pseudo labels per class after the first update of F to verify their accuracy. We adopt the same way to evaluate Label Propagation on Cora dataset with the label rate 20. As shown in Figure EL after the first update of F, the accuracy of the top 180 nodes from each class can be 90%. So is reasonable use these pseudo labels to train MLP. Besides, the accuracy of our method at each K is much better than Label propagation, which suggests the effectiveness of the feature-enhanced label propagation update for F.
8

---

Under review as a conference paper at ICLR 2023
25
690
Lambda
Lambon 2
Figure 3: Top Accuracy.
Figure 4: Parameter Sensitivity.
Hyperparameters Sensitivity. We test the parameter sensitivity of Y and X2 in Eq (2) on Physics and Photo datasets by fixing one with the best parameters and tuning the other. From Figure !! MULTIVIEW4GNN is not very sensitive these hyperparameters at the chosen regions.
RELATED WORK
Graph Neural Network (GNN) is an effective architecture to represent the graph-structure data. Two essential operations in the GNN are feature propagation and feature transformation. Consid- ering how may times feature propagation in the training procedure, we categorize GNNs into Per- sistent propagation GNNs and One-time propagation GNNs. Persistent propagation GNNs (e.g.. GCN (Kipi & Welling M GraphSAGE (Hamilion et all 2017), GAT (Velickovic et all 2 require feature propagation on each training step. GCN (Kipt & Welling 2016) is the most com- monly used method which performs feature transformation then feature aggregation in each layer. More recently, decoupled GNNs are proposed to alleviate the over-smoothness problem (L all 2018; Qono & Suzuki, 2019). APPNP (Kliepera all, 20T)) is the first work to first apply multiple feature transformations then multiple aggregations. Similar architectures are also utilized (Lu etail 20ZI1 2OZ0 Zhou al (IZII) However, these decoupled GNNs still require feature aggrega- tion in each training step. One-time propagation GNNs are more efficient than the above Persistent propagation methods for they only propagate once despite the number of training steps. SGC (Wu et all ((11 is the first one to do feature aggregation and then transformation. SIGN (Rossi all all 2020) adopts a similar strategy with a different aggregation scheme. Recently, more efficient neu- ral networks are proposed which utilize graph structure for post-process. |uuan et all (2020) train base predictor on labeled data and then apply correct and smooth step to post-process. Dong ctal (202I) further understand the relationship between decoupled GNNs and label propagation and utilizes soft pseudo labels for training. PPRGo (Bojchevski et all 202 precomputes the PageRank matrix but it lose much edge information due to aggressive sparsification of the PageRank matrix, which often leads to degraded performance and non-trivial tradeoff between efficiency and accuracy.
Multi-view learning on Graph. Our proposed multi-view learning framework for graph representa- tion learning differs from existing works in literature. Specifically, multi-view graph cluster (Wang ellll 201%; Pan & Kang, 2021) consider totally different setting where multi-view attributes and multiple structural graphs exist. Multi-view graph contrastive learning algorithms (Hassani & Khasahmadı, 2020; Wang et all 2020)) use data augmentation to generate different graph views, then encourage the similarity between different views generated from the same graph while reduce the similarity in other view pairs. In contrast, our multi-view learning framework considers node features, graph structure, and node labels as three views of the nodes.
Unified understanding on GNN. Recent works ef 2021, Maceaal 2021 Yang et all, (0211
aim to provide unified optimization framework for understanding the message passing mechanism of different GNNs and designing new graph filter layers. However, they only focus on the for- ward process without taking the backward learning process into consideration, and they are still following the existing GNN architecture with end-to-end training. In this work, we do not aim to understand the message passing and design new layers based on existing architectures. Instead, MELLIIIEW4GN is a new graph deep learning framework as multi-view learning. It provides a new perspective for graph representation learning with better flexibility, explanability, and efficiency.
5 CONCLUSION
In this work, we provide a new perspective to view the three types of information available for node classification (i.e., graph structure, node feature, and node label) as three views of nodes. This understanding inspires us to design GNN models as multi-view learning. The proposed MULTI- VIEW4GNN framework imental results validate that MULTIVIEW4GNN promising performance on the node classification task especially when the label rate is low.
naturally be trained with the alternating optimization algorithm. Exper- both computational and memory efficient with

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E. OConnor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
James C Bezdek and Richard J Hathaway. Some notes on alternating optimization. In AFSS inter- national conference on fuzzy systems, pp. 288-300. Springer, 2002.
Aleksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Günnemann. Scaling graph neural networks with approximate pagerank. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 2464-2473, 2020.
Michael Collins, Sanjoy Dasgupta, and Robert E. Schapire. A generalization of principal compo- nents analysis to the exponential family. In NIPS, 2001.
Hande Dong, Jiawei Chen, Fuli Feng, Xiangnan He, Shuxian Bi, Zhaolin Ding, and Peng Cui. On the equivalence of decoupled graph convolution network and label propagation. I Proceedings of the Web Conference 2021, pp. 3651-3662 2021.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: 1933:02033 2019.
Matthias Fey, Jan E Lenssen, Frank Weichert, and Jure Leskovec. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings. In International Conference on Machine Learning, pp. 3294-3304. PMLR, 2021.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
S. K. Godunov, A. G. Antonov, O. P. Kiriljuk, and Victor I. Kostin. Singular value decomposition. Practical Numerical Mathematics with MATLAB, 2021.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Will Hamilton, Zhitao Ying, and Jure Leskovee. Inductive representation learning on large graphs Advances in neural information processing systems, 30, 2017.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning graphs. In International Conference on Machine Learning, pp. 4116-4126. PMLR, 2020.
Michael C. Hout, Megan Papesh, and Stephen D. Goldinger. Multidimensional scaling. Wiley interdisciplinary reviews. Cognitive science, 4 1:93-103, 2013.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs Advances in neural information processing systems, 33:221 1121118-221331 2020.
Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang, Erjin Zhou, and Yue Gao. Graph-mlp: node classification without message passing in graph. arXiv preprint arXiv:2106.0400 2021.
Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin Benson. Combining label propa- gation and simple models out- perfoorrssms graph neural networks. arXiv preprint arXiv:2010. 13993, 2020.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In Proceedings the IEEE/VV Conference on Computer Vision and Pattern Recognition, pp. 5070-5079, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.66 2014.
10

---

Under review as a conference paper at ICLR 2023
Thomas N Kipf and Max Welling, Semi Superviuee classification with graph convolutional networks. arXiv preprint wrXiv:160.002907 2016.
Johannes Kliepera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate:
Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05p0 2018.
Guanghui Lan. First-order and stochastic optimization methods for machine learning. Springer, 2020.
Long-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges representation learning, ICMI,, volume pp. 896, 2013.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International conference on machine learning, pp. 6437-6449. PMLR, 2021.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference artificial intelligence, 2018.
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp.
338-348, 2020.
Xiaorui Liu, Wei Jin, Yao Ma, Yaxin Li, Hua Liu, Yiqi Wang, Ming Yan, and Jiliang Tang. Elas- tic graph neural networks. In International Conference on Machine Learning, pp. 6837-6849. PMLR, 2021.
Yao Ma and Jiliang Tang. Deep learning on graphs. Cambridge University Press, 2021.
Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks graph signal denoising. In Proceedings of the 30th ACM International Conference on Information Knowledge Management, pp. 1211, 2021.
Yurii Nesterov et al. Lectures on convex optimization, volume 137. Springer, 2018.
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. arXiv preprint 1rXiv 2019.
Erlin Pan and Zhao Kang. Multi-view contrastive graph clustering, Advances in neural information processing systems, 34:2148 2159, 2021.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: online learning of social representa- tions. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 2014.
Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Fed- erico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.1 11988 2020.
Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 290 5500:2323-6, 2000.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rae Rad. Collective classification in network data. Al magazine, 29(3):93-93, 2008.
Oleksandr Shchur. Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprini arXiv: 18/ 100868, 2018.
Heng Tao Shen. Principal component analysis. In Encyclopedia of Database Systems, 2009.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. stat, 1050:20, 2017.

---

Under review as a conference paper at ICLR 2023
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arxiv: 2017.
Hao Wang, Yan Yang, and Bing Liu. Gme: Graph-based multi-view clustering, IEEE Transactions on Knowledge and Data Engineering, 32(6): 1e11111661.,,9 2019.
Yingheng Wang, Yaosen Min, Xin Chen, and Ji Wu, Multi-view graph contrastive representation learning for drug-drug interaction prediction. In Proceedings of the Web Conference 2021, pp. 2921-2999 2021.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference machine learning, pp- 6861-6871. PMLR, 2019a.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. arXiv preprint S55I:::901100596 2019b.
Tian Xia, Dacheng Tao, Tao Mei, and Yongdong Zhang. Multiview spectral embedding. IEEE
Transactions Systems, Man, and Cybernetics, Part B (Cybernetics), 40(6):1438- 143--1466, 2010.
Chang Xu, Dacheng Tao, and Chao Xu. A survey multi-view learning. arXiv:1304.5634 2013.
arXiv preprint
Liang Yang, Chuan Wang, Junhua Gu, Xiaochun Cao, and Bingxin Niu. Why do attributes propagate in graph convolutional neural networks? In Proceedings the AAAI Conference on Artificial Intelligence, volume 35, pp. 4590-4598, 2021.
Ming Yin, Junbin Gao, and Zhouchen Lin. Laplacian regularized low-rank representation and its ap- plications. IEEE Transactions on Puttern Analysis and Machine Intelligence, 38:504-517, 2016.
Hanqing Zeng. Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. arXiv preprint arXiv: arXiv: 1907.0491, 2019.
Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems, 16, 2003.
Kaixiong Zhou, Xiao Huang, Daochen Zha, Rui Chen, Li Li, Soo-Hyun Choi, and Xia Hu. Dirichlet energy constrained learning for deep graph neural networks. Advances Neural Information Processing Systems, 34, 2021.
Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021, pp. 1215- 1226, 2021.
Yanqiao Zhu, Yichen Xu, Feng Yu, Q. Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. ArXiv, abs/2006.0411 2020.
12

---

Under review as a conference paper at ICLR 2023
A MULTIVIEW4GNN WITH CROSS ENTROPY LOSS
In section 2.2, we instance MULTIVIEW4GNN with a Mean Square Error Loss. In this section, we show it can replaced by Cross Entropy Loss. By replacing the first part of Eq (L) be cross entropy between MLP and F, the formulation becomes:
= A1C (MLP()), F) (F(FF + F
F2 - YL
(11)
where C'E(,,) is the cross entropy function. Adopting the same gradient decent method, the update rule for F becomes:
F pk+1
= FL 1J1. (-X1
log MLP((X 2(LF"), + 222 - -v1)) = (1 [1-21 (1 + X2)) F + A1 log MLP(X)2 + ÄF + 2n
L
(12)
F F F NJU (- (-x1 M + 2(LF*)))
= (1 2n²) F + tu A1 log MIP((X)) + 2n (AF*))
(13)
where the MLP the output after the Softmax function and the step size can be set as 7n 2(1+))) Therefore, the update rule of F is:
22 Y
F++1
log MLP(X,)
+
(AF*)
)
(14)
211 + X22
1 X2
1 A2
A1
A2 1
F
log M
(AF*) 10
F
(15)
2(1 X2)
1 1+ A2
1+ 22
Then we consider the hidden variable F as pseudo label and update the parameters of MLP based on the cross entropy loss CE(MLP, F). In practice, there are some situations that the cross entropy have better performance than the original mean square error loss. Using two different losses would give similar results in most cases, and we report the best one.
B ALGORITHM OF MULTIVIEW4GNN
In this section, we provide the algorithm M and code MULTIVIEW4GNN.
In line 1, first initialize the pseudo label F as label matrix Y. In line 2, preprocess data by feature diffusion. In line 3, we pretrain the MLP on the labeled data for a few epochs. From lines to 9, we update F and MLP alternatively and iteratively.
Algorithm 1: MULTIVVEWAGNN
Input :Adaacet matrix A; Feature matrix X; Labels Y: Hyper parameerrsseterss λ1. A2 a, pseudo node number m; Labeled nodes L; layer p. pretraining step S; update times
Output:Pseute Pseudo Label F, MLP parameter θ
Initialize F Y:
X' + L.((X
Pretraining steps MLP with L= L= ((MLP(z(; Y.);;
while Stopping condition is not met do
Update based on Eq. (1Z) and Eq. (8) for times
exp(F.,/T) Normalize F based Fú EV exp(F.e/T)
Select m top unlabeled nodes U, per class by w. = 1 H(F.)
Update for & times by minimizing (M F;O) EiELUG, ( F,):
end return F,O:
The code implemented by Pytorch is avaliable at http://aaagyooo...guuussseeeeemr
F41 8/.
13

---

Under review as a conference paper at ICLR 2023
C DATASET STATISTICS
In the experiments, the data statistics used in Section 3 are summarized in Table B. For Cora, Cite- Seer and PubMed dataset, we adopt different label rates, i.e., 5, 10, 20, 60, 30% and 60% labeled nodes per class, to get more comprehensive comparison. For label rates 5, 10, 20, and 60, we use 500 nodes validation and 1000 nodes for test. For label rates 30% and 60%, we use half of rest nodes for validation and the remaining half for test. For Ogbn-arxiv dataset, we use the original fixed data split.
Table 5: Dataset Statistics.
Dataset
Nodes
Edges
Features
Classes
Cora
2,708
5,278
1,433
CiteSeer
3,327
4,552
3,703
PubMed
19,717
44,324
500
Coauthor CS
18,333
81,894
6,805
15
Coauthor Physies
34,493
247,962
8,415
Amazon Computer
13,381
245.778
767
10
Amazon Photo
7,487
119.043
745
Flickr 89,250
899,756
500
Reddit 232,965
11,606,919
602
41
Ogbn-Arxiv
169,343
1,166,243
128
40
Ogbn-Produee 2,449,029
61,859,140
100
47
D TRANSDUCTIVE NODE CLASSIFICATION RESULTS
Table 6: The overall results of the transductive node classification task.
Method
Nen-UNN
Prnsistnt propegtion metheds
Chte-time propagation methods
Ours
Daroett
label Rain
MI
OCN
GAT
APYNP
SCR
SIGN
CAS
MulisirostNNN
1.5
S7
5:71
2117
7237
7585
2:12
00.31
36
5:55
78.78 2.30
10
63.70
3.60
$1 34
3.37
1.42
78.03
1.17
80.20
1.00
76.25
76.25
1.26
71.01
11.04-11
3.50
BIL66 1.92
20
67.57
1.43
59.23
2.50
70.1
1.30
81.30
1.41
82.34
0.67
80.30
1.72
79.71
1.11
77.96
2.13
82.66 0.98
13.92
08.35
2,08
1.44
83.11
1.10
35 25
1:25
84.17
$4.16
1.18
82.21
1.45
85.60
1.12
82.26
75.26
1.38
80.0
87.28
87.77
1.13
86.97
87.17
1:28
81.001
81.00
1:19
76
1:15
18
1:78
68:11
1,50
41.05
50.30--
47.99
1.56
73.15
1.47
77.00 1.67
7XS1 4.6
65.30
3.58
3.85
3.700
4 3.75
75.55 5.09
69.07
77.43
78.93
76.03
76.39
75.34
79.16 2.36
75.79
76.20
$1.00
1.40
82.55
1.47
$1.75
1.49
82.5 1.26
30%
$2.51
0.34
85.92 2 0.25
83,07
0.29
$6.51
a: 0.41
87.56
0.39
96.23
0.43
$9.29
0.3
88.44
0.40
88.24 0.36
60%
$3.38
0.64
86.14
0.54
$5.48
0.46
86.52
3 0.56
87.56
0.52
96.63
0.38
99.55
0.56
88.53
0.56
88,83 0.55
CS
20
77.45
1.30
88.12
99.23
11:9
V036
0016
12.35
10.3
9000
039
9202
1 0441
9241
0.4.
92.77 2.000 0.30
Physks
20
56.10
1.00
88.30
1.39
1.39
93.29
1000
VEST
1 1:00
93.39
1007
91.23
1339
9100
1.15
93.23
0.55
MIL0S 2 1.3T
Compulers
20
72.44
2.87
00165
2 2.06
77.17
192
18.8
2.27
1 1.14
Y3.00
240
7508-1111 2113
15.25
2299
79.12 X.20
Pauna
20
$1.38
4.00
75.33
= 1.39
89.344
1.22
19.14 1,42
1088
1.144
13.01 23
16.11:
0.66
8437
1.00
91.23 2 1.20
oghn ands
54%
68.14
0.00
55.68
0.11
71.91
2 0.15
1192
0:30
0.12
1198110.11
7100
2.1.5
72.76 0.37
7408
1 0100
61.17 0.20
75.70.0019
00M
0.15
73.15
0012
30335016
27.11.50000
W1.64F 0.21
E PARAMETER SETTING
E. TRANSDUCTIVE SETTINGS
For all deep models, we use transformation layers with 256 hidden units for OGB datasets and layers with 64 hidden units for other datasets. We use the same learning rate of 0.01. For all methods, hyperparameters are tuned based on the loss and validation accuracy from the following search space: 1) dropout rate: {0, 0.5, 0.8 ]: 2) weight decay: [0, 5e-4, 5e-5); and 3) For the hyperparameters range between 0 and 1, we tune them by step size 0.1. The propagation step K for APPNP and C&S is tuned from (5, 10) and (10, 20, 50), respectively. The d and X in MULTIVIEW4GNN
tuned from (0.1, 0.3, 0.5, 0.7, 1| and ( (1, 3, 5, 7, 10), respectively. Adam optimizer(Kingma & Ba, 2014) used in all experiments.
14

---

Under review as a conference paper at ICLR 2023
E.2 INDUCTIVE SETTINGS
We first filter the training graph that only contains labeled node for training, and the entire graph are used for inference. For all models, dataset, and layers with 64 hidden units for Flickr dataset. For all methods, hyperparameters are tuned based on the loss and validation aceuracy from the following search space: (1) learning rate: {0.01. 0,05 ): (2) dropout: {0, 0.1]: (3) weight decay: {0}; For the hyperparameters range between and 1, we tune them by step size 0.1. The propagation step K for APPNP and C& is tuned from
transformation layers with 256 hidden units for Reddit
(2, 3, 5, 10) and {10, 20, 50), respectively. The λ and X2 in MULTIVIEW4GNN are tuned with granularity of 0.1 in range [0, 1], and [1, 10], respectively.
15


Reviews:
[{"review_id": "xpoRDpPnHrp", "reviewer": "Reviewer_ogUi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way to map node features $\\boldsymbol{X_i}$, graph structure $\\boldsymbol{A}$, and node label $\\boldsymbol{Y_i}$ into a shared latent feature space $\\boldsymbol{F_i}$. Each node $i$ is now represented by a vector $\\boldsymbol{F_i}$, which has the same dimension as one-hot encoded label $Y_i$. $\\boldsymbol{F_i}$ is treated as the soft pseudo label of node $i$.  \n\nThe $X_i \\mapsto F_i $ mapping is done by an MLP with learnable $\\Theta$ params. The graph structure $\\boldsymbol{A}$ is enforced by imposing Laplacian smoothness of $\\boldsymbol{F_i}$ against its neighbors $\\boldsymbol{F_j}$. The node label $\\boldsymbol{Y_i}$ is enforced by otimizing MSE between $\\boldsymbol{Y_i}$ and $\\boldsymbol{F_i}$ directly.\n\nThe paper also proposes an alternative training scheme to learn $\\boldsymbol{F}$ and $\\Theta$ with various training tricks such as: initial feature diffusion, MLP pre-training that uses only $\\boldsymbol{Y_i}$, importance sampling based on $\\boldsymbol{F_i}$ entropy, class balancing...\n\nThe experiment results show that the proposed methods yield good performance when the label rate is very low.\n\n", "summary_of_the_review": "The paper's novelty is not significantly strong. Most of its empirical positive results are from training tricks that are introduced by Iscen et al. 2019. The paper claims that alternative training is more flexible and more efficient than end-to-end training, which are not well-supported.", "strength_and_weaknesses": "Strength:\n  - The paper is well presented, with good support materials in the Appendix, including speudo code, dataset statistics, and parameter setting.\n  - The paper conducts relevant experiments, including ablation study and efficiency comparison. \n\nWeaknesses:\n  - The whole training scheme is very similar to the Iscen et al. 2019 \"Label Propagation for Deep Semi-supervised Learning\" paper, including most of the important training tricks.\n  - Experiment results only show good improvement on very low label setting (5-60 examples per class). For most of the time, the improvements are not statistically significant.\n  - Hyper-param search space for the proposed method is much larger than baseline models. It includes $\\lambda_1$ and $\\lambda_2$, which together create 25 to 100 settings (Appendix E). On the contrary, the baseline models use fixed learning rate = 0.01. Apart from that, there are many training params that are used in the training algorithm (Appendix B) but are not mentioned in the paper. They are usually also hard to tune right, including $S$ pre-training steps, $p$ times $F$ update and $t$ times $\\Theta$ update per iteration.\n  - The paper claims that alternating optimization provides better flexibility than end-to-end training. This is not a well-supported statement. In practice, end-to-end training is much easier to setup, does not require training tricks, and depends less on hyper-params tuning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "USb6cx2mn2", "reviewer": "Reviewer_aMtC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new perspective of graph learning by regarding node features, graph structures, and node labels as three different views. Different from the previous GNN models, the paper brings in a new latent variable to be learned and shared across all three views. Through a progressive optimization framework, the paper claims the new framework can obtain better computational and memory efficiency. ", "summary_of_the_review": "In general, the paper has some interesting insights but the novelty is limited. And the contribution especially the analysis of the advantage of the complexity has some problems. ", "strength_and_weaknesses": "The regularization framework  $min_F tr(F^T L F) + ||F-Y||^2$, which is used to derive the graph learning models, is not new and it has been well known for decades. Early works such as manifold ranking [1] already used it; in the era of graph neural networks, as the paper mentioned, some works [2] proposed the unified optimization view of GNNs and also used it; in addition, from the perspective of graph signal processing, there are also some works [3,4] which used it as a special case. These works generally directly obtained the solution expression of F, and then use MLP(X) to take place of F. This paper differs in that it added || MLP(X) – F||^2 as another regularization and unroll the optimization process (the methodology of [4] is somehow similar but used a different objective). It does provide a new perspective and still looks a little bit trivial. The understanding of updating F as propagation and updating \\Theta as pseudo label generation is interesting.\n\n However, the biggest problem is I did not see much advantage of the new formulation and optimization. The analysis of the time complexity if also doubtable.  If Multiview4GNN and APPNP only have dimension $c$ in time complexity because of the feature dimension transformation. Why do not we use a different step for SGC? $XW$ part can be first calculated (so the feature dimension is changed to c) and then do the feature aggregation with \\sigma(A^n XW). Let us look back at Multiview4GNN, as other methods contains the computation of backpropagation, why is the backpropagation of the MLP in equation (9) not counted in Multiview4GNN? This MLP generally does not only have one layer so the dimension $d$ cannot be omitted, and the backpropagation also takes $t$ steps. \n\n[1] Zhou et al. Learning with Local and Global Consistency. In NIPS 2003.\n[2] Ma et al. A Unified View on Graph Neural Networks as Graph Signal Denoising. In CIKM 2021.\n[3] NT, Maehara. Revisiting graph neural networks: All we have is low-pass filters[J]. arXiv preprint arXiv:1905.09550, 2019.\n[4] Chen et al. BiGCN: a bi-directional low-pass filtering graph neural network. arXiv preprint arXiv:2101.05519, 2021.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IXI1wUpE4nk", "reviewer": "Reviewer_YbZy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes using node features, graph structure, and node labels as three different views and maximizing their agreement.  Therefore, the loss function has three terms and gets optimized with an alternating optimization method. ", "summary_of_the_review": "Most of the points are mentioned in the strength and weaknesses section but in summary, the authors suggest that we can see different types of information as different \"views\" and try to maximize the agreement between these views with an alternation optimization algorithm. They show that the model is efficient and has good performance, mostly in very low node label scenarios. The datasets in the experiment section are small and not diverse.", "strength_and_weaknesses": "**Strengths**\n- The paper is well-written and clear.\n- The method shows strong performance in very low node label scenarios.\n- MULTIVIEW4GNN is memory and time efficient.\n\n**Weaknesses** \n- D_A assumes that graphs have high homophily ratios (the smooth assumption). What would the model perform when the graph is heterophily?\n- The performance is promising mostly in the scenarios where we have very few labels. \n- In my opinion, more rigorous experimentation is needed. The datasets are mostly small, easy to solve, and not diverse. \n- Is it mentioned anywhere in the paper what would be the case if there are no initial node features? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D4AiwO_vpl", "reviewer": "Reviewer_xFqB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers node features, graph structure, and node labels as three views, and designs three loss components correspondingly. Moreover, the latent $F$ and the model parameters $\\Theta$ are proposed to be optimized alternatingly to achieve efficiency and scalability.\n", "summary_of_the_review": "This paper decouples the supervised loss and optimizes it alternatingly to achieve efficiency and scalability. However, the technical contribution is a bit limited.", "strength_and_weaknesses": "Strength:\n\n1. The proposed multi-view loss (with *lazy propagation*) is an effective method to achieve the trade-off between performance and cost.\n2. The authors conduct extensive ablation experiments to verify each model component.\n\nWeaknesses:\n\n1. The three losses after decoupling are not novel individually. Each component seems to be standard and well-studied in graph semi-supervised learning.\n2. The pseudo-label with certainty and class-balancing in graph semi-supervised learning are also discussed in [1]. And there are other works highly related to the core ideas in this paper, e.g., [2,3,4], should be discussed.\n\nReferences:\n\n[1]. Self-Enhanced GNN: Improving Graph Neural Networks Using Model Outputs. Han Yang, Xiao Yan, Xinyan Dai, Yongqiang Chen, James Cheng. IJCNN '2021\n\n[2]. Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination. Yizhen Zheng, Shirui Pan, Vincent Cs Lee, Yu Zheng, Philip S. Yu. NeurIPS ’22.\n\n[3]. Combining Label Propagation and Simple Models Out-performs Graph Neural Networks. Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, Austin R. Benson. ICLR ’21.\n\n[4]. On Graph Neural Networks versus Graph-Augmented MLPs. Lei Chen, Zhengdao Chen, Joan Bruna. ICLR ’21.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject