A Unified Framework for Convolution-based Graph Neural Networks
Authors: Xuran Pan
Shiji Song
Gao Huang
Venue:  (2021)
Keywords: Graph Representation Learning
Graph Convolution
Graph Signal Processing
Oversmoothing

Abstract: Graph Convolutional Networks (GCNs) have attracted a lot of research interest in the machine learning community in recent years. Although many variants have been proposed, we still lack a systematic view of different GCN models and deep understanding of the relations among them. In this paper, we take a step forward to establish a unified framework for convolution-based graph neural networks, by formulating the basic graph convolution operation as an optimization problem in the graph Fourier space. Under this framework, a variety of popular GCN models, including the vanilla-GCNs, attention-based GCNs and topology-based GCNs, can be interpreted as a same optimization problem but with different carefully designed regularizers. This novel perspective enables a better understanding of the similarities and differences among many widely used GCNs, and may inspire new approaches for designing better models. As a showcase, we also present a novel regularization technique under the proposed framework to tackle the oversmoothing problem in graph convolution. The effectiveness of the newly designed model is validated empirically.

Under review as a conference paper at ICLR 2021
A UNIFIED FRAMEWORK FOR CONVOLUTION-BASED GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Graph Convolutional Networks (GCNs) have attracted lot of research interest in the machine learning community recent years. Although many variants have been proposed, we still lack systematic view of different GCN models and deep understanding of the relations among them. In this paper, we take a step forward establish unified framework for convolution-bast baadd graph neural networks, by formulating the basic graph convolution operation as an optimization problem the graph Fourier space. Under this framework, variety of popular GCN models, including the vanila-GCNss, attention based GCNs and topology-basee GCNs, can be interpreted as a same optimization problem but with different carefully de- signed regularizers. This novel perspective enables a better understanding of the similarities and differences among many widely used GCNs, and may inspire new approaches for designing better models. As showcase, we also present a novel regularization technique under the proposed framework tackle the oversmooth- ing problem in graph convolution. The effectiveness of the newly designed model is validated empirically.
INTRODUCTION
Recent years have witnessed a fast development in graph processing by generalizing convolution operation to graph-structured data, which is known as Graph Convolutional Networks (GCNs) (Kipf & Wellingg (017)) Due to the great success, numerous variants of GCNs have been developed and
extensively adopted in the field of social network analysis (Hamilton et al. 2017: Wu et al. 201 9a
Veličković et al. 2018), biology (Zitnik et al. 2018), transportation forecasting (L. et al. 201 and
natural language processing (Wu et al. 2019h Yao et al. 2019).
graph learning approaches are proposed to enhance the generalization performance of graph neural networks. Several research aim to achieve higher expressiveness by exploring higher-order information or introducing additional learning mechanisms like attention modules. Although proposed from different perspectives, their exist some connections between these approaches. For example, attention- GCNs like GAT (Veličkovid) let al. 2018) and AGNN (Thekumparampil et al. 2018) share the similar intention by adjusting the adjacency matrix with a function of edge and node features. Similarly, TAGCN (Du et al., (217)) and MixHop (Kapoor et al., (019) can be viewed as particular instances of PPNP (Klicpera al., 2018) under certain approximation. However, the relations among these graph learning models are rarely studied and the comparisons are still limited in analyzing generalization performances on public datasets. As a consequence, we still lack a systematic view of different GCN models and deep understanding of the relations among them.
Inspired by GCN, wide variety of convolution-
In this paper, we resort to the techniques in graph signal processing and attempt to understand GCN-based approaches from a general perspective. Specifically, we present a unified graph convo- lution framework by establishing graph convolution operations with optimization problems in the graph Fourier domain. We consider a Laplacian regularized least squares optimization problem and show that most of the convolution-based approaches can be interpreted in this framework by adding carefully designed regularizers. Besides vanilla GCNs, we also extend our framework to formulat- ing non-convolutional operations (Xu et al., 2018a, Hamilton et al. 2017). attention- GCNs (Veličković et al. 2018: Thekumparampil et al. 2018) and topology- based GCNs (Kliepera et al. 2018 Kapoor al. 2019), which cover large fraction of the state-of state-of-the-art graph learning ap-

---

Under review as a conference paper at ICLR 2021
proaches. This novel perspective provides re-interpretation of graph convolution operations and enables a better understanding of the similarities and differences among many widely used GCNs, and may inspire new approaches for designing better models.
As conclusion, we summarize our contributions as follow:
1. We introduce unified framework for convolution-basee graph neural networks and interpret various convolution filters as carefully designed regularizers in the graph Fourier domain, which provides a general methodology for evaluating and relating different graph learning modules.
2. Based on the proposed framework, we provide new insights on understanding the limitations of GCNs and show new directions tackle common problems and improve the generalization per- formance of current graph neural networks in the graph Fourier domain. Additionally, the unified framework can serve as a once-for-all platform for expert-designed modules on convolution-base approaches, where newly designed modules can be easily implemented on other networks as a plug- in module with trivial adaptations. We believe that our framework can provide convenience for designing new graph learning modules and searching for better combinations.
As a showcase, we present a novel regularization technique under the proposed framework to alleviate the oversmoothing problem in graph representation learning. As shown in Section 4 the newly designed regularizer can implemented on several convolution-base networks and effec- tively improve the generalization performance of graph learning models.
2 PRELIMINARY
We start with an overview of the basic concepts of graph signal processing. Let 9 = AV denote a graph with node feature vectors where V represents the vertex set consisting of nodes (V1, U....., UN and A = (a.z) E NNN is the adjacency matrix implying the connectivity be- tween nodes in the graph. Let D =
d(N)) E R NxN be the degree matrix of A A the combinatorial Laplacian and L - D (-1/2) (-(-1/2) is the normalized Laplacian of 9. Additionally,
where d(i) Ejev (ij is the degree of vertex i. Then, L=
let Á = A I and D D + I denote the augmented adjacency and degree matrices with added self-loops. Then Faym I D --1--/110 AD-1/2 1/2 (Asym D 1/2 AD 1/2) and A.w I - - Á (Aew D- A) are the augmented symmetric normalized and random walk normalized Laplacian (augmented adja- cency matrices) of 9, respectively.
Let x E N be signal on the vertices the graph. The spectral convolution is defined as a function of filter go parameterized in the Fourier domain (Kipf & Welling 2017):
9a (A)(AA xx
(1)
where U and A are the eigenvectors and eigenvalues of the normalized Laplacian Σ. Also, we follow Hoang & Maehara (2019) and define the variation A and D-inner product as:
A(x) Σ (x)( =(1)² x Læ, veDy Σ (d(1) + 1)x(((((11) x Dy. (2)
which specifies the smoothness and importance of the signal respectively.
3 UNIFIED GRAPH CONVOLUTION FRAMEWORK
With the success of GCNs, wide variety of convolution-base approaches are proposed which pro- gressively enhance the expressive power and generalization performance of graph neural networks. Despite the effecttivnness of GCN and its derivatives specific tasks, there still lack comprehen- sive understanding on the relations and differences among various graph learning modules.
Graph signal processing is powerful technique which has been adopted in several graph learning researches (Kipf & Welling 2017: Hoang Maehara 2019: Zhao & Akoglu 2019). However, existing researches mainly focus analyzing the properties of GCNs while ignore the connec- tions between different graph learning modules. Innovatively, in this work, we consider interpreting convolution-based approaches from general perspective with graph signal processing techniques.
2

---

Under review as a conference paper at ICLR 2021
In specific, we establish the connections between graph convolution operations and optimization problems in graph Fourier space, showing the effect of each module explicitly with specific reg- ularizers. This novel perspective provides systematic view of different GCN models and deep understanding of the relations among them.
3.1 UNIFIED GRAPH CONVOLUTION FRAMEWORK
Several researches have proved that, in the field of graph signal processing, the representative fea- tures are mostly preserved in the low-frequency signals while noises are mostly contained in the high-frreuuenyy signals (Hoang Maehara 2019). Based on this observation, numerous graph rep- resentation learning methods are designed to decrease the high-frequency components, which can be viewed as low-pass filters in the graph Fourier space. With similar inspiration, we consider Laplacian regularized least squares optimization problem with graph signal regularizers and attempt to build connections with these filters.
Definition 1 Unified Graph Convolution Framework. Graph convolution filters can be achieved by solving the following Laplacian regularized least squares optimization:
min X
Σ | - (i)l + >Xregy
NVV
where | = (2x, D denotes the norm induced by D.
In the following sections, we will show that a wide range of convolution- based graph neural net- works can be derived from Definition I with different carefully designed regularizers, and provide new insights on understanding different graph learning modules from the graph signal perspective.
3.1.1 GRAPH CONVOLUTIONAL NETWORKS
Graph convolutional networks (GCNs) Kipf & Welling 2017) are the foundation of numerous graph learning models and have received widespread concerns. Several researches have demonstrated that the vanilla GCN is essentially type of Laplacian smoothing over the whole graph, which makes the features of the connected nodes similar. Therefore, to reformulate GCNs in the graph Fourier space, we consider utilizing the variation ((x) as the regularizer.
Definition 2 Vanilla GCNs. Let (())sev be the estimation of the imput observation x()),ev A low-pass filter:
A
is the first-order approximation of the optimul solution of the following optimizution:
min -1=(0)=<(i111 Σ
aijl< x(j)1l
(5)
iJev
Derivations of the definitions are presented in Appendix
As the cigenvalues of the approximated filter Arw are bounded by 1, it resembles low-pass fil- ter that removes the high- fighffrequency signals. By exchanging Arw with Asym (which has the same eigenvalues A,,), obtain the same formulation adopted in GCNs.
has been stated that the second term A(x) in (q. measures the variation of the estimation a over the graph structure. By adding this regularizer to the objective function, the obtained filter em- phasizes the low-frequency signals through minimizing the variation over the local graph structure, while keeping the estimation close to the input in the graph Fourier space.
3.1.2 NON-CONVOLUTIONAL OPERATIONS
Residual Connection. Residual connection is first proposed by He et al. (016) and has been widely adopted in graph representation learning approaches. In the vanilla GCNs, norms of the eigenvalues of the filter A.w (or Asym) are bounded by which ensures numerical stability in the training procedure. However, convolution layer stacks, leading to a consistent information loss. Therefore, adding the residual connection is deemed preserve the strength of the input signal.
the other hand, signals in all frequency band will shrink as the
3

---

Under review as a conference paper at ICLR 2021
Definition 3 Residual Connection. A graph convolution filter with residual connection:
= +
where e > O controls the strength of residual connection, is the first-order approximation of the
optimal solution of the following optimization:
min X
- 3
Σ aayla(1)) -1)))
TEV
JJEV
By adding the negative regularizer penalize the estimations with small norms, we can induce the same formulation as the vanilla graph convolution with residual connection.
Concatenation. Concatenation is practically a residual connection with different learning weights. Definition 3' Concatenation. A graph convolution filter concatenating with the input signal:
X = Arw X + eX
(8)
is the first-order approximation of the optimal solution of the following optimization: [ii)(=(() a
min X
(Ila(i)-=(iilp) =l(
Σ
LEV
where e controls the strength of concatenation and the learning coefficient.
Although the learning weights 00? has a constrained expressive capability, it can be compensated by the following feature learning modules.
3.1.3 ATTENTION-BASE CONVOLUTIONAL NETWORKS
Since the convolution filters in GCNs are dependent only on the graph structure, GCNs are proved to have restricted expressive power and may cause the oversmoothing problem. Several researches try to introduce the attention mechanism edge weights at each layer based nodes and edges. GAT (Veličkovćć et al., (018)) and AGNN (Thekumparampil al. 2018) compute the attention coefficients as a function of the features of con- nected nodes, while ECC (Simonovsky & Komodakis) 2017) and GatedGCN (Bresson & Laurent 2017) consider the activations for each connected edge. Although these approaches have different insights, they can be all formulated as (See details in Appendix A):
the convolution filter, learning to assign different
Pij = fe (i)),, axi), exj) i, V.
(10)
where Ei denotes the edge representation if applicable. Therefore, we replace az Definition with learned coefficients to enforce different regularization strength on the connected edges.
Definition 4 Attention-based GCNs. An attention- based graph convolution filter:
=
(11)
is the first-order approximation of the optimal solution of the following optimization:
min Σ 10))))=()113
P =()1
s.t.
E DDa=Divee.. JeV
(12)
iJEV
Notice that we use a normalization trick to constrain the degree of attention matrix to be the same as the original degree matrix as we want to preserve the strength of the regularization for each node. The formulated filter P corresponds to the matrix D-1p with row sum equals to 1, which is also consistent with most of the attention-based approaches after normalization. Through adjusting the regularization strength for edges, nodes with higher attention coefficients tend to have similar features while the distance for nodes with low attention coefficients will be further.
3.1.4 TOPOLOGY-BASED CONVOLUTIONAL NETWORKS
Attention ditention-based approaches are mostly designed based on the local structure. Besides focusing on the first-order adjacency matrix, several approaches (Klicpera et al. 2018 2019) Kapoor et al. 2019 Du et al. 2017) propose to adopt the structural information in the multi-hop neighborhond,

---

Under review as a conference paper at ICLR 2021
which are referred to topology-biee saaee convolutional networks. We start with analysis of PPNP Kllippea et al. 2018) and then derive a general formulation for topology- based approaches.
PPNP. PPNP provides insights towards the propagation scheme by combining message-passing function with personalized PageRank. As proved in (Xu et al.] 2018b), the influence of node i on node j is proportional to a k-step random walk, which converges the limit distribution with multiple stacked convolution layers. By involving the restart probability, PPNP is able to preserve the starting node i's information. Similarly, in Definition the first term can also be viewed regularization of preserving the original signal information. Therefore, we may achieve the same purpose by adjusting the regularization strength.
Definition 5 PPNP. A graph convolution filter with personalized propagation (PPNP):
X = aI 121-oj o))A.)) equivalent to the optimal solution of the following optimization:
(13)
min
- + (1 Σ
aiy -,)l=(i)-
(14)
NEV
JJVV
where a E (0, 1] is the restart probability.
Higher means higher possibility to teleport back to the starting node, which is consistent with the higher regularization on the original signal in [4).
Multi-hop PPNP. One of the possible weakness of the original PPNP is that personalized PageRank only utilizes the regularizer over the local structure. Therefore, we may improve the expressive capability by involving multi-hop information, which is equivalent to adding regularizers for higher- order variations.
Definition 6 Multi-hop PPNP.
Let t be the highest order adopted in the algorithm. A graph convo- lution filter with multi-hop hopp personalized propagation (Multi-hop PPNP):
X =oo
) )- -
(15)
k=1
where Ei_o au = ao > 0 and ag 2 0,k = 1,2,.... ,,, is equivalent to the optimal solution of the following optimization:
min X
Co
Σ [ -
Ok
Σ (k)
a
(16)
1EV
(k) is proportional to the transition probability of the k-step random walk and the same where 0ig normalization trick in Section 3.1.3 adopted on {at)]
(k)
Solving Eq.415) directly is computationally expensive. Therefore, we derive a first-order approxi- mation by Taylor expansion and result in the form of:
X
A ew X O(A
(17)
2=0
As the norm of the eigenvalues of A,w are bounded by 1, we can keep the first term in Eq. 17 as close approximation.
By comparing the approximated solution with topology-based graph convolutional networks, we find that most of the approaches can be reformulated as particular instances of Definition 6. For example, the formulation for Mixhop (Kapoor et al. 2019) can be derived as an approximation of Eq. if we let and ag a1 02 1/3. Different learning weights can be applied to each hop as Section 3.2 to concatenate multi-hop signals. See more examples in Appendix B.
3.2 REMARKS
In this section, we build bridge between graph convolution operations and optimization problems in the graph Fourier space and provide insights into interpreting graph convolution operations with regularizers. For conclusion, we rewrite the general form of the unified framework as follow.
5

---

Under review as a conference paper at ICLR 2021
Definition 1' Unified Graph Convolution Framework. Convolution- based graph neural networks can be reformulated (after approximation) as particular instances of the optimal solution of the following optimization problem:
min 00 ((1=(i)-x(())) - X
-
Σ k=1
ON
( Py KEEV
Σ
+
1EV
Mon-Conv
Attention based
Topology based
(18)
where Σi=0 O4k az 2 and Ejev DRs. Vi E
If we let d be the feature dimension of X, then θ ( I are the corresponding learning weights. Lreg corresponds to the personalized regularizer based on the framework, which can be effective if carefully designed as we will show in Section
By establishing the unified framework, we interpret various convolution filters as carefully designed regularizers in the graph Fourier domain, which provides new insights on understanding graph learn- ing modules from the graph signal perspective. Several graph learning modules are reformulated as smoothing regularizers over the graph structure with different intentions. While vanilla GCNs fo- cus on minimizing the variation over the local graph structure, attention-base and topology-basee the differences between connected edges and graph
GCNs take step forward and concentrate structure with larger receptive field. This novel perspective enables a better understanding of the similarities and differences among many widely used GCNs, and may inspire new approaches for designing better models.
4 TACKLING OVERSMOOTHING UNDER THE UNIFIED FRAMEWORK
Based on the proposed framework, we provide new insights on understanding the limitations of GCNs and inspire new line of work towards designing better graph learning models. As a show- case, we present a novel regularization technique under the framework to tackle the oversmooth- ing problem. It is shown that the newly designed regularizer can easily implemented on other convolution-based networks with trivial adaptations and effectively improve the generalization per- formances of graph learning approaches.
4.1 REGULARIZATION ON FEATURE VARIANCE
Here, adopt the definition of feature-wise oversmoothing in (Zhao & Akoglu 2019). Due to multiple layers of Laplacian smoothing, all features fall into the same subspace spanned by the dominated eigenvectors of the normalized adjacency matrix, which also corresponds the similar situation described in (Kliepera et al. 2018). To tackle this problem, we propose to penalize the features when they are close each other. Specifically, we consider the pairwise distance between normalized features, which is summarized as:
8(X) E. 11 /I. [x x.y x..///.x, |
(19)
where d is the feature dimension and R represents the i-th dimension for all nodes. There- fore, (9) can be interpreted as a feature variance regularizer, representing the distance between features after normalization. By adding this regularizer to the unified framework, the proposed filter should have the property to drive different features away.
Definition 7 Regularized Feature Variance. Let & be the Kronecker product operator, xec(X) E Rnd be the vectorized signal X. Let Dx be a diagonal matrix whose value is defined by Dx (i, = ||xll/2 A graph convolution filter with regularized feature variance:
112.
vee(X) (In & [[a: + a2) azārv) aa
D-1) vec(X) -- (20)
equivalent to the optimal solution of the following optimization:
min
+
anj an)l=(i) Jee
(
03
Σ s,jdd
| |r.
- //..>1,,1111 (21)
X

---

Under review as a conference paper at ICLR 2021
Table 1: Test accuracy (%) transductive learning datasets. We report values and standard deviations in 30 independent experiments. The best results are highlighted with boldface.
Method
Dataset
Citeseer
Cora
Pubmed
FastGCN Chen al
2018
68.810..
79.810.3
76.8100.6
DGI /Velcckvić et al
2019
71.8+007
82,3+40.6
76.8
Vanilla
GIN (u et al.
201 8a
66.120.99
77.6+1.1
77.0+1.2
SGC Wu et
2019a
71.9=#0.1
81.0+0.0
78.9++.0
GCN (Kipf Welling
2017
70.310.4
81.5+0..5
79.0::0.4
GCN+reg (ours)
72.2:10.4
83.6110.3
79.8
AGNN (Thekumparampil et al.
2018
71.610.5
82.710..4
78.910.4
GatedGCN
[Bresson & Laurent
2017
72.010.4
82,440.6
78.9+0.3
Attention
MoNet (Monti el al.
201
81.7.00.5
78.8
GAT VVllikkvvi ct al.
2018
72.5+007
83.0+0.66
08.5++0.3
Gre (ours)
73.310044
83.910.6
80.3:10.3
TAGCN (Du ct al. 2017
70.9
82.5
81.1
MixHop Kapoor et al.
2019
71.4400.
81.9++..4
80.8++0.6
Topology
APPNP Klicpera
2018)
70.510.9
82.7:0.88
79.440.6
APPNP+reg (ours)
71.9.10.4
84.0
80.2.10.3
where 0n O, 022, 03 O. For computation efficiency, we approximate ||+,|| with || as assume that single convolution filter provides little effect to the norm of features.
Calculating the Kronecker product and inverse operators are computationally expensive. Neverthe- less, we can approximate Eq. (2)) via Taylor expansion with an iterative algorithm. If we let:
A = (an + 022 azAm.:
B = In: D
(22)
03 D-1
D=
11 D
(23)
Then, t-order approximated formulation is summarized
X (0) X, X (++1) = = + (k) AX B B-X k= 1.
(24)
(25)
Through approximation, computation overhead is greatly reduced. See details in the Appendix A.
As far as we are concerned, the advantages of utilizing feature variance regularization are three- fold. First, the regularizer measures the difference between features, therefore explicitly preventing all features from falling into the same subspace. Second, the modified convolution filter does not require additional training parameters, avoiding the risk of overfitting. Third, the regularizer is designed based on the proposed unified framework, which means it can be easily implemented on other convolution-ban networks as a plug-in module.
4.2 DISCUSSION
Several researches have also shared insights understanding and tackling oversmoothing. It is shown in (Li et al. 2018) that the graph convolution of GCN is special form of Laplacian smooth- ing and the authors try to compensate the long-range dependencies by co-training GCN with random walk model. JKNet (Xu et al. 2018b) proved that the influence score between nodes con- verges to a fixed distribution when layer stacks, therefore losing local information. As remedy, they proposed to concatenate layer-wise representations to perform mixed structural information. More recently, Oono & Suzuki (2020) theoretically demonstrated that graph neural networks lose expressive power exponentially due oversmoothing. Comparing to the aforementioned researches, our proposed method acts explicitly on the graph signals and can be easily implemented on other convolution-basee networks plug-in module with trivial adaptations.

---

Under review as a conference paper at ICLR 2021
0.05
0.5
DON-
Recursyy
0.8
0.75
DOR -GAT
0.85
0.7
0.28
(a) Accuracy
(b) Mean Feature Variance
Figure 1: Accuracy and mean feature variance on Cora. Use GCN and GAT for comparison.
4.3 EXPERIMENT
To testify the effectiveness of the regularizer, we empirically validate the proposed method on sev- eral widely used semi-supervised node classification benchmarks, including transductive and in- ductive settings. As we have stated in Section 4.1 our regularizer can be implemented on vari- convolution-based approaches under the unified graph convolution framework. Therefore, consider three different versions by implementing the regularizer on vanilla-GCNs, attention-based GCNs and topology-basee GCNs. We achieve state-of-the-ar results on almost all of the settings and show the effectiveness of tackling oversmoothing on graph-structured data.
Dataset and Experimental Setup. We conduct experiments on four real-world graph datasets. For transductive learning, we evaluate our method on the Cora, Citeseer, Pubmed datasets, following the experimental setup in (Sen al., 2008). PPI (Zitnik Leskovec 2017) is adopted for induc- tive learning. Dataset statistics and more experimental setups are presented in Appendix C. For comparison, we categorize state-of-the-art convolution-
graph neural networks into three spe- cific classes, corresponding to the three versions of proposed method. The first category is based on the vanilla-GCN proposed by Kipf & Welling (017)), including GCN, FastGCN (Chen al. 2018). SGC (Wu et al. 2019a), GIN (Xu et al. 2018a), and DGI (Velickovćć et al.) 2019). Since GIN is not initially evaluated on citation networks, we implement GIN following the setting in (Xu et al. 2018a). The second category corresponds to the attention-based approaches, includ- ing GAT (Veličković et al., 2018). AGNN (Thekumparampil et al., 2018), MoNet (Monti et al.) 2017) and GatedGCN (Bresson Laurent) 2017). The last category of approaches topology- based GCNs which utilizes the structural information in the multi-hop neighborhood. We consider APPNP (Klicpera al. 2018). TAGCN (Du et al. 2017) and MixHop (Kapoor al. 2019) as the baselines.
Transductive Learning. Table presents the performance of our method and several state- of-the-art graph neural networks on transduc- tive learning datasets. For three classes of convolution-based approaches, we implement regularizer with GCN, GAT and APPNP comparisons with other basclines, respectively. For fair comparison, adopt the same work structure, hyperparameters and training shown
Table 2: Test Micro-F1 Score on inductive learn- ing dataset. We report mean values and standard deviations in 5 independent experiments.
Dataset
PPI
GCN
(Kipf Welling
2017
92.4
GAT
Vcličković et al.
2018
97.3
SGC
Wu et al.
2019a
66.4
JKNet
Xu et al.
2018b)
97.6
configurations as baseline models. that the proposed model achieves state-of-the- results on all three settings. On all of the datasets, we can observe a 1.0% higher performance after adopting the proposed reg- ularizer. Notably, the proposed model achieves the highest improvement on the vanilla GCNs as this simplest version suffers most from the oversmoothing problem. Meanwhile, when combining with GAT, the model achieves the highest results comparing with almost all the baselines. Con- sidering that attention mechanism and the regularization on oversmoothing focus on the local and global properties respectively, this be ideal combination for graph representation learning. We also conduct experiments on three citation networks with random splits and present the result in Appendix D.
GraphSAGE Hamilton el
2017
61.2
DGI Veličković et al.
2019
63.8
GCN+reg (ours)
97.69-1.332
GAT+reg (ours)
98.23-0.08
Inductive Learning. For the inductive learning task, we implement our method on the vanilla GCN and GAT, and adopt the same experimental setup. Table 2 the comparison results
8

---

Under review as a conference paper at ICLR 2021
Table 3: Comparison results transductive learning datasets. We report mean values and standard
deviations in 30 independent experiments. The best results are highlighted with boldface. The number in the brackets represent the number of GCN layers when achieving the best performance.
Method
Citeseer
Cora
Pubmed
GCN + DropEdgeRong et al. 2019
73.2:0..1 (4)
84.2:10.6 (6)
80.110.3 (6)
GCN PairNormZZao Akoglu (2019)
71.0+:0.5
82.2 (2)
79.6=0.66 (4)
GCN regs (ours)
73.6+
84.9+ (5)
81.0==0.4 (5)
inductive learning dataset. It can be seen that our model compares favorably with all the competitive baselines. On the PPI dataset, out model achieves 0.5~1% higher on test Micro-Fl score, showing the effectiveness of applying our method under inductive settings.
Comparison with Other Related Works. To validate the effectiveness of our model, compare the proposed regularizer with two state-of-the-arr approaches on tackling oversmoothing. DropE- dge(Rong al., 2019) and PairNorm(ZZal & Akoglu 2019). For fair comparison, all approaches are adopted on vanilla-GCN with 2~8 layers and show the best performance on three transductive datasets respectively. As shown in Table 3 our regularizer achieves best performance on all three settings. As PairNorm is more suitable when a subset of the nodes lack feature vectors, it is less competitive the general settings.
Analysis. As we have stated above, the regularizer can be interpreted as the mean feature variance, which prevents different features from falling into the same subspace. method, we compute the mean pairwise distance (Eq. (19) of the last hidden layer of GCN and GAT, with and without regularizer on the Cora dataset. We show the result of models with 2-8 layers in Figure As we can observe, the feature variances and the accuracies of models with regularization are comparably higher than vanilla models with obvious gaps. Therefore, after applying the regular- izer, features more separated from each other, and the oversmoothing problem is alleviated.
testify the effect of our
CONCLUSION
In this paper, we develop unified graph convolution framework by establishing graph convolution filters with optimization problems in the graph Fourier space. We show that most convolution- based graph learning models are equivalent to adding carefully designed regularizers. Besides vanilla GCN, our framework is extended to formulating non-convolutional operations, attention- based GCNs and topology-base GCNs, which cover a large fraction of state-of-the- graph learn- tackling the oversmoothing prob-
ing models. On this basis, we propose novel regularization lem as a showcase, proving the effectiveness of designing new modules based on the framework. Through the unified framework, we provide general methodology for understanding and relating different graph learning modules, with new insights on tackling common problems and improving the generalization performance of current graph neural networks in the graph Fourier domain. Mean- while, the unified framework can also serve as a once-for-all platform for expert-designed modules on convolution- based approaches. We hope our work can promote the understandings towards graph convolutional networks and inspire more insights in this field.
REFERENCES
S. Bai, F. Zhang, and Torr. Hypergraph convolution and hypergraph attention. abs/1901. 08 150, 2019.
ArXiv,
Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv:1711.07553 2017.
arXiv preprint
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling. International Conference Learning Representations, 2018.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, S. Bengio, and Cho-Jui Hsich. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. 2019.

---

Under review as a conference paper at ICLR 2021
Jian Du, Shanghang Zhang. Guanhang Wu, José MF Moura, and Soummya Kar. Topology adaptive graph convolutional networks. CoRR, abs/ 100.1077,,,,,,, 2017.
M. Fey, J. E. Lenssen, F. Weichert, and H. Müller. Splinecnn: Fast geometric deep learning with continuous b-spline kernels. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 669-877, 2018.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. CoRR, abs/ 903.02228, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249-256, 2010.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs In Advances Neural Information Processing Systems, pp. 1024-1034 1034, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
NT Hoang and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv:905.009550, 2019.
Amol Kapoor, Aram Galstyan, Bryan Perozzi, Greg Ver Steeg, Hrayr Harutyunyan, Kristina Ler- man, Nazanin Alipourfard, and Sami Abu-El-Haija. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In International Conference on Machine Learn- ing, 2019.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations, 2017.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. International Conference on Learning Rep- resentations, 2018
Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learn- ing, 2019.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net-
work: Data-driven traffic forecasting, arXiv preprint arXiv:1707.01926 2017.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5115- 5124, 2017.
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference on Learning Representations, 2020.
Y. Rong, W. Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards the very deep graph convolutional networks for node classification. 2019.
M. Schlichtkrull, Thomas Kipf, Bloem, R. V. Berg, Ivan Titov, and M. Welling. Modeling rela- tional data with graph convolutional networks. In ESWC, 2018.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad Collective classification in network data. Al magazine, 29(3):93-93, 2008.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. CoRR, abs/181 11 1058688888 2018.
10

---

Under review as a conference paper at ICLR 2021
Martin Simonovsky and Nikos Komodakis. Dynamic conditindd filters in convolutional neu- ral networks on graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3693-3702, 2017.
Kiran K Thekumparampil. Chong Wang. Sewoong Oh, and Li-Jia Li. Attention-based graph neural network for semi-supervised learning. CoRR, abs//803.00 1803.037333 2018.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations, 2018.
Petar Veličkovié, William Fedus, William L Hamilton, Pietro Liô, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. International Conference on Learning Representations, 2019.
Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graph convolutional networks. International Conference on Machine Learming, 2019a.
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang. Xing Xie, and Tieniu Tan. Session-basse rec- ommendation with graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 346-353, 2019b.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful graph neural networks? CoRR, abs/1810.00826, 2018a
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. arXiv preprint Ntti:18006035530 2018b.
Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7370-7377, 2019.
Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. arXiv preprint arXiv:1999.12223. 2019.
Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14):110-ii19; 2017.
Marinka Zitnik, Monica Agrawal, and Jure Leskovee. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13)i457-i466, 2018.

---

Under review as a conference paper at ICLR 2021
APPENDIX
A. PROOFS OF THE DEFINITIONS
Definition 2 Vanilla GCNs. Let (())eev be the estimation of the imput observation ar ())),vv
A
low-pass filter:
X Ar
(26)
is the first-order approximation of the optimal solution of the following optimization:
min X
Σ - - =
Σ
aiyl =())
(27)
ijev
Proof. Let / denote the objective function. We have
= tr( - D( X)] + (
Then,
01 0x
2D X) +2LX.
we let 0x = 0:
D (D+ DX (1+ x - X.
As the norm of eigenvalues of Anw = I- Erw bounded by 1, I Lrw has eigenvalues in range [1, 3], which proves that + Erw is a positive definite matrix. Therefore.
X = (1 (r X.
(28)
Unfortunately, solving the closed-fomm form solution of (2.) less, we can derive simpler form, = (I (I- ) Ar X, via first-order Taylor approxiatiin which establishes the Definition.
(8) computationally expensive. Neverthe-
Definition Residual Connection. A graph convolution filter with residual connection:
X = X + cX,
(29)
where c > O controls the strength of residual connection, is the first-order approximation of the optimal solution of the following optimization:
min X
x()))) =||=(1)1B)+
Σasl -
(30)
TVV
WEV
Proof. Let /. denote the objective function. We have
1= tr
X) D(X X)] ctr ctr(X* + LX).
Then,
Øl
2D X) -
0x
If let 0x 0:
- (1--) L|X DX
X (1-e)I Znw] 1 X
X = [I + (Ō,w -el) -1x. X.
Therefore, the first-order approximation of the optimal solution is
X ~ I - (2 =f))
Anw + eX.
12

---

Under review as a conference paper at ICLR 2021
Definition 3' Concatenation. A graph convolution filter concatenating with the input signal:
X = Arw X + ex
(31)
the first-order approximation of the optimal solution of the following optimization:
min
-(10(()- <l(@()0))) + Σ (u]1=(())((()))
(32)
REV
J.jEV
where > O controls the strength of concatenation and è is the learning coefficients for the
catenated signal.
Proof. Let denote the objective function. We have
1= tr((1(( D( x)] xx)
Then,
Ø1 0x
22 X) +2X - 2DDX00
01 0: If we let
8X
D D - DDx00 Dx
(I (, X eX X.
With the help of the Kronecker product operator and first-order Taylor expansion, have
vec( ()) ( (I f... <((00((()))))))))) = veo
= =2II- @I (I _. + @ ())ve((X)
vec(2X -
+ x
vec(Ara X + +x00").
Definition Attention- GCNs. An attention-bate basiioo---addddd graph convolution filter:
=
(33)
the first-order approximation of the optimal solution of the following optimization:
min
|0(1) - x01) + -
s.l. Σ
Paj ViEV
(34)
iJEV
JEV
Proof. Let denote the objective function. We have
= tr( - x - (XX
Then,
Øl
2D -
ax
If we let 01 = 0:
(21-P)X
Similarly, prove that (21 - P) is positive definite matrix, with eigenvalues in range [1, 3]. Therefore,
X 2 -X-XX
~ PX.
13

---

Under review as a conference paper at ICLR 2021
Definition 5 & 6 Topology-base GCNs Due the fact that most of the topology-base models adopt non-convolutional operations like concatenation, we derive a more general objective function by combining with the non-convolutional operations:
min
|a((i) xi)l +
ON
Σ i.JEV
b |=(1)) di
-
x())
(35)
KEY
1=1
where Eizo 0z = og X. (1)) Raxd correspond to the learning weights for the ky hop neighborhood. Let I denote the objective function, we have:
0 and 2 0, = .,., If we let d be the feature dimension of
Øl 0X
DO X)+ Σ C DA ( X0(0))
By letting 0 = 0, we have: 8x
on X
EII, AK 1 on.
k=1
Therefore, with the help of the Kronecker product operator @ and first-order Taylor expansion, we have
Ja + Σ O (O0)))) @ (L. - A [vec))) (X) = (x)
(36)
k=1
We observe that Et (ar (O))) and (L, ( have non-negative eigenvalues. Due the property of the Kronecker product that the eigenvalues of the Kronecker product (A @ B) equal
to the product of cigenvalues of A and B. the filter (aoIn + E (015)) is proved to be a positive definite matrix. Therefore,
vec( vcc(X)= aalaa In
Σ oon ( @ (I, - -Atw -voc(X"
2 00l(2 Ln Σ (ay (a) (Q(K))) s0 (J, At veo X)
= anveec(2 - -a0)X
ax an II, - A XOI
let
w 2 In
ak (k) (o(
(37)
09
00
(A) (x) (e(k))T)
k == 1.2.....
(38)
we can denote the convolution filter as:
X anit (k)
(39)
As we have stated in the Section 2.2.2, although the learning weights has constrained expressive capability, it can be compensated by the following feature learning module. We omit the proofs of Definition 5 and 6, as they can be viewed as particular instances of 335
Definition 7 Regularized Feature Variance. Let & be the Kronecker product operator, vec(X) Innd be the vectorized signal X. Let Dx be a diagonal matrix whose value is defined by D (i, =
||o.ill A graph convolution filter with regularized feature variance:
vec(X (X) = (In @ [(a] + 022) - azArw az Dz
11 D
D D-1)- -vvo)( (40)
14

---

Under review as a conference paper at ICLR 2021
is equivalent to the optimal solution of the following optimization:
min
Σ LEV
- = + 02
E onj ( ( 0:3
1
Σ
[ 10..//122.1
F.g//
//1.//1. lll
LJEV
i.jdd
(41)
where 0, 02, 2 O. For computation efficiency, we approximate Dx with Dx we assume that a single convolution filter provides little effect the norm features.
Proof. Let denote the objective function. We have
1 1=
X( X)] + 0z ag (X*LX) agtr XD
'I
11--111 111")D21
d
Then,
a1
20: D(X X) 202LX - 223XD -1-
0x
we
0:
0x
[(an + a2) 022 D 0z3 -xg XD D x
With the help of the Kronecker product operator (,, we have
(I, @ [(() oz) a2Â.w] - 03 D 1 ) D ")vee(() veo(X),
(42)
By setting ag with small positive value, the filter Eq 42) is still a positive definite matrix. Therefore complete the proof.
Similarly, we can derive a simpler form via Taylor approximation. If we let:
= (an + 022 azArw:
B In.
(43)
C D- D D 1 11))
(44)
Then, the first-order approximation of Eq. 40 is summarized as:
voc(X) = "B"
+ D & C) -vec'
~ (2I T
@ A - DJ @ C')ee((X)
vec(2X AXB CXD)
Additionally, we can also derive a t-order approximated formulation:
(ec(X (I+ E I- A+ DJ @ CC)]")eec(X)
However, is computationally expensive to calculate the Kronecker product. Therefore, we consider utilizing iterative algorithm. For any 0 < k <
veel voc(X((+1)) (I +
Σ
I
&
A
D & vvcc))
& [ - (BT & A D C -
(B" 8 A CI DJ & ve) veo(X)
1=1
I (B" & A + D C) ( (x)) vec(X)
veo(X X X (k) AX B CX D).
(45)
B. REFORMULATION EXAMPLES
The reformulation examples of GCN derivatives presented in Table 4
15

---

Under review as a conference paper at ICLR 2021
Table Reformulation convolution- based graph neural networks. D and de in the attention- based modules are normalization coefficients.
Models
Non-Comv Module
dlentiin-busee Module
Topology-bae -baseed Module
GIN Xuu et al
2018a
Residual Connection
GraphSAGE Hamilton et al. 2017
Concatenation
RGCN Schlichtkaull 4.al
Concatemation
Ensk LW,
SplineCNN Tey et
P. ho(e.y)
Simonovsky Komodakis
Concatenation
P. ho(ee))
AGNN Thelumparumpil al.
2018
Pij
Esentous
MoNet Monti
2017
Concstenation
Pij djexp(- (iry pa)) S (ey pcl)
GAT Veličkovid al.
2018
Concatenation
Po d, Eisnan cxpleiat, (Feal
eaptolaly (aa,ll
Cluster GCN Chiang et al. 2019
Concatenation
p DíAlo Ading(Are))
SGC Wo al. 201 9m
P= DA
Hyper-Atten Bai et al. 2019
HWB-
APPNP Kliepera
2018
= 7.01
Kliepern
0,
TAGCN
2017
0g +/(k++1)
MixHop Kapoor
Concstenation
0g m1 1/3
Table 5: Dataset Statistics
Dataset
Cora
Citeseer
Pubmed
PPI
Nodes
2.708
3,327
19,717
56,944(24 graphs)
Edges
5,429
4,732
44,338
818,716
Features
1,433
3,703
500
50
Classes
3
(2(
Training Nodes
60
44,906(20 graphs)
Validation Nodes
500
500
500
6,514(2 graphs)
Test Nodes
1,000
1,000
1,000
5,524(2 graphs)
C. DATA STATISTICS AND EXPERIMENTAL SETUPS
We conduct experiments on four weal-world graph datasets, whose statistics are listed in Table For transductive learning, we evaluate our method on the Cora, Citeseer, Pubmed datasets, following the experimental setup (Sen et al. 2008). There are nodes per class with labels to be used for training and all the nodes' features generalization performance is tested on 1000 nodes with unseen labels. PPI (Zitnik & Leskovee) 2017) is adopted for inductive learning, which is protein-protein interaction dataset containing 20 graphs for training, for validation and for testing while testing graphs remain unobserved during training.
available. 500 nodes used for validation and the
To ensure a fair comparison with other methods, we implement our module without interfering the original network structure. In all three settings, two convolution layers with hidden dimen-
16

---

Under review as a conference paper at ICLR 2021
Table 6:
Test accuracy (%) transductive learning datasets with random slits. We report mean
values and standard deviations of the test accuracies over 100 random train/validation/eess splits.
Dataset
Citeseer
Cora
Pubmed
Gip Welling
2017
1.99
81.5+1..
77.8++2.9
GAT Veličković et al.
2018
7..4199
81.8113
78.7.2.3
MoNet MMoni et al
2017
71.2+2.0
81.31:3
78.6+2.3
GraphSAGE (Hamilton et al.
2017)
71.6+119
79.2+7.7
77,4+:222
GCN+reg (ours)
72.911.4
83.611.2
79.9:1.66
Table Training and test time on Cora. We report mean values in independent experiments. The best results are highlighted with boldface.
Method
Training Time
Training Time (ms / epoch)
Test Time (ms)
GCN
Kipf Welling
2017)
1.8
3.8
1.9
GAT (Veličković et al.
2018
5.4
8.5
3.3
AGNN [Thekumprammpil et al.
2018
5.3
7.9
3.2
APPNP (Klicpera al.
2018
9.8
14.2
13.6
GCN regs (ours)
5.7
8.0
3.2
sion h 64. We set on 0.2, 02 0.8 and 0g 0.05 for all four datasets. We apply La regularization with = 0.0005 and use dropout on both layers. For training strategy, we initialize weights using the initialization described in (Glorot Bengio 2010) and follow the method pro- posed in GCN, adopting an early stop if validation loss does not decrease for certain consecutive epochs. The implementations of baseline models are based on the SyToch--Goometee library (Fey & Lenssen 2019) in all experiments.
D. RANDOM SPLITS
As illustrated in (Shchur et al. 2018), using the same train/validationntest splits of the same datasets precludes fair comparison of different architectures. Therefore, we follow the setup in (Shchur jet al. (0188) and evaluate the performance of our model on three citation networks with random splits. Empirically, for each dataset, per class as the validation set, and the rest as the test set. For every model, we choose the hy- perparameters that achieve the best average accuracy Cora and CiteSeer datasets and applied to Pubmed dataset.
use 20 labeled nodes per class the training set, 30 nodes
Table shows the results three citation networks under the random split setting. As observe, our model consistently achieves higher performances on the datasets. On Citeseer, our model achieves higher accuracy than on the original split. On Cora and Pubmed, the test accuracies of our model are comparable to the original split, while most of the baselines suffer from serious decline.
E. TIME CONSUMPTION
As we have shown in Eq.20) 20 the computation of graph filter with the regularizer is greatly increased with Kronecker product and inverse matrix operations. Nevertheless, we approximate the filter with an iterative algorithm as stated in (5) and realize an efficient implement. To empirically testify the computation efficiency, we conduct experiments on Cora and report the training and test time of several GCN models on a single RTX 2080 Ti GPU, Due to the early stopping rule (see details in Appendix C). the training epoch for each module is different. The results shown in Table 7 As we can observe, when combining with vanilla GCNs, the training and test time of our model is similar GAT and AGNN and faster than APPNP.
17

---

Under review as a conference paper at ICLR 2021
Table 8: Ablation study the regularization strength. We report values and standard devia-
tions in 30 independent experiments. The best results are highlighted with boldface.
Regularization Strength
Citeseer
Cora
Pubmed
70.3+
81.5+10.5
79.0:10.4
0.01
71.710.6
83.0.0.5
79.2.10.6
0.05
72.2#0.4
83.62:.3
79.8+0.2
0.1
72.4+0.5
83.5+0.2
79.4+0.7
0.2
1..4
1.3
70.24:2.2
F. ABLATION STUDY
To analyze the effects of the regularization strength, we conduct experiments on three transductive datasets and present the results in Table g As we can observe, with reasonable choice of the regu- larization strength, our approach can achieve consistent improvement under all settings. However, when the regularization strength is too large, the training procedure becomes unstable and the model performance suffers from a severe decrease.
18


Reviews:
[{"review_id": "l9TmkZxM89_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper unifies several variants of the graph convolutional networks (GCNs) into a regularized quadratic optimization framework. Basically, the function to be optimized considers both to preserve node information and to perform graph Laplacian regularization, whose optimal solution gives a convolutional layer.\n\nThe unification is given by equations (3) and (18) and elaborated in section 3, which includes several methods including GCN, graph attentions, residual connections, concatenation, etc. This is not surprising: as a GCN layer (without activation) is a linear transformation, surely it is the optimum of a quadratic function. Broadly, any linear layer can be trivially formulated as a quadratic optimization problem. Still, I appreciate the authors' delicate work on unifying these diverse methods from an optimization perspective, which is useful and could lead to new methods.\n\nFrom a technical perspective, the main novelty is that the authors further extend this framework by adding another feature variance term, so that the learned features have a certain variance. This is similar to the idea of batch normalization. This is reasonable because GCN tends to correlate the learned features with the graph Laplacian embedding (the optimal solution of the 2nd term in the authors' framework).\n\nThis is interesting but empirical. I would like to see how this additional regularization can be equivalent to transforming the original graph with some formal arguments. Unfortunately, this technic is mainly introduced as a heuristic and more detailed analysis is missing.\n\nAs in any regularization framework, there is an additional parameter involved that is the regularization strength (\\alpha_3 in 21). Therefore the performance improvement is not surprising as the model is \"enlarged\". In the experiments or supplementary material, there should be a sensitivity study of this parameter.\n\nOn three citation graphs (that are commonly used to evaluate graph neural networks) and semi-supervised node classification tasks, the authors showed that the regularizer can bring marginal performance improvement.\n\nRegarding Clarity, there are some typos in several places and rarely used phrases.\n\nOverall, I don't feel excited after reading the article (although the contents are useful), as a large part of this work is on summarizing existing literature.  The \"new bit\" is mainly on the additional regularization term that is introduced as a heuristic.\n\nBased on the novelty, a more proper venue for publishing this work could be relevant journals. Overall this submission presents a borderline case and I recommend weak acceptance.\n\nAs a minor comment: Equation (21) why not set \\alpha_1=1?\n\n----\nAfter rebuttal:\n\nNovelty: my assessment remains the same. It is not non-trivial enough to combine several linear operators into a unified optimization framework. Although the unification is useful, it is not a major novelty.\n\nThank you for the additional experiments on testing the hyper-parameter. As you mentioned instability, it is worth to have some toy example to demonstrate the instability and study the cause of such instability and show how to avoid such instability using the proposed regularizer. Clearly (19) is bounded. When \\alpha_3 is large enough, the solution will be trivial.\n\nRegarding non-linearity: the authors' framework is for unifying a graph convolution operator (that is one layer in a graph neural network). Nonlinear activation is another operator. This is not a major problem from my perspective.\n\nOverall, I think this work has some value (although the novelty is not strong) and still recommend weak acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_oBYcrE87EB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a unified framework for graph convolutional networks by interpreting filters as regularizers in the graph Fourier domain.\nIn particular, this framework allows to establish the relationships between standard, attention-based and topology-based GNNs.\nFurthermore, the authors propose a regularization technique based upon the proposed framework which tackles the oversmoothing problem of GNNs, which achieves clear benefits on standard (small) benchmark datasets.\n\nThe paper is mostly well-written, although though to understand on first read.\nI especially liked that it tries to establish a systematic view of different GNN models and their relations, which is a welcome work in the field of graph representation learning (especially with the sheer amount of GNN models available in literature).\nIn my opinion, the proposed framework has the potential to improve our understanding of GCNs and inspire better models in return.\n\nOn the other hand, it is not exactly clear to me how the proposed regularization technique differs from PairNorm (which is build upon similar insights by preventing node embeddings from becoming too similar). I would very much welcome a discussion between key differences and similarities between the two approaches. Furthermore, the authors should consider comparing the proposed regularization technique against related approaches, e.g., PairNorm and DropEdge.\nOverall, the empirical evaluation feels a bit shallow by only evaluating on small benchmark datasets, but might be sufficient for a work that has mostly theoretical contributions.\n\nMinor comments:\n\n* It is not exactly clear to me how ECC can be viewed as an attention-based GNN since this operator learns a weight matrix conditioned on the edge features (instead of performing weighted normalization). Does this operator really fit into the proposed unified framework?\n\n* A GCN baseline is missing on the PPI dataset.\n\n============== Post Rebuttal Comments =================\n\nI would like to thank the authors for their insightful rebuttal and clarifications. Sadly, I cannot find the newly added section regarding the non-linearity analysis in the revised manuscript and therefore cannot judge the findings of the authors.\n\nHence, my rating will stay the same.", "extra_scores": {}, "extra_text": {}}, {"review_id": "V_aMIeRjCg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper shows that several graph networks (GCN, attention GCN, PPNP, residual) can be unified under a common framework of Laplacian-regularised optimisation. Subsequently, different types of regularisation are combined to propose a new method for graph transduction, which is then empirically evaluated.\n\nSignificance: Laplacian regularisation is a classical approach for formulating/justifying graph transduction algorithms (multiple papers by Mikhail Belkin and Xiaojin Zhu around 2004-06). It is interesting to see that so many graph networks can also be unified in the same framework. A unified framework does aid in both theoretical analysis and implementation of GCNs.\nHowever, the claims and derivation do not seem to account for the non-linear activation in the networks, and hence, significance of the work seems limited.\n\nQuality: As noted above, non-linearity is not considered which makes the derivation significantly simpler. Moreover, the first-order approximation is quite misleading since even the proof do not seem to consider non-linear activation. \nSince the proposed method combines multiple types of regularisation, it is expected to perform better than other networks. However, it is not clear if the training time increases due to the complex regularisation.\n\nClarity and orginality: The paper is otherwise well written / organised, and the theoretical contributions (although technically straightforward) seem original and somewhat interesting.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TI7mf0Uooc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a unified framework for graph convolutional neural networks based on regularized optimization, connecting different  variants of graph neural networks including vanilla, attention-based, and topology-based approaches. The authors also propose  a novel regularization technique to approach the oversmoothing problem in graph convolution. Experiments on the standard settings of node classification on Citeseer, Cora, and Pubmed prove the effectiveness of the proposed regularization techniques. \n\nOverall, this is a very interesting paper, proposing a unified framework for different variants of convolution-based graph neural networks. However, I also have a few concerns:\n\n(1) The proposed framework is mainly designed for GNNs without considering the nonlinear transformation matrix. What if we have to consider the nonlinear transformation? Is the whole framework able to unify different GNNs?\n\n(2) In the case of linear GNNs (without nonlinear transformation matrix), it is actually not surprising formulating GNNs as a regularized optimization problem. Such a regularization framework has already been discussed in the original GCN paper (Kipf et al. 2016). \n\n(3) In the case of linear GNNs, the overall framework is also very similar to the traditional label propagation framework (Zhou et al. Learning with Local and Global Consistency). Could you explain the difference?\n\n(4) The new novel regularization technique seems to be similar to the one proposed in PairNorm (Zhao et al. 2020). Could you also explain the difference?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four reviewers have reviewed and discussed this submission. After rebuttal, two reviewers felt the paper is below acceptance threshold. Firstly, Rev. 1 and Rev. 2 were somewhat disappointed in the lack of analysis regarding non-linearities despite authors suggested this was resolved in the revised manuscript, e.g. Rev. 2 argued the paper without such an analysis is too similar to existing 'linear' models, e.g. APPNP, SGC, and so on. While Rev. 3 was mildly positive about the paper, they also noted that combining several linear operators is somewhat trivial. Overall, all reviewers felt there is some novelty in the proposed regularization term but also felt that contributions of the paper could have been stronger. While AC sympathizes with this submission and hopes that authors can improve this work, in its current form it appears marginally below the acceptance threshold.
Decision: Reject