Title: Towards the Universal Learning Principle for Graph Neural Networks
Venue: NeurIPS 2023
Accepted: no
Keywords: Graph Neural Network

Graph Filter

Learning Framework
Abstract: Graph neural networks (GNNs) are currently highly regarded in graph representation learning tasks due to their significant performance. Although various propagation mechanisms and graph filters were proposed, few works have investigated their rationale from the perspective of learning. In this paper, we elucidate the criterion for the graph filter formed by power series, and further establish a scalable regularized learning framework that theoretically realizes GNN with infinite depth. Following the framework, we introduce Adaptive Power GNN (APGNN), a deep GNN that employs exponentially decaying weights to aggregate graph information of varying orders, thus facilitating more effective mining of deeper neighbor information. Moreover, the multiple $P$-hop message passing strategy is proposed to efficiently perceive the higher-order neighborhoods. Different from other GNNs, the proposed APGNN can be seamlessly extended to an infinite-depth network. To clarify the learning guarantee, we theoretically analyze the generalization of the proposed learning framework via uniform convergence. Experimental results show that APGNN obtains superior performance compared to state-of-the-art GNNs, highlighting the effectiveness of our framework.

Abstract

Graph neural networks (GNNs) are currently highly regarded in graph representation learning tasks due to their significant performance. Although various propagation mechanisms and graph filters were proposed, few works have investigated their rationale from the perspective of learning. In this paper, we elucidate the criterion for the graph filter formed by power series, and further establish a scalable regularized learning framework that theoretically realizes very deep GNN. Following the framework, we introduce Adaptive Power GNN (APGNN), a deep GNN that employs exponentially decaying weights to aggregate graph information of varying orders, thus facilitating more effective mining of deeper neighbor information. Moreover, the multiple P -hop message passing strategy is proposed to efficiently perceive the higher-order neighborhoods. Different from other GNNs, the proposed APGNN can be seamlessly extended to an infinite-depth network. To clarify the learning guarantee, we theoretically analyze the generalization of the proposed learning framework via uniform convergence. Experimental results show that APGNN obtains superior performance compared to state-of-the-art GNNs, highlighting the effectiveness of our framework.

1 Introduction

Recently, Graph Neural Networks (GNNs) have shown commendable performance on numerous graph representation learning tasks. In addition, GNNs have been introduced in a variety of application tasks, such as recommendation systems [7, 11, 37], computer vision [4, 13, 23], and traffic forecasting [8, 9]. The fundamental part of GNN is the design of the propagation mechanism or the graph filter [5, 12, 27, 32, 34]. GNNs can be categorized into two groups based on the approach of formulation. Spatial-based GNN formulates propagation mechanisms through the direct aggregation of spatial features. As one of the most simple GNNs, Graph Convolutional Network (GCN) [15] designs graph convolutional layer via aggregating one-hop information on the graph. Graph Attention Network (GAT) [30] learns node relationships using an attention mechanism, enhancing the scalability of the network. For extension of inductive learning, GraphSAGE [10] employs various pooling operations as aggregation functions. Liu et. al proposed DAGNN, which integrates information from multiple receptive fields for adaptive propagation [19]. Spectral-based GNN designs graph filters by constructing filter functions in the graph Fourier domain, which aims to find a proper transformation of the graph spectrum. Chebynet constructs the localized graph filter with Chebyshev polynomial [3]. From the view of the spectrum, GCN could be seen as a Chebyshev filter with first-order truncation [15]. To construct deeper GNN, Personalize PageRank method is employed to design graph filter [16]. GNN-LF/HF [36] concludes various designs of graph filters and constructs the graph filter through a graph optimization framework.

Despite their success, few studies have explored the general rule for devising GNNs from the 36

perspective of learning. In this paper, we start from the graph filter formed by power series and 37

MLP

+

+

Convolution

Classifier

Observed samples

Learning

Principle

,

APGNN

with

discuss what makes a legitimate graph filter for the construction of deep GNN. A learning principle 38

is then proposed to summarize the rule of formulating a graph filter. Following this, we propose 39

Adaptive Power Graph Neural Network (APGNN), which adaptively learns the task-specific graph 40

41

filter for node representation learning.

The main idea of APGNN is depicted in Figure 1.

The

parameterized graph filter is designed with regularization of the exponential decay rate. A multiple 42

P -hop strategy is applied to enhance the capacity of perceiving the higher-order neighborhoods. 43

Furthermore, the generalization bound of APGNN is presented with the setting of the continuous 44

graph, which provides a learning guarantee for the proposed principle theoretically. 45

We conduct evaluations on five benchmark datasets on node classification tasks. The experimental 46

results suggest the superiority of the proposed method over the existing GNNs. The theoretical 47

analysis is also validated via the empirical study. 48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

2 Preliminaries

Notations. Suppose we have an undirected graph G = ( V , E , A ) with node set V and |V| = n . A ∈ R n × n denotes the adjacency matrix indicating the edges in E . Assuming that the self-loops are contained in the graph, i.e., a ii = 1 . Let X = [ x 1 , x 2 , · · · , x n ] ⊤ ∈ R n × d be the graph signals (or features) of the nodes. We use notation [ n ] ≜ { 1 , 2 , · · · , n } for n ∈ N + . Assume that the label of x i is y i ∈ Y for all i = [ n l ] , where n l ≤ n is the number of labeled samples.

Graph Neural Networks. We introduce some essential concepts in GNNs. Let d i = ∑ n j =1 A ij be the degree of i -th node, so the degree matrix of A can be defined as D = diag( d 1 , d 2 , · · · , d n ) . The symmetrically normalized Laplacian is L = I -˜ A , where ˜ A ≜ D -1 / 2 AD -1 / 2 is normalized adjacency matrix. Consider the eigen-decomposition L = UΛU ⊤ , where Λ = diag( λ 1 , · · · , λ n ) is the diagonal matrix of eigenvalues, and U = [ u 1 , · · · , u n ] represents the eigenvectors associated with the eigenvalues. Note that ˜ A shares the same eigenvectors with L .

Spectral convolution on graphs is defined as the following transformation [15, 28]:

where g ( · ) : [0 , 2] ↦→ R is called filter function and g ( Λ ) = diag( g ( λ 1 ) , · · · , g ( λ n )) . The common approach in GNNs is to apply polynomial functions as the filters [3, 12, 15], which leads to U g ( Λ ) U ⊤ = g ( L ) . Therefore, spectral convolution is usually written as g ∗ X = g ( L ) X . The graph representation paradigm in GNN is generally expressed as follows:

where Z ∈ R n × c denotes the node representation, and f ( · ) represents a feature extractor such as 66 multi-layer perceptions (MLPs). 67

3 Learning Principle for GNNs 68

69

3.1 The principle of devising graph filters

Current studies suggest a significant relationship between the performance of GNN and its graph filter 70 [16, 19]. Predominantly, the general graph filters are characterized by polynomials associated with 71 the adjacency matrix ˜ A (or Laplacian matrix L ), i.e., g ( L ) = ∑ K k =0 θ k ˜ A k . However, the existing 72 methods still meet the issue that the depth of GNN is limited. The reason for this phenomenon 73 is that these GNNs are inconsistent with their "infinite-depth" version. That is, the corresponding 74 models lose some essential properties as the depth K →∞ . Consequently, the depth of the models 75 is restricted. To address this issue, it is necessary to study the properties of GNNs with infinite depth. 76 Therefore, we explore the graph filter reformulated as power series: 77

First of all, a well-defined graph filter represented as equation 3 must be convergent. Consequently, it 78 becomes essential to investigate the properties that the coefficients θ k should exhibit. The following 79 lemma provides appropriate constraints for the coefficients of the graph filter. 80

81

82

Lemma 1. Let { a k } and { γ k } be the real number sequences, where γ ∈ ( -1 , 1] and k ∈ N . Then ∑ ∞ k a k γ k converges uniformly and absolutely if and only if the series ∑ ∞ k a k converges absolutely.

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

As a direct corollary, the weights of the graph filter (i.e., θ k ) should satisfy the following theorem.

Theorem 1. Let ˜ A = D -1 / 2 AD -1 / 2 be the normalized adjacency matrix of a graph G with spectral radius ρ ( ˜ A ) ≤ 1 . The matrix series ∑ ∞ k =0 θ k ˜ A k converges uniformly and absolutely if and only if the series ∑ ∞ k =0 θ k converges absolutely.

The proofs are shown in Appendix. Theorem 1 offers a sufficient and necessary condition for the convergence of graph filters formed by power series. Specifically, the condition requires the existence of a finite real number M ≥ 0 ,

Therefore, an arbitrary graph filter formed by power series should satisfy the above convergence condition, which gives the first requirement while designing GNN. Apart from convergence, we expect the graph filter to possess good analytic properties such as smoothness. To this end, Lipschitz continuity should be considered the second requirement of the graph filter. Let g ( · ) be an L -Lipschtiz continuous function, meaning that

This property indicates the stability or robustness of the model [6, 24]. If the graph is contaminated and its eigenvalues are perturbed by at most ϵ , Lipschitz continuity ensures the perturbation of the graph-filtered result is at most Lϵ . For instance, considering g ( λ ) = ∑ ∞ k =0 (1 -λ ) k /k 2 , which is convergent, yet the Lipschitz condition does not satisfy for λ closed to zero. Therefore, this graph filter might be sensitive to the input graph. Subsequently, we conclude the following criterion.

To enhance the scalability of the model, we define θ as a learnable parameter (though its dimension is infinite). In this way, (6) gives a regularized learning framework for GNN. Therefore, for a K -order polynomial graph filter g K θ ( λ ) = ∑ K k =0 θ k (1 -λ ) k , which is what we can implement in practice, the condition (6) should be satisfied to keep the consistency with its infinitely deep version g ∞ θ ( λ ) = ∑ ∞ k =0 θ k (1 -λ ) k . We will present the applications of this criterion in this section, and further analyze the learning guarantee with generalization in section 4.

3.2 Related works

In this subsection, we investigate the relationship between our learning framework and several wellknown Graph Neural Networks (GNNs), focusing on the design of graph filters. Our findings indicate that these GNNs are all special cases of our learning framework, which are summarized in Table 1.

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

GCN/SGC [15, 33]. Graph convolutional network (GCN) aims to learn a node representation by stacking multiple graph convolutional layers. In each layer, GCN applies first-order Chebyshev approximation as the graph filter followed by a fully connected layer. For simplicity, we analyze one-layer GCN, which is formulated as Z = σ ( ˜ AXW ) , where W is a learnable weight matrix for linear transformation. Therefore, the graph filter of one-layer GCN is g GCN ( L ) = I -L = ˜ A , or a trivial matrix power series:

It should be noted that this equation satisfies the condition described in (6).

SGC is a simplified version of GCN that eliminates the activation function and applies a single linear projection to extract features. This simplification reduces the multiple-layer GCN into a more concise model as Z = ˜ A K XW . Similarly, the graph filter of SGC can be represented as:

Both GCN and SGC use a monomial to construct the graph filter. Therefore, in the viewpoint of spectral-GNN, their graph filters are too simple to capture the spectral characteristic. Besides, the small eigenvalue vanishes when K becomes very large, leaving only the largest eigenvalue, which leads to the well-known over-smoothing problem [17].

PPNP [16]. PPNP uses Personalized PageRank as the graph filter, which balances local information preservation and the utilization of high-order neighbor information. The model of PPNP is Z = α ( I -(1 -α ) ˜ A ) -1 H = ( I + β L ) -1 H , where H = f ( X ) is a two-layer MLPs and β = 1 /α -1 . Hence, the graph filter of PPNP is g PPNP ( L ) = ( I + β L ) -1 . Considering its Taylor series, we have

where θ k = β k / (1 + β ) k +1 . It is straightforward to validate that ∑ ∞ k =0 θ k = 1 , and thus the convergence requirement (4) holds. Moreover, the Lipschitz condition is easily verified. Thus PPNP satisfies the criterion of (6). However, the performance of PPNP is heavily dependent on the hyperparameter β , which must be carefully tuned to achieve optimal performance.

DAGNN [19]. DAGNN adaptively adjusts the weight of information aggregation from different neighbors to solve the over-smoothing problem. It designs a parameterized graph filter formulated as a K -order polynomial:

where θ k is the learnable parameter with bounded constraint. Due to this adaptive learning strategy, DAGGN is able to learn a graph filter more suitable for node classification. The empirical studies suggest DAGNN works well with a proper K . However, as K → ∞ , the constraint 0 ≤ θ k ≤ 1 cannot guarantee the convergence of the graph filter. It indicates that DAGNN is 'inconsistent' with its infinitely deep version. Therefore, it can not be naturally extended to significantly deep GNN.

3.3 Instantiation: Adaptive Power Graph Neural Network

We now introduce a novel GNN following the framework in section 3.1, called Adaptive Power GNN (APGNN). We first consider the following graph filter parameterized by β with the form:

where the coefficient of the power series θ k = β k α k , with hyper-parameter α ∈ (0 , 1) ensuring the 143 convergence. Immediately, we check the condition of Lemma 1. 144

Hence, the power series converges on [0 , 2] absolutely and uniformly. Similarly, the associated matrix 145 series g ∞ β ( L ) = ∑ ∞ k =0 β k α k ˜ A k also converges uniformly and absolutely by Theorem 1. Moreover, 146 g ∞ β ( · ) is α (1 -α ) -2 -Lipschitz. To see this, for any | β k | ≤ 1 and 1 -λ ∈ ( -1 , 1] , we have 147

which implies the Lipschitz continuous property. Thus, this graph filter fits the requirement of the 148 proposed criterion. However, the model with this graph filter is unavailable in practice as the number 149 of parameters to be learned is infinite. The K -order truncated polynomial is utilized for substitution, 150 i.e., g K β ( L ) = ∑ K k =0 β k α k ˜ A k . We evaluate the approximation via the upper bound of K -order 151 truncation error: 152

which uniformly holds for ∀ λ ∈ [0 , 2] . Likewise, the approximation error of matrix series is given by 153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

where λ i denotes the i -th eigenvalue of L . This upper bound is independent of the given graph, which can be controlled via tuning α and K . The higher K and smaller α yield a better approximation to the exact graph filter g ∞ β ( · ) . Nevertheless, the small α tends to limit the capability of the graph filter. Extremely, α → 0 gives a trivial function g K β ( λ ) = β 0 . This suggests that α should be elaborately tuned to improve the performance.

Though the aforementioned graph filter is primarily motivated via spectral analysis, we can still present the spatial perspective explanation for its design. Existing GNNs aggregate the neighbor information of different hops with certain weights, which could be either manually assigned or learned adaptively. Typically, methods like GPR-GNN [2] and DAGNN [19] that learn the aggregation weight, tend to treat the neighbor's information of different hops equally. That is, the k -hop's weight are assigned with θ k = O (1) for each k ∈ [ K ] . However, it is shown in the previous research that the propagation with the very high-order neighbor potentially leads to the over-smoothing issue [25, 33]. The current methods magnify this flaw of the high-order graph since they cannot distinguish the significance of the information of different hops. This motivates the design of the decay rate in APGNN, i.e., we employ weights with exponential decaying rate by assigning θ k = O ( α k ) for some 0 < α < 1 . This approach emphasizes the contribution of lower-order neighbors and restricts the over-weighting of the information from high-order neighbors due to θ k → 0 with k →∞ . Therefore, it provides more effective aggregation and thus enhances the model's scalability.

To take a further step in the construction of a deep GNN, we introduce a multiple P -hop strategy for 172 the graph filter of (11), which effectively extends the utmost neighborhood range that the graph filter 173 can perceive by P times. Consider a different perspective regarding the construction of a filter with 174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

the utmost order T = KP . The previous methods can be viewed as a one-hop graph filter by setting P = 1 . For P > 1 , the graph filter is able to aggregate information from a larger neighborhood in the same order. In addition, we will illustrate the advantages of this strategy from the perspective of generalization in the following section.

Summarizing the above analysis, we present the following comprehensive architecture of APGNN:

In short, APGNN incorporates the benefits from the decay rate α that exponentially suppresses the information of extremely high-order neighbors and the multiple P -hop strategy to enlarge receptive fields. These approaches make it possible to realize a sufficiently deep GNN.

4 Generalization analysis

The theoretical analysis of GNN's generalization is widely studied. [31] provides the generalization result of the algorithmic stability of GCN in the discrete graph setting. In contrast, [14] shows the convergence and stability guarantee over the random and continuous graph. In this section, we will present the uniform generalization bound of the proposed GNN learning framework under the continuous setup.

We first introduce some notations for later discussion. Denote x ∈ X as any samples from the input space X (we generally set X as a subset of R d ). Let ρ ( · ) be a probability measure defined over X . Assume x j is the j -th coordinate of x ∈ X and E [ x 2 j ] ≤ c 2 X for any j ∈ [ d ] . To describe the graph relation between each pair ( x , x ′ ) over X × X , we define a continuous graph function A ( · , · ) : X × X ↦→ R + , and its corresponding degree function is

Different from the setting of [18, 26], we assume 0 ≤ A ( x , x ′ ) ≤ c U , and 0 < c L ≤ d ( x ) for any x , x ′ ∈ X . Therefore, we can define the symmetric normalized graph:

Then the corresponding normalized Laplacian is L = I -˜ A , where I indicates the identity operator over X . For a graph filter function g θ ( λ ) = ∑ K k =0 θ k (1 -λ ) k , graph convolution of the continuous graph is defined as the following integral operator:

where ˜ A k = ˜ A k -1 ◦ ˜ A denotes k -order composition of integral operator with ˜ A 0 = I . Note we have ∑ K k =0 θ k ∥ ˜ A ∥ ≤ ∥ θ ∥ 1 ≤ M for any K ∈ N , indicating ∑ ∞ k =0 θ k ˜ A is absolutely summable. This guarantees the existence of graph filter on the continuous graph when K →∞ . For convenience in understanding, we provide the analysis on a simplified GNN, where we consider a semi-supervised learning task with two classes, i.e., y i ∈ Y ≜ {-1 , 1 } , and utilize linear feature extractor f ( X ) = w ⊤ X . Note that we can still extend our result for f ( X ) = MLP( X ) and multi-class cases using the techniques proposed in [1]. With the above setting, the hypothesis set over is described as

However, the integral in each hypothesis h ∈ H X is intractable since the underlying graph function and the data distribution are unknown. Therefore, we should use the 'empirical version' of the hypothesis to estimate h ∈ H X . For this reason, we introduce the hypothesis set defined over the observed samples S and graph G :

Define the generalization error and the empirical error [21] as follows 211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

We have the following theorem on the generalization of the proposed learning paradigm.

Theorem 2. Suppose g θ ( · ) is L M -Lipschitz. Let h w , θ ∈ H X and h w , θ ∈ H S share the same parameter ( w , θ ) . Then there exists a constant C > 0 related to the graph function, with the probability at least 1 -δ , the following inequality holds.

The proof is given by excess risk decomposition, shown in Appendix. The notation " ≲ " denotes "less than or approximately equal to the right-hand side" and guarantees an approximation error of at most O ( √ log(1 /τ ) n l ) with a probability of at least 1 - O ( τ ) . We remind readers the important difference between R ( h w , θ ) and ˆ R ( ˆ h w , θ ) . The former term measures the population error over the whole input space with the continuous graph filter g θ L . In contrast, ˆ R ( ˆ h w , θ ) is the empirical risk (i.e., training risk) on the sample set S with the discrete graph filter g θ ( L ) . h w , θ shares the same learning parameter with ˆ h w , θ . Therefore, the minimization of the right-hand-side of (23) w.r.t ( w , θ ) reduces the upper bound of the population error.

We observe the first term of generalization bound is of order O (( dn -1 l log K ) 1 / 2 ) , which outlines the model's complexity. Although it becomes infinity when K →∞ , the growth of this term is extremely slow as K increases. In practice, we generally set K < n since the neighbor information beyond n -hops is redundant, restricting the complexity away from infinity. Therefore, the generalization of the model is rigorously guaranteed for sufficiently large K , which allows us to construct significantly deep GNN in the proposed framework. We can obtain a more precise estimation for a certain model. In the following proposition, we unveil the generalization of APGNN as a direct application of Theorem 2.

Proposition 1. Let β ∈ R K and g K β ( λ ) = ∑ K k =0 β k α k (1 -λ ) k where 0 < α < 1 and ∥ β ∥ ∞ ≤ 1 . with the probability at least 1 -δ , the following inequality holds.

In (24), the complexity term becomes O ( √ log K (1 -α K )) with K = ⌊ T/P ⌋ , which is relatively tighter than O ( √ log K ) . For this term, we promote further discussion with P -hop. Since it takes ⌊ T/P ⌋ steps to reach the T -order graph, the term becomes O ( √ log ⌊ T/P ⌋ (1 -α ⌊ T/P ⌋ )) . It is observed that the term decreases as P increases. Therefore, the appropriate P reduces the bound, explaining the mechanism of the P -hop method. On the other hand, larger α leads to a higher bound. From the point of spatial view, the information from high-order neighbors is underused, which limits the range of the graph filter. Thus α should be moderate to leverage the generalization and the capability of the model.

5 Experiment

In this section, we conduct node classification experiments on various benchmark datasets to evaluate the performance of APGNN. Specifically, we compare our method with state-of-the-art methods and display the corresponding learned graph filter on different data sets. Moreover, to validate the theoretical analysis, the influence of parameters K , α , and P is also investigated in experiments.

248

249

250

251

252

253

254

255

256

257

258

259

260

5.1 Experiment Setup

Datasets. We perform experiments on five benchmark datasets commonly used in node classification tasks. 1). Cora, Citeseer, Pubmed[29, 35] : These are three standard citation networks where each node is a paper and each edge is a citation link. 2).Wiki-CS[20] : This dataset defines the computer science articles as nodes, while the hyperlinks are edges. 3). MS Acadamic[16] : The nodes represent the author and the edges represent the co-authorships. A co-authorship Microsoft Academic Graph, where the nodes are the bag-of-words representation of the papers' abstract and edges are co-authorship. The data statistics and their partitions are presented in Appendix.

0.0

0.3

0.6

0.9

1.1

1.4

1.7

2.0

0.1

1.4

2.6

3.9

5.2

6.4

g

(

)

Citeseer

Pubmed

Wiki

MS

(a)

0.0

0.3

0.6

0.9

1.1

1.4

1.7

2.0

0.9

1.6

2.4

3.1

3.8

4.6

g

(

)

Citeseer

Pubmed

Wiki

MS

(b)

Baselines. To evaluate the effectiveness of APGNN, we compare it with the following baseline models: 1) MLP [22], a traditional method that does not use graphs, 2) GAT [30] and GraphSAGE [10], spatial methods that aggregate neighborhoods' information, and 3) ChebNet [3], GCN [15], SGC [33], PPNP, APPNP[16], GNN-LF (iteration form), GNN-HF (iteration form) [36], and DAGNN [19], spectral methods analyzing GNNs with graph Fourier transform.

Settings. We conducted 10 runs for each method on each dataset, with a hidden dimension of 64 . For 261 all compared methods, their parameter settings follow the previous practices [19, 36]: the dropout 262 rate is 0 . 5 except for Cora, which had a rate of 0 . 8 . Furthermore, the learning rate is 0 . 01 for Cora, 263 Citeseer, and Pubmed, but 0 . 03 for Wiki-CS and 0 . 02 for MS-Academic, while the weight decay is 264 0 . 005 for Cora and Pubmed, 0 . 02 for Citeseer, 0 . 0005 for Wiki-CS, and 0 . 00525 for MS-Academic. 265 We fix the polynomial order K to 10 in ChebNet, APPNP, GNN-LF, GNN-HF, DAGNN, and APPNP. 266 The best hyperparameters we choose for APGNN are presented in Appendix. To ensure a fair 267

comparison with the compared methods, we also applied our optimal hyperparameters to them, 268 selecting the maximum value to display. 269

0

2

4

6

8

10

12

14

16

18

20

Parameter

K

60

65

70

75

80

85

Accuracy (%)

Citeseer ACC

Pubmed ACC

(a)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9 0.99

Parameter

60

65

70

75

80

85

Accuracy (%)

Max: 83.98%

Max: 72.90%

Max: 80.40%

Citeseer ACC

Pubmed ACC

(b)

1

2

3

4

5

6

Parameter

P

(fixing T)

70

72

74

76

78

80

82

84

Accuracy (%)

Min: 82.75%

Min: 70.78%

Min: 80.03%

Citeseer ACC

Pubmed ACC

(c)

5.2 Analysis

Node Classification. As the metric for evaluation, the mean accuracy of 10 runs is used. We compare the performance of APGNN with other methods on five benchmark datasets. Experiment results are reported in Table 2. We can observe that APGNN achieves the highest accuracy across all five datasets, demonstrating its superior performance.

Learnable Graph Filters. Figure 2 shows the graph filters learned on various datasets via APGNN. When the parity of P varies, the graph filter has a distinctive shape. However, their shapes exhibit minimal impact on their accuracy regardless of the parity of P according to the experiment results. Moreover, the graph filters of each dataset are plotted in Appendix, more details are included in Appendix. Our results show that the graph filters learned from different datasets vary in detail, even when their parameters have similar parity, demonstrating the efficacy of APGNN in learning task-specific graph filters.

Polynomial Order K . To gain insight into the role of polynomial order K , we conduct the experiment tuning K in { 1 , 2 , ..., 20 } on Cora, Citeseer, and Pubmed dataset. Our theoretical analysis supports the observation that a small K can result in a large truncation error, leading to a low accuracy rate. It can be observed that the accuracy rate has little promotion when K is larger than 10 , although at the cost of high computational resources.

Decay Rate α . Figure 3 (b) depicts the accuracy curve corresponding to various α values ranging from 0 . 1 to 0 . 9 and 0 . 99 on Cora, Citeseer and Pubmed datasets. As α decreases, the classification accuracy initially increases and then declines sharply. This phenomenon verifies the theory that the truncation error decreases as α decreases, but it leads to a trivial function when α is extremely small.

P -hop strategy. We investigate the accuracy associated with varying parameters P taken from the set { 1 , 2 , 3 , 4 , 5 , 6 } when fixing T = KP = 60 . As we can see in Figure 3 (c), the accuracy increase when P > 1 . This phenomenon can be attributed to the fact that the generalization bounding decreases when P increases, which suggests that the P -hop strategy can effectively explore deeper information with the same computational complexity.

6 Conclusion

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

This paper proposes a universal learning principle for a valid construction of GNN. An instantiation 297 named APGNN is proposed to verify the effectiveness of our framework. APGNN employs a decay 298 rate and a multiple P -hop strategy to learn the coefficients adaptively, which can efficiently aggregate 299 the information from high-order neighbors. We present a theoretical analysis of the generalization 300 capabilities of both our framework and APGNN, which provides a learning guarantee. Comprehensive 301 experiments show the superior performance of APGNN. In the future, it is worth exploring diverse 302 graph filters based on the proposed principle. As shown in the generalization analysis, the upper 303 bound of the model complexity relies on O ( √ log K ) . How to devise the GNN with complexity free 304 of the hyperparameter K is also a meaningful research direction. 305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

References

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 30, page 6241-6250, 2017.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 29, 2016.

Leyan Deng, Defu Lian, Chenwang Wu, and Enhong Chen. Graph convolution network based recommender systems: Learning guarantee and item mixture powered strategy. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 35, pages 3900-3912, 2022.

Jiarui Feng, Yixin Chen, Fuhai Li, Anindya Sarkar, and Muhan Zhang. How powerful are k-hop message passing graph neural networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 35, pages 4776-4790, 2022.

Fernando Gama, Joan Bruna, and Alejandro Ribeiro. Stability properties of graph neural networks. IEEE Transactions on Signal Processing , 68:5680-5695, 2020.

Francesco Giuliari, Geri Skenderi, Marco Cristani, Yiming Wang, and Alessio Del Bue. Spatial commonsense graph for object localisation in partial scenes. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19496-19505, 2022.

Kan Guo, Yongli Hu, Yanfeng Sun, Sean Qian, Junbin Gao, and Baocai Yin. Hierarchical graph convolution network for traffic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , volume 35, pages 151-159, 2021.

Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatialtemporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , volume 33, pages 922-929, 2019.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 30, 2017.

Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision GNN: An image is worth graph of nodes. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 35, pages 8291-8303, 2022.

Mingguo He, Zhewei Wei, zengfeng Huang, and Hongteng Xu. Bernnet: Learning arbitrary graph spectral filters via bernstein approximation. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 34, pages 14239-14251, 2021.

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, YongDong Zhang, and Meng Wang. LightGCN: Simplifying and powering graph convolution network for recommendation. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , page 639-648, 2020.

Nicolas Keriven, Alberto Bietti, and Samuel Vaiter. Convergence and stability of graph convolutional networks on large random graphs. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 33, pages 21512-21523, 2020.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017.

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In Proceedings of the International Conference on Learning Representations (ICLR) , 2019.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2018.

Shaojie Li, Sheng Ouyang, and Yong Liu. Understanding the generalization performance of 359 spectral clustering algorithms. arXiv preprint arXiv:2205.00281 , 2022. 360

361

362

363

Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD) , page 338-348, 2020.

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

Péter Mernyei and C˘ at˘ alina Cangea. Wiki-CS: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901 , 2020.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Sankar K Pal and Sushmita Mitra. Multilayer perceptron, fuzzy sets, and classification. IEEE Transactions on neural networks , 3(5):683-697, 1992.

Yitong Pang, Lingfei Wu, Qi Shen, Yiming Zhang, Zhihua Wei, Fangli Xu, Ethan Chang, Bo Long, and Jian Pei. Heterogeneous global graph neural networks for personalized sessionbased recommendation. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM) , page 775-783, 2022.

Patricia Pauli, Anne Koch, Julian Berberich, Paul Kohler, and Frank Allgöwer. Training robust neural networks using lipschitz bounds. IEEE Control Systems Letters , 6:121-126, 2021.

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Lorenzo Rosasco, Mikhail Belkin, and Ernesto De Vito. On learning with integral operators. Journal of Machine Learning Research (JMLR) , 11(2), 2010.

Aliaksei Sandryhaila and José M. F. Moura. Discrete signal processing on graphs: Graph filters. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6163-6166, 2013.

Aliaksei Sandryhaila and José M. F. Moura. Discrete signal processing on graphs: Graph fourier transform. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6167-6170, 2013.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI Magazine , 29(3):93, 2008.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018.

Saurabh Verma and Zhi-Li Zhang. Stability and generalization of graph convolutional neural networks. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD) , pages 1539-1548, 2019.

Xiyuan Wang and Muhan Zhang. How powerful are spectral graph neural networks. In Proceedings of the International Conference on International Conference on Machine Learning (ICML) , volume 162, pages 23341-23362, 2022.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In Proceedings of the International Conference on International Conference on Machine Learning (ICML) , 2019.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In Proceedings of the International Conference on Learning Representations (ICLR) , 2019.

Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the International Conference on International Conference on Machine Learning (ICML) , page 40-48, 2016.

Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of The International World Wide Web Conference (WWW) , page 1215-1226, 2021.

[37] Stefano Zorzi, Shabab Bazrafkan, Stefan Habenschuss, and Friedrich Fraundorfer. PolyWorld: 410 Polygonal building extraction with graph neural networks in satellite images. In Proceedings 411 of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 412 1848-1857, 2022. 413

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a spectral GNN with geometrically decaying weight $\\alpha^k$ and $P$-hop polynomial basis. They study the Lipschitzness and generalization bound of the model. Experiment results demonstrate the good performance of the proposed method over baselines\n\nSTRENGTHS:\n(+) Interesting generalization analysis.\n\nWEAKNESSES:\n(-) The recent spectral GNN literature is not discussed or compared, where the advanced methods are also polynomial graph filters but with a more careful design (i.e., different basis).\n\n(-) The motivation of the proposed new designs $\\alpha$ and $P$-hop seems not convincing\n\n(-) The spectral GNN baselines are completely missing in the experiment.\n\n(-) The experiments do not demonstrate the benefit of the proposed design besides prediction accuracy. Since the authors talk about polynomial graph filters, I wonder how the proposed methods can learn different graph filters.\n\n## Detail comments\n\nI have mixed feelings about this paper. On one hand, I find the discussion on requiring the Lipschitz graph filter and generalization analysis interesting. On the other hand, I feel the authors did not conduct a comprehensive literature survey on the recent development of spectral GNNs, which makes me feel the paper is a bit outdated. \n\nNote that the idea of learning polynomial graph filters has been extensively studied recently, as capable of learning high-pass graph filters enables good performance on heterophilic graph datasets [2]. Since GPR-GNN (using a monomial basis), many advances have been made. For instance, Bernnet [He et al. 2021] argue that using Bernstein polynomial basis has better properties. [Wang et al. 2022] further improve this line of work by using Jacobi polynomial basis. Both of them not only demonstrate great performance on homophilic and heterophilic datasets but also conduct experiments to directly examine how well spectral GNNs can learn each target graph filter such as low-pass, high-pass, band-pass and more. That is, these works show how to make $g_{\\theta}$ learn better with different polynomial basis and thus constraints on the coefficients. I feel the proposed APGNN is very relevant to this literature and should be compared with them in both related works and experiments. Furthermore, Since APGNN is claimed to be a ”universal principle”, I wonder how well it can learn on heterophilic datasets and different graph filters (i.e., the experiment in [Wang et al. 2022] Table 1).\n\nAlso, I find the motivation for using $\\alpha$ and $P$-hop not very convincing. For $\\alpha$, I agree that if we set $K = \\infty$ then the geometric decaying weight is needed. However, as the authors also mentioned in Section 4, we cannot use $K= \\infty$ in practice. In this case, I wonder what benefit would $\\alpha$ bring to us. Can not we just learn it automatically? On the other hand, the argument for motivating the use of $P$ seems problematic. Intuitively, we should compare two methods with the same number of hops ($K$). In this point of view, $P$ does not help in any sense. Also, if we fix $T$ (the maximum hop range), then the effect of increasing $P$ is merely reducing $K = T//P$, which apparently will make the generalization bound smaller. However, this is at the cost of increasing the term $\\hat{R}$, otherwise one should simply set $K=0$. \n\nEven from the experiment section, we can see that choosing $\\alpha\\approx 1$ gives roughly the same best-tuned results (Figure 3-(b)). A similar observation can be made for $P=1$ in Figure 3-(c), albeit the difference seems to be a bit larger. According to the argument above, I conjecture that the benefit of $P$ comes from having a smaller number of learnable weights (in the graph filter $g_{\\theta}$). It is interesting to investigate deeper of how we can improve along this line but the current results and explanation are unsatisfactory. At last, I also hope the authors can repeat experiments of Figure 3 on heterophilic datasets and even on experiments of learning different graph filters (maybe too much to do though). I feel the finding therein can motivate the authors to some ideas to further improve the work.\n\nMinor:\n\nHow can the authors answer \"yes\" in the reproducibility but not include their code (only after acceptance)? I feel the authors should be more serious about answering the checklist questions.\n\n\n## References\n\n[He et al. 2021] Bernnet: Learning arbitrary graph spectral filters via Bernstein approximation, He et al. NeurIPS 2021.\n\n[Wang et al. 2022] How powerful are spectral graph neural networks, Wang et al. ICML 2022.\n\nQUESTIONS:\n1. Detail comparison with recent spectral GNN literature in both related works and experiments.\n\n2. Conduct experiments on heterophilic datasets.\n\n3. Conduct experiments in learning different graph filters.\n\n4. Explain the benefit of $\\alpha$ in finite $K$ regime.\n\nLIMITATIONS:\nI do not find potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically studies the criterion for the graph filter formed by power series using Lipschitz smoothness and then proposes a novel Adaptive Power GN architecture. Some convergence and generalization analyses are covered. Experiments also show the advantages of the proposed methods, with some ablation studies on some parameters. Overall, it is a work that combines theory and practice. I think this work can benefit from some revisions. I am going to raise my score if my concerns are addressed. I would like to see a revision in the manuscript (it is OK to appear in the appendix) or a plan for the revisions (if the revision uploading is not allowed).\n\n------------------------------------------------------------------------------------------------\nAfter rebuttal, I increased my score to 6, which is based on my first impression and the responses. I am generally satisfied with the responses.\n\nSTRENGTHS:\n1. It is a paper that combines both theory and experiments.  I like the logic from theory to practice. Hence, the writing is excellent and clear to follow.\n2. As a reviewer from the theoretical area, I think the generalization analysis on non-GCN graph neural networks is novel and exciting. One great part is that this analysis seems to cover a lot of existing graph neural networks. So it is general. \n3. The experimental results support the theory and show the advantages.\n\nWEAKNESSES:\n1. Some parts of the paper are not very clear. It is mainly around lines 235 to 242. (a) For example, in line 239, it says, \"larger $\\alpha$ leads to a higher bound. \" It is not clear what this \"bound\" refers to. At first, I thought it referred to $1-\\alpha^K$ in line 235 or $1-\\alpha^{T/P}$ in line 237. Later, I feel it should be the last two terms (or the second to the last term) in Equation 24. This needs clarification. (b) Another thing is at the end of this paragraph, it claims, \"$\\alpha$ should be moderate to...$. The discussion about small $\\alpha$ is missing. I guess it is because a small $\\alpha$ makes the filter trivial and not powerful (line 157). This also needs clarification. \n\n2. Some discussions about theoretical works of generalization analysis on Graph Neural Networks are needed. I would like to know the comparison and theoretical novelties beyond existing works. Here are some recent related works.\n\n[1] Esser et al., 2021, \"Learning Theory Can (Sometimes) Explain Generalisation in Graph Neural Networks.\"\n\n[2] Cong et al., 2021, \"On Provable Benefits of Depth in Training Graph Convolutional Networks.\"\n\n[3] Li et al., 2022, \"Generalization guarantee of training graph convolutional networks with graph topology sampling.\"\n\n[4] Zhang et al., 2023, \"Joint Edge-Model Sparse Learning is Provably Efficient for Graph Neural Networks.\"\n\n[5] Tang et al., 2023, \"Towards Understanding Generalization of Graph Neural Networks.\"\n\nQUESTIONS:\n1. I think your theory covers other GNNs in Table 1. Can you use your Theorem 2 or another theory to show your proposed GNN is better than other GNNs in Table 1? It does not have to be very rigorous. I expect to see a comparison between different GNNs about $M$ and $L_M$ in equation 23.\n\n2. I think the multiple P-hop message-passing strategy should work better on some heterophilous graphs. The citation graphs are homophilous. Could you please show some experimental results on heterophilous graph datasets?\n\nLIMITATIONS:\nThere is no potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the polynomial filters in GNNs. Specfically, the authors propose a Adaptive Power GNN which employs exponentially decaying coefficients. A theoretically generalization  analysis of  the proposed framework is conducted. Experiments demostrate the proposed method can outperform some baselines on the selected datasets.\n\nSTRENGTHS:\n1. The background provided is detailed and aids readers in comprehending the nuances of the paper.\n2. The authors offer a generalization analysis of the proposed method.\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. The primary concern is the lack of novelty. The concept of polynomial filters in GNNs is well-established. Additionally, the paper looks at infinite orders, a domain where any function can be approximated by infinite polynomial functions. Also, the topic of GNN generalization has already been examined in several papers, such as [1], [2], and [3].\n2. The method proposed in this paper comprises decoupled GNNs, which implies that the framework and analysis may not be applicable to coupled cases, such as GCN and GAT.\n3. The hyperparamters $\\alpha, \\beta$ are hard to choose. For instance, GPR-GNN struggles to learn these hyperparameters effectively without proper initialization. How do the authors suggest these hyperparameters be selected?\n4. The authors opted for fixed data splits across all datasets, which may predispose the model to overfitting on the valid/test sets. Have the authors considered random data splits? Additionally, the study could benefit from incorporating larger datasets like the OGB datasets.\n\n[1] Baranwal, Aseem, Kimon Fountoulakis, and Aukosh Jagannath. \"Graph convolution for semi-supervised classification: Improved linear separability and out-of-distribution generalization.\" arXiv preprint arXiv:2102.06966 (2021).\n\n[2] Verma, Saurabh, and Zhi-Li Zhang. \"Stability and generalization of graph convolutional neural networks.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.\n\n[3] Ma, Jiaqi, Junwei Deng, and Qiaozhu Mei. \"Subgroup generalization and fairness of graph neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 1048-1061.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn GNNs, designing a graph filter or propagation mechanism plays a critical role. Spectral-based GNNs formulate graph filters in the graph Fourier domain, and those filters are usually in the form of polynomials or power series. The manuscript argues that a well-defined graph filter should be convergent when represented as power series and have desirable analytic properties such as the Lipschitz continuity. Graph filters of four existing GNNs are analyzed based on the proposed conditions; it is shown that they have some limitations. The proposed APGNN introduces an exponentially decaying rate to coefficients of the learnable graph filter. Due to the decaying rate, the filter of APGNN is guaranteed to converge, and APGNN can theoretically be extended to an infinite-depth GNN. The filter of APGNN also satisfies the Lipschitz continuity, implying that the model is stable and robust. A mathematical analysis of the generalization bound of APGNN is provided. In practice, a truncated polynomial instead of the power series is utilized as the filter of APGNN to avoid the infinite number of learnable parameters. A multiple P-hop strategy, which uses the P-th power of the adjacency matrix, is introduced to enlarge the receptive field of the graph filter while maintaining the same computational complexity. Experimental results on five real-world benchmark datasets show that APGNN has comparable performance to 11 baselines in accuracy.\n\nSTRENGTHS:\n1. The manuscript points out that there exist inconsistencies between GNN models and their infinite-depth versions. A criterion for the polynomial-based graph filter is proposed with two constraints: the convergence and the Lipschitz continuity of the graph filter. The criterion reflects the stability of a GNN model with respect to the input graph and the consistency between a GNN and its infinite-depth version.\n\n2. The motivation of APGNN is well described. The criterion for a polynomial-based graph filter is proposed, and the formulation of APGNN is provided as an instantiation of a graph filter satisfying the criterion. A truncated polynomial filter is suggested for practical use where the number of parameters should be finite.\n\n3. The effectiveness of the P-hop strategy is shown in experiments. In Figure 3 (c), the P-hop strategy allows the model to keep the performance while decreasing the computational cost by omitting the terms that are not multiple of P from the graph filter polynomial.\n\nWEAKNESSES:\n1. There should be a comparison between APGNN and other GNN methods with an infinite depth. Existing approaches utilize a residual connection [1] or formulate the state of equilibrium [2, 3] to model GNNs with an infinite depth. Those methods should be theoretically and empirically compared to APGNN. \n\n   [1] Chen et al., Simple and Deep Graph Convolutional Networks, ICML 2020.\n\n   [2] Gu et al., Implicit Graph Neural Networks, NeurIPS 2020.\n\n   [3] Liu et al., EIGNN: Efficient Infinite-Depth Graph Neural Networks, NeurIPS 2021.\n\n2. The node classification accuracy of APGNN is relatively low compared to the existing GNN models such as GCNII [1] and G$^2$CN [4]. For example, the accuracy of GCNII is 85.5 on Cora, whereas the accuracy of G$^2$CN is 73.8 on Citeseer. In addition, GNN models with an infinite depth should be compared to APGNN.\n\n   [4] Li et al., G$^2$CN: Graph Gaussian Convolution Networks with Concentrated Graph Filters, ICML 2022.\n\n3. The authors mentioned the hyperparameter sensitivity of PPNP in lines 130-131: \"However, the performance of PPNP is heavily dependent on the hyperparameter $\\beta$, which must be carefully tuned to achieve optimal performance.\" However, the same applies to APGNN since the node classification performance heavily depends on $\\alpha$, a decay weight. Figure 3(b) shows that the accuracy gap between the best and the worst cases is more than 20% on Cora.\n\n4. The polynomial order K is fixed to 10 for some baseline methods, such as ChebNet and DAGNN. However, the optimal value of K can vary depending on the baseline methods. In [1], the node classification results with various depths (Table 3 of [1]) show that the optimal depths are different for each combination of model and dataset. Therefore, the authors should show the node classification results with various polynomial orders, as in [1], or tune the polynomial order for each baseline method, as in [5].\n\n   [5] Liu et al., Towards Deeper Graph Neural Networks, KDD 2020.\n\n5. Minor Comments\n- In line 54, $i = [n_l]$ should be modified to $i \\in [n_l]$.\n- There is an inconsistency between Equations (2) and (3).\n- In line 93, \"L-Lipschtiz\" should be changed to \"L-Lipschitz\".\n- In line 136, “DAGGN” should be modified to “DAGNN”\n- In Equation 13, $(-1)^k$ should be modified to $(-1)$.\n- $\\rho$ indicates the spectral radius in Theorem 1, whereas it represents a probability measure in Section 4.\n- In Lines 192-195, there is no description and definition of $c_{\\mathcal{X}}$, $c_{\\mathcal{U}}$, $c_{\\mathcal{L}}$.\n- In line 206, ‘the hypothesis set over __ is described as’ should be modified to ‘the hypothesis set over \\mathcal{X} is described as’.\n- In line 266, APPNP is mentioned twice.\n\nQUESTIONS:\n1. Is there any tendency on the learned coefficients of the graph filter $\\beta$? APGNN learns the coefficients $\\beta$ during training to deal with arbitrary filter shapes. A higher $\\beta$ value means that the corresponding hop of neighbors is more important than a hop with a lower $\\beta$ value. Thus, by analyzing the learned $\\beta$, more explanations about the neighborhood depth can be added, such as the appropriate polynomial order for the model or important hops of the dataset.\n\n2. What is the total runtime of APGNN concerning K? As K grows, the proposed graph filter will become denser since its receptive field becomes larger. This aspect can increase the computation cost of applying the proposed graph filter. Meanwhile, the graph filter of GCN is relatively sparse since it sets K to 1. Therefore, comparing the runtimes of GNN models as K increases is encouraged.\n\nLIMITATIONS:\n1. There might be some cases where capturing the long-range dependencies is essential to understand the context of the given graph [6]. However, the proposed model might not be appropriate for those graphs. To capture long-range dependencies, either the decay weight $\\alpha$ has to be large enough, or $\\beta$ assigned to the low-order should be small. The former increases the generalization bound of the model, and the latter might lead to a vanishing gradient problem.\n\n   [6] Dwivedi et al., Long Range Graph Benchmark, NeurIPS 2022.\n\n2. The performance of the model is highly sensitive to the choice of $\\alpha$.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a regularized learning framework for creating deep Graph Neural Networks (GNNs), including the Adaptive Power GNN (APGNN) that uses exponentially decaying weights to aggregate graph information of varying orders. The proposed multiple P-hop message passing strategy efficiently perceives higher-order neighborhoods, and the APGNN can be extended to an infinite-depth network.\n\nSTRENGTHS:\n1. The proposed APGNN model effectively captures higher-order neighborhood information, which is a crucial aspect of many graph-based tasks. This is a significant contribution to the field.\n2. The regularized learning framework utilized in the proposed model provides a valuable theoretical guarantee for convergence. This adds to the credibility and reliability of the approach.\n3. The experimental results presented in the paper demonstrate that the proposed method outperforms other state-of-the-art GNN models on several benchmark datasets.\n\nWEAKNESSES:\n1. It would be beneficial for the paper to include a more detailed comparison with other recent GNN models that also aim to capture higher-order neighborhood information. This would provide readers with a clearer understanding of how the proposed APGNN model compares in terms of performance and capabilities.\n2. The paper lacks a detailed analysis of the computational complexity of the proposed method. Considering the potential concerns related to large-scale graphs, it is important to provide insights into the computational requirements of the model.\n3. Similarly, the paper should include a more detailed analysis of the sensitivity of the proposed method to hyperparameters. Understanding how the model's performance varies with different hyperparameter settings is crucial for practical implementation.\n4. The paper lacks a thorough analysis of the interpretability of the learned graph filters. Providing insights into the interpretability of the model's learned representations would enhance the understanding and trustworthiness of the proposed approach.\n\nQUESTIONS:\n1. Can you provide more details on the computational complexity of the proposed method, and how it scales with the size of the graph?\n2. How sensitive is the proposed method to the choice of hyperparameters, and how did you select the hyperparameters used in the experiments?\n3. Can you provide more details on the interpretability of the learned graph filters, and how they can be used to gain insights into the structure of the graph?\n4. Have you considered the robustness of the proposed method to adversarial attacks, and how it compares to other GNN models in this regard?\n5. How do you plan to extend the proposed method to handle dynamic graphs, where the structure of the graph changes over time?\n\nLIMITATIONS:\nConsidering the increasing importance of robustness in real-world applications, it would be valuable to evaluate the model's performance under adversarial scenarios and discuss its limitations and strengths in this regard.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}