Title: Deep Graph Neural Networks via Flexible Subgraph Aggregation
Venue: NeurIPS 2023
Accepted: no
Keywords: Graph Neural Networks

Semi-supervised Learning

Deep Models
Abstract: Graph neural networks (GNNs), a type of neural network that can learn from graph-structured data and learn the representation of nodes through aggregating neighborhood information, have shown superior performance in various downstream tasks. However, it is known that the performance of GNNs degrades gradually as the number of layers increases. In this paper, we evaluate the expressive power of GNN from the perspective of subgraph aggregation. We reveal the potential cause of performance degradation for traditional deep GNNs, i.e., aggregated subgraph overlap, and we theoretically illustrate the fact that previous residual-based GNNs exploit the aggregation results of 1 to k hop subgraphs to improve the effectiveness. Further, we find that the utilization of different subgraphs by previous models is often inflexible. Based on this, we propose a sampling-based node-level residual module (SNR) that can achieve a more flexible utilization of different hops of subgraph aggregation by introducing node-level parameters sampled from a learnable distribution. Extensive experiments show that the performance of GNNs with the our proposed SNR module outperform a comprehensive set of baselines.

Abstract

Graph neural networks (GNNs), a type of neural network that can learn from graphstructured data and learn the representation of nodes through aggregating neighborhood information, have shown superior performance in various downstream tasks. However, it is known that the performance of GNNs degrades gradually as the number of layers increases. In this paper, we evaluate the expressive power of GNNs from the perspective of subgraph aggregation. We reveal the potential cause of performance degradation for traditional deep GNNs, i.e., aggregated subgraph overlap, and we theoretically illustrate the fact that previous residual-based GNNs exploit the aggregation results of 1 to k hop subgraphs to improve the effectiveness. Further, we find that the utilization of different subgraphs by previous models is often inflexible. Based on this, we propose a sampling-based node-level residual module (SNR) that can achieve a more flexible utilization of different hops of subgraph aggregation by introducing node-level parameters sampled from a learnable distribution. Extensive experiments show that the performance of GNNs with our proposed SNR module outperform a comprehensive set of baselines.

1 Introduction

GNNs have emerged in recent years as the most powerful model for processing graph-structured data and have performed very well in various fields, such as social networks [1], recommender systems [2], and drug discovery [3]. Through the message-passing mechanism that propagates and aggregates representations of neighboring nodes, GNNs provide a general framework for learning information on graph structure.

Despite great success, according to previous studies [4, 5], GNNs show significant performance 22 degradation as the number of layers increases, which makes GNNs not able to take full advantage of 23 the multi-hop neighbor structure of nodes to obtain better node representations. 24

25

26

27

28

29

30

31

32

33

The main reason for this situation is now widely believed to be oversmoothing [4, 6, 5, 7]. However, since ResNet [8] uses residual connection to solve a similar problem in computer vision and obtains good results, several new works have been inspired to apply the idea of residual connection to GNNs to alleviate oversmoothing and thus improve the expressive power. For example, JKNet [5] learns node representations by aggregating the outputs of all previous layers at the last layer. DenseGCN [9] concatenates the results of the current layer and all previous layers as the node representations of this layer. APPNP [7] uses the initial residual connection to retain the initial feature information with probability α , and utilizes the feature information aggregated at the current layer with probability 1 -α .

In this paper, we evaluate the expressive power of GNNs from the perspective of subgraph aggregation. 34

Based on this perspective, we show that the single high-hop subgraph aggregation of message-passing 35

GNNs is limited by the fact that high-hop subgraphs are prone to information overlap, which 36 makes the node representations obtained from k-hop subgraph aggregation indistinguishable, i.e., 37 oversmoothing occurs. 38

Based on this perspective, we conduct a theoretical analysis of previous residual-based models and 39 find that previous methods are in fact able to utilize multiple subgraph aggregations to improve the 40 expressiveness of the model. However, most methods tend to utilize subgraph information by fixed 41 coefficients, which assumes that the information from the subgraph of the same hop are equally 42 important for different nodes, which leads to inflexibility in the model's exploitation of subgraph 43 information and thus limits further improvement of the expressive power. Some existing methods try 44 to overcome this inflexibility but lead to overfitting by introducing more parameters, which in turn 45 affects the effectiveness of the model, which is demonstrated by the experiment. 46

47

48

49

50

51

52

53

Considering these limitations, we propose a S ampling-based N ode-level R esidual module ( SNR ). Specifically, we adopt a more fine-grained node-level residual module to achieve a more flexible exploitation of subgraph aggregation, which is proved by the theoretical analysis. On the other hand, to avoid overfitting due to the introduction of more parameters, instead of learning the specific parameters directly, we first learn a correlation distribution through reparameterization trick and obtain the specific residual coefficients by sampling. Experiments verify that this sampling-based approach can significantly alleviate overfitting.

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

Our Contributions. (1) We reinterpret the phenomenon that the effectiveness of traditional messagepassing GNNs decreases as the number of layers increases from the perspective of k -hop subgraph overlap. (2) Based on the idea of subgraph aggregation, we theoretically analyze the previous residualbased methods and find that they actually utilize multiple hop subgraph aggregation in different ways to improve the expressive power of the model, and we point out the limitations of inflexibility and overfitting in previous residual-based methods. (3) We propose a sampling-based node-level residual module that allows more flexible exploitation of different k -hop subgraph aggregations while alleviating overfitting due to more parameters. (4) Extensive experiments show that GNNs with the proposed SNR module achieve better performance than other methods, as well as with higher training efficiency, on semi-supervised tasks as well as on tasks requiring deep GNNs.

2 Preliminaries

2.1 Notations

A connected undirected graph is represented by G = ( V , E ) , where V = { v 1 , v 2 , . . . , v N } is the set of N nodes and E ⊆ V × V is the set of edges. The feature of nodes is given in matrix H ∈ R N × d where d indicates the length of feature. Let A ∈ { 0 , 1 } N × N denotes the adjacency matrix and A ij = 1 only if an edge exists between nodes v i and v j . D ∈ R N × N is the diagonal degree matrix whose elements d i computes the number of edges connected to node v i . ˜ A = A + I is the adjacency matrix with self loop and ˜ D = D + I .

2.2 Graph Neural Networks

A GNNs layer updates the representation of each node via aggregating itself and its neighbors' representations. Specifically, a layer's output H ′ consists of new representations h ′ of each node computed as:

where

h

′

i

indicates the new representation of node

v

i

and

f

θ

denotes the update function.

key to the performance of different GNNs is in the design of the

f

θ

and

AGGREGATE

The function.

Graph Convolutional Network (GCN)[10] is a classical massage-passing GNNs follows layer-wise propagation rule:

where H k is the feature matrix of the k th layer, W k is a layer-specific learnable weight matrix, σ ( · ) 80 denotes an activation function. 81

2.3 Residual Connection 82

Several works have used residual connection to solve the problem of oversmoothing. Common 83 residual connection for GNNs are summarized below. Details are explained in Appendix A.

3 Motivation

Message-passing GNNs recursively update the features of each node by aggregating information from its neighbors, allowing them to capture both the graph topology and node features. For a message-passing GNNs without a residual structure, the information domain of each node after k -layer aggregation is a related k -hop subgraph. Figure 1 shows that, after two aggregation operations, nodes on layer 2 obtain 1-hop neighbor and 2-hop neighbor information in layer 0, respectively. According to the definition of the k -hop subgraph, the information of the node on layer 2 in the figure is composed of all reachable nodes information shown on layer 0. We can consider the result of k -layer residual-free message-passing GNNs is equivalent to k -time aggregation of each node on its k -hop subgraph, which we call k -hop subgraph aggregation.

It is evident that as the number of aggregation operations increases, the reachable information range of a node expands rapidly, that is, the size of its k -hop subgraph grows exponentially as k increases, leading to a significant increase in the overlap between the k -hop subgraphs of different nodes. As a result, the aggregation result of different nodes on their respective k -hop subgraphs becomes indistinguishable. Furthermore, in a specific graph dataset, nodes with higher degrees tend to have a larger range of k -hop subgraphs compared to nodes with lower degrees. As a result, the subgraphs are more likely to overlap between nodes with higher degrees, making their aggregation results more likely to become similar and indistinguishable.

To verify this point, we conduct experiments on three graph datasets, Cora, Citeseer, and Pubmed. First, we group the nodes according to their degrees by assigning nodes with degrees in the range of [2 i , 2 i +1 ) to the i -th group. Subsequently, we perform aggregation with different layers of GCN and GAT, then calculate the degree of smoothing of the node representations within each group separately. We use the metric proposed in [11] to measure the smoothness of the node representations within each group, namely SMV , which calculates the average of the distances between the nodes within the group:

̸

where D ( · , · ) denotes the normalized Euclidean distance between two vectors:

A smaller value of SMV indicates a greater similarity in node representations. 116

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

We select the most representative result illustrated in Figure 2, which shows the result of GAT on 117 Pubmed. The rest of the results are shown in the Appendix B. It can be seen that the groups of nodes 118 with higher degree tend to be more likely to have high similarity in the representation of nodes within 119 the group in different layers of the model. This finding supports our claim. 120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

After verifying the conclusion that subgraph overlap leads to oversmoothing through experiments, a natural idea is to alleviate the problem of large overlap of single subgraph by utilizing multiple hop subgraph aggregations, thereby alleviating oversmoothing. In the following section, we will demonstrate that the previous k -layer residual-based GNNs are actually different forms of integration of 1 to k hop subgraph aggregations.

3.1 Revisiting Previous Models in a New Perspective

In the rest of this paper we will uniformly take GCN, a

classical residual-free message-passing GNNs, as an example. We assume that H is non-negative, so the ELU function can be ignored. In addition, the weight matrix is ignored for simplicity. Combined with the formula of GCN given in Equation 1, we can formulate the specific result of k -hop subgraph aggregation as N k H , where N = ˜ D -1 2 ˜ A ˜ D -1 2 . To show more intuitively how different k -layer-based residual models utilize N j H , j = 0 , 1 , · · · , k . We derive the general term formulas of their final outputs, and the results are shown in Table 2. Details of the derivation of the formula in this part are given in Appendix C.

From the formula in the table, we can see that, in comparison to message-passing GNNs, residualbased variants of GNNs can utilize multiple k -hop subgraphs. There are two methods to exploit them: (1) Summation, such as ResGCN and APPNP. Such methods employ linear summation over the aggregation of different hop subgraphs; (2) Aggregation functions, such as DenseNet and JKNet. Such methods make direct and explicit exploitation of different hop subgraph aggregations through methods such as concatenation.

However, for the first type of methods, they all employ a fixed, layer-level coefficient for linear summation of the subgraph aggregation, which assumes that the information from the subgraph of the same hop are equally important for different nodes. It will limit the expressive power of GNNs, which reveals the need to design a more fine-grained node-level residual module that can more flexibly utilize information from different k -hop subgraphs. For another type of method, they can achieve finer-grained subgraph aggregation, but the experiment find that their performance is not improved because of the more finer-grained structure, mainly because the introduction of more parameters leads to overfitting Phenomenon. In general, neither of these two types of methods has achieved a more effective improvement in the expressive power of GNNs.

4 The Proposed Method

In order to solve the two limitations of flexibility and overfitting encountered by previous residualbased models, we try to propose a node-level, more flexible, general residual module, which can

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

alleviate overfitting caused by more parameters at the same time. Based on this, we propose a sampling-based node-level generic residual module SNR. We define SNR module as:

where h ( i ) k denotes the representation of k -th layer of node i , h ( i ) k ′ denotes the result obtained by an arbitrary GNNs layer with h ( i ) k -1 as input. p ( i ) k is a random number sampled from N ( α ( i ) k , β ( i ) k 2 ) which associated with the i -th node at the k -th layer while α ( i ) k and β ( i ) k are learnable parameters representing the mean and the standard deviation of this distribution, respectively. Next, we will illustrate the superiority of the SNR module in terms of flexibility and overfitting alleviation.

4.1 Flexibility

In this section, we will analyze the expressive power of GCN with SNR module and show that SNRGCN achieves a more flexible utilization of multiple subgraph aggregations. First of all, combined with the previous definition 5, the matrix form of the recurrence formula of SNR-GCN can be written as:

where Λ k is a diagonal matrix whose i -th diagonal element is equal to p ( i ) k . We first try to obtain the general term formula of SNR-GCN according to the recursive formula and demonstrate SNR-GCN's treatment of multiple subgraph aggregations. The following theorem can be proved:

Theorem 1. The general term formula of SNR-GCN can be deduced as: H k = ∑ k -1 i =2 ∏ k -1 j = i ˜ N j ( M i -M i -1 ) + ∏ k -1 i =1 ˜ N i ( H 1 + M 1 ) -M k -1 where ˜ N i = -Λ k -1 N and M k = -(Λ k N + I ) -1 ( I +Λ k ) H 1 .

The details of the proof are provided in Appendix D. From the general term formula of SNR-GCN, we can see that M k is a linear transformation of H 1 . Therefore, the first two terms of the formula can be approximately regarded as a new form of subgraph aggregation. Further we can find that all 1 to k hop subgraph aggregations appear in the formula, which ensures the expressive power. And because Λ k are learnable diagonal matrixes , SNR-GCN's subgraph aggregation is learnable and more flexible, which further makes expressive power stronger. Besides, when we set Λ k = -α I , the first term will be 0, and the rest terms are equivalent to APPNP's formula, which means SNR-GCN can be approximately regarded as a more fine-grained and expressive APPNP.

4.2 Overfitting Alleviation

Another key point of SNR is that it introduces randomness to alleviate overfitting. In our initial idea, we attempt to build a generic module similar to the initial residual at the node level. Based on this, we initially designed the following modules:

where q ( i ) k is a learnable parameter which associated with the i -th node at the k -th layer. After conducting experiments, we discover that the model has a high risk of overfitting when adding this module. However, we also find that if we do not learn q ( i ) k directly through backpropagation, but first learn a normal distribution associated with it via reparameterization trick and obtain q ( i ) k by sampling at each computation, the issue can be resolved, and the performance of the model significantly improves. To prove this, we perform an experimental verification. The details of the experiments are shown in the Appendix E.

It is worth noting that GCNII and SNR-GCN share a similar architecture, so both can be viewed approximately as more refined APPNP-style models. However, when faced with the problem of overfitting due to more parameters, GCNII adds an identity matrix to mitigate the issue. Later experiment results have shown that SNR-GCN's learning distribution-sampling approach is more effective in alleviating overfitting.

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

4.3 Complexity Analysis

Taking vanilla GCN as an example, we analyzed the additional complexity of SNR in model and time. We assume that the number of nodes in the graph is n and hidden dimension is d .

Model Complexity. As described in Section 4, at each layer the SNR module learns a mean and standard deviation of the corresponding distribution for each node, so the complexity can be calculated as O ( n ), and thus the additional complexity of the k -layer model equipped with SNR is O ( kn ).

Time Complexity. The time complexity of a vanilla GCN layer mainly comes from the matrix multiplication of N and H , hence its complexity is O ( n 2 d ) . And the main computational parts of a SNR module are the sampling of p ( i ) k , scalar multiplication and matrix addition, which correspond to a complexity of O ( n ), O ( nd ), and O ( nd ), respectively. Thus the time complexity of the SNR module is O ( nd ) and the time complexity of a GCN layer equipped the SNR module is O ( n 2 d + nd ). Therefore, the introduction of the SNR module does not significantly affect the computational efficiency.

5 Experiment

In this section, we aim to experimentally evaluate the effectiveness of SNR on real datasets. To achieve this, we will compare the performance of SNR with other methods and answer the following research questions. Q1: How effective is SNR on classical tasks that prefer shallow models? Q2: Can SNR help overcome oversmoothing in GNNs and enable the training of deeper models? Q3: How effective is SNR on tasks that require deep GNNs? Q4: How efficient is the training of SNR?

5.1 Experiment Setup

In our study, we conduct experiments on four tasks: semi-supervised node classification (Q1) , alleviating performance drop in deeper GNNs (Q2) , semi-supervised node classification with missing vectors (Q3) , and efficiency evaluation (Q4) .

Datasets. To assess the effectiveness of our proposed module, we have used four data sets that are widely used in the field of GNN, including Cora, Citeseer, Pubmed [12], and CoraFull [13] for testing purposes. In addition, we also use two webpage datasets collected from Wikipedia: Chameleon and Squirrel [14]. Details on the characteristics of these datasets and the specific data-splitting procedures used can be found in Appendix F.1.

Models. We consider two fundamental GNNs, GCN [10] and GAT [15]. For GCN, we test the performance of SNR-GCN and its residual variant models, including ResGCN [9], APPNP [7], DenseGCN [9], GCNII [16] and JKNet [5]. For GAT, we directly equip it with the following residual module: Res, InitialRes, Dense, JK and SNR and test the performance. Additionally, for the SSNC-MV task, we compare our proposed module with several classical oversmoothing mitigation techniques, including BatchNorm [17], PairNorm [18], DGN [19], Decorr [11], DropEdge [20] and other residual-based methods. Further details on these models and techniques can be found in the following sections.

Implementations. For all benchmark and variant models, the linear layers in the models are initialized with a standard normal distribution, and the convolutional layers are initialized with Xavier initialization. The Adam optimizer [21] is used for all models. Further details on the specific parameter settings used can be found in Appendix F.2. All models and datasets used in this paper are implemented using the Deep Graph Library (DGL) [22]. All experiments are conducted on a server with 15 vCPU Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz, A40 with 48GB GPU memory, and 56GB main memory.

5.2 Semi-supervised Node Classification

To validate the performance of SNR, we apply the module to two fundamental GNNs, GCN and GAT, and test the accuracy according to the mentioned experimental setup, and compare it with four classic residual modules, DenseNet, ResNet, InitialResNet and JKNet. We vary the number of layers in the range of { 1 , 2 , 3 , · · · , 10 } and select the best result among all layers. Specifically, we run 10 times for each number of layers to obtain the mean accuracy along with the standard deviation. We select the best results among all layers and report them in the Table 3. We find that GNNs with

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

the SNR module consistently achieve the best performance in all cases (Q1) . However, from the experimental results, many models with residual modules have not achieved the expected results. In many cases, compared with the basic model, the accuracy is even reduced. According to previous research [18], we speculate that overfitting may have contributed to this phenomenon. To verify our hypothesis, we conduct further experiments. Given that most models in the previous experiments achieve their best performance with shallow models, we select models with two layers, train 500 epochs, and report their accuracy on the training and validation sets at each epoch. The results are shown in Appendix G. Most models show signs of overfitting and SNR module demonstrates the best ability to alleviate overfitting. Specifically, in shallow GNNs with limited subgraph aggregation, most models have similar expressive abilities, and overfitting is the main factor affecting their performance. Our proposed method effectively alleviates overfitting by learning a more representative distribution, resulting in a better performance than the base models.

5.3 Alleviating Performance Drop in Deeper GNNs

As the number of layers in GNNs increases, oversmoothing occurs, resulting in performance degradation. Our objective is to investigate the performance of deep GNNs equipped with SNR and observe the impact of oversmoothing on their performance. We evaluate the performance of GNNs with different residual modules on 2, 16, and 32 layers using the Cora, Citeseer, and Pubmed datasets. The "None" column represents vanilla GNNs without any additional modules. According to [16], APPNP is a shallow model, hence we use GCNII to represent GCN with initial residual connection instead. The same settings are used in section 5.4. The experimental results are presented in Table 4.

From Table 4, we can observe that GNNs with SNR consistently outperform other residual methods and the base models in most of cases when given the same number of layers. SNR can significantly improve the performance of deep GNNs (Q2) . For instance, on the Cora dataset, SNR improves the performance of 32-layer GCN and GAT by 53.69% and 56.20% , respectively. By flexibly utilizing multiple subgraph aggregation results with our SNR module, we can enhance the expressive power of the model and produce more distinctive node representations than those of regular GNNs, thereby overcoming the oversmoothing problem. These results suggest that we can train deep GNNs based on SNR, making them suitable for tasks that require the use of deep GNNs.

5.4 Semi-supervised Node Classification with Missing Vectors

When do we need deep GNNs? [18] first proposed semi-supervised node classification with missing vectors (SSNC-MV), where nodes' features are missing. SSNC-MV is a practical problem with various real-world applications. For example, new users on social networks usually lack personal information [23]. Obviously, we need more propagation steps to effectively aggregate information associated with existing users so that we can obtain representations of these new users. In this scenario, GNNs with more layers clearly perform better.

283

284

285

286

287

288

289

290

291

292

Previous research has shown that normalization techniques can be effective in mitigating oversmoothing, and further, exploring deeper architectures. Therefore, we apply several techniques that can overcome oversmoothing and residual modules to GCN and GAT to compare their performance on tasks that require deep GNNs.

We remove the node features in the validation and test set following the idea in [11, 18, 19]. We reuse the metrics that already reported in [11] for None, BatchNorm [17], PairNorm [18], DGN [19], DeCorr [11], and DropEdge [20]. For all residual-based models, the results are obtained by varying the number of layers in { 1 , 2 , 3 , · · · , 10 , 15 , · · · , 30 } and running five times for each number of layers. We select the layer #K that achieves the best performance and report its average accuracy along with the standard deviation. The results are reported in Table 5.

Our experiments show that GNNs with the SNR module outperform all previous methods (Q3) . 293 Additionally, we find that for most models, the number of layers to reach the best accuracy is relatively 294 large, which indicates that it is necessary to perform more propagation to gather information from 295 further nodes so that we can obtain effective representations of nodes with missing features. 296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

5.5 Efficiency Experiment

In real-world tasks, the rate at which a model achieves optimal performance through training is often important, and this affects the true effectiveness and time consumption of the model in real-world applications. To enable concrete measurement and comparison, here we define the following metrics for model training efficiency:

where Accuracy denotes the accuracy of the model when it reaches its optimal performance and Time denotes the time when the model reaches its optimal performance. The definition of this formula shows that a larger Efficiency represents a higher performance per unit time improvement, and therefore a higher training efficiency.

Based on the above equation, we evaluate the training efficiency of vanilla GNNs and SNR-GNNs. We use the 2, 4, 8, 16, 32, and 64-layer and average five Efficiency calculated for each layer of the model. Specifically, each Efficiency is calculated based on the time for the model to reach the highest accuracy on the validation set after 100 epochs of training and the accuracy achieved on the test set at that time. Figure 3 shows the models' Efficiency on Cora. The results on other datasets are shown in the Appendix H. It can be noticed that the training efficiency decreases as the number

1.4

GAT

2.00

GCN

SNR-GAT

SNR-GCN

1.2

1.75

1.50

1.0

Efficiency

0.8

lenc

1.25

1.00

出

0.6

0.75

0.4

0.50

0.2

0.25

0.0

0.00

2

8

16

32

64

2

8

16

32

64

Layer

Layer

of layers increases, which is due to the increase in training time caused by the rise in the number of model parameters. However, in most cases, compared to vanilla GNNs, our SNR module is able to maintain the highest training efficiency (Q4) .

6 Conclusion

Our work proposes a new perspective for understanding the expressive power of GNNs: the k -hop subgraph aggregation theory. From this perspective, we have reinterpreted and experimentally validated the reason why the performance of message-passing GNNs decreases as the number of layers increases. Furthermore, we have evaluated the expressive power of previous residual-based GNNs based on this perspective. Building on these insights, we propose a new sampling-based generalized residual module SNR and show theoretically that SNR enables GNNs to more flexibly utilize information from multiple k -hop subgraphs, thus further improving the expressive power of GNNs. Extensive experiments demonstrate that the proposed SNR can effectively address the issues of overfitting in shallow layers and oversmoothing in deep layers that are commonly encountered in message-passing GNNs, and significantly improves the performance, particularly in SSNC-MV tasks. Our research will facilitate a deeper exploration of deep GNNs and enable a wider range of potential applications.

References

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of SIGKDD , pages 701-710, 2014.

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Yihong Eric Zhao, Jiliang Tang, and Dawei Yin. Graph 331 neural networks for social recommendation. In Proceedings of WWW , pages 417-426, 2019. 332

David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, 333 Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional networks on graphs 334 for learning molecular fingerprints. In Proceedings of NeurIPS , pages 2224-2232, 2015. 335

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks 336 for semi-supervised learning. In Proceedings of AAAI , pages 3538-3545, 2018. 337

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and 338 Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In 339 Proceedings of ICML , pages 5449-5458, 2018. 340

Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for 341 node classification. In Proceedings of ICLR , 2020. 342

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: 343 Graph neural networks meet personalized pagerank. In Proceedings of ICLR , 2019. 344

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image 345 recognition. In Proceedings of CVPR , pages 770-778, 2016. 346

Guohao Li, Matthias Müller, Ali K. Thabet, and Bernard Ghanem. Deepgcns: Can gcns go as 347 deep as cnns? In Proceedings of ICCV , pages 9266-9275, 2019. 348

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional 349 networks. In Proceedings of ICLR , 2017. 350

Wei Jin, Xiaorui Liu, Yao Ma, Charu C. Aggarwal, and Jiliang Tang. Feature overcorrelation in 351 deep graph neural networks: A new perspective. In Proceedings of SIGKDD , pages 709-719, 352 2022. 353

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi354 Rad. Collective classification in network data. AI Magazine , 29(3):93-106, 2008. 355

Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsu356 pervised inductive learning via ranking. In Proceedings of ICLR , 2018. 357

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. 358 Journal of Complex Networks , 9(2), 2021. 359

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, P. Lio', and Yoshua 360 Bengio. Graph attention networks. Proceedings of ICLR , 2017. 361

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph 362 convolutional networks. In Proceedings of ICML , volume 119, pages 1725-1735, 2020. 363

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training 364 by reducing internal covariate shift. In Proceedings of ICML , volume 37, pages 448-456, 2015. 365

Lingxiao Zhao and L. Akoglu. Pairnorm: Tackling oversmoothing in gnns. Proceedings of 366 ICLR , 2019. 367

Kaixiong Zhou, Xiao Huang, Yuening Li, D. Zha, Rui Chen, and Xia Hu. Towards deeper graph 368 neural networks with differentiable group normalization. Proceedings of NeurIPS , 2020. 369

Y. Rong, Wen-bing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph 370 convolutional networks on node classification. Proceedings of ICLR , 2019. 371

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings 372 of ICLR , 2015. 373

Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, 374 Qi Huang, Chao Ma, Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang 375 Li, Alexander J. Smola, and Zheng Zhang. Deep graph library: Towards efficient and scalable 376 deep learning on graphs. CoRR , abs/1909.01315, 2019. 377

Al Mamunur Rashid, George Karypis, and John Riedl. Learning preferences of new users in 378 recommender systems: An information theoretic approach. Proceedings of SIGKDD , 10(2):90379 100, 2008. 380

Peer reviews:
comments: SUMMARY:
This manuscript understands the expressive power of GNNs from a new perspective of subgraph aggregation, and reveals the potential reason for the performance degradation of traditional deep GNNs due to the overlap of aggregated subgraphs. The authors propose a sampling-based generalized residual module SNR and theoretically proves that SNR enables GNNs to more flexibly utilize information from multiple k-hop subgraphs, thereby improving the expressive power of GNNs. Extensive experiments show the effectiveness of the proposed SNR module.

STRENGTHS:
1）The idea of rethinking the expressive power of GNNs from the perspective of subgraph aggregation is novel and interesting.

2）The paper is well presented. The motivation, steps to construct a node-level, more flexible and general residual module to enhance the expressive power of GNNs while alleviating overfitting issue are clearly introduced.

3）The experimental results are convincible.

WEAKNESSES:
1）More related works on dealing with oversmoothing and overfitting issues in deep graph neural networks should be reviewed.

2）Further analysis of the experimental results is needed. For example, in Table 4, the possible reason why the proposed SNR module is weaker than InitialRes on Citeseer dataset should be discussed. 

3）There are some typos, e.g., row 221: “four data sets” should be “six datasets”.

QUESTIONS:
1）Why did the author use different backbone models on Table 3 and Table 4? GCNII is not used in Table 4.

2）What is the full name of the evaluation metric “SMV”.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose a sampling-based module to enhance the expressive power of Graph Neural Networks (GNNs), which traditionally assume that information from the subgraph of the same hop is equally important for all nodes in the graph. They argue that this rigid assumption restricts the models' ability to capture complex relationships. With their proposed module, different nodes can assign varying levels of importance to their neighbors at different hops during information aggregation, which is shown to improve the performance of the GNN variants.\n\nSTRENGTHS:\n1.\tThe proposed idea is interesting.\n2.\tThe paper is well written and clear.\n3.\tThe method is described in sufficient detail and easy to follow.\n\nWEAKNESSES:\n1.\tSome reported baseline methods’ performances are questionable. For example, GCNII [1] shows alleviation of over-smoothing on the benchmark datasets with even 32 and 64 layers and achieved much higher accuracy than reported in Table 3. \n2.\tThe experiments are mainly performed on small-scale graph datasets. It’ll be interesting to test the method on larger graph datasets (e.g., the OGB datasets).\n3.\tSome detailed analysis of the learned mean and variance of the normal distribution would better support the proposed idea. It would be interesting to see how the distribution for nodes of different degrees changes with the increasing number of layers.\n4.\tTo evaluate the method’s robustness to oversmoothing, it will be better to see a plot about how the performance changes as the number of layers increases than presenting the performance in a table. What’s more, the proposed method’s performance significantly dropped as the layer number increased. It suggests it still suffers from oversmoothing and/or overfitting.\n5.\tIt’ll be interesting to see the over-smoothing analysis in Figure 2 to be done for different GCN variants with and without the proposed module.\n\nReferences:\n[1] Chen, Ming, et al. \"Simple and deep graph convolutional networks.\" International conference on machine learning. PMLR, 2020.\n\nQUESTIONS:\n1.\tWas the mean and variance parameters of the normal distribution regularized by using any prior?\n2.\tWhy is GCNII not included in comparison in table 4? Although it is claimed in line 266 that APPNP is a shallow model, GCNII is used to represent GCN with initial residual connection instead. But GCNII is not only about initial residual connection and has significant performance improvements over other models, so the comparison seems not proper.\n3.\tWhat is the rationale to sample p_k from a normal distribution and pass it to a sigmoid function in equation 5? Would directly sampling p_k from a beta distribution do the work? Since its outputs are in the range 0 to 1 and also has a reparameterization form based on a digamma function.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of over-smooth with GNNs. It is argued that the over-smooth problem is caused by the increased overlap of the sub-graph when the respective field of GNNs becomes larger and larger.  In order to alleviate this problem, this paper proposed a method that random weighting the nodes within each layer by node-wise and layer-wise learnable parameters. The proposed method is limited evaluated on several datasets.\n\nSTRENGTHS:\n- Introducing randomness by using re-parameterization trick is interesting and makes sense to alleviate overfitting.\n\n- Formulating different types of residual GNNs from a unified view is good.\n\nWEAKNESSES:\n- Some statements from this manuscript do not stand well. For example, it is argued from the 98-th line that as the k increases, the overlap between k-hop subgraph rises, making the aggregation from the k-hop subgraph from different nodes indistinguishable. It is not evident and the overlap is not sufficient for the over-smooth issues. Think about the transformer architecture where the self-attention within each layer has access to all other nodes/tokens while performing well on pixel-level tasks with several layers. \n\n- Additionally, the experiment that nodes with higher degrees tend to have more similar representations seem can not support the claim of sub-graph theory, as the overlap of the sub-graph does not influence by its degree.\n\n- As for the over-smoothing problem, there are many other methods like drop-edge, which also introduce randomness when training the GNNs. It is highly recommended to include comparisons in the main results with methods along this line to make the contribution clearer (rather than just some comparisons under the setup of missing vectors). \n\n- It is unclear how the proposed method can help under the setup of missing vectors rather than other methods.\n\n- As the sampling parameters are node-wise, I'm wondering how the proposed method can extend to inductive learning.\n\nSome minor issues:\n\n- Missing punctuation at the end of all equations.\n\n- The quality of the figures can be improved a lot. It is highly recommended to use vector graphics for all the visualization.\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nThere is no limitation discussion. The authors are encouraged to include as least the discussions about the limitation for inductive learning.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper focuses on the alleviation of overfitting and over-smoothing of deeper GNNs. It is an interesting topic and the solution seems promising with a sample-based node level residual block. Extensive experiments on public datasets verifies the applicability.

STRENGTHS:
The paper analyses in a new perspective on the performance drop of GNN with the increase of layers, with motivating examples. It proposes a new residual block with the parameters randomly sampled from a natural distribution. The block alleviates the performance drop and seems applicable in general to all GNN models.

WEAKNESSES:
1. The presentation is in great need to improve. There are some long sentences that hinder the understanding of the authors's ideas, such as L54. A research article should describe facts with a 3rd party stand, but not subjective notations.
2. The paper should be self-contained even without the appendix, but it does not provide enough explanations such as on L83.
3. The figure is not reflecting exactly what is written such as L90 for figure 1, nothing shows the overlaps of aggregations via the lines and nodes.
4. Abbreviations should come with the full spellings on the first occurrence, even if it may be obvious in a specific research domain, such as GCN, but not GCNII on L194.
5. The citations should go to clear items such as a formula or a reference, but not a long section as L202 to Sec. 4.
6. It seems to be unnecessary to have Sec. 3.1 if there is only 1 sub-section.
7. Theorem 1 does not mean anything to me as it gets only formulas, but no conditions neither a conclusion.
8. It should be consistent to have the name of the proposed method or model.

QUESTIONS:
See weakness

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07