Title: Learning to Control the Smoothness of GCN Features
Venue: NeurIPS 2024
Accepted: no
Keywords: graph neural networks

activation function

smoothness of node features
Abstract: The pioneering work of Oono \& Suzuki [ICLR, 2020] and Cai \& Wang [arXiv:2006.13318] analyze the smoothness of graph convolutional network (GCN) features. Their results reveal an intricate empirical correlation between node classification accuracy and the ratio of smooth to non-smooth feature components. However, the optimal ratio that favors node classification is unknown, and the non-smooth features of deep GCN with ReLU or leaky ReLU activation function diminish. In this paper, we propose a new strategy to let GCN learn node features with a desired smoothness to enhance node classification. Our approach has three key steps: (1) We establish a geometric relationship between the input and output of ReLU or leaky ReLU. (2) Building on our geometric insights, we augment the message-passing process of graph convolutional layers (GCLs) with a learnable term to modulate the smoothness of node features with computational efficiency. (3) We investigate the achievable ratio between smooth and non-smooth feature components for GCNs with the augmented message passing scheme. Our extensive numerical results show that the augmented message passing remarkably improves node classification for GCN and some related models.

Abstract

The pioneering work of Oono & Suzuki [ICLR, 2020] and Cai & Wang [arXiv:2006.13318] analyze the smoothness of graph convolutional network (GCN) features. Their results reveal an intricate empirical correlation between node classification accuracy and the ratio of smooth to non-smooth feature components. However, the optimal ratio that favors node classification is unknown, and the non-smooth features of deep GCN with ReLU or leaky ReLU activation function diminish. In this paper, we propose a new strategy to let GCN learn node features with a desired smoothness to enhance node classification. Our approach has three key steps: (1) We establish a geometric relationship between the input and output of ReLU or leaky ReLU. (2) Building on our geometric insights, we augment the message-passing process of graph convolutional layers (GCLs) with a learnable term to modulate the smoothness of node features with computational efficiency. (3) We investigate the achievable ratio between smooth and non-smooth feature components for GCNs with the augmented message passing scheme. Our extensive numerical results show that the augmented message passing remarkably improves node classification for GCN and some related models.

1 Introduction

Let G = ( V, E ) be an undirected graph with V = { v i } n i =1 and E be the set of nodes and edges, resp. Let A ∈ R n × n be the adjacency matrix of the graph with A ij = 1 ( i,j ) ∈ E , where 1 is the indicator function. Furthermore, let G be the following (augmented) normalized adjacency matrix

where I is the identity matrix, D is the degree matrix with D ii = ∑ n j =1 A ij , and ˜ A := A + I and ˜ D := D + I . Starting from the initial node features H 0 := [( h 0 1 ) ⊤ , . . . , ( h 0 n ) ⊤ ] ⊤ ∈ R d × n with h 0 i ∈ R d being the i th node feature vector, the graph convolutional network (GCN) [20] learns node representations using the following graph convolutional layer (GCL) transformation

where σ is the activation function, e.g. ReLU [25], and W l ∈ R d × d is learnable. GCL smooths 25 feature vectors of the neighboring nodes. The smoothness of features helps node classification; see 26 e.g. [22, 31, 5], resonating with the idea of classical semi-supervised learning approaches [41, 38]. 27 Accurate node classification requires a balance between smooth and non-smooth components of GCN 28 features [27]. Besides graph convolutional networks (GCNs) stacking GCLs, many other graph neural 29 networks (GNNs) have been developed using different mechanisms, including spectral methods [3, 9], 30 spatial methods [12, 30], sampling methods [13, 36], and the attention mechanism [30]. Many other 31 GNN models can be found in recent surveys or monographs; see, e.g. [15, 1, 33, 39, 14]. 32

Deep neural networks usually outperform shallow architectures, and a remarkable example is convo-

33 lutional neural networks [21, 16]. However, this does not carry to GCNs; deep GCNs tend to perform 34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

significantly worse than shallow models [5]. In particular, the node feature vectors learned by deep GCNs tend to be identical over each connected component of the graph; this phenomenon is referred to as over-smoothing [22, 26, 27, 4, 5, 32], which not only occurs for GCN but also for many other GNNs, e.g., GraphSage [13] and MPNN [12]. Intuitively, each GCL smooths neighboring node features, benefiting node classification [22, 31, 5]. However, stacking these smoothing layers will inevitably homogenize node features. Algorithms have been developed to alleviate the over-smoothing issue of GNNs, including decoupling prediction and message passing [11], skip connection and batch normalization [18, 7, 6], graph sparsification [29], jumping knowledge [34], scattering transform [24], PairNorm [37], and controlling the Dirichlet energy of node features [40].

From a theoretical perspective, it is proved that deep GCNs using ReLU or leaky ReLU activation function learn homogeneous node features [27, 4]. In particular, [27] shows that the distance of node features to the eigenspace M - corresponding to the largest eigenvalue 1 of matrix G in (1) - goes to zero when the depth of GCN with ReLU goes to infinity. Meanwhile, [27] empirically studies the intricate correlation between node classification accuracy and the ratio between smooth and non-smooth components of GCN node features, i.e., projections of node features onto eigenspace M and its orthogonal complement M ⊥ , resp. The empirical results of [27] indicate that both smooth and non-smooth components of node features are crucial for accurate node classification , while the ratio between smooth and non-smooth components to achieve optimal accuracy is unknown and task-dependent. Furthermore, [4] proves that the Dirichlet energy - another smoothness measure for node features - goes to zero when the depth of GCN with ReLU or leaky ReLU goes to infinity.

A crucial step in the proofs of [27, 4] is that ReLU and leaky ReLU reduce the distance of feature vectors to M and their Dirichlet energy. However, [4] points out that over-smoothing - characterized by the distance of features to eigenspace M or the Dirichlet energy - is a misnomer ; the real smoothness should be characterized by a normalized smoothness , e.g., normalizing the Dirichlet energy by the magnitude of the features. The ratio between smooth and non-smooth components of node features - studied in [27] - is closely related to the normalized smoothness . Nevertheless, analyzing the normalized smoothness of node features learned by GCN with ReLU or leaky ReLU remains an open problem [4]. Moreover, it is interesting to ask if analyzing the normalized smoothness can result in any new understanding of GCN features and algorithms to improve GCN's performance.

1.1 Our contribution

We aim to (1) establish a new geometric understanding of how GCL smooths GCN features and (2) develop an efficient algorithm to let GCN and related models learn node features with a desired normalized smoothness to improve node classification. We summarize our main contributions towards achieving our goal as follows:

We prove that there is a high-dimensional sphere underlying the input and output vectors of ReLU or leaky ReLU. This geometric characterization not only implies theories in [27, 4] but also informs that adjusting the projection of input onto eigenspace M can alter the smoothness of the output vectors. See Section 3 for details.

We show that both ReLU and leaky ReLU reduce the distance of node features to eigenspace M , i.e., ReLU and leaky ReLU smooth their input vectors without considering their magnitude. In contrast, when taking the magnitude into account, ReLU and leaky ReLU can increase, decrease, or preserve the normalized smoothness of each dimension of the input vectors; see Sections 3 and 4.

Inspired by our established geometric relationship between the input and output of ReLU or leaky ReLU, we study how adjusting the projection of input onto eigenspace M affects both normalized and unnormalized smoothness of the output vectors. We show that the distance of the output to eigenspace M is no greater than that of the original input - no matter how we adjust the input by changing its projection onto M . In contrast, adjusting the projection of input vectors onto M can change the normalized smoothness of output to any desired value; see details in Section 4.

Based on our theory, we propose a computationally efficient smoothness control term (SCT) to let GCN and related models learn node features with a desired (normalized) smoothness to improve node classification. We comprehensively validate the benefits of our proposed SCT in improving node classification - for both homophilic and heterophilic graphs - using a few of the most representative GCN-style models. See Sections 5 and 6 for details.

As far as we know, our work is the first thorough study of how ReLU and leaky ReLU affect the smoothness of node features both with and without considering their magnitude.

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

127

128

129

130

1.2 Additional related works

Controlling the smoothness of node features to improve the performance of GCNs is another line of related work. For instance, [37] designs a normalization layer to prevent node features from becoming too similar to each other, and [40] constrains the Dirichlet energy to control the smoothness of node features without considering the effects of nonlinear activation functions. While there has been effort in understanding and alleviating the over-smoothing of GCNs and controlling the smoothness of node features, there is a shortage of theoretical examination of how activation functions affect the smoothness of node features, specifically accounting for the magnitude of features.

1.3 Notation and Organization

Notation. We denote the ℓ 2 -norm of a vector u as ∥ u ∥ . For vectors u and v , we use ⟨ u , v ⟩ , u ⊙ v , and u ⊗ v to denote their inner, Hadamard, and Kronecker product, resp. For a matrix A , we denote its ( i, j ) th entry, transpose, and inverse as A ij , A ⊤ , and A -1 , resp. We denote the trace of A ∈ R n × n as Trace( A ) = ∑ n i =1 A ii . For two matrices A and B , we denote the Frobenius inner product as ⟨ A , B ⟩ F := Trace( AB ⊤ ) and the Frobenius norm of A as ∥ A ∥ F := √ ⟨ A , A ⟩ .

Organization. We provide preliminaries in Section 2. In Section 3, we establish a geometric characterization of how ReLU and leaky ReLU affect the smoothness of their input vectors. We study the smoothness of each dimension of node features and take their magnitude into account in Section 4. Our proposed SCT is presented in Section 5. We comprehensively verify the efficacy of the proposed SCT in Section 6. Technical proofs and more experimental results are provided in the appendix.

2 Preliminaries and Existing Results

From the spectral graph theory [8], we can sort eigenvalues of matrix G in (1) as 1 = λ 1 = . . . = λ m > λ m +1 ≥ . . . ≥ λ n > -1 , where m is the number of connected components of the graph. We decompose V = { v k } n k =1 into m connected components V 1 , . . . , V m . Let u i = ( 1 { v k ∈ V i } ) 1 ≤ k ≤ n be the indicator vector of V i , i.e., the k th coordinate of u i is one if the k th node v k lies in the connected component V i ; zero otherwise. Moreover, let e i be the eigenvector associated with λ i , then { e i } n i =1 forms an orthonormal basis of R n . Notice that { e i } i m =1 spans the eigenspace M - corresponding to eigenvalue 1 of matrix G , and { e i } n i = m +1 spans the orthogonal complement of M , denoted by M ⊥ . The paper [27] connects the indicator vectors u i s with the space M . In particular, we have

Proposition 2.1 ([27]) . All eigenvalues of matrix G lie in the interval ( -1 , 1] . Furthermore, the nonnegative vectors { ˜ D 1 2 u i / ∥ ˜ D 1 2 u i ∥} 1 ≤ i ≤ m form an orthonormal basis of M .

For any matrix H := [ h 1 , . . . , h n ] ∈ R d × n , we have the decomposition H = H M + H M ⊥ with H M = ∑ m i =1 He i e ⊤ i and H M ⊥ = ∑ n i = m +1 He i e ⊤ i such that ⟨ H M , H M ⊥ ⟩ F = Trace ( ∑ m i =1 He i e ⊤ i ( ∑ n j = m +1 He j e ⊤ j ) ⊤ ) = 0 , implying that ∥ H ∥ 2 F = ∥ H M ∥ 2 F + ∥ H M ⊥ ∥ 2 F .

2.1 Existing smoothness notions of node features

Distance to the eigenspace M . Oono et al. [27] study the smoothness of features H := [ h 1 , . . . , h n ] using their distance to the eigenspace M as an unnormalized smoothness notion.

Definition 2.2 ([27]) . Let R d ⊗M be the subspace of R d × n consisting of the sum ∑ m i =1 w i ⊗ e i , where w i ∈ R d and { e i } i m =1 is an orthonormal basis of the eigenspace M . Then we define ∥ H ∥ M ⊥ - the distance of node features H to the eigenspace M - as follows:

With the decomposition H = H M + H M ⊥ , ∥ · ∥ M ⊥ can be related to ∥ · ∥ F as follows: 126

Dirichlet energy. The paper [4] studies the unnormalized smoothness of node features using Dirichlet energy, which is defined as follows:

Definition 2.3 ([4]) . Let ˜ ∆ = I -G be the (augmented) normalized Laplacian, then the Dirichlet energy ∥ H ∥ E of node features H is defined by ∥ H ∥ 2 E := Trace( H ˜ ∆ H ⊤ ) .

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

Normalized Dirichlet energy. [4] points out that the real smoothness of node features H should be measured by the normalized Dirichlet energy Trace( H ˜ ∆ H ⊤ ) / ∥ H ∥ 2 F . This normalized measurement is essential because data often originates from various sources with diverse measurement units or scales. By normalization, we can mitigate biases resulting from these different scales.

2.2 Two existing theories of over-smoothing

Let λ = max {| λ i | | λ i < 1 } be the second largest magnitude of G 's eigenvalues, and s l be the largest singular value of weight matrix W l . [27] shows that ∥ H l ∥ M ⊥ ≤ s l λ ∥ H l -1 ∥ M ⊥ under GCL when σ is ReLU. Therefore, ∥ H l ∥ M ⊥ → 0 as l →∞ if s l λ < 1 , indicating node features converge to M and results in over-smoothing. A crucial step in the analysis in [27] is that ∥ σ ( Z ) ∥ M ⊥ ≤ ∥ Z ∥ M ⊥ , for any matrix Z when σ is ReLU, i.e., ReLU reduces the distance to M . [27] points out that it is hard to extend the above result to other activation functions even leaky ReLU.

Instead of considering ∥ H ∥ M ⊥ , [4] shows that ∥ H l ∥ E ≤ s l λ ∥ H l -1 ∥ E under GCL when σ is ReLU or leaky ReLU. Hence, ∥ H l ∥ E → 0 as l →∞ , implying over-smoothing of GCNs. Note that ∥ H ∥ M ⊥ = 0 or ∥ H l ∥ E = 0 indicates homogeneous node features. The proof in [4] applies to GCN with both ReLU and leaky ReLU by establishing the inequality ∥ σ ( Z ) ∥ E ≤ ∥ Z ∥ E for any matrix Z .

3 Effects of Activation Functions: A Geometric Characterization

In this section, we present a geometric relationship between the input and output vectors of ReLU or leaky ReLU. We use ∥ H ∥ M ⊥ as the unnormalized smoothness notion for all subsequent analyses since we observe that ∥ H ∥ M ⊥ and ∥ H ∥ E are equivalent as seminorms. In particular, we have

Proposition 3.1. ∥ H ∥ M ⊥ and ∥ H ∥ E are two equivalent seminorms, i.e., there exist two constants α, β > 0 s.t. α ∥ H ∥ M ⊥ ≤ ∥ H ∥ E ≤ β ∥ H ∥ M ⊥ , for any H ∈ R d × n .

3.1 ReLU

Let σ ( x ) = max { x, 0 } be ReLU. The first main result of this paper is that there is a high-dimensional sphere underlying the input and output of ReLU; more precisely, we have

Proposition 3.2 (ReLU) . For any Z = Z M + Z M ⊥ ∈ R d × n , let H = σ ( Z ) = H M + H M ⊥ . Then H M ⊥ lies on the high-dimensional sphere centered at Z M ⊥ / 2 with radius

In particular, H M ⊥ lies inside the ball centered at Z M ⊥ / 2 with radius ∥ Z M ⊥ / 2 ∥ F and hence we have ∥ H ∥ M ⊥ ≤ ∥ Z ∥ M ⊥ .

3.2 Leaky ReLU

Now we consider leaky ReLU σ a ( x ) = max { x, ax } , where 0 < a < 1 is a positive scalar. Similar to ReLU, we have the following result for leaky ReLU

Proposition 3.3 (Leaky ReLU) . For any Z = Z M + Z M ⊥ ∈ R d × n , let H = σ a ( Z ) = H M + H M ⊥ . Then H M ⊥ lies on the high-dimensional sphere centered at (1 + a ) Z M ⊥ / 2 with radius

In particular, H M ⊥ lies inside the ball centered at (1 + a ) Z M ⊥ / 2 with radius ∥ (1 -a ) Z M ⊥ / 2 ∥ F and hence we see that a ∥ Z ∥ M ⊥ ≤ ∥ H ∥ M ⊥ ≤ ∥ Z ∥ M ⊥ .

3.3 Implications of the above geometric characterizations

Propositions 3.2 and 3.3 imply that the precise location of H M ⊥ (or ∥ H M ⊥ ∥ F = ∥ H ∥ M ⊥ ) depends on the center and the radius r or r a . Given a fixed Z M ⊥ , the center of the spheres remains unchanged, and r and r a are only affected by changes in Z M . This observation motivates us to investigate how changes in

Z

M

impact

∥

H

∥

M

⊥

, i.e., the unnormalized smoothness of node features

̸

.

Propositions 3.2 and 3.3 imply both ReLU and leaky ReLU reduce the distance of node features to eigenspace M , i.e. ∥ H ∥ M ⊥ ≤ ∥ Z ∥ M ⊥ . Moreover, this inequality is independent of Z M ; consider Z , Z ′ ∈ R d × n s.t. Z M ⊥ = Z ′ M ⊥ but Z M = Z ′ M . Let H and H ′ be the output of Z and Z ′ via ReLU or leaky ReLU, resp. Then we have ∥ H ∥ M ⊥ ≤ ∥ Z ∥ M ⊥ and ∥ H ′ ∥ M ⊥ ≤ ∥ Z ′ ∥ M ⊥ . Since Z M ⊥ = Z ′ M ⊥ , we deduce that ∥ H ′ ∥ M ⊥ ≤ ∥ Z ∥ M ⊥ . In other words, when Z M ⊥ = Z ′ M ⊥ is fixed, changing

Z

M

to

Z

′ M

can change the unnormalized smoothness of the output features but cannot

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

199

200

201

202

203

Fig. 1a) in Section 4.1. Notice that without considering the nonlinear activation function, changing Z M does not affect the unnormalized smoothness of node features measured by ∥ H ∥ M ⊥ .

In contrast to the unnormalized smoothness, if one considers the normalized smoothness, we find that adjusting Z M can result in a less smooth output ; we will discuss this in Section 4.1.

4 How Adjusting Z M Affects the Smoothness of the Output

Throughout this section, we let Z and H be the input and output of ReLU or leaky ReLU. The smoothness notions based on the distance of feature to M or their Dirichlet energy do not account for the magnitude of each dimension of the features; [4] points out that analyzing the normalized smoothness of features Z , given by ∥ Z ∥ E / ∥ Z ∥ F , is an open problem. However, these two smoothness notions aggregate the smoothness of node features across all dimensions; when the magnitude of some dimensions is much larger than others, the smoothness will be dominated by them.

Motivated by the discussion in Section 3.3, we study the disparate effects of adjusting Z M on the normalized and unnormalized smoothness in this section. For the sake of simplicity, we assume the graph is connected ( m = 1 ); all the following results can be extended to graphs with multiple connected components easily. Due to the equivalence between seminorms ∥ · ∥ M and ∥ · ∥ E , we introduce the following definition of the dimension-wise normalized smoothness of node features.

Definition 4.1. Let Z ∈ R d × n be the features over n nodes with z ( i ) ∈ R n being its i th row, i.e., the i th dimension of the features over all nodes. We define the normalized smoothness of z ( i ) as follows:

where we set s ( z ( i ) ) = 1 when z ( i ) = 0 .

Remark 4.2 . Notice that the normalized smoothness s ( z ( i ) ) = ∥ z ( i ) M ∥ / ∥ z ( i ) ∥ is closely related to the ratio between the smooth and non-smooth components of node features ∥ z ( i ) M ∥ / ∥ z ( i ) M ⊥ ∥ .

The graph is connected implies that z ( i ) M = ⟨ z ( i ) , e 1 ⟩ e 1 and ∥ z ( i ) M ∥ = |⟨ z ( i ) , e 1 ⟩| . Without ambiguity, we write z for z ( i ) and e for e 1 - the eigenvector of G associated with the eigenvalue 1 . Moreover, we have

It is evident that the larger s ( z ) is, the smoother the node feature z is 1 . In fact, we have

where ∥ z ∥ M ⊥ / ∥ z ∥ decreases as s ( z ) increases. 196

To discuss how the smoothness s ( h ) = s ( σ ( z )) or s ( σ a ( z )) can be adjusted by changing z M , we consider the function

It is clear that

where we see that α only alters z M while pre197 serves z M ⊥ . Moreover, it is evident that 198

1.0

0.5

0.0

0.5

1.0

Parameter

(

)

0

5

10

Smoothness

(

s

)

||

z

||

||

(

z

)||

||

a

(

z

)||

1.0

0.5

0.0

0.5

1.0

1.5

2.0

Parameter

(

)

0.0

0.5

1.0

Smoothness

(

s

)

s

(

z

)

s

(

(

z

))

s

(

a

(

z

))

a) Smoothness

b) Normalized smoothness

It follows that s ( z ( α )) = 1 if and only if z M ⊥ = 0 (include the case z = 0 ), showing that when z M ⊥ = 0 , the vector z is the smoothest one.

4.1 The disparate effects of α on ∥ · ∥ M ⊥ and s ( · ) : Empirical results

Let us empirically study possible values that the unnormalized smoothness ∥ σ ( z ( α )) ∥ M ⊥ , ∥ σ a ( z ( α )) ∥ M ⊥ and the normalized smoothness s ( σ ( z ( α ))) , s ( σ a ( z ( α ))) can take when α varies.

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

We denote z α := z ( α ) = z -α e . We consider a connected synthetic graph with 100 nodes, and each node is assigned a random degree between 2 to 10 . Then we assign an initial node feature z ∈ R 100 , sampled uniformly on the interval [ -1 . 5 , 1 . 5] , to the graph with each node feature being a scalar. Also, we compute e by the formula e = ˜ D 1 2 u / ∥ ˜ D 1 2 u ∥ from Proposition 2.1, where u ∈ R 100 is the vector whose entries are all ones and ˜ D is the (augmented) degree matrix. We examine two different smoothness notions for the input z and the output σ ( z α ) and σ a ( z α ) , where the smoothness is measured for various values of the smoothness control parameter α ∈ [ -1 . 5 , 1 . 5] . In Fig. 1a), we study the unnormalized smoothness measured by ∥·∥ M ⊥ ; we see that ∥ σ ( z α ) ∥ M ⊥ and ∥ σ a ( z α ) ∥ M ⊥ are always no greater than ∥ z ∥ M ⊥ . This coincides with the discussion in Section 3.3; adjusting the projection of z onto the eigenspace M can not change the fact that ∥ σ ( z α ) ∥ M ⊥ ≤ ∥ z ∥ M ⊥ and ∥ σ a ( z α ) ∥ M ⊥ ≤ ∥ z ∥ M ⊥ . Nevertheless, an interesting result is that altering the eigenspace projection can adjust the unnormalized smoothness of the output : notice that altering the eigenspace projection does not change its distance to M , i.e., the smoothness of the input is unchanged, but the smoothness of the output after activation function can be changed.

In contrast, when studying the normalized smoothness s ( · ) in Fig. 1b), we find that s ( σ ( z ( α ))) and s ( σ a ( z ( α ))) can be adjusted by α to values smaller than s ( z ) . More precisely, we see that by adjusting α , s ( σ ( z ( α ))) and s ( σ a ( z ( α ))) can achieve most of the values in [0 , 1] . In other words, both smoother and less smooth features can be obtained by adjusting α .

4.2 Theoretical results on the smooth effects of ReLU and leaky ReLU

̸

In this subsection, we build theoretical understandings of the above empirical findings on the achievable smoothness shown in Fig. 1. Notice that if z M ⊥ = 0 , the inequalities presented in Propositions 3.2 and 3.3 indicate that ∥ σ ( z ( α )) ∥ M ⊥ and ∥ σ a ( z ( α )) ∥ M ⊥ vanish. So we have s ( σ ( z ( α ))) = 1 for any α when z M ⊥ = 0 . Then we may assume z M ⊥ = 0 for the following study.

̸

Proposition 4.3 (ReLU) . Suppose z M ⊥ = 0 . Let h ( α ) = σ ( z ( α )) with σ being ReLU, then

where x := ˜ D -1 2 z , max x = max 1 ≤ i ≤ n x i , and ˜ D is the augmented degree matrix with diagonals d 1 , d 2 , . . . , d n . In particular, the normalized smoothness s ( h ( α )) is monotone increasing as α decreases whenever α < ∥ ˜ D 1 2 u n ∥ max x and it has range [min α s ( h ( α )) , 1] .

̸

Proposition 4.4 (Leaky ReLU) . Suppose z M ⊥ = 0 . Let h ( α ) = σ a ( z ( α )) with σ a being leaky ReLU, then (1) min α s ( h ( α )) = 0 , and (2) sup α s ( h ( α )) = 1 and s ( h ( α )) has range [0 , 1) .

Proposition 4.4 also holds for other variants of ReLU, e.g., ELU 2 and SELU 3 .; see Appendix C. We summarize Propositions 3.2, 3.3, 4.3, and 4.4 in the following corollary, which qualitatively explains the empirical results in Fig. 1.

̸

Corollary 4.5. Suppose z M ⊥ = 0 . Let h ( α ) = σ ( z ( α )) or σ a ( z ( α )) with σ being ReLU and σ a being leaky ReLU. Then we have ∥ z ∥ M ⊥ ≥ ∥ h ( α ) ∥ M ⊥ for any α ∈ R ; however, s ( h ( α )) can be smaller than, larger than, or equal to s ( z ) for different values of α .

Propositions 4.3 and 4.4, and Corollary 4.5, provide a theoretical basis for the empirical results in Fig. 1. Moreover, our results indicate that for any given vector z , altering z M can change both the unnormalized and the normalized smoothness of the output vector h = σ ( z ) or σ a ( z ) . In particular, the normalized smoothness of h = σ ( z ) or σ a ( z ) can be adjusted to any value in the range shown in Propositions 4.3 and 4.4. This provides us with insights to control the smoothness of features to improve the performance of GCN and we will discuss this in the next section.

5 Controlling Smoothness of Node Features

We do not know how smooth features are ideal for a given node classification task. Nevertheless, our theory indicates that both normalized and unnormalized smoothness of the output of each GCL can be adjusted by altering the input's projection onto M . As such, we propose the following learnable smoothness control term to modulate the smoothness of each dimension of the learned node features

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

where l is the layer index, { e i } i m =1 is the orthonormal basis of the eigenspace M , and α l := { α l i } i m =1 is a collection of learnable vectors with α l i ∈ R d being approximated by a multi-layer perceptron (MLP). The detailed configuration of α l i will be specified in each experiment later. One can see that B l α always lies in R d ⊗M . We integrate SCT into GCL, resulting in

We call the corresponding model GCN-SCT. Again, the idea is that we alter the component in eigenspace to control the smoothness of features . Each dimension of H l can be smoother, less smooth, or the same as H l -1 in normalized smoothness, though H l gets closer to M than H l -1 .

To design SCT, we introduce a learnable matrix A l ∈ R d × m for layer l , whose columns are α l i , where m is the dimension of the eigenspace M and d is the dimension of the features. We observe in our experiments that the SCT performs best when informed by degree pooling over the subcomponents of the graph. The matrix of the orthogonal basis vectors, denoted by Q := [ e 1 , . . . , e m ] ∈ R n × m , is used to perform pooling H l Q for input H l . In particular, we let A l = W ⊙ ( H l Q ) , where W ∈ R d × m is learnable and performs pooling over H l using the eigenvectors Q . The second architecture uses a residual connection with hyperparameter β l = log( θ/l +1) and learnable matrices W 0 , W 1 ∈ R d × d and the softmax function ϕ . Resulting in A l = ϕ ( H l Q ) ⊙ ( β l W 0 H 0 Q +(1 -β l ) W 1 H l Q ) . In Section 6, we use the first architecture for GCN-SCT as GCN uses only H l information at each layer. We use the second architecture for GCNII-SCT and EGNN-SCT which use both H 0 and H l information at each layer. There are two particular advantages of the above design of SCT: (1) it can effectively change the normalized smoothness of the learned features, and (2) it is computationally efficient since we only use the eigenvectors corresponding to the eigenvalue 1 of matrix G , which is determined based on the connectivity of the graph.

5.1 Integrating SCT into other GCN-style models

In this subsection, we present other usages of the proposed SCT. Due to the page limit, we carefully select two other most representative models. The first example is GCNII [6], GCNII extends GCN to express an arbitrary polynomial filter rather than the Laplacian polynomial filter and achieves state-of-the-art (SOTA) performance among GCN-style models on various tasks [6, 23], and we aim to show that SCT can even improve the accuracy of the GCN-style model that achieves SOTA performance on many node classification tasks. The second example is energetic GNN (EGNN) [40], which controls the smoothness of node features by constraining the lower and upper bounds of the Dirichlet energy of features and assuming the activation function is linear. In this case, we aim to show that our new theoretical understanding of the role of activation functions and the proposed SCT can boost the performance of EGNN with considering nonlinear activation functions.

GCNII. Each GCNII layer uses a skip connection to the initial layer H 0 and given as follows:

where α l , β l ∈ (0 , 1) are learnable scalars. We integrate SCT B l α into GCNII, resulting in the following GCNII-SCT layers

where the residual connection and identity mapping are consistent with GCNII.

EGNN. Each EGNN layer can be written as follows:

where c 1 , c 2 are learnable weights that satisfy c 1 + c 2 = c min with c min being a hyperparameter. To constrain Dirichlet energy, EGNN initializes trainable weights W l as a diagonal matrix with explicit singular values and regularizes them to keep the orthogonality during the model training. Ignoring the activation function σ , H l - node features at layer l of EGNN satisfies

where c max is the square of the maximal singular value of the initialization of W 1 . Similarly, we modify EGNN to result in the following EGNN-SCT layer

where everything remains the same as the EGNN layer except that we add our proposed SCT B l α .

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

6 Experiments

In this section, we comprehensively demonstrate the effects of SCT - in the three most representative GCN-style models discussed in Section 5 - using various node classification benchmarks. The purpose of all experiments in this section is to verify the efficacy of the proposed SCT - motivated by our theoretical results - for GCN-style models. We consider the citation datasets (Cora, Citeseer, PubMed, Coauthor-Physics, Ogbn-arxiv), web knowledge-base datasets (Cornell, Texas, Wisconsin), and Wikipedia network datasets (Chameleon, Squirrel). We provide additional dataset details in Appendix D.1. We implement baseline GCN [20] and GCNII [6] (without weight sharing) using PyG (Pytorch Geometric) [10]. Baseline EGNN [40] is implemented using the public code 4 .

6.1 Node feature trajectory

We visualize the trajectory of the node features, following [27], for a graph with two nodes connected by an edge and 1D node feature. In this case, (6) becomes h 1 = σ ( w h 0 G + b α ) , where w = 1 . 2 in our experiment, h 0 , h 1 , b α ∈ R 2 , and G ∈ R 2 × 2 . We use a matrix G = [0 . 592 , 0 . 194; 0 . 194 , 0 . 908] whose largest eigenvalue is 1. Twenty initial node feature vectors h 0 are sampled evenly in the domain [ -1 , 1] × [ -1 , 1] . Fig. 2 shows the trajectories in

a)

α

=

-

0

.

25

b)

α

= 0

.

0

c)

α

= 1

.

0

relation to the eigenspace M (red dashed line). In Fig 2a), one can see that some trajectories do not directly converge to M . In Fig. 2b) when α = 0 . 0 , GCL is recovered and all trajectories converge to M . In Fig. 2c), large values of α enable the features to significantly deviate from M initially. We observe that the parameter α can effectively change the trajectory of features.

6.2 Baseline comparisons for node classification

Citation networks. We compare the three representative models discussed in Section 5, of different depths, with and without SCT in Table 1. This task uses the citation datasets with fixed splits from [35] for Cora, Citeseer, and Pubmed and splits from [40] for Coauthor-Physics and Ogbn-arxiv; a detailed description of these datasets and splits are provided in Appendix D. Following [6], we use a single training pass to minimize the negative log-likelihood loss using the Adam optimizer [19], with 1500 maximum epochs, and 100 epochs of patience. A grid search for possible hyperparameters is listed in Table 5 in Appendix D. We accelerate the hyperparameter search by applying a Bayesian meta-learning algorithm [2] which minimizes the validation loss, and we run the search for 200 iterations per model. In particular, Table 1 presents the best test accuracy between ReLU and leaky ReLU for GCN, GCNII, and all three models with SCT 5 . For the baseline EGNN, we follow [40] using SReLU, a particular activation used for EGNN in [40]. These results show that SCT can boost

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

the classification accuracy of baseline models; in particular, the improvement can be remarkable for GCN and GCNII. However, EGNN-SCT (using ReLU or leaky ReLU) performs occasionally worse than EGNN (using SReLU), and this is because of the choice of activation functions. In Appendix D.3, we report the results of EGNN-SCT using SReLU, showing that EGNN-SCT outperforms EGNN in all tasks. In fact, SReLU is a shifted version of ReLU, and our theory for ReLU applies to SReLU as well. The model size and computational time are reported in Table 4 in the appendix.

Table 1 also shows that even with SCT, the accuracy of GCN drops when the depth is 16 or 32. This motivates us to investigate the smoothness of the node features learned by GCN and GCN-SCT. Fig. 3 plots the heatmap of the normalized smoothness of each dimension of the learned node features learned by GCN and GCN-SCT with 32 layers for Citeseer node classification. In these plots, the horizontal and vertical dimensions denote the feature dimension and the layer of the model, resp. We notice that the normalized smoothness of each dimension of the features - from layers 14 to 32 learned by GCN - closes to 1, confirming that deep GCN learns homogeneous features. In contrast, the features learned by GCN-SCT are inhomogeneous, as shown in Fig. 3b). Therefore, we believe the performance degradation of deep GCN-SCT is due to other factors. Compared to GCNII/GCNII-SCT and EGNN/EGNN-SCT, GCN-SCT does not use skip connections, which is known to help avoid vanishing gradients in training deep neural networks [16, 17]. In Appendix D.3, we show that training GCN and GCN-SCT do suffer from the vanishing gradient issue; however, the other models do not. Besides Citeseer, we notice similar behavior occurs for training GCN and GCN-SCT for Cora and Coauthor-Physics node classification tasks. 1 1.0 1 1.0

Other datasets. We further compare different models trained on different datasets using 10-fold cross-validation and fixed 48 / 32 / 20% splits following [28]. Table 2 compares GCN and GCNII with and without SCT, using leaky ReLU, for classifying five heterophilic node classification datasets. We exclude EGNN as these heterophilic datasets are not considered in [40]. We report the average accuracy of GCN and GCNII from [6]. We tune all other models using a Bayesian meta-learning algorithm to maximize the mean validation accuracy. We report the best test accuracy for each model of depth searched over the set { 2 , 4 , 8 , 16 , 32 } . SCT can significantly improve the classification accuracy of the baseline models. Table 2 also contrasts the computational time (on Tesla T4 GPUs from Google Colab) per epoch of models that achieve the best test accuracy; the models using SCT can even save computational time to achieve the best accuracy which is because

1

8

16

dim

16

32

Layer

0.0

0.2

0.4

0.6

0.8

1

8

16

dim

16

32

Layer

0.0

0.2

0.4

0.6

0.8

a) GCN

b) GCN-SCT

the best accuracy is achieved at a moderate depth (Table 8 in Appendix D.4 lists the mean and standard deviation for the test accuracies on all five datasets. Table 9 in Appendix D.4 lists the computational time per epoch for each model of depth 8, showing that using SCT only takes a small amount of computational overhead.

7 Concluding Remarks

In this paper, we establish a geometric characterization of how ReLU and leaky ReLU affect the smoothness of the GCN features. We further study the dimension-wise normalized smoothness of the learned node features, showing that activation functions not only smooth node features but also can reduce or preserve the normalized smoothness of the features. Our theoretical findings inform the design of a simple yet effective SCT for GCN. The proposed SCT can change the smoothness, in terms of both normalized and unnormalized smoothness, of the learned node features by GCN.

Limitations: Our proposed SCT provides provable guarantees for controlling the smoothness of features learned by GCN and related models. A key aspect to establish our theoretical results is demonstrating that, without SCT, the features of the vanilla model tend to be overly smooth; without this condition, SCT cannot ensure performance guarantees.

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

8 Broader Impacts

Our paper focuses on developing new theoretical understandings of the smoothness of node features learned by graph convolutional networks. The paper is mainly theoretical. We do not see any potential ethical issues in our research; all experiments are carried out using existing benchmark settings and datasets.

Our paper brings new insights into building new graph neural networks with improved performance over existing models, which is crucial for many applications. In particular, for applications where graph neural network is the method of choice. We expect our approach to play a role in material science and biophysics applications.

References

Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018.

Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and deep locally connected networks on graphs. In 2nd International Conference on Learning Representations, ICLR 2014 , 2014.

Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318 , 2020.

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3438-3445, 2020.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1725-1735. PMLR, 13-18 Jul 2020.

Zhengdao Chen, Lisha Li, and Joan Bruna. Supervised community detection with line graph neural networks. In International Conference on Learning Representations , 2019.

Fan RK Chung. Spectral graph theory , volume 92. American Mathematical Soc., 1997.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems , 29, 2016.

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Combining neural networks with personalized pagerank for classification on graphs. In International Conference on Learning Representations , 2019.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 1263-1272. JMLR.org, 2017.

William Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

William L Hamilton. Graph representation learning . Morgan & Claypool Publishers, 2020. 421

William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods 422 and applications. arXiv preprint arXiv:1709.05584 , 2017. 423

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im424 age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern 425 Recognition (CVPR) , June 2016. 426

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual 427 networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The 428 Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 630-645. Springer, 2016. 429

430

431

Tatsuro Kawamoto, Masashi Tsubaki, and Tomoyuki Obuchi. Mean-field theory of graph neural networks in graph partitioning. Advances in Neural Information Processing Systems , 31, 2018.

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference on artificial intelligence , 2018.

Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Revisiting heterophily for graph neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Yimeng Min, Frederik Wenkel, and Guy Wolf. Scattering gcn: Overcoming oversmoothness in graph convolutional networks. Advances in Neural Information Processing Systems , 33:1449814508, 2020.

Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) , pages 807-814, 2010.

Hoang Nt and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv:1905.09550 , 2019.

Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference on Learning Representations , 2020.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. In International Conference on Learning Representations , 2020.

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Representations , 2020.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 6861-6871. PMLR, 09-15 Jun 2019.

Xinyi Wu, Zhengdao Chen, William Wei Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. In The Eleventh International Conference on Learning Representations , 2023.

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems , 32(1):4-24, 2020.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning , pages 5453-5462. PMLR, 2018.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 974-983, 2018.

Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In International Conference on Learning Representations , 2020.

Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems , 16, 2003.

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI Open , 1:57-81, 2020.

Kaixiong Zhou, Xiao Huang, Daochen Zha, Rui Chen, Li Li, Soo-Hyun Choi, and Xia Hu. Dirichlet energy constrained learning for deep graph neural networks. Advances in Neural Information Processing Systems , 34:21834-21846, 2021.

Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian 494 fields and harmonic functions. In Proceedings of the 20th International conference on Machine 495 learning (ICML-03) , pages 912-919, 2003. 496

Appendix for 'Learning to Control the Smoothness of GCN 497 Features" 498

A Details of Notations 499

For two vectors u = ( u 1 , u 2 , . . . , u d ) and v = ( v 1 , v 2 , . . . , v d ) , their inner product is defined as

their Hadamard product is defined as

and their Kronecker product is defined as

The Kronecker product can be defined for two vectors of different lengths in a similar manner as 500 above. 501

B Proofs in Section 3 502

First, we prove that the two smoothness notions used in [27, 4] are two equivalent seminorms, i.e., 503 we prove Proposition 3.1 below. 504

Proof of Proposition 3.1. The matrix H can be decomposed as H = ∑ n i =1 He i e ⊤ i , where each e i is the eigenvector of G associated with eigenvalue λ i . This indicates that

Then using the fact that 1 -λ i ≥ 0 for each i , we obtain

That is,

On the other hand, (3) implies

We first show that both ∥ H ∥ M ⊥ and ∥ H ∥ E are seminorms. Since ∥ c H ∥ F = | c | · ∥ H ∥ F for any c ∈ R , we have ∥ c H ∥ M ⊥ = | c | · ∥ H ∥ M ⊥ and ∥ c H ∥ E = | c | · ∥ H ∥ E . Moreover, for any two matrices H 1 and H 2 s.t. H = H 1 + H 2 , we have

Then the triangle inequality of ∥ · ∥ F implies that of ∥ H ∥ M ⊥ and ∥ H ∥ E , respectively. 505

Now since 0 < 1 -λ m +1 ≤ 1 -λ i ≤ 2 for any i = m +1 , . . . , n , we may take α = √ 1 -λ m +1 and β = √ 2 . Then

The result thus follows from ∥ H ∥ E = ∥ ∥ ∥ ∑ n i = m +1 √ 1 -λ i He i e ⊤ i ∥ ∥ ∥ F . 506

B.1 ReLU 507

We present a crucial tool to characterize how ReLU affects its input. 508

Lemma B.1. Let Z ∈ R d × n , and let Z + = max( Z , 0) and Z -= max( -Z , 0) be the positive and 509 negative parts of Z . Then (1) Z + , Z -are (component-wise) nonnegative and Z = Z + -Z -and 510

⟨ Z + , Z -⟩ F = 0 . 511

Proof of Lemma B.1. Notice that for any a ∈ R , we have

This implies that a = max( a, 0) -max( -a, 0) and max( a, 0) · max( -a, 0) = 0 . 512

Let Z ij be the ( i, j ) th entry of Z . Then Z = Z + -Z -follows from Z ij = max( Z ij , 0) -max( -Z ij , 0) . Also, one can deduce that

Before proving Proposition 3.2, we notice the following relation between Z and H .

Lemma B.2. Given Z ∈ R d × n , let H = σ ( Z ) with σ being ReLU, then H lies on the highdimensional sphere, in ∥ · ∥ F norm, that is centered at Z / 2 and with radius ∥ Z / 2 ∥ F . That is, H and Z satisfy the following equation

Proof of Lemma B.2. We observe that H = σ ( Z ) = max( Z , 0) = Z + is the positive part of Z . Then

where we have used Z = Z + -Z -and ⟨ H , Z -⟩ F = ⟨ Z + , Z -⟩ F = 0 from Lemma B.1. 518 Therefore, one can deduce the desired result as follows

519

Applying ∥ H ∥ 2 F = ∥ H M + H M ⊥ ∥ 2 F = ∥ H M ∥ 2 F + ∥ H M ⊥ ∥ 2 F , to both Z 2 and H -Z 2 , we obtain

513

514

515

516

517

and

Then (8) becomes 520

By direct calculation, we have 521

Combining (9) and (10), we obtain the following result 522

Lemma B.3. For any Z = Z M + Z M ⊥ , let H = σ ( Z ) = H M + H M ⊥ , then 523

where Z + M = ∑ m i =1 Z + e i e ⊤ i , Z -M = ∑ m i =1 Z -e i e ⊤ i . 524

Proof of Lemma B.3. Recall that H = σ ( Z ) = max( Z , 0) = Z + . Also, Z = Z + -Z -implies Z M = Z + M -Z -M = H + M -Z -M . Therefore, we see that

525

By using the fact that ⟨ Z + M , Z -M ⟩ F ≥ 0 in Lemma B.3, we reveal a geometric relation between Z 526 and H mentioned in Proposition 3.2. 527

Proof of Proposition 3.2. Since Z + , Z -≥ 0 are nonnegative and all the eigenvectors e i are also nonnegative, we see that Z + M = ∑ m i =1 Z + e i e ⊤ i and Z -M = ∑ m i =1 Z -e i e ⊤ i are nonnegative. This indicates that

Then according to Lemma B.3, we obtain

So we have

531

532

533

534

535

536

537

538

539

which shows that H M ⊥ lies on the high-dimensional sphere that we have claimed. Furthermore, we 528 conclude that 529

This demonstrates that H M ⊥ lies on the high-dimensional sphere we have stated. 530

Since the sphere ∥ ∥ ∥ H M ⊥ -Z M⊥ 2 ∥ ∥ ∥ 2 F = ∥ ∥ ∥ Z M⊥ 2 ∥ ∥ ∥ 2 F passes through the origin, the distance of any

H M ⊥ to the origin must be no greater than the diameter of this sphere, i.e., ∥ H M ⊥ ∥ F ≤ ∥ Z M ⊥ ∥ F . Also, this can be derived from

One can see that the maximal smoothness ∥ H M ⊥ ∥ F = ∥ Z M ⊥ ∥ F is attained when H M ⊥ = Z M ⊥ , the intersection of the surface and the line passing through the center and the origin.

After all, we complete the proof by using the fact that ∥ Z M ⊥ ∥ F = ∥ Z ∥ M ⊥ for any matrix Z , which implies ∥ H ∥ M ⊥ = ∥ H M ⊥ ∥ F ≤ ∥ Z M ⊥ ∥ F = ∥ Z ∥ M ⊥ .

B.2 Leaky ReLU

For the leaky ReLU activation function, we have

Lemma B.4. If H = σ a ( Z ) with σ a being leaky ReLU, then H lies on the high-dimensional sphere centered at (1 + a ) Z / 2 with radius ∥ (1 -a ) Z / 2 ∥ F .

544

Proof of Lemma B.4. Notice that

Then H -Z = (1 -a ) Z -and H -a Z = (1 -a ) Z + . Using ⟨ Z -, Z + ⟩ F = 0 , we have 540

Moreover, we notice that 542

Lemma B.5. For any Z = Z M + Z M ⊥ , let H = σ a ( Z ) = H M + H M ⊥ , then 543

Proof of Lemma B.5. Similar to the proof of Lemma B.3, the orthogonal decomposition implies that

Proof of Proposition 3.3. Similar to the proof of Proposition 3.2, we apply ⟨ Z -M , Z + M ⟩ F ≥ 0 to Lemma B.5 and hence obtain the geometric condition as follows

Then we have the following inequality

Moreover, we deduce that

and hence

Therefore, we obtain a ∥ Z M ⊥ ∥ F ≤ ∥ H M ⊥ ∥ F ≤ ∥ Z M ⊥ ∥ F . (Remark that H M ⊥ achieves its 545 maximal norm when it is equal to Z M ⊥ , the intersection of the surface and the line passing through 546 the center and the origin. ) 547

550

551

552

553

554

C Proofs in Section 4

̸

Throughout this section, we assume that z M ⊥ = 0 .

Proof of Proposition 4.3. Recall that e = ˜ D 1 2 u n /c has only positive entries where ˜ D is the augmented degree matrix and u n = [1 , . . . , 1] ⊤ ∈ R n and c = ∥ ˜ D 1 2 u n ∥ . Let d i be the i th diagonal entry of ˜ D . Then we have e = [ √ d 1 /c, √ d 2 /c, . . . , √ d n /c ] ⊤ and c = √∑ n i =1 d i .

Note that z ( α ) = z -α e = z -α c ˜ D 1 2 u n = ˜ D 1 2 ( ˜ D -1 2 z -α c u n ) = ˜ D 1 2 ( x -α c u n ) , where we assume x := ˜ D -1 2 z . Then we observe that when σ is the ReLU activation function,

and hence

We may now assume x = [ x 1 , . . . , x n ] ⊤ is well-ordered s.t. x 1 ≥ x 2 ≥ . . . ≥ x n . Indeed, there is a collection of indices { k 1 , ..., k l } s.t.

̸

We first restrict the domain of α s.t. h ( α ) = 0 . Note that we have

So we will study the smoothness s ( h ( α )) when α < cx 1 . 556

Let ϵ > 0 and consider α = c ( x 1 -ϵ ) . When ϵ ≤ x 1 -x k 1 +1 = x 1 -x k 2 , we see that

where only the first k 1 entries are positive since x 1 -x i ≥ ϵ for any i ≥ k 1 +1 . Therefore,

and hence we can compute that ∥ h ( α ) ∥ = ϵ √ √ √ √ k 1 ∑ i =1 d i . Also, we have

Then we obtain the smoothness s ( h ( α )) as follows

Now we are going to show that the smoothness s ( h ( α )) is increasing as α gets smaller whenever α < 558 cx 1 , implying K 1 c is the minimum of the smoothness s ( h ( α )) . Remember that we are considering 559 α = c ( x 1 -ϵ ) and we have studied the case when 0 < ϵ ≤ x 1 -x k 1 +1 = x 1 -x k 2 . 560

Let δ j := x 1 -x k j for 1 ≤ j ≤ l . Clearly, we have δ 1 = 0 and δ j < δ j +1 for 1 ≤ j ≤ l -1 . Fix a j ′ ∈ { 2 , . . . , l -1 } , we see that when δ j ′ < ϵ ≤ x 1 -x k j ′ +1 ,

where we have ϵ -δ j > 0 for 2 ≤ j ≤ j ′ and ϵ -( x 1 -x i ) ≤ 0 for any i ≥ k j ′ +1 . Consequently,

Then we can compute

where we set k 0 := 0 for simplicity and K j = √ ∑ k j i = k j -1 +1 d i for j = 1 , . . . , j ′ . Also, we have

A careful calculation shows that ∂ ∂ϵ s ( h ( α )) > 0 whenever δ j ′ < ϵ ≤ x 1 -x k j ′ +1 which implies that s ( h ( α )) is increasing as ϵ increases. Indeed, we have

Then to show that ∂ ∂ϵ s ( h ( α )) > 0 , it suffices to show that the numerator is positive, i.e.

561

562

since the denominator c ∑ j ′ j =1 K 2 j ( ϵ -δ j ) 2 √ ∑ j ′ j =1 K 2 j ( ϵ -δ j ) 2 > 0 is always positive. In fact, this follows from the Cauchy inequality ∥ v ∥∥ u ∥ ≥ ⟨ v , u ⟩ , where we set

Moreover, equality happens only when v is parallel to u . This is, however, impossible since ϵ -δ j > ϵ -δ j +1 for any j = 1 , . . . , j ′ -1 and each K j is positive.

So we see that s ( h ( α )) is increasing as ϵ increases whenever 0 < ϵ , and hence the smoothness 563 s ( h ( α )) is increasing as α decreases whenever cx n ≤ α < cx 1 . 564

For the case j ′ = l where δ l = x 1 -x n < ϵ , we have x n -α/c = x n -( x 1 -ϵ ) = ϵ -( x 1 -x n ) > 0 , implying α < cx n and h ( α ) = z ( α ) . We have shown that the smoothness is increasing as α is going far from ⟨ z , e ⟩ ; in particular, when α < ⟨ z , e ⟩ and α is decreasing. One can check that

which means the smoothness is increasing as α decreases whenever α < cx n .

We conclude that the smoothness increases as α decreases provided α < cx 1 . Also, we have sup α<cx 1 s ( h ( α )) = 1 as the case in the proof of Proposition C.1. One can check that s ( h ( α )) is a continuous function for α < cx 1 and thus it has range [ K 1 /c, 1) by the mean value theorem.

Finally, we can establish the result: K 1 /c = √ ∑ x i =max x d i ∑ n j =1 d j is the minimum of s ( h ( α )) and 1 is the maximum of s ( h ( α )) occurring whenever α ≥ cx 1 = √ ∑ n j =1 d j max i x i . Moreover, s ( h ( α )) has

It is clear that the assumption on the ordering of the entries of x will not affect this result.

To prove Proposition 4.4, we first prove an analogous result for the identity function, that is, h = σ ( z ) = z .

̸

565

566

567

568

569

570

571

572

573

574

575

576

Proposition C.1. Suppose z M ⊥ = 0 , then s ( z ( α )) achieves its minimum 0 if α = ⟨ z , e ⟩ . Moreover, sup α s ( z ( α )) = 1 where s ( z ( α )) is close to 1 when α is far away from ⟨ z , e ⟩ .

Notice that Proposition C.1 does not consider the activation function. 577

Proof of Proposition C.1. We know that 0 ≤ s ( z ( α )) ≤ 1 and

̸

Suppose s ( z ( α )) = 1 . Then we have ∥ z M⊥ ∥ 2 ∥ z M⊥ ∥ 2 + ∥ z M -α e ∥ 2 = 0 which forces ∥ z M ⊥ ∥ = 0 . However, 578 this contradicts the hypothesis z M ⊥ = 0 . So s ( z ( α )) cannot attain its maximum. 579

But for any 0 ≤ t < 1 , one can see that s ( z ( α )) = t if and only if

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

This implies that sup α s ( z ( α )) = 1 and s ( z ( α )) achieves its minimum 0 if and only if α = ⟨ z , e ⟩ . It is clear that s ( z ( α )) get closer to 1 when α is going far away from ⟨ z , e ⟩ . i.e., | α -⟨ z , e ⟩| = ∥ z M -α e ∥ is increasing.

Proof of Proposition 4.4. First, we notice that leaky ReLU has the following two properties

2. σ a is a non-trivial linear map for x ≫ 0 .

Wewill use Property 1 to show that min α s ( h ( α )) = 0 and Property 2 to show that sup α s ( h ( α )) = 1 . Notice that σ a ( x ) < 0 for x ≪ 0 implies that there exists a sufficient small α 2 < 0 s.t. all of the entries of h ( α 2 ) are negative and hence |⟨ h ( α 2 ) , e ⟩| < 0 . Similarly, σ a ( x ) > 0 for x ≫ 0 implies that there exists a sufficient large α 1 > 0 s.t. all of the entries of h ( α 1 ) are positive and hence |⟨ h ( α 1 ) , e ⟩| > 0 . Since |⟨ h ( α ) , e ⟩| is a continuous function of α on [ α 1 , α 2 ] , the Intermediate Value Theorem follows that there exists an α ∈ ( α 1 , α 2 ) s.t. |⟨ h ( α ) , e ⟩| = 0 . Thus by definition s ( h ( α )) = |⟨ h ( α ) , e ⟩| / ∥ h ( α ) ∥ , we see that min α s ( h ( α )) = 0 .

̸

On the other hand, since σ a is a non-trivial linear map for x ≫ 0 , we may assume σ a ( x ) = cx for x > x 0 where c = 0 is some non-zero constant and x 0 > 0 is some positive constant. Then we can choose an α 0 > ⟨ z , e ⟩ s.t. for any α ≥ α 0 , all of the entries of z ( α ) are greater than x 0 . Then whenever α ≥ α 0 , we have h ( α ) = σ a ( z ( α )) = c z ( α ) . This implies

Thus sup α s ( h ( α )) = 1 follows from the Proof of Proposition C.1 where we see that sup α s ( z ( α )) = 1 since s ( z ( α )) gets closer to 1 as α increases.

Remark C.2 . Indeed, it holds for any continuous function f : R → R satisfying the following

f is a non-trivial linear map for x ≫ 0 or x ≪ 0 .

One can check the proof above only depends on these two properties. It is worth mentioning that most activation functions, e.g. leaky LU, SiLU, tanh , satisfy condition 1.

̸

Proof of Corollary 4.5. For any α , we notice that ∥ z ∥ M ⊥ = ∥ z M ⊥ ∥ F = ∥ z ( α ) ∥ M ⊥ since α only changes the component of z in the eigenspace M . Also, Propositions 3.2 and 3.3 show that ∥ z ( α ) ∥ M ⊥ ≥ ∥ h ( α ) ∥ M ⊥ whenever h ( α ) = σ ( z ( α )) or σ a ( z ( α )) . Therefore, we see that ∥ z ∥ M ⊥ ≥ ∥ h ( α ) ∥ M ⊥ holds for any α . Since z M ⊥ = 0 , s ( z ) must lie in [0 , 1) .

D Experimental Details

This part includes the missing details about experimental configurations and additional experimental results for Section 6. All tasks we run using Nvidia RTX 3090, GV100, and Tesla T4 GPUs. All computational performance metrics, including timing procedures, are run using Tesla T4 GPUs from Google Colab.

D.1 Dataset details

In this section, we briefly describe the benchmark datasets used. Table 3 provides additional details about the underlying graph representation.

Citation Datasets: The five citation datasets considered are Cora, Citeseer PubMed, CoauthorPhysics, and Ogbn-arxiv. Each dataset is represented by a graph with nodes representing academic publications, features encoding a bag-of-words description, labels classifying the publication type, and edges representing citations.

Web Knowledge-Base Datasets: The three web knowledge-base datasets are Cornell, Texas, and 618 Wisconsin. Each dataset is represented by a graph with nodes representing CS department webpages, 619 features encoding a bag-of-words description, edges representing hyper-link connections, and labels 620 classifying the webpage type. 621

Wikipedia Network Datasets: The two Wikipedia network datasets are Chameleon and Squirrel. 622 Each dataset is represented by a graph with nodes representing CS department webpages, features en623 coding a bag-of-words description, edges representing hyper-link connections, and labels classifying 624 the webpage type. 625

D.2 Model size and computational time for citation datasets 626

Table 4 compares the model size and computational time for experiments on citation datasets in 627 Section 6.2. 628

D.3 Additional Section 6.2 details for citation datasets 629

Table 5 lists the hyperparameters used in the grid search in generating the results in Table 1. Also, 630 Table 7 reports the classification accuracy of different models with different depths using either ReLU 631 or leaky ReLU.

D.3.1 Vanishing gradients

Figure 4 shows the vanishing gradient problem for training deep GCN - with or without SCT - in comparison to models like GCNII and EGNN. This figure plots || ∂ H out /∂ H l || for layers l ∈ [0 , 32] as the training epochs run from 0 to 100 . Figures 4 (a) and (b) illustrate the vanishing gradient issue for GCN and that it persists for GCN-SCT. Figures 4 (c) and (e) illustrate that GCNII and EGNN do not suffer from vanishing gradients, and furthermore, because these models connect H 0 to every layer, the gradient with respect to the weights in the first layer is nonzero. What is interesting about the addition of SCT to both EGNN and GCNII is that the intermediate gradients become large as the training epochs progress shown in Figure 4 (d) and (f).

D.4 Additional Section 6.2 details for other datasets

632

633

634

635

636

637

638

639

640

641

642

643

644

Table 8 reports the mean test accuracy and standard deviation over ten folds of the WebKB and WikipediaNetwork datasets using SCT-based models.

Table 9 lists the average computational time for each epoch for different models of the same depth 645 - 8 layers. These results show that integrating SCT into GNNs only results in a small amount of 646 computational overhead. 647

0

20

40

60

80

Epoch

0

20

Layer

0.00

0.05

0.10

0

20

40

60

80

Epoch

0

20

Layer

0.00

0.05

0.10

0

20

40

60

80

Epoch

0

20

Layer

0.00

0.05

0.10

(a) GCN

(e) EGNN

0

20

40

60

80

Epoch

0

20

Layer

0.00

0.05

0.10

0

20

40

60

80

Epoch

0

20

Layer

0.00

0.05

0.10

0

20

40

60

80

Epoch

0

20

Layer

0.00

0.05

0.10

(b) GCN-SCT

(d) GCNII-SCT

(f) EGNN-SCT

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See details in Sections 3, 4, 5, and 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Sections 3 and 4 for details.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 6 and supplementary materials for details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See supplementary materials for details.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 6 for details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Section 6 for details.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 6 for details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have fully complied with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section 8 for details.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The data used in this paper are all benchmark tasks established by the community.

Guidelines: 902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have fully acknowledged baseline models, codes, and data in our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have provided details documents for the codes.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Learning to Control the Smoothness of GCN Features\" investigates the impact of activation functions, specifically ReLU and leaky ReLU, on the smoothness of node features in Graph Convolutional Networks (GCNs). It provides a geometric characterization of these effects, showing how altering the input's projection onto eigenspace M can control the smoothness of output features. The study introduces a Smoothness Control Term (SCT) to modulate the smoothness of node features, aiming to improve node classification tasks in both homophilic and heterophilic graphs. Experimental results validate the efficacy of SCT, demonstrating significant improvements in node classification accuracy for several GCN-style models .\n\nSTRENGTHS:\n## Originality\n- The paper introduces a novel approach to control the smoothness of Graph Convolutional Networks (GCNs) features, which is a significant departure from traditional methods. It builds upon and extends the work of Oono & Suzuki and Cai & Wang by integrating geometric insights with the message-passing process in GCNs.\n\n## Quality\n- The paper provides a robust theoretical framework, including geometric characterizations and proofs, that underpin the proposed methods.\n- Extensive experiments validate the theoretical claims, showing significant improvements in node classification accuracy. Detailed descriptions of the experimental setup, including datasets and hyperparameter tuning, enhance the reproducibility of the results.\n\n## Clarity\n- The paper is well-structured, with clear sections that logically flow from introduction to theoretical analysis, experimental validation, and conclusions.\n- The paper provides a comprehensive review of related work, situating its contributions within the broader context of graph neural networks research.\n\n## Significance\n- The ability to control the smoothness of GCN features addresses a to me interesting and important challenge in graph neural networks, with potential applications in various domains such as social network analysis, biological networks, and recommendation systems.\n- The proposed SCT shows improvements in real-world datasets.\n- The insights gained from this work could inform future research on activation functions and feature smoothness in other types of neural networks.\n\nWEAKNESSES:\n## Weaknesses\n- The removal of white space in the paper makes it hard to read. The authors should really try and make the paper easier to read visually by not condensing as much math in the main text as possible. Not only is this arguably in violation of the guidelines, but it also illustrates that the authors need to distinguish clearer what the main contributions are and which parts in the main text can go to an appendix.\n- While the geometric insights provided are valuable, the complexity of the mathematical formulations is challenging for readers not well-versed in advanced geometry and spectral graph theory. Simplifying explanations or providing more intuitive examples could enhance accessibility. Moreover, I feel like the math could be made more intuitive by giving verbal explanations before the theorems. The cramming of the paper and not highlighting enough what the main contributions should be improved.\n- Although the paper compares SCT with a few baseline models, it would benefit from a broader comparison with additional state-of-the-art methods in GCNs and GNNs to provide a more comprehensive evaluation of its effectiveness.\n- The experiments are primarily conducted on benchmark datasets. Incorporating more real-world applications and diverse datasets would demonstrate the practical relevance and versatility of the proposed method.\n- The drop in accuracy for deeper models (16 or 32 layers) is noted but not deeply analyzed. A more thorough investigation into the causes of this performance degradation, beyond mentioning vanishing gradients, could offer insights into potential improvements. What happens if you use techniques that combat eg oversmoothing?\n- While the paper mentions computational efficiency, a more detailed discussion on the computational overhead introduced by SCT, including potential trade-offs between accuracy and efficiency, would be beneficial.\n\nQUESTIONS:\n## Questions\n\n-  Could you provide more intuitive examples or visual aids to help readers better understand the geometric characterization of smoothness in GCN features?\n- Why did you choose the specific baselines for comparison? How would SCT perform against other state-of-the-art GCN and GNN models not included in your study?\n- Can you elaborate on the causes of performance degradation in deeper models (16 or 32 layers)? Have you considered any specific techniques to mitigate this issue?\n\nLIMITATIONS:\n## Limitations\n\nThe authors have acknowledged several limitations of their work, including the over-smoothing issue in deep GCNs and the dependence on specific activation functions, but they could improve by providing more detailed discussions on computational efficiency and potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of balancing smooth and non-smooth features in graph convolutional networks (GCNs) for node classification. Building on previous work that highlighted the correlation between feature smoothness and classification accuracy, the authors propose a novel method to control the smoothness of node features through a geometric approach and an augmented message-passing process. Their strategy involves establishing a geometric relationship between input and output vectors of activation functions like ReLU and Leaky-ReLU, and integrating a learnable term in the graph convolutional layers to modulate feature smoothness. The paper provides an empirical study to showcase the effectiveness of the proposed method.\n\nSTRENGTHS:\nThe paper offers a novel geometric insight into the effects of different activation functions, specifically ReLU and Leaky-ReLU, on smoothing in GCNs. Furthermore, it introduces an innovative method to enhance the message-passing framework for GCNs (and similar networks) to better control smoothing.\n\nWEAKNESSES:\n* The empirical results section is very weak and a major revision is needed in order to be able to judge the merit of the proposed method. The most significant weaknesses are:\n    * The paper does not compare to established baselines from other competing methods. For instance, one could take a look at the tables in https://arxiv.org/abs/2202.04579 or https://arxiv.org/abs/2210.00513 and add the results to the comparison in this paper (i.e., for cornell, texas, wisconsin, squirrel, and so on). \n    * The paper reports different (mostly weaker) performance for the baseline models such as GCN and GCNII. Again, this should be fixed and established results can be taken from the tables in the two papers mentioned above.\n    * The improvement using the proposed SCT is very marginal in many experiments, and in particular the performance still drops (or improves very little) for higher number of layers. It is somewhat expected that increasing the number of layers and solving oversmoothing does not help much in homophilic tasks, i.e., the tasks considered in Table 1. However, if the method works it should lead to significant improvements on heterophilic tasks, even for increasing number of layers. Therefore, please show the results for the datasets in Table 2 in the same way as you show them in Table 1, i.e., for increasing number of layers.\n    * It is claimed in Table 1 that the drop in performance of GCN for increasing number of layers is due to vanishing gradients and not due to oversmoothing. This claim has to be justified empirically. \n    * Figure 4 in the appendix showing gradient norms is not meaningful. The y-axis should be displayed on a logarithmic scale. Vanishing gradients occur for gradients approaching zero (exponentially fast). It is not clear if this is the case here, since it is plotted in linear scale.\n    \n* The structure and readability of this paper should be improved. For instance, it would be helpful for readers who are not very familiar with the graph-learning field, to introduce the concept of GNNs and GCNs in the introduction. It would be advisable to move the technical aspects from section 1 to section 2. In fact, none of the definitions presented at the beginning of the introduction are needed anywhere else in the remainder of the introduction section. Thus, this could be moved to section 2.\n\nQUESTIONS:\nhow expensive is it to compute the eigenbasis for the smoothness modulation term (5)? How does it scale with respect to the number of nodes and edges? How long does it take to compute for the biggest graph considered here, i.e., arxiv graph? Is that a limitation for large-scale graph learning?\n\nLIMITATIONS:\n* The method necessitates a pre-processing step to compute the eigenbasis in equation (5). This may not scale efficiently for very large graphs.\n* The method does not show significant improvements. Notably, there are more effective models and methods available for mitigating oversmoothing, which have not been cited or empirically compared. Examples include https://arxiv.org/abs/2202.04579, https://arxiv.org/abs/2210.00513, https://arxiv.org/abs/2206.05437, https://arxiv.org/abs/2110.14446, https://arxiv.org/abs/2206.10991,\nhttps://arxiv.org/abs/2006.11468.\n* While the theoretical justification of the approach is intriguing, the insights are somewhat limited. For example, there are no provided estimates for choosing the parameter alpha, which instead has to be learned through gradient descent.\n* The empirical results are not convincing.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how GCN smoothes node features in terms of unnormalized and normalized smoothness. The results show that adjusting projection can alter the normalized smoothness to any desired level. Based on this, the paper proposes a new method SCT to let GCN learn node features with a desired smoothness to enhance node classification and verifies it effectiveness in practice.\n\nSTRENGTHS:\n1.\tUnderstanding the effect of nonlinearities in GNNs is an important yet underexplored problem due to the complexity of nonlinearities. The paper offers a new perspective on it.\n\n2.\tOversmoothing is a known issue, while it is also known that some amount of smoothness is desired for graph learning. How to find the ideal amount of smoothness among node features is an important but nontrivial problem.\n\n3.\tThe proposed method SCT is principled and seems effective.\n\nWEAKNESSES:\n1.\tWhile I can see that normalized smoothness has its own merit, this notion could be better motivated and connected to the literature. \n\n   a.\tGiven analysis in [27, 4] is asymptotic, I would not say that “over-smoothing – characterized by the distance of features to eigenspace M or the Dirichlet energy – is a misnomer”, as it is too strong of a claim to make. Those results essentially say that “very deep GCNs are bad due to oversmoothing” which has their limitations but can be well justified by the distance of features to eigenspace M or the Dirichlet energy.\n\n   b.\tMy understanding is that normalized smoothness could be more connected to the non-asymptotic notion of oversmoothing studied in [32], which is defined based on the Bayes error of classification (the distance to the decision boundary) and hence taken the magnitude of features into account.\n\n   c.\tBased on the above, the argument presented in the paper can be strengthened in the following way: the motivation of normalized smoothness should be based on a discussion on how the magnitude (and hence normalized smoothness) is more related to a non-asymptotic notion of oversmoothing, which is directly related to the classification performance of finite-depth/shallow GNNs. Based on this, the results present in this paper is more practically relevant than the previous asymptotic result.\n\nI would suggest the authors modify the relevant text in the introduction and analysis accordingly.\n\n\n2.\tThe analysis only applies for GCNs, while whether the analysis or the proposed method can be extended to more complexed GNNs such as GATs or graph transformers is unclear. \n\n3.\tThe analysis only applies for ReLU and LeakyReLU, which reads a bit specific. I wonder if the results can be generalized to a general family of nonlinearities.\n\n4.  For the experiments, there is a lack of baseline comparisons except the basic backbone architecture. For example, I wonder how it would compare to APPNP, which is proposed to balance the need to explore larger neighborhood and locality and the implicit goal is also to produce node features with the \"right\" amount of smoothness.\n\n4.\tAnother presentation suggestion I have for the authors is that one should minimize the use of in-text math and bold or italic fonts for highlighting (such as line 167-173).  Math is hard to read in-text and when too many texts are highlighted, the paper becomes ever harder to read because everything seems to be emphasized and it kind of messes up with its original purpose.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first shows that in GCN, the output of ReLU or LeakyReLU lies on a sphere whose input is characterized by components parallel and perpendicular to $\\mathcal{M}$, the space spanned by eigenvectors for the maximum eigenvalue of a graph. As a corollary, this paper shows that these activation functions do not increase the component of the feature vector perpendicular to $\\mathcal{M}$. Furthermore, this paper defines the normalized smoothness and evaluates how its range varies with the activation functions. Based on this discussion, this paper proposes an SCT that learns the parallel components of the feature vectors. The proposed method is applied to GCN, GCNII, and EGNN, and verifies its effectiveness by applying it to node prediction tasks with various heterophily.\n\nSTRENGTHS:\n- The theorems presented in the theoretical analysis (Propositions 3.2 and 3.3) enable a unified treatment of ReLU and Leaky ReLU.\n- Numerical experiments show that the proposed method is effective for various data sets and models, showing the versatility of the proposed method.\n\nWEAKNESSES:\n- I need help understanding the explanation in Section 3.3. More specifically, it is difficult to understand that the *independence* of the inequality means that the upper bound of the inequality does not depend on the value of $\\boldsymbol{Z}_{\\mathcal{M}}$. I suggest writing it explicitly. \n- P7, L.265: It seems strange that although SCT changes its architecture depending on whether the underlying GNN is GCT or GCNII, it has the same name. I suggest naming SCT for GCT and SCT for GCNII differently.\n\nQUESTIONS:\nP5, L.196: $\\boldsymbol{e}$ -> $\\boldsymbol{e}_{1}$\nP6, L.209: What is the value of $a$ of $\\sigma_a$?\n\nLIMITATIONS:\nThe authors discuss in the conclusion section that the proposed method's limitation is that it assumes the model's oversmoothing. However, I do not think this is a limitation because if the model does not cause oversmoothing, there is no need to use the proposed method. Rather, I recommend evaluating whether SCT has bad effects when the model does not cause oversmoothing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with (Over-)smoothing is Graph Neural Networks. While it was previously known that GCN-type GNNs oversmooth, this paper reexamines the case for GCN-type architectures with Relu-type activation functions in terms of *normalized* smoothness. The authors show that the convergence behaviour of GCNs can be split into two parts, a \"smooth\" part and a \"non-smooth\" part and that by manipulating the smooth part, one can influence the normalized smoothness of the signal. From these theoretical insights, the authors propose a new system that uses a learnable parameter that modulates the smooth part, making the model able to learn the most beneficial normalized smoothness for the problem at hand.\n\nSTRENGTHS:\n- The paper furthers the understanding of oversmoothing in GNNs\n- The proposed approach is well-founded in theory\n\nWEAKNESSES:\n- The experiments are unconvincing. \n    - There is a slight improvement to be found in models with SCT, but this is not too surprising, as these models also have more parameters, and mostly improvements are quite slim.\n    - You choose GCN, GCNII and EGNN, two of which have built-in skip connections, that are known to help with oversmoothing. This also coincides with the models that cope quite well with a larger number of layers. The vanilla GCN takes a huge hit in all benchmarks apart from ogbn-arxiv. So learning the normalized smoothness of features does not actually seem to help in the case of GCN. \n    - The other two models don't suffer from oversmoothing to begin with\n\nQUESTIONS:\n- The parameter $\\alpha$ can be chosen such that the normalized smoothness does not diminish asymptotically. However, the unnormalized smoothness does; does this mean that the norm of the features $||z|| \\rightarrow 0$ tends towards 0? Doesn't this mean that features are also unusable in deeper layers?\n\n- Is this analysis extendable to other architectures? E.g. GAT and other attention-based methods are also known to oversmooth, can a similar trick be applied there?\n\n- Table 5 details the hyperparameters that were tried for each model. These seem to be very many combinations. Is it correct that you tried approx. 1.2 Million hyperparameter combinations for EGNN?\n\nLIMITATIONS:\nThe limitations are not well-discussed. \nThe only limitation the authors claim for this work is, that: \"without this condition [that oversmoothing happens], SCT cannot ensure performance guarantees.\" There are no performance guarantees given for SCT. This is the only limitation discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper studies how ReLU and Leaky ReLU affect the smoothness of node features in graph convolution layers. The authors demonstrate that adjusting the input projection onto eigenspace $\mathcal{M}$ of the node feature matrix can achieve any desired normalized smoothness. Additionally, they propose a Smoothness Control Term (SCT) to enhance node classification in Graph Convolutional Networks, validated on both homophilic and heterophilic graphs.

STRENGTHS:
1. From a geometric perspective, the authors prove that how ReLU and Leaky ReLU affect the smoothness of node features in graph convolution layers.
2. The experimental results validate the theory proposed by the authors.

WEAKNESSES:
Equation 5 implies that using SCT requires performing eigendecomposition. This paper avoids the high time complexity associated with eigendecomposition, especially when the number of nodes in a graph is very large.

QUESTIONS:
None

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11