Learning to Control the Smoothness of GCN Features

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 17:02:51

Abstract: The pioneering work of Oono \& Suzuki [ICLR, 2020] and Cai \& Wang [arXiv:2006.13318] analyze the smoothness of graph convolutional network (GCN) features. Their results reveal an intricate empirical correlation between node classification accuracy and the ratio of smooth to non-smooth feature components. However, the optimal ratio that favors node classification is unknown, and the non-smooth features of deep GCN with ReLU or leaky ReLU activation function diminish. In this paper, we propose a new strategy to let GCN learn node features with a desired smoothness to enhance node classification. Our approach has three key steps: (1) We establish a geometric relationship between the input and output of ReLU or leaky ReLU. (2) Building on our geometric insights, we augment the message-passing process of graph convolutional layers (GCLs) with a learnable term to modulate the smoothness of node features with computational efficiency. (3) We investigate the achievable ratio between smooth and non-smooth feature components for GCNs with the augmented message passing scheme. Our extensive numerical results show that the augmented message passing remarkably improves node classification for GCN and some related models.

Main paper: 
 introduction: 
 Let G = (V, E) be an undirected graph with V = {v i } n i=1 and E be the set of nodes and edges, resp.Let A ∈ R n×n be the adjacency matrix of the graph with A ij = 1 (i,j)∈E , where 1 is the indicator function. Furthermore, let G be the following (augmented) normalized adjacency matrixEQUATION)where I is the identity matrix, D is the degree matrix with D ii = n j=1 A ij , and Ã := A + I and D := D + I. Starting from the initial node features H 0 := [(h 0 1 ) ⊤ , . . . , (h 0 n ) ⊤ ] ⊤ ∈ R d×n with h 0 i ∈ R d being the i th node feature vector, the graph convolutional network (GCN) [20] learns node representations using the following graph convolutional layer (GCL) transformationEQUATIONwhere σ is the activation function, e.g. ReLU [25] , and W l ∈ R d×d is learnable. GCL smooths feature vectors of the neighboring nodes. The smoothness of features helps node classification; see e.g. [22, 31, 5] , resonating with the idea of classical semi-supervised learning approaches [41, 38] .Accurate node classification requires a balance between smooth and non-smooth components of GCN features [27] . Besides graph convolutional networks (GCNs) stacking GCLs, many other graph neural networks (GNNs) have been developed using different mechanisms, including spectral methods [3, 9] , spatial methods [12, 30] , sampling methods [13, 36] , and the attention mechanism [30] . Many other GNN models can be found in recent surveys or monographs; see, e.g. [15, 1, 33, 39, 14] .Deep neural networks usually outperform shallow architectures, and a remarkable example is convolutional neural networks [21, 16] . However, this does not carry to GCNs; deep GCNs tend to perform Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.significantly worse than shallow models [5] . In particular, the node feature vectors learned by deep GCNs tend to be identical over each connected component of the graph; this phenomenon is referred to as over-smoothing [22, 26, 27, 4, 5, 32] , which not only occurs for GCN but also for many otherGNNs, e.g., GraphSage [13] and MPNN [12] . Intuitively, each GCL smooths neighboring node features, benefiting node classification [22, 31, 5] . However, stacking these smoothing layers will inevitably homogenize node features. Algorithms have been developed to alleviate the over-smoothing issue of GNNs, including decoupling prediction and message passing [11] , skip connection and batch normalization [18, 7, 6] , graph sparsification [29] , jumping knowledge [34] , scattering transform [24] , PairNorm [37] , and controlling the Dirichlet energy of node features [40] .From a theoretical perspective, it is proved that deep GCNs using ReLU or leaky ReLU activation function learn homogeneous node features [27, 4] . In particular, [27] shows that the distance of node features to the eigenspace M -corresponding to the largest eigenvalue 1 of matrix G in (1)-goes to zero when the depth of GCN with ReLU goes to infinity. Meanwhile, [27] empirically studies the intricate correlation between node classification accuracy and the ratio between smooth and non-smooth components of GCN node features, i.e., projections of node features onto eigenspace M and its orthogonal complement M ⊥ , resp. The empirical results of [27] indicate that both smooth and non-smooth components of node features are crucial for accurate node classification, while the ratio between smooth and non-smooth components to achieve optimal accuracy is unknown and task-dependent. Furthermore, [4] proves that the Dirichlet energy -another smoothness measure for node features -goes to zero when the depth of GCN with ReLU or leaky ReLU goes to infinity.A crucial step in the proofs of [27, 4] is that ReLU and leaky ReLU reduce the distance of feature vectors to M and their Dirichlet energy. However, [4] points out that over-smoothing -characterized by the distance of features to eigenspace M or the Dirichlet energy -is a misnomer; the real smoothness should be characterized by a normalized smoothness, e.g., normalizing the Dirichlet energy by the magnitude of the features. The ratio between smooth and non-smooth components of node features -studied in [27] -is closely related to the normalized smoothness. Nevertheless, analyzing the normalized smoothness of node features learned by GCN with ReLU or leaky ReLU remains an open problem [4] . Moreover, it is interesting to ask if analyzing the normalized smoothness can result in any new understanding of GCN features and algorithms to improve GCN's performance.We aim to (1) establish a new geometric understanding of how GCL smooths GCN features and(2) develop an efficient algorithm to let GCN and related models learn node features with a desired normalized smoothness to improve node classification. We summarize our main contributions towards achieving our goal as follows:• We prove that there is a high-dimensional sphere underlying the input and output vectors of ReLU or leaky ReLU. This geometric characterization not only implies theories in [27, 4] but also informs that adjusting the projection of input onto eigenspace M can alter the smoothness of the output vectors. See Section 3 for details.• We show that both ReLU and leaky ReLU reduce the distance of node features to eigenspace M, i.e., ReLU and leaky ReLU smooth their input vectors without considering their magnitude. In contrast, when taking the magnitude into account, ReLU and leaky ReLU can increase, decrease, or preserve the normalized smoothness of each dimension of the input vectors; see Sections 3 and 4.• Inspired by our established geometric relationship between the input and output of ReLU or leaky ReLU, we study how adjusting the projection of input onto eigenspace M affects both normalized and unnormalized smoothness of the output vectors. We show that the distance of the output to eigenspace M is no greater than that of the original input -no matter how we adjust the input by changing its projection onto M. In contrast, adjusting the projection of input vectors onto M can change the normalized smoothness of output to any desired value; see details in Section 4.• Based on our theory, we propose a computationally efficient smoothness control term (SCT)to let GCN and related models learn node features with a desired (normalized) smoothness to improve node classification. We comprehensively validate the benefits of our proposed SCT in improving node classification -for both homophilic and heterophilic graphs -using a few of the most representative GCN-style models. See Sections 5 and 6 for details.As far as we know, our work is the first thorough study of how ReLU and leaky ReLU affect the smoothness of node features both with and without considering their magnitude.Controlling the smoothness of node features to improve the performance of GCNs is another line of related work. For instance, [37] designs a normalization layer to prevent node features from becoming too similar to each other, and [40] constrains the Dirichlet energy to control the smoothness of node features without considering the effects of nonlinear activation functions. While there has been effort in understanding and alleviating the over-smoothing of GCNs and controlling the smoothness of node features, there is a shortage of theoretical examination of how activation functions affect the smoothness of node features, specifically accounting for the magnitude of features.Notation. We denote the ℓ 2 -norm of a vector u as ∥u∥. For vectors u and v, we use ⟨u, v⟩, u ⊙ v, and u ⊗ v to denote their inner, Hadamard, and Kronecker product, resp. For a matrix A, we denote its (i, j) th entry, transpose, and inverse as A ij , A ⊤ , and A -1 , resp. We denote the trace ofA ∈ R n×n as Trace(A) = n i=1 A ii .For two matrices A and B, we denote the Frobenius inner product as ⟨A, B⟩ F := Trace(AB ⊤ ) and the Frobenius norm of A as ∥A∥ F := ⟨A, A⟩.Organization. We provide preliminaries in Section 2. In Section 3, we establish a geometric characterization of how ReLU and leaky ReLU affect the smoothness of their input vectors. We study the smoothness of each dimension of node features and take their magnitude into account in Section 4.Our proposed SCT is presented in Section 5. We comprehensively verify the efficacy of the proposed SCT in Section 6. Technical proofs and more experimental results are provided in the appendix. 
preliminaries and existing results: 
 From the spectral graph theory [8] , we can sort eigenvalues of matrix G in (1) as1 = λ 1 = . . . = λ m > λ m+1 ≥ . . . ≥ λ n > -1, where m is the number of connected components of the graph. Wedecompose V = {v k } n k=1 into m connected components V 1 , . . . , V m . Let u i = (1 {v k ∈Vi}) 1≤k≤n be the indicator vector of V i , i.e., the k th coordinate of u i is one if the k th node v k lies in the connected component V i ; zero otherwise. Moreover, let e i be the eigenvector associated with λ i , then {e i } n i=1 forms an orthonormal basis of R n . Notice that {e i } m i=1 spans the eigenspace M -corresponding to eigenvalue 1 of matrix G, and {e i } n i=m+1 spans the orthogonal complement of M, denoted by M ⊥ .The paper [27] connects the indicator vectors u i s with the space M. In particular, we have For any matrix H := [h 1 , . . . , h n ] ∈ R d×n , we have the decompositionH = H M + H M ⊥ with H M = m i=1 He i e ⊤ i and H M ⊥ = n i=m+1 He i e ⊤ i such that ⟨H M , H M ⊥ ⟩ F = Trace m i=1 He i e ⊤ i ( n j=m+1 He j e ⊤ j ) ⊤ = 0, implying that ∥H∥ 2 F = ∥H M ∥ 2 F + ∥H M ⊥ ∥ 2 F .Distance to the eigenspace M. Oono et al. [27] study the smoothness of featuresH := [h 1 , . . . , h n ]using their distance to the eigenspace M as an unnormalized smoothness notion.). Let R d ⊗ M be the subspace of R d×n consisting of the sum m i=1 w i ⊗ e i , where w i ∈ R d and {e i } m i=1 is an orthonormal basis of the eigenspace M. Then we define ∥H∥ M ⊥ -the distance of node features H to the eigenspace M -as follows:∥H∥ M ⊥ := inf Y ∈R d ⊗M ∥H -Y ∥F = H - m i=1 Heie ⊤ i F .With the decompositionEQUATIONDirichlet energy. The paper [4] studies the unnormalized smoothness of node features using Dirichlet energy, which is defined as follows: Definition 2.Let λ = max{|λ i | | λ i < 1} be the second largest magnitude of G's eigenvalues, and s l be the largest singular value of weight matrix W l . [27] shows that∥H l ∥ M ⊥ ≤ s l λ∥H l-1 ∥ M ⊥ under GCL when σ is ReLU. Therefore, ∥H l ∥ M ⊥ → 0 as l → ∞ if s l λ < 1, indicating node features converge to Mand results in over-smoothing. A crucial step in the analysis in [27] is that ∥σ(Z)∥ M ⊥ ≤ ∥Z∥ M ⊥ , for any matrix Z when σ is ReLU, i.e., ReLU reduces the distance to M. [27] points out that it is hard to extend the above result to other activation functions even leaky ReLU.Instead of considering ∥H∥ M ⊥ , [4] shows that ∥H l ∥ E ≤ s l λ∥H l-1 ∥ E under GCL when σ is ReLU or leaky ReLU. Hence, ∥H l ∥ E → 0 as l → ∞, implying over-smoothing of GCNs. Note that ∥H∥ M ⊥ = 0 or ∥H l ∥ E = 0 indicates homogeneous node features. The proof in [4] applies to GCN with both ReLU and leaky ReLU by establishing the inequality ∥σ(Z)∥ E ≤ ∥Z∥ E for any matrix Z. 
effects of activation functions: a geometric characterization: 
 In this section, we present a geometric relationship between the input and output vectors of ReLU or leaky ReLU. We use ∥H∥ M ⊥ as the unnormalized smoothness notion for all subsequent analyses since we observe that ∥H∥ M ⊥ and ∥H∥ E are equivalent as seminorms. In particular, we have Proposition 3.1. ∥H∥ M ⊥ and ∥H∥ E are two equivalent seminorms, i.e., there exist two constantsα, β > 0 s.t. α∥H∥ M ⊥ ≤ ∥H∥E ≤ β∥H∥ M ⊥ ,for any H ∈ R d×n .Let σ(x) = max{x, 0} be ReLU. The first main result of this paper is that there is a high-dimensional sphere underlying the input and output of ReLU; more precisely, we haveProposition 3.2 (ReLU). For any Z = Z M + Z M ⊥ ∈ R d×n , let H = σ(Z) = H M + H M ⊥ .Then H M ⊥ lies on the high-dimensional sphere centered at Z M ⊥ /2 with radiusr := ∥Z M ⊥ /2∥ 2 F -⟨HM, HM -ZM⟩F 1/2 .In particular, H M ⊥ lies inside the ball centered at Z M ⊥ /2 with radius ∥Z M ⊥ /2∥ F and hence we have ∥H∥ M ⊥ ≤ ∥Z∥ M ⊥ .Now we consider leaky ReLU σ a (x) = max{x, ax}, where 0 < a < 1 is a positive scalar. Similar to ReLU, we have the following result for leaky ReLU Proposition 3.3 (Leaky ReLU). For anyZ = Z M + Z M ⊥ ∈ R d×n , let H = σ a (Z) = H M + H M ⊥ .Then H M ⊥ lies on the high-dimensional sphere centered at (1 + a)Z M ⊥ /2 with radiusra := ∥(1 -a)Z M ⊥ /2∥ 2 F -⟨HM -ZM, HM -aZM⟩F 1/2 .In particular, H M ⊥ lies inside the ball centered at (1 + a)Z M ⊥ /2 with radius ∥(1 -a)Z M ⊥ /Z, Z ′ ∈ R d×n s.t. Z M ⊥ = Z ′ M ⊥ but Z M ̸ = Z ′ M .Let H and H ′ be the output of Z and Z ′ via ReLU or leaky ReLU, resp. Then we have In contrast to the unnormalized smoothness, if one considers the normalized smoothness, we find that adjusting Z M can result in a less smooth output; we will discuss this in Section 4.1.∥H∥ M ⊥ ≤ ∥Z∥ M ⊥ and ∥H ′ ∥ M ⊥ ≤ ∥Z ′ ∥ M ⊥ . Since Z M ⊥ = Z ′ M ⊥ , we deduce that ∥H ′ ∥ M ⊥ ≤ ∥Z∥ M ⊥ . In other words, when Z M ⊥ = Z ′ M ⊥ is fixed, changing Z M to Z ′ M can4 How Adjusting Z M Affects the Smoothness of the Output Throughout this section, we let Z and H be the input and output of ReLU or leaky ReLU. The smoothness notions based on the distance of feature to M or their Dirichlet energy do not account for the magnitude of each dimension of the features; [4] points out that analyzing the normalized smoothness of features Z, given by ∥Z∥ E /∥Z∥ F , is an open problem. However, these two smoothness notions aggregate the smoothness of node features across all dimensions; when the magnitude of some dimensions is much larger than others, the smoothness will be dominated by them.Motivated by the discussion in Section 3.3, we study the disparate effects of adjusting Z M on the normalized and unnormalized smoothness in this section. For the sake of simplicity, we assume the graph is connected (m = 1); all the following results can be extended to graphs with multiple connected components easily. Due to the equivalence between seminorms ∥ • ∥ M and ∥ • ∥ E , we introduce the following definition of the dimension-wise normalized smoothness of node features.Definition 4.1. Let Z ∈ R d×n be the features over n nodes with z (i) ∈ R n being its i th row, i.e., the i th dimension of the features over all nodes. We define the normalized smoothness of z (i) as follows:s(z (i) ) := ∥z (i) M ∥/∥z (i) ∥,where we set s(z (i) ) = 1 when z (i) = 0.Remark 4.2. Notice that the normalized smoothness s(z(i) ) = ∥z (i) M ∥/∥z (i)∥ is closely related to the ratio between the smooth and non-smooth components of node features ∥z(i) M ∥/∥z (i) M ⊥ ∥.The graph is connected implies that z(i) M = ⟨z (i), e 1 ⟩e 1 and ∥z(i) M ∥ = |⟨z (i) , e 1 ⟩|.Without ambiguity, we write z for z (i) and e for e 1 -the eigenvector of G associated with the eigenvalue 1. Moreover, we haveEQUATIONIt is evident that the larger s(z) is, the smoother the node feature z is1 . In fact, we haves(z) 2 + ∥z∥ M ⊥ ∥z∥ 2 = ∥zM∥ 2 ∥z∥ 2 + ∥z M ⊥ ∥ 2 ∥z∥ 2 = 1,where ∥z∥ M ⊥ /∥z∥ decreases as s(z) increases. To discuss how the smoothness s(h) = s(σ(z)) or s(σ a (z)) can be adjusted by changing z M , we consider the functionz(α) = z -αe.It is clear thatz(α) M ⊥ = z M ⊥ and z(α)M = zM -αe,where we see that α only alters z M while pre-serves z M ⊥ . Moreover, it is evident that s(z(α)) = 1 - ∥z(α) M ⊥ ∥ 2 ∥z(α)∥ 2 = 1 - ∥z M ⊥ ∥ 2 ∥z(α)∥ 2 .It follows that s(z(α)) = 1 if and only if z M ⊥ = 0 (include the case z = 0), showing that when z M ⊥ = 0, the vector z is the smoothest one. 
the disparate effects of α on ∥ • ∥ m ⊥ and s(•): empirical results: 
 Let us empirically study possible values that the unnormalized smoothness ∥σ(z(α))∥ M ⊥ , ∥σ a (z(α))∥ M ⊥ and the normalized smoothness s(σ(z(α))), s(σ a (z(α))) can take when α varies.We denote z α := z(α) = z -αe. We consider a connected synthetic graph with 100 nodes, and each node is assigned a random degree between 2 to 10. Then we assign an initial node feature z ∈ R 100 , sampled uniformly on the interval [-1.5, 1.5], to the graph with each node feature being a scalar.Also, we compute e by the formula e = D 12 u/∥ D 1 2 u∥ from Proposition 2.1, where u ∈ R 100 is the vector whose entries are all ones and D is the (augmented) degree matrix. We examine two different smoothness notions for the input z and the output σ(z α ) and σ a (z α ), where the smoothness is measured for various values of the smoothness control parameter α ∈ [-1.5, 1.5]. In Fig. 1a ), we study the unnormalized smoothness measured by ∥•∥ M ⊥ ; we see that ∥σ(z α )∥ M ⊥ and ∥σ a (z α )∥ M ⊥ are always no greater than ∥z∥ M ⊥ . This coincides with the discussion in Section 3.3; adjusting the projection of z onto the eigenspace M can not change the fact that ∥σ(zα )∥ M ⊥ ≤ ∥z∥ M ⊥ and ∥σ a (z α )∥ M ⊥ ≤ ∥z∥ M ⊥ .Nevertheless, an interesting result is that altering the eigenspace projection can adjust the unnormalized smoothness of the output: notice that altering the eigenspace projection does not change its distance to M, i.e., the smoothness of the input is unchanged, but the smoothness of the output after activation function can be changed.In contrast, when studying the normalized smoothness s(•) in Fig. 1b ), we find that s(σ(z(α)))and s(σ a (z(α))) can be adjusted by α to values smaller than s(z). More precisely, we see that by adjusting α, s(σ(z(α))) and s(σ a (z(α))) can achieve most of the values in [0, 1]. In other words, both smoother and less smooth features can be obtained by adjusting α.In this subsection, we build theoretical understandings of the above empirical findings on the achievable smoothness shown in Fig. 1 . Notice that if z M ⊥ = 0, the inequalities presented in Propositions 3.2 and 3.3 indicate that ∥σ(z(α))∥ M ⊥ and ∥σ a (z(α))∥ M ⊥ vanish. So we have s(σ(z(α))) = 1 for any α when z M ⊥ = 0. Then we may assume z M ⊥ ̸ = 0 for the following study.Proposition 4.3 (ReLU). Suppose z M ⊥ ̸ = 0. Let h(α) = σ(z(α)) with σ being ReLU, then min α s(h(α)) = x i =max x di n j=1 dj and max α s(h(α)) = 1,where x := D-1 2 z, max x = max 1≤i≤n x i , and D is the augmented degree matrix with diagonals improve the performance of GCN and we will discuss this in the next section. 
controlling smoothness of node features: 
 We do not know how smooth features are ideal for a given node classification task. Nevertheless, our theory indicates that both normalized and unnormalized smoothness of the output of each GCL can be adjusted by altering the input's projection onto M. As such, we propose the following learnable smoothness control term to modulate the smoothness of each dimension of the learned node featuresEQUATIONwhere l is the layer index, {e i } m i=1 is the orthonormal basis of the eigenspace M, andα l := {α l i } m i=1is a collection of learnable vectors with α l i ∈ R d being approximated by a multi-layer perceptron (MLP). The detailed configuration of α l i will be specified in each experiment later. One can see thatB l α always lies in R d ⊗ M.We integrate SCT into GCL, resulting inEQUATIONWe call the corresponding model GCN-SCT. Again, the idea is that we alter the component in eigenspace to control the smoothness of features. Each dimension of H l can be smoother, less smooth, or the same as H l-1 in normalized smoothness, though H l gets closer to M than H l-1 .To design SCT, we introduce a learnable matrix A l ∈ R d×m for layer l, whose columns are α l i , where m is the dimension of the eigenspace M and d is the dimension of the features. We observe in our experiments that the SCT performs best when informed by degree pooling over the subcomponents of the graph. The matrix of the orthogonal basis vectors, denoted byQ := [e 1 , . . . , e m ] ∈ R n×m , is used to perform pooling H l Q for input H l .In particular, we letA l = W ⊙ (H l Q), where W ∈ R d×mis learnable and performs pooling over H l using the eigenvectors Q. The second architecture uses a residual connection with hyperparameter β l = log(θ/l + 1) and learnable matrices W 0 , W 1 ∈ R d×d and the softmax function ϕ. Resulting inA l = ϕ(H l Q) ⊙ (β l W0H 0 Q + (1 -β l )W1H l Q).In Section 6, we use the first architecture for GCN-SCT as GCN uses only H l information at each layer. We use the second architecture for GCNII-SCT and EGNN-SCT which use both H 0 and H l information at each layer. There are two particular advantages of the above design of SCT: (1) it can effectively change the normalized smoothness of the learned features, and (2) it is computationally efficient since we only use the eigenvectors corresponding to the eigenvalue 1 of matrix G, which is determined based on the connectivity of the graph.In this subsection, we present other usages of the proposed SCT. Due to the page limit, we carefully select two other most representative models. The first example is GCNII [6] , GCNII extends GCN to express an arbitrary polynomial filter rather than the Laplacian polynomial filter and achieves state-of-the-art (SOTA) performance among GCN-style models on various tasks [6, 23] , and we aim to show that SCT can even improve the accuracy of the GCN-style model that achieves SOTA performance on many node classification tasks. The second example is energetic GNN (EGNN) [40] ,which controls the smoothness of node features by constraining the lower and upper bounds of the Dirichlet energy of features and assuming the activation function is linear. In this case, we aim to show that our new theoretical understanding of the role of activation functions and the proposed SCT can boost the performance of EGNN with considering nonlinear activation functions.Each GCNII layer uses a skip connection to the initial layer H 0 and given as follows:H l = σ ((1 -α l )H l-1 G + α l H 0 )((1 -β l )I + β l W l ) ,where α l , β l ∈ (0, 1) are learnable scalars. We integrate SCT B l α into GCNII, resulting in the following GCNII-SCT layersH l = σ ((1 -α l )H l-1 G + α l H 0 )((1 -β l )I + β l W l ) + B l α ,where the residual connection and identity mapping are consistent with GCNII.EGNN. Each EGNN layer can be written as follows:EQUATIONwhere c 1 , c 2 are learnable weights that satisfy c 1 + c 2 = c min with c min being a hyperparameter. To constrain Dirichlet energy, EGNN initializes trainable weights W l as a diagonal matrix with explicit singular values and regularizes them to keep the orthogonality during the model training. Ignoring the activation function σ, H l -node features at layer l of EGNN satisfiescmin∥H 0 ∥E ≤ ∥H l ∥E ≤ cmax∥H 0 ∥E,where c max is the square of the maximal singular value of the initialization of W 1 . Similarly, we modify EGNN to result in the following EGNN-SCT layerH l = σ W l ((1 -cmin)H l-1 G + c1H 0 + c2H l-1 ) + B l α ,where everything remains the same as the EGNN layer except that we add our proposed SCT B l α . 
experiments: 
 In this section, we comprehensively demonstrate the effects of SCT -in the three most representative GCN-style models discussed in Section 5 -using various node classification benchmarks. The purpose of all experiments in this section is to verify the efficacy of the proposed SCT -motivated by our theoretical results -for GCN-style models. We consider the citation datasets (Cora, Citeseer, PubMed, Coauthor-Physics, Ogbn-arxiv), web knowledge-base datasets (Cornell, Texas, Wisconsin), and Wikipedia network datasets (Chameleon, Squirrel). We provide additional dataset details in Appendix D.1. We implement baseline GCN [20] and GCNII [6] (without weight sharing) using PyG (Pytorch Geometric) [10] . Baseline EGNN [40] is implemented using the public code4 . We visualize the trajectory of the node features, following [27] , for a graph with two nodes connected by an edge and 1D node feature. In this case, (6) becomesh 1 = σ(wh 0 G + b α ), where w = 1.2 in our experiment, h 0 , h 1 , b α ∈ R 2 , and G ∈ R 2×2 .We use a matrix G = [0.592, 0.194; 0.194, 0.908] whose largest eigenvalue is 1. Twenty initial node feature vectors h 0 are sampled evenly in the domain[-1, 1] × [-1, 1]. Fig. 2 shows the trajectories in relation to the eigenspace M (red dashed line). In Fig 2a ), one can see that some trajectories do not directly converge to M. In Fig. 2b ) when α = 0.0, GCL is recovered and all trajectories converge to M. In Fig. 2c ), large values of α enable the features to significantly deviate from M initially. We observe that the parameter α can effectively change the trajectory of features. 1 : Accuracy for models of varying depth. We note vanishing gradients occur but not over-smoothing for the accuracy drop using GCN-SCT with 16 or 32 layers. For Cora, Citeseer, and PubMed, we use a fixed split with a single forward pass following [6] ; only test accuracy is available in these experiments. For Coauthor-Physics and Ogbn-arxiv, we use the splits from [40] ; both test accuracy and standard deviation are reported. The baseline results are copied from [6, 40] where the standard deviation was not reported. (Unit:%)Citation networks. We compare the three representative models discussed in Section 5, of different depths, with and without SCT in Table 1 . This task uses the citation datasets with fixed splits from [35] for Cora, Citeseer, and Pubmed and splits from [40] for Coauthor-Physics and Ogbn-arxiv; a detailed description of these datasets and splits are provided in Appendix D. Following [6] , we use a single training pass to minimize the negative log-likelihood loss using the Adam optimizer [19] , with 1500 maximum epochs, and 100 epochs of patience. A grid search for possible hyperparameters is listed in Table 5 in Appendix D. We accelerate the hyperparameter search by applying a Bayesian meta-learning algorithm [2] which minimizes the validation loss, and we run the search for 200 iterations per model. In particular, Table 1 presents the best test accuracy between ReLU and leaky ReLU for GCN, GCNII, and all three models with SCT 5 . For the baseline EGNN, we follow [40] using SReLU, a particular activation used for EGNN in [40] . These results show that SCT can boost the classification accuracy of baseline models; in particular, the improvement can be remarkable for GCN and GCNII. However, EGNN-SCT (using ReLU or leaky ReLU) performs occasionally worse than EGNN (using SReLU), and this is because of the choice of activation functions. In Appendix D.3, we report the results of EGNN-SCT using SReLU, showing that EGNN-SCT outperforms EGNN in all tasks. In fact, SReLU is a shifted version of ReLU, and our theory for ReLU applies to SReLU as well. The model size and computational time are reported in Table 4 in the appendix.Table 1 also shows that even with SCT, the accuracy of GCN drops when the depth is 16 or 32. This motivates us to investigate the smoothness of the node features learned by GCN and GCN-SCT. Fig. 3 plots the heatmap of the normalized smoothness of each dimension of the learned node features learned by GCN and GCN-SCT with 32 layers for Citeseer node classification. In these plots, the horizontal and vertical dimensions denote the feature dimension and the layer of the model, resp.We notice that the normalized smoothness of each dimension of the features -from layers 14 to 32 learned by GCN -closes to 1, confirming that deep GCN learns homogeneous features. In contrast, the features learned by GCN-SCT are inhomogeneous, as shown in Fig. 3b ). Therefore, we believe the Other datasets. We further compare different models trained on different datasets using 10-fold cross-validation and fixed 48/32/20% splits following [28] . Table 2 compares GCN and GCNII with and without SCT, using leaky ReLU, for classifying five heterophilic node classification datasets. We exclude EGNN as these heterophilic datasets are not considered in [40] . We report the average accuracy of GCN and GCNII from [6] . We tune all other models using a Bayesian meta-learning algorithm to max- Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: See Section 6 for details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
concluding remarks: 
 In this paper, we establish a geometric characterization of how ReLU and leaky ReLU affect the smoothness of the GCN features. We further study the dimension-wise normalized smoothness of the learned node features, showing that activation functions not only smooth node features but also can reduce or preserve the normalized smoothness of the features. Our theoretical findings inform the design of a simple yet effective SCT for GCN. The proposed SCT can change the smoothness, in terms of both normalized and unnormalized smoothness, of the learned node features by GCN.Limitations: Our proposed SCT provides provable guarantees for controlling the smoothness of features learned by GCN and related models. A key aspect to establish our theoretical results is demonstrating that, without SCT, the features of the vanilla model tend to be overly smooth; without this condition, SCT cannot ensure performance guarantees.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: See Section 6 for details.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
broader impacts: 
 Our paper focuses on developing new theoretical understandings of the smoothness of node features learned by graph convolutional networks. The paper is mainly theoretical. We do not see any potential ethical issues in our research; all experiments are carried out using existing benchmark settings and datasets.Our paper brings new insights into building new graph neural networks with improved performance over existing models, which is crucial for many applications. In particular, for applications where graph neural network is the method of choice. We expect our approach to play a role in material science and biophysics applications.For two vectors u = (u 1 , u 2 , . . . , u d ) and v = (v 1 , v 2 , . . . , v d ), their inner product is defined as⟨u, v⟩ = d i=1 u i v i ,their Hadamard product is defined asu ⊙ v = (u 1 v 1 , u 2 v 2 , . . . , u d v d ),and their Kronecker product is defined asu ⊗ v = uv ⊤ =     u 1 v 1 u 1 v 2 . . . u 1 v d u 2 v 1 u 2 v 2 . . . u 2 v d . . . . . . . . . . . . u d v 1 u d v 2 . . . u d v d     .The Kronecker product can be defined for two vectors of different lengths in a similar manner as above.First, we prove that the two smoothness notions used in [27, 4] are two equivalent seminorms, i.e., we prove Proposition 3.1 below.Proof of Proposition 3.1. The matrix H can be decomposed as H = n i=1 He i e ⊤ i , where each e i is the eigenvector of G associated with eigenvalue λ i . This indicates thatH ∆ = H(I -G) = n i=1 He i e ⊤ i (I -G) = n i=1 (He i e ⊤ i -He i e ⊤ i G) = n i=1 (He i e ⊤ i -He i (λ i e i ) ⊤ ) = n i=1 (1 -λ i )He i e ⊤ i = n i=m+1 (1 -λ i )He i e ⊤ i .Then using the fact that 1 -λ i ≥ 0 for each i, we obtain∥H∥ 2 E = Trace(H ∆H ⊤ ) = Trace n i=m+1 (1 -λ i )He i e ⊤ i ( n j=1He j e ⊤ j ) ⊤ = TraceEQUATIONF.That is,∥H∥ E = n i=m+1 1 -λ i He i e ⊤ i F.On the other hand, (3) implies∥H∥ M ⊥ = ∥H M ⊥ ∥ F = n i=m+1 He i e ⊤ i F.We He i e ⊤ i ,n i=m+1 1 -λ i H 1 e i e ⊤ i + n i=m+1 1 -λ i H 2 e i e ⊤ i = n i=m+1 1 -λ i He i e ⊤ i .Then the triangle inequality of ∥ • ∥ F implies that of ∥H∥ M ⊥ and ∥H∥ E , respectively.Now since 0 < 1 -λ m+1 ≤ 1 -λ i ≤ 2 for any i = m + 1, . . . , n, we may take α = 1 -λ m+1 and β = √ 2. Then α∥H∥ M ⊥ = α n i=m+1 He i e ⊤ i F ≤ n i=m+1 1 -λ i He i e ⊤ i F ≤ β n i=m+1 He i e ⊤ i F = β∥H∥ M ⊥ .The result thus follows from∥H∥ E = n i=m+1 √ 1 -λ i He i e ⊤ i F.We present a crucial tool to characterize how ReLU affects its input.Lemma B.1. Let Z ∈ R d×n , and let Z + = max(Z, 0) and Z -= max(-Z, 0) be the positive and negative parts of Z. Then (1) Z + , Z -are (component-wise) nonnegative and Z = Z + -Z -and(2) ⟨Z + , Z -⟩ F = 0.Proof of Lemma B.1. Notice that for any a ∈ R, we have max(a, 0) = a if a ≥ 0 0 otherwise and max(-a, 0) = 0 if a ≥ 0 -a otherwise .This implies that a = max(a, 0) -max(-a, 0) and max(a, 0) • max(-a, 0) = 0.Let Z ij be the (i, j) th entry of Z.Then Z = Z + -Z -follows from Z ij = max(Z ij , 0) - max(-Z ij , 0). Also, one can deduce that⟨Z + , Z -⟩ F = Trace((Z + ) ⊤ Z -) = d i=1 j j=1 max(Z ij , 0) max(-Z ij , 0) = 0.Before proving Proposition 3. EQUATIONProof of Lemma B.2. We observe thatH = σ(Z) = max(Z, 0) = Z + is the positive part of Z. Then ⟨H, Z⟩ F = ⟨H, Z + -Z -⟩ F = ⟨H, Z + ⟩ F -⟨H, Z -⟩ F = ⟨H, H⟩ F ,where we have used Z = Z + -Z -and ⟨H, Z -⟩ F = ⟨Z + , Z -⟩ F = 0 from Lemma B.1.Therefore, one can deduce the desired result as followsEQUATIONH - Z 2 2 F = H M ⊥ - Z M ⊥ 2 2 F + H M - Z M 2 2 F .Then (8) becomesEQUATION)By direct calculation, we haveEQUATION)Combining ( 9) and (10) , we obtain the following resultLemma B.3. For any Z = Z M + Z M ⊥ , let H = σ(Z) = H M + H M ⊥ , then Z M ⊥ 2 2 F -H M ⊥ - Z M ⊥ 2 2 F = ⟨Z + M , Z - M ⟩ F .whereZ + M = m i=1 Z + e i e ⊤ i , Z - M = m i=1 Z -e i e ⊤ i . Proof of Lemma B.3. Recall that H = σ(Z) = max(Z, 0) = Z + . Also, Z = Z + -Z -implies Z M = Z + M -Z - M = H + M -Z - M .Therefore, we see that⟨H M , H M -Z M ⟩ F = ⟨Z + M , Z - M ⟩ F .By using the fact that ⟨Z + M , Z - M ⟩ F ≥ 0 in Lemma B.+ M , Z - M ⟩ F = Trace Z + M (Z - M ) ⊤ ≥ 0.Then according to Lemma B.3, we obtainZ M ⊥ 2 2 F -H M ⊥ - Z M ⊥ 2 2 F = ⟨Z + M , Z - M ⟩ F ≥ 0.H M ⊥ - Z M ⊥ 2 F = Z M ⊥ 2 2 F -⟨Z + M , Z - M ⟩ F = Z M ⊥ 2 2 F -⟨H M , H M -Z M ⟩ F ,which shows that H M ⊥ lies on the high-dimensional sphere that we have claimed. Furthermore, we conclude thatEQUATIONThis demonstrates that H M ⊥ lies on the high-dimensional sphere we have stated.Since the sphere H M ⊥ -Z M ⊥ 2 2 F = Z M ⊥ 2 2 Fpasses through the origin, the distance of any H M ⊥ to the origin must be no greater than the diameter of this sphere, i.e., ∥H M ⊥ ∥ F ≤ ∥Z M ⊥ ∥ F . Also, this can be derived from∥H M ⊥ ∥ F - Z M ⊥ 2 F ≤ H M ⊥ - Z M ⊥ 2 F ≤ Z M ⊥ 2 F .One can see that the maximal smoothness∥H M ⊥ ∥ F = ∥Z M ⊥ ∥ F is attained when H M ⊥ = Z M ⊥ ,the intersection of the surface and the line passing through the center and the origin.After all, we complete the proof by using the fact that ∥Z M ⊥ ∥ F = ∥Z∥ M ⊥ for any matrix Z, whichimplies ∥H∥ M ⊥ = ∥H M ⊥ ∥ F ≤ ∥Z M ⊥ ∥ F = ∥Z∥ M ⊥ .For the leaky ReLU activation function, we have H = σ a (Z) = Z + -aZ -.Then H -Z = (1 -a)Z -and H -aZ = (1 -a)Z + . Using ⟨Z -, Z + ⟩ F = 0, we have⟨H -Z, H -aZ⟩ F = 0 ⇒∥H∥ 2 F -2 H, (1 + a)Z 2 F + a∥Z∥ 2 F = 0 ⇒∥H∥ 2 F -2 H, (1 + a)Z 2 F = -a∥Z∥ 2 F ⇒ H - (1 + a) 2 Z 2 F = (1 + a) 2 Z 2 F -a∥Z∥ 2 F = (1 -a) 2 Z 2 F .Moreover, we notice that Lemma B.5. For anyZ = Z M + Z M ⊥ , let H = σ a (Z) = H M + H M ⊥ , then (1 -a) 2 Z M ⊥ 2 F -H M ⊥ - (1 + a) 2 Z M ⊥ 2 F = (1 -a) 2 ⟨Z + M , Z - M ⟩ FProof of Lemma B.5. Similar to the proof of Lemma B.3, the orthogonal decomposition implies that(1 -a) 2 Z M ⊥ 2 F -H M ⊥ - (1 + a) 2 Z M ⊥ 2 F = H M - (1 + a) 2 Z M 2 F - (1 -a) 2 Z M 2 F =⟨H M -Z M , H M -aZ M ⟩ F =⟨(1 -a)Z - M , (1 -a)Z + M ⟩ F =(1 -a) 2 ⟨Z - M , Z + M ⟩ F .Proof of Proposition 3.3. Similar to the proof of Proposition 3.2, we apply ⟨Z - M , Z + M ⟩ F ≥ 0 to Lemma B.5 and hence obtain the geometric condition as followsH M ⊥ - (1 + a) 2 Z M ⊥ F = (1 -a) 2 Z M ⊥ 2 F -⟨H M -Z M , H M -aZ M ⟩ F .Then we have the following inequality0 ≤ H M ⊥ - (1 + a) 2 Z M ⊥ F ≤ (1 -a) 2 Z M ⊥ F .Moreover, we deduce that∥H M ⊥ ∥ F - (1 + a) 2 Z M ⊥ F ≤ H M ⊥ - (1 + a) 2 Z M ⊥ F ≤ (1 -a) 2 Z M ⊥ F .and hence- (1 -a) 2 Z M ⊥ F ≤ ∥H M ⊥ ∥ F - (1 + a) 2 Z M ⊥ F ≤ (1 -a) 2 Z M ⊥ F .Therefore, we obtain a∥ZM ⊥ ∥ F ≤ ∥H M ⊥ ∥ F ≤ ∥Z M ⊥ ∥ F . (Remark that H M ⊥ achieves itsmaximal norm when it is equal to Z M ⊥ , the intersection of the surface and the line passing through the center and the origin. )By using the fact that ∥Z M ⊥ ∥ F = ∥Z∥ M ⊥ for any matrix Z, we conclude that a∥Z∥M ⊥ ≤ ∥H∥ M ⊥ ≤ ∥Z∥ M ⊥ .Throughout this section, we assume that z M ⊥ ̸ = 0. Note that z(α) = z -αe = z -α c D 1 2 u n = D 1 2 ( D-1 2 z -α c u n ) = D 1 2 (x -α c u n ),where we assume x := D- 1 2 z. Then we observe that when σ is the ReLU activation function,h(α) = σ(z(α)) = σ D 1 2 (x - α c u n ) = D 1 2 σ x - α c u n ,and hence⟨h(α), e⟩ = D 1 2 σ x - α c u n , e = σ x - α c u n , D 1 2 e = σ x - α c u n , Du n .We may now assume x = [x 1 , . . . , x n ] ⊤ is well-ordered s.t. x 1 ≥ x 2 ≥ . . . ≥ x n . Indeed, there is a collection of indices {k 1 , ..., k l } s.t.x 1 = . . . , x k1 and x k1 > x k1+1 , x kj-1+1 = . . . = x kj and x kj > x kj +1 for any j = 2, . . . , l -1,x k l-1 +1 = . . . = x k l and k l = n.That is,x 1 = x 2 = . . . = x k1 > x k1+1 = . . . = x k2 > x k2+1 = . . . = x k3 > x k3+1 . . .We first restrict the domain of α s.t. h(α) ̸ = 0. Note that we haveh(α) = 0 ⇔σ x - α c u n = 0 ⇔x i - α c ≤ 0 for i = 1, . . . , n ⇔x 1 - α c ≤ 0 ⇔α ≥ cx 1 .So we will study the smoothness s(h(α)) when α < cx 1 .Let ϵ > 0 and consider α = c(x 1 -ϵ). When ϵ ≤ x 1 -x k1+1 = x 1 -x k2 , we see thatx - α c u n = [ϵ, . . . , ϵ, ϵ -(x 1 -x k1+1 ), . . . , ϵ -(x 1 -x n )] ⊤ ,where only the first k 1 entries are positive since x 1 -x i ≥ ϵ for any i ≥ k 1 + 1. Therefore,h(α) = D 1 2 σ x - α c u n = D 1 2 [ϵ, . . . , ϵ, 0, . . . , 0] ⊤ = [ϵ d 1 , . . . , ϵ d k1 , 0, . . . , 0] ⊤ .and hence we can compute that ∥h(α)∥ = ϵ k1 i=1 d i . Also, we have∥h(α)∥ M = |⟨h(α), e⟩| = [ϵ d 1 , . . . , ϵ d k1 , 0, . . . , 0] ⊤ [ d 1 /c, d 2 /c, . . . , d n /c] = ϵ c k1 i=1 d i .Then we obtain the smoothness s(h(α)) as followss(h(α)) = ∥h(α)∥ M ∥h(α)∥ = ϵ c k1 i=1 d i ϵ k1 i=1 d i = k1 i=1 d i c = K 1 c < 1,where K 1 := k1 i=1 d i . Similarly, we may denote kj i=kj-1+1 d i by K j for j = 2, . . . , l.Now we are going to show that the smoothness s(h(α)) is increasing as α gets smaller whenever α < cx 1 , implying K1 c is the minimum of the smoothness s(h(α)). Remember that we are considering α = c(x 1 -ϵ) and we have studied the case when 0< ϵ ≤ x 1 -x k1+1 = x 1 -x k2 .Let δ j := x 1 -x kj for 1 ≤ j ≤ l. Clearly, we have δ 1 = 0 and δ j < δ j+1 for 1 ≤ j ≤ l -1. Fix a j ′ ∈ {2, . . . , l -1}, we see that whenδ j ′ < ϵ ≤ x 1 -x k j ′ +1 , x - α c u n = ϵ -δ 1 , . . . , ϵ -δ 1 , ϵ -δ 2 , . . . , ϵ -δ 2 , ϵ -δ 3 , . . . , ϵ -δ j ′ , ϵ -(x 1 -x k j ′ +1 ), . . . , ϵ -(x 1 -x n ) ⊤ ,where we have ϵ -δ j > 0 for 2 ≤ j ≤ j ′ and ϵ -(x 1 -x i ) ≤ 0 for any i ≥ k j ′ + 1. Consequently, h(α) = D 1 2 σ(x - α c u n ) = [(ϵ -δ 1 ) d 1 , . . . , (ϵ -δ 1 ) d k1 , (ϵ -δ 2 ) d k1+1 , . . . , (ϵ -δ 2 ) d k2 , (ϵ -δ 3 ) d k2+1 , . . . , (ϵ -δ j ′ ) d k j ′ , 0, . . . , 0] ⊤ .Then we can compute∥h(α)∥ = j ′ j=1 kj i=kj-1+1 d i (ϵ -δ j ) 2 = j ′ j=1 K 2 j (ϵ -δ j ) 2 ,where we set k 0 := 0 for simplicity and K j = kj i=kj-1+1 d i for j = 1, . . . , j ′ . Also, we have∥h(α)∥ M = |⟨h(α), e⟩| = j ′ j=1 kj i=kj-1+1 d i (ϵ -δ j ) c = 1 c j ′ j=1 K 2 j (ϵ -δ j ).A careful calculation shows that ∂ ∂ϵ s(h(α)) > 0 whenever δ j ′ < ϵ ≤ x 1 -x k j ′ +1 which implies that s(h(α)) is increasing as ϵ increases. Indeed, we have∂ ∂ϵ s(h(α)) = ∂ ∂ϵ j ′ j=1 K 2 j (ϵ -δ j ) c j ′ j=1 K 2 j (ϵ -δ j ) 2 = ∂ ∂ϵ j ′ j=1 K 2 j (ϵ -δ j ) j ′ j=1 K 2 j (ϵ -δ j ) 2 - j ′ j=1 K 2 j (ϵ -δ j ) ∂ ∂ϵ j ′ j=1 K 2 j (ϵ -δ j ) 2 c j ′ j=1 K 2 j (ϵ -δ j ) 2 = j ′ j=1 K 2 j j ′ j=1 K 2 j (ϵ -δ j ) 2 - j ′ j=1 K 2 j (ϵ -δ j ) ∂ ∂ϵ j ′ j=1 K 2 j (ϵ-δj ) 2 2 j ′ j=1 K 2 j (ϵ-δj ) 2 c j ′ j=1 K 2 j (ϵ -δ j ) 2 = j ′ j=1 K 2 j j ′ j=1 K 2 j (ϵ -δ j ) 2 - j ′ j=1 K 2 j (ϵ -δ j ) j ′ j=1 K 2 j (ϵ -δ j ) c j ′ j=1 K 2 j (ϵ -δ j ) 2 j ′ j=1 K 2 j (ϵ -δ j ) 2 .Then to show that ∂ ∂ϵ s(h(α)) > 0, it suffices to show that the numerator is positive, i.e.j ′ j=1 K 2 j j ′ j=1 K 2 j (ϵ -δ j ) 2 - j ′ j=1 K 2 j (ϵ -δ j ) 2 > 0, since the denominator c j ′ j=1 K 2 j (ϵ -δ j ) 2 j ′ j=1 K 2 j (ϵ -δ j ) 2> 0 is always positive. In fact, this follows from the Cauchy inequality ∥v∥∥u∥ ≥ ⟨v, u⟩, where we setv := [K 1 , K 2 , . . . , K J ′ ] ⊤ , u := [K 1 (ϵ -δ 1 ), K 2 (ϵ -δ 2 ), . . . , K j ′ (ϵ -δ j ′ )] ⊤ .Moreover, equality happens only when v is parallel to u. This is, however, impossible since ϵ -δ j > ϵ -δ j+1 for any j = 1, . . . , j ′ -1 and each K j is positive.So we see that s(h(α)) is increasing as ϵ increases whenever 0 < ϵ, and hence the smoothness s(h(α)) is increasing as α decreases whenever cx n ≤ α < cx 1 .For the case j ′ = l where δ l = x 1 -x n < ϵ, we havex n -α/c = x n -(x 1 -ϵ) = ϵ-(x 1 -x n ) > 0,implying α < cx n and h(α) = z(α). We have shown that the smoothness is increasing as α is going far from ⟨z, e⟩; in particular, when α < ⟨z, e⟩ and α is decreasing. One can check thatcx n = n i=1 d i x n c = x n u n , Du n c ≤ x, Du n c = D 1 2 x, D 1 2 u n c = ⟨z, e⟩,which means the smoothness is increasing as α decreases whenever α < cx n .We conclude that the smoothness increases as α decreases provided α < cx 1 . Also, we have sup α<cx1 s(h(α)) = 1 as the case in the proof of Proposition C.1. One can check that s(h(α)) is a continuous function for α < cx 1 and thus it has range [K 1 /c, 1) by the mean value theorem.Finally, we can establish the result:K 1 /c = x i =max x di n j=1 djis the minimum of s(h(α)) and 1 is themaximum of s(h(α)) occurring whenever α ≥ cx 1 = n j=1 d j max i x i . Moreover, s(h(α)) hasa monotone property when α < n j=1 d j max i x i and has rangeEQUATIONIt is clear that the assumption on the ordering of the entries of x will not affect this result.To prove Proposition 4.4, we first prove an analogous result for the identity function, that is, h = σ(z) = z.Proposition C.1. Suppose z M ⊥ ̸ = 0, then s(z(α)) achieves its minimum 0 if α = ⟨z, e⟩. Moreover, sup α s(z(α)) = 1 where s(z(α)) is close to 1 when α is far away from ⟨z, e⟩.Notice that Proposition C.1 does not consider the activation function.Proof of Proposition C.1. We know that 0 ≤ s(z(α)) ≤ 1 ands(z(α)) = 1 - ∥z M ⊥ ∥ 2 ∥z(α)∥ 2 = 1 - ∥z M ⊥ ∥ 2 ∥z M ⊥ ∥ 2 + ∥z(α) M ∥ 2 = 1 - ∥z M ⊥ ∥ 2 ∥z M ⊥ ∥ 2 + ∥z M -αe∥ 2 . Suppose s(z(α)) = 1. Then we have ∥z M ⊥ ∥ 2 ∥z M ⊥ ∥ 2 +∥z M -αe∥ 2 = 0 which forces ∥z M ⊥ ∥ = 0. However,this contradicts the hypothesis z M ⊥ ̸ = 0. So s(z(α)) cannot attain its maximum.But for any 0 ≤ t < 1, one can see that s(z(α)) = t if and only if1 - ∥z M ⊥ ∥ 2 ∥z M ⊥ ∥ 2 + ∥z M -αe∥ 2 = t ⇔ ∥z M ⊥ ∥ 2 ∥z M ⊥ ∥ 2 + ∥z M -αe∥ 2 = 1 -t 2 ⇔ ∥z M ⊥ ∥ 2 = (1 -t 2 ) ∥z M ⊥ ∥ 2 + ∥z M -αe∥ 2 ⇔ t 2 ∥z M ⊥ ∥ 2 = (1 -t 2 )∥z M -αe∥ 2 ⇔ ∥z M -αe∥ = t 2 1 -t 2 • ∥z M ⊥ ∥This implies that sup α s(z(α)) = 1 and s(z(α)) achieves its minimum 0 if and only if α = ⟨z, e⟩.It is clear that s(z(α)) get closer to 1 when α is going far away from ⟨z, e⟩. i.e., |α -⟨z, e⟩| = ∥z M -αe∥ is increasing.Proof of Proposition 4.4. First, we notice that leaky ReLU has the following two properties 1. σ a (x) > 0 for x ≫ 0 and σ a (x) < 0 for x ≪ 0.2. σ a is a non-trivial linear map for x ≫ 0.We will use Property 1 to show that min α s(h(α)) = 0 and Property 2 to show that sup α s(h(α)) = 1.Notice that σ a (x) < 0 for x ≪ 0 implies that there exists a sufficient small α 2 < 0 s.t. all of the entries of h(α 2 ) are negative and hence |⟨h(α 2 ), e⟩| < 0. Similarly, σ a (x) > 0 for x ≫ 0 implies that there exists a sufficient large α 1 > 0 s.t. all of the entries of h(α 1 ) are positive and hence |⟨h(α 1 ), e⟩| > 0. Since |⟨h(α), e⟩| is a continuous function of α on [α 1 , α 2 ], the Intermediate Value Theorem follows that there exists an α ∈ (α 1 , α 2 ) s.t. |⟨h(α), e⟩| = 0. Thus by definition s(h(α)) = |⟨h(α), e⟩|/∥h(α)∥, we see that min α s(h(α)) = 0.On the other hand, since σ a is a non-trivial linear map for x ≫ 0, we may assume σ a (x) = cx for x > x 0 where c ̸ = 0 is some non-zero constant and x 0 > 0 is some positive constant. Then we can choose an α 0 > ⟨z, e⟩ s.t. for any α ≥ α 0 , all of the entries of z(α) are greater than x 0 . Then whenever α ≥ α 0 , we have h(α) = σ a (z(α)) = cz(α). This impliess(h(α)) = |⟨h(α), e⟩| ∥h(α)∥ = |⟨cz(α), e⟩| ∥cz(α)∥ = |⟨z(α), e⟩| ∥z(α)∥ = s(z(α)).Thus sup α s(h(α)) = 1 follows from the Proof of Proposition C.1 where we see that sup α s(z(α)) = 1 since s(z(α)) gets closer to 1 as α increases.Remark C.2. Indeed, it holds for any continuous function f : R → R satisfying the following 1. f (x) > 0 for x ≫ 0, f (x) < 0 for x ≪ 0 or f (x) < 0 for x ≫ 0, f (x) > 0 for x ≪ 0, 2. f is a non-trivial linear map for x ≫ 0 or x ≪ 0.One can check the proof above only depends on these two properties. It is worth mentioning that most activation functions, e.g. leaky LU, SiLU, tanh, satisfy condition 1.Proof of Corollary 4.5. For any α, we notice that ∥z∥M ⊥ = ∥z M ⊥ ∥ F = ∥z(α)∥ M ⊥ since αonly changes the component of z in the eigenspace M. Also, Propositions 3.2 and 3.3 show that ∥z(α)∥ M ⊥ ≥ ∥h(α)∥ M ⊥ whenever h(α) = σ(z(α)) or σ a (z(α)). Therefore, we see that ∥z∥ M ⊥ ≥ ∥h(α)∥ M ⊥ holds for any α. Since z M ⊥ ̸ = 0, s(z) must lie in [0, 1).This part includes the missing details about experimental configurations and additional experimental results for Section 6. All tasks we run using Nvidia RTX 3090, GV100, and Tesla T4 GPUs. All computational performance metrics, including timing procedures, are run using Tesla T4 GPUs from Google Colab.In this section, we briefly describe the benchmark datasets used. The SCT is added at each layer and the size of the additional parameters scales with the number of eigenvectors with an eigenvalue of one for matrix G in (2). • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: See supplementary materials for details.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Section 6 for details.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.Here, z ∈ R n is a vector whose i th entry is the 1D feature associated with node i.The ELU function is defined by f (x) = max(x, 0) + min(0, a • (e x -1)) where a > 0.The SELU function is defined by f (x) = c(max(x, 0) + min(0, a • (e x -1))) where a, c > 0.https://github.com/Kaixiong-Zhou/EGNNA comparison of the results using ReLU and leaky ReLU is presented in Appendix D. 
 
 Appendix:  


Decision: The paper investigates how Graph Convolutional Networks smooth node features. The authors show that adjusting the projection can modify the normalized smoothness to any desired level. The paper further introduces a new method, SCT, which allows GCNs to learn node features with a targeted smoothness level, thereby improving node classification.

Reviewers generally agree that this is a borderline paper, with both reasons to accept and reject it. They appreciated the new insights into node smoothness in GCNs and noted that SCT enhanced accuracy across various architectures and datasets, particularly highlighting the significant performance improvement when applying SCT to heterophilic datasets.

However, reviewers also expressed serious concerns about the lack of solid experimental evidence, pointing out that important baselines appear to be missing. They suggested that the paper's concepts should be more thoroughly connected to existing literature. Additionally, they noted that the analysis is relatively narrow, focusing mainly on GCN and ReLU/Leaky ReLU, and that the results are not surprising given the context provided by [4]. They also pointed out that the discussion of homophilic and heterophilic settings is missing. Specifically, since there is a gap between the theory presented (which currently analyzes generic graphs without distinguishing between homophilic and heterophilic settings) and its application, the reviewers recommend including both an ablation study and a discussion addressing this distinction.

For these reasons, I recommend rejecting the paper.

Reviews: review: paper_summary: paper_summary: This paper studies how ReLU and Leaky ReLU affect the smoothness of node features in graph convolution layers. The authors demonstrate that adjusting the input projection onto eigenspace $\mathcal{M}$ of the node feature matrix can achieve any desired normalized smoothness. Additionally, they propose a Smoothness Control Term (SCT) to enhance node classification in Graph Convolutional Networks, validated on both homophilic and heterophilic graphs.
strength_weakness: strength_weakness: Equation 5 implies that using SCT requires performing eigendecomposition. This paper avoids the high time complexity associated with eigendecomposition, especially when the number of nodes in a graph is very large.
questions: questions: None
limitations: limitations: N/A
paper_summary: paper_summary: The paper deals with (Over-)smoothing is Graph Neural Networks. While it was previously known that GCN-type GNNs oversmooth, this paper reexamines the case for GCN-type architectures with Relu-type activation functions in terms of *normalized* smoothness. The authors show that the convergence behaviour of GCNs can be split into two parts, a "smooth" part and a "non-smooth" part and that by manipulating the smooth part, one can influence the normalized smoothness of the signal. From these theoretical insights, the authors propose a new system that uses a learnable parameter that modulates the smooth part, making the model able to learn the most beneficial normalized smoothness for the problem at hand.
strength_weakness: strength_weakness: - The experiments are unconvincing. 
    - There is a slight improvement to be found in models with SCT, but this is not too surprising, as these models also have more parameters, and mostly improvements are quite slim.
    - You choose GCN, GCNII and EGNN, two of which have built-in skip connections, that are known to help with oversmoothing. This also coincides with the models that cope quite well with a larger number of layers. The vanilla GCN takes a huge hit in all benchmarks apart from ogbn-arxiv. So learning the normalized smoothness of features does not actually seem to help in the case of GCN. 
    - The other two models don't suffer from oversmoothing to begin with
questions: questions: - The parameter $\alpha$ can be chosen such that the normalized smoothness does not diminish asymptotically. However, the unnormalized smoothness does; does this mean that the norm of the features $||z|| \rightarrow 0$ tends towards 0? Doesn't this mean that features are also unusable in deeper layers?

- Is this analysis extendable to other architectures? E.g. GAT and other attention-based methods are also known to oversmooth, can a similar trick be applied there?

- Table 5 details the hyperparameters that were tried for each model. These seem to be very many combinations. Is it correct that you tried approx. 1.2 Million hyperparameter combinations for EGNN?
limitations: limitations: The limitations are not well-discussed. 
The only limitation the authors claim for this work is, that: "without this condition [that oversmoothing happens], SCT cannot ensure performance guarantees." There are no performance guarantees given for SCT. This is the only limitation discussed.
paper_summary: paper_summary: This paper first shows that in GCN, the output of ReLU or LeakyReLU lies on a sphere whose input is characterized by components parallel and perpendicular to $\mathcal{M}$, the space spanned by eigenvectors for the maximum eigenvalue of a graph. As a corollary, this paper shows that these activation functions do not increase the component of the feature vector perpendicular to $\mathcal{M}$. Furthermore, this paper defines the normalized smoothness and evaluates how its range varies with the activation functions. Based on this discussion, this paper proposes an SCT that learns the parallel components of the feature vectors. The proposed method is applied to GCN, GCNII, and EGNN, and verifies its effectiveness by applying it to node prediction tasks with various heterophily.
strength_weakness: strength_weakness: - I need help understanding the explanation in Section 3.3. More specifically, it is difficult to understand that the *independence* of the inequality means that the upper bound of the inequality does not depend on the value of $\boldsymbol{Z}_{\mathcal{M}}$. I suggest writing it explicitly. 
- P7, L.265: It seems strange that although SCT changes its architecture depending on whether the underlying GNN is GCT or GCNII, it has the same name. I suggest naming SCT for GCT and SCT for GCNII differently.
questions: questions: P5, L.196: $\boldsymbol{e}$ -> $\boldsymbol{e}_{1}$
P6, L.209: What is the value of $a$ of $\sigma_a$?
limitations: limitations: The authors discuss in the conclusion section that the proposed method's limitation is that it assumes the model's oversmoothing. However, I do not think this is a limitation because if the model does not cause oversmoothing, there is no need to use the proposed method. Rather, I recommend evaluating whether SCT has bad effects when the model does not cause oversmoothing.
paper_summary: paper_summary: The paper studies how GCN smoothes node features in terms of unnormalized and normalized smoothness. The results show that adjusting projection can alter the normalized smoothness to any desired level. Based on this, the paper proposes a new method SCT to let GCN learn node features with a desired smoothness to enhance node classification and verifies it effectiveness in practice.
strength_weakness: strength_weakness: 1.	While I can see that normalized smoothness has its own merit, this notion could be better motivated and connected to the literature. 

   a.	Given analysis in [27, 4] is asymptotic, I would not say that “over-smoothing – characterized by the distance of features to eigenspace M or the Dirichlet energy – is a misnomer”, as it is too strong of a claim to make. Those results essentially say that “very deep GCNs are bad due to oversmoothing” which has their limitations but can be well justified by the distance of features to eigenspace M or the Dirichlet energy.

   b.	My understanding is that normalized smoothness could be more connected to the non-asymptotic notion of oversmoothing studied in [32], which is defined based on the Bayes error of classification (the distance to the decision boundary) and hence taken the magnitude of features into account.

   c.	Based on the above, the argument presented in the paper can be strengthened in the following way: the motivation of normalized smoothness should be based on a discussion on how the magnitude (and hence normalized smoothness) is more related to a non-asymptotic notion of oversmoothing, which is directly related to the classification performance of finite-depth/shallow GNNs. Based on this, the results present in this paper is more practically relevant than the previous asymptotic result.

I would suggest the authors modify the relevant text in the introduction and analysis accordingly.


2.	The analysis only applies for GCNs, while whether the analysis or the proposed method can be extended to more complexed GNNs such as GATs or graph transformers is unclear. 

3.	The analysis only applies for ReLU and LeakyReLU, which reads a bit specific. I wonder if the results can be generalized to a general family of nonlinearities.

4.  For the experiments, there is a lack of baseline comparisons except the basic backbone architecture. For example, I wonder how it would compare to APPNP, which is proposed to balance the need to explore larger neighborhood and locality and the implicit goal is also to produce node features with the "right" amount of smoothness.

4.	Another presentation suggestion I have for the authors is that one should minimize the use of in-text math and bold or italic fonts for highlighting (such as line 167-173).  Math is hard to read in-text and when too many texts are highlighted, the paper becomes ever harder to read because everything seems to be emphasized and it kind of messes up with its original purpose.
questions: questions: See weaknesses.
limitations: limitations: n/a
paper_summary: paper_summary: The paper addresses the challenge of balancing smooth and non-smooth features in graph convolutional networks (GCNs) for node classification. Building on previous work that highlighted the correlation between feature smoothness and classification accuracy, the authors propose a novel method to control the smoothness of node features through a geometric approach and an augmented message-passing process. Their strategy involves establishing a geometric relationship between input and output vectors of activation functions like ReLU and Leaky-ReLU, and integrating a learnable term in the graph convolutional layers to modulate feature smoothness. The paper provides an empirical study to showcase the effectiveness of the proposed method.
strength_weakness: strength_weakness: * The empirical results section is very weak and a major revision is needed in order to be able to judge the merit of the proposed method. The most significant weaknesses are:
    * The paper does not compare to established baselines from other competing methods. For instance, one could take a look at the tables in https://arxiv.org/abs/2202.04579 or https://arxiv.org/abs/2210.00513 and add the results to the comparison in this paper (i.e., for cornell, texas, wisconsin, squirrel, and so on). 
    * The paper reports different (mostly weaker) performance for the baseline models such as GCN and GCNII. Again, this should be fixed and established results can be taken from the tables in the two papers mentioned above.
    * The improvement using the proposed SCT is very marginal in many experiments, and in particular the performance still drops (or improves very little) for higher number of layers. It is somewhat expected that increasing the number of layers and solving oversmoothing does not help much in homophilic tasks, i.e., the tasks considered in Table 1. However, if the method works it should lead to significant improvements on heterophilic tasks, even for increasing number of layers. Therefore, please show the results for the datasets in Table 2 in the same way as you show them in Table 1, i.e., for increasing number of layers.
    * It is claimed in Table 1 that the drop in performance of GCN for increasing number of layers is due to vanishing gradients and not due to oversmoothing. This claim has to be justified empirically. 
    * Figure 4 in the appendix showing gradient norms is not meaningful. The y-axis should be displayed on a logarithmic scale. Vanishing gradients occur for gradients approaching zero (exponentially fast). It is not clear if this is the case here, since it is plotted in linear scale.
    
* The structure and readability of this paper should be improved. For instance, it would be helpful for readers who are not very familiar with the graph-learning field, to introduce the concept of GNNs and GCNs in the introduction. It would be advisable to move the technical aspects from section 1 to section 2. In fact, none of the definitions presented at the beginning of the introduction are needed anywhere else in the remainder of the introduction section. Thus, this could be moved to section 2.
questions: questions: how expensive is it to compute the eigenbasis for the smoothness modulation term (5)? How does it scale with respect to the number of nodes and edges? How long does it take to compute for the biggest graph considered here, i.e., arxiv graph? Is that a limitation for large-scale graph learning?
limitations: limitations: * The method necessitates a pre-processing step to compute the eigenbasis in equation (5). This may not scale efficiently for very large graphs.
* The method does not show significant improvements. Notably, there are more effective models and methods available for mitigating oversmoothing, which have not been cited or empirically compared. Examples include https://arxiv.org/abs/2202.04579, https://arxiv.org/abs/2210.00513, https://arxiv.org/abs/2206.05437, https://arxiv.org/abs/2110.14446, https://arxiv.org/abs/2206.10991,
https://arxiv.org/abs/2006.11468.
* While the theoretical justification of the approach is intriguing, the insights are somewhat limited. For example, there are no provided estimates for choosing the parameter alpha, which instead has to be learned through gradient descent.
* The empirical results are not convincing.
paper_summary: paper_summary: The paper "Learning to Control the Smoothness of GCN Features" investigates the impact of activation functions, specifically ReLU and leaky ReLU, on the smoothness of node features in Graph Convolutional Networks (GCNs). It provides a geometric characterization of these effects, showing how altering the input's projection onto eigenspace M can control the smoothness of output features. The study introduces a Smoothness Control Term (SCT) to modulate the smoothness of node features, aiming to improve node classification tasks in both homophilic and heterophilic graphs. Experimental results validate the efficacy of SCT, demonstrating significant improvements in node classification accuracy for several GCN-style models .
strength_weakness: strength_weakness: ## Weaknesses
- The removal of white space in the paper makes it hard to read. The authors should really try and make the paper easier to read visually by not condensing as much math in the main text as possible. Not only is this arguably in violation of the guidelines, but it also illustrates that the authors need to distinguish clearer what the main contributions are and which parts in the main text can go to an appendix.
- While the geometric insights provided are valuable, the complexity of the mathematical formulations is challenging for readers not well-versed in advanced geometry and spectral graph theory. Simplifying explanations or providing more intuitive examples could enhance accessibility. Moreover, I feel like the math could be made more intuitive by giving verbal explanations before the theorems. The cramming of the paper and not highlighting enough what the main contributions should be improved.
- Although the paper compares SCT with a few baseline models, it would benefit from a broader comparison with additional state-of-the-art methods in GCNs and GNNs to provide a more comprehensive evaluation of its effectiveness.
- The experiments are primarily conducted on benchmark datasets. Incorporating more real-world applications and diverse datasets would demonstrate the practical relevance and versatility of the proposed method.
- The drop in accuracy for deeper models (16 or 32 layers) is noted but not deeply analyzed. A more thorough investigation into the causes of this performance degradation, beyond mentioning vanishing gradients, could offer insights into potential improvements. What happens if you use techniques that combat eg oversmoothing?
- While the paper mentions computational efficiency, a more detailed discussion on the computational overhead introduced by SCT, including potential trade-offs between accuracy and efficiency, would be beneficial.
questions: questions: ## Questions

-  Could you provide more intuitive examples or visual aids to help readers better understand the geometric characterization of smoothness in GCN features?
- Why did you choose the specific baselines for comparison? How would SCT perform against other state-of-the-art GCN and GNN models not included in your study?
- Can you elaborate on the causes of performance degradation in deeper models (16 or 32 layers)? Have you considered any specific techniques to mitigate this issue?
limitations: limitations: ## Limitations

The authors have acknowledged several limitations of their work, including the over-smoothing issue in deep GCNs and the dependence on specific activation functions, but they could improve by providing more detailed discussions on computational efficiency and potential negative societal impacts.

Comments: title: Thank you
Further clarification -- cont'd
Further clarification
Thank you
Thank you for considering our rebuttal
Thank you
Thank you
comment: Thank you for your further feedback and support of our paper. Your thoughtful review and valuable comments have significantly improved our paper.
Thank you for pointing this out. Indeed, I was only looking at the revised paper, not the rebuttal pdf. 
I appreciate the depth and detail of the author's response, which has clarified my concerns. I will increase my score accordingly. That being said, it is very important that the authors include the computational time comparison on the arxiv dataset in the paper as well as a discussion on it, as the computational burden of computing the eigenbasis was pointed out by almost all reviewers.
Dear Reviewer 5MTH, 

We first appreciate your further feedback. We are not sure if we misunderstood your comment that ``unfortunately, I cannot see any of the changes the authors claim to have made.’’ We are happy to address your further comments before the end of the discussion period. 

We believe we have provided detailed responses with additional experimental results in the rebuttal file - see the general response. We have also restructured the paper following the reviewers' feedback and added the additional results together with the references pointed out by the reviewer to the revised paper. Again, the openreview does not allow us to update the paper. Moreover, the rebuttal instruction says we can only post a single-page PDF file for additional figures and tables.



Regards,

Authors
Dear Reviewer 5MTH,
  
   Thank you for your response. Additional experimental results are provided in the one-page pdf file and the other changes have been made in the revised paper, which is straightforward to do. We cannot update the paper on openreview per the rebuttal rule. The system does not allow us to upload the revised version.


Regards,

Authors
Unfortunately, I cannot see any of the changes the authors claim to have made.
Thank you for considering our rebuttal and support of our work. We will reformat the paper following your and other reviewers' suggestions.
Thank you for your further feedback and your support of our work. 

We would like to clarify that while graph attention networks (GATs) may employ asymmetric graph convolutions, our work relies solely on the characterization of oversmoothing. As demonstrated in [1], GATs with asymmetric attention matrices can still exhibit oversmoothing, and our oversmoothing measurement is equivalent to that used in [1]. Importantly, our proposed method, smoothness control term (SCT), analyzes the impact of activation functions on the smoothness of $\bf Z = \bf W\bf H\bf G$, by considering only the input $\bf Z$ and output $\sigma(\bf Z)$ of these functions. This abstraction allows us to isolate the effects of the weight matrices $\bf W$ and graph convolution $\bf G$, providing an understanding of how SCT controls the smoothness of $\sigma(\bf Z)$.



[1] Wu, Xinyi, et al. "Demystifying oversmoothing in attention-based graph neural networks." NeurIPS, 2023.


Thank you again for considering our rebuttal.
Thanks a lot for the rebuttal. I will raise my score to a 5, but really urge the authors to reformat the paper in a better readable way.
I thank the authors for responding to my review. One difficulty I can see regarding extending to GATs or GTs is that unlike graph convolutions considered in this paper, attention matrices are in general not symmetric and the current analysis techniques developed for symmetric matrices might not be trivially applied. Nonetheless, the rebuttal has addressed the rest of my concerns. I will keep my score for now and stay on the positive side.
Thank you for considering our rebuttal.
I thank the authors for responding to my review comments. Their responses answered my questions appropriately. So, I want to keep my scores.