Title: GRANOLA: Adaptive Normalization for Graph Neural Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Neural Networks

Normalization Layer in GNNs
Abstract: Despite the widespread adoption of Graph Neural Networks (GNNs), these models often incorporate off-the-shelf normalization layers like BatchNorm or InstanceNorm, which were not originally designed for GNNs. Consequently, these normalization layers may not effectively capture the unique characteristics of graph-structured data, potentially even weakening the expressive power of the overall architecture. 
While existing graph-specific normalization layers have been proposed, they often struggle to offer substantial and consistent benefits. In this paper, we propose GRANOLA, a novel graph-adaptive normalization layer. Unlike existing normalization layers, GRANOLA normalizes node features by adapting to the specific characteristics of the graph, particularly by generating expressive representations of its nodes, obtained by leveraging the propagation of Random Node Features (RNF) in the graph. We provide theoretical results that support our design choices as well as an extensive empirical evaluation demonstrating the superior performance of GRANOLA over existing normalization techniques. Furthermore, GRANOLA emerges as the top-performing method among all baselines in the same time complexity class of Message Passing Neural Networks (MPNNs).

GRANOLA: Adaptive Normalization for Graph Neural Networks

Abstract

Despite the widespread adoption of Graph Neural Networks (GNNs), these models often incorporate off-the-shelf normalization layers like BatchNorm or InstanceNorm, which were not originally designed for GNNs. Consequently, these normalization layers may not effectively capture the unique characteristics of graph-structured data, potentially even weakening the expressive power of the overall architecture. While existing graph-specific normalization layers have been proposed, they often struggle to offer substantial and consistent benefits. In this paper, we propose GRANOLA, a novel graph-adaptive normalization layer. Unlike existing normalization layers, GRANOLA normalizes node features by adapting to the specific characteristics of the graph, particularly by generating expressive representations of its nodes, obtained by leveraging the propagation of Random Node Features (RNF) in the graph. We provide theoretical results that support our design choices as well as an extensive empirical evaluation demonstrating the superior performance of GRANOLA over existing normalization techniques. Furthermore, GRANOLA emerges as the top-performing method among all baselines in the same time complexity class of Message Passing Neural Networks (MPNNs).

1 Introduction

Graph Neural Networks (GNNs) have achieved remarkable success in several application domains [35, 61], showcasing their ability to leverage the rich structural information within graph data. Recently, a plethora of different layer designs has been proposed, each tailored to address specific challenges in the context of GNNs, such as limited expressive power [63, 41, 40] and oversmoothing [46]. Notably, analogously to architectures in other domains [30, 18], these GNN layers are often interleaved with normalization layers, as the integration of normalization methods has empirically proven beneficial in optimizing neural networks, facilitating convergence and enhancing generalization [34, 7, 55].

In practice, most existing GNN architectures employ standard normalization techniques, such as BatchNorm [34], LayerNorm [3], or InstanceNorm [58]. However, these widely adopted normalization techniques were not originally designed with graphs and GNNs in mind. Consequently, they may not effectively capture the unique characteristics of graph-structured data, and can also hinder the expressive power of the overall architecture [11]. These observations highlight the need for graph-specific normalization layers.

Recent works have taken initial steps in this direction, mainly targeting oversmoothing [72, 64, 74] or the expressive power of the overall architecture [11, 14]. Despite the promise shown by these methods, a consensus on a single normalization technique best suited for diverse tasks remains elusive, with no single normalization technique proving clearly superior across all benchmarks and scenarios.

Our approach. In this paper, we identify adaptivity to the input graph structure as a desirable property for an effective normalization layer in graph learning. Intuitively, this property ensures that the normalization is tailored to the specific input graph, capturing attributes such as the graph size, node degrees, and connectivity. Importantly, we claim and demonstrate that, given the limitations of practical GNNs, achieving full adaptivity requires expressive architectures that can detect and disambiguate graph substructures, thereby better adapting to input graphs.

Guided by this desirable property, which is absent in existing normalization methods, we introduce our proposed approach - GRANOLA (Graph Adaptive Normalization Layer). GRANOLA aims at dynamically adjusting node features at each layer by leveraging learnable characteristics of the node neighborhood structure derived through the utilization of Random Node Features (RNF) [44, 1, 51, 56, 17]. More precisely, GRANOLA samples RNF and uses them in an additional normalization GNN to obtain expressive intermediate node representations. The intermediate representations are then used to scale and shift the node representations obtained by the preceding GNN layer.

We present theoretical results that justify the primary design choices behind our method. Specifically, we demonstrate that GRANOLA is fully adaptive to the input graph, which, in other words, means that GRANOLA can predict different normalization values for non-isomorphic nodes. This property arises from the maximal expressive power of the normalization GNN we employ (MPNN augmented with RNF [1, 51]). In addition, we show that our method inherits this expressive power. Lastly, we show that using standard MPNN layers without RNF within GRANOLA cannot result in a fully adaptive method or in any additional expressive power.

Empirically, we show that GRANOLA significantly and consistently outperforms all existing standard as well as GNN-specific normalization schemes on a variety of different graph benchmarks and architectures. Furthermore, GRANOLA proves to be the best-performing method among all baselines that have the same time complexity as the most widely adopted GNNs, namely the family of MPNNs.

Contributions. Our contributions are as follows: (1) We provide an overview of different normalization schemes in graph learning, outlining adaptivity as a desirable property of normalization layers that existing methods are missing, (2) We introduce GRANOLA, a novel normalization technique adjusting node features based on learnable characteristics of their neighborhood structure, (3) We present an intuitive theoretical analysis of our method, giving insights into the design choices we have made, and (4) We conduct an extensive empirical study, providing a thorough benchmarking of existing normalization methods and showcasing the consistently superior performance of GRANOLA.

2 Normalization layers for GNNs

2.1 Basic setup and definitions

Let G = ( A , X ) denote graph with N ∈ N nodes, where A ∈ R N × N is the adjacency matrix and X ∈ R N × C is the node feature matrix, with C ∈ N the feature dimension. Consider a batch of B ∈ N graphs encoded by the adjacency matrices { A b } B -1 b =0 , and, for simplicity, assume that all graphs in the batch have the same number of nodes N . We consider a model composed of L GNN layers, with L ∈ N . Each GNN layer is followed by a normalization layer NORM and an activation function ϕ . At any layer ℓ ∈ [ L ] , the output of the GNN layer for a batch of graphs consists of (intermediate) node representations, which can be gathered in a matrix ˜ H ( ℓ ) ∈ R B × N × C (since all graphs have the same number of nodes per our assumption 2 ). These undergo a normalization and an activation layer, resulting in node representations denoted by H ( ℓ ) ∈ R B × N × C , which serve as input of the next GNN layer, with H (0) := X . 3 Throughout this paper, for any three-dimensional tensor, we use subscripts to denote access to a corresponding dimension. For example, we denote the intermediate node representations of graph b ∈ [ B ] by ˜ H ( ℓ ) b ∈ R N × C , and by ˜ h ( ℓ ) b,n,c the value of feature c ∈ [ C ] in node n ∈ [ N ] of graph b ∈ [ B ] .

BatchNorm

DiffGroupNorm

GraphSizeNorm

LayerNorm-node

NodeNorm

LayerNorm-graph

C

B

N

C

B

N

C

B

N

InstanceNorm

PairNorm

MeanSubtractionNorm

GraphNorm

C

B

N

Formally, the intermediate, pre-normalized node features for the b -th graph in the batch are defined as

Then, the overall update rule for the batch of B graphs can be written as

for ℓ ∈ [ L ] . Equation (2) serves as a general blueprint, and in what follows we will show different ways to customize it in order to implement different existing normalization techniques.

We consider normalization layers based on the standardization [38] of their inputs, as this represents the most common choice in widely used normalizations. Generally, a standardization-based normalization layer first shifts each element ˜ h ( ℓ ) b,n,c by some mean µ b,n,c , and then scales it by the corresponding standard deviation σ b,n,c , i.e.,

where γ ( ℓ ) c , β ( ℓ ) c ∈ R are learnable affine parameters, that do not depend on b nor n .

2.2 Current normalization layers for GNNs

The difference among different normalization schemes lies in the set of values used to compute the mean and standard deviation statistics for each element, or, more precisely, across which dimensions of ˜ H ( ℓ ) they are computed. We present them below, and visualize them in Figure 1.

BatchNorm. BatchNorm [34] computes the statistics across all nodes and all graphs in the batch, for each feature separately. Therefore, we have

which implies that µ b,n,c = µ b ′ ,n ′ ,c for any b ′ ∈ [ B ] and any n ′ ∈ [ N ] (and similarly for σ 2 b,n,c ). Despite the widespread use of BatchNorm in graph learning [63], it is important to recognize scenarios where alternative normalization schemes might be more suitable. A concrete example illustrating this need is presented in Figure 2, focusing on the task of predicting the degree of each node. 4 In this example, BatchNorm, by subtracting the mean computed across the batch, results in negative values for nodes with outputs below the mean. The subsequent ReLU application, as standard practice [63, 37, 41], zeros out these negative values, leading to predictions of 0 for such nodes, irrespective of their actual degree. Additional insights are further discussed in Example C.1 in Appendix C, where it is demonstrated that relying on the affine parameter β ( ℓ ) c to shift the negative output does not provide a definitive solution, since β ( ℓ ) c is the same for all graphs.

InstanceNorm. InstanceNorm [58] is similar to BatchNorm, but considers each graph separately, that is

which implies that µ b,n,c = µ b,n ′ ,c for any n ′ ∈ [ N ] (and similarly for σ 2 b,n,c ). Notably, the example in Figure 2 can be extended to InstanceNorm by considering all graphs in the batch as disconnected components in a single graph, as we show in Example C.4 in Appendix C.

LayerNorm. LayerNorm [3] can be defined in two ways in the context of graphs [24]. The first, which we call LayerNorm-node , behaves similarly to LayerNorm in Transformer architectures [59], and computes statistics across the features for each node separately, that is

4

5

5

6

5

3

4

6

5

5

6

6

6

6

1

2

3

4

5

6

(a) Node degrees, computed by one message-passing layer.

-1.1

-0.1

-0.1

0.9

-0.1

-2.1

-1.1

0.9

-0.1

-0.1

0.9

0.9

0.9

0.9

-6

-5

-4

-3

-2

-1

0

(b) Subtraction of the mean node degree. The features of nodes with degree less than average turn negative.

0

0

0

0.9

0

0

0

0.9

0

0

0.9

0.9

0.9

0.9

-3.0

-2.5

-2.0

-1.5

-1.0

-0.5

0.0

0.5

and therefore µ b,n,c = µ b,n,c ′ for any c ′ ∈ [ C ] (and similarly for σ 2 b,n,c ). The second variant, which we call LayerNorm-graph , is similar to LayerNorm in Computer Vision [62], and computes statistics across the features and across all the nodes in each graph, that is

and therefore µ b,n,c = µ b,n ′ ,c ′ for any n ′ ∈ [ N ] and any c ′ ∈ [ C ] (and similarly for σ 2 b,n,c ). Example C.5 in Appendix C presents a motivating example similar to Figure 2 for LayerNorm.

Graph-Specific normalizations. Several normalization methods tailored to graphs have been recently proposed. We categorize them based on which dimensions (the batch dimension B , the node dimension N within each graph, and the channel dimension C ) are used to compute the statistics employed within the normalization layer. We illustrate this categorization in Figure 1. DiffGroupNorm [74] and GraphSizeNorm [21] normalize features considering nodes across different graphs, akin to BatchNorm. Similarly to InstanceNorm, PairNorm [72] and MeanSubtractionNorm [64] shift the input by the mean computed per channel across all the nodes in the graph, with differences in the scaling strategies. GraphNorm [11] extends InstanceNorm by incorporating a multiplicative factor to the mean. NodeNorm [75] behaves similarly to LayerNorm-node but only scales the input, without shifting it. We provide additional details in Appendices A and B, and include methods that normalize the adjacency matrix before the GNN layers, which, however, fall outside the scope of this work.

3 Method

The following section describes the proposed GRANOLA framework. We start with identifying adaptivity as a desired property in a graph normalization layer.

𝐆𝐍𝐍

!"#$

(ℓ)

𝐆𝐍𝐍

()*+#

ℓ,𝟏

𝑨

.

𝐇

.

(ℓ,/)

𝑨

.

Node Features

𝐇

.

(ℓ,/)

Compute

GNN Features

𝐇 &

.

(ℓ)

Random

Node

Features

Normalized

Features

𝐇

.

(ℓ)

Predict

Normalization

Parameters

𝐑

.

(ℓ)

𝜸

.

(ℓ)

𝜷

.

(ℓ)

GRANOLA

Node Feature Extraction

NORM(               ,

)

𝐇 &

.

(ℓ)

𝜸

.

(ℓ)

𝜷

.

(ℓ)

Motivation. In the previous section, we observed that current normalization schemes used within GNNs are mostly borrowed or adapted from other domains and that they possess two main limitations: (i) Using BatchNorm and InstanceNorm (as well as many methods derived from them, including graph-specific methods) can limit the expressive power of GNNs that use them, preventing them from being able to represent even very simple functions such as computing node degrees; (ii) As shown in previous works, as well as in our experiments in Section 5, these methods do not provide a consistent benefit in downstream performance on different tasks and datasets.

One possible reason for these limitations is that the discussed methods use the same affine parameters in the normalization process, irrespective of the input graph. Crucially, unlike other more structured data types, such as images and time series, graphs do not differ merely in the values associated with each element or time step. Instead, graphs fundamentally vary in the actual connectivity between the nodes. Therefore, it might make sense to employ different (adaptive) normalization schemes based on the features and the connectivity of the input graph. Next, we explore this concept and show that carefully accounting for the graph structure in the normalization process may enhance the expressive power of the GNN, overcoming the previously mentioned failure cases, and providing consistent experimental behavior and overall improved performance.

3.1 Design considerations and overview

In an adaptive normalization, instead of using the same affine parameters γ ( ℓ ) c and β ( ℓ ) c (Equation (3)) for all the nodes in all the graphs, the normalization method utilizes specific parameters conditioned on the input graph. Importantly, in other domains, adaptivity in normalization techniques has proven to be a valuable property [19, 33, 48, 76, 47, 49]. In GRANOLA, we achieve this property by generating affine parameters using an additional normalization GNN that takes the graph as input, similarly to Hypernetworks [29] that generate weights for another network. Importantly, for full graph adaptivity, where the network can assign different normalization parameters for every pair of non-isomorphic nodes, the normalization GNN should have maximal expressive power. Due to the limited expressive power of MPNNs [41, 63], when designing GRANOLA, we advocate for using a normalization GNN with high expressiveness. Notably, most expressive GNNs come at the expense of significantly increased computational complexity [16, 71, 4, 69, 40, 43]. For this reason, we parameterize the normalization GNN using an MPNN augmented with Random Node Features (RNF), which provide strong function approximation guarantees [1, 51] while retaining the efficiency of standard MPNNs.

3.2 GRANOLA

We are now ready to describe our GRANOLA approach, which is based on the utilization of an additional GNN to compute the affine parameters and the integration of RNF [56]. These affine parameters are then used to normalize node representations obtained by the GNN layer preceding the normalization. An overview of GRANOLA is visualized in Figure 3 and described in Algorithm 1.

GRANOLA layer. At any given layer ℓ ∈ [ L ] , for each graph b ∈ [ B ] , we sample RNF R ( ℓ ) b ∈ R N × K , K ∈ N , from some joint probability distribution, e.g., Normal, R ( ℓ ) b ∼ D . Then, we

Algorithm 1 GRANOLA Layer

Input: Node features ˜ H ( ℓ ) b ∈ R N × C , obtained from GNN ( ℓ -1) LAYER . Output: Normalized features per node n ∈ [ N ] and channel c ∈ [ C ]

.

1: Sample random node features R ( ℓ ) b ∈ R N × K .

2: Compute Z ( ℓ ) b = GNN ( ℓ ) NORM ( A b , ˜ H ( ℓ ) b ⊕ R ( ℓ ) b ) .

3: Compute affine parameters: γ ( ℓ ) b,n = f ( ℓ ) 1 ( z ( ℓ ) b,n ) , β ( ℓ ) b,n = f ( ℓ ) 2 ( z ( ℓ ) b,n ) .

4: Compute mean and standard-deviation µ b,n,c and σ b,n,c of ˜ H ( ℓ ) b .

concatenate R ( ℓ ) b with the intermediate node features ˜ H ( ℓ ) b obtained from GNN ( ℓ -1) LAYER (Equation (1)), and pass the resulting feature matrix through an additional GNN, i.e.,

where ⊕ denotes feature-wise concatenation, and GNN ( ℓ ) NORM is a shallow GNN architecture, with L ( ℓ ) NORM layers. The affine parameters are then computed from Z ( ℓ ) b ∈ R N × C , that is

where f ( ℓ ) 1 , f ( ℓ ) 2 are learnable functions (e.g., MLPs), and γ ( ℓ ) b,n , β ( ℓ ) b,n ∈ R C . Then, a GRANOLA layer is defined as

where µ b,n,c and σ b,n,c are the mean and std of ˜ H ( ℓ ) , computed per node across the feature dimension, exactly as in LayerNorm-node (Equation (6)). We highlight here a noticeable difference compared to standard normalization formulations, as presented in Equation (3): in GRANOLA, γ ( ℓ ) b,n,c , β ( ℓ ) b,n,c not only depend on c , but they also have a dependency on b and n and indeed vary for different graphs and nodes. Notably, this is possible because our normalization is adaptive to the input graph. Methods that disregard this information are compelled to use the same learnable normalization values for all graphs, as they operate without knowledge of the specific input graph being considered.

GRANOLA-NO-RNF. We consider a variant of GRANOLA that does not sample RNF and instead uses only ˜ H ( ℓ ) b to obtain Z ( ℓ ) b in Equation (8), which therefore becomes

We refer to this variant as GRANOLA-NO-RNF, as it allows us to directly quantify the contribution of the expressiveness offered by augmenting it with RNF as in Equation (8).

Complexity. We conclude by remarking that both GRANOLA and its variant, GRANOLA-NO-RNF, do not impact the asymptotic time and space complexity of standard MPNNs, which remains linear in the number of nodes and edges, as we show in Appendix G.

4 Theoretical Analysis

In this section, we explain the main design choices taken in the development of GRANOLA. Specifically, we elaborate on the advantages obtained by utilizing RNF as part of the normalization scheme as opposed to relying solely on the node features. We assume all GNN layers, including those within GRANOLA (Equation (8)) are message-passing layers, as the ones in Xu et al. [63], Morris et al. [41].

The advantages of using RNF-augmented MPNNs for normalization. We start by observing that the integration of our GRANOLA in an MPNN allows to easily default to an MPNN augmented

with RNF [56], as we formalize in Proposition E.2 in Appendix E. The idea of the proof lies in the ability of the first normalization layer to default to outputting its input RNF, enabling the rest of the architecture to function as an MPNN augmented with these RNF. The significance of this result lies in the fact that MPNN + GRANOLA inherits the ( ϵ, δ ) -universal approximation properties previously proved for MPNNs augmented with RNF [1, 51]. This naturally solves the limitations of existing normalizations identified in Section 2.2, as an MPNN + GRANOLA can leverage universality to approximate such functions. Importantly, the universality of MPNNs augmented with RNF further implies that GRANOLA is fully adaptive , as it can approximate any equivariant function on the input graph [1], and therefore can approximate functions returning different normalization values for non-isomorphic nodes.

Why RNF are necessary? While GRANOLA employs RNF to compute the normalization affine parameters γ ( ℓ ) b,n,c and β ( ℓ ) b,n,c , the same procedure can be applied without the use of RNF, a variant we denoted as GRANOLA-NO-RNF in Section 3 (Equation (11)). However, we theoretically demonstrate next that an MPNN + GRANOLA-NO-RNF not only loses the universal approximation properties, but is also not more expressive than standard MPNNs.

Proposition 4.1 (RNF are necessary in GRANOLA for increased expressive power) . Assume our input domain consists of graphs of a specific size. For every MPNN with GRANOLA -NO-RNF (Equation (11) ) there exists a standard MPNN with the same expressive power.

Proposition 4.1 is proven by showing that an MPNN with GRANOLA-NO-RNF can be implemented by a standard MPNN, and, therefore, its expressive power is bounded by the expressive power of a standard MPNN. The proof can be found in Appendix E. This limitation underscores the significance of RNF within GRANOLA. Furthermore, our experiments in Section 5 show that omitting the RNF within the normalization results in degraded performance, as GRANOLA always outperforms GRANOLA-NO-RNF. Finally, we remark that this theoretical result emphasizes the necessity of RNF for increased expressiveness and, consequently, for ensuring full adaptivity to the input graph. That is, any normalization generated by GRANOLA-NO-RNF will be limited by the expressive power of standard MPNNs while GRANOLA does not have this limitation.

Relation to expressive GNNs. The results in this section highlight the connection between GRANOLA and expressive GNNs, as our method inherits enhanced expressiveness of MPNNs augmented with RNF. Notably, while MPNNs with RNF have demonstrated theoretical improvements in expressiveness, their practical performance on real-world datasets has not consistently reflected these benefits [22]. Our experimental results indicate that GRANOLA serves as a valuable bridge between theory and practice. Specifically, our findings address the gap between the theoretical expressiveness of MPNNs that use RNF, and their limited practical utility. This is achieved by effectively incorporating RNF within the normalization process rather than simply treating it as an additional input to the MPNN. Importantly, GRANOLA gives rise to a method that is efficient, provably expressive, and performs well in practice, something that, to our knowledge, has never been accomplished by any previous architecture. This provides an additional perspective to our contribution: beyond designing an effective and efficient normalization layer, which is the main scope of our work, we offer a practical approach to realizing the theoretical benefits of RNF. Finally, we conclude this section by remarking that GNN ( ℓ ) NORM in Equation (8) can be modified to be any other expressive architecture, and our design choice was mainly guided by the computational practicality of RNF, that allows GRANOLA to offer increased expressive power while retaining the linear complexity of MPNNs, as discussed in Appendix G. We refer the reader to Morris et al. [43] for a survey on expressive methods.

5 Experimental Results

In this section, we evaluate the performance of GRANOLA. In particular, we seek to address the following questions: (1) How does GRANOLA compare to other normalization methods? (2) Does GRANOLA achieve better performance than its natural baselines that also leverage RNF, that is, how important is the graph-adaptivity within GRANOLA ? (3) How does GRANOLA compare to its variant GRANOLA -NO-RNF , which does not leverage RNF within the normalization, that is, how important are the RNF within GRANOLA ? In what follows, we present our main results and refer to Appendices F to H for details and additional experiments, including timings and ablation studies. Our code is available at https://github.com/MosheEliasof/GRANOLA .

Baselines. For each task, we consider the following baselines: (1) Standard normalization layers, (2) Graph-designated normalization layers and (3) Our natural baselines, namely: (i) GRANOLA-NO-RNF, which corresponds to the variant of GRANOLA that uses only the intermediate features, without RNF, inside the normalization (Equation (11)) to assess the practical importance of RNF for full graph-adaptivity; (ii) RNF as a positional encoding (RNF-PE), where we augment only the initial input features with RNF, as in Sato et al. [56]; (iii) The normalization that uses only RNF, without any message passing layer inside the normalization, to compute the normalization affine parameters γ ( ℓ ) b,n,c and β ( ℓ ) b,n,c . This is achieved by considering Z ( ℓ ) b = R ( ℓ ) b in Equation (8). We denote this baseline by RNF-NORM and remark that it corresponds to a version of GRANOLA without graph-adaptivity. For a fair comparison, in all the baselines, as well as in our method, we employ GIN [63] or GINE [32] layers, which are maximally-expressive within the MPNN family. Furthermore, we compare GRANOLA to GNNs in the same complexity class and refer the reader to Appendix H.8 for additional comparisons.

ZINC. We experiment with the ZINC-12K molecular dataset [57, 28, 21], where the goal is to regress the solubility of molecules. As can be seen from Table 1, GRANOLA achieves significantly lower (better) mean-absolute-error compared to existing standard and graph-designated normalization layers, while outperforming all its natural baselines. Moreover, GRANOLA emerges as the best-performing model with the same time complexity of a standard MPNN.

OGB. We test our GRANOLA on the OGB collection [31], and report its performance in Table 2. Notably, GRANOLA consistently improves over existing standard normalization methods while retaining similar asymptotic complexity. For instance, on MOLBACE, we achieve an accuracy of 79.92% compared to 72.97% when using BatchNorm, an improvement of 6.95%. Compared with graph-designated normalization methods, GRANOLA also offers a consistent improvement. As an example, on MOLESOL we obtain a RMSE of (lower is bet-

0

100

200

300

400

500

Epoch

10

2

10

1

10 0

Training Loss

BatchNorm + RNF-PE

Identity (No Norm)

BatchNorm

InstanceNorm

LayerNorm-node

LayerNorm-graph

GraphNorm

GRANOLA

(a) ZINC-12K

0

100

200

300

400

500

Epoch

10

3

10

2

10

1

Training Loss

BatchNorm + RNF-PE

Identity (No Norm)

BatchNorm

InstanceNorm

LayerNorm-node

LayerNorm-graph

GraphNorm

GRANOLA

(b) MOLHIV

0

100

200

300

400

500

Epoch

10

2

10

1

Training Loss

BatchNorm + RNF-PE

Identity (No Norm)

BatchNorm

InstanceNorm

LayerNorm-node

LayerNorm-graph

GraphNorm

GRANOLA

(c) MOLTOX21

ter) 0.960, compared to the second best graph normalization layer, GraphNorm, that achieves 1.044. It is also noteworthy to mention the performance gap between GRANOLA and its natural baselines, such as RNF-PE, which emphasizes the practical benefit of graph-adaptivity within GRANOLA.

TUDatasets. We experimented with popular datasets from the TUD [42] repository. Our results are reported in Table 14 in Appendix H.8, suggesting that GRANOLA consistently achieves higher accuracy compared to its natural baselines, as well as standard and graph-designated normalization techniques. For example, on the NCI109 dataset, GRANOLA achieves an accuracy of 83 . 7% , compared to the second-best normalization technique, GraphNorm, with an accuracy 82 . 4% .

Training convergence of GRANOLA. In addition to improved downstream task performance being one of the main benefits of a normalization layer, accelerated training convergence is also an important desired property [34, 11]. Figure 4 shows that GRANOLA offers faster convergence and lower MAE compared to other methods.

Combining GRANOLA with expressive methods. Since our goal is to understand the impact of normalization layers, our experiments focus on studying the benefit of augmenting standard and well-understood MPNNs with GRANOLA. However, it is also interesting to understand if expressive, domain-expert approaches such as GSN [10] can also benefit from GRANOLA. To this end, we augment GSN with GRANOLA, and report the results in Table 3. These results further underscore the versatility of GRANOLA, which can be coupled with any GNN layer and improve its performance.

Discussion. Our experimental results cover standard normalization layers, as well as graph normalization methods, evaluated on 11 datasets from diverse sources, and applied to various tasks. Throughout all experiments, a common theme is the performance consistency of GRANOLA. Specifically, GRANOLA always improves over its natural baselines and other normalization techniques across all datasets. In contrast, other existing methods exhibit less clear trends in their performance. While some methods achieve competitive results on certain datasets, they may struggle on others. Notable examples are GraphNorm and PairNorm, which, despite offering improved performance compared to BatchNorm on most of the OGB datasets, show worse results on ZINC-12K. Furthermore, standard normalization layers also lack consistency. As an example, consider InstanceNorm, which is beneficial in some OGB datasets, yet, it does not offer favorable results on ZINC-12K.

6 Conclusions

In this paper, we discuss the existing landscape of feature normalization techniques in Graph Neural Networks (GNNs). Despite recent advances in designing graph normalization techniques, the optimal choice remains unclear, with methods not offering consistent performance improvements across tasks. To address this challenge, we identify a desirable property of graph normalization layers, namely

adaptivity to the input graph, and argue that it can be obtained only with expressive architectures. To incorporate this property, we present GRANOLA, a normalization layer that adjusts node features based on the specific input graph, leveraging Random Node Features (RNF). Our theoretical analyses support the design choices of GRANOLA, demonstrating its increased expressiveness. Empirical evaluations across diverse benchmarks consistently highlight the superior performance of GRANOLA over existing normalization methods, as well as other baselines with the same time complexity.

Limitations and impact. Although GRANOLA exhibits promising results, there are areas for potential improvement in future research. For instance, investigating alternative designs for GNN ( ℓ ) NORM could further enhance the performance, as, in certain datasets, there is still a gap between GRANOLA and expressive GNNs. Additionally, exploring ways to reduce memory and time complexity (which are still linear in the number of nodes) represents an important avenue for future research. Furthermore, by improving the performance of GNN through GRANOLA we envision a positive impact in domains such as drug discovery.

References

Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. arXiv preprint arXiv:2010.01179 , 2020.

Muhammad Awais, Md Tauhid Bin Iqbal, and Sung-Ho Bae. Revisiting internal covariate shift for batch normalization. IEEE Transactions on Neural Networks and Learning Systems , 32(11): 5082-5092, 2020.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. In International Conference on Learning Representations , 2022.

Beatrice Bevilacqua, Moshe Eliasof, Eli Meirom, Bruno Ribeiro, and Haggai Maron. Efficient subgraph gnns by learning effective selection policies. arXiv preprint arXiv:2310.20082 , 2023.

Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.

Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normalization. Advances in neural information processing systems , 31, 2018.

Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yu Guang Wang, Pietro Liò, Guido Montúfar, and Michael Bronstein. Weisfeiler and lehman go cellular: CW networks. In Advances in Neural Information Processing Systems , volume 34, 2021.

Cristian Bodnar, Fabrizio Frasca, Yu Guang Wang, Nina Otter, Guido Montúfar, Pietro Lio, and Michael Bronstein. Weisfeiler and lehman go topological: Message passing simplicial networks. ICML , 2021.

Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-yan Liu, and Liwei Wang. Graphnorm: A principled approach to accelerating graph neural network training. In International Conference on Machine Learning , pages 1204-1215. PMLR, 2021.

Kaixuan Chen, Shunyu Liu, Tongtian Zhu, Ji Qiao, Yun Su, Yingjie Tian, Tongya Zheng, Haofei Zhang, Zunlei Feng, Jingwen Ye, et al. Improving expressivity of gnns with subgraph-specific factor embedded normalization. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 237-249, 2023.

Yihao Chen, Xin Tang, Xianbiao Qi, Chun-Guang Li, and Rong Xiao. Learning graph normalization for graph neural networks. Neurocomputing , 493:613-625, 2022.

Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count substructures? Advances in neural information processing systems , 33:10383-10395, 2020.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇ ckovi´ c. Principal neighbourhood aggregation for graph nets. In Advances in Neural Information Processing Systems , 2020.

Leonardo Cotta, Christopher Morris, and Bruno Ribeiro. Reconstruction for powerful graph representations. In NeurIPS , 2021.

George Dasoulas, Ludovic Dos Santos, Kevin Scaman, and Aladin Virmaux. Coloring graph neural networks for node disambiguation. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 2126-2132, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics , 2019.

Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv:1610.07629 , 2016.

Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations , 2022.

Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research , 24(43):1-48, 2023.

Moshe Eliasof, Fabrizio Frasca, Beatrice Bevilacqua, Eran Treister, Gal Chechik, and Haggai Maron. Graph positional encoding via random feature propagation. In International Conference on Machine Learning , pages 9202-9223, 2023.

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Fabrizio Frasca, Beatrice Bevilacqua, Michael Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. Advances in Neural Information Processing Systems , 35:31376-31390, 2022.

Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens. Exploring the structure of a real-time, arbitrary neural artistic stylization network. arXiv preprint arXiv:1705.06830 , 2017.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Rafael Gómez-Bombarelli, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science , 4(2):268-276, Jan 2018.

David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 630-645. Springer, 2016.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems , volume 33, 2020.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations , 2020.

Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision , pages 1501-1510, 2017.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

Richard J Larsen and Morris L Marx. An introduction to mathematical statistics . Prentice Hall Hoboken, NJ, 2005.

Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergcn: All you need to train deeper gcns, 2020.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. In NeurIPS , 2019.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 33, 2019.

Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) , 2020.

Christopher Morris, Yaron Lipman, Haggai Maron, Bastian Rieck, Nils M Kriege, Martin Grohe, Matthias Fey, and Karsten Borgwardt. Weisfeiler and leman go machine learning: The story so far. arXiv preprint arXiv:2112.09992 , 2021.

Ryan Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Relational pooling for graph representations. In International Conference on Machine Learning , pages 4663-4673. PMLR, 2019.

Mark Newman. Networks: An Introduction . Oxford university press, 2010.

Hoang Nt and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv:1905.09550 , 2019.

Eduardo Ogasawara, Leonardo C Martinez, Daniel De Oliveira, Geraldo Zimbrão, Gisele L Pappa, and Marta Mattoso. Adaptive normalization: A novel data normalization approach for non-stationary time series. In The 2010 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2010.

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2337-2346, 2019.

Nikolaos Passalis, Anastasios Tefas, Juho Kanniainen, Moncef Gabbouj, and Alexandros Iosifidis. Deep adaptive input normalization for time series forecasting. IEEE transactions on neural networks and learning systems , 31(9):3760-3765, 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

Omri Puny, Heli Ben-Hamu, and Yaron Lipman. Global attention improves graph networks generalization. arXiv preprint arXiv:2006.07846 , 2020.

Omri Puny, Derek Lim, Bobak Kiani, Haggai Maron, and Yaron Lipman. Equivariant polynomials for graph neural networks. In International Conference on Machine Learning , pages 28191-28222. PMLR, 2023.

Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems , 35:14501-14515, 2022.

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Representations , 2019.

Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems , 31, 2018.

Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In Proceedings of the 2021 SIAM international conference on data mining (SDM) , pages 333-341. SIAM, 2021.

Teague Sterling and John J. Irwin. ZINC 15 - ligand discovery for everyone. Journal of Chemical Information and Modeling , 55(11):2324-2337, 11 2015. doi: 10.1021/acs.jcim.5b00559.

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 , 2016.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog) , 38(5):1-12, 2019.

Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV) , pages 3-19, 2018.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019.

Chaoqi Yang, Ruijie Wang, Shuochao Yao, Shengzhong Liu, and Tarek Abdelzaher. Revisiting over-smoothing in deep gcns. arXiv preprint arXiv:2003.13663 , 2020.

Gilad Yehudai, Ethan Fetaya, Eli Meirom, Gal Chechik, and Haggai Maron. From local structures to size generalization in graph neural networks. In International Conference on Machine Learning , pages 11975-11986. PMLR, 2021.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in neural information processing systems , 34:28877-28888, 2021.

Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10737-10745, 2021.

Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small relu networks are powerful memorizers: a tight analysis of memorization capacity. Advances in Neural Information Processing Systems , 32, 2019.

Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests. arXiv preprint arXiv:2302.07090 , 2023.

Bohang Zhang, Shengjie Luo, Liwei Wang, and Di He. Rethinking the expressive power of GNNs via graph biconnectivity. In The Eleventh International Conference on Learning Representations , 2023.

Muhan Zhang and Pan Li. Nested graph neural networks. In Advances in Neural Information Processing Systems , volume 34, 2021.

Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In International Conference on Learning Representations , 2020.

Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any GNN with local structure awareness. In International Conference on Learning Representations , 2022.

Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu. Towards deeper graph neural networks with differentiable group normalization. Advances in neural information processing systems , 33:4917-4928, 2020.

Kuangqi Zhou, Yanfei Dong, Kaixin Wang, Wee Sun Lee, Bryan Hooi, Huan Xu, and Jiashi Feng. Understanding and resolving performance degradation in deep graph convolutional networks. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 2728-2737, 2021.

Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5104-5113, 2020.

A Additional Related Work

Standard normalization layers. BatchNorm [34] is arguably one of the most widely used normalization schemes in deep learning. Despite its success, there is little consensus on the exact reason behind the improvements it generally yields. While some studies argue that the effectiveness of BatchNorm lies in its ability to control the problem of internal covariate shift [34, 2], other works [7, 55] attribute the success to the promotion of faster convergence. Similarly to the other domains, also in graphlearning, BatchNorm normalizes each channel (feature) independently by computing the mean and standard deviation across all the elements in the batch, i.e., across all the nodes in all the graphs in the batch. Another normalization approach that was adopted in graph-learning is InstanceNorm [58], which was originally introduced in image style-transfer tasks to remove image-specific contrast and style information. In the context of graphs, InstanceNorm normalizes each channel (feature) independently by computing the mean and standard deviation across all nodes within each graph separately. Additionally, LayerNorm [3] was originally proposed for recurrent models, and it is also widely used in Transformers architectures [59]. In the context of graph learning, LayerNorm can take two different versions: Layernorm-node and LayerNorm-graph [23]. The former normalizes each node independently, by computing the mean and standard deviation across all channels within each node separately, while the latter uses the mean and standard deviation across all the channels of all nodes in the entire graph. We visualize these variants in Figure 1. The common theme of these three discussed methods is that they were not originally designed with graph-learning in mind. Next, we discuss graph-designated normalization techniques.

Graph-designated normalization layers. PairNorm [72] was introduced to mitigate the issue of oversmoothing in GNNs, where repeated graph convolutions eventually make node representations indistinguishable from one and the other. The key idea of PairNorm is to ensure that the total pairwise feature distances remain constant across layers, preventing the features of distant nodes from becoming overly similar or indistinguishable. Yang et al. [64] proposed a different understanding of the oversmoothing issue in GNNs: the authors show that upon initialization GNNs suffer from oversmoothing, but GNNs learn to anti-oversmooth during training. Based on this conclusion, the authors design MeanSubtractionNorm, which removes the mean from the inputs, without rescaling the features. When coupled with GCN [37], MeanSubtractionNorm leads to a revised Power Iteration that converges to the Fiedler vector, resulting in a coarse graph partition and faster training. To enhance the performance of GNNs on large graphs, Li et al. [39] propose MessageNorm, a method that normalizes the aggregated message for each node before combining it with its node features, demonstrating experimental benefits. DiffGroupNorm [74] was designed to alleviate oversmoothing by softly clustering nodes and normalizing nodes within the same group to increase their smoothness while also separating node distributions among different groups. Zhou et al. [75] explored the impact of transformation operators, such as normalizations, that transform node features after the propagation step in GNNs, and argue that they tend to amplify node-wise feature variance. This effect is shown to lead to performance drop in deep GNNs. To mitigate this issue, they introduced NodeNorm, which scales the feature vector of each node by a root of the standard deviation of its features. This approach also shares similarities with LayerNorm-node. While the aforementioned normalization techniques have made significant strides in mitigating the oversmoothing phenomenon in GNNs, other normalization methods have been proposed to address problems beyond this specific concern. Cai et al. [11] showed that InstanceNorm serves as a preconditioner for GNNs, thus accelerating their training convergence. However, it also provably degrades the expressiveness of the GNNs for regular graphs. To address this issue, they propose GraphNorm, which builds on top of InstanceNorm by adding a learnable multiplicative factor to the mean for each channel. GraphSizeNorm [21] was proposed based on promising empirical results, and aims at normalizing the node features by dividing them by the graph size, before applying a standard BatchNorm layer. UnityNorm [13] was introduced to learn graph normalization by optimizing a weighted combination of existing techniques, including LayerNorm-node, InstanceNorm, and BatchNorm. Finally, SuperNorm [12] first computes subgraph-specific factors, encompassing the number of nodes and the eigenvalues of the neighborhood-induced subgraph for each node. These subgraph-specific factors are then explicitly embedded at the beginning and end of a standard BatchNorm layer. This approach ensures that any arbitrary GNN layer becomes at least as powerful as the 1-WL test, while simultaneously mitigating the oversmoothing issue. We conclude this paragraph by acknowledging the existence of normalization techniques that focus on normalizing the adjacency matrix before the GNN layers. One common example is the symmetric normalization which is used in GCN [37] and subsequent

works. Despite their wide use, especially on node-level tasks, these normalizations may fail to capture structural information and lead to less expressive architectures. For instance, a common practice of normalizing the adjacency matrix by dividing its rows by the sum of their entries is equivalent to employing a mean aggregator over the neighbors, which has been shown to fail to distinguish certain non-isomorphic nodes [63]. For this reason, in this work, we consider only normalization layers applied to node features after every GNN layer.

AdaIN and StyleGANs. Adaptive instance normalization (AdaIN) [33, 19, 26] was originally proposed for real-time arbitrary style transfer. Given a content input and a style input, AdaIN adjusts the mean and variance of the content input to match those of the style input. This allows for the transfer of stylistic features from the style input to the content input in a flexible and efficient manner. Motivated by style-transfer literature, Karras et al. [36] introduced StyleGANs as a variant of GANs [27] which differs in the generator architecture. More precisely, the generator in StyleGANs starts from a learned constant input and adjusts the 'style' of the image at each convolution layer based on the latent code through the usage of AdaIN. Similarly to StyleGANs, we adjust node features at every layer through the usage of noise, i.e., random node features, akin to the role played by the latent code in StyleGANs.

Random Node Features (RNF) in GNNs. Random input features were used in the context of GNNs to improve the expressive power of Message Passing Neural Networks (MPNNs) [44, 51, 1, 56, 17]. Importantly, MPNNs augmented with random node features have been shown to be universal (with high probability) in Puny et al. [51], Abboud et al. [1], a result we naturally inherit when GRANOLA defaults to simply utilizing random node features. Notably, despite the universality results, GNNs augmented with RNF do not consistently improve performance on real-world datasets [22]. We show instead that GRANOLA consistently leads to improved performances.

B Comparison of Different Normalizations

In this section we present the different normalization techniques that have been specifically proposed for graph data and highlight their connection to standard normalization schemes (i.e., BatchNorm, InstanceNorm, LayerNorm-node, LayerNorm-graph) whenever this is present.

PairNorm. PairNorm [72] first centers node features by subtracting the mean computed per channel across all the nodes in the graph, similarly to InstanceNorm. Then, it scales the centered vector by dividing it by the square root of the mean (computed across nodes) of the norm of each node feature vector, where the norm is computed over the channel dimension. That is, the center operation has equation:

where µ b,n,c is as in InstanceNorm (Equation (5)), while the scale operation can be written as

with s ∈ R a hyperparameter and the norm is

MeanSubtractionNorm. MeanSubtractionNorm [64] is similar to InstanceNorm, but it simply shifts its input without dividing it by the standard deviation (and without affine parameters). That is

and µ b,n,c as Equation (5).

MessageNorm. MessageNorm [39] couples normalization with the GNN layer. In particular, it defines the update rule of H ( ℓ ) as follows,

where s ∈ R a hyperparameter and m ( ℓ -1) b,n is the message of node n ∈ [ N ] in graph b ∈ [ B ] , which can be defined as

with ρ ( ℓ ) , ζ ( ℓ ) learnable functions such as MLPs and N b v neighbors of node v in graph b .

DiffGroupNorm. DiffGroupNorm [74] first softly clusters nodes and the normalizes nodes within the same cluster by means of BatchNorm. That is, for each graph b ∈ [ B ] in the batch, it computes a soft cluster assignment as

where W ( ℓ ) ∈ R C × D with D ∈ N the number of clusters, and therefore S ( ℓ ) b ∈ R N × D . Let us denote by S ( ℓ ) ∈ R B × N × D the matrix containing the cluster assignments for all graphs in the batch, where the b -th row of S ( ℓ ) is S ( ℓ ) b . DiffGroupNorm computes a linear combination of the output of BatchNorm applied to each cluster (where the feature is weighted by the assignment value), and adds the result to the input embedding, that is

where BATCHNORM leverages Equation (4), and λ ∈ R is a hyperparameter. Notably, the term ˜ h ( ℓ ) b,n,c in Equation (15) is similar to a skip connection, with the difference lying in the fact that skip connections usually add the output of the previous layer ℓ -1 after the norm instead of the output of the current layer before the norm.

NodeNorm. NodeNorm [75] is similar to LayerNorm-node, but it simply divides its input by a root of its standard deviation, without shifting it and without affine parameters:

with σ as in Equation (6) and p ∈ N a hyperparameter.

GraphNorm. GraphNorm [11] builds upon InstanceNorm by adding an additional learnable parameter α ( ℓ ) ∈ R C which is the same for all nodes and graphs. The equation can be written as

and µ b,n,c , σ b,n,c as Equation (5).

GraphSizeNorm. GraphSizeNorm [21] normalizes the node features by dividing them by the graph size, before applying a standard BatchNorm:

UnityNorm. UnityNorm [13] consists of a weighted combination of four normalization techniques, where the weights λ 1 , λ 2 , λ 3 , λ 4 ∈ R are learnable. That is

where LAYERNORM-NODE, INSTANCENORM and BATCHNORM leverage respectively Equations (4) to (6). ADJACENCYNORM computes the statistics for each node across all features of all its neighbors, that is

SuperNorm. SuperNorm [12] embeds subgraph-specific factors into BatchNorm. First, for each node n in graph b it extracts the subgraph induced by its neighbors, denoted as S b,n . Then, it computes the so called subgraph-specific factors for each subgraph, which are the output of an hash function over the number of nodes in the subgraph and its eigenvalues. That is the subgraph-specific factor of S b,n is

where | V S b,n | denotes the number of nodes in S b,n and Eig S b,n its eigenvalues, with ϕ and ψ injective functions.

Subgraph-specific factors for all subgraphs in a graph b are collected into a vector M G b ∈ R N × 1

which is used to obtain two additional vectors M SN b , M RC b ∈ R N × 1 defined as

where ⊙ denotes the element-wise product. Then, the normalization computes the segment average of ˜ H ( ℓ ) b for each graph b , denoted as ˜ H ( ℓ ) , segment b ∈ R N × C , where a row n is defined as

Then, the input ˜ H ( ℓ ) b is calibrated by injecting the subgraph-specific factors as well as the graph statistics obtained via ˜ H ( ℓ ) , segment b . That is,

where 1 C ∈ { 1 } C × 1 and W ( ℓ ) = 1 N w ( ℓ ) ⊤ with 1 N ∈ { 1 } N × 1 and w ( ℓ ) ∈ R C × 1 is a learnable parameter. After injecting subgraph-specific factors, the normalization layer performs BatchNorm on the calibration features, that is

Finally, subgraph-specific factors are embedded after BatchNorm, by computing

where γ ( ℓ ) , β ( ℓ ) ∈ R C × 1 are learnable affine parameters, and P ( ℓ ) b ∈ R N × C is a matrix where each entry is obtained as

where w ( ℓ ) RE ∈ R C is a learnable parameter and M RE b ∈ R N × C is obtained as

Equation (20) represents the output of SuperNorm, followed by an activation function ϕ .

C Additional Motivating Examples

In the following we elaborate on the failure cases of existing normalization layers. Throughout this section, we will assume that all GNN layers are message-passing layers, and, in particular, we focus on the maximally expressive MPNN layers presented in Morris et al. [41], which have been shown to be as expressive as the 1-WL test. In particular Equation (1) can be rewritten as follows,

Example C.1 (BatchNorm reduces GNN capabilities to compute node degrees (Figure 2)) . Consider the following task: Given a graph G , for each node predict its degree. Assume that our batch contains B graphs, and, for simplicity, assume that they all have the same number of nodes N . 5 Assuming such graphs do not have initial node features, we follow standard practice [63] and initialize H (0) b for each graph b ∈ [ B ] as a vector of ones, i.e., H (0) b = 1 ∈ R N × 1 . The output of the first GNN layer is already the degree of each node, or a function thereof:

where W (0) 1 ∈ R 1 × C , W (0) 2 ∈ R 1 × C , are learnable weight matrices, and C ∈ N is the hidden feature dimensionality. Note that since the input is one dimensional, all output channels behave identically. We consider the case where C = 1 . Importantly, in our example, we have that H (0) b W (0) 1 is the same for all nodes in all graphs in the batch, because H (0) b and W (0) 1 are the same for all nodes and graphs. Therefore, the term H (0) b W (0) 1 acts as a bias term in Equation (22). Thus, for each node, the output of the first GNN layer is simply a linear function of the node degree, which is computed by the term A b H (0) b in Equation (22). Now, consider the normalization layer NORM applied to the output of this function, and assume for now that there are no affine parameters. First, the BatchNorm normalization layer subtracts the mean computed across all nodes in all graphs, as shown in Equation (4). For all nodes having an output smaller than the mean, this subtraction returns a negative number, which is zeroed out by the application of the ReLU which follows the normalization. Therefore, assuming no further layers, for these nodes the prediction can only be 0 regardless of the actual degree, and therefore is incorrect. In Figure 2, we provide a concrete example where this limitation occurs.

Remark C.2 (Deeper networks are also limited) . It is important to note that even when the number of layers is greater than one, and assuming C = 1 in all layers, the problem persists, because the analysis can be applied to every subsequent layer's output by simply noticing that subtracting the mean will always zero out features less than the mean.

Remark C.3 (BN with Affine transformation is also limited) . We highlight that the inclusion of learnable affine parameters in BatchNorm, while potentially mitigating the problem during training by means of a large affine parameter β (1) c that shifts the negative outputs before the ReLU, does not offer a definitive solution. Indeed, it is always possible to construct a test scenario where one graph has nodes with degrees significantly smaller than those seen while training, for which the learned shift β (1) c is not sufficiently large to make the output of the normalization layer positive.

It is important to mention that the aforementioned example potentially serves as a failure case for other methods that are based on removing the mean µ b,n,c as calculated by BatchNorm in Equation (4). For instance, GraphSizeNorm [21] also suffers from the same issue, as this normalization technique scales the node features by the number of nodes before applying BatchNorm. Furthermore, Example C.1 can be adjusted to induce a failure case for the normalization operation (excluding the skip link) in DiffGroupNorm [74], which is also based on BatchNorm. This can be achieved by ensuring that a subset of nodes has a degree less than the average within all the clusters to which it is (softly) assigned.

Example C.1 can be easily adapted to represent a failure case for InstanceNorm. The adaptation involves treating the entire batch as a single graph, which can be further refined to avoid having disconnected components by adding a small number of edges connecting them.

Example C.4 (InstanceNorm reduces GNN capabilities to compute node degree (Figure 2 considering all graphs as disconnected components in a single graph)) . Consider the following task: Given a graph G , for each node predict its degree. Assuming the graph does not have initial node features, we follow standard practice [63] and initialize H (0) b = 1 ∈ R N × 1 . The output of the first GNN layer is already the degree of each node, or a function thereof. Now, consider the normalization layer NORM applied to the output of this function. First, the InstanceNorm normalization layer subtracts the mean computed across all nodes within each graph in the batch, as shown in Equation (5). For all nodes having an output smaller than the mean within their graph, this subtraction returns a negative number, which is zeroed out by the application of the ReLU activation function, applied following the normalization. Therefore, assuming no further layers, for these nodes the prediction can only be 0 regardless of the actual degree, and therefore is incorrect. Similarly to BatchNorm, the inclusion of learnable affine parameters as well as the stacking of additional single-channel layers is not sufficient to solve the problem.

The aforementioned limitation extends directly to other graph-designed normalizations based on InstanceNorm, such as PairNorm and GraphNorm 6 as they also shifts the input in the same way, based on the mean µ b,n,c as defined in Equation (5).

Example C.5 (LayerNorm-node reduces GNN capabilities to compute node degree (Figure 5)) . Consider the following task: Given a graph G , for each node predict its degree. Consider a graph G consisting of a path of 3 nodes. Assuming the graph does not have initial node features, we initialize H (0) b = 1 ∈ R N × 1 , with b = 0 as we only have one graph. We will assume that W (0) 1 = 0 (Equation (21)). Therefore, the output of the first GNN layer can be written as:

which, for each node, is a vector having at each entry the degree of the node multiplied by a learnable weight entry. Compare now one of the node having degree d ∈ N , denoted as v with the node having degree 2 d , denoted as u . Clearly, we have that the vector of node features in u is equal to two times the vector of node features in v . Therefore, by subtracting the mean computed across the channel dimension, as per Equation (6), we obtain

for all c ∈ [ C ] . This implies that it is not possible to then distinguish the degree of these two nodes.

1

2

1

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

0

0

0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Figure 5: A graph, where subtracting the mean of the node features computed on the feature dimension, as in LayerNorm-node and related methods, results in the loss of capacity to compute node degrees.

Remark C.6 (Deeper networks are also limited) . It is important to note that even when the number of layers is greater than one, assuming W ( ℓ -1) 1 = 0 in all GNN layers (Equation (21)), the problem persists. Indeed, in the next layer, both nodes v and u will aggregate neighbors having identical features. Therefore the vector of node features in u will still be equal to two times the vector of node features in v , a difference that is removed by the subtraction of the mean computed per node.

D Variants of GRANOLA

In the main paper, we have additionally considered GRANOLA-NO-RNF, a variant of GRANOLA that does not sample RNF and instead uses only ˜ H ( ℓ ) b (Equation (11)) . In the following, we present an additional variant.

GRANOLA-MS. Equation (10) can be specialized to the case where the affine parameters correspond to the mean and standard deviation computed per node across the feature dimension of Z ( ℓ ) b . That is,

We refer to this variant of Equation (10) which uses the pre-defined mean and standard deviation functions in Equation (24) for f ( ℓ ) 1 , f ( ℓ ) 2 as GRANOLA-MS, where MS stands indeed for mean and standard deviation. The idea behind GRANOLA-MS is to explicitly align the node-wise mean and variance of ˜ H ( ℓ ) b to match those of Z ( ℓ ) b . Note that, while GRANOLA-MS follows a predefined realization of f 1 and f 2 by using the mean and standard deviation, it is still a learnable normalization technique, as Z ( ℓ ) b is learned in Equation (8). In Appendix H.8, we provide empirical comparisons with this specialized case.

E Proofs

Theorem E.1 (Existing normalization techniques limit MPNNs' expressivity) . Let f be a stacking of GIN layers with non-linear activations followed by sum pooling. Let f norm be the architecture obtained by adding InstanceNorm or BatchNorm without affine parameters. Then f norm is strictly less expressive than f .

Proof. All non-isomorphic graphs that can be distinguished by f norm can clearly be distinguished by f as the normalization only shifts and scale its input. To show that f norm is strictly less expressive than f , consider two CSL graphs with different numbers of nodes. These are distinguishable by f . However, applying InstanceNorm to the output of GIN results in a zero matrix, as shown in Cai et al. [11, Proposition 4.1]. Similarly, if the batch consists of these two graphs, applying BatchNorm results in a zero matrix. Since the output of the normalization is a zero matrix, they are indistinguishable by f norm , concluding our proof.

Proposition E.2 (MPNN with GRANOLA can implement MPNN with RNF) . Let f norm be a stacking of MPNN layers interleaved with GRANOLA normalization layers (Equation (10) ), followed by activation functions. There exists a choice of hyperparameters and weights such that f norm defaults to MPNN + RNF [1].

Proof. We only need to show a choice of hyperparameters and weights that makes an MPNN + GRANOLA default to an MPNN + RNF, which is the model obtained by performing message passing on the input graph where the initial node features are concatenated to RNF.

Since RNF in an MPNN + GRANOLA are introduced in the GRANOLA layer, we choose the first MPNN layer, which precedes any normalization (Equation (1)), to simply repeat its inputs using additional channels. In the case of GraphConv layers, this is easily obtained by W (0) 1 = ( I I ) and W (0) 2 = 0 , with I ∈ { 0 , 1 } C × C . With these choices, ˜ H (1) b in Equation (1) becomes ˜ H (1) b = H (0) b ⊕ H (0) b = X b ⊕ X b . Notably, this concatenation is only introduced to make the dimension of ˜ H (1) b match the dimension of the concatenation of the initial node features with RNF having the same dimension.

Consider the first GRANOLA layer, ℓ = 1 . It is sufficient to set the activation functions inside GNN (1) NORM to be the identity function and to properly set the weights of GNN (1) NORM (Equation (8)) and f (1) 1 , f (1) 2 (Equation (9)) such that the normalization layer returns H (0) b ⊕ R (1) b , with R (1) b ∈ R N × K and K chosen such that K = C . For example, if GNN (1) NORM is composed by a single GraphConv layer [41] (with an identity activation function), we have

then for GNN (1) NORM is sufficient to choose W NORM , (1) 2 = 0 , W NORM , (1) 1 = ( 0 0 I 0 0 I ) , where I ∈ { 0 , 1 } C × C is the identity matrix. For f (1) 1 , f (1) 2 its is sufficient to set f (1) 1 to always return a zero vector, and f (1) 2 to be the identity function. With these choices, Equation (2) becomes

which represents the input of the next GNN layer, and matches the input of an MPNN + RNF. Therefore, we are only left to show that subsequent applications of GRANOLA layers behave as the identity function, since the GNN layers will instead behave as those in the MPNN + RNF.

After the first GRANOLA layer, ℓ > 1 , it is sufficient to set GNN ( ℓ ) NORM (Equation (8)) and f ( ℓ ) 1 , f ( ℓ ) 2 (Equation (9)) to return its input ˜ H ( ℓ ) ∈ R N × C (while discarding R ( ℓ ) b ∈ R N × K ). Assuming a single layer in it (with an identity activation function), this can be accomplished by setting W NORM , ( ℓ ) 2 = 0 and W NORM , ( ℓ ) 1 = ( I 0 ) , f ( ℓ ) 1 to always return a zero vector, and f ( ℓ ) 2 to be the identity function. With these choices, Equation (10) becomes

Therefore, these two steps imply that an MPNN with GRANOLA implements an MPNN with RNF.

Corollary E.3 (MPNN + GRANOLA is universal with high probability) . Let Ω N be a compact set of graphs with N ∈ N nodes and g a continuous permutation-invariant graph function defined over Ω N . Let f norm be a stacking of MPNN layers interleaved with GRANOLA normalization layers (Equation (10) ) followed by activation functions. Then, if random node features inside GRANOLA are sampled from a continuous bounded distribution with zero mean and finite variance, for all ϵ, δ > 0 , there exist a choice of hyperparameters and weights such that, ∀ G = ( A , X ) ∈ Ω ,

where f norm ( A , X ) is the output for the considered choice of hyperparameters and weights.

Proof. The proof follows by showing the choice of hyperparameters and weights which makes an MPNN augmented with GRANOLA satisfy the assumptions of Puny et al. [51, Proposition 1]. In all our GRANOLA layers, R ( ℓ ) b ∈ R N × K b is drawn from a continuous and bounded distribution for any graph b in a batch of B graphs. Therefore, we only need to show that the overall architecture can default to an MPNN augmented with these RNF. This follows from Proposition E.2.

Proposition 4.1 (RNF are necessary in GRANOLA for increased expressive power) . Assume our input domain consists of graphs of a specific size. For every MPNN with GRANOLA -NO-RNF (Equation (11) ) there exists a standard MPNN with the same expressive power.

Proof. The proof follows by showing that an MPNN with GRANOLA-NO-RNF can be implemented by a standard MPNN (without any normalization), which therefore represents an upper-bound of its expressive power.

We assume that all the MPNN layers are maximally expressive MPNNs layers of the form of GraphConv [41], and that all activation functions inside GRANOLA-NO-RNF are ReLU activation functions. Since the convolution layers are the same in both f norm and f , we are only left to show that an MPNN (without normalization, with ReLU activations) can implement a layer of GRANOLANO-RNF. Recall that a single GRANOLA layer can be written as

and, since we are not considering RNF, Z ( ℓ ) b is obtained with Equation (11), recalling that GNN ( ℓ ) NORM is also composed by MPNN layers interleaved by ReLU activation functions per our assumption. We denote the number of layers of this GNN by L ( ℓ ) NORM . We next show how to obtain Equation (26) using multiple layers of an MPNN which takes as input ˜ H ( ℓ ) . We will denote intermediate layers of this MPNN as ˆ H ( t ) , t ≥ 1 .

First Step, compute Z ( ℓ ) b . The first layers of this MPNN are used to obtain Z ( ℓ ) b given ˜ H ( ℓ ) b , effectively mimicking GNN ( ℓ ) NORM . Since we will later need also ˜ H ( ℓ ) b , we will use the last feature dimensions in every layer representation to simply copy it in every subsequent layer. Importantly, however, simply copying ˜ H ( ℓ ) b in the last dimensions using an identity weight matrix may not be sufficient, as the application of ReLU non-linearity would clip negative entries to 0. Therefore, we copy both ˜ H ( ℓ ) b and -˜ H ( ℓ ) b in the last dimensions, and at the end recover ˜ H ( ℓ ) b as ˜ H ( ℓ ) b = ϕ ( ˜ H ( ℓ ) b ) -ϕ ( -˜ H ( ℓ ) b ) , with ϕ the ReLU activation.

For t = 1 , the MPNN layer is simply responsible of replicating its input ˜ H ( ℓ ) b , so that we can later use the first C channels to obtain Z ( ℓ ) b and the last to later recover ˜ H ( ℓ ) b . Therefore,

with ˆ W (1) 1 = ( I I -I ) and ˆ W (1) 2 = 0 , with I ∈ { 0 , 1 } C × C the identity matrix, and ϕ is the identity activation function. This means that ˆ H (1) b = ˜ H ( ℓ ) b ⊕ ˜ H ( ℓ ) b ⊕-˜ H ( ℓ ) b . For t > 1 and t ≤ L ( ℓ ) NORM +1 , we need to mimic GNN ( ℓ ) NORM from Equation (11) on the first C dimensions. This is achievable by

exactly the same as the corresponding weights of GNN ( ℓ ) NORM , ϕ is the ReLU activation. Therefore, after L ( ℓ ) NORM +1 layers, we have

We can then use an additional MPNN layer t to recover ˜ H ( ℓ ) b by setting ˆ W ( t ) 1 = ( I 0 0 I 0 -I ) , ˆ W (1) 2 = 0 , and obtain

Finally, we rely on the ability of MLPs to memorize a finite number of input-output pairs (see Yun et al. [68] and Yehudai et al. [65, Lemma B.2]) to implement Equation (26) given its input. This is achieved by making the three subsequent layers of the MPNN behave as a 3-layer MLP (with ReLU activations) by simply zeroing out ˆ W ( t ) 2 . In this way, we have obtained Equation (26) through MPNN layers only.

F Experimental Details

We implemented GRANOLA using Pytorch [50] (BSD-style license) and Pytorch-Geometric [24] (MIT license). We ran our experiments on NVIDIA RTX3090 and RTX4090 GPUs, both having 24GB of memory. We performed hyperparameter tuning using the Weight and Biases framework [6]. To sample the random node features, we followed Abboud et al. [1] and used a standard Gaussian with mean 0 and variance 1, sampling random node features for each forward pass. For each node, we sample independently a number of random features equal to the channel dimension of ˜ h ( ℓ ) b,n , that is K is equal to C in Equation (8). Recall that when augmenting a GNN architecture with GRANOLA, we have two types of networks: (i) the main downstream network, and (ii) the normalization network returning Z ( ℓ ) b (see Equation (8)). In both networks, and throughout all datasets that do not contain edge features (e.g., in TUDatasets), we employ GIN layers [63] to perform message passing. In case edge features are available, such as in ZINC, we use the GINE variant of GIN, as prescribed in Dwivedi et al. [21]. For brevity, we refer to both as GIN throughout the paper. Our MLPs are composed of two linear layers with ReLU non-linearities. Our normalization network is kept small and comprises a number of layers tuned in { 1 , 2 , 3 } and an embedding dimension that is the same as its input ˜ H ( ℓ ) . Each experiment is repeated for 5 different seeds, and we report the average and standard deviation result. Details of hyperparameter grid for each dataset can be found in the following subsections.

F.1 ZINC-12k

We consider the dataset splits proposed in Dwivedi et al. [21], and use the Mean Absolute Error (MAE) both as loss and evaluation metric. For all models, we used a batch size tuned in { 32 , 64 , 128 } . To optimize the model we use the Adam optimizer with initial learning rate of 0 . 001 , which is decayed by 0 . 5 every 300 epochs. The maximum number of epochs is set to 500 . The test metric is computed at the best validation epoch. The downstream network is composed of a number of layers in { 4 , 6 } , with an embedding dimension tuned in { 32 , 64 } .

F.2 OGB datasets

We consider the scaffold splits proposed in Hu et al. [31], and for each dataset we used the loss and evaluation metric prescribed therein. In all experiments, we used the Adam optimizer with initial learning rate of 0 . 001 . We tune the batch size in { 64 , 128 . We employ a learning rate scheduler that that follows the procedure prescribed in Bevilacqua et al. [4]. We also consider dropout in between layers with probabilities in { 0 , 0 . 5 } . The downstream network has the number of layers in { 4 , 6 } with embedding dimensions in { 32 , 64 , 128 } . The maximum number of epochs is set to 500 for all models. The test metric is computed at the best validation epoch.

F.3 TUDatasets

For all the experiments with datasets from the TUDatasets repository, we followed the evaluation procedure proposed in Xu et al. [63], consisting of 10-fold cross validation and metric at the best averaged validation accuracy across the folds. The downstream network is composed of a number of layers tuned in { 4 , 6 } layers with embedding dimension in { 32 , 64 } . We use the Adam optimizer with learning rate tuned in { 0 . 01 , 0 . 001 } . We consider batch size in { 32 , 64 , 128 } , and trained for 500 epochs.

G Complexity and Runtimes

Complexity. As described in Section 3, and specifically in Equation (10), our GRANOLA takes random node feature and hidden learned node features, and propagates them using a GNN backbone denoted by GNNNORM to compute intermediate (expressive) features, which are then used to calculate the normalization statistics, as shown in Equation (9). The calculation can be implemented either by considering their mean and standard deviation, as in GRANOLA-MS, or more generally by employing an MLP in GRANOLA, as described in Equation (24) and Equation (9), respectively. In our experiments, GNN ( ℓ ) NORM in GRANOLA is also an MPNN, similar to the downstream backbone model. Therefore, including our GRANOLA layers does not change the asymptotic computational complexity of the architecture, which remains within the computational complexity of MPNNs (e.g., Morris et al. [41], Xu et al. [63]). Specifically, each MPNN layer is linear in the number of nodes | V | and edges | E | . Since a single GRANOLA layer is composed by L ( ℓ ) NORM MPNN layers, assuming it the same for all ℓ , it has a time complexity of O ( L NORM · ( | V | + | E | )) . Every downstream MPNN layer in our framework uses a GRANOLA normalization layer, and therefore, assuming L downstream MPNN layers, the overall complexity of an MPNN augmented with GRANOLA is O (( L · L NORM ) · ( | V | + | E | )) , compared to the complexity of an MPNN without GRANOLA which amounts to O ( L · ( | V | + | E | )) . In practice, L NORM is a hyperparameter between 1 to 3 and can therefore be considered a constant.

Runtimes. While the asymptotic complexity remains linear with respect to the number of nodes and edges in the graph, as in standard MPNNs, our GRANOLA requires some additional computations due to the hidden layers in the normalization mechanism. To measure the impact of these additional layers, we measure the required training and inference times of GRANOLA, whose results are reported in Table 4. Specifically, we report the average time per batch measured on a Nvidia RTX-2080 GPU. For a fair comparison, in all methods, we use the same number of layers, batch size and number of channels. Our results indicate that while GRANOLA requires additional computational time, it is still a fraction of the cost of more complex methods like Subgraph GNNs (5.2 × faster than efficient expressive models like Subgraph GNNs), while yielding favorable downstream performance. These results indicate that GRANOLA offers a strong tradeoff between performance and cost.

H Additional Experimental Results

H.1 GRANOLA Coupled with Additional Backbones

In this subsection, we evaluate the performance of GRANOLA when coupled with additional backbones beyond GIN and GSN as presented in Section 5. Specifically, we evaluate its integration with GCN [37], GAT [60], and GPS [53], using GRANOLA as the normalization layer. Table 5 shows that adding GRANOLA results in improved performance regardless of the architecture. These results underscore the versatility of GRANOLA, which can be coupled with any GNN layer and improve its performance.

H.2 Using other Expressive Mechanisms in GRANOLA

As explained in Sections 3.1 and 4, in this paper, we chose to use RNF within GRANOLA. Incorporating RNF into our GRANOLA allows it to fully adapt to the input graph, providing different affine parameters for non-isomorphic nodes. Full adaptivity is lost when removing RNF and using a standard MPNN as GNNNORM, as in GRANOLA-NO-RNF. This is because GRANOLA-NO-RNF is not more expressive than an MPNN (Proposition 4.1), and thus, there exist non-isomorphic nodes that will get the same representation (and the same affine parameters). However, any other most expressive architecture used as GNNNORM would achieve the same full adaptivity, and our choice of MPNN + RNF was motivated by its linear complexity.

To make this point clearer, in this subsection, we analyze the performance of a variant of GRANOLA that uses a Subgraph GNN, namely DS-GNN [4], as the GNNNORM, instead of an MPNN + RNF. We denote this variant as GRANOLA-SubgraphGNN. Table 6 shows that GRANOLA-SubgraphGNN behaves similarly to GRANOLA. However, employing a SubgraphGNN as the GNNNORM results in additional complexity coming from the Subgraph GNN itself (which is quadratic rather than linear). Thus, while enhanced expressivity in GNNNORM does not require employing RNF specifically, using an MPNN + RNF provides the advantage of linear complexity.

H.3 GRANOLA building from BatchNorm

As discussed in Section 3, in Equation (10), µ b,n,c and σ b,n,c are the mean and std of ˜ H ( ℓ ) computed per node across the feature dimension, exactly as in LayerNorm-node. This choice was dictated by the fact that we found LayerNorm-node to offer consistent performance across different benchmarks. However, it is also possible to compute µ b,n,c and σ b,n,c across different dimensions, resulting in a variant of GRANOLA that builds on top of other normalization layers rather than LayerNorm-node. In Table 7 we study the effectiveness of the variant of GRANOLA that builds on top of BatchNorm. As can be seen from the table, GRANOLA-BatchNorm outperforms BatchNorm, further highlighting the role of graph adaptivity in normalizations.

H.4 Shared weights vs. Per Layer GNN ( ℓ ) NORM

In our experiments in Section 5, the additional normalization GNN layer, GNN ( ℓ ) NORM , had unique parameters per layer, as evidenced by the superscript ℓ . In Table 8, we show that by using the same GNN ( ℓ ) NORM for all ℓ , that is by sharing the weights across all layers, which overall requires less parameters, our GRANOLA continues to offer competitive results, further highlighting its competitiveness also in cases where parameter budget is low.

H.5 The role of GRANOLA Normalization

To better understand the contribution of GRANOLA, we provide results where we set the normalization term in GRANOLA to zero, leaving only the bias term to be learned. That is achieved by setting γ ( ℓ ) b,n,c = 0 in Equation (10). By following this approach, we isolate the contribution of the normalization itself from the bias in the normalization process. Our results, shown in Table 9, indicate that the normalization term in GRANOLA is significant, and cannot be replaced by a simple bias.

H.6 RNF as PE Ablation Study

GRANOLA benefits from (i) enhanced expressiveness, and (ii) graph adaptivity. Property (i) is obtained by augmenting our normalization scheme with RNF, as shown in Figure 3. Therefore, it is important to ensure that the contribution GRANOLA does not stem solely from the use of RNF, but rather the overall approach and design of our method.

To this end, in addition to the natural baseline of RNF-PE, which uses RNF as positional encoding combined with GIN + BatchNorm (as in [56]), we now provide results of RNF-PE when combined with GIN and different normalization layers. Specifically, we consider Identity (no normalization) and LayerNorm (both graph and node variants). The results are provided in Table 10, together with

the results of our GRANOLA, for reference and convenience of comparison. The results suggest that while the different variants of RNF-PE do not show significant improvement over the baseline of GIN + BatchNorm, our GRANOLA does. These results are further evidence that while RNF are theoretically powerful, they may not be significant in practice, as shown in Eliasof et al. [22]. Instead, it is important to incorporate them in a thoughtful manner, for example, to obtain graph adaptivity within the normalization layer, as in our GRANOLA.

H.7 Normalization GNN depth ablation study

As discussed in Section 3, our GRANOLA utilizes a GNN to learn graph-adaptive normalization shift and scaling parameters, and we denote this GNN by GNN ( ℓ ) NORM . Combined with the RNF as part of the input to GNN ( ℓ ) NORM , we are able to obtain both enhanced expressiveness (from RNF) and graph-adaptivity (by GNN ( ℓ ) NORM ). It is therefore interesting to study the effect of the number of layers in GNN ( ℓ ) NORM on the downstream performance. Specifically, in the case where GNN ( ℓ ) NORM has 0 layers, the experiment defaults to a model very similar the RNF-NORM baseline (with the difference that in RNF-NORM we have Z ( ℓ ) b = R ( ℓ ) b , while in this case we have Z ( ℓ ) b = ˜ H ( ℓ ) b ⊕ R ( ℓ ) b ), thereby losing graph adaptivity. In Table 11, we provide results on a varying number of layers, from 0 to 4. Our results suggest that there is a significant importance in terms of performance to having graph adaptivity in the normalization technique, as offered by our GRANOLA.

H.8 Comparison with additional baselines

In the main paper, in Section 5, we focused on providing a comprehensive comparison with directly comparable methods, i.e., standard normalization methods, graph normalization methods, as well as our own set of natural baselines. In this section, we provide additional comparisons with other expressive approaches, such as positional encoding methods and Subgraph GNNs. Our additional comparisons on ZINC-12K, OGB, and TUDatasets are provided in Table 12, Table 13, and Table 14, respectively.

It is important to note, that while some of these methods achieve better performance than our GRANOLA, they are not within the same complexity class as GRANOLA, and specifically, they are not linear with respect to the number of nodes and edges in the graph, as discussed in Appendix G. For example, RFP-QRˆ L , ˆ A , S learn -DSS [22], which also utilizes the expressive power of RNF, achieves an MAE of 0.1106 on ZINC-12K, while our GRANOLA achieves 0.1203. However, the former is of quadratic complexity with respect to the number of nodes, while GRANOLA is linear, as standard MPNNs. On the other hand, the linear and thus directly comparable RFP ℓ 2 -ˆ L , ˆ A achieves a higher (worse) MAE of 0.1368. Similarly, while there are Subgraph GNNs that can

outperform GRANOLA, these are at least a quadratic in the number of nodes. Therefore, we find that our GRANOLA offers a practical yet powerful approach for utilizing RNF.

Additionally, we observe that in some cases, GRANOLA achieves similar or better performance than other expressive and asymptotically more complex methods. For example, our results on TUDatsets in Table 14 show that GRANOLA offers better performance than DSS-GNN [4] on all considered datasets, despite DSS being quadratic in the number of nodes.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contribution paragraph in the introduction (Section 1) explicitly describes the main contributions of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The conclusion section (Section 6) includes a separate paragraph discussing the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Section 4 presents our main theoretical results, and Appendix E contains the full set of assumptions as well as the proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 3 fully describes our proposed method, and Appendix F contains all experimental details to reproduce our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We offer comprehensive details regarding the implementation and evaluation of our method (Appendix F), our code is available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Appendix F contains all the details, including the hyperparameters and other experimental choices.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All results (Section 5 and Appendix H.8) are accompanied by the corresponding standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix F describes the type of GPUs used in the experiments, Appendix G provides a runtime comparison.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms with the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss potential societal impacts in Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We credited the owners of the assets and the licenses in Appendix F.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel graph-adaptive normalization layer named GRANOLA for Graph Neural Networks (GNNs). The authors argue that existing normalization techniques, such as BatchNorm and InstanceNorm, are not well-suited for GNNs due to their design not considering the unique characteristics of graph-structured data. To address this, GRANOLA is proposed to normalize node features by adapting to the specific characteristics of the graph, using Random Node Features (RNF) to generate expressive node representations. The paper provides theoretical results supporting the design choices and demonstrates through empirical evaluation that GRANOLA outperforms existing normalization techniques across various graph benchmarks.\n\nSTRENGTHS:\n* The paper presents a novel normalization technique specifically tailored for GNNs, addressing a recognized gap in the field where traditional normalization layers do not capture the unique properties of graph data effectively.\n\n* The authors provide a solid theoretical basis for their method, including proofs that GRANOLA can achieve full adaptivity to the input graph, which is a significant contribution to the understanding of normalization in GNNs.\n\n* The paper offers extensive empirical results across multiple datasets and tasks, showing consistent performance improvements of GRANOLA over existing methods, which strengthens the credibility of the proposed technique.\n\nWEAKNESSES:\n* The work focuses on the normalization of GNNs for graph-level tasks. However, there is a type of model that dominates this field: Graph Transformers. I'm curious if this normalization would help with the graph transformers in graph-level tasks.\n\n* Implementation details are not so clear. Is the RNF sampled once and trained like other parameters or just randomly sampled at each mini-batch (or each epoch)?\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GRANOLA (Graph Adaptive Normalization Layer), a novel normalization technique designed specifically for Graph Neural Networks (GNNs). The authors identify a critical gap in existing normalization methods for GNNs, which often fail to capture the unique structural characteristics of graph data or consistently improve performance across various tasks. GRANOLA aims to address these limitations by dynamically adjusting node features based on both the graph structure and Random Node Features (RNF). The key innovation lies in its use of an additional GNN (termed GNN_NORM) to generate normalization parameters that are adaptive to the input graph. The paper provides a comprehensive theoretical analysis of GRANOLA, demonstrating its ability to default to RNF-augmented Message Passing Neural Networks (MPNNs) and proving its increased expressive power compared to standard MPNNs. The authors also show that the use of RNF is necessary for this increased expressiveness. Empirically, the paper presents extensive evaluations across multiple graph benchmarks.\n\nSTRENGTHS:\nGRANOLA presents a novel approach to graph normalization by incorporating graph adaptivity and RNF, addressing a significant gap in existing methods.\n\nThe paper provides solid theoretical analysis, including proofs of GRANOLA's expressive power and its ability to default to RNF-augmented MPNNs.\n\nThe authors conduct extensive experiments across multiple datasets and tasks, demonstrating GRANOLA's consistent performance improvements over existing normalization techniques.\n\nThe paper effectively demonstrates how GRANOLA serves as a valuable bridge between the theoretical expressiveness of RNF-augmented MPNNs and practical performance improvements in graph learning tasks.\n\nGRANOLA maintains the same asymptotic complexity as standard MPNNs while offering improved performance, making it a practical solution for real-world applications.\n\nWEAKNESSES:\nGRANOLA's parameters (gamma and beta) are unique for each node, layer, and feature attribute. This fine-grained adjustment is significantly more detailed than most normalization methods, making GRANOLA resemble a new model architecture rather than a normalization technique. The motivation as a normalization method may be somewhat inappropriate given this level of detail.\n\nThe paper could benefit from additional experiments, such as removing the $\\gamma_{b,n,c}^{(\\ell)}$ term and retaining only the affine intercept term $\\beta_{b,n,c}^{(\\ell)}$. This would help determine whether the improved performance is due to the detailed internal model adjustments or the normalization concept itself.\n\nQUESTIONS:\nThe authors address some limitations of their work in the conclusion section. They could have elaborated more on the practical limitations of GRANOLA, such as potential challenges in implementing it in resource-constrained environments or its scalability to extremely large graphs. While the normalization GNN depth ablation study is available in Appendix H.2, a more detailed discussion on the choice of other hyperparameters for the normalization GNN, such as dimensions, would be beneficial. This is particularly important given that the normalization GNN is at the core of the GRANOLA framework.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper pertains to the problem of using normalisation techniques specifically designed for graph-structured data and Graph Neural Networks (GNNs). Using constructed illustrative examples, the authors claim that existing normalisation techniques (including others designed for graphs) may create expressivity issues to the GNNs to which they are applied. Motivated by these examples, they speculate that one possible reason is the lack of adaptivity of the normalisation (affine) parameters to the input graph. To that end, they propose to replace fixed normalisation parameters with adaptive ones, produced at each layer by an auxiliary GNN for each vertex of the input graph. Additionally (and probably crucially), the auxiliary GNNs do not only receive the current vertex features as inputs but also Random Node Features (typically sampled from a Gaussian). The last one is known to render GNNs universal, and so is the case here. Experimentally, the method is tested on a battery of tasks and ablated across several factors (training convergence speed, combined with more expressive architectures than plain MPNNs), showing consistently improved performance compared to other normalisation methods and competitiveness with the state-of-the-art in several cases.\n\nSTRENGTHS:\n**Clarity**. The paper is well-written, easy-to-follow and all the concepts are clearly explained. It can therefore be understood by a wider audience.\n\n**Contextualisation to related work**. An extensive review and comparisons with related works are provided, allowing the reader to grasp their differences and understand the innovation of the proposed approach.\n\n**Importance to the GNN community.** The techniques proposed by the authors are of general interest to the GNN community since they clearly and consistently improve current normalisation techniques and show good potential for adaptation in practice in various problems. Additionally, it is one of the few works showcasing the practical benefits of RNF (however see also weaknesses), which is a long-standing puzzle for GNN researchers.\n\n**Empirical results**. The proposed method appears to work quite well and consistently in practice, in terms of empirical generalisation. In particular, it provides improvements against all base architectures tested and against all competing normalisation techniques, while both methodological techniques proposed (adaptive normalisation + RNF) are sufficiently ablated and shown to be important to be combined to obtain the desired performance gains.\n\nWEAKNESSES:\n**Reservations regarding claims and the selected way to present findings**. \n- One of my main objections to the presented manuscript is the way the findings were selected to be presented. In particular, the authors present their work as a graph-oriented adaptive normalisation technique and motivate their approach by comparing it against other normalisation techniques, both intuitively using constructed counterexamples and empirically, showcasing consistent improvements. Nevertheless, it seems to me that the main reason behind the empirical success is the random features. For example, see Table 2, where the adaptive normalisation per se does not seem to provide any significant practical advantages. \n- However, as the authors correctly point out, random features are known to behave subpar in practice, despite their theoretical advantages. This leads me to believe that the authors have probably found a way to *improve the performance of random features*. To me, this is an important contribution per se, but the authors have chosen not to present their paper in that way, nor to sufficiently point out this takeaway in their paper. \n- On the other hand, the authors have devoted a substantial part of the paper to motivating adaptive normalisation and discussing other normalisation techniques. In my viewpoint, the intuitive explanations provided (the examples in sec. 2.2. and the last paragraph before section 3.1.) are speculative and do not seem to be the real reason behind the success of the approach. \n- Note also, that in several parts of the paper, the authors claim that existing normalisation techniques limit MPNNs’ expressivity, but I am not sure if this is the case. I think that if the authors choose to keep these claims, they should probably be provided with a more rigorous theoretical statement. \n- I think that the most important question that needs to be addressed is why incorporating RNF into a normalisation layer overcomes their current limitations in providing performance improvements. I recommend that the authors discuss this both in their rebuttal and the paper, as it will provide important insights.\n\n**Efficiency**. Another reservation I have about this work is that computational efficiency might be an issue, but is not adequately discussed.  For example, the runtimes provided by the authors in Table 4 in the appendix, show an almost 3-fold increase in both training and inference time. Although this might not be significant compared to other more expressive GNNs, the comparison between the performance-complexity trade-offs is not clear. Moreover, it is a limitation of this method and should be more clearly discussed in the main paper.\n\n**Limited evaluation against baselines**. Although as I mentioned before, the results are indeed convincing, both from a perspective of a normalisation technique and of a random feature technique, the authors have not sufficiently compared against the state-of-the-art. To their credit, they did include other baselines in the supplementary material and mentioned the gap in performance in their limitations section, but I think it would be fairer to discuss this more prominently (especially since this work can be perceived as an expressive GNN and not as a normalisation technique alone). \n\n**Limited technical novelty**: Finally, a minor weakness is that there is limited novelty from a technical perspective, since if I am not mistaken adaptive normalisation has been proposed before for other domains (e.g. AdaIN, Huang et al., ICCV’17 is a relevant example – discussed by the authors in the appendix) and random features are well-known in the GNN community.\n\nQUESTIONS:\n- Have the authors tried to modify a different normalisation technique, apart from LayerNorm? This might be an interesting ablation study.\n- Do the authors need a GNN normalisation network for each layer of the GNN processing network?\n\nLIMITATIONS:\nThe authors have mentioned some limitations (mainly the gap in performance compared to state-of-the-art), but I think others should be discussed more extensively (e.g. efficiency - see weaknesses). \n\nI do not foresee any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes an adaptive normalization layer for GNNs. It first points out that the traditional normalization layer (BatchNorm, InstanceNorm)  are not specifically designed for graphs and thus may limit the expressive power of GNNs, and single normalization technique can not always be the best for all graphs. Therefore, it proposes the GRANOLA method which is a learnable normalization layer for GNNs, and justify its design with theories.

STRENGTHS:
S1. the paper is well-structured and easy to follow.

S2. GRANOLA consistently bring performance improvement on all benchmark datasets.

S3. Baselines are carefully selected and organized.

WEAKNESSES:
W1. GRANOLA involves an additional GNN module to learn the adaptive normalization, making it more costly and less scalable than its counterparts. It would be better if the authors could provide some experimental results to show how much additional time we will need when applying GRANOLA.

W2. The proposed GRANOLA is only tested on GIN and GSN backbones. It would be better to check if it could be applied to more common backbone models such as GCN, GAT, etc.
 
W3. (This is a minor point). Code is not provided.

QUESTIONS:
Q1. The authors point out that the traditional normalization may not effectively capture the unique characteristics of graph-structured data. I am curious to see is there any empirical study or theoretical results to justify this point?

Q2. Figure 2 provides an example to illustrate that BatchNorm may make the GNN less powerful. Is it possible to have something similar to visualize the performance of BatchNorm, InstanceNorm, LayerNorm, and GRANOLA?

LIMITATIONS:
Please refer to weakness and questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17