DeeperGCN: Training Deeper GCNs with Generalized Aggregation Functions
Authors: Guohao Li
Chenxin Xiong
Ali Thabet
Bernard Ghanem
Venue:  (2021)
Keywords: Graph Neural Networks
Graph Representation Learning

Abstract: Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs. Recent works developed frameworks to train deep GCNs. Such works show impressive results in tasks like point cloud classification and segmentation, and protein interaction prediction. In this work, we study the performance of such deep models in large scale graph datasets from the Open Graph Benchmark (OGB). In particular, we look at the effect of adequately choosing an aggregation function, and its effect on final performance. Common choices of aggregation are mean, max, and sum. It has shown that GCNs are sensitive to such aggregations when applied to different datasets. We further validate this point and propose to alleviate it by introducing a novel Generalized Aggregation Function. Our new aggregation not only covers all commonly used ones, but also can be tuned to learn customized functions for different tasks. Our generalized aggregation is fully differentiable, and thus its parameters can be learned in an end-to-end fashion. We add our generalized aggregation into a deep GCN framework and show it achieves state-of-the-art results in six benchmarks from OGB.

Under review as a conference paper at ICLR 2021
DEEPERGCN: TRAINING DEEPER GCNs WITH GEN- ERALIZED AGGREGATION FUNCTIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs. Recent works developed frameworks to train deep GCNs. Such works show impressive results in tasks like point cloud classification and segmentation, and protein interaction prediction. this work, study the performance of such deep models in large scale graph datasets from the Open Graph Benchmark (OGB). In particular, we look at the effect of adequately choosing aggregation function, and its effect final per- formance. Common choices of aggregation are mean, max, and sum. It has shown that GCNs are sensitive such aggregations when applied different datasets. We further validate this point and propose to alleviate it by introducing a novel Generalized Aggregation Function. Our new aggregation not only covers all com- monly used ones, but also can be tuned to learn customized functions for different tasks. Our generalized aggregation is fully differentiable, and thus its parame- ters can be learned in an end-to-end fashion. We add our generalized aggregation into a deep GCN framework and show it achieves otate-of-the-art results in six benchmarks from OGB.
INTRODUCTION
The rise of availability of non- -un--uulde data (Bronstein et al. 2017 has recently shed interest into the topic of Graph Convolutional Networks (GCNs). GCNs provide powerful deep learning archi- tectures for irregular data, like point clouds and graphs. GCNs have proven valuable for applications in social networks (Tang & Liu 2009), drug discovery (Zitnik Leskovee 2017, Wale al. 2008), recommendation engines (Monti et al. 2017b; Ying et al. 2018). and point clouds (Wang et al.) Li et al. 2019b). Recent works looked at frameworks to train deeper GCN architectures (Li al. 2019b These works demonstrate how increased depth leads to state-of-the-arr performance on tasks like point cloud classification and segmentation, and protein interaction prediction. The power of deep models become more evident with the introduction of more challenging and large- scale graph datasets. Such datasets were recently introduced in the Open Graph Benchmark (OGB) (Hu al. 2020), for tasks of node classification, link prediction, and graph classification.
2018
Graph convolutions in GCNs are based on the notion of message passing (Gilmer et al. 2017). To compute new node feature at each GCN layer, information aggregated from the node and its connected neighbors. Given the nature of graphs, aggregation functions must be permutation invari- ant. This property guarantees invariance/equivarrance
isomorphic graphs (Battaglia et al., 2018 Xu et al. 2019b Maron et al. 2019a). Popular choices for aggregation functions are mean (Kipf & Welling 2016). max (Hamilton ton al. 2017). and sum (Xu et al.) 2019b). Recent works suggest different aggregations have different performance impact depending on the task. For example, mean and sum perform best in node classification (Kipf & Welling 2016), while max is favorable for deal- ing with 3D point clouds (Qi al. 20177] Wang et al. 2019). Currently, all works rely empirical analysis to choose aggregation functions.
In DeepGCNs (Li et al. (2019b)), the authors complement aggregation functions with residual and dense connections, and dilated convolutions, in order to train very deep GCNs. Equipped with these new modules, GCNs with more than 100 layers be reliably trained. Despite the potential of these new modules (Kipf & Welling 2016 Hamilton et al. 2017) Veličković et al. 2018 [u et al. (019a), it is still unclear if they are the ideal choice for DeepGCNs when handling large-scale graphs

---

Under review as a conference paper at ICLR 2021
Permutation Invariant Aggregators
SoftMaxSum_Agg
SoftMax_Ape
Sum
Mean
Max
Min
PowerMeanSum Agg
PowerMean_Agg
Figure 1: Illustration of Generalized Message Aggregation Functions
In this work, analyze the performance of GCNs large-scale graphs. In particular, look the effect of aggregation functions in performance. We unify aggregation functions by proposing novel Generalized Aggregation Function (Figure I suited for graph convolutions. We show how our function covers all commonly used aggregations (mean, max, and sum), and its parameters can be tuned learn customized functions for different tasks. Our novel aggregation is fully differen- tiable and can be learned in end-to-end fashion in deep GCN framework. In our experiments, we show the performance baseline aggregations in various large-scale graph datasets. We then introduce our generalized aggregation and observe improved performance with the correct choice of aggregation parameters. Finally, we demonstrate how learning the parameters of our generalized aggregation, in end-to-end fashion, leads to state-of-ther offthe-art performance in several OGB bench- marks. Our analysis indicates the choice of suitable aggregations is imperative to the performance of different tasks. A differentiable generalized aggregation function ensures the correct aggregation is used for each learning scenario.
We summarize Function. This new function is suitable for GCNs, as it enjoys a permutation invariant property. We show how our generalized aggregation covers commonly used functions such as mean, max, and sum in graph convolutions. Additionally, we show how its parameters can be tuned to improve performance on diverse GCN tasks. Since this new function is fully differentiable, we show how its parameters can be learned in an end-to-end fashion. (2) We run extensive experiments on seven datasets from the Open Graph Benchmark (OGB). Our results show that combining depth with our generalized aggregation function achieves state-of-the-ar in several of these benchmarks.
contributions two-fold: (1) We propose novel Generalized Aggregation
2 RELATED WORK
Graph Convolutional Networks (GCNs). Current GCN algorithms can be divided into two cate- gories: spectral-basee and spatial-baser Based on spectral graph theory, Bruna et al. (2013) firstly developed graph convolutions using the Fourier basis of given graph in the spectral domain. Later, many methods proposed to apply improvements, extensions, and approximations on spectral-based based
GCNs (Kipf & Welling 2016, Defferrard al. 2016 Henaff al.
2015
Levie al., 2018 Li
ct al. 2018 Wu et al. 2019). Spatial-based GCNs (Scarselli et al.
2008
Hamilton et al. 201
Monti al. 70 Niepert al. 2016 Gao al. 2018 Xu et al. 2019b Veličković et 2018
define graph convolution operations directly on the graph by aggregating information from neigh-
bor nodes. To address the scalability issue of GCNs on large-scale graphs, two main categories of
algorithms exist: sampling-based (Hamilton et al.) 2017 Chen et al.) [018h)] Li et al. 2018 Chen
al. 201 8a
Zeng al. 2020) and clustering based (Chiang et al.[20199
Training Deep GCNs. Despite the rapid and fruitful progress of GCNs, most prior work em- ploys shallow GCNs. Several works attempt different ways training deeper GCNs (Hamilton) et al.
2017: Armeni et al. 2017 Rahimi et al.) 2018 Xu et al. 2018). However, all these ap- proaches are limited to 10 layers of depth, after which GCN performance would degrade because of vanishing gradient and wer-smoothinggLi et al.) (2018). Inspired by the merits of training deep CNN-based networks (He et al.] 2016aa] Huang et al.] 2017 Yu & Koltun (016), DeepGCNs (Li
et al. 2019b) propose to train very deep GCNs (56 layers) by adapting residual/dense connections
2

---

Under review as a conference paper at ICLR 2021
(ResGCN/DenssGCN)) and dilated convolutions to GCNs. DeepGCN variants achieve state-of-th of the art results on S3DIS point cloud semantic segmentation (Armeni et al. (017) and the PPI dataset. Many recent works focus further addressing this phenomenon Khicpera et al., 2019; Rong al.) 2020 /hhao & Akoglu 2020; Chen et al. 2020 Gong et al. 2020 Rossi et al. 2020). In particular, Klicpera et al. (20T9) propose a PageRank- based message passing mechanism involving the root node in the loop. Alternatvely, DropEdge (Rong et al.] 2020) randomly removes edges from the graph, and PairNorm (Zhao & Akoglu 2020) develops a novel normalization layer. We find that the choice of aggregation may also limit the power of deep GCNs. In this work, we thoroughly study the important relation between aggregation functions and deep GCN architectures.
Aggregation Functions for GCNs, GCNs update node's 's feature vector by aggregating feature information from its neighbors in the graph. Many different neighhorhood aggregation functions that possess a permutation invariant property have been proposed (Hamilton et al. 2017, Veličkovid jet al. 2018 Xu al.) (019b). Specifically, |Hamilton et al. (2017) examine mean, max, and LSTM aggregators, and they empirically find that max and LSTM achieve the best performance. Graph attention networks (GATs) (Veličković et al.) 2018) employ the attention mechanism (Bahdanau et al. 2015) obtain different and trainable weights for neighbor nodes by learning the attention between their feature vectors and that of the central node. Thus, the aggregator in GAT's operates like learnable weighted mean. Furthermore, Xu et al. (2019h) propose GCN architecture, denoted Graph Isomorphism Network (GIN), with a sum aggregation that has been shown to have high discriminative power according to the Weisfeiler-Lehman (WL) graph isomorphism test (Weisfeiler & Lehhman 1968). In this work, we propose generalized message aggregation functions, a new Family of aggregation functions, that generalizes conventional aggregators including mean, max and sum. With the nature of differentiablity and continuity, generalized message aggregation functions provide a new perspective for designing GCN architectures.
3 REPRESENTATION LEARNING ON GRAPHS
Graph Representation. A graph G is usually defined as a tuple of two sets G = (V,E), where 01, U2, ..., UN } and & C V x V are the sets of vertices and edges, respectively. If an edge ei = (ei, v/)) E E for undirected graph, Eij is an edge connecting vertices v; and By; for directed graph, exy is an edge directed from vi to Vy. Usually, a vertex v and an edge e in the graph are associated with vertex features h, E RR and edge features h, E RC respectively!
GCNs for Learning Graph Representation. We define a general graph representation learning operator F, which takes as input graph 9 and outputs transformed graph g', i.e. G' = F(G). The features even the topology of the graph can be learned or updated after the transformation F. Typical graph representation learning operators usually learn latent features or representations for graphs such DeepWalk (Perozzi et al.) 2014), Planetoid YYang al. 2016), Node2Vec (Grover & Leskovec 2016). Chebyshev graph CNN (Defferrard et al. (0166), GCN (Kipf & Welling (016)),
Neural Message Passing Network (MPNN) (Gilmer et al. 2017). GraphSage (Hamilton al. 2017
GAT (Veličković et al. 2018) and GIN (Xu et 2019bb). In this work, we focus on the GCN family and its message passing framework (Gilmer et al.J 2017 Battaglia et al.) 2018). To be specific, message passing based on the GCN operator F. operating on vertex e E V at the 1-th layer is defined
as follows:
m p 0)) (h
(.),, Vu (V))
m = (00 (ml) ! u (EN(()))
(2)
h. (++1) (h ..),
(3)
where ((), (0)) and ( are all learnable or differentiable functions for message construction, mes- sage aggregation, and vertex update at the I-th layer, respectively. For simplicity, we only consider the case where vertex features are updated at each layer. It is straightforward to extend it to edge
features. Message construction function (9)) is applied to vertex features h. (!) of its neighbor's features ( and the corresponding edge features b to construct an individual message m for
(I)
In some cases, vertex features edge features are absent.
3

---

Under review as a conference paper at ICLR 2021
each neighbor u -v).. Message aggregation function (d) is commonly permutation invari- ant set function that takes as input a countable unordered message set [mo u E (V())) J. where
(1) R., and outputs reduced aggregated message m. (1) R2. The permutation invariance
men
of ( guarantees the invariance/equivarinnee to isomorphic graphs (Battaglia et al. 2018 (()) can
simply be a symmetric function such as mean (Kipf & Welling 2016), max (Hamilton et al. 2017).
or sum (Xu et al. 2019b). Vertex update function e0)) combines the original vertex features h. and
the aggregated message m (!)
to obtain the transformed vertex features h (!+1)
4 BEYOND MEAN, MAX, AND SUM AGGREGATION FUNCTIONS
Property (Graph Isomorphic Equivariance), If a message aggregation function Ç is permutation invariant to the message set { myv u E (u) }, then the message passing based GCN operator F is equivariant to graph isomorphism, i.e. for any isomorphic graphs G1 and G2 = a * Gr, F(G2) =
* F())), where * denotes a permutation operator on graphs.
The invariance and equivariance properties sets GCNs have been discussed in many recent works. Zaheer et al. (2017) propose DeepSets based on permutation invariance and equivariance to deal with sets inputs. Maron et al. (2019c) show the universality of invariant GCNs to any continuous invariant function. Keriven & Peyré (2019) further extend it to the equivariant case. Maron al. (2019b) compose networks by proposing invariant or equivariant linear layers and show that their models are as powerful as any MPNN (Gilmer et al., (017). In this work, we study permutation invariant functions of GCNs, which enjoy these proven properties.
4.1 GENERALIZED MESSAGE AGGREGATION FUNCTIONS
To embrace the properties of invariance and equivariance (Property []. many works in the graph learning field tend to use simple permutation invariant functions like mean (Kipf & Welling 2016).
max (Hamilton et al. 2017) and sum (Xu al.. 2019b). Inspired by the Seasfeiler-Lehmaae (WL) graph isomorphism test (Weisfeiler& & Lehman 1968). Xu et al. (2019b) propose a theoretical frame- work and analyze the representational power GCNs with meun, max and sum aggregators. Al- though mean and max aggregators are proven to be less powerful than sum according to the WI test in (Xu et al.] (2019b), they are found to be quite effective the tasks of node classification (Kipt & Welling 2016 Hamilton et al., (017) and 3D point cloud processing (Qi et al. 2017 [Wang et al., 2019) To go beyond these simple aggregation functions and study their charactcristics, we define generalized aggregation functions in the following.
Definition 2 (Generalized Message Aggregation Functions). We define a generalized message ag- gregation function $2 (.) as a function that is parameterized by a continuous variable z to produce a family of permutation invariant set functions, iee. Vz, (=(+) is permutation invariant to the order of messages in the set { m... u )
In order to subsume the popular mean and max aggregations into the generalized space, we further define generalized mean-max aggregation parameterized by scalar for message aggregation.
Definition 3 (Generalized Mean-Max Aggregation). If there exists pair of say IT. 22 such that for any message set limg-pei (-) = Mcan(-) )) and limg-wa (-) = Max(-), then (.) is a generalized mean-max aggregation function.
The nice properties of generalized mean-max aggregation functions can be summarized as follows: (1) they provide a large family of permutation invariant aggregation functions; (2) they are contin- uous and differentiable in and are potentially learnable; (3) it is possible to interpolate between Ij and x2 to find a better aggregator than mean and max for a given task. To empirically validate these properties,
propose two families of generalized mean- max aggregation functions based on Definition B namely SoftMax aggregation and PowerMean aggregation.
Proposition 4 (SoftMax Aggregation). Given any message set { mya I E ((() }. mau E RDD Aggg(( is generalized mean-max aggregation function, where SoftMax
Mcan(-) denotes the arithmetic mean.
4

---

Under review as a conference paper at ICLR 2021
exp(8meu) EveN(()) EieN(os exp(Sm..) ture.
Myu- Here, 3 is continuous variable called an inverse tempera-
The SoftMax function with temperature has been studied in many machine learning areas, e.g- Energy-Basee Learning (LeCun et al. 2006), Knowledge Distillation (Hinton et al. 2015) and Re- inforcement Learning Gao & Pavel 2017). Here, for low inverse temperatures 3, Softtax-Aggaaa behaves like a mean aggregation. For high inverse temperatures, it approaches a max aggregation. Formally, oimz-oSoftMax Agga (-) Mean(-) and sim.ssooSofMMa A-ga(-) Max(-). It can be regarded as a weighted summation that depends on the inverse temperature 3 and the values of the elements themselves. The full proof of Proposition iis in the Appendix.
Proposition 5 (PowerMean Aggregation). Given any message set { u (N(()) }. a generalized mean-mux aggregation function, where PowerMean Ag8,()) = WWo Σ Ne)N(e) 1/.. Here, is non-zero, continuous variable
mmy R., PowerMean
denoting the p-th power.
auasi-arithmetic mean (Kolmgoorov & Castelnuovo 1930) was proposed to unify the family of mean functions. Power mean one member of the Quasi-arithmetic mean family. It a generalized mean function that includes harmonic mean, geometric mean, arithmetic mean, and quadratic mean. The main difference between Proposition and 5 is that Proposition [5] only holds when message features are all positive, i.e. Møu E IR. In particular, we have PowerMean A and lim PowerMean_Agee (-) = Max(-). PowerMean.Agg, geometric mean aggregation when = or - O, respectively. See the Appendix for the proof.
Mean(-) becomes the harmonic or the
To enhance expressive power according to the WL test (Xu et al. 2019b), we generalize the function space to cover the sum aggregator by introducing another control variable on the degree of vertices. Proposition 6 (Generalized
Bean-Max-SSe Aggregation). Given any generalized mean-max ag- gregation function (.).), we can generalize the function to cover sum by combining it with the degree of vertices. For instance, by introducing variable y, we can compose generalized mean- max-sun aggregation function as [N(v]] (x(+). We can observe that the function becomes Sum aggregation when (x(:) is Mean aggregation and y 1. By composing with SoftMax aggregation and PowerMean aggregation, we obtain SoftMaxSum Agg .a) (-) and PowerMeanSum Agg (p.a)) aggregation functions, respectively.
4.2 GENERALIZED AGGREGATION NETWORKS (GEN)
Generalized Message Passing Layer. Based on the Propositions above, construct simple mes- sage passing based GCN network that satisfies the conditions in Proposition and The key idea is to keep all the message features to be positive, that generalized mean-max aggregation functions (SoftMax (.g and PowerMean (-), can be applied. We define the message construction function (d) as follows:
(1) m ra
p 12) (h ( ( Re + 1(h h + Vu E (Nv)
(4)
where ReLU(-)) is a rectified linear unit (Nair & Hinton 2010) that outputs values to be greater or equal to zero, 1(.) is an indicator function being when edge features exist otherwise 0, and is small positive constant chosen to be 10-7 As the conditions
satisfied, can choose
the message aggregation function (+)(.) to be either SoftMax SoftMax-Agg,(( (-g PowerMean Agg,, (.), SoftMaxSum (gg(s.)) (-), or PowerMeanSum (g (.). As for the vertex update function o we use a simple layer perceptron, where (i()) MLP(h!) + m!!)
Skip Connections and Normalization. Skip connections and normalization techniques are impor- tant to train deep GCNs. Li al. (2019) 2019b) propose residual GCN blocks with components following the ordering: GraphConv - Normalization + ReLU - Addition. He et al. (2016b) studied the effect of ordering of ResNet components their paper, the output range of the residual function should be (-oc, +oc). Activation functions such as ReLU before addition may impede the representational power of deep models. Therefore, adopt pre-activation variant of residual connections for GCNs, which follows the ordering:
CNNs, showing its importance. As recommended in
5

---

Under review as a conference paper at ICLR 2021
GraphConv + Addition. Empirically, find that the pre-activation version performs better. In our architectures, normalization methods such as BatchNorm (loffe & Szegedy 2013) LayerNorm (Ba et al. 2016) are applied to normalize vertex features.
Normalization + ReLU
EXPERIMENTS
We propose GENeralized Aggregation Networks (GEN) equipped with generalized message aggre- gators. To evaluate the effectiveness of these aggregators, perform extensive experiments on the Open Graph Benchmark (OGB) (Hu et al.] 2020), which includes a diverse set challenging and large-scale tasks and datasets. We first conduct a comprehensive ablation study on the task of node property prediction on ogbn-proteins and ogbn-arxiv datasets. Then, we apply our GEN framework on the node property prediction dataset dogbn-produse three graph property prediction datasets googgmmohhiv, ogbg-molpeba and ogbg-ppt spp), and one link property prediction dataset (ogbl-collab).
5.1 EXPERIMENTAL. SETUP
Baseline Models. The PlainGCN model stacks GCNs from 3 layers to 112 layers without skip connections. Each GCN layer uses the same message passing operator as in GEN except the aggre- gation function is replaced by Sum(-), Mean(-), Max(-) aggregation. LayerNorm is used in every layer before the ReLU activation function. Similar to tiLi et al. (2019b), we use Res- GCN layers by adding residual connections to PlainGCN following the ordering: GraphGonv Normalization + ReLU + Addition. We construct the pre-activation version of ResGCN by changing the order of residual connections Normalization ReLU GraphGonv Addition. We denote this as ResGCN+ to differentiate it from ResGCN. The effect of residual connections can be found in Appendix A
BatchNorm
The ResGEN models are designed using the message passing functions described in Sec- tion 4.2 The only difference between ResGEN and ResGCN+ is that generalized message ag- gregators are used instead of Sum(-), Mean(-), ), or Max(-). For simplicity, we study generalized mean-max aggregators (i.e. PoftMax_Agg.o (-) and PowerMean_Age, (-)) which are parameterized by only one scalar. To explore the characteristics of the generalized message aggregators, we in- stantiate them with different hyper-parameterrs
ResGEN.
Here, we freeze the values of B to 10", where
( 2, 0, 1, 2, 4} and p to (-1, 10,110,,,,,2 1, 100..
DyResGEN. contrast to ResGEN, DyResGEN learns variables 3, p or y dynamically for every layer at every gradient descent step. By learning these variables, we avoid the need to painstakingly search for the best byper-parameters. In doing so, DyResGEN can learn aggregation functions that adapt the training process and the dataset. We study the potential of learning these variables for our proposed uggregators: SoftMax_Agg, TowerMean.Agg,. (-), SeetMaxSummAggaae (-), and PowerrMaaSSum (g (+)-
Datasets. Traditional graph datasets have been shown limited and unable to provide reliable evalu- ation and rigorous comparison among methods (Hu et al. 2020 Dwivedi et al., 2020). Reasons in- clude their small-scale nature, non-negligible duplication or leakage rates, unrealistic data splits, etc. Consequently, we conduct our experiments on the recently released datasets of Open Graph Bench- mark (OGB) (Hu et al. 2202),, which overcome the main drawbacks of commonly used datasets and thus are much more realistic and challenging. OGB datasets cover variety of real-world applica- tions and span several important domains ranging from social and information networks to biological networks, molecular graphs, and knowledge graphs. They also span a variety of prediction tasks at the level of nodes, graphs, and links/edges. In this work, experiments are performed on three OGB datasets for node property prediction, three OGB datasets for graph property prediction, and one OGB dataset for link property prediction. We introduce these seven datasets briefly in Appendix More detailed information about OGB datasets can be found in (Hu et al. 2020).
Implementation Details. We first perform ablation studies on the ogbn-proteins and ogbn-arxiv datasets. Then, we evaluate our model on the other datasets and compare the performances with state-of-the-arr (SOTA) methods. Since the peoneiioteins dataset is very dense and comparably large, full-batch training is infeasible when considering very deep GCNs. We simply apply a random partition generate batches for both mini-batch training and test. We set the number of partitions to

---

Under review as a conference paper at ICLR 2021
10 for training and 5 for test, and set the batch size to subgraph. In comparison, the ogbn-arxd arxiv dataset is relatively small, so we conduct experiments via full batch training and test in this case.
5.2 RESULTS
Aggregators may Limit the Power of Deep GCNs. Although pre-activation residual connections alleviate the effect of vanishing gradients and enable the training of deep GCNs, the choice of ag- gregation function is crucial to performance. In (a) ResGCN+, we study how conventional aggregators (i.e. Sum, Mean and Max) behave on ogbn-proteins and ogbn-arxiv. We find that not all of them benefit from network depth. The aggregators perform inconsistently among different datasets and cause significant gaps in performance. For instance, the Max aggregator outperforms the other two by a large margin (~ 1%) for all network depths on ogbn-proteins, but reaches unsat- isfactory results (< 70%) and even becomes worse with depth increasing on ogbn-arxiv. The Mean aggregator performs the worst on ogbn-proteins, but the best (72.31%) with 28 layers ogbn-arxiv.
Table I: Ablation studies of aggregation functions on the ogbn- proteiis and gebi-arxiv datasets
(a)
oghn-proteins
ogbn-arxiv
Model
#Layers
Sum
Mean
Max
SoftMax
Sum
Mcan
Max
PowerMeanSum
3
82.67
79.69
83.47
83.42
70.89
71.17
69.59
72.12
ResGCN+
7
83.00
80.84
84.65
84.81
71.17
71.83
69.57
72.31
14
83.33
82.25
85.16
85.29
71.50
72.03
68.97
72.14
28
83.98
83.28
85.26
85.51
71.32
72.31
66.91
72.40
56
84.48
83.52
86.05
86.12
112
85.33
83.40
85.94
86.15
avg.
83.80
82.16
85.09
85.22
71.22
71.83
68.76
72.24
ogbn-proteins
SoftMax
Model
#Laycrs
10
2
10
10
102
10a
10*
3
79.69
78.90
77.80
81.69
83.24
83.16
83.07
83.21
ResGEN
80.81
80.71
79.83
83.85
84.66
84.60
84.68
14
82.44
82.14
81.24
84.39
85.13
84.96
84.99
84.85
28
83.13
82.47
81.78
85.08
85.07
85.35
85.80
85.82
56
83.62
83.45
82.86
85.76
85.97
86.20
85.98
86.19
112
83.50
83.61
83.16
85.77
86.38
86.27
86.27
86,30
82.20
81.88
81.11
84.42
84.96
85. 85
85.12
85.17
(c)
ogbn-proteins
PowerMean
Model
#Layers
10-3
2
3
4
5
10
3
82.34
81.06
78.52
80.23
82.01
81.61
82.89
82.89
ResGEN
7
83.36
81.08
81.02
83.49
83.67
84.82
84.54
84,50
14
83.73
80.64
82.45
84.15
84.48
84.64
85.00
85.08
28
84.56
80.92
82.58
84.16
85.20
85.87
85.34
85.76
56
84.46
83.49
85.04
85.68
85.90
85.64
85.74
112
85.13
81.10
83.92
85.47
85.70
86.01
86.09
86.31
avg.
83.93
80.95
82.00
83.76
84.46
84.81
84.92
85.05
ogbn-proteins
SoftMax
SoftMaxSum
PowerMcan
PowerMeanSum
Model
#Layers
Fixed
Learned
Fixed
Learned
Fixed
Learned
Fixed
Learned
3
81.69
83.42
83.06
83.42
78.52
82.25
81.70
83.71
DyResGEN
83.85
84.81
84.71
84.63
81.02
84.14
83.23
84.62
14
84.39
85.29
84.77
85.03
82.45
85.04
83.96
84.83
28
85.08
85.51
85.64
85.66
82.58
85.04
84.59
85.96
56
85.76
86.12
85.63
85.50
83.49
85.27
85.37
85.81
112
85.77
86.15
86.11
86.13
83.92
85.60
85.71
86.01
avg.
84.42
85.22
84,99
85.06
82.00
84.56
84.09
85.16
Exploring Generalized Message Aggregators. In Table [ (b) & (c) ResGEN, we examine
SoftMax_Agg,h and PowerMean_Agg, aggregators on eebnn-protein by measuring test ROC- AUC. Since both are generalized mean-max aggregations, they can theoretically perform at least as good as Mean and Max through interpolation. For SoftMax Agg, when 3 = 10 - performs sim- ilarly Mean aggregation (82.20% vs. 82.16%). As increases to 10 it achieves slightly better
7

---

Under review as a conference paper at ICLR 2021
performance than Max aggregation. Remarkably, 112-layer ResGEN with SoftMax-Age reaches 86.38% and 86.30% ROC-AUC when / = 10 and 3 = 10 respectively. For PowerMean_Agg, find that it reaches almost the same ROC-AUC as Mean when p = (arithmetic mean). We also observe that all other orders of mean except = 10-3 (akin to geometric mean) achieve better performance than the arithmetic mean. PowerMean_Agg _gg with p 10 reaches the best ROC-AUC 86.31% % with 1 12 layers. However, due to some numerical issues in PyTorch (Paszke et al. 2019). we are not able to use larger p. These results empirically validate the discussion on existence of better generalized mean-max aggregators beyond mean and max in Section 4.1
Learning Dynamic Aggregators. Trying out every possible aggregator or searching hyper- parameters is computationally expensive. Therefore, we propose DyResGEN to explore the potential of learning dynamic aggregators by learning the parameters 3. p. and even y within GEN. T DyResGEN reports the results of learning B, f&ay p and p&ay for SoftMax_Agg SoftMaxSumAAgg. respectively. In practice, y is bounded from to by sigmoid function. In all experiments, we initialize the values of B, p to 1 and y to 0.5 at the beginning of training. In order to show the improvement of the learning process, we also ablate experiments with fixed initial values. We denote aggregators with fixed initial values as Fixed and learned aggregators as Learned. We see that learning these variables consistently boosts the av- erage performances of all the learned aggregators compared the fixed initialized counterparts, which shows the effectiveness of learning adaptive aggregators. In particular, when / is learned, -EesSGEN-SffMaax achieves 86. 15% 112 layers. We observe that DyResGEN-SoftMMe out- performs the best ResGEN-SoftMax (3 = 10*) in terms of the average performance (85.22% vs. 85.17%). Interesting, we find generalizing the sum aggregation with PowerMean significantly im- prove the average performance from 84.56% to 85. 15.16% We also put the best learned generalizing message aggregators in [ ResGCN+ with gray color for a convenient comparison.
DowerMeanAAgg and PowerMeanSum
Comparison with SOTA. We apply our GCN models to six other OGB datasets and compare re- sults with the published SOTA method posted on OGB Learderboard at the time of this submission (See Table 2} The methods include Deepwalk (Perozzi et al. (014). GCN (Kipf' & Welling] 2016), GraphSAGE (Hamilton et al.) 2017). GIN (Xu et al. 2019b). GIN or GCN with virtual nodes, JKNet (Xu et al., 2019a), GaAN (Zhang al. 20188) GatedGCN (Bresson & Laurent 2018). GAT (Veličković et al. 2018). HIMP (Fey et al. 2020), GCNII (Ming Chen et al.J 2020). DAGNN (Liu et al. 2020). The provided results each dataset obtained by averaging the results from 10 independent runs. It is clear that our proposed GCN models outperform SOTA on all four datasets.
In two of these datasets (ogbn-proteins and ogbg-ppa), the improvement is substantial. The imple- mentation details and more experimental results can be found in the Appendix.
Table 2: Comparisons with SOTA. denotes that virtual nodes are used.
GraphSAGE
GCN
GaAN
Ours
ogba-protcins
77.68 0.20
1 0.35
78.03
0.73
0.16
GrapSAABB
GCN
GaAN 71.97 1 0.24
GCNII
JKNet
DAGNN
ogbn-arxie arxiv
71.49 1 0.27
71.74
0.29
72.74 1 0.16
72.19 1 0.21
72.09 0.25
72.32 0,27
GraphSAGE
GCN
ChusterGCN
GrapSAINN
GAT 79.45 0.59
ogba produpprodutts
78.29 1: 0.16
75.64
0.21
78.97 1: 0.33
80.27 1 0,26
81.64 0.30
GIN 75.58 + 1.40
GCN
GIN
GCN'
HIMP 78.80 0.82
oghg-molhiv
76.06 0.97
77.07
1.49
75.99 + 1.19
78.87 1.24
molg-hhl
22.66 1 0.28
20.20
0.24
27.03 1 0.23
24.24 1 0.34
27.81 1. 0.38*
opha-ppa
68.92 +: 1.00
68.39
0.84
70.37 1.07
68.57 :: 0.61
77.12 0.71
77.12 0.71
GraphAAGE
GCN
DeepWalk
cobl-collab
48.10 1 0.81
44.75
1.07
50.37 0.34
52.73 0.47
CONCLUSION
In this work, we proposed differentiable generalized message aggregation function, which de- fines a family of permutation invariant functions. We identify the choice of aggregation functions is crucial to the performance of deep GCNs. Experiments show that existence of better generalized aggregators beyond mean, max and sum. Empirically, we show the effectiveness of training our pro- posed deep GEN models, whereby we new SOTA on several datasets of the challenging Open Graph Benchmark. We believe the definition of such a generalized aggregation function provides a new view to the design of aggregation functions in GCNs.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
I. Armeni, A. Sax, A. R. Zamir, and S. Savarese. Joint 2D-3D-Semantic Data for Indoor Scene Understanding. ArXiv e-prints, February 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv.1607.06450, 2016
arXiv preprint
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, 2015.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 2018.
Xavier Bresson and Thomas Laurent. An experimental study of neural networks for variable graphs. In International Conference on Learning Representations Workshop, 2018.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.
Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. In International Conference on Machine Learning, pp. 941-949, 2018a.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling. arXiv preprint 1rXvv11801.1024,, 2018b.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. arXiv preprint Xxxv:2007.02/33 2020.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 257-266, 2019.
Gabriele Corso, Luea Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. arXiv preprint arXiv:2004.05718. 2020.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pp. 3844-3852, 2016.
Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint wrXiv:2003.00982. 2020.
M. Fey, J. G. Yuen, and Weichert. Hierarchical inter-message passing for learning on molecular graphs. In ICML Graph Representation Learning and Beyond (GRL+) Workhop, 2020.
Matthias Fey and Jan Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop Representation Learning Graphs and Munifolds, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv: (00:://0000000 2017.

---

Under review as a conference paper at ICLR 2021
Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. Large-scale learnable graph convolutional networks. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1416-1424, 2018.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Shunwang Gong, Mehdi Bahri, Michael M Bronstein, and Stefanos Zafeiriou. Geometrically prin- cipled connections in graph neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11415-11424, 2020.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855- 864, 2016.
Will Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, pp. 1024-1034 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016b.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks graph-structured data. arXiv preprint arXiv: 150.00516,, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1rXiv:1503.0255. 2015.
Weihua Hu, Matthias Fey, Marinka Zimik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.0066 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1700-4708, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167. 2015.
Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. In Advances in Neural Information Processing Systems, pp. 7090-7099, 2019.
Thomas N Kipf and Max Welling. Supi-supe classification with graph convolutional net- works. arXiv preprint arXiv: 1609.02907, 2016.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR), 2019.
Andreui Nikolaevich Kolmogorov and Guido Castelnuovo. Sur la notion de la moyenne. G. Bardi, tip. della R. Accad, dei Lincei, 1930.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial energy based learning. Predicting structured data, 1(0), 2006.
Ron Levie, Federico Monti, Xavier Bresson, and Michael M Bronstein. Cayleynets: Graph volutional neural networks with complex rational spectral filters. IEEE Transactions on Signal Processing, 67(1):97-109, 2018.
10

---

Under review as a conference paper at ICLR 2021
Guohao Li, Matthias Müller, Guocheng Qian, Itzel C. Delgadillo, Abdulellah Abualshour, Ali K. Thabet, and Bernard Ghanem. Deepgens: Making gens go as deep as cnns. CoRR, abs/19 (bss/990,00689, 2019a.
Guohao Li, Mauthias Müller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep as cnns? The IEEE International Conference on Computer Vision (ICCV), 2019b.
Q. Li, Z. Han, and X.-M. Wu. Deeper Insights into Graph Convolutional Networks for Semi- Supervised Learning. In The Thirty-Second AAAI Conference on Artificial Intelligence. AAAI, 2018.
Ruoyu Li. Sheng Wang, Feiyun Zhu, and Junzhou Huang. Adaptive graph convolutional neural networks. In Thirty-second AAAI conference on artificial intelligence, 2018.
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining- ACM, 2020.
Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. In Advances in Neural Information Processing Systems, pp. 2156-2167, 2019a.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations, 2019b.
Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. arXiv preprint (rxiv:1901.090 2019c.
Zhewei Wei Ming Chen, Bolin Ding Zengfeng Huang, and Yaliang Li. Simple and deep graph convolutional networks. 2020.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5115- 5124, 2017a.
Federico Monti, Michael Bronstein, and Xavier Bresson. Geometric matrix completion with recur- rent multi-graph neural networks. In Advances in Neural Information Prucessing Systems, pp. 3697-3707, 2017b.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-I0), pp. 807-814, 2010.
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural net- works for graphs. In International conference on machine learning, pp. 2014-2023, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, pp- 2019.
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710, 2014.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Afshin Rahimi, Trevor Cohn, and Tim Baldwin. Semi-supervised user geolocation via graph con- volutional networks. (4 2018.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Repre- sentations, 2020.
11

---

Under review as a conference paper at ICLR 2021
Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Fed- erico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:200..1119, 2020.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2008.
Lei Tang and Huan Liu. Relational learning via latent social dimensions. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 817-826. ACM, 2009.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Nikil Wale, lan A Watson, and George Karypis. Comparison of descriptor spaces for chemical com- 2008. pound retrieval and classification. Knowledge and Information Systems, 14(3):347-375, 2008.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. arXiv preprint arXiv: 07789, 2018.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (TOG), 38(5): 1-12, 2019.
Boris Weisfeiler and Andrei A Lehman. A reduction of graph to canonical form and algebra arising during this reduction. Mauccho-TTechncchssayy Informatsia, 2(9): 1968.
Felix Wu, Amauri H Souza Jr, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graph convolutional networks. In /CML, 2019.
Bingbing Xu, Huawei Shen, Qi Cao, Yunqi Qiu, and Xueqi Cheng. Graph wavelet neural network. In International Conference on Learning Representations, 2019a.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019b.
Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, pp. 40-48, 2016.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974 983. ACM, 2018
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C. 30.87. URL BRLLhttps://// dx doi /10 5244/2.342 30.7
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391-
2017.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- SAINT: Graph sampling based inductive learning method. In International Conference on Learn ing Representations, 2020.
12

---

Under review as a conference paper at ICLR 2021
Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma. Irwin King, and Yan Yeung. Gaan: Gated attention networks for learning on large and spatiotemporal graphs. Proceedings of the Thirty- Fourth Conference Uncertainty in Artificial Intelligence, pp. 339-349, 2018.
Lingxiao Zhao and Leman Akoglu. Paimorm: Tackling oversmoothing in gnns. International
Conference on Learning Representations, 2020.
Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14):1100ii19, 2017.
13

---

Under review as a conference paper at ICLR 2021
A DISCUSSION ON NETWORK DEPTH
Depth & Residual connections. Experiments in Figure show that residual connections signif- icantly improve the training dynamic of deep GCN models. PlainGCN without skip connections suffers from vanishing gradient and does not gain any improvement from increasing depth. More prominent gains can be observed in ResGCN+ compared to ResGCN as models go deeper. No- tably, ResGCN+ reaches smallest training loss with 112 layers.Thi This validates the effectiveness of pre-activation residual connections.
0.40
0.40
0.40
144
L-28
0.35
0.35
0.35
L-16
112
0.30
0.39
traking 005
0.15
0.15
0.15
0.30
560
1,00
1500
2000
0.15
500
1000
1500
2000
0.10
530
1600
1500
2000
braining coochs
traiting spodis
Saining spochs
Figure 2: Training loss of PlainGCN, ResGCN and ResGCN+
Depth & Normalization. In our experiments, we find normalization techniques play a crucial role in training deep GCNs. Without normalization, the training of deep network may suffer from vanishing gradient or exploding gradient problem. We apply normalization methods such as BatchNorm (Tolie & Szegedy 2015) or LayerNorm (Ba et al., (2016) to normalize vertex features. In addition to this, we also propose message normalization (MsgNorm) layer to normalize features on the message level, which can significantly boost the performance of networks with under-performing aggregation functions. The main idea of MsgNorm is to normalize the features of the aggregated message (!) E R.D by combining them with other features during the vertex update phase. Suppose we apply the MsrNorm to a simple vertex update function MIP((h.) + +m! m!)). The vertex update function becomes as follows:
m. (
h (++1) = 66 m (1) = MLP(hh + ||h"112
(5)
m 11
where MLP(-) is a multi perceptron and s is learnable scaling factor. The aggregated message m (1) is first normalized by its f norm and then scaled by the l, norm of (!) by a factor of s. In practice, we set the scaling factor s to be learnable scalar with an initialized value of 1. Note that when s !|m! ||2/||/b |2. the vertex update function reduces to the original form. In our experiment, we find MsgNorm boosts performance of under- perforppr aggregation functions such as mean and PowerMean on gebn-proteins more than 1%. However, we do not see any significant gain on pell-perfonming aggregation functions such as SoftMax, SoftMaxSum and PowerMeanSum. We leave this for our future investigation.
Depth & Width. In order to gain a larger representational capacity, we can either increase depth or width of networks. In this work, we focus on the depth instead of the width since it is more challeng- ing to train deeper graph neural network compared to wider one because of vanishing gradient (Li et al.) 2019b) and over-smoothing (Li et al.) 2018) problems. Deeper neural networks can learn to extract higher-Ievel features. However, given a certain budget of parameters and computation, a dell-designed wider networks be more accurate and efficient than deep networks. The trade- off of depth and width have already studied in CNNs (Zagoruyko & Komodakis (016)). We believe that it is also important to study the width of GCNs reduce the computational overhead.
Depth & Receptive Flied & Diameter. There are lots of discussion on whether depth can help for graph neural networks. In our experiments, find that graph neural networks can gain better
14

---

Under review as a conference paper at ICLR 2021
performance with proper skip connections, normalization and aggregation functions. A interesting discussion by Rossi et al. (202)) argues that the receptive field of graph neural networks with a few layers can cover the entire graph since most of graph data are 'small-world' graphs with small diameter. Depth may be harmful for graph neural networks. In our experiment, we observe a different phenomenon. For instance, ogbn-proteins dataset with relatively small diameter 9 can gain improvement with more than 100 layers. However, what is the optimal depth and for what certain kind of graphs depth help more are still mysteries.
B PROOF FOR PROPOSITION 4
Proof. Suppose we have N [v) We denote the message set as M m my }
m; E RD. We first show for any message set, SogtMax = Σ Σ exp( m, satisfies Definition
exp(8m,)
2 Let p denotes permutation on the message M. BB R, for any * M = { (mp(),,-- mp(N) ). it is obvious that Σm exp($m:) = E exp(3m; and E 2 exp(3m;) mj Σ exp(3m,) m, since the Sum function is permutation in- variant function. Thus, we have SoftMax Agg, (MM) SoftMax Ag M). SoftMax (-g satisfies Definition 2.
We now prove SoftMax ( satisfies Definition i.e. lim Agg,s() Mean(.) and to A (-) Max(.). For the k-th dimension, we have input message features
as f mm] (K) m (1:
J. Soft-a Agg (m()) (A)
m (k) }) Σド
exp(1m (t))
(A)
Eis exp(_mm) m)
Σj=1 N m N Σÿ= mm = Mean { m . .... m }). Suppose we have elements that
(R)
are equal to the maximum value m When 3 + 0o, we have:
exp(Bmg) (1))
{ 0
(k) 1/c for m
= m²
(N) for m < m
E exp(3m, (k))
(k) m, (k))) 1==1 (xm(3(m)
We obtain SoftiMx Aggs mi (k)
(2) })
my
m
m
Max(f m ....,,, m }). It is obvious that the conclusions above generalize to all the dimensions. Therefore, SoftMax A is generalized mean-max aggregation function.
(k)
C PROOF FOR PROPOSITION 5
Proof. Suppose we have N = [V()). We denote the message set as M = { (m...., my }.
m; E R.P We have PowerMean.Agg,, (M) ( E m
p * 0. Clearly, for any per-
mutation p * M { (mp(1).... m. mp(N) ). PowerMean Agg, * M) PowerMean Agg, (M).
Hence, PowerMean ( satisfies Definition 2 Then we prove TowerMean_Agee Age, (-) satis- fies Definition i.e. PowerMean Aggp=1()
Mean(.) and
Agg,
Max(.). For the k-th dimension, we have input message features as {m (A)
..... m N }.
(k)
SegrrMMeanAAggg1ll (f (k)
(k) ...., m }) 1/2 E -m, (k) Mean(1 m(() (k) (k)
mn }). Assume
we have e elements that are equal to the maximum value m.. When + 00, we have:
lim - PowerMean (1m))) (R)
(8) }) .... m
-
(m) (k) 1/)
N (m*
i=1
= (m+) po)))//// m²>0 m
(8)
We have limp- Pww.sseeeeeMMMMM...e..e. Agg m ...., m (k) conclusions above hold for all the dimensions. Thus, PowerMean Age, (.) is a generalized mean- max aggregation function.
(k)
N }) = m = Max m .... m }). The
(8)
(8)
15

---

Under review as a conference paper at ICLR 2021
D ANALYSIS OF DYRESGEN
We provide more analysis and some interesting findings of DyResGEN in this section. The experimental results of DyResGEN in this section are obtained on ogbn-proteins dataset. We visualize the learning dynamic of learnable parameters B, p and 8 of 7-layer DyResGEN aggregator and LenverMennSumAAggga.. (+) aggregator respectively. Learnable parameters and p initialized as 1 and y are initialized as 0.5. Dropout with rate of 0.1 is used for each layer to prevent over-fitting. The learning curves of learnable parameters of SoftMaxSum (BB (-) are shown in Figure 3 We observe that both 8 and y change dynamically during the training. The 3 and parameters of some layers tend to be stable after 1000 training epochs. Exceptionally, the 1-st layer learns a 3 increasingly from to 3.3 which learns a smaller y 0.1 which make SoftMaxSum (g behave more like Max aggregation at the 1-th layer. PowerMean ,g (-) aggregator also demonstrates similar phenomena learning in Figure The learned of the 1-st layer and the last layer trends to be smaller than the initial value.
with SoftMaaxSumAAg(s.p))
L-16
3.5
L-48
3.0
L-80
L-96
2.5
0.6
L/112
SoftMarSum Sum-y
0.4
1.0
0.5
250
500
1750
2000
500
1000
1750
2000
training epochs
training opochs
Figure 3: Learning curves of 12-llayer DyResGEN with SoftMaxSum (.).
0.8
L-32
64
1.6
0.7-
L-96
TowerkfeanSun Sum--p
1.4
Power SowerMeean MeannSunyy
L-112
1.2
1.0
0.8
0.6
0.4
250
500
750
1000
1250
1500
1750
2000
250
500
750
1800
1250
1500
1750
2000
training epochs
training opochs
Figure 4: Learning curves of 112 112llayer DyResGEN with PowerMeanSum Agg, (.).
E MORE DETAILS ON THE EXPERIMENTS
In this section, we provide more experimental details on the OGB datasets ogbn- arxiv, ogon-pr,,ucts, ogbg-molhiv, ogbg-molpcha, ogbg-ppa and ogbl-collab).
16

---

Under review as a conference paper at ICLR 2021
E. DETAILS OF DATASETS
Node Property Prediction. Three chosen datasets are dealing with protein-protein association networks (ogbn-proteins), paper citation networks (ogbn-arxiv) and co-purchasing network (ogbn- products). Ogbn-proteins is an undirected, weighted, and typed (according to species) graph con- taining 132, 534 nodes and 39, 561, 252 edges. All edges come with 8-dimensional features and each node has an 8-dimensional one-hot feature indicating which species the corresponding protein comes from. Ogbn-arxiv consists of 169, 343 nodes and 166, 243 243 directed edges. Each node is an arxiv paper represented by a 128-dimensional features and each directed edge indicates the ci- tation direction. As an Amazon products co-purchasing network, ogbn-products is an undirected and unweighted graph which is formed by 2, 449, 029 nodes and 61, 859, 140 edges where nodes are products sold in Amazon that are represented by 100-dimensional features, and edges indicate that the connected nodes are co-purchased. For ogbn-proteins, the prediction task is multi-label and ROC-AUC used as the evaluation metric. For ogbn-arxiv and oggn-products, their prediction tasks are both multi-class and evaluated by accuracy.
Graph Property Prediction. Here, we consider three datasets, two of which deals with molecular graphs (ogbg-molhiv and ogbg-molpcba) and the other is biological subgraphs (ogbg-ppa). Ogbg- molhiv has 41, 127 subgraphs and ogbg-molpeba much bigger which contains 437, 929 subgraphs. For ogbg it consists of 158, 100 subgraphs and each subgraph is much denser in comparison to molhiv and ogbg- molppha are both binary classification while the prediction task of ogbg-ppat ppa is multi-class classification. The former two evaluated by the ROC-AUC and Average Precision (AP) metric separately. Accuracy is used to assess ogbg-ppo ppa.
the other two datasets. The tasks of ogbg-molhiv
Link Property Prediction. We select ogbl-collab, an author collaboration network consisting of 235, 868 nodes and 1, 285, 465 edges for link prediction task. Each node in the graph comes with a 128-dimensional feature vector representing an author and edges indicate the collaboration between authors. The task is to predict the future author collaboration relationships given the past collab- orations. Each true collaboration is ranked among a set of 100, 000 randomly-sann sammlld negative collaborations, and the ratio of positive edges that are ranked at K K-place or above (Hits@k, k, k is 50 here) is counted as the evaluation metric.
E.2 DETAILS OF RESULTS AND IMPLEMENTATION
For fair comparison with SOTA methods, we provide results on each dataset by averaging the re- sults from 10 independent runs. We provide the details of the model configuration on each dataset. All models are implemented based on PyTorch Geometric (Fey & Lenssen 2019) and all experi- ments are performed on single NVIDIA V100 32GB.
ogbn-proteins. For both proteins and ogbg-ppa, there is no node feature provided. We ini- tialize the features of nodes through aggregating the features of their connected edges by Sum aggregation, i.e. xj = DjEN(() ei where X₁ denotes the initialized node features and ei denotes the input edge features. We train a 112-layer DyResGEN with SoftMaxAAggsl aggregator. A hidden channel size of 64 is used, A layer normalization and dropout with rate of 0.1 are used for each layer. We train the model for 2000 epochs with an Adam optimizer with a learning rate of 0.001.
ogbn-arxiv. We train a 28-layer ResGEN model with SoftMax (-g aggregator where / is fixed as 0.1. We convert this directed graph into undirected and add self-loop. Full batch training and test are applied. A batch normalization is used for each layer. The hidden channel size is 128. We apply a dropout with a rate of 0.5 for each layer. An Adam optimizer with learning rate of 0.001 used to train the model for 2000 epochs.
ogbn-products. A 14-layer ResGEN model with SoftMax (gg aggregator where fixed as 0.1 is trained for ogbn-products with self-loop added. We apply mini-batch training scenario by randomly partitioning the graph into 10 subgraphs and do full-batch test. For each layer, a batch normalization is used. The hidden channel size is 128. We apply dropout with rate of 0.5 for each layer. An Adam optimizer with a learning rate of 0.001 is used to train the model for 1000 epochs.
17

---

Under review as a conference paper at ICLR 2021
ogbg-molhiv. We train 7-layer DyResGEN model with SoftMax aggregator where / is learnall.. batch normalization used for each layer. We set the hidden channel size as 256. A dropout with rate of 0.2 is used for each layer. An Adam optimizer with learning rate of 0.0001 are used to train the model for 300 epochs.
ogbg-molpcha. A 14-layer ResGEN model with SoftMax (-g aggregator where is fixed as 0.1 is trained. In addition, the original model performs message passing over augmented graphs with virtual nodes added. A batch normalization used for each layer. We the hidden channel size as 256. A dropout with a rate of 0.5 is used for each layer. An Adam optimizer with a learning rate 0.01 are used train the model for 300 epochs.
ogbg-ppa. As mentioned, we initialize the node features via a Sum aggregation. We train a 28- layer ResGEN model with SoftMax A- aggregator where fixed as 0.01. We apply layer normalization for each layer. The hidden channel size is set as 128. A dropout with rate of 0.5 is used for each layer. We use an Adam optimizer with learning rate of 0.01 to train the model for 200 epochs.
ogbl-collab. The whole model used to train on link prediction task consists two parts: a 7- layer DyResGEN model with SoftMax_Agga() aggregator where 3 is learnable and a 3-layer link predictor model. A batch normalization used each layer in DyResGEN model. We set the hidden channel size 128. An Adam optimizer with learning rate of 0.001 used train the model for 400 epochs.
F MORE FUTURE WORKS
We believe generalized aggregation functions will open a new view for designing aggregation func- tions in graph neural networks. Here discuss some more potential directions as follows:
Can we learn the parameters of generalized aggregation functions with mete-learning method such MAML (Finn al., 2017)?
What is the expressive power border of generalized mean-max-sum aggregation functions with respect to WeisfeilerLehman graph isomorphism test (Xu et al. 2019h)?
Can we design Principal Neighbourhood Aggregation (PNA) (Corso et al. 2020 by com-
bining multiple learnable aggregators from generalized aggregation functions?
18


Reviews:
[{"review_id": "p5KaWWRWQkI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose new and, in particular, parameterized aggregation functions for GNNs in order to especially support the construction of deeper GNNs. The paper is fairly understandable and the \"deeper GNN\" topic has gained more attention recently. However, in my opinion, the paper is missing the theoretical justification of its proposals and I have concerns about the results (details below). Hence, I do not vote for acceptance.\n\n(+) The dynamic aggregation considered in Section 5.2 seems really interesting to me.\n\n(-) Approach: The authors define several parameterized aggregation functions but the motivation is left unclear. Indeed, it seems that the main motivation for splitting the definitions in this way seems to be to generalize as many of the existing functions as possible. The paper is missing an explanation why these different aggregation functions are supposed to specifically support deeper GNNs.\n\n(-) Experiments: \n* The main table contains very many similar results but is missing standard deviation. This is especially strange given that the OGB code supports several folds.\n* Table 2 is supposed to show the comparison with SOTA, but I claim that it is lacking. The initial OGB leaderboard contains only the most basic GNNs. However, when proposing an approach for creating deeper GNNs, the paper has to compare to this kind of models too. The row in which the authors compare to GCNII, JKNet, DAGNN, etc. actually seems to show that these models are as good/better. \n----------------------------------------------\n\nSmaller Comments:\n- p.1 \"The power of deep models become more evident with the introduction of more challenging and large-scale graph datasets\" this claim should be supported by references. Why are they especially helpful for this kind of data?\n- Table 1. The gray color is hardly readable and does not really allow for a \"convenient comparison\". I suggest to use bold face or anything else.\n\n----------------------------------------------\n\nUpdate after Rebuttal:\nI have read the authors' response but do no change my scores.\n\nThe experimental results alone do not offer a theoretical justification. The paper does not have to contain the latter but then the evaluation would have to be exhaustive.\n\nI see that the authors have done a huge number of experiments. However, if basic standards like standard deviation are neglected in order to run just more experiments, the results of the entire evaluation remain questionable. Similarly, related approaches have to be considered adequately to have an appropriate comparison to SOTA. The authors have not made them available yet.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "jkmGLpOvLP2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "—Summary:\n\t\n\tThe authors propose a generalized neighborhood message aggregation function for GNNs. The proposed choice of generalized aggregation functions is SoftMax and PowerMean, which generalizes Max and Mean functions and interpolates them. Additionally, they propose a variant of these two methods, which can also encompass the Sum function. By making the components of these generalized aggregator functions differentiable, the GNNs can choose an approximate instantiation of the aggregation function that best optimizes the task. \n——\n\nPros:\n\n\tA straightforward and elegant solution \n\tThe Paper is well written and is easy to follow. \n\n——\nConcerns:\n\n\t(i) It is not clear how generalized aggregation functions to aid training deeper GCNs \n\t\tAlso, from the experiments, the baseline, ResGCN model’s performance also increases with layers and is falling short of the proposed model only by a small margin. \n\n\t(ii) Missing model wise results on OGB benchmark (Table 2)\n\t\tA detailed study is done only on one dataset, and in all other datasets, the best of the models alone is reported.\n\n\t\tFor all the datasets, results for the following compared models from Table1 is required to understand the achieved performance improvement. Since the OGB benchmark is new and the reported GNN models on the OGB leaderboard were only run for 3 layers, it is crucial to analyze all the variants discussed here in detail to appreciate the performance gain achieved by the proposed generalized aggregation function. \n \t\t- baselines: ResGCN and ResGCN+ with mean, max, sum, \n \t\t- Fixed simple variant: Softmax, PowerMean, Softmax-sum, PowerMean-sum\n\t\t- Learnt variants: Softmax-sum and PowerMean-sum\n\n\t(iii) Benefits of the proposed model is inconclusive:\n\t\tComparison with ResGCN+ is reported only for two datasets, and out of which, one dataset, the arXiv dataset, has results reported only for one of the aggregation functions. From the other protein dataset, the avg score for Max-ResGCN+ is 85.09, Learned PowerMean, and SoftMax based models scores are 84.56 and 85.22. There is both a drop and gain in performance with the adoption of the generalized aggregation functions. Hence, it is not clear whether both the generalized functions aid in improving performance without seeing the results on other datasets for all the four or two versions of the learned generalized aggregation functions (as in Table 1.d) \n\n\t(iv) Need a statistical significance test report. \n\tThe results are very similar among models and are not clear whether there is any significant difference in choosing one over the other. \n\n——\nQuestions during rebuttal:\n\n\tKindly address the concerns raised above.\n--- Post rebuttal:\nI've read the author's response and there is no change in my scores. \n\n- The given argument regarding better-generalized aggregation function aid in training deep GCNs is not clear and convincing. \n- I agree that doing a detailed ablation study on a large dataset is expensive. In which case experiments on either synthetic or other smaller real-world datasets would be helpful.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1D_4HWBdQIu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how to train deeper graph convolutional networks by using different aggregation function. Convolutional networks typically show better performance when getting deeper. However, this is not true in graph convolutional networks -- deeper graph neural networks usually show degenerate performance. So designing a deeper model for graph is both necessary and valuable.  This paper proposes a general aggregation function which summarizes sum, max, and softmax operations etc. I list the pros and cons as following. \n\nPros:\n1. This paper studies an important problem for graph convolutional networks. Also, this paper proposes an interesting perspective to study this problem.\n2. This paper conducts extensive experiments and analysis of different aggregation functions and their combinations. \n3. The paper is well written. \n\nCons:\n1. The greatest weakness of this paper is its novelty. Although studying aggregation function is interesting, this paper doesn't propose a new aggregation function beyond analyzing the combinations of existing ones. It would be great if authors can state the novelty of this paper compared to DeepGCNs. Also, what makes difference between the proposed aggregation function and softmax. \n2. The proposed general aggregation function doesn't show significant improvement over softmax with learned temperature. Actually, the proposed powermean and softmaxsum are even worse than max. \n3. The comparison to other methods (GraphSAGE and GCN) is not fair because they are shallow models while the model used in this paper is quite deep. I'd like to see more fair comparisons with the same model size. At least the authors should present the number of parameters and the size of their model. \n4. This paper is based on DeepGCNs. DeepGCNs conducts experiments on point cloud recognition tasks and compare to DGCNN etc. I'd like to know why this paper doesn't consider point cloud recognition tasks as DeepGCNs tested on. Is it possible to compare to DGCNN/DeepGCNs on S3DIS?", "extra_scores": {}, "extra_text": {}}, {"review_id": "-jn_4545phk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#####Summary#####\n\nThis work proposes a generalized aggregation function for graph neural networks. This generalized aggregation function can cover commonly used aggregation functions (i.e., mean, max, and sum) by particular setting of hyperparameters. Also, these hyperparameters can be learned with model in an end-to-end fashion instead of being predefined manually. The experimental results on OGB is impressive and can demonstrate the effectiveness of the proposed approaches. \n\n#####Pros#####\n\n(1) The proposed generalized aggregation functions are intuitively and empirically effective.\n\n(2) The included experiments are comprehensive and can demonstrate the effectiveness of the proposed method.\n\n(3) This paper is well organized and easy to follow.\n\n#####Cons#####\n\n(1) Although the provided experiments are comprehensive, some key experiments are missed. A. The proposed aggregation functions seem need more computations than simple mean/max/sum. A comparison of efficiency between proposed aggregation functions and existing simple functions should be included. B. The effectiveness of the whole model has been shown strongly. However, an ablation study to show the respective improvement brought by the increasing depth and proposed aggregation should be considered.\n\n(2) It would be better to provide some theoretical support if possible.\n\n\n\n#####Update######\n\nThank the authors for the response. I keep my score as 6.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: One referee supports acceptance, whereas three referees lean towards rejection. All referees agree that the idea introduced in the paper is interesting but find that the motivation and evaluation of the proposed aggregation functions could be significantly strengthened. The rebuttal addresses R1's concerns about novelty and unfair comparisons, R2's concerns about computational efficiency of the methods, R3's concerns about motivation of the proposed approach and some missing baselines, and R4's concerns about motivation. However, the rebuttal does not address the reviewers' concerns related to improvements achieved by the proposed approach, statistical significance nor appropriate comparison with SOTA. I agree with the reviewers that the paper tries to address a relevant problem and proposes interesting ideas, which are worth exploring. However, after discussion, the referees agree that further work should be devoted to strengthen the contribution. I agree with their assessment and hence must reject. In particular, I would strongly recommend to follow their suggestions to either provide strong theoretical motivation to support the claims of the paper or work on a strengthened empirical evaluation, following OGB guidelines to report the std of the results and including a proper comparison with the state of the art. 
Decision: Reject