Generalised f-Mean Aggregation for Graph Neural Networks
Authors: Ryan Kortvelesy
Steven Morad
Amanda Prorok
Venue: NeurIPS 2023 poster (2023)
Keywords: Aggregation
Graph Neural Networks
TL;DR: Improving the performance of GNNs with an aggregation method based on the generalised f-mean.

Abstract: Graph Neural Network (GNN) architectures are defined by their implementations of update and aggregation modules. While many works focus on new ways to parametrise the update modules, the aggregation modules receive comparatively little attention. Because it is difficult to parametrise aggregation functions, currently most methods select a ``standard aggregator'' such as mean, sum, or max. While this selection is often made without any reasoning, it has been shown that the choice in aggregator has a significant impact on performance, and the best choice in aggregator is problem-dependent. Since aggregation is a lossy operation, it is crucial to select the most appropriate aggregator in order to minimise information loss. In this paper, we present GenAgg, a generalised aggregation operator, which parametrises a function space that includes all standard aggregators. In our experiments, we show that GenAgg is able to represent the standard aggregators with much higher accuracy than baseline methods. We also show that using GenAgg as a drop-in replacement for an existing aggregator in a GNN often leads to a significant boost in performance across various tasks.

Generalised f-Mean Aggregation for Graph Neural Networks
Ryan Kortvelesy, Steven Morad and Amanda Prorok University of Cambridge {Ik627, sm2558, asp45) @ @cam.ac.uk
Abstract
Graph Neural Network (GNN) architectures defined by their implementations of update and aggregation modules. While many works focus on new ways to parametrise the update modules, the aggregation modules receive comparatively little attention. Because it is difficult to parametrise aggregation functions, cur- rently most methods select "standard aggregator" such as mcan, sum, or max. While this selection is often made without any reasoning, it has been shown that the choice in aggregator has significant impact on performance, and the best choice in aggregator is problem- dependent. Since aggregation is a lossy opera- tion, it is crucial to select the most appropriate aggregator in order minimise information loss. In this paper, we present GenAgg, a generalised aggregation operator, which parametrises a function space that includes all standard aggrega- tors. In our experiments, aggregators with much higher accuracy than baseline methods. We also show that using GenAgg drop-in replacement for existing aggregator in a GNN often leads to significant boost in performance across various tasks.
show that GenAgg able to represent the standard
1 Introduction
Graph Neural Networks (GNNs) provide a powerful framework for operating over structured data. Tak- ing advantage of relational inductive biases, they local filters to learn functions that generalise over high-dimensional data. Given different graph structures, GNNs can represent many special cases, including CNNs (on grid graphs) [12], RNNs (on line graphs) [4], and Transformers (on fully nected graphs) [19]. All of these architectures can be subsumed under the Graph Networks frame- work, parametrised by update and aggregation mod- ules [2]. Although the framework itself is general, the representational capacity is often constrained in practice through design choices, which create hu- man prior [21]. There are two primary reasons for introducing this human prior. First, there are no stan- dard methods to parametrise all of the modules- MLPs can be used as universal approximators the update modules, but it is nontrivial to parametrise the function space of aggregators. Consequently, most GNNs simply make design choice for the aggre- gation functions, selecting mean, sum, or max [21].
(a) (a(a)), 0,-1)
(b) (e" 0.00
(c) (log(I=))00.
0 (tanb((r) O,
(0) (e
6 (log( (ac ),
Figure A qualitative demonstration of the di- versity functions that can be represented by GenAgg. In these visualisations, GenAgg is plot- ted as a function of inputs ao and x for different parametrisations @ (f. (see Equation (1)).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Second, constraints can boost performance in GNNs, either through invariances or a regularisation effect.
In this paper, we focus on the problem of parametrising the space of aggregation functions. The ulti- mate goal is to create aggregation function which can represent the set of all desired aggregators while remaining as constrained as possible. In prior work, one approach is to introduce Iearnable parameters into functions that could parametrise min, mean, and max, such the Powermean and a variant of Softmax [8, 13, 20J. However, these approaches can only parametrise a small set of aggregators, and they can introduce instability in the training process (see Section 4). On the oppo- site end of the spectrum, methods like Deep Sets [22] and LSTMAgg [9] are capable of universal approximation over set functions, but they are extremely complex, which leads to poor sample effi- ciency. These methods scale in complexity (i.e. number of parameters) with the dimensionality of the input, and lack some of the useful constraints that are shared among standard aggregators (see Section 4). Consequently, the complexity of these methods counteracts the benefits of simple GNN architectures.
Although existing approaches present some limitations, the theoretical advantages of a learnable aggregation module are evident. has been shown that the choice of aggregation function not only has a significant impact on performance, but also is problem-specific [21]. Since there is no aggregator that can discriminate between all inputs [5], important to select an aggregator that preserves the relevant information. In this paper, we present method that parametrises the function space, allowing GNNs learn the most appropriate aggregator for each application.
Contributions
We introduce Generalised Aggregation (GenAgg). the first aggregation method based on the generalised f-mean. GenAgg learnable permutation-invaricc aggregator which is provably capable (both theoretically and experimentally) of representing all "standard aggregators" (see Appendix C for proofs). The representations learned by GenAgg are explainable- each learnable parameter has interpretable meaning (Section 6).
Our experiments provide several insights about the role of aggregation funetions in the performance of GNNs. In our regression experiments, we demonstrate that GNNs struggle to "make up for" the lack of representational complexity in their constituent aggregators, even when using state-of-the-aat parametrised aggregators. This finding is validated in our GNN benchmark experiments, where show that a GenAgg-based GNN outperforms all of the baselines, including standard aggregators and other state-of-the parametrised aggregators.
Finally, we show that GenAgg satisfies generalisation of the distributive property. We de- rive the solution for a binary operator that satisfies this property for given parametrisations of GenAgg. The generalised distributive property can be leveraged in algorithms using GenAgg to improve space and nemory-efficiencyy
2 Problem Statement
Consider a multiset X (Ii, 22, 22. .n of cardinality |2| n, where E Rd. We de- fine an aggregation function as a symmetric function O Raxd + glxd The aggregator must be independent over the feature dimension, so without loss of generality can be rep- resented over a single dimension O R" + R. A set of standard aggregators is defined A = {mean, sum, product, min, max, ...).... }} (for the full list see Table 1). Our task is to create an aggregator Do RRa + IR' parametrised by 0 which can represent all standard aggregators: YO, A 39 D = O
3 Method
In this section we introduce GenAgg, a parametrised aggregation function which based on the generalised f-mean [11]. In our formulation, we introduce additional parameters to increase the representational capacity the f-mean, producing the augmented f f-mean (AFM). Then, as the im- plementation is non-trivial, we propose method to implement it. The novel aspect of GenAgg is not

---

Aggregation Function
8
f
GenAgg
SoftmuxAgg
PowerAgg
1
Σ
f(3) =
sum: Σ
1
0
f(x)
/
x
x
product:
[
[ad
1
0
[x) = log((2]])
/
X
X
min (magnitude): min lny
0
0
[(c) 1omp- kr|-
/
X
V
max (magnitude); max [a:]
0
0
f(z) = limp-x P
/
X
/
min: min
0
0
f(a) limp-+o0 e p
/
/
X
max: max
0
0
f(x) limp-aoe CP
/
X
harmonic mean:
0
0
f(x) =
1
x
Σ
geometric mean:
/ II
0
0
f(x) - -ogg(<=])
1
X
root mean square:
Σd
x
0
0
f(x) = x
1
X
euclidean norm:
Σ
1
0
[(x) = x2
X
x
standard deviation:
(21-- 1
0
1
[(x) = x2?
/
X X
log-sum-exp: log
0
f(x) =
x
x
Table A table of all the most common aggregators. For each special case, we specify the values of a, 33 and f for which the augmented f-mean is equivalent (see Appendix C). We also report whether or not SoftmaxAgg and PowermeanAgg can represent each aggregator.
only the augmented f-mean formula, but also the implementation, which allows the mathematical concept of generalised mean be utilised in machine learning context.
3.1 Generalised f-Mean
The generalised f-mean [11] is given by: 1- fa,)). While difficult to define aggre- gation functions, the generalised f-mean provides powerful intuition: most aggregators can be represented by single invertible scalar-valued function f RJ RR. This is a useful insight, because it allows comparisons to be drawn between aggregators by analysing their underlying func- tions f. Furthermore, it provides a framework for discovering new aggregation functions. While classic functions like ex log(x), x! all map aggregators in A, new aggregators can be created by defining new functions f.
3.2 Augmented f-Mean
The standard generalised f-mean imposes strict constraints, such as symmetry (permutation- invariance), idempotency (#((z..... 0). However, this definition is too restrictive to parametrise many special cases of aggregation func- tions. For example, sum violates idempotency (Ese[L.n) 24 = nz), and standard deviation violates
x), and monotonicity (Vi E [1..n), ( > 8a,
02, == < 0). Consequently, we deem these constraints to be counterpro- ductive. our method, we introduce learnable parameters and 3 to impose relaxation on the idetempotency and monotonicity constraints, while maintaining symmetry. We call this relaxed
monotonicity (0r((2)1))

---

formulation the augmented f-mean (AFM), given by:
2; = f-1
n Σ
f(z; 3p)
(I)
sE[1..]
rE(I..m]
The a parameter allows AFM to control its level of dependence on the cardinality of the input then AFM represents the geometric mean: VII M.l However,
X. For example, given (() log([=]), if
4 (f.a.8) = + (1og(1=[),000)
= 1, then the n-th root disappears, and AFM represents a product: (1)00))) = [x/--
The parameter enables AFM to calculate centralised moments, which are quantitative measures of the distribution of the input X [17]. The first raw moment of X' is the mean μ - Σ -.,,, xi and the k-tth central moment is given by μæ -E (x; -1)². With the addition of B, it becomes possible for AFM to represent 4/E,, the k-th root of the k-th central moment. For k: = 2, this quantity is the standard deviation, which in our set of standard aggregators A. If the output is scaled to the k-th power, then it can also represent metrics such as variance, unnormalised skewness, and unnormalised kurtosis. It is clear that these metries about the distribution of data are useful-thee can have real- world meaning (e.g., moments of inertia), and they have been used as aggregators in GNNs in prior work [5]. Consequently, & provides AFM with an important extra dimension of representational complexity. In addition to representing the centralised moments when 3 1 and f(x) x", allows any aggregator be calculated in a centralised fashion. While the centralised moments are the only well-known aggregators that arise from nonzero 3, there are several aggregators with qualitatively unique behaviour that can only be represented with nonzero 3 (see Fig. 1).
With this parametrisation, AFM can represent any standard aggregator in A (Table 1). Furthermore, by selecting new parametrisations @ (f, a, 3), is possible compose new aggregators (Fig. 1).
3.3 Implementation
In Equation (1), the manner in which f-1 implemented is an important design choice, One option is to learn the coefficients for an analytical function (e.g. truncated Taylor Series) and analytically invert it. However, it can be difficult to compute the analytical inverse function, and without carefully selected constraints, there is no guarantee that f will be invertible.
Another possible option is an invertible neural network (e.g. a parametrised invertible mapping from normalising flows [10]). We have tested the invertible networks from normalising flows literature as implementations for f. While they work well on smaller tasks, these methods present speed and memory issues in larger datasets.
In practice, find that the most effective approach is to use two separate MLPs for f and f-1, We enforce the constraint x = (fx)) by minimizing the following optimisation objective:
Cinv (01, 02 02)= = E [(1 (fo. (x)) (x))
- -121)
(2)
The absolute value operations apply relaxation the constraint, allowing (f) to reconstruct either or [z]. This is useful because several of the ground truth functions from Table include an absolute value, making them non-invertible. With this relaxation, becomes possible represent those cases. This optimisation objective ensures that ff is both monotonic and invertible over the domains IR + and F independently. In our implementation, this extra optimisation objective is hidden behind the GenAgg interface and gets applied automatically with a forward hook, so it is not necessary for the user to apply an extra loss term.
While using a scalar-valued f : RR. + R. is the most human-interpretable formulation, it is not necessary, A valid implementation of GenAgg can also be achieved with R² + gd and f-1; Rd + 11_ In our experiments, we found that mapping to a higher intermediate dimension can sometimes improve performance over scalar-valued f (see training details in Appendix E).
3.4 Generalised Distributive Property
Given that GenAgg presents a method of parameterising the function space of aggregators, can also be used as a tool for mathematical analysis. To demonstrate this, we use the aug-

---

f(b)) is an Aggregation Function
Distributive Operations v(a, b2
mean:
sum:
Σ
product:
|o|
min (magnitude): min lad]
mii[[al], [8])
max (magnitude)); max [as]]
mnax[la], [B])
min: min
min(a, 6)
max: maxca,
max
harmonic mean:
asb mes. a
geometric mean:
Ja [6), [a|ios (6]
mean square:
Σ
o4
va++
[a] [b]
euclidean norm:
Σ
[4] [4]-|b]
standard deviation:
(ar 10)
[a] [M
log-sum-exp: log
Σ
a
Table 2: A table of the distributive operations 1/ that satisfy each aggregation function, computed using Equation 3. All aggregation functions have at least one solution, and some special cases have multiple solutions.
mented f-mean to analyse a generalised form of the distributive property, which is satisfied di (c. 2z,xx Fz) Onex (0c, 2.) for binary operator & and aggregator O. For given ag- gregation function parametrised by f (assuming is 0), we derive a closed-form solution for a corresponding binary operator which will satisfy the generalised distributive property (for further explanation and proofs, see Appendix A).
Theorem 3.1. For GenAgg parametrised by 0 = ax 3) (J,a, 0), the binary operator ab which will satisfy the Generalised Distributive Property for Dg is given by:
(aa b) = (f(((( f(b))
(3)
Furthermore, for the special case 0 af, B) (f. 0, 0). there additional solution.
For example, for the euclidean norm where f(z = x² and 1, the binary operator is (a b) (a² b euclidean norm. This is a useful finding, as the distributive property can used to improve algorithmic time and space complexity (e.g. the FFT) [1]. With our derivation of z as a function of f, it is possible to implement similar efficient algorithms with GenAgg.
b, which implies that a constant multiplicative term can be moved outside of the
4 Related Work
Several existing works propose methods to parametrise the space of aggregation functions. These methods can broadly be divided into two categories. Mathematical approaches derive explicit equation in terms of the inputs and one or more learnable parameters. Usually, these approaches represent smooth interpolation through function space from min, through mean, to max. Alter- natively, Deep Learning approaches seek to use the universal approximation properties of neural networks maximise representational complexity.
4.1 Mathematical Approaches
SoftmaxAgg SoftmaxAgg computes the weighted sum of the set, where the weighting is derived from the softmax over the elements with some learnable temperature term [13, 20). This formu-

---

lation allows SoftmaxAgg to represent mean, min, and max (see Table 1). Unfortunately, it fails to generalise across the majority of the standard aggregators.
PowerAgg Based on the p-norm, PowerAgg is a special case of GenAgg where a = 0, B = O. and f(x) x². There some methods which use the powermean directly [13, 20, 8), and others which build on top of it [18]. Theoretically, PowerAgg can represent significant subset of the stan- dard aggregators: min magnitude, max magnitude, mean, root mean square, harmonic mean, and geometric mean (although the geometric mean requires limp-0. so it is not practically realisable) (see Table 1). Unfortunately, there is a caveat to this approach: for negative inputs 2; < 0 and non-integer values p. it is only defined in the complex domain. Furthermore, for negative inputs, the gradient Ou"
Sp with respect to trainable parameter is complex and oscillatory (and therefore is prone to getting stuck in local optima). In order to fix this problem, the inputs must be constrained to be positive. In prior work, this has been achieved by clamping x max(x;, 0) [13]. subtracting the minimum element z = 2, min(2)) [20]. or taking the absolute value x |zy| [8]. However, this removes important information, making impossible to reconstruct most standard aggregators.
4.2
Deep Learning Approaches
PNA While Principle Neighbourhood Aggregation [5] is introduced as a GNN architecture, its nov- city stems from its method of aggregation. In PNA. input signal is processed by a set of aggregation functions, which is produced by the cartesian product standard aggregators {mean, min, max, std} and scaling factors { 1, nj. The output of every aggregator is concatenated, and passed through a dense network. While this increases the representational complexity of the aggregator, it also scales the dimensionality of the input by the number of aggregators multiplied by the number of scaling factors, which can decrease sample efficiency (Figure 3). Furthermore, the representational complexity of the method is limited by the choice of standard aggregators it cannot be used to represent many of the special cases of parametrised general aggregators.
LSTMAgg In LSTMAgg, the input set is treated as sequence (applying some random permuta- tion), and is encoded with a recurrent neural network [9]. While this method is theoretically capable of universal approximation, in practice its non-permutation-ine -inruuttttooonnvvaianceee invariance can cause its performance to suffer (as the factorial complexity of possible orderings leads to sample-inefficiency). SortAgg addresses this issue by sorting the inputs with computed features, and passing the first k sorted inputs through convolutional and dense networks [23]. While this method solves the issue of non- permutation -nneriinnnniaiinneee... it loses the capability of universal approximation by truncating to k inputs. While universal approximation is not a requirement for an effective aggregation function, we note that it cannot represent many of the standard aggregators.
Deep Sets Deep Sets a universal set function approximator [22]. However, because it operates over the feature dimension in addition to the "set" dimension, is not regarded as an aggregation function. Instead, it usually serves as a full GNN layer or graph pooling architecture [14, 15]. One may note that the formulation for Deep Sets ((ee( fz;)) bears some resemblance to our method.
However, there are two important differences. First, our method adds the constraint Φ = f- limiting possible parametrisations to the subspace where all of the standard aggregators lic. Sccond, while the learnable functions & and f in Deep Sets are fully connected over the feature dimension, the f and f-1 modules in our architecture are scller-valued functions which are applied element- wise. To summarise, Deep Sets is useful as a set function approximator, but it lacks constraints that would make it viable an aggregation function.
5 Experiments
In this paper, we run three experiments. First, we show that GenAgg can perform regression to re- cover any standard aggregation function. Then, we evaluate GenAgg and several baselines inside of a GNN. The resulting GNN architectures are given the same task of regressing upon graph-structured data generated with a standard aggregator. This tests if it is possible for a GNN with given ag- gregator to represent data which was generated by different underlying aggregators. Finally, we provide practical results by running experiments on public GNN benchmark datasets: CLUSTER, PATTERN, CIFAR1O, and MNIST [6].

---

Aggregation
GenAgg
P-Agg
S-Age
mean
Aggregation
GenAgg
P-Age
S-Age
mcan
mean
1.000
0.817
1,000
1,000
mean
0.977
0.999
1.000
0.972
1,000
0.761
0.887
0.888
0.971
0.906
0.887
0.903
product (mag)
0.985
0.407
0.172
0.022
product (mag)
0.966
0.644
0.726
0.434
min (mag)
0.962
0.027
min (mag)
0.952
0.876
0.810
0.731
max (mag)
0.990
0.586
0.024
max (mag)
0.734
0.784
0.747
min
0.995
0.734
1,000
0.805
min
0.995
0.986
0.999
0.806
max
0.990
0.920
1.000
0.805
max
0.989
0.976
0.999
0.845
harm. mean (abs)
0.986
0.453
0.088
0.027
harm. mean (abs)
0.931
0.797
0.842
0.697
geom, mean (abs)
0.994
0.481
0.152
0.031
geom. mean (abs)
0.963
0.629
0.836
0.626
rool mean square
0.996
0.532
0.308
0.028
rook mean square
0.975
0.775
0.808
0.899
euclidean norm
0.964
0.585
0.464
0.019
euclidean norm
0.985
0.742
0.680
0.756
stundard dev.
0.999
0.442
0.558
0.013
standard dev.
0.966
0.739
0.805
0.624
log sxppupe-
0.999
0.823
0.947
0.747
log-sum-exy
0.983
0.919
0.952
0.841
(a) Aggregator Regression.
(b) Regression
Figure 2: Results for the Aggregator Regression and GNN Regression experiments, indicating the ability of GenAgg, PowerAgg (P-Agg). Softmax (S-Agg), and mean parametrise each stan- dard aggregator in A. We report the correlation coefficient between the ground truth and predicted outputs. The nighest-performing methods (and those within 0.01 correlation) are shown in bold.
For all baselines, we use the implementations provided in PyTorch Geometric [7]. The only ex- ception is PNA, which is GNN architecture by nature, not aggregation method. For our periments, we adapt PNA into an aggregation method, staying as true to the original formulation as possible: PNA(X) = f([1, n, 11 & [mean(X)), std( min(X),, max(X))), where f is linear layer mapping from IR 12d back to Rd.
x
For more training details, see Appendix E. Our code can be found at: https://github. com/ Acciorocketshipp hwtpokkttshissgggeeraeeeeeed gegneraissddggregatiii.....
5.1 Aggregator Regression
In this experiment, we generate a random graph E) with | 8 nodes and an edge density of 181
TDP = 0.3. For each node E V, we draw an internal state x, E Rd from a normal distribution I; .N(0,, Ia) with d 6. Then, generate training data with
of ground truth aggregators On E A (where & is an index). For each aggregator Os- the dataset X,, Y, is produced with a Graph Network [2], using O₁ as the node aggregation module. The inputs are defined by the set of neighbourhoods in the graph Xx = {2; E []..[V]]} where the neighbourhood X; is defined X, = {x, j N, with N, [ (i.j) E}. The corresponding ground truth outputs defined as Y2 = {us i [I.. where y = (X))
The model that use for regression takes the same form the model used to generate the data, except that the standard aggregator used to generate the training data O₂ is replaced with parametrised aggregator Dg:
0
(4)
X
In our experiments, each type of parametrised aggregator (GenAgg, SoftmaxAgg, PowerAgg, and mean as baseline) is trained separately on each dataset Xx. Ys.
Results. We report the MSE loss and correlation coefficient with respect to the ground truth in Table 2a. GenAgg is able to represent all of the standard aggregators with a correlation at least 0.96, and most aggregators with correlation of greater than 0.99. The only where the performance of GenAgg is surpassed by a baseline are min and max, where SoftmaxAgg exhibits marginally higher accuracy.
One interesting observation is that even if the baselines can represent an aggregator in theory, they cannot necessarily do so in practice. For example, PowerAgg can theoretically represent the geomet-

---

CLUSTER Test Accuracy
PATTERN Accuracy
00
0.52
05
0.66
02
032
Styp
CIFARIO Test Accuracy
MNIST Test Accuracy
0.5
0.4
6.1
0.2
x027
Dataset
GenAgg
P-Agg
S-Age
PNA
mean
SUIn
CIFARIO
0,540
0.451
0.467
0.473
0,502
0.435
0.471
#0.09
#0.14
#0.13
#0.14
20.13
#0.15
#0.13
MNIST
0.928
0.877
0.864
0.717
0.847
0.844
0.831
1:0.07
10.07
1:0.66
+0.19
1.0.8
1:0.88
1:0.8
CLUSTER
0.627
0.610
0.168
0.602
0.170
0.501
::0.02
+0.01
+0.01
::0.02
#:0.1
PATTERN
0.925
0.883
0.861
0.871
0.860
0.860
++0.00
+0001
+-0.03
+.001
+:0.1
+-0.01
+:001
Figure 3: Test accuracy for GNNs with various aggregators on GNN benchmark datasets. this experiment, each trial uses the same base GNN architecture (4-layer GraphConv), and the default aggregator is replaced with either GenAgg, PowermeanAgg (P-Agg), SoftmaxAgg (S-Agg), PNA, mean, sum, max. The plots depict the mean and standard deviation of the test accuracy over 10 trials (note that the y-axis is scaled to increase readability). The table reports the maximum of the mean test accuracy all timesteps, well as the standard deviation (rounded to 0.01).
ric mean with li Σ, x but in practice there are instabilities as approaches 0 because approaches J. Similarly, while theory PowerAgg can represent min magnitude, max magnitude, harmonic mean, and root mean square, it falls short in practice (see Table 2a). likely because of reasons stated Section In other cases, baselines can perform well even they should not be able to represent the target in theory. One such example is PowerAgg, which achieves correlation of 0.92 on max, but only 0.59 on max magnitude, which is the opposite of what theory might suggest. This is likely due to the the clamp operation that Pytorch Geometric's implemen- max magnitude suffers,
tation uses to restricts inputs to the positive domain. The performance as it misses cases where the highest magnitude element is negative. Similarly, the performance of max increases, because it simply selects the maximum among the positive elements. Another base- line which performs unexpectedly well is SoftmaxAgg, which achieves high correlation with the log-sum-exp aggregator. While it cannot compute log, the SoftmaxAgg formulation does include of exponentials, it able to produce close approximation.
5.2 GNN Regression
In GNN Regression, the experimental setup same as that of Aggregator Regression (Section 5.1), with the exception that the observation size is reduced to d = 1. However, instead of using GenAgg Do as a model, we use a multi-layer GNN. The GNN implemented with 4 layers of GraphConv [16] with Mish activation (after every layer except the last), where the default aggrega-

---

tion function is replaced by parametrised aggregator Dg:
2;
(k+1) Mish
W 2
(k)
+W
W(
(8)
(0) 27
2/4
(5)
JEN,
(3) = -W 2 + W
(6)
JEN:
While this experiment uses the same dataset as Aggregator Regression (Section 5.1), it provides several new insights. First, while the aggregator regression experiment shows that GenAgg can represent various aggregators, this experiment demonstrates that training remains stable even when used within larger architecture. Second, this experiment underlines the importance of using the correct aggregation function. While it is clear that it is advantageous to match a model's aggregation function with that of the underlying mechanism which generated particular dataset, we often opt to simply use a default aggregator. The conventional wisdom of this choice is that the other learnable parameters in a network layer can rectify an inaccurate choice in aggregator. However, the results from this experiment demonstrate that even with additional parameters, it is not necessarily pos- sible to represent different aggregator, underlining the importance of aggregators with sufficient representational capacity.
Results. The results show that GenAgg maintains its performance, even when used component within a GNN (Table 2b). GenAgg achieves a mean correlation of 0.97 across all aggregators. While the baselines perform significantly better with the help of a multi-layer GNN architecture, they still cannot represent many of the standard aggregators. The highest-performing baseline is SoftmaxAgg, which only achieves a mean correlation of 0.86.
5.3 GNN Benchmark
In this experiment, examine the performance of GenAgg GNN benchmark datasets [6]. In order to perform a comparison with benchmarks, we train on an existing GNN architecture (a 4-layer GraphConv [16] GNN with a hidden size of 64) where the default aggregator is replaced with new aggregator, selected from [GenAgg, PowerAgg, SoftmaxAgg, PNA, mean, sum, max).
Results. As shown in Fig 3, GenAgg outperforms all baselines in all four GNN benchmark datasets. provides a significant boost in performance, particularly compared to the relatively small differ- ences in performance between the baseline methods.
The training plots in Fig. 3 provide complementary information. One interesting observation is that GenAgg converges at least as fast as the other methods, and sometimes converges significantly faster (in PATTERN, for example). Furthermore, the training plots lend information about the stability of training. For example, note that in MNIST, most of the baseline methods achieve a maximum and then degrade in performance, while GenAgg maintains stable performance throughout training.
Discussion
Results. In our experiments, we present two regression tasks and one GNN benchmark task. The regression experiments demonstrate that GenAgg is the only method capable of representing all of standard aggregators, and a GNN cannot used to compensate for the shortcomings of the baseline aggregators. The GNN benchmark experiment complements these findings, demonstrating that this representational complexity is actually useful in practice. The fact that GenAgg outperforms the standard aggregators (mean, max, and sum) the GNN benchmark experiment implics that it is in fact creating a new aggregator. Furthermore, the fact that it outperforms bascline methods like SoftmaxAgg and PowermeanAge implies that the aggregator learned by GenAgg lies outside the set of functions which can be represented by such methods.
Limitations. While GenAgg achieves positive results on these datasets, it is not possible to make generalisations about its performance in applications. particular, we observe that some datasets fundamentally require less complexity to solve, so simple aggregators are sufficient (i.e., GenAgg fails to provide a significant performance boost). For full list of datasets that we considered and further discussion of limitations, see Appendix D.

---

Parameters. When comparing the performance of different models, it is important to also consider the number of parameters. By introducing additional parameters, some models can improve overall performance at the cost of sample efficiency. While methods like PowerAgg and Softmax only have one trainable parameter, GenAgg has two scalar parameters a and B3, and learnable function f. which has 30 parameters in our implementation (independent of the size of the state). However, we observe that using GenAgg within a GNN is always at least as sample-efficient as the baselines, and sometimes converges significantly faster (Fig. 3 and Appendix B). Furthermore, while GenAgg has more parameters than PowerAgg and softmaxAgg, the increase is negligible compared to the total number of parameters in the GNN. We also note that GenAgg has significantly fewer parameters than the deep learning methods discussed in Section 4. While the deep learning methods scale linearly or quadratically in the dimension of the state, the number of parameters in GenAgg is constant.
Stability. Another observation from our experiments is that GenAgg exhibits more stability during the training process than the baselines (Appendix B). In the GNN Regression experiment, the Pow- erAgg and SoftmaxAgg training curves tend to plateau least once before reaching their maximum value. It is possible that these methods lead to local optima because they are optimised in lower dimensional parameter space [3]. For example, is straightforward to smoothly transform a learned in GenAgg from x² to 2², but to do so in PowerAgg, it is necessary to pass through æ3, which has significantly different behaviour in the negative domain. While PowerAgg restricts inputs to the positive domain to circumvent this particular issue, the problem of local optima can still arise when methods like PowerAgg or SoftmaxAge are used as components in a larger architecture.
Explainability. While in this paper we primarily focus on the performance of GenAgg, we note that it also presents benefits the realm of explainability. The three parameters in GenAgg are all heman-readable (scalars and scalar-valued functions can easily be visualised), and they all provide a unique intuition. The a parameter controls the dependence on the cardinality of the input set. The |8 parameter dictates if the aggregator is computed in a raw or centralised fashion (colloquially, it answers if the aggregator operates over the inputs themselves, or the variation between the inputs). Lastly, the function ʃ can be analysed by considering the sign and magnitude of (ax)). The sign denotes if a given x, increases (z) > 0) or decreases (x,) < 0) the output. On the other hand, the magnitude [xx)] can be interpreted as the relative impact of that point on the output. For example, the parametrisation of product is f(x) = log((z:) (|x1). which implies that a value of has no 0, and extremely small values have large impact, because lim log((<))| = co. Indeed, 1 is the identity element under multiplication, and multiplying by a change the output by many orders of magnitude. The interpretability of GenAgg
impact the output since log([1|)
small value can also be leveraged as a method to select an aggregator-a model can be pre-trained with GenAgg, and then each instance of GenAgg can be replaced with the most similar standard aggregator in A.
7 Conclusion
In this paper we introduced GenAgg, a generalised, explainable aggregation function which parametrises the function space of aggregators, yet remains as constrained as possible improve sample efficiency and prevent overfitting. In our experiments. we showed that GenAgg can repre- sent all 13 of our selected "standard aggregators" with a correlation coefficient of at least 0.96. We also evaluated GenAgg alongside baseline methods within a GNN, illustrating how other approaches have difficulties representing standard aggregators, even with the help of additional Icarnable param- eters. Finally, we demonstrated the usefulness of GenAgg on GNN benchmark tasks, comparing the performance of the same GNN with various different aggregators. The results showed that GenAgg provided significant boost in performance over the baselines in all four datasets. Furthermore, GenAgg often exhibited more stability and faster convergence than the baselines in the training process. These results show that GenAgg is application -ggnostic aggregation method that can provide a boost in performance as a drop-in replacement for existing aggregators.
8 Acknowledgements
Ryan Kortvelesy and Amanda Prorok were supported in part by ARL DCIST CRA 1911NNF11-22011 and European Research Council (ERC) Project 949940 (gAla).

---

References
[1] S.M. Aji and R.J. McEliece. The generalized distributive law. IEEE Transactions on Information Theory, 46(2):325-343, March 2000.
[2] Peter W Battaglia, Jessica B Hamrick, Victor Bapst. Alvaro Sanchez- Gonzalez, Vinicius Zambaldi, Ma- teusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1xKiv:8806.01200 2018.
Sanchez-Gonzalez,
[3] Alan J. Bray and David S. Dean. The statistics of critical points Gaussian fields on large- dinensiime spaces. Physical Review Letters, 98(15), April 2007.
[4] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 14001410000000 1078, 2014.
[5] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neigh- bourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33, 2020.
[6] Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Bench- marking graph neural networks. arXiv preprint arXiv:2003, 00982, 2020.
[7] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with Py Torch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[8] Caglar Gulcehre, Kyunghyun Cho, Razvan Pascanu, and Yoshua Bengio. Learned-nom pooling for deep feedforward and recurrent neural networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 530-546. Springer, 2014.
[9] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Ad- vunces in neural information processing systems, 30, 2017.
[10] Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows:
An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(1 (3999,999,,, 2020.
[11] Andrey Kolmogorov. On the notion of mean. Mathematics and Mechanics, Selected works of A.N. Kolmogorov. Vol. 1:144-146, 1930.
[12] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computa- tion, 1(4):541-551, 1989.
[13] Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergen: All you need train deeper gens. arXiv preprint arXiv:2006.07739, 2020.
[14] Yujia Li. Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. Graph matching networks for learning the similarity of graph structured objects. In International conference on machine learning,
pages 3835-383-3 (335-384,,10 PMLR, 2019.
[15] Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. Graph matching networks for learning the similarity of graph structured objects. In International conference on machine learning. pages 3835-3845. PMLR, 2019.
[16] Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grobe Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4602-4609, 2019.
[17] Athanasios Papoulis and S Unnikrishna Pillai. Probability, random variables and stochastic processes. McGraw-Hii Hill, 1991.
[18] Giovanni Pellegrini, Alessandro Tibo. Paolo Frasconi, Andrea Passerini. and Manfred Jaeger. Learning Aggregation Functions. In Proceedings of the Thirtieth International Joint Conference on Artificial Intel- ligence, pages 2892-2898, Montreal, Canada, August 2021. International Joint Conferences on Artificial Intelligence Organization.
[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[20] Beibei Wang and Bo Jiang. Generalizing aggregation functions in gnns: High-capacity gans via nonlinear neighborhood aggregators. arXiv preprint arXiv:2202.091455 2022.
[21] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design Space for Graph Neural Networks. In Advances in Neural Information Processing Systems, volume 33, pages 17009-17021 Curran Associates, Inc., 2020.
[22] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep Sets, April 2018. arXiv:1703.06111 [cs, stat].

---

[23] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architec- ture for graph classification, In Proceedings of the AAAI conference artifiiil intelligence, volume 32, 2018.


Reviews:
[{"review_id": "mIUowM2AOH", "reviewer": "Reviewer_rgaT", "summary": "EDIT: Due to author responses below, I have increased my score from a 6 to a 7.\n\nThis work proposes a generalized method for parameterizing the aggregation operation of a message passing neural network (i.e. a graph neural network, GNN). This relies on a modification of the generalized $f$-mean that adds two hyperparameters, $\\alpha$ and $\\beta$. The authors describe how their method, GenAgg, can approximate commonly used GNN aggregators by varying the hyperparameters and the definition of the function $f$. To overcome issues with defining analytical forms of  $f^{-1}$, the authors propose to implement $f$ and $f^{-1}$ with two separate MLPs that learn the inversion.\n\nThen they evaluate GenAgg with three sets of experiments. One tests how well their GenAgg method can approximate common aggregation functions. Then they insert GenAgg into a GNN to see if these results hold true when the aggregator is inside a GNN architecture. Finally, they evaluate a GNN with GenAgg on widely-used benchmarks for GNNs.", "questions": "- On lines 145-146, it says that \"we found mapping to a higher intermediate dimension can sometimes improve performance over a scalar-valued $f$\" with a reference to Appendix E. This claim is not mentioned or alluded to at all in Appendix E.\n- Line 151: is $c$ a scalar?\n- Section 6 claims that PowerAgg and SoftmaxAgg training curves plateau at least once (lines 326 - 328). However, I do not notice a difference between these training curves and the others. Can you show this more clearly?\n- The plots in Fig 3 are hard to read because the lines and error bars often overlap a lot.\n    - Please add standard deviation to the table as well, to match the error bars in the plots.\n    - I also recommend putting a color coded label on the right side of the plot to indicate the ordering of the final test accuracy (e.g. from top to bottom for CLUSTER: GenAgg, Softmax-agg, Powermean-agg, mean, max, ...). While this information is in the table, it would make the plots easier to read.\n\nMinor typos:\n- \"the a\" - line 71\n- \"are a quantitative measures\" (line 110) -- \"a\" should be dropped\n", "limitations": "The authors discuss some limitations of the method in Section 6. However, as noted above, including some of the discussion from Appendix D in the main text would be more appropriate.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The described method and parameterized implementation appear sound and practical for researchers optimizing GNN model designs.\n    - In particular, the formulation of the f-mean to overcome various constraints and the demonstration of stable training behavior make this method attractive for GNN practitioners.\n- The proposed GenAgg method clearly and stably outperforms the other aggregator methods evaluated in experiments presented in the main text.\n- The experiments are well-designed and adequately thorough for this venue.\n- The analysis of their experimental results is interesting and insightful (e.g. for the Aggregator regressions, the authors discuss where PowerAgg fails and why theory deviates from practice)\n- The authors share their code. Upon first inspection the release appears to be comprehensive, and providing a ready implementation is a useful contribution.\n\nI cannot comment on the novelty because I am not familiar with other recently proposed methods for aggregation in GNNs. While I checked the math throughout the main text, I did not review all the parameter settings for the aggregation functions in Table 1.", "weaknesses": "- Some claims are made about the sample efficiency of other methods (e.g. lines 47, 194) that should be supported by citations or data.\n- While I do appreciate the inclusion of the data in Appendix D, there should be a more in-depth discussion of the method's limitations in the main text.\n    - I am not fully convinced of the overfitting claim, especially because the pattern of results on PUBMED does not differ much from those presented in Figure 3. If overfitting is the reason that GenAgg does not show strong performance benefits on the smaller datasets in Appendix D, we should see a decrease in its benefit when reducing the size of a dataset in the main text (e.g. PATTERN).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review! We will address the weaknesses:\n\n- The closest thing to a DeepSets-based GNN in prior literature is the original MPNN. Despite its representational capacity, subsequent research has focused on more constrained methods. A direct comparison can be seen in the PNA paper [1]. Since it can represent anything that the more recent methods can in theory, but not in practice, we say it has “poor sample efficiency”. For LSTM aggregation, we say it has poor sample efficiency because it is not permutation-invariant (which is acknowledged in the paper where it was introduced). As for PNA in line 194, our argument about the sample efficiency can be backed up by our results in the experiments section.\n- The discussion about overfitting in the limitations section is based on 1) the observation that GenAgg tends to exhibit a larger performance boost on larger datasets, and 2) the observation that the datasets where GenAgg did not provide as much of a performance increase tended to have a large gap between train and test accuracy. These are the main differences between datasets like PUBMED and those like PATTERN. You are right, if overfitting is an issue, we should see a performance decrease if we significantly shrink the size of the PATTERN training dataset.\n\nAnd your questions:\n- This is alluding to the hidden sizes which are discussed in Appendix E ([1,2,2,4] for $f$, and [4,2,2,1] for f$^{-1}$). If we used the standard formulation for GenAgg, the networks would look something like [1,4,1] (mapping from $\\mathbb{R} \\mapsto \\mathbb{R}$).\n- Yes, $c$ is a scalar, and $x_i$ is as well. Since the feature dimensions are independent, we can consider a feature dimension of size 1 without loss of generality.\n- Most of our training curves are in the Appendix. In this claim, we are specifically referencing Figure 5 in Appendix B.\n- Thank you for the suggestion, we will do this.\n\n[1] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Velickovic. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33, 2020.\n"}, {"review_id": "83doOEnLwZ", "reviewer": "Reviewer_3j4u", "summary": "The paper proposes a generalized aggregation operator, GenAgg, for GNNs that can significantly improve performance in various tasks. The authors argue that existing aggregation methods are limited in their expressiveness and propose GenAgg as a more flexible alternative. They demonstrate the effectiveness of GenAgg on some benchmark datasets. Overall, the paper makes a contribution to the field of GNNs by introducing a new aggregation operator.", "questions": "- How does the computational complexity of GenAgg compare to existing aggregation methods?\n- Is the learned parameters in GenAgg interpretable? Any analysis can be made?", "limitations": "- The paper could benefit from a more detailed discussion of the limitations of the proposed approach, for example, the computational overhead in terms of memory and runtime.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper proposes a new aggregation operator, GenAgg, that provides a more flexible alternative to existing methods.\n- The authors provide a thorough evaluation of GenAgg on several benchmark datasets and show that it outperforms existing methods.", "weaknesses": "- The technical content of the paper is quite limited (only taking 2 pages)\n- The paper does not compare with DeepSet like aggregator, both in terms of theory and experiments, which is proved to be fully injective.\n- The paper could benefit from an ablation study of GenAgg by showing the results of different parameterizations of f.\n- The figures and tables are not professional. They have fonts that are too small to read. Figure 2 should be a table, not a figure.\n- The paper could benefit from a more detailed discussion of the limitations of the proposed approach, for example, the computational overhead in terms of memory and runtime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, you bring up some good points. We will first address the weaknesses:\n\n- Yes, the method section is about 2 pages—the architecture itself is relatively simple (we regard this as a good thing). However, the rest of the paper also contains technical content, such as the experimental approach and the appendix. For example, we regard Theorem 3.1 (referencing Appendix A) as a particularly interesting and useful theoretical contribution.\n- We do not compare against DeepSets because it is widely regarded as a set function, not an aggregation function—it is different from GenAgg and all of the standard aggregators in that it is not independent over the feature dimension (line 77). It is important to define this boundary between set functions and aggregation functions, as otherwise all GNN architectures would be subsumed under aggregators. In this paper, we stress the importance of constraints specifically in aggregation functions. If we were to augment DeepSets with these constraints, the architecture would approach GenAgg, so one could think of GenAgg as the aggregator variant of DeepSets.\n- We do show the performance of a subset of special cases of GenAgg in each experiment (max, sum, and mean, as shown in Figures 2a, 2b, and 3).\n- Thank you, we will fix this. Are there any specific notes that you have about the figures other than their text size?\n- We will include a figure and discussion about the computational and memory overhead in the discussion section.\n\nTo address your questions:\n- If you’re referring to theoretical computational complexity, then sum, PowerAgg/SoftmaxAgg, and GenAgg are in the same category. They scale linearly with the number of features or the number of edges. On the other hand, PNA employs a network which operates over a concatenation of multiple flattened outputs, so the number of parameters (and therefore the number of multiplications) grows quadratically with the number of features. This is discussed in the Parameters section of the Discussion.\n- Yes, the learned parameters can provide intuition about the underlying aggregation function (see the Explainability section of the Discussion for a more detailed explanation).\n\nWe will edit the figures’ readability, and add a figure and discussion to the limitations section about the computational overhead of each aggregator. In addition to these items, is there anything else you would like us to add to our rebuttal results / revisions in order to increase your score?\n\n"}, {"review_id": "ADSafcDOQ8", "reviewer": "Reviewer_uMEa", "summary": "This paper presents GenAgg, a generalised aggregation operator, which parametrises a function space that includes all standard aggregators. They show using GenAgg as a drop-in replacement for an existing aggregator in a GNN usually leads to a boost in performance across various tasks.", "questions": "see weakness.", "limitations": "The authors mentioned overfitting cases in their limitation part. I also point out the latency issues, I would like to know the feedback from authors.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed GenAgg is an unified aggregator that parametrizes a function space, encompassing all standard aggregators.\n\n2. Using GenAgg as a drop-in replacement for the aggregators in a GNN often leads to a performance boost. \n\n3. The paper is well-structured. Table 1 is particularly commendable as it lucidly demonstrates that the proposed GenAgg can accommodate most common aggregators.", "weaknesses": "\n1. The tradeoff is not mentioned. GenAgg requires to uses 8 additional linear layers (4 layers MLPs for f and f inverse). Despite the channel size of each layer is small, it will still lower down inference speed significantly. The latency of using different aggregators should be included in Figure 3 Table. The fair comparison for different aggregators in the same latency (by changing the number of channels) should be considered as well.\n\n2. The authors should consider providing GenAgg with various baselines. Currently, the demonstration of GenAgg is limited to its use with a simple 4-layer GraphConv model. It would be insightful to see experiments using more powerful baselines such as GraphSAGE, DeepGCNs, and state-of-the-art networks. There's a possibility that as network capability increases, the selection of aggregation functions diminishes in significance. This brings into question another aspect of this paper: its practicality.\n\n3. The practical application of GenAgg appears to be limited for large networks. Given that GenAgg slows down the processing speed, it prompts the question: Is it essential to implement GenAgg in a deep network? It would be beneficial if the authors could demonstrate at least one instance of its application in a larger graph (for example, OGB benchmarks) using a larger network (over 50 layers), rather than limiting their illustrations to small networks in toy examples.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! To address the weaknesses:\n\n- You’re right, GenAgg does introduce some computational overhead compared to an aggregator like ‘sum’. Unfortunately, an experiment where we tweak the architectures until all achieve the same latency is not possible—sum is a subcomponent of GenAgg, so it will always be less computationally expensive. However, we note that a forward pass takes less than 1ms (using MNIST data, processing a batch containing over 500,000 edges). Given the scale of this runtime, the difference in runtime for larger architectures using GenAgg vs other aggregators as subcomponents should be relatively small. That being said, the focus of this paper is not on runtime, but on performance. - We examine a different tradeoff: representational capacity vs overfitting (to address this, we report test accuracy, and we discuss the tradeoff with the number of model parameters). However, we agree that latency is useful information for some, so we will include a table of the latencies in the Limitations section (see annexed PDF), as well as a discussion about the results.\n- Ideally we would like to show that GenAgg works with all different GNN architectures and numbers of layers. However, due to the impracticality of running the same experiments for multiple trials on several different setups, we chose to select a representative architecture. To maximise the relevance of the results, we selected GraphConv (a popular architecture), and chose the number of layers to be 4 in accordance with the details in the Benchmarking GNNs paper [1]. As an aside, we note that the Pytorch Geometric implementation of SAGEConv is equivalent to GraphConv with a mean aggregator instead of sum (the project and normalise operations are disabled by default), so implicitly we did test that particular architecture. To address your concern about the usefulness of aggregators as the number of layers increases: you’re right that it is possible that the benefit of aggregators might see diminishing returns after the first layer. However, it is likely that the aggregation in the first layer will remain extremely important, as a naive aggregation can irreversibly remove important information from the raw input (for example, if we wish to predict the standard deviation in the distances of neighbouring agents from their positions, then if we apply a sum in the first layer, it is not possible to reconstruct the correct output, no matter how many layers there are). To some degree, this result can be seen in the second experiment: even as we introduce more than enough representational complexity in a GNN, it is unable to overcome a poor choice in aggregator.\n- Most of this question was addressed in the previous answer, but we will briefly elaborate. We would not consider the GNN Benchmark experiment a “small network in a toy example”. The Pattern and Cluster datasets include 10,000 graphs, each of which contain an average of 4,000-6,000 edges. To maximise relevance to the community, we strove to select the most popular datasets and use the most popular architecture (we note that the models in the papers associated with the datasets on pytorch geometric often use ~4 layers). However, we agree that in the future it would be interesting to test GenAgg with different architectures.\n\n[1] Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.\n"}, {"review_id": "mZytuO22GN", "reviewer": "Reviewer_PwWR", "summary": "The authors deal with the interesting problem of parametrising the space of aggregation functions. They propose a novel aggregation method that allows each MPNN to learn the most appropriate aggregator for each downstream task. In contrast to previous works, their proposed aggregator can interpret a broad pool of known aggregators and enjoys a set of theoretical observations that can guarantee the generalization of the distributive property. \n\nThe proposed aggregator is based on the generalized f-mean, a known operator that imposes though a set of restrictive constraints with respect symmetry, monotonicity, and idempotency. For that reason, they extend the definition of the generalized f-mean operator with parameters $\\alpha$ and $\\beta$ that relax the two latter constraints, while maintaining the symmetry property.\n\nUsing two separate MLPs, they implement the instances of functions $f$ and $f^{-1}$ for the definition of the augmented f-mean operator, and they propose a new optimization objective (Equation 2), in order to ensure the equality of the invertability of function $f$ to the original input signal $x$.\n\nTheir extensive experimentation study suggests the superior performance of their proposed aggregator (GenAgg) in both computer vision and graph downstream tasks. Also, Theorem 3.1 suggests that GenAgg satisfies a generalised distributive property, allowing for further analysis of the algorithmic time  and space complexity of common aggregators. \n\n", "questions": "1. There is a relation of the proposed parameterization of the aggregation operators with a couple of other proposed parametrised operators [1],[2]. Can the authors discuss any potential advantage over these methods, and whether there is a theoretical or intuitive connection among them? \n\n\n[1] Dan Zhang et al. Node-Wise Adaptive Aggregation in GNNs for Recommendation. 2023\n[2] George Dasoulas et al. Learning Parametrised Graph Shift Operators. 2021\n", "limitations": "The authors discuss potential limitations of their method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of extending the generalized f-mean operator for defining a new aggregation framework is novel, and provides insights on the effectiveness of common aggregators in MPNNs. \n- GenAgg can be the framework for defining already defined aggregation operators, as well as enabling the design of new, more sophisticated ones.\n- Theorem 3.1 allows for a further analysis of common aggregators with respect to their time and space complexity.\n- The empirical results suggest a very strong performance of GenAgg with respect to other powerful MPNN aggregators. ", "weaknesses": "- There is not a clear connection of the evolution of parameters $\\alpha$ and $\\beta$ with respect to the downstream tasks. Especially, when the datasets come from different domains (e.g computer vision, and graph data), it is unclear how the GenAgg is formed, and whether the final configuration correspond to an optimal representation of the neighborhood adjacency. \n- It would be very helpful for the authors to discuss the convergence analysis of the loss term defined in Equation 2. It seems from the results in the computer vision datasets, that the model is quickly converged to a state-of-the-art performance. Is that pattern observed, also, in the GNN benchmark datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We will address the weaknesses that you cited:\n\n- If I understand this correctly, this is about the explainability of GenAgg—is that correct? You are correct, some datasets (such as images, as you mentioned) are less likely to lead to an easily interpretable aggregator. Any learned aggregator can be analysed to some degree (e.g. the dependence on the degree, the relative importance of high/low values, the dependence on the variation of the data) by the method outlined in our discussion section, but the aggregator will not always converge to a satisfying special case. To maximise the explainability properties of GenAgg, we would suggest using it in a proxy model with data which is likely to have some symbolic form (e.g. geometric data, physics-based data, etc).\n- This bullet brings up two points. To answer the first: the loss term from Equation 2 converges orders of magnitude faster than the model as a whole. In effect, $f$ and $f^{-1}$ remain true inverses of each other throughout training, and evolve simultaneously as they converge to the desired parametrisation. To answer the second point: All four of the datasets in this paper are GNN datasets, two of which are adapted from computer vision data, and two of which are based on mathematical modelling tasks. \n\nTo answer your question about the the relationship between GenAgg and the methods you have cited:\n\n- *Node-Wise Adaptive Aggregation in GNNs for Recommendation*.\nThis paper is about a new GNN architecture for the purpose of learning on heterogeneous graphs. This is achieved by conditioning the weights on metrics which can distinguish the nodes from each other, such as the degree. This focuses on a different part of the GNN—while GenAgg parametrises the aggregation function, this method parametrises the weights applied to each node. To avoid confusion, note that this paper defines aggregation slightly differently than we do. For example, the paper regards attention as a form of aggregation, while we would consider it a mechanism which *contains* a sum aggregator.\n- *Learning Parametrised Graph Shift Operators*.\nThis method parametrises graph shift operators, generalising special cases like the adjacency, the normalised form of the adjacency used in GCN, and the laplacian. While GenAgg focuses on just aggregation, this method sets the edge weights which are then used in the aggregation step (the aggregation they use is a sum, so it is always linear regardless of the GSO parametrisation). In this sense, the methods are complementary—one could use both approaches at the same time. The one aspect of overlap is normalisation based on the degree. While GenAgg bases the normalisation exclusively on the degree of node $i$, this method can also normalise by a combination of the degrees of node $i$ and node $j$.\n"}]

Author rebuttal:
Two reviewers have requested that we add a table about the computational overhead of GenAgg to our Limitations section. We include our results in the attached PDF.

Meta-review: After the additional experiments performed during the rebuttal period all reviewers voted for acceptance. The authors should include the additional experiments and apply the proposed changes in the revised version of the manuscript.
Decision: Accept (poster)