Learning Aggregation Functions
Authors: Giovanni Pellegrini
Alessandro Tibo
Paolo Frasconi
Andrea Passerini
Manfred Jaeger
Venue:  (2021)
Keywords: Deep learning
Neural networks
Relational and structured data
Aggregation functions

Abstract:  Learning on sets is increasingly gaining attention in the machine learning community, due to its widespread applicability. Typically, representations over sets are computed by using fixed aggregation functions such as sum or maximum. However, recent results showed that universal function representation by sum- (or max-) decomposition requires either highly discontinuous (and thus poorly learnable) mappings, or a latent dimension equal to the maximum number of elements in the set. To mitigate this problem, we introduce LAF (Learning Aggregation Functions), a learnable aggregator for sets of arbitrary cardinality. LAF can approximate several extensively used aggregators (such as average, sum, maximum)  as well as more complex functions (e.g. variance and skewness). We report experiments on semi-synthetic and real data showing that LAF  outperforms state-of-the-art sum- (max-) decomposition architectures such as DeepSets and library-based architectures like Principal Neighborhood Aggregation.

Under review as a conference paper at ICLR 2021
LEARNING AGGREGATION FUNCTIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning on sets is increasingly gaining attention in the machine learning com- munity, due to its widespread applicability. Typically, representations over sets are computed by using fixed aggregation functions such as sum or maximum. However, recent results showed that universal function representation by sum- (or max-) decomposition requires either highly discontinuous (and thus poorly learn- able) mappings, or latent dimension equal the maximum number of elements in the set. To mitigate this problem, we introduce LAF (Learning Aggregation Functions), learnable aggregator for sets of arbitrary cardinality. LAF can ap- proximate several extensively used aggregators (such as average, sum, maximum) as well as more complex functions (e.g. variance and skewness). We report experi- ments on semi-synthetic and real data showing that LAF outperforms state-of-the- art sum- (max-) decomposition architectures such as DeepSets and library- based architectures like Principal Neighborhood Aggregation.
INTRODUCTION
The need to aggregate representations is ubiquitous in deep learning. Some recent examples include max-over-time pooling used in convolutional networks for sequence classification (Kim 2014).
erage pooling of neighbors in graph convolutional networks (Kipf & Welling (017)) max-pooling in Deep Sets (Zaheer al., (017). 7), in (generalized) multi-instance learning (Tibo al., 2017 and in GraphSAGE (Hamilton et al. (017). ). In all the above cases (with the exception of LSTM-poollig in GraphSAGE) the aggregation function is predefined, i.e., not tunable, which may be in general disadvantage (Isse et al.) 2018). Sum-based aggregation has been advocated based on theoretical findings showing the permutation invariant functions can sum-decomposed (Zaheer et al.) 2017] Xu et al., (019). However, recent results (Wagstaff et al., 2019) showed that this universal function representation guarantee requires either highly discontinuous (and thus poorly learnable) mappings, or a latent dimension equal to the maximum number of elements in the set. This suggests that learning set functions that are accurate on sets of large cardinality is difficult.
Inspired by previous work on learning uninorms (Melnikov & Hüllermeier 2016), propose new parametric family of aggregation functions that we call LAF, for learning aggregation functions. A single LAF unit can approximate standard aggregators like sum, max mean well model intermediate behaviours (possibly different in different areas of the space). In addition, LAF layers with multiple aggregation units can approximate higher order moments of distributions like variance, skewness or kurtosis. In contrast, other authors (Corso et al. 2020) suggest to employ a predefined library of elementary aggregators be combined. Since LAF can represent sums, it can be seen as a smooth version of the class of functions that are shown in Zaheer et al. (2017) to enjoy universality results in representing set functions. The hope that being smoother, LAF is more easily learnable. Our empirical findings show that this can be actually the case, especially when asking the model to generalize over large sets.
In particular, in this paper we offer an extensive experimental analysis showing that:
LAF layers can learn a wide range of aggregators (including higher-order moments) on sets of scalars without background knowledge on the nature of the aggregation task
LAF layers on the top of traditional layers can learn the same wide range of aggregators on sets of high dimensional vectors (MNIST images)
LAF outperforms state-of-ther -het---r set learning methods such as DeepSets and PNA on real- world problems involving point clouds and text concept set retrieval.

---

Under review as a conference paper at ICLR 2021
Name
Definition
a b
d
f
9
h/a 8 y limits
constant
E I
0
0
-c 10
max
max; Zi
1/r r
0
1
-10 O + 00
min
min, 24
0
1/
O
1
1 1-110r
sum
Σx 24 {i 2t + 0H
O
1
- 1010
nonzero count
1 0
O
1
1010
mean
1/N
Mi a
I 1
1 0
1010
kth moment
1/N
K
1
O
1010
/th power of kth moment
(1/N
k
I
0 1010
min/max
min, L/// max;
0
1/rr
1/88
11
0:00
max/min
max; z/// min; Zj 1/r
0 1
1011r,
Table 1: Different functions achievable by varying the parameters in the formulation in Eq
LAF performs comparably to PNA on random graph generation tasks, outperforming several graph neural networks architectures including GAT (Veličković et al. 2018) and GIN (Xu et al. 2019)
The rest of this work is structured follows. In Section define the LAF framework and show how appropriate parametrizations of LAF allow to represent wide range of popular aggregation functions. Section 3 we discuss some relevant related work. Section reports synthetic and real- world experiments showing the advantages of LAF over (sets of) predifined aggregators. Finally, conclusions and pointers future work are discussed in Section
THE LEARNING AGGREGATION FUNCTION FRAMEWORK
to denote finite multisets of real numbers I, E R. Note that directly taking x to be multiset, not a vector, means that there is no need to define properties like ex- changeability or permutation equivariance for operations function that returns for any multiset a of arbitrary cardinality N E N a value agg(x) e R.
We use x
An aggregation function agg is any
Standard aggregation functions like mean and We therefore build our parametric LAF aggregator around generalized L,-norms of the form
be understood (normalized) Lp- Lo-nos..
(a.((x)
Σ
(a. b 2020. 0).
(1)
La.b is invariant under the addition of zeros: La, (x, = La,b(x U where 0 is multiset of zeros of arbitrary cardinality. order to also enable aggregations that can represent conjunctive behavior such min, we make symmetric use of aggregators of the multisets 1 x {1 ri|0; æ).. For L - x) to be a bellveehaved, dual version of (x)) the values in æ need to in the range [0, 1]. We therefore restrict the following definition of our learnable aggregation function sets whose clements are in [0, 1]:
LAF(#) = aLa,b(x) + 3Le x)
(2)
( + ô o. (1 x))
defined by tunable parameters a..... O, and a. a..... 5 E R. In cases where sets need to be aggregated whose elements are not already bounded by 0, 1, we apply sigmoid function to the set elements prior to aggregation.
'able shows how number of important aggregation functions are special cases of LAF (for values in [o, I)). We make repeated use of the fact that Ta,11 returns the constant 1. For max and min LAF only provides
asymptotic approximation in the limit of specific function parameters (as indicated in the limits column of Table []. In most cases, the parameterization of LAF for the functions in Table will not be unique. Being able encode the powers of moments implies that e.g. the variance of æ can be expressed as the difference 1/N Σ, x - (1/N Σ, x;)² two LAF aggregators.
Since LAF includes sum-aggregation, we can adapt the results of Zaheer et al. (2017) and Wagstaif al. 2019) the theoretical universality of sum-aggregation as follows.
2

---

Under review as a conference paper at ICLR 2021
Proposition 1 Let X R be countable, and f function defined finite multisets with elements from X. Then there exist fumctions o such that f(x) p(LAF((=)) 3, v, a, b, c, d), where oæ is the multiset {Ø(z)) x}.
[0, 11, p R. - R. and a parameterization of LAF,
A proof in Wagstaff al. (2019) for very similar proposition used mapping from X' into the reals. Our requirement that LAF inputs must in [0. 1] requires a modification the proof (contained in the supplementary material), which for the definition of ø relies randomized construction. Proposition [ shows that we retain the theoretical universality guarantees of Zaheer et al.) (2017). while enabling a wider range of solutions based continuous encoding and decoding functions.
Figure 1: LAF functions with randomly generated parameters
should be emphasized at this point that the primary purpose of LAF is not to provide a uniform representation of different standard aggregators as displayed in Table but enable continuum of intermediate and hybrid aggregators. Figure shows the graphs of 4 different randomly generated LAF functions over the unit square [0, x [0, 1], i.c., evaluated over sets of size 2. Parameters a,....? were randomly sampled in the interval [0, 1]; parameters b, d. are randomly sampled from the integers ...... and a, c, e, g are obtained as 1/i with random integer from O.....5.
The figure illustrates the rich repertoire of aggregation functions with different qualitative behaviors already for non-extreme parameter values.
2.1 LAF ARCHITECTURE
LAF can be easily used module of larger architecture suitable for learning sets. Sev- eral LAF units can be combined as shown in Figure 2 to capture different aspects of the in- put set, which that multiple aggregators are also used in related frameworks such as DeepSets (Zaheer et al. 2017) Graph Neural Networks (Veličković al. 2018 Corso al., 2020). A module with r LAF units takes as input d-dimensional vectors and produces a vector of size r x d as out- put. Each LAF unit performs an element-wise aggregation of the vectors in the set such that Lks = (AF((E .NN } akk, (k, Yke 82, ag bb, Cj. dę) for k = I,...,? and j = I,...,d d.
be in general set of vectors
{2I,...,INN] where x; E Rd. Note
The output vector can be then fed into the next layer.
RELATED WORK
Several studies address the problem of aggregating data over sets. Sum-decomposition strategies have been used in (Zaheer et al.) (017)) for points cloud classification and set expansion tasks and in (Santoro et al. 2017) for question answering and dynamic physical systems computation. Max, sum and average are standard aggregation functions for node neighborhoods in graph neural net- works (Hamilton et al. [017 Kipf Welling 2017 Xu al. 2019 Veličković et al. 2018). Zaheer et al. (2017) first proved universal representation results for these standard aggregators when com- bined with learned mappings over inputs and results the aggregation. However, Wagstaff et al. (211)) showed that these universality results are of little practical use, they either require highly discontinuous mappings that would be extremely difficult to learn, or a latent dimension that is at least the size of the maximum number of input elements.
Uninorms Yager Rybalov 1996) are class of aggregation functions in fuzzy logic that can behave in conjunctive, disjunctive averaging manner depending parameter called neutral element. Melnikov & Hüllermeier (2016) proposed to learn fuzzy aggregators by adjusting these learnalle parameters, showing promising results on combining reviewers scores papers into
3

---

Under review as a conference paper at ICLR 2021
layer
1
LAF((æ1 22. 2N))
END)
Fd
LAFi
LAF
LAF,
N
s; E R
(01, ".....,,2.....................,, NN) from previous layer
Figure 2: End-to-end LAF architecture.
overall decision of acceptance or reject. Despite the advantage of incorporating different behaviours in single function, uninorms present discontinuities in the regions between aggregators, mak- ing them not amenable be utilized in fully differentiable frameworks. Furthermore the range of possible behaviours is restricted to those commonly used in the context of fuzzy-logic.
The need for considering multiple candidate aggregators is advocated in a very recent work that was developed in parallel with our framework (Corso et al., (2020). The resulting architecture, termed Principal Neighborhood Aggregation (PNA) combines multiple standard aggregators, in- cluding most of the ones we consider in the LAF framework, adjusting their outputs with degree scalers. However, the underlying philosophy is rather different. PNA aims at learning to select the appropriate aggregator(s) from a pool of candidates, while LAF explores a continuous space of ag- gregators that includes standard ones as extreme cases. Our experimental evaluation shows that PNA has troubles in learning aggregators that generalize over set sizes, despite having them the pool of candidates, likely because of the cuasi-combinatorial structure of its search space. On the other hand, LAF can successfully learn even the higher moment aggregators and consistently outperforms PNA.
Closely connected, but somewhat complementary to aggregation operators are attention mecha- nisms (Bahdanau al. 2015 Vaswani et al. 2017). They have been explored to manipulate set data learning (llse et al.) (0088). Attention operates
Ln(Lee et al. (2019) and in the context at the level of set elements, and aims at transformation (weighting) of their representations such as to optimize subsequent weighted annaaggregttio.. While the objectives of attention based frame- works and LAF partially overlap, they functionally quite different. Exploring combinations of LAF with attention mechanisms is a possible subject of future work.
EXPERIMENTS
In this section, we present and discuss experimental results showing the potential of the LAF frame- work on both synthetic and real-world tasks"] Synthetic experiments are aimed at showing the ability of LAF to learn wide range of aggregators and its ability to generalize over set sizes (i.e., having test-set sets whose cardinality exceeds the cardinality of the training-set sets), something that alter- native architectures based on predefined aggregators fail to achieve. We use DeepSets, PNA, and these architectures. The LSTM architecture corresponds of DeepSets where the aggregation function is replaced by LSTM layer. Experiments on diverse tasks including point cloud classification, concept retrieval and graph properties prediction are aimed at showing the potential of the framework on real-world applications.
LSTM as representatives
version
'The code available in the supplementary material
4

---

Under review as a conference paper at ICLR 2021
4.1
EXPERIMENTS ON SCALARS
IG
26-2
LAF
1.00
Deepdiets
-
PNA
14
LSTM
94
06
Roan
0:00
Inverse Count
Modian
Min / Max
Variance
Skewness
Kurtasis
2.4
2.4
10
Mojer
5.7
Figure 3: Test performances for the synthetic experiment with integer scalars on increasing test set size. The axis of the figures represents the maximum test set cardinality, whereas the axis depicts the MAE error. The dot, star, diamond and triangle symbols denote LAF, DeepSets, PNA, and LSTM respectively.
This section shows the learning capacity of the LAF framework to learn simple and complex ag- gregation functions where constituents of the sets are simple numerical values. In this setting we consider sets made of scalar integer values. The training set is constructed as follows: for each set,
we initially sample its cardinality K from a uniform distribution taking values in {2, M), and then uniformly sample K integers in O, ....,. ,9 For the training set use M = 10. We construct several test sets for different values of M (M = 5. 10, 15, 20, 25, 30, 35, 40, 45, 50). This implies that models need generalize to larger set sizes. Contrarily to the training set, each test set is con- structed in order to diversify the target labels it contains, so as to avoid degenerate behaviours for large sizes (e.g., maximum constantly equal 9). Each synthetic dataset is composed of 100,000 sets for training, 20,000 set for validating and 100,000 for testing.
The number of aggregation units is set as follows. The model contains nine LAF (Equation 2 units, whose parameters {ax...., }. k 1,...,9 9 are initialized according to a uniform sampling in [0. 1] as those parameters must be positive, whereas the coefficients (a. .,, 8) are initialized with a Gaussian distribution with zero mean and standard deviation of 0.01 to cover also negative values. The positivity constraint for parameters {a, b..., h} is enforced by projection during the optimiza- tion process. The remaining parameters can take negative values. DeepSets also uses nine units: three max units, three sum units, and three mean units and PNA uses seven units: mean, max, sum, standard deviation, variance, skewness and kurtosis. Preliminary experiments showed that expand- ing the set of aggregators for PNA with higher order moments only leads to worse performance. Each set of integers is fed into embedding layer (followed by a sigmoid) before performing the aggregation function. DeepSets and PNA do need an embedding layer (otherwise they would have no parameters be tuned). Although LAF does not need an embedding layer, we used it in all mod- els to make the comparison more uniform. The architecture details are reported in the supplementary material. We use the Mean Absolute Error (MAE) a loss function to calculate the prediction error.
Figure 3 shows the trend of the MAE error for the three methods for increasing test set sizes, for different types of target aggregators. As expected, DeepSets manages learn the identity function and thus correctly models aggregators like sum, max and mean. Even if LAF needs to adjust its parameters in order to properly aggregate the data, its performance are competitive with those of DeepSets. When moving to more complex aggregators like inverse count, median or moments of different orders, DeepSets fails to learn the latent representation. One the other hand, the perfor- mance of LAF is very stable for growing set sizes. While having in principle at its disposal most of the target aggregators (including higher order moment) PNA badly overfits over the cardinality of sets in the training set in all cases (remember that the training set contains sets of cardinality at most 10). The reason why LAF substantially outperforms PNA on large set sizes could be explained in terms of greater flexibility to adapt the learnt representation. Indeed, LAF parameters can
5

---

Under review as a conference paper at ICLR 2021
DeepSets PRA LSTM
2.5
Noan
1.
80
10
15
20
25
35
inverse Count
Median
Min. / Maa
Variance
Skewness
Kurtasis
Moan Absobute Brror.
Figure 4: Test performances for the synthetic experiment on MNIST digits on increasing test set size. The axis of the figures represents the maximum test cardinality, whereas the y axis depicts the MAE error. The dot, star, diamond and traingle symbols denote LAF, DeepSets, PNA and LSTM respectively.
adjust the iaf function to be compliant with the latent representation even if the input mapping fails learn the identity. On the other hand, having bunch of fixed, hard-coded aggregators, PNA needs to be able to both learn the identity mapping and select the correct aggregator among the candidates. Finally, LSTM exhibits generally poor results when compared to the other methods, particularly in the case of the count and the sum.
4.2 MNIST DIGITS
In this section, we modify the previous experimental setting to process MNIST images of digits. The dataset the same as in the experiment on scalars, but integers are replaced by randomly sampling MNIST images for the same digits. Instances for the training and test sets drawn from the MNIST training and test sets, respectively. This experiment aims to demonstrate the ability of LAF to lear from more complex representations of the data by plugging it into end-to-end differentiable architectures. Contrarily to the model of the previous section, here we use three dense layers for learning picture representations before performing the aggregation function. The architecture details are reported in the supplementary material.
Figure 4 shows the comparison of LAF, DeepSets, PNA, and LSTM in this setting. Results quite similar to those achieved in the scalar setting, indicating that LAF is capable of effectively backpropagating information so as drive the learning of an appropriate latent representation, while DeepSets, PNA, and LSTM suffer from the same problems seen in aggregating scalars.
Furthermore, Figure 5 provides qualitative evaluation of the predictions of the LAF, DeepSets, and PNA methods on representative subset of the target aggregators. The images illustrate the correlation between the true labels and the predictions. LAF predictions are distributed over the diagonal line, with clear bias. On the other hand, DeepSets and PNA perform generally worse than LAF, exhibiting higher variances. partieular, for inverse count and kurtosis, DeepSets and PNA predictions are condensed in specific area, suggesting an overfitting on the training set.
4.3 POINT CLOUD
In order to evaluate LAF on real-world dataset, we consider point cloud classification, prototype task for set-wise prediction. Therefore, we run experimental comparisons on the ModeINet40 (Wu) et al. 2015) dataset, which consists of 9,843 training and 2,468 test point clouds of objects dis- tributed over 40 classes. The dataset is preprocessed following the same procedure described by Za- heer et al. 2017). We create point clouds of 100 and 1,000 three-dimensional points by adopting the clint-cloud library's 's sampling routine developed by Rusu & Cousins (201I) and normalizing each set of points to have zero mean (along each axis) and unit (global) variance. We refer with

---

Under review as a conference paper at ICLR 2021
LAF
DeeoSStt
PNA
LAF
Deep
PNA
1.C
Median
Kurtosts
DecoSets
DeepSets
Figure 5: Scatter plots of the MNIST experiment comparing true (x axis) and predicted (y axis) values with 50 as maximum test set size. The target aggregations are max (up-left), inverse count (up-right), median (bottom-left) and kurtosis (bottom-right).
P100 and P1000 to the two datasets. For all the settings, we consider the same architecture and hyper-parameters of the DeepSets permutation invariant model described by Zaheer al.) (2017). For LAF, we replace the original aggregation function (max) used in DeepSets with 10 LAA units, while for PNA we use the concatenation of max, min, mean, and standard deviation, as proposed by the authors. For PNA do not consider any scaler, as the cardinalities of the sets are fixed.
Results in Table 2 show that LAF produces an advantage the lower resolution dataset (i.e. on P100), while it obtains comparable (and slightly more stable) performances in the higher resolution one (i.e. on P1000). These results suggest that having predefined aggregators is not necessarily an optimal choice in real world cases, and that the flexibility of LAF in modeling diverse aggregation functions can boost performance and stability.
Table 2: Results on the Point Cloud classification task. Accuracies with standard deviations (calcu- lated on runs) for the ModelNet40 dataset.
METHOD
P100
P1000
DEEPSETS
82.0+22.05
87.0+1.055
PNA
82.910.7%
86.41006%
LSTM
78.711.11%
82.211.7%
LAF
84.011066%
87.0.1.5%
4.4 SET EXPANSION
Following the experimental setup of DeepSets, we also considered the Set Expansion task. In this task the aim is to augment set of objects of the same class with other similar objects, as explained in (Zaheer et al. 2017). The model learns to predict score for an object given a query set and decide whether to add the object to the existing set. Specifically, Zaheer et al. 2017) consider the specific application of set expansion to text concept retrieval. The idea is to retrieve words that belong to a particular concept, giving as input set a set of words having the same concept. We employ the same model and hyper-parameters of the original publication, where replace the sum- decomposition aggregation with LAF units for our methods and the min, max, mean, and standard deviation aggregators for PNA.
We trained our model on sets constructed from a vocabulary of different size, namely LDA-IK, LDA-3K and LDA-5K. Shols shows the results of LAF, DeepSets and PNA on different evaluation metrics. We report the retrieval metrics recall K, median rank and mean reciprocal rank. We also report the results other methods the authors compared to in the original paper. More details the

---

Under review as a conference paper at ICLR 2021
Results on Text Concept Set Retrieval LDA-1K, LDA-3K, and LDA-Sk. Bold values denote the best performance for each metric.
Table 3:
LDA-IR I& (VOCAB 17R)
LDA-3k (VoCAB 38k)
LDA-5k (VOCAD 61k)
METHOD
RECALL %)
RECALL 9)
RECALL($) @100
G 10
@000 @1k
MRR MED.
@100 @1K
MRR MED.
@ 10
@1K
MRR MED.
RANDOM
0.06
0.6
5.9
0.001
8520
0.02
0.2
2.6
0.000 28635
0.01
0.01
0.2
1.6
0,000 30600
37.2
0.007
3234
12.5
3590
w2v NEAR
6.00
28.1
54.7
0.021
641
4.80
21.2
43.2 0.016
2054
4.03
16.7
35.2
0.013 6900
NN-MAX
4.78
22.5
53.1
0.023
779
5.30
24.9
54.8
0.025
672
4.72
21.4
47.0
0.022
1320
NN SUM CON
4.58
19.8
48.5
0.021
1110
5.81
27.2
60.0
0.027
453
4.87
23.5
53.9
0.022
731
NN-MAX-CON
3.36
16.9
46.6
0.018
1250
5.61
25.7
57.5 0.026
570
4.72
22.0
51.8
0.022
877
DEEPSETS
5.53
24.2
54.3
0.025
426
5.54
26.1
55.5
0.026
616
DEEPSETS"
5.89
26.0
0.026
7.56
349
6.49
27.9
0.030
536
PNA
5.56
24.7
53.2
0.027
753
7.04
27.2
58.7
0.028
502
5.47
23.8
52.4
0.025
807
LSTM
4.29
21.5
52.6
0.022
690
5.56
25.7
58.8
0.026
830
4.87
23.8
55.0
0.022
672
LAF
6.51
26.6
54.5
0.030
650
8.14
32.3
0.037
339
6.71
28.3
56.9
0.031
523
other methods in the table can be found in the original publication. Briefly, Random samples word uniformly from the vocabulary; Bayes Set (Ghahramani & Heller 2006); w2v-Near computes the nearest neighbors in the word2vec (Mikolov et al.) 2013) space; NN-max uses a similar architecture as our DeepSets but uses max pooling to compute the set feature, as opposed to sum pooling: NN- max-con uses pooling query for a final set feature; NN-sum-con is similar to NN-max-con but uses sum pooling followed by concatenation with query representation. For the sake of fairness, have rerun DeepSets using the current implementation from the authors (indicated as DeepSet" in Table 3). exhibiting better results than the ones reported in the original paper. Nonetheless, LAF outperforms all other methods in most cases, especially on LDA-3K and LDA-5K.
set elements but concatenates this pooled representation with that of
4.5 MULTI-TASK GRAPH PROPERTIES
Corso et al. 2020) defines a benchmark consisting of classical graph theory tasks on artificially generated graphs from a wide range of popular graph types like Prdos-RRnyi, Barabasi- or star-shaped graphs. Three of the tasks are defined for nodes, while the other three for whole graphs. The node tasks are the single-source shortest-path lengths (NI), the eccentricity (N2) and the Lapla- cian features (N3). The graph tasks are graph connectivity (GI), diameter (G2), and the spectral radius (G3), For more details about the experimental settings please refer to Corso et al. (2020).
Table 4: Results on the Multi-task graph properties prediction benchmark. Results are expressed in log 10 of mean squared enror.
METHOD
N1
N2
N3
G1
G2
G3
BASELINE
-1.87 -.500-1.6.60
-0.62
-1.0
-1.41
GIN
2.00-1..0-1.60-1..6
-2.17
-2.66
GCN
22.1-1189-160-1.692..144
GAT
2.54-2..0-1.0--2.4444
MPNN (MAX)
MPNN (SUM)
2.36--2.16-2.59
2.54-22-22--
PNA (NO SCALERS)
-2.54-2.42
-2.4 -2.1 2.82-228333
PNA
-2.89
-2.99 -3.77 -3.04 -3.57
LAF
-2.13 2.13-2.200-1 -2.35 -2.77 -3.63
We compare LAF against PNA by simply replacing the original PNA aggregators and scalers with 100 LAF units (see Equation 2). Table that albeit these datasets were designed to highlight the features of the PNA architecture, that outperforms a wide range of alternative graph neural net- work approaches LAF produces competitive results, outperforming -fate-of-the-ar art GNN approaches like GIN (Xu et al. 2019), GCN (Kipf & Welling 2017) and GAT (Veličković et al. 2018) and even improving PNA on spectral radius prediction.
8

---

Under review as a conference paper at ICLR 2021
5 CONCLUSIONS
The theoretical underpinnings for sum aggregation as a universal framework for defining set func- tions do not necessarily provide a template for practical solutions. Therefore we introduced LAF, framework for learning aggregation functions that make use of parametric aggregator to ef- fectively explore a rich space of possible aggregations. LAF defines a new class of aggregation functions, which include as special cases widely used aggregators, and also has the ability to learn complex functions such as higher-order moments. We empirically showed the generalization ability of our method on synthetic settings well as real-world datasets, providing comparisons with state- of-the-art cum-dccomposition approaches and recently introduced techniques. The flexibility of our model is crucial aspect for potential practical use in many deep learning architectures, due to its ability to be easily plugged into and learned in end-to-end architectures. The portability of LAF opens new range of possible applications for aggregation functions in machine learning methods, and future research in this direction can enhance the expressivity of many architectures and models that deal with unstructured data.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
hwtgg/bb/11400 0473
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. arXiv preprint arXiv:2004.057/ 2020.
Zoubin Ghahramani and Katherine A Heller. Bayesian sets, In Advances in neural information processing systems, pp. 435 442, 2006.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long
Beach, CA, USA. pp. 1024-1034, 2017. URL http::/papers nips cc/paper/ 6703
Maximilian Ilse, Jakub M. Tomezak, and Max Welling. Attention-based deep multiple instance learning. In Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmssmässan, Stockholm, Sweden, July 10-15, 2018, pp. 2132-2144 2018. URL http:
//proceedings. ml www...peaas/vee sitees/v80//1
8a html
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25- 29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACI, pp. 1746 1751, 2014. doi: 10.3115/v1/11-11111.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In Pro- ceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 3744-3753, 2019. URL http::/prroceenngss mlr. wwwppsssse/////
html
Vitalik Melnikov and Eyke Hüllermeier. Learning to aggregate using uninorms. In Machine Learn- ing and Knowledge Discovery in Databases European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part 1I, pp. 756-771, 2016. doi: 10. 1007 978-3-319-46277- 41. URL https: /ow. http:/ii.....oo..1/o/ 3111177997777-7-911------1
47

---

Under review as a conference paper at ICLR 2021
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp- 3111-3119, 2013.
Radu Bogdan Rusu and Steve Cousins. 3d is here: Point cloud library (pcl). In 2011 IEEE interna- lional conference on robotics and automation, pp. 1-4. IEEE, 2011.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4967-4976, 2017.
Alessandro Tibo, Paolo Frasconi, and Manfred Jaeger. A network architecture for multi-multi- instance learning. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part doi: 10. 0001007/98--3339--1222929 3 71249-12211 9_44
I. pp. 737-752, 2017.
URL wtwph:ttt///mmm /wo.. org/10.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural In- formation Processing Systems 30: Annual Conference on Neural Information Processing Sys- lems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL URLhttp: llpaper..iig. http:///ss.ss..cepee/e//eee//e 1ataao//1o--atttettio-----------
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR'18, 2018.
Edward Wagstaff, Fabian B Fuchs, Martin Engelcke, Ingmar Posner, and Michael Osborne. On the limitations of representing functions on sets. arXiv preprint arXiv: 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang. and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? In International Conference on Learning Representations, 2019. URL https:
//openzeveew. weeenreview.net/onnieee wet./ooormidddrreeeete.eee.e
Ronald R Yager and Alexander Rybalov. Uninorm aggregation operators. Fuzzy sets and systems, 80(1):111-10, 1996.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3391-3401. Curran Associates, Inc., 2017. URL http: paperspppaaa.aaa/ ni ps. cc cc/paper/ 6931 6931- setpppsst.......
10


Reviews:
[{"review_id": "ukmCfRiCNgL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nUniversal function representation guarantee requires either highly discontinuous mappings or a highly dimensional latent space. For this reason the authors propose a new parametric family of aggregation functions, called LAF (for learning aggregation functions). It can be seen as a smooth version of the class of functions that are shown in DeepSets. LAF aggregator could learn all standard aggregation functions. Moreover in experiments the autors shows that LAF surpasses other aggregation methods.\n\n=============================================================================\n\nPros:\n\n1. The authors shows, that all standard aggregation functions are achievable by varying the parameters in the formulation of LAF. Moreover LAF enables a neural network to use a continuum of intermediate and hybrid aggregators.\n\n2. Comprehensive ablation study that compares LAF to DeepSets and PNA on digits and MNIST images. In the study the goal is to learn a different types of target aggregation. The results shows that LAF could learn all the given types of aggregation methods as well as it could generalize well to the size of the test set (and thus is not overfitting to the size of the training set as the other methods).\n\n3. The authors provide an extensive set of experiments on a wide range of datasets, including point clouds, set expansion and graph properties. On most of the given tasks LAF is superior to other methods.\n\n=============================================================================\n\nCons:\n\nNot all the details about LAF aggregation are clear to me. The authors should consider rewriting a section 2 (with a description of the aggregation), considering the points I list below.\n\n1. In the manuscript the autors state that LAF is using the tunable parameters a,...,h and alpha,...,delta, however they do not show how to initialize these parameters and do not tell whether the model is sensitive to these values.\n\n2. The autors state that tunable parameters a,...,h are greater or equal than zero. However hey do not show how to achieve this condition. Whether they use exponent, non-linearity or do it in another way.\n\n3. In the definition of LAF aggregator, the authors states that x should be a real number, however looking at the experiments, it seems to me that it could also be applied to vectors. Please correct me if I'm wrong, but if I am right, than please answer the question whether in this situation a,...,h,alpha,...,delta are still the scalars or whether are they vecors? \n\n4. The more detailed description of builded networks should be included (even in the appendix). It is not clear to me, how the authors made thir networks. E.g. in section 4.1 they state that 'The LAF model contains nine LAF(x) aggregation functions' - does this mean that in the final aggregation layer you create 9 independently working LAF aggregators and then concatenate them and pass to final prediction layer? If yes, then what in the situation with aggregating the informations from vectors? Do you still make concatenaction?\n\n=============================================================================\n\nQuestions during rebuttal period:\n\n1. Why section 5 (Multi-task graph properties) is not the sub-section of section 4 (Experiments)?\n\n2. Usually using a sigmoid could disturb the training of neural network. Could you create the experiment (on a real dataset), where you delete the sigmoid as well as parts with 1-x form the LAF aggregator?\n\n3. Could you create some experiments with LAF as the final aggregator for neural networks? More specific, for exampe, could you use LAF instead of mean average pooling in image classification using ResNet or some text classification task?\n\n=============================================================================\n\n=============================================================================\n\nReasons for score: \n\nI vote for accepting this paper. The idea proposed by the authors is novel and elegant. Moreover experiments shows that the proposed model is superior to the other models with whom it has been compared. My major concern is about the clarity of the paper. Hopefully the authors can address my concern in the rebuttal period. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iZ7Xy--PhFF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of finding appropriate aggregation functions that can be used for instance in deep neural network architectures. One such function maps a variable length list of reals to a scalar. \n\nThe authors investigate the possibility to learn aggregation functions from data. To that end, they investigate an Lp norm based parametric model of aggregation functions called LAF that allows to learn a wide range of function including usual aggregation functions such as mean, max or min. They however restrict these functions to lists of reals that are in the unit interval. \n\nThe theoretical part of the paper is short. The authors simply show that their aggregation model has a rather high expressive power and invoke a theorem from prior arts to outline some universality guarantee (which could actually be explicitly recalled).\n\nThe limited amount of theory is compensated by extensive numerical experiments. The authors provide experimental evidence that their model generalizes fairly well to large sets of inputs. They also show that their model can be plugged to more conventional neural net layers and is backprop friendly.\n\nThis is an overall fair contribution which is well positioned as compared to prior arts. The main issue of this paper is, in my opinion, that the impact on the ML community will be limited if their LAF aggregation model does not become a gold standard like convolutional layers have become. To achieve such goal, one misses an application in which the LAF allows to bring disruptive results by leveraging features that cannot be derived through more usual architectures. In conjunction with this remark, the practical usability of LAF would be much wider if a generalization to sets of vectors would be proposed which could learn other features such as covariance.\n\nDetailed Remarks: \n\nSec 2 : If b or e is equal to zero, then L_ab is not continuous. I think these should be limiting cases too.\n\nSec 4.1 : I fail to understand to what targets input sets are mapped to: one of the feature shown in Fig. 2 or a subset of them. Similarly, I also do not understand what is the global architecture of the approaches. It seems that have 9-dimensional outputs. Please clarify this.\n\nMaybe the plots would be more informative if the y-axis was in log scale.\n\nSec 4.2 : Do you use convolutional layers before the aggregation layer ? Are they trained together ? \n\nSec 4.3 : Can you be more specific on the dataset pre-processing ? Why not directly use the raw dataset ?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fwm_mZaQpTI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a rational approximation approach for learning aggregation functions. In general the paper is well written and technically sound, although there are some questions that could help the readers better understand the paper. \n\nOne of the problems with learnable rational approximations is the potential of finding a pole, e.g., x/0, you do not mention how you avoid/use? such a situation in your approach, whether this causes instabilities during learning, etc. Could you mention something about this?\n\nRegarding the use of the sigmoid to transform the values of x to the range [0,1], it is not clear to me, how you can recover the mean with such low error, i.e., how can you achieve $\\mu(x) ~=  (\\sum sigmoid(x)^b)^a / N$. Without the sigmoid transformation it is clear as presented in table 1. But with the sigmoid transformation is not as straightforward to see, although you show very good results in Fig. 2. Furthermore the sigmoid transformation destroys the linearity needed when the values of x are larger. Could you also give some intuition on why using the sigmoid is superior compared to a minmax scaling process?\n\nYou mention that you use 9 LAF(x) aggregation functions in the experiments, could you explain more what do you mean exactly? are you using a mixture of LAF(x) models? are they independent of each other and train for the different target functions?\n\nIn general, I find the paper interesting and the results promising, just a bit more explanation could help the reader understand the benefits and intuition behind the decisions. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "CFIfcS-0F2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes a parameterized learnable aggregation function (LAF) that can aggregate a multi-set of numbers (i.e. map them to a single real-valued number). This is different to prior works such as Deep Sets that use fixed aggregation functions such as max, mean, etc.\n\nStrong Points:\n- While Deep Sets have shown that is theoretically sufficient to have a sum aggregation, it is still unclear which kind of aggregation functions work well in practice. Hence, the paper addresses an interesting research question.\n- The presented idea is rather simple, which I think is a strong point of the paper.\n- I like the analysis in Table 1 that shows how different parameterizations of the LAF correspond to different functions such as sum, min, means, etc.\n- I also like the evaluation presented in Figure 4.\n\nWeak Points:\n- The setup of the experiments with scalars is unclear. For example, a LAF is supposed to aggregate all scalars in the input. However, the paper states that the LAF model is comprised of 9 LAF functions. Why do we need 9 functions? And how are they composed into a single architecture? Similarly, the paper states that 'DeepSets contains three max units, three sum units, and three mean units'. However, a Deep Set should have one function mapping the input to an intermediate representation, a sum aggregation, and a function that maps the aggregation to the output. I don't see why the model needs three max, sum, and mean units.\n- In experiment 1, it is reported that the 'input mapping is performed by three layers with the hyperbolic tangent as non-linear activation'. However, the input is simply a multi-set of scalars. It remains unclear to me why it can make sense to map scalars with a 3-layer network. Furthermore, a sigmoid is applied in case of LAF. Hence, it is unclear if the observed performance is due to the Sigmoid or due to the LAF. Furthermore, it is unclear why the paper uses tanh as activation function while Deep Set implementations use ReLUs. Also, the architecture of the output mapping is not described.\n- Similar to experiment 1, the experimental setup in experiment 2 is unclear. Additionally, it is unclear how the aggregation of features (i.e. individual dimensions in MNIST images) is related to the aggregation function used to compute the target output of the multi-set.\n- While the problem investigated in the paper is permutation-invariant, several methods have been proposed to approximate permutation invariant problems with recurrent architectures such as LSTMs and GRUs. However, no comparison to these kinds of methods is performed. A comparison would be interesting since they also learn an aggregation function.\n- It would be interesting to see if the parameters of the LAF function are learned as expected, i.e. if they correlate with the expected values as listed in Table 1. An analysis of this question is missing.\n- The paper does not share code or data to improve the reproducibility of the experiments.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel and elegant way for learning parameterized aggregation functions and show that their approach can achieve good performance on several datasets (in many cases outperforming other state-of-the-art methods). This is also appreciated by most of the reviewers. However, there have been several issues regarding the description of the proposed approach and the conducted experiments. These have been partly resolved in the rebuttal phase but should be more carefully assessed in another iteration of reviews. 

More specifically: Experiments regarding learning of a single LAF versus multiple LAF should partly be included in the main paper (e.g. Figure 4 showing the performance for different numbers of LAFs). When constructing deep sets in this setting with a similar number of aggregation function it appears not very sensible to me to incorporate the same aggregation function multiple times but one would rather include a set of different fixed aggregation functions (these could be derived from the proposed LAFs). The experiments would also benefit from including set transformers as baselines (set transformers are discussed in the paper but not considered in the experiments as the authors argue that this is an orthogonal approach; while I agree that the goal of set transformers is different, I think there would be big value in understanding how these approaches compare and/or can be combined).

Beyond that I think 	a brief discussion of the related topic of learning pooling operations (e.g., in CNNs) is warranted. 

Some reviewers also find that their concerns are only partially addressed in the rebuttal (e.g., regarding the extension from sets to vectors and applications in which the achieved performance differences are bigger).

One point which didn’t come up in the reviews but I would want to see addressed in a future version of the paper is an extended discussion of Figure 4. While there are cases were LAF clearly performs better, there are also cases, where Deep Sets outperform (this seem to be the cases in which the used aggregation units match the considered task). As LAFs can in theory represent these aggregation function it still seems challenging to learn the correct form of the aggregation function — I would appreciate deeper insights an analysis of this aspect. An immediate heuristic solution for many applications for improving performance thus might be to combine LAFs and standard aggregators.

In summary, the submitted paper has big potential but should be carefully revised and the experiments should be extended before the paper is accepted.
Decision: Reject