Learning Aggregation Functions

1 INTRODUCTION . The need to aggregate representations is ubiquitous in deep learning . Some recent examples include max-over-time pooling used in convolutional networks for sequence classification ( Kim , 2014 ) , average pooling of neighbors in graph convolutional networks ( Kipf & Welling , 2017 ) , max-pooling in Deep Sets ( Zaheer et al. , 2017 ) , in ( generalized ) multi-instance learning ( Tibo et al. , 2017 ) and in GraphSAGE ( Hamilton et al. , 2017 ) . In all the above cases ( with the exception of LSTM-pooling in GraphSAGE ) the aggregation function is predefined , i.e. , not tunable , which may be in general a disadvantage ( Ilse et al. , 2018 ) . Sum-based aggregation has been advocated based on theoretical findings showing the permutation invariant functions can be sum-decomposed ( Zaheer et al. , 2017 ; Xu et al. , 2019 ) . However , recent results ( Wagstaff et al. , 2019 ) showed that this universal function representation guarantee requires either highly discontinuous ( and thus poorly learnable ) mappings , or a latent dimension equal to the maximum number of elements in the set . This suggests that learning set functions that are accurate on sets of large cardinality is difficult . Inspired by previous work on learning uninorms ( Melnikov & Hüllermeier , 2016 ) , we propose a new parametric family of aggregation functions that we call LAF , for learning aggregation functions . A single LAF unit can approximate standard aggregators like sum , max or mean as well as model intermediate behaviours ( possibly different in different areas of the space ) . In addition , LAF layers with multiple aggregation units can approximate higher order moments of distributions like variance , skewness or kurtosis . In contrast , other authors ( Corso et al. , 2020 ) suggest to employ a predefined library of elementary aggregators to be combined . Since LAF can represent sums , it can be seen as a smooth version of the class of functions that are shown in Zaheer et al . ( 2017 ) to enjoy universality results in representing set functions . The hope is that being smoother , LAF is more easily learnable . Our empirical findings show that this can be actually the case , especially when asking the model to generalize over large sets . In particular , in this paper we offer an extensive experimental analysis showing that : • LAF layers can learn a wide range of aggregators ( including higher-order moments ) on sets of scalars without background knowledge on the nature of the aggregation task • LAF layers on the top of traditional layers can learn the same wide range of aggregators on sets of high dimensional vectors ( MNIST images ) • LAF outperforms state-of-the-art set learning methods such as DeepSets and PNA on realworld problems involving point clouds and text concept set retrieval . • LAF performs comparably to PNA on random graph generation tasks , outperforming several graph neural networks architectures including GAT ( Veličković et al. , 2018 ) and GIN ( Xu et al. , 2019 ) The rest of this work is structured as follows . In Section 2 we define the LAF framework and show how appropriate parametrizations of LAF allow to represent a wide range of popular aggregation functions . In Section 3 we discuss some relevant related work . Section 4 reports synthetic and realworld experiments showing the advantages of LAF over ( sets of ) predifined aggregators . Finally , conclusions and pointers to future work are discussed in Section 5 . 2 THE LEARNING AGGREGATION FUNCTION FRAMEWORK . We use x = { x1 , . . . , xN } to denote finite multisets of real numbers xi ∈ R. Note that directly taking x to be a multiset , not a vector , means that there is no need to define properties like exchangeability or permutation equivariance for operations on x . An aggregation function agg is any function that returns for any multiset x of arbitrary cardinality N ∈ N a value agg ( x ) ∈ R. Standard aggregation functions like mean and max can be understood as ( normalized ) Lp-norms . We therefore build our parametric LAF aggregator around generalized Lp-norms of the form La , b ( x ) : = ( ∑ i xbi ) a ( a , b ≥ 0 ) . ( 1 ) La , b is invariant under the addition of zeros : La , b ( x ) = La , b ( x ∪ 0 ) where 0 is a multiset of zeros of arbitrary cardinality . In order to also enable aggregations that can represent conjunctive behavior such as min , we make symmetric use of aggregators of the multisets 1−x : = { 1−xi|xi ∈ x } . For La , b ( 1 − x ) to be a well-behaved , dual version of La , b ( x ) , the values in x need to lie in the range [ 0 , 1 ] . We therefore restrict the following definition of our learnable aggregation function to sets x whose elements are in [ 0 , 1 ] : LAF ( x ) : = αLa , b ( x ) + βLc , d ( 1− x ) γLe , f ( x ) + δLg , h ( 1− x ) ( 2 ) defined by tunable parameters a , . . . , h ≥ 0 , and α , . . . , δ ∈ R. In cases where sets need to be aggregated whose elements are not already bounded by 0 , 1 , we apply a sigmoid function to the set elements prior to aggregation . Table 1 shows how a number of important aggregation functions are special cases of LAF ( for values in [ 0 , 1 ] ) . We make repeated use of the fact that L0,1 returns the constant 1 . For max and min LAF only provides an asymptotic approximation in the limit of specific function parameters ( as indicated in the limits column of Table 1 ) . In most cases , the parameterization of LAF for the functions in Table 1 will not be unique . Being able to encode the powers of moments implies that e.g . the variance of x can be expressed as the difference 1/N ∑ i x 2 i − ( 1/N ∑ i xi ) 2 of two LAF aggregators . Since LAF includes sum-aggregation , we can adapt the results of Zaheer et al . ( 2017 ) and Wagstaff et al . ( 2019 ) on the theoretical universality of sum-aggregation as follows . Proposition 1 Let X ⊂ R be countable , and f a function defined on finite multisets with elements from X . Then there exist functions φ : X → [ 0 , 1 ] , ρ : R → R , and a parameterization of LAF , such that f ( x ) = ρ ( LAF ( φx ) ; α , β , γ , δ , a , b , c , d ) , where φx is the multiset { φ ( x ) |x ∈ x } . A proof in Wagstaff et al . ( 2019 ) for a very similar proposition used a mapping fromX into the reals . Our requirement that LAF inputs must be in [ 0 , 1 ] requires a modification of the proof ( contained in the supplementary material ) , which for the definition of φ relies on a randomized construction . Proposition 1 shows that we retain the theoretical universality guarantees of Zaheer et al . ( 2017 ) , while enabling a wider range of solutions based on continuous encoding and decoding functions . It should be emphasized at this point that the primary purpose of LAF is not to provide a uniform representation of different standard aggregators as displayed in Table 1 , but to enable a continuum of intermediate and hybrid aggregators . Figure 1 shows the graphs of 4 different randomly generated LAF functions over the unit square [ 0 , 1 ] × [ 0 , 1 ] , i.e. , evaluated over sets of size 2 . Parameters α , . . . , γ were randomly sampled in the interval [ 0 , 1 ] ; parameters b , d , f , h are randomly sampled from the integers 0 , . . . , 5 , and a , c , e , g are obtained as 1/i with i a random integer from 0 , . . . , 5 . The figure illustrates the rich repertoire of aggregation functions with different qualitative behaviors already for non-extreme parameter values . 2.1 LAF ARCHITECTURE . LAF can be easily used as a module of a larger architecture suitable for learning on sets . Several LAF units can be combined as shown in Figure 2 , to capture different aspects of the input set , which can be in general a set of vectors x = { x1 , . . . , xN } where xi ∈ Rd . Note that multiple aggregators are also used in related frameworks such as DeepSets ( Zaheer et al. , 2017 ) or Graph Neural Networks ( Veličković et al. , 2018 ; Corso et al. , 2020 ) . A module with r LAF units takes as input d-dimensional vectors and produces a vector of size r × d as output . Each LAF unit performs an element-wise aggregation of the vectors in the set such that Lk , j = LAF ( { xi , j , . . . , xN , j } ; αk , βk , γk , δk , ak , bk , ck , dk ) for k = 1 , . . . , r and j = 1 , . . . , d. The output vector can be then fed into the next layer . 3 RELATED WORK . Several studies address the problem of aggregating data over sets . Sum-decomposition strategies have been used in ( Zaheer et al. , 2017 ) for points cloud classification and set expansion tasks and in ( Santoro et al. , 2017 ) for question answering and dynamic physical systems computation . Max , sum and average are standard aggregation functions for node neighborhoods in graph neural networks ( Hamilton et al. , 2017 ; Kipf & Welling , 2017 ; Xu et al. , 2019 ; Veličković et al. , 2018 ) . Zaheer et al . ( 2017 ) first proved universal representation results for these standard aggregators when combined with learned mappings over inputs and results of the aggregation . However , Wagstaff et al . ( 2019 ) showed that these universality results are of little practical use , as they either require highly discontinuous mappings that would be extremely difficult to learn , or a latent dimension that is at least the size of the maximum number of input elements . Uninorms ( Yager & Rybalov , 1996 ) are a class of aggregation functions in fuzzy logic that can behave in a conjunctive , disjunctive or averaging manner depending on a parameter called neutral element . Melnikov & Hüllermeier ( 2016 ) proposed to learn fuzzy aggregators by adjusting these learnable parameters , showing promising results on combining reviewers scores on papers into an overall decision of acceptance or reject . Despite the advantage of incorporating different behaviours in one single function , uninorms present discontinuities in the regions between aggregators , making them not amenable to be utilized in fully differentiable frameworks . Furthermore the range of possible behaviours is restricted to those commonly used in the context of fuzzy-logic . The need for considering multiple candidate aggregators is advocated in a very recent work that was developed in parallel with our framework ( Corso et al. , 2020 ) . The resulting architecture , termed Principal Neighborhood Aggregation ( PNA ) combines multiple standard aggregators , including most of the ones we consider in the LAF framework , adjusting their outputs with degree scalers . However , the underlying philosophy is rather different . PNA aims at learning to select the appropriate aggregator ( s ) from a pool of candidates , while LAF explores a continuous space of aggregators that includes standard ones as extreme cases . Our experimental evaluation shows that PNA has troubles in learning aggregators that generalize over set sizes , despite having them in the pool of candidates , likely because of the quasi-combinatorial structure of its search space . On the other hand , LAF can successfully learn even the higher moment aggregators and consistently outperforms PNA . Closely connected , but somewhat complementary to aggregation operators are attention mechanisms ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) . They have been explored to manipulate set data in Lee et al . ( 2019 ) and in the context of multi-instance learning ( Ilse et al. , 2018 ) . Attention operates at the level of set elements , and aims at a transformation ( weighting ) of their representations such as to optimize a subsequent weighted sum-aggregation . While the objectives of attention-based frameworks and LAF partially overlap , they are functionally quite different . Exploring combinations of LAF with attention mechanisms is a possible subject of future work .

Summary: Universal function representation guarantee requires either highly discontinuous mappings or a highly dimensional latent space. For this reason the authors propose a new parametric family of aggregation functions, called LAF (for learning aggregation functions). It can be seen as a smooth version of the class of functions that are shown in DeepSets. LAF aggregator could learn all standard aggregation functions. Moreover in experiments the autors shows that LAF surpasses other aggregation methods.