On Universal Equivariant Set Networks

1 INTRODUCTION . Many interesting tasks in machine learning can be described by functions F that take as input a set , X = ( x1 , . . . , xn ) , and output some per-element features or values , F ( X ) = ( F ( X ) 1 , . . . , F ( X ) n ) . Permutation equivariance is the property required of F so it is welldefined . Namely , it assures that reshuffling the elements in X and applying F results in the same output , reshuffled in the same manner . For example , if X̃ = ( x2 , x1 , x3 , . . . , xn ) then F ( X̃ ) = ( F ( X ) 2 , F ( X ) 1 , F ( X ) 3 , . . . , F ( X ) n ) . Building neural networks that are permutation equivariant by construction proved extremely useful in practice . Arguably the most popular models are DeepSets Zaheer et al . ( 2017 ) and PointNet Qi et al . ( 2017 ) . These models enjoy small number of parameters , low memory footprint and computational efficiency along with high empirical expressiveness . Although both DeepSets and PointNet are known to be invariant universal ( i.e. , can approximate arbitrary invariant continuous functions ) they are not known to be equivariant universal ( i.e. , can approximate arbitrary equivariant continuous functions ) . On the other hand , several researchers have suggested theoretical permutation equivariant models and proved they are equivariant universal . Sannai et al . ( 2019 ) builds a universal equivariant network by taking n copies of ( n− 1 ) -invariant networks and combines them with a layer that is not permutation invariant in the standard ( above mentioned ) sense . Keriven & Peyré ( 2019 ) solves a more general problem of building networks that are equivariant universal over arbitrary high order input tensors Rnd ( including graphs ) ; their construction , however , uses higher order tensors as hidden variables which is of less practical value . Yarotsky ( 2018 ) proves that neural networks constructed using a finite set of invariant and equivariant polynomial layers are also equivariant universal , however his network is not explicit ( i.e. , the polynomials are not characterized for the equivariant case ) and also of less practical interest due to the high degree polynomial layers . In this paper we close the gap between the practical and theoretical permutation equivariant constructions and prove : Theorem 1 . ( i ) PointNet is not equivariant universal . ( ii ) Adding a single linear transmission layer ( i.e. , X 7→ 11TX ) to PointNet makes it equivariant universal . ( iii ) Using ReLU activation the minimal width required for universal permutation equivariant network satisfies ω ≤ kout + kin + ( n+kin kin ) . This theorem suggests that , arguably , PointNet with an addition of a single linear layer is the simplest universal equivariant network , able to learn arbitrary continuous equivariant functions of sets . An immediate corollary of this theorem is Corollary 1 . DeepSets and PointNetSeg are universal . PointNetSeg is a network used often for point cloud segmentation ( e.g. , in Qi et al . ( 2017 ) ) . One of the benefit of our result is that it provides a simple characterization of universal equivariant architectures that can be used in the network design process to guarantee universality . The theoretical tool used for the proof of Theorem 1 is an explicit characterization of the permutation equivariant polynomials over sets of vectors in Rk using power-sum multi-symmetric polynomials . We prove : Theorem 2 . Let P : Rn×k → Rn×l be a permutation equivariant polynomial map . Then , P ( X ) = ∑ |α|≤n bαq T α , ( 1 ) where bα = ( xα1 , . . . , x α n ) T , qα = ( qα,1 , . . . , qα , l ) T , where qα , j = qα , j ( s1 , . . . , st ) , t = ( n+k k ) , are polynomials ; sj ( X ) = ∑n i=1 x αj i are the power-sum multi-symmetric polynomials . On the other hand every polynomial map P satisfying Equation 1 is equivariant . This theorem , which extends Proposition 2.27 in Golubitsky & Stewart ( 2002 ) to sets of vectors using multivariate polynomials , lends itself to expressing arbitrary equivariant polynomials as a composition of entry-wise continuous functions and a single linear transmission , which in turn facilitates the proof of Theorem 1 . We conclude the paper by numerical experiments validating the theoretical results and testing several permutation equivariant networks for the tasks of set classification and regression . 2 PRELIMINARIES . Equivariant maps . Vectors x ∈ Rk are by default column vectors ; 0,1 are the all zero and all one vectors/tensors ; ei is the i-th standard basis vector ; I is the identity matrix ; all dimensions are inferred from context or mentioned explicitly . We represent a set of n vectors in Rk as a matrixX ∈ Rn×k and denoteX = ( x1 , x2 , . . . , xn ) T , where xi ∈ Rk , i ∈ [ n ] , are the columns ofX . We denote by Sn the permutation group of [ n ] ; its action onX is defined by σ ·X = ( xσ−1 ( 1 ) , xσ−1 ( 2 ) , . . . , xσ−1 ( n ) ) T , σ ∈ Sn . That is , σ is reshuffling the rows of X . The natural class of maps assigning a value or feature vector to every element in an input set is permutation equivariant maps : Definition 1 . A map F : Rn×k → Rn×l satisfying F ( σ ·X ) = σ · F ( X ) for all σ ∈ Sn and X ∈ Rn×d is called permutation equivariant . Power-sum multi-symmetric polynomials . Given a vector z = ( z1 , . . . , zn ) ∈ Rn the powersum symmetric polynomials sj ( z ) = ∑n i=1 z j i , with j ∈ [ n ] , uniquely characterize z up to permuting its entries . In other words , for z , y ∈ Rn we have y = σ · z for some σ ∈ Sn if and only if sj ( y ) = sj ( z ) for all j ∈ [ n ] . An equivalent property is that every Sn invariant polynomial p can be expressed as a polynomial in the power-sum symmetric polynomials , i.e. , p ( z ) = q ( s1 ( z ) , . . . , sn ( z ) ) , see Rydh ( 2007 ) Corollary 8.4 , Briand ( 2004 ) Theorem 3 . This fact was previously used in Zaheer et al . ( 2017 ) to prove that DeepSets is universal for invariant functions . We extend this result to equivariant functions and the multi-feature ( sets of vectors ) case . For a vector x ∈ Rk and a multi-index vector α = ( α1 , . . . , αk ) ∈ Nk we define xα = xα11 · · ·x αk k , and |α| = ∑ i∈ [ k ] αi . A generalization of the power-sum symmetric polynomials to matrices exists and is called power-sum multi-symmetric polynomials , defined with a bit of notation abuse : sα ( X ) = ∑n i=1 x α i , where α ∈ Nk is a multi-index satisfying |α| ≤ n. Note that the number of power-sum multi-symmetric polynomials acting onX ∈ Rn×k is t = ( n+k k ) . For notation simplicity let α1 , . . . , αt be a list of all α ∈ Nk with |α| ≤ n. Then we index the collection of power-sum multi-symmetric polynomials as s1 , . . . , st . Similarly to the vector case the numbers sj ( X ) , j ∈ [ t ] characterize X up to permutation of its rows . That is Y = σ · X for some σ ∈ Sn iff sj ( Y ) = sj ( Y ) for all j ∈ [ t ] . Furthermore , every Sn invariant polynomial p : Rn×k → R can be expressed as a polynomial in the power-sum multi-symmetric polynomials ( see ( Rydh , 2007 ) corollary 8.4 ) , i.e. , p ( X ) = q ( s1 ( X ) , . . . , st ( X ) ) , ( 2 ) These polynomials were recently used to encode multi-sets in Maron et al . ( 2019 ) . 3 EQUIVARIANT MULTI-SYMMETRIC POLYNOMIAL LAYERS . In this section we develop the main theoretical tool of this paper , namely , a characterization of all permutation equivariant polynomial layers . As far as we know , these layers were not fully characterized before . Theorem 2 provides an explicit representation of arbitrary permutation equivariant polynomial maps P : Rn×k → Rn×l using the basis of power-sum multi-symmetric polynomials , si ( X ) . The particular use of power-sum polynomials si ( X ) has the advantage it can be encoded efficiently using a neural network : as we will show si ( X ) can be approximated using a PointNet with a single linear transmission layer . This allows approximating an arbitrary equivariant polynomial map using PointNet with a single linear transmission layer . A version of this theorem for vectors instead of matrices ( i.e. , the case of k = 1 ) appears as Proposition 2.27 in Golubitsky & Stewart ( 2002 ) ; we extend their proof to matrices , which is the relevant scenario for ML applications as it allows working with sets of vectors . For k = 1 Theorem 2 reduces to the following form : p ( x ) i = ∑ a≤n pa ( s1 ( x ) , . . . , sn ( x ) ) x a i with sj ( x ) = ∑ i x j i . For matrices the monomial xki is replaced by x α i for a multi-index α and the power-sum symmetric polynomials are replaced by the power-sum multi-symmetric polynomials . First , note that it is enough to prove Theorem 1 for l = 1 and apply it to every column of P . Hence , we deal with a vector of polynomials p : Rn×k → Rn and need to prove it can be expressed as p = ∑ |α|≤n bαqα , for Sn invariant polynomial qα . Given a polynomial p ( X ) and the cyclic permutation σ−1 = ( 123 · · ·n ) the following operation , taking a polynomial to a vector of polynomials , is useful in characterizing equivariant polynomial maps : dpe ( X ) = p ( X ) p ( σ ·X ) p ( σ2 ·X ) ... p ( σn−1 ·X ) ( 3 ) Theorem 2 will be proved using the following two lemmas : Lemma 1 . Let p : Rn×k → Rn be an equivariant polynomial map . Then , there exists a polynomial p : Rn×k → R , invariant to Sn−1 ( permuting the last n− 1 rows ofX ) so that p = dpe . Proof . Equivariance of p means that for all σ ∈ Sn it holds that σ · p ( X ) = p ( σ ·X ) σ · p ( X ) = p ( σ ·X ) . ( 4 ) Choosing an arbitrary permutation σ ∈ stab ( 1 ) < Sn , namely a permutation satisfying σ ( 1 ) = 1 , and observing the first row in Equation 4 we get p1 ( X ) = p1 ( σ ·X ) = p1 ( x1 , xσ−1 ( 2 ) , . . . , xσ−1 ( n ) ) . Since this is true for all σ ∈ stab ( 1 ) p1 is Sn−1 invariant . Next , applying σ = ( 1i ) to Equation 4 and observing the first row again we get pi ( X ) = p1 ( xi , . . . , x1 , . . . ) . Using the invariance of p1 to Sn−1 we get p = dp1e . Lemma 2 . Let p : Rn×k → R be a polynomial invariant to Sn−1 ( permuting the last n− 1 rows of X ) then p ( X ) = ∑ |α|≤n xα1 qα ( X ) , ( 5 ) where qα are Sn invariant . Proof . Expanding p with respect to x1 we get p ( X ) = ∑ |α|≤m xα1 pα ( x2 , . . . , xn ) , ( 6 ) for some m ∈ N. We first claim pα are Sn−1 invariant . Indeed , note that if p ( X ) = p ( x1 , x2 , . . . , xn ) is Sn−1 invariant , i.e. , invariant to permutations of x2 , . . . , xn , then also its derivatives ∂ |β| ∂xβ1 p ( X ) are Sn−1 permutation invariant , for all β ∈ Nk . Taking the derivative ∂|β|/∂xβ1 on both sides of Equation 6 we get that pβ is Sn−1 equivariant . For brevity denote p = pα . Since p is Sn−1 invariant it can be expressed as a polynomial in the powersum multi-symmetric polynomials , i.e. , p ( x2 , . . . , xn ) = r ( s1 ( x2 , . . . , xn ) , . . . , st ( x2 , . . . , xn ) ) . Note that si ( x2 , . . . , xn ) = si ( X ) − xαi1 and therefore p ( x2 , . . . , xn ) = r ( s1 ( X ) − xα11 , . . . , st ( X ) − x αt 1 ) . Since r is a polynomial , expanding its monomials in si ( X ) and xα1 shows p can be expressed as p = ∑ |α|≤m′ x α 1 p̃α , where m ′ ∈ N , and p̃α are Sn invariant ( as multiplication of invariant Sn polynomials si ( X ) ) . Plugging this in Equation 6 we get Equation 5 , possibly with the sum over some n ’ > n. It remains to show n′ can be taken to be at-most n. This is proved in Corollary 5 in Briand ( 2004 ) Proof . ( Theorem 2 ) Given an equivariant p as above , use Lemma 1 to write p = dpe where p ( X ) is invariant to permuting the last n− 1 rows ofX . Use Lemma 2 to write p ( X ) = ∑ |α|≤n x α 1 qα ( X ) , where qα are Sn invariant . We get , p = dpe = ∑ |α|≤n bαqα . The converse direction is immediate after noting that bα are equivariant and qα are invariant .

Summary: The paper presents proof that the DeepSets and a variant of PointNet are universal approximators for permutation equivariant functions. The proof uses an expression for equivariant polynomials and the universality of MLP. It then shows that the proposed expression in terms of power-sum polynomials can be constructed in PointNet using a minimal modification to the architecture, or using DeepSets, therefore proving the universality of such deep models.