Title: ON UNIVERSAL EQUIVARIANT SET NETWORKS

Abstract: Using deep neural networks that are either invariant or equivariant to permutations in order to learn functions on unordered sets has become prevalent. The most popular, basic models are DeepSets (Zaheer et al., 2017) and PointNet (Qi et al., 2017). While known to be universal for approximating invariant functions, DeepSets and PointNet are not known to be universal when approximating equivariant set functions. On the other hand, several recent equivariant set architectures have been proven equivariant universal (Sannai et al., 2019; Keriven & Peyré, 2019) , however these models either use layers that are not permutation equivariant (in the standard sense) and/or use higher order tensor variables which are less practical. There is, therefore, a gap in understanding the universality of popular equivariant set models versus theoretical ones. In this paper we close this gap by proving that: (i) PointNet is not equivariant universal; and (ii) adding a single linear transmission layer makes PointNet universal. We call this architecture PointNetST and argue it is the simplest permutation equivariant universal model known to date. Another consequence is that DeepSets is universal, and also PointNetSeg, a popular point cloud segmentation network (used e.g., in (Qi et al., 2017)) is universal.
The key theoretical tool used to prove the above results is an explicit characterization of all permutation equivariant polynomial layers. Lastly, we provide numerical experiments validating the theoretical results and comparing different permutation equivariant models.

1. INTRODUCTION:

Many interesting tasks in machine learning can be described by functions F that take as input a set, X = (x 1 , . . . , x n ), and output some per-element features or values, F (X) = (F (X) 1 , . . . , F (X) n ). Permutation equivariance is the property required of F so it is welldefined. Namely, it assures that reshuffling the elements in X and applying F results in the same output, reshuffled in the same manner. For example, if X = (x 2 , x 1 , x 3 , . . . , x n ) then F ( X) = (F (X) 2 , F (X) 1 , F (X) 3 , . . . , F (X) n ).

Building neural networks that are permutation equivariant by construction proved extremely useful in practice. Arguably the most popular models are DeepSets Zaheer et al. (2017) and PointNet Qi et al. (2017) . These models enjoy small number of parameters, low memory footprint and computational efficiency along with high empirical expressiveness. Although both DeepSets and PointNet are known to be invariant universal (i.e., can approximate arbitrary invariant continuous functions) they are not known to be equivariant universal (i.e., can approximate arbitrary equivariant continuous functions).

On the other hand, several researchers have suggested theoretical permutation equivariant models and proved they are equivariant universal. Sannai et al. (2019) builds a universal equivariant network by taking n copies of (n -1)-invariant networks and combines them with a layer that is not permutation invariant in the standard (above mentioned) sense. Keriven & Peyré (2019) solves a more general problem of building networks that are equivariant universal over arbitrary high order input tensors R n d (including graphs); their construction, however, uses higher order tensors as hidden variables which is of less practical value. Yarotsky (2018) proves that neural networks constructed using a finite set of invariant and equivariant polynomial layers are also equivariant universal, however his network is not explicit (i.e., the polynomials are not characterized for the equivariant case) and also of less practical interest due to the high degree polynomial layers.

In this paper we close the gap between the practical and theoretical permutation equivariant constructions and prove: Theorem 1.

(i) PointNet is not equivariant universal.

(ii) Adding a single linear transmission layer (i.e., X → 11 T X) to PointNet makes it equivariant universal.

(iii) Using ReLU activation the minimal width required for universal permutation equivariant network satisfies ω

≤ k out + k in + n+kin kin .

This theorem suggests that, arguably, PointNet with an addition of a single linear layer is the simplest universal equivariant network, able to learn arbitrary continuous equivariant functions of sets. An immediate corollary of this theorem is Corollary 1. DeepSets and PointNetSeg are universal.

PointNetSeg is a network used often for point cloud segmentation (e.g., in Qi et al. ( 2017)). One of the benefit of our result is that it provides a simple characterization of universal equivariant architectures that can be used in the network design process to guarantee universality.

The theoretical tool used for the proof of Theorem 1 is an explicit characterization of the permutation equivariant polynomials over sets of vectors in R k using power-sum multi-symmetric polynomials. We prove: Theorem 2. Let P : R n×k → R n×l be a permutation equivariant polynomial map. Then,

EQUATION

where b α = (x α 1 , . . . , x α n ) T , q α = (q α,1 , . . . , q α,l ) T , where q α,j = q α,j (s 1 , . . . , s t ), t = n+k k , are polynomials; s j (X) = n i=1 x αj i are the power-sum multi-symmetric polynomials. On the other hand every polynomial map P satisfying Equation 1 is equivariant.

This theorem, which extends Proposition 2.27 in Golubitsky & Stewart (2002) to sets of vectors using multivariate polynomials, lends itself to expressing arbitrary equivariant polynomials as a composition of entry-wise continuous functions and a single linear transmission, which in turn facilitates the proof of Theorem 1.

We conclude the paper by numerical experiments validating the theoretical results and testing several permutation equivariant networks for the tasks of set classification and regression.

2. PRELIMINARIES:

Equivariant maps. Vectors x ∈ R k are by default column vectors; 0, 1 are the all zero and all one vectors/tensors; e i is the i-th standard basis vector; I is the identity matrix; all dimensions are inferred from context or mentioned explicitly. We represent a set of n vectors in R k as a matrix X ∈ R n×k and denote X = (x 1 , x 2 , . . . , x n ) T , where

x i ∈ R k , i ∈ [n]

, are the columns of X. We denote by S n the permutation group of [n]; its action on X is defined by σ

• X = (x σ -1 (1) , x σ -1 (2) , . . . , x σ -1 (n) ) T , σ ∈ S n .

That is, σ is reshuffling the rows of X. The natural class of maps assigning a value or feature vector to every element in an input set is permutation equivariant maps:

Definition 1. A map F : R n×k → R n×l satisfying F (σ • X) = σ • F (X) for all σ ∈ S n and X ∈ R n×d is called permutation equivariant.

Power-sum multi-symmetric polynomials. Given a vector z = (z 1 , . . . , z n ) ∈ R n the powersum symmetric polynomials s j (z) = n i=1 z j i , with j ∈ [n], uniquely characterize z up to permuting its entries. In other words, for z, y ∈ R n we have y = σ • z for some σ ∈ S n if and only if s j (y) = s j (z) for all j ∈ [n]. An equivalent property is that every S n invariant polynomial p can be expressed as a polynomial in the power-sum symmetric polynomials, i.e., p(z) = q(s 1 (z), . . . , s n (z)), see Rydh (2007 ) Corollary 8.4, Briand (2004) Theorem 3. This fact was previously used in Zaheer et al. (2017) to prove that DeepSets is universal for invariant functions. We extend this result to equivariant functions and the multi-feature (sets of vectors) case.

For a vector x ∈ R k and a multi-index vector α = (α 1 , . . . , α k ) ∈ N k we define

x α = x α1 1 • • • x α k k , and |α| = i∈[k] α i .

A generalization of the power-sum symmetric polynomials to matrices exists and is called power-sum multi-symmetric polynomials, defined with a bit of notation abuse:

s α (X) = n i=1 x α i , where α ∈ N k is a multi-index satisfying |α| ≤ n.

Note that the number of power-sum multi-symmetric polynomials acting on X ∈ R n×k is t = n+k k . For notation simplicity let α 1 , . . . , α t be a list of all α ∈ N k with |α| ≤ n. Then we index the collection of power-sum multi-symmetric polynomials as s 1 , . . . , s t .

Similarly to the vector case the numbers s j (X), j ∈ [t] characterize X up to permutation of its rows. That is Y = σ • X for some σ ∈ S n iff s j (Y ) = s j (Y ) for all j ∈ [t]. Furthermore, every S n invariant polynomial p : R n×k → R can be expressed as a polynomial in the power-sum multi-symmetric polynomials (see (Rydh, 2007) corollary 8.4), i.e., p(X) = q(s 1 (X), . . . , s t (X)),

(2)

These polynomials were recently used to encode multi-sets in Maron et al. (2019) .

3. EQUIVARIANT MULTI-SYMMETRIC POLYNOMIAL LAYERS:

In this section we develop the main theoretical tool of this paper, namely, a characterization of all permutation equivariant polynomial layers. As far as we know, these layers were not fully characterized before.

Theorem 2 provides an explicit representation of arbitrary permutation equivariant polynomial maps P : R n×k → R n×l using the basis of power-sum multi-symmetric polynomials, s i (X). The particular use of power-sum polynomials s i (X) has the advantage it can be encoded efficiently using a neural network: as we will show s i (X) can be approximated using a PointNet with a single linear transmission layer. This allows approximating an arbitrary equivariant polynomial map using PointNet with a single linear transmission layer.

A version of this theorem for vectors instead of matrices (i.e., the case of k = 1) appears as Proposition 2.27 in Golubitsky & Stewart (2002) ; we extend their proof to matrices, which is the relevant scenario for ML applications as it allows working with sets of vectors. For k = 1 Theorem 2 reduces to the following form: p(x) i = a≤n p a (s 1 (x), . . . , s n (x))x a i with s j (x) = i x j i . For matrices the monomial x k i is replaced by x α i for a multi-index α and the power-sum symmetric polynomials are replaced by the power-sum multi-symmetric polynomials.

First, note that it is enough to prove Theorem 1 for l = 1 and apply it to every column of P . Hence, we deal with a vector of polynomials p : R n×k → R n and need to prove it can be expressed as p = |α|≤n b α q α , for S n invariant polynomial q α . Given a polynomial p(X) and the cyclic permutation σ -1 = (123 • • • n) the following operation, taking a polynomial to a vector of polynomials, is useful in characterizing equivariant polynomial maps:

EQUATION

Theorem 2 will be proved using the following two lemmas: Lemma 1. Let p : R n×k → R n be an equivariant polynomial map. Then, there exists a polynomial p : R n×k → R, invariant to S n-1 (permuting the last n -1 rows of X) so that p = p .

Proof. Equivariance of p means that for all σ ∈ S n it holds that σ

EQUATION

Choosing an arbitrary permutation σ ∈ stab(1) < S n , namely a permutation satisfying σ(1) = 1, and observing the first row in Equation 4we get p 1

(X) = p 1 (σ • X) = p 1 (x 1 , x σ -1 (2) , . . . , x σ -1 (n) ).

Since this is true for all σ ∈ stab(1) p 1 is S n-1 invariant. Next, applying σ = (1i) to Equation 4and observing the first row again we get p i (X) = p 1 (x i , . . . , x 1 , . . .). Using the invariance of p 1 to S n-1 we get p = p 1 .

Lemma 2. Let p : R n×k → R be a polynomial invariant to S n-1 (permuting the last n -

EQUATION

where q α are S n invariant.

Proof. Expanding p with respect to x 1 we get

EQUATION

for some m ∈ N. We first claim p α are S n-1 invariant. Indeed, note that if p(X) = p(x 1 , x 2 , . . . , x n ) is S n-1 invariant, i.e., invariant to permutations of x 2 , . . . , x n , then also its derivatives

∂ |β| ∂x β 1 p(X) are S n-1 permutation invariant, for all β ∈ N k . Taking the derivative ∂ |β| /∂x β 1 on both sides of Equation 6 we get that p β is S n-1 equivariant.

For brevity denote p = p α . Since p is S n-1 invariant it can be expressed as a polynomial in the powersum multi-symmetric polynomials, i.e., p(x 2 , . . . ,

x n ) = r(s 1 (x 2 , . . . , x n ), . . . , s t (x 2 , . . . , x n )). Note that s i (x 2 , . . . , x n ) = s i (X) -x αi

1 and therefore p(x 2 , . . . , x n ) = r(s 1 (X) -x α1 1 , . . . , s t (X) -x αt 1 ). Since r is a polynomial, expanding its monomials in s i (X) and x α 1 shows p can be expressed as p = |α|≤m x α 1 pα , where m ∈ N, and pα are S n invariant (as multiplication of invariant S n polynomials s i (X)). Plugging this in Equation 6 we get Equation 5, possibly with the sum over some n' > n. It remains to show n can be taken to be at-most n. This is proved in Corollary 5 in Briand (2004) Proof. (Theorem 2) Given an equivariant p as above, use Lemma 1 to write p = p where p(X) is invariant to permuting the last n -1 rows of X. Use Lemma 2 to write p(X) = |α|≤n x α 1 q α (X), where q α are S n invariant. We get,

p = p = |α|≤n b α q α .

The converse direction is immediate after noting that b α are equivariant and q α are invariant.

4. UNIVERSALITY OF SET EQUIVARIANT NEURAL NETWORKS:

We consider equivariant deep neural networks f : R n×kin → R n×kout ,

EQUATION

where L i : R n×ki → R n×ki+1 are affine equivariant transformations, and ν is an entry-wise nonlinearity (e.g., ReLU). We define the width of the network to be ω = max i k i ; note that this definition is different from the one used for standard MLP where the width would be nω, see e.g., Hanin & Sellke (2017) . Zaheer et al. ( 2017) proved that affine equivariant L i are of the form

EQUATION

where A, B ∈ R ki×ki+1 , and c ∈ R ki+1 are the layer's trainable parameters; we call the linear transformation X → 1 n 11 T XB a linear transmission layer. We now define the equivariant models considered in this paper: The DeepSets (Zaheer et al., 2017) architecture is Equation 7 with the choice of layers as in Equation 8. Taking B = 0 in all layers is the PointNet architecture (Qi et al., 2017) . PointNetST is an equivariant model of the form Equation 7 with layers as in Equation 8where only a single layer L i has a non-zero B. The PointNetSeg (Qi et al., 2017) architecture is PointNet composed with an invariant max layer, namely max(F (X)) j = max i∈[n] F (X) i,j and then concatenating it with the input X, i.e., [X, 1 max(F (X))], and feeding is as input to another PointNet G, that is G([X, 1 max(F (X))]).

We will prove PointNetST is permutation equivariant universal and therefore arguably the simplest permutation equivariant universal model known to date.

Universality of equivariant deep networks is defined next. Definition 2. Permutation equivariant universality 1 of a model F : R n×kin → R n×kout means that for every permutation equivariant continuous function H : R n×kin → R n×kout defined over the cube K = [0, 1] n×kin ⊂ R n×kin , and > 0 there exists a choice of m (i.e., network depth), k i (i.e., network width) and the trainable parameters of F so that H(X) -F (X) ∞ < for all X ∈ K.

Proof. (Theorem 1) Fact (i), namely that PointNet is not equivariant universal is a consequence of the following simple lemma: Lemma 3. Let h = (h 1 , . . . , h n ) T : R n → R n be the equivariant linear function defined by

h(x) = 11 T x. There is no f : R → R so that |h i (x) -f (x i )| < 1 2 for all i ∈ [n] and x ∈ [0, 1] n .

Proof. Assume such f exists. Let e 1 = (1, 0, . . . , 0) T ∈ R n . Then,

1 = |h 2 (e 1 ) -h 2 (0)| ≤ |h 2 (e 1 ) -f (0)| + |f (0) -h 2 (0)| < 1

reaching a contradiction.

To prove (ii) we first reduce the problem from the class of all continuous equivariant functions to the class of equivariant polynomials. This is justified by the following lemma. Lemma 4. Equivariant polynomials P : R n×kin → R n×kout are dense in the space of continuous equivariant functions F : R n×kin → R n×kout over the cube K.

Proof. Take an arbitrary > 0. Consider the function f ij : R n×kin → R, which denotes the (i, j)-th output entry of F . By the Stone-Weierstrass Theorem there exists a polynomial p ij : R n×kin → R such that f ij (X) -p ij (X) ∞ ≤ for all X ∈ K. Consider the polynomial map P : R n×kin → R n×kout defined by (P ) ij = p ij . P is in general not equivariant. To finish the proof we will symmetrize P :

F (X) - 1 n! σ∈Sn σ • P (σ -1 • X) ∞ = 1 n! σ∈Sn σ • F (σ -1 • X) - 1 n! σ∈Sn σ • P (σ -1 • X) ∞ = 1 n! σ∈Sn σ • F (σ -1 • X) -P (σ -1 • X) ∞ ≤ 1 n! σ∈Sn = ,

where in the first equality we used the fact that F is equivariant. This concludes the proof since

σ∈Sn σ • P (σ -1 • X) is an equivariant polynomial map.

Now, according to Theorem 2 an arbitrary equivariant polynomial P : R n×kin → R n×kout can be written as P = |α|≤n b α (X)q α (X) T , where b α (X) = x α 1 ∈ R n and q α = 1 Or just equivariant universal in short.

(q α,1 , . . . , q α,kout ) ∈ R kout are invariant polynomials. Remember that every S n invariant polynomial can be expressed as a polynomial in the t = n+kin kin power-sum multi-symmetric polynomials s j (X) = 1 n n i=1 x αj i , j ∈ [t] (we use the 1/n normalized version for a bit more simplicity later on). We can therefore write P as composition of three maps:

EQUATION

where B : R n×kin → R n×t is defined by

B(X) = (b(x 1 ), . . . , b(x n )) T ,

b(x) = (x α1 , . . . , x αt ); L is defined as in Equation 8with B = [0, I] and A = [e 1 , . . . , e kin , 0], where I ∈ R t×t the identity matrix and e i ∈ R t represents the standard basis (as-usual). We assume

α j = e j ∈ R kin , for j ∈ [k in ]. Note that the output of L is of the form L(B(X)) = (X, 1s 1 (X), 1s 2 (X), . . . , 1s t (X)).

Finally, Q : R n×(kin+t) → R n×kout is defined by Q(X, 1s 1 , . . . , 1s t ) = (q(x 1 , s 1 , . . . , s t ), . . . , q(x n , s 1 , . . . , s t )) T , and q(x, s 1 , . . . , s t ) = |α|≤n x α q α (s 1 , . . . , s t ) T .

Figure 1 : The construction of the universal network (PointNetST).

The decomposition in Equation 9of P suggests that replacing Q, B with Multi-Layer Perceptrons (MLPs) would lead to a universal permutation equivariant network consisting of PointNet with a single linear transmission layer, namely PointNetST.

The F approximating P will be defined as

EQUATION

where Φ : R n×kin → R n×t and Ψ : R n×(t+kin) → R n×kout are both of PointNet architecture, namely there exist MLPs φ : R kin → R t and ψ : R t+kin → R kout so that Φ(X) = (φ(x 1 ), . . . , φ(x n )) T and Ψ(X) = (ψ(x 1 ), . . . , ψ(x n )) T . See Figure 1 for an illustration of F .

To build the MLPs φ ,ψ we will first construct ψ to approximate q, that is, we use the universality of MLPS (see (Hornik, 1991; Sonoda & Murata, 2017; Hanin & Sellke, 2017) ) to construct ψ so that ψ(x, s 1 , . . . , s t ) -q(x, s 1 , . . . , s t ) ∞ < 2 for all (x, s 1 , . . . , s t ) ∈ [0, 1] kin+t . Furthermore, as

ψ over [0, 1] kin+t is uniformly continuous, let δ be such that if z, z ∈ [0, 1] kin+t , z -z ∞ < δ then ψ(z) -ψ(z ) ∞ < 2

. Now, we use universality again to construct φ approximating b, that is we take φ so that φ kin+t) . Therefore by construction of ψ we have err 2 < 2 .

(x) -b(x) ∞ < δ for all x ∈ [0, 1] kin . F (X) -P (X) ∞ ≤ Ψ(L(Φ(X))) -Ψ(L(B(X))) ∞ + Ψ(L(B(X))) -Q(L(B(X))) ∞ = err 1 + err 2 First, L(Φ(X)) -L(B(X)) ∞ < δ for all X ∈ K and therefore err 1 < 2 . Second, note that if X ∈ K then B(X) ∈ [0, 1] n×t and L(B(X)) ∈ [0, 1] n×(

To prove (iii) we use the result in Hanin & Sellke (2017) (see Theorem 1) bounding the width of an MLP approximating a function f : [0, 1] din → R dout by d in + d out . Therefore, the width of the MLP φ is bounded by k in + t, where the width of the MLP ψ is bounded by t + k in + k out , proving the bound.

We can now prove Cororllary 1.

Proof. (Corollary 1):

The fact that the DeepSets model is equivariant universal is immediate. Indeed, The PointNetST model can be obtained from the DeepSets model by setting B = 0 in all but one layer, with B as in Equation 8.

For the PointNetSeg model note that by Theorem 1 in Qi et al. ( 2017) every invariant function f : R n×kin → R t can be approximated by a network of the form Ψ(max(F (X))), where (F (X) is a PointNet model and Ψ is an MLP. In particular, for every ε > 0 there exists such F , Φ for which Ψ(max(F (X))) -(s 1 (X), . . . , s t (X)) ∞ < ε for every X ∈ [0, 1] n×kin where s 1 (X), . . . , s t (X) are the power-sum multi-symmetric polynomials. It follows that we can use PointNetSeg to approximate 1(s 1 (X), . . . , s t (X)). The rest of the proof closely resembles the proof of Theorem 1.

Graph neural networks with constructed adjacency. One approach sometimes applied to learning from sets of vectors is to define an adjacency matrix (e.g., by thresholding distances of node feature vectors) and apply a graph neural network to the resulting graph (e.g., Wang et al. (2019) , Li et al. (2019) ). Using the common message passing paradigm (Gilmer et al., 2017) in this case boils to layers of the form:

L(X) i = ψ(x i , j∈Ni φ(x i , x j )) = ψ(x i , j∈[n] N (x i , x j )φ(x i , x j )),

where φ, ψ are MLPs, N i is the index set of neighbors of node i, and N (x i , x j ) is the indicator function for the edge (i, j). If N can be approximated by a continuous function, which is the case at least in the L 2 sense for a finite set of vectors, then since L is also equivariant it follows from Theorem 1 that such a network can be approximated (again, at least in L 2 norm) arbitrarily well by any universal equivariant network such as PointNetST or DeepSets.

We tested the ability of a DeepSets model with varying depth and width to approximate a single graph convolution layer. We found that a DeepSets model with a small number of layers can approximate a graph convolution layer reasonably well. For details see Appendix A.

5. EXPERIMENTS:

We conducted experiments in order to validate our theoretical observations. We compared the results of several equivariant models, as well as baseline (full) MLP, on three equivariant learning tasks: a classification task (knapsack) and two regression tasks (squared norm and Fiedler eigen vector). For all tasks we compare results of 7 different models: DeepSets, PointNet, PointNetSeg, PointNetST, PointNetQT and GraphNet. PointNetQT is PointNet with a single quadratic equivariant transmission layer as defined in Appendix B. GraphNet is similar to the graph convolution network in Kipf & Welling (2016) and is defined explicitly in Appendix B. We generated the adjacency matrices for GraphNet by taking 10 nearest neighbors of each set element. In all experiments we used a network of the form Equation 7 with m = 6 depth and varying width, fixed across all layers.2 

Equivariant classification. For classification, we chose to learn the multidimensional knapsack problem, which is known to be NP-hard. We are given a set of 4-vectors, represented by X ∈ R n×4 , and our goal is to learn the equivariant classification function f : R n×4 → {0, 1} n defined by the following optimization problem:

f (X) = arg max z n i=1 x i1 z i s.t. n i=1 x ij z i ≤ w j , j = 2, 3, 4 z i ∈ {0, 1} , i ∈ [n]

Intuitively, given a set of vectors X ∈ R n×4 , (X) ij = x ij , where each row represents an element in a set, our goal is to find a subset maximizing the value while satisfying budget constraints. The first column of X defines the value of each element, and the three other columns the costs.

To evaluate the success of a trained model we record the percentage of sets for which the predicted subset is such that all the budget constrains are satisfied and the value is within 10% of the optimal value. In Appendix C we detail how we generated this dataset.

Equivariant regression. The first equivariant function we considered for regression is the function Hanin & Sellke (2017) showed this function cannot be approxi- As can be seen in the graphs, in all three datasets the equivariant universal models (PointNetST, PointNetQT , DeepSets, PointNetSeg) achieved comparable accuracy. PointNet, which is not equivariant universal, consistently achieved inferior performance compared to the universal models, as expected by the theory developed in this paper. The non-equivariant MLP, although universal, used the same width (i.e., same number of parameters) as the equivariant models and was able to over-fit only on one train set (the quadratic function); its performance on the test sets was inferior by a large margin to the equivariant models. We also note that in general the GraphNet model achieved comparable results to the equivariant universal models but was still outperformed by the DeepSets model.

f (X) = 1 n i=1 k j=1 (X i,j -1 2 ) 2 .

An interesting point is that although the width used in the experiments in much smaller than the bound k out + k in + n+kin kin established by Theorem 1, the universal models are still able to learn well the functions we tested on. This raises the question of the tightness of this bound, which we leave to future work.

6. CONCLUSIONS:

In this paper we analyze several set equivariant neural networks and compare their approximation power. We show that while vanilla PointNet (Qi et al., 2017) is not equivariant universal, adding a single linear transmission layer makes it equivariant universal. Our proof strategy is based on a characterization of polynomial equivariant functions. As a corollary we show that the DeepSets model (Zaheer et al., 2017) and PointNetSeg (Qi et al., 2017) 

A APPROXIMATING GRAPH CONVOLUTION LAYER WITH DEEPSETS:

To test the ability of an equivariant universal model to approximate a graph convolution layer, we conducted an experiment where we applied a single graph convolution layer (see Appendix B for a full description of the graph convolution layers used in this paper) with 3 in features and 10 out features. We constructed a knn graph by taking 10 neighbors. We sampled 1000 examples in R 100×3 i.i.d from a N ( 1 2 , 1) distribution (per entry of X). The results are summarized in Figure 3 . We regressed to the output of a graph convolution layer using the smooth L 1 loss. .

In the paper we tested PointNetQT, an architecture that adds to PointNet a single quadratic equivariant layer. We opted to use only the quadratic transmission operators: For a matrix X ∈ R n×k we define L(X) ∈ R n×k as follows:

L(X) = XW 1 + 11 T XW 2 + (11 T X) (11 T X)W 3 + (X X)W 4 + (11 T X) XW 5 ,

where is a point-wise multiplication and W i ∈ R n×k , i ∈ [5] are the learnable parameters.

:

The code can be found at https://github.com/NimrodSegol/On-Universal-Equivariant-Set-Networks

Figures and Tables:

Figure2: Classification and regression tasks with permutation equivariant models. All the universal equivariant models perform similarly, while the equivariant non-universal PointNet demonstrates reduced performace consistently; MLP baseline (with the same number of parameters as the equivariant models) performs poorly.

Figure3: Using DeepSets to regress to a single graph convolution layer. In the left image we varied the depth and took a constant width of 200. In the right we varied the width and took a fixed depth of 6 layers. The y axes are in log scale. Note that even a 2 layer DeepSets can approximate a graph convolution layer up to an error of ∼ 0.01.

are equivariant universal. Experimentally, we tested the different models on several classification and regression tasks finding that adding a single linear transmitting layer to PointNet makes a significant positive impact on performance. Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015. Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. arXiv preprint arXiv:1804.10306, 2018. Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in neural information processing systems, pp. 3391-3401, 2017.

Table content: <table/>

Peer review (iclr2020):
TLDR: The function these deep set networks can approximate is too limited to call these networks universal equivariant set networks. Authors should scope the paper to the specific function family these networks can approximate. No baseline comparison with GraphNets. The paper proposes theoretical analysis on a set of networks that process features independently through MLPs + global aggregation operations. However, the function of interest is limited to a small family of affine equivariant transformations. A more general function is where is the set of index of neighbors within the set . It is trivial to show that this function is permutation equivariant. Then, can the function family the authors used in the paper approximate this function? No. Can the proposed permutation equivariant function represent all function the authors used in the paper? Yes. 1) If , then the proposed function becomes MLP. 2) If and , then this is , the global aggregation function. Also, this is the actual function that a lot of people are interested in. Let me go over few more examples. 3) If adjacency on a graph and , then this is a graph neural network *convolution* (it is not a convolution) Example adjacency . 4) If where is the color, is the pixel coordinate and pixel neighbors within some kernel size, to be the block diagonal matrix only for the first three dimensions and 0 for the rest, then this is the 2D convolution. Again, the above function is a more general permutation equivariant function that can represent: a graph neural network layer, a convolution, MLP, global pooling and is one of the most widely used functions in the ML community, not MLP + global aggregation. Regarding the experiment metrics and plots: On the Knapsack test, the metric of interest is not the accuracy of individual prediction. Rather, whether the network has successfully predicted the optimal solution, or how close the prediction is to the solution. For example: success rate within the epsilon radius of the optimal solution while satisfying all the constraints. Fail otherwise. If these networks can truly solve these problems, authors should report the success rate while varying the threshold, not individual accuracy of the items which can be arbitrarily high by violating constraints. Also, the authors should compare with few more graphnet + transmission layer (GraphNetST) baselines with the graph layers: and the same single transmission layer in PointNetST. PointNet is a specialization of graphnets and GraphNetST should be added as a baseline with reasonable adjacency. Also experiment figures are extremely compact. Try using log scale or other lines to make the gaps wider. Minor I am quite confused with the name PointNetST. Authors claim adding one layer of DeepSet layer to a PointNet becomes PointNetST, but I see this as a special DeepSet with a single transmission layer. The convention is B -> B*, not A + B -> A*. In this case, A: PointNet, B: DeepSet Lemma 3 is too trivial. The paper is not very self contained. Spare few lines of equations to define what are DeepSets and PointNetSeg in the paper and point out the difference since these networks are used throughout the paper extensively without proper mathematical definition. P.2 power sum multi-symmetric polynomials. *For a vector and a multi-index ...* I think it was moved out of the next paragraph since the same is defined again as again in the next sentence. Also, try using the consistent dimension for x throughout the paper, it confuses the reader.