On the Universality of Rotation Equivariant Point Cloud Networks
Authors: Nadav Dym
Haggai Maron
Venue: ICLR 2021 Poster (2021)
Keywords: 3D deep learning
Rotation invariance
Invariant and equivariant deep networks
Universal approximation
Point clouds

Abstract: Learning functions on point clouds has applications in many fields, including computer vision, computer graphics, physics, and chemistry. Recently, there has been a growing interest in neural architectures that are invariant or equivariant to all three shape-preserving transformations of point clouds: translation, rotation, and permutation. In this paper, we present a first study of the approximation power of these architectures. We first derive two sufficient conditions for an equivariant architecture to have the universal approximation property, based on a novel characterization of the space of equivariant polynomials. We then use these conditions to show that two recently suggested models, Tensor field Networks and SE3-Transformers, are universal, and for devising two other novel universal architectures.

Published as a conference paper at ICLR 2021
ON THE UNIVERSALITY OF ROTATION EQUIVARIANT POINT CLOUD NETWORKS
Naday Dym
Haggai Maron
Duke University na da ddyymm@ gma 1 com
NVIDIA Research hma
ABSTRACT
Learning functions point clouds has applications in many fields, including com- puter vision, computer graphics, physics, and chemistry. Recently, there has been growing interest in neural architectures that invariant equivariant to all three shape-preserving transformations of point clouds: translation, rotation, and permutation. In this paper, present first study of the approximation power of these architectures. We first derive two sufficient conditions for an equivariant architecture have the universal approximation property, based acterization of the space of equivariant polynomials. We then use these conditions show that two recently suggested models (Thomas et al. 2018) Fuchs al. 2020) are universal, and for devising two other novel universal architectures.
novel char-
INTRODUCTION
Designing neural networks that respect data symmetry is powerful approach for obtain- ing efficient deep models. Prominent examples being convolutional networks which re- spect the translational invariance of images, graph neural networks which respect the per- mutation invariance of graphs (Gilmer et al., [017] Maron et al., (019b). networks such (Zaheer et al.) 2017: Qi al. 2017a) which respect the permutation invariance of
sets, and networks which respect 3D rotational symmetries (Cohen et al.] 2018) Weiler al., 2018 Esteves al.] 2018 Worrall & Brostow] 2018 Kondor al. 2018a).
While the expressive power of equivariant models is re-
duced by design to include only equivariant functions, desirable property of equivariant networks is universal- ity: the ability to approximate any continuous equivariant function. This is not always the case: while convolutional networks and networks for sets are universal (Yarotsky 2018 Segol & Lipman| 2019). popular graph neural net- works are not (Xu et al. 2019; Morris et al. 2018).
Continuous functions
Equivariant functions
Cap
Equivariant networks
In this paper, consider the universality of networks that respect the symmetries 3D point clouds: translations, rotations, and permutations. Design- ing such networks is popular paradigm in recent years (Thomas al. 2018 Fuchs et al.) 2020 Poulenard et al.) 2019; Zhao et al. 2019). While there have been many works on the universality of permutation invariant networks (Zaheer et al. 2017: Maron et al.) 2019c Keriven & Peyré 2019). and a recent work discussing the universality of rotation equivariant networks (Bogatskiy et al.) 2020), this first paper which discusses the universality of networks which combine rotations, permutations and translations.
We start the paper with general, mreittecuure-gnnstie discussion, and derive two sufficient con- ditions for universality. These conditions are a result of a novel characterization of equivariant polynomials for the symmetry group interest. We use these conditions in order to prove univer- sality of the prominent Tensor Field Networks (TFN) architecture (Thomas et al.) 2018 Fuchs et al.) 2020). The following is a weakened and simplified statement of Theorem stated later on in the paper:
Theorem (Simplification of Theorem 2
Any continuous equivariant function on point clouds can be approximated uniformly compact by a composition of TFN layers.

---

Published as a conference paper at ICLR 2021
We use general discussion prove the universality of two additional equivariant models: the first is a simple modification of the TFN architecture which allows for universality using only low dimensional filters. The second is a minimal architecture which is based on tensor product represen- tations, rather than the more commonly used irreducible representations of SO(3). We discuss the advantages and disadvantages of both approaches.
To summarize, the contributions of this paper are: (1) A general approach for proving the universal- ity of rotation equivariant models for point clouds; (2) A proof that two recent equivariant models
TTomas et al. 2018: Fuchs et al. 2020) are universal; (3) Two additional simple and novel universal architectures.
2
PREVIOUS WORK
Deep learning on point clouds. (Qi al. 2017a| Zaheer et al., 2017) were the first to apply neural
networks directly to the raw point cloud data, by using pointwise functions and pooling operations.
Many subsequent works used local neighborhood information (Qi et al., 2017b] Wang et al. 2019b)
Atzmon et al. 2018). We refer the reader to recent survey for more details (Guo et al. 2020 In contrast with the aforementioned works which focused solcly on permutation invariance, more related to this paper are works that additionally incorporated invariance rigid motions. (Thomas) et al. 2018) proposed Tensor Field Networks (TFN) and showed their efficacy on physics and chem- istry (asks. et al.) 20) also suggested an equivariant model for continuous rotations. (Li et al. 2019) suggested models that are equivariant to discrete subgroups of SO(3). (Poulenard et al.,
2019 suggested an invariant model based on spherical harmonics. (Fuchs et al. 2020) followed TFN and added an attention mechanism. Recently, (Zhao et al. 2019) proposed a quatermion equivariant point capsule network that also achieves rotation and translation invariance.
Universal approximation for invariant networks. Understanding the approximation power of in- variant models is a popular research goal. Most of the current results assume that the symmetry group is permutation group. (Zaheer et al. 2017 Qi et al. 2017a| [Segol & Lipman| [019)] Maron (ct al. 2020 Serviansky et al. 2020) proved universality for several S, -invariai and equivariant models. (Maron et al. 2019b a Keriven & Peyré, 2019 Machara & NT 2019) studied the approx- imation power of high-order graph neural networks. (Maron et al. 2019c Ravanbakhshi (2020) tar- geted universality of networks that use high-order representations for permutation groups YaroosyYa 2018) provided several theoretical constructions of universal equivariant neural network models based on polynomial invariants, including SF) equivariant model. In recent work (Bogatskiy et al., 2020) presented a universal approximation theorem for networks that are equivariant to sev- eral Lie groups including (3). The main difference from our paper that we prove a universality theorem for a more complex group that besides rotations also includes translations and permutations.
3 A FRAMEWORK FOR PROVING UNIVERSALITY
In this section, we describe a framework for proving the universality of equivariant networks. We begin with some mathematical preliminaries:
3.1 MATHEMATICAL SETUP
An action of group G a real vector space W is collection of maps p(g) : W W defined for any E G. such that p(g1) p(g2)= = p(919) for all 01,92 E G, and the identity element of G mapped to the identity mapping on W. We say p is a representation of G if p(g) is a linear map for every g E G. As is customary, when it does not cause confusion we often say that W itself is representation of G.
In this paper, we are interested in functions on point clouds. Point clouds are sets of vectors in R arranged as matrices:
X = (zi,..., ) E 33x
Many machine learning tasks on point clouds, such as classification, aim to learn a function which invariant rigid motions and relabeling of the points. Put differently, such functions are required
2

---

Published as a conference paper at ICLR 2021
to be invariant to the action of G R 3 SO(3) S, on 3 via
pc(t, R, (X)) R(X P1 where / R defines translation,
(1)
rotation and P is permutation matrix.
Equivariant functions are generalizations of invariant functions: If G acts on Wi via some action Pi(g), and on W2 via some other group action p2(g), we say that function f Wi W2 is equivariant if
f() p2(g) (w=) Ve W1 and g E G.
Invariant functions correspond the special case where p2(g) is the identity mapping for all g E G.
In some machine learning tasks on point clouds, the functions learned are not invariant but rather equivariant. For example, segmentation tasks assign discrete label each point. They invariant to translations and rotations but equivariant to permutations in the sense that permuting the input causes corresponding permutation of the output. Another example is predicting normal for each point of point cloud. This task is invariant to translations but equivariant to both rotations and permutations.
In this paper, we are interested in learning equivariant functions from point clouds into WP. where WT some representation of SO(3). The equivariance of these functions with respect to the action PG point clouds defined in equation and the action of G on W defined by applying the rotation action from the left and permutation action from the right as in I but 'ignoring' the trans- lation component. Thus, G-equivariant functions will be translation invariant. This formulation of cquivariance includes the normals prediction example by taking Wf R², as well as the segmen- tation by setting Wr RR with the trivial identity representation. We focus the harder of functions into WT which are equivariant to permutations, since it easily implies the easier case of permutation invariant functions Wr.
......,)) D} and Notation. We use the notation N4 = N (0) and N4 = Uren Nę. We set [D] = {l,..., [D] (D.
Proofs. Proofs appear in the appendices, arranged according sections.
3.2 CONDITIONS FOR UNIVERSALITY
The semi-lifted approach In general, highly expressive equivariant neural networks can be achieved by using 'lifted approach', where intermediate features in the network belong high dimensional representations of the group. In the context of point clouds where typically n > 3, many papers, e.g., (Thomas et al., 2018 Kondor 2018; BBogatskiy al. (020) use 'semi-lifted' approach, where hidden layers hold only higher dimensional representations of (3), but not high order permutation representations, In this subsection, propose strategy for achieving universal- ity with the semi-lifted approach.
We begin by an axiomatic formulation of the semi-lifted approach (see illustration in inset): we assume that our neural networks are composed of two main compo- nents: the first component functions firat which map the original point cloud 3axn to a semi-lifted point cloud We = +i Wiaut where Wheat is lifted (i.e., high-order) representation of SO(3).
a family Frest of parametric continuous G-equivariant
The second component is family of paramet- ric linear SO(3) -quivarauv functions Fpoot, which map from the high order representa- tion Wiest down to the target representation Wr. Each such What + Wr can extended to a SO(3) x S. cquivariant function W + W by ap- plying elementwise. For every positive inte- ger C, these two families of functions induce family of functions Fc obtained by summing C' different compositions of these functions:
Input
Lifted representation
Output
freat
function A
B 3kn
W/0
fent
WiFi
Fo(Froat, F = (ff(()) Σ (X)( (Aeigo) E Fpoot x Froas).
(2)
3

---

Published as a conference paper at ICLR 2021
Conditions for universality We now describe two conditions that guarantee universality using the semi-lifted approach. The first step is showing, as in (Yarotsky 2018) that continuous G-equivariant functions Co (R3xn, WT) can be approximated by G-equivariant polynomials PR WF).
Lemma 1. Any continuous G-equivariant function in Co (Ráxn WT) can be approximated uni- formly on compact sets by G-equivariant polynomials in Pa P WT))
Universality is now reduced to the approximation of G-equivariant polynomials. Next, we provide Iwo conditions which guarantee that G-equivariant polynomials of degree D can be expressed by function spaces F as defined in equation 2 The idea behind these conditions is that explicit characterizations of polynomials equivariant to the joint action of translations, rotations and permutations is challenging. However, it is possible to explicitly characterize polynomials equiv- ariant to translations and permutations (but not rotations). The key observation is then that this characterization can be rewritten as sum of functions to Weat. high dimensional representa- tions of S3(3) which is equivariant to translations, permutations and rotations, composed with a linear map which is permutation equivariant (but does not respect rotations). Accordingly, our first condition is that Freat contains a spanning set of such functions to Weat We call this condition D-spanning:
Definition 1 (D-spanning). For D N4,, let Freas be a subset of Ca CR Weat). We say that Freat is D-spanning, if there exist f1...., .I E Fleat, such that every polynomial p R + IR" of degree which is invariant to translations and equivariant to permutations, can be written as
K
p(X)
Σ (x((()))),
where A& Wiest + IR are all linear functionals, and Ag. W foat
I are the functions defined
by elementwise applications of Ax-
In Lemma 4 we explicitly construct a D-spanning family of functions. This provides us with concrete condition which implies D-spanning for other function families as well.
The second condition is that Foool contains all SO(3) linear equivariant layers. We call this con- dition Linear universality. Intuitively, taking linear rotation equivariant Aş in equation B ensures that the resulting function will be rotation equivariant and thus fully G-equivariant, and linear universality guarantees the ability to express all such invariant functions.
Definition (Linear universality). We say that a collection Foool of equivariant linear functionals between two representations Wteat and WT of SO(3) is linearly universal, if it contains all linear (eq))--auvaaiian mappings between the two representations.
When these two conditions apply, a rather simple symmetrization arguments leads to the following theorem:
Theorem 1. If Freat is D-spanning and Fooot is linearly universal, then there exists some C(D) N such that for all C 2 C(D) the function space Fo(Frens Fpoot) contains all G-equivariant polynomials of degree <
Proof idea. By the D-spanning assumption, there exist f1...., JKK Fiest such that any vector valued polynomial invariant to translations and equivariant to permutations is of the form
K
p(X) Σ (x(((())),
k=1
While by definition this holds for functions p whose image is R", this is easily extended to functions
to W as well.
It remains to show that when is also (equ)-vrvvaaria,,, we can choose A₁ to be SO(3) equivariant.
This is accomplished by averaging over SO(3).
4

---

Published as a conference paper at ICLR 2021
As result of Theorem I and Lemma we obtain our universality result (see inset for illustration)
FocDa Feraza Fo Cg(( Pm))
Corollary 1. For all C,D E N4, let FC,D denote function spaces generated by pair of functions spaces which are D-spanning and linearly universal as equa- tion Then any continuous G-equivariant function in
Ca sxn, WF) can be approximated uniformly on compact sets by equivariant functions
F= U=U FOID).D
DEN
3.3 UNIVERSALITY CONDITIONS IN ACTION
In the remainder of the paper, we prove the universality of several G-equivariant architectures, based on the framework we discussed in the previous subsection. We discuss two different strategies for achieving universality, which differ mainly in the type of lifted representations of SO(3) they use: (i) The first strategy uses (direct sums of) tensor-product representations; (ii) The second uses (direct sums of) irreducible representations. The main advantage of the first strategy from the perspective of our methodology is that achieving the D-spanning property is more straightforward. The advantage of irreducible representations is that they almost automatically guarantees the linear universality property.
In Section we discuss universality through tensor product representations, and give an example of a minimal tensor representation network architecture that would satisfy universality. section we discuss universality through irreducible representations, which is currently the more common strategy. We show that the TFN architecture (Thomas et al. 2018 Fuchs et al., 2020) which follows this strategy is universal, and describe a simple tweak that achieves universality using only low order filters, though the representations throughout the network are high dimensional.
UNIVERSALITY WITH TENSOR REPRESENTATIONS
In this section, we prove universality for models that are based on tensor product repre- sentations, as defined below. The main advantage of this approach is that D-spanning achieved rather easily. The main drawbacks are that its data representation some- what redundant and that characterizing the linear equivariant layers is more laborious.
Tensor representations We begin by defining tensor rep- resentations. For is N_ denote Tx -²a SO(3) acts on To by the tensor product representation, i.e., by ap- plying the matrix Kronecker product & times: Pk (R) == R The inset illustrates the vector spaces and action for =1,2, 2, With this action, for any 21,...,,4 i E [n]. the map from 3xx to Tx defined by
R
==
T2 R²
T
R
(I1,..., + Tú, X Dig @ Tis
(5)
is (3)) equivariant.
D-spanning family We now show that tensor representations can be used to define a finite set of D D-spanning functions. The lifted representation Wieat will be given by
D
W
T fent
=
θ T=0
Tr-
The D-spanning functions are indexed by vectors F (ri...., rk): where each ra is non-negative
integer. Denoting T = ||r| the functions Q(f) R3xn + T7. Q( (Q) y are defined for fixed j E [n] by
(x)=
Σ
I1
x
2r3
is
... 2 ork
x
(6)
b
5

---

Published as a conference paper at ICLR 2021
The functions 0 SO(3) equivariant they sum of equivariant functions from equation
Thus Q(7) is SO(3) x Sn equivariant. The motivation behind the definition of these functions is that known characterizations of permutation equivariant polynomials Segol & Lipman (2019) tell that the entries of these tensor valued functions span all permutation equivariant polynomials (see the proof of Lemma 2 for more details).
To account for translation invariance, we compose the functions Q with the centralization opera- tion and define the set of functions
1 1r - D).
x11,17 @D = f
n
where i is the natural embedding that takes each Tr into W feat = Dr Tr- In the following lemma, we prove that this set is D-spanning.
Lemma 2. For every D 14,, the set QD is D-spanning.
Proof idea. known (Segol & Lipman 2019) (Theorem 2) that polynomials p R830 + T
which are -qquiivvai,, are spanned by polynomials of the form pa (Dz)}=1, defined as
P)(()) = Σ
(8)
in
where ā = (a1...., ax) and each ag E N& is a multi-index. We first show that these polynomials can be extracted from Q(77 and then use them to represent p.
A minimal universal architecture Once we have shown that Op is D-spanning, we can design D-spanning architectures, by devising architectures that are able span all elements of QD. As we will now show, the compositional nature of neural networks allows us to do this in very clean manner.
We define parametric function V(X, which maps R3xn @ to 3x @ T as follows: For all e [n], we have f5(X, 1/) = (z,, (X, V)), where
V = 01 (x, 2 V2) + 82 Σ (x, V2)
We denote the set of functions (X.V) + f(x, V|0,, obtained by choosing the parameters 01.0, 0g E R. by Fmin While in the hidden layers our network the data is represented using both coordinates (X, V), the input to the network only contains an X coordinate and the output only contains coordinate, To this end, we define the functions
ext(X) (X, 1x) and øV (X, V)
(10)
We can achieve D-spanning by composition of functions in Fini with these functions and central- izing:
Lemma 3. The function set 2D is contained in
Freat fLOTV f .o.... o eext(X 1 11,,1 Fmin T TS ISD).
(11)
Thus Freat is D-spanning.
Proof idea. The proof is technical and follows by induction on D.
To complete the construction a universal network, we now need to characterize all linear equivari- ant functions from WT f to the target representation Wy. In Appendix G we show how this can be done for the trivial representation W,. = R. This characterization gives us a set of linear functions Fpool, which combined with Freas defined in equation 11 coorreppodss SO(3) invariant functions)

---

Published as a conference paper at ICLR 2021
gives universal architecture in Theorem However, the disadvantage of this approach is that implementation of the linear functions in Foool is somewhat cumbersome.
In the next section discuss irreducible representations, which give systematic way to address linear equivariant mappings into any Wr. Proving D-spanning for these networks is accomplished via the D-spanning property of tensor representations, through the following lemma
Lemma 4. If all functions in 2D can be written as
K x1, = Σ Axfa(X),
Lo (X
k=1
where fü E Ficat- Ak Wisa + WT and As W fest application of Ak, then Fieat is D-spanning.
(W ) n is defined by elementwise
feat
We note that as before, Aş are not necessarily equivarian
Proof idea. The lemma follows directly from the assumptions.
UNIVERSALITY WITH IRREDUCIBLE REPRESENTATIONS
In this section, discuss how to achieve universality when using irreducible representations of SO(3). We will begin by defining irreducible representations, and explaining how linear univer- sality is easily achieved by them, while the D-spanning properties of tensor representations preserved. This discussion can be seen as an interpretation of the choices made the construction of TFN and similar networks in the literature. We then show that these architectures are indeed universal.
be
5.1 IRREDUCIBLE REPRESENTATIONS OF SO(3)
In general, any dimenssommnni representation W of a compact group H can be decomposed into irreducible representations: a subspace Wo C W is H-invariant if lww E Wo for all h E H, E Wo- representation W irreducible if it has no non-trivial invariant subspaces.
the case (3), all irreducible real representations are defined by matrices ( (R), called the real Wigner D-matrices, acting We 26+ by matrix multiplication. In particular, the representation for 0, are D (R) = and (R) = R.
Linear maps between irreducible representations As mentioned above, one of the main advan- tages of using irreducible representations is that there is very simple characterization of all linear equivariant maps between two direct sums of irreducible representations. We use the notation Wi for direct sums of irreducible representations, where = .ei,...,&u ex) E N and Wt = +k= Wee-
e Lemma 5. Let ((1) = (ef)) ..... Ki and ( = (c(2) ..... (2)) A function A = (Ai,..., Ax.) is a linear equivariant mapping between Waas and Wen, if and only if there exists K₃ K₂ matrix M with M.j- = 0 whenever ((2) # g22) such that
A,(V) = Σ MyVi
(12)
where V = (V and V E W for all :1,.==1....
1,..., K1.
Proof idea. This lemma a simple generalization of Schur's lemma, a classical tool in representa- tion theory, which asserts that non-zero linear map between irreducible representations is scaled multiply of the identity mapping. Lemma was stated the complex setting in Kondor (2018). While Schur's lemma, and thus Lemma 5 does not always hold for representations over the reals, we observe here that it holds for real irreducible representations of SO(3) since their dimension is always odd.

---

Published as a conference paper at ICLR 2021
Clebsch-Gordan decomposition of tensor products As any dimee--ammnnional representation of SO(3) can be decomposed into a direct sum of irreducible representations, this is true for tensor representations well. In particular, the for decomposing the tensor product of two irreducible representations Wi, and Wi, into a direct sum of irreducible representations. This decomposition can be easily extended to decompose the tensor product Wi, @ Wi, into a direct sum of irreducible representations, where 1 l2 are now vectors. In matrix notation, this means there is a unitary linear equivariant U1,,, (2) mapping of Wi, & Wi, onto Wi, where the explicit values of I = 1(11, (2) and the matrix (1,, L2) can be inferred directly from the case where & and &g are scalars.
Corsch coefficients provide an explicit formula
By repeatedly taking tensor products and applying Clebsch-Gordan decompositions to the result, TFN and similar architectures can achieve the D-spanning property in manner analogous tensor representations, and also enjoy linear universality since they maintain irreducible representations throughout the network.
5.2 TENSOR FIELD NETWORKS
We now describe the basic layers of the TFN architecture TThoas et al. 2018), which are based on irreducible representations, and suggest an architecture based on these layers which can approximate G-equivariant maps into any representation W l E N4. There some superficial differences between our description of TFN and the description in the original paper, for more details see Ap- pendix F
We note that the universality of TEN also implies the universality of Fuchs et al. (2020), which generalization of TFN that enables adding an attention mechanism. Assuming the attention mechanism is not restricted to local neighborhoods, this method is at least as expressive as TFN.
TFNs are composed of three types of layers: (i) Convolution (ii) Self-interaction and (iii) Non- linearities. In our architecture, we only use the first two layers types, which we will now describe D
Convolution. Convolutional layers involve taking tensor products of filter and feature vector to create new feature vector, and then decomposing into irreducible representations. Unlike in standard CNN, filter here depends on the input, and is function F R + Win where lD
[0, [0,1..... DJT. The Ê-th component of the filter F(x) (1)0))))... will be given by
(13) where a x </||=|| if + 0 and 2 0 otherwise, Yf spherical harmonics, and R(() any polynomial of degree I D. In Appendix F we show that these polynomial functions can be replaced by fully connected networks, since the latter approximate all polynomials uniformly.
F(0 (x) = Re (I= (6), m
The convolution of an input feature V E W7; and a filter F as defined above, will give an output feature V = (Va)a E W%, where L, I(ly, Lt), which is given by
V)
V.
Σ F F(z. b=1
(14)
More formally we will think of convolutional layer as functions of the form S(X, V) (X, (XX, These functions are defined by choice of D, a choice of a scalar polynomial R(O), = .......D. D, and a choice of the parameter 0u R in equation 14 We denote the set of all such functions f by FD-
Self Interaction layers. Self interaction layers are linear functions from À WT - W which are obtained from elementwise application of equivariant linear functions A Wr + Wt,- These linear functions can be specified by choice of matrix M with the sparsity pattern described in Lemma
Activation functions. TFN, as well as other papers, proposed several activation functions. We find that these layers are not necessary for universality and thus we do not define them here.
Network architecture. For our universality proof, we suggest a simple architecture which depends on two positive integer parameters (C, D): For given D, we will define Fieat (D) as the set of
Since convolution layers in TFN are not linear, the non-linearities are formally redundant
8

---

Published as a conference paper at ICLR 2021
function obtained by 2D recursive convolutions
Freat (D) = { o f2D f... ext( FD).
where ext. and TV are defined as in equation 10 The output of function in Frean (D) is in W for some t (DD which depends on D. We then define DD) to be the self-interaction layers which map ((D) W This choice of (D) and (D), together with choice of the number of channels C, defines the final network architecture C.D (Feett (D), Foool (D)) (D) as in equation 2 In the appendix we prove the universality of TFN:
Theorem 2. For all n E N, IT E N
1. For D E N+, every G-equivariant polynomial p R2 W7 of degree D is in FTFN C(D)
2. Every continuous G-equivariant function can be approximated uniformly on compact sets by functions in UDEN FIFN
C(D),D
As discussed previously, the linear universality of Fooot guaranteed. Thus proving Theorem amounts to showing that Freat (D) ()) is D-spanning. This is done using the sufficient condition for D-spanning defined in Lemma
Proof idea. The proof is rather technical and involved. A useful observation (see Dai & Xu 2013)
used in the proof is that the filters of orders & = 0, 0......, D, defined in equation 13 span all polynomial functions of degree D on R3. This observation is used to show that all functions Qp can be expressed by Freat (D) and so Fieat is D-spanning, as stated in Lemma 2
Alternative architecture The complexity of the TEN network used to construct G-equivariant polynomials of degree D, can be reduced using simple modifications of the convolutional layer in equation 14 We add two parameters 01, 01,02 E IR to the convolutional layer, which now defined as:
=
=
01 F
F(E xx) 2 Vo + 02
F F(r. - xb) @ Va
(15)
b=1
With this simple change. we can show that Freat (D) D-spanning even if we only take filters of order 0 and throughout the network. This is shown in Appendix
6 CONCLUSION
In this paper, we have presented new framework for proving the universality of G-equivariant point cloud networks. We used this framework for proving the universality of the TFN model Thomas al. (2018): (uucss et al.] (2020), and for devising two additional novel simple universal architectures. In the future hope extend these simple constructions to operational G-equivariant networks with universality guarantees and competitive practical performance.
Our universal architectures do require activation functions, and use single seff-interaction layer. In Appendix H we present an experiment indicating that the performance of TFN is not significantly altered by these simplifications. Our architectures also require high order representations, and our experiments show that using increasingly high order representations does indeed improve perfor- mance. To date, practical TFN implementation included relatively small amount of layers, and did not use very high order representations. We believe our theoretical results will inspire interest stable implementation of larger architectures. On the other hand, an interesting open problem understanding whether universality can be achieved using only low-dimensional representations.
Finally, we believe that the framework we developed here will useful for proving the universality of other G-equivariant models for point cloud networks, and other related equivariant models. We note that large parts of our discussion can be easily generalized to symmetry groups of the form G d x x S, acting ddxn, where H can be any compact topological group.
The authors would like to thank Fabian B. Fuchs for making code available and Taco Cohen for helpful discussion. N.D. is supported by THEORINET Simons award 814643.
Acknowledgments

---

Published as a conference paper at ICLR 2021
REFERENCES
Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by exten- sion operators. arXiv preprint arXiv:1803. 10091, 2018.
Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David W Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. arXiv preprint wrXiv:2006004781 2020.
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical cnns. 1801. 10130, 2018.
arXiv preprint
Feng Dai and Yuan Xu. Approximation theory and harmonic analysis on spheres and balls, vol- ume 23, Springer, 2013.
Carlos Esteves, Christine Allen-BAl Bhen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 52-68, 2018.
Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling, Se transsfforssss: 3d roto- translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020.
William Fulton and Joe Harris. Representation theory: a first course, volume 129, Springer Science & Business Media, 2013.
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. arXiv preprint arXiv: (0dd::/700.0///22 2017.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. CoRR, htss/195504943, 2019. URL
Risi Kondor. N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials. arXiv preprint
2018.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-gordan nets: a fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems, pp. 10117- 10126, 2018a.
Risi Kondor, Hy Truong Son, Horace Pan, Brandon Anderson, and Shubhendu Trivedi. Covariant compositional networks for learning graphs. arXiv preprint (.X.v:8801.02144. 2018b.
Hanspeter Kraft and Claudio Procesi. Classical invariant theory, a primer. Lecture Notes, Version, 2000.
Jiaxin Li, Yingcai Bi, and Gim Hee Lee. Discrete rotation equivariance for point cloud recognition. In 2019 International Conference Robotics and Automation (ICRA), pp. 7269-7275. IEEE, 2019.
Takanori Machara and Hoang NT. A simple proof of the universality of invariant/cquivariant graph neural networks, 2019.
Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. arXiv preprint arXiv: /905. //005////6., 2019a.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations, 2019b. URL https: //openrevies net./ wow.omodddssyemaammaooo
Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In International conference on machine learning, 2019c.
10

---

Published as a conference paper at ICLR 2021
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. arXiv preprint arXiv:2002.08599) 2020.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. 2018.
arXiv preprint
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Adrien Poulenard, Marie-Julic Rakotosaona, Yann Ponty, and Maks Ovsjanikov. Effective rotation- invariant point cnn with spherical harmonics kernels, 2019 International Conference on 3D Vision (3DV), pp. 47-56. IEEE, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2):4 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in metric space. In Advances in neural information processing systems, pp. 5099-5108, 2017b.
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp. and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 2(1).1-7,001.
Siamak Ravanbakhsh.
Universal equivariant multilayer perceptrons.
arXiv preprint
arXiv:2002.02911. 2020.
Nimrod Segol and Yaron Lipman. 1212042, 2019.
On universal equivariant set networks. arXiv preprint
Hadar Serviansky, Nimrod Segol, Jonathan Shlomi, Kyle Cranmer, Eilam Gross, Haggai Maron, and Yaron Lipman. Ser2graph: Learning graphs from sets. arXiv preprint arXiv:2002.08872, 2020.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-an and -euuuiiaioon neural networks for 3d point clouds. arXiv preprint arXiv:1802.082/99 2018.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, et al. Deep graph library: Towards efficient and scalable deep learning on graphs. arXiv preprint 13X.v:1909001355. 2019a.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5): 1-12, 2019b.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. 2018. URL http:
Daniel Worrall and Gabriel Brostow. Cubenet: Equivariance to 3d rotation and translation. In
Proceedings of the European Conference on Computer Vision (ECCV), pp. 567-584, 2018.
Keyulu Xu. Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful are graph neural
networks? In International Conference on Learning Representations, 2019. URL https:
//openreview. wtt./oorum. A5Km
Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. arXiv preprint arXiv.1804.10006, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in neural information processing systems, pp- 3391-3401, 2017.

---

Published as a conference paper at ICLR 2021
Yongheng Zhao, Tolga Birdal, Jan Eric Lenssen, Emanuele Menegatti, Leonidas Guibas, and Fed- erico Tombari. Quaternion equivariant capsule networks for 3d point clouds. arXiv preprint arXiv:1912.12099, 2019.
A NOTATION
We introduce some notation for the proofs in the appendices. We use the shortened notation X X 1 X1,,1 1 and denote the columns of by (#1,..., En). We denote
Σy = fF E No ||]] =T}
B PROOFS FOR SECTION
B. G-EQUIVARIANT POLYNOMIALS ARE DENSE
A first step in proving denseness of G-equivariance polynomials, and in the proof used in the next subsection is the following simple lemma, which shows that translation invariance can be dealt with simply by centralizing the point cloud.
In the following, PWr is some representation of SO(3) on a finite dimensional real vector space WT. this induces an action Pwrxs, of SO(3) x S, on W by
PWrx (R, PY) = Pwr (R)Y
This is also the action of G which we consider, PG PW, xS., where we have invariance with respect the translation coordinate. The action of G on 3x3 is defined in equation |
Lemma B.I. A function f : _23x + WT is G-equivariant if and only if there exists a function which is equivariant with respect to the action of SO(3) Sn R3xa, and
f(x) h(x x1,,1
(16)
Proof. Recall that G-equivariance means SO(3) Sn equivariance and translation invariance. Thus if ff is G-equivariant then equation [6 holds with / = f.
On the other hand, if f satisfies equation 16 then claim it is G-equivariant. Indeed, for all (1, RG P) E Rd x SO(3) x Sn since pf I In1 = 1, PL,
h(R(X + =1,)p" 1 f (pa(t, R, P)))) = (R(X + t1,,
R(( + 1,
n
= =][[X - X1,, 11 PTT = (PEEXS. (R, (p P)( 1 -x1, 1
n1
1 X1 X
PWrrs (R, P)h
= =ol(t, R, P)(((.
We now prove denseness of G-equivariant polynomials in the space of G-invariant continuous func- tions (Lemma]) D.
Lemma 1. Any continuous G-equivariant function in Co (Raxn WP) can be approximated uni- formly on compact sets by G-equivariant polynomials in Pa 3xnn, Wa).
Proof of Lemma 7 Let K C p3 be a compact set. We need to show that continuous G- equivariant functions can be approximated uniformly in K by G-equivariant polynomials. Let Ko denote the compact set which is the image of K under the centralizing map X + X x X11,, 1 By Lemma is sufficient to show that every SO(3) S,, equivariant continuous function f can be approximated uniformly on K, by a sequence of SO(3) x S, equivariant polynomials pe. The argument now concluded by the following general lemma:
12

---

Published as a conference paper at ICLR 2021
Lemma B.2. Let G be a compact group, Let P1 and p2 be continuous representations of G on the Euclidean spaces W1 and W2. Let - W1 be compact set. Then every equivariant function f Wi - W2 can be approximated uniformly K by sequence of equivariant polynomials Ph W1 + W2
Let μ be the Haar probability measure associated with the compact group G. Let K1 denote the compact set obtained as an image of the compact set G' x K under the continuous mapping
(g. + pı(g)
Using the Stone-Weierstne Weierstras theorem, let p& be a sequence of (not necessarily equivariant) polyno- mials which approximate / uniformly K1. Every degree D polynomial p W1 W2 induces G-equivariant function
(x) ( ((()))))))))
This function (p) is a degree D polynomial as well: This is because (p) can be approximated uni- formly on K1 by "Riemann Sums" of the form Σド- Wyp2 ()(() (g)))) which are degree D polynomials, and because degree D polynomials are closed in ((K)).
Now for all X E K1, continuity of the function g + p2(g implics that the operator norm of p2(g-²) !) is bounded uniformly by some constant N > 0, and so
p2 |/ Po Pol
(X) - f(X)
1/
S NII p& (klloc))) +
B.2 PROOF OF THEOREM 1
Theorem 1. Fieat is D-spanning and Foool is linearly universal, then there exists some C(D)
such that for all C 2 C(D) the function space FF(Fiear, Fpool) contains all G-equivariant polynomials of degree S D.
Proof. By the D-spanning assumption, there exist fi,..., JK E
Freat such that any vector valued polynomial p 123= + R invariant to translations and equivariant to permutations is of the form
K p(X) Σ Ax(s((X)),
(17)
where As are linear functions E. If is matrix valued polynomial mapping 3xx to Wy = Rxx=, which is invariant to translations and equivariant to permutations, then it is of the form p =
and each p; = (Piy) is itself invariant translations and permutation equivariant. It follows that matrix valued can also be written in the form equation 17 the only difference being that the image of the linear functions A₃ is now R.
Now let 33 Wa be G-equivariant polynomial of degree < D). It remains to show that we can choose A& to be SO(3) equivariant. We do this by a symmetrization argument: denote the Haar probability measure on SO(3) by v, and the action of SO(3) on Wreat and WT by P1 and P2 respectively Denote p (p,)}= and Se (fRY)= For every j 1,..., n, we use the SO(3) cquivariance of pj and f to obtain
P2(X) = / SOOSS pz( oo
K Σ (2 An o (RX)du(M)
k=1 S0(3)
K
(2(R
Pz
(R o (pI(R) f((x) du(R)
o H())
50(3)
k=
k=1
By this mean that the maps (g. ()(9) 1,2 are jointly continuous
13

---

Published as a conference paper at ICLR 2021
where Ag stands for the equivariant linear functional from Wteat Wr, defined for Wleat by
(r(w)= /S033) SS03)) P2(R As (p1 (R)()) dv(R).
Thus we have shown that is in Fo(Fieat: poool for C' = K, as required.
C PROOFS FOR SECTION 4
We prove Lemma
2
Lemma 2. For every D N4, the set QD D-spanning.
Proof. is known (Segol & Lipman 2019) (Theorem 2) that polynomials p R + R" which are -,-equivaria are spanned by polynomials of the form pa (Pa))= defined as
a
(x)
Σ
....
(18)
where (o1,...,, .g) and each or E No. is a multi-index. It follows that Suuequivari polynomials of degree I D spanned by polynomials of the form p'a where Σ |agl S D. Denoting T lax l 1, ... K, the sum of all Tu by T. and F== (rx k=1 that there exists a linear functional Ag : Tr > such that
(x) = Aa, o (x(X)
where we recall that Q² = (o) (x)) j=1
is defined in equation as
Q Σ
2
2 3r
@rk ig
Thus polynomials p (p:)ī=1 which are of degree
S D, and are S, equivariant, can be written as
Ds (x()) E Σ AaF (0)
Σ Σ Ar (1 0 (x)) , .,...,n,
FEDr dllox/=ra
FEDFEET
where A; = Eallau Ag, lo and li -1 is the left inverse of the embedding t. If p is also translation invariant, then
x(X) p(X - -1 - Σ oF (100 -x-xxx1)))
TED
Thus 2p D-spanning.
We prove Lemma 3
Lemma 3. The function set Qp is contained in
Freat = To o f² . of exx((x 1 -X Fmin. T S DJ.
(1I)
Thus Frest is D-spanning.
Proof. In this proof we make the dependence of Freat. on D explicit and denote Fieat (D)-
We prove the claim by induction D. Assume D 0. Then Qo contains only the constant function
X + In E To and this is precisely the function TV ext E Fieat
Now assume the claim holds for all D' with D 2 D' 2 0. and prove the claim for D. Choose
F= (ri,..... rk) Ey. for some T < D, need to show that the function Q(F) is in Fieat ( (D). Since Freat (D 1) C Freat we know from the induction hypothesis that this true T < D.
Now assume T D. We consider two cases:
14

---

Published as a conference paper at ICLR 2021
1. If 71 > O, we set F (r1 1, ......... ..)) We know that (X) Fleac 1) by the induction hypothesis. So there exist f2.... , fD such that
LO f2
ext(X)
(X).
(19)
Now choose fi E Fmin to be the function whose V coordinate V = (V)) V) - 2j @ V,, obtained by setting 0 = 1, 02 in equation
)=1 is given by
9 Then, we have
(( = Σ
Ty 2 x p0
00 r
2rr 60
6 Rik
P k=
(X)
and so
L LOAV o f1 o 0...... SD ext 1 -1 -
(20)
n
and L Q 1 (D).
n2
2. If = O. We assume without loss of generality that 72 > 0. Set = (r2 1, r..... rx
As before by the induction hypothesis there exist f2. fD which satisfy equation 19
This time choose S1 Fmin to be the function whose V coordinate V (V₂) is
given by (X,,, V) E; & Vj. obtained by setting 0 0, 02 in equation 9 Then we have
=
Tzj ④
0(r2
2 xix
3rx
=1
2rr
Fix
8rk
Thus equation 20 holds, and so again we have that &
11-111,11 -xxx1)11111
Fieee (D).
Finally we prove Lemma 4
Lemma 4. If all functions in 2p can be written as
lo (x-_XXX
_x1,
Σ Axfu(X),
n
where fu E Freat Av W feat WT and Åg. Wheat (WT ) is defined by elementwise
feat application of Ak, then Fieat is D-spanning.
Proof. If the conditions in Lemma Ahold. then since 2D is D-spanning, every translation invariant and permutation equivariant polynomials p of degree D can be written as
p(X)= = Σ
Á (10Q
-
Σ AIFIDD
A
K= Σ A
ALTTRO(X))
Al D
Kr
Σ
Σ
(Je,())
where we denote A&, APoLO AR.. Thus we proved Frent is D-spanning.
15

---

Published as a conference paper at ICLR 2021
D PROOFS FOR SECTION 5
We prove Lemma 5
Lemma 5. Let ( 10) = (
2 Ki and ( = - ......,, (........
) A function A = (A Ak,) ) is
linear equivariant mapping between Wio and Wio, and only if there exists K₁ x K2 matrix
M with Mi = whenever (1)) + e²), such that
K. A,(V) = Σ MyV.
(12)
where = (V:),m) and V; E W for all =
.,...,, K1.
Proof. As mentioned in the main text, this lemma is based on Schur's lemma. This lemma is typi- cally stated for complex representations, but holds for odd dimensional real representation as well. We recount the lemma and its proof here for completeness (see also (Fulton & Harris 2013)).
Lemma D.1 (Schur's Lemma for SO(3)). Let A We, + We, be linear equivariant map. l * lg then = 0. Otherwise a scalar multiply of the identity.
Proof. Let A We, + We, be a linear equivariant map. The image and kernel of A are invariant subspaces of We, and Wt,, respectively. follows that if A * O then necessarily & f2- Now assume & lz. Since the dimension of We, is odd, A has real eigenvalue X. The linear function A >/ is equivariant and has a non-trivial kernel, so A Al-
linear isomorphism
We now return to the proof of Lemma 5 equivariant. Next denote the restrictions of each A, y, 1..... K2 by Aij. and note that
Note that each A, Wao + Wa
gea is linear and SO(3)
Ay(V VK.) Σ (V.)..
(21)
By considering vectors Wea; of the form (O..... 0, Vi, 0....... 0) we see that each Aij : W (())
Wees linear and (Quiieequvvariant. Thus by Schur's lemma, if e! = El then A,y My V;
for some real Mij. and otherwise Mjj 0. Plugging this into equation 21 we obtain equation 12
2 which shows that the TFN network described in the main text is universal: Theorem 2. For all n E N, by E N
We prove Theorem
1. For D E every G-equivariant polynomial p 123x WT: of degree D is in JTPN C(D),D'
2. Every continuous G-equivariant function can be approximated uniformly compact sets by functions in J C(D),D
TTNN
Proof. As mentioned in the main text, we only need to show that the function space Frest (D) is D-spanning. Recall that Freas (D) obtained by 2D consecutive convolutions with D-fiters. In general, we denote the space of functions defined by applying J consecutive convolutions by GJ.D.
y is a space of functions from 3xm YR, we denote by (Y,Tr) the space of all functions p R23x T.7 of the form
K
x(X)
Σ
Axf((x)
(22)
16

---

Published as a conference paper at ICLR 2021
where A& Y - TT are linear functions, A& : Y + T are induced by elementwise application, and fK y. This notation is useful because: (i) by Lemma 4 is sufficient show that Q (X) in (GDD.D, (T)) for all F E Σr and all T S D, and because (ii) it enables comparison of the expressive power of function spaces V/. 3/2 whose elements map different spaces Y?". Y2, since the elements in ():, TT), = 1, 2 both map to the same space. In particular, note that if for every 3/2 there is a 9 E 3/1 and a linear map A Y1 + Y2 such that f(x) = Å g (X), then ()2,, TT) C ()i, Tr).
We now use this abstract discussion to prove some useful results: the first is that for the purpose of this lemma, we can 'forget about' the multiplication by a unitary matrix in equation [4] used for decomposition into irreducible representations: To see this, denote by GID the function space obtained by taking J consecutive convolutions with D-filters without multiplying by unitary matrix in equation [4, Since Kronecker products of unitary matrices are unitary matrices, we obtain that the elements in GIDD and ÂJ.D differ only by multiplication by a unitary matrix, and thus (G,,D, Tr) S (9J.D, T and (G Tr) C (GJ.D, Tr), so both sets are equal.
Next, prove that adding convolutional layers (enlarging J) taking higher order filters (enlarg- ing D) can only increase the expressive power of a network.
Lemma D.2. For all D, E N+.
1. (G,,,, S (9.+1,D, Tr)-
2. (GJ,D. S (9J.D41,
Proof. The first claim follows from the fact that every function f in (GJ,D. TT) can be identified with a function in (GJ+1.D, Tr) by taking the J + convolutional layer in equation 14 with 0g =.F
The second claim follows from the fact that D-filters can be identified with D + 1-filters whose D + 1-th entry O.
The last preliminary lemma we will need is
Lemma D.3. For every J,D E N4, and every 1,8 N4, if E (9I.,, Te), then the function defined by
Yo (X))
a T0) 2 (x()
is in (9)-1.D Tets).
Proof. This lemma is based on the fact that the space of s homogeneous polynomial on R spanned by polynomials of the form (rls--Eyff (2) for L s. s 2, 4... Dai & Xu 2.13 For each such e, and s < D. these polynomials can be realized by filters (() by setting R ([=l)) = | that
-|==)=|=>| = (x))
For every D E N and < D, we can construct a D-filter FS.D. = F(s), f0(0-2) .... are as defined above and the other filters are zero. Since both the entries of (xr), and the entries of 00 span the space of s-homogeneous polynomials on R33 it follows that there exists a linear mapping B, WtD + T, so that
F()) where
0 (F*. R
(23)
Thus, since can be written as sum of compositions of linear mappings with functions in GJ.p as in equation 22, and similarly 00 obtained as a linear image of functions in G1,D as equation 23 we deduce that
n
Ca 2b) 2 (x) = Σ
(T. 5b) 2 (X)
b=1
is in (9,-1,D, Tets)
17

---

Published as a conference paper at ICLR 2021
As final preliminary, note that D-filters can perform averaging operation by setting R and 0o. R(),. ...., R(D) = 0 in equation 13 and equation 14 We call this D-filter an averaging filter.
We are now ready to prove our claim: we need to show that for every D,T E N4 where T S D, for every F E Σr, the function Q(ai is in (G2D.D. D. Tr). Note that due to the inclusion relations in Lemma D.2 it is sufficient to prove this for the case T D. We prove this by induction on D. For D 0, vectors Eg contains only zeros and so
(x) 1n TV ext(X) E (Go,o. To).
We now assume the claim is true for all D' with D> D' 2 and prove the claim is true for D. We need to show that for every F' E Ed the function Q is in (G2D, D. TD). We prove this yet again by induction, this time on the value of r1: assume that F E ED and T1 = O.. Denote by the vector in ΣD-1 defined by
== (r2 - r........ rk)
By the induction assumption on D, we know that (x) E (2(D--1.D-11.-- and so
(x(
Σ
TTa
Ts)
20
Q
Ta
m6) 2 b
1ra-1
Σ
8r3
6
@0
b=1
1=1
n
Fa +
Σ Q (8)
(X)
is in (92D-1,D-1, TD) by Lemma D.3 which is contained in (92n-1,D, Tb) by Lemma D.2 Since Ta has zero mean, while Q.²² (X) does not depend on a since r1 = 0, applying an averaging filter to da gives us constant value - Qe (X) in each coordinate E [n]] and so Q (X) is in DGDD.D TD).
Now assume the claim is true for all F Ed which sum to D, and whose first coordinate is smaller than some rj 2 1, we now prove the claim is true when the first coordinate of 7" equal to r'. The vector F = ('2,-.-,' rrk) obtained from F by removing the first coordinate, sums D' D r D, and so by the induction hypothesis D know that ((f) E (G2Dr,D ,D', TD:). By Lemma D.3 we obtain a function Ia E (G2D'+1,D) TD) S (GDDDD, TD) defined by
(X) =
Σ
aTa- -b,)
881 8 (x)
hm
(i σb)
1
0r'
X ... @ x Brk
b
iK
1=1 Q (X) + additional terms
Eg
where the additional terms are linear combinations of functions of the form PDQ
(r') (X) where
r' E ED and their first coordinate Tı smaller than rí. and PD To + To a permutation. By the induction hypothesis on rı, each such Q(r') is in (G2n, TD). It follows that PoQ!" (x)) 1,..., n, and thus ((F) (X), are in (G2D.D: TT) as well. This concludes the proof of Theorem 2
(x)(='(X)a== (X).a=
E ALTERNATIVE TEN ARCHITECTURE
In this appendix we show that replacing the standard TFN convolutional layer with the layer defined in equation 15
12 2b) 2 + 82 Σ F F(x. xx) 2 Va
vx
0
F(
we can obtain D-spanning networks using 2D consecutive convolutions with 1-filters (that is, filters
in Wi where L [0, 1). Our discussion here is somewhat informal, meant to provide the general
18

---

Published as a conference paper at ICLR 2021
ideas without delving into the details have done for the standard TFN architecture in the proof of Theorem 2 In the end of our discussion we will explain what is necessary to make this argument completely ngorous.
We will only need two fixed filters for our argument here: The first is the 1-filter Fia (F(O), F())
defined by setting ([[=]) = 0 and (2 (1|=<]]) || to obtain
Fia(z) | (0) ||
The second is the filter Fi defined by setting ( (||=]l 1 and (I 0, that
F (z) = 1.
We prove our claim by showing that pair of convolutions with 1-filters can construct any convolu- tional layer defined in equation 9 for the D-spanning architecture using tensor representations. The claim then follows from the fact that D convolutions of the latter architecture suffice for achieving D-spanning, shown in Lemma
Convolutions for tensor representations, defined in equation 9 are composed of two terms:
V Ttensor: 1
V) Io & Va and Vitensor 2 V) = Σ Tb Vo.
To obtain the first term Vitenoor we set 0 = 0.02 = 1/n, F = Fia in equation 15 we obtain (the decomposition into irreducibles of) Vitensor,1 (X, V) = Ha @ V.. Thus this term can in fact be
expressed by a single convolution. We can leave this outcome unchanged by a second convolution, defined by setting 81 0, 02 1/n, F Fj.
To obtain the second term / we apply first convolution with 0₁ = =1, F- FFa 02 = 0, to obtain
(ah na) 2 Vn Σ (a. F.) V. teensor 2 2V, Fa
Vh
By applying an additional averaging filter, defined by setting 01 = n: F Fi, 82 = 0, we obtain 2 2VV XV. This concludes our 'informal proof".
Our discussion here has been somewhat inaccurate, since in practice Fra(z) (0, x) E Wo + W and (z) (1 E Wo ① W1. Moreover, in our proof we have glossed over the multiplication by the unitary matrix used to obtain decomposition into irreducible representations. However the ideas discussed here can be used to show that 2D convolutions with 1-filters can satisfy the sufficient condition for D-spanning defined in Lemma
See our treatment of Theorem 2 for more details.
F COMPARISON WITH ORIGINAL TFN PAPER
In this Appendix we discuss three superficial differences between the presentation of the TFN archi- tecture in T'homas al. (2018) and presentation here:
1. We define convolutional layers between features residing in direct sums of irreducible rep- resentations, while (Thomas et al., 2018) focuses on features which inhabit single irre- ducible representation. This difference non-essential, direct sums of irreducible rep- resentations can be represented as multiple channels where each feature inhabits a single irreducible representation.
2. The term 0g V. in equation appeerss in (Fuchs et al., 2020), but does not appear explicitly in (Thomas et al. (2088). However it can be obtained by concatenation of the input of self- self-interaction layer to the output, and then applying a self-interaction layer.
3. We take the scalar functions R(9) be polynomials, while TTomas et al. 2018)) take them to be fully connected networks composed with radial basis functions. Using polynomial scalar bases is convenient for presentation here since it enables exact expression of
19

---

Published as a conference paper at ICLR 2021
equivariant polynomials. Replacing polynomial bases with fully connected networks, obtain approximation of equivariant polynomials instead exact expression. can be shown that if p is a G-equivariant polynomial which be expressed by some network FC.D defined with filters coming from a polynomial scalar basis, then p can be approx- imated a compact set K, up arbitrary error, by similar network with scalar functions coming from a sufficiently large fully connected network.
G TENSOR UNIVERSALITY
In this section we show how to construct the complete set set F of linear SO(3) invariant functionals from W
feat Dr TT to R. Since each such functional A is of the form
Awwo, ... , Σ Ar(WT),
...,
where each Ar is invariant, it is sufficient to characterize all linear als A To + R.
function-
It will be convenient to denote
W R and W R3 = TDD
We achieve characterization using the bijective correspondence between linear functional A W&D and multi-linear functions Á W + R: each such A corresponds unique Á, such that
eip Mee, ... ei Vii iD) [3]
(24)
02, E3 denote the standard basis elements of R We define spanning set of equivariant linear functionals on W via a corresponding characterization for multi-linear functionals on W Specifically, set
where e1, e2,
KD = {k E N+1 D 3k even and non-negative. }
For KD we define multi-linear functional:
As(w (WI..... wD) = det(w), (1., u2, ws) x... x det Wisk WaR) x (W3641, (33+2) X (WD-1, wp),
(25)
and for () E Kp x Sp we define
(u wp)) = A. Wo(D))
(26)
The space of lineur invariant functions from Tn to lR is spanned by the set of linear invariant functionals An {At.al (ks .)) E KD Sp} induced by the multi-linear functional Ng.
Proposition
described in equation 25 and equation 26
We note that (i) equation 24 provides a (cumbersome) way to compute all linear invariant functionals
Ag. explicitly by evaluating the corresponding Âς on the 3" elements the standard basis and (ii) the set AD is spanning. but is not linearly independent. For example, since (w1, w22 (w2, W1), the space (3(3) invariant functionals on To w is one dimensional while |Aa] = 2.
Proof of Proposition We first show that the bijective correspondence between linear functional W + IR and multi linear functions Á DD I, extends bijective correspondence between innvariant linear/multi-inng functionals. The action of SO(3) on Wr", is defined by
7 (uI,....... ......). (Rw1, Rwp).
The action p(R) R of SO(3) on WMD is such that the map
(wi....,wy + wi X w2.. WD
SO(3)- equivariant. It follows that if Ä and A satisfy equation 24 then for all E SO(3), the same equation holds for the pair À ((R) and A o p(R). Thus SO(3) invariance of Á is equivalent 5O(3) -invaariiee of A.
20

---

Published as a conference paper at ICLR 2021
functionals W invariant to á subset of the of polynomials W invariant to p. It is known (see (Kraft & Procesi 2000), page 114), that all such polynomials are algebraically generated by functions of the form
det(ws, Wig; Wig) and (w,, "ja), where i1, 82, 83, J1, J2 E [D].
Equivalently, the form
polynomials are spanned by linear combinations of polynomials of
det(w,, W wly det(w,,, Wis Wto) (w), Wyy (W Wy.)
(27)
When considering the subset of mulli-linead invariant polynomials, we see that they must be spanned by polynomials in equation 27 where each W1; .... wp appears exactly once in each polynomial in the spanning set. These precisely correspond to the functions in ÅD.
H EXPERIMENTS
Table Results obtained on the QM9 (atase[[aamakiihnan et al. (2014) for different design choices
in the TFN architecture. Results are reported for the Chomo target variable, and are multiplied by 103.
Model variant
Mean & error
Order 0 irreps
111.2 + 0.3
Order 0-1 irreps
82.1 + 0.4
Order 0-2 irreps
62.1 + 1.5
Order 0-3 irreps
53.2 + 1.9
Order 0-4 irreps
51.2 + 0.4
Order 0-3 irreps without non non-lineart -ineariy
53.1 + 0.8
Order 0-3 irreps + self-interaction only in the final layer
55.2 + 0.3
This section provides an experimental evaluation of different design choices of the TFN architecture, inspired by our theoretical analysis. We study the following questions:
The importance of non-linear activation. Our proof shows that using non-linear acti- vation functions is not necessary for proving universality. Here, we empirically test the possibility of removing these layers.
2. The importance of high-dimensional irreducible representations. Our theoretical anal- ysis shows that in order to represent/approxinat high degree polynomials, high-order rep- resentations should be used. Here, we check whether using high-order representations has practical benefits.
3. The effect of self-interaction layers. Our proof suggests that it is enough to use self interaction linear layers at the end of the model. We empirically compare this approach with the more common approach of using self-interaction layers after each convolutional layer.
Dataset. We use the QM9 (Ramakrishnan et al., 2014) dataset for experiments. The dataset contains 134K molecules, with node 31) positions, 5 categorical node features and 4 categorical edge features. The task is molecule property prediction regression task.
Framework. We used pytorch (Paszke et al. 2017 jas the deep learning framework and the Deep Graph Library (DGL) (Wang al. 2019a) as the graph learning framework. All experiments ran on NVIDIA GV100 GPUs.
Experimental setup. We use the the TFN implementation from Fuchs et al. (002)) We trained each model variant for 50 epochs the Chomo target variable using an L loss function and the ADAM optimizer with learning rate 10-3 and report results on the test set on the final epoch aver- aged over two runs. We used the default parameters and data splits from Fuchs al. (020).
21

---

Published as a conference paper at ICLR 2021
Architecture. The architecture consists of 4 TFN convolutional layers, each followed by linear self-interaction layer. We used 16 copies each irreducible representation used. We used norm- based non-linearity in the original TFN paper (Thomas et al. 2018). These convolutional layers are followed by a max-pooling layer and two fully connected layers with 16d features in the hidden layer, where d is the maximal degree of irreducible representations used.
Results. Table 1 and Figure ! present the re- sults. The main conclusions are: (1) The ex- periments show that, at least for this task, using non-linear activations does not improve perfor- mance. This result fits our theoretical analysis which shows that these layers are not needed for universality.
110
Mean absolute Error
100
90
knro
(2) Figure presents plot of error vs representation degrees used. The plot clearly shows that using high-dimensional representations (up to order 3) improves per- formance, which also fits our analysis. Using representation orders higher than is signifi- cantly more time consuming, and was found to have little effect on the results (as in in Fuchs al.] (2020)), though we believe this to be application-dependent. (3) Using self interac- tion layers only at the end of the model is shown
60
0
1
2
3
Irrep degree
Figure & error versus maximal irreducible representation used. It is clear that the error is reduced as higher order representation are used.
to have marginal negative effect on the results.
Semi-transparent color represents standard devia- tion.
22


Reviews:
[{"review_id": "wgKxLH4b9Fc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper mainly explores the representation ability of invariability of a point cloud network from the theoretical perspective. The universal approximation property for equivariant architectures under shape-preserving transformations is discussed. First, the authors derived two sufficient conditions for equivariant architectures with the universal approximation property. Then, they examined two methods based on the Tensor Field Network to prove that such a property holds for both of them. At last, the authors propose alternative methods which also satisfy the universal approximation property. \n\nThis paper is full of theoretical analysis, which is based on investigating the equivariant polynomials in the group theory. The proofs are similar to the previous works, except that more transformations, including rotation, translation and permutation, are considered together. However, the proof of the rotation equivariance has been discussed in previous works, and the theorems for translation and permutation are much easier than the rotation analysis. Moreover, the proposed method based on TFN is simply modified by the authors, resulting in alternative architectures which also satisfies the universal approximation property. However, this paper fails to provide any numerical experiments to demonstrate the performance and the influence of parameters \\theta.\n\npros:\n- provide sufficient conditions for equivariance of shape-preserving architectures to satisfy the universal approximation property\n- prove two methods based of Tensor Field Networks that satisfy the universal approximation property\n- raise two simple models based on the TFN\n \n\ncons:\n- This paper provides complete poofs to the TFN network theoretically, but lacks auxiliary experimental verification. Therefore, it is difficult to verify the correctness and feasibility of the proofs.\n- The authors do not provide experimental results to demonstrate the performance of the proposed new methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bOPRmlmOkAM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors introduce a framework for sufficient conditions for proving universality of a general class of neural networks that operate on point clouds which takes as input a set of coordinates of points and as output a feature for each point, such that the network is invariant to joint translation of the coordinates, equivariant to permutation of the points and equivariant to joint SO(3) transformations of the coordinates and output features of all points. Notably, this class contains Tensor Field Networks (TFN). The authors accomplish this by writing the network as a composition of an equivariant function from a class F_feat and followed by a linear pooling layer. When the F_feat class satisfies a “D-spanning” criterion and the pooling layer is universal, the network is universal. For a simple class of networks and for TFNs, the authors prove D-spanning. Linear universality of the pooling layer follows from simple representation theory.\n\nStrengths:\n-\tIt is useful to know whether prevalent classes of neural networks are universal\n-\tThe authors use a general construction for proving universality of equivariant networks for the point cloud group, rather than being specific to certain architectures.\n-\tReading the proofs along with the main text, the argumentation is clear and relatively easy to follow for me as reviewer, unfamiliar with similar universality proofs.\n\nWeaknesses:\n-\tThe paper would benefit from providing more intuition behind the proposed constructions, lemmas and theorems, in particular this holds for: theorem 1 based on the split between the linear universality and D-spanning; the construction of Segol & Lipman (2019) and how this relates to the Q functions and lemma 2\n-\tIn addition to the previous point, the proofs, currently critical for understanding the paper, are given in the appendix, which is not ideal for the self-containedness of the main paper.\n-\tAs the authors of the TFN paper note: in practice not all higher order irreps of the tensor product of the filter and the features are computed. This seems to indicate a big difference between the theoretical analysis - which includes all irreps and thus is computationally intensive even when modelling low order polynomials – and the practical application of TFNs. It would be interesting to know how expressive such practical low order TFNs are. Another difference between the described networks and practical TFN is that in the described networks, all relevant parameters are in the pooling layer, which sums a large number of terms (looking at the proof of lemma 2, exponential in D), while in practical TFNs, the parameters are in the filters.\n\nRecommendation:\nThe authors proof the useful statement of universality of a prominent class of neural networks, which is why I recommend the acceptance of this paper.\n\nSuggestion for improvement:\n-\tMake the big picture clearer by providing more intuition.\n-\tComment on the differences between the class of networks described and TFNs used in practice.\n\nMinor points/suggestions:\n-\tP3 Add definition of W^n_T as n direct sums of W_T\n-\tP3 “where W_feat is a lifted representation of SO(3)”, what does lifted representation here mean? Just any rep?\n-\tI get a bit confused by the wording in Def 1. Unless I am mistaken, it appears like the quantifiers are reversed. Should it mean “for every polynomial …, there exists f_1, … in F_feat and linear functionals Lambda_1, …, : W_feat -> R ”?\n-\tAround Eq 5, perhaps the authors could clarify the clarify the domain of the Q functions, which I suppose is Q^r : R^3n -> T_T, where T=||r||_1\n-\tAround Eq 7, are X_j and x_j the same?\n-\tIn lemma 4, is A_k any linear map or an equivariant linear map? \n-\tIn Appendix B, perhaps a new subsection B.2 would make sense before theorem 1?\n-\tIn the proof of thm 1, it says “p: R^{d \\times n} \\to W_T”, should that be W_T^n?\n-\tIn the proof of lemma 2, it says “we see that that exists a linear functional”\n\n## Post rebuttal\nI thank the authors for their response and revised version, which has been improved notably with the inclusion of the proof ideas. My previous rating still applies.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UkOoIRJ2i8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summarize what the paper claims to contribute.**\nThe authors claim to: (1) introduce a general approach for proving universality of rotation-translation-permutation equivariant models for point clouds; (2) prove universality of two recent rotation equivariant point cloud networks and (3) introduce two rotation equivariant architectures for point cloud processing\n\n**Strengths:**\nThe paper is well organized and both the language and notation are clear\nThe authors consider equivariant representation learning which is of growing interest to the community\nThe authors present theoretical insights for the success of recent architectures \nThe insights themselves are leveraged to support the introduction of two new architectures\n\n**Weaknesses:**\nThe novel architectures are described but not evaluated; it is therefore unclear what impact the simplification will have in practice\nIt might be nice to point out rotation equivariant architecture that is not universal\n\n**Clearly state your recommendation.**\nAccept. See Strengths.\n\n**Arguments for your recommendation.**\nTheoretical issues in deep learning is a relevant topic area for ICLR. This paper provides a theoretical framework for interpreting the success of deep learning frameworks for equivariant representation learning on point clouds. Moreover, the paper leverages the insights gathered to propose two novel approaches. The paper is written in a clear and accessible way. \n\n**Possible typos:**\n(just before Thm. 1) When these two necessary → when these two sufficient \n(just after Lemma 3)  linear function → linear functions \n\n**Post rebuttal**\nWith consideration of the authors' responses to reviewer questions and revisions to the submitted work I have changed my rating to _clear accept_.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PxJPEXS09xJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work is a theoretical paper investigating the sufficient conditions for an equivariant structure to have the universal approximation property. They show that the recent works: Tensor Field networks and Fuchs et al., are universal under the proposed framework. A simple theoretical architecture is presented as another universal architecture.\n\nPros:\n- Achieving rotation equivariance is important to point cloud network as it is the key to improve the expressiveness of point cloud features. Hence the study of the universality of network with such property is important to the community.\n- Overall, the proposed proof looks plausible to me. \n- A minimal universal architecture is proposed that satisfies the D-spanning property. This provides the theoretical starting point to design a more advanced and complex equivariant point cloud network.\n\nCons:\n- The paper is quite difficult to follow. I'm not an expert in group theory and had a difficult time understanding some of the theorems and proofs. It would be great if the writing can be broken down into more fundamental modules and provide more illustrations.\n- In addition, the paper doesn't provide any evaluation of the proposed new universal architectures. Though this is a theoretical paper, it would be nice to show the proposed theory have some practical use. For instance, it would be great to provide a simple implementation of the minimal universal architecture and show it indeed achieves the rotation equivariant features on the point cloud data. That would make this work much stronger and more practical. \n\nMinors:\nThere are some typos in the inset figure on page 1: \"Equivarint\" -> \"Equivariant\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a theoretical analysis of the expressive power of equivariant models for point clouds with respect to translations, rotations and permutations. The authors provide sufficient conditions for universality, and prove that recently introduced architectures, e.g. Tensor Field Networks(TFN), do fulfil this property. 

The submission received positive reviews ; after rebuttal, all reviewers recommend acceptance and highlight the valuable paper modifications made by the authors to clarify the intuitions behind the proofs. 

The AC also considers that this paper is a solid contribution for ICLR, which will draw interest for both theoreticians and practitioners in the community. \
Therefore, the AC recommends acceptance. 
Decision: Accept (Poster)