A Characterization Theorem for Equivariant Networks with Point-wise Activations
Authors: Marco Pacini
Xiaowen Dong
Bruno Lepri
Gabriele Santin
Venue: ICLR 2024 poster (2024)
Keywords: Geometric Deep Learning
Equivariant Neural Networks
Characterization Theorem
Point-wise Activations
Primary area: learning on graphs and other geometries & topologies
TL;DR: A characterization theorem describing admissibile combinations of representations and activation functions for equivariant layers with point-wise activations

Abstract: Equivariant neural networks have shown improved performance, expressiveness and sample complexity on symmetrical domains. 
But for some specific symmetries, representations, and choice of coordinates, the most common point-wise activations, such as ReLU, are not equivariant, hence they cannot be employed in the design of equivariant neural networks. 
The theorem we present in this paper describes all possibile combinations of representations, choice of coordinates and point-wise activations to obtain an equivariant layer, generalizing and strengthening existing characterizations.
Notable cases of practical relevance are discussed as corollaries. Indeed, we prove that rotation-equivariant networks can only be invariant, as it happens for any network which is equivariant with respect to connected compact groups. Then, we discuss implications of our findings when applied to important instances of equivariant networks. First, we completely characterize permutation equivariant networks such as Invariant Graph Networks with point-wise nonlinearities and their geometric counterparts, highlighting a plethora of models whose expressive power and performance are still unknown. 
Second, we show that feature spaces of disentangled steerable convolutional neural networks are trivial representations.

Published as a conference paper at ICLR 2024
A CHARACTERIZATION THEOREM FOR EQUIVARIANT NETWORKS WITH POINT-WISE ACTIVATIONS
Marco Pacini
Xiaowen Dong
Fondazione Bruno Kessler, Italy mpacini @ eb
University of Oxford, United Kingdom rdoogtrobots.. ex. ac.uk
Bruno Lepri
Gabriele Santin
Fondazione Bruno Kessler, Italy lepri fbk.eu
University of Venice. Italy gabriele. sant santin@unive... it
ABSTRACT
Equivariant neural networks have shown improved performance, expressiveness and sample complexity on symmetrical domains. But for some specific symme- tries, representations, and choice of coordinates, the most common point-wise ac- tivations, such as ReLU, are not equivariant, hence they cannot be employed in the design of equivariant neural networks. The theorem present in this paper de- scribes all possible combinations of coordinates and point-wise activations obtain an exactly equivariant layer, gen- eralizing and strengthening existing characterizations. Notable cases of practical relevance are discussed corollaries. Indeed, we prove that rotation equuiinn-quivariaan networks can only be invariant, as it happens for any network which is equiv- ariant with respect to connected compact groups. Then, we discuss implications of our findings when applied to important instances of exactly equivariant net- works. First, we completely characterize permutation equivariant networks such Invariant Graph Networks with point-wise nonlinearities and their geometric counterparts, highlighting a plethora models whose expressive power and per- formance are still unknown. Second, we show that feature spaces of disentangled steerable convolutional neural networks are trivial representations.
representations, choice
INTRODUCTION
In recent years, equivariant neural networks have improved the performance of standard neural net- works by harnessing symmetries of the training data, and have risen to an entirely new branch of
machine learning known as geometric deep learning (Bronstein et al.) 2021 2017 Kondor & Trivedi]
2018 Those networks (Wood & Shawe-Taylow [996]] [Chhnn & Welling 2016a) have shown im-
proved generalization capabilities across various rescarch areas, including computer vision ( (Worrall
ct al. 2017 Hoogeboom et al.) 2018 Dieleman et al. 2016), computer graphics (Weiler et al. [018
Zaheer al.) [i et al. 2017] Maron al. 2020 Marcos al. 2017 Sifre & Mallat 2013).
and graph learning MMaron et al. 2018 2019a b Geerts 2020 Kondor et al. 2018: et al.
2018
However, these networks pose additional constraints in the architecture design. Namely, the use of pointwise activations, which are fundamental in common neural networks for their simplicitly, easiness to train, and computational efficiency, is limited by a lack of equivariance certain set- tings (Cohen & Welling 2016b) Weiler et al. 2018). Their application in equivariant models is thus restricted and needs a careful choice, which is however only partially explored. Indeed, pioneering work (Wood & Shawe-TTayoon 1996) proved a theorem that enables researchers, having at their dis- posal certain representations, to be automatically informed of all employable point-wise activations and vice-versa. This, to the best of our knowledge, is the first and only characterization theorem for equivariant networks with point-wise activations, which has however significant limitations in several aspects, namely, (i) it only applies to finite groups, (ii) it is redundant, since its classification is unable to identify representations that are equivalent up to isomorphisms or change of bases, and (iii) it is restricted to subclass of possible activation functions.

---

Published as a conference paper at ICLR 2024
In this paper, using tools from representation theory (Fulton & Harris 2004) and matrix group theory (Flor) 1969,, we present a stronger and more general theorem that fills the described gaps in the result of Wood & Taylor (996). In more details, consider classes F of activation functions and groups M of representation matrices, and we investigate when they can be combined to obtain an equivariant layer, i.e., they commute- Section We start by defining operations that map any F to a corresponding maximal admissible group of matrices M(F), and vice-versa map M a maximal admissible family of activations F(M). Crucially, we highlight the dual nature of those operations and how their composition stabilizes, leading to a finite family of explicitly defined maximal classes. Finally, for these few maximal classes we exhibit the dual pairings (F.M,, thus obtaining complete and exhaustive categorization of admissible cetivationreeprsennatiio pairs, which includes the classification of Wood & Shawe-Taylor (996) as special case.
We then apply this theorem to obtain a number of novel results. First, we leverage it to collect new theoretical insights into existing methods. Namely, we prove a stronger result than Wood & Shawe-Taylor (1996) in the case of finite groups (Section [5.]] by identifying isomorphic admissi- ble representations. Moreover, we consider disentangled equivariant networks (Cohen & Welling 2016b), and we show that they admit point-wise activations if and only if the linear representa- tions underlying their feature spaces are trivial (Section (52)). Second, we specialize our results to particular cases of equivariant networks of practical relevance (Section 66) For rotation-equivariant networks (Weiler et al.] 2018), we show that all admissible networks are invariant, and hence unable to learn many equivariant tasks such as segmentation or detection. This proves a significant barrier for the use of equivariant networks with point-wise activations in this kind of tasks. Instead, for Invariant Graph Networks (IGNs) (Maron al. 2018) and geometric IGNs (Dym & Gortler 2022] Finkelshtein et al. 2022, Joshi et al. (023), including the k-order invariant and equivariant networks presented by Maron et al. (2018), we show that the choice of admissible representation layers com- patible with point-wise activations is not only limited to those described by Maron et al. (2018), and in fact we highlight and fully characterize wider class of permutation equivariant networks in terms of subgroups of the symmetric group. This opens new directions and provides novel tools for the design of cquivariant networks beyond the known ones.
We point out that approach has limitations, since we focus solely exact equivariance with re- representations. Those representations encompass disentangled
spect to arbitrary representations as well as regular representations (Cohen & Welling 2016a b). and thus we address a substantial portion of documented use cases in the literature. Nevertheless, our work does not apply infiniteddimensional representations, and in particular to the vast body of literature utilizing harmonic analysis techniques. Also in this case, however, our results have a space of application. Indeed, for computational reasons these infiite-dimmnniioall models are often transformed into fi- nite ones in diverse ways, and when this is achieved through the discretization of both the symmetry group and the domain, the resulting discrete model can be frequently analyzed using our approach. Instead, we can not analyze those cases where the discretized model does not exhibit exact equiv- ariance with respect to the initial symmetry group, even if for these models the empirical evidence suggests that this equivariance is approximately maintained (Cohen et al. 2018). Moreover, it is crucial to emphasize that further alternative approaches exist in the literature, and unexplored possi- bilities remain.
In brief, our contributions are summarized as follows: (i) we provide a characterization theorem for equivariant neural networks with continuous point-wise activations, by showing the existence of finite number of maximal sets of equivariant classes, enumerating them, and providing an explicit dual pairing between activations and representations, (ii) we use this result to give an exhaustive description of networks that are equivariant with respect to finite groups, and to show a barrier in the use of disentangled equivariant networks, and (iii) we discuss implications of this theorem in practical and relevant scenarios. namely highlighting a severe limitation of rotation -qquiarnnquuvariiant networks, while providing new and unexplored design possibilities for (geometric) IGNs.
The paper is organized as follows: Section 2 provides an overview of related work on equivariant models and existing limitations concerning point-wise activations. Section 3 provides preliminar- ies for our work. In Section 4 we present the general formulation of the characterization theorem for equivariant networks with point-wise activations. Section 5 explores significant implications of the theorem for specific but still abstract cases such as finite groups or disentangled networks. In Section 6 we discuss examples of relevant significance in practical scenarios such as equivariance with respect to the symmetric group and the rotation group and their application to graph invari-
2

---

Published as a conference paper at ICLR 2024
ant networks, networks equivariant with respect to Euclidean transformations, and geometric graph processing. Finally, Section casiiionnmmmmiiziii summarizes our findings and discusses future research directions.
RELATED WORK
Historically, early explicit integration of representation theory and harmonic analysis into machine learning can be dated to Kakarala (1992) Knndor (2008). Wood & Shawe- (1996) are the first to bring equivariance into deep learning with general approach. They define equivariant neural networks and give a classification of those models for the case of point-wise activations. In more recent years, Cohen & Welling 2014 2016a b) presented the foundational work of group equiv- ariant convolutional networks and introduced the general model of steerable Convolutional Neural Networks (CNNs) (Cohen & Welling 201ōb), a popular and efficient class of equivariant models. In the following years many equivariant models and many applications to different domains appeared in the literature: rotation-invariince for galaxy morphology prediction (Dieleman et al. (2015), per- mutation invariance for set processing (Qi et al. [017]] Zaheer et al. 2017). permutation invariance for graph and relational structure learning (Maron et al. [2018] Kondor et al. 2018: Pan & Kon- dor 2022), simultaneous roto- tito-translation invariance and permutation invariiaccee for 3D point-cloud
processing (Thomas al., 2018 Dym & Maron 2020). and roto-trait transtrannlttoon invariance for medical
image analysis (Bekkers et al. 2018). Different research directions focused instead on theoretical aspects of equivariant models, including the design of new frameworks (Cohen al. 2019] proofs of expressivviy and universality (Maron et al. |0.992019|// 2019b] [020)] Ravanbakhsh /hou Zhou 2020
Yarotsky| 2018 Joshi al. 2023), generalization bounds and sample complexity (Behboodi al.
2022 Cohen et 20199] Sannai et al. 2021 Zweig& & Brunal 2021 Elesedy (022)). and characteri- zations (Wood & Shawe- Taylor] 1996 Kondor Trivedi 2018)]Lang & Weiler| (02I]. Despite this extensive theoretical investigation, a comprehensive study of the interaction of activations and rep- resentations is still missing. This constituess a potential limitation in the understanding and design of novel equivariant architectures enjoying certain desired properties. This paper aims at filling this gap. We remark finally that there exist activations which are not point-wise, and that they are some- times employed in practice even if they are less computationally favourable, and their investigation is still at its infancy. Indeed, to the best of our knowledge, a general framework to describe and study them in principled manner is yet to be proposed. Examples of those activations are the norm nonlinearities (Worrall et al. 2017), squashing nonlinearities (Sabour et al. (017). tensor product nonlinearities (Kondor (2018), and gated nonlinearities (Weiler al. 2018 We refer to Weiler &
Cesa (2019) for further details.
PRELIMINARIES
3.1 GROUPS, REPRESENTATIONS AND AFFINE TRANSFORMATIONS
We would like to define functions symmetrical with respect to a certain set of transformations. groups (Fulton)
Classes of transformations suitable for computation and technical manipulation & Harris 2004). A group is a set of clements which can be composed together, can be inverted and such that there exists an element neutral with respect composition. For further details we refer to Definition and Example 2 in Appendix A.2
Despite the properties presented by groups, these algebraic structures need adaptation to work with the language of linear algebra typical of neural networks. The right tool to operate this translation is representation theory (Fulton Harris 2004) which studies how abstract groups can be translated to sets of matrices which are group themselves. Given group G, a vector space V on the field R of real numbers, and the set GL.(V) of linear invertible functions from V to itself, a G-representation is a function p G GL(V) compatible with the group structures. When possible we will indicate such a representation by using simply V (Definition 10). We will write GL,, (IR) when = IR.
For our purposes some particular representations will play an important role. Given an action of G on a finite set X with cardinality n. setting V = R and considering the standard basis B = { of X, permutation representation is representation such that g(e,) egi for each g E G and i E X, and a signed permutation representation is such that g(e,) = E Moreover, a monomial representation is such that g(e;) Ggi for some non-null real number ag.i, and if additionally
3

---

Published as a conference paper at ICLR 2024
dg is positive say that the monomial representation is non-negative. If each ag.i belongs to the multiplicative group (b")nez for a real value 0 < # 1, we say that the representation is b-monomial. Similarly if each dg.i belongs to (=b")
say that the representation is =6- monomial (Definition 12). These representations respectively generate the group of permutation matrices, digned-permutatinn matrices, monomial matrices, non-negatove monomial matrices, and b- and =6-monomial matrices in (IR). The following matrices P.S and M are respectively examples of permutation matrices, signed permutation matrices and 2-monomial matrices:
T
17 0 0
0
0
2
P
1
S
0
0 0
M
0 2
2
0
0
0
If V and W are vector spaces underlying representations pV and ow of the same group G. we say that function f V -> W is G-equivariant if fopv pv pw f. We write Hom(V, W) for the set of all linear maps from V to W. and define Homg(V, W) as the set of G-equivariant linear functions from V to W. For the interest of this work, we consider affine maps between V and W, i.e., composition between a linear map Hom(V, W) and a translation on W. We denote as Aff W) those maps and (V, W) the set of equivariant affine functions (see also Appendix A.5
3.2 EQUIVARIANT NEURAL NETWORKS
We now define equivariant neural networks, which were first introduced by Wood & Shawe- (1996) as Group Representation Networks and by Cohen & Welling (2016b) as G-Steerable Con- volutional Networks. The same model later appears in other papers such as Behboodi et al. (2022) under the name equivariant neural networks, which is the notation we adopt in this paper.
Given group G and arbitrary G-representations V; for 1 S i < m, G-equivariant neural network a composition
= Φm o fm--1 pm- ..... fi o ou.
where each activation fi : Vi - V; is a G-equivariant function, and Φ; E (V, Vi+1) is an affine G-equivariant map.
An activation f : V; + V, is point-wise if there exist basis B; = (ui,..., Uml of V and a real scalar function f such that
Mare +... Um) = (a1))r +... ((am Um A aa...... am E R.
In this case, say that induces f on B. Note that we do not constrain activations to be non-linear or non-affine but, from now on, we only consider continuous functions C(IR). This condition is not particularly restrictive continuous functions constitute wide class of function which includes all commonly employed point-wise activations (such as ReLU, tanh, ...). they are compatible with backpropagation, and strictly encompasses activations dealt by Wood & Shawe-Taylor 1996)..
We remark that other works (Kondor & Trivedi 2018 Cohen et al. 2019) have considered different definitions of equivariant networks, which allow one to treat infinite dimensional representations, but are limited to represent signals defined on homogeneous spaces. As mentioned in Section [ by avoiding these infinite dimensional representations we fail to fully describe various physical phenomena that are modeled as continuous, and therefore belonging to infiniteddimensional spaces, which are however discretized and often reduced finite models in the computational practice.
MAIN RESULT
We present stronger and more general version of the characterization theorem proposed in Wood & Shawe-Taylor (199), which is the only existing partial characterization of admissible activa- tion functions to the best of our knowledge. Although building on the existing results of Wood & Shawe-Taylor (199)). we introduce several extensions and improvements, namely i) we generalize the theorem to non-finite groups and continuous activations, such as non-discrete ones as rotations and Euclidean transformations (Weiler et al. 2018); ii) We identify certain network classes by con- sidering representations up to isomorphism; mi) We integrate the theory with several adjustments that make the classification effective, such as the maximality discussed in Lemma T iv) We ground
4

---

Published as a conference paper at ICLR 2024
characterization the classification of multiplicative subgroups of R, which highlights the po- tential to generalize these results to different scenarios. For the reader's convenience, we present the general result in Theorem [ and its specialization to compact groups in Theorem
For our purposes, we consider the following sets F functions as activation functions:
Continuous functions, C(R).
The set of b-multiplicative functions for given real value 1, namely each f E C(TR)
such that f(b"x) = 6". f(x) for each E Σ and for each IR (see Lemma 8 in Appendix for a more explicit definition).
The set of =o-multiplicative functions for a given real value 1, namely each f E C(R) such that x) = f(x) for each Z and for each x I (see Lemma 8 in Appendix A.8 for a more explicit definition),
Odd functions, namely each f E ((R) such that f(-z) = -f(x) for each x E I,
Semilinear functions, namely each f E C(R) that is linear on Ry and on Rçç--
For our construction we will need also the following relations between activations and matrices. Given group G' and representation p G + GL(V) of G. we consider point-wise activation f V + V induced by R R basis B of V. The image of in GL(V) with respect to the basis B forms a group of matrices which we denote M, and note that f is G-equivariant if and only if commutes with respect to the matrices in M, i..., f(Mu) = M f(v) for each M e .M and E V. where both e and M are written on the basis B.
This paper proposes a simple procedure to recover the maximal group of representation matrices M compatible with given class F of activation functions, and vice-versa the widest class of functions F given group of representation matrices M. The precise definition is as follows.
Definition 1. Given F S C(R), we define the maximal group MFF) admitted by F as the set of all matrices in GI,, (TR) which commutes with each f induced by f E F. F. Conversely, given M S (R), the maximal set F(M) admitted by M the set of all functions f E C(R) such that f induced by f commutes with each M E M.
Observe that M(F) has trivially group structure with respect matrix product, since if M1, M2 M(F) commute with f. then also their product M2 does. Moreover, the following stabilization lemma (Lemma [ proves that the maps to the maximal sets (Definition [ stabilize after two it- erations for any initial choice of .M or F, proving the duality of the operators M(-) and F(-) on maximal elements. See Section A.88) for the proof.
Lemma 1. The group of matrices M' = MMF(MM) is the largest group in GL, (R) for which F(M') F(M), and F' F(M(F)) is the largest family of functions in C(TR) for which M(F') M(F).
Thanks to Lemma [ it is sufficient to provide explicit pairings between M and F(M) (or and M(F)) just for maximal admissible groups M, and similarly for maximal admissible families of functions F. Indeed, given any other M (or F) which is not maximal, it is sufficient to find superset M of M which a maximal group in the sense of Definition and this will give a set of admissible functions F which are equivariant also for M.
We can now state our main result, which shows that, under mild assumptions, these maximal groups (or, equivalently, maximal function classes) are only in very limited number. Moreover, for each of these we provide the exact correspondence between M and F, thus providing an exhaustive classification of all possible admissible pairs of M and F. As the J represents the activation functions, we make the basic assumptions that it does not contain only affine functions. We remark that the proof of this theorem. for which we refer Section A.8 uses a novel algebraic approach that makes it casier to generalize the result to other scenarios, as discussed in Section
Theorem 1. Assume that F is not set of affine functions. The following are exactly all dual pairs of maximal admissible families of activations F and associated maximal admissible groups M:
1. Continuous functions and permutation matrices,
5

---

Published as a conference paper at ICLR 2024
2. Odd continuous functions and signed permutation matrices,
3. Semilinear functions and non-negative monomial matrices,
4. Continuous b-multiplicative functions and b-monomial matrices,
5. Continuous =b-mudiiplicativee functions and +b-monomial matrices.
Note that some groups of matrices or families of functions are contained in each other. For example, permutation matrices are signed permutation matrices, while 8 mi-monomial matrices are contained into b-monomial matrices, but they are still maximal following Definition
We classified groups of matrices and families of activation functions commuting with each others, and this classification is exhaustive. However, for compact groups G we show that there is another tool enlarge the set of possible activation functions, namely moving isomorphic representation which admit a wider family of activation functions. This approach solves and generalizes an issue raised Wood & Shawe-Taylor (1996), where in Section 4.2 it is noted that the representation
p: Ee + GL []
given by
1
0
P(1) 0 2
(0)
0
1
only admits semi-linear activation functions, according to the rather narrow set of activations studied by Wyoo & Shawe- Shawo-Taylor (1996). To be more precise, by expanding the class of activation functions to continuous functions, we have shown in Theorem that in fact the broader set of 2- 2-multiplicative functions is admissible. Despite 2- 2-multiplicative functions being larger set than semi-linear func- tions, they still do not include common activations such tanh, sigmoid, softplus. However, by basis change through the matrix B diag (1/ v2), obtain the isomorphic representation
0 [
Bp(0) 0
-
which the standard permutation representation of Z2, and thus commutes with all continuous functions (see Theorem []. This approach is generalized to the case of arbitrary compact groups by the following theorem, showing that in this case representations are always isomorphic (signed) permutation representations, which have the widest admissible family of activations. We refer to Section A.8 for proof.
Theorem 2. Let G be compact group and assume that F is not set of affine functions. Then any representation of G admitted by F is isomorphic to a subgroup of the (signed) permutation matrices. Thus, F can always be chosen as the set of (odd) continuous functions.
5 RELEVANT IMPLICATIONS FOR PRACTICAL SCENARIOS
In this section, we delve into the practical implications of Theorems and focusing on non-odd non-affine activations, the most used activations practice. We further specialize these results the case of finite groups, where the only non-trivial admissible representations is induced by permutation representations, which can be described with more precision. We explore the implications of these results on neural networks designed for processing first-order relational structures like sets and point clouds (Qi et al. 2017) Zaheer et al. (2017). These networks have proven highly effective practice, showcasing efficiency and accuracy when dealing with such unordered data. Finally, in this setting, we show that admissible disentangled representations coupled with point-wise activations are trivial.
5.1 FINITE GROUPS
As the majority of activations used in practice are induced by non-odd non-linear functions, we present a corollary of Theorem 2 that completely describes representations for finite groups that can be used in these cases. We show that admissible representations tions up to positive-scaling of the basis.
only permutations representa-
Corollary 1. Let G be a finite group. and let f V - V be non-odd non non-affine equivariant activation function on a G-representation V defined on the basis B. Then there exists positive scaling of B and collection of finite index subgroups H; < G such that V G/H, x x x Hm

---

Published as a conference paper at ICLR 2024
Proof. We can consider V to be finite permutation representation thanks to Theorem2 2 as the only admissible bounded matrix groups are permutation matrices, while signed permutation matrices are compatible only with odd functions. A permutation representation has underlying permutation set that can be decomposed in the disjoint union of orbits. Recall that, for any X on which G acts transitively, there exists a set bijection X = G/H H for some subgroup H of G. Hence, permutation representation V can be decomposed into the direct sum V IRll G/H, for collection of finite index subgroups H; < G such that there is a bijection between the orbits X; of G in X and the quotients G/H,
Note that in general permutation representation is given by the action of group G set X and then extended on I such that, for each E G and I E X, an element eg of the canonical basis of IR transforms as ge, = Eya. But the action of G on X, i.e., the computation of gr, is not explicit and easy to convert in computational terms, while the action of G on one of its quotients G/H H provides an algebraic, hence computable, alternative. Furthermore, this notion provides the complete list of possible sets admitting G-actions up to isomorphism, which is in bijection with the set of all the quotients of G.
Let us now discuss how representation spaces of permutations equivariant networks on sets (Za- heer et al. 2017, Qi et al., 2017) reduce to a simple instance of the representation space shown in Corollary Indeed, the symmetric group of elements, Sn, is the set of permutations of [rt], and then S, -quivarrrt networks are able to process sets of elements independently of their order. A complete treatment of the representations of Sn, can be found in SSaaan (200I). Now want to construct group quotients able to define representations for Sn -quequiiar networks used in prac- tice. Consider À= = (A1,... (A1,...; A1) to be a partition of n, i.e., a decreasingly ordered tuple of positive integers whose sum is n.. Define Sx= = Sx, x .... X Sx, as the subgroup of S, where the ith factor si-1 X to Lo λÿ. Now set À to be the partition (n - 1, 1). elements permutes the elements form 2j=1
of (, will be represented by the identity element and all the permutations of [r]] that send into an element of {2, ..., n
which we indicate with [(1i)] (See Definition 8 for more information about quotients of groups). The action of a E Sn on [(1)]] E Sn/ ((n1)))) will be [(a((i)] where we can identify [(1)] with the identity element. The bijection [(1)] + from Sm/S(m-1.1) to [r]] is compatible with the action of S,, on those sets, hence we have an equivariant isomorphism be- tween S.. S(n-1.)) and IR" which is the standard representation space for permutations equivariant networks on sets (Zaheer et al. 2017 Qi et al. 2017). Due to Corollary | RR.. (B,a//((-111)) is one of the admissible representation spaces for S,, on the family of non-odd non-affine continuous activations. In similar way. in Section 6.2 we will see how to obtain representation spaces of IGNs and we will highlight that many other admissible representation space could be possible. It is relevant how is possible to obtain efficient models through this procedure and which however offers the possibility of building models starting from the simple knowledge of the group of symmetries of the input.
5.2 DISENTANGLED REPRESENTATION
Disentangled representations CChen & Welling 2016b) can be described as follows. An irreducible representation of G-representation V; is a minimal non-trivial G-invariant subspace, and each representation V22 can be decomposed into a direct sum of irreducible spaces (Definition [S] and Theorem 5) Composing irreducíble representations with each other allows us to construct arbitrary representation spaces and control the number of parameters of each layer at will thanks to Schur's s Lemma (Lemma[). The easiest way to compose these representations with each other is by doing a direct sum. If then, to define activations, we choose a basis whose vectors are contained in a single irreducible component we say that such a space is disentangled (Cohen & Welling (016b). As a consequence of Theorem[] we obtain the following characterization of disentangle representations.
Corollary 2. For finite groups and activation functions f V + V induced by non-odd non- non-affine continuous functions, the representation is disentangled if and only if the representation V is a sum of trivial representations.
The direct sum of trivial representations is clearly disentangled and admissible for Theo- rem [ To prove the inverse implication, thanks to Corollary [ we can consider V to be a per- mutation representation. By disentanglement,
Proof.
suppose V to be irreducible and with basis
7

---

Published as a conference paper at ICLR 2024
B (.i...., Um defining Given V, the subspace [ gea gt) is trivial, non-zero because V is permutations representation, and G-invariant. Hence, V Egec go) is trivial.
L
Even if composability and control of the number of parameters are particularly good properties of disentangled network, being able to admit only trivial irreducible representations is not achiev- able in most cases of practical use. Let us consider again the example of representation spaces of permutation equivariant networks on sets (Zaheer et al. 2017; Qi et al. 2017). The irreducible decomposition of the standard action of S, on R" is the direct sum of trivial component and a (n 1)-dimensional one. Hence, linear layer between this input space and a disentangled admis- sible representation space will send the input contained in the (n - 1) -)-dimensional component to due to Schur's Lemma (Lemma? 2) For large customary. this eliminates the entire information inside the input in the forward pass of the first layer of the network.
PRACTICAL USE CASES
In this section, we aim at understanding how Theorem affects and limits the design choices of networks in a relevant practical scenario such as geometric IGNs (Joshi et al. 2023), broad family of models particularly proficient processing geometric data such as point-clouds or meshes. We achieve this objective in Section 6.3 but beforehand, we need to decompose manageable subproblems. Specifically, in Section we investigate ivation-equivarian case of networks of that will be necessary to understand the following analysis. In Section 6.2 delve into higher-order IGN, widely employed in practical applications. Finally, in Section 6.3 we merge the preceding results to analyze geometric IGN comprehensively.
task into two more
networks,
6.1 ROTATION EQUIVARIANCE AND EQUIVARIANCE FOR COMPACT GROUPS
Rotation-equivariant neural networks (Finkelshtein et al. 2022) have the ability to process geomet- rical data tracking orientation and pose. We now discuss how our results affects the design this kind of networks. The group of rotations around the origin of RR3 is denoted as SO(3), it can be described as the group of real orthogonal 3 3 matrices with positive determinant. More generally, let G (O(n) be the group of real orthogonal x matrices with positive determinant. It acts on R by left multiplication. Note that (O(n) ) is connected and compact topological group and the presented representation is irredueible and non-trivial (Fulton & Harris 2004). The following Corollary 3 implies that ination-equivarian networks are invariant.
Corollary Let G compact group and let Go be the connected component containing the identity element. An admissible G-representation for non-affine activation functions is Go-invariant. In particular, if G is connected, an admissible G-representation is trivial.
Proof. First, suppose to be connected. The image of continuous representation of a compact and connected topological group is compact and connected matrix group due to Theorem 4 The only possible representations described by Theorem 2 are permutation representations and signed permutation representations whose images are finite groups whose only compact and connected subgroup the one containing only the identity. Hence, the original representation is trivial. If G is not connected, a G-representations induced a Co-representations which will be Go-invariant.
This means that in general it will be possible to create neural networks capable of performing in- variant tasks such as classification but more general equivariant tasks such as segmentation or detection (Qi et al. 2017). Further, as SO(3) is subgroup of Euclidean transformations of R this phenomenon afflicts networks equivariant with respect general Euclidean transformations, i.e., they will be invariant to the rotational part of Euclidean transformations although possibly sensitive to reflections and translations.
6.2 ON INVARIANT GRAPH NETWORKS
Let us go back at the example proposed in Section 5.1 and generalize it high-order structures to obtain IGNs. Introduced by Maron et al. (2018), they are a class of neural networks equivariant with respect the symmetric group and their expressiveness is intimately related to graph neural
8

---

Published as a conference paper at ICLR 2024
networks (Maron et al., 2019a Geerts 2020). They permutation equivariant models taking as input a relational structure such as a set, a graph, or an higher-order structure such as simplicial complexes in form of tensor of the corresponding order. For example, a directed graph with nodes can be seen as a tensor in R 69 R". Elements of this space can be represented as linear combinations of ej @ ej for i, [n] and each E Sn acts them permuting their indices si- multancously, (e, @ eg) = (i) @ En(s). More in general, a k-ary relational structure can be represented as k-order tensor in (R")@* with S, simultaneously acting on each R" component. Regardless of the order the input tensor, intermediate representation spaces may be the sum of tensors of arbitrary order, hence a linear layer of an IGN will be the direct sum of linear equivariant maps between spaces (RT)) Sh and (Zm)&h for arbitrary &t and h and they admit point-wise activa- tions, hence, by Corollary 1 they should be able to be represented as IR Sa/H: for some subgroups H; < Sn. In Section 5.1 we have seen how Th = (5 Following Benkart et al. (2016). we get (RR)) @k = -2 R../S S../ where ae's 's are positive integers.
This shows that representation spaces of IGNs are direct sum of R.5./((- for some t. But such (n - t, tt) are only but a fraction of the partitions of n. therefore S (in-i, 's are only but fraction of the subgroups Sx which are only few of the subgroups of S, as they are not transitive on [n] for x + (m), unlike other subgroups such as the alternating group, A,, Indeed, the module R.S.AA AM will be a two-dimensional representation compatible with point-wise activations. Maron et al. (219a) and Geerts & Reutter (2022) prove lower and upper bounds on the expressiveness of k-order IGNs. In the proofs of these bounds, they implicitly employ the decomposition (R*)**
of S5.0,..5///nnm.... as this equality is strongly related to the decomposition in tensors of different partition type. This makes it natural to ask what the expressiveness of models employing other types of admissible spaces would be.
6.3 GEOMETRIC GRAPHS AND PRODUCT GROUPS
Geometric graphs (Joshi et al. 2023 Finkelshtein al.) 2022 Bekkers al. 2023 Gasteiger al., 2022) are utilized as a data structure for modeling systems in computational biology, computational chemistry, and computer graphics. Those are graphs, or higher-order structures, embedded in the Euclidean space and as such may transform according to the symmetries of the ambient space, i.e. isometries of TR3, F(3). Hence, becomes relevant to develop neural networks simultaneously equivariant to such ambient symmetries and node permutations, and in algebraic terms this means that we need B(3) x Sn-equivariant networks able to process elements in I @ (RR")*, where the first tensor factor represent the geometrical features and the second the relational structure. We synthesize the results concerning this case in the following corollary of Theorem A.6 for the proof.
@k
2 See Appendix
Corollary 4. Every SO(3) x Sn-equivarian layer defined on Rd @ (R")) @k coupled with non-affine activations is null. Hence, E(3) x S, equivariat networks are rotation invariant.
CONCLUSIONS AND FUTURE DIRECTIONS
In conclusion, we have provided complete characterization of networks with equivariant layers featuring point-wise equivariant activations up to isomorphism of the linear part. Our analysis inves- tigates relevant examples, including IGNs and rotation-equivaaiiat CNNs. Future work will involve investigating equivalent results for complex-valude networks, as this extension is made possible by the novel algebraic approach that we have introduced for the proof of Theorem T Since complex numbers admit non-dense non-discrete multiplicative subgroups, this property should alleviate the limitations of real-valued networks equivariant with respect to non-discrete groups. Future research directions beyond wisn activations introduce us to wide domain with no explicit guidelines for navigation. Various endeavors have been documented in the literature, some of which are norm activations (Worrall et al. 2017), squashing activations (Sabour et al.) 2017). tensor product activa- tions (Konorr 2018), and gated activations (Weiler et al. 2018). An instance of activation functions amenable to theoretical analysis can be found in equivariant polynomial functions. Structures such as symmetric polynomials have been studied in mathematics for long time and are beginning to provide insights for the construction of efficient equivariant models (Puny al. 2023).

---

Erik J. Bekkers, Sharvaree Vadgama, Rob D. Hesselink, Putri A. van der Linden, and David W. Romero. Fast, Expressive SES(n)S Equivariant Networks through Weight-Sharing in wwtp://rgggabb//.beeeobee Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENT
Bruno Lepri acknowledges the support of the PNRR project FAIR Future Al Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU and the support of the European Union's Horizon Europe research and innovation program under grant agreement No. 120277 (ELIAS).
REFERENCES
Arash Behboodi, Gabriele Cesa, and Taco Cohen. A PAC-BBayssin Generalization Bound for Equiv- ariant Networks. Advances in Neural Information Processing Systems, 2022.
Erik J. Bekkers. Maxime W. Lafarge, Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, and Remco Duits. Transsaannntionn Covariant Convolutional Networks for Medical Image Analysis. Roto- Translation Covariant Convolutional Networks for Medical Image Analysis, 1st Conference on Medical Imaging with Deep Learning (MIDL. 2018), Amsterdam, The Netherlands, April 2018.
Position-Orientaan Space, October 2023. URL arXiv:2310.02970 [cs, math].
Georgia Benkart, Tom Halverson, and Nate Harman. Dimensions of irreducible modules for par- tition algebras and tensor power multiplicities for symmetric and alternating groups, May 2016. arXiv:1605.06543 [math].
Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geo- metric Deep Learning: Going beyond Euclidean data. IEEE Signal Processing Magazine, 34(4): 18-42, July 2017. Conference Name: IEEE Signal Processing Magazine.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. arXiv:2104. 13478 [cs, stat], May 2021. arXiv: 2104.13478.
Taco Cohen and Max Welling. Learning the Irreducible Representations Commutative Lie Groups. In Proceedings of the 31st International Conference Machine Learning, pp. 1755- 1763. PMLR, June 2014.
Taco Cohen and Max Welling. Group Eguivariant Convolutional Networks. In Proceedings of The 33rd International Conference on Machine Learning, pp. 2990-2999. PMLR, June 2016a.
Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Rep- resentations, November 2016b.
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical CNNs. In International Conference on Learning Representations, February 2018.
Taco S Cohen, Mario Geiger, and Maurice Weiler. A General Theory of Equivariant CNNs on Ho- mogeneous Spaces. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Sander Dieleman, Kyle W. Willett, and Joni Dambre. Rotation-invariant convolutional neural net- works for galaxy morphology prediction. Monthly Notices of the Royal Astronomical Society, 450 (2):1441-1459, June 2015.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting Cyclic Symmetry in Con- volutional Neural Networks. In Proceedings of The 33rd International Conference on Machine Learning, pp. 1889-1898. PMLR, June 2016.
Naday Dym and Steven J. Gortler. Low Dimensional Invariant Embeddings for Universal Geometric Learning, May 2022. arXiv:2205.022 00200 [cs, math).
10

---

Published as a conference paper at ICLR 2024
Nadav Dym and Haggai Maron. On the Universality of Rotation Equivariant Point Cloud Networks. In Interaational Conference on Learning Representations, October 2020.
Bryn Elesedy. Group Symmetry in PAC Learning. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022.
Ben Finkelshtein, Chaim Baskin, Haggai Maron, and Nadav Dym. A Simple and Universal Rotation Equivariant Point-Cloud Network. In Proceedings of Topological, Algebraic, and Geometric Learning Workshops 2022, pp. 107-115. PMLR, November 2022.
Peter Flor. On groups of non-negative matrices. Compositio Mathematica, 21(4):376-382, 1969.
William Fulton and Joe Harris. Representation Theory, volume 129 of Graduate Texts in Mathemat- ics. Springer, New York, NY, 2004.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. GemNet: Universal Directional Graph Neural Networks for Molecules 2022. 35th Conference on Neural Information Processing Systems.
Floris Geerts. On the expressive power of linear algebra graphs, February 2020. arXiv:1812.04379 [es].
Floris Geerts and Juan L Reutter. Expressiveness and Approximation Properties of Graph Neural Networks. pp. 43, 2022.
Emiel Hoogeboom, Jorn W. T. Peters, Taco S. Cohen, and Max Welling. HexaConv. In International Conference Learning Representations, February 2018.
Truong Hy, Shubhendu Trivedi, Horace Pan, Brandon Anderson, and Risi Kondor. Predicting molec- ular properties with covariant compositional networks. The Journal of Chemical Physics, 148: 241745, June 2018.
Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pietro Lio. On the Ex- pressive Power of Geometric Graph Neural Networks. International Conference of Learning Representations, 2023.
Ramakrishna Kakarala. Triple correlation on groups. phd, University of California at Irvine, USA, 1992. UMI Order No. GAX93-04094.
Imre Risi Kondor. Group theoretical methods in machine learning. Columbia University, 2008
Risi Kondor. N-body Networks: a Covariant Hierarchical Neural Network Architecture for Learning Atomic Potentials, March 2018. 803.01588 [cs].
Risi Kondor and Shubhendu Trivedi. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups. In Proceedings of the 35th International Conference on Machine Learning, pp. 2747-2755. PMLR, July 2018.
Risi Kondor, Truong Son Hy, Horace Pan, Brandon M. Anderson, and Shubhendu Trivedi. Co- variant Compositional Networks For Learning Graphs. International Conference of Learning Representations, February 2018.
Leon Lang and Maurice Weiler. A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels. In International Conference on Learning Representations, 2021.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation Equivariant Vector Field Networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5058-5067, Venice, October 2017. IEBE.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and Equivariant Graph Networks. International Conference on Learning Representations, September 2018.
Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably Powerful Graph Networks. International Conference of Learning Representations, 2019a.

---

Published as a conference paper at ICLR 2024
Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the Universality of Invariant Networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 4363- 4371. PMLR, May 2019b.
Haggai Maron, Or Litany. Gal Chechik, and Ethan Fetaya. On Learning Sets of Symmetric Ele- ments. In Proceedings of the 37th International Conference Machine Learning, pp. 6734- 6744. PMLR, November 2020.
Horace Pan and Risi Kondor. Permutation Equivariant Layers for Higher Order Interactions. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pp- PMLR, May 2022.
Omri Puny, Derek Lim, Bobak T. Kiani, Haggai Maron, and Yaron Lipman. Equivariant Polyno- mials for Graph Neural Networks, 2023. In Proceedings of the 40th International Conference on Machine Learning
Charles Qi. Su, Hao, Mo, Kaichun, and Guibas, Leonidas J. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In 2017 IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 77-85, Honolulu, HI, July 2017. IEEE.
Siamak Ravanbakhsh. Universal Equivariant Multilayer Perceptrons. In Proceedings of the 37th International Conference Machine Learning, pp. 7996-8006. PMLR, November 2020.
Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic Routing Between Capsules. Ad- vances in Neural Information Processing Systems, 2017.
Bruce Sagan. The Symmetric Group, volume 203 of Graduate Texts in Mathematics. Springer New York, New York, NY, 2001.
Akiyoshi Sannai, Masaaki Imaizumi, and Makoto Kawano. Improved generalization bounds of group invariant / equivariant deep networks via quotient feature spaces. In Proceedings of the Thirty-Seventh Conference Uncertainty in Artificial Intelligence, pp. 771-780. PMLR, De- cember 2021.
Laurent Sifre and Stéphane Mallat. Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1233-1240, June 2013.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-cquivarinn neural networks for 3D point clouds, May 2018.
Maurice Weiler and Gabriele Cesa. General B(2) - EEqiivrrian Stecrable CNNs. Advances in Neural Information Processing Systems, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D steerable CNNs: learning rotationally equivariant features in volumetric data. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 10402-10413, Red Hook, NY, USA, 2018. Curran Associates Inc.
Jeffrey Wood and John -hylor. Representation theory and invariant neural networks, Discrete Applied Mathematics, 69(1-2)) August 1996.
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic Networks: Deep Translation and Rotation Equivariance. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7 7168-7177 July 2017.
Dmitry Yarotsky. Universal approximations of invariant maps by neural networks, April 2018.
arXiv:1804.1 10306 [cs].
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep Sets. In Advances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017.
12

---

Published as a conference paper at ICLR 2024
Ding- Xuan Zhou. Universality of deep convolutional neural networks. Applied and Computational Harmonic Analysis, 48(2):787-794, March 2020.
Aaron Zweig and Joan Bruna. A Functional Perspective on Learning Symmetric Functions with Neural Networks. In Proceedings of the 38th International Conference Machine Learning, pp- 13023-13032. PMLR, July 2021.
A APPENDIX
A.I TOPOLOGICAL SPACES
Definition 2. topological space is pair (X,7), where X a set and is a collection of subsets of X such that:
1. belongs to T.
2. The intersection of any finite number of sets sets
T is also T.
3. The union of any collection of sets in T is also in
The in T are called open sets, and the collection is called the topology X.
Example (Discrete Topology). The discrete topology set X is the topology in which every subset of X an open set, i.e., T = P(X), where ((X) is the power set of X.
The set of integers E with the discrete topology is topological space. In this topology, every singleton set {n.} open for all n E Z.
Definition 3 (Continuous Maps). Let (x, and (Y,Ty) be topological spaces. function f X + Y is said to be continuous for every open set V in Y, the inverse image f-²(V) is an open set in X.
Definition 4 (Compact Spaces). An open cover of topological space X is set U C such that X Urrez U. A refinement of cover U of X is a subset u which still a cover. A topological space X is said to be compact every open cover of X has finite refinement.
Theorem 3. subspace S IR" is compact if and only if is closed and limited.
Definition 5 (Connected Spaces). topological space X is said to be connected if cannot be expressed as the union of two disjoint nonempty open sets. A subset of X who is maximal under containment order and connected called a connected component of X.
Theorem 4. If X and Y two topological and f X + Y is continuous map then the two following statements hold
if X is compact then f(X) is compact
if X connected then f(X) is connected
A.2 GROUP THEORY
Definition 6. A group is pair (G.-) where G is a set and G G G is a function satisfying the following axioms.
Associativity: for each g, h, Is G we havelg h) k k= 9- kh-
Identity: there exists an element G such that g == e-g- g= g for each g G.
Inverse Element: for each element g E G. there exists element g G such that
g.g =g =.
group is finite if it contains finite number of elements. A group is abelian or commutative if gh bg for euch g, h E G.
Aggiungere ref,, va bene Hatcher?
13

---

Published as a conference paper at ICLR 2024
Example 2.
Here present some fundamental examples of groups.
The set of integers with addition.
The set S' } of rotations of angle a centered in the origin of the 2D Cartesian plane with composition.
The set GL.(V) of bijective linear maps of a vector space V into itself with composition. Then, the set GLn(IR) of n. real invertible matrices form group where the operation is row-column multiplication.
Let (O,(R)) or simply SO(n), be the group of orthogonal matrices with positive determi- nant.
Fix [n] {l....,.. n.. The set
S. = (f : [m] + [m] is bijective)
with the composition operation form the symmetric group or the permutation group.
Given two groups G and H, the direct product G x H of them is still a group. The set of the elements is the Cartesian product of G and H while the sum is defined
(g1,hg 9GxH (92, hz)) = (91 9G 92. hr oh hz).
Now, we introduce notion of group homomorphism, a transformation between groups which pre- serves the operation.
Definition 7. A group homomorphism is a map
::
between G and H groups such that, for each g. EG
$0g h) = x(g) ((h).
Example 3. The map 4; 8 (R) defined by
(as sin
sin(a) cos(a)
Pa
is an homomorphism between the group of rotations of angle and the 2 x 2 invertible matrices. Definition (Quotients). Let G be a group and H be a subgroup of G. The quotient of G by H is the set G/ H {6H G}, where gH {gh h E H} are the left cosets of H.
Example 4. 1. Consider G = Z and the subgroup H = nā of integers multiples of n. The quotient G/ H is a group and is isomorphic to the cyclic group of n elements, Zen i.e. integers modulo
2. Consider G Ss. the symmetric group three elements, and the subgroup H {(1), (12)}. The quotient G/H is group and is isomorphic to S2, symmetric group on elements.
3. Consider G Sn, the symmetric group on n elements, and the subgroup H1 An, the alternating group on elements. The quotient G H is a group and is isomorphic to Loz
A.3 TOPOLOGICAL GROUPS
Definition 9. A topological group is a group equipped with topology such that multiplication and inversion continuous maps.
Example 5. der addition.
The real numbers R with the standard topology form topological group
The multiplicative group of non-zero real numbers R with the standard topology a topological group.
14

---

Published as a conference paper at ICLR 2024
Each finite group G with the discrete topology is topological group. Note that the nected components of this group are all the singleton elements {g) for each E G. Note that this group is compact P(G) is finite, hence each open cover is finite.
The additive group of integers Z with the discrete topology is topological group. Note that the connected components of this group are all the singleton elements {n} for each n E Z. All this singletons form an infinite open cover of & which cannot be refined.
A.4 REPRESENTATION THEORY
Definition 10. representation of group in a vector space V is group homomorphism
p: G + GL(V).
Definition 11. Consider G SSw
dim V and p(g) id for each g Sa. This called the trivial representation of Sn. dim V and p(g) sgn(g)id 'dd This is called the sign representation of Sn-
Definition 12. representation p
GL,(RR
Non-negative if all the elements of each p(g) are non-negative.
Monomial if each row and colunn of each p(g) has exactly one non-zero element.
A permutation representation if it monomial and each non-zero element
A signed permutation representation if it is monomial and each non-zero element is +1.
Definition 13 (Representations of Topological Groups). Let G be topological group and V be real vector space. A continuous representation of G on V is continuous group homomorphism p: p:G G GLVV..
Example 6. Consider the general linear group GL(V) of invertible nxn matrices, with the topology induced by the Euclidean norm on matrices. Let V = Rn" be the standard Euclidean space. The map Av is continuous representation. Similarly, restricting this representation to 50(n) we get a representation of SO()..
p G + GL(V) defined by (())(v)
W is G-equivariant with respect to the representations P1: P ~ GL.(V and p2 G GT.(W) if
Definition 14. A linear map /b : V +
P2 -
We will denote the space of all G-equivariant maps between p1 and p2 by Home(p1 p2) or Homo(V, W) when on and P2 will be clear from the context.
Definition 15. representation p space W of V such that (W)) S W for each G.
GL(V) is irreducible if there exists no non-trivial sub-
An important result in representation theory of finite groups states that there is always decomposi- tion into irreducible representations.
Theorem 5. For each representation p: + GL(V). there exists decomposition
V V1 @ Vm
where each V is irreducible for p. This decomposition unique up to isomorphism and permutation of the factors.
Another tool coming from Representation Theory is Schur's Lemma which will be fundamental to understand our results.
Lemma Let and W be non-isomopphic irreducible representations, there is only one G- equivariant linear map between them and it is the trivial one.
Definition 16.
The fixed set for representation p G GL(V) is y = {v: gu v}. Note that VG is representation for G and the action is trivial.
Theorem 6. The following properties are true for representations of finite group G.
dim V the multiplicity of the trivial representation in V,
dim Homa(V, (V.W) dim(V & WyC.
15

---

Published as a conference paper at ICLR 2024
A.5 AFFINE MAPS
Definition 17. Let V and W be two K-vector spaces and define the translation of a vector w in W as a non-linear bijective map Tio - w. Define the space of affine maps from V W
AV, W) (Tw flw E W and f (oom(
Note that is a more general definition with respect to the standard one, where f is an isomorphism of vector space V.
Theorem 7. The decomposition of an affine map E Af, W) in translational part To and linear part is unique.
Proof.
Twi Si = -== Twz f2.
evaluating in 0 leads to
W1 = x(0) W2-
Write w1 w2. and note that by the bijectivity of translations,
Tw [1 = Tw o 12
S1= f2.
Let V and W be G-representation. An affine map ó E Af, W) is G-equivariant if o g for each g G, write the of G-equivariant affine maps from V W as AVfa(V
Theorem 8. á Tw f Aff W) and i E Home W) and is invariant.
Proof. Note that for each g G.
0 Tw o = ggw o (g f).
Observe that if and only if and only by the previous proposition.
dog g god.
Th f o g T Ta 0
Jgw (g f)
gw
and
fog=gof
for each g G.
A.6 REPRESENTATIONS OF GROUP PRODUCTS ON TENSOR PRODUCTS
Remark 1. Lei V W be finite-dimensional G x H-representations and V,'s 's's complete list of irreducible G-representations and W, 's a complete list of irreducible H H-vepresentations, then V. & W, 's is complete list of irreducible (G x H represenntations (See Sagan (200]) for proofs in case G and H are finite). m, is the multiplicity of V; in V and nj is the multipicity W,, in W then the multiplicity of V; @ W, V T W is m.n.j. This be easily seen by writing the irreducible decompositions of V and W and use the distributive property of tensor products and direct sums. Note that the same is true if G and H are compact groups and the representations are continuous. If S is an G-isotypic component of V x W of type p then it is H --imvariant and each h E H acts as G-equivariant endomorphism of S. By Lemma 2 S p 2 G. which + p E T where a; are irreducible H -rrepresentations Iterating the decomposition if necessary and by the finite dimension of V and W, we conclude.
Thanks to these observations can prove Corollary
16

---

Published as a conference paper at ICLR 2024
Proof. Let study SO(3) S, eqqiivriiat linear layers. Irreducible representations of (3) x S are the tensor products of irreducible representations of (3(3) and S, as shown above. The natural action of (3)3) on R3 is an irreducible representation and admissible irreducible representations invariant, i.e., the trivial action SO(3) on IR. Hence, the fist layer of an 5O(3) x S, -quequiirri network processing geometric graph will be linear layer from direct of IR3 @ S direct sum of IR @ S", where S and S" indicate irreducible representations for Sn. But, due to Schur's Lemma, the only possible linear layer would be the null one, i.e., layer without trainable parameters.
A.7 PROOF OF THE STABILIZATION LEMMA
For convenience restate Lemma
Lemma 3. The group of malrices M' M(F(M)) is the largest group in GL,(R) for which F(M)) = F(M), and F F(M(F)) is the largest family of functions in C(TR) for which M(F)= MMFF.
In what follows we state and prove some results necessary for the proof of Lemma the next lemma is trivial.
The proof of
Lemma 4. The two following statements true.
1. For each group of matrices M1. M1 M(F(Mi)) and for each family of functions F1, Fi F(M(F))
2. For each inclusion Mi C. M2 of groups of matrices, F(M2) S F(M1), similarly, for families of activations Fi E F .MF2)) M(FF)
Lemma For each family of functions Fú, M(Fi) M(F(M(Fi))) For each group of matrices MI, F(Mi) F(M(F(Mi)))
Proof. We prove the equality MI, MMFi) M(F(M(FI)) first by proving MI, M(Fi) S M(F(M(Fi))) then we prove the opposite inclusion.
Substituting Mi with M(Fi) in the first point of Lemma get M(Fi) C M(F(M(Fi)).
To prove the opposite inclusion, substitute F2 = F(M((i)) in the second point of Lemma 4 we obtain ((F(((F))) S M(Fi). Implying M(Fi) M(F(M((F))) In a similar way, can prove the other equality.
We are now ready to prove Lemma
Proof. We only prove the first equality as the second have an analogous proof. By Lemma 3 we know that F(M') = F(M). Then, for each S C GL, (R) such that F(S) F(M), know that S C M(F((S) by Lemma Moreover, S C M(F(S) = M(F(M) = M'. Hence each C GL, (R) such that F(S) F(M) we have S M., therefore M M(F(M)) is the
largest group in GL, such that F(M'=
A.8 PROOF OF THE MAIN THEOREM
We can now state our main result, which shows that, under mild assumptions, there are only a very limited number of maximal groups (or equivalently, of maximal function classes). Moreover, for each these we provide the exact correspondence between M and F, thus providing an exhaustive classification of all possible admissible pairs of M and F.
A class of functions fundamental for the understating of what follows will be T-multiplicative func- tions which we define follows.
Definition 18. Let T be multiplicative subgroup of R. We say that continuous function f is T mulliippicaatiee if ftxx) = tz)x) for each t E T and x E R. We write Fr to indicate the set of all continuous T T-multplicatiiv functions.
17

---

Published as a conference paper at ICLR 2024
Note that if (b")nez, the notion of T multipipicaattttttyy reduces to the notion of b-multiplicatiiity provided in Section 4
Similarly, for T = (=b")ne2 and #6-multiplicatiitty.
We now ready to state Theorem 9 one of the two key results fundamental to prove Theorem [ In particular, Theorem 9
characterizes admissible pairs indexing them as the multiplicative subgroups of R but does not provide constructive description of their families of activation functions, this description is given by Lemma 10 In what follows we will write R, for the set of all unit row invertible matrices.
Theorem 9. Maximal admissible pairs for T ((M) are
Af R)) for each group of non non-monomial matrices M in (R) and (GL,, (R), Hom R)) if is dense,
(T,, otherwise.
Let us denote R' as the set of non-zero reals and note that it is a multiplicative group. In the following, will use the following characterization of the multiplicative subgroups of IK" that need to prove Lemmal I0
Lemma 6. Multiplicative subgroups of R250 are the trivial group (1), discrete unbounded groups (Ezz, and dense subgroups of Rzo- Multiplicative subgroups of TK" the trivial one, (1), the other finite one, {=1), discrete positive ones, (b"), the other discrete ones, (+b"")ne the ones dense on Rxo, and dense ones.
Proof. Multiplicative subgroups Ry5 and additive subgroups R are linked by the exponential map which is an isomorphism. Additive subgroups of I are divided into three different type: finite (()),, unbounded and discrete (aZ for each a E Ro and dense. They map through into R as finite (()),, unbounded and discrete (fb" e"a pnn Z) and dense. We can get the multiplicative
subgroups of R* noticing that R L2 X R>o.
The following technical lemmas will be necessary to prove Lemma
Lemma 7. Let b a real number, b > 1, and T = nez, For each positive real number æ there exist unique decomposition b"y such that y (1,b) and Z.
Proof. Choose such that [b", b we have that b"y where we define y 5 E [1, b). Sets {[b", )eez are collection of disjoint intervals covering Hence x belongs only to one of those, say x E 1b".b"=1 we have the uniqueness of the decomposition z = b" such that y E b).
Note that Lemma limples that for each R> there exist a unique n E Z such that z E b). This observation grants the following notion to be well-defined. Let [1. b]
in - R be the function where n the
such that (b)) bry(1). Define fn R>0 + as the function f,(x)) 6d 7 x
only integer such that = E [1, b). Clearly S,, is T-multiplicative for T (b"), IEEZ where b is real number greater than We now state and prove Lemma & which gives a complete description of T multiplicatiie functions for T (b") Furthermore, provides constructive procedure offering a computational implementation of such functions.
Lemma 8. Let > be real number, and let T = (bz) A continuous function f is multiplicative if and only if there exist two continuous functions 7 [1, b] + R. such that 77 (b) bn (1) and
(x)
x >>> 0
f(x)
0
==0
33)
(-x)
x <
Proof. (=) Define 77 f[11110)
Multiplicativity and continuity of f implies all the required
properties n
(+) We only need prove that constructed in this way is continuous R. We will prove the
18

---

Published as a conference paper at ICLR 2024
continuity of f on Rz0 and Reo, and in O. The two former cases boil down to proving continuity of fn= respectively. As those proofs are analogous, we only present the one for fn_. Note that for each Ry0 T, there exists an interval I [ Rxo's [T and an integer such that fr_ (z) b" n( (+)
for each x E T. Hence fat is continuous on Ryo \ T. Now see that
lim Snx (x) = lim 6" (x) = lim 8" g0 = lim Se+ (x) 2-4b-+
x-1+
x-b
I-th"
because 17- (7-(6) (1). It remains prove continuity of / in 0, i.e.,
lim frz (x) --
lim x---
(x).
Note that 7/- is continuous and limited as it is defined on a compact subset of IR, hence we can define m = m1,, 77+ (x) and M = maX 77 (z). note that fm-lu.a S fn+ S fM-11.9) where 1er11 is the constant function with value 1 defined on [1, bj. It is easy to see that lim,-sot fm-1p lim SMM.Ip.M O.
We now state Lemma 9 whose proof will essentially contain the proof of Lemma I0. Lemma 9, Each T T-multiplicative continuous function is linear if and only if'T is dense in R.
Proof. Note that if f is T-multiplicative, by definition, f(t) (f(1) for each f. E T. As T is dense in E. f is linear on its entire domain R by continuous extension. If is dense in R>0, f(t) = t1) and f(-t) t 1) for each & > 0, hence f is semilinear. On the other hand, if T is not dense, suppose 7 is (1) or (+1), then Fr are C(RR) and odd functions respectively, which contain non- linear functions. If T (b") Let us now proceed with the construction of a function that is continuous, T T-multiplicative, and non-lineat Let 77 [1, b) + IR be two continuous bump functionn and let ʃ be continuous function as defined in Equation 3 this function is multiplicative and non- linear. This concludes the proof as we have listed all the possible cases of presented in Lemma 6
Thanks to Lemmas 6 and [ the presented proof of Lemma 9 gives a complete characterization of T T-multiplicative functions with respect to multiplicative subgroups T of IR which we can state in the following Lemma 10
Lemma 10. IfT = (1), then Fr = C(R). IfT = (+1), then Fr = O(E), the odd continuous funtiiuuss functions described in Equation functions. IfT = (b") m0zz) for b 1, then FT are all the continuous
3 arising from two continuous n [1. b] - R such that (b) = (1). If = (==")nee for b> then Fr are all the continuous functions defined as in the previous case but n7+ = n- I
is dense on R. then Fr are all the semilinear functions. Finally, I is dense on R. then Fr are all the linear functions on TR.
To prove Theorem D we need state and prove the two following lemmas. The main ideas behind their proofs are primarily due to Wood & Shawe-Taye Taylor (199)..
Lemma 11. Define the multiplicative group T T(M) = EjES :, S [m], e M, E [x]) (oO and (z) = f(x)- - F(0). Note that f is M-equivariant if and only if fo is M-equivariant and (o is a M-invariant vector. In particular, if f is M-equivariant then fo is -uMliiuuiiiiiaa and f(0) or M Rn.
Proof. Note that f(Mx) = fxx) for each E T if and only if (o = f(Mx))
M fx)) - Mf = M for each x E I= if and only if fo(Mx) = M fo(z) for each x E RR"
and M (0) = fo( In particular, Mf( = f(0) if and only if M1f((0 = 10), where 1 is the vector with all ones, if and only if f(0) = 0 or M Rr, For each S C [ni] and each x E we have that fo fo(Efes Mrjz)= = Ejes Mrz.fo(z) for each E [n], hence fo is T (MM)muuiiiliicativ
as Sol (Maa)) = M.fo(r) for each E R and = 1,2 then Jo(M, M2x) = M, M2
Lemma 12. Let T = T(M) be a non- subset of R.. Then F(M) contains non non-affine func- tions if and only if matrices in M are T-monomial.
19

---

Published as a conference paper at ICLR 2024
Proof. (=) For each f E F(M), fo f- f(0) is m-mulipplicative and fo is M-equivariant by Lemma 11 Suppose M contains a matrix M which fails to be T-monomial, without loss of
generality may assume Mn ti and M12 not zero. Hence, fo is additive. Indeed, for each æ1, 22 E IR, we have that
fo(2) + +2) = (fo F₁6i + x222))
(M 2161
( + x202) e1) & folti 21) f2 x2) fo(x1) fo(22)
Therefore is linear, being both additive and continuous, this implies be affine which contra- dicts the hypothesis.
(+) If M contains only T T-monomial matrices, it is easy check that each T-multiplicative func- tion induces an equivariant activation, which are not all affine by Lemma
Now we are ready to present the proof of Theorem 9 For convenience in what follows we will write P n. the group of n permutation matrices.
Proof. We will study which are the groups of matrices M such that T(M) = 7 as T varies between
subgroups of R" If T is a dense subgroup of IR, Lemma 9 and Lemma 11 implies F(M) Af, R. if M C Rn F(M)
Hom R) otherwise. In those cases, ([Af( R)) S (R. and M(Hom(RR, R)) GL. J. For dense T, Lemma immlies that admissible maximal pairs are (M R,, ATT R)) and (GI.,, Hom(R, (R. R)), whose family of activations only contains affine functions.
If T is non-dense and non-trivial, by Lemma 12 have two cases: if F(M) only contains affine functions we reduce to the maximal ad admissible pairs (M S (R), AR, IR) and (GL,, (IR) Hom R)), otherwise, due Lemma 12 M < (T) and F(M) = Fr by Lemma 11 and M(Fr) = (T) because the inclusion (T) S .M(rr) is obvious and if M(Fr) contains non-monn-nnnee monomial matrices, Lemma 12 would contradict Fr containing non-affine functions.
Applying Lemma we get the admissible maximal pairs FT) Finally, if T' is trivial, M P,, hence we obtain the maximal admissible pair (Pn,,((R)) as verifying F(P..) C(TR) is trivial and Lemma 12 implies M(CRT)) MM((1)) P
We are now ready to prove Theorem I
Proof. Theorem classifies admissible pairs whose relevant families of activations are Fr where T varies on the multiplicative subgroups of R. Thanks to the complete classification of T- multiplicative functions, Fr. provided by Lemma I0, we can substitute them with their concrete counterparts enumerated in the statement.
In what follows we restate Theorem 2 and prove it.
Theorem 10. Let G be compact group and let us restrict to families of activations containing some nof-nefine functions. The only two maximal admissible pairs up to isomorphism of groups of matrices are
Continuous functions and permutation matrices,
Odd continuous functions and signed permutation matrices.
Proof. At first, we want to prove that each admissible representation of G is isomorphic either to a permutation representation or to a sign-permutation representation. We split the proof into two parts. In the first part, G' is represented by non-negative monomial matrices and, in the second, G is represented by arbitrary monomial matrices. Thanks to Theorem [ those two cases covers all the admissible cases for families of activations containing at least one non-affine function.
By Flor 1969). each bounded non-negative group of matrices is isomorphic to a permutation group of matrices by positive scaling of basis. Hence, the image of compact group in (T) with C Ry. can be written as a group of permutation matrices after positive scaling of basis. Hence, all the considered group of matrices isomorphic P, therefore reducing the pair (P., C(R)).
20

---

Published as a conference paper at ICLR 2024
Now consider T' arbitrary non-dense multiplicative subgroup of R. Each monomial matrix can be written as the product SDP, where diagonal matrix containing only =1, D positive diagonal matrix, and P is permutation matrix. Consider the map & is a continuous group homomorphism and that its image is a compact group of non-negative matrices. Indeed, Φ is just the absolute value function defined on all the elements of the matrices, hence it is continuous. Then, let P and S2D2P, be two monomial matrices as before, where Po and Pr are permutation matrices respectively representing permutations a and Then their composition Dy by permuting the diagonal elements through o, i.e. ( SoP which commutes with D₁ being both diagonal matrices. This proves that @ is an homomorphism.
SDP DP. Note that
(10(S2) D₃ Po DzP- where (S2)) is the diagonal matrix obtained
For what we proved at the beginning of the proof, there exists positive scaling, represented by positive diagonal matrix B, such that BDPB-1 image of Φb Note that for each SDP, after scaling by B, we obtain BSDPB-1 SP a signed permutation matrix. This is true because S and B commute being both diagonal matrices. Hence, all the considered groups of matrices are isomorphic to the group signed per- mutation matrices, therefore, therefore reducing the pair of odd continuous functions and signed permutation matrices.
P' is a permutation matrix for each DPP in the
SBDPB-
Now that we have proven that there only two isomorphism classes of admissible representations for G, we want to show that there is only one representation in each class with maximal family of activation functions. Let M be the image of positive monomial matrices representation of G. By Lemma 11 each admissible group of matrices M' isomorphic to A commutes with F(M') Fr for some T. The maximal family Fr is C(R) and happens when T (1) and M' are permutation matrices. Note that the isomorphism between M and some group of permutation matrices is always possible shown at the beginning of this proof. An analogous argument applies to groups of arbitrary monomial matrices.
We conclude by noticing that in general permutation representations and sentations are not isomorphic, e.g. the trivial representation and the sign representation of S, This proves that the two presented pairs are in general disjoint.
repre-
Note that compactness is required condition. Indeed, given b > 1, consider the following dimensional representation representation p Z + (R) IR" such that p(n)ær = b" The image p(2) is not bounded and hence is not compact. This means that p cannot be isomorphic to permutation representation whose image is compact.
21


Reviews:
[{"review_id": "ucVmH34I3v", "reviewer": "Reviewer_sg92", "summary": "The paper describes a theorem to characterize what group representations can be used given certain types of point-wise activation functions, and vice versa. A main result is that either permutation representations can be used, or trivial irreducible representations. The paper discusses several equivariant networks in context of the obtained results.\n\n[update: based on the rebuttal I raised my score]", "questions": "See above comments and smaller details below:\n\n1. On page 3 the symbol $\\mathbb{R}^*$ is used. What does the asterix indicate?\n2. I did not understand the introduction of the main result \"Although employing techniques analogue to already known ones, we highlight how hypothesis can generalize ... of the basis\" Apart from this sentence being too long, I really do not understand what is being said here, please rewrite.\n3. Theorem 1 could benefit from some subsequent, explicit examples. Take ReLU and e.g. a cyclic permutation group. Or any example you see fit. I think the paper remains abstract throughout and I think it could be helpful to see some common examples.\n4. Section 5.1 is also a bit abstract still, dispite being in the section for practical scenarios. Some laymans explainations of the quotient structure could be helpful. In particular I struggled with \"...an equivariant isomorphism between ... which is the standard representation space for permutations equivariant networks on sets\".\n5. Section 5.2 makes sense, but as discussed it is limited in scope regarding the use of irreducible representations exclusively. \n6. Could you clarify Corrolary 3. It is hard to understand what is about. In particular what is the significance of the connected component (are we talking about subgroups here?)\n7. \"... but not more general equivariant tasks such as segmentation or detection\" again this is an artifact of sticking to irreps.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a solid contribution to understanding the limits of equivariant neural networks in light of point-wise activation functions\n2. The paper is thorough and precise in its definitions and results (theorems and proofs)\n3. The paper is open about it's scope (I appreciate the comment above Section 4 about not addressing homogeneous spaces / infinite dimensional vector spaces) and defines promising directions for future research.", "weaknesses": "1. The paper is a theoretical contribution and emphasizes that the results are of practical value. However, I believe that in order for it to be of practical value the paper should be more accessible to a broader audience. Although formal and precise, there is little effort put in describing the main results in plain English.\n2. Although partially covered by clarifying the scope of the paper, I do think the notion of regular representations is insufficiently covered. By sticking to irreducible representations one naturally comes to the conclusion that if one wants to apply point-wise activations one has to use trivial (invariant) irreps. However, if one admits regular representations (either approximations in the full SO(n) case, or exact as in the discrete sub-group cases of SO(n)) then one is not limited to invariants as the regular representations are effectively permutation representations. The regular representations can also be defined in the continuous setting, though the discretization may introduce some equivariance error, however, this need not be a problem as one could use random grids to avoid such biases. See e.g. Kuipers-Bekkers-2023:\n\nKuipers, T.P., Bekkers, E.J. (2023). Regular SE(3) Group Convolutions for Volumetric Medical Image Analysis. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. LNCS, vol 14222. Springer, Cham. https://doi.org/10.1007/978-3-031-43898-1_25\nhttps://arxiv.org/abs/2306.13960\n\nAs for equivariant models with point-wise activation functions, it is often the case that internally the models use (non-trivial) irreps to parametrize the layers, and use an inverse Fourier transform to transform back to the regular representation basis, apply point-wise activation, then project back to irrep basis. I think including such approach in the discussions enable more people to relate to the result of the paper. Although commonly used in practice, I did find it hard to find references to such approaches. See e.g. the library e3nn ( https://docs.e3nn.org/en/latest/api/nn/nn_s2act.html ) and escnn (https://quva-lab.github.io/escnn/api/escnn.nn.html?highlight=fourier#escnn.nn.FourierPointwise). As for papers I could find [Cesa et al. 2021] Appendix H.2:\n\nCesa, G., Lang, L., & Weiler, M. (2021, October). A program to build E (N)-equivariant steerable CNNs. In International Conference on Learning Representations.\n\nand perhaps the following as it represents signals on a grid but performs convs in the Fourier domain\n\nCohen, T. S., Geiger, M., Köhler, J., & Welling, M. (2018, February). Spherical CNNs. In International Conference on Learning Representations.\n\n3. By the previous item, remarks such as (see abstract) \"we prove that rotation-equivariant networks can only be invariant\" is in practice more nuanced, and I consider the statement actually incorrect. Another example of a rotation equivariant method based on equivariantly obtained spherical grids (they call it equivariant mesh) is GemNet:\n\nGasteiger, J., Becker, F., & Günnemann, S. (2021). Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34, 6790-6802.\n\nEffectively, the paper works with scalar fields over the homogeneous space $\\mathbb{R}^3 \\times S^2$, as also discussed in recent work https://arxiv.org/abs/2310.02970 , which also fully operates in the domain of regular representations. I remark that this reference seem to have appeared only after the ICLR deadline so of course I do not expect this to have been included.\n\n4. In conclusion, I think the paper could improve on impact if the relation to regular group convolution methods is further clarified and if the main results are introduced in simpler terms. The claim that equivariant networks can only be invariant is I think an artifact of the choice of sticking to irreducible representations.\n\nThank you for the great work and for considering my comments. I'm open to discussing my comments as I could have surely misinterpreted some parts of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KhCuxwhykb", "reviewer": "Reviewer_uXzC", "summary": "This works considers equivariant neural networks, consisting of equivariant affine maps and pointwise nonlinearities that map between representations of a group $G$. They provide a characterization theorem, which simultaneously characterizes possible types of pointwise nonlinearities and maximal group representations that commute with the families of pointwise nonlinearities. This theorem is applied to several specific types of group representations that have been studied in the equivariant machine learning literature.", "questions": "1. In Section 6.1 / Section 6.3, why is the following case ruled out? Say I let $f$ be linear, so that $\\tilde f$ is just scaling and is hence equivariant to $SO(3)$. Then the output representation is the same as the input, and is in particular not invariant. This can be done in the $SO(3) \\times S_n$ case as well.\n\nOther notes:\n1. Typo: Page 4, sentence with \"Although employing techniques analogue to...\" has several typos that make it confusing, incluidng use of \"hypothesis\" and \"thesis\"\n2. Typo: Page 14, \"Aff\" is incorrectly typeset before Theorem 5", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. Substantially generalizes previous existing results, of which there are not many.\n2. The main result is strong, and remarkably shows that there are not too many pairs of maximal admissible families of activations and groups.\n3. The argument on page 7 for why it is better to use pointwise activations in the usual representation for $S_n$ on $\\mathbb{R}^n$ as opposed to a disentangled representation is nice. The equivariant linear maps and pointwise activation approach is universal with the usual representation due to Segol and Lipman 2019 (https://arxiv.org/abs/1910.02421), but can be very weak in a disentangled representation.", "weaknesses": "1. I find this paper quite difficult to read at times, and I think that it would benefit from more explicitly defining certain concepts and writing out certain arguments. In the proof of Corollary 3, I do not quite understand the application of Theorem 1; why does the representation have to be trivial? Also, Section 6.2 about invariant graph networks is very dense. See also questions.\n2. The applicability of these results to empirical equivariant machine learning is unclear. Already practitioners do not use pointwise nonlinearities for $SO(3)$ equivariant networks. Also, are there any expectations of potential benefits from using other representations of $S_n$ for invariant graph networks?\n\nOverall, I have trouble reading the paper in its current state, but I am very open to changing my assessment of this paper in the discussion stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hDYqjwNlNy", "reviewer": "Reviewer_3Kou", "summary": "The paper studies equivariance of neural network layers consisting of a linear transformation followed by a pointwise activation. The main result is a theorem characterizing largest admissible combinations of activation families and the associated groups for which equivariance holds. After that, the main result and its consequences are discussed in the context of several specific equivariance scenarios.", "questions": "I suggest to: \n- carefully revise the writing;\n- strengthen and clarify sections 5-6 by providing experiments and/or theorems so as to make it clear why the discussions there are important (it might be worth it to focus on fewer applications, but with more depth).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main result - Theorem 1 - seems to be new (though heavily relying on previous research, primarily Wood & Shawe-Taylor (1996)). This theorem provides an explicit classification of equivariant pairs of activation families and groups and seems to be a simple, but useful result. I have not spotted any mistakes in the proofs.", "weaknesses": "**Contribution.** I doubt that the paper in its present state is strong enough for ICLR. \n1. The main result looks like a relatively simple add-on to the study of equivariant networks by Wood & Shawe-Taylor (1996). Most results found in the appendix are either simple facts from the group and representation theory or are borrowed from Wood & Shawe-Taylor (1996) (e.g., the key equivariance condition (3)). Theorem 1 seems to be a relatively simple consequence of these results.    \n\n2. Though the paper includes a discussion of applications of Theorem 1 in Sections 5 and 6, I found these sections not easy to follow. They essentially consist of a long and not so well structured list of comments surrounding a few corollaries. The context of these comments is explained relatively vaguely, partly through references to other papers. It is not really clear to me why any of these corollaries and comments is actually important. The paper does not include specific examples, illustrations or experiments showing how Theorem 1 can, say, help improve designs of equivariant models. \n\n**Writing.** The writing is sloppy in several places, with typos and unclear grammar. For example, only in Lemma 5:\n- the index $i$ seems to be missing in the definition of $\\mathcal{T}(\\mathcal M)$; \n- the indices and coefficients are wrong in the formula for $f(\\sum_{j\\in S}M_{rj}x)$; \n- a word seems to be missing in \"It is a simple to check\".\n\nI could understand the proof of Lemma 5 only with some effort (especially the sentence \" And obviously each..\"). The sentence \"Hence, we can restrict our classification..\" after the proof of Lemma 7 seems to consist of two different statements, but it is unclear where the first one ends and the second begins. In Lemma 7: \"dived into three different type\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4aSXdJakM1", "reviewer": "Reviewer_merJ", "summary": "The paper studies the effect of pointwise activation on multilayer equivariant neural networks and establishes a characterization theorem of which activations can be used to construct a fully equivariant net for a wide variety of group and group action through the lens of representation theory. Notably, they give a unifying view for permutation-equivariant architectures, such as DeepSets and IGNs, which also contain some unexplored architectures; as well as showing that equivariant disentangled steerable CNN (in some definition) must be invariant if the activation is pointwise.", "questions": "Small typos: \n- Page 15, proof of Lemma 4. First sentence in the proof misses “for all i”. Second sentence should start: “Then \\sum_{j = 1}^n M_{ij} = 1” (missing “=1”). \n- Page 15. First equation in Lemma 5 missing “i \\in [n]” (missing “i”).\n- Section 5.2. It is more apt to reference Cohen & Welling 2014 \"Learning the irreducible representation...\" rather than Cohen & Welling 2016b \"Steerable CNN\" in this section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper highlights one key challenge in designing a fully equivariant neural network: namely the choice of activation function (common choices of which are not equivariant - e.g. commute with the linear part). Research into this topic is timely, highly relevant and important in the field.\n\nTheir framework generalizes existing work from Wood and Shawe-Taylor, and is tight in the sense that the characterization theorem enumerates all pairs of pointwise activation and linear layer that commute (and hence can be used in an equivariant network design). \n\nThe result for rotational equivariant architecture (that pointwise activation is equivariant iff representation of the linear layer is trivial) is surprising and may be crucial in opening up efforts to find other approaches to rotational equivariant architecture.", "weaknesses": "The paper is rather dense in representation theoretic notions. While I believe the paper is still self-contained, it may be beneficial for readers if the authors can expand on some intuition on the main theorems, maybe even in the appendix. \n\nIt doesn’t look like the technique in this paper can generalize to infinite dimensional representation, which the author acknowledged in Page 4. \n\nMinor: I think the paper would also greatly benefit from a further discussion of \"what's next?\", what is the loss if we give up point-wise activations for rotational equivariant network but replacing them with other kinds of activation. For permutation-equivariant network, it may also be interesting to work out another nontrivial architecture that is neither DeepSets nor IGN (higher order tensor representation). I'd also understand if the authors deem these as being out-of-scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)