REVISITING MULTI-PERMUTATION EQUIVARIANCE THROUGH THE LENS OF IRREDUCIBLE REPRESENTATIONS
Authors: Yonatan Sverdlov
Ido Springer
Nadav Dym
Venue: ICLR 2025 Poster (2025)
Keywords: deep weight spaces
permutation equivariance
irredicible representations.
Primary area: learning on graphs and other geometries & topologies
TL;DR: Characterization of all permutation equivariant linear mappings using irreducibles approach on several use-cases.

Abstract: This paper explores the characterization of equivariant linear layers for representations of permutations and related groups. Unlike traditional approaches,
which address these problems using parameter-sharing, we consider an alternative
methodology based on irreducible representations and Schur’s lemma. Using this
methodology, we obtain an alternative derivation for existing models like DeepSets,
2-IGN graph equivariant networks, and Deep Weight Space (DWS) networks. The
derivation for DWS networks is significantly simpler than that of previous results.
Next, we extend our approach to unaligned symmetric sets, where equivariance
to the wreath product of groups is required. Previous works have addressed this
problem in a rather restrictive setting, in which almost all wreath equivariant layers
are Siamese. In contrast, we give a full characterization of layers in this case and
show that there is a vast number of additional non-Siamese layers in some settings.
We also show empirically that these additional non-Siamese layers can improve
performance in tasks like graph anomaly detection, weight space alignment, and
learning Wasserstein distances.

Published as a conference paper at ICLR 2025
REVISITING MULTI-PERMUTATION EQUIVARIANCE THROUGH THE LENS OF IRREDUCIBLE REPRESENTA-
TIONS
Yonatan Sverdloy Ido Springer & Nadav Dym
FFcultty of Mathematics
FFauulty of Computer Science
Technion - Israel Institute of Technology
nadavdym& technionneeeiin....ommmmmo ac. ill
(yonatans, ido. springer) campus.. www.:us..tcciiii.c.......
ABSTRACT
This paper explores the characterization of equivariant linear layers for repre- sentations of permutations and related groups. Unlike traditional approaches, consider alternative
which address these problems using parameter-sharing. methodology based on irreducible representations and Schur's lemma. Using this methodology, we obtain alternative derivation for existing models like DeepSets, 2-IGN graph equivariant networks, and Deep Weight Space (DWS) networks. The derivation for DWS networks is significantly simpler than that of previous results. Next, we extend our approach to unaligned symmetric sets, where equivariance to the wreath product of groups is required. Previous works have addressed this problem a rather restrictive setting, in which almost all wreath equivariant layers are Siamese. In contrast, show that there is a vast number of additional non-Siamese layers in some settings. We also show empirically that these additional non-Siamese layers improve performance in tasks like graph anomaly detection, weight space alignment, and learning Wasserstein distances. Our code is available at GitHuu.
give full characterization of layers in this case and
INTRODUCTION
Learning with symmetries has recently attracted great attention in machine learning. In this learning setting, group acts on an input space, and the hypothesis mappings are restricted to be equivariant with respect to the group action. Motivated by the structure of fully connected neural networks, group equivariant models are often defined by composition of parametric linear equivariant functions and non-parametric non-lineat activations functions (Cohen & Welling, 2016). Thus, characterizing all equivariant layers for given group action is fundamental for understanding and designing equivariant deep neural networks. Indeed, this question has attracted considerable amount of attention in various scenarios (c.g., Finzi et al. (2021); Kondor & Trivedi (2018); Cohen et al. (2019); Pearce-Crump (2023a;b)).
For permutation groups, the standard strategy for characterizing equivariant layers is through studying parameter-sharing (Ravanbakhsh et al., 2017). Perhaps the first result in this direction given by Zaheer et al. (2017). who showed that only two parameters are required to characterize all permutation equivariant layers
of scalars. Another famous example is Maron et al. (2018), who described the specific parameter-sharing scheme for graph equivariant networks and other tensor representations of the symmetric group. Recently, parameter-sharing has been applied in Navon al. (2023); Zhou et al. (2024b) to characterize all equivariant layers of weight space neural functionals. These neural networks operate on weights of another neural network, problem with complex multi-permutation equivariant structure.
In this paper, we revisit equivariant linear layer characterization for permutation groups from the perspective of irreducible representations. Given group G acting on a linear space V, we explore
"Equal contribution

---

Published as a conference paper at ICLR 2025
linear equivariant mappings f - V by decomposing V to its minimal invariant components, which are irreducible representations of G. Once this decomposition is performed, a basic result in representation theory called Schur's lemma automatically provides us with full characterization of all equivariant mappings.
As first step, characterization at the basis of the DeepSets (Zaheer et al., 2017), 2-IGN (Maron et al., 2018), and DWSNets (Navon al., 2023) architectures. Notably, for the Deep Weight Space problem, our derivation is significantly simpler than the cumbersome derivations previously obtained by parameter sharing (Navon et al., 2023; Zhou et al., 2024b).
method to get an alternative derivation for known equivariant layer
Next, we consider the problem of sets of unaligned symmetric elements, where equivariance to the joint action of group 9 and the permutation group S₃ is required (also known the wreath product G l S.)). This setting arises naturally when considering alignment, distance prediction, or anomaly detection of sets of unaligned symmetric elements. the assumption that G acts transitively, in which case Siamese Networks capture an overwhelming majority of the equivariant layers. Indeed, in practice, Siamese networks are often employed (Navon et al., 2024; Chen & Wang, 2024). In contrast, we give full characterization of the equivariant layers in the general setting and show that in some settings (c.g.. alignment of weights spaces), there can be large number of -on-Sieeese equivariant layers. Empirically, we show that these additional layers improve performance on a synthetic graph anomaly detection task and the deep weight space alignment task discussed in Navon et al. (2024). In summary, our contributions are the following:
was studied by Wang al. (2020) under
We give an alternative irreducible-based derivation for the DeepSets (Zaheer et al., 2017), 2-IGN (Maron et al., 2018), and DWSNets (Navon al., 2023) architectures. The DWSnets derivation is significantly simpler than previous approaches.
We provide characterization of all equivariant functions on sets of unaligned symmetric elements.
We empirically validate the importance of our equivariant layers for sets of unaligned symmetric elements by showing its superiority on synthetic graph anomaly detection task, Wasserstein set distance computation, and deep weight space alignment task.
2 RELATED WORK
Learning Sets of Symmetric Elements. When dealing with multiple occurrences of symmetric objects as a set, additional symmetries arise as the set elements can be permuted. In this case, the layers are required to be equivariant under the permutation of elements in addition to the original element group action. Maron et al. (2020) suggested a new layer called Deep Sets for Symmetric set of symmetric objects. They showed that incorporating DSS layers is strictly more expressive than using Siamese networks. The DSS framework was also used in sub- -rbbgrapp aggregation networks (Bevilacqua et al., 2021), where the network should be invariant to the order of the sub-graphs. In the main setting of Maron et al. (2020), the same group element g E G acts uniformly elements 9: E G to act on the objects. This corresponds to the action of the restricted wreath product G S& on the set. Maron et al. (2020) deal with this setting as well, but only when G is a subgroup of S, and the action of G is transitive. Later, Wang et al. (2020) characterized all linear mappings for general wreath products, however they also assumed that the group actions are transitive.
Elements (DSS) that operates
all set objects. However, the more challenging setting is where we allow different
Equivariant Characterization Via Irreducibles. Using irreducible representations to characterize equivariant layers is a popular approach for rotation-equivariant networks (Thomas et al., 2018 8; Anderson et al., 2019; Dym & Maron). For permutation groups, the primary work which considered this approach, to the best of our knowledge, is Henning Thiede et al. (2020). They suggest using the known characterization of permutation irreducible representations with Young diagrams (Fulton & Harris, 2013) to characterize permutation -euiiiiriinivvviiint layers and explain how to reconstruct the results from DeepSets and 2-IGN. Our approach for these results is similar but gives a more explicit derivation of the DeepSet and 2-IGN layers. More importantly, our analysis covers deep weight spaces and sets of unaligned symmetric elements, not discussed in this paper.
2

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
Let V be a finite-dimensionll vector space over the field IF C or R. Let & be a finite group acting linearly on V. We will sometimes say that V is representation of g. We will say V is trivial representation of G if the action of g on V is the trivial action g = v for all E G and v e V.
A subspace U S V is invariant to G if g subspace U is irreducible if it does not strictly contain an invariant subspace except the zero space. By Maschke's theorem, each finit-diimensionll space V can be decomposed into direct sum of irreducible invariant spaces (Fulton & Harris, 2013).
LI for all G and u E U. We say invariant
mapping T' : > + V is equivariant T v) g - for every v V.g E g. Two representations V.u of G are isomorphic if there is an equivariant linear bijection Lv V - U, which we notate by u.
Tvv)
The following lemma, named Schur's lemma, is a key component of our work. Schur's lemma describes equivariant mappings between irreducible representations:
Schur's lemma. Let V. W be mnite-dimensionll irreducible representations of G over F, and let T: V W be G-equivariant linear map" Then:
* Either T' is an isomorphism or T = 0.
If V + W is an isomorphism, and FF = C, then T = XL for E
In this paper, we will only consider representations over R. which are the common setting in applica- tions. In all cases, we will discuss, our real irreducible representations are absolutely irreducible. This means that their natural extension to complex vector spaces is irreducible. In this case, if L V W is an isomorphism, F-- = R. and V is absolutely irreducible, then T = XL for À E IR (Boardman). In Appendix B we elaborate more on this topic. We note that when is not absolutely irreducible, the space of isomorphisms from V to W is either 2 or 4 dimensional (Poonen, 2016). In this setting, using an automatic computational method find all 2-4 equivariant layers may be beneficial (Finzi et al., 2021).
Layer characterization using Schur's Lemma Equivariant layers from a representation V to itself can be characterized using Schur's lemma via two steps. The first step is decomposition, where identify the decomposition of V into irreducible representations written uniquely sum Ei= I; where each o; is in V, The decomposition step also requires an algorithm to compute this decomposition. Next, we will need to identify which the space pairs V4 and V, are isomorphic, and if they are, specify isomorphism Tuj- Finally, since are working over the reals, we will need to verify that all irreducible representations are absolutely irreducible.
+ V4. Any 2 E V can be
Once the decomposition step is carried out, the second step uses Schur's lemma. Equivariant mappings T V + V can be written as T = -E,j=1 Tij. where Tij V; + V, are equivariant. By Schur's lemma if V- and V, are isomorphic then Te, = >AgTTyy Lyj for some X E IR, and otherwise Tiy = 0. All in all, we obtain that T' is equivariant if and only if for some choice of the scalar dig we have
T( +... 1...t +...+ +x2)= = Σ AzsLe((z,).
(I)
(i)),
The number of parameters in the expression above depends on the isomorphism relations between the irreducibles. We can assume without loss of generality that the first < < s irreducible spaces are not isomorphic, and by identifying isomorphic irreducibles, we can rewrite the decomposition above as V = 1> (Where y is defined to be the direct sum of aj copies of V,)) where a1 +.+..++a aa = s. The number of parameters Aij in the dccomposition is then Ei_ af- This process can easily be generalized to compute mappings between representations V and W by decomposing both spaces into irrducibles.
We note that the connerstones of this methodology: decomposition into irreducibles and Schur's Lemma, are applicable for all finite dimensional representations of finite groups (and also for compact
"The equivariant map often referenced G-module.
3

---

Published as a conference paper at ICLR 2025
infinite groups like (O(d).. The main challenge in this approach is characterizing and computing the decomposition into irreducibles. This needs to be done on a case to case basis. Much of the remainder of the paper will be devoted to computing these decompositions for important equivariant learning scenarios.
4 COMPUTING LINEAR EQUIVARIANT LAYERS
We now demonstrate how to derive the results in Zaheer et al. (2017); Maron et al. (2018); Navon al. (2023) via decomposition into irreducibles.
4.1
DEEP SETS
We begin with the simple case of the action of the permutation group S, on V = R" by permutation of elements (we assume n 2 2). Here, there are two non-trivial invariant spaces:
S = fa 1 IR), V(n) = fxr E R Σ 24 -
As the first space is one-dimensional, it's irreducible. By Hinton al. (2006), the second one is also irreducible. These two spaces are not isomorphic since the action of S, on S is trivial, but the action of S, on (nn) not (also they do not have the same dimension when 2 3).
The decomposition of x E R" to a sum of elements from the irreducible spaces can be computed as
I = f1 (x nl,)
where T is the average of and 1, is the all one vector. By Equation (1), Schur's lemma, and the fact that all irreducibles of the permutation group are absolutely irreducible (see Appendix B), the linear equivariant mappings T V > V are characterized by two parameters a, E IR, that is:
Tr = aBLn + b(x - xln).
This is exactly the result described in DeepSets (Zaheer et al., 2017).
4.2 EQUIVARIANT GRAPH LAYERS
We next consider the setting where V = Rax" 9 = Sn and the group action is defined by (T X) X, 1()),r (()). This setting natural for graph neural networks, as two adjacency matrices A, B E Rax= represent isomorphic graphs and only if A = T B for some permutation т. This setting was considered in Maron et al. (2018) using parameter sharing scheme, and we show how to obtain similar results using irreducibles (note that Maron al. (2018) also discussed general mappings between k-order and ê-order tensors. In contrast, we only discuss the k = & = 2 case).
We claim that, when n 2 4, the space R x* can be written as a sum of seven irreducible permutation invariant sub-spaces. The behavior for < 4 is discussed in Appendix C.3.
The first two spaces are one-dimensional representations on which action of S, trivial: diagonal matrices with identical diagonal entries and matrices with zero diagonal and identical off-diagonal entries:
Vo= {a- Lala R}. V1 {a- (Ixxn In)la E R}.
Next, we have three spaces of dimension n - which are isomorphic to V(n) from equation 2. The first space is the space V/2 of diagonal matrices whose diagonal sums to and the next two dimensional spaces are the space of matrices whose rows (respectively columns) are constant, and columns (respectively rows) sum to zero:
V33 = {r1
Σ r = 0), V42 = (1
Σ
C ===0 0}
Finally, there are larger irreducible spaces:
V/3 (A)
A -- Al 0n), =o= (Al A AT A1, On, A, 0, Vi I.....n
4

---

Published as a conference paper at ICLR 2025
The dimension of V/3 is (²=²) n'-3n we have the following decomposition:
1, and the dimension of V/ is ("²²) - -1 =²=3n So.
2
V Vo 1 V2 t V3
16 vo
+ V/ + V/
Accordingly, using Schur's s lemma, each linear mapping can be characterized by 3 + 1+ 15 parameters, the same result from Maron et (2018), so we have the expected number of parameters. In appendix C, we give formal proof of these arguments and present algorithm to decompose an input matrix X into its seven irreducible spaces with linear complexity the matrix size n². These results summarized in the following theorem.
Theorem 4.1. For all n 2 4, the space RR can be written as a direct sum of the spaces Vo.... V6- These spaces invariant and irreducible, and the isomorphism relations between them are given by Vo V1, 1/2 V/3 V4-
4.3 DEEP WEIGHT SPACES
Recently, there has been growing interest in devising neural operators: neural networks that operate on an input, which is itself a neural network. This type of problem of interest for various tasks involving post-processing and synthesis of multiple trained neural networks, as well for processing Implicit Neural Representations (INRs), which are a popular alternative for representing certain standard data structures. (see e.g. Kalogeropoulos et al. (2024) for further discussion).
A key concept in the design of 'neural operators' has been the requirement that they are equivariant the input neural data (Kalogeropoulos al., 2024; Kofinas al., 2024; Zhou et al., 2024a; 2023; Lim et al., 2024) In this section we consider the setting discussed in Navon et al. (2023); Zhou et al. (2024b), where the neural data is a collection of weights and biases (Wm. bm )mmm=1 representing an MLP of depth M, and neural operator is constructed by composing standard activation with linear mappings which are equivariant with respect to the multi-permutation action we will now describe:
The output of an MLP architecture is invariant to the permutation of its hidden neurons. For example, the MLP defined by RU .W1x) will remain the same function if we replace weights W2, W1 with the weights W"P,p"W
To define the symmetrics of learning on MLPs in full generality, we will adapt the notation from Navon et al. (2023), We consider the space of MLP parameters with given fixed depth M and layer dimensions do.... d (M and all dj are assumed to be larger than one). These are parameterized
by the vector space V + DM (Wm @ Bm) where Wm := Rd xdm-1 and B,, == Rd represent the weights and biases of the m-th layer. The symmetry group of the weight space is the direct product of symmetric groups 9 = Sd, x... Sau- An element g ( TM-1) in the group acts on an element a = [Wm, bmlmm([ from V as follows:
P(g) [W by BmlmE[M)
(4a)
W = PTT W1, by PT
(4b)
W = PT W., m Pr P 6'n = PT bm, m E 12, M -
(4c)
W = WM PMMPr bas = by
(4d)
RRd.. Xdm. is the permutation matrix Tm Sa... where Pr...
Previous work (Navon et al., 2023; Zhou et al., 2024b) has already characterized all linear equivariant functions from itself. However, this characterization requires tedious bookkeeping and division into large number of different cases. Here show how to decompose 1 into irreducibles in rather straightforward way, and as a result obtain (arguably) simpler characterization of linear equivariant layers.
We claim that V is a direct sum of multiple copies of 2M - 3 irreducible representations of G:
1. The first irreducible representation is the trivial scalar representation S = R with the trivial action gr = xr.
2. The next M representations the vector representations V ms 1...., M of vectors in &d which sum to zero (the spaces (dd,.) from equation 2), with the action being permutation of the vector by the m-th permutation Tm
5

---

Published as a conference paper at ICLR 2025
3. The last M - representations representations Mm: = 2...., M M - 1 of dm d matrices whose rows and columns sum to zero, where the action of g on matrix W in this space is given by PT Wm
The linear equivariant layers from V to itself can be inferred from the following theorem:
Theorem 4.2. The spaces S, Vi,..., M M2...., MM are absolutely irreducible, and are not isomorphic to each other. V is isomorphic to S @ (#m=) VVBB ( (0 (0m=2
1-MM1MM Mm). where
a do + 2d + 2MM -3, B1 = 22 3M--1 = dm + 2, Bm 3, Vm = 2,...,
M M- -2 (5)
In particular, the space of equivariant linear mappings from V itself is thus of dimension a² 1M-1 B2 + (M 2)..
Proof. The spaces S, V1........ Vi..... MM same dimension). The action of 9 on S is trivial while the action on the other spaces is not. The other spaces aren't 't isomorphic to each other since different components of the multi colltipeemuutaiinn acts on each space. The spaces S and V, are absolutely irreducible, as we discussed in Subsection 4.1. We will explain why Mm are absolutely irreducible in Appendix D.
MM-1 not isomorphic (even when they have the
g
Obtaining the decomposition of V is rather straightforward. First, following Navon et al. (2023), we identify the weight spaces Wm. and bias spaces B,, with subspaces Wm. and Bm of V by zero padding, and note that each one of these subspaces is G-invariant and that their direct sum gives us the full parameter space V. We can then decompose each one of these spaces into irreducibles to obtain
Bm = S @ V my
m ..... M -
(6a)
BM S Bdar
(6b)
W = S bdo + Bdo
(6c)
Wm _ S + V m- vm Mm. m = 2, 2....,
(6d)
WM = S adar V
(6e)
M-1
The multiplicities of each irreducible in V, specified in equation 5, can now be found by simply counting how many times each irreducible appears in the decomposition above.
The decomposition in equation 6 can actually be easily obtained from the 'deepsets' decomposition of the 5n on IR" described previously: the action of Ç on B,, with m < .M is isomorphic to the action of Sa., on R.t and hence we get the exact DDepSStts decomposition from Section 4.1. The action of G on Bas is trivial and hence be written direct spaces. The action on W1 (and Wx) multiplies matrix by a permutation from the left (right), and hence can be seen the action of on Wm for m 2,.... .M- is a tensor product of the natural action of Sam_, on Iddm- and the action of S..,, on Rdd and therefore its irreducible decomposition can obtained by taking tensor products of the irreducible representations of dd and pdm-1 explained in more detail in the full proof.
of dı trivial one dimensional S
direct sum of 'deep-sets' actions on the columns (rows) of the matrix. Finally,
We note that equation 6 includes almost all information required to compute linear equivariant maps from V to itself. All is left is the decomposition algorithm to write each (Wms bm)me[M] as direct sum of elements in the irreducible decomposition. This can be done independently for each subspace Wm, and B,.. The decomposition for W.,, in equation 6d is not immediate and we will explain it in Appendix D.1.
The decomposition in equation 6 and equation also provides substantial additional information. For example, we can immediately see that there invariant maps from V to I S. which correspond to the number of copies of the trivial representation in V. Moreover, we are interested in the equivariant maps from a bias space B, (or weight space Wi) to another bias space B, (or weight space W,), we can easily infer the equivariant mappings from the decomposition in equation 6. For example, when
M there will be dz, mappings since B,,. consists of das isomorphic

---

Published as a conference paper at ICLR 2025
representations, and when i j-= M there will be das mappings since S appears a single time in B; and dı times in Bar- Continuing in this way can reconstruct all the different bias-to-bias, bias-to-weight, weight-to-bias, and weight-to-weight cases analyzed Tables 5-8 of Navon et al. (2023), and Tables 8-11 in Zhou et al. (2024b). More importantly, these tables which were necessary for implementing weight space layers in previous work, are not necessary when implementing these layers using Schur's lemma as we suggest.
SETS OF UNALIGNED SYMMETRIC ELEMENTS
Next, consider the setting where data is k-tuple of unnaligned objects' (0ly... U&), each coming from representation V of G, and we want to learn functions which are equivariant with re- spect to the joint action of k-tuple of group elements (gi.....gk) on each coordinate independently. and to a permutation
Sg of the k-tuple.
We define G Sk : 9 X ". "k-times x 9 x Sk, and the action is given by
(T, 91, 9k) (V1, .., vk) (g+(1) (-(1), (7( V+(+))
The group for which this action is defined is also called the restricted wreath product of g and Sk. For more details, see Wang al. (2020).
This type of wreath-equivariantt where our goal is, given a pair of elements V2). E V , 2, to find the group element g² ====g"(v ("1, (2) which makes 'as similar as possible' to U2- This task is equivariant to application of g elements to each coordinate because if g" 01 R 02 then 929" 91 (91 R 9202. Similarly, this task is equivariant to permuting ii and U2, because if g0v, R3 02 then (g") RS v1 For a more detailed derivation, see Chen & Wang (2024); Navon et al. (2024), which discussed these problems for sets and weight spaces, respectively. Additional examples of "wreath-equee euuiaariiuvaiiatt pariiaattttrosssssss are the anomaly detection problem discussed in the experimental section, and problems with hierarchical structures as discussed in Wang et al. (2020).
arises in several settings. One is 'alignment problems',
01
-reath-equivariant layers. Our aim is to characterize all G Sy equivariant mappings from V* to itself. We note that any linear G-equivariant mapping L V induces 'Siamese' g Sk equivariant mapping defined by
L(vi,... .........00. vk) (L,,))...,, ((x)))
The interesting question is how many additional mappings are present. This problem was previously studied Maron al. (2020); Wang al. (2020) when 9 is a finite group acting on R. transitively by permutations (this means that the action of G on [n] has single orbit). In this setting, the equivariant mappings are composed of the Siamese mappings and a single additional non-Siamese mapping. However, the transitivity assumption does not hold in many examples of interest, such as the graph and weight space examples discussed in this paper. In our analysis, we will release the transitivity assumption, and allow G to be general finite group. In some cases, this will lead to substantial number of non-Siamese networks.
To characterize 2 Ss equivariant functions, first aim characterize all invariant irreducible sub-spaces of yk, assuming we know all irreducible sub-spaces of V. An important role will be played by trivial representations: representations of & such that gu for all g and S.
Theorem 5.1. Let V be real representation of G, with irreducible decomposition
V = (+) @ (3)=
(8)
where S are trivial representations and V₂ are not. Then an irreducible decomposition for V with respect to the action of G 2 Sy is given by
*
(0i
@ B e (#)=))²))
k E S
where 5 = ((s,,...,sk)
Σ s; = 0},
st = ((....,,)) S
7

---

Published as a conference paper at ICLR 2025
Proof. The fact that Vi is & invariant implies easily that V is Sg also invariant. In the appendix E. we will show that if the action of 9 on V is not trivial then V is irreducible representation of G S.. contrast, for the spaces S the action of G trivial, so this representation can be identified with the standard representation I2 of Sk. The decomposition S and S then follows from the 'DeepSets decomposition" discussed in Section 4.1.
To count the number of linear equivariant mappings L D - Vi we note that
S = $
sh.1 = S =
S S LI Visisjss
Thus, the number of linear equivariant mappings from the 'trivial part' of V*, that is (0f=1 i (B,,, to istelf, 2s², while the number linear equivariant mappings from the 'trivial part' of V to itself (which is equal to the number of Siamese layers), is s² As result, we obtain s² non-Siamese G Sy equivariant maps.
Examples. We found that the number of non-Siamese layers is s² where is the number of trivial representations in V. Let us consider the implications for the three examples we have discussed earlier:
1. Deep-sets. In the DeepSets setting where V = g S,,, there is single trivial representation of constant vectors, and therefore this setting there is a unique non-Siamese layer for V* Indeed, this is the case for any group acting transitively by permutations on R", and thus obtain the results for transitive actions from Maron et al. (2020); Wang et al. (2020).
2. Graphs. In the graph setting where V = R=*a , = Sn. there are 8 = 2 trivial representations (see equation 3), and therefore 5² non-Siamese layers.
3. Weight Spaces. In the weight space examples the number of trivial representations is rather large, = do + 2dM + 2M 3 from equation 5, and there are a² non-Siamese mappings.
In the next theorem we give an explicit characterization of the non- non-Siamese layers of V* For this characterization, note that the direct sum of all trivial representations of V (as in equation 8) the vector space Viised = {or E Vlgu = u, Vg E G}. Alternatively, every basis C1, Cs of V Viized defines decomposition of Vfized into one-dimensional irreducible invariant sub-spaces S. = {ce;le E IR). Accordingly, our characterization of non-Siamese layers is based on such bases:
Theorem 5.2. Let V be real representation of a finite group G. and let Gi,..., Cg be a basis to the subspace Vizzed Let () be invariant inner product on V. Then every linear equivariant map L. 4k 4 is of the form
A - (ve Cin F=1
I(Ui,.... 0k)
Σ 1
aij
...,
(f(or)....., f(wx))
(9)
where L: V- V a linear equivariant map, and oig are real numbers. Conversely, every linear mapping of the form defined in equation 9 is equivariant.
The theorem is proven in Appendix E. We note that an invariant inner product on V is an inner product satisfying (gu, gu) = (v. u) for all E G and u, v V. When G is finite, an invariant inner product always exists: it can be obtained by starting from arbitrary inner product and then averaging over the group (Fulton & Harris, 2013). ) In the examples we consider this paper, the group & acts by permutations. In this case, the standard lg inner product is invariant. Implementing the Siamese layers defined in equation 5.2 only requires finding basis for Vfized- In particular, if V is one the spaces discussed previously, the basis e; just choice of an element from each of the trivial spaces S; in the decomposition V.
Beyond Sg. So far, we have considered the action of Gi where & is finite. We can also generalize these results to the setting G H, where H is subgroup of S₃ which acts transitively on {1..... k)). By doing this, we are generalizing the results in Wang et al. (2020), which assumes that both & and H act transitively.
8

---

Published as a conference paper at ICLR 2025
The full generalization is described in Appendix E.1. The general idea is this: Since G acts trivially on Vyized, the action of G H on fixee can be identified with the action H on V firedd turn, V is direct sum of different copies of Rt Accordingly, the number of H equivariant maps from V
fixed to itself is 8² h, where h is the number of H equivariant maps from R to itself. One of these maps is the identity map, which is Siamese; therefore, the total number of non-Siamese equivariant maps is (h - 1) s2. In the case where H = Sk, we have that h = 2, and therefore, in our analysis above, we have found s² non-Siamese maps.
EXPERIMENTS
In this section, we consider problems with a wreath-equivariant structure. We compare networks implementing the complete basis of equivariant layers that we have found, with networks that only use Siamese layers or combine partial list of non- -on-Siamese layers suggested in previous works. Implementation details of all experiments described in Appendix A.
Graph Anomaly Detection. We begin with a synthetic wreath equivariant task in which Siamese networks will fail by design: we consider graph anomaly detection problem, where the input is graphs with n nodes, (GI,.... G5), where most graphs are similar, and one is an 'anomaly.' The output a l dimensional probability vector, where the i-th entry denotes the probability that G, an anomaly. This task is G 2 Sk equivariant (where G S,), where the action the input space is as in equation and the action on the output space is just permuting the entries the probability vector. We generate data for this problem as follows: We randomly generate two graphs G and & using the Firdos-Reyn distribution. We take k 1 copies of the graphs (G1, G 1) to be permuted copies of G. and one of them to be G and insert in a random location. We also add some noise with variance 77 to all graphs.
We consider equivariant models for this task, composed of several linear preath-equivarian layers and point-wise ReLU activations. The final layer is point-wise summation of each of the & graphs to obtain a final vector in IRA,, followed by softmax layer to obtain a probability vector. For the linear wreath-cquivarin layers, we consider several alternatives: Siamese layers only, adding the single additional non-Siamese layer suggested Wang et al. (2020); Maron et al. (2020) denoted by DSS, and the full model we suggest, which has four non-Siamese layers (we name model SchurNet). The Siamese layers are implemented using the decomposition computed Subsection 4.2.
The results of this experiment are shown in Ta- ble 1. As we can see, Siamese networks attain 10% accuracy without noise (and even lower accuracy with noise). This is to be expected since Siamese features can be useful differen- tiate between C and G. but not to determine how many times each one of them occurs k--tuple In contrast, networks with -on-Siamese layers attain significantly better performance, whereas our method, which contains the maximal number of non-Siamese layers, attains the best performance.
Model/Noise
1 0.0
1 0.1
Siamese
10%
10%
DSS
97.5%
92%
SchurNet (Ours)
100%
97.0%
Table 1: Performance comparison of models at different noise levels (1)).
Wasserstein Distance Computation We next consider the task of learn- dis-
Dataset
Input
SchurNet
NProductNet
100, 300]
0.0389
0.046
noisy-sphere-3
[300, 500
0.1026
0.158
ing the Wasserstein distance, cussed in Amir & Dym (2024); Chen & Wang (2024); Haviv et al. (2024). The Wasserstein distance between two unordered multisets of vectors in Ted denoted by (zi,..., } and fyI...., }. is defined to be the min- imal distance between the sets, under the optimal permutation giving the best alignment between them (as in the alignment problem discussed in the beginning of Section 5). Comput-
100, 300
0.0217
0.015
noisy-sphere-6
[300, 500]
0.0795
0.049
256
0.0974
0.097
uniform
[200, 300]
0.1043
0.1089
[20. 200]
0.0623
0.084
ModelNet-small
[300, 500]
0.0738
0.111
2048
0.0468
0.140
ModeINet-large
[1800, 2000]
0.0551
0.162
[20, 200]
0.0123
0.012
RNAscq
[300, 500]
0.0334
0,292
Table 2: Comparison of SchurNet and NProductNet.

---

Published as a conference paper at ICLR 2025
ing Wasserstein distances can be computationally intensive, this task aims to devise neural network to learn the distance instead. Learning Wasserstein distances is a Sn l S2 invariant task. In Chen & Wang (2024), Siamese approach named NProductNet is suggested to address this problem. We compare SchurNet to Chen & Wang (2024) to demonstrate that adding our non-siamese layers enhances the performance of standard Siamese layers. We note that Amir & Dym (2024) achieved state-of-the-ar results for this task with a very different method, which does not follow the conventional paradigm of stack of linear layers and non-linear activation functions. Therefore, we have no direct way of adding our non-siamese layers to their method for comparison. In our experiments, we add the additional d² non-Siamese layers to their implementation and compare performance on the datasets addressed in their paper. The results, reported in Table 2, show that adding non-Siamese layers improves performance in most cases. We note that each dataset has two options: the top option in each table row depicts the case where training and test distributions had the same dimension, while the bottom option checks the generalization to test distribution of different sizes.
Weight Space Alignment. We consider the alignment task discussed above in the setting where U1 bg are elements in weight spaces, and the task is find the group element that optimally aligns Yn and U2. One interesting application of this problem is model merging: in Ainsworth et al., it was shown that linear interpolation of 11 02 gives a new network whose performance is considerably worse, but interpolation after alignment leads to much better results.
The weight space alignment problem was considered in Ainsworth et al.: Tatro et al. (2020); Peña et al. (2023). Recently, Navon et al. (2024) outperformed these methods using learning approach based on the DWS layers from Navon et al. (2023), applied to U1, U/ in a Siamese fashion. This experiment aims to check whether adding non -iammee layers improves their results. We take only part of the described layers in Theorem 5.2 layers: the mappings from each weight/hias space to itself, excluding, for example, non-Siamese weight to other weight or weight to bias. This added fewer parameters and generalized better than taking all mappings.
Model
MNIST
CIFARIO
Acc(L)
Loss (L)
Acc(l)
Loss (L)
SchurNet (Ours)
1.5e-5
0.251346
0.0
1.7822
Siamese
1.25e-5
0.262913
1.0e-4
1.7876
Table 3: Comparison of SchurNet and Siamese models on MNIST and CIFARIO.
We two sets of experiments from Navon al. (2024): one MLPs trained MNIST (LeCun et al., 1998) and one on MLPs trained on the CIFARIO (Krizhevsky et al., 2009) dataset. We trained both models for 100 epochs and reported the test accuracy and the reconstruction loss. We ran several hyper-parameter configurations for both methods to ensure fair comparison and reported the results with the best reconstruction loss. As we can see, our model with non- -on-Siamese layers outperforms the Siamese network from Navon et al. (2024) in all settings.
Conclusion. In this paper, we revisited the idea of using irreducible representations instead of parameter-sharing to characterize equivariant linear layers for representations of permutations and related groups. Using this approach, we obtained alternative derivations for the characterizations of equivariant layers from DeepSets and 2-IGN, and a significantly simplified derivation of deep weight spaces equivariant layers. We have also obtained a previously unknown characterization for the linear equivariant layers of wreath products G S,, and showed the benefits of using the full characterization for several greath-equivarian tasks. In general, looking forward to other yet unknown applications, what could be the benefit of using the irreducible approach? One answer is reduced book-keeping as exemplified in the deep weight space example. In general, the number of equivariant layers can be, in the worst case, quadratic in the number of irreducibles. When this happens, the irreducible approach is expected to lead to simpler characterizations that are easier to implement. Additionally, networks like k-IGN (Maron et al., 2018), which are based on tensor representations, use intermediate features in n², and using irreducibles could lead to new equivariant models with intermediate irreducible features of lower dimensions. Finding the irreducible decompositions k-IGN when > and the potential applications of this decomposition is an interesting avenue for future work.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
We would like to extend our thanks to Eitan Rosen for the valuable discussions. We also sincerely thank Idan Tankel for his technical support, as without your help, none of this would have worked as it has. The authors are supported by ISF grant 272/23.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference Learning Representations.
Tal Amir and Nadav Dym. Injective sliced-wasserstel embedding for weighted sets and point clouds. arXiv preprint arXiv:2405.1653 2024.
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neu- networks. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019, URL https: http:::::/pppppppepeeeeeeoo neurips wtw.ipss.ccccae. paapperfiiessppppeeee 035 Ft119/iiee0357333222222 6e
Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. arXiv preprini arXiv:21 2021.
Michael Boardman. Real and complex representations (lecture notes). http: archive. orreeeorr//eeee http://wbw00000/// 0007700002//hhhpp0 h0224/ttt::/ww/// 808mult timeiia. commwwnnnne http://wwnnttteeeee htm. Accessed: 2024-09-12.
Samantha Chen and Yusu Wang. Neural approximation of wasserstein distance via universal archi- tecture for symmetric and factorwise group invariant functions. Advances in Neural Information Processing Systems, 36, 2024.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Research, pp. 2990-2999, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https: /ttp///diccs..c.ceeeeeco mlr press/ 1/88ccohenc1 6 1htmll..6........
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on ho- mogeneous spaces. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buce E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL httpt:tttpte..o http//poooeeeeeeeeooooooooooeoee ngs neurips httpppp...ppppeeee cc/paper__ wffileespppeee Pat.rr.appppp........
Nadav Dym and Haggai Maron. the universality of rotation equivariant point cloud networks, In International Conference on Learning Representations.
Mare Finzi, Max Welling, and Andrew Gordon Gordon Wilson. A practical method for constructing
equivariant multilayer perceptrons for arbitrary matrix groups. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3318-3328. PMLR, 18-24 Jul 2021. URL
httpps:/ http:///pocceiinee..e mll. wwllr.press...oommo / vv39/vv////////////// stteess/139//inii2tt1
William Fulton and Joe Harris. Representation theory: a first course, volume 129, Springer Science & Business Media, 2013.
Doron Haviv, Russell Zhang Kunes, Thomas Dougherty, Cassandra Burdziak, Tal Nawy, Anna Gilbert, and Dana Pe' er. Wasserstein wormhole: Scalable optimal transport distance with transformer. In Forty-first International Conference on Machine Learning, 2024. URL https:://pppeeeeeeee operrevie.. net forum? 3
11

---

Published as a conference paper at ICLR 2025
Erik Henning Thiede, Truong Son Hy, and Risi Kondor. The general theory of permutation equivarant neural networks and higher order graph variational encoders. arXiv e-prints, art. arXiv:2004.03990, April 2020. doi: 10.48550//2Xiv222004039999
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
Ioannis Kalogeropoulos, Giorgos Bouritsas, and Yannis Panagakis. Scale equivariant graph metanet- works, 2024. URL htttps::/azxiv.orggg//
2406. 6. 10685.
Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan Burghouts, Efstratios Gavves, Cees G. M. Snoek, and David W. Zhang. Graph neural networks for learning equivariant represen- tations of neural networks. In The Twelfth International Conference on Learning Representations. 2024. URL https: wwwpre/eeeeeeeeeeeeeeeeeeee net/ forum? sMYDB...
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2747-2755. PMLR, 10-15 Jul 2018. URL https: htttts::/ppoceeeeeeee /ptp//eeccceeg....///////// mlr. www.ppress/vve.. v80 wttpa///vv////oorrr/eo kandor 8a. html.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. Mnist handwritten digit database. 1998. Accessed: 2024-08-23.
http://yaan.lemm..coo/////d////
Derek Lim, Haggai Maron, Mare T. Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for processing diverse neural architectures. In The Twelfth International Conference on Learning Representations, 2024. URL https: www.neeeeeeeeeeeeeeeeeeeeeeeeeee n forum i (idd-isiKKyx0
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint arXiv: 11Xx::88/22099022 2018.
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. In International conference on machine learning, pp. 6734-6744. PMLR, 2020.
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equiv- ariant architectures for learning in deep weight spaces. In International Conference Machine Learning, pp. 25790-25816 16. PMLR, 2023.
Aviv Navon, Aviv Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equivariant deep weight space aligmment. In Forty-first International Conference on Machine Learning, 2024. URL htap:::://pppeevvieeeee net forum? id=nBP runk 6Ee0.
Edward Crumpe Connecting permutation equivariant neural networks and partition diagrams. arXiv preprint 2tt.::2//2.08444. 2022.
Edward Pearce-Crump. Brauer's group equivariant neural networks. In International Conference on Machine Learning, pp. 27461 -24422. PMLR, 2023a.
Edward Pearce-Crump. How jellyfish characterise alternating group equivariant neural networks. In International Conference on Machine Learning, pp. 27483-27499 PMLR, 2023b.
Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differenniation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20237-20246, 2023,
Bjorn Poonen. Real representations, 2016.
Siamak Ravanbakhsh, Jeff Schneider, and Barnabás Póczos. Equivariance through parameter- sharing. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2892-2901. PMLR, 06-11 Aug 2017. URL https: http:///poooceeeeeeeeeee mlr. wrw.s/v700 ravanbakhsh 7a html.
12

---

Published as a conference paper at ICLR 2025
Jean-Pierre Serre. Linear representations finite groups. Springer- Verlag, New York, 1977. ISBN 0-387-90190-6. Translated from the second French edition by Leonard L. Scott, Graduate Texts in Mathematics, Vol. 42.
Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment. In H. Larochelle, M. Ranzato. R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural In- formation Processing Systems, volume 33, pp. 15300-15311. Curran Associates, Inc.. 2020. URL https: httppp://pproeeeeee http://poooeeeeeeee.ooaoeoeso neurips http//ppe__fflec http/pppr/2200 fße Papeee pddf.
Nathaniel Thomas, Tess E. Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equiiarraat neural networks for 3d point clouds. CoRR, abs/1802.08219, 2018. URL http:hh:p///i....trtetee
18001 08219.
Renhao Wang, Marjan Albooych, and Siamak Ravanbakhsh. Equivariant maps for hierarchical structures. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Ben Webster. Answer in mathoverflow.
http:://maatloverfoooo.ttt//// 222831/ reference wwweennceereueettteeeeeee request-irreducbllee -ww.iirreeeeeeeeeeeeeeeieeeeee-- of-the- regular reppruaarrr-rpreeetttttiooood- -hhe pereeea rq=1. Accessed: 2024-09-12.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Allan Zhou, Kaien Yang. Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, Zico Kolter, and Chelsea Finn. Neural functional transformers. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL /
net/ forum d=Johy I
Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals, 2024a. URL https: 05232.
Allan Zhou, Kaien Yang. Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in neural information processing systems, 36, 2024b.
A IMPLEMENTATION DETAILS
In this section, we detail the hyper parameearsm used through learning. In all our models we used Adam or AdamW optimizers.
Graph Anomaly Detection In this set of experiments we run each model with 27 different hyper- parameters and reported the best for each model. We run through different learning rates, weight decays and learning rate decay. In our model, we used four layers, each with hidden dimension of 256 and ReLU activation.
Deep weight space alignment In this section took the model of Navon et al. (2024) and added our non-Siamese common layers. We searched for each model and dataset over four hyper-parameten and for each reported the best performance. We trained for 100 epochs and reported the reconstruction loss.
Wasserstein Distance computation In this set took the model of Chen & Wang (2024) and added the non- Siamese layers. We trained for 200 epochs and reported the relative absolute mean error. In most of the experiments used learning rate of Te and weight decay of 15
13

---

Published as a conference paper at ICLR 2025
B ABSOLUTELY IRREDUCIBLE REAL REPRESENTATIONS
Here we give some more details on the concept of absolutely irreducible real representations.
As discussed in the main text, real irreducible representation V is called absolutely irreducible if its complexification is irreducible over C. The complexification is denoted by cV. It can be defined by choosing basis 01, " Un to V and then defining
cV
G KEC).
2m
Let T be equivariant non-zero mapping, and assume V is absolutely irreducible. Then To = Av for some real X. This is because can be linearly extended to a non-zero linear mapping also be extended to eV. The
: eV. Similarly, the group action of which is linear extension of to cV will be equivariant, and therefore, since cV is irreducible by assumption, Schur's Lemma for complex representations implies that Al for C. Since TV V, we know must be real.
Real irreducible permutation representations absolutely irreducible. Classical represen- tation theory results (see e.g., Fulton & Harris (2013)) characterize all complex irreducible repre- sentation of the permutation group, up to isomorphism. In this process, they show that all these representations can be defined over the rational numbers. This means that, for any complex irreducible representation U/ of the permutation group S., there exists a basis in which for every g E G., the matrix representing the linear mapping defined by will be rational.
As explained, e.g., in Webster, this fact implies that every irreducible dimensional real representation V is absolutely irreducible. Indeed, if cV were not irreducible over C, then cV Vi + V/2 where Vi an invariant (complex) irreducible subspace, and the dimension t of V satisfies 0 appropriate bases U1, uj,..., of V we will have that the matrix representing each g E S, is rational. We can write each Uj as
In an
U
(ajk + (bbz)) )xk
1=1
One can then verify that the dimensional real space 1 defined by
ayk2%
a (real) 9 invariant subspace of and has dimension < 8, which contradicts the fact that is irreducible.
14

---

Published as a conference paper at ICLR 2025
C GRAPH NETWORKS IRREDUCIBLE DECOMPOSITION
In the main text, we claim that, for n 2 4, the representation R"** of the permutation group S, can be decomposed into direct sum of seven irreducible representations:
Vo {a I, R}
V = {a (Iuxn In)la E R}
1/2 (Diag(ai, an)
03 = 0}
1=1
V/3 (r1 Σ Pi 7=
14- = (I,c²
Ci 0}
1=1
V/3 (AIA A" Aln = = n..
V/8 - (AIA= A", Al, Al,
We begin by giving algorithm describing how every by matrix be decomposed into of matrices from these spaces, with a computational complexity O(n²). This is necessary for a 2-IGN implementation which is based on irreducibles, and is also first step towards full proof of Theorem 4.1. This full proof will be presented after the algorithm.
C.I DECOMPOSITION ALGORITHM
We will give an algorithm to decompose a matrix R into sum of matrices from these seven sub-spaces. We assume n 2 4 (actually, when n = 3 the algorithm works as well, but in this case V/8 zero dimensional discussed in the main text).
The first step of the algorithm is to write
0
b
a 0
a
= B+
where the two matrices from the right are in Vo and V1, and and b the average of off-diagonal and diagonal elements of A, respectively, so that the diagonal and off-diagonal elements of B both sum zero. It remains find a decomposition for B. The crucial part of this is the following lemma Lemma C.1. Let n > 2 be a natural number, and let B E Raxn be matrix whose diagonal elements sum to zero, and off-diagonal elements sum to zero. Then there exists a matrix C' in
v = {C E BR" ,Cl, = O, 1 0 and Cn 0, Vi 10i=1,.... 1.....n
(10)
and vectors r, c, R" which all sum to zero, such that
C= B r1 f. + 1, + diag(d).
(11)
Moreover, this decomposition of B is unique.
Stated differently, the matrix B will be linear combination of matrices r1,, L, and diag(d) which are in V3, V4 and V2, repsectively, and the matrix which V. Once we prove the lemma, we will conclude by showing that a matrix in D can be written as a sum of matrices V/3 and Ve. which will conclude the argument.
Proof of the Lemma. Let us denote the sum of the i-th row and column of B by rf and of respec- tively. Denote dh = BiG Note that the vectors B " and d all sum to zero by our assumption on B.
15

---

Published as a conference paper at ICLR 2025
We need to show there exists vectors r, d, R" satisfying equation 11 for appropriate C E V. This will occur and only the following equations are satisfied
0
Σ = 0 im1
Σ d =0
1=1
ri+ + d; = - db
Vi= = 1,....
nr; +
Σ c = --
I,...,
j=1
ne; + Σ r q === 1, I....,n. n.
==1
Here the first three constraints are the requirements that the vectors d all sum zero, and the last three constraints (each which are actually n constraints) follow from our requirement that the diagonal elements of C' will be zero and that the rows and columns of C sum to zero.
Next, we note that since the vectors r, c, d, r c., d all sum to zero, if the last three equations above satisfied for i I....,n they will automatically be satisfied for i n. That is, these equations are equivalent to
Σ
0
Σ d: = 0 1=1
C + d. -d d
Vi 1, ,,....
nr; + d. - r
Vi 1,...,
nc, + d. = C
Vi I....,n- n
This gives 3n linear equations system in 3n variables. We will show this equation has unique solution and compute it explicitly. First, we eliminate the variables rns on and di...., d, by setting
d;
d
Ci i -,..., -
(12)
and
Pn
Tjz
Cr Σ
9 d2 = --=-a d
(13)
This ensures that the first four equations above are satisfied, and we are left with the task of choosing Ti..... Tn-1, Ci...., .., Cn-1 satisfying the last two equations (which are actually 2n 2 equations). By replacing the eliminated variables d; obtain the equations
(n 1)r; - C; -r + d.
Vi= 1, 1....,72
(n 1)c; - rr = c + B
Vi
1,...,7-
16

---

Published as a conference paper at ICLR 2025
For every i, there are two equations in r, and Cs. Since assume 2 4, this equation has unique solution
ri
n
n -1
p d. ( c d.
ci
And we can derive
n - 1
n - 1) )
(-
-- d
Ci
n
2n
1
-c + +
ndd
(n - 1)r B
Ti C2
nd
(n - 1) 1)
We can then reconstruct di...., dn-1 from equation 12 and Coyy T'a dn, from equation 13. This concludes the proof.
Given the proof, it remains to decompose matrix C' E v into a sum of two matrices in Vo, Vo. This can be simply done by writing
C -c( + c"))
and noting that the matrices (C- - C*) and [C + C²)) are in Vi and Vo.
C.2 ISOMORPHISM TYPES
We now restate and prove the theorem on 2-IGN stated in the main text Theorem 4.1. For all 2 4, the space Rxx can be written as a direct sum the spaces Vo...., Ve- These spaces are invariant and irreducible, and the isomorphism relations between them are given by Vo = 11, 1/2 V/3 = V4.
Proof. The decomposition given in Subsection C.I is unique when n 2 4, and thus we see that
R Vo Vo-
It is also clear that all the spaces V; are invariant, that Vo. Vi are isomorphic to the irreducible trivial space, and V/2 V3, V4 are isomorphic to the irreducible space Vnn). It remains to show: (a) That Vs. V/ are not vero-dimensional. (b) that they are irreducible, and (c) that both V/ and Vo are not isomorphic to any of the other spaces.
We handle (a) by computing the dimension of V, and Vo. We note that matrix A E V3 can be parameterized by freely choosing all upper diagonal elements A, with j - 1. The elements A,s with i 1 are then determined by the constraint A, -Ay. and then the coordinates A12 = A21 Agg = - AAg..... are determined recursively by the constraint that the first n - 1 rows of A will sum to zero. The obtained matrix A satisfies A -AT and its first 1 rows sum zero. follows from A = A that the sum of all elements of is zero and hence the last row of A must also sum zero and A is indeed in V/.
To summarize, since V; is parameterized by the upper diagonal elements Ai with j - i > we deduce that
2 3n +1.
dim -n)2-
We compute the dimension of V& by noting that 1> from equation 10 is direct sum of V/ and Vg. The space 1 can be parameterized by choosing (n 10-1] (n 1) matrix A with zero on the diagonal, and whose elements sum to zero. There is then a unique extension of this matrix to an n by n matrix À which is in and satisfies Aij Ai for all 1 < i < 1. Accordingly, the dimension of is
(n 1)2 (n 1) 1 = n² 3n + and
dim(V)) = dim(V) dim(Vs) n²
3n
2
17

---

Published as a conference paper at ICLR 2025
It follows that when 4, both V/s and V6 have positive dimension. Moreover, V/ and Va not isomorphic as they do not have the same dimension. The same argument shows that and V. is are not isomorphic to the other V; with contrast V33 does have the same dimension as V/2 when n = 4, as we get dim(Vs) = 3 = dim(V) still not isomorphic since the action of Sn, on the spaces different. For
......4 4 since they do not have the same dimension for any In
Nonetheless, the spaces example, the subspace of fixed points of the permutation = (1, 2) in the space V2 = V(n) is all dimensional vectors which sum to zero and satisfy æ1 z22- This space is dimensional.
In contrast, one can verify that the subspace of fixed points of in V/s is one dimensional. Hence they cannot be isomorphic.
remains to show that V/ and V6 are indeed irreducible. Let V/ C V/3 and V.8 S Vo be non-zero invariant subspaces. We need to show that V, = Vi for 5, 6. To show this, we consider the space
U = Vo + V1 @... 14 + Vi $ Vo.
It is sufficient show that 21 R"x" Since U contains all diagonal matrices which given by Vo < V2, and since it an invariant subspace, it is sufficient to show that the matrix EL1 which is zero in all coordinates except for a single coordinate E = 1, is in UI
To show that FL.2 is in U, let choose non-zero matrix A E V/ and B V/g Both matrices have non-zero off diagonal matrix element, and by applying an appropriate permutation and rescaling we can assume that A 1 B12. Next, let us average over the group
stab(1, 2) = {T E Sn, ((1) = 1, (2) = 2}
to obtain new matrices
Σ
A. B stab(1, 2)
1
Σ
B
stab(1,2) 2)
rEstab(1.2)
rEstab((.2)
Note that Å (respectively B) is in Vús (respectively Vo) and satisfy A12 = B12 and A Å B1j BIN Vj
and
A₂* = A.c. Byx Ba
for all j, k, s,t which are all larger than 2, and such that + k, t. Defining for convenience
1
2
b
n
2
2 - 3)
follows that
0
0
1
1
a
a 1. 0
0
-a
Q b
A
0
B
diag (0, 0, b......)
0 a
b
It is not difficult to show that any matrix which is constant along the rows is in Vo .... + V4 S U. The same true for any matrix which contant along the columns. Let C(1) denote the matrix with (1) for all and Czy(1) for all j and all k + i. Then
A + aC(1) aC(() aC (1) + aCT (2)
0
+ 2a
0
-
2a
0
0
0
0
0
B: B + diag (0, 0,4, b,..,...))
blaxn + (a
2a + b
1 2a + b
1 + 2a +
2a
b
0
0
18

---

Published as a conference paper at ICLR 2025
where both À and B are in U. Since a, are positive numbers, take the following linear combination which zeros out the (2, 1) entry and sets the (1, 2) entry to one:
0
d 0 ... 0
A
1 1 2a
B
0 0
2
+ 2a
0
for appropriate c, d. Since diagonal matrices are in U, we can remove the diagonal matrix diag(c, d, ......0) 0) finally obtain that the matrix E is in 21
C.3 SMALL n
As discussed in Finzi al. (2021); Pearce- Pearce-Crump (2022), when 4 the dimension of the space of
equivariant mappings is smaller than 15: it is for n = 3 and 8 for n = 2. We now explain this according to derivation.
For n = 3, the space Ve is zero-dimensional In this case, there are only 2 3 + = 14 equivariant 2, both V/s and Vi8
linear mappings. When verify that in this case, we can "drop" 144 as well because _2x² = Vo + V e 1/2 + V3. Since Vo = Vi and 1/2 V/3 we have 2 + 2 equivariant linear mappings.
dimensional. Additionally,
directly
19

---

Published as a conference paper at ICLR 2025
D PROOF FOR DEEP WEIGHT SPACE DECOMPOSITION
In this section we fill some details on the tensor product of representations which were ommitted from the proof of Theorem 4.2 in the main text.
At the base of this discussion is the following lemma (Serre, 1977):
Lemma D.1. Let G1 finite groups, and let V1, V2 be (real or complex) finite dimensional
irreducible representations, with an action denoted by p1, P2. Then the action p1 @ P2 V1 @ V2 is an irreducible representation of 91 G2-
Firstly, we explain the decomposition of Wm into irreducibles. We note that the representation Wm of G can be thought of as a representation of Sam-1 X Sam. which is tensor product of the representation I of Sam-, and the representation Rd of Sa... Therefore (Serre. 1977), its irreducible decomposition is given by the tensor product the irreducible decomposition of its factors, namely
Wm. TR x Rd (Se ((dm- & (S V(dm))
(S S (S & V(dm)) @ (V(dm-1) @ S) @ (V(dm-1) & V(dm))
((d)) @ (dmm-1) @ Mm
All components in the decomposition irreducible
product of irreducibles.
We also need to show that all irreducibles in the decomposition are absolutely irreducible. For S this is obvious. For all representations (dm.) this follows from the fact that these representations can be identified with representations of the permutation group Sam-1. which are always absolutely irreducible. To see that Mn. is absolutely irreducible, we note that M,, the of d dm real matrices whose rows and columns all sum to zero, and its complexification cM,n is the set of complex matrices whose rows and columns sum to zero. eM,, the tensor product cV (dlm-) and cV (dlm,, the complexification of V (dm-11) and V (dm), respectively. Since these spaces are irreducible, so is their tensor product.
D.1 DECOMPOSITION OF WEIGHT SPACES
We explain here how each matrix A E Wm, 2,...,, . can be written as sum of elements from its irreducible decomposition. For convenience denote a = dm, b = dm-11 We want to write A as sum of elements from the spaces
= (claxb), Vm- (lacTIcE C =], Vm E R" I = 0}
and
Mn = =C' E gaxb Ol. = 0s, CT = 0o}
As a first step, we take our given Wm and write it as
A= Ålaxn + (A Alnxn)
where Á is the average of all elements of A, and Ál is in S. Next, need to decompose (A pxn) which we denote by B. Note that Eij Bij = 0.
Let r and e be the vectors describing the average of the rows and columns of B, respectively:
1 Blb, c= cT 1a
Note that 1,, = Ei Byg = and similarly II1 = and thus 1,6TT E Vm-1 and r13 e Vmn Now let us define
C B 1oc r
We can directly verify that Cl 0,, and 1 0a. Therefore C is in Mm and we obtained the decomposition we wanted.
20

---

Published as a conference paper at ICLR 2025
E PROOFS FOR WREATH PRODUCTS
To conclude the proof of Theeoem 5.1, we need to prove the following lemma
Lemma E.1. Let V be an irreducible representation of g, and assume the action of g on V is not trivial. Then 4 is irreducible representation of t Sk.
Proof. The space y is invariant. Let W S y* be non-zero G invariant subspace. We need to show that W = yk
We first claim that there exists an element in W of the form (u, 0, 0) with u * 0. Let (V1, 01,02,-- 1%) be non-zero element in W. By applying permutation if necessary, generality that the first element is non-zero. If o; = 0, 4/ _ < k. we are done. Otherwise, there exists some g E G such that gul A ti and so
U2,--,
assume without loss of
(g U1 U2, vn)) (vi, U2, vb) = (ge) Un 0, 0) E W
The space (u (1, O, 0) W) is invariant under the action of G, and just saw it is not Since V is irreducible, this space equals to V, and thus W contains V + O... @ O. By Ss invariance follows that W contains all k-tuples with single non-zero entry, and since such tuples span all of V we deduce that W = V.
Theorem 5.2. Let V be real representation of a finite group G. and let el...., Cg be a basis to the subspace Vized- Let )) be invariant inner product on V. Then every linear equivariant map L: vi + V is of form
k
L(E 06) =
ai
Σ (ve, (ve + (L(or)....., L(wk))
1,1=1
where L V + V is a linear equivariant map, and auj are real numbers. Conversely, every linear mapping of the form defined in equation 9 equivariant.
Proof. It is straightforward to checking that every mapping of the form equation 9 is equivariant. We would like to prove the converse statement.
It is also straightforward to verify that the space spanned by the non-Siamese mappings
(UI, vR) (ue, ed)e
does not depend on the choice of the basis e1...., e, for V Vinned- In particular, we can assume without loss of generality that e.,... , Ex are an orthonormal basis for Vitned-
Note that V can be written as a direct sum of two G invariant spaces V= =Vfized @ VL, where V1 is the space of vectors in V which are orthogonal to Viized with respect to the given G invariant inner V a decomposition of y into two Gk invariant spaces.
product. It follows that yk ///wed
In this situation, we know that any linear equivariant L D + y* can be written as L = + where Lfized Vk Vineed + V foree and L1 : Vt + uk are equivariant, We know that there are no linear equivariant maps from Vfired to Vperp or vice versa, since Vperp cannot contain a trivial irreducible subspace.
Let L VL + V1 be a linear equivariant map. We want to show that it is a 'Siamese mapping'. Denote
L(vi...., vk) = (Li (v), ,.), Lo. vk)),
We define L: V1 + V1 by
L(v) (1, 0,...... 0)..,,
Note that i. is G equivariant because
((v) Ta (gv, 0, ... .). = Ov.... =g((v), Vg E.
21

---

Published as a conference paper at ICLR 2025
Next, note that for every 02...., Uk E V1, have that
.......,00 uk) 9 In (0, 02, . uk))... L1 (1, 02,..., L1 (g O, 12..... ........
Thus, (0, U2. deduce that
0k) E VL, and since it is also in V fized we deduce that it is equal to zero. We
(u ................................ vb) 0, ..... 0) L(v1)
By S3 equivariance we can deduce that
(V Uy...) (igg ....,...,,,,,........
vI....) . ((2))
and we can also show, as we did for Li, that
Ly(01....... ..... 1y .... I, ...,, O..).
Thus L is a Siamese mapping induced from L, that is
ux) = (i(e1)..... [(xx))
Now, let consider linear equivariant mappings from V fixed itself. Denote S. {ces, E IR}, and note that irreducible decomposition of Viized is given by
Viied S a...... S..
and Vd: fizee can be written direct sum of the invariant subspaces S for i ,,....s. The action of g on St can be identified with the action of Sg on these spaces, which is then isomorphic to the action Sg on IR via the isomorphism t/. S + RR* given by
Uk) ((v1, es)........... (v1,,,)))
Note that since we assumed without loss of generality that cI. e, an orthonormal basis, 4/s is zero all other components of k Similarly we can define isomrphism bh R2 + S by
i ( =
As discussed in Section 4.1, an equivariant mapping from g² to itself can be written as a linear combination of the identity mapping x (which Siamese mapping), and the mapping
Tx
Accordingly, a mapping from S to 8 is a linear combination of a Siamese map and the mapping f To uh which is given by
S o To
2k) (+)+ =
(ue, e
Σ
(ue, ei)e,)
t=1
This concludes the proof.
E. BEYOND Sy
We now generalize the results from Theorem 5.2 the case where Sş is replaced with any subgroup H < S, which acts on {1, {I,...,k; transitively:
Theorem E.2 (Generalization of Theorem 5.2). Let be a real representation of finite group G. and let e, be basis to the subspace Viized- Let (.,)) be G invariant inner product V.
Let H be a subgroup of Sy which acts on {1....,k} k} transitively. Let Tı, T2....,T, be a basis for the H -eequivariaat mappings from R to itself, where Ty, is the identity mapping.
22

---

Published as a conference paper at ICLR 2025
Then every G H linear equivariant map L Vkk + Vi is of the form
L( ; 0k)
Σ Σ
aije (63 Te the) (1), ok)
(=1 =-==
(i(61))...,, (((x))
where L: V + is a linear equivariant map, and aejt are real numbers. Conversely, every linear mapping of the form defined in equation 9 is equivariant.
The proof is identical to the proof of Theorem 5.2.
23


Reviews:
[{"review_id": "N2ms9vYJvU", "reviewer": "Reviewer_iEGE", "summary": "The paper introduces an alternative approach for characterizing equivariant linear layers in neural networks that process permutation and related group representations. The paper derives a simpler method for obtaining existing models such as DeepSets, 2-IGN, and Deep Weight Space networks, based on irreducible representations and Schur’s lemma. The proposed framework also considers unaligned symmetric sets, that build upon equivariance to the wreath product of groups.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a fresh perspective on equivariant layer characterization by applying irreducible representations and Schur’s lemma to obtain simplified derivations of established models, such as DeepSets, 2-IGN, and Deep Weight Space (DWS) networks.\n\n2. The theoretical foundations are well-developed. The work provides a complete characterization of equivariant layers in the context of unaligned symmetric sets, which is an interesting theoretical contribution.", "weaknesses": "1. The presentation and flow of the paper could be improved. The claims and results are challenging to follow, which may limit the broader audience’s ability to appreciate the work.\n\n2. The paper’s contributions lack clarity. The paper offers an irreducible-based derivation for existing results and characterizes equivariant functions on unaligned symmetric elements, but the impact and relevance of these contributions remain unclear. It is not evident how these results benefit the design of novel architectures or enhance our understanding of current ones. This limits the significance of the work and may fall short of ICLR’s standards.\n\n3. The empirical evaluation is limited, and the results are not compelling. Using synthetic data for anomaly detection does not sufficiently demonstrate the method’s practical applicability, as the task is relatively unchallenging and does not show the strengths of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ojg9bbbbkv", "reviewer": "Reviewer_vHFp", "summary": "The paper studies equivariant linear layers for representations of permutations and related groups from a novel irreducible representations perspective. The authors provide an alternative derivation for models including DeepSets, 2-IGN, and Deep Weight Space (DWS) networks. The theory is then extended to unaligned symmetric sets, showing that there is a vast number of additional non-Siamese layers in certain settings. Experiments show that additional non-Siamese layers improve the performance in tasks like graph anomaly detection, weight space alignment, and learning Wasserstein distances.", "questions": "* Can the method be generalized to higher-order $k$-IGN in a principled manner? Can you briefly describe the claim that ``using irreducibles could lead to new equivariant models with intermediate irreducible features of lower dimensions''?\n\n* Can you conduct more experiments on real-world and large-scale datasets, and include more baseline? In addition, can you intuitively explain why non-Siamese layers help in these tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper offers the irreducible representations perspective for deriving classical models like DeepSets, 2-IGN and DWS networks. Some derivations are simpler than the original ones. The writing is clear and easy to follow. I check with the details and they are sound.", "weaknesses": "* While the new derivations align with original methods, the resulting models are not new. The concept of ``irreducible representation'' is also well studied, so the contribution of the paper lies mainly in bridging two topics, which is interesting but natural. In particular for equivariant graph layers, the authors only provide derivations for 2-IGN. As admitted in the limitation section, the paper does not involve higher-order $k$-IGN. The author should explain whether their method is broadly applicable for these networks based on tensor representations, or need case-by-case derivations.\n\n* Although this is a theoretical paper, the experiments could be improved. More baselines and more real-world tasks are strongly encouraged.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vH4donDwQT", "reviewer": "Reviewer_mzEE", "summary": "The paper introduces a novel methodology for characterizing equivariant linear layers for permutation representations, utilizing classical results from representation theory. Specifically, it provides an alternative characterization of equivariant linear layers for DeepSets, $2$-IGNs, and DWSNets, as well as the first comprehensive characterization of equivariant linear layers for unaligned symmetric elements. Importantly, the authors identify novel non-Siamese layers and empirically assess their impact.", "questions": "1. I find the methodology presented in L135-155 valuable to the research community due to its generalizability beyond the provided examples, most of which are already characterized. For this reason, would it be possible to add a *brief* discussion on the generalization of this methodology to strengthen the impact of this contribution and broaden its relevance to a wider community? See the following for more specific questions.\n2. Computing a basis compatible with the irreducible representation decomposition can be challenging. Does this difficulty limit the methodology’s generalization? Are there similar technical challenges for characterizing $k$-IGN layers for $k > 2$?\n3. Can this methodology be applied to other groups beyond $S_n$ and wreath products? If so, could you briefly provide a few examples?\n4. Representations of the symmetric group are relevant in machine learning and its irreducible representation are absolutely irreducible. In contrast, other relevant groups, such as finite cyclic groups, have real irreducible representations that are not absolutely irreducible. Could the framework presented here be extend to these cases? What potential challenges do you envision in extending to non-absolutely irreducible representations?\n5. Could you elaborate on the future directions for $k$-IGNs presented in the conclusions (L537-539)?\n\n**Minor Issues (No Impact on Recommendation):**\n- L073: I recommend specifying \"$2$-IGNs\" for transparency.\n- L183: Is the presentation of $P_\\tau$ unnecessary?\n- L340: The wreath product of groups is introduced but not defined in detail; as this operation is uncommon in machine learning literature, additional explanation would benefit Section 5. Also, consider demonstrating that equation 7 forms a linear representation of this group, perhaps in the appendix.\n- L420: Typo, “is prove”.\n- L379 and L1030: I cannot understand why $\\mathcal{V}^k$ is an irreducible representation of $\\mathcal{G}^k$; is it instead irreducible for $\\mathcal{G} \\wr S_n$?\n- L1040: The closing curly bracket is missing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- Clear presentation and notation, supported by rigorous proofs.\n- The methodology is both valuable and simple, with potential to generalize beyond the examples presented.\n- A novel and complete characterization of representations for unaligned symmetric elements.", "weaknesses": "- Lacks discussion on extending the approach to groups and representations beyond the few presented.\n- In particular, an appropriate discussion on characterizing the more expressive layers of $k$-IGNs for $k>2$ is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WeAYyP8aEP", "reviewer": "Reviewer_QTpm", "summary": "The paper considers the problem of constructing linear equivariant layers for groups acting (linearly) on input and output spaces. Specifically, it proposes to exploit the decomposition into irreducible group representations and then appealing to Schur’s Lemma, which reduces the problem to choosing coefficients for pairs of isomorphic representations. Several specific instances are analyzed, such as permutation groups in the context of graph neural networks, groups acting on weights of deep networks, and wreath products acting on products of representations.", "questions": "I would like the authors to comment on the above points regarding novelty and significance of experiments. \n\nMy current opinion is that the work is exceptionally well-written, and bears several contributions to the geometric deep learning literature. However, I am concerned with the novelty and significance, as outlined above. Still, I am leaning towards accepting the paper, but would like to hear from the authors about my points of criticism.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "-The paper is exceptionally well written. The language is clear and concise, the sections are structured, and the mathematical formalism/notation is elegant. \n\n-The problem considered is a fundamental one in machine learning literature. Constructing (linear) equivariant maps lies at the heart of geometric deep learning, which has been successful in several applications. \n\n-The proposed solution is general, as it applies, in principle, to any input/output group representation. Several existing frameworks are phrased under the same paradigm, contributing with structure and clarity to the geometric deep learning literature.", "weaknesses": "I believe that the proposed approach via Schur’s Lemma comes with disadvantages. To begin with, using Schur’s Lemma to construct equivariant linear maps is not novel in the geometric deep learning community. It is a rather well-known technique – see, for example, Behboodi et al., Section 3.2. This is a major concern, since Schur’s Lemma represents a core point of this work; the other contributions amount to rephrasings of known frameworks from the literature under the lenses of Schur’s Lemma. Moreover, Schur’s Lemma has some restrictions. First, it requires the decomposition into irreducible representations to be known a priori, which is not always the case. Such decomposition is challenging to compute algorithmically for general groups and representations.  Second, Schur’s Lemma applies naively only to complex representations (i.e., over $\\mathbb{C}$). As the authors mention, this is not an issue for permutation groups (appendix B), but it can be for other groups. It is still possible to apply Schur’s Lemma to arbitrary real representations of arbitrary groups, but this involves subtleties – see Behboodi et al., Section 8. \n\nI also find the experimental section rather weak. The experiments reported only consider ideal equivariant tasks, i.e., scenarios where the ground-truth function is equivariant. The experimental results show that adding equivariant layers to the network improves (generalization) performance, as compared to non-equivariant architectures. This is not surprising, since in these cases the inductive bias given by equivariance aligns perfectly with the structure of the task. In typical real-world scenarios (e.g., image classification), the (highly-noisy) ground-truth function is instead not exactly equivariant, or it is not equivariant on all the input data. In my opinion, it would be more informative and less trivial to test the models on these types of real-world tasks. The equivariance bias is often still beneficial in terms of generalization – as works in geometric deep learning have extensively shown – but empirical investigations are required to assess this carefully. \n\nMinor typos: \n\n-The paragraph title on line 86 is not capitalized, while the one on line 100 is. \n\n-The tables in section 6 exceed the margins of the paper.\n\n\nBehboodi et al., A PAC-Bayesian Generalization Bound for Equivariant Networks, NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)