In search of projectively equivariant networks
Authors: Georg Bökman
Axel Flinth
Fredrik Kahl
Venue: Accepted by TMLR ()

Abstract: Equivariance of linear neural network layers is well studied.
In this work, we relax the equivariance condition to only be true in a projective sense.
Hereby, we introduce the topic of projective equivariance to the machine learning audience.
We theoretically study the relation of projectively and linearly equivariant linear layers. We find that in some important cases, surprisingly, the two types of layers coincide.
We also propose a way to construct a projectively equivariant neural network, which boils down to building a standard equivariant network where the linear group representations acting on each intermediate feature space are lifts of projective group representations.
Projective equivariance is showcased in two simple experiments. Code for the experiments is provided in the supplementary material.

Published in Transactions on Machine Learning Research (12/2023)
In search of projectively equivariant networks
Georg Bökman'
Jokmann@chaberss..s
Deportment of Electricol Engineering
Cholmers University of Technology
Axel Flinth'
azel.
Department of Mathematics and Mathematical Statistics
Umeá University
Fredrik Kahl
Kredrikkahl@cchalie
Deportment of Electricol Engineering
Chalmers University of Technology
Reviewed on OpenReview: https: // openreview. net/ forum? id=1s1E16b7]8
Abstract
Equivariance of linear neural network layers is well studied. In this work, relax the equivariance condition to only be true in projective sense. Hereby, introduce the topic of projective equivariance to the machine learning audience. We theorctically study the relation of projectively and linearly equivariant linear layers. We find that in some important cases, surprisingly, the two types of layers coincide. We also propose a way to construct projectively equivariant neural network, which boils down to building standard cquivariant network where the linear group representations acting on each in- termediate feature space are lifts of projectíve group representations. Projective equivari- ance is showcased in two simple experiments. Code for the experiments is provided at equivariant deep nets
1 Introduction
Deep neural networks have been successfully applied across a large number of arcas, including but not limited to computer vision (Krizhevsky el al. 2012), natural language processing (Devlin et al., 2019), game play [Silver et al.] 2018 and biology (Jumper et al.) 2021). In many these areas, the data has geometric propertics or contains symmetrics that can be exploited when designing neural networks. For instance, AlphaFold (Jumper et al. 2021) models proteins in 3D-space while respecting translational and rotational symmetries. Much of the work neural networks respecting geometry and symmetry of the data can be boiled down to formulating the symmetries in termss of group equivariance. Group equivariance of neural networks is a currently very active area of research starting with (Wood & Shawe-Taylle 1996) and brought into the deep networks era by Cohen & Welling (2016). Group equivariant neural net are part of the broader framework of geometric deep learning. Recent surveys include (Bronstein et al. 2021 Gerken et al. 2021). Our interests equivariance are driven by applications in computer vision, (Bökman et al. 2022 Bökman Kahl 2022 2023). but in this paper, we will give application examples in several different research fields, targeting more general machine learning andience.
Given set of transformations T acting two sets X, Y, function f : X Y is called equicariant if applying f commutes with the transformations f E T, i.e.
{[f(x) = f(([=])
(1)
Equal contribution

---

Published in Transactions on Machine Learning Research (12/2023)
Concretely, consider rotations acting on point clouds and f mapping point clouds to point clouds. If applying I and then rotating the output yields the same thing as first rotating the input and then applying f. we call f rotation equivariant. In the most general scenario, t E T could act differently on X and Y; For instance, if the action on Y is trivial, we can model invariance of f in this way.
If the transformations T form a group, results from abstract algebra can be used to design equivariant networks. A typical way of formulating an equivariance condition in deep learning is to consider the neural network as mapping from vector space V to another vector space W and requiring application of the network to commute with group actions on V and W. As V.W are vector spaces the setting is nicely framed in terms of representation theory (we will provide a brief introduction in Section 2 What happens however when V and/or W are not vector spaces? This work covers the case when V and W are projective spaces, i.e,, vector spaces modulo multiplication by scalars. This generalization is motivated, for instance, by computer vision applications. In computer vision is common to work with homogeneous coordinates of 2D points, which are clements of the projective space ((R)). In Scction 2 we will describe more examples.
In the vector space case, any multilayer perceptron can be made equivariant by choosing the linear layers well non-linearities equivariant. This is one of the core principles of geometric deep learning. In this paper, we will theoretically investigate the consequences of applying the same strategy for building a projectively equivariant (to be defined below) multilayer perceptron, i.e. one that fulfills (1) only up to a scalar (which may depend on z). More concretely, we will completely describe the sets of projectively invariant linear layers in a very general setting (Theorem ).15).
The most interesting consequence of Theorem 2.15 is negative one: In two important cases, including the (O(3)-action in the pinhole camera model and permutations acting on tensors of not extremely high order, the linear layers that are equivariant in the standard sense (linearly equivariant) are exeetly the same as the projectively equivariant ones. This is surprising, since projective equivariance is a weaker condition and hence should allow more expressive architectures. Consequently, any projectively equivariant multilayer perceptron must in these cases either also be linearly equivariant, or employ non-trivial projectively equivariant non-linearities.
Theorem 2.15 also suggests natural way of constructing a projectively equivariant network that for certain groups is different from the linearly equivariant one. These networks are introduced in Scction 3 We then describe a potential application for them: Classification problems with class-varying symmetries. In Section 3.2 we describe this application, and perform some proof-of-concept experiments on modified MNIST data. In Section
4 we generalize Tensor Field Networks (Thomas et al.) 2018) to spinor valued data, whereby we obtain a network that is equivariant to projective actions of SO(3).
It should be stressed that the main purpose of this article is not to construct new architectures that can beat the state of the art concrete learning tasks, but rather to theoretically invest what projectively equivariant networks can (and more importantly, cannot) be constructed using the geometric deep learning framework. The boundaries that we establish will guide any practitioner trying to exploit projective equivariance.
We have for convenience collected important notation used in this article in Table 1 Appendix A also contains definitions of the mathematical objects we use in the article.
2 Projective equivariance
In this work, we are concerned with networks that are projectively equivariont. Towards giving a formal definition, let us first discuss how oquivariance can be formulated using the notion of representation of a group.
A representation ه of group G on vector space V over a field F is map which associates each group element with an isomorphism p(g)= V + V in manner that respects the structure of the group, i.c.,
( = Pigh) for all g G.
A more compact way of stating this is that p is a group homomorphism from the group G to the group of isomorphisms (general linear group) GL.(V) on V.
2

---

Published in Transactions on Machine Learning Research (12/2023)
P(V)
Projective space of V, i.e, (/ \ {0)) Space linear maps from to Projective general lincar group
Projection map from P(V)
Hom(V. W)
GL(V)
General linear group Group covering map
PGL(V)
Projective representation
Lincar representation
[n]
[0.1......7 1 1}
Zn
Additive group of integers modulo n
Sn
Permutation group of n elements
An
Permutations of signature 1
80(3)
Rotation group in 3D The identity matrix
SU(2)
Unitary matrices in C with determinant
id
The signum function on Sp-
Table 1: Symbol glossary
Figure The pinhole camera model. The black dot is the center of projection, i.c. the camera center. The red and yellow planes represent two different image planes corresponding to two different. viewing directions of a camera at the black dot. 3D points are projected to the image planes via their viewing rays - the straight lines connecting them to the center of projection. 3D rotations act on points in (say) the yellow plane in a projective manner when we rotate the camera from the yellow plane to the red plane, points on the yellow plane do usually not lie on the red plane but can be identified with points the red plane by projection in the camera center. E.g. the green point the ycllow plane transforms to point on the red plane under rotation of the camera by identilying the point with the green line and then projecting this line to the red plane. When working with homogeneous coordinates in computer vision, we identify points in the image plane with their viewing rays. Rotations act on homogeneous coordinates through projective representation. See also Example 2.1 and Example 2.5
We also assume that G is equipped with a topology, with respect. to which the multiplication and the inverse operation are continuous (that is, G a topological group). The representations are assumed to be continuous with respect to that topology. As we will also consider projective representations this paper, we will refer 'ordinary' representations (2) as linear representations. Given linear representations Po and Pi on spaces V and W, respectively, say that map $ V W is equivariant with respect to Po and f1 if
(g) (P)(g))) for all G.G.V..
If in (3,, (g) the identity transformation for every E G, we say that pi acts trivially and then & is called invariant. Many symmetry conditions can be phrased in terms (3). and ways to design neural networks $
satisfying (3 have boen extensively studied for c.g. rigid motions in 2D/3D/na (Cohen & Welling 2016
Weiler & Cesa 2019 Bökman et al. 2022 Bekkers et al., 2018 Thomas el. al. 2018 Cesa al.) 2021
permutations of graph nodes (Maron al. 2019 Zaheer et al. 2017 Qi et al. 2017) and for more general groups G (Wood & Shawe- Taylor 1996
Kondor & Trivedi 2018 Cohen et al. 2020 Aronsson 2022 F Finzi
2021b. The most famous example is probably the translation equivariance of CNNs (Fukushima & Miyake 1982 LeCun et al. 1998).
ct al.
this work, we want to study networks that are projectively in- and equivariant. To explain this notion, let first introduce the notation P(V) for the projective space associated to a vector space V over field F. That is, P(V) is the space of equivalence classes of V (0) under the equivalence relation w + EE F {0) s.t. == Aw. Furthermore, will write 1ly for the projection that maps V to its equivalence class (w) P(V).
Example 2.1. The pinhole camera model well-known and popular in computer vision (Hartley Zissorman, 2003). We illustrate in Figure | In essence, the idea is to identify a point y R² in an image with the line in &d that projects to y through the pinhole camera. Mathematically, this amounts embedding TR2
3

---

Published in Transactions on Machine Learning Research (12/2023)
to the 3D point ly, 1] E R2, and then considering it as point in P(R9). A 2D point cloud can in the same way be identified with a set of points in homogeneous coordinates, say [u, 1] E RR' , E [m]. In this manner, we can think of point-clouds X as elements of V = (IR)). Of course, multiplying all points in X with a scalar À does not change the image of the points, we can regard X as only being defined modulo R, i.e. as an element P(V).
Example 2.2. In quantum mechanics, the state of a particle is determined via a C-valued wave-fanction, that is an element & of some Hilbert space H. with (1),6) [Hall] 2013). Any measurable property A (modeled through self-adjoint operators on H) of the particle is however determined through sesquilinear products (brackets) of the form (b)) which is not changed when & is multiplied with a normalized scalar E C. Physical observations can hence only reveal vo modulo such a scalar, and should therefore be interpreted as an element of the projective space P(H).
The benefit of projective equivariance is eminent when trying to infer a projective feature transforming in a certain manner from projective data. think for example of inferring the center of mass of a point cloud (which is equivariant to rigid motions) from photo of the cloud. Motivated by these examples, we are interested in maps that fulfil the equivariance relation (3) up to multiplicative scalar. To formalise this notion, let us begin by introducing some notation. PGI((V) is the projective general linear group, i.c., the set of equivalence classes of maps A E GL(V) modulo multiplicative scalar. Note that an element M E PGL(V) defines a projective- linear map P(V) - P(V).
Definition 2.3. A projeclive representation of a group G on a projective space P(V) is group homomorphism p: F:G+PGL(C
Put more concretely, projective reprosentation is a map associating each E G' to an equivalence class p(g) of invertible linear maps. P respects the group structure in the sense of however, only in the sense of elements in PVL(V), i.e. up to a multiplicative scalar.
Example 2.4. The simplest form of projective representation is projected linear one. That is, given linear representation p: G - GL(,V, we immediately obtain a projective representation through p = NTLLVV o p. Example 2.3. SO(3) acts linearly on points in R33 by multiplication with the ordinary x 3 rotation matrices R. The projection of this representation gives a projective representation on P(R)). This projective representation acts pinhole projected points P(R3) by multiplying any vector R in the equívalence class by the ordinary 3 X 3 matrices R and interpreting the result as lying in P(RR))
Not all projective representations are projections of lincar ones, as shown by the next example. Example 2.6. Hall 2013) For every t= 0, 1,3/2, there is complex vector space V2 of dimension 2 +&11 1 on which the group SU( is acting by a linear representation. Together with the fact that SU(2) is a global double cover of SO(3), this can be used to define a map P from SO(3) to GL(VF) up to a scalar, i.e., a projective representation. For integer f, this a projection of linear representation of SO(3), but for half-integers 1/2,3/2,... ..., it mechanics.
not. The parameter l is known as the 'spin' of particle in quantum
Given projective representations Pu, PI on P(V) and P(W), respectively, we can now state equivariance of a map Φ P(V) - P(W) exactly as before 3):
P ( = (po(g))) for all g E G, u P(V).
Note that we only demand that the equality holds in P(W) and not necessarily in W, which was the case in We refer to O satisfying 4 as being projectively equivariant with respect to Po and P1-
2.1 Projectively equivariant linear maps
A canonical way to construct (lincarly) oquivariant ncural networks to alternate equivariant activation functions and equivariant linear layers. More concretely, if we let Vo denote the input-space, V,, =1 1,..., K K-1 the intermediate spaces and Vx the output space, a multilayer perceptron is function of the form
((v) =u((v, Uh+1 = (Au 0,... K K-1 1.
4

---

Published in Transactions on Machine Learning Research (12/2023)
It is now clear that if As and dh all are equivariant, & will also be. This method easily generalizes to projective equivariance: one simply needs to restrict the equivariance condition to a projective equivariance one, In the following, we will refer to such nets as canonical.
This construction motivates the question: Which linear maps A V + W define projectively equivariant transformations P(V) + P(W)? If the projective representations Po and Pi are projections of linear representations 5O and PL respectively, we immediately see that every equivariant A defines projectively equivariant transformation. These are however not the only ones an A satisfying
Apolg) = (1)), gEG
(5)
for some function E G + F will also suffice. For such relation to hold for non-zero A, needs to be (contimuous) group homomorphism. The set of these E, equipped with pointwise multiplication, form the so-called character group.
Desinition 2.7. Let G be topological group. The set of continuous group homomorphisms e: F is called the character group of G, and is denoted G".
Remark 2.8. Given an e E G and a linear representation p G, we may define a new representation f through p (g) = =(g)p(g). Hence, means that A is linearly equivariant w.r.t. representations pu and pi.
The condition (5] is surely sufficient for the map A defining a projectively equivariant linear map. In the following, we will prove that in many important cases, it in fact also is necessary. To make our considerations general enough to include examples like Example 2.6 we will not assume that p is a projection of a p, but only that we can 'lift', p to a linear representation of a so-called covering group of G. We need the following two definitions.
Def. 5.36) Let G be a topological group. A group H is called a covering group of G if there exists a group covering yp H + G, ie. a surjective continuous group homomorphism which maps some neighbourhood of the unit clement eн E H to neighbourhood of the unit element eg E G homeomorphically.
Definition 2.9. Hall 2015
Definition 2.10. Let p: G + PGL(V) be a projective representation, and H a covering group of G, with group covering yp. A lift of p is a linear representation /: H - GT(V) with po p = [TGL(V) o
Enample 2.11. If p: G - PVL(() is a projection of a linear representation p: G - GL(V) as in Example 2.4 then P is a lift of (the covering group is simply G itself).
Example 2.12. The linear representations related to spin discussed in Example 2.6 are defined on SU(2), which is a covering group of SO(3). The linear representations are lifts of the projective representations of SO(3) discussed there.
Remark 2.13. Given a projective representation p of a group G, there is a canonical way of constructing a covering group И (dependent on p) and linear representation X of H which lifts This construction is however mainly of theoretical interest, and our subsoquent results will not give us much insight when this lift. is used. We discuss this further in Appendix B.7
Let us first reformulate the equivariance problem slightly as is routinely done in the linear case WWilerr & Cesa 2019 Finzi et al. 2.21h. Given projective representations Po and #1 on P(V) and P(W), we may define a projective representation p G PGL(HommV, W)) on the space Hom(V, W) of linear maps between V and W, through
p(g) = (1)4))
On the left hand side we use p(g) to transform A E P(Hom (V, W)) to obtain a new element in P(Hom (V. W)).
On the right hand side we write this new element out explicitly it is a composition of the three maps P (g), A and pi))). Just in the linear case, can reformulate projective equivariance of an A E Hom (V, W) invoriance equation under p.
Lemma 2.14. A linear map A: V W projectively equivariant if and only if A is invariant under p. that is p(g) A= A for all g E G. These equations are understood in P(Hom(V, W)).
5

---

Published in Transactions on Machine Learning Research (12/2023)
The proof, which in contrast to the nonlinear case necessitates non-trivial technical effort, can be found in the appendix. The key is that the equivariance condition for A means that any solves the 'eigenvalue problem' P1 (z) = Apo(gg)) from which p = A in P(Hom(V, W)) can be deduced. The lemma implies that we can concentrate on invariance equations
p(g)== = v for all g G
(Proja)
which, importantly, are understood in P(V) for some general vector space V (which covers the case V Hom(V, W)). With this knowledge, the main result of this section is relatively easy to show. It says that given a lift p of p. the solutions of projective invariance equations (Projal of p are exactly the ones of the s-linear invariance equations of w.r.l. the covering group H,
p(h)o E(h)v for all H,
(Lin
where ranges the whole of H Note in particular that Ling is nothing but the standard linear invariance problem for P. For a given E E H", we denote the space solutions to (Lin by U
Thcorem 2.15. Let G be group and H covering group of with covering map p. Further, let p: G PVL(V) be projective represenlation of G and p: H GL(V) lift of p. Then, the following are equivalent
(i) solves Proja
(tt) is the equivalence class of some US, H"
Proof. To prove that (ii) + (i), note that (MM = (IL for M E GL(V) and V. Let æ solve [ For any E G, there is a h such that (p(h) and hence
plg) p( = (P(h)
HI (P(h)=) Ny((()))) Iy(a)),
meaning that the equivalence class of solves [Poojc]
To prove that (i) + (ii), let (x) be a solution of [Proja] for some + 0. ((h)æ must then for all h lie in the subspace spanned by x-in other words, there must exist a map A: : with p(h)x = X(h)æ. Since
X(h)A(k)< = ((h))(()) = ((hk)ar = A(hk)ar, must be group homomorphism. Also, since p continuous, A must also be. That is, À H1, and U².
2.2 The structure of the spaces U
Let study the spaces U=, Before looking at important special cases, which will reveal the important consequences of Theorem 2.15 discussed in the introduction, let us begin by stating two properties that hold in general.
Proposition 2.16. (i) V is finile dimensional and H is compact, U is only non-trivial if maps into the unil. circle.
(ii The U are contained in the space Uhh of solutions of the linear invoriance problem
p(h)== for all h E (H,
(Lint
of the commutator subgroup HH, f.e. the subgroup generaled by commulators (h, 1 1k,,,, F C, V is finite dimensional and H is compact, Uhh is even the direct sum of the US, E H*
h1, k H.
The proofs require slightly more involved mathematical machinery than above. The idea for (1) that the T in this case can be assumed unitary. The "C" part of (ii) is simple calculation, whereas we for the "direct sum" part need to utilize the simultaneous diagonalization theorem for commuting families of unitary operators. The details are given in Appendix B.4
We move on to discuss the structure of the spaces U for three relevant cases. We will make use of a few well-known group theory results for convenience, we give proofs of them in B.8
The rotation group. SO(3) is a perfect group, meaning that SO(3) is equal its commutator subgroup (SO(3), SO(3)) This has the consequence that SO(3)* only contains the trivial character which trivially shows that (Projso(3) is equal to the space U1, In other words, we have the following corollary.

---

Published in Transactions on Machine Learning Research (12/2023)
Corollary 2.17. For projections of linear representations p: P:SO(3) - GL(V), (Projso(3) is equivalent to the linear equivariance problem [inso(3)
This mcans that one wants to construct a canonical projoctively cquivariant neural net (Example 2.1 one either has to settle with the already available linearly equivariant ones, or construct a non-linearity that is projectively, but not linearly equivariant. To construct such non-linearities is interesting future work, but unrelated to the linear equivariance questions we tackle here, and hence out of the scope of this work.
The pinhole camera representation is projection of a linear representation of SO(3). Not all projective representations are, though they can also be linear projections of representations of SU(2). Since SU()) also perfect, there is again an equivalence, but only on the (U))e-leve. Hence, in general the projective invoriance problem for the repees-rprssnntttionn not necessarily equivalent to linear equivariance problem [ but always to one of the form (Lingu(2)
The permutation group. The behaviour of SO(3) is in some sense very special. More specifically, it can be proven that if G is compact, G = {1) only when G is perfect (see Section B.8 However, equivalence of Projso(3) and Ling can still occur when G* is not trivial.
An example of this is the permutation group Sn acting on spaces of tensors. The character group S of the permutation group contains two elements: and the sign function sgn. This means that the solutions to Prois.. form two subspaces U and USee, However, in the important case of p being a projection of the canonical representation p of S, the tensor space (F*)** defined by (f(T)z)) = 2x-1(1) for any multi-index I T [n]* something interesting happens: Usgn empty for n 2 + 2. We prove this in Section B.5 In particular, we can draw another interesting corollary.
Corollary 2.18. For the canonical action of Sn on V = (Fm) for 2 ++ linear equivariance problem [Ling.)
Projs is equivalent to the
In more practical terms, this means that unless tensors of very high order (A: S n - 2) are used, we cannot. construct canonical architectures operating tensors that are projectively, but not lincarly, equivariant to permutations without designing non-standard equivariant non-linearites.
The translation group. Σe is isomorphic to the set of n:th roots of unity: every root of unity w defines a character through (k) = w E Z,, Furthermore, for the standard lincar representation on F", each of the spaces Uhe is non trivial it contains the element defined through i = -k; (p(()v)) = u = w (E)uk, In the case F- = C, this leaves us with n linear spaces of protectively invariant glements-wiin not hard to identify as the components of the Fourier transform of v. If = R, the number of roots of unity depend - if is odd, there are two roots {+1, but if is even, the only root is +1. In the latter case, we thus again have an equivalence between (Projz_ and Ling
are
3 A projectively equivariant architecture
Theorem 2.15 is mainly a result which restricts the construction of equivariant networks. Still, we can use its message to construct a new, projectively equivariant architecture as follows. We describe the architecture as a series of maps Vi- - V between fiiitedimeensionll spaces Vy. Theorem 2.15 states that all projectively equivariant linear maps V, + V1 lie in some space U Hence, for each & E H² we may first multiply the input features Vo with maps Aj Us, to form new, projectively equivariant features. By subsequently applying equivariant non-linearity do V1 + V1, obtain the Vi-valued features
(6) It is not hard to see that as soon as the non-lincarity do commutes with the elements e(h), h E H, the features of will be projectively equivariant. Note that by Proposition 2.16 since the V, are finite dimensional and H is compact, the values <(g) will always lie on the unit circle. Hence, we can use tanh-nonlinearity in the real case, or the modReLU (Arjovsky et al. 2016 in the complex-vaiiee valued one.
of = (Afbt) AB W E H"
Also note that we here get one feature for each e H" so we map from Vo to |tuple of Vi's. U is however nontrivial only for finitely many g this is consequence of Proposition 2.16 and the fiittedimensionallty of
7

---

Published in Transactions on Machine Learning Research (12/2023)
Vo
V1
222
Ov U 3 yya y Ua?
Ua
Figure 2: The structure of three-layer projectively equivariant net for H = Zz. Z consists of three elements of the form c(k) 0h, where = 1, a, = e2xi/3_w we write Z = f1, a, a²). The features vf are combined with linear operators in U to produce new features of, with 7, E 2g. In a final step, the features of the e-indexed tuples are linearly combined with a 'selector' p to yield one single output. Best viewed in color.
the Vi- Said proposition also shows that the sum of their dimensions does not exceed dim(URH) (UR))- Hence, each tuple (Ag) can be represented with no more than dim(Unn) I dim(V&)) dim(Ve+1) scalars.
The features of can be combined with new linear maps AT, HH" to form new features follows:
0 =1 01
Σ
174
7HEH ,6==
This process can continue for = 11:3,,4, 3..,...,, K until we arrive at projectively equivariant Vx-valued features. We record that theorem, which we formally prove in Appendix B
Theorem 3.1. If the o; commute with the elements E(g), H", h E H, the features oi are projectively equicariant.
Also, the above construction inherently produces |H*] output features ufx E VK. If we want a single output feature, we propose to learn a "selection-vector" E Fld1, that linearly decides which feature(s) to 'attend' to:
Σ DeVk- SEH
W
Technically, this does not yield an exactly equivariant function unless all but one entry in equal to zero. Still, believe it is reasonable heuristic. In particular, cannot a priori pick one of the vî output, since that fixes the lincar transformation behaviour of the network. For instance, choosing uk leads to a linearly equivariant network, not able to capture non-trivial projective equivariance. Note also that we may employ sparse regularization techniques to encourage sparse p, e.g. to add an f-term to the loss. We provide a graphical depiction of our approach for the (simple but non-trivial) case of = Σ3 in Figure
3.1 Relation earlier work
As described above, once we fix E, (5) is a linear equivariance condition on the linear map A. Hence the construction proposed here is equivalent to the following linear equivariance approach. For each feature space Vv, & > 0, duplicate it [H** times to form a new feature space V = DDER- Vx and select the linear representation DeeH Hf to act Ve. Then parametrize the neural network by linearly equivariant maps Vi - Ve+1- Depending on the group H and the spaces Vi this can be done by various methods found in the literature (Cohen Welling 2016 Finzi et al. 2021 Cesa et al. 2021 Weiler & Cesa 2019 In particular, if U is empty for E # 1, or if there are no non-trivial E, this construction will be exactly the same as the standard lincarly equivariant one. This is for instance always the case for SO(3) as explained leading up to Corollary 2.17
8

---

Published in Transactions on Machine Learning Research (12/2023)
1.03
1.44
035
055
11:00
0:744
0.1
0.7
031
Figure 3: Left: The group action the example in Section 3.2
Right: Training (left) and test (right)
accuracy for the MNIST experiments. The lines report the median performance for each epoch, wheras the errorbars depict confidence intervals of 80%.
3.2 A possible application: Class-dependent symmetries
Let us sketch a somewhat surprising setting where our projectively equivariant network can be applied: Classification problems which exhibit class-dependent symmetries.
For clarity, let consider very concrete example: The problem of detecting 'T'-shapes in natural images. We may phrase this as learning function Pr R. [0, 1] giving the probability that the image IR., contains 'T' This probability is not changed when the image is translated, when it is horizontally flipped. The same is not true when flipping the image vertically- an image containing 'T' will instead contain 'll symbol. In other words, PT(v) 2 + pr (Uver, flpp) z 0. The action of the two flips, which commute,, can be modclled with the help of Klein's Vierergroug! ZZ (1, corresponds to the vertical flip, and (0, 1) to the horizontal one.
Let write pr = Poe where Po is some small number. If is an image neither containing 'T" or a '1', (T()) should be small letting (T suffices in this case. If instead is an image containing 'T', (r(v) should be large, and Φr (Uver. fip)) should be very small. This is satisfied by functions with ((vver. slip) = (Q()). Hence, fr should modulo a multiplication with -1 not change by any flip - that is, should be projectively invariant. Note that 'projectively invariant here a priori is too unrestrictive it formally means qr (ever. nip) = (q) for arbitrary À + O, which any function that is non-zero for both Uver, and u satisfies. As we have seen, in our architecture, the scalar can however only be equal to #1, which is the behaviour we scarch for.
The exact reasoning can be applied to the probability pn of an image containing 'H. In this case, however, РH equivalently, ди is linearly and therefore projectively, invariant of either flip. Put differently, ęи and qr are lincarly equivariant with respect to different reprosontations p², and therefore both projectively equivariant w.r.t. the same projective represenlation See also Figure
Now imagine training both q and qr on a dataset containing both "T's and 'H'ss. If we were to use canononical lincarly equivariant architocture (with respect to a single p²), we could not expect a good performance. Indeed, one of the funetions would necessarily not have the proper transformation behaviour. In contrast, our architecture has the chance through learning class-dependent selection vectors - to adapt to the correct symmetry for the different classes.
Remark 3.2. In this particular example, since we know the symmetries each class a-priori, we could of course choose two different lincarly equivariant architectures to learn each function. Our architecture does however not need this assumption it can be applied also when the symmetries of the individual classes are not known a-priori.
As a proof of concept, we perform toy experiment in similar setting to the above. We will define an image classification problem which is linearly invariant to translations, and projectively invariant to flips along the
Vier" German for four', referring to the number of elements in the group.
9

---

Published in Transactions on Machine Learning Research (12/2023)
horizontal and vertical axes, i.e. the Vierergroup Z3. Its character group also contains four elements the values in (1,0) and 1) can both be either +1 -1.
Data. We modify the MNIST dataset (LeCun et al. 1998), by adding an additional class, which we will refer to as 'NaN: When an image a E p.." is loaded, we randomly (with equal probability) either use the image as is, flip it horizontally or flip it vertically. When flipped, the labels are changed, but differently depending on the label: if it is either 0, or 2, it stays the same. If it is 3, 4 or 5, we change the label to NaN if the image is flipped horizontally, but else not. If it is 6 or 7, we instead change the label if the image is flipped vertically, but else not. The labels of 8 and 9 are changed regardless which flip is applied. Note that these rules for class assignments are completely arbitrarily chosen, and we do not use the knowledge of them when building our model.
Models. Since we operate on images, we build a linearly translation-eeuuiariatt architecture by choosing layers in the space of 3-convolutional layers. We create ViererNet-a four-layer projectively equivariant network according to the procedure described in this section and four-layer baseline CNN with comparable number of parameters. Details about the models can be found in Appendix [
Results. We train 30 models of cach type. The evolution of the test and training accuracy is depicted in Figure 3 along with confidence intervals containing 80% of the runs, We clearly see that the ViererNet is both better at fitting the data, and is more stable and requires shorter training to generalize to the test-data. To give a quantitative comparison, we for each model determine which epoch gives the best median performance on the test data. The ViererNet then achieves a median accuracy of 92.8%, whereas the baseline only achieves 90.2%. We subsequently use those epochs to test the hypothesis that the ViererNet outperforms the baseline. Indeed, the performance difference is significant (p 0.025).
In Appendix D similar modifications of the CIFAR10 dataset are considered. This dataset is less suited for this experiment (since one of the flips often leads to a plausible CIFAR1O-image), and the results are less clear: The basclinc outperforms the VicrerNet slightly, but not significantly so. To some extent, the trend that the ViererNet requires less training to generalize persists.
4 Generalizing Tensor Field Networks to projective representations of SO(3)
In this scction we consider regression task on point clouds. The task is equivariant under a projective representation of SO(3) corresponding to a linear representation of SU(2) as was briefly described in Example 2.6 We give an introduction to the representation theory of SU(2) in Appendix C In short, SU(2) has 2 +1 dimensional irredueible representation for each e f= 0, 12, 1, 1,3/2,.... The vector spaces these representations act on are isomorphic to C2(+1 and we denote them by V/-
Our main result (Theorem 2.15 and the discussion in Section 2.2 show that we for projective SO(3) equivariance could build a net using lincarly SU(2) equivariant layers. We will here use linear layers of the tensor product type used in Tensor Field Networks (TFNs) (Thomas et al. 2018) and many subsequent works (Geiger & Smidt, 2022 Brandstetter et al. 2021). The filters and features are then functions on IR3, and the resulting construction hence deviates slightly from our theoretical results see Appendix C but we think that gencralizing this canonical approach for point cloud processing networks to projective equivariance is of high relevance to the paper.
Data. The data in the experiment consists of point clouds equipped with spinor features and regression targets that are spinors. By spinors we mean elements of V1/2- That is, one data sample is given by where I; E R3 are the 3D positions of the points, E V1/2 ~ C2 are the attached spinor features and t E V1/2 is the regression target. We define an action of SO(3) on these samples by
R- R- (1(x,,, (i) ==
(8)
Here (R) an element of SU(2) that corresponds to R. We note that since SU(2) double covers S0(3), U (RR) only defined up to sign. Hence, 8; and can be thought of as E-projective, and the action of SO(3) them as projective representation.
10

---

Published in Transactions on Machine Learning Research (12/2023)
Figure 4: Data used for the regression task with S0(3) (quuv--quvvarianee We define four classes, each class being a point cloud with three points in 3D. To each 3D point we attach a spinor feature 8 E CC. These spinor features are plotted inside the frames next to each 3D point, the frames and points are color coded. Re(s) is plotted in red and Im(s) in blue. For every class different spinor should be regressed, which is shown in the black frame. Top: The four classes. They are defined such that if one looks only at the spatial locations are the second and fourth. If one instead looks
of the points, the first and third classes are equivalent only at the spinor features, the first and second classes are equivalent as are the third and fourth. A network that regresses the output spinor correctly for all classes must take into account both the locations and the spinor features of each point. Bottom: The regression should be equivariant under rotations of the point clouds. When a point cloud is rotated by R E SO(3), the spinor features and labels are transformed by a corresponding matrix (R) SU(2). A random rotation of the leftmost point cloud is shown in the center column. Given rotation R, the corresponding ((R is however only defined up to sign. E.g., the identity rotation corresponds to both the identity matrix I E SU(2) and -I. The rightmost column illustrates a correct regression even though all spinor input features have been multiplied by -I, while the output spinor has not.
The task is to regress the larget in a way that is equivariant to 8 and invariant to translations of the point clouds. The data is defined by four prototype samples that are shown in Figure 4 To make the task more difficult we also add Gaussian noise to the spatial coordinates rı, varying the noise level (i.e., std.) from to 2/5. During training, a network sees only the four prototypes + noise, while evaluation is done on rotated prototypes noise. According to Theorem 2.15). and the discussion regarding the rotation group in Section 2.2 the projectively -0uu)--quivariant network architectures are in this case exactly the standard linearly (2euuququaa networks. We illustrate the data and how it transforms under SO(3) in Figure
Models. We will tackle the problem by building an equivariant net in the spirit of Tensor Field Networks (TFNs) (Thomas et al. 2018). Let us first recall how TFNs work. Our data are point clouds {=r]i., in R with features fi in some vector space V altached to each point, on which SO(3) is acting through pv. We map the inputs to point clouds with features fú in some vector space W (on which SO(3) acts through Pw) via convolution with filter function V BR Hom(W, V) through
Σ V x;)fs. j
(9)
Provided the filter v satisfies the invariance condition V (Ra) (C)) defines layer invariant to translations and rotations of R3, and the S0(3) on V and W. We could make an ansatz of Te as member of some finiteddimensional space of functions R33 + Hom(V, and then resolve this invariance condition directly using our theory. However, the resulting arcitecture would be numerically heavy to implement. We have therefore instead used the idea of TFNs.
R SO(3), this
The idea of TFN is to define using a tensor product, (=)x= u V... Here, v(x) lives in some third vector space U, and satisfies the condition ((Rx) Pu (R) R EE v maps equivariantly into the space W U V, which can be decomposed into spaces isomorphic to the irreps Vo, V1, V2,... of the
11

---

Published in Transactions on Machine Learning Research (12/2023)
1/2
0.8
Spacet scakes
Spacen fratinn
9.9
Spaion
Miti no. digi
Figure 5: Results of the spinor regression experiments for varying architectures and spatial noise levels. We plot the mean over 30 runs, showing the standard deviation with errorbars.
0.4
Sparn squzee
Spiners squered
0.2
vodag
0.0
0.0
0.2 Nore lod
033 0.4
action of (O(3). In this manner, we may transform features in different irreps into new ones living in other irreps. For instance, if we combine features and filters living in the space V ~ R (i.e., vectors), we end up with one scalar component Vo, one vector component in V1 and a traceless symmetric 3 3-matrix component. in V22 The higher order component 1/2 can be discarded to keep the feature dimensions low. Another type of layer are self-interaction layers where features of the same irrep at each point are linearly combined using leartt weight matrices.
The idea of our approach is now to let U and V be spaces on which SU( is acting, i.c., also consider features and vectors of half-integer spin. We limit ourselves to é = 0, 1/2 and 1, i.e., scalars, spinors and vectors, and call the resulting nets Spinor Field Networks. The hyperparameter choices for a Spinor Field Network in our implementation are the number of scalar, spinor and vector features output by each layer. We use sigmoid-gate gated nonlinearities (Weiler et al., 2018) for non-scalar features and GeLU for scalar features.
We consider in total five different models. (i) Spinors scalars. A non-equivariant baseline TFN that treats input and output spinors four real scalars. It has the correct equivariance for the 3D structure of the locations but not for the spinors. (ii) Spinors as features. An -quiivaann net as described above, with intermediate scalar,vector vector and spinor features. (iii) Spinors squared as vector features. Prior to the first layer, we tensor square the spinor features s; + 8; @ s; and extract a vector feature at each point, which we then process equivaaiuivr This allows the network to only use real valued (scalar and vector) features in the intermediate layers, which likely an advantage in most modern machine learning frameworks (including PyTorch which we use). In the last layer, the scalar and vector features are tensored with the input spinors to produce output spinor.
The last network types have filters that consist of not only Vi valued functions, but also the input spinors themselves. That is, we define layers = - xx)) @ sj) @ fz- This could be generalizod to tensor spherical harmonics valued filters which we explain C.1 We consider two such types: (vi) Spinors as filters works as just described. (v) Spinors squared as vector filters analogously defines vector valued filters by tensor squaring the input spinors as in the Spinors squared as vector features case. This again allows for a mostly real valued network.
Each network consists of three layers and the hyperparameters are chosen to make the number of parameters approximately equal between all networks. In all networks the input feature at a location I; consists of the average of the vectors x2 - Iy for j # i and for all except the two last networks also the spinor feature s; (interpreted as either 4 scalars, a spinor or vector after squaring). The final output is the mean of the features at each input location, meaning that the last layer maps to single spinor feature- feature-excopt in the non-equivariant variant in which the last layer maps to 4 scalar features. We describe the training setup and further details in Appendix C,2
Results. All networks are trained on just the four point clouds shown in the left part of Figure 4 (plus added spatial noise), except one version of the non-equivarianl net which is trained on randomly rotated versions of these four point clouds as data augmentation. The evaluation is always on randomly rotated versions of the data. We present results in Figure 5) Notably, the networks that tensor square the spinors before the first layer to then work on real-valued data until the last layer perform the best. At low noise levels, the networks that work directly on complex-valued spinors also perform well, but their performance
12

---

Published in Transactions on Machine Learning Research (12/2023)
degrades quickly with added noise. The network without correct equivariance unsurprisingly performs quite poorly with, and extremely poorly without, data augmentation. The networks with filters defined using the input spinors outperform the networks where the inputs are only fed in once at the start of the network. This is likely due to the fact that inserting the inputs in each layer makes the learning casier for the network.
5 Conclusion
In this paper, we theoretically studied the relation between projective and linear equivariance for neural network linear layers revealed that our proposed approach the most general one. We in particular found examples (including 5O(3), or rather SU)) for which the two problems arc, somewhat surprisingly, equivalent. Building on the knowledge of the structure of the set of projectively equivariant layers our main result gave us, proposed a neural network model for capturing projective equivariance, and tested it toy task with class-dependent symmetries. Finally, we experimentally evaluated the merit of using projective equivariance for tasks involving spinor-valued features.
Acknowledgement
All authors were supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NATSS) and the Swedish National Infrastructure for Computing (SNIC) at C3Se Chalmers, partially funded by the Swedish Research Council through grant agrecments no. 2022-06725 and no. 2018-05973.
References
Martin Arjovsky, Amar Shah, and Yoshma Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pp. 1120 1128. PMLR, 2016.
Jimmy Aronsson. Homogeneous vector bundles and g-equivariant convolutional neural networks. Sampling Theory, Signal Processing, and Dala Analysis, 20(2):10, 2022.
Erik J. Bekkers, Maxime W. Lafarge, Mitko Vela, Koen AJ Eppenhol, Josien PW Pluim, and Remco Duits. Translarannn Covariant Convolutional Networks for Medical Image Analysis. Xiv: 1804.0333333 fes, math/, June 2018. URL http::/aaxw...rgg/aaaa.. a80aaaabss///.0. 03393 arXiv: 1804.03393.
L. C. Biedenharn and James D. Louck. Angular Momentum in Quantum Physics: Theory and Application. Encyclopedia of Mathematics and Its Applications. Cambridge University Press, Cambridge, 1984. ISBN 078-0-521-30228- doi: 10.1017/CB097805117550
Georg Bökman and Fredrik Kahl. A case for using rolation invariant features in stale of the art feature matchers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5110-5119, 2022.
Georg Bökman and Fredrik Kahl. Investigating how ReLU-networks encode symmetries. In Thirty- Conference on Neural Information Processing Systems, 2023. URL httpps///penrrevieeenee///eeeeee 81bFupebeu
Georg Bökman, Fredrik Kahl, and Axel Flinth. Zu-net: A universal rotation equivariant architecture for 2d point clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognation, pp. 10976 10985, 2022.
Johannes Brandstetter, Rob Hesselink, Elise der Pol, Erik J Bekkers, and Max Welling, Geometric and physical quantities improve (3) equivariant message passing. arXiv preprint arXic:2110.022) 00.0200, 2021.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličkovié. Geometric Deep Learning: Grids, Groups, Graphs, Goodesics, and Gauges. vrXim:2104.13/33 fes, stat), May 2021. URL http://arxiv.org/ abs/ 2b04. 13478 arXiv: 2104.13478.
13

---

Published in Transactions on Machine Learning Research (12/2023)
Gabricle Cesa, Leon Lang, and Maurice Weiler. A Program to Build E(N)-Equivariant Steerable CNNs. ICL.R, September 2021. URL https://openrewiev.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990 2999. PMLR, 2016.
Taco Cohen, Mario Geiger, and Maurice Weiler. A General Theory of Equivariant CNNs on Homogeneous Spaces. arXic:181100201 (es, stat), Jamuary 2020. URL http://arxiv wtt.:/.rrgbbbbbb.b.aae 02017 ar 1811.02017.
Jacob Devlin, Ming-We, Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North Americon Chapter of the Association for Computational Linguistics: Humon Language Technologies, Volume (Long and Short. Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Compulational Linguistics. doi: 10.18653/ /1/N19-1422, URL httpss://aclanthoolog/ hww.pgyy..r/NN11--111111
Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 Proceedings of Machine Learning Research, pp. 3318 3328. PMLR, 2021. URL htttss:/pproceddingg werpeiinnnnmmr......mmmmmemmm mwr. wwa.r.pess/vvle 1ttps///v19/f/ni2111ee html
Kunihiko Fukushima and Sei Miyake. Noocognitron: A self-organzing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp. 267 285. Springer, 1982.
Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXiv preprint ar arXiv:2307.09450 2022.
Jan E. Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and Daniel Persson. Geometric Deep Learning and Equivariant Neural Networks. arXiv:2105.1890 [hep-th], May 2021. URL Sttt:///arxx..orr//a//211155 13926 arXiv: 2105.13926.
Brian C. Hall. Quantum Theory for Mathematicians. Graduate Texts in Mathematics. Springer, 2013.
Brian C. Hall. Lie Groups, Lie Algebras, and Representations: An Flementary Introduction, pp. 49-76.
Springer International Publishing, Cham, 2015. ISBN 978-3-319-13667-3 URL https://arxiv.orr
nath-ph/0005032.0 pdf
Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge, UK New York, 2nd ed edition, 2003. ISBN 978-0-5211-4051-33
Takeshi Hirai, Akihito Hora, and Etsuko Hirai. Introductory expositions on projective representations of groups. In Projective Representations and Spin Characters of Complex Reflection Groups G(m, n) and G(m.p. p. 0o), volume 29, pp. 1-48. Mathematical Society of Japan, 2013.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowic, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmect Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, July 2021. ISSN 0028-0836, 1476-4687. doi: 111103//41556-020 URL http: : wwwww nature. wwt.eooooaaaeeeeeeee
Aleksandr A Kirillov. Elements of the Theory of Representations, volume 220. Springer Science & Business
Media, 2012.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks the action of compact groups. In International Conference on Machine Learning, pp. 2747-2755. PMLR, 2018.
14

---

Published in Transactions on Machine Learning Research (12/2023)
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch convolutional neural network. Advances in Newral Information. Processing Systems, 31, 2018.
Clebsch-gordan gordan nets: fully fourier space spherical
Alex Krizhevsky, Lya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)22278 2324, 1998.
Gregory T. Lee. Abstract Algebra first course. Springer Undergraduate Mathematics Series. Springer, 2018.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations, 2019.
Oystein Ore. Some remarks on commutators. Proceedings of the American Mathematical Society, 2(2): 307 314, 1951.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IFFE conference on compuler vision and paltern recognition, pp. 652 660, 2017.
Walter Rudin. Fourier Analysis Groups. Interscience publishers, 1962.
David Silver, Thomas Hubert, Julian Schrittwieser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140-11444 2018.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and traanslation--qquivariann neural networks for 3D point clouds. orXiv:1802.088: 08219 arXiv: 1802.08219.
[es], May 2018. URL http:///raaa..oo////////222.2222
Maurice Weiler and Gabriele Cesa. General -quiiiqurar steerable cmns. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Aché--Bu,, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL
www..ne..rcccooo cc/pperr/
pdf
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3D Steerable CNNs: Learning Rotationally Equivariant, Features in Volumetric Data. Advances in Neural Information Processing Systems, 31, 2018.
Jeffrey Wood and John Shawe-Taylor Representation theory and invariant neural networks. Discrete Applied Mathematics, 69(1-2):33-60, August 1996. ISSN 0166218X. doi: 10/c3qmr6.
URL https
//linkinghub..o httpa//gii..llaieeeooo//oooeeeea
Manzil Zahecr, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep Sets. Advonces in neural information processing systems, 30, 2017.
Zee. Group Theory in Nulshell for Physicists. In a Nutshell. Princeton University Press, 2016. ISBN 978-0-691--6269-0.
15

---

Published in Transactions on Machine Learning Research (12/2023)
A Technical definitions
Table contains symbol glossary for various symbols used in this work. The remainder of this section contains definitions used in the text, in particular a few that did not have room in the main text, compiled for the convenience of the reader.
Definition A.1 (Group homomorphism). Given two groups G, H, a group homomorphism from G to H is a map a: G - H that respects the group structure, i.c., such that a(91))) = a( (()))(()))) for all 91, 92 G. Definition A.2. Given a vector space V over field IF, P(V) is the corresponding projective space consisting of equivalence classes under the relation
U W v == Aw for some À + 0
(10)
Definition A.3 (General linear group, Projective linear group). The general linear group GL(V) of a vector
space V is the group of all invertible linear maps from to itself. The set of equivalence classes of GL(V) under the equivalence relation A ~~ B + A= for E F\ is the projective linear group PGL(V).
Definition A.4 (Linear representation). linear represenlation of a group G on a vector space V a group homomorphism p: G GL.(V).
Definition A.5 (Projective representation). A projective representation of a group G on a projective space space P(V) is a group homomorphism p: G PGL(V).
Definition A.6 (Linear (projective) equivariance). Let V and W be vector spaces equipped with linear (projective) representations ev and ew. A linear map A e Hom(V, W) is then linearly (projectively) equivariant if
(a geeo,,gee g
as elements in W (in P(W)).
Definition A.7 (Topological group). A topological group is a set that is simultancously topological space and group such that
The group operation continuous w.r.t. the topology.
2. The inversion map -1 is continuous w.r.t. the topology.
Definition A.8 (Homcomorphism). A homcomorphism is an invertible, continuous map between two topological spaces whose inverse is also continuous. A function maps a set homeomorphically if the restriction of the function to that set is homeomorphism.
Definition A.9 (Character group). Let G be topological group. The set of continuous group homomor- phisms G - IF is called the character group of G, and is denoted G*
Definition A.10 (Covering group). Let G be topological group. A group H is called a covering group of G if there exists group covering yp H - G, ie. surjective continuous group homomorphism which maps some neighbourhood of the unit element eн E H to neighborhood of the unit element eg E G homeomorphically.
Definition A.11 (Lift of a projective representation). Let p G + PGL(V) be a projective representation, and H a covering group of G, with group covering A lift of p is a linear representation p: H + GL(V) with po = (TLL(V)
Definition A.12 (Tensor product of finite dimensional vector spaces). Given two vector spaces U with basis fu and V with basis m their tensor product U & V a vector space with one basis vector for each pair (u;, v5) of one basis vector from U and one from V. The basis vectors U 60 are typically denoted {u, 1
Definition A.13 (Tensor product of vectors). The tensor product of an element == E; 0,4; in U and an element u = Σ, B,u, in V the element & En, a,,8, &y of U & V. This defines a bilinear map (u, u) u u.
16

---

Published in Transactions on Machine Learning Research (12/2023)
Definition A.14 (Tensor power). The tensor power U is simply the tensor product of U with itself k times.
Definition A.15 (Ireedccability) A linear representation p: G + GL(V) is called irreducible there no subspace $ [{0), V} which is invariant under all p(g), 9 E
Omitted proofs
In this section, we collect some further proofs we left out in the main text.
B.1 Induced projected representations
We begin, for completeness, by remarking and proving small statement that use implicitly throughout the entire article.
Proposition B.1. Let G be group, V and W vector spaces, pv and pw linear representations of G V and W, respectively, and and pw their corresponding projected representations. Then, the induced projective represenlation p on Hom(V, W)
p()) = pw (g) A (v)
is the projection of the corresponding induced linear projection
((g) (w( A (1)
Proof. What we need to prove is that if M is equivalent to (g) and N is equivalent to pv(g), the linear map
RMN Hom(V, W) + Hom(V, W), AH MAN-1
is equivalent to p(g). However, the stated equivalences mean that M Xpw (g) and N upv (g) for some non-zero μ1 A. This in turn shows that
RMN(A) (QV())A((vv(o))== -
T.e., that RANN equivalent (g) The claim has been proven.
B.2 Theorem
Proof of Theorem
We will prove that
(Po())) = E(g)Px(g)
for E G, E V for all k. This in particular means (P) (x( as elements in P(Ve), which is to proven.
We procoed with induction starting with & We have for g E G and e E G arbitrary
5 45)) = (())7
(P)(g)))) = co (2) (g) ((Po(()u))
where we in the final step used our assumption on
To prove the induction step k + 1, let us first concentrate on each expression A in 7 Due to the induction assumption, we have (Po(g))) = ((g))x()){{(g) for g G, d G Consequently (o = PT = (1)x) ( (0)p++ A
(x)))
Summing over all 7, G* with 7 = e yiclds a feature that transforms the claimed way. This is not changed through an application of the equivariant o since it by assumption commutes with =(g). The claim has been proven.
17

---

Published in Transactions on Machine Learning Research (12/2023)
B.3
Lemma 2.14
Proof of Lemma 2.14 Equivariance of the map A means that for all g E G and u E P(V), we have Pr Av Apo(g)u. Since all po( are invertible, this is equivalent to
( Av for all VEV.
Now; the above equality is not an equality of elements in W, but rather of elements in P(W). That is, it says that all vectors V are solutions of the generalized eigenvalue problem #i (g) Apo(g) AAe. The aim is to show that this implies that Pi (g) Ap)
XA for some À E F. To show this, proceed in two steps.
Claim Let M E GL(V). If every vector v E V is an eigenvector of M, M is multiple of the identity.
Proof Suppose not. Then, since all vectors are cigenvoctors, there exists F w + O and À + μ with Mv- λu, Mw - pe. Now, again since all vectors are eigenvectors of M, there must exist a third scalar o with Mu + wn o(v + w). Now,
o(v + w) = M(v + = MU Mw = Xe + wm = (o - ---- = - (o-o)u.
Now, the final equation can only be true - = = 0, i.e., X = == - which is a contradiction.
Claim Let E.Fe e Hom(V, W). If every v E is a solution of the generalized eigenvalue problem Ev AFo, E multiple of F.
Proof First, E restricted ker F must be the zero map, since for all ker F, Ev XFu = x 0 0. Secondly, Fº; ker F+ - ran F is an isomorphism, so that the generalized eigenvalue problem of Ev XFe on ker F is equivalent to the cigenvalue problem of E(F9)-1 ran F + ran F. By the previous claim, E(F")-1 Aid for some À F, i..,, E = XF on ker F+, Since both E and F additionally are equal to the zero map on ker F, the relation XF remains true also there. The claim has been proven.
B.4 Proposition 2.16
Proof of Proposition 2.16 Ad (i) If V is finiteddimensional and H is compact, it is well known that we (by modifying the inner product on V) may assume that p is unitary-foo for convenience, a proof is given in Section B.S This means that |o| = |(a) for all h E H. now v E U is non trivial, we have 0 * |o| = p h = ===(h)u], Le., [s(h)] = for all H.
Ad (ti): Let E H", U k, H be arbitrary. Then
Akk = ===
-1x x.
Since X is a group isomorphism, this relation extends to p({)z = r for all é (HI H. This proves the first part.
To prove the second claim, we will show that the operators (h)) H restricted to UHH are commuting operators Uни UHR- Since we are in the case IF = C, and we WLOG can assume that they are unitary, this means that they are simultancously diagonalizable, i.c.,
p())., A,(h)a,
for some basis x; UhH and 'eigenvalue functions' X; :::H + C. With the exact same argument as in the proof of the main theorem, we show that A; E H² Hence, the x; are elements of subspaces U which shows that UHH S span(Us, , H*). Since the other inclusion holds in general by what we just proved, the claim follows.
So let prove that the p(h) are commuting operators UnH - Unn. Both of this follows from the following equality:
(())p(h) = p(kh) D JJ)
18

---

Published in Transactions on Machine Learning Research (12/2023)
To show that p(h) are operators Uhи - UHH, may argue that UHH, and & E (H, H) and h e H are arbitrary, it follows
DEUnn (h)r
which means that p()) UHH- To prove that the maps commute Uhh is even casier: for x E Unn and k, h E H arbitrary, we immediately deduce
(())(()) -())p()))) h "h''))=== (())
The proof is finished.
B.5 Uragu for the standard action on (F")ek
Proposition B.2. Let p be the standard representation of Sn on (F")ok, For 2 k 22 U is nontrivial only for The bound is tight.
Proof. Let. T (F*))k be element of U and let I (in; ... -i--)) be arbitrary k-index. Since 2 k 2, there are at least two elements (i.j) & (( 1)). Consider the transposition 7 of and j. Then, r(i))= iz, L E [k], and sgn(r) -1 Consequently,
Tj = (T) = (P(T)T), = sgn(r)T -T2,
since T was in U Consequently, Tr Since was arbitrary, T must be trivial, and first part of the claim follows.
To show that the bound is tight, it is enough to construct non-zero element in (F++1y8k n Usagn, Let to = (0, 1, 2, ,k) E [k + 1]k. For each multi-index I, there are either multiple indices in 1, or there exists a unique permutation my Sa+1 so that I = ग, (4o). Due to the uniqueness, is not hard to show that - = म. oa, a E Sn. Let us now define a tensor through
if I contains multiple indices sgn(TT) clsc.
We claim that S is the sought element. To show this, let [k + 11kt and S, be arbitrary. We have (ola))), So-1(1)
Now, if contains multiple indices, o (I) must also, so S = 0, and (p(o)S)) = for such indices. I does not contain multiple indices, we deduce
S 2() sgn(T, 1n)) sgn(T) o sgu (s)) sgn(o)S,.
Since I was arbitrary, we have shown that p(o)S = sgn which was the claim.
Remark B.3. In an earlier version of the manuscript, we proved the above proposition in a different, arguably more clumsy, manner. Since it provides nice connection to the results (Maron et al. 2019). we include a sketch of that argument also here.
First, we realise that S being trivial is the same as to say that (Projs.] p is equivalent to the linear invariance problem [Lins.] The solutions of the latter are surely included in the formor (by c.g. Thcorem 2. 2.15 However, Proposition ii) also tells us that the solutions of (Projs.] are included in the solution space of
Linis..s.. The commutator {Sn, is the alternating group An, L.e., the group of permutations of sign ssee Section B.gh. Now, the linear invariance problem (in was treated in (Maron shown that for n 2 + 2, the set of solutions of Lin is exactly equal to the one of (Ling. (in fact, the proof that builds on the same idea as our proot of Proposition B.2 Hence, we conclude
2019 was
Sol. of Ling C Sol. of Projs.
Sol. of Lin
Sol. of [in
Sol. of Lin
19

---

Published in Transactions on Machine Learning Research (12/2023)
U ++ +
U
U
U
Figure 6: The bases for the spaces U for the action ZZ on the space C of 3 x filters.
B.6 The Vierergroup
Let us begin by describing the character group of Σ3. Eg is an abelian group generated by two elements, (1, and (0. 1). Any group homomorphism hence uniquely determined by its values <(11, 0)) and c((0, ()).
Since (1,0) + (1,0) (0, (0, 1), the values must be square rools of one, i.e., either +1 -1. Hence, Zn contains four elements c++, c+- c- E--, defined through the following table:
z
(0,0)
(1,0)
(0, 1)
(1,1)
++
1
1
1
1
6+-
1
1
-1
-1
8 --
-1
-1
E--
The appearance of the spaces e will depend on P. They are however simple to compute given vector V and an E (Z])", the element
= Σ
p(h)e
hE2
is u
pkk)o2 Σ a(h)) p(khju kEzz
[kh = Σ s(k- p(E)) e(k)ve.
FE
Conscquently, given any basis of V, we may construct a spanning set of U via calculating the above vector for each basis vector, and subsequently extract basis. Applying this strategy to the example of the Vierergroup acting through flipping on the spaces of 3 x 3 convolutional filters yields the bases depicted in Figure
B.7 A (not very useful) canonical lifting procedure
A natural question is whether every projective representation can be lifted to a linear one. In fact, this can done, if one allows the group H to depend on
Definition B.4. Let G PGL((V be a projective representation. We define
H, {(g, A) p(g) S G GL(V).
2This strategy generally applicable. Tt will quite inefficient if the group has many elements, though.
20

---

Published in Transactions on Machine Learning Research (12/2023)
Proposition B.5. Hp is subgroup of G GL(V), and covering group of G. The covering map is given by
p(g. A) g.
Proof. The subgroup property follows from the fact that p is projective representation if (), (h, B) is in Hp it per definition means that A E p(g) and B E p(h). However, then
AB E (()) p(gh)
L.c. A) (h, B) = AB) E Hp.
That is continuous, surjective and maps a neighbourhood of the identity (e, id) E H, homeomorphically to a neighbourhood of e in G is clear. Likewise clear is that &p is a group homomorphism:
p((g, A) B)) = (((hh, AB)) = gh = A)p(,)))g. ().gg))) B).
We can now lift p to a representation " on Hp as follows:
p: Hp + GL(V), + A.
Indeed,
Fg. A) A A p(g) = plppl A)),
which is the same as saying that po o0 ITL(Y)) 7
It now however important note that although this construction is theoretically pleasing, using it in conjunction with Proposition 2.15 does not give much additional insight. Said proposition states that solves U for some H However, being in U
[Projc if and only if it is the equivalence class of some
means that
A))a = e(g, A)æ Ax - -
(11)
for every E G and A E p(g) that is, for every in the equivalence class of g, Ar is equal to æ times some constant that may depend on both A and g. which is just a reformulation of (Proja)
For more information about this canonical lift we refer to the textbook (Kirillov 2012) and the historical exposition (Hirai et al. 2013). The cases where becomes useful is when H, known group with known representations and in particular when H, H is independent of p. This is the case for instance for the lift. of representations of SO(n) to SU(n).
B.8 Group theoretic facts
Here we, out of convenience for the reader, present proofs of some well-known group thcoretical facts. Lemma B.6. Let Hl be compuct group and ọ representation of H on a frrite-dimensionde space V. Then there cxist an scalar product on V with respect to which all p(h), h H, are unilary-
Proof. Since H is compact, there exists a unique normalized Haar measure i on H. That is, Jr is measure with the property that u(h-1a A) w(A) for all h e H and Borel measureable A C H. Given any scalar product () on V, we may now define
(0, w) = (()))),)
This is a well-defined expression due V being fnite-dimensional and H compact (and hence, h + (phh)e, p(h)o) is continuous, bounded function). It is easy to show that it is sesqui-linear. To show that it
21

---

Published in Transactions on Machine Learning Research (12/2023)
is definite, note that if 0== 2) (p(h),, p(h)o) is continuous function in h zero almost everywhere, i.e., zero. This means that (P(h)u, ((h)e) = 0 for all h, which immediately implies that e is zero.
We now claim that any (kk) is unitary with respect to this inner product. We have
(()))) [ ( (())) = ((
- J, (T(M),, (ol'),Ple (()d)))u(()) w)
= [ - hk]
where used the invariance of the Haar measure in the substitution step. Lemma B.7. S% (1,sgn):
Proof- Let E S be arbitrary. Since Sn is generated by transpositions -= (ij),, uniquely determined by its values of them. Since 72 = 1, must have s(r)² =1, meaning that cach value e(7) is either +1 or -1. If can prove that they must all have the same value, are done: if e maps the transpositions to +1, 8 is the character 1, and if it maps them to -1, it is sgn.
So let r and 7' be to transpositions. We distinguish two cases.
Case 1: r and share an element, i.e., r (if) and = (ikk). Since (ik) (z)))) we must however have <(T) = (( -((j)²² E(7').
Case 2: 5 and 7' do not share element, i.e., (ij) and r' = (kel). Since (kl) (2)))))((())((())1)()) we have <('') = 8 (e))) =(T).
Lemma B.8. (Sn, } An.
Proof. This proof can be found in e.g. (re 1951
That {S,, S S An follows from the fact that all commutators have signature 1:
( - =
=
To prove the converse, let first note that the cases 1, are trivial in both those cases, A, {id], but Sn is also abelian, so that {Sn, Sn} {id}. For n 2 3, we use the well-known fact that An is generated by 3-cycles (Lee 2018 Cor. 6.3) (ijk). If we show that they commutators, we are hence done. But
(ijk) (4)(())))(()
SO that the claim follows.
Lemma B.9. SU( is perfect.
Proof. For this proof, we are going to use that elements in SU(2) can be thought of as unit quaternions q= (a, v), where a E R is the real part the quaternion, and v E R its vector part. being unit norm means that a² + |u|2 The group action given by the quaternion multiplication, i.e.,
(a, (B, w) (a. v. w. ou + Bw v w),
where and x are the standard dot and cross products of R3, We need to show that all unit quaternions can be generated by commutators. We will do this in three steps.
Step 1. Each unit norm quaternion has a square rool r, i.e., an r with و2 = q. This can be seen through direct calculation: Each unit norm quaternion can be written as (cos(20), sin() u) for e [0, x]] and a unit norm we then define r (cos( sin (8)u), we have
2 (cos2 (() sin sin? (0), cos(() sin( + cin(())in(())) sin(O) u sin 2 (0) x u)
(cos(20), sin(20)ul (20)u) g.
22

---

Published in Transactions on Machine Learning Research (12/2023)
Step 2. Write Bu, let be unit norm ector orthogonal to and If we define (0,
we have
ST g- (0, () (0, u) = (0+ Bu u, av - BB u) (0, B0) uv)
(alu/? Bw v, 000 x Bw v) Bu) =;
T.e., r = 88
Step 3. We now combine the above steps
= -- rsr 1ss- {r.s}.
Hence, q is commutator, and the claim has been proven.
Corollary B.10. SO(3) is perfect.
Proof. The perfectness follows from the perfectness of its cover SU(2). If p SU(2) - SO(3) is a covering map, write any E SO(3) as p(r) for an r E SU(2). We just proved that there exists 8, & E SU(() with r= (s,t). But this implies
R= = ((fs,t)) (p()) ((()).
Lemma B.11. Z, is isomorphic to the set of n-th roots of unity.
Proof. Zn is abelian group, generated by the element 1. Consequently, any group homomorphism is uniquely determined by the value e(1). Since 1 = e(0) e +[ = <(1)", e(1) must be rool of unily.
n times
Hence, all clements of Z is of the form (k) = w with w a root of unity. Since that surcly defines a group homomorphism, Z = w =}. As for the isomorphy, we simply need to remark that Eutur = Eww by direct calculation.
wn
Lemma B.12. G is perfect, G only contains the trivial character 1.
Proof. Let h = (h, be a commulator and e E G". Then
e(k) =(heh- =(N))(()(())=>(((()
Since is group homomorphism, this shows that is equal to 1 the entire commutator subgroup {G, G), and thus, since G is perfect, on the entirety of G.
In fact, to some extent, the converse statement of the last lemma holds. We include the proof for completeness. Lemma B.13. Let F C. If is a compact group with G = {1}, G perfect.
Proof. {G, G} is a normal subgroup of G, meaning that if h E G and g E GG, the conjugation high is also (fh, g) = hgh-'g- is surely in {G, G}, and therefore also hgh {h. gh.).. This implies that we can define the quotient group A A= G/ GGG), G}, which defined via identifying elements h. & E for which hk-le (G, GJ. The group operation inherited from G: For two equivalence classes [h] and [k], its product is defined [k] [hk]. Also, an e E G defines an element of A*. Indeed, any G must by the proof of Lemma. B.12 be identically equal to 1 on GG. This has the consequence that if h,k are in the same equivalence class,
))(kk <(h) e(hk ((k) = e(hk e(hk-))e(k) = =(k),
that is well-defined on A.
23

---

Published in Transactions on Machine Learning Research (12/2023)
The group A is called the Abelianization of G. The naming stems from the fact that it is an abelian group for any elements [b), [b:] of A, (b), [+]] = [h,, k3] [e], due to the definition of A. Since G is compact, A surely also is.
Since G only contains the trivial element, A also does. Since A is an abelian compact group, we may now apply the Pontryagin duality theorem (Rudin) 1962, p.28), which states that A is isomorphic the bidual A = [1}- = {e}. Hence, A only contains the unity element, which in turn means that ke- (G,G) for every & E G. That is, G is perfect.
C Details of the Spinor Field Networks experiments
As advertised, we begin with a review of spinors and the representation theory of SU(2). The reader is referred to e.g. (Biedenharn & Louck 1984; Hall 2013 Zee 2016h for in depth treatments. SU) has one so-called irreducible representation (irrep) for each dimension TI O. Rather than labelling them by n, they are lypically labelled by = 0, 1/2, 1,3/ 3/....., where the l'th irrep (or rather the space it is acting on) has dimension 28 + 1. == 0 corresponds to scalar values that don't 't transform under SU(2) (i.e., SU(2) acts as the identity on them), f= corresponds to vector values on which SU(2) acts as 3D rotation matricos and 1/2 corresponds to spinor values on which SU(2) acts as the ordinary 2 x complex matrix representa aapeentation of SU(2). For integer f, the SU(2)-irrep can be taken consist of only real valued matrices and hence can be interpreted as acting on g2(+1 These irreps are also irreps of SO(3). For non-integer l the other hand, it is not possible to let the representation consist of only real valued matrices. Thus these irreps act on c20+1 Purthermore, these irreps do not correspond to linear representations of SO(3), but only to projective representations of 50(3).
Let us denote the irreps of SU(2) by pe. E = 0, 1,/,,,,,.33 and the spaces that they act on by Ve ~ C (or RR2+1 for integer f). All other finite dimensional representations of SU()) can be decomposed into these irreps, meaning that if (U()) acts linearly on CmM, we can decompose the space
Ve LEL
for some specific multisct of indices such that Eier, (26 1) = m. One particular decomposition is the decomposition of the tensor product of two Ve's. Given 2 l we have that
V& ② Vor ~ Ve-r @ VI-++1 ① @ V @ Ve+e-
(12)
In particular, there exists a change of basis matrix that transforms canonical coordinates on V. & Vy into coordinates which can be split into blocks, cach representing coordinates for one V/. We denote this matrix from the left. hand side of (12) to the right hand side by C. C consists of so-called Clebsch-Gordan coefficients, which are known and hence it is feasible to perform this decomposition.
The basic idea for our -Uiuueqquvvariian neural network architecture is now to have layers that combine features and filters using the tensor product. This is a standard approach to constructing equuiaaquv neural networks (Thomas et al. 2018 Geiger & Smidt 2022 Kondor et al. 2018) which we here generalize slightly to SU)2). Say that we have a feature f that transforms according to Pe and a filter ve that transforms according to pe. Their tensor product will transform according to pe @ pe. Note that 12 means that C(f )) transforms according to
Clf & v) SU(2),
Cipe & pf) = (pe-r + P0- @ ... @ P @ ((4e) 8 8
This mcans that C(f & v is a concatenation of quantities that transform according to known irreps. Since the lensor product is the most general bilinear map, a good way to look at it is that C defines all possible multiplications of f and & that are SU(2) equivariant. If we are only interested in a particular output type, say the part of C(f & v that transforms according to Pe-e', then we can simply let Ō consist of the corresponding rows of C, i.c., the first 2(2- l') + 1 rows, and the quantity Č(f & v)) will be the output with correct behaviour. C typically contains many zeros and we don't actually have to compute the complete
24

---

Published in Transactions on Machine Learning Research (12/2023)
f @ & if we are only interested in some particular output irreps, but can instead only compute those values that will be multiplied by non-zero values of C. Tf our network layers consist of tensor products like this, it is easy to keep track of how the features in the network transform under SU(2), and hence ensure that the output the network transforms correctly. In our experiments we have outputs that transform according to P1/2 and so build networks that guaranlee such output.
Example C.1. Let us walk through the example of tensoring a vector feature f by vector filter v. Both the feature and filter transform according to Pi which we can take to consist of real-valued 3D rotation matrices. This means that according to (12). the output should be decomposable into Vo @ V @ V2, ie., a scalar, a vector and a "higher order" 5-dim. feature. The reader already knows how to combine two vectors to form a scalar and veclor, we use the scalar product and the cross product! Indeed, the tensor product has the following form:
f2, & (Vi, v2, vt)" = (frbi, fiv2, fiva, S2 S212, S24/ Jahi, S8 faaa))"
The first row C' selects those values that yield the scalar product of f and sh:
C = 0000001000)
Ci 2 f.
The second to fourth rows of C' select those values that yield the cross product of f and th:
0
0
0
0
0
1
0
-1
0
C24
0
0 -1
0
0
0
0
0
C2)( = Φb
0
1 0
0
0
0
0
The last rows of O' select values that yield a 5-dim. product of f and ib transforming according to the irrep P2, we don't write it out here as it not too illuminating.
C.1 Tensor spherical harmonics filters
We defined the layers in our Spinor Field Networks like
f
Σ (1b2, 1
(13)
A tensor spherical harmonic (see (Biedenharn & Louck 1984 is the tensor product of a spherical harmonic and some other quantity that transforms under SU) according to some representation p. Then [3 could become for instance
f = (vbl, x;) @ 8,) 8 S3-
(14)
j
Technically our filters in 13 are tensor spherical harmonics where the spherical harmonic tensored by sj is the trivial y
C.2 Training/implemmntttioon details
The implementation is inspired by the e3nn package (Geiger & Smidt, 2022), but differs in some crucial aspects (apart from the obvious fact of using spinors).
A tensor product layer in our framework works as follows. We define the number of input, filter and output scalars, spinors and vectors. For each output requested, certain input types can be used. For instance, to produce an output scalar, we can multiply
(i) an input scalar and filter scalar,
(ii) an input spinor and a filter spinor, or (iii) an input vector and filter vector.
25

---

Published in Transactions on Machine Learning Research (12/2023)
Layer 1
Input Spinors
Filter
Scalars
Vectors
Scalars
Spinors
Vectors
Spinors scalars Spinors as features Spinors os filters Spinors squared as vector features Spinors squared as vector filters
1
0
1
1
0
1
0
0
1
1
1
0
3
0
1
1
0
3
Layer 2
Input
Filter Spinors
Scalars
Spinors
Vectors
Scalars
Vectors
Spinors scalars Spinors as features Spinors as filters Spinors squared as vector features Spinors squared as vector filters
32
8
0
32
12
12
0
32
4
4
1
1
32
8
0
1
8
0
3
Layer 3
Input Spinors
Filter Spinors
Scalars
Vectors
Scalars
Vectors
Spinors as scalars Spinors as features Spinors as fillers Spinors squared as vector features Spinors squared as vector filters
32
8
0
1
0
12
0
0
1
32
4
1
1
32
8
1
0
8
1
0
Table 2: Number of scalars, spinors and vectors in each layer of the five network types. The number of the number of imputs to the next layer. The final layer always outputs
outputs of each type is the one spinor, except for the Spinors scalars net which outputs four scalars.
So for each output scalar requested, we compute a weighted sum of all input scalars and multiply it by a weighted sum of all filter scalars, and similarly for spinors and vectors. These weighted sums contain the learnable parameters the net. We do not use learnt radial functions as in Tensor Field Networks, that did not to be necessary for our data.
When mapping from spinors to scalars/ /vecoor, we get complex valued scalars and vectors. These are split into real and imaginary parts, hence if 10 output scalars are requested we only compute 5 complex output scalars yielding 10 real output scalars. Note that the operation of taking real and imaginary parts of a vector is rotation cquivariant as we work in the basis where rotation matrices are real valued.
The loss used is Lz-loss on the regressed spinor, but accounting for arbitrary sign due to projective ambiguity, i.e.
loss( (Srred 8) min s|l. || + s|])-
The networks contain between 2500 and 3000 parameters and are small enough train on CPU in tens of seconds per net. All nets were trained using the Adam optimiser with default PyTorch settings, for 300 epochs with learning rate 10-2
The number of features of each type in the layers and filters of the Spinor Field Networks is summarised in Table
Details and further results for ViererNet experiments
We begin by providing some details about the model used in the experiments in the main paper.
The layers of the projectively equivariant ViererNet used in the main paper has, in succession, 32, 32, 32 and 11 intermediate RR... valud feature tuples, each of size 4 (Zz)*. These subsequently wverage-pooled to
26

---

Published in Transactions on Machine Learning Research (12/2023)
M
M
M
11
11
11
2+++ ++
2
+++
2C+
+
g
R
u
vC-
v
IF
P
R
uF+-
uf
I
2
c
IR
Figure 7: The ViererNet architechtures. V is the input space. For the MNIST experiment, M 32, and for CIFAR, M 64. Best viewed in color.
11 4-tuples. To transform these to scalars, use 11 selector vectors, and send it through first a batch-norm and then a softmax layer to end up with final output distribution p E R Before sending the data to the first layer, we normalize it to have zero mean. The base-line model a three-layer standard CNN, with 32, 32, 32 and 11 -=*"-valued intermediate features, which after average pooling simply feeds their 11 output features through first a batch-norm and then a softmax layer. Both models use tanh non-lincarities. The two models have different memory footprints, due to the ViererNet having to handle 4 features in each layer, but each input-output channel pair can be described the same number of parameters (due to Proposition 2.16 ii)). The selector vectors do result in 44 additional ViererNet paramcters, but this is miniscule compared to the about 30K in total. The architecture is schematically presented in Figure i
We train for 100 epochs using the Adam algorithm, with the learning rate parameter set to 10-4.. 4
D.1 Additional experiments on CIFAR10
To complement the MNIST experiments in the paper, we tested VicrerNct to classify similarly modified version of the CIFAR1O dataset.
Data We modily the CIFAR1O dataset in two ways. First, we modily the dataset exactly the way MNIST is modified in the main paper: Le., we add a class 'not an image' which we put (or not, depending on the class) images after they are flipped horizontally vertically: The images in the classes 'airplanc', 'automobile' and 'bird' are never put in the 'not image' the 'cat' and 'deer' and 'dog' images are put there after horizontal flip, 'frog' and 'horse' after a vertical flip and 'ship' and 'truck' after both types of flip. We refer to this version of the dataset as the 'both flip' dataset.
The above modification has a problem, and that is that since flipping a CIFAR10-image vertically, in contrast to the MNIST images, most often results in a plausible image. The horizontal flips however results in objects that are upside down, and hence are different from the 'normal' CIFAR images. For this reason, we also consider different type of modification of the dataset. In this version, we still perform flips with the same probabilities as before, but choose to not change the labels after vertical axis flips at all. We refer to this version of the data set as the 'single flip' datasots.
Models We use the same ViererNet and baseline architectures in the main paper, with the only difference that the number of intermediate features are 64, 64, 64, 11, instead of 32, 32, 32, 11.
Results Using the same training algorithm and learning rate (e.g. Adam with ir 10-4) as for the MNIST experiments, we train each model each dataset for 100 epochs. The experiments repeated 30 times. In Figure S the evolutions of the median training and test losses are depicted, along with errorbars encapsulating 80% of the experiments. In comparison to the experiments in the main paper, the models are performing more similar. When comparing the models at the epochs with best median performance, the baseline outperforms the ViererNet slightly on both datasets 2atasets-51.0% vs. 49.9% on the both flip dataset and 59.0% vs. 57.0% on the single flip datasct. The performance difference is however not as significant as in the main paper- the p-values for the baseline outperforming ViererNet is only 0.26 for the both flip dataset, and .11 for the single flip dataset.
27

---

Published in Transactions on Machine Learning Research (12/2023)
(double fllp) (train)
cifer (double flipi
0.55
0.50
0.50
0.45
0.90
0.40
0.35
0.30
0.30
Easeline
0.25
baseline
werer
viener
0.20
20
60
100
20
40
00
200
citar (single flip) (train)
citar (single flii) (test)
0.65
0.00
0.55
0.40
0.35
beseline
0.25
0.2
100
Figure 8: Results for the CIFARIO datasets. Top left: training accuracy on the 'double flip' version. Top right: test accuracy for the 'double flip' version. Bottom left: training accuracy on the 'single flip' version. Bottom right: test accuracy on the 'single flip' version. The crrorbars depict confidence intervals of 80%. Generally, the baseline performs slightly better al the end of the training, but well within the margin of error. The ViererNet is quicker to generalize than the baseline.
Dataset
MNIST
CIFAR (double flip)
CIFARIO (single flip)
Max
50
100
50
50
75
100
Baseline 80.3%
86.6%
89.4%
90.2%
43.2%
48%
49.4%
51.0%
47.3%
53.7%
57.2%
59.0%
ViererNet 89.6%
91.7%
92.5%
92.8%
15.1%
48.1%
19.1%
19.9%
52.1%
55.2%
56.5%
57.0%
Table 3: Median accuracies (over 30 runs) in all experiments for different upper iterations bounds. Statistieally significantly (p < .05) better performing models for each dataset and upper iteration limits are marked in bold font. (In fact, using the higher p-value 0.1 would not change the appearance of the table).
Performance vs. number of iterations Figures and both suggest that the ViererNet can generalize to the test data faster, since it outperforms the baseline in early iterations. To test this quantitatively, we perform the same statistical tests as above, but choose the best performing epoch among the 25, 50 and 75 first iterations instead out of all 100 iterations. We report the median accuracies for all experiments in Table 3 Statistically better figures are highlighted in bold. We see that the ViererNet outperforms the baseline in the MNIST experiments by wider margins, and with high statistical confidence, earlier in the training, Looking the median values in Table the trend that the ViererNet is quicker to train apparently persists to some extent for the CIFAR experiments. However, closer looks reveals that we can only with high statistical confidence say that the ViererNet is better than the baseline after 25 iterations on the single flip dataset.
Discussion Although the CIFAR experiments are in general not statistically conclusive, we can conclude that the ViererNet in comparison to the MNIST experiments the main paper perform worse on the CIFAR set. We believe the main reason behind this simply that the property of projective Zz-equivariance is less well suited for these tasks. For the both flips version of the dataset, as discussed above, flipping vertically
28

---

Published in Transactions on Machine Learning Research (12/2023)
will most often lead to new image which is plausible to come from the same CIFAR-class, and it will be less helpful to apply the strategy of either changing maintaining the class when flipping the images of the class. As for the single flip version of the dataset, the same argument does not apply. However, in this case, it is ultimately only a subgroup of the Vierergroup that changing the images in the datasct. Hence, a VicrerNet is not the best suited architecture Zg-equivariaat would instead be.
A more thorough empirical comparison of the effectiveness of projectively equivariant architectures for classification tasks would be very interesting, but out of scope for this article. We leave it for future work.
D.2 Training details
We used cross-entropy-loss in our experiments, account for the different sizes of the classes: the weight vector vectors chosen
WMNIST [1,1, 1.5, 1.5, 1.5, 3, 1], UCIFAR [L, 1, 1.5, 1,,,,,,,,,,,,,,
for the MNIST and CIFARIO experiments, respectively. batch size of 32 was used for all experiments. The models were trained, on single A40 GPUs, in parallell
high performance computing cluster. The MNIST experiments presented in the main paper used in total about 110 GPU hours, whereas the CIFARI0-=eperrmentts presented in the appendix used about 170 GPU hours each.
D.3 Implementational details
Since all of the basis elements, depicted in Figure 0 sparse, implement the ViererNet layers PyTorr by saving nine parameters per in-layer-out-layer pair, which we turn into four 3 3 filters through multiplying the R vector with a sparse matrix. These are then used in padded standard PyTorch convolution layers, and combined in the manner described in Section 3
29


Reviews:
[{"review_id": "lsQUtGWzaV", "reviewer": "Reviewer_SXuJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. Some theory is given to relate projectively equivariant representation with linearly equivariant representations and this characterization later proves useful for the construction of toy projectively equivariant neural networks for a synthetic symmetry-oriented MNIST task.\n\n2. The projective equivariance-oriented approach seems to clearly help neural networks better capture symmetry in the synthetic spinor regression example of Section 4. \n\n## Weaknesses\n\n1. Although the paper does study projective equivariance, which has not yet been explicitly explored in the equivariance literature, the motivating examples do not provide a particularly convincing case for the study of this notion. Namely, the synthetic class-dependent symmetry example with MNIST was very much a toy construction and even then, the model did not outperform a CNN baseline by a very significant amount. Moreover, the paper itself admits that on a synthetic symmetry-oriented CIFAR-10 test task the CNN baseline performs better than the introduced method. Either a more convincing synthetic dataset task or ideally a real-world dataset task (perhaps from physics) as a motivating example for the study of projective equivariance would help bolster the paper significantly.\n\n2. The theoretical results are fairly straightforwardly obtained as is and are not as pertinent as desirable. Notably, they do not hold for the very spinor example given in the paper (Section 4), since the filters and features lie in an infinite dimensional vector space. If not an additional theoretical result, I would have liked to see further discussion of how one might obtain an extension of the extant theory to encompass this highly pertinent case. Additionally, I feel that this paper unfairly differs the design of projectively but not linearly equivariant nonlinearities to future work; such nonlinearities would be highly relevant for constructing more interesting canonical projectively equivariant neural networks and I would have liked to see some attempt at addressing this topic in this paper.\n\n## Verdict\n\nThe paper presents an investigation of projective equivariance in the context of equivariant neural networks. The authors present some theory regarding the relation between projectively and linearly equivariant representations and propose a way to construct projectively equivariant neural networks (motivated by the theory). Empirically, projective equivariant neural networks are showcased first for a toy symmetry-oriented MNIST task and further for a toy spinor regression task. Although the motivating examples given are highly synthetic and the theory does not encompass the spinor regression example, I believe the claims made in the submission are well supported and the findings of the paper would be interesting to some portion of TMLR's audience. As such, both criteria for acceptance to TMLR are satisfied; ergo, I recommend an accept rating for the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors study projective equivariance in the context of neural networks. The authors present some theory regarding the relation between projectively and linearly equivariant representations, culminating in Theorem 2.13 and the results of Section 2.2. They also propose a way to construct projectively equivariant neural networks, which amounts to building standard equivariant neural networks where the linear group representations acting on each intermediate feature space are lifts of projective group representations. Projective equivariant neural networks are then showcased first for a toy MNIST task and further for a toy spinor regression task where point clouds were equipped with spinor features and the goal was to regress to the spinor targets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This is not crucial for acceptance, but I would like to know if the authors can add some more discussion with respect to my concerns in the weaknesses section above. Notably, I would like to know if they can either introduce or further discuss a better toy motivating example or potentially a real-world application (e.g. from physics) of the introduced networks. Also, a discussion of how the theory may potentially be extended to encompass the spinor example from Section 4 would help tie the paper together. \n\nIn terms of writing, I would like to request some changes that would strengthen the presentation of the work. First, I note that the paper uses in-text citations in place of parenthetical citations where they are needed. For example in the first paragraph we see \"...to computer vision Krizhevsky et al. (2012)\" instead of \"...to computer vision (Krizhevsky et al., 2012)\"; this should be fixed. Many citation packages have a \"\\citep\" option for parenthetical citations that contrasts with \"\\citet\" for in-text citations. I would like the authors to go through and perform the needed citations corrections. Additionally, I would like to suggest some minor writing corrections that will improve the writing via the following non-exhaustive list:\n\n- In Example 2.1: The authors write \"We illustrate in Figure 2.\" This should say \"We illustrate in Figure 1.\"\n- In the paragraph after Example 2.2: \"The benefit of projective equivariance is imminent\" should be \"The benefit of projective equivariance is eminent\"\n- Stylistic comment about how definitions are given: When first defining \"equivariance\" and \"projective equivariance\" it is good to underscore the group action. You can make your definitions more clear by changing e.g. \"We refer to $\\Phi$ satisfying (4) as being _projectively equivariant_\" to \"We refer to $\\Phi$ satisfying (4) as being _projectively equivariant_ to action by G\".\n- Stylistic comment: \"unit root\" on pages 6-7 refers to roots of unity, which is not standard phraseology. I would refer to them simply as roots of unity, e.g., change phrases like \"every unit root\" to \"every root of unity.\" (\"unit root\" typically has a different connotation in probability theory/statistics)\n- First paragraph of Section 3: \"Still, we can still use\" -> \"Still, we can use\"\n- Third-to-last paragraph of Section 3: \"procedure described in this Section\" -> \"procedure described in this section\"\n- Section 4 \"Models\": \"Indata are point clouds\" -> \"Our data are point clouds\"\n- Conclusion: \"we theoretically studied of the relation\" -> \"we theoretically studied the relation\"", "broader_impact_concerns": "The paper is of a fairly theoretical nature; as such, I have no ethical concerns and do not believe the paper requires a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ycQDEgZvTL", "reviewer": "Reviewer_2rmv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- This paper is technically solid, with all major theoretical results being substantiated by proofs and well-defined conditions. The authors have conducted a comprehensive exploration of all crucial aspects of projective equivariance in deep learning.\n- The experiments give simple yet interesting examples, showcasing scenarios where projective equivariance outperforms standard equivariant networks.\n\n**Weaknesses**\n- The paper's presentation could be more accessible; readers are required to grasp and remember numerous concepts and notations throughout the text. More repetitions in the context could help. To enhance the paper's readability and appeal, the authors could consider providing high-level intuition in the beginning, followed by the rigorous mathematical statements later on.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents the concept of projective equivariance to the geometric deep learning community, providing a thorough exploration of several crucial questions related to this topic:\n- How do projectively equivariant linear layers relate to their linearly equivariant counterparts, and under what conditions do these two types of layers coincide?\n- What are the necessary and sufficient conditions for a linear map to exhibit projective equivariance?\n- Can we establish a general framework for constructing projectively equivariant neural networks?\n- Are there simple examples that empirically validate the superiority of our methods over standard equivariant models?", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- While demonstrating through experiments is unnecessary for this theoretical work, can the authors provide non-toy applications where considering projective equivariance holds great potential and standard equivariance can get us into trouble? I think the class-dependent symmetry example is already very interesting, but can projective equivariance fully tackle the problem of partial equivariance [1][2]. (It would be super exciting if it can).\n- The current writing is not accessible enough to the machine-learning and application audience. I suggest (1) adding high-level intuition of projective equivariance in the introduction. (2) Even though a notation has been introduced before, do add some repetitions in the text to help reading.\n\n[1] Miao, Ning et al. “Learning Instance-Specific Augmentations by Capturing Local Invariances.” International Conference on Machine Learning (2022).\n[2] Romero, D. W., & Lohit, S. (2022). Learning partial equivariances from data. Advances in Neural Information Processing Systems, 35, 36466-36478.", "broader_impact_concerns": "I have no concern over the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L2OW1gen2g", "reviewer": "Reviewer_kggx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n**[S1]** For the most part, I found the paper to be well motivated and clearly written. The authors do a good job of motivating each concept they introduce and explaining the key results informally as well as formally.\n\n**[S2]** The theoretical results seem interesting and general. I liked how the authors also explained the consequences for individual example groups as well.\n\n**[S3]** The proposed architecture makes sense and corresponds nicely to the theoretical results.\n\n## Weaknesses\n\n**[W1]** The MNIST example seems badly motivated and doesn't really feel like it has anything to do with projective invariance. For this task, some classes are invariant to flips and others are not, and the authors note that this could be modeled as either flipping a sign of an activation when the class is flipped, or not flipping the sign; both of these behaviors are projectively equivariant because they differ by only a scalar. Thus, the authors consider using their projectively-equivariant architecture to model it. But assuming I'm understanding correctly, this seems pretty silly, because *every* nonzero scalar function is projectively equivariant; there's only one equivalence class of projectively equivariant scalar-output functions. So if all you need is projective equivariance, any nonzero function would work. And ultimately, the authors end up using a loss which is not well-behaved over projective equivalence classes: it gives different scores to outputs that differ by rescaling, so in a sense it feels like projective invariance is not the right equivalence relation for this task.\n\nI guess this result says something about the inductive biases of the proposed network, but I feel like it doesn't really serve as motivation for projective equivariance, and is sort of trivial from a projective equivariance perspective. If the goal is to handle class-dependent symmetries it seems like it would be easier to directly infer different invariances for each class using something like [Benton et al. (2020)](https://proceedings.neurips.cc/paper/2020/file/cc8090c4d2791cdd9cd2cb3c24296190-Paper.pdf)'s approach.\n\n**[W2]** More generally, it's hard for me to think of situations where a task would be well-suited to projectively invariant architectures like the one proposed here. The spinor fields example in section 4 seems a bit more realistic than the MNIST one, but it's a bit contrived as well.\n\n**[W3]** There were a few parts of the paper that I found hard to follow, and a few terms that did not seem properly defined before they were used. (See the requested changes section below.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the conditions under which neural network layers are *projectively equivariant*, in the sense that group actions on the input and group actions on the output are equivalent up to multiplication by a nonzero scalar.\n\nThe authors describe background of linear and projective equivariance, then prove a series of results about projectively equivariant linear maps. The main result is that projective equivariant linear maps can only lie in certain subspaces, each of which is defined in terms of (the equivalence classes of) linearly equivariant maps scaled by a character of the group. The authors present examples of groups where the only projectively equivariant maps are projections of linearly equivariant ones, and examples of groups where projective equivariance adds additional structure. They further build on this theory by constructing a neural net architecture that is projectively-equivariant by construction.\n\nThe authors then conduct a few experiments on synthetic datasets with their proposed architecture. The first is based on MNIST with horizontal and vertical flips, although the motivation of using projective invariance here is pretty weak (see weaknesses below). The second is a toy dataset of point clouds with \"spinor features\" which jointly transform; for this they find that their proposed architecture outperforms some baselines but not others.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Clarity improvements\nThere were a few parts of the paper I found difficult to follow, and I think the paper could be improved by clarifying them.\n\n- At the end of page 4, you say \"For such a relation to hold for generic A, ...\". What's a \"generic\" A in this context? Are you saying essentially \"some A are pathological and for those $\\varepsilon$ can be something else\"? In that case, why aren't those A worth considering?\n- In Remark 2.8, I don't understand what you mean by \"its image is endowed with $\\hat{\\rho}^\\epsilon$\"\n- Definition 2.9 and the sentence before it refer to a \"so-called covering group of G\" and a group covering $\\varphi$, but these are never defined except in the Appendix. Please make sure to define these terms or at least point to a definition before defining other terms that depend on them. (In particular, you never say that $\\varphi$ maps $H$ to $G$, that it's a homomorphism, or that it's surjective, except in the unreferenced appendix, and all of those seem necessary for understanding your later definitions and results).\n- I found the notation in equation $\\rho(g) A = \\rho_1(g) A \\rho_0^{-1}(g)$ to be difficult to understand. Am I understanding correctly that, on the left, $\\rho(g) A$ means $\\rho(g)[A]$, i.e. you're applying the function $\\rho(g)$ to the value $A$ (viewed as a vector), whereas on the right, $\\rho_1(g) A \\rho_0^{-1}(g)$ means $\\rho_1(g) \\circ A \\circ \\rho_0^{-1}(g)$, i.e. you're composing these three linear maps viewed as functions? I'd suggest being more explicit about the notation here, or adding a note clarifying that the implicit concatenation means something different in each case.\n- In Section 2.2., you never explain how $S_n$ acts on spaces of tensors (e.g. what the \"canonical action\" is). I assume this is permuting the tensor elements?\n- In Section 3, I don't see how the finitely-many-nontrivial-$U^\\varepsilon$ property follows from Proposition 2.14. Could you add a bit more explanation of this? (It's also not obvious to me why we can bound the dimensionality of $U_{HH}$ the way you have shown.)\n\n## Questions\n\nTheorem 2.13 is presented in terms of a covering group $H$ of $G$ and a lift $\\hat{\\rho}$, and before Definition 2.9 you indicate that you're assuming such a $\\hat{\\rho}$ exists. Does this mean that there may be some $\\rho$ that cannot be lifted to linear representations of any covering group? Or does every projective representation admit a lift, so it's always safe to assume you have one?\n\nIn Theorem 2.13, is it necessary to say \"$v$ is in the equivalence class of some $x \\in U^\\varepsilon$\"? It seems like if $v$ is in the same equivalence class as $x \\in U^\\varepsilon$, we'd also have $v \\in U^\\varepsilon$? Unless you are using $v$ to denote the equivalence class itself, but then shouldn't it be \"$v$ **is** the equivalence class of some $x \\in U^\\varepsilon$\", not \"is in\"?\n\n## Optional suggestions\n\nI wonder if the MNIST example could be replaced with a different toy task where the loss function actually respects the projective invariance equivalence classes. Having the loss depend on the exact scalar value seems to defeat the purpose of designing a projectively equivariant architecture.\n\nI was surprised that in the MNIST example you changed the label when flipping the digit 8 vertically and horizontally. Doesn't an 8 look the same when flipped horizontally or vertically? So it's label intuitively *should* be invariant to flips, and it doesn't make much sense to me to change the label to NaN when flipping it. (That might contribute to the relatively low accuracy of all models for this task.)\n\nFor Example 2.6., perhaps it would be worth adding a brief note pointing to Appendix C, since you discuss those concepts in more detail there? I also wonder if it would be helpful to show a visualization similar to your \"pinhole camera\" figure for SU(2), since it seems that this particular example is used in many times in the paper, and I still don't have an intuitive grasp of what the projective representations mean in this space.\n\nIn Figure 2, does it make sense to interpret $V_0$ as being $V_0^1$ in a sense? If so, you could move up the white box to line up with the others and make the figure look a bit more consistent.\n\nIn Section 3, I think you could give a bit more motivation for what the $|H^*|$ output features correspond to. At first I was confused why you wouldn't just take $v_K^1$ as the output feature, but perhaps the reason is that which output feature you use determines something about the set of functions you can express? Is that right?\n\n\n## Minor typographical note\n\nMany of your citations seem to use `\\citet` when `\\citep` would be more appropriate. For instance, in the first sentence, I think those citations should be entirely in parentheses.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers unanimously agreed that the claims of the submission are well supported, and that the paper can be of interest to part of the TMLR audience, since projective equivariance has not been discussed much in the literature. Therefore, the bar for acceptance at TMLR is met, and the decision is to accept this paper.
Decision: Accept (poster)