Learning Irreducible Representations of Noncommutative Lie Groups
Authors: Noah Shutty
Casimir Wierzynski
Venue:  (2021)
Keywords: equivariance
object tracking
equivariant neural networks
deep learning
point cloud
lie group
lie algebra
lorentz group
poincaré group

Abstract: Recent work has constructed neural networks that are equivariant to continuous symmetry groups such as 2D and 3D rotations. This is accomplished using explicit group representations to derive the equivariant kernels and nonlinearities. We present two contributions motivated by frontier applications of equivariance beyond rotations and translations. First, we relax the requirement for explicit Lie group representations, presenting a novel algorithm that finds irreducible representations of noncommutative Lie groups given only the structure constants of the associated Lie algebra. Second, we demonstrate that Lorentz-equivariance is a useful prior for object-tracking tasks and construct the first object-tracking model equivariant to the Poincaré group.

Under review as a conference paper at ICLR 2021
LEARNING IRREDUCIBLE REPRESENTATIONS OF NONCOMMUTATIVE LIE GROUPS
Anonymous authors Paper under double-bling review
ABSTRACT
Recent work has constructed neural networks that equivariant continuous symmetry groups such as 2D and 3D rotations. This is accomplished using explicit group representations to derive the equivariant kernels and nonlinearities. We present two contributions motivated by frontier applications of equivariance beyond rotations and translations. First, we relax the requirement for explicit Lie group representations, presenting novel algorithm that finds irreducible representations of noncommutative Lie groups given only the structure constants of the associated Lie algebra. Second, we demonstrate that Lorentz-equivariance is useful prior for object-tracking tasks and construct the first object- model equivariant the Poincaré group.
INTRODUCTION
Many tasks in machine learning exactly or approximately obey a continuous symmetry such as 2D) rotations. An MI. model is said be equivariant such symmetry if the model respects automatically (without training). Equivariant models have been applied to tasks ranging from computer vision molecular chemistry, leading a generalization equivariance techniques beyond 2D rotations to other symmetries such as 3D rotations. This is enabled by known mathematical results about each new set of symmetries. Specifically, explicit group representation matrices for each new symmetry group are required. For many important symmetries, formulae are readily available to produce these representations. For other symmetries we are not so lucky, and the representations may be difficult to find explicitly. In the worst cases, the classification of the group representations is an open problem in mathematics. For example, the important case of the homogeneous Galilean group, which we define in section the classification of the finite dimensional representations is a so-called "wild algebraic problem" for which we have only partial solutions (De Montigny et al. 2006 Niederle & Nikitin 2006 Levy-Leblond 1971).
To construct an equivariant network without prior knowledge of the group representations, novel approaches needed. In this work, we propose an algorithm LearnRep that finds the representation matrices with high precision. We validate that Leannep succeeds for the Poincaré group, a set of symmetries governing phenomena from particle physies object tracking. We further validate LearnRep on two additional sets of symmetries where formulae are known. We apply the Poincaré group representations obtained by LearnRep to construct SpacetimeNet, object-tracking model. As far as we are aware, LearnRep is the first automated solver which can find explicit representation matrices for sets of symmetries which form noncompact, noncommutative Lie groups Further, SpacetimeNet is the first object-tracking model with a rigorous guarantee of Poincaré group equivariance.
Poincaré-equivariant
1.1 GROUP REPRESENTATIONS AND EQUIVARIANT MACHINE LEARNING
Group theory provides the mathematical framework for describing symmetries and building equiv- ariant ML models. Informally, a symmetry group G is a set of invertible transformations a, 3 E G which can be composed together using product operation a:3. We are interested in continuous symmetries for which G is a Lie group. In prior constructions of Lie couppequuvariant models, group representations are required. For group G, n- oo-dimensional (real) group representation p: G + B is a mapping from each element a E to an n x n-dimensional matrix p(a), such that for any elements o, G, we have (()) p(a33)

---

Under review as a conference paper at ICLR 2021
Two parallel techniques have been developed for implementing Lie group equivariant neural networks. The first approach was described in general by Cohen et al. (2019). For the latter approach taken by Thomas al. (2018) Anderson al. (2019) Bogatskly et al. (020), convolutions and nonlinearities are performed directly on the irreducible representations of the group, which we define section 22.4] common thread in these works has been utilize existing formulas derived for the matrix elements of these irreducible representations. However, these formulas are only available for specific Lic groups where the representation theory is well-understood. A more convenient approach for extending equivariance to novel Lie groups would utilize an automated computational technique to obtain the required representations. The primary contribution of this work is such a technique.
1.2 CONTRIBUTIONS
In this work, we automate the generation of explicit group representation matrices of Lie groups using an algorithm called LearnRep. LearnRep poses an optimization problem defined by the Lie algebra associated with Lie group, whose solutions are the representations of the algebra. A penalty term is used to prevent the formation of trivial representations. Gradient descent of the resulting loss function produces nontrivial representations upon convergence. We apply LearnRep to three noncommutative Lie groups for which the fiite-dimensional representations are well-understood, allowing us to verify that the representations produced are irreducible by computing their Clebxch-Gordd Gordan coefficients and applying Schur's Lemma.
One of the Lie groups where LeanRep performs well is the Lorentz group of special relativity. Prior work has applied Lorentz-qquivariant models to particle physics. In this work we explain that the Lorentz group along with the larger Poincaré group also governs everyday object-tracking tasks. We construct boincaré-equivarinn that it can learn to solve 3D object-tracking task subject "motion equivariance," where the inputs are time series of points in space.
neural network architecture called SpacetimeNet and demonstrate
In summary, our contributions are:
LearnRep, an algorithm which can find irreducible representations of a noncompact and noncommutative Lie group.
SpacetimeNet, Poincaré group group-equivariant neural network applied to object-tracking tasks.
Our work contributes towards general framework and toolset for building neural networks equivari- ant to novel Lie groups, and motivates further study of Lorentz equivariance for object tracking.
1.3 ORGANIZATION
We summarize all necessary background and terminology in section 2 We describe the LearnRep algorithm in section 3.1 and SpacetimeNet in section 3.2 We summarize related work in section 4 We present experimental results in section 5 experiments in learning irreducible Lie group representations with LearnRep in section 5.1 and the performance of our Poincaré-equivariant
SpacetimeNet model on 3D object tracking task in section 5.2
TECHNICAL BACKGROUND
We explain the most crucial concepts here and defer to Appendix A.1 for a derivation of the represen- tation theory of the Lorentz group
2.1 SYMMETRY GROUPS 50(n) AND S0(m, n)
A 3D rotation may be defined as matrix A :e gax3 which satisfies the following properties, in which (II, 0)
3 1= U;
(i) det 11
(ii) V7,, 7 E R (Aū, AU) = (ū,
these imply the set of 3D rotations forms a group under matrix multiplication and this group is denoted (3). This definition directly generalizes the n dimiinnsionna rotation group SO(n). For
2

---

Under review as a conference paper at ICLR 2021
2 3. the group (O(n) is noncommutative, meaning there are elements A, B E SO(n) such that AB * BA. Allowing for rotations and translations of n dimensional space gives the n in-dimensional special Euclidean group SE(n). (O()) is generalized by family of groups denoted (0 n), with SO(n) SO 0). For integers m, n 2
define (ū,, m) = -iimi 4 Limm+ UgVi. The group S0(m, n) is the set of matrices A E g(m++)xx(m+u)) satisfying (i-ii) below:
det ==
(ii) Viž, E Rm*" (Aiz, Aü)m, (17, m)m
these imply that 50 n) is also a group under matrix multiplication. While the matrices in SO(n) can be seen to form a compact manifold for any n, the elements of SO n) form noncompact manifold whenever n, m 1. For this reason SO(n) and SO n) are called compact and noncompact Lie groups respectively. The representations of compact Lie groups are fairly well understood, see Bump 2004); Cartan (1930).
2.2 ACTION OF S0(m, n) ON SPACETIME
We now explain the physical relevance of the groups 50(m, n) by reviewing spacetime. We refer to Feynman et al. (20TT) (ch. 15) for a pedagogical overview. Two observers who are moving at different velocities will in general disagree on the coordinates {(t,, ú7,)] c R of some events in spacetime. Newton and Galileo proposed that they could reconcile their coordinates by applying a spatial rotation and translation (i.e., an element of SE)()), a temporal translation (synchronizing their clocks), and finally applying transformation of the following form:
4; in which Ø is the relative velocity of the observers. The transformation equation D is called a Galilean boost. The set of all Galilean boosts along with 3D rotations forms the homogeneous Galilean group denoted HG(1, 3). Einstein argued that equation 1 must be corrected by adding terms dependent on ||0]ee/, in which e the speed of light and | is the &g norm of J. The resulting coordinate transformation is called a Lorentz boost, and an example of its effect is shown in figure [ The set of 3D rotations along with Lorentz boosts is exactly the group SO 1). the case of 2 spatial dimensions, the group is SO(2, 1). Including spacetime translations along with the Lorentz group SO(n, 1) gives the larger Poincaré group P,, with n spatial dimensions. The Poincaré group Pa is the group of coordinate transformations between different observers in special relativity.
+ oli
Consider an object tracking task with input data consisting of spacetime point cloud with n dimensions of space and 1 of time, and corresponding outputs consisting of object class along with location and velocity vectors. A perfectly accurate object tracking model must respect the action of Pr, on the input. That is, given the spacetime points in any observer's coordinate system, the perfect model must give the correct outputs in that coordinate system. Therefore the model should be Pr.-equivariant. For low velocities the symmetries of the homogeneous Galilean groups HG(n, 1) provide good approximation to SO 1) symmetries, so some tasks. Unfortunately the representations of HG(n, 1) are not entirely understood De Montigny et al. (0066); Niederle & Nikitin 2006 Levy-Leblond (191I)
may be sufficient for
2.3
LIB GROUPS AND LIE ALGEBRAS
Here we give an intuitive summary of Lie groups and Lie algebras, deferring to Buump (2004) for a rigorous technical background. A Lie group G gives rise to a Lie algebra A as its tangent space at the identity. This is a vector space along with bilinear product called the Lie bracket: [a, which must behave like the commutator for associative ring R with multiplication operation XR:
(a, b] x b x a
The Lie algebra for SO(3), denoted so(3). has a basis {J1, Ja} satisfying
(2) in which 6jjk E 11.0 is the totally antisymmetric I Levi-Civita symbol Intuitively, the Lie bracket shows how group elements near the identity fail to commute. For example, the matrices R2y Ry, R
[Jr, J,J Jjj.Jk.
"Specifically, the Lie bracket must satisfy the Jacobi identity and (a, a) O. "hh symbol eyy simply expresses in equation 2 that [J1, Jz) = Js. [J2, Ja] J1, [Ja, Jı] J2-
3

---

Under review as a conference paper at ICLR 2021
Original
Transformed: Boost
Transformed: Boost
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0025
0.25
-0.5
-0.50
-0.50
-0.50
-0.55
-0.5
-0.5
-0.5
0.5
Figure 1: Activations of an SO(2, 1) Equqqirari neural network constructed using our framework. The arrows depict the elements of the 3-dimensional representation space (arrows) and are embedded on their associated points within the point cloud. This point cloud is from the MNIST-Live dataset as generated with digits embedded
x plane. The y axis is suppressed. The left plot depicts the "original" activations (with the digit rest). The right plots show what happens if we transform the point cloud with a Lorentz boost in the tz direction before feeding it through the network. As the activation vectors generated by the network transform in the
dictated by parentz-cquivarian same way as the input point cloud.
for rotations about the and axes by small angle 9 satisfy R, R. R 0(0)); more generally the Lie bracket of equation 2 is satisfied to first order in 0. The Lia algebra so(3, 1) of the Lorentz Group SO(3, 1) also satisfies equation 2 for the generators J1, J2, J3 of its subalgebra isomorphic to so(3). It has additional generators denoted K1, K2, K3, which satisfy:
Ke These K, correspond to the Lorentz boosts in the same way that the J, correspond to the rotations. In general, A is t-dimensional Lie algebra with generators Ti.... Ti such that
[J, K2]
K2 Ký] = CijkeJk
(3)
[T, T3]
Σ TR
we call the tensor Aijk the structure constants of A. For connected matrix Lie groups such as S0(m, n), the structure constants Aijk are easily obtained. For example, one may apply the matrix logarithm to several elements the group to obtain elements of the algebra, then find a complete basis for the algebra and write the commutator of all basis elements in this basis.
2.4 GROUP REPRESENTATION AND THE TENSOR PRODUCT
Let G be Lie group and p G + R=x be a representation of G as defined in section 1.1 Then defines a group action on R. given vector i R." and group element G. we can define
xp ū p(a)ū
using the matrix product. We then say that is irreducible if it leaves no nontrivial subspace invariant for every subspace V C Rn with 0 < dim V < n, there exists a E G, it E V such that a *p it & V.
P2 G x we define their tensor product Given two G-representations P1 G R as
P1 c P2 G m
(p1 (a) pı(a) & pe(a),
in which @ the right hand side denotes the usual tensor product of matrices. is easy check that P1 @ P2 is also a representation of G using the fact that for matrices A1, Ag e g X and B1, Bg Xn
(A1 & B1) & B2) = (A1 & (2)
For as above we also define their direct sum as
(p1 2 (2) = (pi(a)
P2(0))
4

---

Under review as a conference paper at ICLR 2021
For two groups H, G say that H is isomorphic G and write H = G there exists bijection f: H - such that f(a.) = (3) For P1,P2 22 as above, their images Pa(G) form groups and we say that P1 and P2 isomorphic and write P1 _ P2 if these groups isomorphic, i.. (G) = p2 Some familiar representations of SO(3) act on scalars E IR, vectors E IR3,, and tensors (e.g., the Cauchy stress tensor) these representations are all nonisomorphic.
For many Lie groups such as SO(n, 1) and (0()), property called complete reducibility guarantees that any representation is either irreducible, isomorphic direct sum of irreducible representa- tions. For such groups suffices to identify the irreducible representations to understand all other representations and construct equivariant models.
2.5 CLEBSCH-GORDAR COEFFICIENTS AND TENSOR- TENSOR-PRODUCT PRODUCT NONLINEARITIES
Clebsch-Gorde Coefficients: Let G be completely reducible Lie group, and let Pl. P2, be irreducible G-representations on the vector spaces R , _R' Consider the tensor product repre- sentation P1 @ P2. Since G is completely reducible, there exists set S of irreducible representations such that @ P2 = 4 P Suppose that Pa E S. Then there exists matrix C E R ((1
which projects the space of the ni-diimnsionna group representation Ps from the tensor product space m @ R2 That is,
= V(a, ū. ( EEExx RR? C(pi(a) @ = (0 C(pi(a) @ =
(5)
The matrices C satisfying equation for various P3 called the Clebsch-Gorde coefficients. In equation 5 there are n 12223 linear constraints on C, and therefore this is well-posed homogeneous linear program (LP) for C. The entries of C' may be found numerically by sampling several distinct G' and concatenating the linear constraints (equation 5) to form the final LP. The solutions for C form a linear subspace of R ((1)22) given by the nullspace of some matrix we denote Clpi, (|,,,,,,/00 P2, Pa].
Tensor Product Nonlinearities: Tensor product nonlinearities, including norm nonlinearities, the Clebsch-Gorde coefficients defined above to compute equivariant quadratic functions of multiple G-representations within the G-equivariant model. This was demonstrated for the case of SE(() by
Thomas et al. (2018); Kondor et al. (2018) and for SO(3, 1) by Bogatskiy et al. (2020).
METHODS
3.1
LEARNING LIE GROUP REPRESENTATIONS
For matrix M E R we denote its Frobenius and L1 norms by
IMP
Σ
[M]
Σ 154.5"
Migl-
IS1JSR
The approach of LearnRep is to first learn Lie algebra representation and then obtain its correspond- ing group representation through the matrix exponential. Fix t-dimensional Lie algebra with structure constants Aiyk as defined in equation Fix positive integer n as the dimension of the representation of A. Then let the matrices T Ti, ...... T, E IR xx be optimization variables, and define the following loss function on the T;:
CTT. ..... Ti] max
1, max ISiSt TTP
[T,T]]
Σ AnjiTe
(6)
Isisis
N[TI-1
This is the magnitude of violation of the structure constants of A, multiplied by norm penalty term [[ (this penalty is plotted separately in figure 2 The purpose of the norm penalty is to avoid convergence solution where T; = Onxn for any which will act trivially when restricted to the nontrivial subgroup ( e I... We pose the optimization problem:
min TeRNn
C(Ti...., Ti]-
The generators initialized with entries from the standard normal distribution. Gradient descent was performed Py Torch with the Adam optimizer (Kingma & Ba 2014) with initial learning rate 0.1. The learning rate decreased exponentially when loss plateaued. The results shown in figure
5

---

Under review as a conference paper at ICLR 2021
3.1.1 VERIFYING IRREDUCIBILITY OF LEARNED REPRESENTATIONS
Suppose we have converged Ti T, such that CTT] 0. Then the Ti, .., T, are a nonzero n--dimensional representation of the Lie algebra A. The groups considered here are covered by the exponential map applied their Lie algebras, so for each a there exist by...., by E R such that
p(o) = exp Σ b.T.
where p any n nidiinnniional representation of G and exp the matrix exponential. This p G H> Rox" is then representation of the Lie group. Throughout this section, p denotes this representation. In general p may leave some nontrivial subspace invariant. this case it is reducible and splits as the direct sum of lower-dimensional irreducible representations Pt as explained in 2.4
p 2 @ ......D. ..... pt-
Recall that any representation may be obtained as such a direct sum of irreducible representations with dimensions ....... satisfying = =Ei-
If is set to the minimum dimension of nontrivial irreducible representation, the only permissible partitions of n have & = and = n as the latter representation is trivial equation diverges, so LearnRep can only converge to an irreducible n dimensional representation 3 It is important to verify that the learned is indeed irreducible with f 1. To validate that p is irreducible. LearnRe computes its tensor product structure and compares with the expected structure. Specifically, it computes the Clebsch-Gordan coefficients for the direct-sum decomposition of the tensor product of the learned representation p with several other known representations Pi. Pr- section 2.3 these coefficients and explains how they are computed from the nullspace of the matrix C = Clp. P1, P2], in which p2 appears in the decomposition of p & Let Pl,P2 denote two other known representations, and consider the Clebsch- Gordan coefficients C' such that Cp & P1 pyC.. The dimension of the nullspace of indicates the number of unique nonzero matrices C' of Clebsch-Gordan coefficients. The singular values of are denoted SV, (C) S ... S (C). The ratio
rC) == (V2(C)/VV(()
diverges only if the nullspace is dimensional which therefore corresponds unique solution for C. The number of expected solutions is known (e.g., it may be computed using the same technique from the formulae for the irreducible representations). Therefore if rCC) diverges for exactly the choices of Pl, P2 where the theory indicates that unigue nonzero Wiebcch-Gordan coefficients exist, then this is consistent with our having learned an irreducible representation of the group G.
Clearly the tensor product with the trivial representation P is & p. In this case, the permissible C correspond to G-linear maps RM >> T2 By a result [chhr (1905) (Schur's Lemma), the only such (nonzero) maps are isomorphisms. Therefore divergent value of r(C) when P1 indicates that p _ p2. This is shown in the top row of figure and discussed further in
section 5.1
3.1.2 STOPPING CONDITION
Similar to (Rao & Ruderman 1999). LearnRep restarts gradient descent several times starting from random initialization points. A restart is triggered if loss plateaus and the learning rate is smaller than the loss by factor of at most 10-1 The tensor product structure is computed upon convergence to loss under 10-9, a restart is triggered if the divergences r(C) do not agree with the theoretical prediction, indicating reducible representation.
3.2 SPACETIMENET ARCHITECTURE
We obtain all Clebsch-Gordar coefficients through the procedure explained in section 2.5 We place them in a tensor: Cnt- This notation corresponds to taking the tensor product of an element of the (th group representation space indexed by with an element of the mh group representation space indexed by f,, and projecting it onto the 7th group representation space indexed by The space
Thiis applies experiments learning SO(3) representations, with

---

Under review as a conference paper at ICLR 2021
of possible Clebsch- Gordan coefficients can be multidimensional We use index g to carry the dimension within the space of (lebsch-Gordan coefficients.
The trainable weights in SpacetimeNet
complex-valuees filter weights denoted f and channel- mixing weights denoted W qegdd" Each layer builds a collection of equivariant convolutional filters
Fk from the geometry of the point cloud. Let o' denote the index of the group representation in whigh the points are embedded. Let Xzár denote the point coordinates, in which indexes the batch dimension, i indexes the points, and r indexes the q' group representation space. Define the (globally) iasslation-invarian quantity AX := Xarr -xiir. The equivariant filters at layer are:
F zijer = -gg AXaige + Caurd a. AX AKzijt-
(
e,t,9
The forward pass consists tensor product nonlinearities between equivariant filters and activations. The input and activations for the h layer of the network are defined on tensor V Vaimett where is the batch dimension, i indexes the points, m is the group representation index, is the channel index, t indexes the group representation space. Our mixing weights are then defined for the Ath layer as gegd with layer update rule:
W
Vk+1 irgeer
Σ
Cg.gr.s.mt
Fik Vll Wh zijls ajmdte
(9)
gogd-
g.S.a.m.dd...
proof that SpacetimeNet is
P.-eeiiu Pu-equivvriant is given in Appendix A.2
RELATED WORK
4.1 LEARNING LIE GROUP REPRESENTATIONS
Several authors have investigated automated means of identifying Lie group representations. (Rao & Ruderman T999) used gradient descent with several starting points to find the Lie group generators, given many examples of data which had been transformed by the group. Applying the technique requires knowledge of how the group acts on a representation space. Here we know the Lie algebra structure but we do not know how to compute its representations. Tai et al. (2019) gave closed-form solution for the canonical coordinates for Lie groups. But their formula only applies for Abelian one- parameter Lie groups, excluding SO(3), SO(,, 1), and SO(3, 1). Cohen & Welling (2014) devised a probabilistic model to learn representations of compact, commutative Lie groups from pairs of images related by group transformations. In the present work we demonstrate a new approach to handle noncompact and noncommutative groups such as (0(3).SO() 1), and SO(3, (3, 1). Computer algebra software such the LiE package developed by Van Leeuwen al. (1992) automates some calculations related to completely reducible Lie groups. Unfortunately this limits us when considering novel Lie groups where the representation theory is less well-understood.
4.2 EQUIVARIANT NEURAL NETWORKS
Beginning with the success of (approximately) translation equivariant CNNs introduced by LeCun
al. (1989) for image recognition. a line of work has extended equivariance to additional continuous
symmetry groups. Most relevant are the architectures for groups SF(2) (Worrall et al. 2017 Weiler
Cesal 2019). SE(3) (Weiler et al., 2018 Cohen et al. 2019 Kondor et al., 2018 Thomas et al.
2018 Cohen et al., 2018 Kondor 2018 Gao et al. 2020: Anderson et al. 2019 Fuchs al. 2020
Eismann et al. 2020), and the group of Galilean boosts (hu et al. 2019).
The work by Thomas et al. (2018); Kondor et al. (2018): Anderson et al. 2201)) Bogatskiy et al. (220)) used Clebsch-Gordan coefficients in their equivariant neural networks. Weiler et al. (2018). generalized by Cohen et al. (2019) showed all equivariant linear maps are convolutions whose kernels satisfy some linear constrants. In our work we obtain Clebsch-Gordan coefficients from similar linear constraints (equation 5) and use them to show that the learned representations irreducible. We also use them in SpacetimeNet, Grifffths & Griffiths (2005) provide an introductory exposition of Rebsch-Gordan coefficients and Gurarie 1992) provides a more general exposition.
*This common if group representation is itself obtained via tensor product.
7

---

Under review as a conference paper at ICLR 2021
One of the first constructions that addressed spatiotemporal symmetries by Zhu et al. (011)) They introduce motion- eiriio--quivaiian networks to handle linear optical flow of an observer moving at fixed speed. They use canonical coordinate system in which optical flow manifests translation, as described for general one dimensional Lie groups by (Tai et al. (201)). This allows them to use the translation equivariance of CNNs to produce Galilean boost-equivariance. However, this gives up equivariance to translation in the original coordinate system. To maintain approximate translatinn-qqiivaraaee the authors apply a spatial transformer network (Jaderberg et al. 2015) to predict a landmark position in each example. This is similar to the work of Esteves et al. (2018). 8).
which achieved equivariance 2D rotation and scale, and approximate equivariance to trans.ation.
The first mention of on the link between covariance in MI. and physies. Concurrently to our work, |Bogatskiy et al. (2020) constructed a Lorentz-equivarian model which operated on irreducible representations of the Lorentz group, derived similarly to Appendix I This work also made use of the (Lebsch-Gordan coefficients, and the model was applied to experimental particle physics rather than object-tracking. Another work by Finzi et al. (2020) concurrent to our own proposed a framework for building models equivariant to arbitrary Lie groups. This work also made use of the exponential and logarithm maps between Lie algebra and group. It does not provide a technique for identifying the Lie algebra representations. Our ideas complement this line of work by providing algorithm (LearnRep) that solves for the representations numerically.
networks appears to be a work by Cheng et al. (2019)
5 EXPERIMENTS
5.1 CONVERGENCE OF LEARNREP TO IRREDUCIBLE REPRESENTATIONS
We apply LearnRep to SO(,), 1). and SO(3, 1) to learn 3, 3, and 4 dimensional irreducible representations respectively. The loss function converges arbitrarily close with the penalty term bounded above by a constant. We exponentiate the resulting algebra representation matrices to obtain group representations and calculate the tensor product structure as described section I The details of this calculation are in Appendix [A.4 and shown in figure 3 The results indicate that the learned representations are irreducible representations of the associated Lie algebras to within numerical error of about 10-6, Schur's Lemma in the special case of the tensor product with the trivial representation indicates the isomorphism class of each learned group representation.
5.2 DOINCARÉ-EQUIVARIANY OBJECT- TRACKING NETWORKS
We created MNIST-Live, benchmark dataset of spacetime point clouds sampled from digits from the MNIST dataset moving uniformly through space. Each sample consists of 64 points with uniformly random times E [-1/ 1/2], and spatial coordinates sampled from 2D probability density function proportional to the pixel intensity. Using instances of the O and 9 classes, we train on examples with zero velocity and evaluate examples with random velocity and orientation. This dataset is analogous to data from an event camera (see (Orchard et al., (015)) or LIDAR system. We train 3 layer SO(2, 1) and SO 1) -qquivariantt SpacetimeNet models with channels and batch size 16 on 4096 MNIST-Live examples and evaluate on a dev set of 124 examples. We obtain dev accuracy of 80 + 5% as shown in figure 4 of the Appendix.
5.3 CONCLUSION
We envision many applications of Poincaré-cquivarian deep neural networks beyond the physics of particles and plasmas. SpacetimeNet can identify and track simple objects as they move through 3D space. This suggests that Lorentz-equivariance is useful prior for object-tracking tasks. With a treatment of bandlimiting and resampling as in Worrall et al. (017)): Weiler et al. (2018), our work could be extended to build Poincaré- eeuuirriauuvaiiiant networks for volumetric data. More broadly, understanding the representations of noncompact and noncommutative Lie groups may enable the construction of networks equivariant to new sets of symmetries such the Galilean group. Since the representation theory of these groups is not entirely understood, automated techniques such as I LearnRep could play a beneficial role.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural net- works. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 14510-14519. Curran Associates, Inc., 2019.
Alexander Bogatskiy, Brandon Anderson, Jan T Offermann, Marwah Roussi, David W Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. arXiv preprint arXiv:2006.04780, 2020.
Daniel Bump. Lie groups. Springer, 2004.
Élie Cartan. La théorie des groupes finis continus l'analysis situs. Mémorial Sc. Math., 1930.
Miranda CN Cheng. Vassilis Anagiannis, Maurice Weiler, Pim de Haan, Taco S Cohen, and Max Welling. Covariance in physics and convolutional neural networks. arXiv preprint arXiv: 1rXivv:906.02440 02481, 2019.
Taco Cohen and Max Welling. Learning the irreducible representations commutative lie groups. In International Conference on Machine Learning, pp. 1755-1763, 2014.
Taco Cohen, Mario Geiger, Jonas K ohheer Pim de Haan, K. T. Sch utt, and Benjamin K. Miller. Lie learn, February 2020. URL
com AMLab Aaw.llaammmmttaammmaaaaaaaaaaaaa wwterdaammiaeeeeeeeeeeeeeeeeeeee lie_ammieee_eee______eeee______e
teleases/tag/vl.
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In International
Conference on Learning Representations, 2018. URL https::/openrev// ww..ewwwwwww.................... nwt.t/forrmm
id=Hkbd5x22Rb
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on ho- mogeneous spaces. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bus E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 9142-9153. Curran Associates, Inc., 2019.
M De Montigny, J Niederle, and AG Nikitin. Galilei invariant theories: I. constructions of inde- composable limte-dimensional representations of the homogeneous galilei group: directly and via contractions. Journal of Physics A: Mathematical and General, 39(29):9365, 2006.
Stephan Eismann, Raphael JL Townshend, Nathaniel Thomas, Milind Jagota, Bowen Jing, and Ron Dror. Hierarchical, rotation-equivariant neural networks to predict the structure of protein complexes. arXiv preprint arXiv:2006.09275. 2020.
Carlos Esteves, Christine Blenchllachhett,, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. In International Conference on Learning Representations, 2018. URL https:/7 openreview www.neeetooriiee d
Richard Feynman, Robert B Leighton, and Matthew Sands. The Feynman lectures on physics, Vol. I: The new millennium edition: mainly mechanics, radiation, and heat, volume 1. Basic books, 2011.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson, Generalizing convolutional neural networks for equivariance lie groups on arbitrary continuous data. arXiv preprint arXiv:2002. 12880, 2020.
Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling, Se (3)-transformers: 3d roto-translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020.
Liyao Gao, Yifan Du, Hongshan Li, and Guang Lin, Roteqnet: Rotation-equuvvarian network for fluid systems with symmetric high-order tensors. arXiv preprint urXiv:2005.04286 2020.
D.J. Griffiths and P.D.J. Griffiths. Introduction Quantum Mechanics. Pearson international edition. Pearson Prentice Hall, 2005. ISBN 9780131 11113311119999 18928. URL /tt//ttts////okk... htt.t:::///ooosssoogeeeec.oo gwogle. com/
book dizz FWARRAMAAN

---

Under review as a conference paper at ICLR 2021
D Gurarie. Symmetries and laplacians. introduction to harmonic analysis, group representations and applications.
mathematics studies, 174:1-448, 1992.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 2017-2025. Curran Associates, Inc., 2015. URL ltttp://papers. nips. cc/paperr//ppr pdf.
J Robert Johansson, Paul D Nation, and Franco Nori. Qutip 2: A python framework for the dynamics of open quantum systems. Computer Physics Communications, 184(4):1234- 1240, 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1eti.6112.6000 2014.
Risi Kondor. N-body networks: covariant hierarchical neural network architecture for learning atomic potentials. arXiv preprint (003v0/88000/55555 2018.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch- gordan nets: fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems, pp. 10117- 10126, 2018.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard. Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Jean-Mare Levy-Leblond. Galilei group and galilean invariance, Group theory and its applications, pp. 221-299. Elsevier, 1971.
J Niederle and AG Nikitin. Construction and classification of indecomposable fiite-ddimensional representations of the homogeneous galilei group. Czechoslovak Journal of Physics, 56(10-11): 1243-1250, 2006.
Garrick Orchard, Ajinkya Jayawant, Gregory Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience, 9:437, 2015.
Didier Pinchon and Philip E Hoggan. Rotation matrices for real spherical harmonics: general rotations of atomic orbitals in space-fixed axes. Journal of Physics A: Mathematical and Theoretical, 40(7): 1597, 2007.
Rajesh PN Rao and Daniel L Ruderman. Learning lie groups for invariant visual perception. In Advances in neural information processing systems, pp. 810-816, 1999.
Issai Schur. Neue begründung der theorie der grappencharaktere. 1905.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant transformer networks. arXiv preprint arXiv: /901. 1399, 2019.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley, Tensor field networks: Rotation-and translation ranslation-equivariann neural networks for 3d point clouds. arXiv preprint arXiv:1802.08211 2018.
Marc AA Van Leeuwen, Arjeh Marcel Cohen, and Bert Lisser. Lie: A package for lie group computations. 1992.
Maurice Weiler and Gabriele Cesa. General e (2)-cquivariant steerable cnns. In Advances in Neural Information Processing Systems, pp. 14334-14345, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems, pp. 10381-10392, 2018
Steven Weinberg. The quantum theory of fields. Vol. I: Foundations. Cambridge University Press, 1995.
10

---

Under review as a conference paper at ICLR 2021
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Alex Zihao Zhu, Ziyun Wang, and Kostas Danilidis. Motion equivariant networks for event cameras with the temporal normalization transform. arXiv preprint
2019.
A APPENDIX
A.1 ANALYTIC DERIVATION OF LORENTZ GROUP REPRESENTATIONS
To compare our learned group representations with those obtained through prior methods, require analytical formulae for the Lie algebra representations for the algebras so(3), so(3, 1), and so(2. 1). The case of so(3) has well-known solution (see Griffiths Griffiths) (2005) ). If complex matrices are permissible the library representation matrices. A formula obtain real-valued representation matrices is given in Pinchon & Hoggan (007) and software implementation is available at Cohen et al. (222)). The three- dimensional Lie algebra s2 1) = span, Kg, y. J2) has structure constants given by equation 3 In fact, these three generators K₃, Kg, J= may be rescaled so that they satisfy equation 2 instead. This due to the isomorphism s3) so(2, 1). Specifically, leting (Le:
et al. (2013) has function "jmat" that readily gives the
Lys L2> denote a Lie
algebra representation of so(3), defining
K₂ = -iLe
=iLy J= Loy
it may be casily checked that Kg, Ky. Jg satisfy the applicable commutation relations from equation 3 This reflects the physical intuition that time behaves like imaginary dimension of space.
The final Lie algebra for which we require explicit representation matrix formulas is so 1). Following Weinberg (1995), define generators A,, B,
A = 2 (J. + iK,)
B; = (J. - -K;);
(10)
we see that the so(3, 1) commutators equation equation become
[Ai, A, = Ax;
[B, B,] ieija Br.
[A:, Bj] 0.
(11)
Therefore so 1) so( @ s3). and the irreducible algebra representations of so 1) may be obtained the direct of two irreducible algebra representations of so(3).
A.2 PROOF THAT SPACETIMENET IS MIINCARÉ-EQUIVARIE
Consider an arbitrary Poincaré group transformation a E Pn. and write = 3t in which B e SO(n, 1) and is a translation. Suppose we apply this a to the inputs of equation 8 through the representations indexed by q: Pa(a)st, in which s, index the representation matrices. Then since the translation leaves AX invariant, the resulting filters will be
Fk Fxeger Oeg'
+ Σ Eggar (3) Po AX
= - Σ (g) AX + Σ
Σ
Co Co (0
f f AXxijo ANaije
mit
= Gug (gr AXnij + Σ
(Po(8)rr of
s.1.g.t'
EPy (3) 8gg Aggi + Σ
Σ Pof (g) F
11

---

Under review as a conference paper at ICLR 2021
where we have used equation 5 The network will be equivariant if each layer update is equivariant. Recall the layer update rule of equation
V+++1 aagee
Σ
Agggr F zifts zimdt gegd-
Vak Wh
g.J.e.m,t.d.j
Suppose for the same transformation = BL above, that V and AX are transformed by a. Then because the activations associated with each point are representations of SO(n, 1), they are invariant to the global translation of the point cloud and have
Vk+1 ssger
Σ
E Pm (B) Va ajmmder W Cg.aris.mt E Pm (B))ss))) F mijts'
gegd
g.l.s.m.I.d..
Σ
E si, g.l.s.m,t.d..
(Ca.ar,temmaPn) (3) Pen. (8) Paigtar V simade W gogd
E s.l.s.m...ddji.r
Gagrr de.me) E Fzisto Vol Vasmat W good
Epm( (.m(B)r siger'>
vh+1
where again we applied equation 5
A.3 EQUIVARIANT CONVOLUTIONS
Consider data on a point cloud consisting of a finite set of spacetime points { R4,, a rep- resentation Po S3(3, 1) + 1 of the Lorentz group defining its action upon spacetime, and
feature maps { C Im (7) c R" associated with representations Pa. SO 1) - mmm and
P. (3, 1) - _" A convolution of this feature map can be written
T Σ -
in which x(z): : I + R"xm, matrix-valued function of spacetime, the filter kernel. Pg-equivariance dictates that for any SO(3, 1).
(o(a) Σn - =,jūū = Σ <(pola)(ET ()))Pa(a)TT,
n² ((AF) (o paloe))aapp (12)
Therefore a single kernel matrix Rx may be learned for each coset of spacetime under the action of S3 1). The cosets are indexed by the invariant
x² 2² =2
The kernel may then be obtained at arbitrary point . E I from equation 12 by computing that relates it to the coset representative zg: = Pa a zg. A natural choice coset representatives for SO(3, acting upon RM is the set of points 0,,,0,0,0 R+] [(t,, ct, 0,0) R+)
R+ (00, 0,00)
A.4 TENSOR PRODUCT STRUCTURE OF LEARNED SO(3), 1), SO(3, 1) GROUP REPRESENTATIONS
We quantify the uniqueness of each set of Clebsch-Gordee coefficients in terms of the diagnostic ratio r(C) defined in equation 7 Recall that the value r becomes large only there a nondegenerate nullspace corresponding the irreducible group representations are labeled by an integer which is sometimes called the spin. We label learned group representations with primed (i') integer. For the case of SO 1) the irreducible group representations are obtained from two irreducible group representations so 53 explained in section A.1 and label these representations with both spins i.e. (81,82). We
unique set of Clebsch-Gordan coefficients. For SO(3) and SO 1),
12

---

Under review as a conference paper at ICLR 2021
30
151
Figure 2: Convergence arbitrary precision group representations of three Lie groups: SO(3), S0(2, 1), and SO(3, 1). The multiplicative norm penalty is plotted in each lower subplot, and demonstrates that this penalty is important early on preventing the learning of trivial rep- resentation, but for later iterations stays at its clipped value of 1. Loss is plotted on each upper subplot.
again label the learned group representations of SO(3, 1) with primed spins, i.c. (81,s)). The tensor product structures of the representations is shown in figure
We have produced software library titled Lie Algebraic Networks (LAN) built PyTorch, which derives all Grebsch-Gordan coefficients and computes the forward pass of Lie group equivariant neural networks. LAN also deals with Lie algebra representations, allowing for operations such as taking the tensor product of multiple group representations. figure demonsstates the LAN library. Starting from several representations for a Lie algebra, LAN can automatically construct neural network equivariant to the associated Lie group with the desired number of layers and channels. We present our experimental results training SO 1) and SO(3, 1) 1)-euiivariant object- abject-tracking networks in section 5.2
A.5 SUPPLEMENTARY FIGURES
13

---

Under review as a conference paper at ICLR 2021
1'80
1/8.
(1/2: 1/27 (0.
50,5VA
10
5V,5555
1012
5Y.8VV
1017
101
109
10F
01
(17) 127
a1. 121
(1.1)
(an2 301
1'
1'e1
(1Z. 1/2)
(1.1)
57,50
10a
5W5855
SV/591
10
20*
1/2
(12. 1/2)
a. (1.1)
on 3.)
10.
077.
(1/2; 1/2)) (1/2. 122
5Y,80
201
595:8W
10"
101
10
9
6)
(12. 121
(1. 12)
(..))
012
1' #
1' 2
(1/2; 1/2") @ (1/2. 1/2)
57,85
10*
595884
10"
SV.5VV
10t
20*
10
10
127
a1 12)
(.1)
an 30)
(0.
017.
1'
@ (22)
10a
100
10"
10a
301
7
(z2 1/2)
aa, 12)
(1.1)
0/2 3/0)
SO(3)
S0 1)
SO(3, 1)
Figure Tensor product structure of the learned group representations p with several known dunalyticallyderiveee group representations P1 for the groups SO(3). SO(2, 1), and SO(3, 1). Each column is for the group indicated at the bottom, each row is for different choice of p1 for that group, and the horizontal axis indicates the p(t) onto which we project the tensor product p p1 @)esp()) The diagnostic r (defined in section 2.5 is plotted on the y-axis with a log scale for each subtigure. The labelling of group representations is explained in section 5.1 recall that the primed integers indicate learned representations. The first row demonstrates by Schur's Lemma that within numerical error of about ~ 10-6 the learned SO(3) group representation denoted is isomorphic the spin-1 irreducible group representation obtained from known formulae, i.e. 1/S013) & 1so(3)-
The first row also indicates that 1'002.1)) 1s0(2,1), and (1/2', 1/22)ss(s.n) 11/2,1/2)so(3.u) The remaining rows indicate that the tensor product structure of the learned group representations matches that of the known irreducible group representations.
14

---

Under review as a conference paper at ICLR 2021
0.8
0.4
test accuracy train (zo)
test accuracy accuracy (tto)
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.0
0.4
(hr)
time (hr)
50000
2500
0.0
0.1
0.2
0.3
0.4
0.2
0.4 time (hr)
0.6
time (hr)
50000
200000
train loss (2o)
train loss (#:o)
0.0
0.1
0,2
0.3
0.4
0.0
0,2
0.4
0.6
time (hr)
(hr)
Figure 4: (Left) SO(2, 1) equivariant neural network learning to recognize digits from the MNIST- Live dataset spatial dimensions. Error bars for train accuracy and loss are computed as the mean and standard deviation across sliding window of 15 batches. (Right) SO(3, 1) J-equivariant neural network training recognize digits from the MNIST-Live dataset in spatial dimensions. Error bars for train accuracy and loss computed as the mean and standard deviation across sliding window of 15 batches.
from 1an import Liballgebraaeppreeenttiinn sig..geeraggreeettttttiooleceser
2.
\
learned_generators [...] 7
known_gencrators = [...]
learned Lidallgbraaleppreeetttttttoo (earreeoeeeeeeeeeeeeeeeertttoaae
scalar_irrep LiiekllgborRRppreennttttoot
11
numpy. zeros(
12
learned ieren wtt.:irreeerprrr.a..a algeora. dim, 1, 1
))
known_irrep heallghrrkereeeetttttiitttooeeee
16
17
representations = Leelggaackepeeenttttoobeeeessss
scalar _erep,
known_irrep
20
lcarned_irrep,
21
22
[irop, learned_irrqp))
})
25
model LieerropEpniivrrittteeeaalllle representations, num layers=11 num_cchaneess=22
26
Figure Our Lie Algebraic Networks (lan) module handles Lie algebra and Lie group representations, derives Gobbsch-Gordate coefficients for the equivariant layer update, and computes the forward pass. This makes simple to build equivariant point cloud network once the representations are obtained.
15


Reviews:
[{"review_id": "UZxhUTvoeUz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposes a new framework based on noncommutative Lie groups to learn irreductible representations. Such representations can manage many kind of operation like rotation, translation, Lorentz boost... The interest of such representations is important since many application must be insensitive to some modification.\n\n##########################################################################\n\nReasons for score:\n\nOverall, I vote for rejecting (see cons for more details). The framework is very interesting but too complex to have a large audience. Furthermore some clues are missing on how the equivariance are declared.\n\n##########################################################################\n\nPros:\n\n1. This paper proposes a very generic framework for learning equivariant representations. This is of broad interest.\n\n2. Lie groups are able to manage many kind of transformations, thus the framework could lead to new application of deep learning.\n\n##########################################################################\n\nCons:\n\n1. The tensor A (the structure constants) of the algebra seems the main element of the whole framework. The construction of such tensor is unclear and seems non-trivial. For a given set of transformations, how can we derive the structure constants? The authors must explain this point.\n\n2. The equilibrium between appendix and main article is not good. Some figures are cited into the main article while they are in the appendix. On the other side, a large part of the paper is used to introduce the Lie groups. Perhaps a good way to reduce the complexity of he paper would be to take one example as introduction and lets the more formal parts in the appendix. As such the paper is too complex to catch the audience it merits.\n\n3. The experiments are not useful as we don't have any description on how there are done. For example on section 5.1 how the structure constants are given. Are they learnt or estimated?\n\n##########################################################################\n\nQuestions during rebuttal period:\n\nPlease address and clarify the cons above, especially point 1.\n\n#########################################################################\n\nSome typos: \n\npage 4: representation instead of \"represenation\"\n\npage 6: Adam instead of \"adam\"\n\npage 6: please put parenthesis for the Adam paper citation", "extra_scores": {}, "extra_text": {}}, {"review_id": "8lMNXKjPPr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThe paper proposes the algorithm LearnRep that uses gradient descent methods to learn Lie algebras from structure constants, before obtaining the corresponding group representation through the exponential map. The algorithm is tested on SO(3), SO(2, 1), and SO(3, 1). In addition to this, the paper proposes SpaceTimeNet, a Poincaré-equivariant neural network architecture, and applies this architecture to an object-tracking task involving MNIST digits moving uniformly through space.\n\n------------------------------------------\nStrengths and weaknesses:\n\nThe paper proposed a well-motivated algorithm for learning irreducible group representations and performed sensible checks against well-studied Lie groups. The proposed Poincaré-equivariant convolutional network was similarly well-motivated, and the experimental results were promising.\n\nAs it stands, I’m assigning a score of 5. I like the paper and think that it would be a good workshop paper but is not ready for the main conference. The reason for this is that the theoretical contributions, while novel, are not large enough on their own, and the experimentation to support the theoretical contributions are not extensive enough to demonstrate that the theoretical contributions demonstrate a major step forward in terms of functionality.\n\nGoing forward, I think the paper could be improved by including more thorough experimentation for both LearnRep and SpaceTimeNet. Space could also be made for this through a more concise presentation of the background material in the first 5 pages.\n\n------------------------------------------\nQuestions and clarification requests:\n\n1)\tWhy did you choose the norm penalty that you did in equation 6? Did you consider other choices?\n2)\tIn section 2.5 you mention Tensor Product Nonlinearities. Where did you end up using them in the paper?\n \n------------------------------------------\nTypos and minor edits:\n\n-\tPage 2, bottom – “det A = 0” -> “det A = 1”\n-\tPage 3, second paragraph, condition (ii) – “R^3” -> “R^{m+n}”\n-\tSection 2.4, paragraph 1, sentence 1 – “represenation” -> “representation”\n-\tSection 2.4, paragraph 2, sentence 2 – “R^{n_{1}}” -> “R^{n_{1} x n_{1}}” for A matrices and similar for B matrices\n-\tSection 3.1.1, paragraph 3, sentence 4 – “section 2.5” -> “Section 2.5”\n-\tAppendix A.1, paragraph 1, sentence 4 – “A formulae to obtain real-valued representation matrices” -> “A formula to obtain real-valued representation matrices”\n-\tAppendix A.1, end of paragraph 1, “it may be easily checked that Kx , Ky , Jz satisfy the applicable commutation relations from Equation equation 3” -> “it may be easily checked that Kx , Ky , Jz satisfy the applicable commutation relations from equation 3”\n-\tAppendix A.2, paragraph 1, sentence 1 – “Lorentz group defining its action upon the spacetime” -> “Lorentz group defining its action upon spacetime”\n-\tAppendix A.2, paragraph 1, sentence 1 – we need u_{i} in R^{m}, not R^{n}\n-\tAppendix A.2, just before equation 10 - \\kappa(\\rho_{1}(\\alpha) …) -> \\kappa(\\rho_{0}(\\alpha) …)\n-\tAppendix A.3, paragraph 1, sentence 2 – “only if there is a nondegenerate nullspace corresponding to a unique set of Clebsch- For SO(3) and SO(2, 1), …” -> “only if there is a nondegenerate nullspace corresponding to a unique set of Clebsch-Gordan coefficients. For SO(3) and SO(2, 1), …”\n-\tAppendix A.3, paragraph 2, sentence 2 – “allowing for operations such as taking the tensor product of mutliple group representations” -> “allowing for operations such as taking the tensor product of multiple group representations”", "extra_scores": {}, "extra_text": {}}, {"review_id": "2MkO_IgSytp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "==== Summary ====\n\nThis work studies the problem of learning irreducible group representation without prior knowledge, and such algorithm (LearnRep)  is further used to build an object-tracking model (SpacetimeNet), which has guarantee of Poincaré group equivariance.\n\n==== Comments ====\n\n- *Strength*: The topics of learning irreducible representation (irrep) and Poincaré group equivariance look interesting to me. The technique of using optimization for finding irrep is simple and appears to be novel and effective. A complete introduction of the preliminary knowledge about group theory is presented in this paper, which reduces some of the difficultly for reader who is not familiar with this topic.\n\n- *Weakness*: One contribution claimed in this paper is the SpacetimeNet for object-tracking task. So I expect the proposed model can be evaluated through more realistic data sets instead of just MNIST. Also, as a person who is not very familiar with this task, I think it would be helpful to add some illustration on how the dataset is built and how the model is evaluated. Currently the experimental section seems to be difficult to follow due to the lack of explanation. \n\n In addition, I am wondering how LearnRep is properly motivated and analyzed? Specifically, I am not sure why the resulted representation from LearnRep is irreducible. Currently the irreducibility has only been demonstrated through experiments, and there is no motivation about how the loss function is derived. So I think it could help if the author can present some theoretical analysis about the correctness of the learned group representations.\n\n====  Reason for scoring ====\n\nOverall, the topic for finding irrep is important and suitable for presenting in ICLR. My main concern is on the experiment section, which needs more realistic dataset and explanation to demonstrate the importance and effectiveness of the proposed model/method. I think moving most of the contents in Sec.2 to the appendix for space, and adding more experiments and illustrations mentioned above can greatly enhance the soundness of this paper.\n\n \n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work investigates an algorithm to learn representations of Lie groups. It first learns a representation of the Lie algebra by enforcing the Jacobi identity using known structure coefficients. Then obtains the group representation via matrix exponentiation.
The paper also proposes a Poincaré-equivariant neural network, and applies this model to an object-tracking task.
The paper is well-motivated, the derivations could be more clearly presented but are otherwise sound. The experimental results are promising but rather limited in scope at the time.
Decision: Reject