Learning Irreducible Representations of Noncommutative Lie Groups

Abstract: Recent work has constructed neural networks that are equivariant to continuous symmetry groups such as 2D and 3D rotations. This is accomplished using explicit group representations to derive the equivariant kernels and nonlinearities. We present two contributions motivated by frontier applications of equivariance beyond rotations and translations. First, we relax the requirement for explicit Lie group representations, presenting a novel algorithm that finds irreducible representations of noncommutative Lie groups given only the structure constants of the associated Lie algebra. Second, we demonstrate that Lorentz-equivariance is a useful prior for object-tracking tasks and construct the first object-tracking model equivariant to the Poincaré group.

Reviews and author responses:

Thank you for your review and pointing out these typos, and for asking how the structure constants (the entries of $A$) are obtained. It is simple to compute the structure constants. One approach was demonstrated by (Rao and Ruderman 1999) which we cite in section 4.1.
We have added an additional comment to make this clear in our manuscript. Briefly, as long as you can define the Lie group explicitly as a connected matrix Lie group, you can obtain the structure constants directly.

For some more detail, take for example the homogeneous Galilean group $\text{HG}(1,3)$. For this group, the representations are not all known, and yet the structure constants of the Lie algebra are available e.g. on the [wiki page](https://en.wikipedia.org/wiki/Galilean_transformation). This is because the structure constants can be obtained from just a single representation, such as the one we use for everyday physical calculations.
More explicitly, we can generate many random elements of this group, apply the matrix logarithm to obtain elements of the Lie algebra, and find a complete basis of this algebra as a vector space. Then we just have to decompose the commutator of each pair $(i,j)$ of basis elements (indexed by $k$) to obtain the constants $A_{ijk}$.

Thank you for your review and suggestions to improve our paper.

To address the theoretical soundness of the LearnRep algorithm,  we added a concise proof in Section 3.1.1 showing that for sufficiently low $n$, if LearnRep converges to an $n$ dimensional representation then it must be irreducible. The main idea is that for low $n$, combinatorial constraints along with the multiplicative norm penalty of our loss function make it impossible to converge to any other representation. This explains why LearnRep can find, say, the $n=3$ dimensional representation of SO(3). The question remains, why does the loss function converge at all, given that it appears nonconvex? Here, once again for a low $n$, the number of parameters is small ($\propto n^2$); heuristically it is therefore unsurprising that LearnRep converges within finite time.

Our experiments were designed to carefully characterize the behavior of LearnRep and SpacetimeNet.
First, we demonstrated that LearnRep converges to irreducible representations of $\text{SO}(3,1), \text{SO}(2,1),$ and $\text{SO}(3)$. To this end we created a novel technique to characterize and visualize the tensor product structure of general Lie group representations.
We then produced artificial 3D and 4D point cloud datasets based on MNIST, which contain just 64 points per sample and are a toy model of event camera or LIDAR data. We trained the first Poincaré-equivariant object tracking models on these datasets and showed they can obtain reasonable accuracy while maintaining a rigorous guarantee of motion-equivariance.
These experiments provide motivation to scale up the SpacetimeNet architecture to real datasets with $\sim 10^3$ points per sample. This will require overcoming engineering hurdles, e.g. there is currently a $O(N^2)$ complexity of the forward pass when there are $N$ points per sample. They also motivate applying LearnRep to Lie groups whose representations are uncharacterized, such as the homogeneous Galilean group.
Both the LearnRep algorithm and the SpacetimeNet model break new ground, and our experimental results provide a solid foundation for the future directions mentioned above.

As you rightly pointed out, some important figures in the appendix ought to be available in the main text. With the additional page we plan to move the tensor product structure figure to the main text.

Thank you very much for the review and especially the list of typos and minor edits.

To answer the questions that you raised:

Why did you choose the norm penalty that you did in equation 6? Did you consider other choices?

This is a great question and we added a bit of additional explanation around this point. The norm penalty of course causes the loss function to diverge at the trivial representation (all 0 matrices). However, it also exhibits other divergences whose importance we did not emphasize. Briefly, the maximum over $T_i$ gives a divergence whenever any $T_i$ approaches the 0 matrix, preventing the formation of any representation which is trivial for a nontrivial subgroup. For instance, $\text{SO}(3)$ is a subgroup of $\text{SO}(3,1)$, so without this property the learned representations may just be $\text{SO}(3)$ representations.


In section 2.5 you mention Tensor Product Nonlinearities. Where did you end up using them in the paper?

Tensor product nonlinearities are used in the forward pass of SpacetimeNet (section 3.2). The layer update rule for the representation at some spacetime point x includes a term for each other point $y$, which is the (decomposed) tensor product of representations at $y$ with (a representation equivariant to) the displacement vector $y-x$. We have tried to make this more clear in section 3.2.

We will upload a revised draft shortly to fix these typos and add some clarification.


##########################################################################

Summary:

This paper proposes a new framework based on noncommutative Lie groups to learn irreductible representations. Such representations can manage many kind of operation like rotation, translation, Lorentz boost... The interest of such representations is important since many application must be insensitive to some modification.

##########################################################################

Reasons for score:

Overall, I vote for rejecting (see cons for more details). The framework is very interesting but too complex to have a large audience. Furthermore some clues are missing on how the equivariance are declared.

##########################################################################

Pros:

1. This paper proposes a very generic framework for learning equivariant representations. This is of broad interest.

2. Lie groups are able to manage many kind of transformations, thus the framework could lead to new application of deep learning.

##########################################################################

Cons:

1. The tensor A (the structure constants) of the algebra seems the main element of the whole framework. The construction of such tensor is unclear and seems non-trivial. For a given set of transformations, how can we derive the structure constants? The authors must explain this point.

2. The equilibrium between appendix and main article is not good. Some figures are cited into the main article while they are in the appendix. On the other side, a large part of the paper is used to introduce the Lie groups. Perhaps a good way to reduce the complexity of he paper would be to take one example as introduction and lets the more formal parts in the appendix. As such the paper is too complex to catch the audience it merits.

3. The experiments are not useful as we don't have any description on how there are done. For example on section 5.1 how the structure constants are given. Are they learnt or estimated?

##########################################################################

Questions during rebuttal period:

Please address and clarify the cons above, especially point 1.

#########################################################################

Some typos: 

page 4: representation instead of "represenation"

page 6: Adam instead of "adam"

page 6: please put parenthesis for the Adam paper citation

==== Summary ====

This work studies the problem of learning irreducible group representation without prior knowledge, and such algorithm (LearnRep)  is further used to build an object-tracking model (SpacetimeNet), which has guarantee of Poincaré group equivariance.

==== Comments ====

- *Strength*: The topics of learning irreducible representation (irrep) and Poincaré group equivariance look interesting to me. The technique of using optimization for finding irrep is simple and appears to be novel and effective. A complete introduction of the preliminary knowledge about group theory is presented in this paper, which reduces some of the difficultly for reader who is not familiar with this topic.

- *Weakness*: One contribution claimed in this paper is the SpacetimeNet for object-tracking task. So I expect the proposed model can be evaluated through more realistic data sets instead of just MNIST. Also, as a person who is not very familiar with this task, I think it would be helpful to add some illustration on how the dataset is built and how the model is evaluated. Currently the experimental section seems to be difficult to follow due to the lack of explanation. 

 In addition, I am wondering how LearnRep is properly motivated and analyzed? Specifically, I am not sure why the resulted representation from LearnRep is irreducible. Currently the irreducibility has only been demonstrated through experiments, and there is no motivation about how the loss function is derived. So I think it could help if the author can present some theoretical analysis about the correctness of the learned group representations.

====  Reason for scoring ====

Overall, the topic for finding irrep is important and suitable for presenting in ICLR. My main concern is on the experiment section, which needs more realistic dataset and explanation to demonstrate the importance and effectiveness of the proposed model/method. I think moving most of the contents in Sec.2 to the appendix for space, and adding more experiments and illustrations mentioned above can greatly enhance the soundness of this paper.

 

 

Paper summary:

The paper proposes the algorithm LearnRep that uses gradient descent methods to learn Lie algebras from structure constants, before obtaining the corresponding group representation through the exponential map. The algorithm is tested on SO(3), SO(2, 1), and SO(3, 1). In addition to this, the paper proposes SpaceTimeNet, a Poincaré-equivariant neural network architecture, and applies this architecture to an object-tracking task involving MNIST digits moving uniformly through space.

------------------------------------------
Strengths and weaknesses:

The paper proposed a well-motivated algorithm for learning irreducible group representations and performed sensible checks against well-studied Lie groups. The proposed Poincaré-equivariant convolutional network was similarly well-motivated, and the experimental results were promising.

As it stands, I’m assigning a score of 5. I like the paper and think that it would be a good workshop paper but is not ready for the main conference. The reason for this is that the theoretical contributions, while novel, are not large enough on their own, and the experimentation to support the theoretical contributions are not extensive enough to demonstrate that the theoretical contributions demonstrate a major step forward in terms of functionality.

Going forward, I think the paper could be improved by including more thorough experimentation for both LearnRep and SpaceTimeNet. Space could also be made for this through a more concise presentation of the background material in the first 5 pages.

------------------------------------------
Questions and clarification requests:

1)	Why did you choose the norm penalty that you did in equation 6? Did you consider other choices?
2)	In section 2.5 you mention Tensor Product Nonlinearities. Where did you end up using them in the paper?
 
------------------------------------------
Typos and minor edits:

-	Page 2, bottom – “det A = 0” -> “det A = 1”
-	Page 3, second paragraph, condition (ii) – “R^3” -> “R^{m+n}”
-	Section 2.4, paragraph 1, sentence 1 – “represenation” -> “representation”
-	Section 2.4, paragraph 2, sentence 2 – “R^{n_{1}}” -> “R^{n_{1} x n_{1}}” for A matrices and similar for B matrices
-	Section 3.1.1, paragraph 3, sentence 4 – “section 2.5” -> “Section 2.5”
-	Appendix A.1, paragraph 1, sentence 4 – “A formulae to obtain real-valued representation matrices” -> “A formula to obtain real-valued representation matrices”
-	Appendix A.1, end of paragraph 1, “it may be easily checked that Kx , Ky , Jz satisfy the applicable commutation relations from Equation equation 3” -> “it may be easily checked that Kx , Ky , Jz satisfy the applicable commutation relations from equation 3”
-	Appendix A.2, paragraph 1, sentence 1 – “Lorentz group defining its action upon the spacetime” -> “Lorentz group defining its action upon spacetime”
-	Appendix A.2, paragraph 1, sentence 1 – we need u_{i} in R^{m}, not R^{n}
-	Appendix A.2, just before equation 10 - \kappa(\rho_{1}(\alpha) …) -> \kappa(\rho_{0}(\alpha) …)
-	Appendix A.3, paragraph 1, sentence 2 – “only if there is a nondegenerate nullspace corresponding to a unique set of Clebsch- For SO(3) and SO(2, 1), …” -> “only if there is a nondegenerate nullspace corresponding to a unique set of Clebsch-Gordan coefficients. For SO(3) and SO(2, 1), …”
-	Appendix A.3, paragraph 2, sentence 2 – “allowing for operations such as taking the tensor product of mutliple group representations” -> “allowing for operations such as taking the tensor product of multiple group representations”

Meta-review: This work investigates an algorithm to learn representations of Lie groups. It first learns a representation of the Lie algebra by enforcing the Jacobi identity using known structure coefficients. Then obtains the group representation via matrix exponentiation.
The paper also proposes a Poincaré-equivariant neural network, and applies this model to an object-tracking task.
The paper is well-motivated, the derivations could be more clearly presented but are otherwise sound. The experimental results are promising but rather limited in scope at the time.

Decision: withdrawn-rejected-submissions