Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an invariant and equivariant unsupervised learning method, i.e., an invariant and equivariant autoencoder.
When considering such a problem, the naive problem is the decoder construction.
This is because when the invariant model maps to an intermediate space, the group action is already obvious in that space, and the information about the group has disappeared.
The method used in this paper avoids this problem by adding group labels to the intermediate space, and from their experimental results with rotated MNIST and others, it appears that this method indeed achieves a better representation than the usual auto-encoder.
strengths_and_weaknesses: Strengths
This paper is based on a natural conception and solves the problem of invariant auto encoders.

Weaknesses
The proposed method and solution are too natural to be non-trivial.
It would be better if situations in which expressions created in this way would be useful could be shown.
questions: See Weaknesses.
limitations: Not applicable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Designs an autoencoder that projects each input element into a pair of vectors: one that is group-invariant, and one that represents the group action necessary to recover the original vector from the decoded group-invariant representation.  The group action encoder is designed as a learnable map followed by a deterministic map; the consequent of proposition 2.7 is a sufficient condition to guarantee that this construction results in a suitable group function.  Learnable maps are proposed (and proven suitable) for 2d and 3d rotation groups, permutation groups, and translation groups.
strengths_and_weaknesses: Experimental examples show that the proposed encoder results in group-invariant representations with very high classification accuracy and very high cluster purity for rotation, permutation, and rotation+permutation+translation groups.

Obviously, many other papers have proposed group-equivariant and group-invariant autoencoders.  This paper seems, to me, to go significantly beyond those previous proposals, because the consequent of Proposition 2.7 provides a sufficient condition for the design of an autoencoder with separable group-invariant and group-action representations.

It's interesting that Prop. 2.7 says that if mu is G-equivariant, then G_x=G_\mu, but only the consequent is necessary in Prop. 2.9 to prove that \xi\circ\mu is a suitable group function.  As a result of affirming the consequent in this way, we never get a proof that a suitable group function is also G-equivariant.  It seems intuitively to be the case, but it is not proven, nor, perhaps, necessary.

questions: The notation is challenging, and its challenges are exacerbated by typos and undefined terms.  For example:

GL(X) is never defined.

On line 82 of p. 3, the application of f_X to g and x is written as f_X(g)x, but in equation (1), it is written as f(\rho_V(g^{-1})x) -- I think that in both cases, the x should be inside the parentheses.

A map \phi is said to be G-equivariant if \psi(...) = ...\psi -- either the \phi should be a \psi or vice-versa

we wish to learn the invariant map \phi_{inv}, thus... --- but \phi_{inv} is never mentioned in the definition on the next line

Just in general, I didn't understand the purpose of the \rho_X(g) notation.  If I had to guess, I would say that it takes a general representation, g, of a member of the set G, and converts it into the instantiation of that member that is applicable to external dataset X?  The definition of this notation in the text is self-contradictory and unhelpful: a group action is defined as f:G,X -> X, then a representation is defined as (1) a group action, but also (2) as a function f:X -> GL(X), which seems to be a contradiction.  Neither (1) nor (2) seems to fit with the way \rho_X(g) is used in the rest of the article.  


limitations: Limitations and ethical implications are not discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: An autoencoder architecture that is based on equivariant and invariant neural layers is proposed. The most important aspect of the architecture is that the encoder stage can map the input to a guaranteed group invariant feature and a group action such that the decoder operates on the group invariant feature and produces the reconstruction of the input in a canonical form. The predicted group action is then applied on this canonical decoder output to reconstruct the input exactly (assuming the autoencoder is trained well). Experiments on images and point clouds are performed for a variety of groups.
strengths_and_weaknesses: Strengths:

1. The method provides guaranteed invariance to the transformation group.
2. The method is general enough to apply to both discrete (such as permutations) and continuous (such as rotations and translations) groups, and to a diverse set of domains -- images, point clouds. 
3. Experiments show that the network is indeed able to learn a group invariant and the corresponding group action when trained with the reconstruction loss alone.


Weaknesses:

1. Missing related work: I think the main weakness in the paper is that the authors have missed plenty of related work in literature. I give some examples here of works which consider unsupervised learning of group invariant representations:

(a) Manifold Learning in Quotient Spaces: https://openaccess.thecvf.com/content_cvpr_2018/papers/Mehr_Manifold_Learning_in_CVPR_2018_paper.pdf

(b) Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance: https://www.ecva.net/papers/eccv_2018/papers_ECCV/papers/Zhixin_Shu_Deforming_Autoencoders_Unsupervised_ECCV_2018_paper.pdf

(c) Rate-Invariant Autoencoding of Time-Series: https://ieeexplore.ieee.org/abstract/document/9053983

In (a), group invariant features are learned in an unsupervised fashion, and a group-invariant loss function is used. Similar ideas have been applied for rotation-invariant autoencoders for spherical images as well. In (b) and (c), the invariant feature and the group transformation are disentangled in an autoencoder which is closely related to this submission. In (b) and (c), the ideas are applicable to much larger groups than considered in the present submission as well -- diffeomorphisms -- which I believe cannot be handled by the method presented submission, at least for now. 

Given these works, I think the authors should significantly improve the related work section. They should put this submission in context of such works and better explain the main advantages their method provides.

2. Missing comparison with all these related work: I do think that the papers I have cited and the ideas therein are closely related to the spirit of this work. And thus, the authors should provide some experimental comparisons to one or more of these works.

3. MNIST and Tetris are toy datasets in my opinion. At least some experiments with more complex datasets like CIFAR-100, Patch Camelyon or ShapeNet are needed to better validate the effectiveness of the method. Authors do not provide quantitative measures of reconstruction quality for all experiments, which makes reproducibility hard. Also, non-equivariant baselines and corresponding quantitative evaluation are useful to compare the proposed method and to find out whether indeed the group invariant and equivariant network can be significantly smaller for the same reconstruction quality. 

Overall, I think there are too many weaknesses in this submission and am recommending a rejection. 


## Post author response

I think the authors have addressed all my questions well. They have also addressed the weaknesses pointed out by the other reviewers well. The additional experiments and comparison with the QAE baseline make the contributions of the paper stronger. 

I hope the authors will expand their related work section and put the work in context of several other closely related papers (I only mentioned three, but there are others). I don't fully agree that because some of the papers can also model local transformations and this submission can handle only global transformations, that these papers are not closely related. In fact, they point to some limitations of the current work. For example, how can this submission be extended to handle infinite-dimensional groups which are important in image and time series registration? I am now recommending a weak accept.
questions: The authors should discuss what, if any, are the downstream applications of such a method. Is it data compression? Or do they envision the group invariant features learned by their method to be useful for high-level tasks like robust classification?

Related to reference (a) above, if the eventual goal is to use these group-invariant features for tasks like classification, I wonder why we need to find the group action at all. If we have a group-invariant latent feature, we can use a decoder and a group invariant reconstruction loss for training the autoencoder. 

I do hope that the authors address the weaknesses in the paper. Particularly, I would like them to clarify the contributions of the paper better, especially in light of other closely related work already in literature. Experiments need to be improved as well, as I have indicated.
limitations: The authors have not discussed the limitations of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Equivariant neural networks have been shown to improve data efficiency and model performance, but require knowledge about the group acting on the data space to be implemented. This work promises to learn the group action from data instead, foregoing the need for ad hoc implementations of the group action on the feature maps / data space. To this end, authors propose learning two separate mappings: (1) from the input sample to a group-invariant canonical representation and (2) from the input sample to a group element / group action. Together, these allow for reconstructing the input sample under its observed pose. The authors show, in a range of experiments, that their framework is able to learn to successfully reconstruct input samples under their respective poses, from an invariant embedding and a learned group action.
strengths_and_weaknesses: **Originality and Significance** The authors propose a sound method for implementing group equivariant auto-encoders. I have some concerns regarding claims the authors make throughout regarding learning the group action. If my understanding is correct, this method only "learns" the group action on the data space in the sense that the $\mu$ operator learns a map from any point in the data space $x \in X$ into an embedding space $Y$, where this map is required to be equivariant and $Y$ is required to be a homogeneous space of the group. Next, an operator $\xi$ essentially maps this homogeneous space into the left-regular group action. The whole point of this encoder is to extract pose, but since you presuppose the encoder to be equivariant to the group action, is this not trivial?

Since $\mu$ is equivariant to the group action it inherently preserves pose, i.e. prop 2.7 is a property of any operator equivariant to the group action, and therefore the group action isn't actually "learned"; its structure in the embedding space is given by the equivariant operator. Furthermore, this separation between equivariant and invariant representations hardly seems like a contribution; as you yourself mention, most applications of equivariant neural networks learn equivariant representations which are then projected to an invariant representation. I don't fully understand the need for learning separate encoders $\eta$ and $\mu$. Can you not reuse the same embedding in both cases and simply apply a symmetric operation to $\mu(x)$ obtain to obtain $z$? In fact, for experiment 5.1 you mention that you use the same architecture for both pipelines, why not share the representation?

I would like to see the authors note on how this work relates to Keller & Welling (2021). The methods seem related in that both authors propose learning group equivariant latent representations from data without explicitly encoding the group action.

**Quality and Clarity** The authors explain their derivations in thorough and clear manner. In section 3 I'm left with some questions. [line 186, 187] "*we turn to describing our construction of $\xi, \mu$ and $Y$ for a number of groups*". For ${\rm SO(2)}$ you only describe how to construct $Y$ and $\xi$, what about $\mu$? For ${\rm SO(3)}$ you only describe $Y$, what about $\xi, \mu$? Same for the rest of the examples. Could you address this?

$\xi$ needs to be constructed by hand, i.e. a new group requires a new implementation of $\xi$ AND a new implementation of $\eta$ and $\mu$ because of new equivariance constraints. There seems to be no actual learning of the group action and although you claim your method "avoids the necessity of ad-hoc group-specific implementations" it does in fact require such group-specific implementations to be implemented itself?

From my view, the experiments used by the authors to investigate and evaluate their approach do not adequately show its merits. For details on my concerns, see below questions 4-7. The experiments do not convince me of the need for separate invariant-equivariant terms. To this end, could the authors, in their experimental setups, compare with a fully equivariant encoder-decoder?

*References*

Keller, T. A., & Welling, M. (2021). Topographic vaes learn equivariant capsules. Advances in Neural Information Processing Systems, 34, 28585-28597.
questions: 1. I would like to see the authors address my concern regarding their claim on avoiding the necessity of ad-hoc group-specific implementations. Is it not correct that you need such group-specific implementations of $\eta, \mu$ and $\xi$ in your framework? If so, how does your framework reflect the claim that you are "learning the group action"?

2. I don't fully understand the need for learning separate encoders $\eta$ and $\mu$. Can you not reuse the same embedding in both cases and to obtain $z$ simply group invariantly project over the equivariant embedding?

3. I'm not completely clear on the need for separating equivariant and invariant representations in the first place, could the authors expand on this a bit? Is it simply because we want to place no constraints on the decoder $\delta$ that we discard information on group action over $X$ in the $\eta\rightarrow\delta$ pathway? But if we want a latent space structured according to a group action, can we not simply use an equivariant encoder and decoder?

4. Regarding the experiments, in 5.1 you mention that, upon inspection, the encoded latent representation is indeed rotation invariant up to machine error for 90 deg rotations, and shows interpolation artefacts otherwise. But this is just a consequence of the method of interpolation you are using to rotate the images and the encoder (SO(2)-CNN) you are using, and isn't necessarily a result of your framework correct? You use an encoder which is invariant, hence the learned representation is invariant. I also believe the invariance property of your encoder completely explains the rotation invariant structure of your latent space. To me this doesn't seem a result of your framework, but of the (group-specific) encoder you are using?

5. For the set of digits experiment in 5.2, the experimental setup is not clear to me. Is the task permutation invariant or equivariant? Do you only evaluate the presence of digits in the reconstruction or their order as well? If you are only evaluating the set contents, is the equivariant pipeline discarded for this experiment? Or, if you are evaluating their order as well, I think Fig 4a is misleading, as in addition to the embedding your framework needs the permutation matrix for reconstruction (which is of larger dimensionality than the embedding).

6. In experiment 5.3 you investigate a SE(3) and S_N autoencoder. From line 322, it seems you are only evaluating your reconstruction up to to a translation and permutation (you are only predicting a rotation with $\psi$)? So the autoencoder as a whole is invariant to permutations and translations and only equivariant to rotations, correct? In which case, why didn't you make it invariant to translations and permutations as well? As your autoencoder is invariant to translation, why do you apply translation augmentations to the input?

7. In line 330 you note that your SE(3)-equivariant model (which it is not, it is only SO(3)-equivariant) can distinguish between mirrored chiral shapes, and that SE(3)-invariant models are unable to do so. I believe this is incorrect; reflections are part of the E(3) group, and not of the SE(3) group, hence SE(3)-invariant models should be able to distinguish between these shapes. Could you comment on this?


**General post-rebuttal response**
I would like to thank the authors for their very thorough rebuttal, addressing most of my concerns, and the concerns raised by the other reviewers. I think the work significantly improved in experimental evaluation and comparison. As indicated in my last comments, parts of the motivation and framing of the model proposed by the authors remains misleading / incorrect to me. I would recommend that, were this work to be accepted, the authors review some of the claims they make in the paper regarding the "learning of a group action". I feel such claims should be addressed before publication.

The authors did address many of my questions regarding the broad motivation for their invariant-equivariant approach, but from my point of view, the authors skip a step by discarding fully invariant aggregation of equivariant self-supervised representations. I would recommend the authors empirically or theoretically show that such representations don't work in practice, as motivation for their (arguably more complex) approach. 

On the condition of inclusion of their responses in an updated version of the manuscript, I raise my recommendation to a borderline accept.


limitations: I did not see the authors address any limitations of their work.
review_title: null

Meta-review: metareview: The paper proposes an auto-encoder that maps a point to a canonical point and  to a group element such that the composition of the group and the canonical point reconstructs the point (the invariance / equivariance is in this sense).   The method is promising since it learn in an unsupervised way the group actions. Experiments were on simple tasks and group actions. Reviewers were overall positive and the paper improved a lot during rebuttal/ revision thanks to their recommendations.  

I have one recommendation to the authors to include the use of the representation learned in a classification task to see how the learned representation alleviates the need of large training samples. 

Accept 
 
confidence: Certain
Decision: Accept