Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: **Summary and Contributions**:

The paper proposes partial group convolution that can learn layer-wise partial equivariance from data. The main idea is to use the Monte Carlo approximation to the group convolution as in LieConv (Finzi 2020), but sample group elements from a learned distribution over groups rather than a uniform one.

They empirically show that allowing the network to learn partial equivariance improves performance upon conventional G-CNNs when full equivariance is harmful, and still match the performance of G-CNNs when full equivariance is beneficial.

main_review: **Strengths**:

- Partial equivariance is an interesting yet under-explored problem. Existing G-CNN literature mainly focuses on full equivariance, and partial equivariance is seldom explored. This paper has demonstrated that studying partial equivariance is of practical relevance: It can improve performance upon conventional G-CNNs when full equivariance is harmful.
- The proposed partial group convolution is a simple yet effective solution. This paper proposes a solution that only requires a small modification of existing G-CNN architectures, and the authors have empirically shown that the solution is effective.
- Clear messages from the experiments. I like the experiments section, because the take-away messages from these experiments are very clear, and the main claims of this paper are empirically well supported: 
  - (Figure 4) Different layers may require different levels of equivariance.
  - (Table 2) Partial G-CNNs improve performance when full equivariance is harmful, e.g. CIFAR10, CIFAR100, and still match the performance of G-CNNs when full equivariance is beneficial, e.g. rot-MNIST
  - (Table 4) SIREN kernel parameterisation is significantly better compared to other choices.

**Weaknesses**:

My main concern of this paper is its presentation and technical statements, which is detailed below:
- Misleading terminology/notations. Even though the overall idea is simple and clear to me. I still find the technical parts very hard to follow and check their correctness, mainly due to the somewhat misleading usage of terminology and notations. Below are a few examples:
  - The input/output space of group convolutions are functions on groups rather than the groups themselves, and the groups are the input space of a particular input/output function. This paper very often does not distinguish the two, hence causing confusion.
  - Does $p_i$ in Section 4.3.1 (the discrete group case) define a categorical distribution over groups? If that is the case, then the probabilities should sum to 1? But then the paper set all the probabilities $p_i$ to $1$. I might misunderstand something here.
- Unclear illustration: Figure 1 and Figure 2 are hard for me to understand. A few suggestions to improve clarity:
  - Provide a mathematical definition of equivariance error.
  - In Figure 1, images on the right are feature maps (functions on groups) rather than group elements. So better not annotate them as group elements and group subsets, or explain what’s the correspondence between group elements and feature maps?
Images in Figure 2 might be too small.
  - The text explanation for these two figures also needs lots of work.

 
**Questions**:

- If there is an image dataset, for some of the data, the corresponding group subset is $[-60, 60]$, for the rest, the corresponding group subset is $[0, 90]$, what would the overall learned group subset be?
- In the definition of partial group convolution, can we constrain the distribution over groups to be a uniform distribution over a subset? That is to say, group elements are either in or out of the subset, and all elements in the subset are equally likely. For me, it seems to be a more ‘natural’ definition of ‘partial’ group convolution, unless there are cases when you would prefer a more flexible distribution $p(g)$?

I will give a negative score for now because the technical part needs significant work before publishing. But I think it still solves an interesting problem with a simple solution, and the experiments are sufficient. If the authors are willing to give a list of changes they will make to the mathematical formulation and statements, I am happy to raise my score to the positive side.

summary_of_the_review: The paper studies an interesting problem proposes a simple solution and has sufficient empirical support for main claims. However, the technique parts and over clarity needs significant work before publishing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to learn partial equivariances from data by training a group equivariant CNN (G-CNN) where the group elements are pruned at each layer. For learning the pruning operation, the paper leverages the idea of Finzi et al [4] to approximate the group convolution with a Monte Carlo integral, and the idea of Benton et al [3] to use the reparametrization trick to learn the bounds of an uniform distribution from where the group elements are sampled. Results show that the partial equivariance performs outperforms the full equivariance when the data itself is not fully rotational-equivariant. 
main_review: # Strengths

The paper addresses the important problem of learning symmetries from data. One of the weaknesses of group equivariant CNNs is that one has to assume what is the group that acts on the dataset, which might be hard to know in practice. Commonly, the chosen group will be too restrictive and still miss relevant transformations. The paper proposes a way to start from some group of transformations and learn to prune it differently at each layer. This is potentially more flexible and is shown to outperform the baselines where the original group is fixed throughout. 

# Weaknesses

If I understand correctly, the partial equivariance as defined is an approximate equivariance to a subset of the group. In this case, the average along the group dimension will not be invariant to transformations from the subset. Furthermore, once equivariance is broken in some layer, it can never be recovered, so the effect of this approach is to monotonically restrict equivariance on each subsequent layer.

On one hand, this is satisfying because the benefits of reducing or breaking equivariance at some mid or final layer have been observed before (eg the group restriction in Weiler and Cesa [1]). On the other hand, what the method can learn from data seems to be straightforward to implement with either a group restriction or including coordinate values on feature channels like in Fuchs et al [2], while the proposal seems also prone to converging to undesirable models such as breaking equivariance too early or having a more restrictive layer following a less restrictive.  

1) Figure 4 shows what I would expect as a reasonable output, where the equivariance is broken at the final layers to make use of the input pose. Is the same observed for CIFAR10/100? Please report similar figures for the SE(2) case.
   
2) Results would be more convincing if compared with the aforementioned handcrafted methods of breaking equivariance. How would something simple like dropping from E(2) to T(2) on the last layer, or using a fully connected layer instead of the final global max pooling perform on CIFAR10/100? These would highlight the advantages learning partial equivariances per layer. 
   
3) The Monte Carlo approximation of the group convolution integral causes some equivariance error. The group pruning to achieve partial equivariance compounds on this error, even when only the group subset is considered. I appreciate the error expressions in Appendix B, but I believe such errors should be quantified and compared with the baselines (both non-equivariant and fully equivariant). Without these numbers it is hard to judge whether (or at what layer) we can consider the equivariance approximate or broken. 

I have a separate concern regarding novelty. The paper seems heavily inspired by Benton et al [3], who, despite focusing on learning invariances from data, also show an extension to learning global equivariances. I believe the differences and similarities between both approaches should be discussed more thoroughly and that quantitative comparison under fair conditions is also necessary. Benton et al [3] also show results for rotated MNIST and CIFAR10. 
   
# Questions

1) Please elaborate on which discrete groups are unstable to train as mentioned in section 6. Table 2 shows the best results for E(2) G-CNN, which has a discrete component. Was the instability observed in this model or others?

# Minor

Figures 1 and 2 are hard to read. It would be much easier to compare the feature maps and see the differences if they were rotated to the same pose. 

# References   

[1] Weiler and Cesa, "General E(2) - Equivariant Steerable CNNs", NeurIPS'19.

[2] Fuchs et al, "SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks", NeurIPS'20.

[3] Benton et al, "Learning Invariances in Neural Networks", NeurIPS'20.

[4] Finzi et al, "Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data", ICML'20.

summary_of_the_review: I think the paper addresses an important problem and proposes a seemingly reasonable method that might be useful in practice. However, in the current state I'm not sure if there is enough evidence to demonstrate the value of the paper (see detailed review). I would be happy to increase my rating if such evidence is presented during the discussion phase.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn partial equivariance where the equivariance property is constrained to a subset of the considered group. Specifically, they propose to learn such subset instead of predefining it, by parameterizing the sampling probability of the MC sampling. Experiments on variants of MNIST and CIFAR10 demonstrate that the partial equivariance brings better performance than whole group equivariance in terms of classification accuracy.
main_review: ## Strengths
  1. The writing is clear and easy to follow. The relationship with relevant works is discussed sufficiently.
  2. The proposed partial equivariance is novel and several different cases of the subsets are properly discussed (e.g., group to subset, subset to group, etc.)
  3. Experiments show certain improvements compared with enforcing equivariance on the whole group.

## Weaknesses
  1. One major concern is that the partial equivariance seems not motivated properly. It is not clear why and when the partial equivariance will be needed in the natural datasets. All experiments on MNIST are artificially constructed in favor of the partial equivariance, and the experiment on CIFAR only show marginal improvement on classification accuracy which cannot prove that the CIFAR dataset demonstrates partial equivariance property. More analysis/visualization/experiments/concrete examples should have been included to motivate the need for partial equivariance and show its specific benefits. 
  2. Another concern is related to the first one: The experiments are conducted only on small datasets and are artificially constructed in favor of the partial equivariance. More experiments on larger and natural datasets are needed.
summary_of_the_review: In summary, although the proposed partial equivariance is novel, it's not well motivated and the current experiments/analysis do not prove the benefits of partial equivariance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the notion of partial equivariances which are defined as being equivariant with respect to a subset of a group. The authors also propose to learn the domain of this subset on the fly as opposed to enforcing this prior to training. The authors validate their proposed model on simple datasets that require different degrees of equivariance such as RotationMNIST, CIFAR10, and CIFAR100. 
main_review: **Strengths**
The main strength of the paper is that idea is conceptually simple and well executed. Indeed, enforcing strict equivariances is often a strong constraint in learning problems and as a result the ability to toggle the amount of equivariance needed should prove beneficial. The writing in this paper is also well done for the most part and the overall presentation is of high quality.

**Weaknesses**
In my opinion there are a few weaknesses in this paper that I believe the authors could address. 

***Presentation Issues:***
Starting with presentation, I found the notation to be unclear or inconsistent at times. For example, in Eqn 3. $f$ as previously defined is a map from $G\to \mathbb{R}$ but $v \in X$. The lifting operation as a result is bit confusing. I'm aware of the lifting operation as defined in Kondor & Trivedi 2018 [1], can you please comment whether this is the same thing being done here? Also, Eqn 3. assumes a continuous/infinite group and discrete/finite groups this integral becomes a sum. Although this is obvious, I feel like the authors should mention this point as the first G-Conv paper was on finite groups. With regards to Fig 1 & 2, I tried my very best but I still couldn't discern what to look for. I appreciate the authors outlining the important samples in red but still it wasn't clear to me. I feel like a better and fine grained qualitative description is needed here as at the moment it takes too much effort to understand what should be a simple figure. Moreover, proposition 4.1 is not novel but the current presentation is a bit ambiguous on this point. I'm sure the authors will agree that this has been proven in other work (e.g. [2]) so it would be necessary to make this more explicit and cite the correct papers. Finally, with regards to the experiments a quantification of the error is needed (e.g. +/- with a statistical test) as this is standard protocol. 

***Technical Issues:***
This paper has a few other technical issues. First, it seems like a large body of Steerable G-CNNs are missed. This is important as the equivariance is guaranteed and not in expectation like in computing the integral. Furthermore, from the experiments it is unclear if the authors used Steerable G-CNNs for their baselines. I suspect that this is not the case as the results do not match [3] for RotationMNIST and CIFAR10. Also, do note that in [3] the authors achieve a variable level of equivariance at different depths which is in line with partial equivariance as proposed here. Admittedly, these are not the same but a similar idea was attempted in [3] and proved useful this diminishes the novelty of this work. If the baselines are indeed not G-Steerable with mixed equivariance then I feel like this is a must have and is quite a relevant comparison. Lastly, I found the quantification of the equivariance error an interesting discussion worthy of the main paper. However, I do not get the main point here as breaking exact equivariance to the whole group is desirable which is the premise of this paper. What should a reader take away from this error quantification if the error is not a bug but a feature? Finally, if I understood correctly the paper claims that we are learning a distribution over a subset but it seems like it is always the uniform distribution and what we instead learn is the domains or support of this distribution over the group. Is there a case where it would be beneficial if the learned distribution is not uniform?


[1] Kondor, Risi, and Shubhendu Trivedi. "On the generalization of equivariance and convolution in neural networks to the action of compact groups." International Conference on Machine Learning. PMLR, 2018.

[2] Cohen, Taco, Mario Geiger, and Maurice Weiler. "A general theory of equivariant cnns on homogeneous spaces." arXiv preprint arXiv:1811.02017 (2018).

[3] Weiler, Maurice, and Gabriele Cesa. "General $ E (2) $-Equivariant Steerable CNNs." arXiv preprint arXiv:1911.08251 (2019).
summary_of_the_review: Overall the paper is of high production quality but misses the mark on claimed novelty. The biggest potential weakness is the omission of G-Steerable CNNs as baselines which are quite relevant here. There are a few minor presentation issues but these can be fixed with a revision.
review_title: null

Meta-review: The paper proposed learning partial and full equivariances from data in an end-to-end way. The problem studied is an important issue of existing equivariant neural networks which always assume full equivariance. However, the paper only got 3 "marginally below threshold" and 1 "marginally above threshold" even after rebuttal. The major challenges include technical parts being hard to follow due to multiple reasons, unsatisfactory paper writing (the updated version has undergone restructuring), the issue of "breaking equivariance" after multiple layers, some important experiments (such as comparing with Steerable G-CNNs) missing, etc. After rebuttal only reviewer RivC raised his/her score to "marginally above threshold". Such scores are difficult to justify acceptance. The AC appreciated the authors for making great efforts on rebuttal and revising the manuscript. However, from the review comments it is clear that the paper still needs further revision (many of those clarified for the reviewers, e.g., explaining distribution for the discrete groups and adding more experiments, should be included in the manuscript). So the AC deemed that the paper is not ready for publication.
Decision: Reject