Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper describes a focussed analysis of non-linearities to be used with steerable (Fourier based) implementations of equivariant neural networks. The authors show how to get exact equivariance with polynomial activation functions and provide guidelines on how to use general activation functions via the route [Inverse Fourier > activation func > Fourier]. Their theoretical insights are confirmed by experimentation.
main_review: The paper is very well written (up to several typos, see minor comments below) and has a high tutorial value. The paper also comes with code, which further increases its usefulness.

While most of the notions addressed in this paper (equivariance and numerical artifacts due to band-limiting or discretization) have been addressed in several other works, it is very valuable to have the essentials in one place and clearly presented. The citations to other works seem accurate and appropriate.

The experiments are sound, though perhaps somewhat limited to easy problems such as MNIST. I can very well imagine that conclusions drawn on MNIST do not generalize to other datasets because the the dataset is so simple. This is my main concern, and it could be addressed by validating on a more representative dataset (STL10, tiny image net, ...) in which equivariance is I think also more important. 

Finally, after reading the paper I am missing a short discussion on the relevance of approximate vs exact equivariance in the context of performance. Namely, ReLU based methods still outperform the exact activation functions. What does this mean? Apparently it is OK to be not perfectly equivariant, but does this mean ReLU is still the best activation function, or does it mean equivariance errors can be exploited to get better performance? Since the main focus of the paper is on achieving exact equivariance through appropriate activation functions I feel a short discussion regarding the above is in place.

All in all, I think this is a valuable paper and only have some small remarks left:

I got a bit confused after equation 5. The idea of output coordinate dependency of the convolution kernel seems to have been analyzed before (ref 31), however, I don't get it. It seems like a very unusual thing to do. Do I understand correctly that usually one does not have beta dependency and thus w_{i,\beta}=w_{i,0} for every beta? Are there other works actually using this. Is this paper using it?

Related is the interpretation of equation 6. It is introduced as a point-wise multiplication, but then in the equation I see a sum over a frequency index which to me looks more like an inner product. Does this have to do anything with the beta dependency of the conv kernel? In a point-wise multiplication I wouldn't expect a reduction over the points... Could this please be clarified.

Also, I don't see how the beta dependency is used in the paper. Could it be mentioned whether this is the case or not? I think the reader (me at least) would be helped with emphasizing the simpler case of no beta dependency.

Line 195 "we can also expect favorable numerical properties". Favorable compared to what?

Some typos:
line 55 "compute the a"
line 145 "correspond do convolution"
line 147 "most case and but also"
line 156 "is still holds"
line 160 "non-linear can"
line 172 "outputs will"
line 186 "anbular"
line 201 "problem of is"
line 202 "by clip the"
and probably several more... please check this.

limitations_and_societal_impact: I think limitations with respect to generalization of the result to more realistic datasets could have been addressed (see MNIST critic above). Otherwise, I think the scope is clearly given and appropriate.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the effect of discretization issues when using different point-wise non-linearities (applied on the regular representation) in SO(2) equivariant neural networks. Unfortunately, the composition of band-limited SO(2) functions (in the features of an equivariant network) with popular point-wise non-linearities (such as ReLU, ELU or tanh) results in SO(2) functions whose spectrum is decaying but is not generally band-limited. A band-limited spectrum can be reconstructed up to desired precision using a larger number of samples but will always include some non-equivariant approximation error.

To guarantee perfect equivariance, the authors propose the use of polynomial activation functions, which guarantee band-limited output funtions and therefore enable perfect reconstruction of the spectrum using a finite number of samples as prescribed by the sampling theorem.

The authors compare different types of non-linearities on MNIST-rot, a classical benchmark for rotation equivariant models, and ModelNet40.
Their models achieve results comparable with the State-of-the-Art while achieving more stable equivariance.
main_review: I think the authors study a very relevant problem which affects many equivariant models in the literature. The work relies on standard tools from signal processing and harmonic analysis to clearly characterize this problem and to propose a solution. I also found the presentation clear overall.

I think the authors should also cite at least [1, 2] (see below), which also used SO(2) point-wise non-linearities based on Fourier Transform.

While I agree polynomial activations are necessary to achieve perfect equivariance, I find the relative error measured when using ELU or ReLU non-linearites sufficiently low for any practical application in a neural network. Indeed, Fig 6 shows that ELU achieves a *relative error* lower than 1% with ~12 samples. The final classification is anyways usually perfomed using an argmax over the output logits, which are usually trained using a cross entropy loss to favor a single class; I expect this to make the model sufficiently robust to small invariance errors in most cases. This also seems to agree with the MNIST-rot experiments in [31], where the performance of the C_N models saturate after N=12. I would like to see some more comments about this in the paper.




[1] RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks, Cheng et al.
[2] Gauge Equivariant Mesh CNNs Anisotropic convolutions on geometric graphs, de Haan et al.

Edit after rebuttal:
I have read the reviews and author responses, and decide to maintain the same rating.
limitations_and_societal_impact: I described the main limitation in the MAIN REVIEW.

As the authors mentioned, this approach only applies to the SO(2) group and might be hard to immediately generalize to more general Lie groups due to their more complex harmonic analysis.
I only find this a minor limitation.

I don't see any issue with the authors' comments on the societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors develop an SO(2)-equivariant convolution based on the Fourier transform. They show that non-linear activations create high frequencies and hence they clip those high frequencies in their convolution. They apply their method to both 2D data (rotated MNIST) and 3D data (rotated ModelNet40). 
main_review: The paper seems to have a very nice contribution, but it was quite hard for me to read it. The presentation of the method should be improved. 

Table 1: Why is it important to have so many versions of the proposed method here? What is the best choice? Also - is there only one other paper that has results on rot-MNIST?

Please cite the paper "Vector Neurons: A General Framework for SO(3)-Equivariant Networks":
https://arxiv.org/pdf/2104.12229.pdf

Table 2: Some comparisons from papers cited in the paper above are missing, and there are some works there that out-perform the proposed method. See table 1 for ModelNet40.  

Some typos:
line 145: "do" -> "to".
line 147: " show that this is not only the most case and but": please rephrase. 
line 186: "anbular" -> angular
line 202: "by clip the" -> "by clipping the" 

=========== Post rebuttal ==================

In light of the comments made by the other reviewers, I upgrade my score to 6. Regarding my comment on the missing comparisons to other methods: the author mention that their method is more stable, yet I do not see a discussion about that in the paper. How can you reach that your method is more stable than others? Is there a theoretical justification for it? Regarding your explanation: the better methods look pretty consistent wrt the two columns that you mention.
limitations_and_societal_impact: No mention.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper investigates the effect of nonlinearities in SO(2)-equivariant networks. When being applied in the spatial domain, such nonlinearities may lead to harmonic distortions, creating higher frequency harmonics in the signal. In a naive implementation this leads to aliasing effects, reducing the equivariance to a finite subgroup C_N < SO(2).
Due to their analytical tractability, the authors start by assuming nonlinearities that are polynomials of finite degree D. By employing the convolution theorem, they show that the Fourier coefficients of the nonlinearly transformed signal can be expressed in terms of convolutions of the input Fourier spectrum. This result implies in particular that inputs which are band limited to frequency K result in outputs that are band-limited to KD. Fully SO(2)-equivariant polynomial nonlinearities require therefore a D-fold oversampling of the input, to which the nonlinearity is applied pointwise.
For non-polynomial nonlinearities, the authors propose a D-fold oversampling, where D is a user-parameter. Equivariance is in this case approximate, but will decrease with increasing D.

The experimental section evaluates different nonlinearities and the effect of their hyperparameters.
The authors implement a G-CNN for G=SE(2), whose features can be seen as functions on fibers SO(2) over a base space R^2. As usual, the nonlinearities are acting on each fiber individually.
A second experiment implements SE(3)-equivariant Surfel networks, which are again relying on features which are functions on SO(2) at each point of a surface.
Figure 5 evaluates the equivariance of ReLU nonlinearities, which are non-polynomial. The relative equivariance error is hereby found to decay exponentially with the number of oversampled Fourier coefficients. The equivariance error of polynomial nonlinearities is in figure 6 shown to drop at the theoretically predicted cutoff to machine precision, which shows that they are indeed continuously SO(2)-equivariant. Classification results on the rotated MNIST dataset are approximately similar to those of previous work, however, the models require less parameters.
The surfel network achieves SOTA results on ModelNet-40, consuming a significantly reduced runtime in comparison to prior work.
main_review: Originality:
SO(2)-equivariant CNNs in prior work relied on non-standard nonlinearities like tensor products and C-ReLUs or applied standard nonlinearities like ReLU to a discretization of the signal on a cyclic subgroup C_N. While the former perform usually unfavorably, the latter are not continuously SO(2)-equivariant. Inspired by harmonic distortion analysis, the authors propose polynomial nonlinearities which guarantee exact SO(2)-equivariance without compromising model performance or efficiency.
The method is new and related work is adequately cited.

Quality:
The submission is technically sound. All claims are well supported by the theoretical analysis and experimental results. Strengths and weaknesses are discussed appropriately.
Table 1 presents results in multiple different settings, however, the results of E2CNNs and the Poly networks are not immediately comparable since the number of numerical coefficients differ. It would be useful to add E2CNNs with 9 coefficients or Poly networks with 17 coefficients (or both).

Clarity:
The mathematical formulation is clear and the paper is well organized.
While the theoretical contribution is considering SO(2)-equivariant networks, the experimental section considers SE(2) and SE(3) equivariant models, which I found initially somewhat confusing. The authors should clarify how the models and the theory relate.
It would furthermore be helpful to clarify that the theory is developed from the viewpoint of regular SO(2)-CNNs with scalar features on SO(2) instead of steerable (representation valued) features. The authors should mention that their Fourier series representation of the features is equivalent to irrep features in steerable CNNs.
The linear maps W that are introduced in section 3 are not used in the experimental section, which may leave their role to some readers unclear. I could furthermore not follow the reasoning in lines 145-147 that w may additionally depend explicitly on a parameter beta.
Instead of the \otimes symbol in eqs. 9,10, I would suggest using \ast. This may help to avoid confusion since equivariant networks rely often on tensor products.
I do not understand the claim in lines 264,265 that the resulting angular functions may be interpreted as functions in the tangent plane. Should the angular functions not rather be identified with a function on unit-vectors?
The authors provide sufficient information to reproduce the empirical results.

Significance:
While the structure of linear equivariant maps is largely understood, the space of equivariant nonlinearities remained somewhat unclear. Specifically, the existing continuously SO(2)-equivariant nonlinearities Weiler and Cesa shown to perform inferior to discretized C_N-equivariant nonlinearities. The authors make an important contribution to the understanding of continuously SO(2)-equivariant nonlinearities.
The results are of particular importance for applications in natural sciences, where SO(2)-equivariance should in many cases hold exactly.

limitations_and_societal_impact: Limitations of the work are discussed appropriately. No immediate negative societal impacts are to be expected.
review_title: null

Meta-review: Congratulations, the paper is accepted to NeurIPS 2021!
Please add a discussion on tradeoffs of approximate versus exact equivariance. 
Furthermore, please add additions to the supplementary as discussed with reviewer 5pXb. 
Please consider to include/elaborate the discussion on the challenges in generalizing your method to other transformation groups. 
Lastly, please incorporate other suggestions and edits as discussed in rebuttal and reviews. 
Decision: Accept (Poster)