Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes a rotationally invariant CNN based on local Fourier-Bessel expansion of input images. To define the new type of convolution, for each local region of the image, the proposed B-CNN takes the averaged (over angles) inner product between kernel and “rotated” Fourier-Bessel coefficients. This convolution is invariant to the rotation of the local patches. Unlike G-CNN that deals with discrete sets of rotations, the B-CNN can handle any rotation in the continuous space $[0, 2\pi]$. The numerical experiments demonstrate the advantage of B-CNN in various settings.

main_review: The paper is in general well-written. In theory, it is a nice property that B-CNN is able to achieve rotation invariance for any continuous rotations. It seems to perform better than G-CNN especially when there is no data argumentation. However, I have the following concerns on the novelty and experiments of the paper:

1). I feel that the contributions of this paper are over claimed. For example, the paper claims itself to be the first work in machine learning that introduces the Bessel functions. However, the following papers already introduced Fourier-Bessel, and in general, steerable basis functions, to achieve rotation invariance in dimension reduction, image classification/recognition/denoising: 

1. “Fast Steerable Principal Component Analysis”.
2. “Steerable PCA for Rotation-Invariant Image Recognition”.
3. “Fourier–Bessel rotational invariant eigenimages”.
4. “Rotationally invariant image representation for viewing direction classification in cryo-EM”.

The motivation of the above works is almost the same as this paper. For example, see (15)-(20) in https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3711886/ that uses similar integral over angle $\alpha$ compared to (12) of this B-CNN paper. The only difference is that (15)-(20) in the above link computes the rotationally invariant (angular averaged) mean (or the inner product with a constant kernel) and covariance, and in this B-CNN paper (12) computes the rotationally invariant (angular averaged) inner product with a kernel (not a constant kernel). I hope the authors can clarify the strong relevance of these papers.

2). The experimental setup looks a bit unfair for G-CNN which uses $\mathbb Z_4$ to represent the discrete rotations. I understand that the original G-CNN paper also uses $\mathbb Z_4$ group, but it seems to be generalizable to e.g. $\mathbb Z_8$, $\mathbb Z_{16}$ (although careful interpolation may be required), at the cost of increasing the number of parameters. However, since in the experiment B-CNN has fewer layers than G-CNN, one can certainly reduce the number of layers in G-CNN and increase its number of discrete rotations to keep the similar number of model parameters. Another simpler way is to consider implementing CNN on the polar-transformed image with cyclic paddings, like in paper “Polar Transformer Networks”. I am curious about the experimental results in this new setting.

3). The paper claims its advantage on handling continuous rotations, but I doubt its importance for discrete images. For example, for images with low resolution, due to the discretization error it is sufficient to use the set of discrete rotations. The number of rotations $n_R$ should be proportional to the image resolution. One may argue that for high resolution more discrete rotations are needed and this increases the model complexity of G-CNN. However, the same issue applies to B-CNN. Indeed, for low resolution images, one should use smaller $\nu_{\max}$ and $j_{\max}$ to avoid the aliasing problem. Namely, one should keep them below the Nyquist frequency of the image, which is proportional to the image resolution. For higher resolution, one should increase the number of angular and radial frequencies accordingly and this increases the model complexity. From this perspective, the parameter $\nu_{\max}$ in B-CNN and $n_R$ in G-CNN play similar roles (although the two are respectively in frequency and real domain). Thus, it is a bit suspicious that you take a large $\nu_{\max}$ in B-CNN but a much smaller number of discrete rotations ($n_R=4$) in G-CNN (see also my point 2) ). The question is that when $\nu_{\max} \approx n_R$ and with the same number of layers, can B-CNN still beat G-CNN? A more convincing experiment set up is to compare the performance between dfifferent neural networks with the same number of layers for varying $\nu_{\max}$ and $n_R$ .

Minor comments:
1. In (5), $\nu', j’$ should be $\nu, j$.
2. In line 105, $J’$ should be $J$.
3. In the first line below line 143, the subscript of the last $\theta$ should be $\nu', j’$. 

Missing references:
1. Dense Steerable Filter CNNs for Exploiting Rotational Symmetry in Histology Images.
2. General E(2) - Equivariant Steerable CNNs.


In summary, this paper applies ideas from Fourier-Bessel steerable basis to CNNs, and is well-written in general. However, its novelty is limited in my opinion (which is overclaimed in the paper), and more experiments and clarifications are expected. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper introduces Bessel convolutions, an approach to computing rotation equivariant representations using the well-known Bessel functions. The operation can be described mathematically as an integral over rotations of  an inner product between a rotated/phase shifted filter and the input. It is shown that this can be computed efficiently without an integral, as a modulus of a complex inner product between filter and input. The idea is validated by training networks on rotated MNIST, Outex (textures), and brain MRI scans. Results show that in among the specific models that were compared, the B-CNN outperforms on several datasets / configurations.
main_review: The paper is well written, and technically sound. The approach is clearly motivated and explained, and enough detail is provided to enable others to reproduce the work. However, despite many strong points of the paper, I am not convinced that this method truly advances the state of the art in rotation-equivariant convolutional networks. There are a number of reasons for this:

- The method is compared to CNNs and G-CNNs that are equivariant to 90 degree rotations. It is natural to expect that a method dealing with continuous rotations can do better than 90 degree rotations, and indeed the results confirm that in several configurations (but not all) the B-CNN outperforms the G-CNNs. However, there are several methods that deal with continuous / finely discretized rotations, and these are not compared to. The main methods are harmonic networks by Worrall and, what I believe to be the current best approach, the "General E(2)-equivariant steerable CNNs" of Weiler & Cesa. Open source code is available for both, so it should not be too hard to benchmark against these.

- The networks trained are relatively small (5-7 layers) without modern design elements such as residual connections and batchnorm. Also, the datasets used for benchmarking either don't have an established state of the art result or these are not reported/compared to. For these reasons it is not clear if the comparisons reported in the paper reflect the very best that the G-CNN and B-CNN can achieve. It would be more convincing to see that B-CNNs can outperform the results reported in other papers on a competitive benchmark. There are a number such benchmarks that can be found in other papers on rotation equivariant CNNs.

- For some of the plots in figure 3, it seems like the baseline methods are not yet converged. Also, slower convergence seen for the CNN/G-CNN could indicate that the weights are not initialized at the right scale or optimizer hyperparameters are not properly tuned. Again, such concerns could be taken away by reporting results on a competitive benchmark. 

- It is claimed that B-CNNs achieve equivariance to continuous rotations, but this is only guaranteed in the continuous setting, not when the Bessel functions are discretely sampled on a grid. To achieve rotation equivariance in practice, one would need to use low frequencies or relatively large filters (which have not been favored in modern CNN designs). Weiler & Cesa have an extensive discussion of this. The same is true for any steerable CNN, including ones based on circular harmonics such as harmonic networks and Weiler & Cesa's networks, so it's no shame but at least should be explained clearly. 

- The property of Bessel functions that makes the B-CNN equivariant is the fact that they undergo a phase shift under rotation. That is, they are steerable filters, and so this methods falls in the category of steerable CNNs. It is not clear why the Bessel functions are a better choice than say circular harmonics, which have the same property and have been used in previous works (Worrall and Cesa & Weiler). The circular harmonics already provide a complete basis for the space of equivariant filters, which implies one does not need anything else. Perhaps there is a numerical reason why Bessel functions are more stable or otherwise preferable, but this should then be explained clearly and demonstrated empirically in a head-to-head comparison with circular harmonics based approaches.

- I'm not 100% clear on the implementation details, but it seems that the current implementation requires nu_max+1 calls to conv2d, which would make it considerably slower than methods that use a single call. Perhaps the implementation can be improved though?

Although I have focussed on points that need to be improved, I want to reiterate that this is not a bad paper at all. In fact it's based on a neat idea, with solid and interesting underlying mathematics. However to really advance the field it is necessary to deeply study the current best methods, understand and address their weaknesses. Studying steerable filter bases for equivariant CNNs is a topic worthy of study, so I encourage the authors to dig deeper into this topic to better understand what makes for a good basis of filters, and perform direct comparisons to the most closely related and best performing existing methods.
limitations_and_societal_impact: As noted in the main review some limitations could be better discussed. Societal impact for this paper is the same as any other paper in (equivariant) CNN design, in that CNNs can be used for positive and negative uses. In practice, equivariant CNNs have mostly been applied in medical imaging and problems in the natural sciences.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper aims at achieving rotation invariance of images by introducing Bessel transforms to deep convolutional neural networks. 
Local operations are carried out on submatrices over sub-regions by shifting kernels and the inner products are taken for Bessel coefficients over radial directions. The experiments show that the proposed Bessel-convolutional neural networks perform extremely well. 
main_review: The main idea is to use Bessel transforms for representing images and to take inner products on coefficients over the radial directions. Since Bessel transforms turn rotations into multiplications with exponentials, the induced inner products becomes rotation invariant. This is a very nice observation and the authors' idea is novel. The experiments with MNIST, Outex and brain MRI provide very convincing results. The paper is also well written. 
limitations_and_societal_impact: It would be great somewhere else if the authors could provide some theoretical justification for the proposed method. Though some computations are given on page 6 about the implementation of the Bessel transform, more details and computational complexity should be provided.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The submission proposes Bessel-convolutional neural networks (B-CNNs), which are by design guaranteed to be equivariant w.r.t. translations and continuous rotations in two dimensions, i.e. the group SE(2). Convolution kernels are hereby expanded in an orthonormal function basis which consists of Bessel functions in the radial part and circular harmonics in the angular part (as commonly done in previous work). The central building block of B-CNNs is a non-linear operation, which (in its final form, eq. 22) computes the convolution response of the image with each angular harmonic frequency component individually, takes their square norm to produce locally rotation invariant responses, and finally aggregates those. This operation is motivated by an initial operation in eq. 12, which performs a group averaging over all filter or image rotations. Sections 4.3 and 4.5 are analytically manipulating this initial formulation in eq. 12 to its final formulation in eq. 22.

B-CNNs are evaluated on three small scale classification datasets, namely MNIST digits, Outex-TC-00010-r textures and brain MRI images. Each experiment compares B-CNNs with a non-equivariant CNN baseline and a G-CNN architecture, which is equivariant w.r.t. translations and rotations by multiples of 90 degrees. Classification accuracies (figure 3) are reported in two settings: First, each model is trained on non-rotated inputs and then tested on rotated samples. This setting evaluates the OOD generalization over rotations which is theoretically ensured by rotation equivariant networks. The second setting trains and tests on rotated samples. While the equivariant models perform significantly better than the standard CNN baseline, neither B-CNNs nor G-CNNs are clearly outperforming each other in all settings.
Section 5.5 and figure 4 present a small experiment which compares the equivariance of B-CNN, G-CNN and standard CNN feature maps. B-CNNs perform hereby slightly better than G-CNNs, which in turn perform better than CNNs. This is not surprising since the G-CNNs which are used are in contrast to the B-CNNs not continuously rotation equivariant.
main_review: Originality:
A main argument of the authors (e.g. lines 6-8 and 31-32) is that their model is the first which is equivariant w.r.t. continuous rotations in SO(2). This claim is incorrect as there exist many continuously rotation equivariant models, for instance:
- Harmonic Networks (Worral et al., CVPR 2017)
- E(2)-CNNs (Weiler and Cesa, NeurIPS 2019)
- LieConv (Finzi et al., ICML 2020)
- Zernet (Sun et al., 2018)
- Harmonic surface networks (Wiersma et al., Transactions on Graphics, 2020)

The method relies essentially on the use of a decomposition of the kernels' angular part into circular harmonics. It has been shown by (Lang and Weiler, ICLR 2021) that any SO(2)-equivariant model necessarily relies on such a decomposition (sometimes implicitly). However, the kernels' radial part is shown to be entirely unconstrained in rotation equivariant models - the use of Bessel functions to expand the radial parts is therefore arbitrary for the rotation equivariance of B-CNNs. The authors should clarify this point since the current submission gives the impression that this choice is essential (e.g. lines 8-11, 28-29). The use of a polynomial radial part is not new (see e.g. Zernet, who use Zernicke polynomials) but is in principle an interesting choice which is worth to be investigated.


Quality:
The presented theoretical results seem in general to be technically correct. The mathematical notation could at some points be made mode formal, however, this is not a big issue in an applied field. The authors may for instance choose to adapt the following points:
(line 51, 55): [0,2pi] should probably be changed to [0,2pi)
(line 51): the group action R on the input and output of an equivariant function f may in general differ
(line 104 / eq. 6): the basis is not only orthogonal but also orthonormal
(lines 101-105): this sentence is a bit confusing. Eq. 5 does not use proper set builder notation (the ", \nu, j in N" should be inside the curly brackets).

The empirical evaluation is formally correct and evaluates the key theoretical claims. However, it would be desirable to evaluate the models beyond toy datasets. The claim in line 184 that MNIST-rot consists of digits which are rotated by multiples of 2pi/8 is incorrect. A comparison against previous work on this dataset would be interesting since it is the main toy benchmark for rotation equivariant models. Such an experiment would be particularly interesting since the current submission evaluates the model only against their own weak baselines. Since a particular focus is put on continuous rotation equivariance, it would furthermore be interesting to add a harmonic network baseline and a G-CNN baseline for G=C_N with N>4.


Clarity:
The submission is mostly clearly written, however, the irrelevant role of the Bessel radial parts for rotation equivariance remains unclear (as stated above, only the circular harmonic angular parts are relevant). The necessity of the derivation in Sections 4.3 - 4.5 is in my view unclear. One could immediately propose eq. (22) as an rotationally equivariant operation without deriving it from eq. (12), which is itself not further motivated.


Significance:
Given that continuously SE(2)-equivariant models which are based on kernels with circular harmonic angular parts, the significant contributions of the submission are 1) the use a Bessel expansion of the radial parts and 2) the nonlinear mapping of non-zero frequency harmonics responses to invariant scalars via the square norm. While the current experiments evaluate the continuous equivariance of B-CNNs, they are not investigating the significance of these two choices. The authors should therefore add experiments with baselines that use 1) different radial parts and 2) different methods to map the higher order responses to scalars.
limitations_and_societal_impact: The authors address some limitations of the model.
No immediate negative societal impact is to be expected.
review_title: null

Meta-review: While reviewers enjoyed aspects of the paper, the key remaining concern after the discussion period was a distinct lack of comparisons (and theoretical advantages) over the state-of-the-art in rotation invariant neural networks. Several reviewers have left their final comments as replies to the replies. Please consider these carefully in preparing your revisions.
Decision: Accept (Poster)