Achieving Rotational Invariance with Bessel-Convolutional Neural Networks

Abstract: For many applications in image analysis, learning models that are invariant to translations and rotations is paramount. This is the case, for example, in medical imaging where the objects of interest can appear at arbitrary positions, with arbitrary orientations. As of today, Convolutional Neural Networks (CNN) are one of the most powerful tools for image analysis. They achieve, thanks to convolutions, an invariance with respect to translations. In this work, we present a new type of convolutional layer that takes advantage of Bessel functions, well known in physics, to build Bessel-CNNs (B-CNNs) that are invariant to all the continuous set of possible rotation angles by design.


Reviews and author responses:

 I'd like to thank the authors for a detailed response. I keep my score (7, accept). I think this is a good paper which gives a good solution to an important problem. I still have some minor concerns over effects of interpolation, statements about low-pass images, imperfect invariance and related guarantees.

This paper proposes a method to achieve rotational equivariance (and to some extent, invariance) in convolutional neural networks. The method works by representing the convolution kernel in a Bessel Function basis (which is naturally rotation equivariant). This results in a convolutional operator which, though requires more convolution operators (one per Bessel coefficient used compared to one in regular CNNs), outputs a rotation equivariant feature map, unlike normal CNNs.
The method is demonstrated to work on three simple datasets - MNIST (with rotations) a Texture dataset and a small MRI dataset.   Originality:
The proposed method is different from prior work and quite an elegant solution to the problem.

Quality:
I think this is quite a solid paper in terms of quality all in all, though there are a few shortcomings. The motivation is clear, and the proposed solution indeed addresses the problem directly. The mathematical analysis is, though I didn't check in detail, extensive and well presented and the bottom line (i.e convolving with Kernels obtained with the projection on the Bessel function) is clear and actionable. I think the main weak point is the experimental evaluation - though I am convinced that the method is able to produce rotationally equivariant feature maps I am not sure it has been demonstrated to be useful, or, in fact, significantly better than competing methods or even a regular CNN (certainly with augmentations). The datasets used are too simple and the results, even in the ideal case (other than MNIST without augmentation) are not convincing.
Another point I wish the authors would address is what happens in cases where one actually wants orientation information to be preserved and processed by the network? would a combination of the proposed method and regular CNNs work? how? 

Clarity:
Though the writing the is rough at times I had no problem following the paper. It is nicely presented and well structured.

Significance:
I think this is another weak point of the paper - I was not convinced this is an actual significant problem - not because learning rotation equivariance is not important, but because it seems current methods solve it quite well, certainly in more complex settings (where the data is rich). Nevertheless the analysis is interesting and may be useful in some specific contexts.

Post discussion update: Following the author rebuttal and discussion with reviewer I have updated my score and I am now leaning for acceptance of this paper. Discussed above, no societal impact I am concerned about.

 Dear reviewer,

Thank you for the time you spent on reviewing our paper. You provided interesting comments that will be added in the final version of the paper.

***“However I think the authors should be more explicit about the fact that they are only addressing local rotational invariance of feature descriptors, as opposed to global rotational invariance to shapes in an image. It would appear that the global rotational invariance is achieved simply by training on objects in all different rotations, which seems pretty brute force. But the other methods being compared to appear to do the same thing.”***

We want to stress that while indeed the global invariance is obtained via a succession of local variations, our approach is critically different from the other methods. Indeed, we define the invariance in the network, while the other architectures need more data (and augmented data) to in fine learn a good rotational invariance for a continuous set of angles.

Concerning your comment on adversarial attacks, we indeed are (like almost all other networks that are not specifically designed to counter them) vulnerable to these attacks. We will add a discussion on what can be the impact of such attacks in the final version of the paper.

Thank you again for these comments as they will add interesting additions to the final version of the paper.

 Dear reviewer,

Thank you for your review and the time you spent on it. We really appreciate your meaningful comments and the additional references, we will take them all into account in the final version of the paper.

***“Please state for which space the used Bessel functions form a basis.”***

This basis can be used to express any square-integrable function $f$ such that $f: D^2 \subset R^2 \longrightarrow R$ (the domain $D^2$ is a disk in $R^2$).

***“When you write "In order to be mathematically exact", what are you referring to?”***

You are right, there is some kind of ambiguity in this sentence. We will replace it with “In order to be mathematically exact when describing $\Psi(\rho, \theta)$ using its Bessel coefficients...”. The $\Psi(\rho, \theta)$ that is mentioned here is the one in Equation (7).

***“I am a bit puzzled by "as \nu or j increases, k_{\nu, j} also increases. As J_\nu(x) tends to zero when x increases, a large value for \nu and j means that \Psi_{\nu, j} will only slightly influence the reconstruction of \Psi". This is true for images that are low-pass in the Bessel basis. If the image contains high-frequencies then it is not true. In particular, how about an image u(x)=Jν(kν,jρ) for large \nu?”***

Yes, this is true. However for the vast majority of practical cases, images are low-pass and this approximation will work pretty well. 
In a future work, it would be interesting to automatically choose the Bessel coefficients to use in order to keep as much information as possible when decomposing the image. It could probably be done by analyzing the frequencies of the image, and choosing the Bessel coefficients which allow the image to be reconstructed in the best possible way. 
There are otherwise established convergence criteria that limit the number of basis functions to consider for a given number of pixels in the image (see “Fast Steerable Principal Component Analysis”, by Zhao et al.). Our basis functions will accurately represent all (squared-integrable) functions that are sampled appropriately by this number of pixels (excluding de facto pathological high-frequency counter-examples). Thank you for pointing this out.

***“One thing that I find at the same time interesting and confusing is that you use the Fourier-Bessel power spectrum. This clearly immediately gives invariance, but a "canonical" way to get invariance (as in some of the cited prior work) is to compute a convolution over rotations. What is the technical difficulty in doing that for all angles?”***

Using Equation (12) directly should lead to the same results (since Equation (12) and (18) are mathematically equivalent, and Equation (22) is derived from Equation (18)), but it is much more computationally expensive. In this case, we need to perform a numerical integration in addition to the summations over nu and j. In Equation (22), only the summations over nu and j remain. However, thank you for pointing this out, more clarity and motivation about that will be added in the paper.

***“And what are the implications or drawbacks of the fact that your representations are non-negative?”***

The fact that the representation is non-negative is not really an issue, since we add a bias term (like in standard CNNs). Therefore, the bias can arbitrarily shift the representation if needed.

***“Could you elaborate why you don't get perfect invariance in experiments? Is there a setting / dataset where you could expect perfect invariance?”***

There are two main reasons why B-CNNs are not perfectly rotational invariant in practice. The first one is due to the low resolution of some images. The second one is due to the fact that the Bessel functions are also discretely sampled on a grid (see line 168 in the paper). Therefore, when using filters that are too small, the perfect rotational invariance may not be achieved. Using larger filters helps to achieve a better rotational invariance. In conclusion, it is true that performing the experiment that leads to Figure 4 with another image of bigger resolution, and using filters of a bigger size, will definitely be more convincing. We will replace Figure 4 with this setup in the final version of the paper and add a discussion about that. 

***“Could you state a theorem?”***

It is probably possible to determine the minimal size of a filter that is required for an image for obtaining the almost perfect rotational invariance. This can probably be assessed by looking at the frequencies in the image. Higher frequencies will require a bigger filter size in order to properly sample the image.

***“In the context of translations, perfect {equi, in}variance is precluded by nonlinearities and downsampling, but it can be restored by modifying the pooling strategy (https://openaccess.thecvf.com/content/CVPR2021/papers/Chaman_Truly_Shift-Invariant_Convolutional_Neural_Networks_CVPR_2021_paper.pdf, https://arxiv.org/pdf/2105.04040.pdf). Could a similar strategy help achieve perfect rotation invariance in your Bessel-Fourier framework?“***

Thank you for pointing out these references. Indeed, similar ideas may be used to make the rotational invariance more robust when the resolution is low. We will add this and the references in the future work.

***“How does the training time / complexity of your neural networks compare to that of standard CNNs?”***

At the moment, the complexity is the same as the one of a CNN, except that B-CNNs require to perform \nu convolutions instead of only one for standard CNNs. This is a good avenue to improve B-CNNs in future work.

Again, we would like to thank the reviewer for his comments and for all his relevant comments. The different points raised by the reviewer will definitely be addressed in the future version of our paper.

 Dear reviewer,

First, we would like to thank you for the time you spent on carefully reviewing our paper. We will take into account all your very interesting comments in the final version of the paper. We also provide below some answers to a question you raised in your review.

***“It would be helpful if the authors can further explain the symbols in equation (9).”***

In Equation (9), R and I stand for the real and imaginary parts, since $\Psi_{\nu,j}$ is a complex number. We will clarify that in the final version. 

***“The observations clearly show that data augmentation helps a lot with improving even the BCNN model. [...] Can the authors provide their opinion on why data augmentation might be providing those improvements to BCNN, when it should be intrinsically invariant to rotations?”***
 
There are two main reasons why B-CNNs are not perfectly rotational invariant in practice. The first one is due to the low resolution of some images. The second one is due to the fact that the Bessel functions are also discretely sampled on a grid (see line 168 of the paper). Therefore, if too small filters are used, perfect rotational invariance may not be achieved. Using larger filters helps to achieve a better rotational invariance. In conclusion, it is true that performing the experiment that leads to Figure 4 with another image of bigger resolution, and using filters of a bigger size, will definitely be more convincing. We will replace Figure 4 with this setup in the final version of the paper and add a discussion about that.

***“It would be interesting to see further exploration into the implications in low data settings and robustness in future work.”***

As we ran the experiment for another reviewer, we can state that our performance is better in a low data setting. This is easily explained by the fact that we have the invariance for the continuous set of angles as a prior in the network and don’t (or less) need data to learn this invariance. This experience is in fact exactly the same as the one performed on the Outex dataset, except that we removed 75% of the training set. When, in the initial experiment presented in the paper, we slightly outperformed G-CNNs by less than 5% accuracy, in this new experiment where less data is provided during training, we outperform G-CNNs by roughly 15%. Furthermore, the learning curves are much more stable with B-CNNs. As this is a key advantage in our contribution, we will elaborate on that and show the results of the additional experiment in the final version of the paper.

Thank you again for your time and your comments.

 Dear reviewer,

Thank you for your review and the time you spent on it. Your feedback will really help us to improve the quality of our work, and we will address all of your different comments for the final version. We also provide below some answers to questions or remarks you raised in your review.

***“[...] the main weak point is the experimental evaluation.”***

The aim of our method is not so much to obtain the best accuracy in the case of rotational invariant problems, but rather to propose a more convenient, better mathematically defined, way to deal with rotational invariant problems with a method that has at least the same accuracy as the literature. In other words, using a complex CNN with a large dataset may lead to the same performance, but with our model, it should be easier to achieve it (with less data, less parameters, etc.).

***“The datasets used are too simple [...].”***

We followed a strategy for choosing our datasets that surely need more description in the final version of the paper. The datasets have been chosen in an increasing difficulty. MNIST is simple, but is used in almost every image analysis task as a benchmark. Outex, which is already way more complex than MNIST, is also used in different works to assess local and global rotational invariance. Then, brain MRI is used as a real-world, complex dataset, which contains local entities (the brain tumors) that can be subject to arbitrary orientations. We do not consider brain MRI to be a simple dataset.

***“I was not convinced this is an actual significant problem - not because learning rotation equivariance is not important, but because it seems current methods solve it quite well, certainly in more complex settings (where the data is rich).”***

G-CNNs and classical CNNs are not designed to intrinsically tackle the rotational invariant problems for a continuous spectrum of angles, but achieve it using a lot of data and more computing power to learn to detect the same rotated features. We ran a new experiment that shows that our method outperforms the competitor when not enough data is gathered to learn the invariance, i.e. our method is more data-efficient. This experience is in fact exactly the same as the one performed on the Outex dataset, except that we removed 75% of the training set. When, in the initial experiment presented in the paper, we slightly outperformed G-CNNs by less than 5% accuracy, in this new experiment where less data is provided during training, we outperform G-CNNs by roughly 15%. Furthermore, the learning curves are much more stable with B-CNNs. We will add more discussions / experiments about that in the future version of the paper.

***“What happens in cases where one actually wants orientation information to be preserved and processed by the network? Would a combination of the proposed method and regular CNNs work? How?”***

It would be possible to combine them in parallel. This way, the B-CNN part should be able to extract rotational invariant features, and the other part other features that are not rotational invariant. However, it is not possible to sequentially combine B-CNNs and CNNs as the invariance is obtained by a succession of equivariant operations that lead to an 1x1 feature map and this would be lost if a CNN is used after a B-CNN.

We would like to thank the reviewer again for taking the time to review and for all his constructive comments.

The paper shows how Bessel functions may be used in the context of a convolutional network to obtain better recognition performance for rotated images.  Conceptually a Bessel function expansion is applied locally at each point in an image, which is then filtered with a standard convolutional filter bank (eq. 18).  The authors describe a more efficient implementation that folds the Bessel function expansion into the kernel convolution (eq. 22).  Performance is described on MNIST, Outex, and MRI images of tumors, showing improvement for the Bessel function approach.  This seems like a reasonable approach.  However I think the authors should be more explicit about the fact that they are only addressing *local* rotational invariance of feature descriptors, as opposed to global rotational invariance to shapes in an image.  It would appear that the global rotational invariance is achieved simply by training on objects in all different rotations, which seems pretty brute force.  But the other methods being compared to appear to do the same thing.  Perhaps this is state of the art, but in my view rather primitive.  There doesn't appear to be any attempt to reason about the geometry of an object, just a bunch of filters and averaging.  

Also the Bessel function descriptor simply averages over all orientations, so it loses all information about the local configuration of shape (with respect to orientation).  It would seem that such an approach could be easily fooled by adversarial examples or nonsensical images that provide the same signature in terms of these descriptors but without a consistent global shape, something to think about.
 Yes

The authors propose a representation based on Bessel functions as a replacement for standard convolutional layers which guarantees rotation invariance. They discuss the properties of Bessel functions and motivate the new rotation-invariant layers. Experiments show that the proposed layers indeed improve rotation invariance; in fact, the proposed scheme seems to outperform the existing schemes based on group representations which only implement invariance for a discrete set of angles.
  Overall this is a solid paper about a creative idea. It would be great to see a bit more discussion about the peculiarities of the proposed layers and to tighten the presentation. Another welcome addition would be a precise statement about when perfect rotation invariance can be achieved, and a discussion about why we don't see it in experiments. Perhaps a reference to similar issues that plague ordinary convolutional layers (see below) and a comparison with the proposed Bessel representations. Below I list a bag of comments not ordered by any particular criterion:

- Please state for which space the used Bessel functions form a basis

- To streamline notation (e.g. in (7), (8)) I suggest to define normalized versions of Bessel functions

- When you write "In order to be mathematically exact", what are you referring to? 

- I am a bit puzzled by "as \nu or j increases, k_{\nu, j} also increasse. As J_\nu(x) tends to zero when x increases, a large value for \nu and j means that \Psi_{\nu, j} will only slightly influence the reconstruction of \Psi". This is true for images that are low-pass in the Bessel basis. If the image contains high-frequencies then it is not true. In particular, how about an image $u(x) = J_\nu(k_{\nu,j}\rho)$ for large \nu? 
- Certainly the above makes sense for images with finite pixels, but then the requisite value of \nu_max can be determined exactly from the sampling theorem. (NB: I strongly suggest to replace "The values of \nu_max, j_max can be determined through a mathematical insights about..." by something like "The values... can be determined by considering ... ").

- One thing that I find at the same time interesting and confusing is that you use the Fourier-Bessel power spectrum. This clearly immediately gives invariance, but a "canonical" way to get invariance (as in some of the cited prior work) is to compute a convolution over rotations. What is the technical difficulty in doing that for all angles? And what are the implications or drawbacks of the fact that your representations are non-negative? 

- Fourier-Bessel representations are commonly used in signal processing to obtain rotation-invariant representations. One good example is https://arxiv.org/abs/1211.1968 or fast steerable PCA (https://arxiv.org/abs/1412.0781). It would be good to compare your work to these representations. There are many other examples which would be acknowledging and commenting under prior work.

- Some of your experimental results where you improve over C5 convolutions in G-CNNs are not surprising since 4 rotation angles is a coarse discretization. It would be great to compare to finer discretizations of 8 or 16 angles.

- Section 5.2 gives a great amount of detail about the architectures which is a priori good, but it throws a brick in the flow of the paper. Perhaps some of it could be put in the appendix? (Not all of it is really important to appreciate the experiments)

- Could you elaborate why you don't get perfect invariance in experiments? Is there a setting / dataset where you could expect perfect invariance? Could you state a theorem? In the context of translations, perfect {equi, in}variance is precluded by nonlinearities and downsampling, but it can be restored by modifying the pooling strategy (https://openaccess.thecvf.com/content/CVPR2021/papers/Chaman_Truly_Shift-Invariant_Convolutional_Neural_Networks_CVPR_2021_paper.pdf, https://arxiv.org/pdf/2105.04040.pdf). Could a similar strategy help achieve perfect rotation invariance in your Bessel-Fourier framework?

- How does the training time / complexity of your neural networks compare to that of standard CNNs?


Some misprints / suggestions:

- mathematical basis -> basis
- which really conveniently expresses rotation -> which conveniently expresses rotations in the Fourier-Bessel transform domain (perhaps: analogously to how the Fourier transform maps translations to multiplications by complex exponentials)
 Yes.

This paper proposes a new type of convolutional layer that takes advantage of Bessel functions, which are very well known in physics, to achieve rotational equivariance in convolutional neural networks. The main idea of the approach is as follows: in a general convolutional layer, a convolution is performed with a sub region of the image and the filter. But here, the sub region is first transformed to a polar coordinate system using a Bessel function as the basis, and the properties of the Bessel function are utilized here to define a filter operation that integrates over all possible angles of the sub region, thus making the operation rotation invariant, while ensuring its scalable to deep networks. The main contribution of the paper, therefore, is to propose a technique that achieves both local and global rotational equivariance in convolutional neural networks over the continuous set of all possible angles, while making it scalable and mathematically rigorous.   Originality: 
- The proposed method is definitely very novel, and the authors claim that to the best of their knowledge, this is the first method to introduce Bessel functions in machine learning, as well as to propose a rigorous rotational invariance over the continuous set of all possible angles in CNNs. 
- The related work is comprehensive and very well compiled. Also, the authors make it clear how their approach stands out, with the main drawbacks of the other approaches being not providing rigorous rotational invariance or not covering the entire spectrum of angles. 

Quality: 
- The main idea of the paper, using Bessel functions in polar coordinate system to define a filter that is rotationally invariant, is supported by rigorous mathematical evidence. All derivations involved in obtaining the final form of the proposed filter operation are very clearly provided.
- Comprehensive experiments are conducted, while comparing with the state of the art techniques, to provide empirical evidence to support their claims.
- This is a complete piece of work with no unexplained or incomplete sections.  

Clarity: 
- The paper overall is very well written. Its clear, understandable, and very well organized. The code is also provided in the supplementary material. 
- It would be really helpful if the authors can expand a bit more, and add some derivations, in section 4.1, especially for equations 5 and 6, if required in an appendix. 
- It would be helpful if the authors can further explain the symbols in equation 9. 
- In Figure 3, one observation is that data augmentation (which includes only random rotations) provided considerable improvements to BCNN. GCNN with data augmentation gave better results than BCNN without data augmentation is several scenarios. Further, in section 5.4, the authors commented that "GCNNs need data augmentation in addition to the groups of symmetry to learn rotational invariance, while Bessel convolutions intrinsically impose the invariance". But the observations clearly show that data augmentation helps a lot with improving even the BCNN model. The authors too acknowledge this in section 5.4. Can the authors provide their opinion on why data augmentation might be providing those improvements to BCNN, when it should be intrinsically invariant to rotations? 

Significance: 
- I think the introduction of Bessel functions is a very important contribution of the paper. It has a lot of scope for further work from the community. Further, with the strong guarantees provided by the paper, unlike any other preceding work, this paper definitely advances the state of the art in the area of rotational invariance in CNNs. 

Further Comments: 
- It's very interesting to note that this approach seemed to provide an advantage in low data scenarios, and provide more stability due to equivariance of the feature maps. It would be interesting to see further exploration into the implications in low data settings and robustness in future work. 
- No large datasets were used in the experiments, which can be used to test the scalability of the proposed approach. It would be very helpful if the authors provide results (if required in an appendix) on some large dataset like ImageNet, with a well-known deep network like a ResNet, etc. This is to analyze how the proposed approach will scale with large models and datasets, and if similar gains can be noticed in such a scenario too.  I think the main limitation of this work is the proposed Bessel convolutional layer is computationally more expensive than a normal convolutional layer. The authors did acknowledge this and provided an explanation on how the prior knowledge helps in achieving similar results with smaller networks that can compensate for the computational differences. In addition to that, I think it will be very helpful if the authors provided some results with a large dataset and a deep network too.

Meta-review: After lively discussion, all reviewers found this paper interesting and thought the paper ought to be accepted. Everyone liked the idea of using Bessel functions to achieve local rotational invariance clever and found the technique to be potentially quite impactful -- especially in the low data regime. Reviewers also found the experimental results compelling.

Decision: accept