Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes a new convolution operation for neural networks that is simultaneously equivariant to rotations, translations and scaling. The crux of the method is representing the images and filters using in what the authors call the scalable Fourier-Argand representation. With this operation defined, equivariance can be shown theoretically for continuous signals, which also holds empirically. Implementation of the idea via discretization is also discussed.  Experiments on multiple benchmark datasets clearly show that the proposed ideas are better at classification than other equivariant methods, especially when unknown scale, rotation and translations are applied simultaneously at the input. 
strength_and_weaknesses: Strengths:

1. The ideas in the paper are novel. The scalable Fourier-Argand representation is clever and allows for enabling the required equivariance, given that the convolution operation is defined as shown. 

2. Equivariance has also been experimentally verified. 

3. Experiments with SRT-MNIST clearly show that the proposed method outperforms methods that do not take into the right equivariance into account.

Weaknesses:

1. I am confused by the authors stating at multiple points that they are describing the first method that achieves rotation and scaling equivariance simultaneously. But the authors themselves describe several works in the related work section that do it. The most recent example I know is from CVPR 2022: Enabling Equivariance for Arbitrary Lie Groups which shows a framework for equivariance to any finite-dimensional Lie group: https://openaccess.thecvf.com/content/CVPR2022/papers/MacDonald_Enabling_Equivariance_for_Arbitrary_Lie_Groups_CVPR_2022_paper.pdf

The authors should definitely make this point clearer in the paper and discuss why these could not be included as baselines for their experiments. I also don't see what the authors mean by their method not being based on group theory.

2. There are some minor very minor issues in the formulation section 3.2. In many places, the equations use dimension $d$, while the text says two-dimensions and that the convolution integral is a double integral.

 3. While the experiment with STL-10 is appreciated, it would be nice if the authors can have another experiment to clearly show that it is because of the joint scale and rotation equivariance that the experiments improve i.e. are the OOD samples OOD because of unseen scale and rotation transformations?
clarity,_quality,_novelty_and_reproducibility: The paper is written well and is generally very clear. The central ideas in the paper are indeed novel. I am not very sure about the reproducibility however, and strongly suggest that the authors try to release their code. 

summary_of_the_review: I think the novel scalable Fourier-Argand representation and how it enables equivariance is very interesting. Experimental results provide validation. There are some weaknesses which I have listed above which I hope the authors can address. Overall, I recommend acceptance.


UPDATE AFTER AUTHOR RESPONSE:

I thank the authors for their response. I think the contributions are clearer now and the authors should have this discussion in the main paper. Having also read the other reviews, I maintain my original score of accepting the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors highlight the fact that CNNs, which are capable of tackling translation, scale and rotation transformations, would be useful for computer vision, but are less studied than less general counterparts. Then the authors propose scalable Fourier-Argand representation as the key for building such models. In their experiments they demonstrate that the proposed SREN model is more accurate than the previous methods.
strength_and_weaknesses: The paper is well-written. Although, I am going to propose several adjustments, the paper is already clear enough to be understood. The paper has a good structure which makes it easy to follow the  idea of the authors. Additionally, the mathematical language is high level which allows to stick to very accurate formulations.

The idea discussed in the paper is novel. And it is interesting to read it. Although it may be rough, and may be at an early stage of its development, it made me start thinking about the area of the field, the problem and the solutions, which authors highlighted. It is a big plus.

The main weakness of the paper is that the novelty sounds very important, it seems to be of a large scale, while the experiments just slightly demonstrate its advantage over the previous methods.

**Issues, which affect my rating**
1. If I understood correctly, the rotation part of the proposed filters is very close to the idea of Harmonic Nets by Worral et al. [1] and E(2)-steerable CNNs with irreducible representations by Cesa & Weiler [2]. If so, then I think that a more accurate comparison to [1, 2] should be presented. If not, please add some remarks to the sections 4.1 to highlight the difference.
2. The authors mention several times that the method is not based on group theory. I think such a distinction is misleading, artificial and somewhat incorrect mathematically. Moreover, the exact roots or each cited paper go far beyond than just "group theory". If it is not based on group theory, could you demonstrate how this fact may be useful to a reader. For example, can it be generalized to a non-group transformation?
3. Introduction, the "To achieve this property," paragraph. I find several phrases here a bit misleading and are of less respect to the previous approaches. I suppose it is just wording. For instance, the phrase "these “copy-paste” approach with group theory" may give a reader an incorrect understanding of the previous appoaches. What the authors call here as "copy-paste" is better to be referred to as "weight-sharing". It is a) more convenient and b) less vulgar. Finally, for some methods, such as [1, 2] it is not necessary to reuse filters to obtain their rotated versions.
4. One of the arguments for the proposed method is that it allows for equivariant models with lower complexity than quadratic. In Worrall & Welling [3] and Sosnovik et al. [4, 5] it is demonstrated that scaling can be implemented as dilation (integer and fractional). Moreover, it is possible to implement it by utilizing multiple cores of GPUs to make it O(1) in terms of time, which is also discussed in the papers, as the weight sharing scheme is replaced with dilation in these papers. I think a more accurate comparison of computational complexities would help.
5. I suppose that the results in Table 5 are copied from the paper [6]. The main issue with equivariant models when it comes to comparing a new model to the previous approaches is the hyperparameter tuning. In the original papers the authors spent some time to perform such a process. For example in [2, 4, 5, 7] the authors train and test by using exactly the same protocol and their reported results are more accurate than what is present in Table 2. One of the main reasons is because they used a WideResNet. If it is possible to additionally compare a WideResNet with the proposed blocks to the previous models. It would significantly increase the contribution of section 5.2

**Issues, which do not affect my rating**
1. The paper contains typos and writing issues. For instance, the word "trannels" (p1) seems rather misleading. The word "convolute" (p5) is incorrectly used (I suppose it must be "convolve").
2. In Eq. 1 it makes sense to write the transformation as a 3x3 Matrix from the very start by using the block notation. It will improve the clarity of the equation. Because then, you introduce the 3rd coordinate to $[x_1, x_2, w]$. It also makes sense to write what the $w$ is.
3. A more detailed illustration of how the equivariance of the SimBlock works would help. 
4. Very bottom, page 2. I am not sure if "Worrall and Welling" is the right reference for "Scale-space theory". I suppose the placing of the reference is just incorrect.

- [1] Worrall D. E. et al. Harmonic networks: Deep translation and rotation equivariance. CVPR – 2017
- [2] Weiler M., Cesa G. General E(2)-equivariant steerable CNNs. NeurIPS - 2019. 
- [3] Worrall D., Welling M. Deep scale-spaces: Equivariance over scale. NeurIPS – 2019.
- [4] Sosnovik I., Moskalev A., Smeulders A. Disco: accurate discrete scale convolutions. BMVC - 2021
- [5] Sosnovik I., Moskalev A., Smeulders A. How to Transform Kernels for Scale-Convolutions. CVPR – 2021, VIPriors Workshop
- [6] Gao L., Lin G., Zhu W. Deformation robust roto-scale-translation equivariant cnns. TMLR - 2022
- [7] Sosnovik I., Szmaja M., Smeulders A. Scale-equivariant steerable networks. ICLR 2020
clarity,_quality,_novelty_and_reproducibility: The paper is clear, the illustrations are of good quality. The approach and reasoning are novel. I am not able to evaluate it's reproducibility because the code is not provided. However, from the provided information it is possible to implement the main building blocks of the paper. However, I want to highlight the fact that if the authors release the code, it will improve the visibility of such an approach in the community, which is always a huge contribution!
summary_of_the_review: The paper contains a novel idea and highlights an important issue: combining translation, scale and rotation in one networks is still challenging from the implementation point of view.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides a new way to construct convolution-like neural networks that are equivariant to continuous translation, rotation, and scaling without relying on group theory. More specifically, they use the local scale/orientation of the input signal to adapt the scale/orientation of the filters (with the proposed Fourier-Argand representation) and implement spatially-varying convolutions. The proposed method does not introduce any significant computational overhead. They empirically verify the equivariance and generalization ability of their proposed models on scaled and rotated versions of MNIST and STL-20 datasets.
strength_and_weaknesses: **Strength:**

1. The proposed construction of equivariant networks does not rely on group theory, so it can potentially generalize to transformations that are hard to describe with transformation groups. Furthermore, their implementation is efficient and does not have computational overhead for larger groups.
2. The technical parts are solid, and their empirical study sufficiently supports their claim.

**Questions (potential weaknesses):**

1. The modulation of filters depends on the optimal scale/orientation (computed in equation 13) which has the form of argmax function. I think it can lead to computational instability because: Firstly there might be multiple maximum values, how can you choose under this scenario? Secondly, even though this extreme case does not happen in practice, there may be cases where a little bit of noise can lead to dramatic change, e.g. orientation $0^{\circ}$ is only slightly better than $180^{\circ}$, a small amount of noise could lead to the filters rotate by $180^{\circ}$. How are you planning to fix this issue?
2. Still on equation 13, it seems to me that the scale/orientation of the filters depends on the input signal over the entire domain. That is to say, to compute local features, a change far away could change the orientation/scale of the filters. Would that break the locality of the convolutional layers? 

However, even with all the potential drawbacks mentioned above, I still think this is an interesting work that can inspire valuable research.


clarity,_quality,_novelty_and_reproducibility: **Clarity:**

The writing is well-paced and clear to me. However, I think the authors should give more intuition rather than math in the main text. And the intuition parts should be before the rigorous mathematical statements parts. Moreover, I think it might help a lot if the authors can design some simplified concrete 1D examples before talking about the general cases.

**Quality:**

The technical parts are solid. Although I am not able to check all the math line-by-line, I believe the main results are all correct. The experimental results are sufficient to support their claim.

**Novelty:**

The way of constructing equivariant networks presented in this paper is new and has the advantages of computational efficiency and being generalizable outside of the group theory framework. In addition, according to the author, no previous work has achieved equivariance w.r.t. continuous translation/rotation/scaling at the same time. This paper offers a very simple solution.

**Reproducibility:**

The paper has provided sufficient technical details. The code has not been provided yet but one should be able to implement their model with the given details.
summary_of_the_review: I recommend acceptance of this paper based on the following:
This paper provides a new way to construct convolution-like neural networks that is equivariant to translation/rotation/scaling, with appealing computational efficiency, and can potentially be generalized to other transformations without relying on the group theory framework. The technical parts are solid and the empirical study is sufficient. Even though I have some technical concerns outlined in the question section, they do not stop this paper from being valuable research work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method to generate scale-rotation equivariant feature maps in a novel cnn architecture. This is achieved by constraining the filters to be steerable with respect to a scale-rotation equivariant basis. The authors define this basis to be image dependent and validate the approach on STL-10 and variants of MNIST.

strength_and_weaknesses: *Strengths*: The authors address the challenging and important problem of designing cnns to produce representations equivariant to larger transformation groups. 

*Weaknesses:*
- I found the paper somewhat difficult to read:
  - I believe the paper would benefit from proofreading by an editor.
  - Some additional language in the appendices could make the proofs easier to follow
- I think it is incorrect to say that previous methods can only achieve equivariance to rotation or scaling; the authors themselves cite Esteves et al. 2018 as an example

*Questions:*
- Does the size of the filters impact performance? I ask since (I imagine) the size of the filter determines the scale range.
- I think there may be a typo in Appendix A, I don’t see 4→5 in equation 33, specifically, what happened to exp(\rho m_{m_{t_1}})? Is it possible it should cancel with exp(-\rho m) in 5→6?
- I don’t see 3→4 in Appendix B, should the last quantity be \lambda^{-m}?
- What is meant by a “covariance indicator” (Sec 4.1 para 1)?
- The last step in (19) seems inconsistent with (17) (i.e., pre- vs. post- multiplication with T^{-1}), am I missing something here? Is this because the group is Abelian?
- Is P \circ L_T = P proved somewhere?
- Are \Lambda_f (x), and \Gamma_f (x) computed for every layer at every iteration? Have the authors compared the computational costs of the proposed method to the baseline methods?

*Possible typos:*
- “Specifically, We” → Specifically, we
- ““equivariance in rotation, scale” → equivariant in rotation, scale
- ““if we want to convolute…” → if we want to convolve
- ““We can pre-convolve the filter” → we can pre-convolve the image(?)
- ““implementable formulation (Equation (41))” → implementable formulation (Equation (14))
- ““following summarized condition: T” → following summarized condition: T^{-1}
- ““the communicated operator” → the commutator operator
- ““satisfies the equivalent property” → satisfies the equivariance property
- “Eq 23: “f (x + Mf (x)” → f (x + M_f (x)t)
- ““y_t = x + M_f (x” →y_t = x + M_f (x)t

clarity,_quality,_novelty_and_reproducibility: *Quality:* The approach appears to be technically sound and the claims seem well supported theoretically and empirically.

*Clarity:* although parts of the paper are difficult to read, the paper is generally well organized and the figures are useful.

*Originality:* As far as I know the work is novel. 

summary_of_the_review: I think the proposed work is interesting and of value to the community; however, there are several typos which should be resolved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new approach to building convolution-like networks that are equivariant to rotation, translation and scale. The idea is to extract a local scale and orientation and use this to filter at that scale/orientation. The reviewers appreciated the novelty of the idea, the experiments validating the improved OOD performance, and experimental validation of low equivariance error. 

Some concerns were raised about certain claims made in the original version (e.g. "not based on group theory") and comparison to previous work. I agree with the reviewers that the original paper, and to some degree the current paper still, makes some unsuported claims and is not entirely balanced in discussion previous work. For instance, already the first "Steerable CNNs" paper mentions the idea of decoupling the feature dimension from the size of the group, and indeed all steerable CNNs achieve this, so this is not a novel feature of the current paper. How this is achieved is however novel; since the SimConv is not linear in the input (since it depends on the nonlinear M_f(x)), it is not a steerable convolution.

One additional note is that the work bears some similarity to "Deformable Convolutional Networks" by Dai et al., and to a lesser extent "Spatial Transformer Networks" by Jaderberg et al. It would be nice to discuss these in the related work as well.

Personally, I would have liked to see larger-scale experiments to show that the method really works on important problems. Adding more details on implementation (e.g. in an appendix), and a code release, is also strongly encouraged.

In summary, this paper proposes a novel and promising approach to joint rotation/translation/scale equivariant CNNs. There are some minor issues remaining and we encourage the authors to address them as much as possible. Nevertheless I recommend that the paper be accepted to ICLR.
justification_for_why_not_higher_score: Some concerns about claims (see above); lack of large-scale experiments; writing quality is somewhat lacking.
justification_for_why_not_lower_score: The paper has an original idea that seems to work well on small-scale problems
Decision: Accept: poster