Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a scale-equivariant Neural Networks model to solve the multi-scale image classification task. The main contributions are the proposed joint convolution across the spatial and scale space, and the decomposed filters to reduce computation cost and improve robustness.

The paper is generally well-written and well placed in the literature. Different from existing work on the scale-equivariance CNNs, they build a more general model that allows different scale information to transfer through different layers. The experiments also suggest the model achieving scale-equivariance and the superior performance of the proposed method compared with several baselines. However, for the novelty part of the proposed method (the separable basis decomposition), it is not convincing enough because the differences between this paper and Cheng et al. (2019) and Weiler et al. (2018b) are not clear.

Questions:
1. The authors claim that the joint convolutions across the space and the scaling group are both "sufficient" and "necessary". Although the experiments show that the proposed method achieves better performance, which might indicate the "sufficiency" of the architecture, however, how should the "necessity" be proved?
2. How should "K" and "K_\alpha" be selected (Table 1)? When data is enough (e.g., 5000), it seems that only some specific values can improve performance.
3. How much can the model size be reduced compared with other multiscale image classification models? And could you provide some concrete comparisons about the computation cost?

Typos:
- 1st line in 2nd paragraph of Introduction: rotation-equivarianc(e)
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper is introducing a scale-equivariant CNN architecture with joint convolutions over spatial and scale space. Moreover, the authors used decomposable convolutional filters to reduce the number of parameters. Based on Therom 1, it is shown that scale-equivariance is achieved if and only if joint convolutions are conducted over spatial and scale space. 

Overall, the contribution seems meaningful but incremental that the method is almost similar to the 'RotDCF: Decomposition of convolutional filters for rotation-equivariant deep network'.

- to verify scale equivariance, visualizing features with tsne would be interesting
- in table 1, what is the reason for doing experiments with and without batchnorm?
- what if the baseline's size is close to the proposed method, how it would be improved?
- it would be better to show how the decomposed filters reduce the total number of parameters.
- also comparing flops would be informative.
- the author's name of 'Locally scale-invariant convolutional neural networks' is wrongly written.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: *Paper summary*

The authors propose a CNN architecture, that is theoretically equivariant to isotropic scalings and translations. For this they add an extra scale-dimension to activation tensors, along with the existing two spatial dimensions. In practice they implement this with scale-steerable filters, which are discretised and truncated in both the spatial and scale-dimensions. They also provide a deformation robustness analysis.

*Paper decision*

Thank you for writing a very interesting paper indeed. I have to admit I am somewhat on the fence about this paper. I think it contains many nice ideas, but the experimental section is somewhat lacking in terms of comparisons or insights which I can gain, and the theory has some missing elements too (which I shall discuss below). For that reason, I am recommending a weak reject, but would very easily upgrade this if the authors provide strong rebuttal to my comments below.

*Supporting arguments*

-Experiments: The experiments are quite light, although I must admit that many other works in the area of equivariance are also light on experimentation and if there is enough theory, that is not such a great issue. The main issues I have are 1) the choice of experiments, 2) the comparisons against an insufficient number of baselines, and 3) the ablation studies.
1) The choice of experiments: I think looking at scaled-MNIST is not particularly useful as an experiment nowadays, unless used as a toy experiment. A larger dataset with real-life scale variations would have been better. Furthermore, I’m not sure what the image reconstruction task is supposed to tell the reader, that the ScDCFNet is able to generalise to new scales?
2) There is a lot of concurrent work on multiscale architecture. To name a few:
Multigrid Neural Architectures, Ke et al., 2017
Feature Pyramid Networks for Object Detection, Lin et al., 2017
Multi-Scale Dense Networks for Resource Efficient Image Classification, Huang et al., 2018

Deep Scale-spaces, Worrall and Welling, 2019

I believe these works should at least be cited, but ideally compared against.

3) I would have liked to have seen some numerical results for the verification of scale equivariance. In the first layer the equivariance improbably going to be close to perfect because there is no truncation of the scale-dimension in the network, but after this layer I predict the equivariance error increases due to truncations effects. This would be a similar effect to other works, such as “Deep Scale-spaces” (Worrall and Welling, 2019).

-Theory: I think the theory is very interesting and a meaningful contribution in its own right. The authors treat scale-translation in continuous space as a group action on signals. This motivates the convolution presented in Theorem 1. This is a group convolution as per Cohen and Welling, (2016) modified to continuous space for a non-compact group. (Actually, this should be mentioned in the text as a matter of good scholarship). This is nice, since the group convolution has not been used in for a regular representation of a non-compact group (other than translation) as far as I can tell. What is perhaps not clear for me is how the theory breaks down in practice, since the implementation requires discretisation in space AND scale, which is not discussed much and furthermore, filters are restricted in spatial AND scale dimensions, leading to truncation errors in the equivariance. This last perspective was not discussed, and I feel it rather should be.
The theory goes further into deformation stability, which is a fresh perspective in the equivariance literature, so I am happy for its inclusion. Perhaps more motivation for why you think this is necessary would be warmly welcomed. 

*Smaller questions/notes for the authors*

- Technically this is scale-translation equivariance, you even write this in the method section of your paper, why is it not in the title? The reason I mention this Is because there are scale equivariant networks, which are not translation equivariant in the literature, see “Warped Convolutions: Efficient Invariance to Spatial Transformations” (Henriques and Vedaldi, 2019).
- Please make the link between Theorem 1 and the group convolution of Cohen and Welling (2016)
- Last paragraph of page 1: A steerable-in-scale filter does exist, see “Deformable kernels for early vision” (Perona, 1991)
- Please use numbering for all display-mode equations.
- This scheme works perfectly in the continuous-image setting, but how about for discretized images? In that case it cannot be scale-translation equivariant because the scaling-action is no longer part of a group.
- In equations 6 and 7, what is the specific motivation for using the laplacian eigen-decompositions as a basis? Is it for steerability with respect to the scale-translation action? Otherwise, surely any basis will do?
- Remark 2: If you are considering a truncation of the scale-axis, surely you can still use an L2 norm when quantifying the robustness of your representation?
- Bandlimiting of the filters: I would consider citing “Structured Receptive Fields in CNNs”, (Jacobsen et al., 2016)
- Pooling: A useful citation here would be “Making Convolutional Networks Shift-Invariant Again” (Zhang, 2019). They precise low pass filter before pooling.
- Experiments: 
—I’m not clear on the reason to include reasons with and without batch normalization. This is quite unconventional
— Did you ever use scale augmentation? What is the effect of training on one scale and then testing on another? 
review_title: Official Blind Review #3

Meta-review: This paper presents a CNN architecture equivariant to scaling and translation which is realized by the proposed joint convolution across the space and scaling groups. All reviewers find the theorical side of the paper is sound and interesting. Through the discussion based on authors’ rebuttal, one reviewer decided to update the score to Weak Accept, putting this paper on the borderline. However, some concerns still remain. Some reviewers are still not convinced regarding the novelty of the paper, particularly in terms of the difference from (Chen+,2019). Also, they agree that experiments are still very weak and not convincing enough. Overall, as there was no opinion to champion this paper, I’d like to recommend rejection this time. 
I encourage authors to polish the experimentations taking in the reviewers’ suggestions. 

Decision: Reject