Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a framework (SESN) for learning deep networks that possess scale equivariance in addition to translation invariance. The formulation is based on group convolution on the scale-translation group. Filters are represented as the coefficients of a set of continuous basis functions, which are sampled (once) at a discrete set of scales. The theoretical formulatioin is clear and interesting. The approach is evaluated in terms of image classification accuracy. The set of baselines is quite exhaustive, including recent papers and papers that are not widely-known.

The most significant improvement for the STL-10 dataset was obtained by the SESN-B variant. This is interesting, because it applies the same operation independently at multiple scales and periodically performs global pooling over scale.

The effectiveness of the approach was demonstrated in the low-data regime, where the inductive bias of scale equivariance is more likely to help.

Overall I found the paper to be thought-provoking and well-executed. There are a number of questions that I would still like to see investigated, but nevertheless I feel that this paper already represents a worthwhile contribution.

Most important issues and questions:

(1.1) The SESN-B architecture resembles quite closely the SI-ConvNet architecture of Kanazawa et al. (except that that paper resized the images instead of the filters). While your approach may be more computationally efficient, it's not clear what leads to the improvement in accuracy here? Can you explain the difference?

(1.2) I would have preferred to see the approach demonstrated on a task which possesses scale equivariance, such as semantic segmentation.

(1.3) To argue in favour of the continuous basis, it would have been more convincing to compare against directly learning the filters at the highest resolution and obtaining the other filters by downsampling. This would not represent a runtime cost during inference.

(1.4a) It seems that SESN-C should contain SESN-A as a special case. However, SESN-C is worse than SESN-A. Do you have any idea whether this is due to optimization difficulty or over-fitting? Could you compare the training objectives?
(1.4b) It is stated that the scale equivariance of SESN-C is worse than SESN-A and -B. However, it should still be scale equivariant, except for boundary effects in scale? What is the parameter N_S in this experiment compared to the number of scales S? And the same question for the plot on the right in Figure 2.
(1.4c) How does SESN-C have the same number of parameters as SESN-A and SESN-B? I thought that more parameters would be required to compute interscale interaction. Was the number of channels reduced?

Issues with clarity:

(2.1) The explanation of equation 10 is not clear. In particular, the diagonal structure in Figure 1 is not stated anywhere in the text, it is simply explained as an expansion from [C_out, C_in, S, V, V] to [C_out S, C_in S, V, V].

(2.2) It's not immediately apparent how multiple applications of convHH are used to provide interscale interaction. I assume it is achieved by shifting f or psi in the scale dimension for each application of convHH, or equivalently by modifying the base-scale in the basis?

(2.3) The explanation of "scalar" and "vector" variants in the experimental section was not perfectly clear. It is stated that "all the layers have now scalar input instead of vector input." However, I understood that the max-reduction was only over the scale dimension, not the channel dimension, so that the inputs are still vectors? This is confusing as a reader.

(2.4) The expansion of the filters to a diagonal structure is described in the "implementation" section. However, it seems that this would entail wasteful multiplications by zero. Nevertheless, SESN is shown to be highly efficient in the appendix. Do you avoid these pointless operations in the actual implementation?

(2.5) It was not immediately obvious how $\psi_{\sigma}(s, t)$ was related to $\psi_{\sigma}(x)$.

Other details:

(3.1) In Figure 2, how many layers does the network have which was used to construct the middle plot?

(3.2) It would have been useful to include a study of the effect of the range and resolution of the scale space.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper describes a method for integrating scale equivariance into convolutional networks using steerable filters.  After developing the theory using continuous scale and translation space, a discretized implementation using a fixed set of steerable basis elements is described.  Experiments are performed measuring the error from true equivariance, varying number of layers, image scale and scales in scale interactions.  The method is evaluated using MNIST-scale and STL-10, with convincing results on MNIST-scale and bit less convincing but still good results on STL-10.

Overall, I think this is a nice paper with generally good explanations and experiments probing the behavior.  I would have liked to see more probing into the effects of number and distance between scales.  Table 1 and corresponding text say that a significant advantage of the approach is that it can handle arbitrary scale values, but there was no explicit exploration of the effects of using this beyond one set of scales per experiment/dataset.  What scale values can be sampled, which work best, and why?

Also, while the MNIST-scale experiment seems convincing, I think the STL-10 is a bit less (but still OK):  Although the method outperforms other methods and appropriate baseline models, it's a little disappointing that pooling over scales (which I would would convert the equivariance to invariance) is best, and inter-scale interactions increase error.  (Perhaps this is not too surprising in retrospect, as images may have limited scale variation from camera position in this dataset, but significant within-class viewpoint variation.)

Even so, I still find the method concise and of interest, with the basics evaluated, even if some of its unique advantages may have been better explored.


Additional Questions:

* Inter-scale interaction could be elaborated a bit more.  End of sec 4 says, "use convHH for each scale sequentially and .. sum".  I believe this is sequencing over scales in the kernel; explaining a bit better how this is implemented, including the shape of w in this case, would be helpful.

* Which scales were chosen for the fixed basis?  How large in spatial extent are the kernels in the basis elements, at each scale?

* In the implementation, what is the value of V (sampled 2d conv kernel size)?

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper proposed scale-equivariant steerable convolutional neural networks that is able to preserve both the translation and scaling symmetry of the data in the representation. To achieve this, the authors developed the scale-convolution blocks in the network, and generalized other common blocks, such as pooling and nonlinearity, to remain scale-equivariant. Extensive experiments have been conducted to show that the proposed scale-equivariant network
(a) is indeed scale-equivariant even with numerical discretization
(b) achieves better classification performance when compared to non-scale-equivariant networks as well as previously proposed locally-scale invariant networks.

Overall, this is a very good paper. The paper is well-written and well-organized. The newly proposed scale-convolution is the most general way of achieving scale-equivariant representations. Experiments are convincing and justifies the usage of the proposed architecture in dealing with multi-scale inputs.

One question to ask that does not effect my rating:
There is a very similar paper submitted to this conference:
https://openreview.net/forum?id=rkgCJ64tDB
Would you care to make a comparison between these two manuscript?
review_title: Official Blind Review #2

Meta-review: This work presents a theory for building scale-equivariant CNNs with steerable filters. The proposed method is compared with some of the related techniques . SOTA is achieved on MNIST-scale dataset and gains on STL-10 is demonstrated. The reviewers had some concern related to the method, clarity, and comparison with related works. The authors have successfully addressed most of these concerns. Overall, the reviewers are positive about this work and appreciate the generality of the presented theory and its good empirical performance. All the reviewers recommend accept.
Decision: Accept (Poster)