Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes DiffStride as a drop-in replacement to standard downsampling layers. It extends previous work Spectral Poolnig and learns the size of the cropping box in the frequency domain by backpropagation. Experiments are conducted on audio and image classification, and the results show that the model can learn non-integer stride and adapt different initial stride well.
main_review: The idea is interesting and makes sense to me. The writing is clear and easy to understand.  However, from the experiment results (especially table3 and 4), it seems that the proposed method has marginal improvement compared to regular strided conv and spectral pooling baselines. It seems that the default setting can already achieve very good results on ImageNet and how it will affect the model performance when the model is large enough remains unclear. The benefit of learnable strides is not fully demonstrated. It would be great if the authors could implement one or more future works in Sec. 4 to showcase its capability further and show the proposed module's overheadein detail to let the readers better understand its limitation.


summary_of_the_review: Overall I think this paper presents a novel idea to learn stride in the downsampling layer. However, the current results are not good enough to showcase its effectiveness by making the stride learnable. The authors may want to find more ways, as discussed in Sec.4, to show its value.

-- Post rebuttal
After reading the authors' response, I raise my rating to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose DiffStride, a technique for learning the stride of downsampling operations in neural networks by gradient descent. Specifically, similarily to SpectralPool, the feature map is transformed to the frequency domain by a Discrete Fourier Transform, where it is then cropped according to learnable parameters. The authors envision a simple soft-relaxation of the cropping in order to allow the gradient to flow towards cropping parameters. Performances against both standard and random stride policies are reported on a number of datasets, both for audio and image recognition. Moreover, the authors introduce a regularization objective that penalizes the use of small strides, in the interest of encouraging downsampling for improving computational and memory cost.
main_review: PROs
- The paper is very well written and pretty much clear at a first read. All the introduced components are properly motivated intuitively, and figures and algorithms really help the reader in understanding.
- The authors excel in framing their work within related art. Specifically, they discuss both standard alternatives such as pooling, as well as alternatives based on fractional stride. For all of them, issues are properly pointed out such that the motivation behind this paper is very clear.
- The paper is, to the best of my effort, technically sound. In terms of novelty, one might argue it being a bit incremental over SpectralPool, as it substitutes the cropping hyperparameters with learnable values. However, i) this step is not trivial, as the cropping operation is non-differentiable, and the authors introduce a solution to that and ii) this modification enables a significant improvement in performances, registered across a number of datasets.
- The authors support their proposal by carrying out experiments on multiple datasets. Specifically, 6 datasets are employed for audio recognition and 3 (CIFAR-10, CIFAR-100 and Imagenet) are used for image recognition. In my opinion, for a paper of this type, attacking a very fundamental and obiquitous operator in modern architectures, being able to showcase improvements on non-toy datasets such as Imagenet is remarkable and noteworthy.
- For every experiment, the authors report the mean and standard deviation over multiple trials, strenghtening the reliability of the conclusions.
- The authors properly discuss limitations of their work, by highlighting the computational cost, some failure cases on DenseNets and implementation challenges on specific types of hardware.

---
CONs

I think this paper is very solid, and I consider the following points as minor concerns.
- In Sec 3.1, the authors claim that as "DiffStride learns different strides for the time and frequency axes", and as such "this justifies using a different parameter for each dimension rather then sharing strides". However, this validation is flawed. Just because the model takes advantage of this flexibility, it doesn't necessarily mean that it is beneficial in the end. This might be the case if we completely trusted gradient descent to reach global optimums of the cost function, which is not the case. Indeed, the final stride configuration might depend much on the initialization, as also suggested by the experiment in Fig. 3. Therefore, to validate such a claim the authors should simply include a baseline model where DiffStride is applied with shared striding parameters for the time and frequency axes.
- Fig. 4 would be much clearer if on the x axis MACs or FLOPs were represented, instead of the regularization term. This would help quantify how much computation DiffStride can save without losing significant performances. As it is, the value of the regularization term dos not tell much about actual computational cost.
- To my understanding, within experiments the authors substitute a few downsampling layers (not even all of them) in a network with DiffStride. As a reader, I would have expected that every convolutional layer would be equipped with its own learnable striding. I grasp that this may not be practical but I don't have a clear view of the reason. The authors should consider adding some motivation behind this choice.
- Another point that may be worth discussing as an extension/future work. May the DiffStride technique be employed within a conditional computation framework, by predicting the striding parameters conditioned on the current example? I think that this strategy might succeed especially when aiming at optimizing the computational cost of a model. I do not expect such an extention to be implemented and tested within the rebuttal period. I would just like the authors' opinion about that (potential, concerns etc).
- In Tab. 1, the Speaker Id dataset provides no information whatsoever, as all methods score a perfect accuracy of $100.0 \pm 0.0$. I suggest the authors to remove this dataset, as it is apparently beyond trivial, and including it downgrades the quality of the experiment rather than increasing it.
- On CIFAR-100, the authors show that SpectralPool is significantly better than strided convs even with the same downsampling hyperparameters. Can the authors provide an intuition for this behavior?  
summary_of_the_review: Overall, I recommend the paper for acceptance. The contribution is novel and clearly motivated, and experimental results are encouraging across different datasets. I think this paper can be of interest for many within the ICLR community. There are some improvement points as discussed above, but in my opinion the strengths clearly outnumber the weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach to learning optimal striding parameters in convolutional networks. The proposed approach, DiffStride, is a downsampling layer that builds on spectral pooling to allow for integer output dimensions, but arbitrary strides by cropping in the Fourier domain. Unlike spectral pooling, DiffStride relaxes the parameters of the cropping mask to be differentiable, and using the stop-gradient operator in the cropping. Results show that the proposed approach can work well as a drop-in replacement for normally fixed pooling layers. It is also demonstrated that from different random initialisations a variety of different pooling approaches can be learned that acheive similar performance. Use of a regularisation term the attempts to encourage time and space efficiency reduces this variability and allows accuracy to be traded off.
main_review: Positives
---------

Overall I feel that the paper presents a really neat idea well. Besides a few minor issues (see below) the paper was enjoyable to read and describes the ideas well. There is thorough experimentation on a range of tasks and model architectures that demonstrate the power of the proposed approach. 


Concerns
--------

Really, the only major concern I have is around the rather limited discussion of the limitations. In particular, I'd at least like some discussion (or thoughts) on why it doesn't improve densenet performance, and I'd like to see experimental results presented on the impact of the use of DiffStride during training and inference, both in terms of measurable memory usage and impact on training & inference wall time. Obviously such results can be caveated with a statement that the implementation could be improved as per the existing discussion in the limitations section.

 
Minor points
------------

Please check and fix algorithm 1 - I believe line 4 should be using the output from the filtering with the mask, rather than the raw FFT result. It would also be better not to reuse the $\tilde{y}$ symbol on lines 4 and 5, and it would also be helpful if the symbols (the variants of the intermediate computations $y$, $\tilde{y}$}) were clearly labelled on Fig 1.

summary_of_the_review: Overall, as can be seen from my review I'm satisfied that this paper makes a good contribution.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work the authors introduce a differentiable stride formulation, which allows for learning the stride value. To this end, they propose learning the size of a cropping mask in the Fourier domain, which allows for learning how to perform resize in a differentiable way. Authors present experiments on several datasets on different domains (image, audio), including large scale datasets.
main_review: + The paper is well-written and the proposed formulation is sound. 

+ Also extensive experiments are provided on several datasets to demonstrate the benefits of the proposed method.


- Authors demonstrate that the proposed method can recover from different initial strides and learn the optimal one. However, my most important concern is the lack of appropriate comparisons with neural architecture search approaches, since - in my view - these are the most direct competitors of the proposed method. Therefore, I would expect experiments and appropriate discussions between the cost and benefits between the proposed method and neural architecture search approaches.

- The proposed method seems to target only residual architectures. Discussion on how the proposed method could be used with other architectures would be beneficial

- Also, providing experiments with different architectures (apart from resnet) on the same dataset would also further improve the confidence on the obtained results.

- What is the actual overhead of the using the proposed method in real applications? How much does the training time and memory usage increases?
summary_of_the_review: Overall, I think this is a good paper with an interesting approach on differential stride learning. There is no theoretical discussion and I would like to see some additional experiments, however I feel that this work is slightly above acceptance threshold, mainly given the novelty of the application.
review_title: null

Meta-review: The paper proposes a method to learn the stride of downsampling in deep networks using a  gradient-based learning approach. The main idea is to work in the frequency domain and to learn the cropping mask in that domain. The authors also introduce a regularization for applications seeking computationally and memory efficiency. The authors investigate the interest of the approach on a number of datasets with audio and image data. 

The reviewers praised the paper, appreciating the elegance of the approach and the effort made to thoroughly evaluate it. The reviewers also appreciated the clarity of the exposition and the care in the reporting of the results. The reviewers also expressed some concerns about several choices of the design (stride sharing) and the lack of detail in some experiments (computational /memory efficiency). Finally, the reviewers wished the paper had more theoretical grounding.

The authors submitted detailed responses to the reviewers' comments. After reading the responses, updating the reviews, and discussion, the reviewers found that the responses were ‘reinforcing [their] initial assessments' and their several concerns were satisfactorily addressed. Moreover several of the reviewer’s suggestions clearly already led to an improved manuscript with very thorough experimental evaluation and simpler approaches for stride sharing. 

The paper proposed an elegant, learning-based, approach to one of the most important design choices in deep network architecture design: the strides in the convolutions. The authors provided a careful and thorough experimental evaluation, and moreover improved it during the review process following the reviewer’s feedback. 

Accept, definitely. 
Decision: Accept (Oral)