Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Authors improve DiffStride, a pooling layer with learnable strides, by replacing the Fourier transform with a DCT.
strength_and_weaknesses: Strengths:
* DCT-DiffStride is significantly better than DiffStride in the low computational cost regime.

Weaknesses:
* The main weakness of the paper is the minimal technical novelty (replacing DFT by DCT in DiffStride), which is hidden behind a lot of verbosity.
* Section 2.2 is very verbose and could be summarized. For instance, the example given for non-periodic signals is artificial. Also, how are properties of the DCT on AR(1) processes relevant unless authors justify they are a good model for the natural signals considered? What would have needed more details is the last paragraph which vaguely describes the implementation details.
* Similarly, the appendix provides a generic description of KLT/DCT/DFT that seems unconnected to the narrative of the paper.
* The weight of the regularization (100) is not discussed, neither how it was cross-validated.
* The results show that DCT-DiffStride only improves in the very low complexity regime. As such, and given that the technical contribution is minimal, it is hard to consider this paper as a significant contribution on top of DiffStride.
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper has almost 4 pages of "methods" yet no pseudo-code or implementation details are given to the reader, and there are rather a lot of high-level description of time-frequency representations which could be useful if it served the narrative and the reader's comprehension.

Novelty: minimal change to DiffStride (replacing a time-frequency representation by another one)

Reproducibility: No code, no pseudo-code and a lack of implementation details make it difficult to reimplement.
summary_of_the_review: DCT-DiffStride applies a simple tweak to DiffStride. Simplicity is not a flaw but rather a quality when 1) it serves better performance 2) is clearly justified and explained. Given the inconsistent experimental results, and the lack of analysis of the high performance in the low-computation regime, it's hard to consider this paper worth being published in its current state.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work extends the baseline model DiffStride by replacing the discrete Fourier transform (DFT) with the discrete cosine transform (DCT), which leverages the energy compaction properties of DCT. Experiments are conducted on natural signals non-natural signals. The results in image classification and modulation classification demonstrate the advantageous tradeoff in model complexity and model performance for DCT-DiffStride against DiffStride. In speech classification, at low model complexity, these two methods are comparable

strength_and_weaknesses: 1. This paper is well-written and easy to understand. The structure is very clear.

2. The architectures of this work are similar to DiffStride. The main innovation is the use of DCT. Knowledge in signal process is utilized. The energy compaction properties of DCT are well-known and widely used in lossy compression. 

3. In Figure 2, DCT-DiffStride achieves the best performing model at 40% model complexity. However, for CIFAR-100, the baseline DiffStride seems achieve better performance at similar model complexity (around 40%) than DCT-DiffStride. Moreover, for ImageNet dataset, is it possible to provide a comparison figure between DCT-DiffStride and DiffStride as Figure 2?

4. For audio dataset, DCT-DiffStride and DiffStride is comparable. In Figure 3 (b), It seems DiffStride even achieves slightly higher accuracy than DCT-DiffStride at the lowest model complexity. Moreover, why the performance of DiffStride considerably decreases at the highest model complexity (the red point between 0.6 and 0.8)?

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read. Improvement is acceptable, but experiments should improved especially on audio domain. 
summary_of_the_review: Overall, the idea of this paper makes sense to me, but I think the experiments results and the novelty of this paper may be marginally below the acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a DCT-based approach to learning strides to perform spatial pooling. The approach is an improvement over DFT-diffstride and shows benefits over it in a low-complexity regime.
strength_and_weaknesses: ## Strengths
1. DCT is shown to be better to represent signals compactly and the paper shows its benefits compared DFT-diffstride.
2. Overall the paper is clearly written.

## Weaknesses
1. The novelty is somewhat limited as it is a straightforward modification to DFT-diffstride. Even though, marginal improvements are shown in practice it is not clear if it is sufficient for a publication.
2. Experiments simply compare against DFT and it is not clear if this could outperform hand-tuned strides in standard architectures. Furthermore, how difficult is it to optimize for the decimation rates, it seems it involves differentiating through DCT. Please comment on this. 
clarity,_quality,_novelty_and_reproducibility: There are concerns about the novelty and quality of the paper. Overall the paper is clear. 
summary_of_the_review: The paper is a straightforward improvement over DFT-diffstride and marginal improvements are shown against it.

## Post-rebuttal
I acknowledge the authors' response. As noted by other reviewers, the contribution is incremental and not sure if it is sufficient for publication at ICLR. Therefore I'm keeping the original score of marginal-accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors consider the problem of reducing the size of hidden layers in convolutional neural networks (CNN). Size reduction is usually achieved by setting an appropriate stride for the convolutional layers. The stride is a hyperparameter and is usually fine-tuned using cross-validation. An alternative approach is DiffStride (DS, [1]). DS is based on the observation that strided convolutions over the activations of a CNN can be viewed as a low-pass filter over the Fourier-transformed activations. The cutoff frequency of a low-pass filter is a continuous parameter that can be learnt using gradient descent. Thus, the original recipe for DS is to compute the activations' discrete Fourier transform (DFT), remove frequencies above the learnable cutoff frequency and inverse DFT the filtered frequencies.

The authors propose extending DS by using the discrete cosine transform (DCT) instead of the DFT, as they argue that its properties are better suited for the problem at hand. The authors call their method DCT-DiffStride (DCT-DS). Furthermore, the authors propose a regularisation term to ensure that the model does learn a useful cutoff frequency instead of just keeping all the activations.

The authors benchmark their approach on a couple of standard image and audio classification tasks and a less standard modulation scheme classification task on radio signals.

References
[1] Rachid Riad, Olivier Teboul, David Grangier, and Neil Zeghidour. Learning strides in convolutional neural networks. ICLR, 2022
strength_and_weaknesses: ## Strengths
DCT-DS is a sound, well-motivated idea, and the proposed model complexity measure and regularization term based on it seem reasonable too. The authors perform a reasonable amount of experiments on a good variety of data types. The authors show that in the low model complexity domain, DCT-DS can significantly outperform DS.

## Weaknesses
 - While the idea is sound and reasonably well-executed, it is quite incremental. This is not a problem in itself, but it is unclear how beneficial the method is. This is because, with the exception of the modulation scheme classification task, the authors only show comparisons between DS and DCT-DS, so it is not possible to assess how well these methods perform compared to more traditional methods.
 - As I understand, the only difference between learning the cutoff frequency for a given model complexity for DS or DCT-DS and setting the stride size manually for standard convolutional layers is that during training, the model can allocate the size reduction more flexibly. Therefore, it would also be important to discuss how the  "distribution" of size reduction across different layers differs between DS, DCT-DS and the traditional methods.
 - The authors note, "To ensure stability during training, all methods utilize unit l2-norm global gradient clipping"  - it is not obvious to me what causes instability in the case of DS and DCT-DS. Is this a weakness of these methods, or is training unstable anyways?

## Questions 
 - Why does the squared penalty term in Eq 4 ensure that the model complexities of DiffStride and DCT-DiffStride will be better matched compared to Eq 3? Is it merely the fact that it encourages that the model complexity exactly matches $\hat{c}$ rather than just making it an upper bound?
 - Could the authors comment on why DiffStride overtakes DCT-DiffStride in Figure 2? Furthermore, why does the accuracy decrease as the model complexity increases? Is this simply showing overfitting behaviour, or is there a different reason?
clarity,_quality,_novelty_and_reproducibility: The paper is reasonably clearly written, with a few more striking things that need fixing:
 - "However, in many applications the DFT can often be outperformed by the discrete cosine transform (DCT) (Ahmed et al., 1974) due to relaxed assumptions of periodicity and better energy compaction" - I believe this statement is imprecise; the assumptions are not necessarily relaxed. They are just more fit for the problem.
 - "Naturally occurring signals are often defined as signals that originate in nature..." - this sentence needs rewriting.
 - "We note that this does not include complexity of linear layers as they are also kept constant in the network"- do the authors mean fully connected layers (convolutional layers are linear too)?
 - "In our regularization term, we normalize each output shape by the maximum shape that the output could be (i.e., no decimation is performed)" - I believe this normalization is implied for C in Eqs 3 & 4. However, it is not present in Eq 2 and should probably be fixed. 
 - "However, it is clear that the use of frequency-based decimation methods can be applied" - this sentence needs to be rewritten.
summary_of_the_review: Overall reasonable, though quite incremental contributions mainly lacking some empirical studies/comparisons. I will be happy to raise my score if the authors address my concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Thank you for your submission to ICLR.  The reviewers agree that while there do appear to be some benefits of the approach, the contributions here (which largely involves replacing the DFT, from past work on differentiable strides, with the DCT) are likely too incremental to warrant acceptance to ICLR in its current form.  Though the authors do importantly point out the larger scope of these contributions, ultimately the consensus feeling of the reviewers is that the higher-level arguments of the authors were not fully satisfactory when it came to justifying this change from a broader perspective.
justification_for_why_not_higher_score:  The reviewers generally felt that the proposed contribution is quite incremental and not yet up to the bar of an ICLR publication.
justification_for_why_not_lower_score:  NA
Decision: Reject