Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new downsampling method for sequential data, which supports varying downsampling ratios across the same data instance. To achieve this, it rearranges each time step on a one-dimensional line segment, which is used to learn an alignment matrix for performing the downsampling. Extensive experiments across several audio processing tasks validate the effectiveness of the proposed method.
strength_and_weaknesses: **Strength**

1. It is interesting and novel to apply varying downsampling ratios across the same data instance to better extract the task-relevant information.

2. The proposed method is extensively validated across several audio processing tasks, indicating its general effectiveness on audio processing.


**Weakness**

1. The major concern for the proposed method is its generality for 2D inputs like images, considering all the baselines, including spectral pooling and DiffStride, also work for 2D images. Intuitively the principles discussed in Section 3.1 can be applicable for 2D images but the alignment matrix A should be defined on 2D distances and the introduced computational overhead should be quite large. A discussion for such generality is expected, otherwise the proposed method is limited to sequential data and not so general as the baselines.

2. Considering the fully connected layers in AdaStride are executed for each input token, this may incur larger overheads and limit the scalability of long sequences.

3. The author analyzes that the spectral pooling is a better choice for the TUT dataset with high-frequency noises in Table 1. Although this is reasonable, I wonder whether the proposed AdaStride can be also applied in the frequency domain on top of spectral pooling for improving the accuracy on the TUT dataset.

4. The WER improvements in Table 3 is limited. Experiments on low-resource speech, e.g., LibriSpeech-10h/100h, may lead to more extensive comparisons.

5. The margins around figures and tables are sometimes too large, e.g., Table 3 and Figure 4, which can be better organized.





clarity,_quality,_novelty_and_reproducibility: This paper is clearly written with good clarity and has made novel contributions. Source codes are provided in the supplementary materials.
summary_of_the_review: Considering the interestingness and the achieved experimental improvements, I tend to rank this paper as marginally above the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors propose AdaStride, a differentiable layer that learns an input-dependent and irregularly sampled downsampling.
strength_and_weaknesses: Strengths:
 * Learning adaptive downsampling is a fundamental deep learning problem with very large potential impact across all types of tasks, models and modalities.
 * This approach displays two characteristics that are offered by strided convs, spectral pooling or DiffStride: 1) the striding is input dependent 2) the striding is uneven along the temporal axis and can emphasize certain segments
 * The algorithm is simple and despite the computational overhead it could be useful in practice
 * It's really nice to see this approach tested in a generative VQ-VAE setting, while the benchmarks are typically classification networks. 
 * Lots of details on the experimental setup in appendix

Weaknesses:
 * Model:
  * I suspect that authors do not show how close their approach is to the learnable alignment layer of [Efficient-TTS](http://proceedings.mlr.press/v139/miao21a/miao21a.pdf) which seems to accomplish the same thing (learn an alignment between x and its downsampled version). If the models are indeed very close and authors simply apply it in the context of downsampling, the contribution is still worth publication but they should be honest about that while currently the reference to Efficient-TTS is kind of hidden in the text.
  * Authors should specify more clearly how they replace strided convs by adastride, even though I assume it first applies an unstrided convolution.
  * AdaStride-F is presented as an implementation trick, yet performance between AdaStride and AdaStride-F vary a lot (e.g. for ASR). Why?
  * This approach is conceptually very different from spectral pooling or DiffStride that exploit global spatial information for downsampling, while AdaStride only repeats or removes frames, as would be done in DTW. It would be interesting to have more discussion on this fundamental difference.
 * Experiments:
  * The gains on the audio classification tasks are not significant, and I find the justification for the particularly weak performance on TUT not convincing, or at least incomplete. The authors explain that the "information density is uniform" in acoustic scenes which is somehow incorrect (and speculative anyway) since a car honking or a bird chirping are transient events that can hint at the type of acoustic scene.
  * it is explained that AdaStride-S improves IEMOCAP but this is not what is shown by Table 1
  * The audio classification dataset being quite small and subject to wide variations in results based on confounding factors such as the random seed, I find it not very convincing to cross-validate configurations (from AdaStride to AdaStride-S) or hyperparameters (sigma) on them, in particular on the test set.
  * DiffStride is excluded from ASR experiments due to learning the downsampling factor, why is it a limitation?

Questions:
* Why is Adastride limited to sequential data? What are the technical challenges to extend it to an arbitrary number of dimensions?
* Section 3.1: why is the Miao reference put after the definition of a simple vector?


Typos:
* 4.1: "dowmsampling" -> "downsampling"
* Section 3.1: "a exponential" -> "an exponential"
clarity,_quality,_novelty_and_reproducibility: Very clear paper.
Novelty wrt Efficient-TTS is questionable.
Reproducibility is favored by extensive details on hyperparameters and experimental settings in appendix.
summary_of_the_review: The potential impact is large but the authors may have disregarded the main baseline they should compare against.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new downsampling method using adaptive strides for sequential data, especially audio data. Specifically, it rearranges each time step of an input on a one-dimensional line segment by using a method called vector positioning. By doing so, it builds an alignment matrix for the downsampling. The paper shows the results of the method on different tasks including audio classification, automatic speech recognition, and discrete representation learning, and claims the method is generalizable and effective.
strength_and_weaknesses: Strength: The paper written is good; the idea is easy to follow; the motivation is clear.

weaknesses: See the next Section. 
clarity,_quality,_novelty_and_reproducibility: Some main concerns:

1. The method belongs to pooling methods used for networks. Some related works are missed.
      LiftPool: Bidirectional ConvNet Pooling. J. Zhao and C. Snoek. ICLR 2021.
     Refining activation downsampling with SoftPool. A. Stergiou et. al. ICCV 2021.

2. Except for results gain, it lacks some analysis for shift-invariance and shift-equivariance. 

3. It claims that the inferior results on the dataset 'TUT urban' is due to the property of the dataset itself. Then how to show the method's generality. Since it uses adaptive strides, it should also work well for the dataset with uniform information density, right?

4. To illustrate its generality, how the method performs on other kinds of datasets, like image?

4. Besides the results comparisons, computation should also be compared among different methods.

5. The captions for figures and tables lack details.
summary_of_the_review: Please refer to the weaknesses and details above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper an adaptive down-sampling method, AdaStride. The aim of AdaStride is to learn to deploy adaptive strides in a sequential data instance, i.e., preserving more information from task-relevant parts by using smaller strides while using larger strides for less-relevant parts. This idea is implemented by the cumulative-sum + normalization algorithm followed by a soft feature aggregation. Experimental results of speech classification and self-supervised learning are provided.
strength_and_weaknesses: Strength:

- The motivation is clear and natural. The proposed method is novel on top of DiffStride.
- Both the empirical results of supervised learning and self-supervised learning are provided.
- AdaStride is a plug-in module, and can be applied to many deep networks.

Weaknesses:
 
- Only the audio data is considered. Can AdaStride be applied to 2D image data?
- It would be better to present an analysis of the computation cost of AdaStride.


##Post-rebuttal##
After reading the rebuttal from the authors, I decided to keep my score. I think that this paper is just around the borderline and leaning toward accepting. However, as I have mentioned, I'm not quite familiar with the area of audio processing. Therefore, I may not be able to properly evaluate the novelty of this paper. 

clarity,_quality,_novelty_and_reproducibility: The writing of this paper is generally clear. Personally, I think this paper is of high quality.

However, as a matter of fact, I'm not quite familiar with the area of audio processing. Therefore, I may not be able to properly evaluate the novelty of this paper. 
summary_of_the_review: I think that the quality of this paper is well. My only concern is that I'm not quite familiar with the area of audio processing. Therefore, I may not be able to properly evaluate the novelty of this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new adaptive downsampling method for sequential data. Adaptive strides allow for preserving more information for task-relevant parts by using smaller strides for those and larger strides for less important parts, while preserving a global downsampling factor. Experiments across several audio processing tasks validate the effectiveness of the proposed method on audio data. Reviewers raised some concerns about quadratic time complexity of AdaStride; although authors propose a variant AdaStride-F which has linear complexity but reviewers noted in the virtual meeting that performance is not so good. A major concern that came up in the virtual meeting with reviewers was that the application scope is limited to 1D data with major focus on audio classification tasks in the paper and ASR experiments on Librispeech does not use full data. Reviewers believed that these points make the paper fall slightly below the acceptance bar for ICLR. 
justification_for_why_not_higher_score: Reviewers raised some concerns about quadratic time complexity of AdaStride; although authors propose a variant AdaStride-F which has linear complexity but reviewers noted in the virtual meeting that performance is not so good. A major concern that came up in the virtual meeting with reviewers was that the application scope is limited to 1D data with major focus on audio classification tasks in the paper and ASR experiments on Librispeech does not use full data. Reviewers believed that these points make the paper fall slightly below the acceptance bar for ICLR.
Decision: Reject