AdaStride: Using Adaptive Strides in Sequential Data for Effective Downsampling
Authors: Yoonhyung Lee
Kyomin Jung
Venue: Submitted to ICLR 2023 (2023)
Keywords: Downsampling
Pooling
Strides
Learning algorithm

Abstract: The downsampling layer has been one of the most commonly used deep learning (DL) components in sequential data processing due to its several advantages. First, it improves the generalization performance of networks by acting as an information bottleneck, where it extracts task-relevant features and discards others. Second, it reduces data resolution allowing CNN layers to have larger receptive fields with smaller kernel sizes. Third, the reduced data resolution facilitates the use of Transformer networks in case of high-resolution data. Accordingly, there have been many studies on downsampling methods, but they have a limitation in that they apply the same downsampling ratio across a data instance. Using the same downsampling ratio uniformly for an entire data instance does not reflect the fact that the task-relevant information is not uniformly distributed in real data. In this paper, we introduce AdaStride, a downsampling method that can apply adaptively varying downsampling ratios across a sequential data instance given an overall downsampling ratio. Specifically, AdaStride learns to deploy adaptive strides in a sequential data instance. Therefore, it can preserve more information from task-relevant parts of a data instance by using smaller strides for those parts and larger strides for less relevant parts. To achieve this, we propose a novel training method called vector positioning that rearranges each time step of an input on a one-dimensional line segment without reordering, which is used to build an alignment matrix for the downsampling. In experiments conducted on three different tasks of audio classification, automatic speech recognition, and discrete representation learning, AdaStride outperforms other widely used standard downsampling methods showing its generality and effectiveness. In addition, we analyze how our AdaStride learns the effective adaptive strides to improve its performance in the tasks.

Under review as a conference paper at ICLR 2023
ADASTRIDE: USING ADAPTIVE STRIDES IN SEQUEN- TIAL DATA FOR EFFECTIVE DOWNSAMPLING
Anonymous authors Paper under double-blind review
ABSTRACT
The downsampling layer has been one of the most commonly used deep learning (DL) components in sequential data processing due to its several advantages. First, it improves the generalization performance of networks by acting as an information bottleneck, where it extracts task-relevant features and discards others. Second, it reduces data resolution allowing CNN layers to have larger receptive fields with smaller kernel sizes. Third, the reduced data resolution facilitates the use of Transformer networks in case of high-resolttion data. Accordingly, there have been many studies on downsampling methods, but they have limitation in that they apply the same downsampling ratio across a data instance. Using the same downsampling ratio uniformly for entire data instance does not reflect the fact that the task-relevant information is not uniformly distributed in real data. In this paper, we introduce AdaStride, a downsampling method that can apply adaptively varying downsampling ratios across a sequential data instance given an overall downsampling ratio. Specifically, AdaStride learns to deploy adaptive strides in sequential data instance. Therefore, it can preserve more information from tesk-relevant parts of a data instance by using smaller strides for those parts and larger strides for less relevant parts. To achieve this, we propose novel training method called vector positioning that rearranges each time step of an input a one-dimensional line segment without reordering, which is used to build an alignment matrix for the downsampling. In experiments conducted on three different tasks of audio classification, automatic speech recognition, and discrete representation learning, AdaStride outperforms other widely used standard downsampling methods showing its generality and effectiveness. In addition, we analyze how our AdaStride learns the effective adaptive strides to improve its performance in the tasks.
INTRODUCTION
Recently, deep learning (DL) has achieved remarkable performance in various machine learning domains such as image classification (Krizhevsky et al. 20122) He et al.] | (0166)), machine translation
(Bahdanau et al. 2015 Vaswani al., (0172, 1), audio classification (Yoon al. 2019 Li et al. (019)).
and speech recognition CCha et al. [2015] Gulati et al. 2020 Kim et al. 2022). This is because many
DL architectures such as CNN (Fukushima & Miyake 1982 LeCun
1989) RNN (Rumelhart
ct al. 1985 Hochreiter & Schmidhuber 1997) and Transformer (Vaswani et al. 2017) can be easily
employed for various types of input and output.
Especially, downsampling layers have brought many benefits when they are used in combination with other DL layers in many sequential processing tasks. For example, many classification networks (Li et al.) 2019 Ma et al., (021). the downsampling layer was used with CNN layers where it gradually reduces the data resolution while providing several benefits: (1) it improves the generalization performance of the networks by acting as an information bottleneck that preserves task-relevant information and discards other trivial information (Li & Liu 2019): (2) it reduces the amount of computation because the reduced resolution allows intermediate CNN layers to have virtually larger receptive fields with smaller kernel sizes. Other than CNN, there have been many studies reporting remarkable results and even the of performance by using downsampling layers and Transformer layers (Dhariwal et al., 2020; Gulati et al., 2020; Kim et al., 2022) Karita al. 2019 Synnaeve et al. 2019 Collobert et al. (2200) together. In these studies, the downsampling

---

Under review as a conference paper at ICLR 2023
AvgPool uniform strides
AdaStride adaptive strides
Figure 1: Examples of two different downsampling methods, where they reduce the lengths of melspectrograms in half: (1) average pooling with uniform strides; (2) AdaStride downsampling that uses adaptive strides.
layer was used to alleviate the problem of quadratically increasing memory and computational cost of Transformer, which has hindered Transformer from being used in case of high-resolution data.
Due to the advantages of the downsampling layers, there have been many studies on downsampling
(Radford al. 2016 Zhang 2019| Rippel et al. 2015 Riad et al. 2022 Zhaa & Snoek [02T]]
Stergiou et al. 2021 to preserve more task-relevant information or to discard task task-interfering
information. Radlord al. (2016) replaced the max- or avg- pooling with strided convolutions
to improve the performance by allowing the networks to learn their own downsampling, Zhhng (2019) pointed out that aliasing can occur in downsampling because the reduced sampling rate decreases the Nyquist frequency (Nyquist 1928). Therefore, [Zhagg (2019) proposed to apply low- pass filtering before downsampling to remove the high-frequency components over the Nyquist frequency. Mcanwhile, (Rippel et al. (2015) proposed spectral pooling that performs downsampling in the frequency domain. Specifically, inspired by the fact that the power spectrum of images mostly consists of lower frequencies, it removes the figh-frequency components in the frequency domain. Furthermore, Riad et al.] (2022) proposed DiffStride, which improves the performance and narrows down the architecture search space by making the stride factor of the spectral pooling learnable. However, there have been no studies about using varying downsampling ratios across a data instance fully utilize the given overall downsampling ratio.
In this paper, we introduce AdaStride, a novel downsampling method that applies adaptively varying downsampling ratios in sequential data sample given an overall downsampling ratio. To be specific, our AdaStride learns to deploy adaptive strides in sequential data sample. Figure I illustrates the usefulness of deploying adaptive strides in a data instance. The upper part of Figure I shows average pooling using uniform strides, which inevitably uniformly reduces the data resolution in half for every part of the data. On the contrary, as shown in the lower part of Figure [ deploying adaptive strides allows for preserving more information for task-relevant parts by using smaller strides for that parts and larger strides for less important parts. To achieve this, we propose novel training method called vector positioning (VP), which rearranges each time step of the input signal on mie-dimensional line segment without reordering. Based on VP. AdaStride constructs an alignment matrix and performs downsampling based on the matrix. In addition, we also introduce variant of AdaStride called AdaStride-F, which speeds up the computation by replacing the matrix multiplication in the downsampling with faster paatter_add operation. This reduces the training time of AdaStride by around 13% while showing similar performance compared to AdaStride.
To evaluate AdaStride, we compare our method with widely used downsampling methods in three different tasks: audio classification, automatic speech recognition (ASR), and discrete representa- tion learning based on VQ-VAE (Van Den Oord et al. 2017). In audio classification, AdaStride outperforms all the standard downsampling methods including DiffStride on three of four datasets without hyperparameter tuning. Also, we identify that it is possible to obtain better performance of AdaStride by adjusting the trade-off between information loss and aliasing depending on the dataset. Indeed, AdaStride ranks first on all the datasets when we optimize the trade-off independently by tuning hyperparameter according to the dataset. Furthermore, our AdaStride achieves significant performance improvement over the strided convolution, even when the increase in memory usage of AdaStride is minimized. On other tasks, show that AdaStride also outperforms all the other downsampling methods by learning effective adaptive strides, and we provide analyses of how the adaptive strides are learned in the tasks.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND
In this section, we present the background for developing a better understanding our work. First, we demonstrate how the amount of information that be preserved in downsampling is bounded, mainly focusing on the relationship between downsampling and aliasing. Then, we will explain spectral pooling based downsampling methods that will appear in comparative experiments, which will also help understand the motivation of our work.
2.1 DOWNSAMPLING AND ALIASING
Zhang et al. (019) showed that commonly used downsampling methods such max-pooling,
vverage-pooling or strided convolution can be seen as combination of "densely refining an input' and 'naively resampling the signal'. For example, let's assume that we perform max-pooling with a window size of w and a stride of s. Then, it can be decomposed into the following two steps: (1) applying max operator of window size of the input signal with a stride of I; (2) naively resampling the refined input signal with sampling rate reduced by 1/s.
In the paper, it was pointed out that aliasing can occur in step (2), and it hurts the shift-invariance and shift-qquivariance of networks. Therefore, it proposed to apply low-pass filtering to the signal before step (2), and it actually improved the shift-invariance and shift-equivariance. Furthermore, the low-pass filtering surprisingly even improved the classification performance of networks despite of the information loss for the high-frequency components. It implies that, in downsampling, the components above the Nyquist frequency rather disturb the networks than convey task-relevant information. other words, it is essential to use lower downsampling ratio to raise the level at which frequency components become meaningful.
2.2 SPECTRAL POOLING AND DIFFSTRIDE
Rippel et al. 2015) proposed S spectral pooling that performs downsampling in the frequency domain.
Let
Tou ..... - E Rº be the input and be stride factor. Then, spectral pooling first sends
X to the frequency domain based on Discrete Fourier Transform (DFT) as = F(x) E [c as
follows:
T-1 Yn = Σ T4 e
2xm
T 2
0 < n
(1)
Then, y is cropped by maintaining only the lower 2x back the original domain through the inverse DFT as F (crop(y). Inspired by the fact that the power spectrum of natural images mostly consists of low-frequee wwwfrreuency components, spectral pooling could improve the network performance, and the aliasing problem naturally resolved here.
1 frequency components, and it brought
Furthermore, Riad et al. (2022) proposed DiffStrrie that rather applies soft mask m R RLEJ+1 to than directly crops y as follow:
T + -0+++1-900 n), 0J, 1]. 3=mOy,
m. min [max 1 +R+
28
where R is a hyperparameter and @ denotes element-wise multiplication. Then, DiffStride crops ý by maintaining only the non-zero frequency components and the downsampled signal is obtained as x = F== (cropcr(())) By doing so, it allowed the model to learn the stride factor through gradient descent algorithm, resulting improved classification performance and smaller architecture search space. However, it is also the same for these two spectral pooling methods that the downsampling ratio determines the level at which frequency components become meaningful.
METHODOLOGY
In this section, we introduce AdaStride, novel downsampling method that can adaptively apply different downsampling ratios to different parts of sequential input based on vector positioning process. Also, introduce variant of AdaStride called AdaStride-F, which is efficient ver- sion of AdaStride that performs downsampling based on scatter_ add operation instead of matrix multiplication.
3

---

Under review as a conference paper at ICLR 2023
0L-2
q
Vector Positioning
FC-layer Exponential
Cumulative-sum Normalization
Figure 2: An abstract diagram about how AdaStride downsampling is performed. It is performed based on an alignment matrix, and the matrix calculated based on the vector positioning process.
3.1 ADASTRIDE
Let X (x.. XT-1) E RTXD be the input of AdaStride and stride factor. Then, AdaStride reduces the time steps of X by the factor of and outputs Y {Vo..... ............ V1-17 } R L [T/s] based on a vector positioning (VP) process. Inspired by Efficient-TTS (Miao et al. 2021), we developed the VP where first define one-dimensional line segment represented by list of indices q = [0, 1,.... L-1] E IR as shown in Figure Then, unlike Efficient- that aligns the text and speech modalities based on the cross-attention mechanism, each input vector X is positioned on the line segment based only on the input vectors. First, we obtain relevant displacements of the input vectors, Ap- {Apo. .... Apr-1}
IRT Here, each Ap, is calculated by passing each x, through position-wisd wise fully-connected layer followed by an exponential function. Then, the positions of X&S are calculated using the cumulative summation operation, and they are lastly normalized to be positioned between [0, I. 1] as follows:
Po PT-1
Apn:
* (L 1).
(3)
Po
n=0
In this way, we can guarantee that the positions of the input vectors start from 0 and end at L 1 without reordering.
Then, an alignment matrix A E LLXT is computed based on the distances between pz and qa (D RIXT), and X is downsampled to Y based A as follows:
exp(di))
d; = o 2p; - 4:)3
0ij E exp(d..)
Y A@X.
(4)
where hyperparameter that controls the sharpness of the alignment and denotes matrix multiplication. To sum up, X is downsampled to Y based A calculated in VP, and we name this downsampling method AdaStride because can seen as a sub-sampling with adaptive strides for significantly small value.
In addition, we empirically observed that it is erucial prevent every stride from heing less than one (i.e., doing upsampling) to stabilize VP learning (Appendix D) In other words, every P should have at most one qi where the D is the nearest position to the q, Thereefore, we add an additional position loss to the original loss of each task satisfy the condition:
Lpos T
Σ max p. 1.0, 0.0).
(5)
All the calculation steps of AdaStride are summarized in algorithm Din Appendix A
3.2 ADASTRIDE-F
To reduce the memory and computational cost of AdaStride, we also propose a variant of AdaStride called AdaStride-F that replaces the matrix multiplication based downsampling with scatter_ade
4

---

Under review as a conference paper at ICLR 2023
operation based downsampling. In AdaStride-F, distances calculated only from each pj to its nearest index 9i, where the gi is equivalent to the index obtained by rounding the pf:
de 1 { (p} - 9;)², (q: = 1p (q: + P)])
(6)
Then, the alignment matrix A is calculated as (4), but it not used for matrix multiplication. Instead, since the alignment matrix A has only one non-zero element in column, the matrix multiplication in (2) can be calculated using weighting vector
{wo, -..y WT-1) E R follows:
L-1 Σ Dijo
W/ =
y== =
' Wp E XT.
C = { E oo.... 1]1 i = qi }.
DEC;
where C; is of indices of pãs of which the nearest index is q. and the weighted sum is imple- mented using the scatter add operation. All the calculation steps of AdaStride- are summarized in algorithm 2 in Appendix
4 EXPERIMENTS
In this section, we describe the experimental results on the superiority of AdaStride as downsampling method, where we conduct three different tasks to verify its wide applicability: audio classification, automatic speech recognition (ASR), and discrete representation learning based on VA-VAE. In all of the tasks, we use audio representaions by converting waveforms to melspectrogram features and the conversion parameters summarized in Table iin Appendix C
4.1 AUDIO CLASSIFICATION
Experimental setup We used four audio classification datasets: IEMOCAP dataset (Busso et al. 2008) for speech emotion recognition. Speech commands dataset (Warden 2018) for speech vocab- ulary recognition, BirdSong dataset (Stowell et al. 2018) for birdsong detection, and TUT urban dataset (Heittola al. 2018) for acoustic scene classification. For all of the tasks, used the same architecture consisting of 6-layer ResBlocks (He et al.] 2016b), and downsamplings were performed in the first, third, and fifth ResBlocks where the downsampling ratios were 2. For the downsampling methods other than the strided convolution, we first apply an unstrided convolution and then apply the downsampling methods. At the end of the blocks, hidden representations were globally average-pooled to obtain single vector, and it was fed to a fully-connected classifier. For more details, please refer to Appendix C
Results As shown in T our AdaStride and AdaStride-F outperformed the other downsampling methods three four datasets without hyperparameter tuning. However, in TUT urban dataset, spectral downsampling methods outperformed AdaStride. We attribute this to the characteristics of the TUT urban dataset, of which audios mostly consist of plain background sound recorded at certain place: (1) the audios have relatively uniform information density (Figure 3 (c)), so it is difficult for AdaStride to fully utilize the adaptive strides; (2) the noisy nature of the TUT urban dataset is advantageous likely to incur aliasing in downsampling, Namely, the benefits of learning adaptive strides were likely smaller than the benefits of anti-aliasing. In Appendix E there an analysis of how effectively AdaStride utilizes the adaptive strides according to datasets, and in Appendix [ there is discussion about the relationship between learning adaptive strides and anti-aliasing with additional experiments.
the spectral downsampling methods, where the high-frequency noise is
Computation overhead To minimize the computation overhead of the AdaStride compared to the strided convolution, we trained a smaller model called AdaStride-S, where the AdaStride layer was only used at the most upper downsampling layer and strided convolution layers were used for the rest downsampling layers. In this case, it still outperformed the strided convolution in all datasets and learned effective alignments at the last AdaStride layer (Figure 13), even though the time and memory overhead was only about 10% (Table 10). Especially, significantly improved the accuracy on the IEMOCAP dataset, indicating there tasks that benefit specifically much from using AdaStride. In Appendix
G we provide the computation complexities of different downsampling methods with a comparison of memory and time consumption that actually used.
5

---

Under review as a conference paper at ICLR 2023
Table Audio classification accuracies (% mean + std) different downsampling methods measured on four datasets. Each experiment was conducted five times with different random seeds.
Dataset
Method
IEMOCAP
Speech commands
TUT urban
BirdSong
Strided Conv.
46.7 + 1.9
95.2 0.1
94.5 0.3
82.3 0.1
Spectral Pool.
51.4 + 1.8
95.2 + 0.2
95.4 0.4
83.1 + 0.3
Dift'Stride
51.1 + 1.3
95.4 + 0.1
95.4 + 0.3
82.8 + 0,2
AdaStride
52.2 = 1.0
95.9 0.1
95.1 0.6
83.3 0.5
AdaStride-F
51.9 = 1,2
95,9 + 0.1
95.1 0.5
83.3 0.4
AdaStride-S
51.0 = 1.9
0.1
94.6
82.9
(>) Speech commands
(c)
(d) Birdsong
Figure 3: Each melspectrogram in the first row is sample from each dataset, and the alignments
below a melspectrogram are obtained by AdaStride layers when classifying the melspectrogram.
Learned alignments Figure 3 shows that AdaStride learns effective alignments for classifying audio samples. For example, in Figures (a). (b), AdaStride applies much more downsampling ratios for the silence part and less downsampling ratio for the speech part. By doing so, AdaStride can preserve much more information from the speech part, which contains most of the task-relevant information. contrast, the alignments in (c) have the shape of an almost straight diagonal line, and it means that AdaStride applies almost the same downsampling ratio to the entire input. Therefore, there is less gain from using AdaStride for the dataset such as the TUT urban dataset.
Relationship between and aliasing In equations (4) and (6), the hyperparameter controls the sharpness of the alignment, where the smaller the o is, the sharper the alignment matrix is. In other words, we can balance the trade-off between preserving high-frequency components and anti-aliasing by adjusting the a value. Therefore, we conducted an experiment in which we trained various AdaStride models with different values on the audio classification tasks. From the experiment, we found out that was crucial to select an appropriate a value depending on which dataset was used. For example, AdaStride with a - = 0.1 showed the best performance on the TUT urban dataset and even outperformed the spectral downsampling methods, unlike the results Table [ Meanwhile, AdaStride with which means that preserving high-frequency components is preferred to anti-aliasing on those datasets compared to the TUT urban dataset. To see the shape of the alignments learned for different values, please refer to Figure 14
5.0 achieved the best performance on the IEMOCAP and Birdsong datasets,
4.2 AUTOMATIC SPEECH RECOGNITION
Experimental setup In this experiment, compare various downsampling methods in ASR by using them subsampling module to reduce the time resolution an input. Specifically, the subsampling module consists of two (Conv-Pool-Sillu) blocks where each block downsamples the

---

Under review as a conference paper at ICLR 2023
Table Audio classification accuracies (% mean std) measured on four datasets using different values. Each experiment was conducted five times with different random seeds.
Dataset
ΙЕMOCAP
Speech commands
TUT urban
Birdsong
a = 0.1
50.4 + 1.7
93.0 + 0.1
95.6 + 0.3
76.1 + 0.6
a a- = 1.0
52.1 = 0.7
95.6 + 0.1
95.2 + 0.5
82.3 + 0.7
0-2 5.0
52.2 + 1.0
95.9 + 0.1
95.1 + 0.6
83.3 + 0.5
o 2 = 10.0
52.2 = 1.0
96.0 + 0.1 95.2 0.6
83.3 + 0.2
input by the case of an experiment on DiffStride. the experiment was conducted by using a DiffStride block with spectral pooling layer block to fix the total downsampling ratio to 4. Then, the output of the subsampling module is fed to an encoder consisting of the 16 Squeezeformer (Kim et al.) 2022) blocks. We trained ASR models with different downsampling methods based CTC loss (Graves et al. 2006) on LibriSpeech ASR corpus (Panayotov et al., 2015), and we compared phoneme error rate (PER) and word error rate (WER) of the models. We used g2p module (Park (20T9) for PER models and 1k subword tokenizer for WER models, which is trained on the train splits using WordPiece algorithm (Wu et al., 2016). Also, we used SpecAugment (Park al. (019)) with one frequency-mask with mask parameter (F = 27) and five manes with maximum time-mask ratio (ps = 0.05). For more details such as audio representation and training parameters, please refer to Appendix
Table 3: PER and WER (%) of different downsampling methods measured on each data split of the LibriSpeech dataset. A data split with 'oother' consists of speakers that are more difficult to recognize.
dev-clean
dev-other
test-clean
test-other
Method
PER
WER
PER
WER
PER
WER
PER
WER
Strided Conv.
1.46
4.87
4.66
12.29
1.55
4.88
4.43
11.91
Spectral Pool.
1.53
4.54
4.79
11.63
1.59
4.62
4.62
11.53
DifiStride
1.48
5.02
4.78
12.88
1.60
4.93
4.62
12.81
AdaStride
1.40
4.05
4.56
10.98
1.47
4.11
4.28
10.75
AdaStride-F
1.38
4.50
4.62
11.71
1.53
4.38
4.26
11.50
Results As shown in Table 3 our AdaStride and AdaStride-F outperformed the other downsampling methods for both PER and WER for all data splits while combined with the state-of-the-art ASR Squeezeformer architecture. Since recognizing a speech is to extract a word sequence (sequential data) from audio input (time series data), applying varying downsampling ratios to the audios in ASR is effective rather than hurting the performance. Moreover, the alignments that AdaStride learns show that it learns effective adaptive strides even if Spec we conducted the ASR experiments in low-resource setting for more extensive comparisons, where we used only a small fraction of the LibriSpeech dataset much as 10% and 1%, cf. Appendix H
is used (Figure [6p. In addition,
4.3 DISCRETE REPRESENTATION LEARNING
In this section, demonstrate experiment conducted based VQ-VAL (Van Den Oord et al.) 2017) to see whether AdaStride superior in discrete representation learning over the other down- sampling methods. Here, we compare AdaStride with strided convolution and spectral pooling, all of which use a fixed overall downsampling ratio. VQ-VAE is one of the representation learning frameworks that learns to represent data with discrete codes shown in Figure
Experimental setup In this experiment, we used LISpeech-1.J dataset (Ito & Johnson 2017). and we adopted 6-ResBlock Encoder- -ecoddr architecture as our baseline model, where downsamplings conducted alternately in the encoder ResBlocks, and nearest neighbor upsamplings alternately in the decoder ResBlocks. As a result, learns to obtain code representations which time steps reduced by 1/8 from the original time steps of a melspectrogram. When we used AdaStride,
conducted
7

---

Under review as a conference paper at ICLR 2023
Codebook
9:12
Encoder
Decoder
vector quantization (nearest code)
Auy -Softmax(D, dim-1)
Figure 4: A schematic diagram of the VQ-VAE architecture. (Down- / Up-) samplings are conducted in the colored ResBlocks existing the encoder and decoder. When AdaStride downsampling layer is used, an alignment matrix is also passed from the encoder to the decoder.
we used it only for the downsampling conducted in the second last encoder ResBlock for simplicity, This is because AdaStride encodes parts of information using the alignment matrix so the alignment information should be passed the decoder as well as the discrete code representations. Specifically, passed alignment matrix Aup to the second decoder ResBlock for upsampling where Aup was calculated by applying softmax to the distance matrix along the first dimension. Therefore, AdaStride-F was excluded in this experiment because applying softmmx operation to D along the first dimension is meaningless for AdaStride-F For more details such as audio representation, training parameters, architecture, and training of VQ-VAE, please refer to Appendix and C
Table 4: PER-R and PER-C (%) for comparing VQ-VAE models with different downsampling methods. PER-R represents a performance of an ASR model trained on reconstructed audios. PER-C represents a performance of ASR model trained on discrete code representations.
Method
PER-R
PER-C
Strided Conv.
11.44
21.23
Spectral Pool.
18.14
25.53
AdaStride
10.85
19.51' / 2
each of these indicates whether it uses alignments or not
To see the effectiveness of AdaStride in discrete representation learning, we evaluated the representations that the VQ-VAE models using different downsampling methods learn. First, we evaluated the performance of VQ-VAE itself by comparing the reconstruction performance based on phoneme error rate (PER-R) For this, we generated a hundred audio samples by applying HiFi- GAN (Kong al. 2020) to the melspectrograms reconstructed by each VQ-VAE model. Then, measured PER-R on the generated audio samples via the Google Cloud Speech-to- Tpeech-to-Text API Second, evaluated the quality of the discrete representations by training ASR models using the learned discrete representations and measuring phoneme error rate (PER-C). the case AdaStride, we separated the cases of whether the alignments that AdaStride learns are used training. Specifically, the representations are upsampled by two based on the learned alignments or by repeating each representation twice as shown Figure X in Appendix C The result of the experiment is shown in Table 4 where our AdaStride outperforms the strided convolution and spectral pooling in terms of both PER-R and PER-C. However, since the code representations and alignments are not learned targeting ASR, PER-C becomes worse when the alignments are not used in ASR training. To see alignments that AdaStride learns, please refer to Figure 15
Results
used in ASR
Encoding data with codes and durations To see how AdaStride effective compared the other downsampling methods, we analyzed how the data encoding is learned in terms of codes and their durations. Here, durations of codes mean lengths of consecutive groups equivalent codes when codes upsampled back to the length right before the last downsampling. For AdaStride, the code durations were calculated after upsampling the codes using the alignment matrix which was converted
https://cclot google com/ spwech-
8

---

Under review as a conference paper at ICLR 2023
165
Stricd Conv.
AdaStride
Count
5
3
2
2
15 16
Duration
Figure 5: The counts of code durations when VQ-VAE models with different downsampling methods encode the LJSpeech-1.. 1.1 dataset. The durations are calculated after the codes are upsampled to the original length, the durations are even numbers for strided convolution and spectral pooling.
Table The number of unique codes out of a total of 512 codes when VQ-VAE models with different downsampling methods encode the LJSpeech-1.. dataset.
Strided Conv. Spectral Pool. AdaStride
Unique codes
390 512
/12
365 / 512
to one-hot matrix via argmax operation. Figure Sillustrates the counts of durations of codes that the different downsampling methods learn. As is expected, AdaStride learned to represent data with codes with variable durations so it can encode information-rich parts of data using more codes in finer resolution.
contrast, strided convolution and spectral pooling ineffectively encoded data with codes most of which have durations of two. Furthermore, the fact that there were no durations longer than two represents that even long-lasting part in a data sample was encoded using various codes. Therefore, this way of encoding can lead waste of codebook capacity, and it can result in a performance drop when the size of the codebook decreases. Indeed, Table 5) illustrates how many unique codes existed when each VQ-VAE model encoded the full LJSpeech- 1.1 dataset, where AdaStride used the least number of codes.
Future work Although it was shown that Adastride has superiority over other downsampling methods, there are still areas be studied in the future. First, extending AdaStride to the higher dimensional data. For instance, in image classification, it would be beneficial to preserve more infor- mation for the target object instead of the background (Appendix [[]. Second, addressing the difficulty of finding the optimal value to control the trade-off between preserving high-frequency components and anti-aliasing. For instance, combining an explicit anti-aliasing process with AdaStride (Appendix F). Lastly, improving its efficiency in terms of memory and time consumption. Even though introduce AdaStride-F F, there is still much room for efficiency improvement in terms of memory and computational cost.
5 CONCLUSION
In this paper, we proposed novel downsampling method called AdaStride, which learns to apply adaptively varying downsampling ratios within sequential data sample via the vector positioning process. In this way, it can preserve information for task-relevan parts of a data sample better than the fixed overall downsampling rate. audio classification, AdaStride outperformed other standard downsampling methods on three of four datasets without hyperparameter tuning and on all of the datasets with hyperparameter tuning. In addition, even when we minimized the memory increase of AdaStride compared to strided convolution, it still achieved significant performance improvement, especially on the IEMOCAP dataset. In ASR, AdaStride also outperformed the other downsampling methods in terms of both WER and PER for all data subsets. In VQ-VAE, saw that AdaStride superior in reconstructing the input melspectrogram due to its effective utilization of codes and durations, and indeed leads to better representations for sub-tasks such as ASR.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY
To help reproduce the experiments, we provided a separate paragraph describing the experimental setup for each sub-task. Also, we elaborated on other details such as data preprocessing, training parameters, and the architectural details in appendices with many figures describing the architectures for accurate re-implementation. Furthermore, we provided source code of AdaStride as supple- mentary material, where you can reproduce the audio classification experiment using the AdaStride. Since we took great care to modularize the AdaStride layer while writing the code, it is also possible to easily employ the AdaStride in other tasks.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, JCLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. htt.tttp///arx.v.rrr//////// 1409. 047333993373
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-33) 359, 2008.
William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. Listen, attend and spell. CoRR, abs/1.508.011 152201011 2015. hRt::: wwtp abs/ 1508. 01211
Hyeong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee. Neural analysis and synthesis: Reconstructing speech from self-supervised representations. Advances in Neural Information Processing Systems, 34, 2021.
Ronan Collobert, Awni Hannun, and Gabriel Synnaeve. Word-level speech recognition with letter to word encoder. In International Conference on Machine Learning, pp. 2100-2110 10. PMLR, 2020.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution im- age synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873-12882 2021.
Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp. 267-285. Springer, 1982.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Anmol Gulati, James Qin, Chung-Chern Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmeet Transformer for Speech Recognition. In Proc. Interspeech 2020, pp. 5036-5040, 2020. doi: 10221457/ierrppecc2222-333
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016b.
Toni Kristian Heittola, Annamaria Mesaros, and Tuomas Virtanen, 2018. Heittola, Toni, Mesaros, Annamaria, & Virtamen, Tuomas. (2018). TUT Urban Acoustic Scenes 2018, Development dataset 1 htt.e/ooooo.22.11.... 1221142.
[Data set]. Zenodo. https://doi..rrg/1...281//
10

---

Under review as a conference paper at ICLR 2023
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Keith Ito and Linda Johnson.
The lj speech dataset.
https://keithito.com
LJ--ppacch-Datasett/ 2017.
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al. A comparative study on transformer rn in speech applications. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 449-456. IEEE, 2019.
Schoon Kim, Amir Gholami, Albert Shaw, Nicholas Lee, Karttikeya Mangalam, Jitendra Malik, Michael W Mahoney, and Kurt Keutzer. Squeezeformee: An efficient transformer for automatic speech recognition. arXiv preprint arXiv:2206.000880 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
org. abs abs//4b/...1 6980
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL https: stttps://proo http://e//c/cceeeeieeceo. cewpiiseerrpppp.p cc/ httppaaper//// 2012////21///aa c39986243bb966b7766 668 (wwee9ee666664336662222 (1986228b96676633333222446663333 Papeo pdf
Yann LeCun, Bernhard Boser, John Denker, Donnic Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back-propagationn network. Advances in neural information processing systems, 2, 1989.
Junjie Li and Ding Liu. Information bottleneck theory on convolutional neural networks. arXiv preprint arXiv: MXiv:I911.0377 2019.
Yuanchao Li, Tianyu Zhao, and Tatsuya Kawahara. Improved End-to-End -oe--oo--ddddd- Speech Emotion Recog- nition Using Self Attention Mechanism and Multitask Learning. In Proc. Interspeech 2019, pp. 2803-2807, 2019. doi:
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
Youxuan Ma, Zongze Ren, and Shugong Xu. RW-Resnet: A Novel Speech Anti-Spoofing Model Using Raw Waveform. In Proc. Interspeech 2021, pp. 4144-4148, 2021. doi: 10.2 10.21437/Interspecch. 2021-438.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp.
2017.
Chenfeng Miao, Liang Shuang, Zhengehen Liu, Chen Minchuan, Jun Ma, Shaojun Wang, and Jing Xiao. Efficienttss: An efficient and high-quality text-to-speech architecture. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7700-7709. PMLR. 18 24 Jul 2021. URL https: htttps://proceed: mgs.mlmmmm.....m...ng.mmm....m.. m wtw.sppssseeeo 39/mi a 21 html
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
URL htR.htttps::///ppreeeee www.:/nettooommmmmmmmm wtt.:rroommmool-000gg1111

---

Under review as a conference paper at ICLR 2023
H. Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 47(2))617 644, 1928. doi: 1..1009/-AAEE. AIE....E.....000. 192.5005524..
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5216 5206..52..166 IEEE, 2015.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recogni- tion. In Proc. Interspeech 2019, pp. 2613-2611 2019. doi: 10224457/mmcxpecch2210--2266
Jongseok Park, Kyubyong & Kim. g2pe. https::/ggithbb.cmmm htt.:///www.o.ooooyoonoo//gg////
2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: http://axivv.co abs///b///// 1511
06434
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Rachid Riad, Olivier Teboul, David Grangier, and Neil Zeghidour. Learning strides in convolutional neural networks. In International Conference on Learning Representutions, 2022. URL https
//openreview.o www.eiiwww.... net/ Forum ? wddmmm12--m22m
Oren Rippel, Jasper Snoek, and Ryan P Adams. Spectral representations for convolu- tional neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015, URL https: pestps://proceediingg www.ine.uriccmm ptt.rrpps.ccc/pppe//////////// pdff
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
A. Stergiou, R. Poppe, and G. Kalliatakis. Refining activation downsampling with softpool. In 2021 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 10337-10346, Los Alamitos, CA. USA, oct 2021. IEEE Computer Society. doi: 10. 10.11991CCC448022000...0000 ht fetpps:/ddon htt.:aaa......cooeoooooreeoore..
URL 01019
D. Stowell, Y. Stylianou, M. Wood, H. Pamula, and H. Glotin. Automatic acoustic detection of birds through deep learning: the first bird audio detection challenge. Methods in Ecology and Evolution, 2018. httpp:
org abs /10071101000 05812]
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap. Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end asr: from supervised to semi-supervised learning with modern architectures. arXiv preprint 1rXiv:1911.08440 2019.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition. arXiv preprint arXiv: 1804. 2018.
12

---

Under review as a conference paper at ICLR 2023
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint 2016.
Seunghyun Yoon, Seokhyun Byun, Subhadeep Dey, and Kyomin Jung. Speech emotion recognition using multi-hop attention mechanism. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2822- IEEE, 2019.
Richard Zhang. Making convolutional networks shift-invariant again. In Proceedings of the 36th International Conference on Machine Leurning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. www.nnla..aheee..e html
PMLR,
2019. URL http:tt/prrceeednggg
mlr. www.spress/vvv
Ya-Jie Zhang, Shifeng Pan, Lei He, and Zhen-Hua Ling. Learning latent representations for style control and transfer in end-to-end speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1945-6944) IEEE, 2019.
Jiaojiao Zhao and Cees G. M. Snoek. Liftpool: Bidirectional convnet pooling, In International Conference on Learning Representations, 2021. URL httpp:://operrevlew wwtp://ee/ffrurt
id=kE.vv6 639RRV
13

---

Under review as a conference paper at ICLR 2023
A ALGORITHMS
Algorithm 1: Pseudo-code describing how AdaStride works
Data:
X RT'D; input of AdaStride, a sequential data consisting of T time steps
8: overall stride factor that determines the ratio of the length of the input to the output
a: hyperparameter that controls the sharpness of the alignment matrix
Result:
Y Lx output of AdaStride, sequential data consisting
[T/s] time steps
pos position loss that stabilizes the training of AdaStride
/* Caculate cormalized positions of the input time steps
Ap < Exponential
p k Cumsum (Ap, dim = 0);
(p p(0 (p) 1 p(0 1J) (L - 1);
* Construct an alignment matrix based on p" and q f range(0, L);
D < -o - "p" tra.spsss)))
+ Softmax (D, dim = 1);
/+ Perform wiinsmmpiig and calculate the position loss
Y A@X;
Lyss t- T 2i= max (p" + p' 1.0, 0.0)
Algorithm 2: Pseudo-code describing how AdaStride-F works
Data:
X RTxD; input of AdaStride, sequential data consisting of T time steps
8: overall stride factor that determines the ratio of the length of the input to the output
a: hyperparameter that controls the sharpness of the alignment matrix
Result:
Y IRLXD, output of AdaStride, sequential data consisting of L [T/s] time steps
Cpos: position loss that stabilizes the training of AdaStride
/* Caculate normalized positions of the input time steps
Ap + Exponential (PositionalF((X))
p f Cumsum (Ap, dim 0);
(p p(0 (p) 1: pp0: 1]) (I 11:1;);
* Construct an al alignment matrix based on p'
+ o 2 [p-]2
D k- (L, T);
D < D.scatter (dim 0, index (t)].t(), src d.t()):
A Softmax (D, dim = 1);
W +- A.sum(dim u)..usqueezee
/* Perform downsampling and calculate the position loss
Y + O(L
Y + Y @scatter_add(ding = 0, index = repeat(], D), src W X);
Epos + - Σ max((p' + - pil 1.0, 0.0)
14

---

Under review as a conference paper at ICLR 2023
B VQ-VAE
VQ-VAE In this paper, VQ- Van Den Oord et al. 2017 Razavi et al. (019) used to represent melspectrogram with sequence of positive integers called codes. During training, the encoder obtains a sequence of continuous vector representations (ze (X)) from a melspectrogram. Then, each of the vector representations is replaced with among the set of code representations (codebook) based on the nearest neighbor lookup. Lastly, the quantized vector representations (2g (X)) are passed to the decoder, and the decoder learns to reconstruct the original input based mean absolute error (Lmar).
To successfully achieve the discrete representation learning, several techniques used in VQ-VAE training: (1) during backpropagation, the gradients from the decoder input 2q x,(X) (X) are passed to the encoder output ze (X), so encoder can learn to obtain better representations; (2) there are two additional losses used to minimize the gap arisen from the vector quantization process:
Loodebook ()
1 (X); (X2
(8)
5
commmit (X)
Σ
|و X)=] - ze (x)
(9)
where S is the length of the representations and sg denotes the stop-gradient operation that passes zero gradients during backpropagation.
Furthermore, prevent the codes from being not used and to speed up training, we adopted several more techniques called code restart and exponential moving average (EMA) code update (Dhariwal et al. 2020). Code restart is method of replacing code that is rarely used during training with representation randomly sampled from 2₃ (X). EMA code update is to update code representations based on EMA instead of the codebook loss Lendebook speed up training. As result, the total loss of VQ- VAE becomes as follows:
Luquue (X) L.mac (X, G(X)) frommii
(10)
where the VQ- VAE model and G(X) is the melspectrogram reconstructed by the model.
0,-1
-128
Convild (er3,, d:1,, wnl
h2=128
Conuiid [a-3, dr3, wn))
Blockock
x
Convid [b-1, d=1, wil
h,-128
Convöd [b-3, 6-3, wr))
LeokyRelU (0.2)
Convid (e-1, dr1,
L-80
1,=128
X
X
(a) Discriminator
(b) ResBlock
Figure 6: A schematic diagram of the discriminator architecture, 'sn' denotes spectral normalization (Miyato al. 2018) and 'wn' denotes weight normalization (Salimans & Kingma 2016).
Adversarial training To guide the code representations capture information, we employed an additional discriminator and adopted the GAN approach in our VQ- VAE training (Esser al. 2021). We adopted the architecture of the discriminator following (Choi et al. 202T) and is shown in Figure For GAN loss, we used LSGAN losses (Mao et al. additional nntture-matching loss, and they are as follows:
perceptually important
2017) with
Codo Matn(D: (D; (X) = Ex [(((X) - 1)² (D(G(X)))))
(11)
Coai Laa (G; (X = Ex [((G(()) - 1
(12)
fym (G; ()()) = =x
|| -ID(X)-D'GGXX))))
(13)
Ni
1=1
(14)
15

---

Under review as a conference paper at ICLR 2023
where denotes 1-th layer of the discriminator, and Dj and N, denote the features and the number of features in the (-th layer of the discriminator, respectively. As a result, the final loss of VQ-VAE becomes as follows:
LD Lade (D; G): LG = Luquae + B * Ladv D) + +
(15)
(G;
(16)
C ADDITIONAL DETAILS
Dataset In Table 6 the information about the datasets used in this paper demonstrated.
Table 6: Information about the license and about whether the datasets include the personal information or offensive content.
License
Personal information
Offensive content
IEMOCAP
Custom (non mnon-commercial""
X
X
Speech commands
CC-BY
X
TUT urban Birdsong LibriSpeech LJSpeech-1.1
Custom nonncommeeriil))*
X
CC-BY 4.0
X
CC-BY 4.0
X
CCO
X
X
http:///aall..cceee//////c//aass Re Release Form TEMOCAP pdff
BN-ReLU-Conv
FC-Down
BN-ReLU-Conv
BN- EN--ReLU-Conv
X
X
(a) ResBlock without downsampling
(b) ResBlock with downsampling
Figure 7: A schematic diagram of the ResBlock architectures with and without downsampling.
boobbabbabilities
Transformer Encoder
ResBlock
Upsampling
ResBlock
code representations
(alignment)
Figure A schematic diagram of the ASR network architecture trained with code representations.
Architecture of ResBlock We adopted FreAct-RessMe block (He et al, 2016b) shown in Figure 7 for audio classification and discrete representation learning. Specifically, downsampling methods by replacing the strided convolution of the ResNet block with a normal con- volution followed by one of the downsampling layers, shown in Figure When the downsampling
compare various
16

---

Under review as a conference paper at ICLR 2023
layer is AdaStride, the vector positioning process is conducted in the main path of ResNet, and in the shortcut path of ResNet, downsampling is conducted using the alignment matrix obtained the main path.
ASR network architecture trained with code representations We trained ASR networks (Figure with code sequences obtained via VQ-VAE encoder based on CTC loss. Also, we optionally used alignments that were obtained by AdaStride Encoder, and if alignments were not used, we expanded the code sequences by two based on nearest neighbor upsampling to match the time resolution of the code sequences equally for fair experiment. Each network was trained for 100k iterations with batch size of 32 and learning rate of le-3 using AdamW optimizer (Loshchilov Hutter 2019).
Training instability and layer normalization In ASR and VQ- VQE training, sometimes found out that it was unstable to train models using AdaStride due to the exponential function existing in the vector positioning process. However, could easily resolve this problem by putting layer normalization layer (Ba et al. 2016) before the AdaStride layer.
Hyperparameters
Table the hyperparameters used in each task are demonstrated.
Table 7: Hyperparameters used in each task.
Audio classification
Speech recognition
VQ-VAE
hidden dimension
256
144
512
kernel size
3
3
0 (# of codes)
5.0
5.0
10.0
512
0.1
1.0
1.0
sampling rate
16000
16000
22050
filter banks
80
80
80
windows hops
1024
400
1024
256
160
256
optimizer
Adam"
Adam
Adam
learning rate batch size
le-4
5e-4
3e-4
128
256
64
Kingma & Ba 2015)
Loshchilov & Hutter 2019)
17

---

Under review as a conference paper at ICLR 2023
D ABALATION STUDY ON THE POSITION LOSS
Figure 9: Learned alignments and the training losses and validation accuracy curves during training of the Birdsong classification task when position loss used or not.
We empirically observed that, witout position loss, training of the models becomes unstable leading to radical alignment learning and poor performance. For instance, EegureWshowe the training logs when ResNet equipped with AdaStride layers is trained the Birdsong dataset: (1) the alignments that AdaStride layers learn; (2) training losses and validation accuracy curves when position loss is used not.
E EFFECTIVENESS OF ADAPTIVE STRIDES ACCORDING TO DATASETS
IEMOCAP
SpeechCommands
1.0
AXX):
HXX) 1.1031
0.8
0.8
0.6
0.6
0.4
0.0
0.0
TUTurban
BirdSong
1.0
1.0
0.7409
0.8
0.8
0.6
0.6
0.2
0.2
0.0
0.0
Figure 10: Approximated probability mass functions of the number frames that are pooled into single frame. These are approximated by counting the numbers of the pooled frames after applying argmax operation the learned alignments. On upper right corner, an entropy calculated from each approximated probability mass function is written.
To see how effectively AdaStride utilizes the adaptive strides according to different datasets, we counted the numbers of frames that pooled into single frame after applying the argmax operation to the learned alignments. Herein, the alignments are obtained from the test splits of each audio clas- sification dataset after training the models using the AdaStride layer for the uppermost downsampling
18

---

Under review as a conference paper at ICLR 2023
layer. Figure 10 shows the probability mass functions of the number of the pooled frames which are approximated based on the counts, with the entropies written on the upper right corner. As we guessed in Section 4.1 the model trained the TUT urban dataset utilized the adaptive strides to the smallest extent, indicating that does not utilize the adaptive strides effectively.
F ADAPTIVE DOWNSAMPLING AND ANTI-ALIASING
Table 8: Audio classification accuracies (% mean + std over 5 runs) of LPFAdaStride measured on four datasets. Each experiment conducted five times with different random seeds.
Dataset
Method
IEMOCAP
Speech commands
TUT urban
BirdSong
LPFAdaStride
51.7 + 0.8
96.0 + 0.1
95.2 + 0.6 83.3 + 0.4
As mentioned in Section both preserving more task-relevant information (adaptive downsampling) and discarding task-interfering information (anti-aliasing) are significantly important for effective downsampling. Therefore, we again conducted the audio classification experiments in Section 4.1 with model called LPFAdaStride, where we combine the two beneficial design elements. Specifically, we located low-pass filter that removes the upper-half frequency components from the input representations before an AdaStride downsampling layer. The result of this experiment shown in Table g Although expected that the accuracies would significantly increase based on the adaptive downsampling and anti-aliasing that work complementarily. they did not. We attribute this to the varying Nyquist Frequencies according the adaptive strides the frequency threshold of the low-pass filters should also be set adaptively according to the adaptive strides.
StrideConv (1.0211)
2-
SpectralFool (0.5557)
DiffStride (0.6112)
Accuracy (96)
AlignStride (0.5594)
-1
10
Temporal shift
Figure 11: A graph of the accuracy changes based on the average accuracy when shifting the input melspectrogram from 0 to 10 time frames. On the upper right corner, the standard deviations of the accuracy changes are written for different downsampling methods.
In addition, we compared how the accuracies of different downsampling methods change IEMO- CAP dataset when shifting the input melspectrograms. We conducted this experiment to see how invariant AdaStride is the input shift was done in Zhang et al.) (2019). The result shown in Figure 11 In this experiment, AdaStride ranked second in terms of robustness based on the standard deviation of the accuracy changes, which indicates that the distance-bact based soft downsampling of AdaStride indeed has some anti-aliasing effect.
19

---

Under review as a conference paper at ICLR 2023
G MEMORY AND TIME CONSUMPTION
Table 9: Computation complexities of the downsampling methods. n the sequence length and is
the hidden dimension.
Strided Conv.
Spectral Pool.
DiffStride
AdaStride
AdaStride-F
Complexity
(n d) lnnnogn d) O log d) O d)
O d)
Table 9 shows the computational complexities of the downsampling methods. Strided convolution has the computation complexity of O(n d). Spectral pooling and DifiSttrie have the same computation complexity of O(n log d), which corresponds to the Fourier transform. AdaStride has the computation complexity of O( d), which corresponds the matrix multiplication between the alignment matrix and the input representations. AdaStride-F has the computation complexity of On d), which corresponds to the position-wise linear layer and the scatter_ace _add operation.
Table 10: Memory and time consumption per epoch for different downsampling methods relative to strided convolution.
Strided Conv.
Spectral Pool.
AdaStride
AdaStride-F
AdaStride-S
Mem 1.00
1.38
2.53
2.34
1.11
Training
Time
1.00
1.25
1.38
1.20
1.08
Mem
1.00
1.00
1.24
1.25
1.08
Inference
Time
1.00
1.70
1.66
1.65
1.08
In addition, actually measured the memory and time consumption of the different downsampling methods on the IEMOCAP dataset. Specifically, Table showss the memory and time consumption used process epoch for training and inference for different downsampling methods relative to strided convolution.
20

---

Under review as a conference paper at ICLR 2023
H AUTOMATIC SPEECH RECOGNITION ON LOW-RESOURCE DATASETS
Table 11: PER and WER results (%) of different downsampling methods measured on each data split of the LibriSpeech dataset. A data split with '-other' consists of speakers that are more difficult to recognize. Here, only 10% of the LibriSpeech train dataset is used.
dev-clean
dev-other
test-clean
test-other
Method
PER
WER
PER
WER
PER
WER
PER
WER
Strided Conv.
5.23
20.94
12.79
35.81
5.37
21.38
12.87
36.31
Spectral Pool.
4.94
25.18
12.52
40.68
5.02
25.58
12.30
41.31
DiftStrrie
4.73
24.75
12.06
40.19
4.78
25.07
12.03
41.20
AdaStride
4.50
15.47
11.65
29.27
4.66
15.51
11.70
29.97
AdaStride-F
4.56
15.40
11.64
29.29
4.78
15.61
11.76
30.16
Table 12: PER and WER results (%) of different downsampling methods measured on each data split of the LibriSpeech dataset. data split with '-other' consists speakers that are more difficult to recognize. Here, only 1 % of the LibriSpeech train dataset is used.
dev-clean
dev-other
test-clean
test-other
Method
PER
WER
PER
WER
PER
WER
PER
WER
Strided Conv.
28.52
69.86
39.32
77.82
28.33
70.21
39.98
78.17
Spectral Pool.
32.50
70.12
42.83
78.13
32.69
70.13
43.40
78.33
DiffStride
34.06
69.51
43.67
77.72
33.78
69.75
44.51
77.79
AdaStride
21.68
65.35
33.68
74.45
21.82
65.49
34.41
75.10
AdaStride-F
21.53
65.33
33.31
74.53
21.46
65.96
34.25
75.12
For more extensive experiments, we also did the speech recognition experiments under the low- resource setting, where used only small fraction of the LibriSpeech dataset as much as 10% and 1 1.. In the experiments, AdaStride consistently outperformed the other downsampling methods.
I ADASTRIDE FOR IMAGE DATA
Table 13: Image classification accuracies (% mean std over 5 runs) of different downsampling methods measured on two datasets. Each experiment was conducted five times with different random seeds.
Dataset
Method
CIFAR-10
CIFAR-I00
Strided Conv.
91.6 0.1
68.4 + 0.4
Spectral Pool.
91.9 0.1
68.8 = 0.2
DiffStride
92.3 0.1
69,5 = 0.1
AdaStride
92.1 + 0.1
68.8 + 0.3
Although the concept of adaptively downsampling the input representations can be extended to various domains such as images, it not straightforward to apply AdaStride in the two-dimensional space. Especially, it is challenging to locate the spatial features onto the two-dimensional plane defined by indices based on the cumulative sum operation while preserving their spatial ordering. This is because, the summation of the relevant displacements along arbitrary course is not conservative in the two-dimensional space. Nevertheless, we compare the different downsampling methods the image classification using CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al. 2009).
21

---

Under review as a conference paper at ICLR 2023
(a) Training
(b) Inference
Figure 12: Input images and alignments obtained from an AdaStride layer of an image classification model during training and inference, where downsamplings are applied twice sequentially for height and width axes. The alignment matrix shown on the upper right corner is used to downsample the image along the height axis, and the alignment matrix shown on the below left corner is used to downsample the image along the width axis. The image shown on the below right corner is the outer product of the two alignment matrices.
When using AdaStride, we applied it twice sequentially along the height and width axes. Table TO shows the accuracies of different downsampling methods measured on CIFAR-10, 100 datasets. In the experiment, AdaStride outperformed the strided convolution and spectral pooling but did not outperform the DiffStride. We conjecture that this because the downsamplings along the different axes happen independently so the effect of learning adaptive downsampling is limited. Actually, in Figure [T] is shown that the independently learned alignments are not flexible enough preserve much information from the task-relevant object.
J ALIGNMENT SAMPLES
(a) IEMOCAP
(b) Speech commands
TUT urban
(d) Birdsong
Figure 13: Each melspectrogram in the first row is sample from each dataset. Also, the alignment below melspectrogram the alignment obtained from an AdaStride layer of an audio classification model when the AdaStride layer is used only for the last downsampling layer.
22

---

Under review as a conference paper at ICLR 2023
( 0.1
(b) - 100
(c) 5.0
(d) 0- 10.0 0
Figure 14: Each melspectrogram in the first row is a sample from the IEMOCAP dataset. Also, the alignments below the melspectrogram are the alignments learned by AdaStride layers when classifying the melspectrogram during training. Each column represents different sigma values.
Figure 15: Melspectrogram and alignments obtained from the ResBlocks of the VQ-VAE. The alignment the bottom is the alignment used for upsampling and transposed here.
(a) Training
(b) Inference
Figure 16: Melspectrograms and alignments obtained from the AdaStride layers of the ASR networks. During training. SpecAugment is applied the input features.
23


Reviews:
[{"review_id": "_-qoWTSEUl", "reviewer": "Reviewer_kiYh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new downsampling method for sequential data, which supports varying downsampling ratios across the same data instance. To achieve this, it rearranges each time step on a one-dimensional line segment, which is used to learn an alignment matrix for performing the downsampling. Extensive experiments across several audio processing tasks validate the effectiveness of the proposed method.", "summary_of_the_review": "Considering the interestingness and the achieved experimental improvements, I tend to rank this paper as marginally above the acceptance threshold.", "strength_and_weaknesses": "**Strength**\n\n1. It is interesting and novel to apply varying downsampling ratios across the same data instance to better extract the task-relevant information.\n\n2. The proposed method is extensively validated across several audio processing tasks, indicating its general effectiveness on audio processing.\n\n\n**Weakness**\n\n1. The major concern for the proposed method is its generality for 2D inputs like images, considering all the baselines, including spectral pooling and DiffStride, also work for 2D images. Intuitively the principles discussed in Section 3.1 can be applicable for 2D images but the alignment matrix A should be defined on 2D distances and the introduced computational overhead should be quite large. A discussion for such generality is expected, otherwise the proposed method is limited to sequential data and not so general as the baselines.\n\n2. Considering the fully connected layers in AdaStride are executed for each input token, this may incur larger overheads and limit the scalability of long sequences.\n\n3. The author analyzes that the spectral pooling is a better choice for the TUT dataset with high-frequency noises in Table 1. Although this is reasonable, I wonder whether the proposed AdaStride can be also applied in the frequency domain on top of spectral pooling for improving the accuracy on the TUT dataset.\n\n4. The WER improvements in Table 3 is limited. Experiments on low-resource speech, e.g., LibriSpeech-10h/100h, may lead to more extensive comparisons.\n\n5. The margins around figures and tables are sometimes too large, e.g., Table 3 and Figure 4, which can be better organized.\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AtqCURSLQC", "reviewer": "Reviewer_MsFg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose AdaStride, a differentiable layer that learns an input-dependent and irregularly sampled downsampling.", "summary_of_the_review": "The potential impact is large but the authors may have disregarded the main baseline they should compare against.", "strength_and_weaknesses": "Strengths:\n * Learning adaptive downsampling is a fundamental deep learning problem with very large potential impact across all types of tasks, models and modalities.\n * This approach displays two characteristics that are offered by strided convs, spectral pooling or DiffStride: 1) the striding is input dependent 2) the striding is uneven along the temporal axis and can emphasize certain segments\n * The algorithm is simple and despite the computational overhead it could be useful in practice\n * It's really nice to see this approach tested in a generative VQ-VAE setting, while the benchmarks are typically classification networks. \n * Lots of details on the experimental setup in appendix\n\nWeaknesses:\n * Model:\n  * I suspect that authors do not show how close their approach is to the learnable alignment layer of [Efficient-TTS](http://proceedings.mlr.press/v139/miao21a/miao21a.pdf) which seems to accomplish the same thing (learn an alignment between x and its downsampled version). If the models are indeed very close and authors simply apply it in the context of downsampling, the contribution is still worth publication but they should be honest about that while currently the reference to Efficient-TTS is kind of hidden in the text.\n  * Authors should specify more clearly how they replace strided convs by adastride, even though I assume it first applies an unstrided convolution.\n  * AdaStride-F is presented as an implementation trick, yet performance between AdaStride and AdaStride-F vary a lot (e.g. for ASR). Why?\n  * This approach is conceptually very different from spectral pooling or DiffStride that exploit global spatial information for downsampling, while AdaStride only repeats or removes frames, as would be done in DTW. It would be interesting to have more discussion on this fundamental difference.\n * Experiments:\n  * The gains on the audio classification tasks are not significant, and I find the justification for the particularly weak performance on TUT not convincing, or at least incomplete. The authors explain that the \"information density is uniform\" in acoustic scenes which is somehow incorrect (and speculative anyway) since a car honking or a bird chirping are transient events that can hint at the type of acoustic scene.\n  * it is explained that AdaStride-S improves IEMOCAP but this is not what is shown by Table 1\n  * The audio classification dataset being quite small and subject to wide variations in results based on confounding factors such as the random seed, I find it not very convincing to cross-validate configurations (from AdaStride to AdaStride-S) or hyperparameters (sigma) on them, in particular on the test set.\n  * DiffStride is excluded from ASR experiments due to learning the downsampling factor, why is it a limitation?\n\nQuestions:\n* Why is Adastride limited to sequential data? What are the technical challenges to extend it to an arbitrary number of dimensions?\n* Section 3.1: why is the Miao reference put after the definition of a simple vector?\n\n\nTypos:\n* 4.1: \"dowmsampling\" -> \"downsampling\"\n* Section 3.1: \"a exponential\" -> \"an exponential\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7uVXAeBxILm", "reviewer": "Reviewer_PyHD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper an adaptive down-sampling method, AdaStride. The aim of AdaStride is to learn to deploy adaptive strides in a sequential data instance, i.e., preserving more information from task-relevant parts by using smaller strides while using larger strides for less-relevant parts. This idea is implemented by the cumulative-sum + normalization algorithm followed by a soft feature aggregation. Experimental results of speech classification and self-supervised learning are provided.", "summary_of_the_review": "I think that the quality of this paper is well. My only concern is that I'm not quite familiar with the area of audio processing. Therefore, I may not be able to properly evaluate the novelty of this paper. ", "strength_and_weaknesses": "Strength:\n\n- The motivation is clear and natural. The proposed method is novel on top of DiffStride.\n- Both the empirical results of supervised learning and self-supervised learning are provided.\n- AdaStride is a plug-in module, and can be applied to many deep networks.\n\nWeaknesses:\n \n- Only the audio data is considered. Can AdaStride be applied to 2D image data?\n- It would be better to present an analysis of the computation cost of AdaStride.\n\n\n##Post-rebuttal##\nAfter reading the rebuttal from the authors, I decided to keep my score. I think that this paper is just around the borderline and leaning toward accepting. However, as I have mentioned, I'm not quite familiar with the area of audio processing. Therefore, I may not be able to properly evaluate the novelty of this paper. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3-kBfNPpsZS", "reviewer": "Reviewer_PVam", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new downsampling method using adaptive strides for sequential data, especially audio data. Specifically, it rearranges each time step of an input on a one-dimensional line segment by using a method called vector positioning. By doing so, it builds an alignment matrix for the downsampling. The paper shows the results of the method on different tasks including audio classification, automatic speech recognition, and discrete representation learning, and claims the method is generalizable and effective.", "summary_of_the_review": "Please refer to the weaknesses and details above. ", "strength_and_weaknesses": "Strength: The paper written is good; the idea is easy to follow; the motivation is clear.\n\nweaknesses: See the next Section. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject