LEARNING THE SPECTROGRAM TEMPORAL RESOLUTION FOR AUDIO CLASSIFICATION
Authors: Haohe Liu
Xubo Liu
Qiuqiang Kong
Wenwu Wang
Mark D Plumbley
Venue: Submitted to ICLR 2023 (2023)
Keywords: audio classification
differentiable temporal resolution
feature dimension reduction

Abstract: The audio spectrogram is a time-frequency representation that has been widely used for audio classification. The temporal resolution of a spectrogram depends on hop size. Previous works generally assume the hop size should be a constant value such as ten milliseconds. However, a fixed hop size or resolution is not always optimal for different types of sound. This paper proposes a novel method, DiffRes, that enables differentiable temporal resolution learning to improve the performance of audio classification models. Given a spectrogram calculated with a fixed hop size, DiffRes merges non-essential time frames while preserving important frames. DiffRes acts as a "drop-in" module between an audio spectrogram and a classifier, and can be end-to-end optimized. We evaluate DiffRes on the mel-spectrogram, followed by state-of-the-art classifier backbones, and apply it to five different subtasks. Compared with using the fixed-resolution mel-spectrogram, the DiffRes-based method can achieve the same or better classification accuracy with at least 25% fewer temporal dimensions on the feature level, which alleviates the computational cost at the same time. Starting from a high-temporal-resolution spectrogram such as one-millisecond hop size, we show that DiffRes can improve classification accuracy with the same computational complexity. 

Under review as a conference paper at ICLR 2023
LEARNING THE SPECTROGRAM TEMPORAL RESOLU- TION FOR AUDIO CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
The audio spectrogram time-frequency representation that has been widely used for audio classification. The temporal resolution of spectrogram depends on hop size. Previous works generally assume the hop size should be constant value such as ten milliseconds. However, fixed hop size or resolution is not al- ways optimal for different types of sound. This paper proposes a novel method, DiffRes, that enables differentiable temporal resolution learning to improve the performance of audio classífication models. Given a spectrogram calculated with fixed hop size, DiftRes merges non-essential time frames while preserving im- portant frames. DiffRes acts as a "drop-in" module between an audio spectro- gram and classifier, and can be jointly optimized with the classification task. We evaluate DiffRes on the mel-spectrogram, followed by state-of-the-- aar classifier backbones, and apply five different subtasks. Compared with using the fixed- resolution mel-spectrogram, the DifiRes-based method can achieve the same or better classification accuracy with least 25% fewer temporal dimensions on the feature level, which alleviates the computational cost at the same time, Starting from a ter--eepporl-rssluuutiee spectrogram such as onemmillseccond hop size, we show that DiffRes can improve classification accuracy with the same compu- tational complexity.
INTRODUCTION
Audio classification refers to a series of tasks that assign labels an audio clip. Those tasks in- clude audio tagging (Kong et al., 2020), speech keyword classfication (Kim et al., 2021), and music genres classification (Castellon ally a one- one-dimensional audio waveform, which can be represented by discrete samples. Although there are methods using time-domain samples as features (Kong et al., 2020; Luo Mesgarani, audio classification convert the waveform into
al., 2021). The input to an audio classification system usu-
2018; Lee et al., 2017), the majority of studies spectrogram as the input feature (Gong et al., 202 202)b;a). Spectrogram is usually calculated by the Fourier transform (Champeney & Champeney. 1987), which is applied in short waveform chunks multiplied by a windowing function, resulting in a two-dimensional time-freuunccy representation. According to the Gabor's ' uncertainty principle (Gabor, 1946), there is always trade-off between time and frequency resolutions. To achieve desired resolution on the temporal dimension, it is a common practice (Kong al., 2021a; Liu al., 2022a) to apply a fixed hop size between windows to capture the dynamics between adjacent frames. With the fixed hop size, the spectrogram has a fixed temporal resolution, which we will refer simply resolution.
Using a fixed resolution is not necessarily optimal for an audio classification model. Intuitively, the resolution should depend on the temporal pattern: fast-changing signals are supposed have high resolution, while relatively steady signals or blank signals may not need the same high resolution for the best accuracy (Huzaifah, 2017). For example, Figure shows that by increasing resolution, more details appear in the spectrogram of Alarm Clock while the pattern of Siren stays mostly the same. This indicates the finer details in high-resolution Siren may not essentially contribute to the classifi- cation accuracy. There are plenty of studies on learning suitable frequency resolution with a similar spirit (Stevens et al., 1937: Sainath et al., 2013; Ravanelli & Bengio, 2018b; Zeghidour et al., 2021), I), but learning temporal resolution is still under-explored. Most previous studies focus investigat- ing the effect of different temporal resolutions (Kekre et al., 2012: Huzaifah, 2017; Ilyashenko et al., 2019; Liu et al., 2022c). Huzaifah (2017) observe the optimal temporal resolution for audio classifi-

---

Besides, the potential of the high-resolution spectrogram, e.g. with one milliseconds (ms) hop size, is still unclear. Some popular choices of hop size including 10 ms (Böck et al., 2012; Kong et al., 2020; Gong et al., 2021a) and 12.5 ms (Shen et al., 2018; Rybakov et al., 2022). Pre- vious studies (Kong et al., 2020; Ferraro et al., 2021) show classification performance can be steadily improved with the increase of resolution. One remaining question is: Can even finer resolution improve the performance? We conduct a pilot study for this question on a limited- vocabulary speech recogmition task with hop sizes smaller than 10 ms (see Figure 10b in Appendix A.3). We noticed that accuracy can still be improved with smaller hop size, Under review as a conference paper at ICLR 2023
cation is class dependent. Ferraro et al. (2021) experiment on music tagging with coarse-resolution spectrograms, and observes similar performance can be maintained while being much faster to compute. Kazakos et al. (2021) propose a two-stream architecture that process both fine-grained and coarse-resolution spectrogram and shows the state-of-the-an result on VGG-Sound (Chen al., 2020). Recently, Liu et al. (2022c) propose a spectrogram spectrogram-pooling-baeee classification efficiency with negligible performance degradation. In addition, our pilot study shows the optimal resolution is not the same for different types of sound (see Figure 10a in Appendix A.3). The pilot study shows that when increasing resolution, the improvement on different types of sound is not consistent, in which some of them even degrade with higher resolution. This motivates us to design a method that can learn the optimal resolution.
module that can improve
Alarto Clock mis hop siz
Alarm Clack
10 ms hop size
Sirep
Sirep 10 mz hop size
49 mp hpo size
Figure 1: The spectrogram of Alarm Clock and Siren sound with 40 ms and 10 ms hop sizes. All with a 25 ms window size. The pattern of Siren, which is relatively sta- ble, does not change significantly using a smaller hop size (i.c., larger temporal reso- lution), while Alarm Clock is the opposite.
at cost of increased computational complexity. This in- dicates there is still useful information in the higher tem- poral resolution.
In this work, we believe that we are the first to demon- strate learning temporal resolution the audio spectro-
gram. We show that learning temporal resolution leads to efficiency and accuracy improvements over the fixed-resolution spectrogram. We propose a lightweight algorithm, DiffRes, that makes spectrogram resolution differentiable during model optimization. DiffRes can be used as a "drop- in" module after spectrogram calculation and optimized jointly with the downstream task. For the optimization of DiftRes, we propose a loss function, guide loss, to inform the model of the low importance of empty frames formed by SpecAug (Park et al., 2019). The output of DiffRes is a time-freque frequency representation with varying resolution, which is achieved by adaptively merging the time steps of a exeedresolution spectrogram. The adaptive temporal resolution alleviates the spec- trogram temporal redundancy and can speed up computation during training and inference. We per- form experiments on five different audio tasks, including the largest audio dataset AudioSet (Gem- meke et al., 2017). DiffRes shows clear improvements on all tasks over the fixed-resolution mel- spectrogram baseline and other learnable front-ends (Zeghidour et al., 2021; Ravanelli & Bengio, 2018b; Zeghidour et al., 2018). Compared with the fixed-resolutied spectrogram, we show that using DiffRes can achieve a temporal dimension reduction of at least 25% with the same or better audio classification accuracy. On high-resolution spectrogram, we also show that DiffRes classifier performance without increasing the feature temporal dimensions. Our code is publicly available
improve
LEARNING TEMPORAL RESOLUTION WITH DIFFRES
We provide an overview of DiffRes-based audio classification in Section 2.1. We introduce the detailed formulation and the optimization of DiftRes in Section 2.2.1, 2.2.2, and 2.3.
2.1 OVERVIEW
Let æ E R& denote a one-dimensional audio time waveform, where L is the number of audio samples. An audio classification system can be divided into a feature extraction stage and a classi- fication stage. In the feature extraction stage, the audio waveform will be processed by a function Qi R - which maps the time waveform into a two-dimensional time- fmeefregy repre-
hhttps://annnnnnnt www.n.sccenceeeeeeeoe.ecc wetteeeneee//fiiiieeeeeeeeeeeeee ffres-frreeeee 8H22
2

---

Under review as a conference paper at ICLR 2023
DiffRes FV
Melfa
ErfciintNee Clacsifeer
ResCondlb
Que
Gv
Input
Calculate
Estimate
Rescale
Calculata warp metrix
Manga
Down stream task
audio waveforr mer spectrogeam
frame importance
Importance score
fromes
Figure Audio classification with DiffRes and mel-spectrogram. Green blocks contain learnalle parameters.
DiflRes is a "drop-in" module between spectrogram calculation and the downstream task.
sentation X, such as a mel-speetrogramm where X,r = (.., XF,)) is the 7-th frame. Here, T and F stand for the time and frequency dimensions of the extracted representation. We also refer to the representation along the temporal dimensions as frames. We use I and h to denote window length and hop size, respectively. Usually T x 4. We define the temporal resolution At by frame per second (FPS), which denotes the number of frames in one second. In the classification stage, X will be processed by a classification model Ge parameterized by 8. The output of Go is the label pre- dictions ÿ. in which y/: denotes the probability of class i. Given the paired training data (æ, y) D, where y denotes the one-hot vector for ground-truth labels, the optimization of the classification system can be formulated as
arg min (eyy)) C(Go (X).))))))
(1)
where C is a loss function such as cross entropy (De Boer et al., 2005). Figure show an overview of performing classification with DiffRes DiffRes is "drop-in" module between X and Go fo- cusing on learning the optimal temporal resolution with a learnable function F& R FxT - Rxx, where t is the parameter denoting the target output time dimensions of DiffRes, and ø is the learn- able parameters. DiffRes formulates Fs with two steps: i) estimating the importance of each time frame with a learnable model Hg: X - s. where s is a X T shape row vector, and ii) warping frames based on frame warping algorithm, the warping is performed along a single direction on the temporal dimension. We introduce the details of these two steps in Section 2.2.1 and Section 2.2.2. We define the dimension reduction rate 5 of DiffRes by 8 = (T L)/T. Usually, 5 S 1 and / & T because the temporal resolution of the DiffRes output either coarser or equal to that of X. Given the same T, larger å means fewer temporal dimensions in the output of DiftRes, and usually less computation is needed for Gg. Similar to Equation 1, Fo can be jointly optimized with Go by
arg min E( ([ (x)())).
0,6
2.2 DIFFERENTIARLE TEMPORAL RESOLUTION MODELING
Figure illustrates adaptive temporal resolution learning with DiffRes. We introduce frame impor- tance estimation in Section 1. and introduce the warp matrix construction function and frame warping function in Section 2.2.2. Figure 4 shows an example of DiffRes on the mel-spectrogram.
2.2.1 FRAME IMPORTANCE ESTIMATION
Convid: Cu-CCuur
LeakyReLU..
We design a frame importance estimation module He to decide the propor- tion of each frame that needs to be kept in the output, which is similar to the sample weighting operation (Zhang & Pfister, 2021) in previous studies. The frame importance estimation module will output row vector s' with shape x T. where the element s' is the importance score of the 7-th time frame X. s. The frame importance estimation can be denoted as
BatchNormid
Convld: Caur+Cuun
LeakyRelUpo:
BatchNormid
s' (HH
(3)
where s' is the row vector of importance scores, and ar is the sigmoid func- tion (Han & Moraga, 1995). A higher value in 8' indicates the 7-th frame is important for classification. We apply the sigmoid function to stabilize training by limiting the values in s' between zero and one. We implement He with a stack of one-dimensional convolutional neural networks (CNNs) (Fukushima & Miyake, 1982; LeCun al., 1989). Specifically, H. is a stack of five one-dimensional convolutional blocks (ResConv1D), We design the ResConv1D block following other CNN based methods (Shu et al., 2021; Liu et al., 2020;
Figure 3: ResConv1D
3

---

Under review as a conference paper at ICLR 2023
Fik-temporall-esohiee mel- speetregremm....m.. x
Frame importance Estimation gixr
Adaptive-temgoraleeed Spectrogram ge*
Warp Matrix Wist
Resolution encoding gei
Figure 4: Visualizations of the DiffRes using the mel-spectrogram. The part with the shaded background is the input feature of the classifier. For more examples please refer to Figure 12 and 13 in Appendix A.8.
Kong et al.. 2021b). As shown in Figure 3, each ResConv1D has two layers of mne-dimessiona CNN with batch normalization (Ioffe & Szegedy, 2015) and leaky rectified linear unit activation functions (Xu et al., 2015). We apply residual connection (He et al., 2016) for easier training of the deep architecture (Zaeemzadeh et al., 2020). Each CNN layer is zero-padded to ensure the temporal dimension does not change (LeCun et al., 2015). We use exponentially decreasing channel numbers to reduce the computation (see details in Table 10). In the next frame warping step (Section 2.2.2), elements in the importance score will represent the proportion of each input frame that contributes to output frame. Therefore, we perform rescale operation s', resulting an 8 that satisfies
[0, 1] and Σ 8k S t. The rescale operation can be denoted as max( where å is an intermediate variable that may contain elements greater than one, max denotes the maximum operation. To quantify how active Ho trying to distinguish between important and less important frames, we also design measurement, activeness which is calculated by the standard derivation of the non-empty frames, given by
8 -
1 Eies 1 8
8:)2
Snetive = B( min((((..)) +e},
(4)
netive
where Sactive is the set of indices of non-empty frames, is small value, S] denotes the size of set S, function E(-) calculates the root-mean-square energy (Law & Rennie, 2015) frame in the spectrogram, and function min(.) calculates the minimum value within matrix. We use 5 to unify the value of p for easier comparison between different 6 settings. The activeness p can be used as an indicator of how DiffRes behaves during training. A higher P indicates the model is more active at learning the frame importance. A lower p such as zero indicates learning nothing. We will discuss the learning process of DiffRes with p in Section 3.3.
2.2.2 TEMPORAL FRAME WARPING
We perform temporal frame warping based on and X to calculate adaptive temporal resolution representation O which similar to the idea of generating derived features (Pentreath, 2015). Gen- erally, the temporal frame warping algorithm can be denoted by W = a(s) and O = B(X,, W), where ax(-) is a function that convert s into a warp matrix W with shape x T, and B(-) is a function that applies W to X to calculate the warpped feature O. Elements in W such W denote the contribution the j-th input frame X., to the i-th output frame O.,.. We will introduce the real- ization of (-) and (()) in the following sections. Figure 11 in Appendix A.7 provides an example of the warp matrix construction function.
Warp matrix construction Function a(-) calculates the warp matrix W with by:
8j. if El 8 Si+1
W
(5)
0, othherwise
where we calculate the cumulative sum of & to decide which output frame each input frame will be warped into. The warp matrix W here can be directly used for frame warping function 3(-)- However, we found the DiffRes can learn more actively if the warp matrix satisfies Σ W 1, which means each output frame is assigned equal amount of total warp weights on input frames. We further process W to meet this requirement with Algorithm in Appendix A.6. We also provide vectorized version of Algorithm that can be run efficiently GPUs (see Appendix A.7).
Frame warping Function 3(-) performs frame warping based on the warp matrix W. The i-th output frame is calculated with X and the i-th row of W, given by
Oi A((X j.) (,,)).
where A I + R stands for the frame aggregation function such as averaging, O is the final output feature with shape F f.
4

---

Under review as a conference paper at ICLR 2023
Resolution encoding The final output O does not contain the resolution information at each time step, which is crucial information for the classifier. Since the temporal resolution can be represented with W, we construct resolution encoding with W in parallel with frame warping. Firstly, construct a positional encoding matrix with shape x T, using the similar method described in Vaswani et al. (2017). Each column of E represents a positional encoding of a time step. Then we calculate the resolution encoding by & = EW where W T stands for the transpose of W. The shape of the resolution encoding is F x Both and O are concatenated on the channel dimension as the classifier input feature.
2.3 OPTIMIZATION
We propose a guide loss to provide guidance for DiffRes on learning frame importance. Since we do not know the ground truth frame importance, we cannot directly optimize 8. We introduce gu as an inductive bias (Mitchell, 1980) to the system based on the assumption that an empty frame should have low importance score. Specifically, we propose the guide loss by
Cyuide
a)+, Sempty {i & Sactive and E
T}}
(7)
Sempyy
iESengy
where Sempty is a set of time indexes that have low energy, and is a constant threshold. Given that the output of DiffRes has fewer temporal dimensions than X, the DiffRes layer forms an information bottleneck (Tishby et al., 2000; Shwartz-Ziv & Tishby, 2017) that encourages DiffRes to assign higher score to important frames. We analyze the information bottleneck effect of DiffRes in Section 3.3. The parameter À is a threshold for the guide loss to take effect. This threshold can alleviate the modeling bias toward energy. For example, if 0, the importance scores of empty frames are strongly regularized, and the model will also tend to predict low importance scores for lower energy frames, which may contain useful information. Lace the standard binary cross entropy loss function (Shannon, 2001) for classification, given by Equation 8, where is the label prediction and N the total number of classes.
Loce Σ y log((:) + +(1 - y:) log(1 9.))),
(8)
The loss function of the DifiRes-based audio classification system includes our proposed guide loss Eguide and the binary cross entropy loss Crees given by = Lbee + guuide.
EXPERIMENTS
We focus on evaluating DiftRes on the mel-specttogram, which one of the most popular features used by state-of-the-art systems (Chen et al., 2022; Gong et al., 2022; Verbitskiy et al., 2022; Koutini al., 1). We evaluate DiffRes on five different tasks and datasets (see Table Appendix A.5), including audio tagging on AudioSet (Gemmeke et al., 2017) and FSD50K (Fonseca et al., 2021), environmental sound classification on ESCS0 (Piczak, 2015), tion on SpeechCommands (Warden, 2018), and music instrument classification NSynth (Engel et al., 2017). All the datasets are resampled at sampling rate of 16 kHz. Following the evaluation protocol in the previous works (Zeghidour et al., 2021; Riad et al., 2021; Kong et al., 2020; Gong et al., 2021b bb, we report the mean average precision (mAP) as the main evaluation metric on Au- dioSet and FSD50K, and report classification accuracy (ACC) other datasets. In all experiments, we use the same architecture as used by Gong et al. (2021b), which is an EfficientNet-B2 (Tan & Le. 2019) with four attention heads (13.6 M parameters). We reload the ImageNet pretrained weights for EfficientNet-B2 in similar way to Gong et al. (2021a;b). We also provide ablation study on other model architectures in Appendix A.1.2. For the training data, we apply random spec- augmentation (Park et al., 2019) and mixup augmentation (Zhang et al., 2017) following Gong et al. (2021b). All experiments are repeated three times with different seeds to reduce randomness. We also report the standard derivation of the repeated trails along with the averaged result. We train the DiffRes layer with À = 0.5 and = x 10-4, For the frame aggregation function A (see Equation 6), use both the max and mean operations, whose outputs are concatenated with the resolution encoding & on the channel dimension as the input feature to the classifier. The frame importance estimation module used in this paper is stack of five ResConv (see Table 10 in
voitted-ovcabuarry speech recogni-
5

---

Under review as a conference paper at ICLR 2023
Task name 100 FPS baseline (%)
Metric
FPS
Change hop size (%)
AvgPool (%)
ConvAvgPool (%)
Proposed (%)
25
38.6
+ 0.3
39.9 + 0.2
10.1 + 0.2
41.7 t 0.1
AudioSet tagging 43.7 + 0.1
mAP
50
41.8
+ 0.2
42.4
0.1
42.7 + 0.2
43.6
0.1
75
42.7 + 0.2
43.6 + 0.0
43.5 + 0.2
44.2 = 0.1
25
48.9
+ 0.4
51.4 + 0.3
19.2 + 0.4
56.9 = 0.2
FSDSOK tagging 55.6 + 0.3
mAP
50
53.3
0.4
54.5 0.4
52.2 + 0.8
57.2 = 0.2
75
+ 0.4 54.8
55.3 + 0.3
54.4 + 0.2
57.1 0.4'
25
74.6 + 0.6
75.6 + 0.3
72.4 + 1.2
82.9 + 0.5
Environmental sound 85.2 + 0.5
ACC
50
82.4
0.5
83.2 0.3
77.3 1 0.8
85.5 0.4
75
84.9 + 0.3
85.2 + 0.4
81.8 1 0.6
86.8 - 0.3!
25
98.5 + 0.1
94.9 + 0.4
95.8 = 0.3
95.0 = 0.3
Speech recognition 97.2 = 0.1
ACC
50
96.1 + 0.1
96.0 + 0.2
96.0 + 0.1
96.7 1 0.2
75
96.8 + 0.2
96.9 + 0.1
97.0 + 0.1
97.2 = 0.0
25
79.7 1 0.2
78.3 1 0.7
78.0 + 0.5
80.5 - 0.2
Music instrument 79.9 + 0.2
ACC
50
79.9
+ 0.01
79.5 + 0.3
79.4 + 0.3
81.0 0.51
75
79.8 + 0.2
79.6 + 0.3
79.7 + 0.4
80.8 = 0.21
Table 1: Comparison of different temporal dimension reduction methods. The numbers under the task name show the baseline performance. Baseline methods fix-temporal-resolution mel-spectrogram with 10 hop size. Numbers with mean better or comparable performance compared with the 100 FPS baseline.
Appendix A.5) with a total 82371 of parameters. We calculate the mel-spectrogram with a Han- ning window, 25 ms window length, 10 ms hop size, and 128 mel-filterbanks by default. We list our detailed hyperparameters setting in Table 8 (Appendix A.5). We also encourage readers to read through Figure 12 and Figure 13 in Appendix A.8 for examples of DiftRes on the mel-specttogram.
3.1 ADAPTIVELY COMPRESS THE TEMPORAL, DIMENSION
Compression of mel-spectrograa temporal dimension can lead to considerable speed up on train- ing and inference (see Figure 5), thus it is a valuable topic for efficient classification (Huang & Leanos, 2018) and on-device application (Choi al., 2022). The experiments in this section aim to evaluate the effectiveness of DiffRes in compressing temporal dimensions and maintaining pre- cision. We compare DiffRes with three temporal dimension reduction methods: i) Change hop size (CHSize) reduces the temporal dimension by enlarging the hop size. The output of CHSize has fixed resolution and may lose information between output frames.; ii) AvgPool is a method that performs average pooling on a 100 FPS spectrogram to reduce the temporal dimensions. Avg- Pool also has a fixed resolution, but it can aggregate information between output frames by pooling; úii) ConvArgPool is the setting that the 100 FPS mel-spectrogram will be processed by stack of ResConv (mentioned in Section 1),. followed by an average pooling for dimension reduc- tion. ConvAvgPod has total of 493 824 parameters and provide the detailed structure in Ta- ble 10 (Appendix A.5). Based on a learnable network, ConvAvgPool has the potential of learning more suitable features and temporal resolution implicitly.
Baseline comparisons Table shows our experimental result. The baseline of this experiment is performed on mel-spectrogram without temporal compression (i.e., 100 FPS) and the baseline result is shown under each task name. When reducing 25% of the temporal dimension (i.e., 75 FPS), the proposed method can even considerably improve the baseline performance on most datasets, except on speech recognition tasks where we maintain the same performance. We assume the improvement comes from the data augmentation effect of DiffRes, which means divergent temporal compression on the same data at different training steps. With 50 FPS, four out of five datasets can maintain comparable performance. With only 25 FPS, the proposed method can still improve the FSD50K tagging and music instrument classification tasks, which indicates the high temporal redundancy these datasets. Our proposed method also significantly outperforms other temporal dimension reduction baselines. With fixed resolution and fewer FPS, the performance of CHSize degrades more notably. AvgPool can outperform CHSize by aggregating more information between output frames. Although ConvAvgPool has extra learnable neural network, it does not show significant improvements compared with AvgPool. ConvAvgPool even has an inferior performance on FSD50K and environmental sound classification tasks. This indicates employing neural network for feature

---

Under review as a conference paper at ICLR 2023
learning is not always beneficial. Also, by comparison, the interpretability of our proposed method is much better than ConvAvgPool (see Figure 9).
On variable-length audio data Note that the proposed method even improves the mAP perfor- mance by 1.3% with only 25 FPS on the FSDS0K dataset. This is because the audio clip durations in the FSD50K have high variance (see Table 9). In previous studies (Gong et al., 2021a;by Kong et al., 2020), a common practice is padding the audio data into the same duration in batched training and inference, which introduces a considerable amount of temporal redundancy in the data with a significantly slower speed. By comparison, DittRes can unify the audio feature shape regardless of their durations. Model optimization becomes more efficient with DiffRes. As a result, the proposed method can maintain an mAP of 55.6 + 0.2 on the FSD50K, which is comparable to the baseline, with only 15 FPS and 28% of the original training time. This result shows that DiffRes provides a new mind map for future work on elassifying variable-length audio files.
3.2 LEARNING WITH SMALLER HOP SIZE
Previous studies have observed that a higher resolution spectrogram can improve audio classification accuracy (Kong et al., 2020; Ferraro et al., 2021). However, a hop size smaller than 10 ms has not been widely explored. This is partly because the computation becomes heavier for a smaller hop size. For example, with 1 ms hop size (i.e., 1000 FPS), the time and space complexity for an EfficientNet classifier will be 10 times heavier than with a common ms hop size. Since DifiRes can control the temporal dimension size, namely FPS, working on small hop size spectrogram becomes computationally friendly. Table 2 shows model performance can be considerably improved with smaller hop sizes. AudioSet and enviromment sound dataset achieve the best performance on 6 ms and 1 ms hop size, and other tasks benefit most from 3 ms hop sizes. In later experiments, we will use these best hop size settings on each dataset.
Hop size
10 ms
ms
3 ms
1 ms
FPS (Dimension reduction rate)
100 (0%)
100 (10%)
100 (70%)
100 (90%)
AudioSet tagging
43.7 + 0.1
44.1
+ 0.1
43.8
+ 0.0
43.7
+ 0.1
Environmental Sound Classification
85.2 + 0.4
87.2
4 0.3
88.0
+ 0.6
88.4 + 0.5
Speech recognition
97.2
+ 0.1
97.6
+ 0.0
97.9
+ 0.1
97.8
+ 0.1
Music instrument
79.9 + 0.2
81.3
+ 0.3
81.8
+ 0.2
80.6
+ 0.4
Average
76.5 + 0.2
77.6
+ 0.2
77.9
+ 0.1
77.5
+ 0.2
Table 2: Learning with high temporal resolution spectrograms. FPS is controlled at 100, so the computational complexity of the classifier is the same in all hop-size settings. Results are reported in the percentage format.
Comparing with other learnable front-ends The DiffRes is learnable, so the Mel+DifiRes setting as a whole can be viewed as a learnable front-end. Table compares our proposed method with SOTA learnable front-ends, our best setting is denoted as Mel+DiffRes (Best), which achieves the best result on all datasets. For a fair comparison, we control the experiment setup to be consistent with Zeghidour et al. (2021) in Mel+DiffRRe. Specifically, we change the backbone to EfficientNet-- BO (5.3 M parameters) without ImageNet pretraining. We also remove spec-augment and mixup, except in AudioSet, and change our Mel bins from 128 40, except the AudioSet experiment where we change to 64. The result shows Mel+DiffRes can outperform SOTA learnable front- end (Zeghidour et al., 2021; Ravanelli & Bengio, 2018b: Zeghidour al., 2018) by a large margin.
Front-end
Mel
TD-fbank
SincNet
LEAF
Mcl+DiffRes
Mel+DiffRes (Best)
Parameters
0
51
256
448
82
82 k
AudioSet tagging 96.8 0.1
96.5
0.1
96.1
1 0.0 96.8
1 0.1
97.0
0.0
97.5 + 0.0
Speech recognition
93.6 + 0.3
89.5
1 0.4
91.4
1 0.4
93.6
1 0.3
95.4
: 0.2
97.9 + 0.1
Music instrument
70.7 +:201 0.6 66.3
+ 0.6
67.4
+ 0.6
70.2
1 0.6
78.5
4 0.7
81.8 1 0.2
Average
87.0 1 0.3 84.1 1 0.4 85.0 1 0.3 86.9
1 0.3
90.3
1 0.3
92.4 1 0.1
Table 3: Comparison with SOTA Icarnable front-ends. All the methods use 100 FPS. Results are reported in the percentage format. Mel-DiffRes controls the experimental settings mentioned in Section 3.2 to be consistent with Mel, TD-fbank, SincNet, and LEAF. Mel+DifiRes (Best) use the best possible settings.
7

---

Under review as a conference paper at ICLR 2023
Mell Proposed
Throughout second
#PS-50 8m75%
FPS
21
LEAF
DifRee activeness
Guide loss valuee
FPS-50
0-1
0-50%
22
FPS-25
6-255%
2*-
0.2
0.3
21-
50
100
Frame per second
Steps (K)
Steps (K)
Figure 5: Audio throughput in one second. Evaluated on 2.6 GHz Intel Core i7 CPU.
Figure 6: Trajectories of DitlRes learning activeness (p) on differ- ent training steps and FPS set- tings.
Figure The training curve of guide loss (Eguide) with different FPS settings.
Computational cost We assess the one-second throughput of different front-ends on various FPS settings compare their computational efficiency. We control the FPS of Mel and LEAF by av- erage pooling. The computation time is measured between inputting waveform and outputting la- bel prediction (with EfficientNet-B3 fair comparison with 128 mel- filttebtbbbbks in Mel and DiffRes. As shown in Figure 5, our proposed DiffRes only introduces marginal computational cost compared with Mel. The state-of- oftte-of-thearrd of-tthe-rt learn able front-end, LEAF, is about four times slower than our proposed method. The majority of the cost computation in LEAF comes from multiple complex-van valued convolutions, which are computed in the time-domain with large kernels (e.g., 400) and stride of one.
We use 128 filters in LEAF (Zeghidour et al., 2021) for a
3.3 ANALYSIS FOR THE LEARNING OF DIFFRES
Learning activeness DiffRes does not explicitly learn the optimal frame importance score because the ground truth frame importance is not available. Instead, DiffRes is optimized with the guid- ance of guide loss ggudde (Equation 7), which is strong assumption introduced to the model. Figure shows the trajectories of the DiffRes learning activeness (defined in Section 2.2.1) during the optimization with different FPS settings on the speech recognition task in Table 1. According to the final converged value, DiffRes with a smaller FPS tend to be more active at learning frame importance. This is intuitive since smaller FPS leads more information bottleneck effects (Saxe et al.. 2019) in DiffRes. With a 25 FPS, the activeness even keeps increasing with more training steps, indicating the active learning of DifiRes. Figure 7 shows the guide loss curve during training with different FPS settings. Intuitively, when the FPS is small, model needs to preserve more non-empty frames and fewer empty frames for better accuracy. This assumption aligned with our experiment result, which shows the model tends to have a lower guide loss with smaller FPS.
Data augmentation and regularization effect As reflected in the curve of p and Lquide in Figure and 7, DiffRes is optimized along with the classifier during training. Hence DiffRes may produce different outputs for the same training data at different epochs. This is equivalent to performing data augmentation the audio data. We suppose this is the main reason for the improved performance shown in Table 1. Also, DiffRes reduces the sparsity of the audio feature by adaptive temporal compression. This is equivalent performing implicit regularization (Neyshabur, 2017; Arora et al., 2019) on the feature level, which beneficial for the system efficiency.
Ablation studies We perform the ablation study on AudioSet (Gemmeke et al., 2017), the largest audio dataset by far. The ablation study is performed on the mel-spectrogram with 10 ms hop size and 75% DiftRes dimension reduction rate (i.e., 25 FPS). Figure 8 shows the value of guide loss, activeness, and mean average precision (mAP) with different training steps and ablation setups. As the red solid line shows, if we remove the guide loss, the DiffRes activeness becomes very high, but the mAP becomes worse. Mcanwhile, the value of guide loss increases gradually, which is counter-intuitive because the empty frames should not have high important scores. This indicates guide loss is an effective prior knowledge we can introduce to the system. As the dotted green line shows, we remove the resolution encoding, the curve of guide loss and activeness almost show no changes, while the mAp degrades 1.15%. This indicates resolution encoding is crucial information for the classifier. If we remove the max aggregation function (see the dotted black line), both the activeness and the mAP have a notable degradation. As shown by the green solid line, if we do not apply Algorithm 1, the DiftRes activeness will converge to a small value, which means DiffRes is not working active enough, indicating Algorithm is crucial for DiffRes to take effects.
8

---

Under review as a conference paper at ICLR 2023
0.8
0.15
Mean average precision
0.42
Diffies activeness
wio resolut encoding
loss value
weo gurde loss
0.10
wio apgregation
0.4
w/o iimm
0.38
N~
0.05
0.2
0.00
50
100
200
250
100
150
200
250
150 200
250
Step (K)
Step (K)
Step (K)
Figure 8: The ablation study on AudioSet tagging task with 10 hop size and 75% dimension reduction rate. Three figures visualize guide loss, DiffRes activeness p. and the mean average precision (mAP), respectively, with different training steps and ablation setups.
For a more detailed analysis of different hyper-paramete values in the guide loss, please refer to Appendix 1.1.1. We also provide weakness analysis of DiffRes in Appendix A.4.
RELATED WORKS
Neural-network based methods have been successfully applied on audio classification and achieved state-of-the-art performance, such as the pretrained audio neural networks (PANNs) (Kong et al., 2020), pretraining. sampling, labeling, and aggregation based audio tagging (PSLA) (Gong et al., 2021b), and audio spectrogram transformer (AST) (Gong et al., 2021a). We will cover two related topics on audio classification in the following sections.
Learnable audio front-ends In recent years, learning acoustic features from waveform using train- able audio front-ends has attracted increasing interest from researchers. Sainath al. (2013) in- troduced one of the earliest works that propose to jointly learn the parameter of filter banks with a speech recognition model. Later, SincNet (Ravanelli & Bengio, 2018b) proposes to learn a set of bandpass filters the waveform and has shown success speaker recognition (Ravanelli & Bengio, 2018b;a). Most recently, (Zeghidour et al., 2021) proposes to learn bandpass, and low- pass filtering well as per-channel compression (Wang al., 2017) simultaneously in the audio front-end and shows consistent improvement audio classification. Different from existing work on learnable audio front-ends, which mostly focus the frequency dimension, our objective is learning the optimal temporal resolution. We show that our method can outperform existing audio front-ends for audio classification on both accuracy and computation efficiency (see Table and Figure 5). Note that our proposed method can also be applied after most learnable front-ends (Zeghidour et al., 2021), which will be our future direction.
Learning feature resolution with neural networks One recent work on learning feature resolu- tion for audio classification is DiffStride (Riad al., 2021), which learns stride convolutional neural network (CNN) in differentiable way and outperforms previous methods using fixed stride settings. By comparison, DiffStride needs to be applied each CNN layer and can only learn a sin- gle fixed stride setting, while DiftRes is a one-layer lightweight algorithm and can personalize the best temporal resolution for each audio during inference. Recently, Gazneli et al. (2022) proposed to use stack of one-dimension- CNN blocks to downsample the audio waveform before the audio classification backbone network, c.g.. Transformer, which can learn temporal resolution implicitly for audio classification. In contrast, DiffRes can explicitly learn temporal resolution on the feature level with similar interpretability as mel-spectrogram.
5 CONCLUSIONS
In this paper, we introduce DiffRes, "drop-in" differentiable temporal resolution learning module that can be applied between audio spectrogram and downstream tasks. For the training of DiffRes, our proposed guide loss is shown to be beneficial. We demonstrate over large range of tasks that DifiRes can improve or maintain similar performance with 25% to 75% reduction on tempo- ral dimensions, and can also efficiently utilize the information in high-resolution spectrograms to improve accuracy. In future work, we will move forward to evaluate DiffRes on different kinds of time- frequenyy representations with more sophisticated frame importance prediction models. Also, we believe the idea DiffRes can work on other time series data as well for learning optimal tem- poral resolutions, such video seismic data.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
We believe that we have covered the details of our proposed method with figures (see Figures 13), algorithms (see Algorithm I to 3) and mathematical equations (see Equation 2 to 8). As shown in Table and 8, the hyperparameters are provided as exhaustive as we can, both in the main text and the appendix. The realization the LEAF module we used for throughput comparison weeaeeaaaawaaaaaaaaaaaaaa https: h httg///ttubbbtt com/ eetaaaaaaarreeeaappoorrrhh pytorch. We also open-source our code https: anonymous 4open. wwt.eeeneeereefeeeeeeeeeeee 8F22 facilitate reproducibility.
REFERENCES
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
Sebastian Böck, Andreas Arzt, Florian Krebs, and Markus Schedl. Online real-time onset detection with recurrent neural networks. Proceedings of the International Conference on Digital Audio Ejffects 2012.
Rodrigo Castellon, Chris Donahue, and Percy Liang. Codified audio language modeling learns useful representations for music information retrieval. arxivv2107.05677, 2021.
David C Champeney and DC Champeney. A Handbook of Fourier Theorems. Cambridge University Press, 1987.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio- visual dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 721-725. IEEE, 2020.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor KingKKikkaatrick, and Shlomo Dubnov. HTS- AT: A hierarchical token-semantic audio transformer for sound classification and detection. In IEEE International Conference Acoustics, Speech and Signal Processing, pp. 646-650, 2022.
Kwanghee Choi, Martin Kersner, Jacob Morton, and Buru Chang. Temporal knowledge distillation for on-device audio classification. In IEEE International Conference Acoustics, Speech and Signal Processing, pp. 486-490, 2022.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134(1):19667, 2005.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi. Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with WaveNet autoencoders. In International Conference Machine Learning, pp. 1068-1077 PMLR, 2017.
Andres Ferraro, Dmitry Bogdanov, Xavier Serra Jay, Ho Jeon, and Jason Yoon. How low can you go? Reducing frequency and time resolution in current CNN architectures for music auto-tagging. In IEEE European Signal Processing Conference, pp. 131-135, 2021.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSD50K: An open dataset of human--labeled sound events. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829 852, 2021.
Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and Cooperation in Neural Nets, pp- 267-285. Springer, 1982.
Dennis Gabor. Theory of communication. Part 1: The analysis of information. Journal of the Institution of Electrical Engineers, 93(26):429 1946.
Avi Gazneli, Gadi Zimerman, Tal Ridnik, Gilad Sharir, and Asaf Noy. End-to-end audio strikes back: Boosting augmentations towards an efficient audio classification network. arXiv:2204. 11479, 2022.
10

---

Under review as a conference paper at ICLR 2023
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. AudioSet: An ontology and human-labeles dataset for audio events. In IEEE International Conference on Acoustics. Speech and Signal Processing, pp- 776-780, 2017.
Yuan Gong, YuAn Chung, and James 011777 2021a.
Glass.
AST: Audio
spectrogram transformer.
Yuan Gong, YuAn Chung, and James Glass. PSLA: Improving audio tagging with pretraining, sampling. labeling, and aggregation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3292-3306, 2021b.
Yuan Gong, Cheng-I Lai, Yu-An Chung, and James Glass. SSAST: Self-supervised audio spectro- gram transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10699-10709, 2022.
Jun Han and Claudio Moraga. The influence of the sigmoid function parameters on the speed of backpropagation learning. In International Workshop on Artificial Neural Networks, pp. 195- 201. Springer, 1995.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Jonathan Huang and Juan Jose Alvarado Leanos. AclNet: Efficient end-to-end audio classification CNN. 1xXi::811.00666 2018.
Muhammad Huzaifah. Comparison of time-frequency representations for environmental sound clas- sification using convolutional neural networks. arXiv preprint: 2017.
ID Ilyashenko, RS Nasretdinov, YA Filin, and AA Lependin. Trainable wavelet-like transform for feature extraction to audio classification. In Journal of Physics: Conference Series, volume 1333, pp. 032029. IOP Publishing, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456. PMLR, 2015.
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Slow-fast auditory streams for audio recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 855-859 855, 2021.
HB Kekre, Vaishali Kulkarni, Prashant Gaikar, and Nishant Gupta. Speaker identification using spectrograms of varying frame sizes. International Journal of Computer Applications, 50(20), 2012.
Byeonggeun Kim, Simyung Chang, Jinkyu Lee, Dooyong Sung. Broadcasted residual learning for efficient keyword spotting. arxiv:2106.08 10o.04400604444 140, 2021.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. AudioCaps: Generating captions for audios in the wild. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 119-132, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.69880 2014.
Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley, PANNs: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 08:2880-28941 2020.
Qiuqiang Kong. Yin Cao, Haohe Liu, Keunwoo Choi, and Yuxuan Wang. Decoupling magnitude and phase estimation with deep ResUNet for music source separation. arXiv:2109.05418, 2021a.
Qiuqiang Kong, Haohe Liu, Xingjian Du Li Chen, Rui Xia, and Yuxuan Wang. Speech enhance- ment with weakly labelled data from AudioSet. 1rXiv:2/02.09977 2021b.

---

Under review as a conference paper at ICLR 2023
Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient training of audio transformers with Patchout. arXiv:2110.05069, 2021.
Jonathan Law and Richard Rennie. Dictionary Of Physics. OUP Oxford, 2015.
Yann LeCun, Bemhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with back- proo-propagttion network. Advances in Neural Information Processing Systems, 2, 1989.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-44 2015.
Jongpil Lee, Taejun Kim, Jiyoung Park, and Juhan Nam. Raw waveform- baaed audio classification using sample-level CNN architectures. arxiv: 2017.
Haohe Liu, Lei Xie, Jian Wu, and Geng Yang. Channel-wise subband input for better voice and accompaniment separation on high resolution music. Interspeech 2020, pp. 1241-1245, 2020.
Haohe Liu, Xuho Liu, Xinhao Mei, Qiuqiang Kong, Wenwu Wang, and Mark D Plumbley, Segment- level metric learning for few-shot bioacoustic event detection. arXiv:2207.0777 2022a.
Xubo Liu, Qiushi Huang, Xinhao Mei, Haohe Liu, Qiuqiang Kong, Jianyuan Sun, Shengchen Li, Tom Ko, Yu Zhang, Lilian H Tang, et al. Visually-aware audio captioning with adaptive audio- visual attention. 1.Xi::2210016428 2022b.
Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei, Mark D. Plumbley, and Wenwu Wang. Simple pooling front-ends for efficient audio classification. arXiv:2210.00944 2022c.
Xubo Liu, Xinhao Mei, Qiushi Huang, Jianyuan Sun, Jinzheng Zhao, Haohe Liu, Mark D Plumbley, Volkan Kilic, and Wenwu Wang. Leveraging pre-trained BERT for audio captioning. In European Signal Processing Conference, pp. 1 1145-12 1149. IEEE, 2022d.
Yi Luo and Nima Mesgarani. TasNet: time-domain audio separation network for real-time, single- channel speech separation. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 696-700, 2018
Xinhao Mei, Qiushi Huang, Xubo Liu, Gengyun Chen, Jingqian Wu, Yusong Wu, Jinzheng Zhao, Shengchen Li, Tom Ko, H Lilian Tang, et al. An encoder-decoder based audio captioning system with transfer and reinforcement learning. arXiv:2108.02752, 2021.
Xinhao Mei, Xubo Liu, Haohe Liu, Jianyuan Sun, Mark D Plumbley, and Wenwu Wang. Automated audio captioning with keywords guidance. Technical report, Technical Report in DCASE2022 Challenge, 2022a.
Xinhao Mei, Xubo Liu, Mark D Plumbley, and Wenwu Wang. Automated audio captioning: an overview of recent progress and new challenges. arXiv:2205.05945 2022b.
Tom M Mitchell. The need for biases in learning generalizations. pp. 184 191, 1980.
Behnam Neyshabur. Implicit regularization in deep learning. arXiv: 2017.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. SpecAugment: A simple data augmentation method for automatic speech recognition. arXiv:1904.08777 08779, 2019.
Nick Pentreath. Machine Learning with Spark. Packt Publishing Birmingham, 2015.
Karol J Piezak. ESC: Dataset for environmental sound classification. In ACM International Confer- ence on Multinedia, pp. 1015-1018, 2015.
Mirco Ravanelli and Yoshua Bengio. arXiv:1811.09725 2018a.
Interpretable convolutional filters with SincNet.
Mirco Ravanelli and Yoshua Bengio. Speaker recognition from raw waveform with SincNet. In IEEE Spoken Language Technology Workshop, pp. 1021-1028, 2018b.
12

---

Under review as a conference paper at ICLR 2023
Rachid Riad, Olivier Teboul, David Grangier, and Neil Zeghidour. Learning strides in convolutional neural networks. In International Conference on Learning Representations, 2021.
Oleg Rybakov, Marco Tagliasacchi, Yunpeng Li, Liyang Jiang. Xia Zhang, and Fadi Biadsy. Real time spectrogram inversion on mobile phone. 2r030:2203.00756, 2022.
Tara N Sainath, Brian Kingsbury, Abdelrahman Mohamed, and Bhuvana Ramabhadran. Learning filter banks within deep neural network framework. In JEEE Workshop Automatic Speech Recognition and Understanding, pp. 297-302, 2013.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12) 2019.
Claude Elwood Shannon. A mathematical theory of communication. ACM Mobile Computing and Communications Review, 5(1) 311):3555, 2001.
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan, et al. Natural TTS synthesis by condi- tioning WaveNet on mel spectrogram predictions. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp.
2018.
Xiaofeng Shu, Yehang Zhu, Yanjie Chen, Li Chen, Haohe Liu, Chuanzeng Huang, and Yuxuan Wang. Joint echo cancellation and noise suppression based on cascaded magnitude and complex mask estimation. N1XYv:2107.092900 2021.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv: 1703.00000, 2017.
Stanley Smith Stevens, John Volkmann, and Edwin Broomell Newman. A scale for the measurement of the psychological magnitude pitch. The Journal of the Acoustical Society of America. 8(3); 185-190, 1937.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114 14. PMLR, 2019.
Naftali Tishby, Fernando Percira, and William Bialck. The information bottleneck method. arXiv physics/0004057, 2000.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neurul Informa- tion Processing Systems, 30, 2017.
Sergey Verbitskiy, Vladimir Berikov, and Viacheslav Vyshegorodtsev. ERANNs: Efficient residual audio neural networks for audio pattern recognition. Pattern Recognition Letters, 161:38 44, 2022.
Yuxuan Wang, Pascal Getreuer, Thad Hughes, Richard F Lyon, and Rif Saurous. Trainable fron- tend for robust and far-field keyword spotting. In IEEE International Conference Acousties, Speech and Signal Processing, pp. 5670-5674, 2017.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition.
arXiv:1804.00 000,:000443399 2018
Bing Xu, Naiyan Wang. Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. 15Xiv:1555.0085. 2015.
Alireza Zaeemzadeh, Nazanin Rahnavard, and Mubarak Shah. Norm- pomm-preservation: Why residual networks can become extremely deep? IEEE Transactions on Pattern Analysis and Machine Intelligence, 43( 13(11):3980-3990 2020.
Neil Zeghidour, Nicolas Usunier, Iasonas Kokkinos, Thomas Schatz, Gabriel Synnaeve, and Em- manuel Dupoux. Learning filterbanks from raw speech for phone recognition. IEEE Acoustics, Speech and Signal Processing, 2018
13

---

Under review as a conference paper at ICLR 2023
Neil Zeghidour, Olivier Teboul, Félix de Chaumont Quitry, and Marco Tagliasacchi. LEAF: A learnable frontend for audio classification. arxiv:2101.08500 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. MixUp: Beyond empir- ical risk minimization. arX1v:1710.09412, 2017.
Zizhao Zhang and Tomas Pfister. Learning fast sample re-weighting without reward data, In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 725-734, 2021.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.I ABLATION STUDIES
We report ablation studies on hyper-parameters and different model architectures in Appendix A.I.1 and A.1.2, respectively. In Appendix A.1.3, also discuss whether DiffRes only learns remove silent frames.
A.1.1
In this section, we provide ablation studies and discussions on the hyper-pan saper-parameters in DiffRes, including the threshold À mentioned in Equation 7, dimension reduction rate 5, and the used in Equation 4. We choose conduct the experiment on the SpeechCommands dataset since it has reasonable amount of data and is computationally friendly on model training compared with large datasets such as AudioSet (Gemmeke et al., 2017). The ablation study result on hyper-parameter presented in Table 4.
Metric
Hop size (ms)
A-0.0
A-0.3
A-0.5
A-0.8
Average
3
70%
98.0
97.9
98.0
98.0
98.0 0.0
97.8
Accuracy (%)
90%
98.0
98.0
98.0
98.0
98.0 + 0.0
97.8
0.5
95%
97.9
97.9
97.9
98.0
97.9 + 0.0
97.7
3
70%
32.4
29.4
28.1
30.6
30.1 1.6
20.6
Activeness p (%)
1
90%
42.3
42.8
43.9
42.4
42.9 - 0.6
17.9
0.5
95%
43.1
41.6
44.7
40.5
42.5 = 1.6
8.4
Average importance scores on empty frames (%)
3
70%
0.2
13.2
27.2
41.0
20.4 = 17.6
81.1
1
90%
0.2
4.6
11.8
30.5
11.8 - 13.4
91.6
0.5
95%
0.2
3.0
11.0
17.2
7.8 + 7.7
102.3
Guide loss applied
X
Table Ablation study on the SpeechCommands dataset. All experiments use 100 FPS. The bascline performance is 97.2 = 0.1 with 10 ms hop size and = 0%. We report the accuracy, activeness, and average importance score on empty frames on different hop size, dimension reduction rate 6, guide loss, and threshold À settings. The column "Average" denotes the average result on each metric with four different À values.
The effect of guide loss. Table 4 shows that even without guide loss, the model can still improve over the baseline performance (97.2 + 0.1) using DiffRes. At the same time, applying guide loss further improve the activeness p (see Equation 4) of DiffRes and classification performance. For example, without the guide loss, the with ms, ms, and 0.5 ms hop size are 20.6, 17.9, and 8.4, respectively, while after applying guide loss, the average P become 32.6, 45.4, and 45.0, respec- tively. The improvement on indicates guide loss can encourage the model to better discriminate between the importance of frames. The model classification accuracy improve by about 0.2% after applying guide loss, which is significant enough for the SpeechCommande dataset. Moreover, without guide loss, the model tends predict high-importance scores on empty frames, which also counterintuitive.
The effect of dimension reduction rate 8 With the same hop size, a smaller will lead larger temporal dimension in the DiffRes output feature, which also leads to heavier computational cost (see Figure 5). Even though smaller hop size and smaller tend to achieve better performance because finer temporal details can be preserved, in practice, the exact value of å still should be determined by the computation limit.
The effect of guide loss threshold À As shown in Table 4, tried different À on different hop sizes. The experimental result shows model accuracy is not sensitive to À thus the value usually does not need careful finetuning.
The effect of the small value e We use e Equation to control the threshold of deciding whether each frame is active or empty. In practice, gram, in which the empty frames Semply in Equation will the masked time steps. To verify e is not essential for model training, We try construct Sempty in Equation 7 training data with five
will apply SpecAug (Park al., 2019) on the spectro-
15

---

Under review as a conference paper at ICLR 2023
different values between 1 x 10 and 1 10 Our result shows more than 98% training data have the same Sempyy with different e values, which indicates is not an essential paramee for model training.
A.1.2 MODEL ARCHITECTURE
To verify the generality of the proposed approach, we also conduct experiments on two more state- of-the-art architectures, CNN6 and CNN14 (Kong et al., 2020). Experiments are conducted on the SpeechCommands dataset with the same setting as Table 1.
Hop Size (ms)
Frames per second
EfficientNet-b2
CNN6
CNN14
0%
100
97.2.10.1
96.4 0.1
97.9 0.1
25%
75
97.2 0.0
96.4 0.0
98.0 0.0
50%
50
96.7 0.2
96.1 1:1::00 0.1
97.7 + 0.0
10
75%
25
95.0 + 0.3
95.7 + 0.1
97.1 + 0.1
40%
100
97.6 0.0
96.8 0.0
98.1 + 0.0
3 70%
100
97.9 + 0.1
97.2 - 0.1
98.1 + 0.0
90%
100
97.8 + 0.1
97.2 + 0.0
98.1 + 0.2
Table 5: Ablation study the model architectures. We 8 denote the dimension reduction rate. Large 8
indicates less computational cost.
Table 5 shows ablation study result on different architectures. Three results exhibit similar trend as Table and Table 2. All three models can maintain similar or better performance after reducing 25% of the temporal dimensions. With the same number of frames per second, namely the same computational cost, all the models show clear improvement with a smaller hop size. This improvement indicates DiffRes is effective in selecting informative frames across different architec- tures. We do not experiment with other non-neural architecture because the optimization of DiffRes requires gradient pack-propagations (LeCun al., 2015).
A.1.3 REMOVE EMPTY FRAME OR SELECT IMPORTANT FRAME?
To study whether DiffRes learns to remove only silent frames, or it would be also effective when the signal has consistent energy, we design pitch classification experiment the NSynth dataset following (Zeghidour et al., 2021). We will refer to this task as NSynth-Pitch. We design the pitch classification task for the following two reasons: (i) The data in NSynth is mostly instrumental sounds, which have stable spectral patterns and are highly redundant for the pitch classification task. Thus NSynth-Pitch is an ideal use case of DiftRes. (ii) Our statistic shows about 19. 7% frames in this dataset are silent frames, thus any dimension reduction rate 5 larger than 19.7% means DiffRes need remove some non-empty frames to benefit classification accuracy.
Frames per second Dimension reduction rate
25/8 75%
50/6 50%
75/6 25%
AvgPool
90.5 0.3
91.3 + 0.2
92.6 + 0.2
Proposed
92.1 + 0.1
92.4 0.2
92.6 = 0.1
Table 6: Experiment result on the pitch classification task the NSynth dataset. All the experiments 10 ms hop size. The baseline performance 92.5 0.2, with 10 ms hop size 100 FPS.
Table 6 shows our result on the NSynth-Pitch task. All the settings use dimension reduction rate 5 19.7%, which means DiffRes have to remove part of the non-empty frames. If we reduce the temporal dimension with AvgPool, the performance will degrade significantly, while our proposed method can remain similar performance even after reducing 75% temporal dimensions. This result suggests DiffRes not only remove the silent frames but also preserves important frames for classi- fication. The high activeness (see Equation 4) in the non-empty frames shown in Figure 6 and Table 4 also indicate the model has leamed to distinguish the importance of the non-empty frames.
16

---

Under review as a conference paper at ICLR 2023
A.2 AUTOMATIC AUDIO CAPTIONING WITH DIFFRES
To further verify the generality of our proposed method, we conduct an extra set of experiments on the automatic audio captioning (AAC) task (Mei al., 2022b), which can automatically generate natural language descriptions for audio clips. We use the architecture proposed by Mei et al. (2021) and the same DiffRes setting introduced in Section for the experiments. Our experiments are on the AudioCaps dataset. We will try to reduce the input feature size of the AAC task and observe the change in model performance.
We conduct experiments on AudioCaps (Kim et al., 2019), which is the largest public audio cap- tioning dataset with around 50 000 10-second audio clips, and is divided into three splits: training, validation and testing sets. The audio clips are annotated by humans through the Amazon Mechani- cal Turk (AMT) rrowd-sourced platform. Each audio clip in the training sets has human-annotated caption, while each clip in the validation and test set has five ground-truth captions.
For model evaluation, we use the metrics calculated based on n-gram matching (n-gram refers to consecutive words) following previous works (Liu et al., 2022d:b). BLEU,, measures the preci- sion of n-gram matching and a sentence-brevity penalty is introduced to penalize short sentences. ROUGE, caleulates an F-measure by considering the longest common subsequence between the can- didate and ground truths. METEOR calculates uni-gr mui-gram precision and recall, taking into account the surface forms, stemmed forms, and meanings of words. CIDEr computes the cosine similarity of weighted ri- r--gamms between candidates and references. SPICE parses each caption into scene graphs and an F-measure is calculated based on the matching of the graphs. SPIDEr the average of SPICE and CIDEr and is used as the official ranking metric in DCASE challenge (Mei et al., 2022a).
8 (>=FLPs reduction)
BLEU-1
BLEU-4
METEOR
ROUGE,
CIDEr
SPICE
SPIDEr
0%
0.658
0.235
0.232
0.473
0.643
0.168
0.406
25%
0.665
0.247
0.228
0.471
0.657
0.171
0.414
50%
0.674
0.266
0.230
0.475
0.646
0.167
0.407
75%
0.659
0.252
0.225
0.478
0.649
0.164
0.407
Table 7: Applying DiffRes on the automatic audio captioning task, which exhibits a similar trend with audio classification tasks shown in Table I and Table 2. By removing 25% of the input dimensions, the performance some metrics even got improved. After removing 75% of the input temporal dimensions with DiffRes, the model still retain comparable result.
The result in Table 7 shows that applying DiffRes on the AAC task can significantly reduce the computational cost while preserving similar performance on most of the metries. We perform ex- periments with four different temporal dimension reduction rate settings, including 0%. 25%, 50%, and 75% reductions. The reduction on the temporal dimension also significantly benefits model throughput at the same time (see Figure 5). After removing 25% temporal dimensions, the perfor- mance of AAC even shows an improvement, which might be due to the data augmentation effect mentioned in Section 3. After removing 75% of the input temporal dimensions, the model can still achieve on-par results compared with the baseline 0% reduction. The 75% reduction setting can even improve five metrics out of the total seven metrics. The result of the AAC task further indicates our proposed method is generalizable to other similar audio tasks.
17

---

Under review as a conference paper at ICLR 2023
A.3 FIGURES
Mel-spectrogram
Proposed
ConvAvgPool
sheila la
sheilp
sheila-
Figure 9: Comparison mel-spectrogram, DiffRes feature, and AogPov learned feature. The DiffRes feature preserves more details in the original mel-spectrogram and is more interpretable than the ConvAvgPool feature.
Distribution the Improvement
mAP impravement (hop size 40ms 10ms)
mAp provement
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
-0.1
-0.2
Train whisti
Jingle bell
Vocal music
loot
Water tap, faucet
Beatboxing
Telephone bell ringing
Cymbel
cápella
Trobonne
Female singing
Progress Pogressive jvve rock
tone
Chants Ship
Sad music 1
Emergency vehicle
Buzzer
Gobble
Fusillade
Cougth
Printed
Skateboard
foothbrush
Goose
Whistling
Squall
Oink
Owh
Camerar
Duck
Single-lens reflex camera
Zipper (clothing)
-0.2
(a) Class-wise improvement after changing hop size from 40 ms to 10 ms.
100
Baseline
0.98
ReleLive complexity
Proposed
10
Accuracy
0.96
Baseline hop
1
0.94
5/. 6
Hop size (ms)
Hop size (ms)
(b) Accuracy and the classifier computational complexity with different hop size settings on the speech recognition task. black dotted lines show the accuracy, and complexity with 10 ms hop size.
Figure 10: Pilot studies. The mAP improvement for each class in the AudioSet after decreasing the hop size from 40 ms to ms. The violin plot on the right shows improvement distribution. where red dashed line the median value. The inconsistency of improvement in different sound classes indicates they need different temporal resolutions to achieve optimal classification performance. The accuracy can he improved with a smaller hop size at the cost of computation. DiffRes can achieve similar improvements without increasing computational complexity.
18

---

Under review as a conference paper at ICLR 2023
A.4 WEAKNESS ANALYSIS
Table 2 shows DiffRes does not improve the model performance on 1 ms setting on most datasets. This may be due the insufficient receptive field of the convolutions in DiffRes, which is around 41 time steps. By comparison, the temporal dimension of X on AudioSet is = 3333 and / = 000 with ms and ms hop size, respectively. DiftRes may not effectivelyy capture the useful information with only 41 temporal receptive field in this case. Future work will address this problem by designing the resolution prediction model with large receptive field.
A.5 DATASET AND EXPERIMENT DETAILS
Learning rate scheduler
Dataset
Learning rate
Epoch
Batchsize
GPU(s)
(start epoch, gamma, every epnch)
Audioset
1.0 10-4
30
22
(11, 0.5,
FSD50K
5.0 x 10
40
15
(21, 0.5, 5)
1
ESC50
2.5 10
80
32
(41, 0.95,
1
SpeechCommands
2.5 10
60
128
(25, 0.9, 1)
1
NSynth
1.0 10-4
30
48
0.85,
1
Table 8: Hyper-parameter setting. We all the experiments with ADAM optimizer (Kingma & Ba, 2014) GeForce RTX 2080 Ti GPU(s).
Audio tagging
Audio tagging
Environmental sound
Speech recognition
Music instrument
Task
Dataset
AudioSet
FSD50K
ESC50
SpeechCommands
NSynth
Classes
527
200
Train examples
1912 134
36 799
2000
84 771
289 205
Test examples
18 887
10231
10 700
12 678
Duration (mean,std)
9.91, 0.50
7.63, 7.82
5.00, 0.00 500
0.98, 0.07
4.00, 0.00
Pad to length
1000
3000
400
Evaluation metric
mAP
mAP
Accuracy
Accuracy
Accuracy
5-fold cross-validann validation
Class re- ee-balancing SpecAug
Table Detailed information of datasets we used paper. perform padding to unify the data length. The last row shows the mel-spectrogram temporal dimension we used for batched training.
Frame importance estimation module H
ConvAvgPool Encoder
Parameters (k) Kernel size
82.4
493.8
5
5
ResConvlll 1D blocks (input chnanel, onput chnanel)
(128, 64), (64, 32), (32, 16), (16, 8), (8, (128, 128), (128, 128), (128, 128)
Table 10:
The structure of the frame importance estimation module and the front-end structure of ConvAvg-
Pool (baseline method in Table l). The structure of ResConv1D is discussed in Section 2.2.
19

---

Under review as a conference paper at ICLR 2023
A.6 ALGORITHM FORMULATION OF DIFFRES
Algorithm 1: Inplace warp matrix update
Inputs : W Equation 5. W satisfies D -i=lj== Wi, t.
Output: The updated W that satisfies both E W = and Σ 1-j= W. t
i 1; j 1; 8 4- 0
> i.j for indexing, & store the sum value in each row.
while i < /nddd
do
if W. > 0 then
s - s + Wifi
Add up the weights row i. b Move to the next column in row i.
j { + 1:
else
Wij + -88; 8;
Assign weight to Waj make Σ W.,: b The assigned weight is taken from W ++1,f. Move to row + in column j. Moved to the next row. Reset the row sum value.
W f W Wasi
i bitt
8 8 0:
Algorithm 2: Perform audio classification with the DiffRes layer
Inputs: a R dinesinnnsional waveform. Fg: DiffRes layer. Gg: Classifier backbone.
Output: y: Classification result.
FFXT
(x); b Project input to the Fourier domain with fixed temporal resolution.
O Fxt gFxt
x 4- (x()); & DiffRes layer (see Algorithm 3). & is the resolution encoding. Classifier forward propagation.
(.,EE;
if training then
loss = Line + Lguidei backprop;
> Lguide the loss function for DiffRes (See Equation 7)
Algorithm 3: DiftRes layer Fo
Inputs: X E RxT, Fix-time-resolution spectrogram. E E FxT', Positional encoding.
Ho: Frame importance estimation network. E Z: Target temporal dimensions.
Output: O E Fxt, Two-ddmensional representation with adaptive temporal resolution.
E E FRt, Resolution encoding.
IXT - Norm (a (H)()))))
Calculate frame importance (see Equation 3).
WIXT t a(s:
> Construct a warp matrix (see Equation and Algorithm 1). b Warping frames with the warp matrix (see Equation 6). Encode the information of W into
FXt t 3(X, W);
Fxt - EWT;
20

---

Under review as a conference paper at ICLR 2023
A.7 FAST IMPLEMENTATION OF ALGORITHM
We provide a pure matrix operation-based version of Algorithm for efficient model optimization on GPUs. Given the output of Equation 5, Wn. will process it into matrix W with the same shape so that Σ W and Σ Li=1,j=1 Wij .j S Figure 11 provides an example of the fast implementation discussed in this section.
- W W T= 3.
S
Wo
(9)
P
(11)
[0.3,0.9.0.]]
10.
0.7
0.7
0.7
0
01
(12)
(12)
LO
0.9
0.81
- 0.7
-07
-0.71
0
0.7
0
+
(10)
(12)
K
M
Q
W
C
H
0
0
-1
0l
0
0.7
01
0..
0.2
0.7
0
Figure 11: An example of the Algorithm I vectorized implementation. means the equation number.
The number inside the parentheses
First, we calculate the cumulative sum the distance between one and the total input weight as- signed to each output frame, given by
T P = cumsum(1, - Σ Wo
(9)
where 1, denotes the all-one column vector with shape t. For each row in Wo. only the first non- negative element and the first zero after the last non-negative element need update (see Algorithm I). And we locate those elements by performing the following convolution: convolving (no padding) M with kernel K of shape x 2, given by
1
Q KM, K
M sgn WW),
(10)
tx2
where sgn(.) stands for sign function. We pad column of zero on the first row of the output Q.
The output Q E {0, 1, -1 a matrix with shape t x T. Then we calculate the value that needs updates with P and Q, given by
Or P Q
U=PO(-Q)+-
(11)
where P (Pu))eettta- denotes slicing of the matrix row, Or is the all-zero row vector with length T, and & denotes element- multiplication. Note that here the start index of matrix one. U and V store the values that need addition and subtraction for each element in Wo. The
final warp matrix calculated by
W = - V W..
(12)
21

---

Under review as a conference paper at ICLR 2023
A.8 EXAMPLES
Fixtlltpora-resoluttin spectrogram (100 FPS)
shei
left
forward
Frame importance
Warp Matrix
30
Maaptive-ermorrakkesolddbi spectrogram (25 FPS)
Resolution encodling
Figure 12: Examples of DiffRes adapiiv-temooonn-eessluuin spectrogram the SpecchCommands dataset.
Fixstemporal-resaluutonn mel-spectrogram (100 FPS)
animal, squcaak
laug coUg
speech
Frame importance estimation
100
300
330
Warp Matrix
1200
Adaptive temporalrrssooiitiionn spectrogram (25 FPS)
Resolution encoding
Figure 13: Examples DiffRes addapiiie-emmoonlcccsuttiii spectrogram on AudioSet dataset.
22


Reviews:
[{"review_id": "vHk41eaauM", "reviewer": "Reviewer_wxoQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a drop-in differentiable module (DiffRes) that automatically adjusts temporal resolutions of spectrogram input for audio classification.  The DiffRes module computes frame-level importance score and calculate a warp matrix to dynamically scale temporal resolution. The guide loss encourages empty frames to have a low importance score. The DiffRes module was validated with various audio classification tasks including audio tagging, speech command recognition, and musical instrument classification. The experimental results show the effectiveness of the proposed module. In addition, it was compared to other learnable front-end models, showing superior results. The empirical analysis validates the efficiency in computational cost and the effectiveness of the resolution encoding and guide loss. \n\n  ", "summary_of_the_review": "This paper is a great contribution to audio classification tasks and has a potential to be used for any high-dimensional time series data such as video as the authors suggested in the conclusion. \n\nThe proposed DiffRes module is neat and handy. It improves not only the classification accuracy but also reduces the computational cost.  \n\nMy only concern is whether DiffRes works well only when the audio signals have sufficient empty frames (e.g. environmental sounds) or it would be also effective when the audio signals have consistent energy (e.g. music tracks). I also wonder if DiffRes would be useful for automatic speech recognition. \n", "strength_and_weaknesses": "Strengths\n- The propose module was well motivated in the introduction section with appropriate references\n- The technical description is delivered well with the illustrated figure of the computational procedure, visual examples, and detailed algorithm formulation in the appendix section. \n- The experiment is comprehensive and rigorous, covering various audio classification tasks and experimental settings  \n- The ablation study is very convincing and explains the effectiveness of the proposed ideas  \n- The visual animation in the demo link is very impressive  \n\nWeaknesses\n- There are some unclear part in writing (See below) \n- The performance increment is notable for environmental sounds, whereas it is marginal in speech command and music instrument. This may indicate that DiffRes is more effective when the audio examples has a lot of silent parts. This result could limit the use of DiffRes.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "vHYEbfUTY_l", "reviewer": "Reviewer_qgYk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a \"drop-in\" module and a loss function to compress the information codified in spectrograms for Audio classification tasks. The proposed module is based on standard 1D convolution layers, batch normalization, and residual connections. It can be used as an intermediate module between the input (Spectrogram) and a particular architecture for audio classification. As the authors highlighted, It could also be used with other speech representations either following a feature learning or a feature engineering fashion. The output of the module is a vector containing some frame importance indexes (s_i). Since the proposed module must act as a bottleneck to compress the original information in fewer frames, the proposed loss function promotes the scenarios where the \"mass of importance\" is distributed in fewer s_i.\n\n", "summary_of_the_review": "The paper proposes a module for spectrogram compression in audio classification tasks. Even though the authors assert that the module learns the temporal resolution, it is not clear how the proposed approach can do that. The proposed formulation lets us see that the module compresses the information of a given Spectrogram, but the temporal resolution is defined during the Spectrogram estimation, so the proposed module cannot do anything about it. Either way, the idea of applying a compression module to Spectrograms is interesting since the information contained in spectrograms is mostly sparse so data compression can lead to a reduction in the computational load of audio classification systems. The estimation of a vector of importance indices is not new since it is the base of the self-attention mechanism. Still, the way such a vector is used to build the matrix W and the corresponding bottleneck representation O is interesting as it follows a less data-driven (more heuristic) but coherent approach. A significant limitation of the proposed approach is the inclusion of many hyper-parameters that could significantly affect the performance of the proposed module.", "strength_and_weaknesses": "The paper is interesting and addresses a relevant problem since the information contained in spectrograms is mostly sparse, so data compression can lead to a reduction in the computational load of audio classification systems, such as the authors showed. The authors performed a large enough battery of experiments to show the effects of their proposal on different audio classification tasks and regarding also different hyper-parameters involved in the process.\n\nThe main drawback of the proposed approach is the introduction of several free parameters (hyper-parameters) that make the proper training of the module a problematic task. In this sense, the proposed method has a significant limitation regarding the definition of the dimension reduction rate. Such a parameter is similar to any feature reduction rate that must be adjusted following a validation methodology, which goes against the authors' claim that the proposed approach is an end-to-end approach. Following this idea, I also find the title of the paper problematic; the authors claim that the proposed method \"learns the temporal resolution,\" but it is not changing the temporal resolution of the spectrogram; it is applying a compression algorithm to identify the frames with less or not none information at all, in order to reduce the dimensions of the audio representation processed for further modules. The experiments and results presented in section 3 show that temporal resolution must be changed manually, so there is no any gain in this respect by using the proposed module. \n\nOn the other hand, the proposed loss function to adjust the s coefficients is quite arbitrary as it is the selection of its parameters. The authors should elaborate more about why it is a good choice for the frame compression task. They should also present results and discussion about the sensitivity of the model to delta, epsilon, and lambda parameters. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EkMk-bnG7M", "reviewer": "Reviewer_Uoii", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes approaches to identify the optimal frame rate for audio classification tasks. Specifically, they propose a series of strategies to identify the optimal rate by inserting the frame-rate identification module between the traditional feature extraction step and the final classifier and casting the problem as a joint optimization problem. Experiments on several classification tasks show that the proposed approach improves over recent approaches. ", "summary_of_the_review": "The paper proposes an end-to-end framework to optimize the time resolution and classification accuracy in audio classification tasks. For ICLR, the generality of the approach is not quite established. There are a lot of good engineering tactics though that would be of interest to the audio classification community. I recommend instead audio/speech-based venues such as ICASSP, Interspeech, and MLSP.", "strength_and_weaknesses": "Strengths\n1. A novel approach to identify the frame rate that is optimal for different audio classification tasks. \n2. Code is available online.\n3. Experimental details are parameters are clearly stated. \n4. Results on several tasks show that the proposed approach results in higher classification accuracy compared to other recent models.\n\nGrowth Opportunities\n1. The paper makes sweeping novelty claims and misses key references. e.g. Kekre, H. B. et al. “Speaker Identification using Spectrograms of Varying Frame Sizes.” International Journal of Computer Applications 50 (2012): 27-33. and Huzaifah, Muhammad. \"Comparison of time-frequency representations for environmental sound classification using convolutional neural networks.\" arXiv preprint arXiv:1706.07156 (2017).\n2. Section 2.2.1 indicates a frame-weighting approach. This is similar to the sample weighting approaches common in ML tasks. Temporal frame warping in 2.2.2 is similar to the use of derived features. These similarities are worth pointing to the reader.\n4. The importance of guide loss is understated. It is only in Figure 8 that the large contribution becomes clear. Incorporating that in the abstract, introduction, and conclusion prominently will result in a more faithful representation of the novelty.\n5. Ablation study on other neural and event non-neural classification architecture will also be helpful to understand the generalizartion ability of the proposed feature-optimization approach.\n6. A lot of experimental tactics, e.g., section 3.1, seem like good engineering recipes. Probably this work is a better fit for the ICASSP or Interspeech community.\n7. Legend in table 3 is missing. The difference between the last 2 columns is unclear. \n8. Why is the computational cost of LEAF so much higher? An explanation will be helpful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject