LEAF: A Learnable Frontend for Audio Classification
Authors: Neil Zeghidour
Olivier Teboul
Félix de Chaumont Quitry
Marco Tagliasacchi
Venue: ICLR 2021 Poster (2021)
Keywords: audio understanding
frontend
learnable
mel-filterbanks
time-frequency representations
sound classification

Abstract: Mel-filterbanks are fixed, engineered audio features which emulate human perception and have been used through the history of audio understanding up to today. However, their undeniable qualities are counterbalanced by the fundamental limitations of handmade representations. In this work we show that we can train a single learnable frontend that outperforms mel-filterbanks on a wide range of audio signals, including speech, music, audio events and animal sounds, providing a general-purpose learned frontend for audio classification. To do so, we introduce a new principled, lightweight, fully learnable architecture that can be used as a drop-in replacement of mel-filterbanks. Our system learns all operations of audio features extraction, from filtering to pooling, compression and normalization, and can be integrated into any neural network at a negligible parameter cost. We perform multi-task training on eight diverse audio classification tasks, and show consistent improvements of our model over mel-filterbanks and previous learnable alternatives. Moreover, our system outperforms the current state-of-the-art learnable frontend on Audioset, with orders of magnitude fewer parameters.

Published as a conference paper at ICLR 2021
LEAF: A LEARNABLE FRONTEND FOR AUDIO CLASSIFICATION
Neil Zeghidour, Olivier Teboul, Félix de Chaumont Quitry & Marco Tagliasacchi Google Research
(neilz, oliviert, feq, mtagl fasacchi)@googlllee...
ABSTRACT
Mel-filterbakks are fixed, engineered audio features which emulate human percep- tion and have been used through the history of audio understanding up to today. However, their undeniable qualities are counterbalanced by the fundamental lim- itations of handmade representations. In this work we show that we can train a single learnable frontend that outperforms mel-filterbanks on wide range of au- dio signals, including speech, music, audio events and animal sounds, providing a reneral-purposee learned frontend for audio classification. To do so, we introduce a new principled, lightweight, fully learnable architecture that can be used as a drop-in replacement of mel-filterbanks. Our system learns all operations of dio features extraction, from filtering to pooling, compression and normalization, and can be integrated into any neural network negligible parameter cost. We perform multi-task training on eight diverse audio classification tasks, and show consistent improvements of our model over mel-filterbanks and previous learn- able alternatives. Moreover, our system outperforms the current state-of-the-aad learnable frontend on Audioset, with orders of magnitude fewer parameters.
INTRODUCTION
Learning representations by backpropagation in deep neural networks has become the standard in audio understanding, ranging from automatic speech recognition (ASR) (Hinton al., 2012: Se- nior et al., 2015) to music information retrieval (Arcas et al., 2017), as well as animal vocaliza- tions (Lostanlen et al., 2018) and audio events (Hershey et al., 2017; Kong et al., 2019). Still, a strik- ing constant along the history of audio classification is the mel-filterbanks, fixed, hand-engineered representation of sound. Mel-filterbanks first compute spectrogram, using the squared modulus of the short-term Fourier transform (STFT). Then, the spectrogram is passed through bank of triangu- lar bandpass filters, spaced on a logarithmic scale (the mel-scale) to replicate the non-linear human perception of pitch (Stevens & Volkmann, 1940). Eventually, the resulting coefficients are passed through a logarithm compression, to replicate our non-linear sensitivity to loudness (Fechner et al., 1966). This approach of drawing inspiration from the human auditory system design features for machine learning has been historically successful (Davis & Mermelstein, 1980; Mogran et al., 2004). Moreover, decades after the design of sileriiteraanss, Andén & Mallat (2014) showed that they coincidentally exhibit desirable mathematical properties for representation learning, in particu- lar shift-invariance and stability to small deformations. Hence, both from an auditory and a machine learning perspective, mel-filterbanks represent strong audio features.
However, the design of mill-riitssaanks also flawed by biases. First, not only has the mel-scale been revised multiple times (O Shaughnessy, 1987: Umesh et al., 1999), but also the auditory experiments that led their original design could not be replicated afterwards (Greenwood, 1997). Similarly, better alternatives to cog-comprssioonn have been proposed, like cubic root for speech enhancement (Lyons & Paliwal, 2008) or 10th root for ASR (Schluter et al., 2007). Moreover, even though matching human perception provides good inductive biases for some application domains, e.g., ASR music understanding, these biases may also be detrimental, e.g. for tasks that require fine-graines resolu- tion at high frequencies. Finally, the recent history of other fields like computer vision. in which the rise of deep learning methods has allowed learning representations from raw pixels rather than from engineered features (Krizhevsky et al., 2012), inspired to take the same path.

---

Published as a conference paper at ICLR 2021
MEL-FILTERBANKS
1-12
MEL A
TD-FILTERBANKS
CONVID
|-|
LOWPASS
LOG
SINCNET
SINC Sap
LEAF
GABOR
GAUSSIAN LOWPASS
SPCEN
Figure 1: Breakdown of the computation of mel-filterbanks, Time-Domain filterbanks, SincNet, and the proposed LEAF frontend. Orange boxes are fixed, while computations in blue boxes leamable. Grey boxes represent activation functions.
These observations motivated replacing nnelfiilerbanks with learnable neural layers, ranging from standard convolutional layers (Palaz et al., 2015) to dilated convolutions (Schneider et al., 2019), as well as structured filters exploiting the characteristics of known filter families, such as Gamma- tone (Sainath et al., 2015), Gabor (Zeghidour et al., 2018a: Sa; Noé et al., 2020), Sinc (Ravanelli & Bengio, 2018; Pariente al., 2020) or Spline (Balestriero et al., 2018) filters. While tasks such as speech separation have already successfully adopted learnable frontends (Luo & Mesgarani, 2019; Luo et al., 2019), observe that most state-of-the art approaches for audio classification (Kong et al., 2019), ASR (Synnaeve et al., 2019) and speaker recognition (Villalba et al., 2020) still em- ploy neelfilterbanns input features, regardless of the backbone architecture.
In this work, we argue that credible alternative to mel-filterband for classification should be evaluated across many tasks, and propose the first extensive study of learnable frontends for audio over a wide and diverse range of audio signals, including speech, music, audio events, and ani- mal sounds. By breaking down nel-filterbans into three components (filtering, pooling, compres- sion/normalization). we propose LEAF, novel frontend that is fully learnable in all its operations, while being controlled by just a few hundred parameters.
a multi-task setting over datasets, show that can learn single set of parameters that outperforms mel- hel-filterbanks, previously proposed learnable alternatives. Morcover, these findings are replicated when training a different model for each individual task. We also confirm these results challenging, large-scale benchmark: classification on Audioset (Gemmeke et al., 2017). In addition, we show that the general inductive bias of our frontend (i.e., learning bandpass filters, lowpass filtering before downsampling, learning per-channel compression) is general enough to benefit other systems, and propose a new, improved version of SincNet (Ravanelli & Bengio, 2018). Our code is publicly available'.
well as
RELATED WORK
In the last decade, several works addressed the problem of learning the audio frontend, as an alter- native mel-filterbanks. The first notable contributions in this field emerged for ASR, with Jaitly & Hinton (2011) pretraining Restricted Boltzmann Machines from the waveform, and Palaz et al. (2013) training hybrid DNN-HMM model, replacing mel-tilterbanks by several layers of convo- lution. However, these alternatives, well others proposed more recently (Tjandra et al., 2017; Schneider al., 2019), are composed of many layers, which makes fair comparison with mel- filterbanks difficult. In the following section, we focus on frontends that provide a lightweight, drop-in replacement to mel-filtrrbanks, with comparable capacity.
2.1 LEARNING FILTERS FROM WAVEFORMS
first attempt at learning the filters mel-filterbanks was proposed by Sainath al. (2013), where filterbank is initialized using the mel-scale and then learned together with the rest of the net- work, taking a spectrogram as input. Instead, Sainath et al. (2015) and Hoshen et al. (2015) later
httpp::// thub. www./oo///o....o..oooooooooooo.. 1 e resea rcch audio
2

---

Published as a conference paper at ICLR 2021
proposed to learn convolutional filters directly from waveforms, initialized with Gammatone fil- ters (Schluter et al., 2007). In the same spirit, Zeghidour et al. (2018a) used the scattering transform approximation of mel-filterbanks (Andén & Mallat, 2014) to propose the time-domain filterbanks, learnable frontend that approximates mel-filterbanks at initialization and can then be learned without constraints (see Figure 1). More recently, the SineNet (Ravanelli & Bengio, 2018) model was pro- posed, which computes a convolution with sine cardinal filters, non-linearity and a max-pooling operator (see Figure 1), as well variant using Gabor filters (Noé et al., 2020).
We take inspiration from these works to design new learnable filtering layer. As detailed in Sec- tion 3.1.2, we parametrize complex-valueds filtering layer with Gabor filters. Gabor filters are opti- mally localized in time and frequency (Gabor, 1946), unlike Sinc filters that require using window function (Ravanelli Bengio, 2018). Moreover, unlike Noé et (2020), who use complex-valus layers in the rest of the network, we describe in Section 3.1.2 how using a squared modulus not only brings back the signal to the real-valued domain (leading to compatibility with standard architec- tures), but also performs shift-invariant Hilbert envelope extraction. Zeghidour et al. (2018a) also apply a squared modulus non- -emeliierrit,, however as described Section 3.1.1, II, training unconstrained filters can lead to overfitting and stability issues, which we solve with our approach.
2.2 LEARNING THE COMPRESSION AND THE NORMALIZATION
The problem of learning a compression and/or normalization function has received less attention in the past literature. A notable contribution is the Per-Channel Energy Normalization (PCEN) (Wang et al., 2017; Lostanlen et al., 2019), which was originally proposed for keyword spotting, outper- forming log-comppnn comprrsssoo.. Later, Battenberg al. (2017) and Lostanlen et al. (2018) confirmed the advantages of PCEN, respectively for large scale ASR and animal bioacoustics. However, these pre- vious works learn a compression on top of fixed mel-filterbanks. Instead, this work we propose a new version of PCEN and show for the first time that combining learnable filters, learnable pooling, and learnable compression and normalization outperforms all other approaches.
MODEL
Let 2 R.T denote one-dimensional waveform of T samples, available at the sampling frequency F, [Hz]. We decompose the frontend into a sequence of three components: i) filtering, which passes x through bank of bandpass filters followed by non-linearity, operating at the original sampling rate F,; ii) pooling, which decimates the signal to reduce its temporal resolution; iii) compression/normalizttooo which applies non- non-linearity reduce the dynamic range. Overall, the frontend can be represented as a function F₂ I - RMN,, which maps the input waveform to 2-dimensional feature space, where M denotes the number of temporal frames (typically M < T), N the number of feature channels (which might correspond to frequency bins) and ei the frontend parameters. The features computed by the frontend are then fed to a model ge(-) parametrized by B. The frontend and the model parameters are estimated by solving supervised classification problem:
0*, arg min E(Z,DED C(ge(F(()))
0.
where (xx y) are samples in a labelled dataset D and L is loss function. Our goal is to learn the frontend parameters v end-to-end with the model parameters 0. To achieve this, is necessary to make all the frontend components learnable, that can solve the optimization problem in equa- tion 1 with gradient descent. In the following we detail the design choices of each component.
3.1 FILTERING
The first block of the learnable frontend takes x as input, and computes convolution with bank of complex-valued filters (fpa)==1.NN followed by a squared modulus operator, which brings back its output to the real-valued domain. This convolution step has a stride of 1, therefore keeping the input temporal resolution, and outputs the following time-freyuency representation:
fn |xr sp 1 RT. n =1,.... II...,N. N,
(2)
where P.. E CW is complex-valued I-D filter of length W. It is possible to compute equation without explicitly manipulating complex numbers. As proposed by Zeghidour et al. (2018a), to
3

---

Published as a conference paper at ICLR 2021
NormalleedGamviD
GebertonnVD
-0.4
-52
0.
5.2
0.1
Figure 2:
Frequency response of filters at convergence for two parametrizations: normalized ID filters and Gabor filters, both initialized mel scale. We highlight two filters among the 40 of the filterbank: one in yellow in the low frequency range and one in pink at high frequencies.
produce the squared- squared-modulus of a complex -eaplex--alued convolution with N filters, we compute instead the convolution with 2.V real-valued filters pn. 1...., 2N, and perform squared fz-pooling with size and stride along the channel axis obtain the squared modulus, using adjacent filters as real and imaginary part of (pn. Formally:
fn r * 1 1 +
62nl 1 R
N.
(3)
We explore two different parametrizations for Pr. One relies on standard fully parametrized convo- lutional filters while the other makes use of learnable Gabor filters.
3.1.1 NORMALIZED ID-CONVOLUTION
Inspired by Zeghidour et (2018a), the first version of our filtering component is a standard ID convolution, initialized with a bank of Gabor filters, which approximates the computation of a mel- filterbank. Thus, initialization, the output of the frontend is identical to that of mel-filterbank, but during training the filters can be learned by backpropagation. In our early experiments, we observed several limitations of this approach. First, due to the unconstrained optimization, these filters not only learn to select frequency bands of interest, but also learn a scaling factor. This might lead to instability during training, because the same form of scaling can be also computed by the filter-wist wise compression component, which is applied at later stage in the frontend. We address this issue by applying ez-oommalization to the filter coefficients before computing the convolution. Second, the unconstrained parametrization increases the number of degrees of freedom, making training prone to overfitting. The first panel in Figure shows the frequency response of a bank of = 40 filters, each parametrized with W 401 coefficients. We observe that, at convergence, they widely spread across the frequency axis, and include negative frequencies. Moreover, the filters are very spiky rather than smooth. To alleviate these issues, the next section we introduce a parametrization based on a bank of Gabor filters, which reduces the number of parameters to learn, while at the same time enforeing during training a stable and interpretable representation.
3.1.2 GABOR ID-CONVOLUTION
Gabor filters are produced by modulating a Gaussian kernel with a sinusoidal signal. These filters provide several desirable properties. First, they have optimal trade-off between localization in time and frequency (Gabor, 1946), which makes them a suitable choice for a convolutional network with finite-sized filters. This is in contrast Sine filters (Ravanelli Bengio, 2018), which require using window function to smooth abrupt variations on each side of the filter. Second, the time and frequency response of Gabor filters have the same functional form, thus leading to interpretable bandpass filters, unlike the unconstrained filters described in the previous section. Finally, Gabor filters are quasi-analytic (i.e., their frequency response is almost zero for negative frequencies) and when combined with the squared modulus the resulting filterbank can be interpreted as a set of subband Hilbert envelopes, which are invariant to small shifts. Due to this desirable property, they have been previously used as (fixed) features for speech and speaker recognition (Falk & Chan, 2009; Thomas et al., 2008). Formally, Gabor filters are parametrized by their center frequencies (n 1 and inverse bandwidths (on)n=1..N as follows:

---

Published as a conference paper at ICLR 2021
1
22 20
(nl()) e 2 V2TTn
n- N, 1= -= W/2.
(4)
The frequency response fnn a Gaussian centered at frequency 7nn and of bandwidth 1/ 1/ both expressed in normalized frequency units in [-1/2, +1/2]. Therefore, learning these parameters al- lows learning a bank of smooth, quasi-analytic bandpass filters, with controllable center frequency and bandwidth. In practice, compute the output of the filtering component, pulse response of the Gabor filters Pn. over the range = - -W/2.... W/2. and convolve these impulse responses with the input waveform. To ensure stability during training, we clip the center frequencies (1...= to be in [0, 1/2]. so that they lie in the positive part of the frequency range. We also constrain the bandwidths (o.)n= in the range [4v2 log 2, 2W /2 2), such that the full-width at half- milimmm of the frequency response is within 1/W and 1/2.
obtain the im-
Gabor filters have significantly fewer parameters than the normalized ID-convolutions described in Section 1. N filters of length W are described by 2N parameters, N for the center frequencies and N for the bandwidths, against W .V for a standard ID-convolution. a window length of 25 ms, operating at sampling rate of 16 kHz, then W = 401 samples, and Gaabo-bbasd filtering accounts for 200 times fewer parameters than their unconstrained alternatives.
particular, when using
3.1.3 TIME-FREQUENCY ANALYSIS AND LEARNABLE FILTERS
A spectrogram, linear or mel scale, provides an ordered time-frequency representation: adjacent frames represent consecutive time-steps, while frequencies monotonically increase along the feature axis. A learnable frontend that performs filtering by means of convolution with a set of bandpass filters also preserves ordering along the temporal axis. However, the ordering along the feature axis is unconstrained. This can be problematic when applying subsequent operations that rely on frequency ordering. These include, for example: i) operations that leverage local frequency infor- mation, e.g.. two-dimensional convolutions, which compute a feature representation based on local üime-frequency patches; ii) operations that leverage long-range dependencies along the frequency axis, e.g., to capture the harmonic structure of the underlying signal; and iii) augmentation methods that mask adjacent frequency bands, like SpecAugment (Park al., 2019). To evaluate the impact of enforcing an explicit ordering of the center frequencies of the learned filters, we compared the result of training a frontend using Gabor filters with or without explicitly enforcing ordering of the center frequencies. Interestingly, we observe that even without explicit constraint, filters that are ordered at the initialization tend to keep the same ordering throughout training, and that enforcing sorted filters has no effect on the performance.
3.2 LEARNABLE LOWPASS POOLING
The output of the filtering component has the same temporal resolution as the input waveform. The second step of a Icarnable frontend is to downsample the output of the filterbank to a lower sam- pling rate, similarly to what happens in the STFT when computing mel-filterbanks Previous work relied on max-pooling (Sainath et al., 2015; Ravanelli & Bengio, 2018; Noé et al., 2020), lowpass- filtering (Zeghidour al., 20) average pooling (Balestriero al., 2018). Zeghidour et al. (2018b) compare these methods on a speech recognition task and show a systematic improvement when using lowpass filtering instead of max-pooling. More recently, Zhang (2019) showed that in standard 2D convolutional architectures, including ResNet (He et al., 2016) and DenseNet (Huang al., 2017), drop-in replacement of max- max-pooling and average pooling layers with (fixed) low- pass filter improves the performance for image classification. In the proposed frontend, we extend the pooling layer of Zeghidour al. (2018a) in two ways. First, while Zeghidour et al. (2018a) adopt a single shared lowpass filter for all input channels, we implement lowpass filtering by means of depthwise convolution, such that each input channel is associated with one lowpass filter. This is useful because each channel in the learnable frontend is characterized by different bandwidth, and specific lowpass filter can learned for each of them. Second, we parametrize these lowpass filters to have Gaussian impulse response:
(t) V2TOn
20
t= -W/ W/2.
(5)
5

---

Published as a conference paper at ICLR 2021
Note that this is particular case of Gabor filters with center frequency equal to 0 and learnable bandwidth. With this choice, we can learn per-channel lowpass pooling functions while adding only N parameters to the frontend model. We initialize all channels with bandwidth of 0.4, which gives a frequency response close to the Hann window used by mel-filterhanks.
3.3 LEARNING PER-CHANNEL COMPRESSION AND NORMALIZATION
When using a traditional frontend based on mel- filterbank or STFT, the freeffruyency features are typically passed through logarithmic compression, to replicate the non-linear human perception of loudness (Fechner et al., 1966). The first limitation of this approach is that the same compression function is applied to all frequency bins, regardless of their content. The second limitation is the fixed choice of the non-linearity used by the compression function. While, in most cases, a loga- rithmic function is used, other compression functions have been proposed and evaluated in the past, including the cubic root (Lyons & Paliwal, 2008) and the 10th root (Schluter et al., 2007). This mo- tivates learning the compression function as part of the model, rather than relying handerafted choice. In particular, Per-Channel Energy Normalization (Wang et al., 2017) was proposed as a learnable alternative log-compression and mean-variance normalization:
(CEN(F(Lm) - (1e F(t, n)
6,p
(6)
(Mt, n²)
where t 1,...,, .M denotes the time-step and = =1 I...... N the channel index. In this parametriza- tion, the freeeerrruuncyy representation F is first normalized by an exponential moving average of its past values MM(t, n) = (1 exponent On, with e being a small constant used to avoid dividing by zero. An offset 6, is then added before applying compression with the exponent v. (typically in [0, 1)) Wang et al. (2017) train On 5n, and Tn. while treating s as hyperparameter, or learning a convex combination of exponential moving averages for different, fixed values of s. Instead, in this work we learn -hannel-dependent smoothing coefficients s,, jointly with the rest of the parameters. We call this version SPCEN. This approach was previously used by Schlüter & Lehner (2018) for singing voice detection, except that they did not learn the exponents On,, while we learn all these parameters jointly. Our final frontend cascades a Gabor ID-convolution, a Gaussian lowpass pooling, and sPCEN. In the rest of the paper, we refer to this model as LEAF, for "LEarnable Audio Frontend".
nt), controlled by smoothing coefficient and an
4 EXPERIMENTS
We evaluate our frontend on three supervised learning problems: i) single-task classification; ii) multi-task classification and iii) multi-label classification on Audioset. As baselines, we compare system to log- cogpeesede mel-filterbanks, Iearnable Time-Domain filterbanks (Zeghidour et al., 20)8a)² and SincNet (Ravanelli & Bengio, 2018)². In all our experiments, we keep the same com- mon backbone network architecture, which consists of a frontend, convolutional encoder, and one (or more) head(s). We adopt the lightweight version of EfficientNet (Tan & Le, 2019) (Efficient- NetBO, with 4 M parameters) as convolutional encoder. On AudioSet, we also experiment with state-of-the-art CNNI4 encoder (Kong et al., 2019), with 81 M parameters. A head is a single linear layer with number of outputs equal the number of classes. In the multi-task setting we use different head for each of the target tasks, all sharing the same encoder.
The input signal sampled F. 16 kHz is passed through the frontend which feeds into the convo- lutional encoder, As baseline, we use a log-compressed mel-filterbank with 40 channels, computed over windows of 25 ms with a stride of 10 ms. For fair comparison, both LEAF and the learnable baselines also have N = 40 filters, each with W 401 coefficients (=:25 ms at 16 kHz). The learn- able pooling is computed over 401 samples with a stride of 160 samples (10ms iss at 16 kHz), giving the same output dimension as mel-filterbanks. On AudioSet we use 64 channels instead of 40 as Kong et al. (2019) observed improvements from using 64 mel-filters.
To address the variable length of the input sequences, we train on randomly sampled I second win- dows. We train with ADAM (Kingma & Ba, 2014) and a learning rate of 10-4 -4 for M batches, with
/tttps://gi http:/thwc.cooocccccooeeeeeee htt.reeeaearcctt//aasss.. 'https://g http../..//maaaaaaaaeeeeee 1/81 niNeet.

---

Published as a conference paper at ICLR 2021
Table I: Test accuracy (%) for single-task classification.
Task
Mel
TD-fbanks
SineNet
LEAF
Acoustic seenes
99.2 = 0.4
99.5 0.3
96.7 = 0.9
99.1 0.5
Birdsong detection
78.6 = 1.0
0.9
78.0 1.0
81.4 0.9
Emotion recognition
49.1 2.1
57.1
44.2 2.4
57.8
Speaker ld. (VC)
31.9
0.7
33.1 4.8
Music (instrument)
72.0
70.0
70.3 0.6
72.0
Music (pitch)
0.3
9.3
0.5
0.3
Speech commands
92.4 0.4
87.3=0.4
8.4
93.4 0.3
Language Id.
76.5 = 0.4
71.6
78.9
86.0 0.4
Average
73.9 = 0.8
72.9 + 0.8
73.1 + 0.9
76.9 + 0.8
Table 2: Impact of the compression layer the performance (single task, average accuracy in %).
mel-filtrbannk
LEAF- filterhank
log
PCEN
sPCEN
log
PCEN
SPCEN
Average 73.9 + 0.8
76.4 + 0.8 76.0 + 0.7 74.6 0.7
76.4 + 0.8 76.9 1 0.8
batch size 256. For Audioset experiments, we train with mixup (Zhang et al., 2017) and SpecAug- ment (Park et al., 2019). During evaluation, consider the full-length sequences, splitting them into consecutive non-overlapping I second windows and averaging the output logits over windows.
4.1 SINGLE-TASK AUDIO CLASSIFICATION
We train independent single-task supervised models on 8 distinct classification problems: acoustic scene classification on TUT (Heittola et al., 2018) birdsong detection (Stowell et al., 2018), emotion recognition on Crema-D (Cao et al., 2014), speaker identification on VoxCeleb (Nagrani et al., 2017), musical instrument and pitch detection on NSynth (Engel et al., 2017), keyword spotting on Speech Commands (Warden, 2018), and language identification on VoxForge (Revay & Teschke, 2019). A summary of the datasets used in experiments is illustrated in Table A.I.
Table reports the results for each task, with 95% confidence intervals representing the uncertainty due the limited test sample size. No class rebalancing is applied, neither during training, during testing. On average, we observe that LEAF outperforms all alternatives. When considering results for each individual tasks, observe that LEAF outperforms or matches the accuracy of other frontends, with the notable exception of SincNet on Voxceleb. This is consistent with the fact that SineNet was originally proposed for speaker identification (Ravanelli & Bengio, 2018) and illustrates the importance of evaluating over a wide range of tasks. To evaluate the robustness of our results with respect to the choice of the datasets, we apply statistical bootstrap (Efron & Tibshirani, 1993) to compute the non-parametric distribution of the difference between the accuracy obtained with LEAF and each of the other frontends, when sampling datasets with replacement among the eight datasets. We test the null hypothesis that the mean of the difference is zero and measure the following one-sided p-values: PMel < 10-5, PTD-fhanks < 10-5, PSinaNer = 0.059. Figure A.1 illustrates the corresponding bootstrap distribution, showing the statistical significance of our results with respect to the choice of the datasets. In Table 2, we study the effect of the compression function. PCEN improves significantly over com-compression on both mel- and learned filterbanks. We also observe that for any choice of the compression function, the learnable frontend matches outperforms the corresponding eelffilterban.
4.2 MULTI-TASK AUDIO CLASSIFICATION
To learn a general-purpose frontend that generalizes across tasks, we train multi-task model that uses the same LEAF parameters and encoder for all tasks, with task ask-sppecfiic linear head. More specifically, single network with K heads (he,...., hax) is trained on mini-batches, uniformly sampled from the K datasets. A mini-batch of size B can be represented (x ,,, )==..B where k; is the associated task the example has been sampled from. The multi-task loss function is now computed mini-batch as the sum of the individual loss functions:
7

---

Published as a conference paper at ICLR 2021
Table 3: Test accuracy (%) for multi-task classification.
Task
Mel
TD-fbanks
SincNet
LEAF
Acoustic scenes
99.1 + 0.5
9.6
91.0 1.4
98.9 0.5
Birdsong detection
81.3 = 0.9
82.3 21.4
78.8 - 0.9
0.9
81.9 0.9
Emotion recognition
21.1 + 2.1
2:1
2.3
Speaker Id. (LBS)
100.0 0.0
100
100.0 0.0
100.0 0.0
Music (instrument)
70.7 0.6
66.3
0.6
70.2 0.6
Music (pitch)
0.4
8.4
0.5
18.6
Speech commands
08.6
8..5
91.4
93.6 0.3
Language ld.
64.9 0.5
58.9
60.8 10.8000 0.5
69.6 0.5
Average
77.8 + 0.7
75.8 0.7
74.6 +: 0.8
79.3 1 0.7
Table 4: Test AUC and d-prime (# standard deviation over three runs) on Audioset, with the number of learnable parameters per frontend.
EtficientNetBO
CNN14 (ours)
CNN14 (Kong et al., 2019)
Frontend
#Params
AUC
d d-prime
AUC
d-prime
AUC
d-prime
Mel
0.968 + .001
2.61 + .02
0.972 + .000
2.71 + .01
0.973
2.73
Mcl-PCEN
256
0.967 .000
2.60 001
0.973 .000
2.72 + .00
Wavegram
300
0.958 + .000
2.44
+.4
0.961 .001
2.50
+ .02
0.968
2.61
TD-Ibanks
51
0.965
.001
2.57
167
0.972
.000
2.70
SincNet
256
0.961 + .000
2.48
.00
.00
0.970
.000
2.66 1 .01
SincNct+
448
0.966 + .002
2.58
+ .04
0.973
.001 .001
2.71
LEAF
448
0.968
+ 0001
2.63 .01
000 0.974
2.74
C
Σ
Σ Lk (ho (go(Fo(att)) .vt 8(kss k),
(
k=
where 8 and vb represent the shared parameters of the encoder and frontend respectively, 0g.
I,...,A K the task-specific parameters of the heads hag (-), and 5 is the Kronecker delta function.
We use the same set of tasks described in Section 4.1 with one exception, replacing VoxCeleb with Librispeech for speaker identification. As illustrated in Table A. VoxCeleb has much more classes (s/1200) than any other task. This creates an imbalance in the number of parameters in the heads, making training significantly slower and reducing the accuracy all other tasks, regardless of the frontend. Table reports the accuracy obtained on each task, for all the frontends. LEAF shows the best overall performance while matching tasks. We repeated the bootstrap analysis (Figure A.2) to evaluate the statistical significance of these results with respect the choice of the datasets and observed the following p-values: PMel 0.048, PTD-fbanks < 10-5, PSincNer = 10-5, To the best of our knowledge, it is the first time a learnable frontend is shown outperform filteebankks over several tasks with a unique parametrization.
outperforming all other methods every
4.3 MULTI-LABEL CLASSIFICATION ON AUDIOSET
Table 4 shows the performance (averaged over three runs) of the different frontends on Au- dioset (Gemmeke et al., 2017), a large-scale multi-label dataset of sounds of all categories, described by an ontology of 527 classes. Audioset examples are weakly labelled with one or more positive labels, so we evaluate with the standard metric of this dataset, the balanced d-prime, non-linear transformation the AUC averaged uniformly across all classes. EffficientNetB0 trained with LEAF achieves a d-prime of 2.63, outperforming the level achieved when using mel-file aalfilterbanks (d-prime: 2.61) or other frontends. We compare this result with the state-of-the-an PANN model (Kong al., 2019), which reports d-prime of 2.73 when trained on mel-filterbanks. Kong et al. (2019) also train a learnable "Wavegram" frontend made of layers of ID- and 2D- convolutions, for total of 300 k parameters, reporting d prime equal to 2.61. EfficientNerBO on LEAF thus outperforms this system, while having a frontend with 670x fewer parameters, and a considerably smaller encoder. To confirm these results, also train EfficientNetB0 on the Wavegram and get similar findings.
8

---

Published as a conference paper at ICLR 2021
All these findings replicated when replacing EfficientNetBO with our implementation of CNN14: even though the gap between LEAF and baselines reduces as the scores get higher, CNN14 on LEAF matches the current state-of-the-an afrt AudioSet. To conclude these experiments and show that in- dividual components of LEAF can benefit other frontends, we replace the max- max-pooling of SincNet with our Gaussian lowpass filter, and LayerNorm (Ba et al., 2016) with the proposed sPCEN. This version, named "SincNet+" in Table 4, significantly outperforms the original version.
4.4 ANALYSIS OF LEARNED FILTERS, POOLING AND COMPRESSION
Figure A.3 illustrates the center frequencies learned by the Gabor filtering layer of LEAF on Au- dioSet, and compares them to those of mel-filterbanks. At a high level, these filters do not deviate much from their mel-scale initialization. On the one hand, this indicates that the mel-scale is a strong initialization, a result consistent with previous work (Sainath et al., 2015; Zeghidour al., 2018b). On the other hand, there are differences at both ends of the range, with LEAF covering a wider range of frequencies. For example, the lowest frequency filter centered around 60 Hz, as opposed to 100 Hz for mel-filterbanks. We believe that is one of the reasons that explain the out out-performance of LEAF on Audioset, as it focuses on a more appropriate frreuuecy range to represent the underly- ing audio events. Figure A.4 shows the learned Gaussian lowpass filters at convergence. We see that they all deviate towards a larger frequency bandwidth (characterized by a smaller standard deviation in the time domain), and that each filter has different bandwidth, which confirms the utility of using depthwise pooling. Figure A.5 shows the learned exponents rn (initialized at 2.0) and smoothing coefficients S, (initialized 0.04) of SPCEN, ordered by filter index. The exponents are spread in [1.9, 2.6). which shows the importance of learning per-channel exponents, as well as the appropriate choice of using fixed cubic root in previous work (Lyons Paliwal. 2008), as an alternative to log- log-compression. Learning per-channel smoothing coefficients also allows deviating from the ini- tialization. Interestingly, most filters keep a low coefficient (slowly moving average), except for the filter with the highest frequency, which has significantly faster moving average (s=0).15..
4.5 ROBUSTNESS TO NOISE
We compare the robustness of LEAF and mel-fittrbankss to synthetic noise, on the Speech Com- mands dataset. To do so, we artificially add Gaussian noise to the waveforms both during training and evaluation, with different gains to obtain Signal-to-Noise Ratio (SNR) from + inf (no noise) to dB. Figure A.6 shows that while performance (averaged over three runs) degrades for all models as SNR decreases, LEAF is more resilient than nel-filterbanks. In particular, when using a logarithmic compression, the Gabor 1-D convolution and Gaussian pooling of LEAF maintain significantly higher accuracy than mel-filterbanks. Using PCEN has an even higher impact performance when using mel-filterbanks, the best results being obtained with LEAF + PCEN.
5 CONCLUSION
In this work we introduce LEAF, fully learnable frontend for audio classification as an alternative to using handcrafted mel-filterbanks. We demonstrate over a large range of tasks that our model is good drop-in replacement to these features with adjustment to the task at hand, and can even learn a single set of parameters for general-purpos audio classification while outperforming previously proposed learnable frontends. In future work, we will move yet step forward in removing hand- crafted biases from the model. In particular, our model still relies on an underlying convolutional architecture, with fixed filter length and stride. Learning these important parameters directly from data would allow for an easier generalization across tasks with various sampling rates and frequency content. Moreover, we believe that the general principle of learning filter, pool and compress can benefit the analysis of non-audio signals, such as seismic data or physiological recordings.
6 ACKNOWLEDGMENTS
Authors thank Dick Lyon, Vincent Lostanlen, Matt Harvey, and Alex Park for helpful discussions. Authors are also thankful to Julie Thomas for helping with the design of Figure 1. Finally, authors thank the reviewers of ICLR 2021 for their feedback that helped improving the quality of this work.

---

Published as a conference paper at ICLR 2021
REFERENCES
Joakim Andén and Stéphane Mallat. Deep scattering spectrum. IEEE Transactions on Signal Pro- cessing, (46))4114441222 2014.
Blaise Agüera Arcas, Beat Gfeller, Ruiqi Guo, Kevin Kilgour, Sanjiv Kumar, James Lyon, Julian Odell, Marvin Ritter, Dominik Roblek, Matthew Sharifi, and Mihajlo Velimirovié. Now Playing: Continuous low low-power music recognition. Technical report, nov 2017. URL http: arxiv. org/abs 71 1. 10958.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06400 2016.
Randall Balestriero, Romain Cosentino, Hervé Glotin, and Richard Baraniuk. Spline filters for end-to-end deep learning. In International Conference on Machine Learning, pp. 364-373, 2018.
Eric Battenberg, Rewon Child, Adam Coates, Christopher Fougner, Yashesh Gaur, Jiaji Huang, Heewoo Jun, Ajay Kannan, Markus Kliegl, Atul Kumar, et al. Reducing bias in production speech models. arXiv preprint orXiv:1705. 04400, 2017.
Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390. 2014.
Steven Davis and Paul Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE transactions on acoustics, speech, and signal processing. 28(4):357-366, 1980.
Bradley Efron and Robert J. Tibshirani. An Introduction the Bootstrap. Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, Boca Raton, Florida, USA, 1993.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. Technical report, apr 2017. URL http: pttet:://arxee arxiv. org. aasp//////////// 1obbs//70 17044.14000 01279.
Tiago H Falk and Wai- Yip Chan. Modulation spectral features for robust far-field speaker identifi- cation. IEEE Trunsactions on Audio, Speech, and Language Processing, 18(1):90-100, 2009.
Gustav Theodor Fechner, Davis H Howes, and Edwin Garrigues Boring. Elements of psychophysics, volume 1. Holt, Rinehart and Winston New York, 1966.
Dennis Gabor. Theory of communication. part 1: The analysis of information. Journal of the Institution of Electrical Engineers-Pun Part III: Radio and Communication Engineering, 93(26):429 441, 1946.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 20/7 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780. IEEE, 2017.
Donald D. Greenwood. The mel scale's disqualifying bias and a consistency pitch- -itch-difffernne equisections in 1956 with equal cochlear distances and equal frequency ratios. Hearing Research, 103:199-224: 1997.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Toni Heittola, Annamaria Mesaros, and Tuomas Virtanen. TUT Urban Acoustic Scenes 2018, De- velopment dataset, April 2018. URL peattts:://di..o
5281 zenodo. 12281 42.
10

---

Published as a conference paper at ICLR 2021
Shawn Hershey, Sourish Chaudhuri, Daniel P.W. Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. CNN architectures for large-scale audio classification. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing- Proceedings, pp. 131- 135, 2017. ISBN 9781509041 176. doi: 10.1 1A11//ASSPP2117.755555.
Geoffrey Hinton, Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29()) 2012.
Yedid Hoshen, Ron Weiss, and Kevin W Wilson. Speech acoustic modeling from multichannel waveforms. In International Conference on Acoustics, Speech, and Signal Processing, 2015.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and patlern recognition, pp.
2017.
Navdeep Jaitly and Geoffrey Hinton. Learning better representation of speech soundwaves using restricted boltzmann machines. In 201 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884-5887. IEEE, 2011.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint orXiv:1412.6900 2014.
Qiuqiang Kong, Yin Cao, Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley, Panns: Large-scale pretrained audio neural networks for audio pattern recognition. ArXiv, abs/1912.10211, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
V. Lostamlen, J. Salamon, M. Cartwright, B. McFee, A. Farnsworth, S. Kelling, and J. P. Bello. Per-channel energy normalization: Why and how. IEEE Signal Processing Letters, 26(1):39-43, 2019.
Vincent Lostanlen, Justin Salamon, Andrew Farnsworth, Steve Kelling, and Juan Pablo Bello. firdvox-full-nige dataset and benchmark for avian flight call detection. In 2018 IEEE Inter- national Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 266 270. IEEE, 2018.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE/ACM Trans. Audio, Speech Language Processing, 2766):1256-1266, 2019.
Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path mn: efficient long sequence modeling for time-domain single-channel speech separation. arXiv preprint 1rXiv:1910.06337 2019.
James G Lyons and Kuldip K Paliwal. Effect of compressing the dynamic range of the power spectrum in modulation filtering based speech enhancement. In Ninth Annual Conference of the International Speech Communication Association, 2008
Nelson Mogran, Hervé Bourlard, and Hynek Hermansky. Automatic speech recognition: An audi- tory perspective. In Speech processing in the auditory system, pp. 309-338. Springer, 2004.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: A large-scale speaker iden- tification dataset. Interspeech 2017, Aug 2017. doi: 1022447/inerspech.20777 2011cr/..pccc...75500. URL http:// /xxx/xx///////oo doi. wow.oorg/...... 21 2113 2133///nterseeeceee
Paul-Gauthier Noé, Titouan Parcollet, and Mohamed Morchid. Cgenn: Complex gabor convolu- tional neural network on raw speech. ICASSP 2020 -202 IEÉE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7724-7728, 2020.

---

Published as a conference paper at ICLR 2021
D. O' Shauggnessy. Speech communication. human and machine. 1987.
Dimitri Palaz, Ronan Collobert, and Mathew Magimai Doss. Estimating phoneme class condi- tional probabilities from raw speech signal using convolutional neural networks. arXiv preprint arXiv: 1018, 2013.
Dimitri Palaz, Mathew Magimai Doss, and Ronan Collobert. Convolutional neural networks- continuous speech recognition using raw speech signal. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4295-4299. IEEE, 2015.
Manuel Pariente, Samuele Cornll,, Antoine Deleforge, and Emmanuel Vincent. Filterbank de- sign for end-to-end speech separation. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6364-6368. IEEE, 2020.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition.
arXiv preprint arXiv: 1904.08779 2019.
Mirco Ravanelli and Yoshua Bengio. Speaker recognition from raw waveform with sincnet. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 1021-1028. IEEE, 2018.
Shauna Revay and Matthew Teschke. Multiclass language identification using deep learning on spectral images of audio signals. CoRR, abs/ /t/p/1905.0444 04348, 2019. URL http:://atii...///////// org/ 04348.
T. Sainath, Brian Kingshury, Abdel rahman Mohamed, and B. Ramabhadran. Learning filter banks within deep neural network framework. 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 297-302, 2013.
Tara N Sainath, Ron J Weiss, Andrew Senior, Kevin W Wilson, and Oriol Vinyals. Learning the speech front-end with raw waveform cldnns. In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
Jan Schlüter and Bernhard Lehner. Zero-mean convolutions for level-invariant singing voice detec- tion. In ISMIR, 2018
Ralf Schluter, Iija Bezrukov, Hermann Wagner, and Hermann Ney. Gammatone features and feature combination for large vocabulary speech recognition. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICAASSP00 07, volume 4, pp. IV-649. IEEE, 2007.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. arXiv preprint 1rX4v:1904.05862, 2019.
Andrew Senior, Hasim Sak, Felix de Chaumont Quitry, Tara N. Sainath, and Kanishka Rao. Acoustic modelling with ed-cte-smbr lstm rnns. In ASRU, 2015.
Stanley S Stevens and John Volkmann. The relation of pitch to frequency: A revised scale. The American Journal of Psychology, 53(3):329 353, 1940.
Dan Stowell, - Mike Wood, Hanna Pamula, Yannis Stylianou, and Hervé Glotin. Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge. Tech- nical report, 2018 URL https: arxiv. wwtpoot/////oooooooootoooott.opp 1807 05812. pdf.
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Edouard Grave, Tatiana Likhomanenko, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end asr: from super- vised to semi-supervised learning with modern architectures. arXiv preprint arXiv: 1.08460. 2019.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114 PMLR, 2019. URL http:://///. ptttt::/procee http:////ooooodeeoeoo ngs ml www.ssrssssiee / atn 9a html.
Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali-
12

---

Published as a conference paper at ICLR 2021
Samuel Thomas, Sriram Ganapathy, and Hynek Hermansky. Hilbert envelope based features for far-field speech recognition. In International Workshop on Machine Learning for Multimodal Interaction, pp. 124. Springer, 2008.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Attention-based wav2text with feature trans- fer learning. In 2017 IEEE automatic speech recognition and understanding workshop (ASRU), pp. 309-31110000 315. IEEE, 2017.
Umesh, L. Cohen, and Nelson. Fitting the mel scale. 1999 IEEE International Conference Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No. No.99CI136258), 1: 217-220 vol.1, 1999.
Jesús Villalba, Nanxin Chen, David Snyder, Daniel Garcia-Romero, Alan McCree, Gregory Sell, Jonas Borgstrom, Leibny Paola García-Perera, Fred Richardson, Réda Dehak, et al. State-of-the- art speaker recognition with neural network embeddings in nist sre18 and speakers in the wild evaluations. Computer Speech & Language, 60:101026, 2020.
Yuxuan Wang, Pascal Getreuer, Thad Hughes, Richard Lyon, and Rif A Saurous. Trainable frontend for robust and far- far-field keyword spotting. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5670-5674. IEEE, 2017.
Pete Warden, Speech Commands: A Dataset for Limited- Voccauulayy Speech Recognition. Technical report, 2018. URL https: /atess///arxiiee arxivv tttp//wwwww........./o///// 1804. 03209 pd...
Neil Zeghidour, Nicolas Usunier, lasonas Kokkinos, Thomas Schatz, Gabriel Synnaeve, and Em- manuel Dupoux. Learning filterbanks from raw speech for phone recognition. In 2018 IEEE international conference on acoustics, speech and signal Processing (ICASSP), pp. 5509-5513. IEEE, 2018a.
Neil Zeghidour, Nicolas Usunier, Gabriel Synnaeve, Ronan Collobert, and Emmanuel Dupoux. End- to-end speech recognition from the raw waveform. Interspeech, abs/1806.0700 011880000799,,, 2018b.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paze mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.0942, 2017.
Richard Zhang. Making convolutional
networks shift-invariant again.
arXiv preprint
arXiv:/904.1 11486, 2019.
13

---

Published as a conference paper at ICLR 2021
A
APPENDIX
Table A.I: Datasets used the experiments. Default train/test splits are always adopted.
Task
Name
Classes
Train examples
Test examples
Audio events
Audioset
527
1,832,720
17,695
Acoustic scenes
TUT Urban 2018
10
7,829
810
Birdsong detection
DCASE2018
32,129
3,561
Emotion recognition
Crema-D
5,146
820
Speaker Id.
Voxceleb
1,251
138,361
8,249
Speaker Id.
Librispeech
251
25,740
2,799
Music (instrument)
Nsynth
11
289, 289,205
12,678
Music (pitch)
Nsynth
289.205
12,678
Speech commands
Speech commands
35
84,771
10,700
Language Id.
Voxforge
6
126,610
18,378
LEAF mel 0.000)
LEAF tdfbanks (p 0.000)
LEAF SincNet (p 0.059)
0.25
0.20
0.15
Figure A.I:
Distribution of the difference between the accuracy (%) obtained with LEAF and each of the other frontends when sampling 8 datasets with replacement, in the single task setting.
LEAF mel 0.048)
LEAF tdfbanks (p 0.000)
LEAF SincNet (p 0.000)
0.4
1.0.0-75--4 -25
5.0
10.0
-10.0-75
0.0
5.0
100
-10.075
-5.0
0.0
5.0
10.0
Figure A.2: Distribution of the difference between the accuracy obtained with LEAF and each of other frontends when sampling datasets with replacement, multi-task setting.
14

---

Published as a conference paper at ICLR 2021
104
mel-spectrogram
Frequency [Hz]
LEAF (on AudioSet)
10a
10a
0
5
10
15
20
25
30
35
40
Filter index
Figure A.3: Comparison between the filters learned by LEAF and the mel scale.
Learned depthwise Gaussian lowpass filters
1.0
init
0.8
0.6
0.4
0.2
0.0
0
50
100
150
200
250
300
350
400
Figure A.4: Learned Gaussian lowpass filters of LEAF on AudioSet. The dotted line represents the initialization, identical for all filters.
PCEN smooth coefficients
PCEN root coefficients
0.16
2.6
0.14
2.5
0.12
2.4
0.10
2.3
0.08
2.2
0.06
2.1
0.04
2.0
10
20
30
40
50
60
0
10
20
30
40
50
60
Figure A.5: Values of learned sPCEN parameters of LEAF trained AudioSet. The dotted lines represent the initialization, identical for all filters.
15

---

Published as a conference paper at ICLR 2021
LEAF-PCEN
90%
LEAF-log
Mel-PCEN
80%
Mel-log
Accuracy
70%
60%
50%
40%
8
11
5
-5
SNR
Figure A.6: Test accuracy (%) on the test set of the Speech Commands dataset with varying SNR.
16


Reviews:
[{"review_id": "vgQYI2SlOS9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors introduce a learnable front-end (LEAF) for audio. The paper evaluates it on several tasks in the audio domain such as acoustic event classification, speaker identification, keyword spotting, language identification, music classification etc. The study reports that proposed features outperform or perform similar to Mel filterbanks for many tasks. This paper is well written and easy to follow. The references are excellent as well. I have summarized my comments below which will help in improving the quality of this manuscript: \n\n1. While the authors evaluate the proposed front-end on several tasks, the evaluation protocols are missing. Especially, the training partition and test partition and number of segments in each class. For example, did they calculate accuracy using balanced classes? Was classification done at the frame level or the utterance level? I would highly recommend authors to add the experiment protocol to the manuscript as Table 1A does not provide enough information. \n\n2. The authors evaluate LEAF on several tasks however the model architecture for classification models remains constant. Typically Mel FBs are utilized across architectures and one of the major shortcomings of the proposed work is the integration of LEAF with state-of-the-art models (for example x-vectors for SID). This would give more understanding about the generalization capability of the LEAF.\n\n3. As pointed out by authors in the introduction, Mel FBs are invariant to deformation and noise-robust. All the audio problems addressed in this work encounter varying levels of noise in the real-world. It is extremely important to assess the noise robustness of the proposed front-end. I would recommend the authors test LEAF with different levels of noise.\n\n4. For results in Table 1 the statistical significance should be computed. For the acoustic scene task, the difference in performance is very less and this dataset only has 810 samples overall for testing. Therefore, without confidence intervals, it is very hard to conclude the performance of LEAF. \n\nMinor comments:\n1. Please add this missing reference: \nDavis, S. and Mermelstein, P., 1980. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE transactions on acoustics, speech, and signal processing, 28(4), pp.357-366.\n\n2. In Table 1, Mel and LEAF has the same accuracy for Music instrument and needs to be bolded. \n\n3. mel ---> Mel \n\n4. In Fig 2, you may improve the font size of x-axis labels. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q_5PI0NaIvZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new learnable representation fo audio signal classification and compares it to the classical mel-filterbanks representation and two other learnable representations on a broad range of audio classification tasks, from birdsongs to pitch, instrument, language or emotion recognition. The proposed representation combines several parameterized representation techniques from the recent litterature. It is reported to yield on par or better classification results than the other methods on several of these tasks using single- or multi-task learning.\n\nPros:\n- Learning an ultimate, universal, generic representation for all audio signals that renders the 80 years old mel-frequency scale obsolete is certainly an attractive goal\n- The proposed representation carefully and elegantly combines the best parts of several recently proposed parameterized representations and enjoys a nice interpretability while requiring few parameters to learn.\n- Comparing different audio representations on such a broad range of audio classification task is a welcome and unmatched effort, to the best of my knowledge.\n\nCons:\n- The paper lacks humility in its story-telling and its style. It employs formulations such as \"lived through the history of audio\", or \"challenging the historical statu quo\" when refereing to mel-frequency representations, although by the authors' own admition in the paper, a large amount of research effort has already been given in recent years towards learnable audio representations (the authors cite a dozen papers but there are more). Hence, this paper is not a first attempt. And despite the pompous use of \"universal\" in the title, I believe it is not a last attempt either. The authors claim that the proposed representation \"outperform mel-filterbanks over several tasks with a unique parametrization\" but this is far from clear when looking at the results carefully. In the majority of the tasks, the representation performs either slightly worse, equal, or about 0.5% better than Mel-filterbanks. It is not clear whether such improvement is significant, since no error bar or standard deviation is provided in the results (a sadly common habit in the audio litterature). The only tasks where a truly significant improvement is reported are language identification and emotion recognition, which are also the tasks where all the methods perform the poorest. It looks like any significant difference between the 4 compared approaches would vanish if these two tasks were omitted. The reason why the proposed representation performs well on these two very specific tasks is not clear and not discussed.\n- More generally, the paper would be much more valuable if it gave a sense of WHAT is actually learned by the proposed method. Is the final representation significantly different from a mel-filterbank? Given how close to mel most reported results are, this is doubtful. In fact, Fig. A.1. strongly suggests that LEAF just re-learned mel, but strangely this figure is never commented. Some comments on the learned compression-parameters would also be appreciated.\n- At least one important comparison point is clearly missing in the reported results: STFT + PCEN or mel-filterbanks + PCEN, e.g., Wang et al. (2017) or Schlüter & Lehner (2018) [note that the latter already uses sPCEN rather than PCEN, contrary to the authors' claim] . Omitting this from the comparisons prevents one from knowing whether the proposed parameterized Gabor filterbank brings any advantage over another time-frequency representation like STFT or mel-filterbanks. Less critically, another missing comparison point is LEAF + CNN14, in Table 4.\n- What the authors refer to as \"audio\" in the title and throughout the paper is in fact much more narrow, namely \"audio classification\". Learnable audio representations have been studied in a broader context in recent years, e.g., speech enhancement, source separation, dereverberation, sound localization or audio (re-)synthesis. In fact, one of the important breakthroughs recently brought by learnable audio frontends was in source separation with the paper TasNet (Luo et al. 2018) which is not cited by the authors. In the same context, (Ditter and Gerkmann 2020) presented a learnable gammatone-like filterbank and showed that fully-parameterized learned filterbanks tended to have logarithmic spread in frequencies. Moreover, the use of learnable analytical filterbanks/Hilbert pairs due to their envelop extraction/shift invariant properties was already discussed in depth in (Pariente et al. 2019) [cited in the paper]. \n\nOverall, while comparing different learnable audio representations on a broad range of audio classification tasks is a timely and worthwhile topic, and while the proposed representation elegantly combines several recent ideas in this area, the general presentation and angle of the paper strongly lacks humility. Instead of the proposed title, something like \"Benchmark of learnable audio representations on a broad range of classification tasks\" would be more truthful to the work. To make the investigation more worthwhile and insightful, additional comparison points (STFT + PCEN, mel-frequency + PCEN, Gabor + log, etc.) as well as an analysis of what the model has actually learned would be needed.\n\n======= Review edit after authors' revisions ======\nThe changes made by the authors in the title, abstract, introduction and conclusion to narrow the scope of the paper, better contextualize it, and make it more humble and truthful are very welcome. The extra experiments, figures, addition of error bars and new statistical tests are also a real plus. In doing so, the authors addressed all of my major concerns.\n\nFor these reasons, changed my evaluation score from 5 to 8.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NYlNefvv5X", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a learnable frontend for classification tasks on audio signals. The proposed learnable audio frontend (LEAF) is a generalization of a mel filterbank, used commonly in machine audition.\nLEAF consists of a Gabor filterbank, magnitude-squared nonlinearity, Gaussian lowpass filter and previously-proposed per-channel energy normalization. Learnable parameters of LEAF are the center frequencies and bandwidths of Gabor filters, bandwith of lowpass Gaussian filters and smoothing coefficients of sPCEN. The proposed LEAF matches the performance of competing frontends in most of the cases and leads to some improvements in some cases.\nIn general, I like the paper and the proposed frontend is very sensible from perspective of audio-related applications.\nHowever, I believe there’s some exaggeration in terms of impact of the proposed frontend based on the results in this paper. The proposed system is not actually challenging status quo, as many learnable frontends have been proposed in the literature in the past (as also listed in the references).\nThe paper is easy to read and authors communicate their contribution clearly.\nHowever, I believe that the title may be somewhat too general: LEAF is evaluated only on classification tasks, and IMHO that should be indicated in the title as well. There are other tasks, such as speech enhancement, where LEAF-like frontend may work well, but that is out of scope of this paper.\nExperimental results show that LEAF is performing well in the considered tasks. However, it would be interesting to understand the differences in performance when the encoder & head are changed and/or increased.\nMore specifically, the current results are obtained using a lightweight EfficientNet and linear heads. Is there any particular reason for this setup? Would the conclusions change with a different encoder/head?\nAlso, understanding of the influence of the filterbank setup (number of channels, window length, stride) would be beneficial.\n\nDetails:\n(1) Title should reflect the fact that LEAF has been evaluated only on classification tasks\n(2) Abstract: “over a wide range of audio domains” —> It would be more appropriate to talk about a range of applications in audio domain.\n(3) Abstract: “unprecedented” -> I believe this is a bit exaggerated\n(4) Introduction: “this might not be the optimal approach for non-human sounds” —> This is a strange argument, and I believe the authors are confusing sound perception and sound production. Human sound perception works quite well for recognition on non-human sounds. The authors imply that a system which replicates a system mimicking human perception is suboptimal for non-human sounds. However, human auditory system is not designed for processing of only human-made sounds. Furthermore, optimality depends on the application, so stating that something is not optimal for a class of sounds makes no sense without the defined application, which in this case could be recognition of “acoustic events or animal vocalizations”\n(5) Introduction, last paragraph: “wide and diverse range of tasks, including speech, music, audio events, and animal vocalizations” —> Signals, such as speech or music are not tasks. A task can be, e.g., speech recognition.\n(6) Conclusion: Stating there’s a “historical statu quo of using hand-crafted mel-filterbanks” with so many end-to-end systems giving the best performance in different applications is a bit too much.\n\n\n======= Review edit after authors' revisions ====== \n\nMost of my concerns have been resolved in the significantly-improved revised version of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AR3JT8RiR71", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper shows a detailed interpretation on the relationship between each component of hand-crafted audio front-ends (such as mel-spectrograms) and learnable counterparts. To do that, they followed the narratives presented from the previous works such as SincNet and improved the model by changing the several components of it. The authors grouped the audio front-ends into mainly three parts which are filtering, pooling, and compression. And, the contributions were made at each stage. For filtering stage, instead of learning all the parameters of the convolution layer, they let the model to learn only center frequency and bandwidth of the filterbanks that are initially assigned with Gabor filters. For pooling stage, instead of using simple average or max poolings, they let the model to learn low pass filtering with small parameters. For compression, instead of using log based dynamic compression, they extended Per-Channel Energy Normalization by replacing a fixed smoothing factor to learnable parameters and named it to sPCEN.\n\nTo evaluate the proposed approach, they evaluated the models on 8 audio classification tasks which might have diverse audio and label characteristics (such as acoustic scene sound, animal sound, music, speech). The compared models are mainly mel-spectrogram and SincNet. The results shows that the proposed model outperforms the comparisons for most tasks. Then, they further ran a multi-task classification experiment to obtain universal audio front-ends. And, the results show the proposed learnable front-ends is showing some generalization ability on most tasks. Finally, they evaluated the proposed model on large-scale audio classification dataset (AudioSet) and verified that the proposed front-ends is also showing the good performance on it.\n\nIn page 4, the authors mentioned that the l2 normalization helps distinguishing the role of filtering and compression. I think this contribution is not trivial, so if the authors can add more experiments (or plots) to show the difference between models with and without l2 normalization, then it would be helpful.\n\nThe backbone model used in the paper is fixed, and showing that the proposed audio front-ends shows similar trends with multiple backend models can verify better generalization ability of the proposed approach. So, if the authors can add additional experiments with multiple backends, then it would be helpful.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All Reviewers agree that the paper has a clear and solid contribution. Furthermore, all of them highlight that the paper has improved significantly after revision. Hence, my recommendation is to ACCEPT the paper. As a brief summary, I highlight below some pros and cons that arose during the review and meta-review processes.

Pros:
- Comparison across network architectures.
- Comparison across a broad range of different data sets.
- Compactness of the representation (few parameters to learn).
- Authors will share code.

Cons:
- Role of L2 normalization could be further discussed.
Decision: Accept (Poster)