LEAF: A Learnable Frontend for Audio Classification

1 INTRODUCTION . Learning representations by backpropagation in deep neural networks has become the standard in audio understanding , ranging from automatic speech recognition ( ASR ) ( Hinton et al. , 2012 ; Senior et al. , 2015 ) to music information retrieval ( Arcas et al. , 2017 ) , as well as animal vocalizations ( Lostanlen et al. , 2018 ) and audio events ( Hershey et al. , 2017 ; Kong et al. , 2019 ) . Still , a striking constant along the history of audio classification is the mel-filterbanks , a fixed , hand-engineered representation of sound . Mel-filterbanks first compute a spectrogram , using the squared modulus of the short-term Fourier transform ( STFT ) . Then , the spectrogram is passed through a bank of triangular bandpass filters , spaced on a logarithmic scale ( the mel-scale ) to replicate the non-linear human perception of pitch ( Stevens & Volkmann , 1940 ) . Eventually , the resulting coefficients are passed through a logarithm compression , to replicate our non-linear sensitivity to loudness ( Fechner et al. , 1966 ) . This approach of drawing inspiration from the human auditory system to design features for machine learning has been historically successful ( Davis & Mermelstein , 1980 ; Mogran et al. , 2004 ) . Moreover , decades after the design of mel-filterbanks , Andén & Mallat ( 2014 ) showed that they coincidentally exhibit desirable mathematical properties for representation learning , in particular shift-invariance and stability to small deformations . Hence , both from an auditory and a machine learning perspective , mel-filterbanks represent strong audio features . However , the design of mel-filterbanks is also flawed by biases . First , not only has the mel-scale been revised multiple times ( O ’ Shaughnessy , 1987 ; Umesh et al. , 1999 ) , but also the auditory experiments that led their original design could not be replicated afterwards ( Greenwood , 1997 ) . Similarly , better alternatives to log-compression have been proposed , like cubic root for speech enhancement ( Lyons & Paliwal , 2008 ) or 10th root for ASR ( Schluter et al. , 2007 ) . Moreover , even though matching human perception provides good inductive biases for some application domains , e.g. , ASR or music understanding , these biases may also be detrimental , e.g . for tasks that require fine-grained resolution at high frequencies . Finally , the recent history of other fields like computer vision , in which the rise of deep learning methods has allowed learning representations from raw pixels rather than from engineered features ( Krizhevsky et al. , 2012 ) , inspired us to take the same path . These observations motivated replacing mel-filterbanks with learnable neural layers , ranging from standard convolutional layers ( Palaz et al. , 2015 ) to dilated convolutions ( Schneider et al. , 2019 ) , as well as structured filters exploiting the characteristics of known filter families , such as Gammatone ( Sainath et al. , 2015 ) , Gabor ( Zeghidour et al. , 2018a ; Noé et al. , 2020 ) , Sinc ( Ravanelli & Bengio , 2018 ; Pariente et al. , 2020 ) or Spline ( Balestriero et al. , 2018 ) filters . While tasks such as speech separation have already successfully adopted learnable frontends ( Luo & Mesgarani , 2019 ; Luo et al. , 2019 ) , we observe that most state-of-the art approaches for audio classification ( Kong et al. , 2019 ) , ASR ( Synnaeve et al. , 2019 ) and speaker recognition ( Villalba et al. , 2020 ) still employ mel-filterbanks as input features , regardless of the backbone architecture . In this work , we argue that a credible alternative to mel-filterbanks for classification should be evaluated across many tasks , and propose the first extensive study of learnable frontends for audio over a wide and diverse range of audio signals , including speech , music , audio events , and animal sounds . By breaking down mel-filterbanks into three components ( filtering , pooling , compression/normalization ) , we propose LEAF , a novel frontend that is fully learnable in all its operations , while being controlled by just a few hundred parameters . In a multi-task setting over 8 datasets , we show that we can learn a single set of parameters that outperforms mel-filterbanks , as well as previously proposed learnable alternatives . Moreover , these findings are replicated when training a different model for each individual task . We also confirm these results on a challenging , large-scale benchmark : classification on Audioset ( Gemmeke et al. , 2017 ) . In addition , we show that the general inductive bias of our frontend ( i.e. , learning bandpass filters , lowpass filtering before downsampling , learning a per-channel compression ) is general enough to benefit other systems , and propose a new , improved version of SincNet ( Ravanelli & Bengio , 2018 ) . Our code is publicly available1 . 2 RELATED WORK . In the last decade , several works addressed the problem of learning the audio frontend , as an alternative to mel-filterbanks . The first notable contributions in this field emerged for ASR , with Jaitly & Hinton ( 2011 ) pretraining Restricted Boltzmann Machines from the waveform , and Palaz et al . ( 2013 ) training a hybrid DNN-HMM model , replacing mel-filterbanks by several layers of convolution . However , these alternatives , as well as others proposed more recently ( Tjandra et al. , 2017 ; Schneider et al. , 2019 ) , are composed of many layers , which makes a fair comparison with melfilterbanks difficult . In the following section , we focus on frontends that provide a lightweight , drop-in replacement to mel-filterbanks , with comparable capacity . 2.1 LEARNING FILTERS FROM WAVEFORMS . A first attempt at learning the filters of mel-filterbanks was proposed by Sainath et al . ( 2013 ) , where a filterbank is initialized using the mel-scale and then learned together with the rest of the network , taking a spectrogram as input . Instead , Sainath et al . ( 2015 ) and Hoshen et al . ( 2015 ) later 1https : //github.com/google-research/leaf-audio proposed to learn convolutional filters directly from raw waveforms , initialized with Gammatone filters ( Schluter et al. , 2007 ) . In the same spirit , Zeghidour et al . ( 2018a ) used the scattering transform approximation of mel-filterbanks ( Andén & Mallat , 2014 ) to propose the time-domain filterbanks , a learnable frontend that approximates mel-filterbanks at initialization and can then be learned without constraints ( see Figure 1 ) . More recently , the SincNet ( Ravanelli & Bengio , 2018 ) model was proposed , which computes a convolution with sine cardinal filters , a non-linearity and a max-pooling operator ( see Figure 1 ) , as well as a variant using Gabor filters ( Noé et al. , 2020 ) . We take inspiration from these works to design a new learnable filtering layer . As detailed in Section 3.1.2 , we parametrize a complex-valued filtering layer with Gabor filters . Gabor filters are optimally localized in time and frequency ( Gabor , 1946 ) , unlike Sinc filters that require using a window function ( Ravanelli & Bengio , 2018 ) . Moreover , unlike Noé et al . ( 2020 ) , who use complex-valued layers in the rest of the network , we describe in Section 3.1.2 how using a squared modulus not only brings back the signal to the real-valued domain ( leading to compatibility with standard architectures ) , but also performs shift-invariant Hilbert envelope extraction . Zeghidour et al . ( 2018a ) also apply a squared modulus non-linearity , however as described in Section 3.1.1 , training unconstrained filters can lead to overfitting and stability issues , which we solve with our approach . 2.2 LEARNING THE COMPRESSION AND THE NORMALIZATION . The problem of learning a compression and/or normalization function has received less attention in the past literature . A notable contribution is the Per-Channel Energy Normalization ( PCEN ) ( Wang et al. , 2017 ; Lostanlen et al. , 2019 ) , which was originally proposed for keyword spotting , outperforming log-compression . Later , Battenberg et al . ( 2017 ) and Lostanlen et al . ( 2018 ) confirmed the advantages of PCEN , respectively for large scale ASR and animal bioacoustics . However , these previous works learn a compression on top of fixed mel-filterbanks . Instead , in this work we propose a new version of PCEN and show for the first time that combining learnable filters , learnable pooling , and learnable compression and normalization outperforms all other approaches . 3 MODEL . Let x ∈ RT denote a one-dimensional waveform of T samples , available at the sampling frequency Fs [ Hz ] . We decompose the frontend into a sequence of three components : i ) filtering , which passes x through a bank of bandpass filters followed by a non-linearity , operating at the original sampling rate Fs ; ii ) pooling , which decimates the signal to reduce its temporal resolution ; iii ) compression/normalization , which applies a non-linearity to reduce the dynamic range . Overall , the frontend can be represented as a function Fψ : RT → RM×N , which maps the input waveform to a 2-dimensional feature space , where M denotes the number of temporal frames ( typically M < T ) , N the number of feature channels ( which might correspond to frequency bins ) and ψ the frontend parameters . The features computed by the frontend are then fed to a model gθ ( · ) parametrized by θ . The frontend and the model parameters are estimated by solving a supervised classification problem : θ∗ , ψ∗ = argmin θ , ψ E ( x , y ) ∈D L ( gθ ( Fψ ( x ) ) , y ) , ( 1 ) where ( x , y ) are samples in a labelled dataset D and L is a loss function . Our goal is to learn the frontend parameters ψ end-to-end with the model parameters θ . To achieve this , it is necessary to make all the frontend components learnable , so that we can solve the optimization problem in equation 1 with gradient descent . In the following we detail the design choices of each component . 3.1 FILTERING . The first block of the learnable frontend takes x as input , and computes a convolution with a bank of complex-valued filters ( ϕn ) n=1 .. N , followed by a squared modulus operator , which brings back its output to the real-valued domain . This convolution step has a stride of 1 , therefore keeping the input temporal resolution , and outputs the following time-frequency representation : fn = |x ∗ ϕn|2 ∈ RT , n = 1 , . . . , N , ( 2 ) where ϕn ∈ CW is a complex-valued 1-D filter of length W . It is possible to compute equation 2 without explicitly manipulating complex numbers . As proposed by Zeghidour et al . ( 2018a ) , to produce the squared-modulus of a complex-valued convolution with N filters , we compute instead the convolution with 2N real-valued filters ϕ̃n , n = 1 , . . . , 2N , and perform squared ` 2-pooling with size 2 and stride 2 along the channel axis to obtain the squared modulus , using adjacent filters as real and imaginary part of ϕn . Formally : fn = |x ∗ ϕ̃2n−1|2 + |x ∗ ϕ̃2n|2 ∈ RT , n = 1 , . . . , N. ( 3 ) We explore two different parametrizations for ϕn . One relies on standard fully parametrized convolutional filters while the other makes use of learnable Gabor filters .

Summary: The paper shows a detailed interpretation on the relationship between each component of hand-crafted audio front-ends (such as mel-spectrograms) and learnable counterparts. To do that, they followed the narratives presented from the previous works such as SincNet and improved the model by changing the several components of it. The authors grouped the audio front-ends into mainly three parts which are filtering, pooling, and compression. And, the contributions were made at each stage. For filtering stage, instead of learning all the parameters of the convolution layer, they let the model to learn only center frequency and bandwidth of the filterbanks that are initially assigned with Gabor filters. For pooling stage, instead of using simple average or max poolings, they let the model to learn low pass filtering with small parameters. For compression, instead of using log based dynamic compression, they extended Per-Channel Energy Normalization by replacing a fixed smoothing factor to learnable parameters and named it to sPCEN.