SpectroBank: A filter-bank convolutional layer for CNN-based audio applications

1 INTRODUCTION . In audio signal processing , time-frequency representations such as spectrograms are central tools . They have an intuitive interpretation and reveal insightful information to the human expert . It is not a surprise that many deep learning approaches to audio signals use such representations as well ( Choi et al. , 2017 ; Purwins et al. , 2019 ) . It is also convenient as most of the deep network architectures have been developed for image processing and require 2D arrays of values as inputs . The network learns to detect time-frequency patterns , similarly to what is done on images . Depending on the task , it may then output a classification of a sound ( Piczak , 2015 ; Salamon & Bello , 2017 ) , a denoised signal ( Lu et al. , 2013 ) or separated sources ( Chandna et al. , 2017 ) . However , natural images and spectrograms do not possess the same properties and turning an audio file into an image has some limitations . Among them , spectrogram representations can be defined in many different ways , with different time window shapes and sizes or different frequency spacing . Choi et al . ( 2017 ) and Purwins et al . ( 2019 ) give a review of the different time-frequency representations used in deep learning . In addition , patterns in the time-frequency plane are different from those that can be found in images : the former are usually less complex , with smoother edges and limited textures . Furthermore , the axes are not equivalent in the spectrogram as frequency is different from time . For example a frequency-shifted pattern may result in a different sound classification ( Lee & Nam , 2017 ) , while a temporal shift does not . Moreover , the spectrogram is the magnitude of the short-time Fourier transform and the information contained in the phase is not taken into account . Lastly , computing a spectrogram , and possibly inverting it for synthesis , adds a computational burden which can be important for large audio datasets . To overcome these limitations , an alternative direction has been chosen consisting of taking an end-to-end approach where the raw audio file is the input of the network . The recent success of Wavenet ( Oord et al. , 2016 ; Paine et al. , 2016 ; Oord et al. , 2017 ) demonstrates the efficiency of this approach for audio synthesis . Raw audio input is also beneficial for speech separation tasks . Tasnet ( Luo & Mesgarani , 2018 ) as well as Wave-U-Net ( Stoller et al. , 2018 ) show better performances for speech separation and faster processing compared to spectrogram-masking approaches . In end-to-end approaches , one-dimensional convolutions are applied to raw audio signals . However , kernel size needs to be much larger than the one used for image applications . Indeed , at a sampling rate of 44kHz , 44 samples represent 1 ms of audio signal . To capture audio patterns that have duration of 10 , 100 ms or more , in particular low frequency patterns , either large kernels are needed or deeper convolutional architectures ( to allow for combinations of kernels at many different positions in time ) . Both solutions lead to a large increase in the number of parameters to be learned and hence require more training time and more data . The `` atrous '' convolution have been introduced in Wavenet in order to increase the time length of the kernel without increasing the number of weights to learn . Finding alternative ways for unlocking the time-length limit is an important challenge for raw audio processing in deep learning . Motivation . In the case of end-to-end learning , several studies investigate convolution kernels learned from the raw audio signal ( Dieleman & Schrauwen , 2014 ; Tüske et al. , 2014 ; Golik et al. , 2015 ; Sainath et al. , 2015 ) . They all show that the input kernel ’ s focus in frequency is similar to the one of the Mel or auditory scale . The kernel shapes in the spectral domain are similar to band-pass filters , with more narrow-band kernels localized on the low frequency spectrum than in the high frequency . This behavior does not depend on the network architecture nor on the application such as speech recognition ( Hoshen et al. , 2015 ; Zeghidour et al. , 2018 ) or audio tagging ( Dieleman & Schrauwen , 2014 ) . All of these results suggest that the logarithmic spacing of frequencies and bandwidth properties first established in the psycho-acoustics studies with the Mel/Bark scales are somehow universal in audio analysis tasks . These works point out the tendency of the input convolution kernels to adopt band-pass filter shapes . Hence , we hypothesize that designing kernels with a band-pass property results in an inductive bias that helps the network converge more rapidly and possibly reduces overfitting . Our first motivation is to confirm this hypothesis . The studies cited above remain experimental without , yet , precise spectral and temporal properties of the kernels . In addition , most of them initialize the kernels as band-pass filters with a Mel scale frequency spacing . So the influence of the kernel initialization remains unclear . Our second motivation is to investigate more precisely these filters ’ properties . Adopting a hybrid approach , halfway between the raw audio and the spectrogram , we propose to learn particular filters ’ shapes having a limited number of parameters that fully define them . These filters are the kernels of the first convolutional input layer of the network . This set of kernels may be seen as a filter bank . Consequently , the new input layer acts on the raw audio and outputs a learned time-frequency representation , adapted to the task . The functions we propose are modulated Gaussian windows , Gammatone and Gammachirp functions . The goal of these filters is two-fold . Firstly , it reduces the number of parameters to learn . It makes the size of the kernel independent of the number of weights to learn and enables the usage of large temporal inputs . Secondly , this layer of parameterized functions helps understanding the filtering process done within the first layer of deep networks . This opens the way to a better interpretation of the neural networks and beyond , of the intricate relationship and the shape of audio patterns in the time-frequency space . Our contribution 1 . We propose new parameterized functions and compare them to recent works on the same topics that use learnable Wavelets or sinc functions ( see related work section below ) . We confirm that this approach improves the accuracy on several audio classification tasks . Moreover , combining the Spectrobank layer with simpler networks can lead to a higher accuracy . We explore the influence of different parameters on the learning , such as the numbers of kernels and their length . Our classification experiments show that the number of filters required to obtain the best results remains small , around 20-30 . We also demonstrate that the performances of different functions proposed in audio signal processing ( modulated Gaussian , Gammatone and Gammachirp functions ) give close results and are better than Wavelets at classifying sounds . Last but not least , a relationship between the central frequency of the filter and its temporal width emerges with the learning . We provide evidences that the network converges to an auditory frequency spacing , close to the ERB ( Equivalent Rectangular Bandwidth ) and Bark scales found in psycho-acoustic studies . Related work . The kernel shapes proposed in the present work are based on specific signal processing functions . They are used for performing short-time Fourier transforms or more generally for designing filterbanks . Modulated ( truncated ) Gaussian are emblematic examples . Gammatones and Gammachirp functions are used in cochlear models ( Saremi et al. , 2016 ) . They provide interesting results when combined with deep learning models for speech enhancement ( Baby & Verhulst , 2018 ) . Learning parametric filters is halfway between 1 ) learning a standard convolutional layer , where all the weights of the kernels are learnable and unconstrained and 2 ) having a layer of kernels being fixed functions , where only the combination of these predefined functions may be learnt . The first approach is the most versatile but is more prone to overfitting . The second approach used for example in the 1The SpectroBank code will be made publicly available after the review process . Scattering transform ( Bruna & Mallat , 2013 ; Andén & Mallat , 2014 ; Cotter & Kingsbury , 2019 ) , or in Jacobsen et al . ( 2016 ) benefits from an inductive bias through the chosen kernel functions but is less flexible . The concept of learning filters aims at making an ideal compromise between flexibility and inductive bias . It has been first introduced in three recent works by Seki et al . ( 2017 ) , Ravanelli & Bengio ( 2018 ) and Khan & Yener ( 2018 ) . The first one introduces Gaussian filters in the input layer . Parameters are the amplitude , the Gaussian width and the modulation frequency . An increase of the classification accuracy is reported with the learned parameters . However , the filter learning is seen as a fine-tuning of the network after the first training pass with fixed Gaussian parameters . In the present work , the filter layer is fully integrated in the learning process , the parameters are learned from the beginning . In Ravanelli & Bengio ( 2018 ) , the authors introduce a layer , called SincNet , made of sine modulated functions that approximate band-pass rectangular windows in the frequency domain . The learned parameters are the minimal and maximal cut-off frequencies of each band-pass filter . One of the main results is given by the cumulative frequency response of the SincNet filters . The network tends to focus more on particular regions of the frequency space , where formants are localized . This is interesting as it shows how the parameterized filters enable a precise interpretation of the learning and underline particular spectral properties of the data . The present work goes further in this direction . Eventually , Khan & Yener ( 2018 ) introduce Wavelet filter banks learned for speech recognition . Each kernel is a Wavelet defined by a single parameter , its scale . It shows evidences both of the efficiency of this approach and of the possibility to interpret the shape of the learned kernels . We compare the efficiency of the Wavelet filters with several other modulated windows and show that the former under-performs on audio signals . More recently and in line with our approach , Loweimi et al . ( 2019 ) present complementary results , on a different dataset , with a focus on the sinc-square function , learning either the frequency or the bandwidth of the filters . 2 LEARNABLE FILTER BANKS ( SPECTROBANK ) . We design a new convolutional neural network layer , called SpectroBank . In this layer the kernels are functions defined by a few parameters that are learned . We call these functions filters , making a parallel with filters in signal processing . Indeed , these functions have the property of being band-pass filters and are well known in audio signal processing . One of the trainable parameters of each filter is the central frequency of the band-pass filter . The second parameter is the bandwidth of the filter ( or a quantity closely related to it ) . Hence this set of filters forms a filter bank where the frequency and bandwidth of the filters may be adapted to the data and to the learning task . Note that the learned filterbank may not cover the entire spectrum but should focus on important spectral regions that are the most discriminative for classification . The input of the SpectroBank layer is a 1D audio signal and the output is a 2D representation . The output representation axes are time and filter number . Since each filter is associated to a particular frequency band , this 2D representation can be seen as a time-frequency one ( or time-scale in the case of Wavelets ) . Initializing the filters by increasing frequencies ( or scales ) , we can influence the frequency ordering to follow the filter number . In all the definitions , N denotes the filter length and n is the variable ( sample number ) . The time in second can be expressed using the sampling frequency fs with t = n/fs and the frequency in Hertz with f × fs , where f ∈ [ 0 , 0.5 ] is the normalized frequency in the formulas . Mexican hat Wavelet . In order to compare to the state-of-the-art , we use the Mexican hat Wavelet introduced in the paper by Khan & Yener ( 2018 ) : w ( n ) = 2 π1/4 √ 3s ( n2 s2 − 1 ) e− n2 s2 , ( 1 ) with n ∈ [ −N/2 , ( N − 1 ) /2 ] and s > 0 being the scale parameter . Gaussian filter . Here , n ∈ [ −N/2 , ( N − 1 ) /2 ] . The Gaussian filter g is defined as follows : g ( n ) = √ 2√ πσ e− n2 2σ2 ( cos ( 2πfn ) + i sin ( 2πfn ) ) . ( 2 ) The parameter σ > 0 is the variance of the Gaussian ( temporal window width ) and f is the oscillating frequency . It is a complex-valued function that we split into its real and imaginary parts . For each f and σ two kernels are created , one with the cosine modulation and one with the sine one . Gammatone filter . The Gammatone filter ( Darling , 1991 ; Patterson et al. , 1992 ; Hohmann , 2002 ) is another example of kernel . It is defined on the interval n ∈ [ 0 , N − 1 ] as : h ( n ) = A ( γ , b ) nγ−1e−2πbn ( cos ( 2πfn ) + i sin ( 2πfn ) ) , ( 3 ) where A is the normalization , A ( γ , b ) = √ 2 ( 4πb ) ( 2γ+1 ) /Γ ( 2γ + 1 ) . The parameter γ is the order of the Gammatone . It can be learned or fixed to 2 or 4 . These two orders are the best suited ones for modeling the human hearing related filter bank ( Patterson et al. , 1987 ) . In the experiments , we will fix γ = 2 or γ = 4 . The other learnable parameters are b , related to the width of the function , and f the frequency . The symbol Γ denotes the Gamma function . The bandwidth B of h depends linearly on b and is given by the following formula ( Darling , 1991 ) : B ( γ , b ) = 2 ( 21/γ − 1 ) 1/2b . ( 4 ) Gammachirp filter . This function is similar to the Gammatone family ones but possesses an oscillating frequency that may evolve with time . The Gammachirp function ( Irino & Patterson , 1997 ) is defined on the interval n ∈ [ 0 , N − 1 ] as follows : k ( n ) = A ( γ , b ) nγ−1e−2πbn [ cos ( 2πfn+ c ln ( n+ ) ) + i sin ( 2πfn+ c ln ( n+ ) ) ] , ( 5 ) where A is defined above . In the present work , γ is fixed to γ = 4 . This filter possesses 3 parameters , b related to the width of the window , f to the frequency and c to the chirp value . To avoid the logarithmic singularity at the origin , we add a small positive value = 10−4 to the expression . Remark 1 : All the functions are defined and normalized in the continuous domain . In our application , the filters are discretized and truncated in order to be implemented in the convolution layer . Since they all vanish away from zero , it remains a good approximation , provided that the function ’ s width does not exceed the fixed filter length N . Remark 2 : The modulated window functions are defined with a cosine ( real part ) and a sine ( imaginary part ) term , relating them to the Fourier transform , the spectral domain and the standard definition of filters in signal processing . For the sake of simplicity , in our experiments , we have chosen to use only the cosine term . The absence of the sine term did not affect the accuracy of our classification results . The network is able to adapt and detect discriminative patterns with a shifted cosine modulation . Remark 3 : It is important to distinguish the filter length N from the filter temporal width σ or b ( or s for the scale ) . The filter length is fixed , can not be learned and is the size of the vector on which the filter is defined . The temporal width is learned and specifies the spread of the function over the vector of size N . Therefore , the filter temporal width is always smaller than the filter ’ s length .

Summary: This paper proposes to specify the first layer of a CNN for audio applications with predefined filterbanks from the signal processing community. Those latter are only specified by a limited number of parameters, such as the bandwidth or the central frequency of the filter, and those parameters are then optimized through the standard back-propagation algorithm. Some accuracy improvements are obtained on non trivial datasets.