SpectroBank: A filter-bank convolutional layer for CNN-based audio applications
Authors: Helena Peic Tukuljac
Benjamin Ricaud
Nicolas Aspert
Pierre Vandergheynst
Venue:  (2020)
Keywords: audio
classification
convolutional neural network
deep learning
filter
filter-bank
raw waveform

Abstract: We propose and investigate the design of a new convolutional layer where kernels are parameterized functions. This layer aims at being the input layer of convolutional neural networks for audio applications. The kernels are defined as functions having a band-pass filter shape, with a limited number of trainable parameters. We show that networks having such an input layer can achieve state-of-the-art accuracy on several audio classification tasks. This approach, while reducing the number of weights to be trained along with network training time, enables larger kernel sizes, an advantage for audio applications. Furthermore, the learned filters bring additional interpretability and a better understanding of the data properties exploited by the network.

Under review as a conference paper at ICLR 2020
SPECTROBANK: A FILTER-BANK CONVOLUTIONAL LAYER FOR CNN-BASED AUDIO APPLICATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose and investigate the design of convolutional layer where kernels are parameterized functions. This layer aims at being the input layer of convolutional neural networks for audio applications. The kernels are defined functions having a band-pass filter shape, with a limited number of trainable parameters. Building on the literature on this topic, we confirm that networks having such an input layer can achieve state-of-the-ar accuracy on several audio classification tasks. We propose new filter functions and explore the effect of different parameters on the network accuracy and learning ability. This approach reduces the number of weights to be trained and enables larger kernel sizes, an advantage for audio applications. Furthermore, the learned filters bring additional interpretability and better understanding of the data properties exploited by the network.
INTRODUCTION
In audio signal processing, time-frequency representations such spectrograms are central tools. They have an intuitive interpretation and reveal insightful information to the human expert. is not a surprise that many deep learning approaches audio signals use such representations et al. 2017 Purwins et al. 2019). It is also convenient as most of the deep network architectures have been developed for image processing and require 2D arrays of values as inputs. The network learns to detect time- frequency patterns, similarly to what is done on images. Depending on the task, it may then output a classification of sound (Piczak) 2015; (allmon & Bello 2017). a denoised signal (Lu et al. 2013) or separated sources (Chandna et al. 2017).
well (Choi
However, natural images and spectrograms do not possess the same properties and turning an audio file into an image has some limitations. Among them, spectrogram representations can be defined in many different ways, with different time window shapes and sizes or different frequency spacing. Choi et al. (2017) and Purwins et al. (2019) give review of the different freeuuery representations used in deep learning. In addition, patterns in the free-feeyyncy plane are different from those that can be found in images: the former usually less complex, with smoother edges and limited textures. Furthermore, the axes are not equivalent in the spectrogram as frequency is different from time. For example frequency-shifted pattern may result in different sound classification (Lee & Nam 2017), while a temporal shift does not. Moreover, the spectrogram is the magnitude of the short-time Fourier transform and the information contained in the phase is not taken into account. Lastly, computing a spectrogram, and possibly inverting it for synthesis, adds a computational burden which can be important for large audio datasets.
To overcome these limitations, an alternative direction has been chosen consisting of taking an end-to-end approach where the raw audio file is the input of the network. The recent success of Wavenet (Oord et al. 2016)] Paine et al.) 2016: Oord et al., 2017) demonstrates the efficiency of this approach for audio synthesis. Raw audio input also beneficial for speech separation tasks. Tasnet (Luo& & Mesgarani] 2018) as well as Wave-U-Net (Stoller et al. (2018) show better performances for speech separation and faster processing compared to spectrogram- maskinn approaches.
In end-to-end approaches, one-dimensional convolutions are applied to raw audio signals. However, kernel size needs to be much larger than the one used for image applications. Indeed, at a sampling rate of 44kHz, 44 samples represent 1 ms of audio signal. To capture audio patterns that have duration of 10, 100 ms or more, in particular low frequency patterns, either large kernels are needed or deeper convolutional architectures (to allow for combinations of kernels many different positions in time).

---

Under review as a conference paper at ICLR 2020
Both solutions lead to large increase in the number of parameters to be learned and hence require more training time and more data. The "atrous" convolution have been introduced in Wavenet in order to increase the time length of the kernel without increasing the number of weights to learn. Finding alternative ways for unlocking the time-length limit is an important challenge for raw audio processing in deep learning.
Motivation. In the case of end-to-end learning, several studies investigate convolution kernels learned from the raw audio signal (Dieleman & Schrauwen 2014; Tüske et al. 2014; Golik et al. [0155] Sainath et al. 2013). They all show that the input kernel's 's focus in frequency is similar to the one of the Mel or auditory scale. The kernel shapes in the spectral domain are similar to band-pass filters, with more narrow-band kernels localized on the low frequency spectrum than in the high frequency. This behavior does not depend on the network architecture nor on the application such as speech recognition (Hoshen et al., 2015: Zeghidour et al., 2018) or audio tagging (Dieleman & Schrauwen, 2014). All of these results suggest that the logarithmic spacing of frequencies and bandwidth properties first established in the psycho-acousties studies with the Mel/Bark scales are somehow universal in audio analysis tasks. These works point out the tendency of the input convolution kernels to adopt band-past pass filter shapes. Hence, we hypothesize that designing kernels with a band-pass property results in an inductive bias that helps the network converge more rapidly and possibly reduces overfitting. Our first motivation is confirm this hypothesis.
The studies cited above remain experimental without, yet, precise spectral and temporal properties of the kernels. In addition, most of them initialize the kernels as band-pass filters with a Mel scale frequency spacing. So the influence of the kernel initialization remains unclear. Our second motivation is to investigate more precisely these filters' properties.
Adopting a hybrid approach, halfway between the raw audio and the spectrogram, we propose to learn particular filters' shapes having a limited number of parameters that fully define them. These filters are the kernels of the first convolutional input layer of the network. This set of kernels may be seen as a filter bank. Consequently, the new input layer acts on the raw audio and outputs a learned time- frequeeyy representation, adapted to the task. The functions we propose are modulated Gaussian windows, Gammatone and Gammachirp functions. The goal of these filters is two-fold. Firstly, it reduces the number of parameters to learn. It makes the size of the kernel independent of the number of weights to learn and enables the usage of large temporal inputs. Secondly, this layer of parameterized functions helps understanding the filtering process done within the first layer of deep networks. This opens the way to a better interpretation of the neural networks and beyond, of the intricate relationship and the shape of audio patterns in the time-frequency space.
Our contribution We propose new parameterized functions and compare them to recent works on the same topies that use learnable Wavelets or sine functions (see related work section below). We confirm that this approach improves the accuracy on several audio classification tasks. Moreover, combining the Spectrobank layer with simpler networks can lead to higher accuracy. We explore the influence of different parameters on the learning, such as the numbers of kernels and their length. Our classification experiments show that the number of filters required obtain the best results remains small, around 20-30. We also demonstrate that the performances of different functions proposed in audio signal processing (modulated Gaussian, Gammatone and Gammachirp functions) give close results and are better than Wavelets at classifying sounds. Last but not least, a relationship between the central frequency of the filter and its temporal width emerges with the learning. We provide evidences that the network converges to an auditory frequency spacing, close to the ERB (Equivalent Rectangular Bandwidth) and Bark scales found in psycho-acoustic studies.
Related work. The kernel shapes proposed in the present work are based on specific signal process- ing functions. They are used for performing short-time Fourier transforms or more generally for designing filterbanks. Modulated (truncated) Gaussian are emblematic examples. Gammatones and Gammachirp functions are used in cochlear models (Saremi et al. 2016). They provide interesting results when combined with deep learning models for speech enhancement (Baby & Verhulst 2018).
Learning parametric filters is halfway between 1) learning a standard convolutional layer, where all the weights of the kernels are learnable and unconstrained and 2) having layer of kernels being fixed functions, where only the combination of these predefined functions may be learnt. The first approach the most versatile but is more prone to overfitting. The second approach used for example in the
The SpectroBank code will he made publicly available after the review process.
2

---

Under review as a conference paper at ICLR 2020
Scattering transform (Bruna & Mallat 2013 Andén & Mallat 2014 Cotter & Kingsbury 2019).
in inJacobsen et al. (016)) benefits from an inductive bias through the chosen kernel functions but is ideal compromise between flexibilityy and inductive bias. It has been first introduced in three recent works by |Seki et al. (2017). Ravanelli & Bengio (2018) and Khan & Yener (2018). The first one introduces Gaussian filters in the input layer. Parameters are the amplitude, the Gaussian width and the modulation frequency. An increase of the classification accuracy reported with the learned parameters. However, the filter learning is seen as a fine-tuning of the network after the first training pass with fixed Gaussian parameters. In the present work, the filter layer fully integrated in the learning process, the parameters are learned from the beginning. In RRavanelli & Bengio (0188), the authors introduce a layer, called SincNet, made of sine modulated functions that approximate band-pass rectangular windows in the frequency domain. The learned parameters are the minimal and maximal cut-off frequencies of each band-pass filter. One of the main results is given by the cumulative frequency response of the SincNet filters. The network tends to focus more on particular regions of the frequency space, where formants are localized. This is interesting as it shows how the parameterized filters enable a precise interpretation of the learning and underline particular spectral properties of the data. The present work goes further in this direction. Eventually. Khan & Yener (2018) introduce Wavelet filter banks learned for speech recognition. Each kernel is a Wavelet defined by single parameter, its scale. It shows evidences both of the efficiency of this approach and of the possibility to interpret the shape of the learned kernels. We compare the efficiency of the Wavelet filters with several other modulated windows and show that the former under-penn mederr-erfooms on audio signals. More recently and in line with our approach, Loweimi al. (2019) present complementary results, on different dataset, with a focus on the sine-square function, learning either the frequency or the bandwidth of the filters.
less flexible. The concept of learning filters aims at making
2 LEARNABLE FILTER BANKS (SPECTROBANK)
We design a new convolutional neural network layer, called SpectroBank. In this layer the kernels are functions defined by a few parameters that are learned. We call these functions filters, making parallel with filters in signal processing. Indeed, these functions have the property of being band-pase filters and are well known in audio signal processing. One of the trainable parameters of each filter is the central frequency of the band-pass filter. The second parameter is the bandwidth of the filter (or a quantity closely related to it). Hence this set of filters forms filter bank where the frequency and bandwidth of the filters may be adapted to the data and to the learning task. Note that the learned filterbank may not cover the entire spectrum but should focus on important spectral regions that are the most discriminative for classification.
The input of the SpectroBank layer is a ID audio signal and the output is a 2D representation. The output representation axes are time and filter number. Since each filter is associated to particular frequency band, this 2D representation can be seen as frequenrreyyy one (or time-scale in the case of Wavelets). Initializing the filters by increasing frequencies (or scales), we can influence the frequency ordering to follow the filter number.
In all the definitions, N denotes the filter length and n is the variable (sample number). The time in second can be expressed using the sampling frequency fa with t = n/fa and the frequency in Hertz with f x fa. where f E 0.5) is the normalized frequency in the formulas.
Mexican hat Wavelet. order to compare to the state-of-the-art, we use the Mexican hat Wavelet introduced in the paper by Khan & Yener (2018):
2
nt
w(n) x1/4
(1)
$2
with E (N 1)/2] and > 0 being the scale parameter.
Gaussian filter. Here, n E [-N/2, (N- 1)/2]. The Gaussian filter g defined as follows:
2 2 (nn) = ViT"
E (cos(2 + isin((7ff))
(2)
The parameter is the variance of the Gaussian (temporal window width) and f is the oscillating frequency. It is a complex-valuee function that we split into its real and imaginary parts. For each f and two kernels created, one with the cosine modulation and one with the sine one.
3

---

Under review as a conference paper at ICLR 2020
Gammatone filter. The Gammatone filter (Darling
1991
Patterson al. 1992. Hohmann 2002 is
another example of kernel. It is defined on the interval n E [0, - 1 as
h(n) A(T, b)n - e-2bb
2xbn (cos( 2TT fm) + sin((2
where is the normalization, A(y, = 22(4Tb) /T +). The parameter y is the order of the Gammatone. It can be learned fixed to 2 or 4. These two orders are the best suited ones for modeling the human hearing related filter bank (Patterson et al. 1987). In the experiments, we will fix 7 7 4. The other learnable parameters are b, related the width of the function, and the frequency. The symbol I' denotes the Gamma function. The bandwidth B of h depends linearly on b and is given by the following formula (Darling 1991):
B(7, b) = 2(2 /7 1).
(4)
Gammachirp filter. This function is similar to the Gammatone family ones but possesses an oscillating frequency that may evolve with time. The Gammachirp function (Irino & Patterson (997)
is defined on the interval n E [0, N 1] as follows:
knn) A(7, b)mm 2mhn [cos(2 + cln(n + e)) + ssn + cln(n c))],
(5)
where A is defined above. In the present work, y is fixed to y 4. This filter possesses 3 parameters, b related to the width of the window, f to the frequency and e to the chirp value. To avoid the logarithmic singularity at the origin, we add small positive value = 10-4 to the expression.
Remark 1: All the functions are defined and normalized in the continuous domain. In our application, the filters discretized and truncated in order to be implemented in the convolution layer. Since they all vanish away from zero, remains a good approximation, provided that the function's 's width does not exceed the fixed filter length N.
Remark 2: The modulated window functions are defined with cosine (real part) and a sine (imaginary part) term, relating them to the Fourier transform, the spectral domain and the standard definition of filters in signal processing. For the sake of simplicity, in our experiments, we have chosen to use only the cosine term. The absence of the sine term did not affect the accuracy of our classification results. The network is able to adapt and detect discriminative patterns with a shifted cosine modulation.
Remark 3: is important distinguish the filter length N from the filter temporal width a or b (or s for the scale). The filter length is fixed, can not be learned and is the size of the vector on which the filter is defined. The temporal width is learned and specifies the spread of the function over the vector of size N. Therefore, the filter temporal width is always smaller than the filter's 's length.
3 EXPERIMENTS AND RESULTS
We upply SpectroBank to several classification tasks described in the following sections. We want to assess it on standard tasks found in the literature presented in the introduction. We have chosen freely available speech datasets: AudioMNIST (Becker et al.) 2018) and Google Speech Commands v2 (Warden 2018). Both datasets contain words pronounced by different speakers. These datasets are dedicated to correctly recognize the word present in each audio sequence. We also investigate the performances of SpectroBank on an environmental sound dataset in order cover more diverse audio patterns. We have chosen the UrbanSound8K dataset (Salamon et al. 2014). This dataset have been used recently for end-to-end learning (Dai et al.) Abdoli al. 2019). Statistics and spectral energy
speech recognition tasks and the goal is to train the network to
distribution per class of these datasets are given in the Appendix
In order to compare the impact of the SpectroBank layer on the learning and classification results, we use existing network architectures and modify the first layer. For networks with raw audio input, the first convolutional layer (performing a standard ID convolution) is replaced by our proposed parameterized convolution layer. Our layer is then followed by a non-linear ReLU activation function. A stride parameter is available allowing to define the overlap in time of consecutive convolutions. Our focus being to learn from audio, decided to compare our approach only to similar techniques, despite the fact that image-bare based network achieve sometimes higher accuracy than the purely audio- based ones. All the models used for the experiments were implemented using the Keras framework
4

---

Under review as a conference paper at ICLR 2020
and will be made publicly available along with the final version of the paper. Detailed architectures of all networks can be found in appendix B Training was performed using a NVidia GTX1080Ti having 11 GB of RAM.
Input layer initialization. When initializing a filter bank for learning, most of the available solutions start from filter bank with Mel-scale (or log-scale) frequency spacing (Sainath al. 2013 Zeghidour et al.) 2018 Ravanelli & Bengio 2018 Khan & Yener 2018). This frequency distribution supposed to be optimal for audio processing and learning. However, in the present work, want to check this assumption. Hence in all the experiments (except when comparing with SincNet where we retain the mel-based initialization from Ravanelli & Bengio (2018)), we use linearly spaced frequencies (or scales), distributed over the entire spectrum and constant initialization value for the bandwidth (a or b).
3.1 THE IMPACT OF SPECTROBANK PARAMETERS
The learnable parameters of the SpectroBank filters are not the only values that may influence the network accuracy. The choice of the filter type important as well as the filter length and the layer stride (filter overlap). We have tested different configurations and the results are shown on Fig. [ On the left, the accuracy increases with the number of filters up around 30. Beyond this, improvement reported. This number is hence a good compromise between accuracy and network complexity. Similar trend holds for all filter types. These results highlight the better performance of the modulated windows compared to the Wavelets. On Fig. Tb the impact of the filter overlap
0.92
accuracy
0.88
0.75
8.70
Gaussian
0.65
Sinc
Cammatone
Gammachip
evelap 25%
overlap 50%
0.60
16
32
64
0.82
evertap 90%
75%
Number of fiters
10
(a) Accuracy for different filter types and numbers filters. The modulated windows filters achieve similar performances and reach better accuracy than the Wavelets.
(b) Accuracy of Gammatone filters for different lengths and overlap ratios.
Figure Influence of several SpectroBank layer properties on the network accuracy. (Dataset: Google Speech Commands)
(or kernel stride) is shown, exhibiting two different behaviors. First, for large windows (beyond 5ms), a large stride lead to a drop in accuracy. Indeed, the filter width (spread of the modulated window or Wavelet) may be much smaller than the filter total length N. Nevertheless, the overlap measured on the total length. Narrow windows may not overlap at all and information lost during the convolution process. Secondly, short kernels (less than 4ms) with large overlap (or small stride), can render the network short-sighted in time. In that case, long temporal patterns require the combination of large amount of successive output values. The convolutional layers following the SpectroBank layer, deeper inside the network, may not be able capture these long patterns. This results as well in drop of the accuracy observed on Fig. lb
3.2 AUDIOMNIST RESULTS
The original AudioMNIST paper (Becker al., 2018) performs digit classification using raw audio as input to network called AudioNet. The code supplied with the paper has been re-used to perform 5-fold validation on the data. AudioNet is made six convolutional layers, each convolution
https://githhb.... www..///w.....///..o..o.......ee
5

---

Under review as a conference paper at ICLR 2020
being followed by a max-pooling layer, and two dense layers, connected to output layer. In all tests performed using this dataset, the models were trained using the Adam optimizer with default parameters during 50 epochs. Batch size used was set to 256 and loss function used was the categorical cross-entropy. Test accuracy was then computed after this training phase and the same process was repeated for each fold.
On the AudioMNIST dataset sampled at 8 kHz, AudioNet has ca. 17 million trainable parameters. The original paper from (Becker al. 2018) claims an accuracy of 92.53% + 2.04%, whereas our implementation of AudioNet using Keras and Adam optimizer (instead of SGD in the original paper) yields an average accuracy of 94.9% 1.54%, which already a significant improvement. We performed the same 5-fold validation using a modified version of AudioNet where the first convolutional layer is replaced by a SpectroBank layer. This layer consists in 32 4th-order Gammatone filters of length 80 (corresponding to 10 ms at 8 kHz). The stride has been set such that the overlap between two consecutive convolution steps is equal to 75%. In this modified network, the number of trainable parameters drops to ca. 3.5 million trainable parameters, i.e. reduction in size by factor 5. Using the SpectroBank-enableed AudioNet the average accuracy increases to 96.8% + 1.22%.
Another SpectroBaank-nabbld network was used to perform the classification task on AudioMNIST. The architecture. loosely adapted from the one used in the paper by Abboll et al. (019)). is described in appendix (Table Despite its much smaller number of trainable parameters (ca. 300' (000,00), its average accuracy improves to 98.0% + 0.41% 1. For the sake of completeness, we also trained this network, replacing the Gammatone filters by the learned wavelets in (Khan & Yener 2018), and the learned SincNet filters from (Ravanelli & Bengio 20188. A summary of all results achieved using AudioMNIST can be found in Table
Table AudioMNIST mean test accuracy
Network
# Trainable parameters
Avg. accuracy
AudioNet
17 M
94.9% + 1.54%
SpectroBank-AudioNG
3.5 M
= 1.22%11222%
SpectroBank-custom (Gammatone)
300 k
98.0% 10.0%=00 0.41%
SpectroBank-custee (SineNet)
300
97.2% = 1.0%
SpectmBank-custom (Wavelet)
300
89.9% + 1.18%
3.3 GOOGLE SPEECU COMMAND RESULTS
The Google Speech Command dataset provides similar data to the AudioMNIST one, with larger number of classes (35) to distinguish. In the original setting, the goal was to classify 15 unwanted words together unknown. However, in the experiments we performed, we classify each word independently. This dataset does not have pre-defined folds, but train, test and validation data are specified explicitly. We focus on the "Basic" network of the SampleCNN group described in Kim et al.) (2019). Using an input signal resampled to 22.05 kHz, the Basic network has identical blocks, each block being made of ID convolution (size 3). followed by batch normalization, ReLU activation and max pooling. In our experiments, we adapted the proposed setting in order to keep the original 16 kHz sampling of the dataset and ended up with a blocks (vs. 8) network in order to avoid empty dimensions. The code co provided by Kim et al. (2019) was used as basis for our experiments. Reducing the number blocks to 7 and keeping the original 16 kHz sampling rate yields networks having similar number of trainable parameters (ca. 2.3 million vs ca. 2.5 million respectively for 7 blocks/16 kHz and 8 blocks/22.05 kHz).
Given that Google Speech Commands does not possess pre-defined folds for n-fold validation, the experiments were repeated 5 times in order to compute the mean accuracy. The original results from Kim al. (2019) give average accuracy of 92.5% + 0.7% (averaged over 3 training runs). When reproducing a similar experiment (training performed with SGD optimizer, with early stopping), with the simplified SampleCNN using 16 kHz data, we found the average accuracy be 93.34% + 1.26%.
https://gi cot.t/..oomaaeeeeeeeeeeeeo hatmm/aa-juunn jun/ samp eaudi

---

Under review as a conference paper at ICLR 2020
We created SpectroBank-enatiee version of SampleCNN, replacing the first block by spectrobank layer and modifying the other basic blocks introduced by/Kim et al. (2019), as described in appendix|Bi table 8 The SpectroBank layer is made of 80 order-4 Gammatone filters, overlapping by 80% and having a length representing 10 ms. As our initial layer contains less filters than the initial implementation (80 128), the basic block modifications allow to keep non-empty sizes when the number of basic blocks increases. The number of basic blocks is identical (7), reducing the number of trainable parameters to I million. Unlike the original paper, this network was trained using the Adam optimizer, while keeping the same learning rate reduction strategy. The early stopping is usually activated after less than 20 epochs. The mean accuracy achieved using this network improves slightly to 93.45% + 1.35%.
3.4 URBANSOUNDSK RESULTS
One of the main interests of this dataset resides in the fact that the environmental sounds exhibit spectral characteristics that are quite different from speech datasets studied in the previous sections. It however more challenging dataset, firstly because its size is almost an order of magnitude smaller, and secondly because of the longer input data (each sample being 4 seconds long).
We base our experiments on the works from Dai et al. (2017) and more recently Abdoli al. (2019) that also perform classification task using convolutional networks on raw audio input. Dai et al. (0117 define several network architectures, with numbers of trainable parameters ranging from 200 000 to millions. We will focus on the two smallest networks, referred to as M3 and M5 in the original paper. Despite dataset being split into 10 folds for training and validation, only one test (using the 10th fold for validation) has been done in Dai et al. (2017). We tested those networks using an existing Keras implementation and performed 10-fold validation to get the mean accuracy over all folds, using data resampled to 8 kHz. The average accuracy for M3 was found to be 58.94% + 3.83% (vs. 56.12% in the original paper) and the one for M5 66.98% = 6.37% (vs. 63.4% initially).
SpectroBank-enattee enabled versions of M3 and M5 have been created for comparison. The first layer consists in 24 4th-order Gammatone filters, overlapping by 75% and having a length representing 10 ms. All networks were trained for 100 epochs using the Adam optimizer, reducing the learning rate by a factor after 10 epochs without improvement of the validation loss. Spectro M3 accuracy is very close to the one achieved with initial M3, namely 59%17% = 5.33%. However, the SpectroBank-MF has ca. 22'000 parameters, i.c. close to ten times less than initial M3. In the case of SpectroBank-MS, mean accuracy is improved trainable parameters very close to initial M5, i.e. slightly more than 500'000). We also tested the SpectroBank-Sumphl
67.45% + 5.48% (with a number of
architecture described in section 3.3 and achieved a mean accuracy of 69.16% + 5.95%. When comparing more specifically the 10-th fold best accuracy achieved by Dai jtt al. (2017) is 71% % using M18 model (3.7 million parameters), while our approach reaches an accuracy of 75.8%. Higher accuracies have been achieved on this dataset using raw audio (Li et al.) 2018). they however resort to data augmentation, which was not used in our experiments. All results summarized in table
Table 2: UrbanSound8K mean test accuracy
Network
# Trainable parameters
Avg. accuracy
M3
222 k
58.94% + 3.83%
SpectroBank-MB
22.5 k
59.17% + 5.33%
M5
561 k
66.98% + 6.37% 67.45% 5.48% 69.16% + 5.95%
SpectroBank-M5
513 k
SpectroBank-SampleCNNN
1M
The approach taken by Abdoli et al. (201)) is to perform classification on overlapping splits of initial audio data (usually having a length of second). They however single block of data (having a length of ca. 3 seconds). While the code was supposed to be made
also compare to network taking
wwwwwwveeooro
7

---

Under review as a conference paper at ICLR 2020
available after final publication, the repository was still empty. The model was then reimplemented and trained according to the description found in the paper. using all 4 seconds of input data instead of trimming it to 3 seconds. Instead of the mean aceuracy claimed (83% = 1.3%, from Table 2 in original paper), our tests only achieved 63.8% + 5.68%, which is significant difference. We have been unable so far to explain this discrepancy.
3.5 PERFORMANCE CONSIDERATIONS
While having a smaller number of trainable parameters compared to a classical convolutional layer, the SpectroBank layer is slightly more complex than simple convolution, as it requires compute the filter coefficients from the parameters, then perform the convolution. Our implementation showed SpectroBank was roughly 50% slower compared convolution layer having the same number of filters. This performance drop is however mitigated by the fact that the bigger stride (usually an order of magnitude bigger) used in the experiments decreases the overall number of convolution operations required, resulting in faster training times, even when using non-logarithmic or random frequency initialization scheme.
3.6 PROPERTIES OF LEARNED FILTERS
The learned parameters of the SpectroBank filters can reveal insights about the data and the learning process. As stated in the introduction, several studies have shown a tendency governing the spacing in frequency of their learned kernels, approximations of band-pass filters. The spacing becomes exponentially large as the frequency increases, following what is called a Mel scale. This is in agreement with psycho-acoustics tests on the human cochlear system. In order to go further in this direction, we investigate 1) the frequency spacing and 2) we test the relationship between the temporal width of the filters and their central frequency. Indeed, rectangular bandwidth (ERB) model (Glasberg & Moore (1900) and the Bark model (Zwicker & Terhardt 1980)) provide such relationship. This is made possible by our approach where the temporal width as well as the filter central frequency are well defined for each filter.
aaycho-acoustic models (the equivalent
Frequency spacing. The SpectroBank layer is initialized with linear frequency spacing from 0 to the Nyquist frequency. After the learning phase. the filter frequencies have evolved and moved away from their initial value as be seen Fig. 2a The frequency distribution is not exponential but we can point out several interesting facts. Firstly, the final curve is flatter than the initialization in the range 0-2kHz (more filters in this range). It shows that the network tends to favour filters with a band-pass in this range for its discriminative process. Secondly, beyond 4kHz, the filters stay close to their original value. This suggests that there not enough meaningful information in this frequency range for a correct learning. This is indeed the case for speech where the main information resides below 4kHz (see Appendix
Bandwidth and frequency. The learned filter banks can be compared to filter banks modeling the human auditory system. Two main models can be found in the literature, the Equivalent Rectangular Bandwidth (ERB) model (Glasberg & Moore, 1990). and the Bark model (Zwicker & Terhardt] 1980). In these models the bandwidth B of the filter is related to its central frequency f by explicit formulas given in Appendix D The ERB and Bark curves plotted on Fig. 2b together with the learned parameters of the Gammatone filters (black dots). We observe a very good agreement between the ERB curve and the learned filters for frequencies below 2kHz. Ravanelli & Bengio (2018) show that for a neural network applied to a speech dataset, the focus of the learning is situated around the pitch frequency located at 130Hz (male) and 230Hz(female), and the first and second formants, which around 500Hz and IkHz respectively. This is exactly the frequency region where our learned filters match the ERB scale.
Cumulative distribution. In Figure 3 we can see the cumulative energy distribution, in the frequency domain, of the learned filters for Gammatone and Sine filters. We have used 32 filters during the training. From the Gammatone distribution, we can observe that filters focus on at least some of the frequencies relevant for speech, as discussed earlier, in the range 100Hz. 1kHz. The sinc distribution has the same global shape as in Ravanelli & Bengio (2018) but is less conclusive about the formants.
ht / / thub. com/ sa jabdo1 1 1twpnnneentttt_n_n_____e_ee
_ss_s_ddddee __ lasi f ca last ac-
cessed Nov 15th 2019
8

---

Under review as a conference paper at ICLR 2020
0.05
8000
GoogleSpechCommands
7000
Intial
frequercy (Htz]
6000
5000
bardwidh
0.03
4000
3000
2000
1000
0.00
1250
1752
filters
(b) Bandwidth frequency learned filters (black dots) over the range 0-2kHz The curves are the psycho- acoustical relationships given by the ERB and Bark scales. Black dashed line: initial bandwidth value for all filters.
(a) Frequency distribution of the filters before (straight line) and after training (green curve)
Bandwidth and frequency of the learned Gammatone filters (B of Eq. (4) and f parameters) using the Google Speech Commands dataset
Figure 2:
also note difference in the low- low-frequency region below 100Hz, where the distribution drops in our case. We point out that both our dataset and classification task are different, which could explain the discrepancies. It still shows the high focus of the filters on the range 100Hz IkHz, where the distribution curve is the highest.
1000 1500 2000 2500 3000 3500 4000 500 1000 1500 2000 2500 3000 3500 4000 frequency [Hz]
frequency [H2]
Figure 3: Cumulative frequency energy distribution for learned filters on AudioMNIST dataset, SpectroBank-XS network trained with Gammatone (order 4) and SincNet first layer, with 32 filters.
CONCLUSION
Decades of research in audio signal processing have brought us an important knowledge about sounds, speech and audio information. This knowledge may be inserted within neural networks as priori information and turned into efficient inductive biases. This is what show with the example of the SpectroBank layer, a layer of parameterized filters adapted to the extraction of audio information. Moreover, the trained network possesses properties than can, in turn, bring new insights about audio data back to the audio signal processing community.
Future work in this direction and further developments of convolutions with parameterized functions may lead to important progress both in deep learning and audio signal processing. The reduction of the number of trainable parameters decreases the network complexity, along with the training time. also enables better interpretation of the network adaptation to the data.
REFERENCES
Sajjad Abdoli, Patrick Cardinal, and Alessandro Lameiras Koerich. End-to-end environmental sound classification using 1D convolutional neural network. Expert Systems with Applications, 2019.

---

Under review as a conference paper at ICLR 2020
Joakim Andén and Stéphane Mallat. Deep scattering spectrum. IEEE Transactions on Signal Processing, 62(16):4114 4128, 2014.
Deepak Baby and Sarah Verhulst. Machines hear better when they have ears. arXiv preprint arXiv:1806.01 145,, 06 2018.
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Rober Müller, and Wojeiech Samek. Interpreting and explaining deep neural networks for classification of audio signals. CoRR, abs/1807. 03418, 2018.
http:///aaxii..oora/////////// 03418
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872-1886, 2013.
Pritish Chandna, Marius Miron, Jordi Janer, and Emilia Gómez. Monoaural audio source separation using deep convolutional neural networks. In International conference on latent variable analysis and signal separation, pp. 358-266..3 Springer, 2017.
Keunwoo Choi, György Fazekas, Kyunghyun Cho, and Mark Sandler. A tutorial deep learning for music information retrieval. arXiv preprint
2017.
Fergal Cotter and Nick Kingsbury. A learnable scatternet: Locally invariant convolutional layers. arXiv preprint 190vv:9033.031.7 2019.
Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das. Very deep convolutional neural networks for raw waveforms. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 421-425. IEEE, 2017.
AM Darling. Properties and implementation of the gammatone filter: a tutorial. Speech Hearing and Language, Work in Progress, University College London, Department Phonetics and Linguistics, pp. 43-61, 1991.
Sander Dieleman and Benjamin Schrauwen. End-to-end learning for music audio. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6964-6968. IEEE, 2014.
Brian R Glasberg and Brian CJ Moore. Derivation of auditory filter shapes from notched-noise data. Hearing research, 47(1-2):103-133 1990.
Pavel Golik, Zoltán Tüske, Ralf Schlüter, and Hermann Ney. Convolutional neural networks for acoustic modeling of raw time signal in Ivesr. In Sixteenth annual conference of the international speech communication association, 2015.
Volker Hohmann. Frequency analysis and synthesis using a gammatone filterbank. Acta Acustica united with Acustica, 88(3):433 442, 2002.
Yedid Hoshen, Ron Weiss, and Kevin W Wilson. Speech acoustic modeling from raw multichannel waveforms. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4624-4628. IEEE, 2015.
Toshio Irino and Roy Patterson. A time-domain, level-dependent auditory filter: The gammachirp.
The Journal of the Acoustical Society of America, 101(1):412-419, 1997.
Jacobsen, Jan van Gemert, Zhongyu Lou, and Arnold WM Smeulders. Structured receptive fields in cmns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 26 2610-2619, 2016.
Haidar Khan and Bulent Yener. Learning filter widths of spectral decompositions with wavelets. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 1601-4612, Curran Associates, Inc., 2018.
T. Kim, J. Lee, and Nam. Comparison and analysis of samplecnn architectures for audio classi- fication. IEEE Journal of Selected Topics in Signal Processing, 13(2):285-22 297, May 2019. doi: 10.1 N0.1109//SSSP.2199.99990 9,2999479.
10

---

Under review as a conference paper at ICLR 2020
Jongpil Lee and Juhan Nam. Multi-level and multi-scale feature aggregation using pretrained convolutional neural networks for music auto-tagging. IEEE signal processing letters, 24(8): 1208-12122 2017.
Shaobo Li, Yong Yao, Jie Hu, Guokai Liu, Xuemei Yao, and Jianjun Hu. An Ensemble Stacked Convolutional Neural Network Model for Envirommental Event Sound Recognition. Applied Sciences, 8(7):1152, July 2018. doi: 10.3390app80771 152. URL URLhttps: /www://wwww/...///////////ww... mdpi com/
2076-3417/8/7/121
Erfan Loweimi, Peter Bell, and Steve Renals, On learning interpretable cnns with parametric modulated kernel-based filters. In Proc. Interspeech, pp. -480-3484, 2019.
Xugang Lu, Yu Tsao, Shigeki Matsuda, and Chiori Hori. Speech enhancement based on deep denoising autoencoder. In Interspeech, pp. 436-440 2013.
Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single- channel speech separation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 696-700. IEEE, 2018.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.0344
2016.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. arXiv preprint arXiv: 2017.
Tom Le Paine, Pooya Khorrami, Shiyu Chang, Yang Zhang, Prajit Ramachandran, Mark A Hasegawa-Johnson, and Thomas S Huang. Fast wavenet generation algorithm. arXiv preprint arXiv:1611.00442, 2016.
Roy Patterson, Ian Nimmo-Smin John Holdsworth, and Peter Rice. An efficient auditory filterbank based on the gammatone function. In meeting of the 1OC Speech Group on Auditory Modelling at RSRE, volume 2, 1987.
Roy Patterson, KEN Robinson, John Holdsworth, Denis McKeown, C Zhang, and Michael Allerhand. Complex sounds and auditory images. In Auditory physiology and perception, pp.
429-446. Elsevier, 1992.
Karol Piczak. Environmental sound classification with convolutional neural networks. In 2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6.
IEEE, 2015.
Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schlüter, Shuo- Miio Chang, and Tara Sainath. Deep learning for audio signal processing. IEEE Journal of Selected Topics in Signal Processing, 13(2): 206-219, 2019.
Mirco Ravanelli and Yoshua Bengio. Speaker recognition from raw waveform with sincnet. 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 1021-1028. 2018.
Tara N Sainath, Brian Kingsbury, Abdel-rahman Mohamed, and Bhuvana Ramabhadran. Learning filter banks within a deep neural network framework. In 20/3 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 297-302. IEEE, 2013.
Tara N Sainath, Ron J Weiss, Andrew Senior, Kevin W Wilson, and Oriol Vinyals. Learning the speech front-end with raw waveform cldnns. In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
Justin Salamon and Juan Pablo Bello. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Processing Letters, 24(3): 2017.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041-1044. ACM, 2014.
11

---

Under review as a conference paper at ICLR 2020
Amin Saremi, Rainer Beutelmann, Mathias Dietz, Go Ashida, Jutta Kretzberg, and Sarah Verhulst. A comparative study of seven human cochlear filter models. The Journal of the Acoustical Society of America, (4003):1618-1634 2016. doi: 10.1121/1.490086. URL httpt:::///i.ooe/eeteo
1121/1 49604486
H. Seki, K. Yamamoto, and S. Nakagawa. A deep neural network integrated with filterbank learning for speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5480-5484, March 2017. doi: 10.1 10.1099ICASS22217.753330
Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. arXiv preprint arXiv:1806.03185. 2018.
Zoltán Tüske, Pavel Golik, Ralf Schlüter, and Hermann Ney. Acoustic modeling with deep neural networks using raw time signal for lvesr. In Fifteenth annual conference of the international speech communication association, 2014.
Pete Warden. Speech commands: A dataset for limited--vocabulary speech recognition. CoRR, abs/1804.03209, 2018. URL
Neil Zeghidour, Nicolas Usunier, Iasonas Kokkinos, Thomas Schaiz, Gabriel Synnaeve, and Em- manuel Dupoux. Learning filterbanks from raw speech for phone recognition. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5509-5513. IEEE, 2018.
Eberhard Zwicker and Ernst Terhardt. Analytical expressions for critical-band rate and critical bandwidth as function frequency. The Journal of the Acoustical Society of America, 68(5): 1525, 1980.
A FILTER BANK PARAMETERS
In this section we provide an example of visualization of the filter functions introduced for Spec- troBank. These functions and their parameters are recalled on Table 3] Their shape in time is illustrated on Fig. 4 with increasing oscillating frequency (or scale for wavelet) from blue to purple (starting from f 0).
Table 3: Deseription of the filter bank types and the parameters used during training. In most of our experiments, fixed to 4.
Filter Type
# of parameters
Parameters
Wavelet
scaling
Gaussian
2
frequency width
Gammatone
3
frequency, b bandwidth, 7 order
Gammachirp
f- frequency, b bandwidth, e chirp trend
B NETWORK ARCHITECTURES
Detailed architecture for SpectroBank-enabled networks used in the experiments given in this section. All convolutional and dense layers use ReLU activation, except for the output layer using softmax.
12

---

Under review as a conference paper at ICLR 2020
Figure 4: Examples of filter banks in time domain. From left right: Wavelet filters, Gaussian filters (cosine modulation), Gammatone filters (envelope, cosine and sine modulations) and Gammachirp filters, for fixed bandwidth and different frequencies.
Table 4: SpectroBank custom architecture for AudioMNIST.
Layer
Output size
Input
8000 x
SpectroBank (32 filters, size 80, stride 20)
400 x 32
Convolution (32 filters, size 32, stride 2)
200 32
MaxPooling (stride 4)
50 x 32
Convolution (64 filters, size 16, stride 2)
25 64
Convolution (128 filters, size 8, stride 2) Convolution (256 filters, size stride 2)
13 x 128
7 x 256
MaxPooling (stride 4)
1 x 256
Dense (128)
128
Dropout 0.5
128
Dense (64)
Dropout 0.5
64
Dense 10
10
Table 5: SpectroBank-XS custom architecture for AudioMNIST.
Layer
Output size
Input
8000 x
SpectroBank (32 filters, size 80, stride 20)
400
32
MaxPooling (stride 4)
100 x 32
Dense (16)
16
Dropout 0.5
16
Dense 10
10
13

---

Under review as a conference paper at ICLR 2020
Table M3-SpectroBank custom architecture for UrbanSound8K.
Layer
Output size
Input
32000 x
SpectroBank (24 filters, size 80, stride 20)
1600 x 24
Batch Normalization
1600 x 24
MaxPooling (stride 4)
400 24
Convolution (256 filters, size 3, stride 1)
400 256
MaxPooling (stride 4)
100 256
Global Average Pooling
256
Dense 10
10
Table 7: M5-SpectroBank custom architecture for UrbanSound8K.
Layer
Output size
Input
32000
SpectroBank (24 filters, size 80, stride 20)
1600 x 24
Batch Normalization
1600 x 24
MaxPooling (stride 4)
400 x 24
Convolution (128 filters, size 3, stride 1)
400 128
Batch Normalization
400 128
MaxPooling (stride 4)
100 128
Convolution (256 filters, size 3, stride 1)
100 256
Batch Normalization
100 x 256
MaxPooling (stride 4)
25 256
Convolution (512 filters, size 3, stride 1)
25 x 512
Batch Normalization
25 512
MaxPooling (stride 4)
6 x 512
Global Average Pooling
512
Dense 10
10
Table 8: SampleCNNN-SpectroBall basic block. Choice of k: is detailed in Kim et al.
2019
Layer
Output size
Input
N
Convolution (k filters, size 4, stride 1) Batch Normalization
N k
N
k
MaxPooling (stride 2)
N
2
14

---

Under review as a conference paper at ICLR 2020
Table SampleCNN--SecttoBBin architecture for Google Speech Commands (n 35) Urban-
sound8K (n 10). BB stands for 'Basic Block', and GMP for 'Global Max Pooling'
Layer
Output size
Input
16000
SpectroBank (80 filters, size 160, stride 40)
200 x 80
Batch Normalization
200 x
BB 0 (k: 80)
100 x 80
BB (A: 80)
50 x 80
BB 2 (k: 160)
25 160
BB 3 (k: 160)
12 x 160
BB (k: 160)
6 160
BB (k 160)
3 X 160
BB (A: 320)
320
Concatenate (GMP(BB 4), GMP(BB 5), GMP(BB 6))
640
Dense
640
Batch Normalization
640
Dropout (0.25)
640
Dense
C DATASETS
The overall statistics for all the datasets used in the experiments is given in Table T0 In addition the distribution of spectral energy per class is provided on Fig. Most of the speech energy is located in the 0-1.5kHz band.
Table 10: Class statistics over different datasets.
Database
of samples
# of classes
largest class size
smallest class size
AudioMNIST GoogleSpeccCComandds
30000
10
3000
3000
105829
35 10
4052
1557
UrnnnSo
9732
1000
374
D ERB AND BARK SCALES
Two main models auditory filter bank system provide the expression of filter bandwidth B with respect to its frequency f. In the Bark model (Zwicker & Terhardt 1980) the expression is the following:
f
B,(f) = 25 75 1.4
10.69.
(6)
1000
and in the ERB scale (Glasberg & Moore
1990):
BÆRE(S) = 24.7(4.37f + 1).
(7)
These expression are the ones used in the present work.
In addition, these auditory models provide expressions for the frequency spacing between consecutive filters, that follow logarithmic law. For given filter number k in the set of filters, its frequency can
be obtained by using the following formula: f = 228.888800 246 (ct /9.265 - 1). This relationship is more often expressed in terms of function of the frequency:
KERB 9.265 log (1
(8)
228.846
15

---

Under review as a conference paper at ICLR 2020
one
two
three
four
seven
egit
nine
500
1000
1500
2000
2500
3000
3500
4000
1000
1500
2000
2500
3000
3500
4000
frequency [Hiz]
frequeency
(a) AudiOMNIST digit label
(b) AudioMNIST gender label
conditioner
children playing dog bark drillng
three
four
5ix
gun, shot jickhammer siren
up
down
left
right
500
1000
1500
2000
2500
3000
3500
4000
2000
4000
6000
8000
10000
frequency [Hz]
frequency
(c) GoogleSpeechCommands
(d) UrbanSound8K
Figure 5: Dataset energy distribution per class and corresponding labels.
The Bark model has a similar expression:
hb 13 arctan 0.76 1000
f
3.5 arctan
7500
One can also compare with the Mel-scale. Sampling linearly on the Mel-scale m leads to logarithmic frequency sampling:
1127 In (1 (1 700
f
(10)
16


Reviews:
[{"review_id": "rkge3Hj6tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to specify the first layer of a CNN for audio applications with predefined filterbanks from the signal processing community. Those latter are only specified by a limited number of parameters, such as the bandwidth or the central frequency of the filter, and those parameters are then optimized through the standard back-propagation algorithm. Some accuracy improvements are obtained on non trivial datasets.\n\nI think there are a lot of interesting ideas and the numerical improvements seem consistent with the method. However, I find that this study would benefit of more careful comparisons to understand which particular component is responsible for some of their success! Also, I think some relevant papers are missing in the introduction.\n\n\nPros : \n- Good numerical performances.\n- Interesting study of the impact of predefined filters; an analysis at the end of the paper(bandwidth, principal frequency chosen by the algorithm) is shown, which is a positive aspect of the paper.\n\nCons :\n- Several attempts to employ hybrid architectures (as defined in the text) have been already proposed. References to hybrid architecture from Mallat's group are missing, e.g.:  https://arxiv.org/abs/1809.06367/ https://arxiv.org/abs/1605.06644 . Another line of work concerns the steerable filters, which is another manner to parametrized the filters and learn them (e.g.: https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Jacobsen_Structured_Receptive_Fields_CVPR_2016_paper.pdf ) Another manner could be to directly learn the filters as wavelets: https://arxiv.org/pdf/1811.06115.pdf . I agree some of those references are only considering images, but those methods are definitely not specific to them.\n- Comparing the number of parameters of hybrid and non hybrid architectures is meaningless in this setting, as in all the experiments, the number of parameters of the layers above the first layer are kept identical: one only sees the difference due to the first layer, whose kernel is indeed relatively high-dimensional.\n- Also, my understanding is that the general pipeline is slower: indeed, a parameter update aims to compute @f/@w=@f/@x*@x/@w. The computation of the term @f/@x is unavoidable and is identical to its non-hybrid counter-part. However, @x/@w might be sometimes higher because the computations can involve potentially more complex functions(e.g., exponential, cos, sin contrary to linear functions). Would you mind to clarify this thought?(A small fair timing comparison would be welcome!)\n- Furthermore, the improvement in performances is clearly thanks to those a-priori incorporated. As stated in the text, many works propose to initialize the CNN with a specific filter bank. Have the authors tried to compare their performances if the first layer is simply initialized with those filters and then freely evolve? I feel this is missing and would make the claim of the paper stronger. If this has been already done, please highlight it in the text.\n- Abstract: it is claimed that this technique leads to a training speedup (i.e., less epochs) but I do not understand where this is shown.\n- Section 3: Sometimes(e.g., AudioMnist), the hybrid training pipeline is quite different from the original implementation, for instance, because of the use of ADAM when the original implementation was using SGD. Did the use of a different optimizer affect the performances?(e.g., SGD?)\n- Section 3.4: Why not comparing with data augmented settings?\n\nSuggestions of improvement:\n- I would have liked to see a littlewood-paley plot (eg, the sum of the modulus of the filters in the frequency domain) to understand better the distribution of the filters in the Fourier domain, in particular w.r.t. the high-frequency.\n- \"the output maybe too redundant\"(page 5) - I don't understand why this would be an issue? In this case, the network should decide which coefficients to discard if the classifier is good enough, shouldn't it?\n\nPost-discussion:\nR1 made several relevant comments about the technical novelty and my concerns weren't fully solved. Thus I decided to maintain my score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BklbJDXJqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an interesting signal processing-based extension of CNNs, where the first layer convolution is replaced by some pre-defined filter banks. Since those filter banks are parameterized with a smaller number of parameters, while they have been proven to be effective in audio processing, I was convinced that this approach could produce better performance than a generic CNN with no such consideration. \n\nI am still wondering though, what is the main difference between this approach and Wavelet transform-based scatter transform networks that Stephane Mallat has proposed for years, for example in (Andén and Mallat 2014). I figure the proposed method in this paper is more flexible as it does not use the pre-defined filterbanks; instead it tries to learn the parameters to specify the only necessary filters for the particular problem. But I think the authors may need to address the difference from this previous work done by Mallat's group, because they at least share a similar philosophy. \n\nAnother thing that's not entirely clear for me was the effect of the filter length. Obviously, it should depend on the particular classification problem. For example, for speech, there needs to be consideration about the shortest stationary period of speech, while in some other cases like music and urban sound, it should be in different lengths to capture the specifics. It's a bit hard for me to believe that the different choices of filter banks from 1 to 100 ms all gave the same results (in Figure 1b). I think, if there is an optimal filter length depending on the problem, which has to be found to guarantee the performance, it has to be better investigated in the paper. \n\nIt is a confusing message to me, because the paper claims that the first layer of their network can cover a large area, which responds to a large receptive field, with a single filter by using a different parameter. It is a clearly a different kind of observation than the computer vision networks where the large receptive fields are defined with a deeper architecthre and strides. However, the shortest filter (1ms) and the longest one (100ms) doesn't make any difference, empirically? More discussion is needed to resolve this confusion.\n\n\nJ Andén and S Mallat, \"Deep scattering spectrum\", IEEE Transactions on Signal Processing, 2014", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgpQEuhtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new type of convolutional layer for (band-pass) filtering of input signals (e.g., audio recordings). The main benefit is that the layer can be specified with a small number of parameters (+ filter configurations that are typically fixed beforehand get to be tuned to improve inductive bias). This is achieved via modulated windows or wavelets. While this is interesting, I do not see any conceptual novelty. Namely, previous work has already proposed and considered such layers. More specifically, in [1] the authors have considered exponentially-modulated Gaussian windows (detailed experiments, influence of different initialization strategies, properties of learned filters, distribution of modulation frequencies etc.). In [2] the layer is realized using wavelets. In [3] the filter is expressed as a difference between two sinc functions. The authors might argue that the conceptual difference compared to [1] is cosine modulation (see Remark 2 on page 4). Well, cosine modulated filters were considered in [4] as Parzen filters (v1 was on arXiv in June 2019). The latter work has not even been cited by the authors. Moreover, the paper does not discuss the consequences of using cosine modulations instead of exponentials. Section 2.2 in [4] explains why the use of cosine modulations is well suited for real-valued signals. In particular, the moduli of Fourier coefficients are symmetric around the origin for real-valued signals and for this reason spectrograms are typically computed over positive frequencies only. Thus, from this perspective it does not make much difference whether one uses cosine or exponential modulation (when it comes to standard feature extraction approaches for speech processing).\n\nIn the empirical evaluation the focus is on showing the utility of filter optimization on different tasks. The first experiments investigates basic properties such as how the number of filters and their overlap influence the effectiveness of a model. It is unclear why a single learning task is sufficient to conclude that more than 30 filters does not amount to an improvement in accuracy (128 and 64 filters are used in [3] and [4], respectively). This lack of reference to findings in previous work make the analysis incomplete. The approach is evaluated in total on three datasets: audio-mnist, google speech command, and urban-sound. While the reported results indicate a good performance of the considered approach over different tasks, the experiments completely ignore previous approaches for filter learning. This lack of baselines and reference to related work makes the experiments inadequate.\n\nIn general, my main concern with the experiments is that the section is written as if this is the first work proposing filter learning. I feel that a comparison to at least on of the baselines [1-4] would be required for a non-trivial assessment of the approach.\n\n\n[1] N. Zeghidour, N. Usunier, I. Kokkinos, T. Schatz, G. Synnaeve, and E. Dupoux (ICASSP 2018). Learning filterbanks from raw speech for phone recognition.\n[2] H. Khan and B. Yener (NIPS 2018). Learning filter widths of spectral decompositions with wavelets.\n[3] M. Ravanelli and Y. Bengio (arXiv:1812.05920 2018). Speech and speaker recognition from raw waveform with SincNet.\n[4] D. Oglic, Z. Cvetkovic, P. Sollich (arXiv:1906.09526 2019). Bayesian Parznets for Robust Speech Recognition in the Waveform Domain.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a parameterized convolution layer using predefined filterbanks. It has the benefit of less parameters to optimize and better interpretability. The original submission failed to inlcude many related work into the discussion which was addressed during the rebutal.

The main concerns for this paper is the limited novelty and insufficient experimental validation and comprisons: 
* There have been existing work using sinc parameterized filters, learnable Gammatones etc, which are very similar to the proposed method. Also in the rebutal, the authors acknowledged that "We did not claim that cosine modulation was the novelty in our paper" and it is "just a way of simplifying implementation and dealing with real values instead of complex ones" and "addressing the question of convergence of parametric filter banks to perceptual scale".
* Although the authors addressed the missing related work problem by including them into discussions, the expeirmental sections need more work to include comparisons to those methods and also more validations on difference datasets to address the concern on the generalization of the proposed method. 
Decision: Reject