Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper presents an interesting signal processing-based extension of CNNs, where the first layer convolution is replaced by some pre-defined filter banks. Since those filter banks are parameterized with a smaller number of parameters, while they have been proven to be effective in audio processing, I was convinced that this approach could produce better performance than a generic CNN with no such consideration. 

I am still wondering though, what is the main difference between this approach and Wavelet transform-based scatter transform networks that Stephane Mallat has proposed for years, for example in (Andén and Mallat 2014). I figure the proposed method in this paper is more flexible as it does not use the pre-defined filterbanks; instead it tries to learn the parameters to specify the only necessary filters for the particular problem. But I think the authors may need to address the difference from this previous work done by Mallat's group, because they at least share a similar philosophy. 

Another thing that's not entirely clear for me was the effect of the filter length. Obviously, it should depend on the particular classification problem. For example, for speech, there needs to be consideration about the shortest stationary period of speech, while in some other cases like music and urban sound, it should be in different lengths to capture the specifics. It's a bit hard for me to believe that the different choices of filter banks from 1 to 100 ms all gave the same results (in Figure 1b). I think, if there is an optimal filter length depending on the problem, which has to be found to guarantee the performance, it has to be better investigated in the paper. 

It is a confusing message to me, because the paper claims that the first layer of their network can cover a large area, which responds to a large receptive field, with a single filter by using a different parameter. It is a clearly a different kind of observation than the computer vision networks where the large receptive fields are defined with a deeper architecthre and strides. However, the shortest filter (1ms) and the longest one (100ms) doesn't make any difference, empirically? More discussion is needed to resolve this confusion.


J Andén and S Mallat, "Deep scattering spectrum", IEEE Transactions on Signal Processing, 2014
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes to specify the first layer of a CNN for audio applications with predefined filterbanks from the signal processing community. Those latter are only specified by a limited number of parameters, such as the bandwidth or the central frequency of the filter, and those parameters are then optimized through the standard back-propagation algorithm. Some accuracy improvements are obtained on non trivial datasets.

I think there are a lot of interesting ideas and the numerical improvements seem consistent with the method. However, I find that this study would benefit of more careful comparisons to understand which particular component is responsible for some of their success! Also, I think some relevant papers are missing in the introduction.


Pros : 
- Good numerical performances.
- Interesting study of the impact of predefined filters; an analysis at the end of the paper(bandwidth, principal frequency chosen by the algorithm) is shown, which is a positive aspect of the paper.

Cons :
- Several attempts to employ hybrid architectures (as defined in the text) have been already proposed. References to hybrid architecture from Mallat's group are missing, e.g.:  https://arxiv.org/abs/1809.06367/ https://arxiv.org/abs/1605.06644 . Another line of work concerns the steerable filters, which is another manner to parametrized the filters and learn them (e.g.: https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Jacobsen_Structured_Receptive_Fields_CVPR_2016_paper.pdf ) Another manner could be to directly learn the filters as wavelets: https://arxiv.org/pdf/1811.06115.pdf . I agree some of those references are only considering images, but those methods are definitely not specific to them.
- Comparing the number of parameters of hybrid and non hybrid architectures is meaningless in this setting, as in all the experiments, the number of parameters of the layers above the first layer are kept identical: one only sees the difference due to the first layer, whose kernel is indeed relatively high-dimensional.
- Also, my understanding is that the general pipeline is slower: indeed, a parameter update aims to compute @f/@w=@f/@x*@x/@w. The computation of the term @f/@x is unavoidable and is identical to its non-hybrid counter-part. However, @x/@w might be sometimes higher because the computations can involve potentially more complex functions(e.g., exponential, cos, sin contrary to linear functions). Would you mind to clarify this thought?(A small fair timing comparison would be welcome!)
- Furthermore, the improvement in performances is clearly thanks to those a-priori incorporated. As stated in the text, many works propose to initialize the CNN with a specific filter bank. Have the authors tried to compare their performances if the first layer is simply initialized with those filters and then freely evolve? I feel this is missing and would make the claim of the paper stronger. If this has been already done, please highlight it in the text.
- Abstract: it is claimed that this technique leads to a training speedup (i.e., less epochs) but I do not understand where this is shown.
- Section 3: Sometimes(e.g., AudioMnist), the hybrid training pipeline is quite different from the original implementation, for instance, because of the use of ADAM when the original implementation was using SGD. Did the use of a different optimizer affect the performances?(e.g., SGD?)
- Section 3.4: Why not comparing with data augmented settings?

Suggestions of improvement:
- I would have liked to see a littlewood-paley plot (eg, the sum of the modulus of the filters in the frequency domain) to understand better the distribution of the filters in the Fourier domain, in particular w.r.t. the high-frequency.
- "the output maybe too redundant"(page 5) - I don't understand why this would be an issue? In this case, the network should decide which coefficients to discard if the classifier is good enough, shouldn't it?

Post-discussion:
R1 made several relevant comments about the technical novelty and my concerns weren't fully solved. Thus I decided to maintain my score.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes a new type of convolutional layer for (band-pass) filtering of input signals (e.g., audio recordings). The main benefit is that the layer can be specified with a small number of parameters (+ filter configurations that are typically fixed beforehand get to be tuned to improve inductive bias). This is achieved via modulated windows or wavelets. While this is interesting, I do not see any conceptual novelty. Namely, previous work has already proposed and considered such layers. More specifically, in [1] the authors have considered exponentially-modulated Gaussian windows (detailed experiments, influence of different initialization strategies, properties of learned filters, distribution of modulation frequencies etc.). In [2] the layer is realized using wavelets. In [3] the filter is expressed as a difference between two sinc functions. The authors might argue that the conceptual difference compared to [1] is cosine modulation (see Remark 2 on page 4). Well, cosine modulated filters were considered in [4] as Parzen filters (v1 was on arXiv in June 2019). The latter work has not even been cited by the authors. Moreover, the paper does not discuss the consequences of using cosine modulations instead of exponentials. Section 2.2 in [4] explains why the use of cosine modulations is well suited for real-valued signals. In particular, the moduli of Fourier coefficients are symmetric around the origin for real-valued signals and for this reason spectrograms are typically computed over positive frequencies only. Thus, from this perspective it does not make much difference whether one uses cosine or exponential modulation (when it comes to standard feature extraction approaches for speech processing).

In the empirical evaluation the focus is on showing the utility of filter optimization on different tasks. The first experiments investigates basic properties such as how the number of filters and their overlap influence the effectiveness of a model. It is unclear why a single learning task is sufficient to conclude that more than 30 filters does not amount to an improvement in accuracy (128 and 64 filters are used in [3] and [4], respectively). This lack of reference to findings in previous work make the analysis incomplete. The approach is evaluated in total on three datasets: audio-mnist, google speech command, and urban-sound. While the reported results indicate a good performance of the considered approach over different tasks, the experiments completely ignore previous approaches for filter learning. This lack of baselines and reference to related work makes the experiments inadequate.

In general, my main concern with the experiments is that the section is written as if this is the first work proposing filter learning. I feel that a comparison to at least on of the baselines [1-4] would be required for a non-trivial assessment of the approach.


[1] N. Zeghidour, N. Usunier, I. Kokkinos, T. Schatz, G. Synnaeve, and E. Dupoux (ICASSP 2018). Learning filterbanks from raw speech for phone recognition.
[2] H. Khan and B. Yener (NIPS 2018). Learning filter widths of spectral decompositions with wavelets.
[3] M. Ravanelli and Y. Bengio (arXiv:1812.05920 2018). Speech and speaker recognition from raw waveform with SincNet.
[4] D. Oglic, Z. Cvetkovic, P. Sollich (arXiv:1906.09526 2019). Bayesian Parznets for Robust Speech Recognition in the Waveform Domain.

review_title: Official Blind Review #1

Meta-review: The paper proposed a parameterized convolution layer using predefined filterbanks. It has the benefit of less parameters to optimize and better interpretability. The original submission failed to inlcude many related work into the discussion which was addressed during the rebutal.

The main concerns for this paper is the limited novelty and insufficient experimental validation and comprisons: 
* There have been existing work using sinc parameterized filters, learnable Gammatones etc, which are very similar to the proposed method. Also in the rebutal, the authors acknowledged that "We did not claim that cosine modulation was the novelty in our paper" and it is "just a way of simplifying implementation and dealing with real values instead of complex ones" and "addressing the question of convergence of parametric filter banks to perceptual scale".
* Although the authors addressed the missing related work problem by including them into discussions, the expeirmental sections need more work to include comparisons to those methods and also more validations on difference datasets to address the concern on the generalization of the proposed method. 
Decision: Reject