Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a learnable frontend for classification tasks on audio signals. The proposed learnable audio frontend (LEAF) is a generalization of a mel filterbank, used commonly in machine audition.
LEAF consists of a Gabor filterbank, magnitude-squared nonlinearity, Gaussian lowpass filter and previously-proposed per-channel energy normalization. Learnable parameters of LEAF are the center frequencies and bandwidths of Gabor filters, bandwith of lowpass Gaussian filters and smoothing coefficients of sPCEN. The proposed LEAF matches the performance of competing frontends in most of the cases and leads to some improvements in some cases.
In general, I like the paper and the proposed frontend is very sensible from perspective of audio-related applications.
However, I believe there’s some exaggeration in terms of impact of the proposed frontend based on the results in this paper. The proposed system is not actually challenging status quo, as many learnable frontends have been proposed in the literature in the past (as also listed in the references).
The paper is easy to read and authors communicate their contribution clearly.
However, I believe that the title may be somewhat too general: LEAF is evaluated only on classification tasks, and IMHO that should be indicated in the title as well. There are other tasks, such as speech enhancement, where LEAF-like frontend may work well, but that is out of scope of this paper.
Experimental results show that LEAF is performing well in the considered tasks. However, it would be interesting to understand the differences in performance when the encoder & head are changed and/or increased.
More specifically, the current results are obtained using a lightweight EfficientNet and linear heads. Is there any particular reason for this setup? Would the conclusions change with a different encoder/head?
Also, understanding of the influence of the filterbank setup (number of channels, window length, stride) would be beneficial.

Details:
(1) Title should reflect the fact that LEAF has been evaluated only on classification tasks
(2) Abstract: “over a wide range of audio domains” —> It would be more appropriate to talk about a range of applications in audio domain.
(3) Abstract: “unprecedented” -> I believe this is a bit exaggerated
(4) Introduction: “this might not be the optimal approach for non-human sounds” —> This is a strange argument, and I believe the authors are confusing sound perception and sound production. Human sound perception works quite well for recognition on non-human sounds. The authors imply that a system which replicates a system mimicking human perception is suboptimal for non-human sounds. However, human auditory system is not designed for processing of only human-made sounds. Furthermore, optimality depends on the application, so stating that something is not optimal for a class of sounds makes no sense without the defined application, which in this case could be recognition of “acoustic events or animal vocalizations”
(5) Introduction, last paragraph: “wide and diverse range of tasks, including speech, music, audio events, and animal vocalizations” —> Signals, such as speech or music are not tasks. A task can be, e.g., speech recognition.
(6) Conclusion: Stating there’s a “historical statu quo of using hand-crafted mel-filterbanks” with so many end-to-end systems giving the best performance in different applications is a bit too much.


======= Review edit after authors' revisions ====== 

Most of my concerns have been resolved in the significantly-improved revised version of the paper.
review_title: A simple but interesting frontend for audio classification tasks

final_score: rating: 7: Good paper, accept
review_content: The paper shows a detailed interpretation on the relationship between each component of hand-crafted audio front-ends (such as mel-spectrograms) and learnable counterparts. To do that, they followed the narratives presented from the previous works such as SincNet and improved the model by changing the several components of it. The authors grouped the audio front-ends into mainly three parts which are filtering, pooling, and compression. And, the contributions were made at each stage. For filtering stage, instead of learning all the parameters of the convolution layer, they let the model to learn only center frequency and bandwidth of the filterbanks that are initially assigned with Gabor filters. For pooling stage, instead of using simple average or max poolings, they let the model to learn low pass filtering with small parameters. For compression, instead of using log based dynamic compression, they extended Per-Channel Energy Normalization by replacing a fixed smoothing factor to learnable parameters and named it to sPCEN.

To evaluate the proposed approach, they evaluated the models on 8 audio classification tasks which might have diverse audio and label characteristics (such as acoustic scene sound, animal sound, music, speech). The compared models are mainly mel-spectrogram and SincNet. The results shows that the proposed model outperforms the comparisons for most tasks. Then, they further ran a multi-task classification experiment to obtain universal audio front-ends. And, the results show the proposed learnable front-ends is showing some generalization ability on most tasks. Finally, they evaluated the proposed model on large-scale audio classification dataset (AudioSet) and verified that the proposed front-ends is also showing the good performance on it.

In page 4, the authors mentioned that the l2 normalization helps distinguishing the role of filtering and compression. I think this contribution is not trivial, so if the authors can add more experiments (or plots) to show the difference between models with and without l2 normalization, then it would be helpful.

The backbone model used in the paper is fixed, and showing that the proposed audio front-ends shows similar trends with multiple backend models can verify better generalization ability of the proposed approach. So, if the authors can add additional experiments with multiple backends, then it would be helpful.

review_title: The paper well connects the relationship between hand-crafted audio frontends (mel-spectrogram) with learnable frontends.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a new learnable representation fo audio signal classification and compares it to the classical mel-filterbanks representation and two other learnable representations on a broad range of audio classification tasks, from birdsongs to pitch, instrument, language or emotion recognition. The proposed representation combines several parameterized representation techniques from the recent litterature. It is reported to yield on par or better classification results than the other methods on several of these tasks using single- or multi-task learning.

Pros:
- Learning an ultimate, universal, generic representation for all audio signals that renders the 80 years old mel-frequency scale obsolete is certainly an attractive goal
- The proposed representation carefully and elegantly combines the best parts of several recently proposed parameterized representations and enjoys a nice interpretability while requiring few parameters to learn.
- Comparing different audio representations on such a broad range of audio classification task is a welcome and unmatched effort, to the best of my knowledge.

Cons:
- The paper lacks humility in its story-telling and its style. It employs formulations such as "lived through the history of audio", or "challenging the historical statu quo" when refereing to mel-frequency representations, although by the authors' own admition in the paper, a large amount of research effort has already been given in recent years towards learnable audio representations (the authors cite a dozen papers but there are more). Hence, this paper is not a first attempt. And despite the pompous use of "universal" in the title, I believe it is not a last attempt either. The authors claim that the proposed representation "outperform mel-filterbanks over several tasks with a unique parametrization" but this is far from clear when looking at the results carefully. In the majority of the tasks, the representation performs either slightly worse, equal, or about 0.5% better than Mel-filterbanks. It is not clear whether such improvement is significant, since no error bar or standard deviation is provided in the results (a sadly common habit in the audio litterature). The only tasks where a truly significant improvement is reported are language identification and emotion recognition, which are also the tasks where all the methods perform the poorest. It looks like any significant difference between the 4 compared approaches would vanish if these two tasks were omitted. The reason why the proposed representation performs well on these two very specific tasks is not clear and not discussed.
- More generally, the paper would be much more valuable if it gave a sense of WHAT is actually learned by the proposed method. Is the final representation significantly different from a mel-filterbank? Given how close to mel most reported results are, this is doubtful. In fact, Fig. A.1. strongly suggests that LEAF just re-learned mel, but strangely this figure is never commented. Some comments on the learned compression-parameters would also be appreciated.
- At least one important comparison point is clearly missing in the reported results: STFT + PCEN or mel-filterbanks + PCEN, e.g., Wang et al. (2017) or Schlüter & Lehner (2018) [note that the latter already uses sPCEN rather than PCEN, contrary to the authors' claim] . Omitting this from the comparisons prevents one from knowing whether the proposed parameterized Gabor filterbank brings any advantage over another time-frequency representation like STFT or mel-filterbanks. Less critically, another missing comparison point is LEAF + CNN14, in Table 4.
- What the authors refer to as "audio" in the title and throughout the paper is in fact much more narrow, namely "audio classification". Learnable audio representations have been studied in a broader context in recent years, e.g., speech enhancement, source separation, dereverberation, sound localization or audio (re-)synthesis. In fact, one of the important breakthroughs recently brought by learnable audio frontends was in source separation with the paper TasNet (Luo et al. 2018) which is not cited by the authors. In the same context, (Ditter and Gerkmann 2020) presented a learnable gammatone-like filterbank and showed that fully-parameterized learned filterbanks tended to have logarithmic spread in frequencies. Moreover, the use of learnable analytical filterbanks/Hilbert pairs due to their envelop extraction/shift invariant properties was already discussed in depth in (Pariente et al. 2019) [cited in the paper]. 

Overall, while comparing different learnable audio representations on a broad range of audio classification tasks is a timely and worthwhile topic, and while the proposed representation elegantly combines several recent ideas in this area, the general presentation and angle of the paper strongly lacks humility. Instead of the proposed title, something like "Benchmark of learnable audio representations on a broad range of classification tasks" would be more truthful to the work. To make the investigation more worthwhile and insightful, additional comparison points (STFT + PCEN, mel-frequency + PCEN, Gabor + log, etc.) as well as an analysis of what the model has actually learned would be needed.

======= Review edit after authors' revisions ======
The changes made by the authors in the title, abstract, introduction and conclusion to narrow the scope of the paper, better contextualize it, and make it more humble and truthful are very welcome. The extra experiments, figures, addition of error bars and new statistical tests are also a real plus. In doing so, the authors addressed all of my major concerns.

For these reasons, changed my evaluation score from 5 to 8.

review_title: Worthwhile investigation but lack of humility and truthfulness

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this work, the authors introduce a learnable front-end (LEAF) for audio. The paper evaluates it on several tasks in the audio domain such as acoustic event classification, speaker identification, keyword spotting, language identification, music classification etc. The study reports that proposed features outperform or perform similar to Mel filterbanks for many tasks. This paper is well written and easy to follow. The references are excellent as well. I have summarized my comments below which will help in improving the quality of this manuscript: 

1. While the authors evaluate the proposed front-end on several tasks, the evaluation protocols are missing. Especially, the training partition and test partition and number of segments in each class. For example, did they calculate accuracy using balanced classes? Was classification done at the frame level or the utterance level? I would highly recommend authors to add the experiment protocol to the manuscript as Table 1A does not provide enough information. 

2. The authors evaluate LEAF on several tasks however the model architecture for classification models remains constant. Typically Mel FBs are utilized across architectures and one of the major shortcomings of the proposed work is the integration of LEAF with state-of-the-art models (for example x-vectors for SID). This would give more understanding about the generalization capability of the LEAF.

3. As pointed out by authors in the introduction, Mel FBs are invariant to deformation and noise-robust. All the audio problems addressed in this work encounter varying levels of noise in the real-world. It is extremely important to assess the noise robustness of the proposed front-end. I would recommend the authors test LEAF with different levels of noise.

4. For results in Table 1 the statistical significance should be computed. For the acoustic scene task, the difference in performance is very less and this dataset only has 810 samples overall for testing. Therefore, without confidence intervals, it is very hard to conclude the performance of LEAF. 

Minor comments:
1. Please add this missing reference: 
Davis, S. and Mermelstein, P., 1980. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE transactions on acoustics, speech, and signal processing, 28(4), pp.357-366.

2. In Table 1, Mel and LEAF has the same accuracy for Music instrument and needs to be bolded. 

3. mel ---> Mel 

4. In Fig 2, you may improve the font size of x-axis labels. 

review_title: Review for A Universal Learnable Audio Frontend

Meta-review: All Reviewers agree that the paper has a clear and solid contribution. Furthermore, all of them highlight that the paper has improved significantly after revision. Hence, my recommendation is to ACCEPT the paper. As a brief summary, I highlight below some pros and cons that arose during the review and meta-review processes.

Pros:
- Comparison across network architectures.
- Comparison across a broad range of different data sets.
- Compactness of the representation (few parameters to learn).
- Authors will share code.

Cons:
- Role of L2 normalization could be further discussed.
Decision: Accept (Poster)