Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The paper proposes a CNN-based based approach for speech processing using raw waveforms as input. An analysis of convolution and pooling layers applied on waveforms is first presented. An architecture called SimpleNet is then presented and evaluated on two speech tasks: emotion recognition and gender classification. 

This paper propose a theoretical analysis of convolution and pooling layers to motivate the SimpleNet architecture. To my understanding, the analysis is flawed (see comments below). The SimpleNet approach is interesting but not sufficiently backed with experimental results. The network analysis is minimal and provides almost no insights. I therefore recommend to reject the paper. 

Detailed comments:

Section 1:

* “Therefore, it remains unknown what actual features CNNs learn from waveform”. This is not true, several works on speech recognition have shown that a convolution layer taking raw speech as input can be seen as a bank of learned filters. For instance in the context of speech recognition, [9] showed that the filters learn phoneme-specific responses, [10] showed that the learned filters are close to Mel filter banks and [7] showed that the learned filters are related to MRASTA features and Gabor filters. The authors should discuss these previous works in the paper.

Section 2:

* Section 2.1 seems unnecessary, I think it’s safe to assume that the Shannon-Nyquist theorem and the definition of convolution are known by the reader.

* Section 2.2.2 & 2.2.3: I don't follow the justification that stacking convolutions are not needed: the example provided is correct if two convolutions are directly stacked without non-linearity, but the conclusion does not hold with a non-linearity and/or a pooling layer between the convolutions: two stacked convolutions with non-linearities are not equivalent to a single convolution. To my understanding, the same problem is present for the pooling layer: the presented conclusion that pooling introduces aliasing is only valid for two directly stacked pooling layers and is not correct for stacked blocks of convolution/pooling/non-linearity.

* Section 2.2.5: The ReLU can be seen as a half-wave rectifier if it is applied directly to the waveform. However, it is usually not the case as it is applied on the output of the convolution and/or pooling layers. Therefore I don’t see the point of this section. 

* Section 2.2.6: In this section, the authors discuss the differences between spectrogram-based and waveforms-based approaches, assuming that spectrogram-based approach have fixed filters. But spectrogram can also be used as input to CNNs (i.e. using learned filters) for instance in speech recognition [1] or emotion recognition [11]. Thus the comparison could be more interesting if it was between spectrogram-based and raw waveform-based approaches when the filters are learned in both cases.  

Section 3:

* Figure 4 is very interesting, and is in my opinion a stronger motivation for SimpleNet that the analysis presented in Section 2.

* Using known filterbanks such as Mel or Gammatone filters as initialization point for the convolution layer is not novel and has been already investigated in [7,8,10] in the context of speech recognition. 

Section 4:

* On emotion recognition, the results show that the proposed approach is slightly better, but there is some issues: the average recall metric is usually used for this task due to class imbalance (see [1] for instance). Could the authors provide results with this metric ? Also IEMOCAP is a well-used corpus for this task, could the authors provide some baselines performance for comparison (e.g. [11]) ? 

* For gender classification, there is no gain from SimpleNet compared to the baselines. The authors also mention that some utterances have overlapping speech. These utterances are easy to find from the annotations provided with the corpus, so it should be easy to remove them for the train and test set. Overall, in the current form, the results are not convincing.

* Section 4.3: The analysis is minimal: it shows that filters changed after training (as already presented in Figure 4). I don't follow completely the argument that the filters should focus on low frequency. It is more informative, but one could expect that the filters will specialized, thus some of them will focus on high frequencies, to model the high frequency events such as consonants or unvoiced event. 
It could be very interesting to relate the learned filters to the labels: are some filters learned to model specific emotions ? For gender classification, are some filters focusing on the average pitch frequency of male and female speaker ?

* Finally, it would be nice to see if the claims in Section 2 about the fact that only one convolution layer is needed and that stacking pooling layers can hurt the performance are verified experimentally: for instance, experiments with more than one pair of convolution/pooling could be presented.

Minor comments:

* More references for raw waveforms-based approach for speech recognition should be added [3,4,6,7,8,9] in the introduction.

* I don’t understand the first sentence of the paper: “In the field of speech and audio processing, due to the lack of tools to directly process high dimensional data …”. Is this also true for any pattern recognition fields ?  

* For the MFCCs reference in 2.2.2, the authors should cite [12].

* Figure 6: Only half of the spectrum should be presented.

References: 

[1] H. Lee, P. Pham, Y. Largman, and A. Y. Ng. Unsupervised feature learning for audio classification using convolutional deep belief networks. In Advances in Neural Information Processing Systems 22, pages 1096–1104, 2009.

[2] Schuller, Björn, Stefan Steidl, and Anton Batliner. "The interspeech 2009 emotion challenge." Tenth Annual Conference of the International Speech Communication Association. 2009.

[3] N. Jaitly, G. Hinton, Learning a better representation of speech sound waves using restricted Boltzmann machines, in: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011, pp. 5884–5887.

[4] D. Palaz, R. Collobert, and M. Magimai.-Doss. Estimating Phoneme Class Conditional Probabilities from Raw Speech Signal using Convolutional Neural Networks, INTERSPEECH 2013, pages 1766–1770.

[5] Van den Oord, Aaron, Sander Dieleman, and Benjamin Schrauwen. "Deep content-based music recommendation." Advances in neural information processing systems. 2013.

[6] Z.Tuske, P.Golik, R.Schluter, H.Ney, Acoustic Modeling with Deep Neural Networks Using Raw Time Signal for LVCSR,
in: Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), Singapore, 2014, pp. 890–894.

[7] P. Golik, Z. Tuske, R. Schlu ̈ter, H. Ney, Convolutional Neural Networks for Acoustic Modeling of Raw Time Signal in LVCSR, in: Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2015, pp. 26–30.

[8] Yedid Hoshen and Ron Weiss and Kevin W Wilson, Speech Acoustic Modeling from Raw Multichannel Waveforms, International Conference on Acoustics, Speech, and Signal Processing, 2015.

[9] D. Palaz, M. Magimai-Doss, and R. Collobert. Analysis of CNN-based Speech Recognition System using Raw Speech as Input, INTERSPEECH 2015, pages 11–15.

[10] T. N. Sainath, R. J. Weiss, A. Senior, K. W. Wilson, and O. Vinyals. Learning the Speech Front-end With Raw Waveform CLDNNs. Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2015.

[11] Satt, Aharon & Rozenberg, Shai & Hoory, Ron. (2017). Efficient Emotion Recognition from Speech Using Deep Learning on Spectrograms. 1089-1093. Interspeech 2017.

[12] S. Davis and P. Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech and Signal Processing, 28(4):357–366, 1980.
review_title: Review

final_score: rating: 3: Clear rejection
review_content: Summary: 

The authors aim to analyze what deep CNNs learn, and end up proposing “SimpleNet”, which essentially reduces the early feature extraction stage of the network to a single convolutional layer, which is initialized using pre-defined filters. The authors step through a specific example involving bandpass filters to illustrate that multiple layers of filtering can be reduced to a single layer in the linear case, as well as the limitations of pooling. Results on a 4-class speaker emotion recognition task demonstrate some advantage over other, deeper architectures that have been proposed, as well as  predefined feature processing.

Review:

The authors’ arguments and analysis are in my assessment rudimentary---the effects of pooling and cascading multiple linear convolutions are well appreciated by researchers in the field. Furthermore, the adaptation of “front-end” signal processing modules in and end-to-end manner has been considered extensively before (e.g. Sainath et al., 2015), and recent work on very deep networks for signal processing that shows gains on more substantial tasks have not been cited (e.g. Dai, Wei, et al. below). Finally, the experimental results, considering the extensive previous work in this area, are insufficient to establish novel performance in lieu of novel ideas.

Overall Recommendation: 

Overall, this paper is a technical report that falls well below the acceptance threshold for ICLR for the reasons cited above. Reject. 

review_title: Review of "How do deep convolutional neural networks learn from raw audio waveforms?"

final_score: rating: 2: Strong rejection
review_content: The paper provides an analysis of the representations learnt in convolutional neural networks that take raw audio waveforms as input for a speaker emotion recognition task. Based on this analysis, an architecture is proposed and compared to other architectures inspired by other recent work. The proposed architecture overfits less on this task and thus performs better.

I think this work is not experimentally strong enough to draw the conclusions that it draws. The proposed architecture, aptly called "SimpleNet", is relatively shallow compared to the reference architectures, and the task that is chosen for the experiments is relatively small-scale. I think it isn't reasonable to draw conclusions about what convnets learn in general from training on a single task, and especially not a small-scale one like this. 

Moreover, SoundNet, which the proposed architecture is compared to, was trained on (and designed for) a much richer and more challenging task originally. So it is not surprising at all that it overfits dramatically to the tasks chosen here (as indicated in table 1), and that a much shallower network with fewer parameters overfits less. This seems obvious to me, and contrary to what's claimed in the paper, it provides no convincing evidence that shallow architectures are inherently better suited for raw audio waveform processing. This is akin to saying that LeNet-5 is a better architecture for image classification than Inception, because the latter overfits more on MNIST. Perhaps using the original SoundNet task, which is much more versatile, would have lent some more credibility to these claims.

The analysis in section 2.2 is in-depth, but also not very relevant: it ignores the effects of nonlinearities, which are an essential component of modern neural network architectures. Studying their effects in the frequency domain would actually be quite interesting. It is mentioned that the ReLU nonlinearity acts as a half-wave rectifier, but the claim that its effect in the frequency domain is small compared to aliasing is not demonstrated. The claim that "ReLU and non-linear activations can improve the network performance, but they are not the main factors in the inner workings of CNNs" is also unfounded.

The conclusion that stacking layers is not useful might make sense in the absence of nonlinearities, but when each layer includes a nonlinearity, the obvious point of stacking layers is to improve the expressivity of the network. Studying aliasing effects in raw audio neural nets is a great idea, but I feel that this work takes some shortcuts that make the analysis less meaningful.




Other comments:

The paper is quite lengthy (11 pages of text) and contains some sections that could easily be removed, e.g. 2.1.1 through 2.1.3 which explain basic signal processing concepts and could be replaced by a reference. In general, the writing could be much more concise in many places.

The paper states that "it remains unknown what actual features CNNs learn from waveforms.". There is actually some prior work that includes some analysis on what is learnt in the earlier layers of convnets trained on raw audio: 
"Learning the Speech Front-end With Raw Waveform CLDNNs", Sainath et al.
"Speech acoustic modeling from raw multichannel waveforms", Hoshen et al.
"End-to-end learning for music audio", Dieleman & Schrauwen
Only the first one is cited, but not in this context. I think saying "it remains unknown" is a bit too strong of an expression.

The meaning of the following comment is not clear to me: "because in computer vision, the spatial frequency is not the only information the model can use". Surely the frequency domain and the spatial domain are two different representations of the same information contained in an image or audio signal? So in that sense, spatial frequency does encompass all information in an image.

The implication that high-frequency information is less useful for image-based tasks ("the spatial frequency of images is usually low") is incorrect. While lower frequencies dominate the spectrum more obviously in images than in audio, lots of salient information (i.e. edges, textures) will be high-frequency, so models would still have to learn high-frequency features to perform useful tasks.

WaveNet is mentioned (2.2.4) but not cited. WaveNet is a fairly different architecture than the ones discussed in this paper and it would be useful to at least discuss it in the related work section. A lot of the supposed issues discussed in this paper don't apply to WaveNet (e.g. there are no pooling layers, there is a multiplicative nonlinearity in each residual block).

The paper sometimes uses concepts without clearly defining them, e.g. "front-end layers". Please clearly define each concept when it is first introduced.

The paper seems to make a fairly arbitrary distinction between layers that perform signal filtering operations, and layers that don't - but every layer can be seen as a (possibly nonlinear) filtering operation. Even if SimpleNet has fewer "front-end layers", surely the later layers in the network can still introduce aliasing? I think the implicit assumption that later layers in the network perform a fundamentally different kind of operation is incorrect.

It has been shown that even random linear filters can be quite frequency-selective (see e.g. "On Random Weights and Unsupervised Feature Learning", Saxe et al.). This is why I think the proposed "changing rate" measure is a poor choice to show effective training. Moreover, optimization pathways don't have to be linear in parameter space, and oscillations can occur. Why not measure the difference with the initial values (at iteration 0)? It seems like that would prove the point a bit better.

Manually designing filters to initialize the weights of a convnet has been done in e.g. Sainath et al. (same paper as mentioned before), so it would be useful to refer to it again when this idea is discussed.

In SpecNet, have the magnitude spectrograms been log-scaled? This is common practice and it can make a dramatic difference in performance. If you haven't tried this, please do.

review_title: useful idea, but experimental validation and analysis need to be much stronger

Meta-review: The reviewers rightly point out that presented analysis is limiting and that the experimental results are not extensive enough. Moreover, several existing work that use raw waveforms have interesting analysis of what the network is trying to learn. Given these comments, the AC recommends that the paper be rejected.

Decision: Reject