Title: Fast Chirplet Transform to Enhance CNN Machine Listening - Validation on Animal calls and Speech
Authors: Herve Glotin
Julien Ricard
Randall Balestriero
Venue: ICLR 2017
Keywords: Applications
Supervised Learning
Deep learning
Speech

Abstract: The scattering framework offers an optimal hierarchical convolutional decomposition according to its kernels.  Convolutional Neural Net (CNN) can be seen asan optimal kernel decomposition, nevertheless it requires large amount of trainingdata to learn its kernels.  We propose a trade-off between these two approaches: a Chirplet kernel as an efficient Q constant bioacoustic representation to pretrainCNN. First we motivate Chirplet bioinspired auditory representation. Second we give the first algorithm (and code) of a Fast Chirplet Transform (FCT). Third, we demonstrate the computation efficiency of FCT on large environmental data base: months of Orca recordings, and 1000 Birds species from the LifeClef challenge. Fourth, we validate FCT on the vowels subset of the Speech TIMIT dataset. The results show that FCT accelerates CNN when it pretrains low level layers: it reduces training duration by -28% for birds classification, and by -26% for vowels classification. Scores are also enhanced by FCT pretraining, with a relative gain of +7.8% of Mean Average Precision on birds,  and +2.3% of vowel accuracy against raw audio CNN. We conclude on perspectives on tonotopic FCT deep machine listening, and inter-species bioacoustic transfer learning to generalise the representation of animal communication systems.

TL;DR: Proposing a chirplet transform in order to regulate the input of deep-CNN and possible extension to chirplet learning for deep learning bioacoustics

Review discussion:
First, few typos:
2.2: " A wavelet is an atom with compact support in time and frequency domain which integrates to 0"
You mean "often localized". (Haar wavelets are not and the compact support in fourier and spatial domain is mathematically not possible)
2.2: Otherwise, few remarks: The wavelet should be normalized. There should be a downsampling on Ux(lambda,t). The phase is not an information here, it’s a variability since it corresponds to a local translation.
I would have a few questions:
It seems that you are using first order coefficients of scattering. Why so? The representation could be potentially more discriminative with higher order coefficients. Do you have some suggestions of operators that could be used to get second order coefficients? Did you try to remove the low-pass filtering?
Indeed, your representation might lose discriminative information because of this averaging.
Have you done a comparison of your pipeline with a translation scattering transform or a joined scattering transform( https://arxiv.org/pdf/1512.02125.pdf )? It is claimed that in the case of chirp(and especially birds), the representation is more sparse.
What is the Littlewood-paley of your representation? Before and after averaging, is the representation sparse?
Did you apply a logarithm on the Chirplet coefficients, with the CNN+Chirplet? This dataset seems to have a lot of noise, have you tried to use a threshold?
How did you chose the parameters to create your chirplet filters? Are they standard or cross-validated? Do you think you will have to adapt them when using a new dataset?(e.g. are they generic?)
What is the final size of the representation of a signal of size T? I think there is 110 channels, but it is not clear to me!
How did you design the Audio2Chirp CNN, described in Appendix B? Could you initialize the filters with the Chirplet transform?
Thank you!
" A wavelet is an atom with compact support in time and frequency domain which integrates to 0"
You mean "often localized". (Haar wavelets are not and the compact support in fourier and spatial domain is mathematically not possible)
->This holds for continuous wavelets such as Morlet, DOG. For the case of discrete wavelets, this is indeed inappropriate.
Otherwise, few remarks: The wavelet should be normalized. There should be a downsampling on Ux(lambda,t). The phase is not an information here, it’s a variability since it corresponds to a local translation.
->Indeed a downsampling should be done on each of the Ux(lambda,t) but this would lead to non constant size across lambda which would not be suited for a CNN input.
It seems that you are using first order coefficients of scattering. Why so? The representation could be potentially more discriminative with higher order coefficients. Do you have some suggestions of operators that could be used to get second order coefficients? Did you try to remove the low-pass filtering?
Indeed, your representation might lose discriminative information because of this averaging.
->We use the local scattering since the averaging is not global. This is to reduce some transient artifacts and high frequency noise. For birds which has quite "long" songs, the low-pass will not affect them a priori.
It is indeed first order scattering and the higher order coefficients are not computed explicitly since they should be learned through the next CNN layer.
Have you done a comparison of your pipeline with a translation scattering transform or a joined scattering transform( https://arxiv.org/pdf/1512.02125.pdf )? It is claimed that in the case of chirp(and especially birds), the representation is more sparse.
->The joint scattering or even the more recent spiral scattering would indeed be more suited for analysis.
Yet since the aim is to feed the representation into a 2D CNN these representation would have too much parameters.
What is the Littlewood-paley of your representation? Before and after averaging, is the representation sparse?
->Naturally the representation is more sparse after averaging due to the disregard of high frequency information.
Littlewood Paley usually refers to the renormalization of the filters in the Fourier domain as referred in Mallat paper. This ensures that the scattering coefficients converge to 0 layers after layers. Is this what you are referring too ? If yes, not littlewood paley was applied on the filter-bank.
Did you apply a logarithm on the Chirplet coefficients, with the CNN+Chirplet? This dataset seems to have a lot of noise, have you tried to use a threshold?
->We did not applied logarithm compression, as the optimal compression shall be learned in the next CNN stage.
Similarly, thresholding functions would be learnt for any denoising.
How did you chose the parameters to create your chirplet filters? Are they standard or cross-validated? Do you think you will have to adapt them when using a new dataset?(e.g. are they generic?)
->It is standard for bird signals but would need some fine tuning for different dataset. No cross-validation as part of the CNN pipeline was done due to the time needed for training one model.
What is the final size of the representation of a signal of size T? I think there is 110 channels, but it is not clear to me!
->Each example has shape 80 (frequency bins) x 110 (time bins).
How did you design the Audio2Chirp CNN ?
->The hyperparameters of the audio2chirp network were chosen to satisfy the following constraints:
- long filters to capture modulations on long time scale (for the lower chirplet coefficients)
- not too much  downsampling between layers (1 layer only would imply 100x downsampling to go from 11025 time bins to 110)
Thank you for your consideration and questions !
Regards
"This holds for continuous wavelets such as Morlet, DOG. For the case of discrete wavelets, this is indeed inappropriate."
Theorem 2.7, page 45, A wavelet tour of signal processing, Mallat, 2009
"Indeed a downsampling should be done on each of the Ux(lambda,t) but this would lead to non constant size across lambda which would not be suited for a CNN input."
It depends on the way you implement it, I just meant there is a redundancy of the coefficients.
"We use the local scattering since the averaging is not global. This is to reduce some transient artifacts and high frequency noise. For birds which has quite "long" songs, the low-pass will not affect them a priori.
It is indeed first order scattering and the higher order coefficients are not computed explicitly since they should be learned through the next CNN layer."
No, there is absolutely no reason second order coefficients of scattering should be learned yet I would be glad if you could indicate such works.
Besides, this argument is wrong in the sens that the low-pass filter could be learned as well and might avoid a loss of information.
With this argument, one could claim that the first layer should be chirplets, since it will be learned by the CNN...
However, structuring the first layer seems to be a good idea.
"The joint scattering or even the more recent spiral scattering would indeed be more suited for analysis.
Yet since the aim is to feed the representation into a 2D CNN these representation would have too much parameters."
Ok.
"Naturally the representation is more sparse after averaging due to the disregard of high frequency information."
This is incorrect. A wavelet transform is sparse on many natural signals.
When you average it, you reduce the sparsity: the modulus+averaing avoids this kind of phenomenon. That might be checked numerically and it really depends on the nature of your data.
"Littlewood Paley usually refers to the renormalization of the filters in the Fourier domain as referred in Mallat paper. This ensures that the scattering coefficients converge to 0 layers after layers. Is this what you are referring too ? If yes, not littlewood paley was applied on the filter-bank."
I will clarify this comment which is indeed not clear. I was thinking to \sum |\hat \psi_\lambda|^2 for instance.
"We did not applied logarithm compression, as the optimal compression shall be learned in the next CNN stage.
Similarly, thresholding functions would be learnt for any denoising."
This is a strong claim and I do not understand why a classifier should necessarily compress the information/explictely denoise. Do you mean the loss is invariant to noise, thus the representation should be as well?
"It is standard for bird signals but would need some fine tuning for different dataset. No cross-validation as part of the CNN pipeline was done due to the time needed for training one model."
It is very promising for future research!
"This holds for continuous wavelets such as Morlet, DOG. For the case of discrete wavelets, this is indeed inappropriate."
Theorem 2.7, page 45, A wavelet tour of signal processing, Mallat, 2009
->So indeed in practice the support is on the whole domain since by definition they are exponentials.
However, in our case, we redefine the support of the wavelet as the set on which it is greater than a specific given threshold \epsilon and in this case, the Morlet and DOG for example have indeed a compact support around their center frequency and given the scale.
"We use the local scattering since the averaging is not global. This is to reduce some transient artifacts and high frequency noise. For birds which has quite "long" songs, the low-pass will not affect them a priori.
It is indeed first order scattering and the higher order coefficients are not computed explicitly since they should be learned through the next CNN layer."
No, there is absolutely no reason second order coefficients of scattering should be learned yet I would be glad if you could indicate such works.
Besides, this argument is wrong in the sens that the low-pass filter could be learned as well and might avoid a loss of information.
With this argument, one could claim that the first layer should be chirplets, since it will be learned by the CNN...
However, structuring the first layer seems to be a good idea."
->I don't understand why you claim that second order scattering coefficients shouldn't be learned.
For example, learning the \lambda_2 wavelets could be interesting since in the joint-scattering, it could lead to using chirplets instead of standard 2D Gabor wavelets for example. That being said, I agree on learning the low-pass filter.
"Naturally the representation is more sparse after averaging due to the disregard of high frequency information."
This is incorrect. A wavelet transform is sparse on many natural signals.
When you average it, you reduce the sparsity: the modulus+averaing avoids this kind of phenomenon. That might be checked numerically and it really depends on the nature of your data.
->Indeed, this was incorrect. The representation before averaging and after averaging were sparse but indeed, after averaging you loose sparsity since the nonlinearity accumulates energy around w=0.
However, the SNR was increased after averaging.
"We did not applied logarithm compression, as the optimal compression shall be learned in the next CNN stage.
Similarly, thresholding functions would be learnt for any denoising."
This is a strong claim and I do not understand why a classifier should necessarily compress the information/explictely denoise. Do you mean the loss is invariant to noise, thus the representation should be as well?
->The classifier is trained with clean data up to now, and the objective function yields to adapted / optimal non linear transformations of the input features.
The authors advocate use of chirplets as a basis for modeling audio signals.  They introduce a fast chiplet transform for efficient computation.
Also introduced is the idea of initializing (pre-training) CNN layers to mimic chirplet transform of audio signal (similar to ideas proposed by Mallet et al. on scattering transforms).  The paper is fairly easy to follow but in a few places contains undefined terms (e.g. AM-FM, MAP).
While the idea of using chirplet transform is interesting, my main concern is that the empirical evidence provided is in a rather narrow domain of bird call classification.
Furthermore, the accuracy gains shown in that domain are relatively small (61% MAP for log-Mel features vs 61.5% for chirplet transforms).  I would recommend that authors provide evidence for how this generalizes to other audio (including speech) tasks.
Thank you for your questions.
This idea of pre-training is not similar to what is proposed by Mallat, in fact in the scattering transform, everything is deterministic and not just "initialized" the only learning is done by the final classifier. Only recently they changed this to pre-training due to the success of neural networks.
The used bird call classification challenge provides the largest sound corpus for this kind of bioacoustic classification tasks. The gain is not just concerning the gain in classification accuracy but also the speed of convergence of the neural network.
We are currently working on the TIMIT (speech) dataset to present more results on the chirpnet performances, hopefully before the review deadline.
Feel free to ask anymore points.
Regards.
We clarified terms and ran the model on SPEECH TIMIT, we show an improvement of +2.3% of vowel accuracy against raw audio CNN, while the training is divided by a factor 2.
We added the code of the FCT.
Pros:
- Introduction of a nice filter banks and its implementation
- Good numerical results
- Refinement of the representation via back propagation, and a demonstration that it speeds up learning
Cons:
- The algorithms (section 3.1) are not necessary, and they even affect the presentation of the paper.
However, a source code would be great!
- The link with a scattering transform is not clear
- Sometimes (as mentionned in some of my comments), the writing could be improved.
From a personal point of view, I also believe the negative points I mention can be easily removed.
Thank you for the comments and reviewing process.
We now have new performances results on the TIMIT (speech) dataset. This plus a reworking of the paper had been done.
All the changes are present in the updated paper version.
Regards
The current available revised version contains the code and link to GitHub, the writing is improved, and we added SPEECH TIMIT experimentation showing improvement on vowel recognition using proposed FCT.
I still see claims such that:
"A wavelet is an atom with compact support in time and frequency domain which integrates to 0".
For example, is a Morlet wavelet with compact support in time and frequency?
And what are the benefits of the "Algo 1" and "Algo 2"? Maybe you could put them into an appendix.
In the conclusion " First we’ll work".
Write explicitly the setting for the TIMIT dataset (like T=300ms) and potentially add some references, since I thought you were using T=32ms where the accuracy is quite higher(like ~85% as far as I remember). Please add as well more numerical results from other papers in Table 2.
Thanks for your comments :
a)
' "A wavelet is an atom with compact support in time and frequency domain which integrates to 0".
For example, is a Morlet wavelet with compact support in time and frequency?
Indeed the analytical support of the wavelets (even continuous wavelets) is not compact since by definition of the exponential exp(w)>0 but they are very well localized. It can be considered compact in the applied case where roundoff error lead to a value of 0 quickly after moving around the center frequency. We precised it in the revised paper V3.
b)
"And what are the benefits of the "Algo 1" and "Algo 2"? Maybe you could put them into an appendix."
Right, we moved Algo 1 & 2 in appendix.
c)
We precised TIMIT data set definitions (it was only in appendix before). Note that we run at T= 310 ms as recommended in Palaz & Collobert 2013. Note that we experiment on vowel only, due to lack of time, so we are not able to compare to global Phone error rate.
However we focus in this paper on the gain using FCT pretraining.
While I understand the difficulty of collecting audio data from animals, I think this type of feature engineering does not go in the right direction. I would rather see a model than learns the feature representation from data.  I would think it should be possible to collect a more substantial corpus in zoos / nature etc, and then train a generative model. The underlying learned feature representation could be then used to feed a classifier. I'm not familiar with the particularities of this task, it's hard to judge the improvements by using chirplets.
This is indeed the direction that we discuss in this paper. We trained a CNN from raw audio (see page12) to then we show faster training and better MAP with our approach.
A full feature learning should be taken for large scale problems being tackled with deep learning approaches.
However, in order to use fully learned models, the number of observations must be important (at least as many as the number of free parameters in the model), and from clean mono species recordings, and important regularization should be applied precisely.
Nowadays, bioacoustic research (see http://sabiod.org to get links to the largest available challenge on bird classification, including some we organized at NIPS and ICML workshops), the volume of available mono-species clean recordings to learn the underlying distribution per species is limited.
The experiment we run in this paper is based on the largest Amazon avian dataset (cf LifeClef 2015, 16, 17).  The selected species we train are represented by the real available files for each species. It is not so much as you can see.
An alternative is thus to use our prior knowledge from advanced Q constant acoustic representation, to try to bias the network towards this direction which in our case corresponds to applying a chirplet transform to the raw waveform, and then retrain the system to adapt the chirpnet to the training data.
We ran the model on TIMIT, we show an improvement of +2.3% of vowel accuracy against raw audio CNN.

Meta-review:
role: PC
data: comment: This paper studies efficient signal representations to perform bioacoustic classification based on CNNs. Contrary to image classification, where most useful information can be extracted with spatially localized kernels, bioacoustic signatures are more localized in the frequency domain, requiring to rethink the design of convolutional architectures. The authors propose to enforce the lower layers of the architecture with chirplet transforms, which are localized in the time-frequency plane as wavelets, but with time-varying central frequency. They present preliminary numerical experiments showing promising improvements over existing baselines. 
 
 The reviewers found interest in the method, but raised concerns on the relatively narrow scope of the method, as well as the clarity and rigor of the presentation. Whereas the first concern is up to debate, I agree that the paper currently suffers from poor english which affects its clarity. 
 
 Despite these concerns, the AC finds the contribution useful in the broader context of inductive bias and injecting priors in neural networks. This is an example where the inductive priors that work well on images (localized convolutions rather than generic fully connected layers) are not sufficient unless given massive amounts of data. The AC thus recommends rejection, but invites the contribution to the workshop track.

Decision: Invite to Workshop Track