Title: FAST CHIRPLET TRANSFORM TO ENHANCE CNN MA-CHINE LISTENING -VALIDATION ON ANIMAL CALLS AND SPEECH

Abstract: The scattering framework offers an optimal hierarchical convolutional decomposition according to its kernels. Convolutional Neural Net (CNN) can be seen as an optimal kernel decomposition, nevertheless it requires large amount of training data to learn its kernels. We propose a trade-off between these two approaches: a Chirplet kernel as an efficient Q constant bioacoustic representation to pretrain CNN. First we motivate Chirplet bioinspired auditory representation. Second we give the first algorithm (and code) of a Fast Chirplet Transform (FCT). Third, we demonstrate the computation efficiency of FCT on large environmental data base: months of Orca recordings, and 1000 Birds species from the LifeClef challenge. Fourth, we validate FCT on the vowels subset of the Speech TIMIT dataset. The results show that FCT accelerates CNN when it pretrains low level layers: it reduces training duration by -28% for birds classification, and by -26% for vowels classification. Scores are also enhanced by FCT pretraining, with a relative gain of +7.8% of Mean Average Precision on birds, and +2.3% of vowel accuracy against raw audio CNN. We conclude on perspectives on tonotopic FCT deep machine listening, and inter-species bioacoustic transfer learning to generalise the representation of animal communication systems.

1. INTRODUCTION:

Representation of bioacoustic sequences started with 'Human' speech in the 70'. Speech automatic processing yields to the efficient Mel Filter Cepstral Coefficients (MFCC) representation. Today new bioacoustic representation paradigms arise from environmental monitoring and species classification at weak Signal to Noise Ratio (SNR) and with small amount of data per species.

Several neurobiological evidences suggest that auditory cortex is tuned to complex time varying acoustic features, and consists of several fields that decompose sounds in parallel (Kowalski et al., 1996; Mercado et al., 2000) . Therefore it is more than reasonable to investigate the Chirplet timefrequency representation from acoustic and neurophysiological points of view.

Chirps, or transient amplitude and frequency modulated waveforms, are ubiquitous in nature systems (Flandrin (2001) ), ranging from bird songs and music, to animal vocalization (frogs, whales) and Speech. Moreover the sinusoidal models are a typical attempt to represent audio signals as a superposition of chirp-like components. Chirp signals are also commonly observed in biosonar systems.

The Chirplet transform subsumes both Fourier analysis and wavelet analysis, providing a broad framework for mapping one-dimensional sound waveforms into a n-dimensional auditory parameter space. It offers the processing described in different auditory fields, i.e. cortical regions with systematically related response sensitivities. Moreover, Chirplet spaces are highly over-complete because there is an infinite number of ways to segment a time-frequency plane, the dictionary is redundant: this corresponds well with the overlapping, parallel signal processing pathways of auditory cortex.

Then we suggest that low level CNN layers shall be pretrained by Chirplet kernels. Thus, we define and code a Fast Chirplet Transform (FCT). We conduct validation on real recordings of whale and birds, and on Speech (vowels subset of TIMIT). We demonstrate that CNN classification benefits from low level layers FCT pretraining. We conclude on the perspectives of tonotopic FCT machine listening and inter-species transfer learning.

2. FORMAL DEFINITION OF CHIRPLET:

A chirplet can be seen as a complex sinus with increasing or decreasing frequency over time modulated by a Gaussian window to have a localized support in the time and Fourier domain. It is a broad class of filters which includes wavelets and Fourier basis as special cases. As a result, and as presented in (Mann & Haykin, 1991; 1992) , the Chirplet transform is a generalization of many known time-frequency representations. We first present briefly the wavelet transform framework to extend it to Chirplets. Given an input signal x one can compute a wavelet transform (Mallat, 1999) through the application of multiple wavelets ψ λ . A wavelet is an atom with localized support in time and frequency domain which integrates to 0. The analytical support of the wavelets is not compact but they are very well localized. It can be considered compact in the applied case where roundoff error lead to 0 quickly after moving around the center frequency. The whole filter bank is derived from a mother wavelet ψ 0 and a set of dilation coefficients following a geometric progression defined as Λ = {2 1+j/Q , j = 0, ..., JQ -1} with J being the number of octave to decompose and Q the number of wavelets per octave. As a result, one can create the filter-bank as the collection {ψ 0 ( t λ ) := ψ λ , λ ∈ Λ}. After application of the filter-bank, one ends up with a time-scale representation, or scalogram, U x(λ, t) := |(x ψ λ )(t)| where the complex modulus was applied in order to remove the phase information and contract the space. It is clear that a wavelet filter-bank is completely characterized by its mother wavelet and the set of scale parameters. Generalizing this framework for Chirplets will be straightforward by now allowing a nonconstant frequency for each filter. As for wavelets, filters are generated from a Gaussian window determining the time support however the complex sinus has nonconstant frequency over time with center-frequency f c . Since the scope of the parameters leads infinitely many different possible filters, we have to restrain ourselves, and thus create only a fixed Chirplet filter-bank allowing fast computations. The parameters defining these filters include the time position t c , the frequency center f c , the duration ∆ t and the chirp rate c:

g tc,fc,log(∆t),c (t) = 1 √ π∆t e -1 2 (t-tc ) 2 ∆ 2 t e j2π(c(t-tc) 2 +fc(t-tc))

.

(1)

3. PROPOSITION OF A FAST CHIRPLET TRANSFORM (FCT):

The parameter space is basically of infinite dimension. Similarly to continuous wavelet transform however, it is possible to use some a priori knowledge in order to create a finite bank-filter. For example, wavelets are generated by knowing the number of wavelets per octave and the number of octave to decompose. As a result, we used the same motivation in order to reduce the number of possible Chirplets required. The goal here is not to compute an invertible transform, but rather provide a redundant transformation highlighting transient structures which are not the same tasks as discussed in (Coifman et al., 1992; Meyer, 1993; Coifman et al., 1994) . As a result, we keep the same overall framework as for wavelets with the Q and J parameters. For example parameters for bird songs in this paper are J = 6 and Q = 16 with a sampling rate (SR) of 44100Hz, and J = 4 and Q = 16 on speech and Orca with SR=16 kHz). Finally, since we are interested in frequency modulations, we compute the ascendant and descendant chirp filters as one being the symetrized version of the other. As a result, we use a more straightforward analytical formula defined with a starting frequency F 0 , an ending frequency F 1 , and the usual wavelet like parameters σ being the bandwidth. Finally the hyperparameter p defining the polynomial order of the chirp is constant for the whole bank-filter generation. For example, the case p = 1 leads to a linear chirp, p = 2 to a quadratic chirp. The starting and ending frequencies are chosen to approximately cover one octave and are directly computed from the λ parameters which define the scales. Finally, following the scattering network inspiration from (Bruna & Mallat, 2013) , in order to remove unstable noisy pattern, we apply a low-pass filter (a Gaussian blurring) and thus we increase the SNR of the representation.

EQUATION

EQUATION

EQUATION

σ = 2 d λ , λ ∈ Λ.

(5)

4. LOW COMPLEXITY FCT ALGORITHM AND IMPLEMENTATION:

We give here our code of Fast Chirplet Transform (FCT), taking advantage of the a priori knowledge for the filter-bank creation and the fast convolution algorithm1 . Therefore, we first create the Chirplet with the ascendant and descendant versions in once (see Annexe Algo 1).

Then we generate the whole filter-bank (see Algo 2 in annexe) with the defined λ and hyperparameters.

Finally, we use the scattering framework (Bruna & Mallat, 2013; Andén & Mallat, 2014) : we apply a local low-pass filter to the obtained representation. In fact, the scattering coefficients Sx result from a time-averaging on the time-frequency representation U x bringing local and up to global time-invariance. This time-averaging is computed through the application of the φ filter, usually a Gabor atom with specified standard deviation and such that

EQUATION

As a result, one computes these coefficients as: Sx(λ, t) = (|x ψ λ | φ) (t), where ψ λ is a Chirplet with λ parameters and φ. Similarly, we perform local time-averaging on the Chirplet representation in the same manner.

We present some possible filters in Fig. 2 , and some bird features Fig. 3 .

The third step in our FCT consists in the reduction of the convolution task. The asymptotic complexity of the Chirplet transform is O(N. log(N )) with N being the size of the input signal. This is the same asymptotic complexity as for the continuous wavelet transform and the scattering network. However, it is possible to reach lower asymptotic complexity simply by a division of the convolution task. usually the convolutions are carried through application of an element-wise multiplication of the signal and the filter in the frequency domain and then compute the inverse Fourier transform to end up with x ψ λ . However, if we denote by M the length of the filter ψ λ it is possible to instead perform multiple times this operation on different overlapping chunks of the signal to then concatenate the results to obtain at the end the same convolution result but now in O(N. log(M )).

Finally a last improvement induced by this approach is to allow easy tackling of signals with a length just above a power of 2 which otherwise would require to be padded in order to obtain a FFT with real O(N. log(N )) complexity through the Danielson-Lanczos lemma (Press, 2007) . Applying this scheme allowed to compute the convolutions between 3 to 4 times faster. The variations came from the distance between N and the closest next power of 2 depending on the desired chunk size.

We validate the efficiency of FCT on real bioacoustic recordings. We processed on 10 medium speed CPUs of 4 years old, 100 hours of recording of LifeClef bird challenge (16 kHz Sampling Rate (SR), 16 bits) in 2 days. Second, we processed in 7 days the equivalent of 1 month of 

5. ENHANCING CNN BIOACOUSTIC REPRESENTATION WITH FCT:

A strategy for CNN fine-tuning can be to retrain a classifier on top of a CNN on a new dataset, or to fine-tune the weights of a pretrained network by continuing the backpropagation. It is possible to fine-tune all the layers of the CNN or to freeze some of the earlier, later or central layers, and to only fine-tune some portion of the network. As the features propagate deeper and deeper in the network layers, they become increasingly invariant and discriminative (Seltzer 2013). Thus usually only the higher level are fine-tuned, the earlier features of a CNN contain more generic features that should be useful to many tasks. As denoted in later layers of the CNN becomes progressively more specific to the details of the classes contained in the original dataset.

In this paper we adapt our parametric Chirplet decomposition to a specific acoustic domain with a specific CNN. We compare a CNN trained on raw audio to one trained on Mel and Chirplet. The best model is the one trained on parametric Chirplet. Second, we show that the CNN can be enhanced by pretraining Chirp in low level layer.

5.1. CNN BIRDS CLASSIFICATION ON FCT, RAW AUDIO, VERSUS MEL:

The first demonstration is conducted on complex Bird songs. We use the BIRD10 subset of LifeClef 2016 bird classification challenge. It was used as ENS Ulm data challenge 2016, and contains 3 species in a total of 15 minutes of recordings (SR 44100 Hz, 16 bits), and is available (.wav, Mel and FCT features) at http://sabiod.univ-tln.fr/workspace/BIRD10. 

5.2. ENHANCING BIRDS CLASSIFICATION STACKING PRETRAINED CHIRPNET CNN:

In order to test the efficiency of the FCT, we pretrain a CNN to encode audio to Chirplets (a.k.a. the Audio2Chirp CNN) and a CNN to convert parametric Chirplet to classes (a.k.a. the Chirp2Class CNN). The topology of these CNNs (Tab. 2, 3) is set for reasonable time of training. We also speed up the training with shorter time overlap of the time windows (only 30% instead of 90% in the previous experimentation). We then decrease the average MAP, however the objective here is to compare the gain in MAP and time of convergence in stacked Chirplet deep representations.

We then simply stack at low level layer the audio2chirp with the chirp2class CNN to build a complete audio2class CNN. We train it from random initialization, or from pretrained CNN. Note that the random seed in all the experimentation of this paper is fixed to allow fair comparisons. 

5.3. ENHANCING VOWELS CLASSIFICATION STACKING PRETRAINED CHIRPNET CNN:

In this section we run the same demonstration on the subset of speech vowels of all the TIMIT acoustic-phonetic corpus JS et al. (1993) : 3,696 training utterances (sampled at 16kHz) from 462 speakers. The cross-validation set consists of 400 utterances from 50 speakers. The core test set of the 8 vowels subset was used to report the results: 192 utterances from 24 speakers, excluding the validation set. There are 61 hand labeled phonetic symbols but the experiments in this paper run on the time windows of 310ms centered on each of the 8 vowels of TIMIT (= iy, ih, eh, ae, aa, ah, uh, uw).

Due to similar bioacoustic voicing dynamics of the two species (near 4 Hz), we simply set the FCT parameters for vowel to the one used for Orca presented above (p = 3,j = 4,q = 16,t = 0.001,s = 0.01). The time windows are set to 310 ms as recommended in Palaz et al. (2013) .

The results of the different training stages of the audio2chirp and chirp2class and stacked model are given in Tab.2 and Annexe. We run due to lack of time the experiment only on vowel classification, which does not really allow comparison with other papers, however this seminal work only aims to study the relative gain between CNN pretrained or not by FCT.

The results demonstrate that FCT pretraining of the audio2class model is improved by 2.3% of relative gain of accuracy while the training time is decreased by 26%.

6. DISCUSSION AND CONCLUSION:

In this paper we propose for the first time at our knowledge the definition and implementation of a Fast Chirplet Transform (FCT). Due to its low complexity, FCT can be computed as fast as FFT. These gains may be due to the sparsity of the Chirplet, and the denoising step in the FCT. These experiences bring to light the problem of deep learning for small and biased dataset for which a full learning strategy is sub-optimal due to local optimum convergence. As a result, FCT prior knowledge can be used to mitigate this drawback by reducing the complexity of the deep-net architecture.

Three main perspectives are then opened. Future work will consist on sparse Chirpnet inspired from tonotopic net Strom (1997) , auditory nerve and cortex topology Pironkov et al. (2015) . The acoustic vibrations are transmitted to the base of the cochlea, thus each region of the basilar membrane are excited by different frequencies. The higher frequencies excite areas closer to the cochlea base, whereas lower frequencies are closer to the apex. This implies that neurons connected to a specific zone of the basilar membrane will be simultaneously stimulated inducing tonotopic representation.

A second perspective is to integrate Chirplet computation into the CNN training itself, as a constrained embedded layer, in a framework similar to a Wavelet Neural Network (Adeli & Jiang, 2006) but with Chirplet activation functions.

Last, we currently work on transfer learning of Chirpnet from animal to speech (and reverse), in order to generalize a deep Chirpnet representation of the animal communication systems.

A BIRD DATASET:

The experiment is conducted on BIRD10, an online data set http://sabiod.univ-tln.fr/ workspace/BIRD10/ which is a subset of the training LIFEClef 2016 challenge on bird classification. BIRD10 contains 454 audio files (22050 Hz SR, 16 bits) from 10 bird classes, split in 0.5s segments. 20% of the training set was used as the validation set.

Only segments with detected bird activity were kept, assuming a bird sound to have prominent energy and to be mostly harmonic. This bird detection is for a given segment:

if (energy_ratio > energy_threshold and spectral_flatness_weighted_mean < spectral_flatness_threshold) bird_detected = True else bird_detected = False

where the energy and the spectral flatness are computed on 50% overlapping frames of 256 samples:

er = energy_ratio = mean(seg_energy) 95thpercentile(f ile_energy) swf = spectral_f latness_weighted_mean =

sum(seg_spectral_f latness × seg_energy) sum(seg_energy) This naive algorithm performed quite well on a manually labelled dataset of bird vocalizations (precision=0.89, recall=0.57 for er=0.2 and sfw=0.3) after a quick grid search on the two parameters.

B BIRDS CLASSIFICATION : BASELINE CNNS:

The first experiment consisted in running similar CNNs to compare the performance of using raw audio and two time-frequency representations as the input: a standard log-amplitude Mel spectrum and the Chirplet representation described in the first part of this paper. In this experiments the segments were overlapping by 90%. The topologies of the networks are given in Tab. 2. The cost function is the cross-entropy, learning rate = 0.0001 = L2 regularisation coefficient. The Mel spectrum is computed from 64 bands between 0 and 11025 Hz (=SR/2). Both Mel spectrum and Chirplets were normalized by Z-score. 

Input:



E EXPERIMENT ON SPEECH VOWEL:

The Tab. 5 gives the topology of the audio2chirp and chirp2class, and stacked models, for these vowel experiments.

In all experiments, each CNN is initialized using the same random seed. The symbol "*" refers to the optimal trained parameters of a net. F ALGORITHM FOR THE FAST CHIRPLET TRANSFORM (FCT) 



G THE PYTHON CODE FOR THE FAST CHIRPLET TRANSFORM (FCT):

This code, in GPL licence (c) DYNI team, is in Github : https://github.com/DYNI-TOULON/fastchirplet.git. " " " l o w e s t f r e q u e n c y where t h e c h i r p l e t i s a p p l i e d " " " s e l f . m i n _ f r e q u e n c y = F0 " " " h i g h e s t f r e q u e n c y where t h e c h i r p l e t i s a p p l i e d " " " s e l f . m a x _ f r e q u e n c y = F1 " " " s a m p l e r a t e o f t h e s i g n a l " " " s e l f . s a m p l e r a t e = s a m p l e r a t e " " " d u r a t i o n o f t h e c h i r p l e t " " " s e l f . t i m e _ b i n = s i g m a / 1 0 " " " d e g r e e o f t h e p o l y n o m e t o g e n e r a t e t h e c o e f f i c i e n t s o f t h e c h i r p l e t " " " s e l f . p o l y n o m e _ d e g r e e = p o l y n o m e _ d e g r e e " " " c o e f f i c i e n t s a p p l i e d t o t h e s i g n a l " " " s e l f . f i l t e r _ c o e f f i c i e n t s = s e l f . c a l c u l _ c o e f f i c i e n t s ( ) d e f c a l c u l _ c o e f f i c i e n t s ( s e l f ) :

" " " c a l c u l a t e c o e f f i c i e n t s f o r t h e c h i r p l e t s " " " # p r i n t ( s e l f . _ s a m p l e r a t e ) t = l i n s p a c e ( 0 , s e l f . t i m e _ b i n , i n t ( s e l f . s a m p l e r a t e * s e l f . m i n _ f r e q u e n c y ) * t ) e l s e : w = c o s ( 2 * p i * ( ( s e l f . m i n _ f r e q u e n c y * ( s e l f . m a x _ f r e q u e n c y / s e l f . m i n _ f r e q u e n c y ) * * ( t / s e l f . t i m e _ b i n )-s e l f . m i n _ f r e q u e n c y ) * s e l f . t i m e _ b i n / l o g ( s e l f . m a x _ f r e q u e n c y / s e l f . m i n _ f r e q u e n c y

) ) )

c o e f f s = w * h a n n i n g ( l e n ( t ) ) * * 2 #Low f r e q u e n c i e s f o r a s i g n a l s t a r t _ f r e q u e n c i e s = ( s a m p l e r a t e / l a m b d a s ) / 2 . 0 # h i g h f r e q u e n c i e s f o r a s i g n a l e n d _ f r e q u e n c i e s = s a m p l e r a t e / l a m b d a s d u r a t i o n s = 2 . 0 * d u r a t i o n _ l a s t _ c h i r p l e t / f l i p u d ( l a m b d a s ) C h i r p l e t . s m a l l e s t _ t i m e _ b i n s = d u r a t i o n s [ 0 ] c h i r p l e t s = l i s t ( ) f o r f0 , f1 , d u r a t i o n i n z i p ( s t a r t _ f r e q u e n c i e s , e n d _ f r e q u e n c i e s , d u r a t i o n s ) : c h i r p l e t s . a p p e n d ( C h i r p l e t ( s a m p l e r a t e , f0 , f1 , d u r a t i o n , p ) ) r e t u r n c h i r p l e t s d e f a p p l y _ f i l t e r b a n k ( i n p u t _ s i g n a l , c h i r p l e t s , e n d _ s m o o t h i n g ) :

r

" " " g e n e r a t e l i s t o f s i g n a l w i t h c h i r p l e t s " " " r e s u l t = l i s t ( ) f o r c h i r p l e t i n c h i r p l e t s : r e s u l t . a p p e n d ( c h i r p l e t . smooth_up ( i n p u t _ s i g n a l , 6 , e n d _ s m o o t h i n g ) ) r e t u r n a r r a y ( r e s u l t ) d e f f f t _ s m o o t h i n g ( i n p u t _ s i g n a l , s i g m a ) :

" " " s m o o t h t h e f a s t t r a n s f o r m f o u r i e r " " " s i z e _ s i g n a l = i n p u t _ s i g n a l . s i z e # s h o r t e n t h e s i g n a l n e w _ s i z e = i n t ( f l o o r ( 1 0 . 0 * s i z e _ s i g n a l * s i g m a ) ) h a l f _ n e w _ s i z e = n e w _ s i z e / / 2 M=h . s i z e h a l f _ s i z e = M/ / 2 i f ( b o u n d a r y ==0) : #ZERO PADDING i n p u t _ s i g n a l = pad ( i n p u t _ s i g n a l , ( h a l f _ s i z e , h a l f _ s i z e ) , ' c o n s t a n t ' , c o n s t a n t _ v a l u e s = 0 ) h= pad ( h , ( 0 , i n p u t _ s i g n a l . s i z e -M) , ' c o n s t a n t ' , c o n s t a n t _ v a l u e s = 0 ) newx= i f f t ( f f t ( i n p u t _ s i g n a l ) * f f t ( h ) ) r e t u r n newx [M-1: -1] e l i f ( b o u n d a r y ==1) : # s y m m e t r i c i n p u t _ s i g n a l = c o n c a t e n a t e ( [ f l i p u d ( i n p u t _ s i g n a l [ : h a l f _ s i z e ] ) , i n p u t _ s i g n a l , f l i p u d ( i n p u t _ s i g n a l [ h a l f _ s i z e : ] ) ] ) h= pad ( h , ( 0 , x . s i z e -M) , ' c o n s t a n t ' , c o n s t a n t _ v a l u e s = 0 ) newx= i f f t ( f f t ( i n p u t _ s i g n a l ) * f f t ( h ) ) r e t u r n newx [M-1: -1] e l s e : # p e r i d i c r e t u r n r e a l ( r o l l ( i f f t ( f f t ( i n p u t _ s i g n a l ) * f f t ( h , i n p u t _ s i g n a l . s i z e ) ) ,-h a l f _ s i z e ) ) 

f f t x = f f t ( i n p u t _ s i g n a l ) s h o r t _ f f t x = [ ] f o

:

We provide our implementation in Annexe and: https://github.com/DYNI-TOULON/ fastchirplet.git

https://github.com/Lasagne/Lasagne

https://github.com/librosa/librosa

https://github.com/bastibe/PySoundFile

Figures and Tables:

Figure 1: Top: Chirplet of Orca call with p=3, j=4, q=16, t=0.001, s=0.01, with usual FFT spectrogram below, Sampling Rate (SR) 22 kHz, 16 bits. Waves and Chirplets of Orca are: http://sabiod. univ-tln.fr/orcalab. Bottom: same on bird calls from Amazonia (BIRD10 data set). SR 16 kHz, 16 bits.

Figure 2: Some FCT displayed in the physical domain and in the time-frequency domain through a spectrogram. The first one reduces to a wavelet since the chirp rate is 0. One can see the importance of the time duration and the chirp rate and well as the center frequency depending on what one wishes to capture..

Figure 3: FCT of 4 species of amazonian birds LifeClef 2015 challenge including BIRD10 dataset available online. The call patterns are the high SNR (red) regions. The species international codes are, from top to bottom, right to left: nnbhgj, aethwv, aksucy, nipfbr..

CNNs (LeCun & Bengio, 1995)  on the Lasagne Theano platform. The baseline CNN is trained from the raw audio. A second CNN, with similar topology (see annexe) is trained on a simple log of the simple 64 channels Mel scale of FFT spectrum ( http://pydoc.net/Python/librosa/0.2.0/librosa.feature/ ). We overlap by 90% the time windows. A third CNN is trained on our FCT. The parameters of both CNN are similar, with 64 frequency bands each (we remove top and bottom band from the Chirplet to set to 64 bands only). Then the input layer is 64 x 86, the Conv layer has 20 filters of size 8 x 10. All activation functions are relu. We maxpool 2 x 2, follow the 20 filters of size 8 x 10, maxpooling, dense layer (200), dropout at 10%, with a final softmax dense layer with 3 classes and same dropout. Each CNN is trained by cross-entropy, L2 reg., with a learning rate set of 0.001. The Fig.4gives the MAP of these two CNNs having similar hyperparameters. The CNN on FCT gives the best MAP with 61.5% at epoch 280 compared to later epoch (820) for Mel with a similar MAP of 61%. Audio is slower and weaker (58% MAP at epoch 1140).

Figure 4: The Mean Average Precision on BIRD10 of the CNNs on Mel, raw audio, or FCT. The training conditions are the same on the three CNNs, and they have similar size and topology (see Annexe). The CNN trained on FCT is slightly better than on Mel or raw audio, and is learning faster.

Figure 5: FCT (top) versus Fourier spectrogram (bottom) of two utterances of Speech vowel (TIMIT), (p = 3,j = 4,q = 16,t = 0.001,s = 0.01)..

Figure 6: Training stacked CNN. Blue: random initialization of audio2chirp(0) and chirp2class(0). Red: Initialization with optimal audio2chirp(*) and chirp2class(*).

Figure 7: Training stacked random initialized CNNs: audio2chirp(0) and chirp2class(0).

Figure 8: Training stacked pretrained CNNs: audio2chirp(*) and chirp2class(*).

Figure 9: Training stacked pretrained CNNs audio2chirp(*) and chirp2class(*) but freezing chirp2class(*) (no weight update).

Figure 10: Training stacked pretrained CNNs audio2chirp(*) and chirp2class(*) but freezing au-dio2chirp(*) (no weight update).

Figure 11: Training stacked CNNs : pretrained audio2chirp(*) and chirp2class(0) and freezing audio2chirp (no weight update).

Figure 12: Training stacked CNN from pretrained CNN: Initialized with optimal audio2chirp(*) and chirp2class(0).

Figure 13: Trained and loss audio2chirp (TIMIT).

import l i b r o s a import o s import numpy a s np from p y l a b import * import s y s from numpy . l i b import pad c l a s s C h i r p l e t : " " " s m a l l e s t t i m e b i n among t h e c h i r p l e t " " " g l o b a l s m a l l e s t _ t i m e _ b i n s d e f _ _ i n i t _ _ ( s e l f , s a m p l e r a t e , F0 , F1 , sigma , p o l y n o m e _ d e g r e e ) :

t i m e _ b i n ) ) i f ( s e l f . p o l y n o m e _ d e g r e e ) : w = c o s ( 2 * p i * ( ( s e l f . m a x _ f r e q u e n c ys e l f . m i n _ f r e q u e n c y ) / ( ( s e l f . p o l y n o m e _ d e g r e e + 1 ) * s e l f . t i m e _ b i n * * s e l f . p o l y n o m e _ d e g r e e ) * t * * s e l f . p o l y n o m e _ d e g r e e + s e l f .

e t u r n c o e f f s d e f smooth_up ( s e l f , i n p u t _ s i g n a l , sigma , e n d _ s m o o t h i n g ) : # g e n e r a t e f a s t f o u r i e r t r a n s f o r m f r o m a s i g n a l and s m o o t h i t new_up = b u i l d _ f f t ( i n p u t _ s i g n a l , s e l f . f i l t e r _ c o e f f i c i e n t s , s i g m a ) r e t u r n f f t _ s m o o t h i n g ( f a b s ( new_up ) , e n d _ s m o o t h i n g ) d e f compute ( i n p u t _ s i g n a l , s a v e = F a l s e , d u r a t i o n _ l a s t _ c h i r p l e t = 1 . 0 1 , n u m _ o c t a v e s =5 , n u m _ c h i r p s _ b y _ o c t a v e =10 , p o l y n o m e _ d e g r e e =3 , e n d _ s m o o t h i n g = 0 . 0 0 1 ) : " " " main f u n c t i o n . F a s t C h i r p l e t T r a n s f o r m f r o m a s i g n a l " " " d a t a , s a m p l e r a t e = l i b r o s a . l o a d ( i n p u t _ s i g n a l , s r =None ) s i z e _ d a t a = l e n ( d a t a ) n e a r e s t _ p o w e r _ 2 = 2 * * ( s i z e _ d a t a -1) . b i t _ l e n g t h ( ) d a t a = np . l i b . pad ( d a t a , ( 0 , n e a r e s t _ p o w e r _ 2 -s i z e _ d a t a ) , ' c o n s t a n t ' , c o n s t a n t _ v a l u e s = 0 ) c h i r p l e t s = i n i t _ c h i r p l e t _ f i l t e r _ b a n k ( s a m p l e r a t e , d u r a t i o n _ l a s t _ c h i r p l e t , n u m _ o c t a v e s , n u m _ c h i r p s _ b y _ o c t a v e , p o l y n o m e _ d e g r e e ) c h i r p s = a p p l y _ f i l t e r b a n k ( d a t a , c h i r p l e t s , e n d _ s m o o t h i n g ) c h i r p s = r e s i z e _ c h i r p s ( s i z e _ d a t a , n e a r e s t _ p o w e r _ 2 , c h i r p s ) i f s a v e : i f n o t o s . p a t h . e x i s t s ( " c s v " ) : o s . m a k e d i r s ( " c s v " ) np . s a v e t x t ( " c s v / " + o s . p a t h . basename ( i n p u t _ s i g n a l ) . s p l i t ( ' . ' ) [ 0 ] + ' . c s v ' , c h i r p s , d e l i m i t e r = " , " ) r e t u r n c h i r p s d e f r e s i z e _ c h i r p s ( s i z e _ d a t a , n e a r e s t _ p o w e r _ 2 , c h i r p s ) : s i z e _ c h i r p s = l e n ( c h i r p s ) r a t i o = s i z e _ d a t a / n e a r e s t _ p o w e r _ 2 s i z e = i n t ( r a t i o * l e n ( c h i r p s [ 0 ] ) ) t a b f i n a l = np . z e r o s ( ( s i z e _ c h i r p s , s i z e ) ) f o r i i n range ( 0 , s i z e _ c h i r p s ) : t a b f i n a l [ i ] = c h i r p s [ i ] [ 0 : s i z e ] r e t u r n t a b f i n a l d e f i n i t _ c h i r p l e t _ f i l t e r _ b a n k ( s a m p l e r a t e , d u r a t i o n _ l a s t _ c h i r p l e t , n u m _ o c t a v e s , n u m _ c h i r p s _ b y _ o c t a v e , p ) : " " " g e n e r a t e a l l t h e c h i r p l e t s f r o m a g i v e n s a m p l e r a t e " " " l a m b d a s = 2 . 0 * * ( 1 + a r a n g e ( n u m _ o c t a v e s * n u m _ c h i r p s _ b y _ o c t a v e ) / f l o a t ( n u m _ c h i r p s _ b y _ o c t a v e ) )

r e l e i n f f t x [ : h a l f _ n e w _ s i z e ] : s h o r t _ f f t x . a p p e n d ( e l e ) f o r e l e i n f f t x [-h a l f _ n e w _ s i z e : ] : s h o r t _ f f t x . a p p e n d ( e l e ) a p o d i z a t i o n _ c o e f f i c i e n t s = g e n e r a t e _ a p o d i z a t i o n _ c o e f f i c i e n t s ( h a l f _ n e w _ s i z e , sigma , s i z e _ s i g n a l ) # a p p l y t h e a p o d i z a t i o n c o e f f i c i e n t s s h o r t _ f f t x [ : h a l f _ n e w _ s i z e ] * = a p o d i z a t i o n _ c o e f f i c i e n t s # a p p l y t h e a p o d i z a t i o n c o e f f i c i e n t s i n a r e v e r s e l i s t s h o r t _ f f t x [ h a l f _ n e w _ s i z e : ] * = f l i p u d ( a p o d i z a t i o n _ c o e f f i c i e n t s ) r e a l i f f t x w = r e a l ( i f f t ( s h o r t _ f f t x ) ) r e t u r n r e a l i f f t x w d e f g e n e r a t e _ a p o d i z a t i o n _ c o e f f i c i e n t s ( n u m _ c o e f f s , sigma , s i z e ) : " " " g e n e r a t e a p o d i z a t i o n c o e f f i c i e n t s " " " a p o d i z a t i o n _ c o e f f i c i e n t s = a r a n g e ( n u m _ c o e f f s ) a p o d i z a t i o n _ c o e f f i c i e n t s = a p o d i z a t i o n _ c o e f f i c i e n t s * * 2 a p o d i z a t i o n _ c o e f f i c i e n t s = a p o d i z a t i o n _ c o e f f i c i e n t s / ( 2 * ( s i g m a * s i z e ) * * 2 ) a p o d i z a t i o n _ c o e f f i c i e n t s = exp (-a p o d i z a t i o n _ c o e f f i c i e n t s ) r e t u r n a p o d i z a t i o n _ c o e f f i c i e n t s d e f f f t _ b a s e d ( i n p u t _ s i g n a l , h , b o u n d a r y = 0 ) :

d e f b u i l d _ f f t ( i n p u t _ s i g n a l , f i l t e r _ c o e f f i c i e n t s , n =2 , b o u n d a r y = 0 ) :" " " g e n e r a t e f a s t t r a n s f o r m f o u r i e r by windows " " " M= f i l t e r _ c o e f f i c i e n t s . s i z e # p r i n t ( n , boundary ,M) h a l f _ s i z e = M/ / 2 s i g n a l _ s i z e = i n p u t _ s i g n a l . s i z e # power o f 2 t o a p p l y f a s t f o u r i e r t r a n s f o r m w i n d o w s _ s i z e = i n t ( 2 * * c e i l ( l o g 2 (M * ( n + 1 ) ) ) ) number_of_windows = f l o o r ( s i g n a l _ s i z e / / w i n d o w s _ s i z e ) i f ( number_of_windows ==0) : r e t u r n f f t _ b a s e d ( i n p u t _ s i g n a l , f i l t e r _ c o e f f i c i e n t s , b o u n d a r y ) o u t p u t = e m p t y _ l i k e ( i n p u t _ s i g n a l ) # pad w i t h 0 t o h a v e a s i z e i n a power o f 2 z e r o p a d d i n g = pad ( f i l t e r _ c o e f f i c i e n t s , ( 0 , w i n d o w s _ s i z e -M) , ' c o n s t a n t ' , c o n s t a n t _ v a l u e s = 0 ) h _ f f t = f f t ( z e r o p a d d i n g ) # t o b r o w s e t h e w h o l e s i g n a l c u r r e n t _ p o s =0 # a p p l y f f t t o a p a r t o f t h e s i g n a l . T h i s p a r t h a s a s i z e w h i c h i s a power o f 2 i f ( b o u n d a r y ==0) : #ZERO PADDING # window i s h a l f padded w i t h s i n c e i t ' s f o c u s e d on t h e f i r s t h a l f window = i n p u t _ s i g n a l [ c u r r e n t _ p o s : c u r r e n t _ p o s + w i n d o w s _ s i z e -h a l f _ s i z e ] z e r o p a d d e d w i n d o w = pad ( window , ( l e n ( h _ f f t )-l e n ( window ) , 0 ) , ' c o n s t a n t ' , c o n s t a n t _ v a l u e s = 0 ) x _ f f t = f f t ( z e r o p a d d e d w i n d o w ) e l i f ( b o u n d a r y ==1) : #SYMMETRIC window = c o n c a t e n a t e ( [ f l i p u d ( i n p u t _ s i g n a l [ : h a l f _ s i z e ] ) , i n p u t _ s i g n a l [ c u r r e n t _ p o s : c u r r e n t _ p o s + w i n d o w s _ s i z e -h a l f _ s i z e ] ] ) x _ f f t = f f t ( window ) e l s e : x _ f f t = f f t ( i n p u t _ s i g n a l [ : w i n d o w s _ s i z e ] ) o u t p u t [ : w i n d o w s _ s i z e -M] = i f f t ( x _ f f t * h _ f f t ) [M-1: -1] c u r r e n t _ p o s += w i n d o w s _ s i z e -M-h a l f _ s i z e # a p p l y f a s t f o u r i e r t r a n s o f m t o e a c h windows w h i l e ( c u r r e n t _ p o s + w i n d o w s _ s i z e -h a l f _ s i z e <= s i g n a l _ s i z e ) : x _ f f t = f f t ( i n p u t _ s i g n a l [ c u r r e n t _ p o s -h a l f _ s i z e : c u r r e n t _ p o s + w i n d o w s _ s i z e -h a l f _ s i z e ] ) o u t p u t [ c u r r e n t _ p o s : c u r r e n t _ p o s + w i n d o w s _ s i z e -M] = r e a l ( i f f t ( x _ f f t * h _ f f t ) [M-1: -1]) c u r r e n t _ p o s += w i n d o w s _ s i z e -M # a p p l y f a s t f o u r i e r t r a n s f o r m t o t h e r e s t o f t h e s i g n a l i f ( w i n d o w s _ s i z e -( s i g n a l _ s i z e -c u r r e n t _ p o s + h a l f _ s i z e ) < h a l f _ s i z e ) : window = i n p u t _ s i g n a l [ c u r r e n t _ p o s -h a l f _ s i z e : ] z e r o p a d d e d w i n d o w = pad ( window , ( 0 , i n t ( w i n d o w s _ s i z e -( s i g n a l _ s i z e -c u r r e n t _ p o s + h a l f _ s i z e ) ) ) , ' c o n s t a n t ' , c o n s t a n t _ v a l u e s = 0 ) x _ f f t = f f t ( z e r o p a d d e d w i n d o w ) o u t p u t [ c u r r e n t _ p o s : ] = r e a l ( r o l l ( i f f t ( x _ f f t * h _ f f t ) , h a l f _ s i z e ) [ h a l f _ s i z e : h a l f _ s i z e + o u t p u t . s i z ec u r r e n t _ p o s ] ) o u t p u t [-h a l f _ s i z e : ] = c o n v o l v e ( i n p u t _ s i g n a l [-M: ] , f i l t e r _ c o e f f i c i e n t s , ' same ' ) [-h a l f _ s i z e : ] e l s e : window = i n p u t _ s i g n a l [ c u r r e n t _ p o s -h a l f _ s i z e : ] z e r o p a d d e d w i n d o w = pad ( window , ( 0 , i n t ( w i n d o w s _ s i z e -( s i g n a l _ s i z e -c u r r e n t _ p o s + h a l f _ s i z e ) ) ) , ' c o n s t a n t ' , c o n s t a n t _ v a l u e s = 0 ) x _ f f t = f f t ( z e r o p a d d e d w i n d o w ) o u t p u t [ c u r r e n t _ p o s : ] = r e a l ( i f f t ( x _ f f t * h _ f f t ) [M-1:M+ o u t p u t . s i z e -c u r r e n t _ p o s -1]) r e t u r n o u t p u t

Results are reported in Tab. 1 for each of the stacked CNN, with the epoch giving the best MAP on the dev. set, and the corresponding MAP on the test set. Results demonstrate that the pretraining of low level layers by FCT enhances CNN. More details are given in Annexe. Summary of the CNN enhanced by our FCT representation, on BIRD. For each model, we detail the time of convergence on dev. and corresponding Mean Average Precision on test set.

Table content: <table><tr><td>Model</td></tr></table>

Summary of the CNN enhanced by our FCT representation on vowel (TIMIT): time of convergence and vowel accuracy on TIMIT test set.it increases MAP by 4 points (Tab. 1). The experiment on Vowels demonstrates a training of 30 epochs on FCT, versus 60 on raw audio (for same 65% accuracy level), and an increase of 1.5 point of accuracy (Tab. 2).

Table content: <table><tr><td>Model</td></tr></table>

CNN topologies for the 3 different inputs.In all experiments, a given topology is always initialized using the same set of random parameters, unless specified otherwise. The value * (resp. 0) after the name of the net refers to the pretrained net (resp. random initialization).

Table content: <table><tr><td colspan="2">C BIRD AUDIO2CHIRP -CHIRPLET ENCODER</td></tr><tr><td colspan="2">The chirp encoder, aka audio2chirp, aims at training a net to get a Chirplet-like representation. It is a</td></tr><tr><td colspan="2">simple CNN taking audio as input, Chirplets as output and minimizing the square error. It converges</td></tr><tr><td colspan="2">easily in 180 epochs. The topology of the audio2chirp net is given Tab. 4.</td></tr><tr><td>Input</td><td>Topology</td></tr><tr><td/><td>conv_1: 40 filters of shape (1, 1001) (nonlinearity: relu)</td></tr><tr><td/><td>pool_1: (1, 4) max pooling</td></tr><tr><td/><td>conv_2: 40 filters of shape (1, 501) (nonlinearity: relu)</td></tr><tr><td>Audio, shape (1, 11025)</td><td>pool_2: (1, 4) max pooling conv_3: 40 filters of shape (1, 101) (nonlinearity: relu)</td></tr><tr><td/><td>pool_3: (1, 4) max pooling</td></tr><tr><td/><td>dense_1: 8800 units (nonlinearity: relu, 10% dropout)</td></tr><tr><td/><td>reshape_1: 8800 -&gt; (80, 110)</td></tr></table>

CNN topology of the chirp encoder (audio2chirp).

Table content: <table/>

CNN topologies for TIMIT vowel experiments.

Table content: <table/>

Peer review (iclr2017):
Pros: - Introduction of a nice filter banks and its implementation - Good numerical results - Refinement of the representation via back propagation, and a demonstration that it speeds up learning Cons: - The algorithms (section 3.1) are not necessary, and they even affect the presentation of the paper. However, a source code would be great! - The link with a scattering transform is not clear - Sometimes (as mentionned in some of my comments), the writing could be improved. From a personal point of view, I also believe the negative points I mention can be easily removed.