Title: BIOACOUSTIC SEGMENTATION BY HIERARCHICAL DIRICHLET PROCESS HIDDEN MARKOV MODEL

Abstract: Understanding the communication between different animals by analysing their acoustic signals is an important topic in bioacoustics. It can be a powerful tool for the preservation of ecological diversity. We investigate probabilistic models to analyse signals issued from real-world bioacoustic sound scenes. We study a Bayesian non-parametric sequential models based on Hierarchical Dirichlet Process Hidden Markov Models (HDP-HMM). The model is able to infer hidden states, that are referred here as song units. However, using such a model raise one main issue: defining the number of hidden states the model has to learn. In bioacoustic problems we often do not know the number of song units (unlike in human speech recognition). Hence, we work with the Hierarchical Dirichlet Process (HDP)-HMM, which is a Bayesian non-parametric (BNP) model that offers a way to tackle this challenging problem. We focus our work on unsupervised learning from bioacoustic data. It consists in simultaneously finding the structure of hidden song units and automatically infer the unknown number of the hidden states to represent the data. Two real bioacoustic sound scene applications are investigated in this work: on whale and multi-species birds segmentation. The learning of these models is proceeded by using Markov-Chain Monte Carlo (MCMC) sampling techniques on Mel Frequency Cepstral Coefficients (MFCC) of audio signals. The results show an interesting song unit segmentation of the bioacoustic signals and open new insights for unsupervised analysis of such signals. This paper illustrates the potential of chunking non-human animal signals into structured parts. This can yield to a new species representation and help experts to better understand the behaviour of such species as Kershenbaum et al. (2014) wanted.

1. INTRODUCTION:

Acoustic communication is common in the animal world where individuals communicate with sequences of some different acoustic elements (Kershenbaum et al., 2014) . An accurate analysis is important in order to give a better identification of some animal species and interpret the identified song units in the course of time. In this paper, we automatically model the sequence of a non-human signal and determine their acoustic song units. As highlighted in Kershenbaum et al. (2014) , the way according to which non-human acoustic sequences can be interpreted can be summarized as shown in Acoustic units can be determined either manually (e.g. from spectrogram representation), or automatically (e.g. based on a model). Manual segmentation is time consuming and not possible for a large acoustic dataset. That is why automatic approaches are needed. Furthermore, in bioacoustic signals, the problem of segmenting signals of many species, is still an issue, including for bioacoustic. Hence, a well-principled learning system based on unsupervised approach can help them to have a better understanding of bioacoustics species. In this context, we investigate statistical latent data models to automatically identify song units.

First, we study Hidden Markov Models (HMMs) (Rabiner & Juang, 1986) . Which are the gold standard for sequential data, and thus could be relevant for acoustic data modeling and segmentation. The typically used algorithm to learn the model is the Expectation-Maximization (EM) algorithm (Dempster et al., 1977) , also known as Baum-Welch in HMMs (Baum et al., 1970) . The main issue with HMMs is the one of selecting the number of hidden states. Because of the lack of knowledge on non-human species, it is hard to have this number. This rises a model selection problem, which can be addressed by information selection criteria such as BIC, AIC (Schwarz, 1978; Akaike, 1974) , which select an HMM with a number of states from pre-estimated HMMs with varying number of states.

Such approaches are limited because they require learning multiple HMMs. On the other hand, nonparametric derivations of HMMs constitute a well-principled alternative to address this issue. This approach is more flexible than using a Bayesian non-parametric (BNP) formulation for HMMs (Teh et al., 2006) , also called the infinite HMM (iHMM) (Beal et al., 2002) . It allows to infer the number of states (segments, units) from the data. The BNP approach for HMMs relies on Hierarchical Dirichlet Process (HDP) to define a prior over the states (Teh et al., 2006) . It is known as the Hierarchical Dirichlet Process for the Hidden Markov Models (HDP-HMM) (Teh et al., 2006) . The HDP-HMM parameters can be estimated by MCMC sampling techniques such as Gibbs sampling. The standard HDP-HMM Gibbs sampling has the limitation of an inadequate modeling of the temporal persistence of states (Fox et al., 2008) . This problem has been addressed by Fox et al. (2008) by relying on a sticky extension which allows a more robust learning. Hence, we have a model to separate non-human signals into states that represent different activities (song units) and exploring the inference of complex data such as bioacoustic data in surroundings cases is not yet resolved.

In this paper, we investigate the BNP formulation of HMM, that is the HDP-HMM, into two challenges involving real bioacoustic data. First, a challenging problem of humpback whale song decomposition is investigated. The objective is the unsupervised structuration of whale bioacoustic data. Humpback whale songs are long cyclical sequences produced by males during the reproduction season which follows their migration from high-latitude to low-latitude waters. Singers from the same geographical region share parts of the same song. This leads to the idea of dialect (Helweg et al., 1998) . Different hypotheses of these songs were emitted (Medrano et al., 1994; Frankel et al., 1995; Baker & Herman, 1984; Garland et al., 2011) . Next, we investigate a challenging problem of bird song unit structuration. Catchpole & Slater (1995) ; Kroodsma & Miller (1996) show how birds sing and why birds have such elaborate songs. However, analysing bird song units is difficult due to the transientness of typical bird chirps, the large behavioural intra-class variability, the small amount of examples per class, the presence of wildlife noise, and so forth. As shown later in the obtained segmentation results, such automatic approaches allow large-scale analysis of environmental bioacoustics recordings 1.1 RELATED WORK Discovering the call units (which can be considered as a kind of non-human alphabet) of such complex signals can be seen as a problem of unsupervised call units classification as Pace et al. (2010) . Picot et al. (2008) also tried to analyse bioacoustic songs using a clustering approach. They implemented a segmentation algorithm based on Payne's principle to extract sound units from a bioacoustic song. In this paper we reformulate the problem of song decomposition as an unsupervised data classification problem. Contrary to the approach used by Pace et al. (2010) , in which the number of states (call units in this case) has been fixed manually, or the one used by Picot et al. (2008) , where a K-means algorithm is used for automatic classification and then automatically define the optimal number of classes by maximizing the Davies Bouldin criterion.

Our approach is based on a probabilist approach on the MFCC; it is a non-parametric formulation, that is well-suited to the problem of automatically inferring the number of the states in the data. In the next section we describe the real-world bioacoustic challenges we used and explain our approach.

2. DATA AND METHODS:

The data used represent the difficulties of bioacoustic problems, especially when the only information linked to the signal is the species name. Thus, we have to determine a sequence without ground truth.

2.1. HUMPBACK WHALE DATA:

Humpback whale song data consist of a recording (about 8.6 minutes) produced at few meters from the whale in La Reunion -Indian Ocean (NIP, 2013), at a frequency sample of 44.1kHz, 32 bits, one channel.

We extract MFCC1 features from the signal, with pre-emphasis: 0.95, hamming window, FFT on 1024 points (nearly 23ms), frameshift 10 ms, 24 Mel channels, 12 MFCC coefficients plus energy and their delta and acceleration, for a total of 39 dimensions as detailed in the NIPS 2013 challenge (NIP, 2013) where the signal and the features are available. The retained data for our experiment are the 51336 first observations.

2.2. MULTI-SPECIES BIRD DATA:

Bird species song data from Fernand Deroussen Jerome Sueur of Musee National d'Histoire Naturelle (F. Deroussen, 2006) , consists of a training and a testing set (not used here). Theses sets were designed for the ICML4B challenge2 . The recordings have a frequency sample of 44.1kHz, 16 bits, one channel. The training set is composed of 35 recordings, 30 seconds each taken from 1 microphone. Each record contains 1 bird species in the foreground for a total of 35 different birds species.

The feature extraction for this application is applied as follows. First, a high pass filter is processed to reduce the noise (set at 1.000 kHz to avoid noises). Then, we extract the MFCC features with windows of 0.06 seconds and shift of 0.03 seconds, we keep 13 coefficients, with energy as first parameter, to be compact and sufficient accurate, considering only the vocal track information and removing the source information. Also, we focus on frequencies below 8.000 kHz, because of the alterations into the spectrum. We obtain 34965 observations with 13 dimensions each for train set, that is used to learn our model.

2.3. METHOD: BIOACOUSTIC UNSUPERVISED LEARNING FOR SIGNAL REPRESENTATION:

To solve bioacoustic problems and finding the number of call units we propose to use the HDP-HMM model to model the complex bioacoustic data. Our approach automatically discovers and infers the number of states from the non-human song data.

In this paper we present two applications on bioacoustic data. We study the song unit structuration, for the humpback whale and for the multi-species birds signal.

In the next section we give a brief description of the Hidden Markov Model and it's Bayesian nonparametric alternative used in our bioacoustic signal representation applications.

3. BAYESIAN NON-PARAMETRIC ALTERNATIVE FOR HIDDEN MARKOV MODEL:

The finite Hidden Markov Model (HMM) is very popular due to its rich mathematical structure and its stability to model sequential data (e.g. acoustic data). It assumes that the observed sequence X = (x 1 , . . . , x T ) is governed by a hidden state sequence z = (z 1 , . . . , z T ), where x t ∈ R d is the multidimensional observation at time t and z t represents the hidden state of x t taking values in a finite set {1, . . . , K}, K being the possible number of states, that is unknown. The generative process of the HMM can be described in general by the following steps. First, z 1 follows the initial distribution π 1 . Then, given the previous state (z t-1 ), the current state z t follows the transition distribution. Finally, given the state z t , the observation x t follows the emission distribution F (θ zt ) of that state. The HMM parameters, that are the initial state transition (π 1 ), the transition matrix (π), and the emission parameters (θ) are in general estimated in a maximum likelihood estimation (MLE) framework by using the Expectation-Maximization (EM) algorithm, also known as the Bauch-Welch algorithm (Baum et al., 1970) in the context of HMMs.

Therefore, for the finite HMM, the number of states K is required to be known a priori. This model selection issue can be addressed in a two-stage scheme by using model selection criteria such as the Bayesian Information Criterion (BIC) (Schwarz, 1978) , the Akaike Information Criterion (AIC) (Akaike, 1974) , the Integrated Classification Likelihood criterion (ICL) (Biernacki et al., 2000) , etc to select a model from pre-estimated HMMs with varying number of states. Such approaches is limited, it requires learning N HMMs, N being sufficiently high to have an equivalent of a non parametric approach. Regardless this, a non parametric approach is more efficient because it theoretically tends to an infinite number of states. Thus, we use a Bayesian non-parametric (BNP) version of the HMM, that is able to infer the number of hidden states from the data. It is more flexible than learning multiple HMM, because in bio-acoustic problems the model have to characterize multiple species/individuals, thus it possibly tends to a large number of hidden states. Thence, exploring the inference of complex data such as bioacoustic data in surroundings cases is new.

The BNP approach for the HMM, that is the infinite HMM (iHMM), is based on a Dirichlet Process (DP) (Ferguson, 1973) needs to be used. However, due to the transitions of states take independent priors, there is no coupling across transitions between different states Beal et al. (2002) , therefore the DP is not sufficient to extend the HMM to an infinite model. The Hierarchical Dirichlet Process (HDP) prior distribution on the transition matrices over countability infinite state space, derived by Teh et al. (2006) , extends the HMM to the infinite state space model and is briefly described in the next subsection.

3.1. HIERARCHICAL DIRICHLET PROCESS (HDP):

Suppose the data subdivided into J groups, each produced by a related, yet distinct process. The HDP extends the DP by an hierarchical Bayesian approach such that a global Dirichlet Process prior DP(α 0 , G 0 ) is drawn from a global prior G j , where G 0 is itself a Dirichlet Process distribution with two parameters, a base distribution H and a concentration parameter γ. The generative process of the data with the HDP can be summarized as follows. Suppose data X, with i = 1, . . . , T observations that is grouped into j = 1, . . . , J groups. Note that the observations of the group j are given by X j = (x j1 , x j2 , . . .), all observations of group j being exchangeable. Assume each observation is drawn from a mixture model, thus each observations x ji is associated with a mixture component, with parameter θ ij . Note that from the DP property, we observe equal values in the components θ ij . Now, giving the model parameter θ ji , the data x ji is drawn from the distribution F (θ ji ). Assuming a prior distribution G j over the model parameters associated for group j, θ j = (θ j1 , θ j2 , . . .), we can define the generative process in Eq. ( 1).

G 0 |γ, H ∼ DP(γ, H), G j |α 0 , G 0 ∼ DP(α 0 , G 0 ), ∀j ∈ 1, . . . , J, θ ji |G j ∼ G j , ∀j ∈ 1, . . . , J and ∀i ∈ 1, . . . , T , x i |θ ji ∼ F (x i |θ ji ), ∀j ∈ 1, .

. . , J and ∀i ∈ 1, . . . , T .

(1)

The Chinese Restaurant Process (CRP) (Pitman, 1995) is a representation of the Dirichlet Process that results from a metaphor related to the existence of a restaurant with possible infinite tables (clusters) where customers (the observations) are sitting in it. An alternative of such a representation for the Hierarchical Dirichlet Process can be described by the Chinese Restaurant Franchise (CRF) process by extending the CRP to multiple restaurants that share a set of dishes.

The idea of CRF is that it gives a representation for the HDP by extending the Chinese Restaurant Process to a set of (J) restaurants, rather than a single restaurant. Suppose a patron of Chinese Restaurant creates many restaurants, strongly linked to each other, by a franchise wide menu, having dishes common to all restaurants. As a result, J restaurants are created (groups) with a possibility to extend each restaurant with an infinite number of tables (states) at witch the customers (observations) sit. Each customer goes to his specified restaurant j, where each table of this restaurant has a dish that shares between the customers that sit at that specific table. However, multiple tables of different existing restaurants can serve the same dish.

3.2. THE HIERARCHICAL DIRICHLET PROCESS FOR THE HIDDEN MARKOV MODEL (HDP-HMM):

The HDP-HMM uses a HDP prior distribution providing a potential countability infinite number of hidden states and tackles the challenging problem of model selection for the HMM. This model is a Bayesian non-parametric extension for the HMM also presented as the infinite Hidden Markov Model (Beal et al., 2002) . To derive the HDP-HMM model we suppose a doubly-infinite transition matrix, where each row corresponds to a CRP. Thus, in a HDP formalism, the groups correspond to states, with CRP distribution on next states. CRF links these states distributions.

We assume for simplicity a distinguished initial state z 0 . Let G j describes both, the transition matrix π k and the emission parameters θ k , the infinite HMM can be described by the following generative process:

EQUATION

where, β is a hyperparameter for the DP (Sethuraman, 1994) distributed according to the stick-breaking construction noted GEM(.); z t is the indicator variable of the HDP-HMM that follows a multinomial distribution Mult(.); the emission parameters θ k , are drawn independently, according to a conjugate prior distribution H; F (θ zt ) is a data likelihood density with the unique parameter space of θ zt equal to θ k .

Suppose the observed data likelihood is a Gaussian density N (x t ; θ k ) where the emission parameters θ k = {µ k , Σ k } are respectively the mean vector µ k and the covariance matrix Σ k . According to Gelman et al. (2003) , the prior over the mean vector and the covariance matrix is a conjugate Normal-Inverse-Wishart distribution, denoted as N IW(µ 0 , κ 0 , ν 0 , Λ 0 ), with the hyper-parameters describing the shapes and the position for each mixture components: µ 0 is the mean of Gaussian should be, κ 0 the number of pseudo-observations supposed to be attributed, and ν 0 , Λ 0 being similarly for the covariance matrix.

In the generative process given in Eq. ( 2), π is interpreted as a double-infinite transition matrix with each row taking a CRP. Thus, in the HDP formulation "the group-specific" distribution, π k corresponds to "the state-specific" transition where the CRF defines distributions over the next state.

In turn, Fox et al. (2008) showed that HDP-HMM inadequately models the temporal persistence of states, creating redundant and rapidly switching states and proposed an additional hyperparameter κ that increase the self-transition probabilities. This is named as sticky HDP-HMM. The distribution on the transition matrix of Eq. ( 2) for the sticky HDP-HMM is given as follows:

EQUATION

where a small positive κ > 0 is added to the k th component of αβ, thus of self-transition probability is increased by κ. Note that setting κ to 0, the original HDP-HMM is recovered. Under such assumption for the transition matrix, Fox et al. (2008) proposes an extension of the CRF to the Chinese Restaurant Franchise with Loyal Customers. A graphical representation of (sticky) HDP-HMM is given in Fig 5 .

The inference of the infinite HMM (the (sticky) HDP-HMM) with the Block Gibbs sampler algorithm is given in Algorithm 3 of Supplementary Material in Fox et al. (2008) paper. The base idea of this sampler is to estimate the posterior distributions over all the parameters from the generative process of (sticky) HDP-HMM given in Eq. ( 2). Here, the CRF with Loyal Customers, hyperparameter κ of the transition matrix can be sampled in order to increase the self-transition probability.

Hence, the HDP-HMM model resolves the problem of advanced signal decomposition using acoustic features with respect to time. It allows identifying song units (states), behaviour and enhancing populations studies. From the other point, modelling data with the HDP-HMM offers a great alternative of the standard HMM to tackle the challenging problem of selecting the number of states, identifying the unknown number of hidden units from the used features (here: MFCC). The experimental results show the interest of a such approach.

4. EXPERIMENTS:

In this section we present two applications on bioacoustic data. We study the song unit structuration, for the humpback whale signal and for multi-species birds signals.

4.1. HUMPBACK WHALE SOUND SEGMENTATION:

The learning of the humpback whale song, applied via the HDP-HMM, is done with the Blocked Gibbs sampling. A number of iterations was fixed to N s = 30000 and a truncation level, that corresponds to the maximum number of possible states in the model (being sufficient big to approximate it to an infinite model), is fixed to L k = 30. The number of states estimated by the HDP-HMM Gibbs sampling is 6.

The Fig 6 shows the state sequences partition, for all 8.6 minutes of humpback whale song data, obtained by the HDP-HMM Gibbs sampling. For more detailed information, the result of the whole humpback whale signal segmentation is separated by several parts of 15 seconds. All the spectrograms of the humpback whale song and the obtained segmentation are made available in the demo: http://sabiod.univ-tln.fr/workspace/ICLR2017/whale/. This demo highlights the interest of using a BNP formulation of HMMs for unsupervised segmentation of whale signals.

Three examples of the humpback whale song, with 15 seconds duration each, are presented and discussed in this paper (see Fig 1) .

Figure 1 represents the spectrogram and the corresponding state sequence partition obtained by the HDP-HMM Gibbs inference algorithm. They respectively represent examples of the beginning, the middle and the end of the whole signal. All the obtained state sequence partitions fit the spectral patterns. We note that the estimated state 1 fits the sea noise, state 5 also fits sea noise, but it is right before units associated to whale songs. The presence of this unit can be due to an insufficient number of Gibbs samples. For a longer learning the fifth state could be merged with the first state.

State 2 fits the up and down sweeps. State 3 fits low and high fundamental harmonic sounds, state 4 fits for numerous harmonics sound and state 6 fits very noisy and broad sounds. Fig 7 shows two spectrograms extracted from the 6 th song unit (left) and from the 2 nd song unit (right) of the whole humpback whale signal. We can see that the units fit specific patterns on the whole signal.

Pr. Gianni Pavan (Pavia University, Italy), an undersea NATO bioacoustic expert analysed the results on the humpback whale song segmentation, during his stay at DYNI in 2015. He validated the proposed representation. This highlight the interest of learning BNP model on a single species.

Next, we will see how such model reacts with multiple species.

4.2. BIRDS SOUND SEGMENTATION:

In this section we describe the obtained bird song unit segmentation. We segment the bird signals into song units by learning the HDP-HMM model on the training set (containing 35 different species). The main goal is to see if a such approach can model multiple species. Note that in this set, we assume there is no multiple species singing at the same time. For this application, we considered 145000 Gibbs iterations and a truncation level of 200 for the maximum number of states. We suppose them to be sufficiently big for this data problem. Moreover, we use one mixture component per state, that appeared to give satisfactory results and we use a sticky HDP-HMM with the hyper-parameter κ set to 0.1.

We discovered 76 song units with this method. For more detailed information over the signal, we separated the whole train set into parts of 15 seconds each. All the spectrograms and the associated segmentation obtained are made available in the demo: http://sabiod.univ-tln.fr/ workspace/ICLR2017/bird/. 

4.2.1. EVALUATION OF THE BIRD RESULT:

To evaluate the bird results, we used a ground truth produced by Simone Clemente (bird expert). We ask him to segment each recording of the dataset according to the different patterns on the signal. Then we compare this ground truth with the segments produce by the model using NMI (Strehl & Ghosh, 2002) which calculates shared information between two clustering sets.

First, we compute the NMI score between these two segmentations and we obtain a score of 0.490. Thus, the global segmentation from the model isn't near from a segmentation done by an expert. Second, we compute the NMI score for each species to see the possible mistake done by the model.

Tab 1 shows the different scores obtained with a resulting mean score of 0.367. The highest score is 0.680 (corvus corone) and the lowest score is 0.003 (garrulus glandarius). Thus, for some species, the model has difficulties to segment the data. Sometimes, it uses less states than the expert to segment the data: for the oriolus oriolus (golden oriole), the model identifies 12 song units versus 50 identified by the expert. Nevertheless, the model also uses more states than the expert to segment the data: for the fringilla coelebs (chaffinch), the model identifies 15 song units versus 3 identified by the expert. In other cases, the model can't differentiate 2 distinct vocalizes if they have close frequencies (phylloscopus collybita and columba palumbus), background and foreground species (streptopelia decaocto). This can be due to the feature used or an insufficient number of iterations of the Gibbs sampling. For most of species, the model and the ground truth have similar patterns observable on Fig 8.

To improve the model, we can investigate better feature representation for species with different acoustic characteristics. We can also improve noise reduction which could be useful for background activities. Nevertheless, the application highlights the interest of using BNP formulation of HMMs for unsupervised segmentation of bird signals.

5. CONCLUSIONS:

In this work, we relied on real world bioacoustic applications and propose to use a BNP formulation of HMMs to realize a representation of bio-acoustic signals. It is a response for (Kershenbaum et al., 2014) . We investigated this approach on real-world bioacoustic signals from two challenges. The demo for the two applications are available online.

The obtained signal segmentation on the bioacoustic data is recovered in a fully automatic way and is proceeded by a Hierarchical Dirichlet Process for Hidden Markov Model on MFCC. The BNP formulation give an estimate number of cluster needed to segment the signal and our experiments highlight the interest of such formulation on bioacoustic problems. Furthermore we compare the segmentation obtained for birds with the segmentation from an expert and the model using NMI, the results are promising. We describe a full bioacoustic perspective in the annexes.

However, the model used is computational expensive and not suitable larger dataset. Future work will consist in studying methods that should accelerate the MCMC sampling and dealing with larger data problems, like variational inference (Jordan et al., 1999) or stochastic variational inference used for HMMs (Foti et al., 2014) .

Future work will consist in considering our segmentation results for classification task, where the goal is to identify the species. automatic detection of vocalized species, which would help specialists in ecological works during their monitoring activities. Although some works already have reach good classification percentage 3 4 , there's a lack of methodologies available for works focused on real world data, and with further applications in ecology and wildlife management. One of the major bottlenecks for the application of these methodologies is their inability to work under heavy complex acoustic environment, where different taxa may sing together or conversely, their extreme sensitivity which may result in an "over classification" due to the high degree of variability insight many repertoire of the vocal species.

Our unsupervised method for automatic annotation of bioacoustics sequences seems to overtake these obstacles, by the identification of specie-specific pattern, and seems to be not influenced by the inter-individual variation whiting the song's structure. Furthermore, during the study, we have worked whit recording that may contain more than a species vocalizing, or partial overlapping from different species and specimen, and even in these circumstance our model has shown a great ability of categorisations and generalisations, identifying the main pattern and almost all the other sequences recorded (including silence).

Good examples are provided, from the birds' dataset, by recordings including the common wood pigeon (Columba palumbus; Figure 3 (top)), or from the files containing golden oriole (Oriolus oriolus) as a main species. Their vocalisations are partially overlapped by vocalisation of different species, such as Eurasian blue tit (Cyanistes caeruleus, figure 3 (bottom)) or cricket, and is clear in the analysis, the distinction between the bird vocalising and other animals in the background. Finally, a last example could be taken from the carrion crow (Corvus corone, figure 3 (middle) recordings. Here, multiple specimens are vocalising at the same time, but it does not influence the efficiency of our method.

By the other hand, in presence of species with complex acoustics behaviours, such as the common nightingale (Luscinia megarhynchos) the method could lead to identify as different classes each one of the different phrasing elements in the song. This last approach could be useful in order to perform behavioural analysis focused on the identification of hidden significance in the songs, but may loose his benefit when the main achievement is an ecological topic. We finally consider this method as a promising tool, as we move forward, but further analysis will be needed to build, an efficient tool whit relevant application in the acoustic monitoring and conservation of biodiversity. 

Species:



:

The MFCC are features that represent and compress short-term power spectrum of a sound. It follows the Mel scale.

http://sabiod.univ-tln.fr/nips4b/challenge2.html

Automatic large-scale classification of bird sounds is strongly improved by unsupervised feature learning; Dan Stowell and Mark D Plumbley; 2014; PeerJ

Automatic classification of a taxon-rich community recorded in the wild; Potamitis and Ilyas; 2014; PloS on

Figures and Tables:

Fig 4. We distinguish four common properties that are used to define potential criteria for segmenting such signals into song units. The first way, shown in Fig 4(A), consists in separating the signals using silent gaps. The second way, shown in Fig 4(B), consists in separating the signals according to the changes in the acoustic properties in the signal. The third way, shown in Fig 4(C) consists in grouping similar sounds separated with silent gaps as a single unit. The last common way, shown in Fig 4(D) consists in separating signal in organized sound structure, considered as fundamental units.

Figure 1: Obtained song units starting at 60 seconds (left), 255 seconds (middle) and 495 seconds (right). The spectrogram of the whale song (top), and the obtained state sequence (bottom) by the Blocked Gibbs sampler inference approach for the HDP-HMM. The silence (unit 1 and 5) looks well separated from the whale signal. Whale up and down sweeps (unit 2), harmonics (unit 3 and 4) and broad sounds (unit 6) are also present. See fig 13 for bigger figures.

Fig 2 contains three examples of bird bioacoustic segmentation. The species are: Carduelis chloris (left), Luscinia megarhynchos (middle) and Parus caeruleus (right).

Figure 2: Obtained song units on 15 seconds of Carduelis chloris bird song. The spectrogram of the bird song (top), and the obtained state sequence (bottom) by the Blocked Gibbs sampler inference approach for the HDP-HMM. Three song units persist in this sound. The silence looks well separated from the bird signal. Furthermore, we can denote that song units fit well the song and similarity in spectrum persists for each song unit. See fig 14 for bigger figures.

Figure 3: The spectrogram of the bird song and the obtained state sequence by the Blocked Gibbs sampler inference approach for the HDP-HMM. Columba palumbus with cricket noises activities (top); different specimens of Corvus corone (middle); Oriolus oriolus partially overlapped whit Cyanistes caeruleus on the backgound (bottom).

Figure 8: Fringilla coelebs results. The first graph represents the labelled ground truth over 30s where label 0 is always the "none" label and the other labels are not the same from one graph to an other. The second graph represents our model with the 76 classes. The last one is the spectrogram.

Figure 9: Corvus corone results. See caption of 8 for details.

Figure 10: Garrulus glandarius results. See caption of 8 for details.

Figure 11: Motacilla alba results. See caption of 8 for details.

NMI score for the obtained segmentation using HDP-HMM.

Table content: <table><tr><td/><td>NMI Score</td></tr><tr><td>sturnus vulgaris</td><td>0.467</td></tr><tr><td>turdus philomelos</td><td>0.398</td></tr><tr><td>emberiza citrinella</td><td>0.534</td></tr><tr><td>certhia brachydactyla</td><td>0.417</td></tr><tr><td>columba palumbus</td><td>0.352</td></tr><tr><td>picus viridis</td><td>0.602</td></tr><tr><td>anthus trivialis</td><td>0.332</td></tr><tr><td>phasianus colchicus</td><td>0.272</td></tr><tr><td>cuculus canorus</td><td>0.205</td></tr><tr><td>sylvia atricapilla</td><td>0.405</td></tr><tr><td>corvus corone</td><td>0.68</td></tr><tr><td>phylloscopus collybita</td><td>0.267</td></tr><tr><td>streptopelia decaocto</td><td>0.306</td></tr><tr><td>turdus viscivorus</td><td>0.417</td></tr><tr><td>dendrocopos major</td><td>0.481</td></tr><tr><td>erithacus rubecula</td><td>0.394</td></tr><tr><td>pavo cristatus</td><td>0.437</td></tr><tr><td>fringilla coelebs</td><td>0.565</td></tr><tr><td>aegithalos caudatus</td><td>0.202</td></tr><tr><td>turdus merula</td><td>0.395</td></tr><tr><td>branta canadensis</td><td>0.339</td></tr><tr><td>parus palustris</td><td>0.521</td></tr><tr><td>sitta europaea</td><td>0.332</td></tr><tr><td>alauda arvensis</td><td>0.169</td></tr><tr><td>prunella modularis</td><td>0.476</td></tr><tr><td>oriolus oriolus</td><td>0.316</td></tr><tr><td>carduelis chloris</td><td>0.385</td></tr><tr><td>phoenicurus phoenicurus</td><td>0.291</td></tr><tr><td>strix aluco</td><td>0.2</td></tr><tr><td>parus caeruleus</td><td>0.413</td></tr><tr><td>parus major</td><td>0.27</td></tr><tr><td>motacilla alba</td><td>0.105</td></tr><tr><td>luscinia megarhynchos</td><td>0.497</td></tr><tr><td>troglodytes troglodytes</td><td>0.407</td></tr><tr><td>garrulus glandarius</td><td>0.003</td></tr><tr><td>mean</td><td>0.367</td></tr></table>

Peer review (iclr2017):
The paper is a novel application for the sticky HDP-HMM, focused on correctly identifying the number of components in bird and whale song across a variety of datasets. It*s nice to see the model applied to an interesting dataset. My main issues with the paper have to do with structure and the choice of representation used in the model. Namely: The organization of the paper could be significantly improved. There is a lot of repetitive introduction that adds little to the paper. The first and last two sentences of the abstract could be cut. Many other parts of the abstract basically repeat the introduction. The second paragraph of section 2.3 also repeats your introduction - by now we know what you*re doing. I think most people reading this will have no idea what Kershenbaum (2014) is. The description of the data should go in the experiments section. *Different hypotheses for the songs were emitted* in the introduction is odd phrasing. Figure 4 should be the first figure and go in the introduction. Figure 5 should be in the methods section. A summary of Table 1 should be in the experiments section. Generally the writing could be tightened quite a bit, which would make space for these figures. The description of the HDP-HMM, which mostly follows the existing literature, is well done. Some general questions about the methods used: If you*re interested in scalable inference, why use Gibbs sampling? Why not the beam sampler (van Gael 2008), which at least recently was the state of the art for MCMC inference in the HDP-HMM? More generally, why use MCMC at all? For very large datasets, most of the Bayesian ML community has converged on stochastic variational inference as the most practical method (eg Wang, Paisley and Blei 2011). If your interest is mainly in the number of clusters, how would you address the fact that DP mixture models are known not to be consistent for estimating the true number of clusters (Miller and Harrison 2013)? MFCC features are calibrated to the human auditory system, not bird or whale auditory systems. In your data, do you calibrate the MFCC scale to be closer to the auditory systems of the animals that generated the song? And a final suggestion for future work, which could use the results presented here as a baseline: Given the success of LSTMs in speech recognition in recent years, it may be the case that deep learned representations are superior to linear features (like the means of each cluster in an HDP-HMM) for animal song as well. Have you considered a hybrid model, similar to recent work combining autoencoders and graphical models (Johnson, Duvenaud, Wiltschko, Datta and Adams 2016)?