Acoustic Event Detection for Spotting “Hot Spots” in Podcasts
Kouhei Sumi† , Tatsuya Kawahara† , Jun Ogata‡ , and Masataka Goto‡†

Graduate School of Informatics, Kyoto University
Sakyo-ku, Kyoto 606-8501, Japan,‡
National Institute of Advanced Industrial Science and Technology (AIST)
Tsukuba, Ibaraki 305-8568, Japan
Abstract
This paper presents a method to detect acoustic events thatcan be used to ﬁnd “hot spots” in podcast programs. We focus on meaningful non-verbal audible reactions which suggest
hot spots such as laughter and reactive tokens. In order to detect this kind of short events and segment the counterpart utterances, we need accurate audio segmentation and classiﬁcation,
dealing with various recording environments and backgroundmusic. Thus, we propose a method for automatically estimating and switching penalty weights for the BIC-based segmentation depending on background environments. Experimental
results show signiﬁcant improvement in detection accuracy byproposed method compared to when using a constant penalty
weight.
Index Terms: acoustic event detection, laughter detection, podcast, Bayesian Information Criterion

Hot spot

Figure 1: Audible reaction and hot spotin back-channel responses during dialogue suggest a level of interest in the topic of conversation [4]. By detecting these meaningful audible reactions, we expect we can detect “hot spots”
without recognizing the speech. Here, we deﬁne hot spots assegments in which the listeners would be interested in the audio
data 4 , and hot spot candidates exist right before these eventsas shown in Figure 1. Our goal is to detect not only the audible reactions but also segment the preceding utterances, which
potentially make up hot spots.
The rest of the paper is organized as follows: section 2
describes the conventional segmentation method based on BIC
and the issues involved in detecting acoustic events in podcasts.
In section 3, we present a method for optimally deciding penalty
weights for the BIC-based method. Our proposed system is described in section 4, and the experimental results are reported in
section 5. Finally, conclusions and future work are discussed in
section 6.

1. Introduction
In recent years, there has been an increase of audio media (MP3audio ﬁles, etc.) on the internet, including podcasts, internet radio, and audio blogs. Unlike text-based content, audio content
cannot be fully understood if we do not listen to the entirety of
the content. Thus, it is difﬁcult to search and browse audio content because speech, music, and sound effects are not visible.
As a solution to this problem, some applications such as
Podscope 1 , Google Audio Indexing 2 , and PodCastle 3 havebeen developed to search and browse through transcriptions ofaudio data by applying automatic speech recognition (ASR). In
Podcastle, language model is adaptated using RSS metadata,
and acoustic model training is enhanced by user-corrected transcripts [1, 2]. Google Audio Indexing transcribes election video
material and makes the content searchable [3]. However, eventhe state-of-the-art ASR systems cannot accurately transcribe
all audio data, because their content and recording environments are of a wide variety. They contain not only speech but
also music, sound effects, and environmental sound. Anotherproblem for ASR is that most of speech is spontaneous, conversational, and often in dialogue-style.
To circumvent these difﬁculties in ASR, we focus on audible reactions, present in the content, which have non-verbal
information and were ignored in the traditional ASR. For example, laughter occurs after talking about interesting things.
Applause often comes after utterances which impress the audience favorably. In addition, a certain kind of reactive tokens

2. Acoustic event detection in podcasts
In this work, we focus on laughter and reactive tokens which arefrequently observed acoustic events in podcasts and presumed
to be closely related to hot spots. We also address the problem
of segmentation of speech by different speakers and cassiﬁcation of speech, music, and speech mixed with music. We regard
all these categories as acoustic events in this paper.
2.1. Audio segmentation and classiﬁcation in podcasts
The conventional framework for acoustic event detection(AED) or audio segmentation and classiﬁcation is based oneither explicit models or metrics. These methods are mainly
studied targeting at broadcast news and recordings of meetings.
Whereas broadcast news rarely contain dialogue, there are many
dialogue-style programs in podcasts, thus containing a numberof short utterances. Moreover, background music is often usedin podcasts while it never appears in meetings. These issues
combined make it very difﬁcult to conduct audio segmentationand classiﬁcation of podcasts.

1 http://www.podscope.com/

4 Wrede et al. [5] used the term “hot spots” for regions in which two
or more participants are highly involved in a meeting discussion. Ourdeﬁnition is different from this.

2 http://labs.google.com/gaudi
3 http://podcastle.jp/

Copyright © 2009 ISCA

Audible reaction1143

6 - 10 September, Brighton UKNmix-GMM

2.1.1. Model-based methods

Gaussian 1

In the model-based methods, classiﬁers are trained for a given
set of pre-determined acoustic classes, using Gaussian Mixture
Models (GMMs), Hidden Markov Models (HMMs), or Support Vector Machines (SVMs) [6, 7]. For example, Knox et al.
presented a method for detection of laughter in meetings usingneural networks [8].
In podcasts, we observe frequent speaker changes as well as
acoustic events of short duration. Thus, it is not easy to apply
explicit models because the decision made with the features ofshort duration is not reliable, inﬂuenced by local variations.

Gaussian m
Gaussian N(1)where di is the number of parameters of model Mi , and P is
the maximum likelihood of the data under model Mi .
BIC-based segmentation is realized by computing the BIC
difference between two models: model M0 where X =xk , k = 1, 2, · · · , N is identically distributed to a single Gaussian N (µ0 , Σ0 ), and model M12 where xk , k = 1, 2, · · · , j isdrawn from one Gaussian N (µ1 , Σ1 ) and xk , k = j + 1, j +2, · · · , N is drawn from another Gaussian N (µ2 , Σ2 ). Then,∆BIC(j) = BIC(M0 ) − BIC(M12 ) is derived as follows:∆BIC(j) =Dual partitioning

Gaussian 1-2

Gaussian m-1

Gaussian m-2

Gaussian N-1

Gaussian N-2

Figure 2: Dual partitioning of each Gaussian

Segmentation of the audio input is accomplished by measuring
the “uniformity” or distance with different segmentation models. The methods do not require training data though they cannot classify the segments into some category. One of the mostwidely-used method is based on the Bayesian Information Criterion (BIC) [9, 10]. BIC is used to determine which of parametric models M = M1 , M2 , · · · , Mm best represents N data
samples D = D1 , D2 , · · · , DN . According to the BIC theory,the best model maximizes BIC(Mi ) as follows:1
λdi log N
2

Dual partitioning

Gaussian 1-1

Optimal mixture size

2.1.2. Metrics-based segmentation method

BIC(Mi ) = log P (D1 , D2 , · · · , DN |Mi ) −Dual partitioning

and tones, apparently acoustic features in music segments havemore variations than in speech segments, and music segments
tend to be split too much if we use the same penalty weight
λ. On the other hand, when several persons speak in the presence of background music (mix segments), it is hard to ﬁnd
speaker changes because acoustic features are smeared by the
background music, usually monotone music, while there can bea variety of music in music-only segments.
Based on the observation, we set up these global threeclasses (speech, music and mix) and corresponding classiﬁers based on GMM as a preprocessing step before the BICbased segmentation. For each classiﬁed segment, the BICbased segmentation is applied by selecting the penalty weights
λspe , λmix , λmus accordingly.
3.2. Automatic estimation of penalty weight of BIC for segmentation

1(N log |Σ0 | − j log |Σ1 | − (N − j) log |Σ2 |)2
1
1− λ(d + d(d + 1)) log N(2)22

where N is the sample number of the merged segment, d isthe dimension of the feature space, and λ is called the penalty
weight. If j is chosen such that j = arg max1<j<N ∆BIC(j)and ∆BIC(j) > 0, the frame j is identiﬁed as a segment
boundary. One problem with this method is that the penalty
weight λ is usually task-dependent and must be tuned for every
new task [10].

We estimate the appropriate penalty weights in the GMM training phase. When a GMM for each class is properly set up (withthe same mixture size), we expect each Gaussian of the GMM
represents a proper cluster size, and the penalty weight λ for the
BIC-based segmentation should be decided depending on thedispersion of the Gaussian distributions. We assume that when
sufﬁcient training data is available and the number of the mixtures is large enough, each Gaussian of the GMM corresponds
to a uniform segment and should not be split any more. On theother hand, clusters of a larger size (smaller mixture size) musthave been split. Thus, ∆BIC for one single Gaussian of the
ﬁnal GMM and dual partitioned Gaussians from it, as shown in
Figure 2, should satisfy the following formula.
∆BIC =In this work, we adopt a scheme that ﬁrst applies the BICbased segmentation and then classiﬁes each segment using the
GMMs. For segmentation of podcasts with the BIC-based
method, it is hard to ﬁx the appropriate penalty weight because
acoustic characteristics are different when background music
exists and when it does not. Therefore, we propose to switch
the penalty weight λ depending on the audio characteristics.

1((nGm1 + nGm2 ) log |ΣGm |2−nGm1 log |ΣGm1 | − nGm2 log |ΣGm2 |)1
1− λm (d + d(d + 1)) log(nGm1 + nGm2 ) ≈ 0
22(3)

where m = 1, · · · , N is the index of the Gaussians, and nGm1
and nGm2 are the number of samples contributing to Gaussian
m-1 and Gaussian m-2 in the context of the EM estimation. Actually, the prseudo-equation (3) would stand on the average for
all Gaussians, m = 1, · · · , N , thus we calculate ∆BIC for all
Gaussians of the GMM, and compute the average of λ derived
by ∆BIC being equal to zero, to determine the penalty weight
for each global class.

3. Estimating penalty weight of BayesianInformation Criterion
3.1. Statistical characteristics of speech, music, and theirmix

3.3. Estimation result

The variance of acoustic features is different for speech, music,
and also their mix. Because of a wide variety of instruments

Using the training data described in section 5, the penalty
weights λspe , λmus , λmix for the global three classes were esti-

1144

Estimation of
GMM parameters

Training
data

Input audio

GMMs
for each class

Feature extraction
3-class segmentationand classification

Penalty weights
Estimation of penalty for each class

BIC-based segmentation

value of BIC

Training phase

Filled pauses

Table 1: Baseline training data set
clustertraining data
Speech (male and female)
JNAS [12]Music
RWC-MDB [13]Mix (male and female)
JNAS+RWC-MDB
Silence
JNAS, Self-synthesized
Laughter
IMADE corpus [14], Web

GMM-classification

Proposed method

75

Speech Recognition

Detectingreactive tokens

DetectinglaughterLambda=1.0

74.5

83.5

73

83

72.5

82.5

72
71.5

mated to be 1.68, 3.48, and 1.22, respectively. We set the number of mixtures for each GMM to 256, which is obtained by
the optimization by maximizing the BIC (Equation 1) for iterative splits of mixtures from N = 1, 2, 4, to 512. The values ofpenalty weights are reasonable because music segments, whichshould not be frequently segmented, obtained a larger penalty
weight, and mix segments, which is not easily segmented, received a smaller penalty weight compared to speech segments.

Lambda=2.0

84

73.5

Figure 3: System overview

85

84.5

74

Speech and music
detection

Lambda=1.5(a) program-open

82(b) program-closed

Figure 4: Frame-wise accuracy of eight-class segmentation andclassiﬁcation
4. Until the end of input stream, Step 2 and 3 are repeated.
4.3. Laughter, speech and music segment detection

4. Proposed AED system

Each segment obtained by the BIC-based segmentation is classiﬁed among laughter, male-speech, female-speech, male-mix,
female-mix, music, and silence on the basis of the log likelihood of respective GMMs. Speech segments which are shorter
than a constant duration tres (2.0 sec) and have larger log likelihoods of the reactive-token-GMM than a threshold are extracted
as candidates of reactive tokens.

An overview of our proposed acoustic event detection system
is shown in Figure 3. The following sub-sections describe thedetails of each process.
4.1. Training phase and feature extraction
The parameters of each GMM are trained with feature vectors
calculated at frame level. Next, the EM algorithm is used toestimate means, diagonal variance matrices, and weights for
Gaussian components. Additionally, we estimated the penalty
weights for global classes, as mentioned above.
As acoustic features, we use a 26-dimensional vector which
consists of mel-frequency cepstral coefﬁcients (MFCC), delta
MFCC, energy, and delta energy. They are calculated with a 25
msec window (10 msec shift), and normalized by the mean andvariance for the entire training data.

4.4. Reactive token detection
Chang et al. [4] reported that three reactive tokens of “hu:N”,“he:” and “a:” suggest interest level of listeners in Japanese dialogue. Apparently the prolonged vowel can be used as a clue todetect these tokens, thus we utilize the same algorithm as ﬁlled
paused detection [11]. In addition, there is a lexical difference
between these tokens and ﬁllers, so we conduct simple ASR to
identify the tokens.
In summary, detection of reactive tokens is realized as follows:

4.2. Preprocessing and BIC-based segmentation

1. Filled pause detection is applied to each candidate, andthe segments which include a prolonged vowel are selected.

As a preprocessing step, we coarsely segment and classify the
input stream using the GMMs for the global classes (speechGMM, music-GMM, and mix-GMM). By selecting the different penalty weight for each segment, the BIC-based segmentation is expected to operate appropriately for further segmentation. For accurate segmention, we adopt the variable window
scheme described as follows:

2. ASR with a lexicon of ﬁller entries and the three tokens
is conducted, and reactive tokens are detected.

5. Experimental results
We tested our system on eight podcast episodes by choosing two
episodes from four different programs. For training the GMMs,
we used the baseline training data set as listed in Table 1. In
addition, we prepared two sets of podcasts: (a) program-open
set: 19 episodes from programs other than those in the test set,and (b) program-closed set: 23 (19+4) episodes including one
previous episode of each program used in the test set.
To evaluate the effectiveness of proposed method, we measure the frame-wise accuracy, laughter detection accuracy, and

1. The window size is initialized by the minimal window
size Wmin (100 frames).
2. If no segment boundary is found in the current window,
the size of the window is increased by adding Wmin until
a segment boundary is found.
3. When a segment boundary is found, the next window begins with the detected boundary, using the minimal window size.

1145

of approximately 70% if we use only ﬁlled pause detection. A
trade-off between the recall and the precision can be adjusted bytuning the thresholds in ﬁlled pause detection and GMM-basedclassiﬁcation.

Table 2: Detection accuracy of laughter
Measures
Proposed method
λ = 1.0
λ = 1.5
λ = 2.0

R
65.0
91.3
74.2
60.0

P
71.3
26.4
42.2
57.5

F
68.7
30.5
45.9
57.5

6. Conclusions
We have presented a method for detecting acoustic events in
podcast programs based on the BIC-based segmentation whichuses dedicated penalty weights for different global classes(speech, music and mix). We also enhance the detection of reactive tokens with the dedicated GMM, ﬁlled pause detecter and
ASR. Experimental results for real podcast programs demonstrate that our system can detect eight acoustic events withframe-wise accuracy of 74.5%, laughter with F-measure of
68.7%, and reactive tokens with F-measure of 64.0%. It is also
shown that the use of different penalty weights is effective insegmentation of podcasts.
Future work includes incorporation of clustering of speechsegments for speaker diarization. We also plan to design andimplement a “hot spot browser”.

Table 3: Detection accuracy of reactive tokens
Measures
Proposed method
λ = 1.0
λ = 1.5
λ = 2.0

R
34.0
35.3
33.1
29.2

P
85.2
67.9
79.3
81.2

F
64.0
54.7
59.9
57.5

reactive token detection accuracy of the following four methods.

7. References

1. Proposed method (switching penalty weight λ)2. Using the constant penalty weight: λ = 1.0
3. Using the constant penalty weight: λ = 1.5
4. Using the constant penalty weight: λ = 2.0[1] M. Goto, J. Ogata, and K. Eto, “PodCastle: A Web2.0 Approachto Speech Recognition Research,” in Proc. of Interspeech, pp.
2397-2400, 2007.
[2] J. Ogata, M. Goto, and K. Eto, “Automatic Transcription for a
Web 2.0 Service to Search Podcasts,” in Proc. of Interspeech, pp.
2617-2620, 2007.

The frame-wise accuracy is the accuracy of detection (segmentation and classiﬁcation) of the eight classes (male speech,
female speech, male mix, female mix, music, silence, laughter,and reactive tokens). In overlapping segments of two different
events, if one of the correct classes is included, we regard thatthe correct class is labeled for the frame. As the detection accuracy of laughter and reactive tokens, we adopted the recall R,
the precision P , and the F-measure F . F is deﬁned as follows:
F-measure =(1 + α2 )RP
R + α2 P[3] C. Alberti, et al., “An Audio Indexing System for Election Video
Material,” in Proc. of ICASSP, pp. 4873-4876, 2009.
[4] Z. Chang, “Analysis on Morphological and Prosodic Features of
Aiduti for Indexing Poster Presentations,” Masters Thesis, Graduate School of Informatics, Kyoto University, 2009.
[5] B. Wrede and E. Shriberg, “Spotting “Hot Spots” in Meetings:
Human Judgments and Prosodic Cues,” in Proc. of Eurospeech,
pp. 2805-2808, 2003.

(4)[6] X. Zhou, et al., “HMM-based Acoustic Event Detection with AdaBoost Feature Selection,” Multimodal Technologies for Perception of Humans, pp. 345-353, 2008.

where α is a measure of the relative importance of recall andprecision. In this work, α was set to 0.5 because there are alarge number of these events including indistinct ones, and weshould focus on the detection of distinct ones rather than recallof everything.
The results are shown in Figure 4 and listed in Table 2
and 3. The proposed method switching the penalty weight λimproved the frame-wise accuracy in both program-open andprogram-closed cases. The accuracy was drastically improved
when using previous episodes of the same program (programclosed case) because there are often same speakers and same
music in the same program even if the episode is different.
In detection of laughter and reactive tokens, there was nota signiﬁcant difference between the program-open case and theprogram-closed case, thus we report the program-open case. As
seen in Table 2, proposed method signiﬁcantly improved thedetection rate of laughter. Although the recall rate was degraded because it was difﬁcult to detect subtle laughter, mostloud laughter could be detected. Loud laughter is related moreclosely to hot spots than subtle laughter, thus we expect that the
low recall does not cause major problems in ﬁnding hot spots.
The results listed in Table 3 show that the proposed method
achieved best performance of detection accuracy of reactive tokens. The main reason for the low recall is strict constraints
for rejecting ﬁllers and falters. In fact, we obtained the recall[7] A. Temko and C. Nadeu, “Classiﬁcation of Acoustic Events using
SVM-based Clustering Schemes,” Pattern Recognition, Vol. 39,
issue 4, pp. 682-694, 2006.
[8] M. T. Knox and N. Mirghafori, “Automatic Laughter Detection
Using Neural Network,” in Proc. of Interspeech, pp. 2973-2976,
2007.
[9] S. Chen and P. Gopalakrishnan, “Speaker, environment and channel change detection and clustering via the Bayesian Information
Criterion,” in Proc. of DARPA Broadcast News Transcription andUnderstanding Workshop, pp. 127-132, 1998.
[10] A. Tritschler and R. Gopinath, “Improved speaker segmentationand segments clustering using the Bayesian Information Criterion,” in Proc. of Eurospeech, pp. 679-682, 1999.
[11] M. Goto, K. Itou, and S. Hayamizu, “A Real-time Filled Pause
Detection System for Spontaneous Speech Recognition,” in Proc.
of Eurospeech, pp.227-230, 1999.
[12] K. Itou, et al., “JNAS: Japanese Speech Corpus for Large Vocabulary Continuous Speech Recognition Research,” J. Acoust. Soc.
Jpn. (E), Vol. 20, No. 3, pp. 199-206, 1999.
[13] M. Goto, et al., “RWC Music Database: Popular, Classical, andJazz Music Databases,” in Proc. of ISMIR, pp.287-288, 2002.
[14] T. Kawahara, et al., “Multi-Modal Recording, Analysis and indexing of Poster Sessions,” in Proc. of Interspeech, pp. 1622-1625,
2008.

1146