The Use of Semantic and Acoustic Features for
Open-Domain TED Talk Summarization
Fajri Koto∗† , Sakriani Sakti∗ , Graham Neubig∗ , Tomoki Toda∗ , Mirna Adriani† , and Satoshi Nakamura∗∗ Graduate School of Information Science, Nara Institute of Science and Technology, Japan

E-mail: {ssakti,neubig,tomoki,s-nakamura}@is.naist.jp Tel: +81-743-725264† Faculty of Computer Science, University of Indonesia, Indonesia
E-mail: fajri91@ui.ac.id, mirna@cs.ui.ac.id Tel: +62-21-7863419

Abstract—In this paper, we address the problem of automatic
speech summarization on open-domain TED talks. The large vocabulary and diversity of topics from speaker-to-speaker presents
significant difficulties. The challenges increase not only how tohandle disfluencies and fillers, but also how to extract topicrelated meaningful messages within the free talks. Here, we
propose to incorporate semantic and acoustic features within the
speech summarization technique. In addition, we also propose a
new evaluation method for speech summarization by checking
semantic similarity between system and human summarization.
Experiments results reveal that the proposed methods are effective in spontaneous speech summarization.

I. I NTRODUCTION
Recently, information in Internet is available with various
data such as text, images, sound, and also video. Consequently,
many researchers start to study how to retrieve informationfrom these various data. Automatic speech summarization has
been also actively investigated. By using audio and videoof speech data, many researchers have investigated summarization based on the output of automatic speech recognition(ASR) [2][3][5]. Here, the summarization process is performed
over the text output of ASR system without involving audio
features information. For example, the study by Hori et al.,extract and calculate the word significance score and the
linguistic likelihood from the ASR output [2]. Furthermore,some other techniques like random walk, words and sentence extraction, weighted finite-state transducers, and Hidden
Markov Model have been also studied by some researchers to
improve speech summarization technique [3][14][15][16].
However, despite a lot of progress in speech summarization,
most works focused primarily only on news content, news
broadcast, and other non spontaneous speech data. On theother hand, there are many spontaneous speech exist where
people are willing to have a summarization of the talks but
difficult to obtain. One of the cases is open-domain talklike TED talks1 that are still limited to be used. TED is a
nonprofit devoted to Ideas Worth Spreading. It started out
in 1984 as a conference bringing together people from three
worlds: Technology, Entertainment, Design. TED talks bring
together the world’s most fascinating thinkers and doers, who
are challenged to give the talk of their lives in 18 minutes
1 http://www.ted.com/

978-616-361-823-8 © 2014 APSIPA

or less. Here, we initiate to address the problem of automatic
speech summarization (ASR) on open-domain TED talks.
It is obvious that spontaneous speech in TED talks is very
different from speech in broadcast news in which speakers do
not have any text guidance in their hand. This resulted in output of ASR system will have higher error than broadcast news
speech recognition. Furthermore, The large vocabulary and diversity of topics from speaker-to-speaker presents a significant
difficulties. The challenges increase not only how to handle
disfluencies and fillers, but also how to extract topic-related
meaningful messages within the free talks. In this study, we
propose to incorporate semantic features in automatic speechsummarization. In this way, the topic related sentences arescored higher than unrelated sentences. As preliminary study,
we start with incorporating the proposed methods within the
widely-used MMR summarization technique [1].
In addition, we also include acoustic features to the summarization framework, since the acoustic features are also
one of significant factors in speech summarization [17]. The
MMR technique is done by processing the output of ASR
in term frequency (TF) and term frequency-inverse document
frequency (TF-IDF) model. Then, various combination withacoustic features, semantic features, as well as acoustic andsemantic features together are investigated.
We also propose a new evaluation method for speech summarization by checking semantic similarity between systemand human summarization. We argue that Common evaluation like Recall-Oriented Understudy for Gisting Evaluation(ROUGE) and Longest Common Subsequence (LCS) have
limitation for spontaneous speech because they are only basedon the number of overlapping units such as n-gram and word
sequences [13]. Whereas, public speeches like TED are more
unstructured and rich with dictionary. Therefor, performing
evaluation with semantic similarity is more promising.
II. OVERVIEW OF MMR-BASED S UMARIZATIONT ECHNIQUESMMR has been widely used for text summarization. MMR
is a measure where the retrieval status value (RSV) of adocument is influenced by other already retrieved documents:
documents similar to retrieved documents have their RSV
lowered, thus boosting dissimilar documents [1]. Carbonell

APSIPA 2014

and Goldstein proposed this formula as follow:
M M R(Si ) = α ∗ Sim1 (Si , D) + (1 − α) ∗ Sim2 (Si , Summ)(1)
where Si is i-th sentence in document D, Summ is summary
result that is being built according to highest MMR score forevery iteration, and Sim1 and Sim2 are similarity measures,
which can be the same, or can be set to different similarity
metrics. In this study, we use cosine similarity to calculate
Sim1 and Sim2 as follows:P
t1i , t2i(2)sim(D1 , D2 ) = pP 2i pP 2i t1i ∗i t2i
The value of α allows a readjustment of the behavior of MMR
to control diversity ranking between unselected sentence withselected summary sentences. Here, TF and TF-IDF model to
MMR are performed.
III. T HE P ROPOSED S UMMARIZATION T ECHNIQUES

Fig. 1. Build summary label by performing Semantic Similarity.

Eq. 6 above uses two kinds of vector: noun and verb. These
vectors are described simply below:|V |1
V v1k = M axi=1(Sim(V v1i , V bk ))|V |2
V v2k = M axi=1(Sim(V v2i , V bk ))|N |N n1k = M axi=11 (Sim(N n1i , N bk ))|N |N n1k = M axi=12 (Sim(N n2i , N bk ))A. Acoustic and Semantic Feature
1) Acoustic Feature: Acoustic feature that is used in thisstudy is based on INTERSPEECH 2010 paralinguistic challenge configuration (IS10 Paraling features) [18]. It consistsof 1582 features described in Table 1, which are obtainedin three steps: (1) the 38 low-level descriptors are extracted
and smoothed by simple moving average low-pass filtering;(2) their first order regression coefficients are added in full
HTK compliance; (3) 21 functionals are applied. However, 16
zero-information features (e. g. minimum F0, which is alwayszero) are discarded. Finally, the 2 single features F0 number of
onsets and turn duration are added. More details of descriptionof each feature can be found in [18].
2) Semantic Feature: Semantic similarity feature is a similarity score that describes the similarity between a sentence
and document. We proposed this formula to re-rank thesentences according to similarity score between sentence andwhole document.
Pj=|S|
j=1∧j6=i Simsem (si , sj )Simsem (si , D) =(3)
|S| − 1
where si is the i-th sentence in document D, and |S| represents
the number of all sentences in document D. That formulacalculates all semantic similarity score between one sentencewith other sentences. We take the mean score as our finalscore to make the sentence rank. The Simsem is calculated
according to [6], in which one sentence is divided into noun
set and verb set and the similarity score between two sentenceis then calculated based on the similarity of those noun andverb set described in Eq. 6.
S1 = {V1 , N1 } and S2 = {V2 , N2 }(4)

N b = N1 ∪ N2 and V b = V1 ∪ V2

(5)

SimSem (S1 , S2 ) = β1 ∗ Simv (v1 , v2 ) + β2 ∗ Simn (n1 , n2 )
(6)

(7)
(8)
(9)
(10)

To calculate semantic similarity score between two words
above, we also use words-relationship-Tree based on Wu andPalmer’s Algorithm [7]. This function utilized online lexical
database WordNet [8][9]. While, the similarity score (Simvand Simn ) between this two vector can be calculated easily
by using cosine similarity formula.
B. Summarization Method
1) Incorporating Semantic Features: In summarization process, MMR re-ranks every sentence by calculating cosine
similarity score between two term vectors: sentence and documents. To boost the system accuracy we propose modified
MMR by replacing Sim1 (Si , D) in Eq. 1 with Eq. 11. Our
modified MMR incorporates Semantic Similarity in similarity
calculation which also considers the cosine similarity. In thisstudy, we use β equals to 0.5.
β ∗ SimSem (Si , D) + (1 − β) ∗ Sim1 (Si , D)

(11)2) Incorporating Acoustic Features: The motivation of incorporating acoustic features within summarization framework
is to give more score into the sentences that are considered
as candidate summary based on acoustic characteristic of
the sentences. This is done by naive bayes (NB) classifier,implemented based on this formula below, where it output”1” if sentence Si is considered as candidate summary forthat document, and 0 otherwise:
M M Rspeech (Si ) = 0.5 ∗ M M R(Si ) + 0.5 ∗ N BM odel(Si )(12)
We train the classifier by training acoustic feature of eachsentences as described in Fig. 1. We create labels by calculating semantic score between sentences of every documentwith existing summary provided by TED website. We built
the label by selecting top 10 sentences with highest semantic
similarity score for each document.

summarization. The ASR system that is used here was trainedusing 157 hours of TED talks released before the cut-off dateof 31 December 2010, downloaded from the TED websites
with the corresponding subtitles.
V. E XPERIMENTAL SET- UP
A. The TED Data
The TED talks that are used in summarization are thesame data which were used to evaluate the speech recognition
system. There are 20 TED talks in total. The reference
summarization is obtained from human summarization. In thisstudy, five native speakers are required to pick ten sentencesthat were considered as most representative sentences to the
speaker topic for each speech document.

Fig. 2. Automatic Speech Summarization stage3) Incorporating Acoustic and Semantic Features: To get
more elaboration we also implement Naive-bayes-based acoustic classifier to our new method in speech summarization. We
formulate the new score of similarity by adding the score with
NBModel function like Eq. 12. In this study, we use γ equals
to 0.1.
Sim0sem (si , D) = (1−γ)∗Simsem (si , D)+γ∗N BM odel(Si )(13)
C. Evaluation Metric: Semantic Similarity Checking (SSC)
The well-known automatic evaluation method for Summarizer: ROUGE and LCS have been introduced by Lin [13].
Formally, ROUGE-N is an n-gram recall between a candidate
summary and a set of reference summary. Whereas, LCS does
not require consecutive matches but in-sequence matches thatreflect sentence level order as n-grams [12]. Since we focusedon unstructured document like Spontaneous Speech, in thisstudy we propose semantic similarity checking (SSC) in Eq. 14
as a new automatic evaluation method for summarizer which
utilizes semantic similarity calculation. Intuitively, this method
will be more powerful than ROUGE and LCS because theresulting score is not just built by counting matched words orsequences.
Pi=|D1 |SSC(D1 , D2 ) =i=1

Simsem (si , D2 )
|D1 |

(14)

In Eq. 14, the D1 and D2 represents document of resulting
summary and document of reference summary consecutively.
The equation simply calculates the average of all semantic
similarity score between every sentence si in D1 and D2 .
The similarity score Simsem (si , D2 ) is also calculated byaveraging the semantic similarity score between si and all
sentences in D2 .
IV. OVERALL A RCHITECTURE OF S UMMARRIZER
Fig. 2 above shows our summarization experiment stage. We
use output of ASR system to build sentence-based summary by
doing some techniques: MMR, Semantic Similarity and theirincorporation with the audio model. This model is built bytraining acoustic feature. Then we do evaluation by calculating
similarity score between the resulting summary and human

B. Preprocessing
To build the vector space models (TF and TF-IDF) we did
preprocessing to all TED speech data. We replace all capitalletters of transcription file with lowercase and eliminate allpunctuations that exist in the transcription file. We also remove
some of the unimportant word or segment like laugh andapplause. We use all TED documents to build idf(t,D) scorein calculating TF-IDF.
For acoustic features, we perform segmentation based ontime sequences obtained from the srt file and our ASR system.
It aims to get the valid timing of every sentence in document.
Segmented audio file will be extracted by openSMILE toolkit[11]. openSMILE is a feature extraction toolkit, which unites
feature extraction algorithms from the speech processing andthe Music Information Retrieval communities [10].
The noun and verb vector that will be used to calculatesemantic similarity are processed by implementing python
code with NLTK library. The similarity checking will be
performed for the same tagging of words (only between two
nouns or two verbs).
VI. E XPERIMENT R ESULTAs our baseline system, we perform MMR in TF and
TF-IDF model. Various value of alpha parameters in MMR
formula are investigated. The results are then compare withtwo proposed methods: incorporating semantic similarity andacoustic features. We do evaluation with SSC and take top 30%
highest MMR score of sentences in document as its summary.
Our first experiment is comparing SSC and ROUGE tolook their tendency in evaluating system summary. In Fig.
3 we use ROUGE-3 gram and SSC to perform evaluationof MMR-TF for various alpha value. The results reveal thatour proposed metric evaluation has in line performance with
ROUGE. However, SSC is still better to be used because it iscalculated based on semantic.
In MMR experiment we use some alpha parameters: 0.1,
0.3, 0.5, 0.7, and 0.9. Fig. 4 and Fig. 5 show that Semantic
Similarity can boost the accuracy of MMR for all alphaparameters. The incorporation of MMR, semantic and acoustic
feature is shown by the top line for both graph. It affirmsthat semantic and acoustic have important role for optimizing

Fig. 5. MMR-TFIDF Experiment result.

Fig. 3. SSC and ROUGE performance for MMR-TF

VIII. ACKNOWLEDGMENT
Part of this work was supported by JSPS KAKENHI Grant
Number 26870371.
R EFERENCES[1]

Fig. 4. MMR-TF Experiment result.

automatic speech summarization. According to both line graph(Fig. 4 and Fig. 5), the highest accuracy is achieved by MMR
and Semantic at alpha equals with 0.7. They are 55.29% and55.30% for TF model and TF-IDF model consecutively.
In Table. I, we present the best performance of each methods
for vary parameters. Here we compare MMR, MMR+Audio,and MMR+Audio+Semantic. And the result reveals that the
combination of MMR, Acoustic and Semantic feature always
give better performance than standard technique for both
vector space model.
TABLE I
Highest accuracy performance of MMR and its incorporation
Incorporation
MMR
MMR+Audio
MMR+Audio+Semantic

TF
51.38%
51.77%
54.64%

TFIDF
50.71%
53.18%
55.10%

VII. C ONCLUSION
In this study, we attempt to incorporate both semantic
and acoustic features in automatic speech summarization for
open-domain TED talks. The experimental results reveal thatthey can improve textual speech summarization. In short, ourstudy reveals that semantic similarity can be used in speechsummarization: 1) as summarization feature and 2) evaluation
method. Our experiments also show that the incorporationof MMR, Semantic and Acoustic feature can achieve best
performance. It affirms the both features have important rolein speech summarization. In our future work, we will further
investigate various incorporation approaches of semantic andacoustic features into MMR as well as the combination withother summarization techniques

Carbonell, J., and Goldstein, J. ”The Use of MMR, dicersity-based
reranking for reordering documents and producing summaries”. SIGIR
1998, pp. 335-336. ACM, New York, 1998.
[2] Hori, C., and Furui, S. ”Automatic speech summarization based on word
significance and linguistic likelihood”. Proc. ICASSP2000, Istanbul,
Vol.3, pp.1579-1582, 2000.
[3] Hori, C., and Furui, S. ”Improvements in automatic speech summarization and evaluation methods”. ICSLP2000 4: 326-329, 2000
[4] Maskey, S., and Hirschberg, J. ”Comparing lexical, acoustic/prosodic,
structural and discourse features for speech summarization”. In INTERSPEECH, pp. 621-624, 2005
[5] Xie, S., and Yang L. ”Using confusion networks for speech summarization in Human Language Technologies”. The 2010 Annual Conferenceof the North American Chapter of the Association for Computational
Linguistics, pp. 46-54. Association for Computational Linguistics, 2010.
[6] Liu, D., Liu, Z., and Dong, Q. ”A dependency grammar and WordNetbased sentence similarity measure”. Journal of Computational Information Systems 8:3 1027-1035, 2012.
[7] Wu, Z., and P. Martha. ”Verbs semantics and lexical selection”. In Proceedings of the 32nd annual meeting on Association for Computational
Linguistics, pp. 133-138. Association for Computational Linguistics,
1994.
[8] Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., and Miller, K.
J. ”Introduction to wordnet: An on-line lexical database”. International
journal of lexicography 3, no. 4: 235-244, 1990
[9] Available: http://wordnet.princeton.edu/[10] Eyben, F., Woellmer, M., and Schuller, B. ”openSMILE the Munich open
Speech and Music Interpretation by Large Space Extraction toolkit”.
Institute for Human-Machine Communication, version 1.0.1, 2010.
[11] Available: hhtp://opensmile.sourceforge.net/[12] Eyben, F., Martin W., and Bjrn S. ”Opensmile: the munich versatileand fast open-source audio feature extractor”. In Proceedings of theinternational conference on Multimedia, pp. 1459-1462. ACM, 2010.
[13] Lin, C.-Y. ”ROUGE: A Package for Automatic Evaluation of Summaries”. In Text Summarization Branches Out: Proceedings of the ACL04 Workshop, pp. 74-81, 2004.
[14] Chen, Y.-N., Huang, Y., Yeh C.-F., and Lee. L.-S. ”Spoken Lecture
Summarization by Random Walk over a Graph Constructed with Automatically Extracted Key Terms.” In INTERSPEECH, pp. 933-936, 2011.
[15] Furui, S., Hirohata, M., Shinnaka, Y., and Iwano, K. ”Sentence
extraction-based automatic speech summarization and evaluation techniques.” In Proceedings of the Symposium on Large-scale Knowledge
Resources, pp. 33-38, 2005.
[16] Maskey, S. and Hirschberg, J. ”Summarizing speech without text using
hidden markov models.” In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers,
pp. 89-92. Association for Computational Linguistics, 2006.
[17] Inoue, A., Mikami, T. and Yamashita Y. ”Improvement of Speech
Summarization Using Prosodic Information.” In Speech Prosody 2004,
International Conference, 2004.
[18] Schuller, B., Steild, S., Batliner, A., Burkardt, F., Devillers, L., Muller,C., Narayanan, S. ”The INTERSPEECH 2010 Paralinguistic Challenge”.
In INTERSPEECH, pp. 2794-2797, 2010.