論文 / 著書情報
Article / Book Information
Title

Automatic Digest Generation by Extracting Important Scenes from theContent of Presentations

Author

Hieu Hanh LE, Thitiporn LERTRUSDACHAKUL, Tetsutaro
WATANABE, Haruo Yokota

Journal/Book nameProc. of DEXA2008 Workshops (AIEMPro'08), , , pp. 590-594

Issue date2008, 9DOI

10.1109/DEXA.2008.21

URL

http://www.ieee.org/index.html

Copyright(c)2008 IEEE. Personal use of this material is permitted. Permission
from IEEE must be obtained for all other users, including
reprinting/republishing this material for advertising or promotional
purposes, creating new collective works for resale or redistribution toservers or lists, or reuse of any copyrighted components of this work inother works.

Note

このファイルは著者（最終）版です。
This file is author (final) version.

Powered by T2R2 (Science Tokyo Research Repository)

Automatic Digest Generation by Extracting Important Scenes
from the Content of Presentations
Hieu Hanh LE1

Thitiporn LERTRUSDACHAKUL2
Haruo YOKOTA3,4

Tetsutaro WATANABE3

1

Department of Computer Science, Falculty of Engineering, Tokyo Institute of Technology
2
Application Lab, Software R&D Group, Ricoh Company, Ltd.
3
Department of Computer Science, Graduate School of Information and Engineering,Tokyo Institute of Technology
4
Global Scientiﬁc Information and Computing Center, Tokyo Institute of Technology{hanhlh@de.cs.titech.ac.jp,thitiporn.lertrusdachakul@nts.ricoh.co.jp,tetsu@de.cs.titech.ac.jp,
yokota@cs.titech.ac.jp}
Abstract
Recently, combining a video recording of a presentation along with the digital slides used in it has become
popular in E-learning and presentation of archives. For
users of the archives, it is useful to preview a digest ofsuch content to grasp the atmosphere and/or an outline of the presentation. This paper proposes a methodof automatic digest generation by extracting importantscenes from the presentation content. The extracted
scenes are chosen based on several factors such as frequency and speciﬁcity of words, scene duration and order. Finally, the eﬀectiveness of the proposed methods
are evaluated by comparing with testers’ answer sets
for actual lectures.

1

Introduction

In recent years, the combination of digital slides used
in a presentation along with a video recording of thepresentation has been increasingly used in E-learning
or for presentation-archival purposes. The development of software to easily create web-publishable content of presentations, such as MPMeister [8] developedby Ricoh Co. Ltd., has accelerated this trend.
To improve the eﬀectiveness of E-learning andarchiving, it is important to provide functions forsearching for appropriate scenes described by given
keywords. For this type of content, a scene is deﬁned as a part of video bounded by changes in theslides. In previous studies, we have proposed a number of methods to distinguish the scene described by

given keywords from other scenes containing the same
slide, caused by reuse or backtracking by the presenter. We developed a system named UPRISE (Uniﬁed
Presentation Slide Retrieval by Impression Search Engine) [11], and demonstrated the eﬀectiveness of thesystem’s methods [12].
However, in actual situations, one cannot presume
that users can always choose good keywords to search
for the desired portions in the video. Additionally,
there has been a high demand for a brief summaryof the presentation before watching the whole content.
Although a number of studies on video summarization
have been done, their results could not be directly applicable to our application because of the diﬀerences incontent characteristics.
In this paper, we propose a method for generatinga digest of a presentation by concatenating importantscenes extracted from the content created by MPMeister. Here, we assume that scenes showing the representative topic of the presentation are important. To judge
the representativeness, we pay attention to factors suchas frequency and speciﬁcity of words in slides, duration,and order of scenes, without analyzing the semantics ofslides. Obviously, the semantics is important but still
hard to be analyzed correctly. We then evaluate theproposed method by comparing scenes selected by ourexperimental system with testers’ selections to verify
the eﬀectiveness of the approach.
This paper is organized into the following four sections. First, we take an overview of related work in
section 2. Then, we propose our method in section 3,
and report experiments and discuss the results in section 4. Concluding remarks and possibilities for future

work are given in section 5.

2

Related WorkIn the ﬁeld of summarizing text documents, there
are two major approaches: those that extract sentences that best match a given user query [9], and thoseshowing the general topic of the document [1, 3]. Although our objects, i.e., presentation content, are different from simple text documents, concepts developed
for text summarization are also useful for our approach.
There has also been much previous research on
speech [2] and video summarization [7, 10]. One of thestudies most related to our approach was done by Liwei et al. [4]. However, the slide content that we focus
on in this paper has not been considered yet.
We have previously proposed a uniﬁed presentation
content search system named UPRISE [11] (Uniﬁed
Presentation Slide Retrieval by Impression Search Engine) focusing not only on the textual information inslides, but also on the duration and order of scenes.
Later, we improved our precision by integrating speech[6] and laser pointer [5] information. Although the
purpose of this study is the extraction of importantscenes representing the presentation topics, we utilize the techniques developed in the UPRISE research
project.

Because we use Japanese lectures as experimental
data, we apply the Japanese language morphologicalanalysis Sen to extract only the meaningful words fromthe text content. Finally, we use Japanese-English Ontology to discover the same English and Japanese words
in meaning and treat them as the same words.

3.2

Important scene extraction

3.2.1

Assumptions

Suppose that a presentation Ll is one of therecorded presentations in the database. Then, S ={s1 , s2 , . . . , sN } represents the set of all slides usedin the presentation Ll . In our methods, we focus on the frequency of words in slides. Therefore, each slide si can be expressed as an array ofwords si = [vi1 , vi2 , . . . , vim ]. Additionally, let W ={w1 , w2 , . . . , wM } be the set of words that appear in all
slides. Obviously, vij ∈ W . Furthermore, the presentation Ll can be represented by an array of scenes determined by slide transitions, Ll = [cl1 , cl2 , . . . , cln ]. The
scene clk corresponds to one of the slides in S, which
can be expressed as clk = [vi1 , vi2 , . . . , vim ], ∃i : si ∈ S.
Note that because of backtracking or reuse, a slide canappear in more than one scene.
3.2.2

3

In this paper, we propose a method of automatic
digest generation by extracting important scenes frompresentation content created by MPMeister. With thewords and timing information extracted from MPMeister, we propose a digest generation method based on
several important scene extractions, which consists ofthe following three steps: 1) preprocessing, 2) important scene extraction, and 3) digest generation.

3.1

Calculation of scene importance

Method Proposal

Preprocessing

First, we pick up the text content of the slides usedin the scenes, and the timing information from therecorded presentation. Second, considering slide traverses and errors in operation, we ﬁlter out scenes
with durations less than three seconds. In addition,
we remove exceptional scenes that may have extremely
long duration, but no meaningful information, such as
scenes in which students silently carry out exercisesin the lecture. In this paper, we tentatively set thisthreshold time to ten minutes, but a more suitable timeor other feature, such as distinguishing the exercise,
could be considered in the future.

Based on the idea of “The concept that is repeatedly
mentioned is an important concept”, all of these formulas focus on word frequency. In this process, we also
consider the structure of the slide by using weights p(v)of the word v:

 ρt
ρb
p(v) =
 l0v appears in the title
v appears at indent level bl .
v does not appear(1)

Considering the other factors that eﬀectively distinguish scenes in E-learning content, we select four inﬂuential factors: scene duration, scene order, word speciﬁcity (such as idf), and number of words appearing ina slide. We propose ﬁve formulas to put those factors
into the importance calculation, where the inﬂuence ofthe scene duration factor is changed by a parameterand those of the other three factors are implemented
by changing combinations.
Scene duration importance Id Suppose that thelonger a scene is, the more detailed the concepts in thisscene are explained, so we rate the importance of thisscene highly. Therefore, as in UPRISE, we propose aformula for Id that considers the duration of scenes:
Id (clk , θ) = Ip (clk ) · t(clk )θ ,Table 1. Summary of factors considered in
the proposed formulas(2)
Formulawhere t(clk ) is the duration of scene clk , θ is the timeparameter, and Ip can be calculated as:
Ip (clk ) =

∑

(

∑

(

∑

wx ∈clk cy ∈Ll vz ∈cy ,vz =wx

p(vz ))).

(3)

Scene order importance Idc
When a word appears not only in a certain scene, but also in the neighboring scenes, it can be seen that this word is explained
well in this scene. Based on this idea, as in UPRISE, werate the importance of such scenes highly. The formula
Idc captures this:Considered factors
scene
scene
spec ofduration
order
word
°
°
°
°
°
°
°
°
°

word frequency
°
°
°
°
°

Id
Idc
Idr
Idrc
Idf

#wordsin slide°formula expressing both scene order and word speciﬁcity:
Idrc (clk , θ, δ, ε1 , ε2 ) =

j=γ+δ
∑

Idr (clk , θ)·E(j −γ, ε1 , ε2 ).

j=γ−δ

(7)
Idc (clk , θ, δ, ε1 , ε2 ) =

j=γ+δ
∑

Id (clk , θ) · E(j − γ, ε1 , ε2 ),

j=γ−δ

(4)
where δ is a window-size parameter that determines
how many neighboring slides are taken into account.
E(j −γ, ε1 , ε2 ) speciﬁes the eﬀect of neighboring scenes
in the context window and is calculated as follows:{exp(ε1 x)(x < 0)E(x, ε1 , ε2 ) =.
exp(−ε2 x)(x ≥ 0)

Number of words appearing in slide importance
Idf
Through the above proposed formulas, we cansee that the more words there are in a slide, the higher
the importance of the scene using this slide. However,where there are few words in the slide, it can be suggested that those words are better explained comparedwith others. Therefore, these kinds of scenes should
be given high importance. To achieve this idea, we
proposeIdf (clk , θ) =Note that the smaller ε is, the more likely neighboringslide information is used.
Word speciﬁcity importance Idr When focusing
on the importance of a word itself, such as idf (Inverse
Document Frequency), it is thought useful to consider
the word frequency in other scenes. Therefore, we propose a formula Idr to capture the speciﬁcity of words:
Idr (clk , θ) =

∑
wx ∈W

app(clk , wx )∑· Ipl (clk ) · t(clk )θ ,app(cy , wx )cy ∈Ll(5)where, app(cy , wx ) is a function that counts the number of appearances of word wx in scene cy , and Ipl is
calculated according to:
Ipl (clk ) =∑vx ∈clkp(vx ).

(6)

Scene order and word speciﬁcity importanceIdrc
By combining the factors above, we obtain a

∑
wx ∈W

1
· Id (clk , θ).
app(clk , wx )(8)

The factors considered in the proposed formulas are
summarized in Table 1.
3.2.3

Important scene extraction method

By applying each formula to calculate the importanceof scenes, we then extract those scenes that havegreater than average importance as important scenes.

3.3

Automatic digest generation

To make the digest video, we truncated the extracted important scenes and concatenated them. The
durations of the truncated scenes are determined by
the ratios of importance of the scenes.

4

Experimentation and Evaluation

The purpose of our research is to create an automatic digest generation method from presentation content, and so far we have evaluated the important scene               









       


 















4.2.1

                                                                    Figure 1. Difference between importantscenes extracted by testers and our proposal
extraction methods that we proposed in this paperby two-tester-based experiments. The ﬁrst experiment
was aimed to verify the eﬀectiveness of our proposed
method. The second one was to evaluate the impactof the two parameters, ρt and θ, which are used in our
proposed ﬁve formulas.

4.1

Eﬀectiveness veriﬁcation

4.1.1

Experiment circumstance and method

We applied our proposed formulas (2),(4),(5),(7),(8) toa lecture in Japanese that had taken place at our university.
Next, we asked six testers who have not previously
seen the original video of the lecture to choose the important scenes that they would like to include in a digest. Then, we deﬁne a correct set (symbolized as CS),
which contains those scenes that were chosen by morethan four testers. Here, we set values of parameters
used in proposed formulas as ρt = 5, ρbl = 1, ∀bl ,θ = 1, δ = 3, ε1 = 5, and ε2 = 0.5.
4.1.2

Experimental results and discussion

The experimental results are shown in Figure 1. Thisﬁgure indicates that almost all of the scenes includedin CS can be found by our proposal. Therefore, ourimportant scene extraction methods are thought to beeﬀective for generating digests.

4.2

appearing in the title of the slide, and θ, which determines the eﬀect of scene duration.

Evaluation of impact of factors

In this experiment, we evaluated the impact of thetwo parameters ρt , which is the weight given to a word

Experiment circumstance and method

In this experiment, we used recordings of four otherlectures that took place at our university.
¿From the previous experiment, we selected two representative testers and let them choose scenes that theywould like to include in a digest from the lecture video.
Then, the CS was deﬁned as those scenes chosen byboth testers. Next, we evaluated the impact of the twoparameters of ρt and θ by F measure (symbolized as
F ), which is calculated from the CS and the set of important scenes extracted by our proposal. The greater
F is, the better the performance of the system.
At ﬁrst we changed the value of ρt from 1 to 10 insteps of 1 and calculated F to evaluate the impact of
ρt . Then, for each lecture and formula, we changed thevalue of θ from 0 to 5 in steps of 0.5. Note that inthis last step, we set ρt to that which resulted in thehighest F in the previous step.
4.2.2

Experimental results and discussion

The experimental results of the average value of F are
shown in Figure 2 and Figure 3, as a function of ρt andθ, respectively.
First, Figure 2 indicates that for almost all formulas,
it can be seen that the values of F were higher when
ρt > 1. Therefore, for extracting important scenes, theconsideration of ρt is somewhat important; however,
the results are not too sensitive to this parameter.
¿From the experimental results shown in Figure 3,
it is easily recognized that F is lowest when θ = 0. It
indicates that the treatment of scene duration is very
important in the proposed approach for extracting important scenes. However, the results also indicate that
there was a trend of worsening F as θ grows large.
Thus, in order to extract important scenes from presentation content, the impact of duration time of scenes
should not be too large.
Last, but not least, Figure 2 and Figure 3 also showthat method Idf was the best for extracting importantscenes, so the consideration of number of words appearing in the slides was the most eﬀective.

5

Conclusion and Future WorkIn this paper, we propose methods for generating adigest of a presentation using digital slides, which concatenates important scenes extracted from all scenes
in the presentation. To select the important scenes, we 

Acknowledgment

 
 



 



 



 



 



 



 

This work is partially supported by a Grant-in-Aid
for Scientiﬁc Research of MEXT Japan (#19024028),
Tokyo Institute of Technology 21 COE Program“Framework for Systematization and Application of
Large-Scale Knowledge Resources”, and CREST of
JST (Japan Science and Technology Agency).

 















References

Figure 2. The average value of F as ρt varied
from 1 to 10





 
 


 



 



 


 



 



 



 





 



 



Figure 3. The average value of F as θ varied
from 0 to 5
deﬁne a number of formulas focused on factors in thepresentation content: the frequency of words appearing
in slides, scene duration, and the order of scenes. We
then developed an experimental system and evaluatedthe extraction method by comparing scenes selected by
the system with those selected by testers from recordings of lectures that took place in our university. The
experimental results indicated that the considerationof two factors of scene duration and number of words
appearing in a slide were inﬂuential.
In the future, we plan to do the following:• Perform a more detailed veriﬁcation of the proposed method using a higher volume of presentation content.
• Consider other Information, such as speech andanimation, to segment the scenes into smaller
units.
• Integrate the ontology by considering the relationship between words and contriving semanticallyaware ﬁltering methods to remove noise from thepresentation content.

[1] R. Barzilay and M. Elhadad. Using Lexical Chains for
Text Summarization. In Proc. of the Intelligent Scalable Text Summarization Workshop (ISTS’97), 1997.
[2] H. Chiori. A Study on Statistical Methods for Automatic Speech Summarization. PhD thesis, Tokyo Institute of Technology, 2002.
[3] Y. Gong and X. Liu. Generic Text Summarization Using Relevance Measure and Latent Semantic Analysis.
In Proc. of the 24th annual international ACM SIGIR
conference on Research and development in information retrieval, pages 19–25, 2001.
[4] L. He, E. Snocki, A. Gupta, and J. Grudin. AutoSummarization of Audio-Video Presentation. In Proc.
of the seventh ACM international conference on Multimedia (Part 1), pages 489–498, 1999.
[5] W. Nakano, T. Kobayashi, Y. Katsuyama, S. Naoi,
and H. Yokota. Treatment of Laser Pointer and Speech
Information in Lecture Scene Retrieval. In Pro. of theEighth IEEE International Symposium on Multimedia(ISM’06), 2006.
[6] H. Okamoto, W. Nakano, T. Kobayashi, S. Naoi,
H. Yokota, K. Iwano, and S. Furui. PresentationContent Retrieval Itergrated with the Speech Information. In IEICE Transactions on Information and
System, pages 209–222, 2007.
[7] M. J. Pickering, L. Wong, and S. M. Rüer. Image andVideo Retrieval. Springer Berlin / Heidelberg, 2003.
[8] Ricoh Japan.
MPMeister II — Ricoh Japan.
http://www.ricoh.co.jp/mpmeister.
[9] M. Sanderson. Accurate User Directed Summarizationfrom Existing Tools. In Proc. of the 7th International
Conference on Information and Knowledge Management (CIKM98), 1998.
[10] J. Wang, C. Xu, E. Chng, and Q. Tian. Sport Highlight Detection from Keyword Sequences using HMM.
In Proc. of IEEE International Conference on Multimedia and Expo, ICME ’04, volume 1, pages 599–602,
2004.
[11] H. Yokota, T. Kobayashi, T. Muraki, and S. Naoi.
UPRISE: Uniﬁed Presentation Slide Retrieval by Impression Search Engine. IEICE Trans. on Info. andSyst., E87-D(2):397–406, 2 2004.
[12] H. Yokota, T. Kobayashi, H. Okamoto, and
W. Nakano. Uniﬁed Contents Retrieval from an Academic Repository. In Pro. of International Symposiumon Large-scale Knowledge Resources, 2006.