論文 / 著書情報
Article / Book Information
Title(English)

Presentation Scene Retrieval Exploiting Features in Videos Including
Pointing and Speech Information

Authors(English)

Takashi Kobayashi, Wataru Nakano, Haruo Yokota, Koichi Shinoda,
Sadaoki Furui

Citation(English)

Proc. Symposium on Large-Scale Knowledge Resources(LKR2007).,Vol. , No. , pp. 95-100

発行日 / Pub. date2007, 3

Powered by T2R2 (Science Tokyo Research Repository)

Presentation Scene Retrieval Exploiting Features in Videos
Including Pointing and Speech Information
Takashi Kobayashi† , Wataru Nakano‡ , Haruo Yokota†‡ , Koichi Shinoda‡ and Sadaoki Furui‡† Global Scientific Information and Computing Center, Tokyo Institute of Technology‡ Dept. of Computer Science, Grad. School of Info. Sci. & Eng., Tokyo Institute of Technology{tkobaya@gsic, wnakano@de.cs, yokota@cs, shinoda@cs, furui@cs}.titech.ac.jp

AbstractchangechangechangeMovie Stream

Unified presentation contents are widely used for e-learning
and/or e-training. There is a strong need for efficient
search mechanism for unified presentation contents. We haveproposed a unified presentation contents search mechanism
named UPRISE (Unified Presentation Slide Retrieval by Impression Search Engine), and have also proposed a method
to use laser pointer and speech information in lecture scene
retrieval. In this paper, we discuss how to exploit various features in lecture videos such as the laser pointer and speech
information for the efficient retrieval method. We evaluate
our approach by using actual lecture videos and presentation
slides.

Metadata

scene i

scene i+1

scene i+2xx xx

xxx xxx xx xxxx xxxxxxx

Pointing
Information

Speechxxx xx xxxxx xx

Presentation
Slidesslide kIndex Terms: Video Scene Retrieval, e-learning, Information
Integration, Speech Information, Pointing Information, Metadata.

1changeslide k+1

slide k+2

Figure 1. Unifying a presentation video andslidesspeech and pointing information. The UPRISE retrieves a relevant scene by weighting schemata considering keyword position in the slides, duration of a scene, and context in thepresentation by using metadata. In our previous work[16], wehave shown that the precision of the UPRISE is much better
than that of ordinary tf-idf based approaches.

Introduction

Unified presentation contents, which consist of multimedia
contents, such as presentation slides, video and documents,
are widely used in a variety of contexts, such as e-learning ande-training. Many systems to store and distribute such unified
presentation contents have been proposed [10, 3, 1].
In order to adapt to the increasing number of contents, there
is a strong need for efficient search mechanism for unified presentation contents. In particular, it is important for e-learning
systems that users can not only retrieve suitable contents but
also find a scene they should start to see effectively.
There are several existing works investigating cross-overretrievals for lecture contents [9, 5, 4, 2]. However, some ofthem do not realize actual retrieval methods and systems. [2]provides a lecture passages retrieval system using transcription by speech recognition. However, they do not consider thecase of backtracking or reuse of slide materials.

We have proposed an integration method of the speech information in the lecture videos into the search functions toimprove the precision of scene retrieval [13]. In [13], we haveproposed four ways to integrate the influence of speech into
the scene search functions of the UPRISE. The experimental
results indicate that the speech information is effective to improve the precision.
We have also proposed methods to reflect the influence oflaser pointer on the weighting schemata[12] and three improved weighting schemata by using methods to filter the irrelevant pointing information based on keyword occurrencesin slides and speech[11].

We have proposed the UPRISE (Unified Presentation Slide
Retrieval by Impression Search Engine) [16, 7, 17, 12, 11, 13]
which combines slides in a presentation and a video recording
the presentation, and retrieves a sequence of desired presentation scenes from archives of the combined contents.
In UPRISE, we modeled a unified presentation contents asa sequence of scene divided by switching slides. Figure 1
shows our model of unified presentation contents. Each scene
has metadata extracted from various information such as slide,In this paper, we discuss how to exploit various features in
lecture videos such as the laser pointer and speech information for the efficient retrieval method. We introduce our information integration approach and weighting schemata withlaser pointer information and speech information. Moreover,
we evaluate our approach by using actual lecture videos andpresentation slides.
The reminder of this paper is organized as follows. In
Sect. 2, we introduce the base weighting schemata to retrieve
12.4

the unified contents we have proposed. Then we discuss thetreatment of the speech information in lecture scene retrieval
in Sect. 3 and introduce weighting schemata using the laser
pointer information in Sect. 4. Sect. 5 reports experiments
and results using actual lecture materials. We summarize thepaper’s main points in the final section.

2

Base Weighting Schemata in UPRISE

2.1

Considering Slide Structure Information

Considering the rareness of a term is important for theweighting schemata in information retrieval. For tf-idf , theinverse document frequency, idf, is the rareness factor whichindicates the rareness of a term. While the appropriatenessfactor, tf, can be calculated simply in a document, the range
for calculating the rareness factor is variable. For example, inscene retrieval for lecture videos in universities, there are several types of target range, such as a class, a course, courses ina school, and courses in the university.
We have proposed iafr(k, λ), the inverse keyword k appeared scene frequency in range specified by λ, as a dedicatedrareness factor for scene retrieval [17].

We consider the structure of each lecture slide. If a given
word appears in the title of the slide or less indented lines, thevalue of the position impression is high, whereas if the keyword only appears in deep indented lines, the value is lower.
The expression used to calculate the weighting schemata combined with the slide structure is:I p (s, k) =L(s)∑Considering Rareness of a Term

iafr(k, λ) = log# of scenes in range λ# of scenes having keyword k in its slide

We can apply iafr(k, λ) to all weighting schemata we previously discussed as the rareness factor.

P(s, l) · C(s, k, l)

l=1

where s denotes an identifier of the objective scene, k a target keyword, L(s) the total number of lines in a slide used inscene s, P(s.l) a function of the assigned point in the line l
in the slide for scene s, and C(s, k, l) a function of counting
keywords k in the line l of the slide for scene s.

2.2

3

We have proposed a method for using the speech information in a video [13]. If a target keyword not only appears inthe slide for a scene but also is frequently uttered in the scene,
the scene has to be highly ranked because the keyword is explained well in the scene.
In [13], we have proposed skc(s, k), the number of utterance
target keyword k in scene s. We have also proposed weighting
schemata to reflect the influence of speech by combination skc
and Ic .
Since the slide sequence information and speech information are independent, we should consider how the influenceof these information spreads to the neighbor scenes. We haveproposed the separated context consideration methods and theintegrated context consideration methods. Moreover, to deal
with speech recognition errors, we have proposed two criteria for the calculation of the influence of speech information.
The first criterion is that the influence of speech information
is calculated only when the uttered keyword appears in a slideof the scene(I p , 0). As the second criterion, we relax thecondition of the appearance range to neighbor scenes(Ic , 0).
As the combination of above consideration, we have proposedfour indicators.

Addition of Scene Duration Information

Duration information is useful in distinguishing multiple
appearances of the same slide caused by backtracking or reuseby the lecturer. To reflect the duration information of scenes
to the weighting schema, we have proposed the durationimpression indicator. The value of the position-impression
indicator is modified by the presentation time with a duration
parameter:
Id (s, k, θ) = T (s)θ · I p (s, k),where T (s) denotes the time used for scene s, and θ is the duration parameter for changing the influence of the time factor.

2.3

Addition of Context Information

We have combined the information of slide appearance
sequence to reflect the influence of context on the weighting schemata, which accumulates values of the durationimpression indicator within a presentation window indicatedby a window-size parameter δ:Ic (s, k, θ, δ, ε1 , ε2 ) =δ∑Exploiting Speech Information

First indicator is Ic+skc/p , which reflect the influence ofspeech information only when the keyword appear in slidesof the scene and is uttered in same scene. The definition of
Ic+skc/p is as follows:E(γ, ε1 , ε2 ) · Id (s + γ, k, θ)

γ=−δ

Ic+skc/p (Φ, ψ){
Ic (Φ) + ψ · T (s)θ · skc(s, k)=
Ic (Φ)

where E(x, ε1 , ε2 ) is a function to specify the effect of neighboring scenes in the context window. The effect of distance
between scenes is decided using the exponential function with
distance-effect parameters of ε1 and ε2 .
We describe these parameters (s, k, θ, δ, ε1 , ε2 ) as Φ in thispaper.

(I p , 0)(I p = 0)

where ψ is the spoken-keyword-count parameter to change theeffect of speech on the rating.
As next indicator Ic+skc/c , we relax the condition of the ap2

pearance range to neighbor scenes.
Ic+skc/c (Φ, ψ){
Ic (Φ) + ψ · T (s)θ · skc(s, k)=
0

To reflect the effect of laser pointer information on theweighting schemata, we modify Ic by adding phd(s, k) to theterm for the scene duration, and we denote it as Ic[d+phd] :(Ic , 0)(Ic = 0)Ic[d+phd] (Φ, ωd )s+δ∑=E(γ − s, ε1 , ε2 ) · Id[d+phd] (γ, k, θ, ωd )On the other hand, Ic+skc/c is another variation of the firstone. We consider the influence of speech information ofneighbor scenes when the keyword appears in the scene.
{Ic[p+skc] (Φ, ψ)(I p , 0)
Ic[p+skc/p] (Φ, ψ) =Ic (Φ)(I p = 0)γ=s−δId[d+phd] (s, k, θ, ωd )= {T (s) + ωd · phd(s, k)}θ · I p (s, k),where ωd is the pointer-hit-duration parameter that change theeffect of the duration of a hit by the laser pointer.
We have evaluated the weighting schemata combined withthe laser pointer information using an actual lecture materialin [12], and the experimental results indicated that the laser
pointer information is effective in improving the precision of
retrieval.

Ic[p+skc] (Φ, ψ)
=

s+δ
∑

E(γ − s, ε1 , ε2 ) · T (γ)θ · {I p (γ, k) + ψ · skc(γ, k)}γ=s−δAs fourth indicator, we have proposed Ic[p+skc/c] which reflects the influences of speech information of neighbor scenes
when the keyword appears in neighbor scenes:{
Ic[p+skc] (Φ, ψ)(Ic , 0)Ic[p+skc/c] (Φ, ψ) =0
(Ic = 0)4.2

Since the laser pointer pointing has several meanings forexample, to emphasize the keyword, to illustrate a relationshipbetween multiple concepts, and ambiguously pointing. We
have proposed two filtering methods to moderate the influenceof irrelevant laser pointer information for scene retrieval [11].
As the first filtering method, we have proposed “p-filter”, inwhich the laser pointer is ignored except when all keywords
in a query exist in the slide. We have suggested that the laser
pointer pointing to a line not containing all target keywords
have few meaning for the query. We have defined phd/p (s, k)
as phd(s, k) considering the condition:{∏phd(s, k)I (s, k) , 0∏k∈K pphd/p (s, k) =0
k∈K I p (s, k) = 0

We have also proposed isfr(k, λ), the inverse keyword k spoken scene frequency in a range λ, as another rareness factor
dedicated to speech information[13].
isfr(k, λ) = log# of scenes in range λ# of scenes uttering keyword k

To apply rareness factor to an indicator I x , we combine itwith isfr and iafr using ψ.
I x · ia f r · is f r(Φ, ψ, λ){Ic (Φ) · ia f r(k, λ)(I p · Ic = 0)=
Ic (Φ) · ia f r(k, λ) + ψ · I x · is f r(k, λ) (otherwise)4

Exploiting Pointing and Speech Information

4.1

Extraction of Laser Pointer Information

where K is a set of keywords in a query.
We modify Ic[d+phd] by replacing phd(s, k) by phd/p (s, k),
and we denote it as Ic[d+phd/p] as follows:Ic[d+phd/p ] (Φ, ωd )In [12], we first extract the laser pointer information byusing the method of extracting radiant information of thelaser pointer as coordinates in the slide by an image analysis
technique[6]. We extract subscenes from a scene to make each
subscene contains a continuous pointer information. Since
there is ambiguity regarding target keywords caused by shaking or any other habit of the lecturer, we distribute the possibilities of a hit by the pointer in a subscene to the neighborhoodlines and make their sum equals to 1. H(l, q) denotes the hit
probability of line l in subscene q of scene s. We make H(l, q)
related to the distance between the line l and the point. We
then propose an indicator phd(s, k) by multiplying the possibilities of a hit by the duration of each subscene.
phd(s, k) =

L(s)
∑∑

Pointing Information Filtering based on Keyword Occurrence in Slides and/or Speech

=

s+δ
∑

E(γ − s, ε1 , ε2 ) · Id[d+phd/p ] (γ, k, θ, ωd )γ=s−δ

Id[d+phd/p ] (s, k, θ, ωd )={T (s) + ωd · phd/p (s, k)}θ · I p (s, k)

When a laser pointer hits target keywords in a slide andkeywords are not spoken by a lecturer in the scene, we canconsider that the lecturer do not use the laser pointer for emphasizing the keywords. To moderate the effects of such laser
pointer information, we have proposed “s-filter” in which thelaser pointer is ignored except when the keywords are spoken
by the lecturer in the scene. We have defined phd/s (s, k) as
phd(s, k) considering this condition:{phd(s, k)skc(s, k) , 0
phd/s (s, k) =0
skc(s, k) = 0

H(l, qi ) · T (qi )qi ∈s l=1

3

0.63

We replace phd(s, k) by phd/s (s, k) in Ic[d+phd] , and add
skc(s, k) . We denote it as Ic[d+phd/s] :Ic+skc/c
0.61=

Mean Reciprocal Rank

Ic[d+phd/s ,p+skc/p ] (Φ, ωd , ψ)s+δ∑E(γ − s, ε1 , ε2 ) · Id[d+phd/s ,p+skc/p ] (γ, k, θ, ωd , ψ)

γ=s−δ

Id[d+phd/s ,p+skc/p ] (s, k, θ, ωd , ψ)=

Ｉc+skc/p

0.62

θ{T (s) + ωd · phd/s (s, k)} · {I p (s, k) + ψ · skc/p (s, k)},Ic[p+skc/s]
Ic[p+skc/c]

0.6
0.59
0.58
0.57
0.56

Ic

0.55

Since a keyword’s occurrences in slides and in speech are
independent, we have proposed the weighting factor that considers both of the two filtering methods. We have defined anindicator as phd/ps (s, k)Experimental Evaluation

5.1

Experimentation Setting and Data

0

2

3

4

5

ψ

6

7

8

9

10

0.63

We evaluate our proposed methods to exploit speech andpointing information using a series of videos from two actual
lecture courses on databases and on computer architecture respectively.
For the experimentations, we run 124 retrievals using different sets of keywords. These keywords consist of 78 keywords about computer architecture and 46 keywords about
databases. Testers decide only one relevant scene as the answer scene corresponding the keywords.
We use the open-source speech recognition software
Julius 1 to derive the speech information from the lecture
videos. As the language and acoustic model for the speech
recognition, we use a general model generated using CSJ[8]in [15]. We add some words from the lecture slides into thedictionary for speech recognition, which has not been originally included in it.
As the evaluation measure, we use mean reciprocal rank
(MRR). MRR is commonly used for evaluation of thequestion-answering system such as TREC[14]. The definitionof MRR is as follows:

0.62
0.61
0.6
0.59
0.58

Ic[p+skc/p]

0.57

Ic[p+skc/p] ・iafr ・isfr

0.56
0.55
0

5

10

15

20

ψ

Figure 3. Comparison of impression indicators
considering the rareness of a term
retrieval and the influence of speech information of neighborscenes is useful.
To evaluate the rareness factor of speech information, we
compare Ic[p+skc/p] to Ic[p+skc/p] · ia f r · is f r. Figure 3 shows
MRR of Ic[p+skc/p] and Ic[p+skc/p] · ia f r · is f r varying ψ.
An indicator considering rareness factor, “Ic[p+skc/p] · ia f r ·is f r” improves the MRR and its peak value is 0.623(ψ = 15).
This result shows the consideration rareness factor of speech
information is effective for scene retrieval.

5.3

1 ∑
1
MRR =
N i=1 ranking by the i-th retrieval
N

Evaluate of Pointing Information Integration

We evaluate our proposed methods to filter the laser pointerinformation using actual lecture materials. First we describe
the setting and data used in our experiments, then we show
the experimental results and discuss characteristics of the lectures.
We fix the parameters: θ = 0.4, δ = 4, ε1 = 5.0，ε2 = 0.5
and ψ = 1. We also distributed the probabilities of a hit by
the pointer H(l, q) to five neighborhood lines as 0.4, 0.3, 0.15,
0.1, and 0.05.
We compare our proposed weighting schemata, Ic[d+phd/p ] ，Ic[d+phd/s ,p+skc/p ] ，Ic[d+phd/ps ,p+skc/p ] , to the existing methods, Ic
and Ic[d+phd] .

where N is the number we retrieve.

5.2

1

Figure 2. Comparison of four indicators varying
ψ per 1 point
Mean Reciprocal Rank

5

0.54

Evaluation of Speech Information Integration

To evaluate the integration method, we calculate the MRR
of Ic+skc/p ，Ic+skc/c ，Ic[p+skc/p] ，Ic[p+skc/c] varying ψ per 1
point. We fixed the parameters: θ = 0.5，δ = 4，ε1 = 5.0，
ε2 = 0.5，λ = course.
Figure 2 shows the comparison of the four indicators varying ψ. When ψ = 0, each indicator equals Ic and MRR
is 0.560 which is not considering speech information. The
peak MRR of each method is: Ic+skc/p =0.595，Ic+skc/c =0.592，
Ic[p+skc/p] =0.619 Ic[p+skc/c] =0.610. This result shows that theconsideration of speech information is positive effect on scene

Figure 4 shows the MRR of the two lectures. The experimental results in Fig. 4 indicate that the proposed weighting schemata are more precise than Ic and Ic[d+phd] . The
MRR decreases in Ic[d+phd] when ωd increases excessively, but
not in our proposed methods. This means that the influence

1 http://julius.sourceforge.jp/

4

0.67

0.59

0.66

0.58

0.65
MRR

MRR

0.6

0.57
0.56

0.64
0.63
0.62

0.55

0.61

0.54
0.6

0.53

1

30 ω dIc[d+phd]
Ic[d+phd/s,p+skc/p]

0.45
0.43

1

0.41
25

30 ω d51

0.47

Database
0 .1
0 .0 90 .0 80 .0 70 .0 60 .0 50 .0 40 .0 30 .0 20 .0 1
0

56

0.49

20

30 ω d

25

46

MRR

0.51

15

20

Ic[d+phd]
Ic[d+phd/s,p+skc/p]

41

0.53

10

15

36

# o f qu e rie s / to tal # o f
qu e rie s

0.55

5

10

Figure 6. Comparison of MRR of the Lecture
about Computer Architecture

Figure 4. Comparison of MRR of the two Lectures

1

5

Ic
Ic[d+phd/p]Ic[d+phd/ps,p+skc/p]

26

25

31

Ic
Ic[d+phd/p]Ic[d+phd/ps,p+skc/p]

20

16

15

21

10

6

5

11

1

# o f sc e n e s t h at ke ywo rd appe ars in th e slide
Ic
Ic[d+phd/p]Ic[d+phd/ps,p+skc/p]

Ic[d+phd]
Ic[d+phd/s,p+skc/p]

Figure 7. Distribution of the Search Keywordsof the Database Lecture

Figure 5. Comparison of MRR of the Lecture
about Databasefect the ranking scenes. For example, the lecture of databaseincludes some exercise scenes. Exercises scenes usually havelong duration because students need some time to solve problems, and lecturers sometimes use a slide for exercises thata particular keyword appears many times. These distinctions
make scene retrievals difficult.
We also suppose that the difference of keyword distribution influences the difficulty of ranking scenes. Figure 7 and 8
show the distribution of the search keywords using this experiments. Most keywords of the computer architecture lectureappear in a few particular scenes in contrast to some keywordsof the database lecture which appear in more than 20 scenes.
Table 1 shows the average number of scenes that a search
keyword appears in the slide for each lecture, and standard
deviation of it. This result means that the keywords of thedatabase lecture distribute twice more widely than the computer architecture lecture. That is to say, the scene retrieval
on the database lecture is more difficult than on the computerarchitecture lecture.

of irrelevant pointing grows larger as ωd steadily enlarging,whereas the proposed methods moderate the influence of thelaser pointer information.
Figure 5 and 6 illustrate the MRR on each lecture. According to Fig. 5, the laser pointer information does not affectpositively on the scene retrieval by the existing methods. Thisis because the lecture has construction of slides and topics thatis not suitable for ranking based on the information of the text
on the slides. However, this graph shows that the laser pointerinformation filtered by speech information improve the MRR
when parameter ωd increases. As a result, we can moderate
the effect of the laser pointer for purposes expect emphasis
keywords.
In contrast, Fig. 6 shows that the laser pointer informationin the existing methods favorably influence the outcome ofscene retrieval, and the weighting schema filtered by text inthe slide gets a best deal. This means that this lecture’s characteristics differ to the former’s, and is suitable for methods
based on slide construction.

5.4

Analysis of Lecture Characteristics

Table 1. Search Keywords Distribution on each
Lectureaverage # of scenes standard deviation
Database
17.9
13.8
Architecture
8.2
6.6

As characteristics of the lectures are various, and effective
information for ranking scenes differ by lectures, we analyze
some lecture characteristics of slide materials so as to examine the influence of the characteristics on scene retrieval. First,
we suppose that the structural differences between lectures af5

References# o f qu e rie s / to tal # o f
qu e rie s

Co mpu te r Ar c hit ec tu r e
0 .1 40 .1 2[1] G. D. Abowd. Classroom 2000: an experiment with the instrumentation of a living edu cational environment. IBM Syst. J.,
38(4):508–530, 1999.
[2] A. Fujii, K. Itou, and T. Ishikawa. Lodem: A system for ondemand video lectures. Speech Communication, 48(5):516–
531, May 2006.
[3] A. G. Hauptmann and M. J. Witbrock. Informedia: news-ondemand multimedia information acquisition and retrieval. In
M. T. Maybury, editor, Intelligent multimedia information retrieval, pages 215–239. MIT Press, 1997.
[4] G. J. Jones and R. J. Edens. Automated alignment and annotation of audio-visual presentations. In Proc. Proceedings of the
6th European Conference on Research and Advanced Technology for Digital Libraries(ECDL’02), pages 276–291, Rome,
Italy, Sep. 2002.
[5] Y. Kambayashi, K. Katayama, Y. Kamiya, and O. Kagawa. Index generation and advanced search functions for multimedia
presentation material. In Proc. of ER97 Workshop on Conceptual Modeling in Multimedia Information Seeking, 1997.
[6] Y. Katsuyama, N. Ozawa, J. Sun, H. Takebe, T. Kobayashi,
H. Yokota, and S. Naoi. A new solution for extracting
laser pointer information from lecture videos. In Proc. of
E-learn2004, pages 2713–2718, Washinton DC, USA, Nov.
2004.
[7] T. Kobayashi, T. Muraki, S. Naoi, and H. Yokota. An implementation of experimental system for storing and retrieving
unified presentation contents (in japanese). IEICE Transactions on Information and Systems, J88-D-I(3):715–726, Mar.
2005.
[8] K. Maekawa, H. Koiso, S. Furui, and H. Isahara. Spontaneous
speech corpus of japanese. In Proc. LREC2000, volume 2,
pages 947–952, Athens, Greece, May 2000.
[9] O. Marques and B. Furht. Content-Based Image and Video
Retrieval. Kluwer, 2000.
[10] R. Müller and T. Ottmann. The ”Authoring on the Fly” system for automated recording and replay of (tele)presentations.
Multimedia Syst., 8(3):158–176, 2000.
[11] W. Nakano, T. Kobayashi, Y. Katsuyama, S. Naoi, andH. Yokota. Treatment of laser pointer and speech informationin lecture scene retrieval. In Proc. Eighth IEEE Intl Symp. on
Multimedia, pages 927–932, San Diego, USA, Dec. 2006.
[12] W. Nakano, Y. Ochi, T. Kobayashi, Y. Katsuyama, S. Naoi, andH. Yokota. Unified presentation contents retrieval using laser
pointer information. In Proc. of SWOD2005, pages 170–173,
Apr. 2005.
[13] H. Okamoto, W. Nakano, T. Kobayashi, S. Naoi, H. Yokota,
K. Iwano, and S. Furui. Presentation-content retrieval integrated with the speech information. IEICE Transactions on
Information and Systems, 2007. (to appear).
[14] E. M. Voorhees and D. M. Tice. The trec-8 question answering
track evaluation. In Proc. of TREC-8, pages 83–105, 1999.
[15] H. Yamazaki, K. Iwano, K. Shinoda, S. Furui, and H. Yokota.
Using presentation slide information for lecture speech recognition. Technical Report SP2006-122, IEICE, Dec. 2006.
[16] H. Yokota, T. Kobayashi, T. Muraki, and S. Naoi. Uprise:
Unified presentation slide retrieval by impression search engine. IEICE Transactions on Information and Systems, E87D(2):397–406, Feb 2004.
[17] H. Yokota, T. Kobayashi, H. Okamoto, and W. Nakano. Unified contents retrieval from an academic repository. In Proc.
of International Symposium on Large-scale Knowledge Resources LKR2006, pages 41–46, Mar. 2006.

0 .1
0 .0 80 .0 60 .0 40 .0 2

51

56

46

36

41

26

31

21

16

6

11

1

0

# o f sc e n e s t h at ke ywo rd appe ars in th e slide

Figure 8. Distribution of the Search Keywordsof the Computer Architecture Lecture

6

Conclusions and Future WorkIn this paper, we discussed how to exploit various featuresin lecture videos such as the laser pointer and speech information for the efficient retrieval method. We introduced our
information integration approaches. We explained our weighting schemata with speech information considering the criteriaof appearance of the keyword in each slide and how influenceof speech information spreads to neighbor scenes. We also
explained our weighting schemata by using the filtered laser
pointer based on keyword occurrences in slides and speech.
Moreover, we evaluated our approach by using actual lecture
videos and presentation slides.
Our experimental results using actual lecture materials indicate that the consideration of speech information and its
rareness factor can improve the precision of retrieval and the
dedicated weighting schemata for filtered laser pointer information are also effective for scene retrieval. We also evaluated on each lecture, confirmed that the influence of irrelevantlaser pointer information was moderated on scene retrieval,and analyzed difference of each lecture’s characteristics foreach information.
As future direction to this work, we should build up methods that apply the laser pointer information and the speech information comprehensively and more effectively. Moreover,consideration of the influence of the speech recognition rateis also an important issue for the integration. We have to discuss about several problems when we use the speech information, for example, recognition error, and notation difference
between dictionaries used in speech recognition and in text inslides.

Acknowledgment
This work is partially supported by a Grant-in-Aid for
Scientific Research of MEXT Japan(#15017233, 16016232,
18049026), Tokyo Institute of Technology 21COE Program”Framework for Systematization and Application of LargeScale Knowledge Resources”, and CREST of JST(Japan Science and Technology Agency).
6