論文 / 著書情報
Article / Book Information
Title

Treatment of Laser Pointer and Speech Informationin Lecture Scene
RetrievalAuthorWataru NAKANO, Takashi Kobayashi, Yutaka KATSUYAMA, Satoshi
NAOI, Haruo YOKOTA

Journal/Book nameProc. of IEEE International Symposium on Multimedia (ISM2006), Vol. ,No. , pp. 927-932

Issue date

2006, 12

DOI

10.1109/ISM.2006.153

URL

http://www.ieee.org/index.html

Copyright(c)2006 IEEE. Personal use of this material is permitted. Permission
from IEEE must be obtained for all other users, including
reprinting/republishing this material for advertising or promotional
purposes, creating new collective works for resale or redistribution toservers or lists, or reuse of any copyrighted components of this work inother works.

Note

このファイルは著者（最終）版です。
This file is author (final) version.

Powered by T2R2 (Science Tokyo Research Repository)

Treatment of Laser Pointer and Speech Informationin Lecture Scene Retrieval
Wataru NAKANO1
1Takashi KOBAYASHI2
Yutaka KATSUYAMA3
Haruo YOKOTA2,1

Satoshi NAOI3,2

Department of Computer Science, Graduate School of Information Science and Engineering,Tokyo Institute of Technology
2
Global Scientiﬁc Information and Computing Center, Tokyo Institute of Technology
3
FUJITSU LABORATORIES Ltd.
{wnakano@de.cs, tkobaya@gsic}.titech.ac.jp, {katsuyama, naoi.satoshi}@jp.fujitsu.com,
yokota@cs.titech.ac.jp
Abstract

We have previously proposed a uniﬁed presentation contents search mechanism named UPRISE (Uniﬁed Presentation Slide Retrieval by Impression Search Engine), and havealso proposed a method to use laser pointer information in
lecture scene retrieval. In this paper, we discuss the treatment of the laser pointer and speech information, and propose two methods to ﬁlter the laser pointer information using keyword occurrence in slides and speech. We also propose weighting schemata with ﬁltered laser pointer information using slide text and speech information. We evaluate our approach by using actual lecture videos and presentation slides.

1. Introduction
Recently, systems to store and retrieve integrated multimedia contents, such as video and documents, have been
proposed[1, 2, 3] and widely used in a variety of contexts,
such as web-based training and e-learning. In particular, it isimportant for e-learning systems that users can both retrieve
suitable content and ﬁnd eﬀectively the particular scenesthat they wish to study.
We have proposed the UPRISE (Uniﬁed Presentation
Slide Retrieval by Impression Search Engine) system[4, 5,
6]. It combines slides in a presentation and a video recording of the presentation, and retrieves a sequence of desired presentation scenes from archives of the combined
content. UPRISE retrieves a relevant scene by weighting
schemata that consider keyword positions in slides, duration of a scene, and context in a presentation. In our previous work[4], we showed that UPRISE’s precision is muchbetter than those of ordinary tf-idf -based approaches.
We have also proposed a method to reﬂect the inﬂuenceof laser pointer activity on the weighting schemata, and weevaluated the proposed method with actual presentations[7].
The experimental results showed that the UPRISE’s precision was improved by using laser pointer information. However, by analyzing the result, we found that various aspectsof laser pointer pointing needed to be destinguished because
several ones, such as to illustrate relationships between multiple concepts, and ambiguous pointing, had no positive effect on the scene retrieval.
In this paper, we discuss the treatment of laser pointer activity and speech information in lecture scene retrieval, andwe address the problem of the inﬂuence of irrelevant pointing on retrieval. We propose two methods to ﬁlter the laser
pointer information using keyword occurrence in slides andspeech to moderate the inﬂuence. We also propose weighting schemata to combine ﬁltered laser pointer information
with slide text and speech information.
Some previous studies have investigated crossover retrievals for lecture contents[8, 9, 10, 11]. However, someof them do not result in actual retrieval methods and systems. [11] provided a lecture passage retrieval system usingtranscription by speech recognition. However, they did notconsider the case of backtracking or reuse of the slide materials.
The remainder of this paper is organized as follows. In
Section 2, we consider a number of weighting schemata to
retrieve the uniﬁed contents. Then, we discuss the treatmentof laser pointer information in lecture scene retrieval in Section 3. In Section 4, we propose a method to ﬁlter the laser
pointer information, and Section 5 reports experiments andresults using actual lecture materials. We summarize the paper’s main points in the ﬁnal section.

2. Weighting Schemata in UPRISEIn UPRISE, we modeled a lecture’s contents as a sequence of scenes divided by slide changes and proposeda slide identiﬁcation technique[12] for automatic contents
creation.
Because our deﬁnition of a scene is the duration between
slide changes, there are many scenes in which the same slide
appears because of backtracking or reuse by the lecturer,
as shown in Fig 1. In this case, we cannot distinguish thescenes in which the same slide appears by using only text
information in the slide.
To distinguish these scenes, we have incorporated otherinformation such as the context and duration of scenes,
speech information and laser pointer information. We haveproposed impression indicators as the weighting schemata
in UPRISE[4].

2.1. Adding Structure Information
First, we consider the structure of a lecture slide. If
a given word appears in the title of the slide or in lines
less indented, the value of the position impression is high,whereas, if the keyword only appears in lines indented deep,
the value is lower. The expression used to calculate theweighting schemata combined with the slide structure is:I p (s, k) =L(s)∑P(s, l) · C(s, k, l),l=1

where s denotes an identiﬁer of the objective scene, k a target keyword, L(s) the total number of lines in a slide in s,P(s.l) is a function of the assigned point in the line l in theslide for s, and C(s, k, l) is a function of counting keywords
k in the line l of the slide for scene s.

2.2. Adding with Duration Information
Duration information is useful for distinguishing multiple appearances of the same slide caused by backtracking orreuse by the lecturer. To add the duration information to theweighting schema, we propose the duration-impression indicator; its value is modiﬁed by the presentation time witha duration parameter:
Id (s, k, θ) = T (s)θ · I p (s, k),where T (s) denotes the time used for scene s, and θ is theduration parameter for changing the inﬂuence of the time
factor.

sc en e i

sc en e i+ 1

slide Aslide B

concept A
concept B

ex a m pl e A

sc en e i+ 2

sc en e i+ 3

slide Aslide C

concept A
concept BTime

ex a m pl e B

b a ck tr a ck i ng or r eu s e

Figure 1. Example of scenes in which thesame slide appears
schemata, which accumulates values of the durationimpression indicator within a presentation window with its
duration deﬁned as a window-size parameter δ:Ic (s, k, θ, δ, ε1 , ε2 ) =s+δ∑E(γ − s, ε1 , ε2 ) · Id (γ, k, θ),

γ=s−δwhere E(x, ε1 , ε2 ) is a function to specify the eﬀect ofneighboring scenes in the context window. The eﬀect of distance between scenes is decided using the exponential function with distance–eﬀect parameters of ε1 and ε2 .
{exp(ε1 x)(x < 0)E(x, ε1 , ε2 ) =exp(−ε2 x) (x ≥ 0)This means that we can alter the eﬀect of the information
for its initial context and later contexts. For example, when
we want to emphasize the starting point of an explanation
related to the keyword, we set ε1 = 5 and ε2 = 0.5.
We describe these parameters (s, k, θ, δ, ε1 , ε2 ) as Φ inthis paper.

2.4. Adding with Speech Information
We proposed a method for using the speech information
in a video[6]. If a target keyword not only appears in theslide for a scene but is also frequently uttered in the scene,
the scene should be ranked highly because the keyword is
explained we in detail in the scene.
In [6], we extracted the speech information from a lecture video using a technique of speech recognition, and thenwe proposed skc(s, k), the number of utterances of target
keyword k in scene s. We also proposed weighting schemata
to reﬂect the inﬂuence of speech by the combination of skc
and Ic .

3. Weighting Schemata Combined with Laser
Pointer Information

2.3. Adding Context Information
We add information about the slide appearance sequence to reﬂect the inﬂuence of context on the weighting

Lecturers usually use a laser pointer to emphasize partof the text in a slide. In other words, when a laser pointer isused in a presentation, the information about points selectedand the experimental results indicate that the laser pointerinformation was eﬀective in improving the precision of retrieval.

title
P’s o th er
lin esp o in ter P

P’s lin e

a s lid e o f p r es en ta tio n

Figure 2. Extracting laser pointer informationby the laser pointer can be used to improve the precision of
retrievals. We have already proposed a method of reﬂecting
the laser pointer information on the weighting schemata[7].
In [7], we ﬁrst extracted the laser pointer information byusing the method of extracting radiant information from thelaser pointer as coordinates in the slide by the technique ofimage analysis[13]. We extracted subscenes from a scene
to make each subscene contain continuous pointer information. Because there is ambiguity regarding target keywords
caused by shaking or habits of the lecturer, we distributethe possibilities of a hit by the pointer in a subscene to theneighborhood lines and make their sum unity:L(s)∑H(l, q) = 1,

l=1where H(l, q) denotes the hit probability of line l in subscene q of scene s. We make H(l, q) a function of the distance between the line l and the point, as Fig. 2.
We then create an indicator phd(s, k) by multiplying thepossibilities of a hit by the duration of each subscene.
phd(s, k) =

L(s)
∑∑

H(l, qi ) · T (qi ).

qi ∈s l=1

To reﬂect the eﬀect of laser pointer information in theweighting schemata, we modify Ic by adding phd(s, k) to
the term for the scene duration, and we denote it as Ic[d+phd] :Ic[d+phd/p ] (Φ, ωd )s+δ∑=E(γ − s, ε1 , ε2 ) · Id[d+phd/p ] (γ, k, θ, ωd )γ=s−δ

Id[d+phd/p ] (s, k, θ, ωd )= {T (s) + ωd · phd/p (s, k)}θ · I p (s, k),where ωd is the pointer hit duration parameter that changes
the eﬀect of the duration of a hit by the laser pointer.
We evaluated the weighting schemata combined with thelaser pointer information using actual lecture material in [7],4. Filtering the Laser Pointer Information
As mentioned before, we need to consider diﬀerence invarious aspects of laser pointer pointing. To moderate theeﬀect of several ones, we ﬁlter out laser pointer information related to these aspects. In this section, we propose two
methods to ﬁlter the laser pointer information based on keyword occurrence in slides and in speech. In addition, we
propose a method considering both conditions together.

4.1. Filtering Based on Keyword Occurrence in
SlidesWhen we retrieve scenes, we usually use multiple keywords in a query. However, in our previous work, we considered the inﬂuence of the laser pointer for each keyword
independently. Therefore, the laser pointer information affected the ranking of scenes even if not all the keywords ina query existed in a slide.
To correct this problem, we propose a method in which
the laser pointer is ignored except when all keywords in aquery exist in a slide, because we suggest that lines highlighted by the laser pointer that do not contain all keywords should not add emphasis for the query. We deﬁne
phd/p (s, k) as phd(s, k) considering the condition:{∏phd(s, k)I (s, k) , 0∏k∈K pphd/p (s, k) =,0
k∈K I p (s, k) = 0
where K is the set of keywords in a query.
We modify Ic[d+phd] by replacing phd(s, k) by
phd/p (s, k), and we denote it as Ic[d+phd/p] , as follows:Ic[d+phd/p ] (Φ, ωd )s+δ∑=E(γ − s, ε1 , ε2 ) · Id[d+phd/p ] (γ, k, θ, ωd )γ=s−δ

Id[d+phd/p ] (s, k, θ, ωd )= {T (s) + ωd · phd/p (s, k)}θ · I p (s, k)

4.2. Filtering Based on Keyword Occurrence in
SpeechWhen a laser pointer hits target keywords in a slide but
the keywords are not spoken by the lecturer, we can assume
that the lecturer did not use the laser pointer to emphasize
the keywords. To moderate the eﬀects of such laser pointerinformation, we propose that the laser pointer is ignored except when the keywords are spoken by the lecturer in thescene. We deﬁne phd/s (s, k) as phd(s, k) considering thiscondition:{phd(s, k) skc(s, k) , 0
phd/s (s, k) =0
skc(s, k) = 0




We replace phd(s, k) by phd/s (s, k) in Ic[d+phd] , and add
skc(s, k). We denote it as Ic[d+phd/s] :Ic[d+phd/s ,p+skc/p ] (Φ, ωd , ψ)s+δ∑=E(γ − s, ε1 , ε2 ) · Id[d+phd/s ,p+skc/p ] (γ, k, θ, ωd , ψ)
γ=s−δ

Id[d+phd/s ,p+skc/p ] (s, k, θ, ωd , ψ)
= {T (s) + ωd · phd/s (s, k)}θ · {I p (s, k) + ψ · skc/p (s, k)},where ψ is the spoken-keyword-count parameter to change
the eﬀect of speech on the rating, and skc/p(s, k) is the function that calculates skc(s, k) only when the keyword k exists
in the slide using in the scene s:{skc(s, k) I p (s, k) , 0
skc/p (s, k) =0
I p (s, k) = 0

4.3. Filtering Based on Both Conditions
Because keyword occurrences in slides and in speech are
independent, we propose a weighting factor that considers
both. We deﬁne an indicator phd/ps (s, k), as follows:

Figure 3. MRRs for the two lectures
We used the open-source speech recognition software
Julius 1 to derive the speech information from the lecture
videos. We added some words from the lecture slides to thedictionary for speech recognition, which were not originally
included in it.
We ﬁxed the parameters: θ = 0.4, δ = 4, ε1 = 5.0, ε2 =0.5, and ψ = 1. We also distributed the probabilities of a hit
by the pointer H(l, q) to ﬁve neighborhood lines as 0.4, 0.3,
0.15, 0.1, and 0.05.
In this paper, we use mean reciprocal rank (MRR) for theevaluation measure. MRR is commonly used for the evaluation of question-answering systems such as TREC[14]. The
deﬁnition of MRR is:

phd/ps (s, k){∏phd(s, k)k∈K I p (s, k) , 0 ∧ skc(s, k) , 0=0otherwise
We also propose a weighting schema Ic[d+phd/ps ,p+skc/p ] toreplace phd/s (s, k) by phd/ps (s, k) in Ic[d+phd/s ,p+skc/p ] .

5. Experimental Evaluation
We evaluated our proposed methods to ﬁlter laser pointerinformation using actual lecture materials. First, we describe the setting and the data used in our experiments, andthen we show the experimental results and discuss characteristics of the lectures.

5.1. Experiment Setting and Data
We evaluated our proposed methods to ﬁlter laser pointerinformation using a series of videos from two actual lectures: one about databases and one about computer architecture.
For the experiments, we ran 124 retrievals using diﬀerent sets of keywords, 78 about computer architecture and 46
about databases. Testers selected one relevant scene as thebest scene corresponding to the keywords.

                                  1 ∑1,
N i=1 ranking by the ith retrieval
N

MRR =

where N is the number of retrieval.

5.2. Experimental Results
We compare our proposed weighting schemata,Ic[d+phd/p ] , Ic[d+phd/s ,p+skc/p ] , Ic[d+phd/ps ,p+skc/p ] , to the existing methods, Ic and Ic[d+phd] .
Figure 3 shows the MRRs for the two lectures. The
results indicate that the proposed weighting schemata are
more precise than Ic and Ic[d+phd] . The MRR decreases inIc[d+phd] when ωd is made too large, but the proposed methods do not. This means that the inﬂuence of irrelevant pointing grows larger with a steadily enlarging ωd , whereas
the proposed methods moderate the inﬂuence of this laser
pointer information.
Figures 4 and 5 illustrate the MRRs for each lecture. According to Fig. 4, the laser pointer information had no positive eﬀect on the scene retrieval with the existing methods inthe databases lecture. This is because the lecture’s construction of slides and topics is not suitable for ranking based on
1

http://julius.sourceforge.jp/                                                                        

Figure 4. MRRs for the lecture about
databases

Figure 6. Distribution of the search keywordsof the database lecture                                












        




         


















                                    Figure 5. MRRs for the lecture about computer architecture
information about the text on the slides. However, this graphshows that the laser pointer information ﬁltered by speech
information improves the MRR when the parameter ωd increases. As a result, we can moderate the eﬀect of the emphasis the laser pointer adds to keywords.
In contrast, Fig. 5 shows that, with the existing methods, the laser pointer information favorably inﬂuenced theoutcome of scene retrieval in the computer architecture lecture, and the weighting schema ﬁltered by text in the slideachieves the best score. This means that this lecture’s characteristics diﬀer from the those of the ﬁrst lecture, and issuitable for retrieval methods based on slide construction.

5.3. Analysis of Lecture Characteristics
As the above discussion shows, the characteristics of thetwo lectures diﬀer. We analyze some lecture characteristics
of slide materials to understand the inﬂuence of the characteristics on scene retrieval. First, we assume that structural
diﬀerences between lectures aﬀect the ranking of scenes.

Figure 7. Distribution of the search keywordsof the computer architecture lecture
For example, the lecture on database includes some exercise scenes. Exercise scenes usually have long durations because the students require some time to solve problems, andlecturers sometimes use a slide for exercises in which a particular keyword appears many times. These characteristics
make scene retrievals more diﬃcult.
We also assume that diﬀerences in keyword distribution
inﬂuence the diﬃculty of ranking scenes. Figures 6 and 7
show the distribution of the search keywords used in theseexperiments. Most keywords of the computer architecture
lecture appear in a few scenes in contrast to some keywords in the databases lecture, which appear in more than
20 scenes.
Table 1 shows the average number of scenes containing
Table 1. Search keywords distribution on eachlectureAverage # of scenes Standard deviation
Database
17.9
13.8
Architecture
8.2
6.6

a search keyword in the slide for each lecture, and the standard deviation. These results show that the keywords in the
database lecture are distributed twice as widely as those in
the computer architecture lecture. That is, scene retrieval forthe database lecture is more diﬃcult than for the computerarchitecture lecture.

6. Conclusions and Future WorkIn this paper, we have discussed the treatment oflaser pointer and speech information in lecture scene retrieval, and propose two methods to ﬁlter the laser pointerdata phd(s, k) based on keyword occurrences in slides andspeech. We then combine the ﬁltered laser pointer information with the weighting schemata in UPRISE.
Our experimental results using actual lecture materials
indicate that the dedicated weighting schemata for ﬁlteredlaser pointer information are eﬀective in improving the precision of retrieval. We also evaluated each lecture, conﬁrming that the inﬂuence of irrelevant laser pointer information
is moderated on scene retrieval, and we analyzed diﬀerencesin each lecture’s characteristics for each information type.
In this paper, we use the speech information only to ﬁlter the laser pointer information, but we have proposed thatspeech information is eﬀective in retrieving scenes. Therefore, we plan to develop a method in future work that applies laser pointer and speech information comprehensively
and more eﬀectively.
We also plan to evaluate the inﬂuence of the speech
recognition rate. We must consider some problems when
we apply speech information; for example, recognition error and notation diﬀerence between the dictionary used inspeech recognition and the text in slides. We can simulate
the situation where we can correct recognized speech by using manual transcription of speech text.
The rareness factor is important for the weighting schemata in information retrieval. For tf-idf, the inverse
document frequency, idf, is the rareness factor. We have reported on the eﬀectivity of the rareness factor for scene
retrieval [6].
We will consider the rareness factor of keywords inslides and speech in our proposed methods.
Moreover, we will focus on the case of users submitting
synonymous keywords with text on slides and resolve theproblem by, for example, using thesaurus expansion of the
keywords.

Acknowledgment
This work is partially supported by a Grant-in-Aid for
Scientiﬁc Research of MEXT Japan(#15017233, 16016232,
18049026), Tokyo Institute of Technology 21COE Program”Framework for Systematization and Application of LargeScale Knowledge Resources”, and CREST of JST(Japan
Science and Technology Agency).

References[1] R. Müller and T. Ottmann.
The ”Authoring on theFly” system for automated recording and replay of(tele)presentations. Multimedia Syst., Vol. 8, No. 3, pp. 158–
176, 2000.
[2] A. G. Hauptmann and M. J. Witbrock. Informedia: news-ondemand multimedia information acquisition and retrieval. In
M. T. Maybury, editor, Intelligent multimedia information retrieval, pp. 215–239. MIT Press, 1997.
[3] G. D. Abowd. Classroom 2000: an experiment with the instrumentation of a living edu cational environment. IBM
Syst. J., Vol. 38, No. 4, pp. 508–530, 1999.
[4] H. Yokota, T. Kobayashi, T. Muraki, and S. Naoi. UPRISE:
Uniﬁed Presentation Slide Retrieval by Impression SearchEngine. IEICE Trans. on Info. and Syst., Vol. E87-D, No. 2,
pp. 397–406, 2 2004.
[5] T. Kobayashi, T. Muraki, S. Naoi, and H. Yokota. A Searching System on Uniﬁed Presentation Contents (in Japanese).
IEICE Trans. on Info. and Syst., Vol. J88-D-I, No. 3, pp. 715–
726, 3 2005.
[6] H. Yokota, T. Kobayashi, H. Okamoto, and W. Nakano. Uniﬁed Contents Retrieval from an Academic Repository. In
Proc. of International Symposium on Large-scale Knowledge
Resources LKR2006, pp. 41–46, 3 2006.
[7] W. Nakano, Y. Ochi, T. Kobayashi, Y. Katsuyama, S. Naoi,
and H. Yokota. Uniﬁed Presentation Contents Retrieval Using Laser Pointer Information. In Proc. of SWOD2005, pp.
170–173, 4 2005.
[8] O. Marques and B. Furht. Content-Based Image and Video
Retrieval. Kluwer, 2000.
[9] Y. Kambayashi, K. Katayama, Y. Kamiya, and O. Kagawa.
Index Generation and Advanced Search Functions for Multimedia Presentation Material. In Proc. of ER97 Workshop
on Conceptual Modeling in Multimedia Information Seeking, 1997.
[10] G. J. F. Jones and R. J. Edens. Automated Alignment
and Annotation of Audio-Visual Presentations. In Proc. of
ECDL2002, pp. 276–291, Sep. 2002.
[11] A. Fujii, K. Itou, and T. Ishikawa. LODEM: A system foron-demand video lectures. Speech Communication, Vol. 48,
No. 5, pp. 516–531, 2006.
[12] N. Ozawa, H. Takebe, Y. Katsuyama, S. Naoi, and H. Yokota.
Slide Identiﬁcation for Lecture Movies by matching Characters and Images. In Document Recognition and Retrieval XI,
Vol. 5296-10 of Proc. of SPIE, pp. 74–81, Jan 2004.
[13] Y. Katsuyama, N. Ozawa, J. Sun, H. Takebe, T. Kobayashi,
H. Yokota, and S. Naoi. A New Solution for Extracting Laser
Pointer Information from Lecture Videos. In Proc. of Elearn2004, pp. 2713–2718, 10 2004.
[14] E. M. Voorhees and D. M. Tice. The trec-8 question answering track evaluation. In Proc. of TREC-8, pp. 83–105, 1999.