Automatic Video Indexing
Based on Shot Classi cation
Ichiro Ide, Koji Yamamoto? , and Hidehiko Tanaka
Graduate School of Electrical Engineering, The University of Tokyo
7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8656 JAPANTEL: +81-3-3812-2111 ext.7413, FAX: +81-3-5800-6922
f ide j kyama j tanaka g @mtl.t.u-tokyo.ac.jp

Abstract. Automatic indexing to video data is in strong demand tocope with the increasing amount. We propose an automatic indexing
method for television news video, which indexes to shots considering
the correspondence of image contents and semantic attributes of keywords. This is realized by rst, (1) classifying shots by graphical feature,and (2) analyzing semantic attributes of accompanying captions. Next,
keywords are selectively indexed to shots according to appropriate correspondence of typical shot classes and semantic attributes of keywords.
The method was applied to 75 minutes of actual news video, and resulted in indexing successfully to approximately 50% of the typical shots(60% of the shots were classi ed as typical), and 80% of the typical shots
where captions existed.

1 Introduction
As the amount of broadcast video data increases, it is becoming more and more
important to store them in a well organized manner considering recycling andsearching. Above all, television news programs are worthwhile indexing considering the importance and usefulness. Currently this process is mostly done
manually, but automatic indexing is in big demand to cope with the increasing
amount and to achieve sucient precision for detailed searching.
We are trying to accomplish this task by referring to both video data andaccompanying natural language data in Japanese television news video. There
are several notable attempts to automatically index television news video fromthis approach. Most of their indexing strategies are based on frequency or justsimple occurrence of words or phrases. On the other hand, others search forwords in a full text searching manner. These methods are relatively simple, andin that sense quite practical approaches, but the critical point is that they do
not necessarily ensure the correspondence of the image contents and the index.
Re ecting these background issues, in this paper we will propose and evaluate an indexing method, which indexes keywords with appropriate semantic
attributes to classes of shots with graphically typical feature. The base of this? Currently at Research & Development Center, Toshiba Corporation.

method lies in the characteristics of television news programs; graphically similar (in a certain perspective) shots contain semantically similar contents. Since
keywords are tagged selectively according to the contents of each typical shotclass, the correspondence of image contents and keywords is guaranteed to acertain extent.
We will rst take an overview of the characteristics speci c to television
news video and related works in the next Sect., and then introduce the proposed
method in Sect. 3. The succeeding Sects. 4, 5 and 6 discuss in-depth matters ofthe method, and Sect. 7 concludes the paper.

2 Indexing to Television NewsFirst, we will overview the characteristics speci c to television news video, andnext introduce several related works by other people.

2.1 Structure of Television News Video
News videos have both graphical and semantic structures as described here.

Graphical Structure. The graphical structure that television news videos haveare not speci c to the genre. As shown in Fig. 1, they are generally hierarchicallystructured.
video
shot

scene
frame

...

...
cut

Graphical structure of video. Frame: still images that constitute amovie, Shot: group of graphically continuous frames, Scene: group of shots withgraphically and/or semantically similar contents, Cut: discontinuous point between
adjoining shots.
Fig. 1.

Semantic Structure. As shown in Fig. 2, the semantic structure of television
news video is quite unique.
It is very important to detect the boundaries of topics and to grasp thesemantic structure of the video before indexing, so this structure could be used
as an opportune key. As mentioned in Sect. 1, since news video tend to be taken
in similar situations, most shots could be classi ed to several typical shot classes
referring to graphical feature.

Report1

Topic
Report2

Next Topic...
Anchor Shot

Semantic structure of news video; Each block represents a shot. Basically, each
topic begins with an `anchor shot', and between them are miscellaneous shots including`report shots'.

Fig. 2.

2.2 Natural Language Data in Video
Variety of Natural Language Data Source. There are various natural lan-

guage data sources accompanying the video; main audio, sub audio, closed caption (mostly same contents as the main audio) and caption. Particularly, captions
are usually used to describe important matters in a digestive form, so they could
be considered as adequate keyword candidates for indexing. According to our
statistics, they appear approximately once every 15 seconds in news programs,
which is a moderate frequency for nding keyword candidates. Since main audio (or closed caption) require complicated process to be used as a keyword
extraction source, captions that have these characteristics are employed in the
proposed system.

Characteristics of Captions. Captions have speci c characteristics that dif-

fer from normal texts, which makes the analysis employing conventional natural
language processing methods dicult. This problem is solved to a certain extent
in this application by the method described in Sect. 5.
On the other hand, semantic characteristics of captions could be classi ed asshown in Tab. 1. Among these types, (a) and (b), which consist about half of thecaptions, represent the contents in the image directly. This allows (a) and (b) toalmost directly become keyword candidates. Although (c) does not necessarily
re ect the graphical contents of the video, it is an important information which
explains the topic. Thus, nearly 60% of the captions; (a), (b) and (c), could bedirectly used as keywords.

2.3 Related Works
As a general video database creating and browsing system, Informedia project[8, 10, 13, 21] at CMU is the most signi cant work in this eld. They have createdTable 1. Types of television news captions. (f) signi es broadcast technical captions
such as `Live'.
Types(a) Locational/Organizational(b) Personal(c) Title(d) Speech Summary/Translation(e) Temporal(f) Technical(g) Descriptive(h) Etc.

Ratio

30%
15%
14%
10%
7%
3%
2%
19%

an automatic archiving and presenting system for CNN (Cable News Network)news video. It automatically recognizes main audio speech and extract keywords
from the text deduced from it by evaluating the rarity of words by the TF-IDF(Term Frequency Inverse Document Frequency) method. Although such statistic
approaches are relatively simple, and in that sense quite practical, they do notnecessarily ensure the correspondence of image contents and keywords, which isessential for video database.
On the other hand, Nakamura and Kanade [5] have proposed an automatic
indexing method that classi es shots into several typical classes, and tag key
sentences derived from syntactic and semantic analyses of the closed caption.
Although the basic idea of classifying shots into typical classes is similar to ourapproach, the point that they utilize closed captions and that they execute key
sentence extraction, di erentiates the two methods.
Similar to this approach, Satoh et al. [6] proposed an automatic facial image
and personal name associating method that associates facial regions extracted
from the image and personal names derived from closed captions analysis. Although this is completely automated and performs fairly well, it concentrates on
associating personal faces and names, which is an acceptable limitation, but notsucient for news video database.

3 Indexing Based on Shot Classi cation
Considering the issues discussed in Sect. 2, we will propose an automatic indexing
method based on shot classi cation. The basic idea of the method is based on
the characteristic speci c to television news video; graphically similar images
contain similar contents. Based on this assumption, graphically typical shots
will be indexed keywords with certain attributes.
The overall indexing scheme is shown in Fig. 3. A simple overview of eachphase is introduced in this Sect. In-depth description and evaluation on shotclassi cation, caption analysis, and indexing are discussed in the succeeding
Sects.

News Video
Video

Audio

Closed Caption(Motion Image)(Speech)(Digitized Text)

Video Capturing

Speech Recognition

FramesText

Cut Detection

Key Phrase Extraction
Image Characteristics
Extraction

Caption Detection
Character Region

Image Characteristics
Color
Information
Hue,
Saturation,
Intensity,...

Character Recognition

FacialRegion... Location,
Size,
Number,...

Shot Classification

Key Phrase
Morphological Analysis
Words&
Part of Speeches
Semantic Attribute
Analysis

Integration of Results

Semantic Attributes

Classified Shots
Personal
SpeechReportWalking

AnchorGatheringCG...

Indexing Based on
Shot Classification
Indexed Shots
SpeechReportAnchor

Personal

Title

Walking

Gathering

Personal

Locational
Organizational

Locational
Organizational

Natural Language Processing

Image Processing

Shots

Title
...

...

Overall indexing scheme. The left half shows the ow of the image processing
phase, the right half shows the natural language processing phase, and the bottom
shows the integration phase. Audio and Closed Caption are not currently used.

Fig. 3.

Although we do not currently employ main/sub audio and closed caption
data, the scheme could be easily extended to handle them as shown in the gure.
This may also seem as a mere combination of conventional algorithms, but theoverall ability of indexing by integrating video and natural language data is far
superior than a simple combination.

3.1 Image Processing Phase
Video Capturing. First, analog video is digitized by an image capture board

connected to a PC at a sampling rate of 5 frames per second. We consider thisrate is sucient for indexing, although the original frame rate of the NTSC
standard is approximately 30 frames per second.

Cut Detection. Next, cuts are detected, since shots are the most primary unit

to handle video data. Cut detection has been challenged in two approaches;
detecting from non-compressed video, and from compressed video. The former
approach detects cuts by analyzing the graphical similarity of adjoining frames[12, 17]. The latter takes advantage of the compression algorithm. Especially
MPEG video is t for the purpose due to its compression algorithm based ongraphical correlation between frames [3].
Among these methods, we chose the Nagasaka-Tanaka (segmented 2 examination) algorithm [17], which is very simple but quite e ective. The NagasakaTanaka method evaluates the similarity of color histograms of corresponding
equally segmented blocks of adjoining frames, applying the 2 examination function. The function is de ned as follows: 2 (i) =X (H (c) 0 H +1(c))2

cmax
c=0

n
i

n
i

(1)

Hin (c)where Hin (c) and Hin+1 (c) represent the color histogram of the ith block of two
adjoining frames n and n + 1 respectively. When more than half of the values of2 (i) exceeds the threshold, a cut is detected in between the frames.

100%

Precision

80%
60%
40%
20%
0%
0%

20%

40%
60%
Recall80%

100%

Recall and precision of cut detection by Nagasaka-Tanaka (segmented 2 examination) method applied to 30 minutes of video with 210 cuts. If 60% of recall could
be considered sucient, more than 80% of precision is ensured.
Fig. 4.

Figure 4 shows the relation between recall and precision of cut detection,
when the method was applied to 30 minutes of actual news video with 210 cuts.
After these pre-processes, extraction of graphical feature is performed before
the shot classi cation.

Caption Recognition. As a tributary to the main-stream image processing,
caption detection and character recognition should be performed. This is not currently implemented, and captions are written down manually. Although OCR(Optical Character Recognition) softwares with high recognition rates do exist,
the resolution of TV captions are limited due to their size and the number ofscanning lines (525 lines per frame in the case of NTSC broadcasting standard).
Fragments of background image ltering through the characters should be eliminated before character recognition, which is also a dicult process. These restrict
the application of conventional OCR techniques to television caption character
recognition, especially to complicated Japanese characters. Nonetheless, several
attempts are made to accomplish the task [9, 14], although their recognition rates
have still room for improvement. We may hope for digitized caption texts to bebroadcast along with the video, following the future digitalization of television
broadcasting.
3.2 Natural Language Processing Phase
First, morphological analysis to digitized texts derived from captions are performed using the Japanese morphological analysis system JUMAN [20]. This isa pre-process for analyzing the semantic attribute of the entire caption, whichrefers to suxable nouns.

3.3 Integration Phase
After image classi cation and caption analysis of the shot are done, the integration phase indexes shots with captions with appropriate semantic attributes
for the typical shot class; i.e. when the shot is a `speech shot', the speaker's
name is an appropriate keyword. When it is a `gathering shot', the name of thegathering, say a conference, is considered appropriate.
Such indexing scheme ensures the correspondence of image content and keyword, which is essential for video database.

4 Shot Classi cation
After the pre-process, each shot is classi ed based on its graphical feature. Notethat the classi cation rules are based on combinations of relatively simple graphical feature extraction process, which makes the method applicable to large
amount of incoming video data.
We have de ned ve shot classes:{ Speech/Report{ Anchor{ Walking{ Gathering{ Computer Graphics (CG)
These classes covered 57% of the entire news video that we used for experiment.
Details on each shot class are described in this Sect.

4.1 Speech/Report Shot
When a person is addressing a speech, or a reporter is reporting from a relay
spot, there is usually one person speaking in the middle of a frame. In order todetect such a shot, (1) human face, and (2) lip movement should be detected.
Condition (2) is employed to avoid detecting a portrait picture, or a video witha person just standing in the middle of a frame without speaking.
An anchor shot is also detected from these conditions, but they will be separated later.

Face Detection. Face detection is a very popular research eld that has developed various algorithms, but the following method is considered sucient andsimple enough to serve our purpose.

Modified Saturation (S m )

100
80
60
40
20
0
0

60

120

180
Hue (H )240

300

Fig. 5.

Skin colored regions on the H - Sm plane.

360

1. Skin colored region extraction
The modi ed HSI color system is used to detect skin colored regions [16].
I (Intensity) is used only for excluding dark regions. A certain rectangularregion in the H (Hue) - Sm (Modi ed Saturation) plane was de ned as skincolor. The distribution of sample skin colored regions on the H - Sm plane isshown in Fig. 5. Pixels whose H and Sm exist in this region are determinedas skin colored. Small regions that consist of less than a certain number ofpixels are deleted, and adjoining isolated regions are merged by spreadingout their boundaries for a few pixels.
2. Template matching
Template matching is performed to exclude hands, walls, desks and so onthat were extracted as skin colored regions. Average faces in several di erent resolutions are prepared from the I of various facial regions, and are
selectively used for matching, according to the size of the extracted skin colored region. In order to decrease the in uence of optical states, the I of theextracted region is regularized by the overall I of the frame.

Lip Movement Detection. Once a facial region is detected, it is easy to estimate the mouth location. This is because faces in speech/report shots andanchor shots are usually full faces. If the temporal change of the area aroundthe estimated mouth location is relatively bigger than the change of the entire
facial region during a shot, lip movement is detected.
4.2 Anchor Shot
Separation of Anchor Shots Among Speech/Report Shots. Anchor shots
initially satisfy the classi cation conditions for the speech/report shot. One distinctive feature of anchor shots is that they are graphically extremely similar
among themselves, and also frequent compared to other speech/report shots.
Therefore, after speech/report shot classi cation, anchor shots are separated byclustering the speech/report shots by evaluating the mutual similarity between
all the classi ed shots. The largest and most dense cluster would be the anchor
shots. The similarity is evaluated by the comparison of color histograms applying the segmented 2 examination previously used for cut detection. Each shotis regularized so that the facial region should be located in the same position.

Detection of Topic Boundary. The main purpose for detecting anchor shots
is to detect boundaries of news topics. However, since anchor shots may appear
in the middle of a long topic, just separating them from speech/report shots is
not sucient to ful ll this purpose. A distinctive feature of anchor shots in the
beginning of a new topic is the presence of a title caption. As shown in Fig. 6,
this could be detected by observing the transition of the overall edge intensityof frames, caused by the superimposion of the title caption CG. Therefore, anchor shots with prominent edge intensity transitions are used to detect topic
boundaries.
4.3 Walking Shot
When a person is walking, the upper half of the body oscillates up and down
following the steps. A television camera is usually stabilized on a tripod anddoes not oscillate along the vertical axis. Facial region detection is performed
likewise the speech/report shot to the yet unclassi ed shots. As shown in Fig. 7, awalking shot is classi ed by detecting the up and down oscillation of the bottomof a facial region.

Relative Transition of Edge Intensity

1.3

1.2

Title Appearance

1.1

1.0

0.9
Frame
Fig. 6.

Relative transition of edge intensity following the appearance of title caption.

y Coordinates of the Head

175
170
165
160
155
150
145
140
Fig. 7.

Frame

Oscillation of facial region following the person's step.

4.4 Gathering Shot
When many people are gathering, there are usually more than two similar sizedpeople in a frame. Facial region detection is performed likewise the speech/report
shot to the yet unclassi ed shots. A gathering shot is classi ed by detecting morethan two similar sized facial regions in a frame.

4.5 Computer Graphics (CG) Shot
CG shots are quite tricky, since they may contain a referential portrait picture,or have explanatory gures and texts, which may cause mal-e ects to shot clas-

si cation and caption analysis. Therefore, CG shots need to be detected andexcluded from the indexing scheme. A distinctive feature of CG shots is thatthey are usually motionless. Nevertheless, there are occasional movements, forexample in an explanatory owchart. So, CG shots are classi ed by the total duration of motionless frames, not by the overall motionlessness. In the following
experiment, the duration was set to one second.

4.6 Classi cation Experiment
Table 2 shows the result of the shot classi cation applied to 75 minutes of news
video. The numbers of true answers (= NClassified + NUnclassif ied ) were determined and counted manually for evaluation. Recall and precision are de ned asfollows:

NClassif ied
NClassif ied + NUnclassif ied
NClassif ied
P recision =NClassif ied + NMisclassif ied
Recall =(2)(3)

Result of shot classi cation. Most shots were classi ed well in terms of precision, but recall for some classes were relatively low. This is prominent in `gathering'shots, since facial regions were not always exposed in the video.
Table 2.

Shot Class

Speech/Report
Anchor (Overall)Anchor (Topic Head)Walking
GatheringComputer Graphics

NClassified NMisclassified NUnclassified
55
53
37
16
70
14

3
0
0
0
13
1

12
0
6
15
62
0

Recall

83%
100%
86%
52%
53%
100%

Precision

95%
100%
100%
100%
84%
93%

Major reasons for misclassi cation and unclassi cation were:{ Lip movement could not be detected in a speech/report shot, since the face
was not a full face.
{ Topic boundaries were not detected for some very short topics, which did
not have title captions in the beginning.
{ Facial regions were not correctly detected in a gathering shot, since the faces
were too small and/or hidden by hair. This is often the case with indoor
meetings, when a camera shoots from the rear.
{ Up and down oscillation of a head could not be detected in a walking shot,since the person was walking too far away from the camera.
{ A completely still image of an object was misclassi ed as a CG shot.

5 Caption Analysis
Caption analysis is necessary to index typical shots with appropriate keywords
re ecting their typical contents. As shown in Tab. 1, captions that have (1) personaland (2) locational/organizational attributes are adequate keyword candidates.
They are mostly noun phrases (often simple arrays of nouns). In Japanese, theutmost tail nouns, i.e. suxable nouns, de ne their attributes2 .
As related research on semantic disambiguation of nouns, several methods do
exist. Nasukawa [7] proposed a method that determines semantic attributes ofproper nouns (i.e. whether a proper noun indicates a place or a person) referringto the context of neighboring sentences. On the other hand, Watanabe et al. [11]proposed a method that analyzes television news captions by referring to both
locations and grammatical characteristics as keys.
Although these methods perform fairly well, the former method is dicult
to serve our purpose since captions do not have enough neighboring information
to analyze contexts, and also since it is purposed to handle only proper nouns.
The latter method is originally designated to serve similar purpose to ours, but is
not generally applicable to various news programs, which have di erent designingpolicies where layouts of captions vary.
Similarly to our task, the Named Entity task de ned for the Message Understanding Conference (MUC) [15] assigns the participants to classify personal,organizational, locational, temporal and numerical phrases. The di erence is that
our aim is not limited to proper nouns, where the Named Entity task limits thetagging to personal, organizational and locational phrases to proper names.
Considering these issues, we decided to analyze captions on their own byreferring to suxable nouns.

5.1 Collecting Suxable Nouns
To enable caption analysis based on suxable nouns, rst such suxable nouns,
i.e. (1) personal nouns, and (2) locational/organizational nouns, should be collected. These were collected according to certain conditions from two text corpora that consist of newspaper articles [18, 19]. These were manually morphological analyzed beforehand, which ensures the basic reliability of the collection
process.
Details are discussed elsewhere [1, 4] since they meddle with language speci cissues, but as a result, 3,793 nouns were collected as personal nouns, and 11,166
as locational/organizational nouns. Note that the collected suxable nouns include those that represent people or locations/organizations alone, such as `volunteer' and `kitchen', but does not include proper names.
2This is presumably common in some other East Asian languages, but di erent inmajor European languages. For example, personal noun `President' as in `President
Clinton' comes to the head, and locational/organizational noun `City' as in `New
York City' comes to the tail.

5.2 Semantic Analysis Experiment
Table 3 shows the result of the caption analysis applied to the same 75 minutesof news video used for the shot classi cation experiment. The numbers of true
answers (= NClassif ied + NUnclassified ) were determined and counted manually
by a third person, and recall and precision are de ned as noted in formulae (2)and (3), respectively.

Table 3. Result of caption analysis. The result shows fairly well performance for analyzing `personal' captions, but analysis for `locational/organizational' captions needs
further more improvement for practical use.

Caption Attribute

Personal
Locational/
Organizational

NClassified NMisclassified NUnclassified
83

23

4

139

121

23

Recall95%

Precision

86%

78%
54%

Major reasons for misclassi cation and unclassi cation were:{ Some nouns were essentially applicable to both categories (Semantic diversity).

{ Some nouns in the collected dictionary were inappropriate (Noise).
{ The collected nouns were insucient (Lack of vocabulary).
The latter two could be solved by further improvement of the collection scheme,
but the semantic diversity is an essential issue when dealing with semantics ofwords.
The reason for the low rates of locational/organizational captions is due tothe loose conditions of the collection rule. It is dicult to tighten the rule without
more precise grammatical information tagged beforehand in the corpora.

6 Indexing to Classi ed Shots
Following shot classi cation and caption analysis, appropriate keywords aretagged to each classi ed shots according to their classes.

6.1 Indexing Scheme
Appropriate semantic attributes of keywords are de ned to each typical shotclass as shown in Tab. 4. Anchor shots with captions are referred to detect boundaries of topics for this process. On the other hand, CG shots are excluded fromindexing, considering their uniqueness.

Shot class and semantic attributes of indexes. Each shot class (except `CG
shots' and intermediate `anchor shots') are indexed with captions with corresponding
semantic attributes. Titles are extracted from `anchor shots' referring to the special
designs around them.
Table 4.

Shot Class

Semantic Attribute of Index

Speech/Reportpersonal
Anchor (Topic Head) title
Walking
personal
Gathering
locational/organizational

Following these principles, indexing is performed to all the classi ed shots
except intermediate anchor shots and CG shots according to the following procedure:
1. Search for a caption with an appropriate attribute inside the shot, and index
it if found.
2. If not found, search for it in graphically similar shots inside the topic, andindex it.

6.2 Indexing Experiment
Table 5 shows the result of indexing applied to the same 75 minutes of news
video used in previous experiments. In order to evaluate \indexing based onshot classi cation" independently, true answers of semantic attributes of captions
given by a third person were used.

Result of indexing. The evaluation was done from two points of view: `/All'
indicates the ratio to the number of all classi ed shots, and `/Indexable' the ratio tothe number of shots that are manually indexable. This was done since some topics
lacked of caption information necessary for indexing. They can not possibly be indexedin any way by the current method without external knowledge, so they were excluded
to evaluate the method independently. `Walking shot' had too small number of samples
to discuss the result.
Table 5.

Shot Class

Speech/Report
Anchor (Topic Head)Walking
Gathering

All

38
15
4
59

Indexable Indexed
30
25
15
14
1
1
44
33

/All

66%
93%
25%
56%

/Indexable

83%
93%
100%
75%

The result does not necessarily show practical performance as a whole (/All),but as an evaluation of the proposed method itself (/Indexable), all classes
showed more than 75% of recognition rate. The overall performance should improve by employing other natural language source as shown in Fig. 3.

7 Conclusion
In this paper, we have proposed and evaluated an indexing method that indexes
television news video, considering the correspondence of image contents andkeywords. The overall result is not necessarily practical, but the performance ofthe method itself is quite promising. Although the techniques applied for shotclassi cation and pre-process for caption analysis were conventional, the e ectiveness of integrating image and language information was also shown throughthe indexing.
The problem is that the numbers of typical shot classes and caption attributes are relatively small, since the classi cation rules were given in a topdown manner. We are currently examining an automatic classi cation rule acquisition method based on statistic relations of graphical feature and semantic
attributes of captions [2]. The result of a preliminary experiment showed promising performance, although the amount of data was not large enough to discuss
statistics. We will apply the acquired classi cation rules to enable better automatic indexing based on shot classi cation in the near future.

Acknowledgments
The Japanese Morphological Analysis System JUMAN is a free software developed at and distributed by Nagao Lab., Kyoto University and Matsumoto Lab.,
Nara Institute of Science and Technology. The Kyoto University Text Corpus isa product of Nagao Lab., Kyoto University. RWC text database is a product ofReal World Computing Partnership (RWCP), and is used by the authors undera licensed agreement.

References
1. Ide, I. and Tanaka, H.; \Automatic Semantic Analysis of Television News Captions"; Proc. 3rd Intl. Workshop on Information Retrieval with Asian Languages, Oct1998 (to appear).
2. Ide, I., Hamada, R., Tanaka, H. and Sakai, S.; \News Video Classi cation basedon Semantic Attributes of Captions"; Proc. 6th ACM Intl. Multimedia Conf. -Art
Demos-Techinical Demos-Poster Papers-, pp.60-61, Sep 1998.
3. Kaneko, T., and Hori, O.; \Cut Detection Technique from MPEG CompressedVideo Using Likelihood Ratio Test"; Proc. 14th Intl. Conf. on Pattern Recognition,
Aug 1998.
4. Ide, I. and Tanaka, H.; \Semantic Analysis of Television News Captions by Suxes"; Trans. IPS Japan, Vol.39, No.8, pp.2543-2546, Aug 1998 (in Japanese).

5. Nakamura, Y. and Kanade, T.; \Semantic Analysis for Video Contents Extraction{Spotting by Association in News Video{"; Proc. 5th ACM Intl. Multimedia Conf.,
pp.393-402, Nov 1997.
6. Satoh, S., Nakamura, Y. and Kanade, T.; \Name-It: Naming and Detecting Faces
in Video by the Integration of Image and Natural Language Processing"; Proc.
IJCAI'97, pp.1488-1493, Aug 1997.
7. Nasukawa, T.; \Keyword Categorization based on Discourse Information"; Proc.
11th Annual Conf. JSAI, pp.348-349, Jun 1997 (in Japanese).
8. Hauptmann, A. G. and Witbrock, M. J.; \Informedia News-on-Demand: Using
Speech Recognition to Create a Digital Video Library"; Proc. AAAI'97 Spring
Symp. on Intelligent Integration and Use of Text, Image, Video and Audio Corpora,
pp.120-126, Mar 1997.
9. Kurakake, S., Kuwano, H. and Odaka, K.; \Recognition and Visual Feature Matching of Text Region in Video for Conceptual Indexing"; SPIE Proc. of Storage andRetrieval for Image and Video Database V, Vol.3022, Feb 1997.
10. Smith, M. A. and Kanade, T.; \Video Skimming and Characterization throughthe Combination of Image and Language Understanding Techniques"; CMU Tech.
Rep. CMU-CS-97-111, Feb 1997.
11. Watanabe, Y., Okada, Y. and Nagao, M.; \Semantic Analysis of Telops in TV
Newscasts"; Tech. Rep. IPS Japan 96-NL-116, Vol.96, No.89, pp.107-114, Nov 1996
(in Japanese).
12. Ariki, Y. and Saito, Y.; \Extraction of TV News Articles Based on Scene Cut
Detection Using DCT Clustering"; Proc. 1996 Intl. Conf. on Image Processing,pp.847-850, Sep 1996.
13. Wactlar, H. D., Kanade, T., Smith, M. A. and Stevens, S. M.; \Intelligent Accessto Digital Video: Informedia Project"; IEEE Computer, Vol.29, No.3, pp.46-52,
May 1996.
14. Motegi, Y. and Ariki, Y.; \Indexing to TV News Articles Based on Character
Recognition"; Tech. Report IEICE, PRU-95-240, Vol.95, No.584, pp.33-40, Mar
1996 (in Japanese).
15. United States Defense Advanced Research Projects Agency (DARPA), Information
Technology Oce; \Named Entity Task De nition, Version 2.1"; Proc. 6th Message
Understanding Conference, pp.317-332, Nov 1995.
16. Matsuhashi, S., Nakamura, O. and Minami, T.; \Human-Face Extraction Using
Modi ed HSV Color System and Personal Identi cation Through Facial Image
Based on Isodensity Maps"; Canadian Conf. on Elec. and Comp. Eng. '95, Vol.2,
pp.909-912, Sep 1995.
17. Nagasaka, A. and Tanaka, Y.; \Automatic Video Indexing and Full-Video Search
for Object Appearances"; IFIP Trans., Vol.A, No.7, 1992.
18. Kurohsashi, S., Saito, Y. and Nagao, M.; \Kyoto University Corpus version 2.0";
Jun 1998.
Available from http://www-lab25.kuee.kyoto-u.ac.jp/nl-resource/corpus.html
19. Real World Computing Partnership (RWCP); \RWC Text Database"; Mar 1996.
20. Kurohsashi, S. and Nagao, M.; \Japanese Morphological Analysis System JUMAN
version 3.5"; Mar 1998.
Available from http://www-lab25.kuee.kyoto-u.ac.jp/nl-resource/juman-e.html
21. \The Informedia Project";
http://www.informedia.cs.cmu.edu/.
This article was processed using the LaTEX macro package with LLNCS style