Systems and Computers in Japan, Vol. 34, No. 12, 2003
Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J85-D-II, No. 7, July 2002, pp. 1201–1210

Compilation of Dictionaries for Semantic Attribute Analysis of
Television News Captions
Ichiro Ide,1 Reiko Hamada,2 Shuichi Sakai,3 and Hidehiko Tanaka3
1

National Institute of Informatics, Tokyo, 101-8430 Japan

2

Graduate School of Engineering, The University of Tokyo, Tokyo, 113-8656 Japan

3

Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, 113-0033 Japanin Wiley InterScience (www.interscience.wiley.com). DOI
10.1002/scj.10417

SUMMARY
With the increase in the amount of video that is
broadcast daily, there is an increasing need for storage ofvideo in a systematic way for future reuse and retrieval. In
particular, from the viewpoint of importance and usability,
it is desirable to index news videos. For adequate automatic
indexing based on the text information in the video, it is not
sufficient to apply the simple index extraction and annotation methods which have been widely used in conventional
methods. It is important to select index candidates withreference to semantic attributes. The purpose of this study
is to compile dictionaries which are needed for analyzing
the semantic attributes of captions (noun phrases) in TV
news videos. We describe the process by which words are
extracted from text corpora and a thesaurus for storage on
the basis of specified conditions. The quality of the dictionaries is examined by analysis of the semantic attributes of
the words appearing in actual news videos, and the results
are presented. In evaluation experiments in which an existing proper noun dictionary and temporal noun dictionary
were combined and used, a recall of 79 to 93% and a
precision of 41 to 71% were obtained. Although the precision is low in this result, it is concluded that the compiled
dictionaries are of practical use for indexing since the recall
is more important in that case. © 2003 Wiley Periodicals,Inc. Syst Comp Jpn, 34(12): 32–44, 2003; Published online

Key words: indexing; caption; semantic attribute;
suffix noun; dictionary.

1. Introduction
With the increasing amount of broadcast video, there
is an increasing need to store videos in a systematic way forreuse and retrieval. In particular, from the viewpoint of theimportance of their contents and their usability it is desirable to index news videos. At present, this procedure is
performed manually in an approximate way, but automationof the process is desirable in order to achieve more precise
indexing that meets the requirements for detailed retrieval.
The objective of the authors is to realize automatic
indexing of the news videos by comprehensively utilizing
image information and text information. In order to handle
the requirements for detailed retrieval, the use of simple andapproximate indexing, such as is applied in most of theexisting methods, is not sufficient, and the analysis ofsemantic attributes must be emphasized in the selection ofindex candidates.
As prominent in the News-on-Demand system [17]in the Informedia Project [18] at Carnegie Mellon Univer© 2003 Wiley Periodicals, Inc.
32

sity, there are various efforts to perform automatic indexingof news videos by integrated media processing. Most ofthese, however, are simple indexing methods based on
statistical information such as term frequencies, or are
based on the occurrence timing of words or phrases. Correspondence between the index and the image content is notguaranteed, although the requirement for automatic indexing is satisfied to some extent. These methods can be usedin rough indexing for each topic, but in more detailed
indexing, such as indexing for each shot, the fit between theimage and the index in the corresponding range may not beadequate.
In this context, the authors undertook to construct anautomatic indexing system as shown in Fig. 1, considering
the correspondence between the index and the image content in terms of attributes. In the figure, the index is determined with reference to correspondence between the image
and the text with the same attributes. In the natural language
processor used in this system, the semantic attributes must
be analyzed for the text that is the index candidate. For thispurpose, dictionaries were compiled in this study for use in
the semantic attribute analysis of superimposed captions innews videos. This paper describes the construction process,and evaluates the performance of the dictionary by analysisof actual news captions.
More precisely, the analysis of semantic attributes inthis study consists of the classification of noun phrases interms of the following four semantic attributes: (1) person,(2) location or organization, (3) time, and (4) other. The
evaluation presented in this paper considers the analysis ofnews captions, but the semantic attribute suffix noun dictionaries that were constructed in this study will be applied
to information retrieval such as extraction of proper expressions, to other similar indexing procedures [6, 12, 15], andto general natural language understanding procedures. In
the following, Section 2 describes the method of analysisof semantic attributes. Section 3 describes the compilationof the dictionary. Section 4 presents an evaluation based on
application to actual news captions. Section 5 presents asummary.

Furthermore, although the performance of speech
recognition technology is improving, it has not reached thelevel at which speech can be used as an accurate text
information source. Closed captions have the possibility tomake up for this defect of speech recognition technology.
In fact, there is a widespread use of such captions in Europe
and the United States. However, Japan has just begun toprovide the technique to a limited extent via speech recognition limited to particular speakers (announcers) [1]. It has
not been developed to a stage at which the technique is fully
utilized.
On the other hand, the caption represents important
information in a straightforward way, and the process of
extracting the important words is not needed as it is if otherinformation sources are used. One problem is that when thecaption is superimposed on a background image, it is difficult to apply existing OCR (optical character recognition)techniques, since the resolution of the characters is degraded as a result of the small number of scan lines (525 in
NTSC broadcasts). Consequently, there have been several
studies focusing on the recognition of caption characters [3,9], which have had some success.
For these reasons, this paper considers the direct use
of captions as the text information source for indexing, anddiscusses the compilation of the dictionaries which areneeded for the analysis. Although captions written down
manually were used in this experiment, in the future it isplanned to utilize the results of the above-mentioned studies
and data acquired from data broadcasts.
2.1. Linguistic properties of captions
The caption has special properties, from the viewpoints of both grammar and content, in contrast to othergeneral texts. From the viewpoint of grammar, almost halfof the caption text consists of noun phrases, in the form of
isolated substantives and noun sequences. Captions representing persons, places or organizations, and time which are
to be classified in terms of the attributes are mostly nounphrases. Consequently, the noun phrase is taken as theobject of analysis. From the viewpoint of content, captions
representing person, place/organization, and time accountfor 48.3% of all captions, and there are 0.39 such nounphrases per shot on average.* Thus, a relatively large
amount of information is acquired for indexing.

2. Analysis of News Captions with
Reference to Suffix Nouns
This section describes the properties of news captions
and presents the method of analysis of semantic attributes
using the dictionary. The video contains various kinds oftext information, such as speech and closed caption texts inaddition to the ordinary captions (open captions). These
information sources generally contain various kinds ofinformation, but since their content is redundant, it is necessary to extract important words and phrases.

2.2. Semantic attribute analysis of nounphrases
Considering the linguistic properties described in theprevious section, captions composed of noun phrases con*This is a result obtained when 2842 captions occurring in 370 minutes ofnews videos were analyzed manually.

33mation, it is difficult to determine the semantics of a proper
noun such as “mori” (forest, see later discussions) as an
isolated word. The semantics must be inferred from the casestructure of the sentence and the context of neighboring
sentences, as in the method of Nasukawa [14] and various
other methods of proper expression extraction.
Watanabe and colleagues proposed a method of caption analysis for news videos that takes account of thelocation of the caption and its grammatical features [19].
However, this method has the problem that the designingpolicy, including the position in the image, differs fromprogram to program, which reduces the versatility of themethod.

cerning (1) persons, (2) locations or organizations, and (3)time were analyzed. The reason for not separating places
and organizations, which are usually handled separately inthis kind of analysis, is as follows. When the correspondence to image content is considered in indexing videos, it
often happens that the organizational name indicates aparticular place in the image (“XX bank,” for example,
indicates the image of a bank building, as well as theorganization occupying it), so we do not consider a clear
distinction between places and organizations in this work.
In the following, existing studies related to the analysis of nouns (noun phrases) that have some use for the
purpose of this study are discussed. Then, the semantic
attribute analysis method adapted to the nature of captions
which is used in this study is described.

2.2.2. Analysis based on suffix nouns
2.2.1. Related studies

The above properties of caption and problems ofexisting methods were considered, and a method was
adopted in which semantic attributes are analyzed for thecaption alone, with no possibility of referring to adjacentcontext and case structure.
It is in general very difficult even for a human todecide whether a particular noun is a common noun or aproper noun. It is also difficult to decide, in the case of aproper noun, whether it is a personal name, locational name,or organizational name from that noun alone, without considering the context. In practice, the semantics of stems orof whole noun phrases in Japanese seems to be governedby the tail noun (suffix noun) as shown in Table 1.
In the above example, it is difficult to determine thesemantics from just the word “mori” (forest). However, if
it is connected to “prime minister,” it is seen to be a personalname, and if it is connected to “town,” it is seen to be alocational name. Thus, in Japanese in general, the semantics
of whole noun phrases can be determined by analyzing thesuffix noun. Studies based on such an assumption have been
applied to the recognition of nouns, including proper nouns[7, 13]. In this study such a property is assumed, and thesemantic attributes of news captions composed of nounphrases are analyzed by focusing on the suffix nouns.
Figure 2 shows an example of semantic attributeanalysis of a caption based on this principle. First, morphological analysis is applied to a caption, and the suffix nounis extracted. Then the extracted suffix noun is analyzed bycomparing it to the words in attribute-tagged suffix noun
dictionaries.
In this example and the following discussion thesuffix noun is emphasized, but the proposed method essentially considers and analyzes not only noun phrases including proper nouns, but also the nouns, such as “actor” and“kitchen,” which by themselves indicate a person or a place.

An example of a similar semantic attribute analysis
problem is the named entity task [16, 20], which is defined
as a task in the Message Understanding Conference (MUC)
for English, and the Information Retrieval and Extraction
Exercise (IREX) for Japanese. In this problem, a task isposed in which the participant is asked to extract representations concerning persons, places/organizations, times,
numbers, and so on (the first three are limited to proper
nouns) from presented text, and to annotate them.
In contrast to these approaches, there is an approachto analyze the semantics of a proper noun by syntax analysis
of general text [14]. These problems and methods are close
to the technique required in this study, in the sense that theexpressions concerning persons and places/organizations
are extracted and classified. They differ, however, in thatthe object of analysis is limited to proper expressions,especially proper nouns. An analysis for only proper nouns
and an analysis that includes general noun phases differ
fundamentally in their difficulties. Without auxiliary infor-

Fig. 1. Indexing considering correspondences between
indices and image contents.

34

As already discussed, there are nouns such as “actor,”“kitchen,” and “today,” which by themselves represent
persons, locations/organizations, and time, in addition topure suffix nouns. These nouns are also acquired. Proper
nouns are excluded from the acquisition process, since it isdifficult to determine their semantics from the words inisolation.
In the compilation of the dictionary, two text corpora
were used. These text corpora are tagged with the results ofmanual morphological analysis. The segmentation andclassification of morphemes were assumed to be correct,
and were utilized in the study:• RWC-DB-TEXT-95-2 in the RWC Text Database(2nd version) [2]. This consists of 27,418 sentences from 1994 issues of the Mainichi Shimbun
newspaper.
• The Kyoto University Text Corpus (2nd version)[11]. This consists of 19,956 sentences from 1995
issues of the Mainichi Shimbun newspaper.
These two corpora were combined, to comprise
47,374 sentences, and the suffix nouns were extracted undercertain conditions. Below, the compilation procedure foreach semantic attribute-tagged suffix noun dictionary is
described, together with the results of acquisition.
3.1. Acquisition of personal nouns
Fig. 2. Example of caption analysis referring to suffixes.
This section considers the personal nouns, that is,common nouns with a suffix noun indicating a person, suchas “XX hakase (Dr. XX),” as well as common nouns thatthemselves indicate a person, such as “actor.” The acquisition procedure and the results are described.

3. Compilation of Semantic
Attribute-Tagged Suffix Noun Dictionaries

3.1.1. Extraction from the corpus
Personal nouns were extracted from the corpus according to the criteria shown in Table 2. We took advantageof the property that the suffix auxiliaries “(ra)” and “(tachi)”are usually applied in Japanese to represent (exclusively)
multiple persons. The examples of extraction by this criterion are shown in the bottom row of Table 2. Table 3 lists
some of the extracted words in decreasing order of frequency.

This section discusses the construction of the semantic attribute-tagged suffix noun dictionaries which areneeded in performing semantic attribute analysis of new
captions focusing on the suffix noun, as described in theprevious section. More precisely, separate dictionaries areconstructed for suffix nouns representing (1) persons, (2)locations/organizations, and (3) time.

Table 1. Example of determination of stem’s meaning by the suffix
Mori
Mori + “shusho”(prime minister)Mori + “machi” (town)

...
...

Mori
Mori

...

Mori

[noun–?][noun–proper noun–personal name] + “shusho” (prime [noun–common noun]
minister)[noun–proper noun–locational name] + “machi” (town) [noun–common noun]

35

Table 2. Personal noun extraction rules: Extract the underlined word from a sequence that matches the following patternsof both words and morphemes. “*” denotes that any word or morpheme is applicable. Patterns for both RWC corpus andKyoto University corpus are shown, since they have different morphological systems. The extraction example shows theresult from automatic morphological analysis performed by the Japanese morphological analysis system JUMAN.
Word*

RWC morpheme[noun–*]

Kyoto Univ.
morpheme
Example*...

[“ra” (plural suffix)][“tachi” (plural
suffix)][noun][noun–nonautonomous–*][noun–suffix][noun–common noun]...
[suffix–noun type suffix
noun]“kyoju” (professor) [noun–
Ichiro[noun–personal name] common noun]...

[noun–*][suffix–noun typesuffix noun]
Tanaka [noun–
personal name][noun–suffix][suffix–noun typesuffix noun]“ra” (plural suffix)[suffix–noun typesuffix noun]

As the thesaurus we used the “lexical class table inthe classified lexical table format (revised edition)” [8](below called the “new classified lexical table”). It is aconcept class dictionary in which 87,743 words (70,858
distinct words) are assigned to 4 main classes, 842 subclasses, and 10,334 class items. The class item, which is theminimum unit of classification, contains 6.8 words on
average. Table 4 shows the change of vocabulary size in thecourse of dictionary compilation. A total of 1795 personalnouns were acquired in the final stage.

3.1.2. Expansion of vocabulary by use of athesaurus
By the criteria shown in Table 2, 136 suffix nouns
were extracted from the corpus. However, the vocabulary
was still insufficient for use in actual news caption analysis.
Consequently, the vocabulary was expanded by using athesaurus. In this expansion, a word belonging to the sameclass as the extracted word was always judged to belong to
the semantic class in question.

Table 3. List of extracted personal nouns (Top 30)
Order

Extracted word

Occurrences

Order

1
2
3
4
5
6
7
7
9
10
11
12
13
14
14

“sha” (person)“shi” (Mr./Ms.)“san” (Mr./Ms.)“kaicho” (president)“cho” (head)“giin” (representative of House)“nin” (person)“kyoju” (professor)“ka” (specialist)“kanbu” (key person)“shusho” (prime minister)“hikoku” (accused)“gyosha” (contractor)“chokan” (director)“sho” (minister)

75
57
51
32
26
21
16
16
15
14
13
10
9
8
8

14
14
14
19
20
20
20
20
20
20
20
27
27
27
27

36

Extracted word
“shokicho” (secretary general)“kyaku” (guest)“in” (member)“daihyo” (representative)“yogisha” (suspected)“shokuin” (clerk)“jokyoju” (associate professor)“kyokucho” (head of department)“kankeisha” (related person)“kan” (clerk)“iincho” (chairman)“taishi” (ambassador)“shacho” (president of company)“kanji” (secretary of party)“kacho” (head of section)

Occurrences
8
8
8
7
6
6
6
6
6
6
6
5
5
5
5

or organization, such as “kitchen,” were acquired. Thissection describes the acquisition procedure and the results.

Table 4. Number of collected personal nouns: Numbersof classes and subclasses indicate those that extracted
words listed in the thesaurus belong to

3.2.1. Extraction from corpus

Expansion
by New
Classified
Extraction
Lexicalfrom corpus
Table
Word in New Classified
Lexical Table(number of classes)(number of subclasses)
Words outside of New
Classified Lexical Table
Total thesaurus

117

1,776

125
59
19

125
59
19

136

1,795

Locational/organizational nouns were extracted fromthe corpus on the basis of the criteria shown in Table 5. In
this process, the fact that the postpositional case auxiliaries“kara” (from), “de” (at), “ni” (to), “e” (to), “yori” (from),and “nite” (at) are used to indicate location or direction was
utilized. Examples of extraction by this criterion are givenin the second row from the bottom in Table 5.
3.2.2. Elimination of personal nouns
The criteria shown in Table 5 are not rigorous, andwords which are essentially personal nouns were includedin the extraction based on these criteria, as shown in thebottom row. This is due to the multiple usage of the postpositional auxiliaries. The postpositional case auxiliary “e”(to), for example, indicates direction in the extraction described in this section, but can also indicate a person in thedative case in the included personal nouns. The included
personal nouns were eliminated by removing the personalnouns extracted in Section 3.1 from among the words
extracted in this section. Table 6 lists some of the extracted

3.2. Acquisition of locational/organizational
nouns
Locational/organizational nouns—suffix nouns indicating a location or organization, such as “XX station”—
and common nouns which by themselves indicate a location

Table 5. Locational/organizational noun extraction rules
Word*

RWC morpheme[noun–propernoun] [noun–
proper noun–
organization][noun–proper
noun–location–*]Kyoto Univ. morpheme [noun–locational
name][noun–
organizational
name]Example“Tokyo”[noun–locational
name]Example of mistakenly “Tokyo”
mixed personal name [noun–locational
name]

*

...

[noun–*]...

[noun–*][suffix–noun typesuffix noun][postpositional[noun–
common noun] auxiliary–caseauxiliary][suffix–noun
type suffix
noun]“konai” (within “e” (to)“kogakubu”[postpositional
campus)(faculty ofauxiliary–caseengineering) [noun– [noun–
common noun] auxiliary]common noun]“kogakubu”“kyoju”“e” (to)(faculty of(professor)[postpositional
engineering) [noun– [noun–
auxiliary–casecommon noun]
common noun] auxiliary]“daigaku” (university)[noun–common noun]“daigaku” (university)[noun–common noun]

37

...

*

[“kara” (from)][“de” (at)][“ni” (to)][“e” (to)][“yori” (from)][“nite” (at)][postpositional[noun]
auxiliary–case[noun–nonautonomous–*] auxiliary][noun–suffix]

words after elimination of personal nouns, in decreasingorder of frequency.

Table 7. Number of collected locational/organizational
nouns

3.2.3. Expansion of vocabulary by use of athesaurus

Extraction from
corpus

A total of 696 suffix nouns were left after eliminating
personal nouns. This vocabulary was insufficient for use inactual caption analysis. Consequently, the vocabulary was
expanded by using the thesaurus as described in Section 3.1.
Table 7 shows the change in the vocabulary size
during the construction of the dictionary. Some 7908 locational/organizational nouns were acquired in the final stage.
3.3. Acquisition of temporal nouns

Expansion
After
by Newdeleting
Classified
personal
Lexicalname
Table

Word in New Classified
Lexical Table

674

607

7,819

(number of classes)

764

697

697

(number of subclasses)

318

307

307

Words outside of New
Classified Lexical Table

91

89

89

Total thesaurus

765

696

7,908

Temporal nouns—suffix nouns indicating time, suchas “XX go” (after XX)—and common nouns indicatingtime by themselves, such as “today,” were acquired. Thissection describes the acquisition procedure and the results.
shown in the bottom row of Table 8. Table 9 lists some ofthe extracted words in decreasing order of frequency.

3.3.1. Extraction from corpus
Temporal nouns were extracted from the corpus
based on the criteria shown in Table 8. The property of the
postpositional case auxiliaries that “kara” (from), “ni” (to),and “yori” (since) are used to indicate time was utilized.
Although not shown in Table 8, the “noun–temporal noun”
morphemes were prespecified in the Kyoto University Text
Corpus, and all words with those morphemes were extracted. An extraction example based on this criterion is

3.3.2. Expansion of vocabulary by use of athesaurus
Some 156 suffix nouns were extracted from the corpus based on the criteria shown in Table 8. However, thisnumber is still insufficient for actual caption analysis. Consequently, the vocabulary size was expanded by using thethesaurus as described in Section 3.1. The change in the

Table 6. List of extracted locational/organizational nouns (Top 30)
Order

Extracted word

Occurrence

Order

Extracted word

1
2
3
4
5
6
7
7
7
10
11
12
13
14
14

“shi” (city)“nai” (in)“gawa” (side)“taikai” (convention)“ken” (prefecture)“shinai” (in a city)“eki” (station)“seifu” (government)“sen” (war)“mondai” (problem)“sho” (department)“chiho” (district)“ku” (ward)“daishinsai” (catastrophic earthquake)“kokunai” (in a country)

163
108
78
60
46
43
39
39
39
38
36
31
30
29
29

16
16
18
19
20
20
22
22
24
25
25
25
28
28
28

“kaidan” (conference)“sen” (election)“chisai” (regional court)“machi” (town)“ichiba” (market)“kan” (between)“chiku” (region)“kai” (party)“kan” (building)“oki” (offshore)“kaigi” (conference)“chiiki” (region)“mura” (village)“ba” (site)“gun” (troops)

38

Occurrence
28
28
27
25
23
23
22
22
20
19
19
19
18
18
18

Table 10. Number of collected temporal nouns

Table 8. Temporal noun extraction rules
Word*

Expansion
by New
Extraction
Classified
from corpus
LexicalTable[“kara” (from)][“ni” (to)][“yori” (from)]

RWC morpheme [noun–adverb–*] [postpositional
auxiliary–case[noun–nonauxiliary]
autonomous–adverbpossible–*]Kyoto Univ. mor- [noun–adverb
phemetype noun][postpositional
auxiliary–caseauxiliary]

Example“kara” (from)[postpositional
auxiliary–caseauxiliary]“koro” (about)[noun–adverb
type noun]

Word in New Classified
Lexical Table(number of classes)(number of subclasses)
Words outside of New
Classified Lexical Table
Total thesaurus

136

2,124

213
104
20

213
104
20

156

2,144

captions were analyzed and the performance of the dictionaries evaluated.

vocabulary size in the course of dictionary compilation isshown in Table 10. A total of 2144 temporal nouns wereacquired at the final stage.

4.1. Conditions of experiment
The captions were analyzed by the procedure shownin Fig. 2. In this study, the caption area was detected andthe characters were recognized by human observation andthe results were transcribed. For the morphological analysis, the Japanese morpheme analysis system JUMAN [10]was used. JUMAN outputs the borders between morphemes and their attributes/parts of speech. In this study, if
the tail of the caption to be analyzed was a noun, the caption

4. Evaluation of Dictionary Performanceby Analysis of News Captions
Using the semantic attribute-tagged suffix noun dictionaries compiled as described above, actual news video

Table 9. List of extracted temporal nouns (Top 30)
Order
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Extracted word
“tame” (for)“jidai” (era)“chu” (in)“kan” (between)“zen” (before)“ji” (time)“tokoro” (place)“uchi” (within)“sai” (when)“baai” (case)“ue” (above)“chokugo” (immediately after)“irai” (since)“nichi” (day)“go” (after)

Occurrence

Order

Extracted word

Occurrence

465
145
128
123
117
116
79
65
54
47
46
45
43
40
31

16
17
18
19
19
19
22
23
24
24
26
26
26
29
29

“hoka” (other)“koro” (about)“sengo” (after war)
“toji” (at that time)“jiki” (time)“mama” (remain)“toki” (when)“kekka” (result)“kotoshi” (this year)“genzai” (at present)“tabi” (time)“chokuzen” (immediately before)“choki” (long time)“iko” (after)“naka” (in)

30
27
25
24
24
24
23
22
20
20
16
16
16
15
15

39

was compared to the semantic attribute-tagged suffix noun
dictionaries and was analyzed.
JUMAN itself contains a proper noun dictionary anda temporal noun dictionary. For captions with a personalname, locational name, or organizational name at the tail,or some temporal nouns, the output result of JUMAN was
given priority for use. Only nouns which could not be
judged to be proper nouns or temporal nouns were analyzed
by the proposed method. Figure 3 shows the method ofutilizing the dictionaries.
A total of 2546 captions appearing in approximately
370 minutes of news videos were used in the experiment.
Captions other than noun phrases, such as headline title, as
well as captions appearing in schematic diagrams using CG
or flips, were excluded from the experiment. The reason isthat these captions rarely describe the contents of the image
directly. These captions should be easily identified by image recognition and natural language processing.
The correct solution in the evaluation is provided bya third party. When the result of analysis indicated morethan one possible attribute, the result was judged to becorrect if the correct result was included.

temporal nouns, were analyzed by JUMAN. “Combined”
denotes the results of analysis by combining the two procedures.
The following definitions are used. The precision andthe recall were used as evaluation measures. The precisionin each result expresses the performance by that analyticalprocedure alone. The recall expresses the contribution ofeach analytical procedure to the overall recall.
precision =recall =number of correct results (Nc)number of correct results (Nc) + number of mistakes (Nm)number of correct results (Nc)number of correct results (Nc) + number of oversights (No)4.2.2. Discussion
In the experiments, high recall was obtained for eachsemantic attribute. This is an important factor in deriving
the index for particular semantics, constrained by otherprocessing operations such as image processing. The precision is generally low, both in the proposed method alone
and in the combined method, except for captions indicatinga person. The reasons are that inadequate words existed in
the compiled dictionary, and that the proposed method is
not sufficient to assure correct analysis, due to semantic
diversity.
In the analysis of captions indicating a person, a recall
and precision that were almost entirely satisfactory forpractical purposes were obtained. Although the precision of
the proposed method is lower than the results of IREX,
which similarly handled Japanese text [21], the recall isalmost equal. The accuracy is sufficient for practical purposes in an application to index videos in which a larger
amount of information is collected, with the expectationthat the excess information will be selected in later image
processing and integration processing.
JUMAN can analyze only the attributes of proper
nouns and some temporal nouns. By combining JUMAN
with the proposed method, however, noun phrases with a

4.2. Experimental results and discussion
The experimental results obtained under the above
conditions are now presented and discussed.
4.2.1. Experimental results
The results of the analysis are shown in Tables 11 to
13. In the tables, the numbers under “analysis by proposed
method” are the results of analysis by the method shown in
Fig. 3. The numbers under “analysis by JUMAN” are theresults of analysis by JUMAN. As already described, captions with a pronoun, which is a personal name or a locational or organizational name at the tail, as well as some

Table 11. Results of personal caption analysis by thecreated dictionary: Nc, Nm, and No stand fornumbers of correct, mistaken, and oversightanswers, respectively

Fig. 3. Analysis process by combination of JUMAN
and the proposed method.

40

Nc

Nm

No

Precision Recall

Analysis by
proposedmethod

210

54

−

79.6%

68.9%

Analysis by
JUMAN

33

45

−

42.3%

10.8%

Combined

243

99

62

71.1%

79.7%

Table 12

Table 14. Reasons for mistakes and oversights

Results of locational/organizational caption
analysis by the created dictionary
Nc

Nm

No

Precision Recall

Analysis by
proposedmethod

307

491

−

38.5%

Analysis by
JUMAN

220

12

−

94.8%

33.6%

Combined

527

503

127

51.3%

80.5%

ReasonRatio(a)Insufficient vocabulary for common nounin the compiled dictionaries

44%

(b)

Insufficient vocabulary in JUMAN proper
noun and temporal noun dictionaries

39%

(c)

Diversity of semantics(example) “Konishiki-zeki” (name of a
Sumo wrestler, person), “Hakone-seki”(Hakone checkpoint, place/organization)

9%

(d)

Incorrect morpheme analysis by JUMAN(example) ! “bnen-sei” (6th grader,person), X “b-nensei” (others)

6%

(e)

Existence of inadequate words in thecompiled dictionaries

2%

46.9%

common noun as the tail noun can be analyzed regardlessof whether or not a proper noun is included. Thus, theanalysis performance was improved to 46 to 83% in termsof recall.
The main reasons for errors and oversights were thefive shown in Table 14. Items (a) and (e) are direct reflections of the performance of the dictionaries that were compiled. Approximately 40% of the cases attributed to item (a)should have been analyzed correctly if the sentence wasdecomposed into more detailed morphemes. One methodof handling such a situation may be to use the criterion of
longest late fit, not complete fit as is used at present.
However, this approach may increase mistakes.
Item (e) is a composite problem due to the existenceof inadequate words. It results from the less rigorous extraction conditions shown in Tables 2, 5, and 8, as well as
the presence of inappropriate words that were included
while expanding the vocabulary by using the thesaurus.
This problem can be resolved by manual correction of thedictionaries. Most of the inadequate words in the former
problem were extracted due to the use of postpositional
auxiliaries in a manner different from the use assumed in
the extraction conditions. In order to resolve this problem,the morpheme information in the corpus should include theuse of postpositional auxiliaries. Most of the inappropriate
words associated with the latter problem are due to discrep-

Nc

Nm

No

Precision

RecallAnalysis by
proposedmethod

142

221

−

39.1%

83.5%

ancies between the class specified by the classification
principle in the thesaurus that was employed and the classused in the proposed method. Consequently, improvement
should result from using a dictionary with a classification
principle geared to the proposed method.
Item (c) is an essential problem which arises in handling semantics. It cannot be easily resolved by a simple
analysis as used in this study. To deal with items (b) and (d),there must be improvements in the vocabulary of the morphological analysis tool and improvement of analytical
performance.
Table 15 compares the results obtained by using
dictionaries (ideal dictionaries) in which adequate classification items were manually selected from the thesaurus,
and the results shown in Tables 11 to 13. This is a comparison of the upper bound of semantic attribute analysis by the
proposed method. Since the number of correct results isincreased only slightly if an ideal dictionary is used, we seethat item (a) is not a serious problem in terms of the numberof occurrences. On the other hand, the number of mistakes
is greatly reduced, which indicates that item (e) is a serious
problem in terms of the number of occurrences. Since theincidence of oversights (No) is not reduced much if the ideal
dictionary is used, it is possible that the vocabulary notincluded in the thesaurus that we employed should becomplemented in order to fill gaps. That is, additional
words should be extracted from a larger-scale corpus byusing the proposed method.

Analysis by
JUMAN

17

0

−

100.0%

10.0%

5. Conclusions

Combined

159

221

11

41.8%

93.5%

This paper has described the process of composing a
dictionary for the semantic attribute analysis of captions, toTable 13. Results of temporal caption analysis by thecreated dictionary41

Acknowledgments. The “Lexical Classification
Table in the ‘Classified Lexical Table’ Format (Revised
Version)” [8] was used under a monitor contract with the
National Institute of Japanese Language. The “RWC Text
Database” [2] was used under a license contract with theReal World Computing Project (RWCP). A tremendous
amount of labor was contributed by Mr. H. Taira and Ms.
T. Oda in transcribing the caption texts to be used in theexperiment, for which the authors are grateful.

Table 15. Results of caption analysis by manuallycreated dictionary

PersonNc

Nm

No

Precision

Recall

243
(±0)

93
(–6)

62
(±0)

72.3%
(+1.2%)

79.7%(±0.0%)

Place/
534
172
120
75.6%
81.7%
organization (+7) (–331) (–7) (+24.3%) (+1.2%)
Time

159
29
11
84.6%
93.5%
(±0) (–192) (±0) (+42.8%) (±0.0%)

REFERENCES
1. Ando A, Imai T, Kobayashi A, Homma S, Goto A,
Kiyoyama N, Mishima T, Kobayakawa T, Sato S,
Onoe K, Segi H, Imai A, Matsui A, Nakamura A,
Tanaka H, Togi T, Miyasaka E, Isono H. A broadcast
news caption composition system based on speech
recognition. Trans IEICE 2001;J84-D-II:877–887.
2. Real World Computing Project (RWCP). RWC Text
Database, 2nd version, 1998.
3. Hori O, Mita Y. A robust method of character recognition from video for telop recognition. Trans IEICE
2001;J84-D-II:1800–1808.
4. Ide I, Hamada R, Sakai S, Tanaka H. An attributebased news video indexing. Proc ACM Multimedia
2001 Workshops—Multimedia Information Retrieval, p 70–73.
5. Ide I. A study on automatic indexing of video. Indexing by integrated media processing and its application to news video. Doctoral dissertation, Graduate
School of Engineering (Electrical Engineering), The
University of Tokyo, 1999.
6. Ide I, Yamamoto K, Hamada R, Tanaka H. Automatic
indexing of videos based on shot classification. Trans
IEICE 1999;J82-D-II:1543–1551.
7. Kato N, Uraya N, Aizawa T, Nakase S. Processing ofproper nouns in English–Japanese translation. 40thNatl Conv Inf Process Soc 1990;2F-2, p 421–422.
8. National Institute of Japanese Language Lexical
Classification Table in ‘Classified Lexical Table’ Format (Revised Version), Revised Electronic Data for
Monitoring, 1996.
9. Kurakake S, Kuwano H, Odaka K. Recognition andvisual feature matching of text region in video forconceptual indexing. Proc SPIE Conf 3022: Storageand Retrieval for Image and Video Databases V, p
368–379, 1997.
10. Artificial Intelligence Language Media Laboratory,
Graduate School of Information Science, Kyoto University. Japanese Morphological analysis system JUMAN version 3.6. http://www-lab25.kuee. kyoto-u.
ac.jp/nl-resource/juman.html, 1998.

be used in automatic indexing of news videos. A performance evaluation has been presented. Specifically, the noun
(suffix noun) at the tail of the caption (noun phrase) is noted.
To support an analysis of classifications of personal, locational/organizational, and temporal uses, the dictionaries
were constructed by extracting suffix nouns indicating semantic attributes from corpora.
The resulting dictionaries consisted of 1795 words inthe personal noun dictionary, 7908 words in the locational/organizational noun dictionary, and 2144 words inthe temporal noun dictionary, for a total of 11,847 words.
It is conceivable that a similar dictionary could be constructed by manually selecting the classification items in
the New Classified Lexical Table. However, automaticcomposition saves the effort of manual composition andyields the supporting sample data. Another advantage is thatwords not contained in the New Classified Lexical Table
can be included. In this study, 19 personal words, 89 locational/organizational words, and 20 temporal words, or a
total of 128 words, not included in the table were acquired.
As the next step, the proposed method was applied tothe analysis of actual news videos and its performance was
evaluated. Our analysis showed that the recall, which isimportant in indexing, was high, indicating the practicalusefulness of the method, although the precision was low.
Comparison experiments using an ideal dictionary failed to
remedy the problem of insufficient vocabulary of the dictionary. It will be necessary to add words not contained in
the thesaurus in order to improve the recall, which is important in indexing. The above result suggests that it will
be necessary to enlarge the range of word acquisition froma larger-scale corpus.
For the contents of the dictionary that was constructed (classification items and words outside the classified lexical table), see Ref. 5. Regarding the results ofindexing actual news videos by using the dictionary compiled in this study and the mechanism shown in Fig. 1, see
Ref. 4.

42

11. Artificial Intelligence Language Media Laboratory,
Graduate School of Information Science, Kyoto University. Kyoto University Corpus, version 2.0. http://
www-lab25.kuee.kyoto-u.ac.jp/nl-resource/corpus.
html, 1998.
12. Nakamura Y, Kanade T. Semantic analysis for video
contents extraction—Spotting by association in news
video. Proc Fifth ACM Int Multimedia Conf (ACM
Multimedia ’97), p 393–402.
13. Nagase T. Pre-processing for Japanese syntax analysis based on morpheme type. 41st Natl Conv Inf
Process Soc 1990, 1S-6, Vol. 3, 109–110.
14. Nasukawa T. Determination of keyword semantics
based on context information. 11th Natl Conv Soc
Artif Intell 1997; 17-1, p 348–349.
15. Satoh S, Nakamura Y, Kanade T. Name-It: Namingand detecting faces in news videos. IEEE MultiMedia
1999;6:22–35.

16. Sundheim BM. Named entity task definition, version
2.1. Proc Sixth Message Understanding Conf (MUC6), 1995, p 317–332.
17. Wactler HD, Hauptmann AG, Witbrock MJ. Informedia News-on-Demand: Using speech recognition tocreate a digital video library. CMU Tech Rep CMUCS-98-109, Carnegie Mellon University, 1998.
18. Wactler HD, Christel MG, Gong Y, Hauptmann AG.
Lessons learned from building a terabyte digital
video library. IEEE Comput 1999;32:66–73.
19. Watanabe Y, Okada N, Nagao M. Semantic analysisof telops used in video news. Tech Rep Nat Lang
Process Inf Process Soc 1996, 96-NL-116, p 107–
114.
20. Appendix H: NE definition. Proc IREX Workshop
1999, p 264–273.
21. Appendix J: Results of NE evaluation. Proc IREX
Workshop 1999, p 286–294.

AUTHORS (from left to right)

Ichiro Ide (member) received his B.S. degree from the Department of Electronics Engineering, The University of Tokyo,
in 1994, completed the M.E. program (information engineering) and doctoral program (electrical engineering) in 1996 and2000, and joined the National Institute of Informatics as a research associate. His research interests are natural language
processing, video understanding, and integrated media processing. He received a 1995 Encouragement Award from the 51st
National Convention of the Information Processing Society of Japan. He holds a D.Eng. degree, and is a member of the JapaneseSociety for Artificial Intelligence and the Information Processing Society of Japan.
Reiko Hamada (student member) received her B.S. degree from the Department of Information and Communication
Engineering, The University of Tokyo, in 1998, completed the M.E. program (electrical engineering) in 2000, and is now in the
doctoral program. She is a JSPS Special Researcher. She received a 2000 Encouragement Award from the 63rd National
Convention of the Information Processing Society of Japan. She is a student member of the Information Processing Society ofJapan.

43

AUTHORS (continued) (from left to right)

Shuichi Sakai (member) received his B.S. degree from the Department of Information Science, The University of Tokyo,
in 1981, completed his graduate course in information engineering in 1986, and joined the Electro-technical Laboratory. He
was an invited researcher at MIT in 1991–92; head of the RWC Super-Parallel Architecture Laboratory, 1993–94; associate
professor, Institute of Electronics and Information Engineering, University of Tsukuba, 1996–98; associate professor, Graduate
School of Engineering, The University of Tokyo, 1998; professor, Graduate School of Information Science and Engineering,
2001. He is engaged in research on computer systems in general, especially architecture, parallel processing, schedulingproblems, and multimedia. He received a 1990 Information Processing Society of Japan Best Paper Award, 1991 IBM Japan
Award, 1995 Ichimura Science Award, and 1995 71CCD Outstanding Paper Award. He holds a D.Eng. degree, and is a memberof the Information Processing Society of Japan and the Japanese Society for Artificial Intelligence.
Hidehiko Tanaka (member) received his B.S. degree from the Department of Electronics Engineering, The Universityof Tokyo, in 1965, completed the doctoral program (electrical engineering) in 1970, and became a lecturer there. He became aprofessor on the Faculty of Engineering in 1987, and is now a professor in the Graduate School of Information Science andEngineering. He was a visiting professor at New York City University in 1978–79. His research interests are computerarchitecture, parallel processing, artificial intelligence, media processing, natural language processing, distributed processing,and CAD. He is the author of Non-Neumann Computers and Information Communication Systems; a coauthor of Computer
Architectures, VLSI Computing I, II, and Software-Oriented Architecture; and Editor of New Generation Computing. He holds
a D.Eng. degree, and is a member of the Information Processing Society of Japan, the Japanese Society for Artificial Intelligence,and the Japan Society for Software Science and Technology.

44