Automatic Semantic Analysis of Television News Captions
Ichiro IDE and Hidehiko TANAKA
Graduate School of Electrical Engineering, The University of Tokyo
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, JAPANTEL: +81-3-3812-2111 ext.7413, FAX: +81-3-5800-6922
f ide j tanaka g @mtl.t.u-tokyo.ac.jp

Abstract
Automatic indexing to image data is in strong demand.
Utilizing accompanying natural language information isconsidered e ective to accomplish the task. As a basis
for semantic indexing, we propose an automatic television
caption semantic analysis method, which analyzes semantic attributes of Japanese television news captions referring
to suxes. This is a basic pre-process required to enableadvanced indexing, which considers semantic attributes of
keywords. Classi cation in conventional concept-based dictionaries are not fully applicable for such purpose, thus
we extracted such suxes from corpora. The result wasapplied to actual television news programs for evaluation,
which showed fairly high recognition rates.

1 Introduction
As the amount of broadcast video data increases, it is
becoming more and more important to store them in awell organized manner considering recycling and searching. Above all, television news programs are worthwhile
indexing in accordance with the importance and usefulness.
Currently this process is mostly carried out manually, but
automatic indexing is in big demand to cope with the increasing amount of data and to achieve sucient precision
for detailed searching.
We are trying to accomplish this task by referring toboth video data and natural language data accompanying
them in Japanese television news programs. There are various natural language data sources accompanying the image
data; main audio, sub audio, closed caption (mostly sameas the main audio) and caption. Particularly, captions areusually used to describe important matters in a digestive
form, so they could be considered as adequate keyword candidates for indexing. According to our statistics, they appear approximately once per every 15 seconds, which is anadequate frequency for nding candidate keywords. Thisis one reason why we do not use main audio (or closed caption) as natural language data source, since they requirecomplicated key-phrase extraction process. Nevertheless,captions have their own peculiar character, which does notnecessarily allow good performance of conventional natural
language processing methods.
There are several notable attempts made to automatically index television news programs by utilizing accompanied natural language source. Most of their indexing
strategies are based on statistics or just simple occurrences.
TF-IDF (Term Frequency Inverse Document Frequency)
method is frequently used as a statistic method to extract
keywords from main audio (or closed caption) as in the Informedia project's [Smith97, Wactlar96, Informedia] `News
on Demand' system [Hauptmann97]. On the other hand,some others search for keywords in a full text searchingmanner. These are relatively simple, and in that sensequite practical approaches, but the critical point is thatthey do not necessarily ensure the correspondence of theindex and the image contents.
Re ecting these background issues, we are currently developing an image driven keyword extracting method that
ensures the correspondence of the index and the image contents [Ide98]. In order to realize this method, semantic
analysis of captions becomes important on the natural language processing side. Thus, we will introduce a semantic
analysis method for Japanese television news captions inthis paper. Although we have not applied this method toother languages, it should be applicable to other East Asian
languages that have similar characteristics in suxes.

2 Semantic Analysis of TV News CaptionsFirst we will discuss the characteristics of television
news captions, and then introduce the basic idea of theirsemantic analysis referring to suxable nouns.

2.1 Characteristics of Captions
2.1.1 Grammatical Characteristics

From the grammatical point of view, captions are peculiar compared with normal texts handled by conventional
natural language processing methods. Major characteristics are listed below:

Sentence-like caption

1. Simple sentence
2. Omission of the subject
3. Ends with noun
4. Ends with speci c post-positional particles, suchas `he', `ni', `ka', `mo', `ha' and `wo', which indicate estimation, conjecture and so onNon-sentence-like caption

5. Omission of particles
i.e. consists mostly of nouns
6. Noun phrase consisted only of nouns
Due to these characteristics, although high quality natural language processing methods exist, they are not fully
applicable to analyze television captions.

2.1.2 Semantic Characteristics

Besides grammatical characteristics, captions have semantic characteristics as shown in Table 1. Types listed
here are mostly based on [Watanabe96], and the percentages of each type were derived from sample television news

Table 1: Types of television news captions. Numbers cited in `Grammatical Characteristics' refer to those listed in 2.1.1.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)

Types
Ratio Grammatical Characteristics
Locational, Organizational
30% 5, 6
Personal
15% 5, 6
Title
14% 1 - 6
Speech (Include summaries and translations) 10% 1, Colloquial
Temporal
7% 6Technical (i.e. `live')
3% 6
Descriptive
2% 2, 3, 5
Others19% 1 - 6

programs, where 464 captions appeared in total. Grammatical characteristics associated with semantic characteristics are also cited in the Table.
Among these types, (a) and (b), which cover 45% of thecaptions, represent simply what is actually shown in theimage. This allows them to almost directly become candidate keywords. Nevertheless, when attempting to index
conscientiously, it is important to analyze the semantics ofthe candidates. In this case, captions should be analyzed
whether they are (1) personal, (2) locational / organizational, or (3) others.
Although (c) does not necessarily re ect the graphicalcontents of the video, it is an important piece of information which explains the topic. Thus, occupying nearly 60%of the captions, (a), (b) and (c) could be directly used as
candidate keywords.

2.2 Semantic Analysis based on Suxable Nouns

As related researches, several methods to analyze nouns
that satisfy our purpose to some extent exist.
Similar to our task, `the Named Entity task' de ned for
the Message Understanding Conference (MUC) [NEtask]
assigns the participants to classify personal, organizational,
locational, temporal and numerical phrases. The di erenceis that our aim is not limited to proper nouns, while theNamed Entity task limits the tagging to personal, organizational and locational phrases to proper names.
Nasukawa [Nasukawa97] proposed a semantic disambiguation method that determines semantic attributes ofproper nouns (i.e. whether a proper noun indicates a place
or a person) referring to the context of neighboring sentences. On the other hand, Watanabe et al. [Watanabe96]
proposed a method that analyzes television news captions
by referring to both location and grammatical characteristics.
Although these methods perform fairly well, Nasukawa's
method is dicult to serve our purpose since (1) captions
do not have enough neighboring information to analyze
contexts, and (2) it is mainly purposed to handle proper
nouns. Watanabe et al.'s method does serve our purpose,but is not generally applicable to various news programs,
which have di erent designing policies where layouts ofcaptions vary.
Considering these issues, we decided to analyze captions on their own. As an example, it is impossible even fora human being to distinguish whether `' (Hashimoto)alone indicates a person or a place or even a common noun,
which means `the foot of a bridge'. Then how do we distinguish them? We know that `' (Prime Minister- ' (Hashimoto
Hashimoto) indicates a person and `City) indicates a place from the suxable nouns, `'(Prime Minister) and ` ' (City), respectively.
As shown in this example, it is generally possible inJapanese (and also presumably in other East Asian languages) to analyze semantic attributes of noun phrases referring to suxable nouns. Accepting this assumption, we

will analyze captions that are mostly noun phrases, by referring to suxable nouns.
Although in this example and also in the following Sections, issues on suxable nouns are mainly discussed, ourmethod also includes analysis to individual nouns such as`volunteer' and `closet' that on their own indicate people
and places, respectively.

3 Collection of Suxable Nouns from Text Corpora
First of all, we need to know which suxable noun determines the semantic attributes i.e. (1) personal or (2) locational / organizational, of the whole noun phrase. There
are several renowned concept oriented Japanese dictionaries such as the EDR electronic word / concept dictionaries[EDR] and the Classi ed Lexical Table [Lexical], but these
are not structured in the way that satisfy our purpose, inthe sense that they are not organized considering the use ofwords. This makes it dicult to determine which concept
class in these dictionaries contains the appropriate suxable nouns. So, we rst collected such nouns automaticallyfrom two text corpora based on simple rules.
The following two corpora we used are morphologicallyanalyzed manually beforehand, which ensures the basic reliability of the collection process.


RWC-DB-TEXT-95-2 from the RWC Text Database [RWC]
Consists of approximately 27,000 sentences selectedfrom Mainichi newspaper's 1994 edition.

Kyoto University Text Corpus[KUTC]
Consists of approximately 20,000 sentences from theMainichi newspaper's 1995 edition. Strictly speaking,morphological analysis was carried out rst automatically using the Japanese morphological analysis system JUMAN [JUMAN] and the Japanese structural
analysis system KN Parser [KNP], and then correctedmanually.

Both of these corpora consist of texts from newspapers,
which should include vocabularies similar to those used in
television news captions.
Although these contain quite an amount of data, thenumber of the collected nouns is still relatively small. We
expanded the vocabulary by gathering nouns from the same
concept classes (consists of 10 words on an average) of theClassi ed Lexical Table (Enlarged Edition) in which thecollected nouns belong. This expansion was done basedon the assumption that although the structure of the Table does not satisfy our purpose, each concept class should
contain similar words.
In this Chapter, we will introduce the methods and theresults of the collection of the two groups of nouns.

Table 2: Result of personal noun collection. The top twenty frequent nouns among the 1,630 collected nouns (include
overlaps) are listed.
Rank Noun
1
2
3
4
5
6
7
8
9
10

(person)(children)(I)(person)(Mr.)(Mr./Ms.)(myself)(chairman)(athlete)(children)

Frequency Rank Noun
160
11
100
12
91
13
76
14
57
14
53
16
36
16
32
16
31
19
30
19

(congressman)(chief)(executive)(professor)(prime minister)(student)(housewife)(women)(-er as in `photographer')(corporate president)

Frequency
28
26
21
18
18
17
17
17
15
15

Table 3: Result of locational / organizational noun collection. The top twenty frequent nouns among the 4,861 collected
nouns (include overlaps) are listed.
Rank Noun
1
2
3
4
5
6
6
8
9
10

(city)(inside)(person)(ward)(side)(prefecture)(town)(conference)(government)(inside the city)

Frequency Rank Noun
302
11
161
12
142
13
123
14
111
15
85
16
85
16
80
18
58
19
53
20

3.1 Collection of Personal Nouns

In this Section, the method and the result of collecting\personal nouns"; suxable nouns that indicate people like
`Mr. |' and `actor', are discussed.

3.1.1 Collection Rules

1. Search for speci c suxes `ra' and `tachi' that only
succeed personal nouns
2. If the preceding morpheme is a common noun or asux, collect it as a \personal noun"\-'(Hashimoto)[Proper Noun](Prime Minister) [Common Noun]" (`ra' = et al.)[Sux]

Note that proper nouns are not collected in rule 2.

3.1.2 Result

Frequency
51
41
40
39
37
36
36
35
34
33

recorded in the Classi ed Lexical Table, and the remaining
56 were not. The recorded 287 were classi ed in 251 subclasses under 102 classes. 3,737 nouns were recorded inthese sub-classes, and as a nal result, a total of 3,793
personal nouns were collected.

3.2 Collection of Locational / Organizational Nouns

Personal nouns were collected by the following rules:

Following this rule, from the following phrase, `(Prime Minister) [Common Noun] will be collected.

(meeting)(language)( ght)(station)(state)(between)(region)(countries)(problem)(army)As a primary result, 343 di erent personal nouns werecollected including `ra' and `tachi' used as seeds. The top
twenty nouns are shown in Table 2. Among these, 287 were

In this Section, the method and the result of collecting\locational nouns" and \organizational nouns"; suxable
nouns that indicate locations and/or organizations like `|station' and `kitchen', are discussed.

3.2.1 Collection Rules

Locational / organizational nouns were collected by thefollowing rules:
1. Search for a place name (proper noun)1
2. Track down succeeding nouns continuously
3. When the chain of nouns ends, collect the last common noun if the succeeding morpheme is one of thefollowing post-positional particles that are related tolocations: `kara' (from), `de' (at), `he' (to), `yori'(from), and `nite' (at)Following this rule, from the following phrase, ` ' (City)[Common Noun] will be collected.
Note that, again proper nouns are not collected in rule3.
1 This information is available from the corpora.

\-(Hashimoto) [Proper Noun](City)[Common Noun]" (`de' = at) [Post-positional Particle]

3.2.2 Result

As a primary result, 1,030 di erent locational / organizational nouns were collected. The top twenty nouns areshown in Table 3. Among these, 930 were recorded in theClassi ed Lexical Table, and the remaining 100 were not.
The recorded 930 were classi ed in 1,049 sub-classes under
399 classes. At this point, we excluded all the sub-classes
that were assigned to personal nouns in the previous experiment (for example, sub-class derived from `人' (person)in Table 3). This ended in reducing the sub-classes to 926
under 379 classes. In these sub-classes, 11,070 nouns wererecorded, and as a nal result, a total of 11,166 locational/ organizational nouns were collected.
The exclusion may delete potential locational / organizational noun classes, but the mal-e ect from the wrongly
collected classes seems to be more harmful.

4 Evaluation
In order to evaluate the suxable nouns collected in
the previous Chapter, we applied each dictionary to actual
television news programs. Five 20 minute news programs
were used, in which 587 captions appeared.

4.1 Caption Analysis Procedure

For each caption:
1. Apply the Japanese morphological analysis system
JUMAN [JUMAN] to it. If the utmost tail morpheme
is analyzed as person's, place, or organization nameat this point, the analysis terminates here.
2. Else if the utmost tail morpheme is a common noun,
look up for it in the dictionaries collected in the last
Chapter to distinguish its semantic attribute.
For example, JUMAN will analyze a phrase as follows:\-

Table 4: Result of caption analysis (personal).

NCorrect
NReported
NTrueAnswer
Precision
Recall

4.2 Results

The result of the analysis is shown in Tables 4 and 5.
The true answers were given by a third person, and wascompared with the reported answer to evaluate the result.
Precision and recall are de ned as follows:

Correct
Precision = NNReported
NCorrect
Recall = NTrueAnswer

JUMAN
Combined
Alone
16
79
24
99
|
83
66.7%
+ (19.3%) =

79.8%
95.2%

Table 5: Result of caption analysis (locational / organizational).

NCorrect
NReported
NTrueAnswer
Precision
RecallProposed
Method Alone
80
176
|
45.5%
(50.0%)JUMAN
Combined
Alone
60
140
61
237
|
160
98.4%
+ (37.5%) =

59.1%
87.5%

In the Tables, \Proposed Method Alone" indicates theresult derived solely from the collected noun dictionary,and \JUMAN Alone" indicates the proper nouns analyzed
by JUMAN as person's, place, or organization names. Since
it is dicult to count the true answers for each method separately, individual recalls are calculated as how much eachmethod contributed to the overall recall.

4.3 Examination

Misrecognition and oversight are due to the following
reasons:Misrecognition
1. Misrecognition of proper noun (JUMAN)

There are proper nouns that belong to both categories in JUMAN proper noun dictionary.
2.
(Proposed
method)Inappropriate nouns that sneaked in during thecollection process, due to the incompleteness ofthe collection rules, and to the mismatch of theclassi cation policy of the Classi ed Lexical Table with our purpose. This is prominent in locational / organizational nouns.
3.
(Proposed method)
Some nouns were essentially applicable to bothcategories.

(Tanaka) [Person's Name](Tokyo)[Place Name](university) [Common Noun]" (professor) [Common Noun]

This does not give us any information whether the phrase
indicates a person, location / organization or somethingelse.
On the other hand, the proposed method looks up forthe last noun of the above result in the collected dictionary. Since the collected personal noun dictionary has thenoun `' (professor), the whole phrase is considered asa personal noun phrase.

Proposed
Method Alone
63
75
|
84.0%
(75.9%)

Noise in the collected dictionary

Semantic diversityOversight
4. Lack of proper noun vocabulary (JUMAN)
5.

JUMAN's proper noun dictionary did not haveenough vocabulary.

Lack of vocabulary in the collected dictionary (Proposed method)

Our dictionary did not have enough vocabulary
either.

Among these, we will have to wait for the upgrade of
JUMAN's dictionary to solve problem 4, but problem 2
will be solved by reconsideration of the collection rules,

and problem 5 by collecting nouns from larger corpora. It
is dicult to solve problems 1 and 3, since they arise fromdiversity; an essential property of words.

5 Conclusion
We proposed and evaluated an automatic television caption semantic analysis method in this paper. First, we
showed that a large number of captions could be used almost directly as keyword candidates by analyzing captions
manually. Next, we collected suxable nouns that could
be used as keys when analyzing semantic attributes of nounphrases, and accordingly, of most captions.
As a result, although there is still room for improvementof the dictionaries, the evaluation shows fairly good results.
The proposed method is a basis for using captions ascandidate keywords for automatic indexing. We have been
applying the method to actual indexing [Ide98], which shows
limited but promising results.

Acknowledgments[NEtask] United States Defense Advanced Research
Projects Agency (DARPA), Information TechnologyOce; \Named Entity Task De nition, Version 2.1";
Proc. 6th Message Understanding Conference, pp.317332, Nov 1995.
[KNP] Kurohsashi, S. and Nagao, M.; \Japanese Structural Analysis System KNP version 2.0 beta6"; Jul
1998 (in Japanese).
Available from http://www-lab25.kuee.kyoto-u.ac.jp/
nl-resource/knp-e.html[KUTC] Kurohsashi, S., Saito, Y. and Nagao, M.; \Kyoto University Corpus version 2.0"; Jun 1998 (inJapanese).
Available from http://www-lab25.kuee.kyoto-u.ac.jp/
nl-resource/corpus.html[JUMAN] Kurohsashi, S. and Nagao, M.; \Japanese Morphological Analysis System JUMAN version 3.5"; Mar
1998 (in Japanese).
Available from http://www-lab25.kuee.kyoto-u.ac.jp/
nl-resource/juman-e.html

The Japanese Morphological Analysis System JUMAN
is a free software developed at and distributed by Nagao
Lab., Kyoto University and Matsumoto Lab., Nara Institute of Science and Technology. Kyoto University Text
Corpus is a product of Nagao Lab., Kyoto University.
The Classi ed Lexical Table (Enlarged Edition) wasused for monitoring under a licensed agreement with National Language Institute of Japan. RWC text database isa product of Real World Computing Partnership (RWCP),
and was used under a licensed agreement.

[EDR] Japan Electronic Dictionary Research Institute,Ltd.; \EDR Electronic Dictionary"; Mar 1996 (inJapanese).

References[Informedia] \The Informedia Project";
http://www.informedia.cs.cmu.edu/.

[Ide98] Ide, I., Yamamoto, K., and Tanaka, H.; \Automatic Video Indexing based on Shot Classi cation";
Proc. 1st Intl. Conf. on Advanced Multimedia Content, Nov 1998 (to appear).

Processing[Nasukawa97] Nasukawa, T.; \Keyword Categorization
based on Discourse Information"; Proc. 11th Annual
Conf. JSAI, pp.348-349, Jun 1997 (in Japanese).
[Hauptmann97] Hauptmann, A. G. and Witbrock, M. J.
\News-on-Demand: Using Speech Recognition to Create a Digital Video Library"; Proc. AAAI'97 Spring
Symp. on Intelligent Integration and Use of Text, Image, Video and Audio Corpora, pp.120-126, May 1997.

[Hisamitsu97] Hisamitsu, T. and Niwa, Y.; \Acquisition of
Personal Names from Newspaper Articles by Lexical
Knowledge and Co-occurrence Analysis"; Tech. Rep.
IPS Japan 97-NL-118, Vol.97, No.29, pp.1-6, Mar 1997
(in Japanese).
[Smith97] Smith, M. A. and Kanade, T.; \Video Skimmingand Characterization through the Combination of Image and Language Understanding Techniques"; CMU
Tech. Rep. CMU-CS-97-111, Feb 1997.
[Watanabe96] Watanabe, Y., Okada, Y. and Nagao, M.;\Semantic Analysis of Telops in TV Newscasts"; Tech.
Rep. IPS Japan 96-NL-116, Vol.96, No.89, pp.107-114,
Nov 1996 (in Japanese).
[Wactlar96] Wactlar, H. D., Kanade, T., Smith, M. A. andStevens, S. M.; \Intelligent Access to Digital Video:
Informedia Project"; IEEE Computer, Vol.29, No.3,
pp.46-52, May 1996.

[Lexical] National Language Research Institute of Japan;\The Classi ed Lexical Table (Enlarged Edition)[Monitoring Version]"; Mar 1996 (in Japanese).
[RWC] Real World Computing Partnership (RWCP);\RWC Text Database"; Mar 1996 (in Japanese).