Proceedings of the Third NTCIR Workshop

Exploitation of Newspaper-article Characteristics for Article
Retrieval and Answer Extraction in QAC Task 2
Ruck THAWONMAS† Takayuki TOMOIKE††
Tomohiko KAWACHI††† Akio SAKAMOTO†††† Department of Computer Science, Ritsumeikan University
1-1-1 Noji Higashi Kusatsu City, Shiga 525-8577, Japan†† Course of Information Systems Engineering, Kochi University of Technology††† Department of Information Systems Engineering, Kochi University of Technology
185 Miyanokuchi, Tosayamada-cho, Kami-gun, Kochi 782-8502, Japan
ruck@cs.ritsumei.ac.jp
Abstract
In this paper, we discuss a system for newspaper article retrieval and answer extraction. Due to the rapidly
increasing amount of accessible information, systems
that allow search in natural language are expected toplay a much more important role in the very near future. Our system, called RAIK-Prassie, is designed
for TASK 2 of QAC. The design of the RAIK-Prassie
system focuses mainly on practical use of characteristics of newspaper articles and on questions related toperson names. In addition, we implement the systemsuch that it replies at most one answer to each query.
Keywords: Newspaper Article Characteristics, Question Answering, Information Retrieval, Information
Extraction

1Introduction

The amount of information spreading out over theInternet is dramatically increasing due to the popularization of WWW. Accordingly, WWW has become an indispensable tool to search/retrieve suchinformation. However, compared with the quality
and amount of transmitted information, still there
are not suﬃcient tools and/or techniques for precisely and promptly searching/retrieving the designated information.
At present, the most commonly used information retrieval techniques are those by which users
directly use keywords to specify the designatedinformation. Though they impose fairly simple implementations to achieve information services, such techniques are hardly considered userfriendly. For example, sometimes, users may ﬁnd
it is diﬃcult to specify a proper set of keywords,
when a single keyword just does not work. In
this case, the users are usually required to understand some search syntaxes such as AND/OR con© 2003 National Institute of Informatics

ditional statements. Thus, these techniques might
not be considered as good candidates for preciselyand promptly searching/retrieving the designatedinformation.
To solve the above problem, researchers are currently developing information retrieval techniquesthat accept queries in natural language, cf., worksin [1] and [2]. With these techniques, users canmake queries to the systems as if they do tohumans. For non-expert users, these techniques
are hence much simpler to use compared to thekeyword-based ones.
Question and Answering Challenge (QAC)[4], apart of the NTCIR Workshop, has been conductedwhich uses newspaper articles as benchmark datafor information retrieval in natural language. We
have participated in the TASK2 of QAC, and conducted research and development of our originalsystem. Main features of our system include (a)exploitation of characteristics in newspaper articles and (b) specialization to queries related toperson names.
In the previous version of our system, called
RAIK-Prassie, [3], techniques for extraction of asingle answer were proposed that use the distancesamong terms together with one characteristic innewspaper articles that important terms usuallylie in the ﬁrst sentence. Other newspaper article
characteristics are further exploited in the current
version of the RAIK-Prassie system, used in theFormal Run, discussed below in detail.

2System Description

The RAIK-Prassie system is designed such that it
will give at most one answer to a given query. Inaddition, it does not give an answer when the calculated reliability is below a pre-deﬁned threshold.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

The ﬂowchart of the RAIK-Prassie system used in
the Formal Run is shown in Fig. 1.

Acquire the question pattern• The main points of each article usually appear in the ﬁrst sentence.

Preprocessing

Delete the question pattern
Conduct morphological analysis
Combine back the substring residing
between specific symbols• Important matters are stated early in an article if not in the ﬁrst sentence.

Set N = number of index words
and i = 2

• Person names followed by brackets frequently
indicate the highlighted persons in an article.

Count for each index word the number oftimes of appearance in all articles
Yes
Does one article remain?
No
Narrows articles down to those whose
headline contains the fewest
appearance index word

Yes
Does one article remain?

2.2

Yes

Newspaper articleretrieval

NoYes
Select the most recent article

Delete sentences that do not contain any index word
Delete substrings that reside between
YesNo
Set i = 1
Any person names in thesentence with the i-thhighest weight?
i=i+1

Yes

and

Answer extraction

Delete sentences that contain

Any person namesin the 1st sentence?

If there exist more than one person
name, the following rules from highest priority to lowest one is applied
1. Select the one followed
by brackets
2. Select the one having
both name and sir name
3. Select one of the rests randomly

Noi = N?

NoYes
Reply "none" as the answer

The RAIK-Prassie system exploits the above
characteristics to perform the QAC task 2.

Preprocessing

i=i+1

Noi = N?

• The name of the reporter is given within thesymbols ”【 ” and ” 】”.

No

Narrows articles down to those whose
main body contains the i-th fewest
appearance index word

Does one article remain?

Exploitation of Characteristics
in Newspaper Articles

Newspaper articles must convey a relatively large
amount of information within limited writing
space. As a result, some relevant characteristics
below can be seen.

Begin

Narrows articles down to those whose
main body constains the fewest
appearance index word

2.1

Reply the selected person nameas the answer

Figure 1: Flow chart of the RAIK-Prassie System.

Though the system is universally designed tocope with queries related to person names, places,
times, quantities; this paper shall discuss mainly
on how to deal with those queries related to personnames.

Given a query sentence, the RAIK-Prassie system ﬁrst performs acquisition of the corresponding
question pattern. For queries in Japanese, substrings implying such patterns normally come at
the end of sentences. For example, in the query「 ノーベル物理学賞を受賞し た日本人は誰ですか。
(Who are the Japanese laureates for the Nobel
Prize for Physics?) 」, the substring は誰です
か at the end of the sentence indicates that thisquery is asking for a person name. We call sucha substring a question pattern. Acquisition of thecorresponding question patterns allows us to specify the type of answers in the answer extraction
part. A typical list of question patterns related toperson names is shown in Table 1.
Table 1: A typical list of question patterns related
to person names.
Pattern
Typeは誰でしたか Whoは誰ですか
Whoは誰
Who
Once the answer type is known, the question
pattern no longer contains important information.
Hence, it is deleted from the query.
Next morphological analysis is done for the remaining query. Chasen [5] is used for this task.
Chasen has been widely used as a tool for performing morphological analysis of Japanese sentences.
This tool has also a function to describe the typeof each segmented substring.
Usually substrings residing in the symbols suchas ”「 ” or ” 」” are also segmented after performingProceedings of the Third NTCIR Workshop

morphological analysis. However, such substrings
frequently represent proper nouns such as the titles of books, etc. They should therefore not besegmented. In our system, those terms are combined back to their original forms. The resulting
segmented substrings are called index words.

2.3

Article Retrieval

Using the index words obtained in the preprocessing part, article retrieval is then performed thatsearches for articles containing those words. In theRAIK-Prassie system, the fewer frequently do theindex words appear throughout the whole articles,the more important they are considered. Such index words are given higher weights.
In the article retrieval part, the following procedure is performed for narrowing down the article
search space. The procedure stops when it ﬁnds
one candidate article or does not ﬁnd any.
1. Narrow down the articles to the ones whose
main body contains the fewest appearance index word. If there are multiple resulting articles, go to the next step.
2. Narrow down the resulting articles to the ones
whose headline contains the fewest appearance index word. If there are multiple resulting articles, go to the next step.
3. Iterate narrowing down the resulting articles
to the ones whose main body contains thenext fewest appearance index word. If there
are multiple resulting articles, go to the nextstep.
4. Select the article whose published date ismost recent.

• All substrings residing in the following symbols ”「 ” and ” 」” are deleted. These particular symbols are usually used for dialogues orspeeches of a highlighted person whose name,though, might be the answer.
Next, the answer is extracted. We employ aheuristic that the main points are stated in theﬁrst sentence. So if the ﬁrst sentence contains
a person name, we consider this person name as
the answer. However, if there are multiple personnames appearing in the ﬁrst sentences, we selectone of them according to the answer selection rules
listed below from the highest priority to the lowest
one.
1. Select the one followed by brackets.
2. Select the one having both name and sirname.
3. Select one of the rests randomly.
If the ﬁrst sentence does not contain any person
name, all other sentences are weighted according
to the index terms they contain. Sentences withlower appearance index words are assigned higher
amounts of weights. The sentence with the highest weight will be examined ﬁrst to see whether
or not it includes any person names. If a single
person name exists in the sentence, it will be theanswer. If multiple person names exist, then theabove answer selection rules are applied. If thissentence contains no person name, then the sentence with the second highest weight will be examined. The procedure is repeated until a personname is eventually selected. If no person nameexists in the article, the system outputs ”none” asits answer.

32.4

Answer Extraction

From the resulting retrieved article, discussed below is how the answer is extracted.
First, the article is segmented. The following
procedure is then executed.
• All sentences containing the following symbols ”「 and 」”are deleted. This is because such sentences have tendency to correspond to information related to the reporterand hence are not related to the answer of thequery.
• All sentences that do not contain any index
word are deleted. Such sentences are considered to represent auxiliary information in thearticle. Hence they have low tendency to contain the answer of the query.

Formal Run Results

Results of eleven systems, participating in theQAC TASK 2, were announced by the QAC organizer. In QAC TASK 2, each system was asked
to give answers to 200 given queries. These results are summarized in Table 2. In the table,
the RAIK-Prassie system is given the identiﬁer
SysID S20005.
The deﬁnitions of the upper stack in a row aregiven as follows: Answer-the number of diﬀerent
answers in the task, Output - the number of answers that the user’s system replied, Correct - thenumber of correct answers that the user’s system
replied. In addition, each value of the lower stack
in a row is computed as follows:• Recall = (the number of correct answers thatthe user’s system output)/(the number of correct answers)

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

• Precision = (the number of correct answers
that the user’s system output)/(the numberof answers that the user’s system output)• F-measure = (2 * Recall * Precision)/(Recall+ Precision)

Table 2: Formal Run Results of Eleven Systems
SysID
Answer Output
Correct
RecallPrecision F-measure
S20001 305
729
74
24.262
10.15114.313
S20002 305
20081
26.55740.5
32.079S20003 305
325
76
24.91823.38524.127
S20004 305
456
82
26.88517.98221.551
S20005 305
194
144.597.216
5.611
S20006 305
200011437.377
5.7
9.892
S20007 305
2763712.131
13.40612.737
S20008 305
2414
6320.656
2.61
4.634
S20009 305
371
0000
S20010 305
29245
14.75415.41115.075
S20011 305
586
34
11.148
5.802
7.632
Ave.
305
713
56.36418.48012.92413.423We’d like to point out here that among 11teams, though our system is ranked 10th and 9thfor ”Recall” and ”F-measure”, respectively, thesystem is ranked 7th in terms of ”Precision”. This
performance is fairly reasonable for a ﬁrst timeand relatively late participant team like ours. The
RAIK-Prassie system has been designed to give
only a single answer for a given query, though thequery might essentially require a set of multipleanswers. Another emphasis in the design is tohave a system that will not give wrong answers tothe users.

4System Analysis

As shown in Fig. 1, the RAIK-Prassie system consists of three modules, namely, the preprocessing module, the newspaper article retrieval module, and the answer extraction module. In thissection, we discuss our analysis results done forthe newspaper article retrieval module. There are
200 queries in the Formal Run. Among them 43
queries are related to person names, which are ourmain targets for testing the developed system. Inour analysis below, we focus on these 43 queries.
There are 18 queries for which wrong articles
were retrieved. By wrong articles, we mean articles that do not contain the correct answers.
Among these, 13(72%) articles were retrieved because they contain the fewest appearance index
word in their headline. However, for the remaining 25 queries, the RAIK-Prassie system could retrieve articles that contain the correct answers for
22(88%) queries because those articles contain thefewest appearance index word in their headline.
As a result, our heuristic to retrieve articles according to what their deadline contains might not
be as viable as we thought. Reconsideration ofthe procedure for narrowing down articles is necessary.

5Remaining Problems andFuture WorkThe Formal Run version of the RAIK-Prassie system retrieves only one article and attempts to extract an answer from the retrieved article. However, as discussed above, we have found that retrieved articles might not always contain the correct answers (Table 3). To solve this problem,
we are now extending the article retrieval module
such that it allows retrieval of multiple articles bycombining the narrowing down results from both
article bodies and headlines.
Table 3: Successful and Failed Examples
Successful Example
Query ID
QAC1-2033-01
Query「 速水優の前の日銀総裁は誰ですか 。
(Who was the head of
the Bank of Japan
before Yu Hayami?) 」
Article ID
980414353
Answer「 松下康雄(Yasuo Matsushita) 」
Failed Example
Query ID
QAC1-2063-01
Query「 源頼朝の弟は誰ですか。
(Who is the younger
brother of the shogun
Yoritomo Minamoto?) 」
Article ID
991228141
Answernone(could not ﬁnd the answer)

Proceedings of the Third NTCIR Workshop

In addition, the length of each newspaper article in the repository varies. As a result, in the article retrieval module, articles with longer lengths
have higher tendency to be retrieved than thosewith shorter lengths. This is a problem because
longer articles do not always contain correct answers. To solve this, we are applying text abstraction techniques such as Posum[6] which is publiclyavailable. The resulting abstract of each article
should have similar length. This might improve
the performance of our system.

References[1] Wataru HIGASA. Dialogue Helpsystem
based on Flexible Matching of User Querywith Knowledge Base. Master Thesis, Kyoto
University, 2000 (in Japanese).
[2] Takayuki TOMOIKE. A Study on Construction of a Log Analysis System Utilizing Similar Case Data. Senior Thesis, Kochi University of Technology, 2001 (in Japanese).
[3] Takayuki TOMOIKE, Ruck THAWONMAS,and Akio SAKAMOTO. A Consideration
on Information Retrieval and Information
Extraction in a Question Answering System. Proc. Shikoku Information Processingof Japan Research Symposium, 2002 (inJapanese).
[4] QAC Homepage:
http://www.nlp.cs.ritsumei.ac.jp/qac/[5] Yuji MATSUMOTO. Morphological Analysis System ChaSen version 2.2.8 Manual.
NAIST, Matsumoto Laboratory, 2001.
[6] Hajime MOCHIZUKI. Posum version 1.50.2
Manual. JAIST, 2002.