Proceedings of the Third NTCIR Workshop

A Question-Answering System Using Unit Estimation
and Probabilistic Near-Terms IR
Masaki Murata, Masao Utiyama, and Hitoshi Isahara
Communications Research Laboratory
2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0289, Japan
murata, mutiyama, isahara  @crl.go.jp
Abstract
Our question-answering system incorporates several new methods. One is unit estimation, which isuseful when the answer is a numerical expression andthe question sentence does not include any unit expression. This method can be used to estimate a unit
expression for the answer by using a statistical test
and corpus data and thus improves the results for suchquestions. Another new method is called probabilistic
near-terms information retrieval. This method enables
us to use full-size documents in document retrieval
without dividing documents into passages and is usefulwhen the answer is not within the passage where relevant terms occur. We confirmed the effectiveness of
the unit estimation by using a statistical test (a T-test)
and found that the probabilistic near-terms information retrieval improved the performance in Task-1 andTask-2.
Keywords: Statistical test, Unit estimation, Probabilistic near-terms IR

1 Introduction
We expect question-answering systems to become
increasingly important as a more convenient alternative to systems designed for information retrieval andto be a basic component of future artificial intelligence systems. We have investigated the potentialof question-answering systems [10] and have participated in the Question-Answering Challenge (QAC)
of NTCIR 3. Our question-answering system reported
at NTCIR 3 incorporates two new methods. One isunit estimation which is useful when the answer is anumerical expression and the question sentence does
not include any unit expression. This method can beused to estimate a unit expression for the answer by using a statistical test and corpus data and thus improves
the results for such questions. Another new method
is probabilistic near-terms information retrieval. This
enables us to use full-size documents in document re© 2003 National Institute of Informatics

trieval without dividing documents into passages. Thisis useful when the answer is not within the paragraphwhere relevant terms occur.

2 Importance of a question-answeringsystem
We have investigated question-answering systems
with regard to various aspects [10, 11, 7]. Our experience has led us to believe that question-answeringsystems are particularly important in two respects:

 Question-answering systems are more convenient for users than information retrieval systems.
Since a question-answering system shows users
direct answers to their questions, users can easily acquire the information they want without having to read the pertinent documents.
An information-retrieval system only identifies
such documents, thus making it necessary forusers to read through the documents to find theinformation they seek.

 Question-answering systems can be applied as acomponent of other intelligent systems.

A question-answering system that can automatically extract answers can be used as a component of other intelligent systems. We thinkthat a question-answering system will be a fundamental component of future artificial intelligence systems.

3 Explanation of our system
3.1 Outline of our system
Our system uses three ordinary algorithms as follows:
1. Estimation of answer type

The Third NTCIR Workshop, Sep.2001 - Oct. 2002



7. When ’(what) + particles’ is in a ques’
tion sentence and NOUN-FOCUS is not ’’ (area), or so on, ARTIFACT(company), ’is given as the answer type.

The system estimates the answer to be a particular type of expression based on the expressionsof an interrogative pronoun, adjective, or adverbof an input question sentence. For example,
if the input question sentence is ”How large isthe area of Japan?”, the expression ”How large”
suggests that the answer will be a numerical expression.

(what) + particles’ is in a ques8. When ’’tion sentence and the NOUN-FOCUS is ’(area), or so on, LOCATION is given as the answer type.
9. When ’ (what) + suffix’ is in a question sentence, the suffix is extracted as a unit expression.

The system extracts terms from a question sentence and retrieve documents by using theseterms. The retrieval thus gathers documentsthat are likely to contain the answer. For example, for the input question ”How large is thearea of Japan?”, we extract ’large’, ’area’, and’Japan’ as terms and retrieve documents containing these terms.

!"’ (when) is in a question sentence,
10. When ’
TIME is given as the answer type.

#%$'&(!#X1’ (how many X1) is
11. When ’in a question sentence, the expressions corresponding to X1 are extracted as the UNITFOCUS.

#)$*&+!'#

3. Answer detection’ (how many) is in a ques12. When ’tion sentence, NUM-EXP is given as the answer
type.

The system extracts linguistic expressions thatmatch the estimated expression type of the answer as estimated in 1 from the documents retrieved in 2. It then outputs extracted expressions as the answer. For example, for the question ”How large is the area of Japan?”, the system extracts numerical expressions as the answer from documents containing ’large’, ’area’,
and ’Japan’.

Estimation regarding the answer type in our systemis created based on manually created heuristic rules.
Some of the heuristic rules are shown here.



1. When ’ (what) X1 (what) X2’ is in a question sentence, WH-WH is given as an answer” (what year what month))type. (e.g. “
2. When ’X1   ’ (what do we call X1) is'#,’ (what country) is in a ques13. When ’tion sentence, SEM-COUNTRY is given as theanswer type.

-/.(’ (what spellings) is in a ques14. When ’tion sentence, SEM-SPELL is given as the answer type.
Our system uses a new method, which we call unit
estimation, to obtain a correct unit expression for theanswer. (This method is one main topic of this paper.) In this method, we gather sentences containing
expressions like ”UNIT-FOCUS + ’ ’ + ’numericalexpressions’ + ’unit expressions’” and extract the unit
expressions. We then eliminate unnecessary unit expressions by applying a statistical test based on a binomial distribution. Eliminated expressions are as follows:0in a question sentence, expressions corresponding to X1 are extracted as the NOUN-FOCUS.



Unnecessary expressions’ (where is X1) is in a question
3. When ’X1
sentence, expressions corresponding to X1 areextracted as the NOUN-FOCUS.



4. When ’ ’ (who) is in a question sentence,PERSON is given as the answer type.

’ (where) is in a question sentence
5. When ’’ (area), ’and the NOUN-FOCUS is not ’’ (location), or so on, ORGANIZATION isgiven as the answer type.








’ (where) is in a question sentence
6. When ’’ (comand the NOUN-FOCUS is not ’’ (organization), or so on, LOCApany), ’TION is given as the answer type.







2. Document retrieval

3.2 Estimation of answer type





124365 798;:=<?>@=ACB

DFEHGCI(1)

J

is calculated by the following equation
where
and
is a constant that is set based on experimental
results.

M

KCL

Extracting unit expressions from the corpus was proposed by us
in our previous paper [11].
In this study, we used the summation of 0 to , but the summacould also be used. When the summation of 0 totion of 0 to
is used, an expression having a lower value foris judged to
be an expression that occurs less often than the average occurrence
when UNIT-FOCUS co-occurs and it is eliminated. When the sumis used, an expression having a higher value
mation of 0 to
is judged to be an expression that occurs more often thanforthe average occurrence when UNIT-FOCUS co-occurs and the expressions other than such an expression are eliminated. We plan topatent these methods.

N

OQP+R

O

SUTZV[XOYP'RO

SUTWV6X

Proceedings of the Third NTCIR Workshop)
¢E}I(£ [¨ ©[© 0¤¥ 04©[¥ ª¤¦ ª ¦[¥ § §[Jy« £­¬¯® ¬°¬¬°¬°±±°±³²°´qis 533,366,720, the frequency of ’
’ is 11,887,. As shown
andin Table 1, our method can correctly eliminate ’ ’(minute) and ’ ’ (piece). When using our unit estimation, we do not need a dictionary for unit expressions. Another good point regarding the unit estimation is that it gives various expressions that appear in
the corpus. Unit estimation can also be used to construct a dictionary of unit expressions. Thus, our unit
estimation method offers various benefits.

Table 1. An example of using unit estimation
P(e)50128175 1.000000(meter)28
47050 1.000000(centimeter)
1125897 1.000000(millimeter)
1199618 0.999996(kilometer)2
538 1.000000(light-year)2955808 0.000000(minute)12744 0.998205(yard)
11865 0.999160(inch)
1 1625073 0.000000(piece)12146 0.998892(shaku)_] ^a`cG b
dfe_g
hi
j/k

m

l

K

\



3.3 Document retrieval

n ^o
p e_g
q
r

lated by

E}I

(3)

is the frequency of the UNIT-FOCUS
where freq
appearing in the corpus andis the number of all
characters in the corpus. In this study, we used newspaper issues from over a 10-year period [5] as the corpus for the unit estimation calculation.
An example of using unit estimation is as follows.
Consider the question sentence ’X’ (How much is the length of X?). In this’ (length) as the UNITcase, we extract a noun ’
FOCUS and gather candidate unit expressions using++ ’numerical expressions’ + ’unit expres”’ (meter), ’’ (censions’”. We obtain ’’ (millimeter), ’’ (kilometer), ’timeter), ’’ (light-year), ’ ’ (minute), ’’ (yard), ’’ (inch), ’ ’ (piece), and ’ ’ (shaku) as canfor each candidate anddidates. We calculate
obtain the results shown in Table 1. In this case,





e(g

K°µ6¶ 0

documents that
1. We first retrieve the top
have the highest scores from the equation



¡

hi

q

m

]+^ `YbDFEGCI

jk
n ^/o
r

de*g



l

p

¼½

·}¸[¹ º:C8»<t1 v

3 ¿8»ÀÁ3<3¿8»ÀÁ3<{Â@CÃ=Ä :4~ÆÅCÉ 3ÇÂ8»@ <{È Â@³È
term 3Ê ¹Å 
(4)Ä »¿8Z3<ºË#*_f#$*&!'
a

We extracted terms from a question sentence by using a morphological analyzer, ChaSen[6]. We eliminated terms whose parts of speech were postpositional
particles or something, then performed retrieval using
the extracted terms. We developed a method that enables us to use a full-size document for document retrieval. In general, we divide each document into small
passages before document retrieval. For example, we
divided each document into sets of three sentencesin our previous study [10, 11]. This overcomes theshortcoming of common information retrieval methods that cannot handle long documents for retrieval.
In a question-answering system, the answer often occurs in the terms extracted from a question sentence.
In this case, the condition that terms occur near eachother is very important, but existing information retrieval methods have not taken this condition into consideration. As a result, divided passages have generally been used for document retrieval. The information
retrieval method that we use for document retrieval
in our question-answering system, however, considers
the condition that terms occur near each other by using
a probability theory. Our method is as follows.

©

vu
798;:s<t1 wyx{z}| 8~<W8{< w 8 }8{<< w(2)
where Ey{I is the number of combinations when
we select  items from  items, \ is the number oftimes expression G occurs in the corpus, K is the number of times the unit expression G occurs in the patternof ”UNIT-FOCUS + ’  ’ + ’numerical expressions’ +’unit expressions’” in the corpus, and ?E;I is calcu}8{<1 freq 8{<

m

¾

Ì

Í
ÍyÎÏEHÌ¯¤ÍyIÌ ÌÎÏEÍyIÍ

where is a document, is a term extracted fromis the frequency ofa question sentence,occurring in a document ,is the numberof documents in which appears, is the totalÍÐ We used down-weighting methods [3, 9, 8] for the terms in Eqs.

(5), (9), and (12) which allowed us to use compound terms. We also
used synonyms in the EDR dictionary [2] to expand terms. When weused synonyms as terms in Eqs. (5), (9), and (12) instead of terms ina question sentence, we used the total frequency of the original term.
and its synomyms asÑZÒÀTÔÓX

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

HÕ G=\}ÖÀÍy×¢EÌÀI
K°µ6¶ 0 KÙ
K°Ú

ÌÌ°÷øsÍäEÍ4ù¤Íy±I

Í4ù £If we suppose that
= 0.5 when
, the above equation becomes similar to Eq.
(5). When the probability of pattern occurrenceis low, the pattern is more valuable, so we extract documents that have low scores from Eq.
(8). Since Eq. (5) is multiplied by  , weextract documents having high scores from Eq.
(5).

number of documents,
is the lengthof a document , and is the average length of, , andare constants
the documents.
that are set according to experimental results.

Íy±

Ø

úù

This equation was based on Robertson’s equation [13, 14]. This method is very effective
and we have used it for information retrieval[9, 12, 8].
2. Next, we re-rank the extracted documents using the following equation and extract the top
documents. The
extracted documents
are used in the answer extraction phase whichcomes next.

We call this information retrieval method that uses
the pattern of nearby terms based on probability theory
probabilistic near-terms information retrieval.

·}¸¤¹ =:C8»<Mßºà
1aÝÜ Þ~¯Ã Ä ¹ ÅâN á ßà Ð 8ãä»CÞHå63[8Z3[=3ã=< »¿ 8Z3ãº< <æÆçèéäê Ã NëÃ(5)Mßºà v¹
1Ü ìí Ã N ßà Ð°îï w N 8Z3ãº< Ä Å ãs»CÞHå[368Z3¤º3ãº<ñð¢»¿8Z3ãº<Ã

To detect answers, our system first generates candidate expressions for the answer from the extracted
documents. We initially used morpheme n-grams forthe candidate expressions, but this led to the generation of too many candidates. Therefore, we now use
only candidates that consist of only nouns, unknown
words, and symbols. Also, we use ChaSen so that we
can use morphemes and their parts of speech. Hiragana
characters in personal names having the suffix ’’ (Mr.) were sometimes incorrectly recognized as
particles, so we added expressions
whose first word’ (Mr.), and whose
are nouns, whose last word is ’length is less than 10 characters to the candidates as
exceptions. We call this addition use of exceptional
person candidates. Also, we added expressions surrounded by Japanese brackets ’  ’ and ’  ’ to the candidates because such expressions were often correct
answers [15] and these expressions contain words suchas particles.
Our method of judging whether each candidate is acorrect answer is to add up the score for each candidate
under the condition that the candidate is a nearby termwith the score based on heuristic rules according to theanswer type. Our system selects the candidates having
the highest total points as correct answers.
We developed two methods for calculating the scoreunder the condition that the candidate be near theterms of each candidate  . These are as follows.

K°µ6¶ JKÛµ6¶ J(6)

òôó 1a26345 3Ïõ ò ãs»CÞHå[368Z3[=3< »Á¿ 8Z3< >sB¨

3.4 Answer detection





(7)

Ì is a document, ö is a set of terms, andÌ÷ø=ÍäE;Í4ù°yÍy±I is the distance between Í4ù and Íy±(using the number of characters as the distance)
where Ì÷ø=ÍäE;Í4ù°yÍy±I = 0.5 when Í4ùú£Íy± . ÌÎÏE;ÍyI is
the number of times Í occurs in all documents,
 is the total number of characters of all documents, and ûYµ6¶ EÍy±I is a function of Íy± which isJadjusted according to experimental results.
Equation (5) can be deduced when ûYµ6¶ E;Íy±°Ic£
Jù through a probability theory. We consider
the probability of a pattern of the occurrenceof terms. First, we select Í4ù . The occurrenceµZüýþÙ 0yÿ . Next, we consider the
probability of Í4ù isother terms Íy± . The probability of each of theseterms occurring within the distance Ì°÷øsÍäEÍ4ù¤Íy±IµZüýþÙ J[ÿ  Therefore, the probto Í4ù is ±³Ì°÷øsÍäEÍ4ù¤Íy±I
whereability of an occurrence pattern is·}¸[¹ =: w M 8 ¸ <!#"¹1a Ä Å N á ßà Ð 8ãä»CÞÁå[3[8 ¸ 3ãº< »Á¿ 8Z3ãº< < æ çèÁé ê Ã NëÃv ßºà w N ¹
1 N Ð î ï 8Z3ã=< Ä Å ãs»CÞHå[368 ¸ Á3ãº<ñð?»¿8Z3ãº<Ã»Á¿ 8Z3[ä< ºß à á M 8ãs»CÞHå[368Z3¤º3ãº< »Á¿ 8Z3ãº< < (8)Ã N Ð  Ã Nx  ÃMStrictly Ã Në N
X ï Ã ê Ã  Ã Në . When
speaking, the probability is RÝPTHRÝP ï  ê
we use Taylor’s formula ( TRP X   R¢P  T  R¤X ), the probÑ yÓTÔÓR Ó äX ï  ê Ã Në .
ability becomes 4
$òôó 126345 3õ ò yãä»CÞÁå[3[8 ¸ 3< »¿ 8Z3< >äB
We plan to patent this method also.

(9)
(10)

(11)

Proceedings of the Third NTCIR Workshop·}¸[¹ º: w N 8 ¸ <v %&"
1 N ßà Ð @ %ï #" Ã w ê('N  Ã Në î ï w N 8Z3ã=< Ä ¹ Å »¿8Z3ãº<Ã

NE data. We made six NE extraction systems
for these six categories. Each system judges
whether a candidate expression belongs to a certain category. For example, the NE system for
LOCATION judges whether a candidate is a
LOCATION. In our system, one candidate canbelong to more than one NE category. We used
the support vector machine for machine learning [1, 4, 17]. When a candidate is judged to
be a DATE, we classify it as a TIME because
our question-answering system does not use a
DATE category.

(12)òôó 126345 3Ïõ ò yãä»CÞHå63[8Z3[=3< »Á¿ 8Z3< >sB

(13)

ö

is a set ofwhere  is a candidate for the answer,)is the distance between  andterms, and.
is the number of times occurs in all theis the total number of characters in all
documents,
is a function of which
the documents, andis adjusted according to experimental results. +*,.is a/ constant set according to experimental results.
1032 *+,.) is a very similar method to that of
Eq. 5. This method also has a function to select a
candidateby using nearby terms.
/
1032 *+,.) is a heuristic function using the condition that the answer is near certain terms.
Next, we describe how the score is calculated basedon heuristic rules using the estimated answer type. We
used many heuristic rules to award points to candidates
and used the total score as the deciding value. Someof the heuristic rules are shown below.

Ì÷ø=ÍäE C¤Íy±I
Íy± ÌÎÏE;ÍyIû µ6¶ J EÍy±IÍ

CG ¶ 0 E sIÍy±

Our method for judging whether a candidate
matches LOCATION, ORGANIZATION, PERSON, ARTIFACT, or TIME uses two other powerful techniques. One improves the recall rate
and the other improves the precision rate. The
first technique is that we did not use the morphemes previous to and succeeding the analyzed
NE as features in the NE judgment. This improves the recall rate. The second techniqueis that we used heuristic rules to eliminate any
wrong NEs after the NE judgment by machinelearning. For example, we used the following
rules for elimination.

K ¶J

ª

CG ¶ J E sI(a) A candidate whose last character is ’ E ’(system), ’ ’ (company), or so on iseliminated in the case of PERSON classification.



1. We add -1,000,000 to candidates that consist of
unnecessary
characters such as ’ 4 ’, ’ 5 ’, and ’
687:9<;<=’ (has a related document in).

(b) Acandidate whose first characters is ’ FG’ (because), ’’ (thing), or so on, iseliminated in the case of PERSON classification.

/

2. We add 1000 to candidates that match one of theestimated answer types (LOCATION, ORGANIZATION, PERSON, ARTIFACT, or TIME)
when they are numerical expressions + UNITEXP or estimated unit expressions; when they(year) X2(month) X3 > (day)’are ’X1@>
where the question sentence contains ’ ?
’ (what is your birthday); when they havethe meaningof jobs where the NOUN-FOCUS=’ (work), ’ BDC ’ (job), or so on; andis ’ A
when they have Noun X where the question sentence contains ’ (what) + Noun X’, and so on.



f!("



(c) A candidate containing words other thanwords we selected in advance is eliminatedin the case of TIME classification. (Wefirst selected ’ :H ’ (past), ’ I<J ’ (half),’ K ’ (summer), and so on as good words
for TIME and eliminated candidates containing other words.)







The rules shown here represent a very small
sample of our heuristic rules. These rules weremade by manually checking a large body of results from our NE system using a large amountof raw data as input data.

This is the most fundamental rule, and we adjust
it so that a candidate that is a good answer willalmost always satisfy the rule.
Named entity (NE) extraction techniques areused to judge whether a candidate matches LOCATION, ORGANIZATION, PERSON, ARTIFACT, or TIME. We use a machine learning approach for our NE extraction system,
and CRL (Communications Research Laboratory) NE data was used for training in the machine learning [16]. Only LOCATION, ORGANIZATION, PERSON, ARTIFACT, TIME,
and DATE were used as categories in the CRL§

3. When WH-WH is one of the estimated answer
types, we add 1000 to candidates that match a
L

We also used other methods to improve the recall rate.
Katakana expressions whose parts of speech are unknown words areclassified as LOCATION, ORGANIZATION, PERSON, or ARTIFACT. A kanji character + any character + ’ MON ’ (Mr.) is classified
as PERSON. Candidates that are in our dictionary of companies areclassified
as ORGANIZATION.
PThis raw data had no NE tags, and was part of the documentsthat were used for the QAC (Mainichi newspaper 1998, 1999).

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

pattern of ’X1 P1 X2 P2’ which uses the interrogative pronoun ’ (what) P1 (what) P2’.





=

’ (work), ’
4. When the NOUN-FOCUS is ’ A
B<C ’ (job), or so on, we add 1000 to candidates
that have the meaning of jobs. The judgment as
to whether a candidate has the meaning of a job
is made by using the EDR thesaurus [2].
5. When SEM-COUNTRY is one of the estimatedanswer types, we add 1000 to candidates that arein our dictionary of countries which includes thenames of almost all countries.
6. When SEM-SPELL is one of the estimated answer types, we add 1000 to candidates that consist of only hiragana characters and the symbol’ Q ’.
7. When a question sentence includes ’ RDSDTDU’ (what prefecture), we add 1000 to candidates that are location expressions and whose
suffix is ’ R ’, ’ S ’, ’ T ’, or ’ U ’ (prefecture).



8. When the NOUN-FOCUS is ’ VXW ’ (masterpiece), ’ WZY ’ (literary work), or so on, we add1000 to candidates that are surrounded by brackets ’  ’ and ’  ’.
9. When NUM-EXP is one of the estimated answer
types, we add 1000 to candidates that consistof only numerical expressions and unit expressions.

# ;\[ 10. When a question sentence contains ’’ (what symbol), we add 1000 to candidates
that consist of symbols.

[

11. When a question sentence contains ’ ]D^D_<_ ’ (what is the telephone number), we add1000 to candidates that consist of numerical expressions and the symbol ’ Q ’.

(

12. When a question sentence contains ’ (what) +Noun X’, we add 1000 to candidates having theNoun X.



13. When a question sentence contains ’ (what)+ `a> (day of the week)’, we add 10,000 to
candidates that are ’ >b`c> ’ (Sunday), ’ D`c> ’(Monday), or so on.

Our system has additional functions that are used
after answers are selected based on the scores. The
first one is the compiling of similar answers. Our
system compiles answers that are part of other answers and the difference in their scores is less than
1000. The compiling is done by eliminating answers
other than the longest one. We call this method answer compiling. The second function expands foreign personal names. For example, when we obtain XXXXX (katakana characters) as an answer for

PERSON, we gather ’YYYYY Q XXXXX’ and the’YYYYY Q XXXXX’ that has the highest frequency
is selected as the answer. We call this method nameexpansion.
Now, we will describe our strategy for the QAC. In
task-1, where the MRR is used in the evaluation, oursystem outputs the top five answers. In task-2 andtask-3, where the mean of the f-measure (MF) is usedin the evaluation, our system outputs the top answer. d
In task-3, we use as the second question sentence theconnection of the first and second question sentence
where an interrogative pronoun, adjective, or adverb
in the first sentence is changed to dummy symbols.

¦

4 Experiments0«

Our experimental results are shown in Table 2.
Experiments S2 to S11 were performed after the formal run to confirm the effectiveness of methods usedin our system. S4 is the method by which documents
are divided into paragraphs after using Eq. 4 and before using Eq. 5. S5 did not use unit estimation at
all and did not use any estimated unit expression. S6
did not use a statistical test to eliminate unnecessary
unit expressions and used all the candidate unit expressions gathered from the corpus as unit expressions.
S10 is the method using as the second question theconnection of the first question sentence, the highestranked answer of the first question sentence, and thesecond question sentence for task-3. S11 is a selecttwo method that outputs the two top answers. The
experiments for S12 to S14 were done to obtain thehighest score using the results of S2 to S10. We useda one-sided T-test to statistically confirm the effectiveness. We used S1 as the base method for comparison. The figures denoted by ’ e ’ were superior to thebase method result at a significance level of 0.05 andthe figures denoted by ’  ’ were inferior at a signifi,
cance level of 0.05. The parameters were,
, andf whena/ term was a verb and otherwise was 1. We used
1032 *+,.) for other than S7 and S8. The heuristic
rules in our system were manually devised by referring to the questions in a dry run and about a hundred
questions constructed by us. We used documents retrieved from Mainichi newspaper articles (taken fromthe issues over two years: 1998 and 1999) [5] whichKÙ*£ ¬{® ¬¬¬°¬{ù KÚ £ ±³¬
ÍCG ¶ 0 E sIg

KCL£ ¬{® ´
ûYµ6¶ J E;ÍyI£ ¬{®

h We explain MRR and MF in Section 4
Our system has no function for selecting multiple answers, so
we use only one answer to maintain good precision. We call thisstrategy the select-one method.
Our submitted systems obtained the highest MF in Task-2. In
Task-1 and Task-3, our systems obtained the second highest MRR
and MF. (The highest MRR in Task-1 was 0.608. The highest MF
in Task-3 was 0.187.) This indicated that our system worked reasonably well. Therefore, our results from the comparison of several methods in our system can be used for future studies regarding
question-answering systems.

MzProceedings of the Third NTCIR Workshop

S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14

Table 2. Experimental results
MethodTask-1(MRR)
Submitted system
0.524
No use of name expansion
0.531
i +fkjNo use of answer compiling
Documents divided into paragraphs
0.513
j
l
No use of unit estimation
No use / of statistical test for unit estimation 0.521
Use of / 1032 3*+,.- with *+,.0.521
Use of 1032 *+,.- with *+,.0.513
No use of exceptional person candidates
0.526
Use of the answer of the first question—
Use of the select-two method—
Combination of S2 and S9
0.532
Combination of S2 and S8—
Combination of S2, S4, S7, and S10—¬¯® Û´¬¯® ¬{ùCG ¶ JCG ¶ JK ¶ J £)¬{® ´´K ¶J )£ ¬{® ´´°´

were the documents that we extracted answers from in
the QAC task. We did not use any other documents for
document retrieval. The QAC had three tasks (Task-1,
Task-2, and Task-3). Task-1 and Task-2 used the same
200 questions. Task-3 used 40 sets of two questions.
The second question of each set was related to the firstquestion. For example, the first sentence would be:mcn” (How many people are
”>
there
in
Japan?)
andthesecondsentence would be ”op” (What is the size of the area?). In
Task-3, only the second sentences were evaluated. In
Task-1, we could submit five or less answers for eachquestion and MRR was used for evaluation. In Task1, the first correct answer among the five answers wasused for the MRR calculation. In the MRR, we obtained a score of 3qr2 when the r-th submitted answer
was correct. In Task-2 and Task-3, we could submit
any number of answers for each question and the MF(mean of the f-measure) was used for evaluation. In
Task-2 and Task-3, when the question had more than
one answer, we had to submit the corresponding multiple answers to obtain full marks. The recall rate in
the f-measure was the ratio of the number of correct
answers submitted to the number of correct answers.
The precision rate was the ratio of the number of correct answers submitted to the number of submitted answers.
We found the following from the experimental results.

q Our method of probabilistic near-terms information retrieval sometimes provided better results(c.f. S1 and S4 in Task-1 and Task-2). However,
we could not confirm its effectiveness throughthe T-test.

 / 1032CG%*,.- ¶ E&sI was good in Task-1.
/
1032C%G *,.- ¶ J0 &E sI was good in Task-2 and Task-3(c.f. S7 and S8).

 Use of name expansion and exceptional person

ù

MM

0[0

We examined the results of using unit estimation in Task 2 more¬¯® fÛ±+l Ú

confirmed by statistical testing in Task-1 (c.f.
S3).

+! $_& '! $_& (be effective by statistical testing (the T-test) (c.f.
S5). Our method of using statistical test for unit
estimation slightly increased the performance ofunit estimation (compare S1 and S6).

¬¯® f+i¯ù j

0.361
0.378
0.382
0.361
—
0.341
—
0.385
—

Task-3
(MF)
0.167
0.192
0.167
0.250
0.167
0.167
0.225
0.167
0.167
0.225
0.213
—
—

 Answer compiling was also effective, as was# Our method of unit estimation was confirmed to

Task-2
(MF)
0.364
0.366
0.339
0.355

candidates lowered the scores (c.f. S2 and S9).

 In Task-3, we applied a new method of using theanswer of the first question because our system
obtained higher scores for the top answer thanwe estimated. This addition greatly improved
the score (c.f. S1 and S10).

0yJ

 We created a tune-up system for each task using the results of S2 to S10 (c.f. S12, S13, andS14). The scores slightly increased in Task-1
minutely. There were fourteen questions where our system judgedthat a question has no unit expression and its answer type is a numerical expression. Among them, our system using unit estimation (S1)obtained more than zero score in eight quetions. When our system
did not use unit estimation (S5), it obtained more than zero scorein only three quetions. (S1 always obtained correct answers when
S5 obtained correct answers.) When our system did not use statistical test for unit estimation (S6), it obtained more than zero score inseven quetions.(We compared S1 and S6 and found that there were
two question when the answer of S1 was correct and the answer of
S6 was incorrect and there was one question when the answer of S6
was correct and the answer of S1 was incorrect.)In the formal run, we thought our system’s performance was
low and adding the answer of the first question decreased the score.

MN

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

and Task-2 and greatly increased in Task-3. The
improvement in Task-3 was significant according to the T-test.

 In Task-3, we observed very few significant dif-ferences. This may have been due to the small
number (40) of questions in Task-3.

 In S11, we used the select-two method. In Task-

2, the score decreased as we estimated. However, we did not estimate the increased scoresin Task-3. The performance of our system for
Task-3 was not particularly good. We foundthat when the system performance was poor,
the select-two method could be superior to theselect-one method.

0©

5 Conclusion
Our question-answering system uses several new
methods. One is unit estimation. This method is very
useful when the answer is a numerical expression anda question sentence does not include any unit estimation. This method can be used to estimate a unit expression for the answer by using a statistical test andcorpus data and thus improves the results for suchquestions. Another new method is the probabilistic
near-terms information retrieval. This method enables
us to use full-size documents in document retrieval
without dividing the documents into passages. Thismethod is useful when the answer is not within theparagraph where relevant terms occur. We confirmed
the effectiveness of the unit estimation by using a statistical test (T-test) and found that probabilistic nearterms information retrieval improved the performancein Task-1 and Task-2.
In our previous question-answering system, weused other original methods based on syntactic structures and rewriting rules (paraphrasing) [10, 11, 7].
However, we did not use these techniques in the QAC
because they are time consuming. In our future work,
we hope to use these techniques to improve our systemwith regard to the QAC.
The QAC has drawn many researchers into the fieldof question-answering systems and has helped to improve the question-answering performance that can beattained. We hope to see further rapid progress in thisfield and the future development of artificial intelligence using such systems.

References[1] N. Cristianini and J. Shawe-Taylor. An Introduction
to Support Vector Machines and Other Kernel-based
Learning Methods. Cambridge University Press, 2000.

MÐWhen we used the select-two method for S14 of Task-3, we
obtained an MF of 0.279 which was lower than the score for S14.

[2] EDR. EDR Electronic Dictionary Technical Guide.
EDR (Japan Electronic Dictionary Research Institute,Ltd.), 1993.
[3] S. Fujita. Notes on phrasal indexing JSCB evaluation
experiments at ntcir adhoc. Proceedings of the NTCIR
Workshop 1, pages 101–108, 1999.
[4] T. Kudoh. TinySVM: Support Vector Machines.
http://cl.aist-nara.ac.jp/ taku-ku// software/TinySVM/
index.html, 2000.
[5] Mainichi Publishing. Mainichi Newspaper 1991-2000,
2000.
[6] Y. Matsumoto, A. Kitauchi, T. Yamashita, Y. Hirano,
H. Matsuda, and M. Asahara. Japanese morphological analysis system ChaSen version 2.0 manual 2ndedition. 1999.
[7] M. Murata and H. Isahara. Universal model for paraphrasing — using transformation based on a defined
criteria —. In NLPRS’2001 Workshop on Automatic
Paraphrasing: Theories and Applications, 2001.
[8] M. Murata, Q. Ma, and H. Isahara. High performance
information retrieval using many characteristics andmany techniques. Proceedings of the Third NTCIR
Workshop, 2002. (to appear).
[9] M. Murata, K. Uchimoto, H. Ozaku, Q. Ma,
M. Utiyama, and H. Isahara. Japanese probabilistic
information retrieval using location and category information. the Fifth International Workshop on Information Retrieval with Asian Languages, pages 81–88,
2000.
[10] M. Murata, M. Utiyama, and H. Isahara. Question
answering system using syntactic information. 1999.
http://xxx.lanl.gov/abs/cs.CL/9911006.
[11] M. Murata, M. Utiyama, and H. Isahara. Question answering system using similarity-guided reasoning. Information Processing Society of Japan, WGNL 2000NL-135, pages 181–188, 2000.
[12] M. Murata, M. Utiyama, Q. Ma, H. Ozaku, and H. Isahara. CRL at NTCIR2. Proceedings of the Second
NTCIR Workshop Meeting on Evaluation of Chinese& Japanese Text Retrieval and Text Summarization,
pages 5–21–5–31, 2001.
[13] S. E. Robertson and S. Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of theSeventeenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 1994.
[14] S. E. Robertson, S. Walker, S. Jones, M. M. HancockBeaulieu, and M. Gatford. Okapi at trec-3. In TREC-3,
1994.
[15] Y. Sasaki, H. Isozaki, H. Taira, T. Hirao, H. Kazawa,J. Suzuki, K. Kokuryo, and E. Maeda. SAIQA: A
Japanese QA system based on a large-scale corpus.
IPSJ-WGNL 2001-NL-145, 2001. (in Japanese).
[16] K. Uchimoto, M. Murata, H. Ozaku, Q. Ma, and
H. Isahara. Named entity extraction based on maximum entropy model and transformation rules. Proceedings of 33rd Annual Meeting of the Association of
the Computational Linguistics, 2000.
[17] H. Yamada, T. Kudo, and Y. Matsumoto. Japanese
named entity extraction using support vector machine.
Transactions of Information Processing Society ofJapan, 43(1):44–53, 2002.