Proceedings of the Third NTCIR Workshop

RitsQA: Ritsumeikan question answering system used for QAC-1
Jun’ichi FUKUMOTO Tetsuya ENDO Tatsuhiro NIWA
Ritsumeikan University
1-1-1 Noji-higashi, Kusatsu-shi, Shiga 525-8577, Japan
fukumoto, t endo, t niwa @nlp.cs.ritsumei.ac.jp

Abstract
In this paper, we describe RitsQA: Ritsumeikan
question answering system. In our QA system, input query sentence is firstly analyzed using question
patterns consist of description of NE elements, POSof words and surface expressions. Then, query typeand some clue words, which are used for document
retrieval, are determined. For answer extraction, oursystem uses word distance between answer candidate
and clue words of retrieved documents.
We have participated Task 1 and Task 2. (it wasnot formal one because we are a member of QAC taskorganizer.) The results were not so good than we expected. However, this experience was helpful to develop QAC task.
Keywords: Question answering, word distance, query
pattern analysis.

1 Introduction
We have started Question answering task called
QAC as one task of NTCIR Workshop 3 as a member of QAC task organizer. Participation to QAC will
be very helpful for organization of QAC tasks and development of our QA technology.
We have been working on development of Named
Entity extraction tool called NExT [7] with Mie university in order to open this NE tool for NLP researcher and other research groups. Applicability ofthis tool is also our purpose of this participation.
We participated Task 1 and Task 2 of QAC although
it was not formal one because we are a member ofQAC task organizer. For Task1, we have developedQA system based on word distance for answer extraction. For Task2, we used the same system as a system
for Task 1. That is, the best answer in Task 1 will be
the answer for Task2.
In this paper, we describe RitsQA: Ritsumeikan
question answering system used for QAC-1. Our QA
system analyzes input query sentence using question
pattern consists of NE elements, POS of words andsurface expression. After this analysis, query type and© 2003 National Institute of Informatics

some clue words that are used for document retrieval
are determined. For answer extraction, our system isbased on word distance from retrieved documents.

2 Query patterns
At the first step of our QA system, query type of
input sentences is analyzed. We assumed 5 types ofquery based on surface expression as follows: Who type (“dare” in Japanese)

This type is a query to get a person name.

 When type (“itsu” in Japanese)

This type is a query about date or time.

 Where type (“doko” in Japanese)

This type of a query is used to get information oflocation name or organization name.

 What type (“nani” in Japanese)

This type of a query covers a variety of query elements such as person name, organization name,location name, and so on.

 How type (“donokurai” in Japanese)

This type of a query is used for some numericexpressions such as information of money, ratio(percent), length, weight, speed and so on.
In order to get answers correctly, it is necessary tounderstand what is required in a query sentence. We
have prepared about 70 query patterns to analyze answer type information. A query pattern consists of thefollowing elements.

 Named Entity which is indicated by PERSON (person name),  DATE (date), ORGANIZATION (organization name) andso on Table 1 summarizes all the type of Named
Entity 1 which are used in our QA system.

1 As you can see, there NE elements beyond the definition ofNamed Entity used in IREX [3] and MUC [4]. We used these type ofNE elements although all the elements are not tagged in the current
implementation of our QA system.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

 a word which are indicated by its part of speech( noun  , verb  and so on) surface expressions

Table 1. NE tags

 PERSONtag

 ARTIFACT ORGANIZATION
 LOCATION DATE TIME MONEY PERCENT UNIT DISTANCEtag type
person nameartifact
organization namelocation namedatetime expression
money
percentage
unit namedistance expression

Each query pattern has its answer type of information. Example of query patterns and their answer types
which are in the brackets shown as follows:  noun ( PERSON )  verb  ( PERSON )  noun ( TIME or  DATE )  verb  ( TIME or  DATE )  noun [   ] ( TIME or DATE )  noun [ !" #$! % ]( TIME  or DATE )  verb% [ & '(&) ] ( TIMEor DATE  )  noun'*+ ( LOCATION )  verb '*+ ( LOCATION )  ARTIFACT'-, ( ARTIFACT )  PERSON-, ( PERSON )  NAME'-, ( PERSON )  ORGANIZATION-,
( ORGANIZATION  )  PERSON/.&0-, ( PERSON )  PERSON-,21345.&0 ( PERSON )  NAME'-,21346.&0 ( PERSON )  ORGANIZATION 7, 18 49.:0
( ORGANIZATION  )  LOCATION  , 14;. 0( LOCATION  )  ARTIFACT  , 14 . 0
( ARTIFACT  )  noun <, 1= 4  ARTIFACT( ARTIFACT  )  noun ',=1>?4  ORGANIZATION( ORGANIZATION  )  verb @=, 17 4  PERSON( PERSON  ) LOCATION  noun( LOCATION ) ,  verbA,  ORGANIZATION
( ORGANIZATION)  noun-BC"DE ( number )  noun'*FE ( number )  noun-BC ( MONEY )  noun-, [ GH5IKJ ..] ( MONEY )  nounL%, [ M>NPOQJRS'TU@VW ..]( UNIT )  noun [XZYH\[ZY^]Y ..] *-F3

( UNIT )  nounZ*/ZF 3 [XZYZ_[HY`a]HY ..]( UNIT )  noun-,)b ( PERCENT )  nounc) [0dfe ] ,)b ( PERCENT )  noun g  noun Zh*gF`( DISTANCE  )3 Analysis of queries
At first, input query is morphologically analyzed
using ChaSen [5] system. Then, NE elements are detected using some simple patterns and clue words fortext retrieval are extracted from the query sentence.
Clue words that will be extracted are a series of noun,verb, adjective, adverb, number expression and unknown words. NE elements are also used for cluewords. After that, answer type is detected by matching
the analyzed sentence with query patterns.

Proceedings of the Third NTCIR Workshop

4 Text retrieval using query words
We utilized full text search system “NAMAZU”[6]for document retrieval from newspaper database. In
our QA system, all clue words extracted from a query
sentence are used for Namazu index words. The retrieved documents are order in some probability obtained by NAMAZU system. If there is a clue wordthat does not match with documents during document
retrieval, such a word will be paraphrased to the otherexpression using thesaurus dictionary. We utilized asmall size (about 100 words) dictionary that is madeby ourselves.

5 Answer extraction
In our QA system, answer extraction is based onword distance between answer candidate and cluewords. For each answer candidate, word distance withall clue words are calculated and the sum of the distances will be score of the answer candidate.
At first, top ten documents retrieved by NAMAZU
system are morphologically analyzed using ChaSen
system and Named Entity elements in the documents
are tagged by NExT system [7]. NExt system tags person name, organization name, location name, percentage and money, therefore answer candidates will belimited to these elements in our current implementation.
For each answer candidate who is required answer
type, word distance between the candidate and cluewords are calculated. For the following query sentenceij)Nand “”) are clue words extracted from query
sentence. In the clue words, our system recognize topical word as an important one. In the above case, “” can be recognized as an important one and theothers are not. For each answer candidate, reciprocal
number of the ranking value of word distance between
the candidate and a clue word will be its score. For an
important clue, the score will be double in our current
implementation. Table 2 shows scores for each answer candidate with clue words. The number attached
to clue words in the table means the number of appear1” means the first word
ance in the document. “” and “2” means the second one.
“l%[ij Nikj"N

Table 2. Sample score (each clue words)j"N 2r s tu l&2[ ij"0.5N 1 ik0.33
s  110.5
Á>ÂÃ Ä 0.67 0.33
1

And the sum of score for clue words will be thescore of an answer candidate as shown in Table 3. Per” will be chose for the answer ofson name “
query sentence.

rs&tu

Table 3. Sample score (sum)order12
3ikj"N -l [ mon &p%q”“(Who is the president of SONY?)
Figure 1 shows a sample text which include the answer of the query sentence.

r stu
K& N l [ w
3vyx"z%p{kJ%/|}>16&~>)EO/N  /qv  N  -"/ij"N  s    ¡ ¦ r§ k¨-&©ª« ¬"­&| ®.&¯'¢ )£ O±[%°&¤²³µ¥´/À ¶·¸ 3¹>º m&)Å »- ­kq¼ ½¾  ½`¿
Ê ! À Á>Ë ÂnÃ>Ä 1±£Ì[ ¾  E|oÍ&¤Î /À !/vÇqÆikÈj"ª&N ÉÐÑ&Ò&Ñ±ÓCÒ5Ô  · Ë À !kÕ Ö"×)I±Ø  °²³ Ï´Ùv o N /Ú oÛÏÝÜÞ'ßoàâá Ó Qãä q
Figure 1. Sample documentrs&tu ”, “s " ”
In this Figure, boxed parts (“and “ÁÂ&Ã Ä ”) are person name which are recognized by NExT system and underlined parts (“l%[ ”ij&Nr stus Á>ÂoÃ Äanswer candidate

score
2.83
2.5
26 Results
Task1
The results were 19.4 marks out of 197.0 in TASK1and Average Score was 0.099. Table 4 shows the output of scorer and Table 5 shows the number of correct
queries in the five orders.

Table 4. Results of Task 1
Question
200

Answer497

Output551

Correct
43

Recall0.087

Precision
0.078

F-measure
0.082

MRR0.099

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

Table 5. The number of correct results
the order
firstsecond
thirdforth
fifth

number of correct one
1315
10814

Task2
The results was 9.9 marks out of 197.0 in TASK2and Average Score was 0.066. Table 6 shows the output of scorer.
Table 6. Results of Task 2
Question
200

Answer497

Output200

Correct
13

Recall0.026

Precision
0.065

F-measure
0.037

MRR
0.066

7 Conclusion
In this paper, we described our QA system that consists of query pattern analysis for answer type detection and clue word extraction, document retrieval using NAMAZU system, and answer extraction based onword distance.
According to Formal Run evaluation, our QA system could not get better results but query pattern analysis and document retrieval modules are not so bad.
We have roughly analyzed performance of these modules using Formal Run evaluation data. In the top five
retrieved documents, almost 60% of them include correct answer. That means major problem of our QA
system is answer extraction using NE system. Current NE system extracts only person name, organization name, location name and so on. In order to robust
QA system, it is necessary to handle the other type ofinformation. Improvement of answer extraction basedon word distance is our future work. The other modules are also points of improvement.

References[1] J. Fukumoto and T. Kato, An Overview of Question andAnswering CHallenge (QAC) of the next NTCIR Workshop, in Proceedings of the Second NTCIR Workshop
Meeting 2001.
[2] NTCIR (NII-NACSIS Test Collection for IR Systems)
Project http://research.nii.ac.jp/ntcir/index-en.html[3] Information Retrieval and Extraction Exercise (IREX)
http://cs.nyu.edu/cs/projects/proteus/irex/[4] Proceedings of 7th Message Understanding Conference(MUC-7), DARPA, 1998.
[5] ChaSen URL http://chasen.aist-nara.ac.jp/.
[6] Full-text search engine intended for easy use
http://www.namazu.org/index.html.en[7] Named Entity Extraction Tool (NExt) Homepage,
http://irmscher.shiino.info.mie-u.ac.jp/next/