Proceedings of the Third NTCIR Workshop

SiteQ/J: A Question Answering System for JapaneseSeungwoo Lee
Gary Geunbae Lee
Dept. of Computer Science & Engineering, POSTECH
San 31, Hyoja-Dong, Pohang, South Korea, 790-784{pinesnow, gblee}@postech.ac.kr
Abstract
This paper describes our Question Answering system
participated in QAC Task1 of NTCIR3 and reports
the results with some observations. Throughanalyzing the previous TREC QA data, we definedpassage and developed passage selection method
suitable for Question Answering. Using LexicoSemantic Patterns (LSP), we identify answer type ofa question and detect answer candidates without any
deep linguistic analysis of the texts. Answer
candidates are ranked by passage scores anddistances between answer candidates and matchedterms. As a result of better engineering, our systemshowed excellent performance when evaluated bymean reciprocal rank (MRR) in NTCIR 3.
Keywords: question answering, passage selection,
lexico-semantic patterns

1. Introduction
Question Answering (QA) is a task to retrieve
answers rather than documents in response to aquestion and an effort to come nearer to true
information retrieval. Since the test collection was
prepared through TREC-8 [16], many researches
were progressed vigorously. We also jumped into thistask, participated in the last TREC QA task (TREC10) and achieved relatively good performance [10].
Encouraged by the result, we decided to participate in
the
QAC
task
of
NTCIR
3
(http://research.nii.ac.jp/ntcir/workshop/qac/cfpen.html) and apply the similar but more advanced
techniques to Japanese Question Answering.
Almost all QA systems use document retrieval
systems and even passage retrieval techniques toreduce search space to small document snippets.
Passage retrieval techniques were originallyintroduced to improve the precision of document
retrieval systems. However, in the view of question
answering, passage retrieval may show different
characteristics. Usually the goal of question
answering task is to find out an exact answer (oranswer string) to a question, whereas the goal of
document retrieval task is to find out as many
relevant documents as possible to a requested topic.
Relevance information to a topic tends to occur
across several sentences in a document, but query© 2003 National Institute of Informatics

terms of question and its answer tend to occur within
one sentence or two. Thus we need to define new
passage and develop a method suitable for question
answering systems
One of main differences between the previous
TREC QA tasks and NTCIR QAC task is exact
answer constraint on returned answers. Returned
answers containing any dummy word will be judged
as incorrect. This requires methods that candetermine the answer type of a question and detect
exact answers belonging to the answer type moreaccurately. These are achieved by using LexicoSemantic Patterns in our system.
The remainder of this paper is organized asfollows. In section 2, we look into some previous
researches on Question Answering. We explain how
passages are defined and selected for further
processing in section 3. In section 4, we describe
what Lexico-Semantic Patterns are, how answer typeof a question is determined, and how answer
candidates are detected using these Lexico-Semantic
Patterns and some scores.

2. Previous Work
Many question-answering systems have beendeveloped since TREC QA test collection wasconstructed. They employed various techniques toanswer a question from the test collection. We will
review some of them in this section.
Identification of answer type of a question wasused to constraint semantic type of potential answers
and can narrow search space considerably. Answer
type was defined based on ontology such as WordNet[7] or Named Entity labels used in MUC-7 [8]. [7]identified answer type of a question through parsing,while [8] identified through learning several features
based on Maximum Entropy model.
Another technique to narrow search space forpotential answers is passage retrieval (or selection).
Passage retrieval techniques were initially developed
for high precision of information retrieval systems[2][6][13][19][21] and also used for selecting
passage that might contain potential answers[3][7][11]. Many QA systems defined their own
passage (sentence, paragraph, topical segment, etc)
and developed various ranking measures. However, it
was not yet evaluated which passage definition andranking method is most effective for Question

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

Answering systems.
To detect answer candidates to a question, manysystems utilized Named Entity recognition
techniques. This was achieved by pattern matchingsuch as regular expression [3] or external NE taggers[7]. To justify answer candidates, [5] employedabductive reasoning with deep analyses of texts,
while [7][14] used only surface patterns and showed
good performance in the latest TREC QA evaluation.

3. Query-based
Selection

AnswerPassage

Most question answering systems utilize existing
document retrieval systems to reduce search space toseveral documents. However a document is usually
not suitable for detecting answer candidates within
itself because it is too long and contains too much
extra information and various topics. By analyzingquestions and their answers used in the past TREC
QA tasks, we could find that an answer occurs
comparatively near to keywords matched to a queryin a document. This means that we can focus on only
short part of a document rather than the whole toextract answer candidates and thereby considerably
reduce the computational load. Considering this fact
we employ passage selection method after retrieving
documents through a probabilistic document retrieval
system, POSNIR[10], like several previous
researches ([3][7][11]).

3.1

Definition of a Passage

Passage selection or passage retrieval techniques
were originally used to improve the precision,
especially top-level precision, of information
retrieval systems [9]. These techniques can bedivided into two different approaches: static anddynamic passage retrieval. In static approach, eachdocument is segmented into several passages inindexing time and each passage, not document, is
indexed. Therefore, a retrieved passage is always
definite regardless of a request [6][13][19][21].
However, this approach may often degrade theperformance to some queries since it is difficult tosegment a document into passages equally suitable toall the queries. In contrast to the static approach, thedynamic approach determines the passages with
fixed or variable length in response to requests [2][9].
This method can get passages more suitable to eachquery but needs an algorithm to efficiently determine
them since all possible passages must be computed in
retrieval time. According to Kaszkiel [9], we knowthat the dynamic approach gives substantial
improvements in effectiveness than the static and so
can be more suitable for question answering. Passage
was defined in several ways in previous works:
sections (or paragraphs) [19][13], pages (which are
adjacent paragraphs within limited bytes)[21], tiles(which are adjacent sentences divided by topic
shift)[6], fixed-length windows[2] and variablelength windows[9]. Sections, pages and tiles are

static passages, whereas fixed-length and variablelength windows are dynamic. Kaszkiel comparedthese various passages through experiments andshowed that passages based on windows were more
effective and especially 150 to 350-word passages
had good performance in retrieval precision.
However, in the view of question answering,passage retrieval may show different characteristics.
We analyzed 492 questions and evaluation results of
the runs submitted by each participant group of
TREC-10 [18] to investigate the effect of passage
retrieval when it was applied to question answering.
Using answer patterns and judgment information
provided by NIST, we gathered instances of eachanswer passage which consists of 11 adjacentsentences: one sentence containing an answer string
and the previous and the next 10 adjacent sentences.
And then we investigate the distribution of query
terms in each answer passage. 1 Throughout thisanalysis, we found that about 80% of query terms ofeach question occurred on average within 50-word
window including answer string in its center andwithin 3-sentence window including answer string in
the middle sentence. This is very small relatively to
the window size suggested by Kaszkiel[9] and might
be due to the difference between the two tasks.
Usually the goal of document retrieval task is to findout as many relevant documents as possible to arequested topic, but the goal of question answeringtask is to find out an exact answer (or answer string)to a question. Relevance information to a topic tends
to occur across several sentences in a document, butquery terms of question and its answer tend to occur
within one sentence or two.
According to this preliminary examination, we
defined a passage as consecutive three sentences for
QAC task. We prefer sentence window to word
window to prevent a possible answer from being cut
off by window boundary even though some query
terms and the possible answer co-occur in the samesentence.

3.2

Passage Scoring

We developed a scoring measure to rank eachpassage and an algorithm to compute efficiently
scores of all possible passages. We can first think of
the following assumptions for scoring measure of thepassages:

1!

The more query terms a passage contains,the more probably the passage also contains
an answer to the question.

!

Duplication of terms in a passage or aquestion is not important and can be ignored.

!

Document-specificterms
important than general terms.

are

more

Stop words and optional terms in a query were ignored andlemmatization was applied to each query term except thesuperlative. For example, in a question, “What is the largest city in
the U.S.?”, ‘city’ is optional and therefore was ignored.

Proceedings of the Third NTCIR Workshop

The first and second assumptions can be measured by
the number of unique query terms occurred in a
passage divided by the number of unique query terms
and the third can be expressed by using inverse
document frequency. Let qtuc be count of uniquequery terms in a question, ptuc be count of uniquepassage terms matched with the query terms, ptfi be
frequency of ith query term in a passage (we call thispassage term frequency) and idfi be inverse document
frequency of ith query term. Then the score of apassage can be calculated using expression (1).

PScore= α ×

ptuc2000 qtuc+ (1 − α ) ×× ∑ ptfi × idfi (1)qtuc
qtuc i =1, where α is a constant.

3.3

Efficient Passage Selection

Passage selection algorithm must cover all
possible passages from each retrieved document toselect most probable passages. If a document consistsof five sentences (S1, S2, S3, S4, S5), there are threepossible passages (P1={S1, S2, S3}, P2={S2, S3, S4},P3={S3, S4, S5}). Then the algorithm must calculate
the score of each passage P1, P2 and P3 respectively
by expression (1). Since qtuc and idfi are constantregardless of passages, it only needs to count ptucand ptfi. These can be easily obtained from term
position information (TPI) which is a database that
contains the position (i.e. jth token of ith sentence) of
each index term occurred in each document andwhich is constructed in IR indexing time. When adocument is retrieved, we can count the occurrencesof each query term in each sentence and then countthe occurrences of each query term in each passage
and calculate the score of each passage.
We rank the passages according to the scores ofthem and select top N passages for further processing.
It is allowed to select more than one passage froma document, but only one passage is allowed to be
selected among three consecutive and overlappedpassages to prevent one sentence from being
processed more than once in the next module. In theabove example, if query terms occur in only S3, thethree passages, P1, P2 and P3 have same scores. In
this case, we prefer P2 to others since answers tend tooccur near the query terms.
After selecting top passages, answer candidates
are extracted from them through Lexico-Semantic
Pattern matching and most probable answers are
returned as a response to the input question.

4. Question Answering using LexicoSemantic Patterns
Various Question Answering systems andtechniques have been developed and tested throughrecent TREC QA tracks on the English test collection[15][17][18]. Some systems like [5] showed good
performance through complicated analyses of texts
such as parsing and theorem proving while somesystems like [14] obtained surprising results using

only simple surface patterns extensively and showed
the remarkable power of such lexical patterns by
exploiting redundancy of the corpus.
The first Question Answering Challenge (QAC)
task in NTCIR3 has a goal similar to that of theprevious TREC QA tasks (that is, uses fact-basedquestions that require short answers) but requires theexact answer to each question rather than fixed-byte
answer strings (TREC2002 QA track also adopts thispolicy). This exact answer constraint may degrade
the performance of QA systems since a returned
answer is judged as incorrect if it contains any
dummy words. For example, a returned answer “ＤＤ
Ｉ・ＫＤＤ・ＩＤＯ” to a question “２０００年１０月１日に合併することが決まった通信三社はどこですか。 (What are
three communication companies which were decided
to merge in Oct. 1, 2000?)” is incorrect though each
‘ＤＤＩ’, ‘ＫＤＤ’ and ‘ＩＤＯ’ is judged as correct.
Therefore we need a technique to exactly extract the
only entities that can be answers and well-engineered
Lexico-Semantic Patterns (LSP) are developed forsuch purposes.
Instance
LSP…名作/NC は/EH 何/NRC です/U か/EM

(What is the masterpiece …)(%work)(は)(何)(です)(か)会長/NC は/EH 誰/NRC です/U か/EM

(Who is the president …)(@position)(は)(誰)(です)(か)３０/NN 歳/NUN(30 year-old)(@number)(@unit_age)
夏 目 /NPPS 漱 石 /NPPG の /EY 「 /SO こ こ ろ/NC 」/SL(Souseki Natsume’s ‘Kokoro’)(@person)(の)(@bracket)
バイ/NC スフ/NC ロク/NC さん/NUP(Mr. Baisuhuroku)(@np)(NUP)
Table 1. Example of Lexico-Semantic Patterns

4.1

2

Lexico-Semantic Patterns

In this subsection, we describe Lexico-Semantic
Patterns (LSP), which is used to determine theanswer type of an input question and also to extract
answer candidates from selected passages.
LSP is a pattern that is expressed by lexical
entries, part-of-speeches (POS), syntactic categories
and semantic categories. Unlike surface patterns,
which are expressed literally using only lexical
entries, LSP has more flexibility, can reduce thenumber of necessary patterns and gives theexpression power to handle the complex syntactic2

Each instance was attached with POS tag, and the parentheses in
LSP were inserted only to separate each component of the LSP.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

and semantic phenomena in human language sinceeach lexical can be generalized by POS, syntactic orsemantic category.
Table 1 shows some examples of LSP’s that canbe constructed from instances. ‘%work’, ‘@position’,‘@unit_age’ and ‘@person’ are semantic categoriesof ‘名作’, ‘会長’, ‘歳’ and ‘夏目漱石’, respectively. Of
course each number expression is generalized to‘@number’. ‘@bracket’ and ‘@np’ are syntacticallygeneralized from ‘ 「 こ こ ろ 」 ’ and ‘ バ イ ス フ ロ ク ’,respectively. ‘NUP’ is a POS tag of ‘ さ ん ’ andrepresents a suffix following person’s name.
We use POSTAG/J, our part-of-speech (POS)
tagger for Japanese, to generalize each lexical entry
to its POS. For syntactic category such as ‘@np’ and‘@vp’, we implemented simple verb and noun phrase
chunker. ‘@bracket’ and ‘@parenthesis’ are made bychecking the boundaries surrounded by brackets andparentheses within small window and is very usefulfor extracting titles of books, movies and TV
programs and acronyms of entities, respectively. To
generalize a lexical entry to a semantic category, wedefined 68 semantic categories by referring theprevious TREC data and gathered about 250,000
instances belonging to each semantic category fromlots of web sites and dictionaries. Semantic
categories include person, location, school, city,
company, bird, drug, etc.

4.2

Determining Answer Type using LSP

It is important for a QA system to predict what
type of answer the question requires (i.e. answertype): person name, location, organization, or any
others since it can fairly reduce the number of answer
candidates. Referring to the previous TREC QA
questions, we defined 62 answer types and developed
a method that classifies questions into the answer
types using the LSP’s.
Usually an interrogative in a question is an
important factor but it is not enough to determine theanswer type of a question because it also can has
sense ambiguities like other words. For example, ‘どこ’ indicates a ‘location’ in a question, “東京ディズニ
ー ラ ン ド は ど こ に あ り ま す か 。 (Where is Tokyo
Interrogatives
何/なん/なに(what/which)
誰/だれ/どなた(who)
どこ/何処/何所, どちら/どっち(where/which direction)
いつ(when)

Disneyland?)” while it indicates a ‘company’ inanother question, “日本サブウェイはどこの会社の子会
社 で す か 。 (Nihon Subway is which company’s
subsidiary?)”. This means that LSP’s for classifying
questions must include its surrounding contexts aswell as an interrogative itself. ‘どこにあり’ and ‘どこの会社’ are key phrases that can tell the answer type of
two questions and expressed in LSP grammar for ‘どこ’ question as follows:(どこ)(に)(ある) " 1|3|location(どこ)(の)(%company) " 1|3|company‘%company’ is a semantic class to represent
synonyms of ‘会社’. The first number in the righthand side of arrow ("), divided by vertical bar,
indicates the location of an interrogative (‘どこ’ in thiscase) in the LSP of the left-hand side and the second
number indicates number of components of the LSP.
The third is an answer type. In other words, theabove grammars represent that the LSP consists ofthree components and the first one is an interrogativeand if some part of a question matches with the LSP,
then the answer type of the question is ‘location’ (or‘company’). If more than one answer type is possible
to each LSP then all possible answer types can beenumerated with a separating vertical bar. Table 2
shows some more examples of LSP grammars fordetermining answer types with sample questions.
Determined answer type can be extended toseveral subsumed answer types. For example,‘location’ is extended to “location, state, city, country,etc” before detecting answer candidates. If a question
is not assigned to any predefined answer type, weassign ‘language_unit’ to the question.

4.3 Detecting Answer Candidates using
LSP
Unlike the previous TREC QA tasks, we have toreturn only exact answers in the QAC task. This
makes answer detection module more importantsince an answer string involving any dummy words
will be judged as incorrect.
LSP is used for extracting answer candidates
from text as well as classifying questions. Once ananswer type of a question is determined, what we

Sample question“夏目漱石の名作は何ですか。”(What is the masterpiece of Souseki Natsume?)“千葉県の県庁所在地は何市ですか。”(Which city is the provincial office of Chiba
located?)“大学審議会の会長は誰ですか。”(Who is the president of university council?)"タージ・マハールはどこにありますか。"(Where is Taj Mahal?)“米ソの冷戦が終わったのはいつですか。”(When was the Cold War between USA andRussia terminated?)

Grammar(%work)(は)(何)(です)" 3|4|movie|book|music(は)(何)(%city)(です)" 2|4|city(@position)(は)(誰)(です)" 3|4|person(どこ)(に)(ある)" 2|3|location(は)(いつ)" 2|2|date

Table 2. Examples of LSP grammars for classifying questions into answer types

Proceedings of the Third NTCIR Workshop

Instances
３０歳 (30 year-old)
夏目房之介さん（４７）(Mr. Husakorekai Natsume (47))
夏目漱石の「こころ」 (Natsume’s ‘Kokoro’)…主演の「Ｌ．Ａ．コンフィデンシャル」(‘LA Confidential’ starring …)
バイスフロクさん (Mr. Baisuhuroku)
村上雅則投手 (A pitcher Masanori Murakami)
スカイマークエアラインズ（本社・東京）(Sky Mark Airlines (head office, Tokyo))

Grammars(@number)(@unit_age) " age|1|2|2|1.0(@person)(NUP)(（)(@number)(）) " age|4|4|5|1.0(@person)(の)(@bracket) " book|3|3|3|0.85(主演)(の)(@bracket) " movie|3|3|3|0.9(@np)(NUP)" person|1|1|2|0.85(NPPS)(NPPG)(@position) " person|1|2|3|1.0(K)(（)(本社)(・)" company|1|1|4|0.9

Table 3. Sample grammars for extracting answer candidates

have to do is to find out entities belonging to theanswer type within passages selected in the previous
step. Named Entity (NE) tagger was used for thispurpose in many QA systems and is substituted by
LSP grammars in our system.
From instances belonging to each answer type,
gathered from texts, we can construct LSP grammars
for extracting answer candidates. For example, in atext snippet, “村上雅則さん（５３）”, we can extract ‘村
上 雅 則 ’ and ‘ ５ ３ ’ as ‘person name’ and ‘age’,respectively, using following LSP grammars:(NPPS)(NPPG)(さん) " person|1|2|3|1.0(NPPS)(NPPG)(さん)(（)(@number)(）) "age|5|5|6|1.0
The first grammar means that the LSP is composedof three components and if it matches with a text
snippet, then corresponding parts to between the firstcomponent and the second including boundaries is
extracted as ‘person name’ with weight 1.0. Weightvalue indicates the precision of the pattern and can beobtained by counting correct instances among ones
extracted by applying the pattern to test collection.
Table 3 shows some more examples of LSP
grammars for detecting answer candidates.
If a question is not assigned to any predefined
answer type and so assigned to ‘language_unit’, any
noun phrase can be considered as answer candidates,
but we prefer proper noun or bracketed phrase toothers.
Answer detection must be applied to a question
itself since it also can contain entities belonging to its
answer type. An answer candidate to a question may
be filtered out if it occurs in the question itself.

4.4

Answer Filtering and Scoring

Before scoring each answer candidates, they arefiltered out by the following assumptions:

Some heuristics specific to each answer type arealso applied for filtering. For example, the number
representing a year usually consists of 2 to 4 digits.
One character person name is also filtered out.
If date constraint was specified in a question, e.g.,“９９年７月の完全失業率は何パーセントでしたか。(Whatpercent was the whole-unemployment rate in July,99?)”, then answer passage should satisfy it too. Thatis, the date should occur in the answer passage or besame with the date of document from which theanswer passage came.
Some questions contain a word that can become aclue for finding answers. For example, in a question,“チュニジアの人口は何人ですか。(How many people
are there in Chunizia?)”, a word, ‘人’, is a clue forfinding an answer, “９０９万人 (909 ten thousandpeople)”. This clue word can be useful for filteringanswer candidates especially for ‘count’ type answers.
The score of each extracted answer candidate isbased on the number of unique terms matched in thepassage and distances from each matched term. This
assumes that a correct answer to a question will occur
probably near terms matched with the question. This
assumption is not necessarily true but makes a good
hit for many questions of the previous TREC QA
tasks.
The score of an answer candidate is calculatedusing the following four features and linearly
combined with the passage score (expression (2))!

LSPwgt : the weight of the LSP’s used to
extract the answer candidate. This reflects
the confidence of LSP’s applied to extract
the answer candidate.

!

qtuc : the count of unique query terms.

!

ptuc : the count of unique terms matchedwith query terms in a selected passage. The
ratio of ptuc to qtuc reflects the weight of apassage from which the answer candidate
was extracted.
avgdist : the average distance between theanswer candidate and each matched term.

!

A word itself occurring in the question
cannot be an answer.

!

An answer candidate cannot be a correct
answer if it also occurred in the question!

!

An answer candidate cannot be a correct
answer if it came from the phrase presenting
the origin of the news article. This heuristic
is specific to the test collection.

AScore = β × PScore +(1 − β ) × LSPwgt ×

(2)
ptuc
ptuc
×
qtuc ( ptuc + avgdist)

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

, where β is a constant, PScore is the score of apassage, from which the answer candidate was
extracted.
When ranking answer candidates, we check any
duplicates and remove them out. For person name,
we also check partial duplicates between familyname and full name.

93.3% had at least one within top 100.
Top N docs
5 docs
10 docs
15 docs
20 docs
30 docs
100 docs
200 docs
500 docs
1000 docs
Retrieved
Relevant
Rel_ret5. Experiments in NTCIR3
We participated in the QAC Task1 of NTCIR 3.
The test collection was composed of 230,000
documents (about 280MB) from 98-99 Mainichi
Newspaper articles and relatively small compared to
the TREC test collection (about 3GB).
To test each participating system, 200 factoid
questions were provided by National Institute ofInformatics (NII) and each participant returned top
five exact answer candidates to each question. This
exact answer constraint is one of the main differencesfrom the TREC-8,9,10. Among the questions, fivequestions were excluded from evaluation since they
had no answer in the test collection.
Returned answers were judged by NII assessors
and evaluated by mean reciprocal rank. In thissection we report our system’s performance
evaluated from NII and performances of each module
in our system.

5.1

5.2

Table 4. Performance of document retrieval

Top N
passages
5 psgs10 psgs
15 psgs
20 psgs
30 psgs
50 psgs
100 psgs
200 psgs
500 psgs

Experimental Environment

We indexed the test collection using ourdocument retrieval system for Japanese, POSNIR/J,
which maintains each document with POS tag
provided by POSTAG/J, our POS tagger for Japanese,and term position information (TPI) of each indexedterm.
We then manually constructed LSP grammars foranswer type identification and answer candidate
detection from the extra web data and the texts of thetest collection.
Our system used top 1000 documents retrievedby POSNIR/J to select answer passages and top 500
passages to detect answer candidates and returned
exact answer candidates in 8.74 seconds per question
averagely on a PC server with P-III 650MHz dual
CPUs.

Effect of Document Retrieval

Based on the answer file provided by NII, weevaluated result of our document retrieval system.
We constructed relevant document list for eachquestion from the answer file and ran the evaluation
tool, trec_eval, on the retrieved document list. The
evaluation result is shown in Table 4.
Unlike general document retrieval, just one
relevant document is enough to answer to a question.
So we inspected rank of the first relevant document
to each question. All questions among 195 questions
except only one (q#24) had at least one relevant
document within top 500 retrieved documents andPrecision
0.16720.12670.10020.08740.06970.03280.02060.00960.0053
1926671123
1037

Rate ofquestions
0.78970.85130.87180.89230.89740.91230.91790.93850.9487

Table 5. Performance of passage selection

5.3

Effect of Passage Selection

We also evaluated the performance of passage
selection module. Based on (answer instance,
document id) pairs of the answer file, we judged aselected passage as correct if it contains an answer
instance and its document id is same with document
id of the answer instance. Table 5 shows the rate ofquestions whose answer was included within top N
selected passages. Only 10 questions had no answers
within top 500 selected passages and 89.2% of thequestions had an answer within top 20 selectedpassages. For two questions (q#78 and q#82) that had
no answer passages within top 20, our system
returned correct answers at top rank. As evaluated bymean reciprocal rank (MRR) of the first relevant
passage, our system got 0.6560.

5.4

Performance of Question Answering

Finally, returned answers of our system were
evaluated. Table 6 shows the performance of oursystem by the scoring tool (version 1.40) provided by
NII. Five among 200 questions were excluded fromthe computation of mean reciprocal rank (MRR) as
they have no answers in the test collection.
Some questions have more than one answer from
more than one document. There are 288 correctProceedings of the Third NTCIR Workshop

answers to 200 questions in the test collection. Our
system returned 994 answer candidates to 200
questions and, among them, 220 responses were
judged as correct and 183 were distinct answers.
Recall and precision were calculated from thenumbers of answers, outputs and correct outputs. Fmeasure was calculated by F-measure = (2 * Recall *Precision) / (Recall + Precision).
In MRR, the standard evaluation criteria in QAC
Task1, our system recorded 0.608, which is best
performance among 15 participating systems(average 0.303). Our system returned correct answers
within top five for 149 questions and at the first rankfor 98 questions. Table 7 shows the number ofquestions according to the rank of the first correct
answer and Table 8 shows the performance of oursystem according to each answer type. Our system
was successful to return the correct answers to about
half of the questions at top rank, and the deviation of
performance according to answer type is not verylarge.
Compared to the performance (MRR=0.320) wemade at TREC-10, this performance is very higheven though the exact answer constraint was given.
This may be because the test collection is very small
and the ‘no answer’ questions were excluded in
NTCIR 3. Another reason can be that we used ourown document retrieval system this time. In TREC10, we used document list provided by NIST since
we couldn’t finish indexing of the test collection
within the given amount of time. However, after theevaluation, we finished indexing and found that theresults of our own document retrieval system aremuch better than the document lists given by NIST.
Question
200
Recall0.635

Answer288
Precision
0.184

Output994
F-measure
0.285

Correct
183
MRR
0.608

Table 6. Performance of our system

Rank
1
2
3
4
5

# of questions
98
27
14
6
4

Table 7. Number of questions according to therank of the first correct answer

Answer type
Artifact
Date
Living thing
Location
NumberOrganization
PersonSubstance# of questions
21
14
6
31
31
19
42
6

MRR
0.579
0.649
0.389
0.610
0.661
0.543
0.614
0.750

Title
Others
Total

10
15
195

0.683
0.536
0.608

Table 8. Performance according to each answer type3

6. Discussions
In this section we analyze our system’s resultswith some examples, especially in the case of failures.
Answer types we used were not enough to cover
all the questions. Q#36 and q#132 require a name ofprize and legislative bill, respectively, but they are
not defined in our system. In questions like q#7 andq#72, artifact is too rough to detect correct answers.
We must define more general answer types using
general or domain-specific thesaurus so as to covervarious questions.
For some questions like q#90, q#169, q#173,
q#186, etc., our system failed to determine answer
types and thus failed to return correct answers
because manually constructed LSP’s and semantic
class dictionaries were not complete yet.
Failure to determining answer type does notalways mean failure to detecting correct answers. For
q#21, q#152, q#187, q#195, etc., our system could be
successful to return correct answers by selecting
bracketed entities.
Some questions can be assigned to multiple
answer types. In this case, answer candidates
belonging to each answer type are detected andranked by the scoring measure. For example, q#29
was assigned ‘location’, ‘company’, etc. and q#136
was assigned ‘year’ and ‘count’, this increased theambiguity of answer candidates though correct
answers to some questions could be ranked within
top five by the scoring measure.
We did not divide ‘count’ type in detail except
age, volume, weight, area, speed, size, temperature,
duration, rate, length, power and money. That is, we
did not distinguish number of people from number ofanimals. Instead, we used unit word as a clue fordetecting correct answer candidates. Unit word,
however, is not always available with all ‘count’ typequestions (e.g., q#49, q#52, q#194, etc). This also
increased the ambiguity of answer candidates andmade it difficult to rank correct answers high.
Our LSP grammar for detecting answer
candidates was not enough and complete, and thus
failed to detect correct answers or extracted
incorrect-type candidates to some questions.
Success in determining answer type and detecting
correct answer candidates does not always mean
correct answers. Many of other failures were caused
that our system ranked answer candidates only
depending on the probabilistic measures without any
3

Living thing includes ‘bird’ and ‘plant’, location includes ‘city’,‘country’, ‘mountain’, ‘planet’, ‘river’ and ‘state’, number includes‘age’, ‘area’, ‘count’, ‘duration’, ‘length’, ‘money’, ‘rate’, ‘speed’and ‘temperature’, organization includes ‘company’ and ‘team’,and title includes ‘book’, ‘magazine’, ‘movie’ and ‘music’.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

semantic answer justification.

7. Conclusions
We developed our Question Answering system
for Japanese, SiteQ/J, to participate in QAC Task1 of
NTCIR3.
Our system narrows a search space using
dynamic answer passage selection, and determines
answer type of a question and extracts answer
candidates using Lexico-Semantic Pattern matching
without deep linguistic analysis of the texts. Detected
answer candidates are ranked by passage scores andthe measure of distance between answer candidates
and matched terms. As a result of LSP engineering,
our system showed the best performance(MRR=0.608) among 15 participating systems.
In the future, we will try to reduce manual efforts
to construct LSP’s and semantic category dictionaries
by automatically learning them from extra web data.
We will develop a method and construct knowledge
to justify answers for higher precision. Automatic
contextual taxonomy [20] learning can be one of thealternatives to go.

References[1] E. Brick, J. Burger, D. House, M. Light, and L. Mani.
Question answering from large document collections.
In The 1999 AAAI Fall Symposium on Question
Answering Systems, pages 26-31. North Falmouth,
Massachusetts, 1999. AAAI.
[2] J. P. Callan. Passage-level evidence in document
retrieval. In The 17th Annual International ACM SIGIR
Conference on Research and Development inInformation Retrieval, pages 302-309, Dublin, Ireland,
1994. ACM.
[3] C. L. A. Clarke, G.V. Cormack, D. I. E. Kisman, and T.
R. Lynam. Question Answering by passage selection(MultiText Experiments for TREC-9). In The 9th Text
Retrieval Conference (TREC-9), pages 673-683,
Maryland, 2000. NIST.
[4] C. L. A. Clarke, G. V. Cormack, and T. R. Lynam.
Exploiting redundancy in question answering. In The
24th Annual International ACM SIGIR Conference on
Research and Development in Information Retrieval,
pages 358-365, Louisiana, 2001. ACM.
[5] S. Harabagiu, D. Moldovan, M. Pasca, R. Mihalcea, M.
Surdeanu, R. Bunescu, R. Girju, V. Rus, and P.
Morarescu. Falcon: Boosting knowledge for answer
engines. In The 9th Text Retrieval Conference (TREC9), pages 479-488, Maryland, 2000. NIST.
[6] M. A. Hearst and C. Plaunt. Subtopic structuring for
full-length document access. In The 16th Annual
International ACM SIGIR Conference on Researchand Development in Information Retrieval, pages 5968, Pittsburg, 1993. ACM.
[7] E. Hovy, L. Gerber, U. Hermjakob, M. Junk, and C.-Y.
Lin. Question answering in webclopedia. In The 9th
Text Retrieval Conference (TREC-9), pages 655-664,
Maryland, 2000. NIST.
[8] A. Ittycheriah, M. Franz, W.-J. Zhu, and A.

Ratnaparkhi, IBM’s Statistical Question Answering
System, In The 9th Text Retrieval Conference (TREC9), pages 229-234, Maryland, 2000. NIST.
[9] M. Kaszkiel and J. Zobel. Passage retrieval revisited.
In The 20th Annual International ACM SIGIR
Conference on Research and Development inInformation Retrieval, pages 178-185, Philadelphia,
1997. ACM.
[10] G. G. Lee, J. Seo, S. Lee, H. Jung, B.-H. Cho, C. Lee,
B.-K. Kwak, J. Cha. D. Kim, J. An, H. Kim, and K.
Kim. SiteQ: Engineering high performance QA systemusing lexico-semantic pattern matching and shallow
NLP. In The 10th Text Retrieval Conference (TREC-10),
pages 437-446, Maryland, 2001. NIST.
[11] D. Moldovan and S. M. Harabagiu. Lasso: A tool forsurfing the answer net. In The 8th Text Retrieval
Conference (TREC-8), pages 175-184, Maryland, 1999.
NIST.
[12] J. Prager. Question-answering by predictive annotation.
In The 23rd Annual International ACM SIGIR
Conference on Research and Development inInformation Retrieval, pages 184-191, Athens, 2000.
ACM.
[13] G. Salton, J. Allan, and C. Buckley. Approaches topassage retrieval in full text information systems. In
The 16th Annual International ACM SIGIR Conference
on Research and Development in Information
Retrieval, pages 49-58, Pittsburg, 1993. ACM.
[14] M. M. Soubbotin. Patterns of potential answer
Expressions as clues to the right answers. In The 10th
Text Retrieval Conference (TREC-10), pages 293-302,
Maryland, 2001. NIST.
[15] E. M. Voorhees and D. Tice. The TREC-8 question
answering track evaluation. In The 8th Text Retrieval
Conference (TREC-8), pages 83-105, Maryland, 1999.
NIST.
[16] E. M. Voorhees and D. Tice. Building a question
answering test collection. In The 23rd Annual
International ACM SIGIR Conference on Researchand Development in Information Retrieval, pages 200207, Athens, 2000. ACM.
[17] E. M. Voorhees. Overview of the TREC-9 Question
Answering Track. In The 9th Text Retrieval Conference(TREC-9), pages 71-79, Maryland, 2000. NIST.
[18] E. M. Voorhees. Overview of the TREC 2001
Question Answering Track. In The 10th Text Retrieval
Conference (TREC-10), pages 42-51, Maryland, 2001.
NIST.
[19] R. Wilkinson. Effective retrieval of structured
documents. In The 17th Annual International ACM
SIGIR Conference on Research and Development inInformation Retrieval, pages 311-317, Dublin, Ireland,
1994. ACM.
[20] W. A. Woods. Conceptual indexing: A better way toorganize knowledge. In Technical Report SMLI TR9761, Sun Microsystems Laboratories, Mountain View,
CA., 1997.
[21] J. Zobel, A. Moffat, R. Wilkinson, and R. Sacks-Davis.
Efficient retrieval of partial documents. Information
Processing Management, 31(3):361-377, 1995.