Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, JapanAn Open-domain Question Answering System
for NTCIR-8 C-C Task
Guiping Zhang, Wubei Zhang, Yu Bai, Shengqiao Kang, Peiyan Wang
Knowledge Engineering Research Center, Shenyang Institute of Aeronautical Engineering

Email:nlpxiaobai@yahoo.com
Abstract
In this paper, we described our CCLQA system andthe evaluation results for the C-C task at NTCIR-8
ACLIA. The system consists of a Question Analysis
module, IR module and Answer Extraction module. The
Question Analysis module was developed for NTCIR-7
CCLQA, which is based on the Question pattern libraryand HowNet. The IR module was developed for
NTCIR-8 IR4QA task, and the results of
KECIR-CS-CS-01-T were used as question related
documents. For answer extraction, a Surface-BasedMulti-Strategy approach was used. It deals retrieval
results with different Strategies. The evaluation results
show that our system achieves 0.3450 average F-score(beta = 3).

Keywords: Question Answering, Surface-Based,
Question Analysis, Answer Extraction
1. Introduction
Current research in QA is moving beyond factoid
questions, so there is significant motivation to evaluate
more complex questions in order to move the researchforward.[1]We
participateinsimplified
Chinese-Chinese(C-C) QA subtask in NTCIR-8, whose
goal is to develop effective CCLQA(Complex
Cross-lingual Question Answering) evaluations forcomplex questions as well as factoid questions.
According to the requirement of the subtask, we
implement a surface-based QA system that can handle
questions with different types. We focused on theprocessing of five types of complex questions:
DEFINITION,
BIOGRAPHY,RELATIONSHIP,
EVENT, and WHY; four types of factoid questions:PERSON, ORGANIZATION, LOCATION, and DATE.
In this paper, we described our CCLQA system andthe evaluation results for the C-C task at NTCIR-8
ACLIA. The system consists of a Question Analysis
module, IR module and Answer Extraction module. The
Question Analysis module was developed for NTCIR-7
CCLQA, which is based on the Question pattern libraryand HowNet. The IR module was developed for
NTCIR-8 IR4QA task, and the results of
KECIR-CS-CS-01-T were used as question related
documents. For answer extraction, a Surface-BasedMulti-Strategy approach was used. It deals retrieval
results with different Strategies. The evaluation results
show that our system achieves 0.3450 average F-score(beta = 3).
The rest of this paper is organized as follows:

In section 2, we present related studies. Section 3
provides an overview of the system architecture. In
section 4 and 5, we introduce the question analysis
methods and answer extraction strategies respectively.
Section 6 gives the evaluation results and error analysis.
Finally, the conclusion is given in section 7.

2. Related Studies
2.1 Question Pattern Built
We focused on the evaluation of five types ofcomplex questions and four types of factoid questions.
Examples are shown in Table 1 below.
Table 1. Example of Questions#
Type
Example of Question
DEFINITION
10What is the Human
Genome Project?
BIOGRAPHY
10
Who is Howard Dean?
RELATIONSHIP
20
What is therelationship between
Saddam Hussein and
Jacques Chirac?
EVENT
20
What are the major
conflicts between
India and China on
border issues?
WHY
20
Why doesn’t U.S.
ratify the Kyoto
Protocol?
PERSON5
Who is the Finland’s
first woman president?
ORGANIZATION
5
What is the name of
the company that
produced the first
Fair-trade coffee?
LOCATION
5
What is the name of
the river that separates
North Korea from
China?
DATE
5
When did Queen
Victoria die?
The Question pattern library is used to the Question
Analysis module in our QA system, which is built
manually according to the different question types.
To build the question pattern, the Xinhua corpus(1998-2001) and evaluation questions of NTCIR-7 wereused. For a question, each named entity was replaced by
the named entity type correspondingly. The Table 2
gives some sample of the question pattern library. For
details of building patterns, see [2].

― 25 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Question

Query

IR4QA

Q-typeComplex Answer
Extraction

RDs

Question
Analysis

HowNetFactoid Answer
Extraction

Question patterns

AnswerFigure 1. C-C QA architectural module
Table 2. Examples of Chinese question patternwith Query typeQ-typeQuestion patterns
Eve.
[߫В/Вߎ/㔫߫/ग़᭄]<EVENT>[᳝/ᰃ@>ાѯҔМ]
Bio.
>ҟ㒡ㅔҟ@䇕ᰃ<PERSON>;>ҟ㒡ㅔҟ@ <PERSON>ᰃ䇕Rel.
>ҟ㒡䇈ᯢᦣ䗄@<Entity1>੠<Entity2> [᳝/ᰃ][ҔМ/ાѯ][݇㋏㘨㋏ऎ߿@Def.
>ҔМᰃ/ԩ䇧]< term >;< term >ⱘ[ᅮН/㾷䞞/৿Н]ᰃҔМ
Why[ᰃ]ЎҔМ˗ҔМॳ಴˗ॳ಴ᰃҔМPers.
ᰃ䇕; 䇕ᰃ; ાϾ<def:=Person>Loc.
[ ೼ / ԡ Ѣ ] ા [ 䞠 ] ˗ ા Ͼ<def:=Location>DateҔ М ᯊ ‫ ; ׭‬ԩ ᯊ ; ા[Ͼ]<def:=time>Org.
ા[Ͼ/ᆊ/ᬃ]<def:=org>ⱘ 㣅 ᭛ 㓽 ‫" ݭ‬, and then, by using rules matchingapproach, we can get “䴲‫ ”݌‬as the expansion of “SARS⮙↦”.

3. System Description

2.2 Name Entity Recognition
The NE recognition is one of the most important partsfor answering factoid questions. We employ a NE
recognition system based on conditional random fields[3], and its F-score in close test is 0.8801. In order toimprove the effect of NE recognition, a classified
dictionary of entities was built mainly by using the open
database of the Chinese Wikipedia, and about 300
thousand entities were content in it.

2.3 Wiki-based Abbreviation Expansion
We can get some keywords in question analysis step, butthey only appear in the question, and it’s possible forthem to appear in corpus in another way, especially forabbreviations. Thereby, for C-C task we must find out
the synonyms of those abbreviations in Chinese, so as toimprove recall of our system. The online-Wiki(http://baike.baidu.com) was used as abbreviation
expanding sources.
For example, the explanation of the term “SARS⮙↦” given by online-Wiki contains "SARSᰃ’䴲‫’݌‬ᄺৡ

As in NTCIR-7, we follow a modular architectural
approach to QA represented in Figure.1.
The system contains three modules: Question
Analysis module, IR module and Answer Extraction
module. It carries out the following steps: For a given
question, we did question classification first, thequestion was classified into its expected answer category
by using pattern matching with predefined templates in
the Question pattern library, and then, a query form was
generated by the original question, it could be used toretrieve relevant documents from the target corpus. A
few sentences were extracted from those relevant
documents to form a candidate pool. Here, a scoring
approach was used to rank candidate answers in thisperiod.
Since the ACLIA task also contains IR4QA(Information Retrieval for Question Answering) subtask,
During the evaluation, the question text and QA system
question analysis results were provided as input to the
IR4QA system, which produced retrieval results thatwere subsequently fed back into the end-to-end QA
systems[4]. Our group also participated in the IR4QA
task at NTCIR8. Therefore, we use this IR4QA system
as the document retrieval module. For details on it, werefer the reader to the KECIR IR4QA system description
paper [5]. Details of other modules will be introduced in
the following sections.

4. Question Analysis
For a given question, the entire passage retrieval andanswer extraction process relies on having the correctquestion (or answer) type and question forces.
Meanwhile, the purpose of question analysis is to
identify the question (or answer) type and detect
question forces of the inputted question. Therefore,
Question Analysis is an important task in most Question
Answering systems [5, 6, and 7]. It contains Question
classification and Keyword extraction.
The purpose of question classification is to obtain thequestion type and the answer type. Question
classification can be formulated as:― 26 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Here, Q represents the question set; T is the set ofquestion types; and f means to assign a particular class
to a question based on the type of answer entity thequestion represents.
In NTCIR-8 Question Analysis Track: Question
Analysis results contain key terms and answer types
extracted from the input question. The released formal
run topics were described in the xml-structure as shownin Figure.2.
<TOPIC ID="ACLIA2-CS-0001"><QUESTION
LANG="EN"><![CDATA[Who is the best
actor in the 76th Oscar's?]]></QUESTION><QUESTION
LANG="CS"><![CDATA[㄀76⬠༹ᮃव᳔Շ
⬋Џ㾦ᰃ䇕˛]]></QUESTION><NARRATIVELANG="EN"><![CDATA[The user would liketo know the name of the best actor in the 76th
Oscar's.]]></NARRATIVE><NARRATIVELANG="CS"><![CDATA[Փ⫼㗙ᛇⶹ䘧㄀76ሞ༹ᮃव᳔Շ⬋Џ㾦ᰃ
Figure2. A released formal run topic
For C-C Subtask, only Chinese QUESTION field isused. Usually, in a question, there are dominant words
and the indicate words. dominant words like “䇕(who)”,“ Ў Ҕ М (why)”, “ ԩ ᯊ (when)”, “ ԩ ഄ (where)” andother words such as “ԩҎ(which person)”, “ાϾ೑ᆊ(which country)”, etc. These can be regarded as
dominant words that show the question type directly.
Take a Chinese question sentence “⬉ᕅǉ᮴ᵕǊҔМᯊ‫݀׭‬᯴˛(When is "The Promise" screened?)” as an
example, we can easily identify the question type as
DATE type because of the dominant word “ҔМᯊ‫׭‬(When)”. Since the traditional regular expression model
performs well with those questions, which containing
dominant words, using the templates and rules, we build
a Question pattern library.
Table 3. The coverage of Chinese question typeclassification with patterns
Type
Patterns matched/ All questions
DEFINITION
10/10
BIOGRAPHY
10/10
RELATIONSHIP
18/20
EVENT
10/20
WHY
19/20
PERSON
5/5
ORGANIZATION
3/5
LOCATION
4/5
DATE
5/5
As shown in Table 3, Because of the limitation of thequestions set that was used to construct patterns, we can
not design all the patterns that could match every

question. For instance, some interrogative words like“ҔМ(what)” and “ાϾ(which)”, they could make apair with many different words and represent different
types. For example, the question type of “೼‫ܟ‬Ҕ㉇ᇨഄऎ᳝ાϾ೑೑ᆊⱘ䖍⬠˛(Which countries have borders
in the Kashimir region?)” is LOCATION, however, the⧗ 䯳ⱘ˛(Which baseballquestion “⥟ᓎ⇥ᰃાϾ⧗team does Chien-Ming Wang belong to?)” is an
ORGANIZATION type question. Therefore, we
consider the indicate words which always follow theinterrogative words in a question to identify the query
type.
HowNet is an on-line extra linguistics knowledge
system for computation of meaning in Human Language
Technology (HLT) [9]. HowNet unveils inter-concept
relations and inter-attribute relations of the concepts as
connoting in lexicons of the Chinese and their English
equivalents. The concept definition (DEF) in HowNet is
not written in natural language as in LDOCE or
WordNet, but in a mark-up language, whose basic units
are sememes such as “human|Ҏ”, “InstitutePlace|എ᠔”,“Occupation|㘠ԡ”, “medical|ए”, “doctor|ए⊏”, andsemantic roles and features such as “HostOf”, “agent”,“modifier” and “domain”. Therefore, the definition of“doctor” can be literally paraphrased as “A doctor is ahuman being, who has the attribute of occupation; hedoctors (gives medical treatment to); he belongs to thedomain of medicine”.
NO=127941
W_C=ए⫳
G_C=N[yi sheng]
E_C=
W_E=doctor
G_E=N
E_E=
DEF={human|Ҏ:HostOf={Occupation|㘠Ϯ}.
Domain={medicl|ए}.{doctor|ए⊏:agent={~}}}
Figure 3. The description of “doctor” in
HowNetTo find the indicate words we use the strategy
proposed in [8], following the algorithm in figure 4.
Step1.Get interrogative word in question Q
Step2.Select the first noun behindinterrogative word as the indicate wordof the question.
Step3.If there is no noun exist behindinterrogative word, then, choose thenearest one left the interrogative word
as the indicate word of the question.
Figure 4. Approach of question
indicate word selection
Then, we gain the concept definition of the indicate
word of the question. The first part of the concept
definition in HowNet was used to identify the type ofthe question.

― 27 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

5. Answer Extraction
5.1 Number of Snippets
The experiment [8] results has already shown that whenvaried the number of snippets from 1 to 1000,
performance improves sharply as the number of snippets
increases from 1 to 50 (0.243 MRR for 1 snippet, 0.370
MRR for 5, 0.423 MRR for 10, and 0.501 for 50),
somewhat more slowly after that, and then falling off
somewhat after that as more than 200 snippets areincluded. Since that, the top 100 results of
KECIR-CS-CS-01-T were used as question related
documents for answer extraction. Then we utilize
different strategies to deal with different type ofquestions.

5.2 Answer factoid Question
For answering question in ‘PERSON’, ‘LOCATION’and ‘ORGANIZATION’ type, we first use the NER tool
to recognize the NE which is correspond to the question
type as the candidate answers, and then A Abbreviations
was used to normalize similar candidate answers. At last
we used the voting approach to ranking them by thefollowing formula:

Score( Ai)nCount ( Ai)¦ Count ( Aj ) ( f 5.2.1)
j 0Here, the Score(X) is the povo (percentage of votes
obtained) of candidate answer X; the Count(X) is theoccurrence number of candidate answer X in relatesnippets; and the n is the number of different candidate
answers.
If some candidate answers got the same score, thenthe rank of them will be decided by the minimal mean
distance weighting method:
1

m
§
·
¨ ¦ min( Ai, Qj ) ¸¸ ( f 5.2.2)
W ( Ai ) ¨1  j 1
¨
¸
m
¨
¸
©
¹

The W(Ai) is the weight of the candidate answer Ai,
the m is number of keywords contained in the relatesnippet. min(Ai,Qj) is the minimal number of words
between candidate answer Ai and keyword Qj.
For answering question in ‘DATE’ type, thedocument ID was used for time reasoning. Rules sample
is shown in Table 4.
Table4. Rules sample of date expanding
Document ID

Time
AnswerEntityXIN_CMN_Ҟᑈ
2005ᑈ
20050620.0000
ᑈᑩ*2005ᑈ12᳜ᴀ᳜
2005ᑈ6᳜Ҟ໽*
2005ᑈ6᳜20᮹
X᮹*
2005ᑈ6᳜X᮹
X᳜*2005ᑈX᳜* If there is another time entity before this in thesame sentence, then don’t follow the rules.

And then, we use formula f5.2.1 and f5.2.2 tocalculate the score of the candidate answer. Here, theCount (Ai) is the weighted score of time expressions incandidate answers.
Here, we consider a whole time expression will be
including three items, day, month and year. Define thetime expression which is including all of three as
T-expression, including two of them as T2-expression,and just including one of them as T1-expression, andthen Count (Ai) will be the following formula:­½°if Ai is an T-expression °°°°then:°°°°C ( Ai)  0.5 u C (T 2)°°0.3 u C (T1)°°°
°if Ai is an T2-expression °Count ( Ai) ®
¾ ( f 5.2.3)°then:°°C ( Ai)  0.3 u C (T1)°°°
°if Ai is an T1-expression °°then :°°°°C ( Ai)°°°¯¿
C (Ai) is the occurrence number of candidate answer
Ai in relates snippets; C (T2) and C (T1) will be theoccurrence number of candidate answer inT2-expression and T1-expression respectively, and all ofthem were included in Ai.

5.3 Answer complex Question
The complex questions of CCLQA task in NTCIR-8
iscomposed
of
the
types
of
‘DEFINITION’,’BIOGRAPHY’,’EVENT’,’RELATION’ and ’WHY’. Answer types for thesequestions are not explicit. On the other hand, nuggets,
which represent the minimum unit of correct
information in answer sentences, are required as answer
candidates in the task. For answer extraction, we utilize
different strategies to deal with different type ofquestions.

5.3.1. ‘DEFINITION’ typeTo a definition question, we first split the document
based on punctuation. If a sentence is an interrogative
sentence, we filter the sentence. Some hint words werecollected to identify whether it’s a candidate answer ornot. The list of the hint words for “DEFINITION” isshown in Table5.
Table5. Hint words of ‘DEFINITION’ type
sets
words
definition_verbs ᰃǃህᰃǃিǃি‫خ‬ǃ⿄Ўǃ⿄԰ǃ⿄(П)Ўǃੑ
ৡ(Ў) ǃㅔ⿄ǃজ⿄ ǃ֫⿄ǃ˄㹿˅䅸Ў
property_verbs ᵘ៤ǃ㒘៤ǃ៤ߚǃሲѢǃࡳ⫼

We also summarized the patterns to extract the― 28 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japananswer of these questions. The patterns are shownas follow:

Table8. The description of ”relationship”
answer extraction method
Relationshipmethodstype
person-person
1) if a sentence contains
two named entities
which exist in aquestion and has asegment between two
nameentities,
thesegment will be ananswer.
2) NE
੠
NE
+
<ANSWER>, NEϢNE+ <ANSWER>ˈ NE +<ANSWER> + NE.
If a sentence matches theabove pattern, extract thesegment between the first
NE and verb.
person-organizat extract the sentences thationcontain all keywords
person-event
extract the sentences that
contain all keywords
event-event
extract the sentences that
contain two eventsotherextract the sentences that
contain all keywords

Table6.Patterns to extract the answers ofthe ‘DEFINITION’ type of questions
pattern
remark9ęGHILQLWLRQBYHUEVRU
keyword + V+
9ęSURWHUW\BYHUEV<ANSWER>
keyword + ⬅
+<ANSWER> +VVęSURWHUW\BYHUEV

5.3.2. ‘BIOGRAPHY’ typeTo a biography question, the answer actually refers to
the description of a person. In this case, we utilize some
attributes of a person. Like ’೑㈡’(nationality) , ’ߎ⫳ᑈ᳜’(birthday) , ’㘠Ϯ’(career) , ‘៤ህ’(achievements) ,etc. We believe these kinds of information are helpful to
extract the answer of ‘BIOGRAPHY’ type. Answer
patterns are shown in Table7.
Table7. Examples of Answer Patterns for“BIOGRAPHY” Type
attributepattern<PERSON>⫳Ѣ<PLACE>
Place<PLACE>[Career]<PERSON>ⓨਬǃ⾥ᄺᆊǃ乚ᇐҎǃሔ䭓ǃ
Careerᘏ㒳<PERSON>⫳Ѣ<TIME>,<PERSON>[Ѣ]<TIME>[ߎ⫳
Date
/⫳Ҏ]
<PERSON>[⦄ᑈ]<NUM>ቕ
<PERSON>[㦋ᕫ/㤷㦋/ᕫࠄ]<PERSON>[߯ࡲ/乚ᇐ/㒘㒛/Achievements খࡴ]<OGANIZATION><OGANIZATION>߯ྟҎ<PERSON>

Table9. Hint word of classification
event typehint words
List߫Вǃ߫ߎǃⳌ݇ǃાѯǃВߎDescription
Other
Table10.The methods to extract the answers
idpattern1಴Ў+ <ANSWER> +᠔ҹ+ keyword ..
2<ANSWER>.ˈ಴ℸ + .keyword
3<ANSWER> ˈ᠔ҹ+ .keyword4ᰃ⬅Ѣ+ <ANSWER>
5ᰃЎњ+ <ANSWER>
6
Aⱘॳ಴ᰃ+ <ANSWER>
7⬅Ѣ+ <ANSWER> + ᇚ˄Ӯ˅+keyword
8<ANSWER> +ᇚᇍ+ keywords
9<ANSWER> + .䗴៤+ keywords
10<ANSWER> + ᓩ䍋+ keywords

5.3.3. ‘RELATION’ type
For this type of questions, answers mostly refer to therelationship between two name entities. The heuristic
rule to identify the potential targets is : if a sentence
contains two named entities which exist in the question
then the items between them are most likely to be ananswer. We classify the relationship of two nameentitiesas:person-person,
person-organization,
person-event, event-event , and others. The answer
extraction method of every relationship pair are shownin Table8.

5.3.4. ‘EVENT’ type
We consider ‘EVENT’ type as compounded by ‘List’
question and ‘Description’ question. For the ‘List’ type,some works, places and persons are frequently
mentioned. For the ‘Description’ type, the answer
should indicate relate facts about the event. ‘List’ typeand ‘Description’ type are classified by use words
shown in table9.

5.3.5. ‘WHY’ type
For this type of questions, the answer actually refers
to the reason of phenomena. We use some rules andpatterns to identify the potential targets. Patterns andmethods to extract are shown in table10.

For a ‘WHY’ type question, if we can’t use patterns to
identify any sentence as an answer, instead, we extract
the sentences which contain keywords. If the sentence incandidates is an interrogative sentence simile to thequestion, we just take the next sentence as an answer.

5.3.6. Answer Ranking
For the “List” type of the “EVENT” question, theranking method is the same to factoid ones. For othercomplex types, two strategies were used separately, as
shown in table11.

― 29 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, JapanAnswering System at NTCIR7 CCLQA. The
proceedings of the 7th NTCIR workshop meeting. 2008.

Table11.The methods of answer ranking
methods
CS-CS Runs
KECIR-CS-CS With minimal mean distanceof keywords.
-01-T

KECIR-CS-CS-02-T[3]. Zhou Bo, Cai Dongfeng. Recognition of Chineseorganization name based on conditional random fields.
Journal of Shenyang Institute of AeronauticalEngineering. 2009.

Re-rank answers which
got same score in Run-01
with answer length.

[4]. Tetsuya Sakai etc. Overview of NTCIR-7 ACLIAIR4QA Task. The proceedings of the 7th NTCIR
workshop. 2008.

6. Evaluation
To evaluate system responses, Official per-topic F-score
definition based on nugget pyramid method [1] wasused.
Table12.The official human evaluation
all
CS-CS Runs

KECIR-CS-CS-01-T
KECIR-CS-CS-02-T

0.34500.3354

The average F-scores of every type of question are
shown in figure5, it shows run-01 and run-02
respectively.

7. Conclusion
In this paper, we described our CCLQA system andthe evaluation results for the C-C task at NTCIR-8
ACLIA. For answer extraction, a Surface-BasedMulti-Strategy approach was used. It deals retrieval
results with different Strategies. The evaluation results
show that our system achieves 0.3450 average F-score(beta = 3).

Reference[1]. Teruko Mitamura, Hideki Shima, Tetsuya Sakai, etc.
Overview of the NTCIR-8 ACLIA Tasks:Advanced
Cross-Lingual Information Access. The proceedings of
the 8th NTCIR workshop meeting. 2010.
[2]. Yu Bai, Li Guo, Lei Liu, etc. KECIR Question[5]. Thamar Solorio, Manuel Pérez-Coutiño, ManuelMontes-y-Gémez, et al. A language independent methodfor question classification. In: Proceedings of the 20thinternational conference on Computational Linguistics,
Switzerland, pp.1374-es. 2004.
[6]. CUI Huan, CAI Dong-feng, MIAO Xue-lei.
Research on Question Comprehension for Question
Answering System [C] (in Chinese). In: Process for
Artificial Intelligence in China [M]. Beijing Universityof Posts and Telecommunications (BUPT) Publishing
House. 2003.
[7]. Dongfeng Cai, Yu Bai etc. Chinese Question
Classification Using Combination Approach. The
proceeding of the International conference on semantics,
knowledge and Grid. 2007.
[8]. Susan Dumais, Michele Banko. Web Question
Answering: Is More Always Better?. The proceedings ofthe 25th Annual International ACM SIGIR conference
on Research and development in Information Retrieval.
2002.
[9]. DONG Zhen-dong, DONG Qiang. HOWNET and
the Computation of Meaning. World Scientific
Publishing Co. Pte. Ltd. 2006.











581
581

3(5

/2&

25*

'DW

%LR

'()

(9(

5(/

:+<

Figure 5.The average F-scores of every type of questions― 30 ―