Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, JapanDLUT Question Answering System at NTCIR-8 CCLQA
Wei Wang
School of Computer Science &Technology, Dalian University ofTechnology
1-12-3, No.8, XiangLong Street
GanJingZi District, Dalian, China
0086-411-87185161

Degen HuangˈLishuang Li

Xiuying Li

School of Foreign Languange, Dalian
School of Computer Science &University of Technology
Technology, Dalian University of
No.2 Linggong Road, Ganjingzi
Technology
District, Dalian, China
No.2 Linggong Road, Ganjingzi District,
Dalian, China
xyli1@163.com
0086-411-84708140, 84706002-3930

wangwei.dl@hotmail.com

huangdg@dlut.edu.cn,
computer@dlut.edu.cn

ABSTRACTThis paper describes the Chinese question answering system
DLUT for the CS-CS Subtask evaluation of NTCIR-8 CCLQA.
The system utilizes Situation Unit (SU), combined syntactic andsemantic information, as a basic processed unit to get candidate
answers, wherein the SUs in the question sentences are matchedwith those in the texts of corpus. In this evaluation, answers arepresented in the form of whole sentences instead of theirsimplified versions. The average F3 Score reaches 0.1954, andthe average Recall is ı 0.75 for the six question types,
BIOGRAPHY,DATE,
DEFINITION,
LOCATION,ORGNIZATION and PERSON. As the current system only
employs the partial information in SUs, the evaluation result only
indicates that the SU-based question answering system can get
promising Recall.

Categories and Subject Descriptors
H.3.3 [Information Search and Retrieval]:Retrieval models andSearch process
H.3.4 [Systems and Software]:Question-answering (fact retrieval)systems

General Terms

apply syntactic and semantic information in their system designand have achieved satisfying results. However, as these systems
are mainly targeted at single sentence processing, they seem to beinadequate in dealing with complex sentences with nested clauses
or modifiers and contextual sentence information extraction.
Moreover, the insufficient number of semantic roles also makes it
impossible to do an elaborate analysis based on the coarse
semantic labeling. These drawbacks might be a hindrance to theperformance of the complex question answering systems.
Suppose there is such an information retrieval system, where
the documents in each file are “segmented” into lexical-level“fragments” with sense in sentence before they are saved into
database. When answering questions, the system only needs tosegment the input question sentence into “fragments” of a similarkind, and then retrieve the combination of these “fragments” fromthe database to produce an answer corresponding to the question
sentence. In other words, the system is trying to realize a transfer,in which a document is first segmented into a number of lexicallevel “fragments” with sense in sentence, then the relevant
information among these “fragments” is picked up and finally an
original document is generated by reorganizing the “segmented”“fragments”. Moreover, these “fragments” can also be retrieved
separately. Figure 1 below is an illustration of the point.

Algorithms, Theory᭛ḷ 001Doc
001……………
.......
………………
……

Keywords
Information Retrieval, Question Answering System, Situation
Unit (SU)᭛ḷ 001Doc
001……………
…….
………………
……

1. INTRODUCTION
The question answering system is a further developmentof the conventional information retrieval techniques. Most of thecurrent question answering systems are built on syntactic andsemantic information. Boris Katz, Gary Borchardt, et al.
integrated syntactic and semantic interpretation approach into
their system START[1]; Hyeon-Jin Kim, Ji-Hyun Wang, et al.
proposed to construct Answer Index Unit (AIU) and thencomputed answer candidate weight by taking into accountsyntactic information [2]; Powerset conducted syntactic analysis
for every sentence on the webpage and extracted entities andsemantic information to generate index [3]; PARC builds theBridge system to realize the abstract knowledge representation ofnatural language sentences and semantic role labeling[4]˗Dan
Shen and Mirella Lapata testified the validity using semantic roles
in question answering system[5]. These previous approaches all

Figure 1. The segmentation and generation of document
Note: The “fragments” here refer to the lexical-level units withsense in sentence rather than the words in the conventional sense.
In addition to the morphological information of words, the“fragments” also contain the syntactic and semantic information.
For instance, when such a question as “ ᯼ ໽ ∸ ྚ 㒭 㑺 㗄 ҔМ˛”(What did Tom give John yesterday?) is input, the system
can retrieve a sentence like “᯼໽∸ྚ㒭㑺㗄 l ᴀк” (Tom gave
John a book yesterday.), excluding the results like “㑺㗄᯼໽㒭∸ྚϔϾ㣍ᵰ” (John gave Tom an apple yesterday.) and “∸ྚⱘહહ᯼໽㒭㑺㗄 1 ᴀк”(Tom’s brother gave John a bookyesterday.), though these sentences all contain “Tom”, “gave” and“John” three words.

― 55 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, JapanIf such an information retrieval system could be realized, thequestion answering system would be able to locate accurate
information, generate automatically answers on the basis of theclues provided by the “fragments”, and get a “Nugget” level
simplified answer. We seem to have found a close equivalent tosuch a “fragment”, namely, the Situation Unit (SU), on which ourChinese question answering system DLUT is based.

2. SYSTEM DESCRIPTION

Table 2. The sample with 6 SUs

What follows is an elaboration on the procedure behind the SUbased question answering system.

The Term of Constituents of the SU

2.1 Defining Situation Unit (SU)

Unit_Type: If the word of a SU acts as the subject, predicate,object of a sentence, or as an adverbial word modifying a verb, theSU is classified into the EVENT unit type; if it modifies the
subject or object of a sentence/clause, the SU is classified into an
ATTRIBUTE unit type.

No._Record: the only record number corresponding to eachspecific SU in the database.

2.1.1 Situation
Situation semantics was originally conceived by theAmerican logician J.Barwise and psychologist J.Perry in the early
1980s and subsequently developed by K.Devlin. “Situation” inthis paper is cited in the sense defined: “The world consists notjust of objects, or of objects, properties and relations, but ofobjects having properties and standing in relations to one another.
And there are parts of the world, clearly recognized (although notprecisely individuated) in common sense and human language.
These parts of the world are called situations. Events and episodes
are situations in time, scenes are visually perceived situations,changes are sequences of situations, and facts are situations
enriched (or polluted) by language.”[7]. Masahito Kawamori
applies situation semantics to the analysis of Japanese argument
roles [8].

Syntactical_Constituent_Code: Different code is assigned
according to the role the SU word plays in a sentence/clause. It
might be the subject, predicate, object, adverbial, complement, orthe appositive of a subject or object. For example, “01” represents
the subject, “02” represents the appositive of a subject, “03”represents the predicate. As one sentence might contain several
the same syntactic elements, it is necessary to indicate the order ofthem through marking. For example, if a subject marked as “01”is the second subject in a sentence, then its
Syntactical_Constituent_Code would be “012”.
Semantic_Code: The SU is semantically marked according to theǉৠН䆡䆡ᵫǊ (Chinese Synonym Dictionary), in which only

2.1.2 The Situation Unit ˄SU˅
Every situation can be divided into some small parts bydifferent ways. In situation semantics, these small parts are spacetime location, properties, individuals, relations, Infon, Profon, etc,
but we divide situations into some small parts by other ways. As
the situation can be described by natural language sentences andevery sentence consists of some words, we think that the every
word of sentence surely embodies the partial information ofsituation. In order to avoid confusion with those small parts
existed in situation semantics, we call such partial information ofsituation “Situation Unit” (SU), which is embodied by the word ofsentence. A SU is expressed by a word coupled with a
combination of the information attributed to this word after
syntactic and semantic analysis. Therefore, the quantity of the SU
in a natural language sentence is dependent on the quantity ofwords of that sentence.

2.1.3 The Constituents of the SU
Each SU comprises of 10 constituents, as shown below:
Table1. The constituent of the SU

An example is given to illustrate the point. The sentence “䩶⨈(Piano)䖯ܹ(Enters)Ё೑(China)ජᏖ(City)ᇏᐌ(Ordinary)ⱒྦྷ
ᆊ (Family)” consists of 6 words, whose corresponding SU islisted below:

the first 4 marking numbers are adopted, so the word “೑ᆊ”(country) and “Ҏ”(people) would be marked as “Dn04” and“Af01” respectively.
Word: The word in sentence or clause, such as “೑ᆊ” (country) .
No._Document: the serial number of the document where the SU
word appears. So, “XINHUA_CMN_20020102.0100” would
indicate the 100th document of ǉ ᮄ ढ ᡹ Ǌ (Xinhua News)(Chinese version) published on January 2, 2002.
No._Sentence: the marking number of the sentence in which the
SU word occurs, hence, “024” refers to the 24th sentence.
No._Clause: the marking number of the clause in which the SU
word occurs, hence, “002” represents the second clause.
Hierachical_Code: As a situation sentence may contain several
clauses, and the syntactic element may have modifiers, a SU may
have been inherited from its paternal hierarchy, yet it has its own
sub-categories as well. Take the nested structure marking of a SU
“0-011-031” for example. If the Unit_Type of the SU is“ATTRIBUTE”, the attributive of the first subject (“011”) of theroot sentence (“0”) is the No.031 of the categories of the“ATTRIBUTE”. Otherwise, if the Unit_Type of the SU is“EVENT”, the attributive of the first subject (“011”) of the root
sentence (“0”) is a clause marked by the first predicate (“031”).
Syntactical_Constituent_Frame: The major syntactic element ina situation sentence is expressed as a frame. Every threecharacters symbolize one syntactic element. For instance,“011031041” indicates the first subject “011”, the first predicate“031” and the first object “041”.

― 56 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

2.1.4 The features of the SU are as follows:I.

II.

The answer to question can be retrieved throughmatching the SUs in the questions sentence with those in the
text of corpus. Flexibility is allowed in the matching process,which means that it might get the rest of constituents of SU
by matching one of constituents of SU, or matching a
combination of multiple constituents of SU. The matching a
combined multiple constituents of SU might improveaccuracy.
As Semantic_Code constituent of SUs is directly related
to the coding in accordance with the ǉ ৠ Н䆡 䆡 ᵫǊ
(Chinese Synonym Dictionary) which includes 95 major
categories and 1428 medium categories rather than about 20
categories of semantic role labeling [9], the classification ismore specific and the treatment is more exact. In addition,
this can avoid the errors which are likely to appear insemantic role labeling.

III.

Some semantically similar synonyms can be matched at
one time by matching Semantic_Code of SU, which is moretime-efficient than the conventional keyword extension
approach which matches synonyms one by one.

IV.

As Semantic_Code of SU is given from the word insentence rather than an isolated word, it is possible to matcha word according to the exclusive meaning appointed in thesentence by matching Semantic_Code of SU. This is very
helpful to ambiguous word matching.

V.

VI.

Through adjusting the number of the No._Sentence or
No._Clause of SU by “+1” or “-1”, the relevant SU
information about the preceding or the following of thegiven sentence or clause may be accessed quickly, whichmakes it possible to get the contextual information ofsentence by cross-sentence access.

from the question sentences; matching SUs and the answer output,
as illustrated below in figure 2:

2.2.1 Extracting the SUs from the texts of corpus
2.2.1.1 Analyzing the given text corpus syntacticallyand semantically
To perform syntactic and semantic analysis of the text ofcorpus, the Chinese language processing free software package
LTP 2.0 from the Information Retrieval Center of HIT has been
applied [10]. The syntactic and semantic analysis result for “䩶⨈䖯ܹЁ೑ජᏖᇏᐌⱒྦྷᆊ”(Piano enters the ordinary family in
China city ) is shown as below:
Table 3. The sample for syntactic and semantic analysis result

2.2.1.2 Transferring the syntactic and semantic
analysis results into the syntactic and semantic frame
As it is still not an easy task to extract SU directly from thesyntactic and semantic analysis results, a syntactic and semantic
frame needs to be generated so that an SU can be transferredthrough this syntactic and semantic frame. The syntactic andsemantic frame for “䩶⨈䖯ܹЁ೑ජᏖᇏᐌⱒྦྷᆊ.”(Piano
enters the ordinary family in China City.) is shown as below˖
Table 4. The sample for syntactic and semantic frame

It is possible to know whether a syntactic element is
modified by an attributive word or a clause by the
combination of the Unit_Type and the Hierarchical_Code ofSU, which is extremely essential for getting the structuralclues of a complex sentence with hierarchical clauses. Thismight be used to analyze some complex sentences.

VII.
The original answer sentence “segmented” into SUs canbe generated by “reorganizing” the SUs of the answer
sentence and can be simplified by some rules. This meansthat the corresponding answer sentence to questions can beaccessed just through SUs while the text corpus is not used
for reference, which might save the space for corpus storage
and avoid analyzing repetitively the same text of corpus toget syntactic and semantic information..
VIII. The drawback of SU is that it needs more storage spacethan a single word, even more when the corresponding
indexes are set, though the problem may become lessprominent with the advance of the storage techniques andthe decline of the cost, coupled with some data compressing
techniques.

2.2 The Procedure of the SU-based Question
Answering System

Note: the Semantic_Code of the SU for “ⱒྦྷᆊ (Family)” is “-1”,
indicating there is no corresponding Semantic_Code. In order todo post-processing, the system will automatically produce a
Semantic_Code “Wn01” for it according to its POS “n” (noun).

2.2.1.3 Extracting the SU directly from syntactic
and semantic frame
Extract “EVENT” or “ATTRIBUTE” according to“Unit_Type”; extract the Hierarchical_Code of the SU according
to the inherited relations of the modifiers; extract the rest
corresponding information in turn by referring to the
Semantic_Code, Words, No._Document, etc. The extraction of
the SU is realized by fully extracting all of constituent information.

The working procedure is mainly divided into three sections:
extracting the SUs from the texts of corpus; extracting the SUs― 57 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

2.2.1.4 Writing the SUs of the Text of Corpus into a
Database

2.2.1.5 Using the Search Engine Sphinx SE and the
Database MySQL to Set Indexes and Speed Retrieval

Every constituent in the SU can be written into the databasefor storage. The data may be compressed according to thedifferent situations. For instance, “EVENT/ATTRIBUTE” can be
represented by a single binary code “0/1”, Words by their index in
the vocabulary list, No._Document information by the index of
their document number, etc. Compressed storage can reduce thecost of the data storing space.

In order to increase the retrieval speed, some indexes of theessential information needs to be set, such as Semantic_Code,
Syntactical_Constituent_Code, Word, etc. Our system DLUT
adopts the high quality full text search engine Sphinx SE by
Andrew Aksyonoff [11], as it is compatible with the database
MySQL, which is used for SU data storage in our system.

Figure 2: The Procedure of the SU-based Question Answering System― 58 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Table 7. The SUs matched with Word “༹䖤Ӯ
༹䖤Ӯ”
༹䖤Ӯ

2.2.2 Extracting the SUs of the Input Questions
Basically, the procedure to extract the SU of the input
questions is similar to the procedure to extract the SU from thetext of corpus except involving one more step, the question typeanalysis, an essential component of the question answering system.
Our system applies the template matching approach in which thequestion type analysis is conducted by matching it with thequestion type template bank. If necessary, the key words of thequestions are extracted and extended by Semantic_Code. For
example, if “Di02” is used for “೑ᆊ” (country) as in “ાϾ೑ᆊ”(which country) , it can cover many words which are relevant to“೑ᆊ”(country).
For example, the question “ACLIA2-CS-0009: 2008 ༹䖤Ӯ೼ ા 䞠 В ࡲ ˛ ”(where will Olympic Games 2008 be held?),
because it is known that such question's type is “LOCATION”
through the pattern matching method, the SU corresponding to thequestion word of “where” will be not used, but the system will
provide the Semantic_Code “Di02/Di03” of “LOCATION” formatching. The SUs of this question sentence are shown as below:

c. As the Unit_Type of the SU embodied by the word “2008”
is “ATTRIBUTE” , the Hierarchical_Code is “0-011” (thefirst subject “011” of the root sentence “0”), and the
Semantic_Code of the SU is “Dn04” which is a numeral andas the attribute, the Word “2008” of the SU is selected tomatch in the database, the partial matched results are shown
as below:
Table 8. The SUs matched with Word “2008”

Table 5. The SUs of question sentence “2008 ༹䖤Ӯ೼ા䞠Вࡲ˛˅Вࡲ˛”(where will Olympic Games 2008 be held?˅

d. As the Semantic_Code“Di03”of the SU is embodied by thequestion word “where” of “LOCATION” question type, theSemantic_Code “Cb25” is selected to match in the database,the partial matched results are shown as below:

2.2.3 Matching the SUs and the Answer Output

Table 9. The SUs matched with Semantic_Code “Di03”

The system is able to get answers by matching the SUs
contained in a question sentence with those in text of corpus in the
database. The specific matching conditions are determined mainlyby the question type and the matching accuracy requirement. The
following is an example on how to get the answer to the question“ACLIA2-CS-0009: 2008 ༹ 䖤 Ӯ ೼ ા 䞠 В ࡲ ˛ ”(where will
Olympic Games 2008 be held).

2.2.3.1 Matching the SUs of the Question Sentencein the Databasea.

As the Syntactical_Constituent_Code “031” of the SU
embodied by the word “hold” is as the predicate, select the
Semantic_Code “Hc05” of the SU to match in the database,the partial matched results are shown as below:
Table 6. The SUs matched with Semantic_Code “Hc05”b.

As the Syntactical_constituent_Code “011” of the SU
embodied by the word “Olympic Games” is as the subject, theword “Olympic Games” of the SU is selected to match in the
database, the partial matched results are shown as below:

2.2.3.2 Grouping the Matched SUs Based on theSame No._Document, No._Sentence and No._ClauseIn the above mentioned matching, the matched results are
obtained by matching a certain constituent of the SU of thequestion sentence, and they may scatter in different document,
sentence or clause but we only need the matched results that allSUs of the question sentence are contained in the same sentence orclause of the same document, so it is necessary to group thematched SUs based on the No._Document, No._Sentence andNo._Clause , and the matched SUs can be classified into several
groups.
As for that whether each group is in a descending orderaccording to the appointed weight of the SUs matched with thoseof the question sentence wholly or partly, in this participation, thesystem only takes the groups wholly containing SUs of thequestion sentence as the corresponding candidate answer groups
regardless of the Syntactical_Constituent_Code of the SUs
matching. For example, the two candidate groups corresponding― 59 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japanto the same No._Document, No._Sentence and No._Clause to thequestion “ACLIA2-CS-0009: 2008 ༹ 䖤 Ӯ ೼ ા 䞠 В ࡲ ˛ ”( where will Olympic Games 2008 be held ? ) are shown as below:

2.2.3.4 Generatting the Sytactical_Constituent_CodeChain
In order to generate an answer sentence the system need togenerate a Sytactical_Constituent_Code chain by the Unit_Type,Sytactical_Constituent_Code and Hierachical_Code of the all SUsof a group. For example, the three Sytactical_Constituent_Codechains of the Group 0011("XIN_CMN_20020501.0113 # 013")
are shown as below:

Table 10. The matched SUs grouped according to the same
No._Document, No._Sentence and No._Clause

ATTRIBUTE 1*0 : 681+911+471
ATTRIBUTE 1*0-911 : A41
ATTRIBUTE 1*0-911-A41: 031 + 011 + 081

2.2.3.5 Generating the Answer Sentence byMatching Sytactical_Constituent_Code Chains in
Template Bank of Generating Sentence
An answer sentence is generated according to the word orderwhich is given by matching Sytactical_Constituent_Code chainsin the pre-established template bank for generating sentence. For
example, the whole answer sentence generated by organizing the
Words of SUs with the word orders matched by the
Sytactical_Constituent_Code chain from template bank. The
details are shown as below:

2.2.3.3 Getting the All SUs from the databaseaccording to the SameNo._Document ,No._Sentence and No._Clause
As the SUs of a sentence are grouped in the same group, all
SUs of the sentence can be got from database by retrieving theNo._Document, No._Sentence and No._Clause of the group. For
example, the two groups of all SUs are shown as below:

Sytactical_Constituent_Code Chain:

Word OrderATTRIBUTE 681-911-471:[471]

Table 11. The all SUs retrieved from the database according to
the same No._Document, No._Sentence and No._Clause

[911]

[681]

Æ “ⱘ”(of) + ” 2008” +Ā༹䖤Ӯā
ATTRIBUTE
EVENT

A41:no content ( only link modifiers)
031-011-081:

[011]

[081]

[031]

Æ ࣫Ҁ(Peking) ᇚ(will) Вࡲ(held)
the whole sentence generated:࣫Ҁ ᇚ Вࡲ + ⱘ 2008 ༹䖤ӮÆ࣫ҀᇚВࡲⱘ 2008 ༹䖤ӮNotes: The above describes how to generate answer sentencesby matching Sytactical_Constituent_Code chains. However, in
this evaluation, our system directly called the whole answer
sentences from the text of corpus rather than generated answer
sentence by matching templates in template bank of generating
sentence during to time limit and unfinished template bank.

3. EVALUATION RESULTS
The average F3 score of our system in this evaluation is F3 Score = 0.1954 (Official); F3 Score = 0.3154(Auto).

3.1 Scores of Various Question Types
Table 12. The average F3 Score, Recall, Precision of various question types, Average quantity of candidates
AnswerType

BIOGRAPHYDATE

DEFINITIONEVENTLOCATIONORGNIZATIONPERSONRELATIONSHIP

WHY

ALL

Average F3
Score

0.50140.07170.31640.16380.12080.24430.15590.20780.04830.1954

Average
Recall0.87660.6

0.68210.2904

0.8

0.6

0.8

0.32080.05610.4332Average
Precision

0.11470.00950.07520.07930.0156

0.06740.02220.06720.02710.0595

Averagequantity ofcandidates

18.6

20.8

22.4

12.1

19.6

9

8.6

13.9

5.6

13.32

― 60 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

3.2 F3 Score, Recall, Precision of various Question Types according to Type Distribution
BIOGRAPHY ˖ 1-10 ˗ DATE ˖ 11-15 ˗ DEFINITION ˖ 16-25 ˗ EVENT ˖ 26-45 ˗ LOCATION ˖ 46-50 ˗ ORGNIZATION ˖ 51-55 ˗ PERSON ˖ 56-60 ˗
RELATIONSHIP˖61-80˗WHY˖81-100˗





5HFDOO
3UHFLVLRQ
)







































Figure3. The Recall, Precision, F3 of Various Question Types According to Type Distribution
The evaluation results indicate that the average recall score isı0.75 for the six types of questions, BIOGRAPHYǃDATEǃ
DEFINITION ǃLOCATIONǃORGNIZATIONǃPERSON. As
our system does not perform cross-sentence access and text
structure analysis, questions like EVENTǃRELATIONSHIP and
WHY are not taken into account. The recall scores of the six
question types are as follows:
3 of them ı0.8; 3 of them ı0.6, among which one DATE
type question might not have a wrong answer if the system record
had not been wrong. If this is taken into account, the average
recall score would reach 0.7598.
Moreover, it can be seen from figure3. that 33 of 100
questions, Recall = 1 ˈ indicating one-third of the question
sentences have successfully recalled their answers.

4. ERROR ANALYSIS
Comparing our results with the gold-standard answers of the
NTCIR-8, the errors can be categorized as follows:i.
The parser applied in this evaluation, the Chinese
language processing package LTP2.0 (June 2009 version)from the Information Retrieval Center of HIT still generates
some syntactic analysis errors. For example, because “⽣‫ܟ‬ᮃ”in “ACLIA2-CS-0037˖ࡴ㧆⣿੠ 20 Ϫ㑾⽣‫ܟ‬ᮃ݀ৌⱘ݇
㋏ᰃҔМ˛” is mistakenly processed in syntactic analysis, theresult of this question is completely wrong.
ii.
There are some question that needs cross-sentence analysis
to get the right answer, as shown by the answer to the question“ACLIA2-CS-0089 ˖Ϲ߃⋄ϢЁ೑П䯈᳝ԩ⏞⑤˛”. The
words “Ё೑” and “Ϲ߃⋄” in the answerĀ.....ᮼ೼㸼ᕄЁ೑‫ݙ‬ഄঞ␃▇ৄഄऎ䴦ᑈ㒣⌢㊒㣅ⱘ䆘䗝⌏ࡼ᮹ࠡᧁᰧǄ⼲Ꮂ᭄ⷕ᳝䰤݀ৌᘏ㺕䛁Ўǃৄ⑒㺩䱚䲚ಶᠻ㸠䭓Ϲ߃⋄......Ǆābelong to two clauses. Although our system has theability for cross-sentence analysis, it did not perform thisfunction in this evaluation, leading to analysis errors to thequestions that need such a function.
iii.
Further analysis is needed to some question types togenerate a right answer. For instance, “ACLIA2-CS-0032˖
߬㖨੠㑺㗄䗞ҔМ݇㋏” belongs to the last few candidate
documents in “XIN_CMN_20050325”. Our system fails toperform sufficient analysis in this aspect, so too many
candidate answer sentences are generated. When only the first
30 candidate sentences are used as answer, some valid answers
are excluded. As a result, the recall score for these types ofquestions are very low.
iv.
Some question sentences need to be analyzed from theperspective of a paragraph. For instance, the answer for“ACLIA2-CS-0027˖㳬㲯ЎҔМ㛑೼໽㢅ᵓϞ㸠䍄˛” hasto be generated through text structure analysis, which oursystem does not perform, finally leading to result errors. Thisis mainly shown through the “WHY” question types. Amongthe 20ĀWHYāquestions, the F3 for 18 of them is 0.
v.
As our system extracts the corresponding sentences directly
instead of simplifying them for the final answer, the final
evaluation results are very much affected. The figure 3 showsthat for many question types, the Recall = 1 ˈ but as thePrecision is very low, F3 is often very low. A case in point isĀACLIA2-CS-0041 ˖ 䇋䯂䇕ᰃ 2004 ᑈ㕢೑⇥Џ‫ܮ‬ᘏ㒳‫׭‬䗝Ҏā, whose Recall = 1 , F3 =0.0223.

5. CONCLUSION
This paper describes the question answering system DLUT
for the evaluation of NTCIR-8 CS-CS SubTask. Owing to the time
limit, the rich resources of the SU are only used partly in oursystem. So the preliminary conclusion to be reached so far throughour present experiment is that a better recall score may be got by
an SU-based question answering system in view of the average
recall score ı 0.75 for the six question types BIOGRAPHY,DATE, DEFINITION, LOCATION, ORGNIZATION, PERSON.
The errors of word segmentation and named entity recognition
may result in analysis mistakes. The performance of the parser
decides the quality of the whole system. If answers output in theform of whole sentences instead of simplified answers may greatly
affect the final F3 Score. The extracting answer without crosssentence access and text structure analysis has a big impact on the
results. All this is to be improved in our future system.

ACKNOWLEDGEMENTOur gratitude goes to the Information Retrieval Center of HIT
for their Chinese language processing package LTP2.0.

― 61 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

REFERENCES[1] Boris Katz, Gary Borchardt, et al. 2005.Syntactic and
Semantic Decomposition Strategies for Question Answeringfrom Multiple Resources. Proceedings of the AAAI 2005
Workshop on Inference for Textual Question Answering, 3541, July, 2005.
[2] Hyeon-Jin Kim, Ji-Hyun Wang, et al. 2005. LF Based
Answer Indexing Method for Encyclopedia QuestionAnswering System. AIRS 2005, LNCS 3689, pp. 679–684.
[3] Barney Pell. 2007. Natural Language and the Semantic Web.
ISWC 2007.
[4] Daniel G. Bobrow, Bob Cheslow, Cleo Condoravdi, et al.
2007.PARC's Bridge and Question Answering System.
Proceedings of the GEAF2007 Workshop.
[5] Dan Shen, Mirella Lapata. 2007. Using Semantic Roles to
Improve Question Answering. Proceedings of the 2007 Joint
Conference on Empirical Methods in Natural Language
Processing and Computational.
[6] Jon. Barwise, John Perry. 1983. Situations and Attitudes ,Bradford Books, The MIT Press.
[7] Keith Devlin. 2008. Situation theory and situation
semantics.Handbook of the History of Logic, Volume 7,
Elsevier, pp. 601-664.
[8] Masahito Kawamori. 1991. A Situation Semantic Approachto Argument-roles in Japanese. THE SIXTH JAPANESEKOREAN JOINT CONFERENCE ON FORMAL
LINGUISTICS.
[9] Daniel Gildea, Daniel Jurafsky. 2002. Automatic labeling of
semantic roles. Computational Linguistics, 28(3):245–288.
[10] http://ir.hit.edu.cn/demo/ltp/[11] http://sphinxsearch.com/― 62 ―