NiCT/ATR in NTCIR-7 CCLQA TrackYouzheng WU, Wenliang CHEN, Hideki KASHIOKANiCT/ATR, JapanNTCIR CCLQA
 Complex Cross-Lingual Question Answering.
List/Event questions
List major events in formation of European Union.
R l i hi questions
Relationshipi
Does Iraq possess uranium, and if so, where did it come from?
Biography questions:
Wh iis H
WhoHowarddDDean??
Definition questions:
What are stem cells?
Questions in English and getting answers from Chinese((SimplifiedSimplified,p, Traditional)) or JapaneseJpcorpuspRelated studies
 Pattern-matching-based [Xu, et al. 2005] [Harabagiu,et al. 2004] [Cui, et al. 2004]
Basic syntactic/semantic structures like appositives, copulas;predicates and relations.

 Centroid-vector-based [Xu, et al. 2003] [Chen, et al.

2006] [Kor, et al. 2007]


Build a target profile for each question, and then compute thesimilarities between candidates and the target profile.

 Others
O h [Bi[Biadsy,d et al.l 2008]
Unsupervised classification model to Biography production using
Wikipedia.
Wikipedia

Centroid-vector-based
WikipediaBiography.com
W dN
WordNetGoogle Definition
NewsLibrary.com
yGoogleCentroid-vector-based cont.
 Easy to implement, and fast in speed
 InI essence, a type off question-sidei
id expansion.
i Hard to obtain sometimes


Wikipedia, WordNet, and Biography.com contain only
82.0%, 40.4%, and 24.6% of TREC05 questions,respectively.
i l Not always contribute positively
Wikipedia negatively impacts Biography questions [Kor,et al. 2007].

Our solution
SVM-based model
Centroid-vector-based model
Regarding complex QA as a SVM
SVMas a retrieval process
based classification
Applying
pp y g sentence-sidequestion-side expansion
qpexpansion
Requiring no specific resources, A number of external resources
except a general search engine (Google)

such as Wikipedia, Biography.com,
WordNet, etc.

Incorporating multiple features

TF-IDF-similarity
TF
IDF similarity score

SVM-based
Same to centroid-vector modelLearning Evidences by Sentence-side
Sentence sideExpansion
For each si in S
1
1.

Extract 2 or 3 nouns nearest to question target from candidate
si as topic terms of si, labeled as R.

2.

Combine topic terms R and question target to compose a web
query and submit it to Google.

3.

Download the top 100 Google snippets.

4.

Retain those snippets {ei,i1, …ei,ik} that contain words inquestion target and R as Web evidences for candidate si.

endAn Example
Who is Anwar Sadat

Question target...
c255 = before zoweil,, late egyptian
gyppresident anwar sadat wonp
the nobel prize for peace after making peace with israelin 1979 ...
...
c274 = in 1970 , anwar sadat was elected president of egypt ,succeeding the late gamal abdel nasser ...

Topic terms

An Example

Bridge lexical
gap between
candidates andprofileSVM-based modelTrain-Classifier
Features

Description

Bfull

If question target occurs in the exact form.

BbeginIf question target occurs at the beginning of an instance or not

Bpattern

If one of predefined patterns occurs or not

Btime

If time expression occurs or notnot. Candidates with time expression
tend to capture important events involving target

Unigram-overlap Overlap of unigrams between an instance and target profile
Bigram-overlap

Overlap of Bigrams between an instance and target profile

TF-IDF similarity TF-IDF-based similarity between an instance and target profile
Freq

The number of relevant pages returned by Google

SVM Classifier

Rules for the Train-Classifier Manually generate from the abstract of Wikipedia.
 Useful to Biography
g p y and Definition questions.
qSelect-Answer

descennding ordeer

Ideal AAnswer<tp1, p1>
<tp2, p2>...
<ttpj, pj>...
<tpn, pn>assuming that all
the features fire

Which topic is thenearest topic to ideal
answerj=15 if the tpj > average probability (1/n)
j=10 else;

Experiments
 Three runs for the EN-CS and CS-CS tasks
 RUN-3: The Centroid-vector model (55 external resources).
resourcesresources)

Wikipedia (0.2 million Chinese entries);
 Baidu Baike (1 million Chinese entries, http://baike.baidu.com);
 Google Definition (e.g., define: Nobel prize); Google news (1000 news sources updated continuously); Google.
Google RUN
RUN--1: The SVM
SVM--based model (Google).
 RUN
RUN--2: The SVM
SVM--based model (5 external resources).
resources)
To compare with the RUNRUN-1

Official Result Three findings: 1. the precision is low; 2. the ranking ofdifficulties of answeringg questions;q; 3. Cross-lingual
g vs.
monolingual (only 10% include at most 1 error).

The best(the secondis 19.30%)
The second(the best is43.29%))Comparison of the Three Runs
EN-CS task
RUN-1

RUN-2

RUN-3

E t
Event

14 54
14.54

14 08
14.08

8 09
8.09

Definition

22.16

23.37

12.57

Biography

31 58
31.58

30 27
30.27

20 77
20.77

Relation

23.35

22.80

12.10

all

22.11

21.79

12.73

CS-CS task
Event

14.30

14.07

10.86

Definition

24.15

25.65

16.18

Biography

33.76

32.53

18.06

Relation

24.29

23.76

16.50

all

23.16

22.98

15.06

The conclusion:• The proposed SVM-basedmodels are much better
than the baseline bycomparing the RUN-1 andRUN-2 with RUN-3.
• Target profile does not play
an important role in the
proposed SVM-basedmodeld lb
by comparing thehRUN-1 with RUN-2.

Automatic Scores
The best((thesecond is22.90%)
The second(the best is37.75%)

IR4QA + CCLQA
 Three retrieval results are CMUJAV1-EN-CS-01
01-T-limit50, CMUJAV1CMUJAV1-EN-CS-02
EN CS 02
02-T-limit50,
T limit50, and MITEL-ENMITEL EN
CS-01
01-T-limit50.

>>

IR4QA + CCLQA
Table 8
8. The Mean
Mean-APAP scores of the CJAV1 and the
MITEL over types of questions

CJAV1

MITEL

Event

19.53

< 26.57

Definition

48.65> 37.10Biography

46.65> 45.15

Relation

32.00< 41.37

T bl 9Table
9. Th
The FF-scores off theh RUN
RUN-1
1 basedb d on theh CJAV1and MITEL over types of questions

CJAV1

MITEL

Event

17.74

< 18.92Definition

23.08> 17.23

Biography

38.37> 37.87

Relation

33.40< 36.01

The conclusion:• The impacts of the IR4QA
system on the CCLQA
system are roughly
consistent.
• However, the extent of theimpacts are not the same.

Discussion
 Hard to directly evaluate the quality of web evidences

learned by sentence-side
sentence side expansion


Has the same underlying logic as that of the ROUGE metric [Lin, etal. 2003] and the nugget-pyramid metric [Lin, et al. 2006]: use
unigram overlap to match semantically Speed problem
Have to train an SVM-classifier for each question.

Summary Propose an SVM-based classification model for NTCIR

complex QA system:
Each candidate represents a topic
 Learn
earn training data for each topic by sentence
sentence-sideside expansion
 Assume an ideal answer, and classify this ideal answer into topics
to find real answers
 The SVM-based model achieves competitive

performances, and relies on no specific external
resources other than Google.

Thanks!