DIRO, Université de Montréal

Lixin Shi, Jian-Yun Nie, Guihong Cao

RALI Experiments in IR4QA
at NTCIR-7

What information retrieval techniques canhelp identify documents that containanswers to different types of question.

◦ passage/document retrieval (ACLIA-IR4QA)◦ answer identification within the retrieved
passages/documents (ACLIA-CCLQA)

2

Question answering is usually composed oftwo main steps:

Goal

3

We try to create a better IR model by using
some knowledge about each type of question.

◦ List/Event, Biography, Definition, Relationship

In NTCIR7 ACLIA, we deal with four types ofquestions:

Basic idea◦ Wikipedia is used in order to improve proper names
translation.

In Ferrández et al'07, the external sources
are also used for cross-lingual questiontranslation.

◦ Chan et al’07, Kata et al'07, …

Many previous studies have tried to use
external sources for question answering in
the later answer identification phase.

Related works

4

Google translate (+enhanced personal name translation)◦ Translation
5

Wikipedia contains articles about different persons in different
languages
Google search: used to identify additional translations of personalnames◦ Personal name and translation

Biography-documents are more likely to contain biographyrelated keywords (birth, education, …)
Extract these keywords from Wikipedia.

In our experiments, we intend to exploit external
knowledge more extensively at the retrieval phase.
◦ Biography question:

We will consider …

2. DOCUMENT/QUESTIONPROCESSING
AND RETRIEVAL MODEL

2008-12-24

Accuracy: 92% for Chinese questions and89% English questions.

7◦ E.g. questions include keywords 谁是, 是何人, 是哪位,生平事迹 or “who”, “biography” biography category.

Type of question: We used a simple rulebased approach for question classification
based on keywords.

Question analysis

2008-12-24

8

Passage retrieval:◦ Window model: text portions with fixed size
Choose the index unit
◦ Cutting into n-grams – unigrams or bigrams ofcharacters◦ Segmenting into words by a word segmentation tool(ICTCLAS), but with an augmented dictionary (personalnames)

Document processing

2008-12-24

9Translations are from the cross-referenced entries
in different languages.
For some personal names not in Wikipedia, using
a search engine to find translation according to thepattern “English-name (Chinese-name)” or“Chinese-name (English-name)”.

{{BD|1887年|10月31日|1975年|4月5日|catIdx=C蒋}}“{{BD|…}}”, …
◦ E.g. [[Category:1887 births]]

Detect personal name entries in Wikipedia: using
Template such as “[[Category:<year>(出生|逝世)]]” ,Extraction of personal names and theirtranslations from external resources◦2008-12-24

E.g. 出生(born), 毕业(graduate), 大学(college,university), 岁(age, year).
10Biography-related words: the most distinctivewords which have a high frequency in biography
documents and a low frequency in otherdocuments
Extracted 110 biography related words

Separate Wikipedia documents into two groups:(1) Biography: title is a personal name(2) Other: title is not a personal name

Extracting Biography-RelatedKeywords

Wikipedia personal name2008-12-24

之间的关系是什么x和跳舞 Yang Liping 杨丽萍之间的关系是什么杨丽萍和跳舞Google Translation

E.g. What is the relationship between Yang Liping and dance?

NEpersonx(1) Use Google to translate the whole question except the personalnames.
(2) Translations of the personal name obtained from Wikipedia.

11Google translation, but it fails to translate several personalnames
Two-stage translation approach

Question Translation

2008-12-24

t1 ,..., t n , # max( p11 ,.. p1m1 ), # max( p 21,.., p 2 m2 )...

Expanding personal name variants

w bio # wsum ( w 1 ,... w m ) )# weight ( 1 . 0 # combine ( t 1 ,..., t n )12Expending biography-related words for biography
questions (wbio=0.2)

wbi # combine (#1(t1 , t 2 ), #1(t 2 , t 3 ),...) )# weight (1.0 # combine (t1 ,..., t n )Combining words and word-Bigrams (wbi=0.1)

Retrieval Models

2008-12-24◦ E.g. “name/年(year)/出生(born)”, “name/加入(join)/null”
13

Passage Filtering:
We define some patterns to impose a penalty to documentsthat do not match patterns: 600 triples for biography
questions form Chinese Wikipedia.

where vi is the value of i-th best retrieved passage of a document, andvdoc is the document relevant value to be assigned

We perform passage retrieval, the candidate documents(required in IR4QA) is determined in the two following
ways:(1) Best-P: vdoc = v1
n +1(2) All-P: vdoc = v1 + v2 / 4 + v3 / 8 + ... + vn / 2

From passages to the document

3. EXPERIMENTS

2008-12-24◦ CS-CS (5 runs), CT-CT (5 runs), EN-CS (4runs), EN-CT (4 runs).
Personal name expansion, biography-related,combination of words with bigrams and “All-P”
for all of our official runs.
Pseudo-relevant feedback, passage filteringfor some of runs

We submitted four sets of IR4QA results:

Official Results

15

.6337
.6490
.8270
.5839
.6018
.7873
.5724
.7656

.5930
.6055
.7951
.5521
.4684
.4812
.7242
.3952
.4096
.6516

Best-1 Best-2 RALI*
Official Official Official
Best-P
.6399
.6486
.8235
.5530
.5678
.7582

All-P
.6428
.6521
.8269
.5596
.5747
.7629

BaselineCT

Our official submission is not good due to a bug in our
program for document re-ranking.
Highest novelty score: 63 unique documents for CS and 32 forAP
CS Q-measure
MSnDCG1KAPCT Q-measure
MSnDCG1K(Title only)

16BASELINE OF MONOLINGUAL

%B

%B

Bio(0.2)%B

Bigram(0.1)
%B

PRF
(0.5)

%B

PNE

%B

Bio+Bi+
PRF+PNE

AP .6399 .6390 -0.1 .6412 +0.2* .6429 +0.5 .6763 +5.7 .6495 +1.5* .6888 +7.6

Baseline

FilterImpact of using biography keywords,
filtering, bigrams, PRF, and PNE

C Q. .6485 .6486 +0.0 .6499 +0.2* .6519 +0.5 .6848 +5.6 .6569 +1.3* .6972 +7.5
S
M. .8235 .8243 +0.1 .8238 -0.0 .8265 +0.4 .8507 +3.3 .8303 +0.8 .8603 +4.5

AP .5530 .5530 -0.1 .5538 +0.1 .5517 -0.2 .5896 +6.6 .5670 +2.5 .6002 +8.5⁂⁂⁂

2008-12-24

Q: Q-measure, M: MscDcG1000, ⁂: t-test≤0.01, *: t-test≤0.05⁂

17C Q. .5678 .5561 -0.3 .5688 +0.2* .5658 -0.3.6076 +7.0 .5824 +2.6 .6150 +8.3
T
M. .7582 .7863 -0.2 .7591 +0.1* .7582 0.0 .7883 +4.0 .7692 +1.5* .7923 +4.5

⁂⁂

⁂

⁂

⁂

⁂⁂

⁂

⁂

⁂

2008-12-24

Overall, the improvement in MAP is relatively small.
The assumption that biography questions aim to findbiographies seems too strong for the IR4QA task

CS-T68谁是张瑞敏 (Who is Zhang Ruimin): 0.9111 → 0.9049 (-0.7%).

◦ 3 topics decreased by more than 0.5%.

18CS-T376谁是奥尼尔(Who is Shaquille O'Neal): 0.6215 → 0.6659 (+7.1%)
CT-T194谁是小泉纯一郎(Who is Junichiro Koizumi): 0.5630 → 0.5940 (+5.5%)40 biography questions for CS and CT.
◦ 18 topics have been improved by more than 0.5%
in MAP

Impact of the biography words (Bio)Personal name expansion is helpful.

2008-12-24◦ CS-T42谁是本拉登 (Who is Osama bin Laden).
PNE: 本拉登 → 宾拉登, 乌萨玛 (Osama), 拉登, …, 拉丹.
MAP: 0.0508 → 0.2112 (+315%).
◦ CS-T324谁是小泉纯一郎 (Who is Junichiro Koizumi).
PNE:小泉纯一郎 → 小泉 (Koizumi).
MAP: 0.6147 → 0.7196 (+17%)

19

34 topics contain various personal names.
After PNE, 12 topics are improved more than 1% in MAP,
Others topics are only changed within ±0.5%.

Impact of the personal name expansion(PNE)

2008-12-24

Slight improvement for CS collection, and no for CT.
20◦ Topic CS-T385: 列出与北京大学百年校庆相关的大事 (Listevents related to the Centenary Celebration of Peking University).
0.7083 drops to 0.6741.
◦ 百年-校庆 (Centenary Celebration) for other universities

However, also some negative results.

◦ MAP using word bigrams 法国-世界杯: 0.03050.2180.

(1) France World Cup(2) France football team in any World Cups

◦ E.g. 法国 (France) and 世界杯 (World Cup) in 列举法国世界杯引发的争议(List the disputes triggered by the France World Cup)

Solve some ambiguities of independent words.

Impact of combing bigram (Bi)2008-12-24

Pseudo-relevant feedback usually
produces better result for IR, especiallyfor short queries.
PRF greatly improved MAP and all theimprovements are statistically significant.

21

Impact of the pseudo-relevant feedback
(PRF)

-M.

-AP
-

.7947

M.

Q.

.6124

Q.

.5959

.4845

.2868

.2723

.6701

.4191

.4033

.4914

.3258

.3067

.7304

.5606

.5526

RALI
Googleofficial online Trans.

.5512

.3769

.3561

.7768

.5927

.5801

+12.2*+15.7*+16.1*

+6.4*

+5.7

+5.0%GGoogle+Wiki

2008-12-24

Using both Google and Wikipedia (G+W), theperformance is largely increased
Closer to the best submission in the case of EN-CS
Google translation in EN-CS is better than in EN-CT

EN-CT

EN-CS

AP

Best
official

22CROSS-LINGUAL RESULTS

EN-CT

EN-CS

.5512

M.

.3561

AP
.3769

.7768

M.

Q.

.5927

.5801

Q.

AP
.5957
.7798
.3605
.3813
.5524

+6.4*
+16.1*+15.7*+12.2*

.5830

+0.2

+1.2*

+1.2*

+0.4

+0.5

+0.5

%G+W

Filter

+5.7+5.0%GTGoogle+Wiki.5945

.4108

.3883

.8084

.6355

.6214

+7.9⁂+9.0⁂

+9.0⁂

+4.1⁂

+7.2⁂

+7.1⁂

%G+W

Bio+Bi+PRF

2008-12-24

.7923

.6150

.6002

.8603

.6972

.6888

23+4.5+8.3+8.5+4.5+7.5

+7.6

%B

Mono: Bio+Bi+ PRF+PNE

Using filtering, biography, bigram, andPRF techniques

⁂

⁂

⁂

⁂

⁂

⁂

◦ Pseudo-relevance feedback (**)◦ Personal name expansion (*)◦ Biography-related words◦ bi-grams of words◦ Filtering

2008-12-24

By combining all the enhancements, we obtained
very good results and the improvements are
statistically significant for both mono-lingual andcross-lingual QA.

CONCLUSION

24

Thanks

Questions?