{shilixin, nie, caogui}@iro.umontreal.caRALI, Dept. d'Informatique et de Recherche
Opérationnelle
Université de Montréal

Lixin Shi, Jian-Yun Nie and Guihong Cao

RALI Experiments in IR4QA at NTCIR-7

••

oo

Mining personal names and their translations:•
Detecting entries in Wikipedia with patterns “{{BD|…}}”,“[[Category:<year>(出生|逝世)]]”…
•
Translations are from the cross-referenced entries in different languages.
•Using a search engine to find translation according to the pattern“English-name (Chinese-name)” or “Chinese-name (English-name)”
Biography-Related Keywords: the most distinctive words which have a highfrequency in biography documents and a low frequency in other documents

METHOD: create a better IR model by using external knowledge for eachtype of question

documents that contain answers to different types ofquestion.

GOAL: what information retrieval techniques can help identify

Overview

Personal name(from Wikipedia)

ICTCLAS

Indri

Passagesegmenter• Use Indri from Lemur Toolkit as our basic IR system• Documents are segmenting into words by a word segmentation tool – ICTCLAS• Personal names extracted from Wikipedia are added into the dictionary of segmentation tool

• Using passage retrieval method• Documents are cut into overlapped passages• The title of document is duplicated to each passages.
• Index unit is segmented word

Documents

wordsegmentation

Document processing

t1 ,..., t n , # max( p11 ,.. p1m1 ), # max( p 21,.., p 2 m2 )...

 Expanding personal name variants (PNE):

wbio # wsum ( w1 ,... w m ) )# weight (1 .0 # combine (t1 ,..., t n ) Expending biography-related words (biography questions) (Bio):

wbi # combine (#1(t1 , t 2 ), #1(t 2 , t 3 ),...) )# weight (1 .0 # combine (t1 ,..., t n ) Combining words and word-Bigrams (Bi):• Enhancement models under Indri framework: Baseline model: based on language modeling approach andconsidering segmented words only.

Retrieval Models









Relationship question. A passage must include both of therelationships words.
Definition question. The term to be defined should have definition
words following it.
Biography question. We consider some typical three-word patterns(triples) for biographies. For example “name/年(year)/出生(born)”,“name/加入(join)/null”. We have extracted about 600 triples from
Wikipedia biographies.
Event. No special rule is used.

 Passage Filtering:where vi is the value of ith best retrieved passage of a document, and vdoc isthe document relevant value to be assigned

 The document score is assigned according the following two ways:(1) Best-P: vdoc = v1
n +1v
=v+v
/
4+v
/
8+...
+v
/23n(2) All-P: doc 1 2

From passages to the document

1926

3263

Collection

XinHua &ZaoBao

CIRB020&
CIRB040

CSCT

9597

#Q

.7873

.5839

AP

MscDcG1K.8270

MscDcG1K

.6018

.6490

Q-measureQ-measure

.6337

AP

Best-1
Official.7656

.5724

.5521

.7951

.6055

.5930

Best-2
Official.6516

.4096

.3952

.7242

.4812

.4684

RALI*
Official

.7582

.5678

.5530

.8235

.6486

.6399

Best-P

.7629

.5747

.5596

.8269

.6521

.6428

All-PBaseline Our official submission contains a bug.
 The corrected baselines (none of enhancements), Both Base-P and All-P, are slight better (forCS) or close to (for CT) the best official results.

#doc.
(K)

(Question type: List/Event, Biography, Definition, Relationship)

Characteristics of IR4QA Chinese collections and Baselines

Monolingual IQ4QA







.6485
.8235
.5530
.5678
.7582

Q.

M.

AP

Q.

M.

%B

%B

Bio+Bi+
PRF+PNE

-0.1

.7863

.5561

.5530

-0.2

-0.3

-0.1

+0.1

+0.0.5517.7591 +0.1* .7582+8.3**+4.5**+7.0** .5824 +2.6** .6150+4.0** .7692 +1.5* .7923

0.0.7883.6076+8.5**-0.2 .5896 +6.6** .5670 +2.5** .6002-0.3+4.5**+0.8.8603.8265 +0.4 .8507 +3.3** .8303.5688 +0.2* .5658.5538.8243 +0.1 .8238.6390+7.5**%B

PNE.6486 +0.0 .6499 +0.2* .6519 +0.5 .6848 +5.6** .6569 +1.3* .6972%B

PRF(0.5)+7.6**%BBigram(0.1).6412 +0.2* .6429 +0.5 .6763 +5.7** .6495 +1.5* .6888%B

Bio(0.2)

Bio: small improvement.
The assumption that biography questions aim to find biographies may be too strong for theIR4QA task.
Bi: slight improvement of MAP for CS, but not for CT collection.
PNE: Helpful, most of the improvements are statistically significant
PRF: can greatly improve MAP
Bio+Bi+PRF+PNE: better than using any single technique alone, outperforms largely the best
official Chinese monolingual result.

CTCS.6399

APBaseline

FilterImpact of using biography keywords, filtering,
bigrams, PRF, and PNE



AP
Q.
M.
AP
Q.
M.

.5959
.6124
.7947
-Best
official.4033
.4191
.6701
.2723
.2868
.4845

RALI
official.5526
.5606
.7304
.3067
.3258
.4914

GoogleTrans.
G+W
.5801
.5927
.7768
.3561
.3769
.5512

%G+5.0+5.7+6.4*+16.1*+15.7*+12.2*

Google+Wiki
.5830
.5957
.7798
.3605
.3813
.5524

%G+W
+0.5+0.5+0.4
+1.2*
+1.2*
+0.2

Filter.6214
.6355
.8084
.3883
.4108
.5945

%G+W+7.1**+7.2**+4.1**+9.0**+9.0**+7.9**

Bio+Bi+PRF

Use Google to translate the whole question except the personal names.
Translations of the personal name obtained from Wikipedia.

“G+W”: increased from G, closer to the best submission of EN-CS and better
than best submission of EN-CT.
Bi, Bio, and PRF: further improvements.

EN-CT

EN-CS

(1)
(2)

Question Translation:

Cross-lingual IR4QA By combining all the enhancements, weobtained very good results and theimprovements over the baseline methods are
statistically significant for both mono-lingualand cross-lingual QA
 This series of experiments showed thatspecific knowledge for each type of questionis highly useful.

Conclusion