presented in NTCIR-7
2008-12-18

Multilingual Interaction Technology Laboratory (MITEL)
Institute of Computing Technology, Chinese Academy of
Sciences

Weihua Luo, Tian Xia, Ji Guo, Qun Liu

ICT-Crossn: The System of Crosslingual Information Retrieval of
ICT in NTCIR-7

INSTITUTE OF COMPUTING TECHNOLOGY12/18/2008

z Conclusion

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Results on the formal run

z Experiments on the dry-run set

z Document retrievalz Query translationz The system architecturez Background

Outline2

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

z Conclusion

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Results on the formal run

z Experiments on the dry-run set

z Document retrievalz Query translationz The system architecturez Background

Outline3

INSTITUTE OF COMPUTINGTECHNOLOGY

z



















CS-CS
EN-CS √
CT-CT √
EN-CT
JA-JA
EN-JA
IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

CLIR has made a great progress
Find out which IR technique would help CCLQA

Subtasks

A novel task in ACLIA task cluster in NTCIR-7
Motivation

Advanced Cross-lingual Information Access (ACLIA)z Information Retrieval for Question Answering (IR4QA)

Tasks in NTCIR-7

12/18/2008z

Background

4

INSTITUTE OF COMPUTINGTECHNOLOGY<QUESTION LANG="EN"><![CDATA[ List the hazards of global warming.]]></QUESTION><QUESTION LANG="CS"><![CDATA[ 列举全球气候变暖的危害。]]></QUESTION><NARRATIVE LANG="EN"><![CDATA[ Users need to know the harm of global warming to humanbeings and the environment.]]></NARRATIVE></NARRATIVE></TOPIC><NUM>001</NUM><SLANG>CH</SLANG><TLANG>EN</TLANG><TITLE>Time Warner, American Online (AOL), Merger, Impact</TITLE><DESC>Find reports about the impact of AOL/Time Warnermerger.</DESC><NARR><BACK>Time Warner and American Online (AOL) announced a merger on
January 10th, 2000. The market value was estimated at $US350 billion
making it the biggest merger in the US.</BACK><REL>Comments on AOL/Time Warner merger's effects on Internet andentertainment media businesses are relevant. Descriptions of thedevelopment of the AOL/Time Warner merger are partially relevant.
Information about the total amount and the transformation of ownership
structure are irrelevant.</REL></NARR>

12/18/2008</TOPIC><![CDATA[ 用户需要知道全球气候变暖对人类和环境有什么危害。]]>

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu<CONC>Time Warner, American Online, AOL, Gerald Levin, merger, M&A,
Merger and Acquisition, media, entertainment business</CONC><TOPIC ID="ACLIA1-CS-T41"><TOPIC><NARRATIVE LANG="CS">

IR4QA

5

INSTITUTE OF COMPUTINGTECHNOLOGY

CLIRz Query

Difference between CLIR and
IR4QA

z

z

Accurate keywords
Detailed background and relevance judgement

TITLE Æ DESC Æ NARR

z

z

z

QUESTION ≈ NARRATIVEIR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

No keywords available
No background and relevance judgement in detail
Question type provided with topics



IR4QA
z Bilingual description of INz One grained description of IN

z

z



CLIR
z Monolingual description of Information need (IN)z Different grained description of IN

Query

12/18/2008z

Difference between CLIR and
IR4QA

6

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008zz

AP--primary
Q
nDCG



F-scoreIR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z End-to-end evaluation with a QA system







z IR evaluation

IR4QA

z Traditional IR metrics: AP

CLIRz Metrics of evaluation

Difference between CLIR and
IR4QA

7

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

8

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

z Conclusion

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Results on the formal run

z Experiments on the dry-run set

z Document retrievalz Query translationz The system architecturez Background

Outline9

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

z

z

z







IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

The judgement of document relevance is vague for a newtask
Which IR model is more suitable?

Improper translation of a few words may lead to retrieval ofmore irrelevant documents

10

INSTITUTE OF COMPUTINGTECHNOLOGY

Modification should be
Different evaluation metrics (reference set)made!

z Shorter query

Major concerned issuesz IR4QA is still an IR task

A traditional CLIR framework still worksz Basic idea

Our Implementation

12/18/2008

Dictionarybased,
corpusbased, MT
modulebased…

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

document list

Result Reranking

INSTITUTE OF COMPUTINGTECHNOLOGY11

VSM model,
Probabilistic
model,LM model…

Formal Representation

CS Document

Relevance Estimation

Formal Representation

CS Query

Query Translation

EN Query

Traditional CLIR Flowchart12/18/2008

use ofintermediate dataof SMT

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

document list

Result Reranking

INSTITUTE OF COMPUTINGTECHNOLOGY

12

Borrow Idea ofsystem combinationof SMT

Formal Representation

CS Document

Relevance Estimation

Formal Representation

CS Query

Query Translation

EN Query

Our Improvement for IR4QA

12/18/2008

ICT-Crossn

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

13

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

z Conclusion

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Results on the formal run

z Experiments on the dry-run set

z Document retrievalz Query translationz The system architecturez Background

Outline

14

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008zz

INSTITUTE OF COMPUTINGTECHNOLOGY

engines
IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Corresponding translation may be found with searchz Any dictionary has a limited coverage

15

z Many OOV words provide key information of questions

OOV words translationz Phrase based SMT creates huge phrase tablesz Phrases resolve some ambiguities

z Full text translation is unfit for short questions

Phrase translationz Basic ideas

Query Translation

INSTITUTE OF COMPUTINGTECHNOLOGY

12/18/2008z

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

16

Trained on 5M pairs of EN-CS sentencez Word alignment with Giza++z Phrase extraction and probabilities estimation witha tool in Mencius (our phrase-based SMT
decoder)(F.J. Och, Hermann Ney, 2008)z Size up to 17GBz The phrase table is filtered by the test set

z Phrase table

Phrase Translation

1 0.000132428 0.5 0.000141711 2.718 1

0:0 0:1 1:3
0:0 1:1 2:2

to coordinate its own 2:0 3:1 3:2 3:3

0:0 2:1 2:2 3:1

0:0 1:0

0:0 1:2

0:0 1:0

0:0 1:0 1:1" to deal with" so that the" so that

Broadcastbroadcast and tv

broadcasting

TV station

feeding radio and televisionInternational Development Bureau

International Development Bureau Assistance 0:0 1:1 2:2 3:3" , 以 对付" , 以 协调" , 以便 使" , 以便 使

广播 电视

广播 电视

广播 电视

广播 电视台广播 电视台国际 发展 局国际 发展 局 援助

0.333333 6.48644e-05 0.5 0.000702031 2.718 1

0.2 0.19695 0.75 0.0735244 2.718 1

0.818182 0.0421837 0.115385 0.0142804 2.718 1
0:2 0:3 1:0

0.4 0.042495 0.0833333 0.0592912 2.718 1
0:0 0:1 1:2
0:0 0:1 1:2
0:0 0:1 1:2
0:0 0:1 1:2of the Foreign Affairs 0:2 0:3

0:1 0:2

0:1 0:2

the Foreign Affairs

the foreign affairs

personnel of foreign affairs

Affairs Committee

Foreign Affairs Committee ,Foreign Affairs Committee of theForeign Affairs Committee of

Foreign Affairs Committee

外事

外事

外事

外事 人才

外事 委员会

外事 委员会

外事 委员会

外事 委员会

外事 委员会

12/18/2008

0.0384615 0.0118484 0.25 0.069292 2.718 1
0.75 0.0421837 0.0769231 0.00190038 2.718 1

international terrorism 0:0 1:1 2:1

国际 反 恐0:0 1:1

0:0 1:1 2:2

international counter terrorism

国际 反 恐IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

0.763285 0.0268476 0.365741 0.0120779 2.718 1

0.777778 0.0268476 0.0324074 0.00160727 2.718 1

0.666667 0.0268476 0.0185185 0.000445208 2.718 1

0.8 0.0268476 0.0833333 0.00196545 2.718 1

1 0.000712337 1 9.76949e-05 2.718 1

0.214286 0.00529871 0.0384615 0.0015063 2.718 1

1 0.0077033 0.166667 0.000422512 2.718 1

1 0.0935168 0.166667 0.0128866 2.718 1

0.6 0.111924 0.0714286 5.25604e-05 2.718 1

0:0 1:1 2:2

international anti terrorism

国际 反 恐1 0.111924 0.0238096 8.55327e-06 2.718 1

0:0 1:1 1:3
0:0 1:1 1:3

International Court of Justice ,0.5 0.19695 0.25 0.020366 2.718 1International Court of Justicethe international law

国际 法

1 0.00106589 1 1.82349e-07 2.718 1

国际 法庭

0:0 1:1

0:1 1:2

international law

国际 法

0.117647 0.000173761 1 0.091007 2.718 1

0:0 1:1 2:2 3:3 4:4

1 0.0941678 1 2.42523e-05 2.718 1

1 0.280224 1 0.00156186 2.718 1

1 0.0685612 0.5 2.82344e-05 2.718 1

0.0754717 0.00252064 0.5 0.0257354 2.718 1

0.142857 0.0856858 0.166667 0.0846071 2.718 1

1 0.106304 0.0833333 0.000460165 2.718 1

0.444444 0.105186 0.166667 0.024001 2.718 1

0.333333 6.48644e-05 0.5 0.00253444 2.718 1

国际 法庭

0:0 1:0 1:1 2:0

Delegation Visits国际 发展署 代表团

1 0.00932347 1 0.00318336 2.718 1

1 0.0129453 1 0.0073817 2.718 1

0.333333 8.20519e-06 1 1.46173e-06 2.718 1

国际 发展 局 援助 局 International Development Bureau Assistance Department

0:0 2:1 2:2 3:1

0:0 2:2 3:2 3:3

1 0.000132428 0.5 0.000511601 2.718 1" to deal with the" , 以 对付0:0 2:2 3:2 3:3

0:0 1:2 2:2 3:1 3:2 4:3 5:4

0:0 1:2 2:2 3:1 3:2 4:3

1 0.018866 1 0.120792 2.718 1! 三 井 住友 金融 集团 ! Sumitomo Mitsui Financial Group! Sumitomo Mitsui Financial! 三 井 住友 金融

0:0 1:2 2:2 3:1 3:2! Sumitomo Mitsui! 三 井 住友

Sample of CS-EN Bilingual Phrase Table

17

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

OOV words
translation

Not found

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu人类基因组序列测定人类基因组序列确定…

Translation assemble人类基因组序列测定，确定，决心

Human genetic sequence, determination

Choose the combination
minimizing the joining times

Human, genetic sequence, determination,
Human genetic, sequence, determination,
Human genetic sequence, determination…

Sub-phrases enumeration“human genetic sequence determination”

Output top N
translations by p(c|e)

Output top N
translations byp(c|e)

found

What is human genetic sequence determination?
Template based phrase
extraction

Search for translation
In the phrase tablewords ≤3

Algorithm of phrase
identification andtranslation

18

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

zz …

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Parade of The Centuryz Edmund Ho Hau Wahz Balkan Syndrome

19

INSTITUTE OF COMPUTINGTECHNOLOGY

The particle phrase/word is removed
z The phrase occurring in many questions is removedz A word sequence with the first letter of its each word
capitalized is viewed as a whole phrasez Additional comments

Phrase translation

INSTITUTE OF COMPUTINGTECHNOLOGY

12/18/2008zz

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

Weights are tuned in a greedy wayscore(c, e) = ∑ ωi f i (c, e)iz Parenthesis
z Similarity of transliterationz co-occurrence probability

20

Non-exists in the phrase tablez Retrieve candidate translation with search engine(SE) (Ying Zhang, Phil Vines, 2004)z Rank candidates with a linear functionz OOV words

OOV words translation

12/18/2008

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

Rank candidates by score(c,e)

Extract candidate translation in awindow around key words

Duplication deletion & word
segmenation

Select top N snippets

Search documents with aChinese SE

English query construction

21

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

z Conclusion

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Results on the formal run

z Experiments on the dry-run set

z Document retrievalz Query translationz The system architecturez Background

Outline22

INSTITUTE OF COMPUTINGTECHNOLOGYINSTITUTE OF COMPUTINGTECHNOLOGY

12/18/2008zz

Lemur toolkit

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Various models are complementary23

Different models have different performance on
different taskz System combination in proper way improves overa single model (L.X.Shi, J.Y.Nie, 2007)z Basic ideas

Document Retrieval

12/18/2008z

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

24

INSTITUTE OF COMPUTINGTECHNOLOGY

Index models
z Various models with minor difference is easy to
combinez Index models have less parameters to be tunedz A linear function and the objective function

z Index models? Retrieval models?

Document Retrieval

0.27240.26090.2931

0.23690.30310.30450.3443fb_tf_idf

okapi

fb_okapi

kl_jm

kl_dirkl_absmixfb_kl_dir

12/18/2008

0.2497

Unigram

tf_idf

IR model0.28600.25190.26150.23660.30050.2646

0.31620.2664

Bigram

T run

0.29330.23020.25880.22180.19460.20650.25690.2164

Unigram

0.29830.22170.22700.21510.31300.23930.30880.2465

BigramIR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

0.30570.25780.2706

0.24990.32770.25820.29500.2515

Word

D run

0.25130.18230.20130.1851

0.23450.17690.26080.2052Word

M ean Average Precision

0.33430.31830.31540.29160.16780.28350.28440.2716

Unigram

0.33780.30130.30190.29700.33900.3172

0.33620.3032

Bigram

N run

25

0.29920.26240.27900.26850.27730.27370.28370.2729Word

INSTITUTE OF COMPUTINGTECHNOLOGY

The results of different IR models and index units for Chinese monolingual IR on
the NTCIR-5 test set

IR model selection

12/18/2008

0.36
0.345
0.33
0.315
0.3
0.285
0.27

Word

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

feedback document number

8 10 15 20 25 30

Bigram2 3 4 5 6 7

Unigram

The results of different index units and feedback document number

Parameter optimaztion

MAP

26

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

0.36
0.345
0.33
0.315
0.3
0.285
0.27

40

60
0
12

0
16

BigramIR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

feedback term count90

Unigram

20

0

Word

The results of different index units and feedback term countParameter optimaztion

MAP

27

INSTITUTE OF COMPUTINGTECHNOLOGYdivide by the score of top 1 docÆ[0,1]

12/18/2008z

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

Each index model finishes retrieval and feedback
respectivelyz the system combines their final scorez Index model combinationz

28

INSTITUTE OF COMPUTINGTECHNOLOGY

language model, Dirrichlet smoothing, with feedbackz Score normalizationz

z Retrieval model

Index Unit Combination

12/18/2008

z Conclusion

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Results on the formal run

z Experiments on the dry-run set

z Document retrievalz Query translationz The system architecturez Background

Outline29

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008zz

INSTITUTE OF COMPUTINGTECHNOLOGYIR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Mark documents containing answers as relevantz Convert according to specified XML formatz Download topics and answers in the dry run set

Process

unfit for IR4QAz Parameters tuned on previous test sets may well be

30

z Evaluation metric of IR4QA ≠ Evaluation metric of CLIR

Motivationz Construction

Development set

12/18/2008

CT-CT

EN-CS

Task
14
10
21
33
8
10
24
29

Event
Relationship
Biography
Definition
Event
Relationship
Biography
Definition

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

Question number

Question Typez Development set

Experiments on the dry-run set

31

INSTITUTE OF COMPUTINGTECHNOLOGYza1999_010451 za1999_007192 za1999_012952
XIN19990425.0127
udn_xxx_19980911_0015 udn_xxx_19980517_0264
udn_xxx_19980521_0269 udn_xxx_19990512_0009
udn_xxx_19981012_0046
udn_xxx_20010926_1110898 mhn_xxx_20010901_1073944
mhn_xxx_20010926_1110315 edn_xxx_20011203_1214182
udn_xxx_20010521_0913048 udn_xxx_19990322_0237
udn_xxx_19990902_0408 udn_xxx_20000309_0049710

What is the relationship between Shintaro

Ishihara and the literary circle?

印尼暴動與峇里島觀光有何關連？

(What is the relationship between riots inIndonesia and tourism in Bali?)禽流感是在說什麼呢?

(What is Avian influenza?)

誰是連戰？

(Who is Lien Chan?)

12/18/2008

udn_xxx_19991124_0194 udn_xxx_20000108_0255269(List events about Shenzhou spacecrafts.)

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

udn_xxx_19991122_0147 udn_xxx_20001106_0618907

udn_xxx_20010111_0719983 udn_xxx_20010118_0730638

udn_xxx_19991122_0061 udn_xxx_19991123_0131

udn_xxx_19991123_0131 udn_xxx_19991130_0249

神舟號太空船udn_xxx_19990507_0272

XIN19980728.0164 XIN19980617.0100 za2000_061113

What is the International Criminal Court?
za1999_007192

za2001_000149

XIN20011009.0072

Kursk explosion.

XIN20000815.0073

XIN20010312.0245

za2001_042120 za2001_003573 za2000_014785 za2001_033102

za2001_038799 za2001_041818 za2001_040008 za1999_028092

Relevant documents

List events related to Russian submarine

What is Mayday?

TopicsSample topics in development data set

32

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

U+W

W+B

0.5U+0.5W+B

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

05 .15 .25 .35 .45 .55 .65 .75 .85 .95
.
0
0
0
0
0
0
0
0
0
0

0.185

0.19

0.195

0.2

0.205

0.21

0.215

0.22

U+B

The Results of Index Models Combination in EN-CS Task

MAP

33

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

U+W

B+W

0.3U+0.7B+W

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

05 .15 .25 .35 .45 .55 .65 .75 .85
.
0
0
0
0
0
0
0
0
0

0.32
0.31
0.3
0.29
0.28
0.27
0.26
0.25
0.24
0.23
0.22
0.21

U+B

The Results of Index Models Combination in CT-CT Task

MAP

34

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

z Conclusion

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Results on the formal run

z Experiments on the dry-run set

z Document retrievalz Query translationz The system architecturez Background

Outline35

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

MITEL-CT-CT-01-T
MITEL-CT-CT-02-T
MITEL-CT-CT-03-D
MITEL-CT-CT-04-T

Run IDMITEL-EN-CS-01-T
MITEL-EN-CS-02-T
MITEL-EN-CS-03-T
MITEL-EN-CS-04-D
MITEL-EN-CS-05-TD

z Our runs

INSTITUTE OF COMPUTINGTECHNOLOGYIR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

36

Description
0.5U+0.5W+0.15B, question part is used
0.5U+0.5W, question part is used
0.45U+0.55B, question part is used
0.5U+0.5W+0.15B, narrative part is used
0.5U+0.5W+0.15B, both question and narrative partare used
0.3U+0.7B, question part is used
0.3U+0.7B+0.05W, question part is used
0.3U+0.7B, narrative part is used
Bigram, question part is used

Results on the formal run12/18/2008

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

AP

nDCG

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

QmetricMITEL-EN-CS-01-T
MITEL-EN-CS-02-T
MITEL-EN-CS-03-T
MITEL-EN-CS-04-D
MITEL-EN-CS-05-TD

The performance based on pseudo-qrels in the NTCIR-7 formal
result (EN-CS)

Performance in CLIR

score

37

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

AP

nDCG

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

metricQ

MITEL-EN-CS-05TD

MITEL-EN-CS-04-D

MITEL-EN-CS-03-T

MITEL-EN-CS-02-T

MITEL-EN-CS-01-T

The performance based on real qrels in the NTCIR-7 formal result(EN-CS)

Performance in CLIR

score

38

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

0.5

0.55

0.6

0.65

0.7

0.75

0.8

AP

nDCG

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

QmetricMITEL-CT-CT-01-T
MITEL-CT-CT-02-T
MITEL-CT-CT-03-D
MITEL-CT-CT-04-T

The performance based on pseudo-qrels in the NTCIR-7 formal
result (CT-CT)

Performance in CLIR

score

39

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

AP

nDCG

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

QmetricMITEL-CT-CT-01-T
MITEL-CT-CT-02-T
MITEL-CT-CT-03-D
MITEL-CT-CT-04-T

The performance based on real qrels in the NTCIR-7 formal result(CT-CT)

Performance in CLIR

score

40

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008

0.2750ATR/NiCT
0.1982 (0.2168)

CMUJAVIR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

EN-CS Collaboration Task (F3 scores based on automatic evaluation)

MITEL

IR4QA

CCLQA

0.1829KECIRz Performance in CCLQA with a QA system

Performance in CCLQA with a
QA system

41

INSTITUTE OF COMPUTINGTECHNOLOGY

z

z

z

12/18/2008z

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu“Who is Hu Jintao?”
Bibliography of Hu is more appropriate than a meeting heattends
Model combination doesn’t change this!

42

INSTITUTE OF COMPUTINGTECHNOLOGY

Overall recall with top 1000 over 90% (dry run)z Precision relatively low
z Model combination helps recall morez Evaluation metric of IR4QA differs from the one of
CLIR

Analysis

12/18/2008

z Conclusion

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

z Results on the formal run

z Experiments on the dry-run set

z Document retrievalz Query translationz The system architecturez Background

Outline43

INSTITUTE OF COMPUTINGTECHNOLOGY12/18/2008z

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

44

INSTITUTE OF COMPUTINGTECHNOLOGY

A phrase table of SMT + OOV words translation
based on SE
z Index model combinationz The construction of Development set based on
the dry run set

z ICT-Crossn achieves a good performancez IR4QA is a novel but challenging task

Conclusion

identification and translation of phrasesz “natural gas hydrates”
rerankingz A reranking method for CLIR of I2R doesn’t work in
IR4QA(L.P.Yang, D.H.Ji, 2005)zz

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

45

INSTITUTE OF COMPUTINGTECHNOLOGY

Phrases translation
High recall & low precision mean…
z Reranking should workz A better objection function for IR4QA should be explored

Future workzzProblems

12/18/2008zzConclusion

12/18/2008

IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu

46

INSTITUTE OF COMPUTINGTECHNOLOGY

Many thanks to Dr. Zhongjun He for his SMT system -- Mencius

Many thanks to my co-authors Tian Xia, Ji Guo and Qun Liu

Thank you!

Information retrieval for question answering:a challenging task but a long way to go