Overview of the ACLIAIR4QA (Information Retrieval forQuestion Answering) Task
Tetsuya Sakai Noriko Kando
ChChuan-Jie
Ji LiLinT k Mitamura
Teruko
MitDonghong JiKuang-Hua Chen
Ei NEricNybergb18th December 2008 @NTCIR-7, TokyoTALK OUTLINE
1. Task Objectives
2 Relevance Assessments
2.
3. Evaluation Metrics
4. Participating Teams
5 Official Results
5.
6. Lazy Evaluation
7. Unanswered Questions

What are the effective IR techniquesfor QA?

Traditional “ad hoc” IR vs IR4QA• Ad hoc IR (evaluated using Average Precision etc.)- Find as many (partially or marginally) relevant
documents as possible and put them near the top of
the ranked list
• IR4QA (evaluating using… WHAT? )- Find relevant documents containing different correct
answers?
- Find multiple documents supporting the same correct
answer to enhance reliability of that answer?
- Combine partially relevant documents A and B todeduce a correct answer?

TALK OUTLINE
1. Task Objectives
2 Relevance Assessments
2.
3. Evaluation Metrics
4. Participating Teams
5 Official Results
5.
6. Lazy Evaluation
7. Unanswered Questions

Pooling for relevance assessments
System 1

TargetDocuments

Run
depth=1000

Run 1Pool
depth>= 30

Relevance
assessments:::

CS: Simplified
Chinese
CT: Traditional
Chinese
JA: Japanese

Topic A

L2-relevant
Pool

L1-relevant
L0

System NRund th
depth
=1000

Run N

Pool
depth>= 30

L2: relevant
L2lt
L1: partially relevant
L0: judged
nonrelevantDifferent pool depths for different topics
Mandatory for all topics
Assess depth-30
depth 30 pool
Assess depth-50
depth 50 pool (minus depth-30
depth 30 pool)

Assess depth
depth-7070 pool (minus depthdepth-5050 pool)

See IR4QAOverview
A
Assessdepth-90d th 90 pooll ((minusiddepth-70th 70 pool)l)Tables 29-31
for details
Assess depth-100 pool (minus depth-90 pool)
Relevance assessments coordinated independently by
Donghong Ji (CS), Chuan-Jie Lin (CT) and Noriko Kando (JA)

Sorting the pooled documents forassessors••

Traditional approach: Docs sorted by IDs
IR4QA approach: Sort docs in depth-X
depth X poolby:- #runs#containingt i i ththe ddoc att or abovebrankkX(primary sort key)
- Sum of ranks of the doc within these runs
(secondary sort key)
Present ``popular’’ documents first!

Assumptions behind the sort
1. Popularpdocs are more likely
y to be relevant thanothers.
Supported by [Sakai and Kando EVIA 08]
2. If relevant docs are concentrated near the top of
the list to be assessed
assessed, this is easier for theassessors to judge more efficiently and consistently.
At NTCIR-2,NTCIR 2 th
the assessors actuallyt ll did nott lik
like ddoc liliststsorted by doc IDs(But we need more empirical evidence)TALK OUTLINE
1. Task Objectives
2 Relevance Assessments
2.
3. Evaluation Metrics
4. Participating Teams
5 Official Results
5.
6. Lazy Evaluation
7. Unanswered Questions

Average Precision (AP)Precision
Pi i
at rank r

Number ofrelevant
docs

1 iff doc
d att ris relevant• Used widely since the advent of TREC
• Mean over topics is referred to as “MAP”MAP• Cannot handle graded relevance(but many IR researchers just love it)Q measure (Q)Q-measure• Generalises AP andhandles graded relevance• Properties
psimilar to AP
and higher discriminativepowerp• Not widely-used, but
has been used for QA
and INEX as well as IR

PersistenceParameter β
set to 1

Blended ratio at rank r(Combines Precisionand normalisedCumulative Gain)
SakaiSk i anddR
Robertsonb t
EVIA 08
provides a user modelfor AP and Q

nDCG (Microsoft version)
Sum of discounted gains
f a systemfortoutputt t

Sum of discounted ggains• Fixes a bug of the original
for an ideal output
nDCG• But lacks a parameter that reflects
the user’s persistence• Most popular gradedgraded-relevance
relevance metricIR4QA evaluation package(Works for ad hoc IR in general)

ComputesAP, Q, nDCG,RBP, NCU[Sakai and Robertson
EVIA 08] and so on
http://research.nii.ac.jp/ntcir/tools/ir4qa_eval-en

TALK OUTLINE
1. Task Objectives
2 Relevance Assessments
2.
3. Evaluation Metrics
4. Participating Teams
5 Official Results
5.
6. Lazy Evaluation
7. Unanswered Questions• 12 participants from China/Taiwan
China/Taiwan, USAUSA, Japan• 40 CS runs (22 CS-CS, 18 EN-CS)• 26 CT runs (19 CT-CT
CT CT, 7 EN-CT)
EN CT)• 25 JA runs (14 JA-JA, 11 EN-JA)
MonolingualCrosslingualOral presentations• RALI (CS-CS,(CS CS, EN
EN-CS,CS, CT
CT-CT,CT, EN
EN-CT)CT)- Uses Wikipedia to extracts cue words for
BIOGRAPHY; Extracts person names using
Wikipedia and Google;
G
Uses Google
Gtranslation• CYUT (EN-CS, EN-CT, EN-JA)- UsesUWikiWikipedia
di ffor query expansion
i andd translation;l iUses Google translation• MITEL (EN-CS,(EN CS CT-CT)
CT CT)- Uses SMT and Baidu for translation; data fusion• CMUJAV (CS(CS-CS,CS EN
EN-CS,CS JA-JA,
JA JA ENEN-JA)
JA)- Proposes Pseudo Relevance Feedback using LexicoSemantic Patterns (LSP-PRF)(LSP PRF)

Other interesting approaches• BRKLY (JA-JA) A very experienced TREC/NTCIR participant• HIT (EN-CS) PRF most successful• KECIR (CS-CS) Query expansion length optimised foreachh questionti typet(d fi iti(definition,bibiography…)h )• NLPAI (CS-CS) Uses question analyses files from otherteams (next slide)• NTUBROWS (CT-CT) Query term filtering, data fusion• OT (CS(CS-CSCS, CT
CT-CTCT, JAJA-JA)
JA) Data fusion-likefusion like PRF• TA (EN-JA) SMT document translation from NTCIR-6• WHUCC ((CS-CS)) Document reranking
gPlease visit the posters of all 12 IR4QA teams!

NLPAI (CS-CS) used question
analysis files from other teams.
CSWHU-CS-CS-01-T:<KEYTERMS><KEYTERM SCORE="1.0">宇宙大爆炸</KEYTERM><KEYTERM SCORE="0.3">理论</KEYTERM></KEYTERMS>
Apath-CS-CS-01-T:<KEYTERMS><KEYTERM SCORE="1.0">宇宙大爆炸理论</KEYTERM></KEYTERMS>/KEYTERMSCMUJAV-CS-CS-01-T:<KEYTERMS><KEYTERM SCORE="1.0">宇宙</KEYTERM><KEYTERMKEYTERM SCORE="1.0">大</KEYTERM>
SCORE
大 /KEYTERM<KEYTERM SCORE="1.0">爆炸</KEYTERM><KEYTERM SCORE="1.0">理论</KEYTERM><KEYTERM SCORE="1.0">宇宙 大 爆炸 理论</KEYTERM><KEYTERMKEYTERM SCORE="1.0">宇宙大爆炸理论</KEYTERM>SCORE "1 0" 宇宙大爆炸理论 /KEYTERM<KEYTERM SCORE="1.0">宇宙 大 爆炸</KEYTERM><KEYTERM SCORE="1.0">宇宙大爆炸</KEYTERM></KEYTERMS>

Different teams
come up withdifferent set ofquery terms withih
different weights.
This clearly affects
retrieval
performance.
pSpecial thanks toMaofu Liu (NLPAI)TALK OUTLINE
1. Task Objectives
2 Relevance Assessments
2.
3. Evaluation Metrics
4. Participating Teams
5 Official Results
5.
6. Lazy Evaluation
7. Unanswered Questions

CS T-runs:
T runs: Top 3 teams
MeanAP

MeanQ

MeannDCGO
OTCS-CS-04-T

.6337
633

O
OTCS-CS-04-T

.6490
6 90

O
OTCS-CS-04-T8270*.8270

MITELMITEL
EN-CS-03-T

.5959
5959

MITELMITEL
EN-CS-03-T

.6124
6124

CMUJAVCMUJAVCS-CS-02-T

.7951
7951

CMUJAVCMUJAVCS-CS-02-T

.5930
5930

CMUJAVCMUJAVCS-CS-02-T

.6055
6055

MITELMITEL
EN-CS-01-T

.7949
7949

- MITEL isi very goodd even thoughthh it is
i a crosslinguallil run- OT significantly outperforms CMUJAV with Mean nDCG
(two-sided bootstrap test; α=0
α=0.05)05)- nDCG disagrees with AP and Q

CT T
T-runs:runs: Top 3 teams
MeanAPMeanQ

Mean
nDCG

MITELCT-CT-02-T

.5839
5839

MITELCT-CT-02-T

.6018
60 8

MITELCT-CT-02-T

.7873
8 3

OTOT
CT-CT-04-T

5521**.5521

OTOT
CT-CT-04-T

5724**.5724

OTOT
CT-CT-04-T

7656 **.7656

RALIRALI
CT-CT-05-T

.3952
3952

RALIRALI
CT-CT-05-T

.4096
4096

RALIRALI
CT-CT-05-T.6559 **- MITEL andd OT nott significantly
i ifi
tl different
difft fromfeachh otherth- OT significantly outperforms RALI
(two-sided bootstrap test; α=0α=0.01)
01)
but RALI’s performance is actually very high after bug fix

JA T
T-runs:runs: Top 3 teams
MeanAP

MeanQ

MeannDCG

OTJA-JA-04-T.6979 **OTJA-JA-04-T.7090 **OTJA-JA-04-T.8650 **CMUJAVJA-JA-01-T.5932

CMUJAVJA-JA-01-T.5996

CMUJAVJA-JA-01-T.7832

BRKLYJA-JA-02-T.5838 **BRKLYJA-JA-02-T.5996 **BRKLYJA-JA-02-T.7831 **- OT significantly outperforms CMUJAV- BRKLY significantly outperforms the 4th team (CYUT
crosslingual run)
(two-sided bootstrap test; α=0.01)

System ranking
by Q/nDCG vsthat by APCSCTJA

By definition,nDCG is more
forgiving forlow-recallo eca runs
u sthan AP and Q.

The most “novel’’ runs
Uniquerelevant

Relevant docs
retrieved byy
Run A

Relevant docs
retrieved by all other teams

RALI-EN-CS-04-T found
63 unique relevant docs
(53 for topic CS-T42)
RALI-EN-CT-05-T found
32 unique relevant docs
(16 for topic CT-T442)
OT-JA-JA-01-T found
51 unique relevant docs
(12 for JA-T236)
JA T236)

These runs are valuable for making
the relevance assessments asexhaustive as possibleSuccessful PRF
Mean AP

Mean Q

Mean nDCG

HIT-EN-CS-01-DN.5690**.5840 **

.7560 **HIT-EN-CS-02-DN

.4634

.4827

.6910

OT-CT-CT-04-T.5521 **

.5724 **

.7656 **OT-CT-CT-02-T

.5111

.5339

.7432

BRKLY-JA-JA-02-T.5838 *.5996 **.7831 **BRKLY-JA-JA-03-T

.5407

.5509

.7475

OT-JA-JA-04-T.6979 *.7090 *.8650 **OT-JA-JA-02-T
OTJA JA 02 T

.6698
6698

.68086808

.8473
8473

Other teams appear to be less successful with PRF.
This may be partly because the qrels are very incomplete.

Per-topic AP/Q/nDCG
averagedd over runs (CS)“Topicdifficultydifficulty”
variesPer-topic AP/Q/nDCG
averagedd over runs (CT)“Topicdifficultydifficulty”
variesPer-topic AP/Q/nDCG
averagedd over runs (JA)“Topicdifficultydifficulty”
variesTALK OUTLINE
1. Task Objectives
2 Relevance Assessments
2.
3. Evaluation Metrics
4. Participating Teams
5 Official Results
5.
6. Lazy Evaluation
7. Unanswered Questions

Forming pseudo-qrels
pseudo qrels
QUESTION: Can we get away with not doing
any relevance assessments at all?
1 Sort pooled docs by1.
(1) Number of runs that retrieved it; and then(2) Sum of its ranks within these runs.
2 Take the top 10 docs in the sorted pool and2.
treat them all as L1-relevant!
Sakai and Kando EVIA 08 actually showsthat the top 10 docs are more likely to be relevant
than others on average

Systemyranking
g byy real MAP vs that byypseudo MAP (CS)

“Pseudo MAP” assumes that “popular” documents are relevantSystemyranking
g byy real MAP vs that byypseudo MAP (CT)

Systemyranking
g byy real MAP vs that byypseudo MAP (JA)
Pseudo-qrels are not very useful for
predicting the ranking of the highest
performers

Kendall’s rank correlation
PPseudod vs real:l aroundd 0.707(cf. Soboroff SIGIR 01: around 0.4)

But they may
be useful for
predictingthe lowperformers(for CT andJA)TALK OUTLINE
1. Task Objectives
2 Relevance Assessments
2.
3. Evaluation Metrics
4. Participating Teams
5 Official Results
5.
6. Lazy Evaluation
7. Unanswered Questions

Unanswered Questions• What IR strategies are good for QA?
(e.g. How does question classification help?)• What are the general/language-specificchallenges for mono/crosslingual IR4QA?
• How incomplete are the IR4QA testcollections? How reusable are they?
• What are the best evaluation methods?
• HowH
ddo IR4QA andd ththe entire
ti ACLIA results
ltcorrelate?