NTCIR-13 OpenLiveQ
Task

Open Live Test for Question Retrieval
Makoto P. Kato, Takehiro Yamamoto (Kyoto University),
Sumio Fujita, Akiomi Nishida (Yahoo Japan Corporation)

GoalImprove
the REAL

Performance evaluatedby REAL users

performance ofquestion retrieval systems
in a production

environment
Yahoo! Chiebukuro(a CQA service of Yahoo!
Japan)Task• Given a query, return a ranked list ofquestions that can satisfy many
REAL users in Yahoo! Chiebukuro (aCQA service)
Effective for Fever

INPUT

Three things you should not do in fever

While you can easily handle most fevers at home, you should call 911
immediately if you also have severe dehydration with blue .... Do not blow yournose too hard, as the pressure can give you an earache on top of the cold. ....

10 Answers

Posted on Jun 10, 2016

Effective methods for fever

Apply the mixture under the sole of each foot, wrap each foot with plastic, andkeep on for the night. Olive oil and garlic are both wonderful home remedies
for fever. 10) For a high fever, soak 25 raisins in half a cup of water.

2 Answers

Posted on Jan 3, 2010

OUTPUT

OpenLiveQ provides an OPEN LIVE TEST EVIRONMENTReal users

InsertClick!
Click!
Click!

InsertInsertTeam ATeam B

Team C

Ranked lists of questions from participants’systems are INTERLEAVED, presented to realusers, and evaluated by their clicks

Multileaving-based Evaluation• Multileaved comparison methods
are used in the online evaluation– Schuth, A., Sietsma, F., Whiteson, S., Lefortier, D., de Rijke, M.:
Multileaved comparisons for fast online evaluation. In: CIKM. pp.
71–80(2014)
Systemresults
Multileavings
submitted by participants

Rank 1

AAA

Rank 1AAARank 2

DBCRank 2

BDCRank 3

BCDRank 3

CB

DSystem 1

System 2

System 3

Multileaving 1 Multileaving 2 Multileaving 3

Shown to the user

• Therefore, submitted runs areevaluated by REAL users’ clicks

Data (tentative)

valuable data

Plan to provide,
which have not been available for universities•1,000 training queries• Top 1,000 search results for each query•Clickthrough data
– How many users have clicked on eachresult– What types of users have clicked(age, sex, job, etc.)

Challenges
How to satisfy diverse relevance criteria ofreal users?

While topical relevance has been considered in many IR researches,the other types of relevance criteria (e.g. popularity and freshness)have notPopularity!
Freshness!

What are major spots
in Kyoto?

Topic Relevance!!

Humor!! Objectivity!
Conciseness!

How to answer ambiguous/underspecified
queries?

In community Q&A, there may be more diverse intents behind thesame query“cheap office”

Q. How to rent a cheap office?
OR
Q. How to buy cheap MS office?

Possible Research Topics
Effective for Fever• Search intent detection– e.g. Estimate important relevance criteria(e.g. popularity) and rank questions bythem (e.g. number of user views)• User demographic estimation– e.g. Estimate major demographics of thequery (e.g. F1) and estimate therelevance mainly for such user groups• Eye catching title detection– e.g. Witty titles can be good for “searchfor fun”

Schedule
Date

Event

Details

Oct 2016

Offlinetraining data
releaseData will be released fortraining retrieval systems

Dec 2016

Registration
due

Registration form will beavailable at the NTCIR-13 siteJan-Mar2017

Offline test
runsubmission

Systems will be evaluated
without real users

Apr-Jun 2017 Online test

Systems will be evaluatedwith real users

Jul 2017

Online test
Evaluation results will be
result release released to participants

Looking forward to your participation. Thanks!