Welcome!

Twitter: #ntcir10
Ust: ntcir10‐kickoff

NTCIR-10 Kick-Off Event
2011.03.08
⽇本語セッション: 13:30English
li h Session:Si
15:3030

Program• About
Ab t NTCIR• About NTCIR-10• Accepted Tasks• Why participate?
• How to participate• Important Dates• Q&AAbout NTCIR
IntroductionNTCIR: NII Testbeds and Community for Information access Researchg IA
Research Infrastructure for Evaluating

A series of evaluation workshops designed to enhanceresearch in information‐access technologies
providingg byy pg an
infrastructure for large‐scale evaluations.
■Data sets, evaluation methodologies, and forum

Project started in late 1997
Once every 18 months

Data sets (Test collections or TCs)
Scientific, news, patents, web, CQA,
Wikipedia, Entrance Exams
Chinese KoreanChinese,Korean, JapaneseJapanese, and English

NTCIR‐917NTCIR‐817NTCIR‐71582

NTCIR‐61285NTCIR‐515NTCIR‐3

9NTCIR‐2

8NTCIR‐11
NTCIR

6
0

IR: Cross‐lingual tasks,
tasks patents
patents, web
web, Geo
QA：Monolingual tasks, cross‐lingual tasks
Summarization, trend info., patent maps
Opinion analysis, text mining

Community‐based Research Activities

65

119
109
# of countries

77

10

NTCIR‐4

Tasks (Research Areas)90# of Active
participant groups# of registered

74
65
36
28
50

100

150

Information retrieval (IR)• Retrieve RELEVANT information from vast collection to meetusers’’ informationi f
ti needsd• Using computers since the 1950s• First CS uses human assessments as success criteria– Judgments vary
– Comparative evaluations on the same infrastructure

Informationo at o access ((IA))
Whole process to make information usable by users.
ex.: IR, text summarization, QA,
Q text mining, anddclustering

Tasks at Past NTCIRs
Community
C
it QA
Opinion Analysis■ ■Contents■ Inference + QA
Module-Based
Cross-Li ngual QA + IR
■ ■
E t
Exam
E
■ ■ Entrance
IR for Focused■ ■ Spoken Doc IR
Geo Temporal
Domain■ ■Patent■ ■ ■ ■ □Complex/ Any Types■ ■ ■ □
Question
Dialog■ ■Cross-Li ngual
Answeri ng■ ■ ■ ■
Factoid, List
■ ■ ■ ■
■ ■ Cross Link Discovery
Extracti on■ ■ Inference + QA
Semantic
Text Mi ning / Classification■ ■ ■ ■ ■Summarization /Trend Info Visualization■ ■ ■Consolidation
Text Summarizati on■ ■ ■Visualization + IIRInteractive■■ ■ Intent Mini ng
Web■ ■ Quality of SERP
Web
■ ■ ■
St ti sticalti l MT
Crosslingual■ ■ ■ ■ Stati
Cross-Li ngual IR■ ■ ■ ■ ■ ■ ■ ■Non-English Search■ ■ ■ ■ ■ ■ ■ ■Ad Hoc IR, IR for QA
Text Retrieval■ ■ ■ ■ ■ ■ ■ ■The Years the meetings were held.
held The tasks started 18 months before

User Generated■■

Procedures in NTCIR Workshopsp• Call for Task Proposals• Selection of Task Proposals by Committee• Discussion about Experimental Designs and Evaluation Methods(can beb continued
d to Formall Runs)• Registration to Task(s)– Deliver Training Data (Documents,(Documents Topics,
Topics Answers)• Experiments and Tuning by Each Participants– Deliver Test Data (Documents and Topics)• Experiments by Each Participants• Submission of Experimental Results• Pooling the Answer Candidates from the Submissions,
Submissions andConduct Manual Judgments• Return Answers (Relevance Judgments) and Evaluation Results• Workshop Meeting
Discussion for the Next RoundNTCIR Workshop Meeting
NTCIR:

http://research nii ac jp/ntcir/
http://research.nii.ac.jp/ntcir/
8

NTCIR‐10 Organization<NTCIR‐10NTCIRPProgram
Co‐Chairs>
C Ch i‐ Hideo Joho（University of Tsukuba, Japan）‐ Tetsuya Sakai（Microsoft Research Asia, China)<EVIA2013 Co‐Chairs>‐Ruihua Songg ((Microsoft Research Asia,, China))‐William Webber (University of Maryland, USA)<NTCIR 10 General Co<NTCIR‐10
Co‐Chairs>
Chairs>‐ Noriko Kando (NII, Japan)‐ Tsuneaki Kato (The University of Tokyo, Japan)‐ Douglas W.
W Oard (University of Maryland,
Maryland USA)
‐ Mark Sanderson (RMIT University, Australia)‐ex‐general chair: Eiichiro Sumita (NICT)‐48 Task Organizers from all over the world

NTCIR Test Collections

est Coect o s = Docs
ocs + Topics/Questions
op cs/Quest o s + Answers
s es
Test
Collections

Available to Non‐participants for Research Purpose

Focus of NTCIR
Lab‐type
yp IR Test

New ChallengesgAsian Languages/cross‐language
Variety of Genre
Parallel/comparable Corpus

Intersection of IR + NLP
To make information in the documents
more usable for users!
Realistic eval/user task
Interactive/Exploratory search
QA typest
att topict i crea

Forum for Researchers andOther Experts/users
Idea Exchange
Discussion/Investigation on Evaluation
methods/metrics

Difficulty of retrieval varies with topics
1.0000J-J Le ve l1 D auto
EffectivenessacrossSYSTEMS

Effectiveness across TOPICS on
system 検索システム別の11pt再現率精度
10.8000A

Average over 50
topicsBC

0.8

0.6000D

precision

E

precision

F

0.4000

G

0.6

H
I
J

0.4

K
L

0.2000

M
N

0.2

O

0.0000
0

2

4

6

8

0

0.

0.

0.

0.

0.

1.

P

recall

0
0

0.1

0.2

0.3

0.4

0.5
recall

0.6

0.7

0.8

0.9

1

101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

Difficulty of retrieval varies with topics
J JL
J-J
Le ve l1 D autot

1.0000Effectiveness across
SYSTEMS

検索システム別の11pt再現率精度
Effectivenessacross TOPICS on
system1“Difficult
topics”
vary with systems
Averageover 500.8000A
B

topicsJ-JJJ Le ve l1 D auto
0.8

Mean av. precision preecision

0.60001.0000

A

D

B

E

C

precisioonF

0 80000.80000.4000

D

G

06
0.6

E

H

0.60000.4

0.40000.2000

I

F

J

G

K

H

L

I

M

J

N

0.2

0.2000K

O

0.0000LP

7110 .2
0
11
3
0
11 .4
6
11
9
120 .
26
12
5
120
8.8
13
1
131
4.0
13
7
14
0
14
3
14
6
14
9

0

0

10

1

0
1 .

04 0

0.000010

preecisionC

recallT opic#Requests#101 150#101‐150

0.1

0.2

0.3

For reliable
and stableMevaluation,using substantial #
N
0.4
0.5
0.6
0.7
0.8
0.9
1
of recall
topicsisnecessary.
O
P

101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

IR Systems Evaluation• Engineering Level: Efficiency• Input Level: ex. Exhaustivity, quality, novelty of DB• Process Level: Effectiveness ex. recall, precision• Output Level: Display of output• UserU Level:L l ex. Eff
Effortt th
thatt users needd• Social Level: ex. Importance(Cleverdon & Keen 1966)

gWhat are TCs usable for evaluating?

Pharmeceutical R & D
Phase I:
In vitro
experiments

Phase II:

Phase III:

Phase IV:

Animal experiments

Tests with healthy
human subjects

Clinical tests

gWhat are TCs usable for evaluating?
NTCIR

Users’ information‐seeking
tasks

Test collections
Phase I:

Phase II:

Phase III:

Phase IV:

Laboratory‐type
testing

Sharing modules,
P t tPrototype
testing

Controlledinteractive testing
using humansubjects

Uncontrolled pre‐operational testing

Phase II:

Phase III:

Phase IV:

Animal experiments

Tests with healthy
human subjects

Clinical tests

Pharmeceutical R & D
Phase I:
In vitro
experiments

2.Input level、Levels ofevaluation
1. Engineering level:
3.Process level: Effectiveness6.Social level

4.User level、5.Output level

Efficiency

• Information Seeking
g Task– document types + user community– useruser’ss situation,
situation purpose of searchsearch, realistic
Experiments are Abstraction of the Real World Tasks.
Tasks
Trade‐off between “reality” and “contorable”• Testing & Bench marking
To learn how and why the system works better (worse)
than others
To learn how it can be improved
Scientific Understanding
g of the effectivenesspp
Improvementof Effectiveness byy Evaluation Workshops
1.5 – 2 times in 3
years

Mean Avverage P
M
Precisionn0.5

Cornell University TREC Systems

0.4
03
0.3

TREC-1
TREC-2TREC
2TREC-3

0.2

TREC-4

0.1

TREC-6

0
'92
'93
'94
'95
'96
'97
'98
System System System System System System System

TREC-5
TREC 7
TREC-7

Research Trends
Number of Papers Presented at ACM‐SIGIR
450

400

WEB
350

User
Evaluation# of pap
pers

300

Non‐Text
QA & Summarization

250

NLP
Cross‐Lingual

200

ML
Clusteringg

150

Efficiency
Filtering

100

Query Processign
IR Models

50

General0''77‐79'8 8'80‐84'8 8'85‐89''90‐94

PpublicationYears''95‐99''00‐04''05‐09

Some Thoughts
g on Future• Requirements for Evaluating Individual Applications etc.
• Interactive and Exploratory Information Access– Users’ Intention, Diversity– Collaborative Search– Expertise, Knowledge, Literacy, etc.

• Ex. Key Topics discussed at SWIRL 2012
– ConversasionalC
il Retrieval
R ti l– MobileIR– Zero Queryy ((or Less))– SmarteIR– Query by walking around– Structure,
Structure Entity,
Entity Relation

About NTCIR‐10
What’ss new?
What

NTCIR 10: Objectives
NTCIR‐10:• Solid foundation– Even stronger
structure• Task diversity– Covers a wide context
in Information Access– Studies rich media
types• Community‐led taskorganisation– Sustainability of
research• Promotion of researchresources– Show case in the
NTCIR‐10 Meeting

NTCIR‐10:
NTCIR 10: Structure• GeneralGl Co‐Chairs
C Ch iTO• TaskT k OrganisersOi– Noriko Kando (NII)– 48 researchers worldwide

ki K
k
– T
Tsuneaki
Kato (T(TokyoUniversity)– Participants (You!)– Douglas W.
W Oard(University of Maryland)• EVIA 2013 Co‐Chairs– Mark Sanderson (RMIT)– Ruihua Songg (MSRA)• Program Co‐Chairs– Hideo Joho ((Tsukuba
University)– Tetsuya Sakai (MSRA)– William Webber(University of Maryland)http://research.nii.ac.jp/ntcir/ntcir‐10/organizers.html

NTCIRNTCIR‐10:
10: Development so far
Oct 2011
Nov 2011

New structure formedfffor NTCIR‐10
Call for task proposal announced andl were submitted
b i d
10 proposals

Dec 2011
Jan 2012

NTCIR‐9 Workshop Meeting
7 proposals were accepted by the programcommittee and co‐chairs
Calls
ll forf taskk participation preparedd
NTCIR‐9 Kick‐Off EventFebb 2012
Mar 2012

NTCIR 10 Program Committee
NTCIR‐10• Charles
Ch l Cl
Clarkek (University(U i
i off Waterloo,
W
l Canada)Cd )• Kalervo Järvelin (University of Tampere, Finland)• Hideo Joho (Co‐chair, University of Tsukuba, Japan)• Gareth Jones (Dublin City University, Ireland)• Noriko Kando (NII, Japan)• Tsuneaki Kato (The University of Tokyo, Japan)• Douglas W. Oard (University of Maryland)• Tetsuya Sakai (Co‐chair, Microsoft Research Asia, PRC)
• Mark Sanderson (RMIT, Austraria)• Ian Soboroff ((NIST,, US))NTCIR‐10 Evaluation Tasks
Calls for task participation

Tasks accepted for NTCIR‐10
NTCIR 10CORE TASKS• [Intent‐2] Search intent and diversification• [1Click‐2][ Cli k ] One‐ClickO Cli k A
Access• [RITE‐2] Recognizing Inference in Text
• [SpokenDoc‐2] IR for Spoken Documents
g access to Patent Docs
• [[PatentMT‐2]] Cross‐lingual• [CrossLink‐2] Cross‐lingual Link Discovery
PILOT TASKS• [Math] Access to mathematical contents

CrossLink‐2

Crosslink Introduction• Cross‐lingual link discovery (Crosslink or CLLD) isconcerned with automatically finding potential
links between documents in different languages.
• CLLD algorithms actively recommend a set of
meaningful anchors in the context of a source
document and establish links to documents in analternative language.
• CLLDC
i helpfulish l f l for
f complimentarylikknowledge
l ddiscovery in different language domains.

Cross-lingualg Link Discoveryy
Article: Australia…
Ranked third in the Index of Economic Freedom(2010),[178] Australia is the world's thirteenthlargest economy and has the ninth highest percapita GDP; higher than that of the United
Kingdom, Germany, France, Canada, Japan, andthe United States.
States The country was ranked
second in the United Nations 2010 Human
Development Index and first in Legatum's 2008
Prosperity Index.[179] All of Australia's major
cities fare well in global comparative livability
surveys;[180] Melbourne reached first place on
Th Economist's
TheE
i ' 2011 World's
W ld' Most
M
Li bl
LivableCities list, followed by Sydney, Perth, andAdelaide in sixth, eighth, and ninth placerespectively.[181] Total government debt inAustralia is about $190 billion.[182] Australia
has amongg the highestghouse pprices and someof the highest household debt levels in theworld.
…

No link was created for this term, forfinding articles in languages we
prefer traditionally we do:

SearchTranslate

이코노미
스트
…

经济学人
…
The
Economist…
エコノミスト…

g
How to automaticallyy create cross‐lingual
links for a document if no links existingyet?

 Links in other languages?
 New articles?
 Missing
Mi i links?
li k ?
 Not what we are looking for?
 What about other relevant
links?

Cross‐lingualCrosslingual Link Discovery

 Cross-lingualg Links New Links
 Better Links
 More options

•Allll aboutbmulti‐linguall i lil kknowledgel d discovery
diin knowledgekl d basesb(e.g.
(
Wikipedia)
k d )•All about easy and efficient information accessThe Goal of Crosslink Task
• It is aimed to create a reusable resource forg g link
evaluatingg automated cross language
discovery approaches. The results of thisresearch will be used in building and refining
systems for automated link discovery.

Crosslink task at NTCIR‐9NTCIR 9• Cross‐lingual
Cross lingual link discovery (Crosslink) as a pilot task of
NTCIR‐9 has been successfully held in 2011
• Below is a list of participating teams with submissions:
GROUP
DUIIS
HITS
IISR
ISTIC
KMI
kslab_nut
KSLP
nthuisa
QUT
UKP
WUST

ORGANISATION
Daegu University
Heidelberg Institute for Theoretical Studies
Yuan Ze University
Institute of Scientific and Technical Informationof China
The Open UniversityggyNagaoka
Universityy of Technology
Kyungsung University
Academia Sinica
Queensland University of Technology
TU Darmstadt
Wuhan University of Science and Technology

Crosslink task at NTCIR‐9NTCIR 9 (Cont.)(Cont )• Therehwere ini totall 57 runs fromf
11 teams
were received.
Group
DUIIS
HITS
IISR
ISTIC
KMI
kslab_nut
KSLP
nthuisa
QUT
UKP
WUST
Sub‐total
b
l
Total

En‐2‐Zh
0
3
0
1
4
0
0
3
5
5
4
25

En‐2‐Ja
0
3
0
0
0
1
0
0
2
5
0
11

En‐2‐Ko
2
3
5
0
0
0
5
0
1
5
0
21
57

Crosslink Task at NTCIR
NTCIR‐10
10• CLLD is
i a core taskk off NTCIR‐10• New Subtasks– Chinese to English CLLD (C2E)– Japanese to English CLLD (J2E)– Korean to English CLLD (K2E)• New Document Collections– ECJK Wikipedia collections• New Topics– 25 topics
p for each languageg gEvaluation Framework• Assessment Types: Automatic (Wikipedia Ground Truth), Manual(h(humaniin theh loop)l)• Test data, training data, gold standard, ECJK Wikipedia collections,validation tool, assessment tool and evaluation tool (with systeml)evaluation
metrics).
• Snapshots of the validation, assessment and evaluation tools:
A
Assessmentt TTooll

Validation Tool

Evaluatuion Tool

System Evaluation Metrics



∑ 1

∑

∑ 1

1

@

where n is the number of topics; m is thenumber of identified items (links or
anchors); Pkt is the precision at top k items(links or anchors) for topic t; Pt@R ( =number of correct items ( links or anchors) /number of items (links or anchors) in qrel) is
the precision calculated using number oflinks / anchors in qrel as denominator for
topic t.
t

 Precision-at-N is computed using the average
precisione i i forf all
ll topicst i (source(
e articles)ti le ) att a
pre-defined position N in the results list.
V lValuesoff N were chosenhas: 5,
5 10,
10 20,
20 30,
30 50,
50
and 250.

ExpectedpOutcomes• More good submissions• More original and innovative
approaches can be seen in
identifying
meaningful anchorsand suggesting high quality cross‐lingual links• The research results can reallyhelp the cross‐lingual knowledge
discovery in knowledge bases
Evaluation Framework• The evaluation framework will befurther refined• The evaluation methods will befurther perfected to distinguish
the good and the bad CLLD
algorithms for the new subtasks

Expected Participants• Previousiactivei participantsi i
off NTCIR‐9
Crosslink task
• The registered participants of NTCIR‐9
Crosslink task.
• Participants of CLIR or IR4QA task in previous
NTCIR workshops.
workshops• Participants of previous INEX Link‐the‐Wiki
t k
track
• Other researchers in the CLIR fieldContacts• Organisers:
Shlomo Geva

Queensland University of Technology, Australia

Andrew Trotman Universityof Otago, New Zealand
Yue XuQueensland University of Technology, Australia

Eric Tang

Queenslandl d University off Technology,h lAustralial

In‐Su Kang

Kyungsung University, South Korea

Fuminori Kimura

Ritsumeikan University,
University Japan

Haitao Mi

Chinese Academy of Sciences, China• Mailing list: crosslink@lists.otago.ac.nz

INTENT‐2

INTENT‐2@NTCIR‐10 Kickoff(English version)
TASK ORGANISERS: Tetsuya Sakai, Ruihua Song,
Zhicheng Dou (Microsoft(
Research Asia)) Min
Zhang, Yiqun Liu (Tsinghua University) Takehiro
Yamamoto,, Makoto Kato ((KyotoyUniversity)y)Mayu Iwata (Osaka University)
http://research.microsoft.com/en‐us/people/tesakai/intent2.aspx

March 8, 2012@NII, Tokyo

What do you expectfrom a search engine?

Ambiguous query

Microsoft software!
Workplace!

What do you expectfrom a search engine?

Underspecified query

Harry Potter
films!

Harry Potter
b k!
books!

Harry Potter thecharacter!

Search Result Diversification

SERP (Search En
ngine Resu
ult Page)

Given an ambiguous/underspecified
Gi
bi
/ d
ifi d query,produce a single Search Engine Result Page thatsatisfies different user intents!

INTENT 2 Subtasks
INTENT‐2• Subtopic Mining (Chinese,(Chinese Japanese,
Japanese English)
INPUT: query (e.g. “harry potter”)
OUTPUT: ranked
k d llist off subtopicb
strings((e.g.
g “harryy potterpbook,, harryy potterpfilm,, harryy
potter the character…”)• Document Ranking (Chinese,(Chinese Japanese)
INPUT: query (e.g. “harry potter”)
OUTPUT: diversified ranked list of web pages
Main challenge: Balancing relevance and diversityDepartures from INTENT‐1
INTENT 1 (1)• English Subtopic Mining (new!) will share thep set with TREC 2012 diversityy task
topic
• Inclusion of queries that do not require
diversification (one item search) – systems
have to selectively diversify• Organisers will provide query suggestions andbaseline search results to participants –researchers who do not have a search engine
can easily participate

Departures from INTENT‐1
INTENT 1 (2)INTENT‐1@NTCIR‐9

INTENT‐2@NTCIR‐10

INTENT‐1
topicsINTENT‐2
topicsINTENT‐1
systems

INTENT‐2
systems

INTENT‐1 systems
rerun against
INTENT‐2 topics((Aree two
o topic
op c sets
se scomparable?)

INTENT‐2 systems
also run with
INTENT‐1 topicsp(Measure progress)

IMPORTANT DATESPARTICIPANTS

ORGANISERS

May 2012
M
TTopicsi andd baselinesb lireleasedldJulyy 2012
All submissions due
Aug‐Dec 2012 Identify intents from subtopics ‐>per intent relevance assessments
per‐intent
Jan 2013
Evaluation results released
March 2013? Draft papers due
May 2013?
Camera ready papers due
NTCIR‐10
June 18‐21
INTENT‐1 had 17 research teams from 7 countries/regions!
But we’d like to see more!

1Click‐2

1CLICK‐2@NTCIR‐10 Kickoff(English version)
TASK ORGANISERS: Makoto Kato, Takehiro Yamamoto (Kyoto U),
T t
Tetsuya
Sakai,
S k i Young‐InY
I Song,S
Zhi h
Zhicheng
D
Dou(MSRA),(MSRA)
Virgil Pavlu, Matthew Ekstrand‐Abueg, Shahzad Rajput (Northeastern U)Mayuy Iwata ((Osaka U))

http://research.microsoft.com/en‐us/people/tesakai/1click2.aspx

March 8, 2012@NII, Tokyo

Traditional Search= MoreMore‐than‐One
than One Click AccessEnter query湘南厚木病院(Shonan Atsugi Hospital)

Cli k SEARCH button
Click
Scan ranked list of URLs

Cli k URL
Click
Read URL contents

Get all desired information

One Click Access
湘南厚木病院(Shonan Atsugi Hospital)

Enter query

The system outputs X‐string

Cli k SEARCH button
Click
Phone: 046‐223‐3636. Fax: 046‐223‐3630. Address: 118‐
1 Nurumizu, Atsugi, 243‐8551. Email: soumu@shonan‐
atsugi jp Visiting hours: general ward Mon
atsugi.jp.
Mon‐FriFri 1515‐20;
20;
Sat&Holidays 13‐20 / Intensive Care Unit (ICU) 11‐11:30,
15:30, 19‐19:30.

Get all desired information

Phone: 046‐223‐3636.
Fax: 046‐223‐3630.
Address: 118‐1
Nurumizu,, Atsugi,
g , 243‐8551. Email:soumu@shonan‐
atsugi.jp. Visiting hours:
general ward Mon‐Fri
15‐20; Sat&Holidays 13‐
20 / Intensive Care Unit
(ICU) 11‐11:30, 15:30,
19‐19:30.

Particularly important formobile search

Go beyond the "ten‐blue‐link" paradigm in Web search3

Benefit of 1CLICK Access

Immediate & Direct Access
When you have to make a call to the Shonan
Atsugi Hospital as soon as possible

Traditional
search engine

1CLICK access system
Phone: 046‐223‐3636. Fax: 046‐223‐3630.

…I found it

Address: 118‐1 Nurumizu, Atsugi, 243‐8551. Email:soumu@shonan‐atsugi.jp. Visiting hours: general ward
Mon‐Fri 15‐20; Sat&Holidays 13‐20 / Intensive Care Unit
(ICU) 11‐11:30,
11 11 30 15:30,
15 30 19‐19:30.
19 19 30

I found it!

Queries• 8 query types based on [Li et al., SIGIR09](for which the user's information need can be satisfied by thesearch results page)– ARTIST, ATHELETE, POLITICIAN, ACTOR– FACILITY,, LOCAL– DEFINITION (of a term)– QA (Natural language questions)• Queries were selected out of a large query log ofa commercial search engine
J. Li, S. Huffman, and A. Tokuda. Good abandonment in mobile and PC internet search. ACM SIGIR 2009

Evaluation with iUnits• Manual/automatic matching between the system
output and iUnits (atomic pieces of information)
Ph046 223 3636 Fax:
F 046‐223‐3630.
046 223 3630
Phone:
046‐223‐3636.
Address: 118‐1 Nurumizu, Atsugi, 243‐
8551 Email: soumu@shonan
8551.
soumu@shonan‐atsugi
atsugi.jp.
jp• Phone number: 046‐223‐3636

System output• Fax number: 046‐223‐3630• Address: 118‐1 Nurumizu, Atsugi

iUnits• Systems are evaluated from viewpoints of
INFORMATION retrieval rather than document
retrieval1CLICK‐2
1CLICK
2 Subtasks
Participants must submit at least one run to• Main Tasks– Mandatoryy Runs: ggenerate X‐stringg based on
baseline web search results– Oracle Runs ((optional):p) ggenerate X‐stringg based on
Webpage URLs supporting the gold standard iUnits– Openp Runs ((optional):p) ggenerate X‐stringg
by searching the live Web on their own

OR• Query classification subtasks– Given a query, return its query type (ARTIST, QA, etc.)Departures from 1CLICK
1CLICK‐1
1• The language scope was expanded:
J
Japanese& English
E li h• More query types (4 → 8 types)• A new info. unit iUnit will be used for evaluation• Automatic/manual matching between
the X‐string and iUnits• DiversifiedDi
ifi d tasksk– Fair comparison by a common dataset(mandatory runs)– You can join us
even only with query classification tasks!

IMPORTANT DATESPARTICIPANTS

Apr 2012
Aug 2012
Oct 2012
Nov 2012‐
Jan 2013
Feb
b 2013
Mar 2013
Jun 2013

ORGANISERS

Sample queries and nuggets released
Test queries released
Run submissions due
Nugget match evaluation
Evaluationl i results
l releasedldDraft ppapersp due
NTCIR‐10

J i us if you are interested
Join
id iin IR,IR IE,IE MOBILE SEARCHSEARCH,SUMMARISATION, or the next search paradigm

PatentMT‐2

The	
  Patent	
  Machine	
  Transla/on	
  Task	  	
  (PatentMT)	

Isao	
  Goto	  (NICT)	
  
Bin	
  Lu	
  (City	
  Univ.	
  of	
  Hong	  Kong	  /	
  Hong	  Kong	  Ins/tute	
  of	
  Educa/on)	
  
Ka	
  Po	
  Chow	
  (Hong	
  Kong	  Ins/tute	
  of	
  Educa/on)	
  Eiichiro	
  Sumita	
  (NICT)	
  Benjamin	
  K.	
  Tsou	  (Hong	
  Kong	  Ins/tute	
  of	
  Educa/on	
  /	
  City	  Univ.	
  of	
  Hong	  Kong)	
1	

Mo/va/on	

There	  is	
  a	
  signiﬁcant	
  prac%cal	
  need	  for	
  patent	
  transla/on.	
  to	
  understand	
  patent	
  informa/on	
  wriLen	
  in	
  foreign	
  languages.	
   to	
  apply	
  for	
  patents	
  in	
  foreign	
  countries.	
  	
  




Patents	
  cons/tute	
  one	  of	
  the	
  challenging	
  domains.	
  Patent	  sentences	
  can	  be	
  quite	  long	  and	  contain	
  complex	  structures.	
   Transla/on	
  between	
  languages	
  with	  largely	
  diﬀerent	
  word	
  order	
  is	
  diﬃcult	
  for	
  long	  sentences.	
  


2	

Goals	  of	
  PatentMT	












To	
  develop	
  challenging	
  and	
  signiﬁcant	
  prac%cal	
  research	
  into	  patent	
  machine	  transla/on.	
  To	
  inves%gate	
  the	  performance	
  of	
  state-­‐of-­‐the-­‐art	
  machine	
  transla/on	
  systems	
  in	
  terms	
  of	
  patent	
  transla/ons	
  involving	
  Japanese,	
  English,	
  and	
  Chinese.	
  To	
  compare	
  the	
  eﬀects	
  of	
  diﬀerent	
  methods	
  of	
  patent	
  transla/on	
  by	
  applying	
  them	  to	
  the	
  same	  test	  data.	
  To	
  create	
  publicly-­‐available	
  parallel	
  corpora	
  of	
  patent	
  documents	
  and	  human	  evalua/ons	
  of	
  MT	
  results	  for	
  patent	
  informa/on	
  processing	  research.	
  To	
  drive	
  machine	
  transla%on	
  research,	
  which	
  is	
  an	
  important	
  technology	
  for	
  cross-­‐lingual	  access	
  of	
  informa/on	
  wriLen	
  in	
  unknown	
  languages.	
  The	
  ul/mate	
  goal	  is	
  fostering	
  scien%ﬁc	
  coopera%on.	
  3	

Findings	
  of	
  PatentMT	
  at	
  NTCIR-­‐9	SMT	
  was	  the	  best	  system	
  for	
  Chinese	
  to	
  English	
  and	
  English	  to	
  Japanese	
  patent	
  transla/on.	
  

This	
  is	
  the	
  ﬁrst	  %me	
  for	
  SMT	  to	
  be	
  demonstrated	
  equal	  or	
  beBer	
  quality	
  than	  that	  of	
  the	
  top-­‐level	  RBMT	
  for	  English	
  to	
  Japanese	
  patent	
  transla/on.	
   The	  pre-­‐ordering	
  method	  of	  NTT-­‐UT	
  for	
  SMT	  is	  very	  eﬀec/ve	
  for	  English	
  to	
  Japanese	
  patent	
  transla/on.	
  






80%	
  of	
  patent	
  sentences	
  could	
  be	
  understood	
  in	
  the	  best	
  system	
  for	
  Chinese	
  to	
  English	
  patent	
  transla/on.	
  RBMT	
  was	  the	  best	  system	
  for	  Japanese	
  to	
  English	
  patent	  transla/on.	
4	Remaining	
  Issues	
  of	
  NTCIR-­‐9	

Prac/cal	
  evalua/on	
  The	
  quality	
  of	
  translated	
  sentences	
  was	  evaluated	  at	
  NTCIR-­‐9.	
  
 More	  prac/cal	
  evalua/ons	  are	
  also	  expected.	
  


5	

Outline	
  of	
  the	  Plans	
  for	
  NTCIR-­‐10	

Three	
  subtasks:	  Subtasks	

Training data	

Chinese to English	
 1 million sentence pairs	
Japanese to English	

Approximately 3.2 million sentence pairs

English to Japanese	(Subtasks	
  and	  training	
  data	  are	  the	
  same	  as	
  at	
  NTCIR-­‐9)	

Par/cipants	
  select	
  subtasks	
  in	
  which	  they	  wish	  to	
  par/cipate.	
   Large	  scale	  parallel	
  corpora	
  and	  new	
  test	  sets	
  will	  be	
  provided.	
   Prac%cal	
  evalua%on	
  will	  be	
  added	  (under	  considera/on).	
   Human	
  evalua%on	
  will	  be	
  carried	
  out.	
  

6	

Diﬀerences	
  from	  NTCIR-­‐9	
Prac/cal	
  Evalua/on	
   New:	
  To	
  explore	
  prac%cal	
  MT	
  performance	
  in	
  appropriate	
  ﬁelds	
  for	
  patent	
  machine	
  transla/on.	
(under	  considera/on)	

Similar	  to	
  the	
  NTCIR-­‐9	
  evalua%on.	
  Quality	
  of	
  translated	
  sentences	
  will	  be	
  evaluated.	
  Addi/ons:	
  Intrinsic	
  Evalua/on	

Chronological	  evalua/on	
  Comparison	
  between	
  NTCIR-­‐10	
  and	
  
NTCIR-­‐9	  to	
  measure	
  progress.	

Comparison	
  of	
  CE	
  and	  JE	  transla/ons	  Mul/lingual	  using	  the	
  same	  English	
  reference	
  will	  evalua/on	
  be	
  added.	
  7	Can the machinetranslated texts showthat patent document A
should be rejected? 	

MT	
Japanese/
ChineseEnglish	

English	＝	Preparation	

Evaluation	

Possible	
  Approach	
  to	
  Prac/cal	
  Evalua/on	English	
Manualtranslation	Rejected patentdocument A	

Text that shows
Human with patentpatent document A examiner experience	
should be rejected. 	

(The	
  feasibility	
  of	
  this	  is	
  under	
  inves/ga/on.	
  	
  
	
  We	
  are	
  working	
  hard	  to	
  make	  necessary	  arrangements.)	
  8	Chronological	
  Evalua/on	
In	
  addi/on	
  to	
  the	  new	
  NTCIR-­‐10	
  test	  sets,	  the	
  
NTCIR-­‐9	
  test	  sets	
  will	  be	
  also	  translated.	
   Transla/ons	
  of	
  the	
  NTCIR-­‐9	
  test	  sets	  at	
  NTCIR-­‐10	
  will	  be	
  compared	
  to	
  the	
  NTCIR-­‐9	
  submissions.	
   This	  allows	
  measurement	
  of	
  the	
  progress	  from	
  NTCIR-­‐9.	
  


9	

Mul/lingual	
  Evalua/on	




We	
  will	  produce	
  a	  C-­‐J-­‐E	
  mul%lingual	
  test	  set.	
  	
  A	
  C-­‐J-­‐E	
  mul/lingual	
  test	  set	
  enables	
  comparison	
  of	
  the	  CE	  transla/ons	
  and	  the	  JE	  transla/ons	
  using	  the	
  same	  English	
  reference	
  data.	
  

10	

The	
  Flow	
  and	  Tenta/ve	
  Schedule	
  of	
  the	  Task	
Par/cipants	
2012/7	
Training	
  period	
  
	
  (4	
  months)	
2012/11	Test	
  period	  (2	
  weeks)	

2013/2	
2013/6	
Workshop	  Organizers	

Training	
  and	
  development	
  data	Test	
  sentences	
Translated	
  results	
Evalua/on	
  results	  and	
  reference	
  data	
11	

Why	
  is	
  it	
  so	
  exci/ng	
  to	  par/cipate	
  in?	

Patents	
  are	
  one	  of	
  the	
  challenging	
  domains	
  for	
  MT.	  Patent	  sentences	
  could	
  be	
  quite	  long	  and	  contain	
  complex	  structures.	
   Transla/on	
  between	
  languages	
  with	  largely	
  diﬀerent	
  word	  order	
  is	
  diﬃcult	
  for	
  long	  sentences.	
  

Par/cipants	
  will	  receive	
  evalua%on	
  results	  for	  their	
  MT	
  quality.	
   Par/cipants	
  can	
  use	
  large-­‐scale	
  patent	
  parallel	
  and	
  monolingual	
  corpora.	
   Par/cipants	
  can	
  choose	
  subtasks	
  from	  three	  language	
  direc/ons,	
  including	
  the	
  language	
  direc%on	
  of	
  Chinese	
  to	
  English.	
  




We	
  look	  forward	
  to	
  many	  groups	
  par/cipa/ng	
  in	
  PatentMT	
  at	
  NTCIR-­‐10!	
  12	SpokenDoc‐2

NTCIR‐10

SpokenDoc‐2
IR (Information Retrieval) for Spoken Documents

Kiyoaki Aikawa1, Tomoyosi Akiba2 , Seiichi Nakagawa2,
Hiroaki Nanjo3, Hiromitsu Nishizaki4, Yoichi Yamashita5
1Tokyo University of Technology, Japan
2Toyohashi University of Technology, Japan
3Ryukoku University, Japan
4University of Yamanashi,
Yamanashi Japan
5Ritsumeikan University, Japan

What is “SpokenDoc
SpokenDoc‐22”??
• Second round of the IR for spoken documents• Finding the information related to given aquery ffrom too muchh speechhd
data
SpeechpdataTranscribed text byy automatic speechprecognitiong
ASR“information retrieval”

Information retrieval (word‐unit transcription)
In for ma tion re triev al (sub‐word‐unit)
Problem Including a lot of errors!!

Participants of SpokenDoc‐2 will challenge

Information retrieval from very noisy text data
Techniques for SpokenDoc may be used for OCR or Machine Translated text retrieval

Target Speech Data
• Type off speechh datad– Broadcast news speech, podcast, lecture speech…
Having noisier words

Our target• Databases– CSJ (Corpus of Spontaneous Japanese)• 2,702
2 702 lecture speeches,
speeches 628 hours– New target! Real academic meeting lectures collection• Over 70 speeches from the spoken document processing
workshops

SpokenDoc 2 Task Overview
SpokenDoc‐2• Sub‐task1: Spoken Term Detection• Sub
Sub‐task2:
task2: Spoken Document Retrieval

Spoken Term Detection
1 Finding
1.
i di utterances including
i l di a query term
QUERY TERM:“Information Retrieval”

Utterances…Spoken term detection…

Search…
… information retrieval …

Find!

…spoken document retrieval …

Spokenp
documents collection

2. A qqueryy term is included or not included in a
spoken document?

Spoken Document Retrieval• Ad‐hoc Information Retrieval from lecturepspeeches• Finding the passages including the relevant
information related to a given query topic• Query– Text query– Spoken Query (optional)

Merits for Task Participants• The
Th organizers
iwill
ill provide
id theh richi h transcriptionsi ibbyan automatic speech recognition system– Participants can focus on developing IR technique for noisy
data, although you have not any techniques for speech
recognition• Participants will get the real lecture speeches:• Academic meeting lectures– All lectures have reference transcriptions– SomeS
lectures
l
have
h
slide
lid d
data andd digitalized
di i li d papers

All data is Japanese• Schedule– Mar.
M 2012:
2012 releasel
off theh taskk d
descriptioni i– June 2012: release of the reference automatic transcription– Sept.
Sept 2012: dry
dry‐runrun– Nov. 2012: formal‐run evaluation– Nov. 2012‐Feb.2013: relevance judgment– Feb. 2013: release of the evaluation results– May 2013: camera ready submission due– JuneJ2013 NTCIR‐10
2013:
NTCIR 10 workshop
k h meetingi• Contact– SpokenDoc‐2
SpokenDoc 2 organizers:• E‐mail: ntcadm‐spokendoc2@cl.ics.tut.ac.jp– Web ( coming soon! )• http://www.cl.ics.tut.ac.jp/~sdpwg/index.php?ntcir10– Twitter: @spokendoc2

RITE‐2

RITE-2R e c o g n i z i n g 	
  I n f e r e n c e 	
   i n 	
  TExt@NTCIR10	  IntroducIon	
  to	
  NTCIR-­‐10	
  RITE-­‐2	
  Task	  (Recognizing	
  Inference	
  in	
  TExt)	

Yotaro	  	
  Junta	
  Watanabe1	
 Mizuno1	
1Tohoku	
  University	
  Shuming	
  
Shi6	
  
6MicrosoS	
  Research	
  Asia	  Yusuke	  	
  Miyao2	

Tomohide	  Shibata3	

2NaEonal	
  InsEtute	  	
  3Kyoto	  of	  InformaEcs	
  Hiroshi	  Kanayama7	  University	
  Koichi	  Takeda7	
  7IBM	
  Research	
  Cheng-­‐	  Wei	
  Lee4	  Chuan-­‐	  Jie	
  Lin5	  

4Academia	  	
  5NaEonal	
  Taiwan	
  Sinica	  Hideki	  Shima8	  Ocean	
  University	
  Teruko	  Mitamura8	
  8Carnegie	
  Mellon	
  University	
  NTCIR-­‐10	
  Kick-­‐oﬀ	
  Event	
  March	  8th,	
  2012	RITE-2R e c o g n i z i n g 	
  I n f e r e n c e 	
   i n 	
  TExt@NTCIR10	
  Overview	
  of	
  RITE-­‐2	

• RITE	  is	
  a	
  benchmark	
  task	  (not	  a	
  compeIIon!)	
  for	
  automaIcally	
  detecIng	
  semanIc	
  relaIons	
  between	  two	
  sentences	  – Entailment,	
  paraphrase	
  and	  contradicIon	
  in	
  text	  • Given	
  t1(text),	
  can	  a	
  computer	
  infer	
  that	
  
	
   	
   	
   	
   	
   	
  t2(hypothesis)	
  is	
  most	  likely	
  true?	
  – t1:	
  Yasunari	
  Kawabata	
  won	  the	
  Nobel	  Prize	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  in	
  Literature	
  for	
  his	
  novel	  “Snow	  Country”	
  – t2:	
  Yasunari	
  Kawabata	
  is	
  the	
  writer	  of	
  “Snow	  Country”	  • Target	
  languages	  – Japanese,	
  Simpliﬁed	
  Chinese,	
  TradiIonal	
  Chinese	RITE-2

RITE-­‐2	
  Subtasks	

R e c o g n i z i n g 	
  I n f e r e n c e 	
   i n 	
  TExt@NTCIR10	
  • Binary-­‐Class	
  (BC)	
  Subtask	  – Given	  a	
  text	  pair	  (t1,	
  t2),	
  idenIfy	
  whether	
  t1	
  entails	
  t2	
  or	
  
not	
  

• MulE-­‐Class	
  (MC)	
  Subtask	  – Given	
  (t1t2	
,	  t2),	
  classify	
  
our	
  
relaIons	
  Entail	 a	
  text	  t1p
	  eair	  ntails	  t1	
  has	
  tfhe	  same	
  meaning	
  as	
  t2	
  (paraphrase)	
Not	
  Entail	

t1	
  contradicts	
  t2	

otherwise	
  (independence)	

• Entrance	  Exam	  Subtasks	
  – BC	
  Subtask	  • Same	  as	
  the	  BC	  subtask	
  in	
  terms	
  of	
  input	
  and	
  output.	
  All	  the	
  data	  is	
  created	
  from	  actual	
  college-­‐level	
  entrance	
  exams.	
  – Search	
  Subtask	  • Given	
  a	
  hypothesis	
  t2	
  and	  resources	
  (Wikipedia,	
  textbook),	
  idenIfy	
  whether	
  t2	
  is	
  true	  or	
  not	
  

RITE-2R e c o g n i z i n g 	
  I n f e r e n c e 	
   i n 	
  TExt@NTCIR10	
  RITE-­‐2	
  Overview	

Does	
  t1	
  entail	
  (infer)	
  t2?	
  t1: Yasunari Kawabata won the Nobel Prize in Literature for his novel “Snow Country”.
t2: Yasunari Kawabata is the writer of “Snow Country”.
Subtask	
  Input	
  Output	  EvaluaEon	
  BC	
   (t1,	  t2)	  Y	
  (Yes,	
  t1	
  ⇒ t2)	
  N	
  (No)	
  Accuracy	  MC	
   (t1,	  t2)	
  

Yes	
  
	
  	
  F	
  (forward;	
  t1	
  ⇒ t2)	
  
	
  	
  B	
  (bidirecIonal;	
  t1⇔t2)	
  	
  
No	
  
	
  	
  C	
  (contradicIon)	
  	
  
	
  	
  I	
  	
  (independence)	
  Accuracy	
  RITE
SystemEntrance	
   (t ,	
  t )
Exam	
  BC	
   1 2 	  Entrance	
  Exam	
  Search	
   (t2,	
  book)	
  Eval(Automatic)

Y	
  /	
  N	
  

Accuracy	  T	  (True)	
  /	
  N	
  (Not)	
  Accuracy	
  

9	
  

RITE-2

	
  	
  	
  	
  	
  	
  	
  	
  Why	
  you	
  should	  parIcipate	R e c o g n i z i n g 	
  I n f e r e n c e 	
   i n 	
  TExt@NTCIR10	
  In	
  addiIon	
  to	
  researchers	
  in	
  entailment	
  and	  paraphrases,	
  various	
  research	
  ﬁelds	
  can	
  beneﬁt	
  from	  RITE:	
  • Core	  technologies:	
  SemanIc	
  processing,	
  Lexical	
  acquisiIon,	
  Machine	
  learning,	
  etc.	
  • ApplicaEons:	
  InformaIon	
  retrieval,	
  QuesIon	
  answering,	
  Document	
  summarizaIon.	
  
	
  
We	
  try	
  hard	  to	
  welcome	  wide	  variety	
  of	
  parIcipaIons	
  –	
  from	  	
  undergraduate	
  students	
  to	
  industry	
  researchers,	
  from	  all	
  over	  the	
  world.	
  • Resource	
  pool:	
  will	  be	
  available	
  to	
  help	
  you	
  build	  a	
  prototype	  system	
  quickly	
  or	
  parIcipate	
  in	
  collaboraIon	
  by	
  sharing	
  useful	
  resource	
  and	
  tools	
  with	  others.	
  Website:	
  hQp://www.cl.ecei.tohoku.ac.jp/rite2	
  	
  

Math

NTCIR‐10 kickoff
M h8
March
8, 2012NTCIR‐10 MATH PILOT TASKThe Goal of NTCIR
NTCIR‐10
10 Math Task
• NTCIR Math Task aims at exploring methods
g its
for mathematical content access throughtask design and the construction of theevaluation dataset
dataset.
l ][ Formulaa mathematical relationship or rule expressed in symbols(Oxford Dictionary)

In science, a formula is a concise way of expressing
information, or a ggeneral relationshipp between quantities.
q(Wikipedia)

Math Information Access
Representations
Embedded image (png, gif, ...)

Character sequence (latex source)

Resourcesmathematical knowledge‐base andmath ontology

OpenMathlog(z_1)+log(z_2)log(z1)+log(z 2) == log(Z_1,log(Z 1,
Z_2)¥¥; z_1+z_2 ¥geq 0

Web‐browsable XML<math
h xmlns='http://www.w3.org/1998/Mathl 'h //
3/1998/M h/MathML' mathematica:form='TraditionalForm'
xmlns:mathematica='http://www.wolfram.com/XML/'> <semantics> <mrow> <mrow> <mrow> <mrow><mi> log </mi> <mo> &#8289; </mo> <mo> ( </mo><msub> <mi> z </mi> <mn> 1 </mn> </msub>

XML for math semantics<annotation‐xml encoding='MathML‐Content'><apply> <ci> Condition </ci> <apply> <eq /><apply> <plus /> <apply> <ln /> <apply> <ci>
Subscript </ci> <ci> z </ci> <cn type='integer'> 1</cn> </apply> </apply> <apply> <ln /> <apply><ci> Subscript </ci>

Wolfram MathWorld :13,081 entries (Sep.
13, 2011)
Wolfram Functionssite :
307,409 formulas(Sep. 15, 2011)

Strict Content
MathML(W3C
recommendation)

Requirement
NISTEP Policy Study

Mathematics as deserted sciencein Japanese S&T policy‐ Current situation on mathematicalsciences research in major countries
and need for mathematical sciencesfrom the science in Japan (2006.5)

Q. Is mathematics related
to your research?
Strongly related

Somewhat
relatedWikipedia:
26,566
mathematics
articlesRelated77% researchers
across diversity of
disciplines answered‘YES’.

Task Overview• [Math Retrieval Subtask]– Given a document collection,, retrieve relevant
mathematical formulae or documents for a given
q yquery.

• [Math[M th U
Understandingd t di SSubtask]
bt k]– Extract natural language definitions ofmathematical expressions in a document for theirsemantic interpretation.

Math Retrieval Subtask• Dataset (scheduled)– Scientific Articles from ArXiv e‐print server http://arxiv.org/
http://kwarc info/projects/arXMLiv/• ConvertedCt d iintot XML+MathML
XML+M thML byb arXMLiv
XMLi projectj t http://kwarc.info/projects/arXMLiv/• 10,000 docs for a dry run, additional 100,000 docs for a formal run• Search Types– The Math retrieval task uses the above 110K docs and can beenvisaged in three different search scenarios• Formula Search– Search for formula queries within the formulae database of the used dataset• Full‐Text Search– Search the document collection using formula queries. Combinations of keywords
and formulae.

• Open Information Retrieval– Search the document collection using free textual queries.

MathRetrieval
Subtask:
mathrepresentation
DatasetExample(MathML)

papers from arXiv.org

xhtml/xml

Math Retrieval Subtask :Query example

Formula search
1.


2.

h
l

f ( x) 2 dx

a b  b a

Full‐text search

Open IR

1.
Pythagorean Theorem

1.
What is the fifthsummandd in thehTaylor expansion ofsinus hyperbolicus?

2.
Bell curve in the form
of
x
2



1

3
3.



2

expp

2

2

lim a  b  cx 3.
NOT Bayes
B
Th
Theorem,but written asP( X | Y )2.
For which $n$ and$k$ is $PSL(n,k)$$PSL(n k)$ notcommutative.?

Math Retrieval Subtask :Formula Search Example
Retrieved formula

Query formulaMath Understanding Subtask• TaskT k definitiond fi iti– Extract natural language definitions of mathematicalexpressions in a document (Basic Task) with their semanticinterpretation (Challenge( h llTask,
k TBA).
)• Dataset (scheduled)()– Development Data
• 10 papers selected from ACL‐Anthology Reference Corpus• 30 papers selected from ArXiv.org
ArXiv org dataset which will be also used in
Math Retrieval Task.

– Data for Formal Run (submission period: five days)• 10 papers selected from ACL‐Anthology Reference Corpus• 10 papers selected from ArXiv.org dataset which will be also used in
Math Retrieval Task.

Math Understanding Subtask :Original text (pdf)
DDatasetEExamplelExample of Presentation MathML<mrow><mrow id="math_b6a1eb64.1"><mi mathvariant="italic">groupSim</mi><mo>?</mo><mrow><mo>(</mo><mrow><msub><mi>G</mi><mn>1</mn></msub><mo>,</mo><msub><mi>G</mi><mn>2</mn></msub></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow id="math_b6a1eb64.2"><mrow><mi mathvariant="italic">RI</mi><mo>?</mo><mrow><mo>(</mo><mrow><msub><mi>G</mi><mn>1</mn></msub><mo> </mo><mo>,</mo><msub><mi>G</mi>(http://www.nactem.ac.uk/brat‐annotation/) <mn>2</mn></msub></mrow><mo>)</mo>Definitions of mathematical objects in text

* Annotation tool

MathML parallel markup
Example of Content MathML<annotation‐xml encoding="MathML‐Content"<apply><eq /><apply xref="math_b6a1eb64.1"><ci>groupSim</ci><apply><selector /><ci>G</ci><cn>1</cn></apply><apply><selector
l t //><ci>G</ci><cn>2</cn></apply></apply><apply xref="math_b6a1eb64.2"><times /><apply><ci>RI</ci><apply><selector /><ci>G</ci><cn>1</cn>/</apply><apply><selector /><ci>G</ci><cn>2</cn></apply></apply><apply><ci>RC</ci><apply><selector /><ci>G</ci>
1 /Schedule
February‐April, 2012

Task framework development

April, 2012

Call for participation

May, 2012

Dataset and example topics release

Early‐mid October, 2012

Topic release for Open IR search type

Late October, 2012

Topic release for Formula, Full‐text search types, andMath understandingg subtask

Early November, 2012

Results submissions due for Formula, Full‐text searchtypes, and Math Understanding subtask

Mid November,Nb 2012R l submissions
Results
b i i
d for
due
f OpenO
IR searchh typeFebruary, 2013

Evaluation Results Released

March, 2013

Draft papers for NTCIR‐10
NTCIR 10 Proceedings Due

May, 2013

Camera ready for NTCIR‐10 Proceedings Due

June, 2013

NTCIR‐10 MeetingTask information• Contact
C t t– ntcir10adm‐math@nii.ac.jp• Task Web page
– http://ntcir‐math.nii.ac.jp/• Task Organizers– Akiko Aizawa (National Institute of Informatics, Japan)– Michael Kohlhase (Jacobs Universityy Bremen)– Iadh Ounis (University of Glasgow)• Task Advisors– Noriko Kando (National Institute of Informatics, Japan)– Fredric C. Gey (University of California, Berkeley)NTCIR‐10 Task Map
Summary

Context of Information AccessHistoryHit
Temporaldata

EconomyTechnology
Society

Infrastructure

Roles
Roles,
work task
Organisation
Contextual‐
taskInteraction

Hyperlink
Reference
Multi‐docsDocument
structure

Between‐
document

Time

Adapted from Ingwersen & Järvelin (2005)

User andInformation
User andsystemCharacter
WorddSyntax
Structure

NTCIRNTCIR‐10
10 Tasks
InfrastructuretaskWork task,
roles

IntentInteraction
Between‐
document

Covers a wide
range of context
d richi h media
diandsources

1Click

Time

RITE
Math

Document
structure
Math

News

CrossLink
PatentMT

Web

Legal

SpokenDoc

SpeechEntrance
Exam

NTCIR’s
NTCIR s Long
Long‐Term
Term View
InfrastructuretaskWork task,
roles

IntentInteraction
Between‐
document

Impact to real
cchallengesa e ges inour society

1Click
NTCIR’s

Time

CrossLink for
RITE Infrastructure

IA Evaluation
Math

Document
structure
Math

News

PatentMT
Web

Legal

SpokenDoc

SpeechEntrance
Exam

Why participate?
Case for students and industry

Why participate? (Students)• EasyEstart‐up– Much of experimental
setup isi provided
id d– Performance measuresare (often) defined• Publications
P bli i
– Comparison with otherparticipants can produce
stronger arguments– Inspiredpbyy theinternational
community for future
work• Diverse tasks– Range of Information
access tasks to tackle

Why participate? (Industry)• Establish your brand– To yyour end‐users andcompetitors– Recruit smart ppeoplep• Fair benchmarking– Comparison withh yourown products can bebiased– Critical self‐assessments• Faster development– Brush up
p yyour p
product oreliminating bugs in ashort period of time

• Early access to resulted
resources– Secondary resources
developed
p byy the task
are yours, too

How to participate
Simple six steps

How to participate
1. R
Read
d the
h taskkdescription and CFP
carefully
2. Contact a TO if you
have questions
3. Decide a task toparticipate
4. Register as aparticipant at NTCIR
website5. Fill ini U
User A
Agreement
Forms
6 Keep6.
K
an eye on atask’s ML, website,
etc to follow theetc.
activity
Don’t hesitate
to send a
feedback to TO

Reminder for participants

Participant’ss agreement
Participant• Submit a paper to NTCIR-10
Workshop Meeting• Attend to the meetingg and givegaposter presentation

Important Dates
For your diaryImportant Dates
08/03/2020
8/ /Ki k ff event inKick‐off
i T
Tokyo
k

30/06/2012

Task registration due

01/07/2012

Document set release

07 ‐ 12/2012

Dry run

09/2012 ‐ 02/2013

Formal run

01/02/2013

Evaluation results due

01/02/2013

Task overview partial release

01/03/2013

Participant paper submission due

01/05/2013

All camera‐ready copy for the Proceedings due

18‐23/06/2013

NTCIR‐10 Workshop Meeting,Meeting NII,NII Tokyo,
Tokyo Japan

Contact TO forthe exact
schedule

Memorial

Wrap‐up
Wrap up• The tenth cycle of NTCIR has started– Even stronger structure• Seven exciting tasks are running– Organised by 48 researchers worldwide• Lots
L t off opportunities
t iti forf iinnovative
ti workk– Exchange
g ggreat ideas with the communityy• What’s missing is your participation!

http://research.nii.ac.jp/ntcir/ntcir-10/

Thank you for your attention!
For further enquiries, contact the NTCIR officentc‐secretariat
ntc
secretariat nii.ac.jp

Q&A