Overview off the 7th NTCIR
WorkshopN k Kando
NorikoK dNational Institute of Informatics, Japanh
http://research.nii.ac.jp/ntcir/
//
h ii
j /
i /
kando (at) nii. ac. Jp
With thanks
th k ffor Tetsuya
T t
S
Sakai
k i forf th
the slides
lid
NTC7 OV 2008-12-17

Noriko Kando

1

NTCIR: NII Test Collection for Information Retrieval

Research Infrastructure for Evaluating IA

A series of evaluation workshops designed toenhancehresearch
h in
i information-accessi ftit h l i b
technologies
byproviding an infrastructure for large-scale evaluations.
■Data sets, evaluation methodologies, and forum

Project started in late 1997
Once every 18 months

Data sets (Test collections or TCs)
Scientific, news, patents, and web
Chin s K
r n J
p n s and
nd En
lish
Chinese,Korean,Japanese,
English

7th
6th
5th
4th
3rd
2st
1st
st

Tasks

0

20

40

# of groups

60

80

100

# of countries

IR: Cross-lingual tasks, patents, web,
QA：Monolingual tasks, cross-lingual tasks
Summarization, trend info., patent maps
Opinion analysis, text mining

Community-based
Research
Activities
C
it b
d R
h A
ti iti
NTC7 OV 2008-12-17

Noriko Kando

2
NTCIR-7 participants

82 groups from 15 countries

Information access (IA)• Whole process ofpreparing information fromthe vast collection of documents usable byusers.
• For
F example,l IR,
IR ttextt summarization,
i ti
QA
QA,
text mining, and clustering• Use human assessments as success criteria

NTC7 OV 2008-12-17

Noriko Kando

3

Focus of NTCIR
N
New
Challenges
Ch ll

Lab-type IR Test
Asian Languages/cross-language
Variety of Genre
Parallel/comparable Corpus

Intersection of IR + NLP
To make information in the
documents more usable forusers!
Realistic eval/user task

Forum for Researchers
Idea Exchange
Discussion/Investigation on
Evaluation methods/metrics
NTC7 OV 2008-12-17

Noriko Kando

4

Tasks Areas)at pastTasks (Researchof NTCIRs
NTCIR Workshops
pJapanese IR
Cross lingual IR
T Cross-linguala Patent Retrievalsmap/classifk Web
W b RetrievalRilsNavigational
Geo
Result Classification
Term Extraction
QuestionAnswering
Info Access Dialog
S
Summ
metricst i s
Cross-Lingual
Text Summarization

1st

2nd

3rd 4th

5th 6thnews

sci

Trend Information
Opinion Analysis
NTC7 OV 2008-12-17

Noriko Kando

5

NTCIR-7 Clusters

Cluster 2. User-Generated :- Multilingual Opinion Analysis
Cluster 3. Focused Domain : Patent- Patent
P t t TranslationT
sl ti ; English
E li h -> Japanese,J- Patent Mining paper -> IPC
Cluster 4. MuST :- Multi-modal Summarization of Trends
NTC7 OV 2008-12-17

Noriko Kando

MuuST; V
Visualizationn Challenge

Cluster 1. Advanced CLIA
- Complex CLQA (Chinese, Japanese, English)- IR for QA (Chinese, Japanese, English)

6

NTCIR 7 is made up of…
NTCIR-7of• Cluster 1: Advanced Cross-lingual
Cross lingual Information
Access (ACLIA) = CCLQA + IR4QA• Cluster 2: Multilingual Opinion Analysis task
(MOAT) + CLIRB• Cluster 3: Focused Domains= PATMT + PATMN• Multimodal
Multim d l Summarization
Summ i ti n of
fT
Trend
nd
information (MuST)• The 2nd International
Internati nal W
Workshoprksh p on
n Evaluating
Evaluatin
Information Access (EVIA)

Evaluation Workshops• "evaluation“
It is not an competition!

not an exam!

• Constructs a common data set usable forexperiments.
• provides to participants the data sets and unified
procedures for evaluation– Each participating research group conducts experiments
withh various approaches
h andd can participate withh own
purpose.

• Successful examples; TREC,TREC CLEF,
CLEF DUC,
DUC INEX,
INEX
and TAC, FIRE (new!) Community-based activities• Implications are various
NTC7 OV 2008-12-17

Noriko Kando

8

IA Systems Evaluation• Engineering Level:
Level Efficiency• Input Level: ex. Exhaustivity, quality, novelty of DB• ProcessP
Level:
Ll Effectiveness
Eff tiex. recall,precisionlli i• Output Level: Display of output• User Level: ex. Effort that users need
• Social L
Level: ex.. Importance
mp r n

NTC7 OV 2008-12-17

Noriko Kando((Cleverdon
r n&K
Keen
n 1966)966)

9

Retrieval Difficulty Varies with Topics
1.0000J-J Le ve l1 D auto
Effectiveness
Across SYSTEMS

Effectiveness Across TOPICS
on a System
検索システム別の11pt再現率精度

101
102

0.8000103

Average over
50 topics

1

A

104

B

105
106

C

107

0.8

precision

precision

0.60000.4000

D

108

E

109

F

110
111

0.6

0.4

0.2000

G

112

H

113

I

114
115

J

116

K

117

L

118
119

M
0.2

0.0000

120

N

121

O

122
123

P

124

0

2

4

6

8

0

0.

0.

0.

0.

0.

1.

0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

125

1

126
127

recallrecall

NTC7 OV 2008-12-17

Noriko Kando

10Retrieval Difficulty Varies with Topics
J JL
J-J
Le ve l1 D autot

1.0000Effectiveness
Across SYSTEMS

Effectiveness Across TOPICS
on a System
検索システム別の11pt再現率精度“Difficult
Topics”
Vary with
Averageover
50 topics
Systems

0.80001

J-J Level1 D auto

preciision

Mea
an Ave P
Precisionnecisionpre0.60001.00000 80000.80000.6000

0.4

104

B

105
106
107
108

E

A

F

B

G

C

112

H

D

113

I

E

J

F

116

G

117

K

M
0.2

N

0.2000

P
0

0

2

4

6

8

0

0.

0.

0.

0.

1.

0.00000

0.1

Topic##101-150
Requests#101 150

recall

NTC7 OV 2008-12-17

Noriko Kando

110
111

114
115

H

119

I

120
121

J

O

0.0000

109

118

L

0.40000.20000.

preecision

0.4000

103

D

0.6

102

A

C

0.8

101

0.2

0.3

122

K

123

L

124

For N reliable andOstableevaluation,using substantial #topics is inevitable.
110.4

0.5 M 0.6

recall

0.7

0.8

0.9

1

125
126
127

TC usable to evaluate?

Pharmaceutical R & D
Phase I:

Phase II:

Phase III:In Vitro

AnimalExperiments

Clinical Test
Test with
Healthyy Human
Subject

NTC7 OV 2008-12-17

Noriko Kando

Phase IV:

12

NTCIRTC usable to evaluate what?

Test Collections

Phase I:
Laboratorytype Testing

Phase II:
Sharing
Modules ,Prototype
testingPharmeceutical R
Phase II:
&PhaseD I:
In Vitro

Levels of
Evaluation

AnimalExperiments

Users’ information seeingtasks
Phase III:

Phase IV:

Phase III:

Phase IV:

Controlled
UncontrolledInteractive
Pre operational
Pre-operational
Testing using
Testing
human Subjects

Clinical Test
Test with
Healthyy Human
2.Input Level、Subject
6.Social Level

4.User Level、5.Output Levle

1.Engineering Level efficiency
3.Process Level: effectiveness
NTC7 OV 2008-12-17

Noriko Kando

13

Summary of “What
What is NTCIR”• Providing a scientific basis for understanding
the effectiveness of automated information
access technologies• Leveraging the R&D and technology transfer• ReusableR
bl T
Test collectionll
is a key
k component• Evaluatingg search effectiveness is not easy.
yA small-scale or carelessly-designed TCs may
skew the test results

NTC7 OV 2008-12-17

Noriko Kando

14

NTCIR-7: Advanced CLIA
Teruko Mitamura (CMU)
Eric Nyberg (CMU)
Ruihua Chen (MSRA)Fred Gey (UCB),
Donghong Ji (Wuhan Univ)
Noriko Kando (NII)
Chin-Yew Lin (MSRA)
Chuan-Jie Lin (Nat Taiwan Ocean Univ)
Tsuneaki Kato (Tokyo Univ)
Tatsunori Mori (Yokohama N Univ)
Tetsuya Sakai (NewsWatch)
Ad i
Advisor:K L K k (Queen
K.L.Kwok(Q
C
College)ll)
NTC7 OV 2008-12-17

Noriko Kando

15

NTCIR-7: Advanced CLIA
Question
Analyzers

CCLQAXML,AP
I Questions

Translation& Retrieval

Q with qtypes

IR for QA

AnswerExtraction &Formatting

Retrieved
documents

CLIR

-Test
T t effectiveness
ff ti
of
f
OOV, PRF, QE in QA

Answers

Eval. By
- IR Effectiveness- QA Effectiveness- Focused Retrieval
NTC7 OV 2008-12-17

Noriko Kando

16

ACLIA
Complex Cross-lingual
Cross lingual Question Answering(CCLQA) Task

Small teams thatd nott possessdoan entire QA
systemcan contribute

Different
teams
can exchange
and create adream team“dream-team”
QA system
IR andd QA communities
i i can collaboratll b

CCLQA= Complex CLQA
• Moving
g towards Advanced ComplexpQuestions
Q
from
Factoid Questions (NTCIR-5, NTCIR-6)• 4 questions types (events, biographies, definitions,and relationships)• Examples of Complex Questions– Definition questions: What is the Human Genome

Project?

– Relationshipp questions:qWhat is the relationshippbetween Saddam Hussein and Jacques Chirac?
– Event questions: List major events in formation ofE
European
Union.
U i– Biography questions: Who is Kim Jong-Il?

CCLQA = Complex Cross
Cross-lingual
lingual QA• Three document languages:
Simplified Chinese (CS)
Traditional Chinese (CT)Japanese (JA)• Four question languages:
CS, CT, JA plus English (EN)• Complex questions for Cross-Lingual QA
EN-CS, EN-CT and EN-JA• Monolingual QA with the same complex questions
CS-CS, CT-CT and JA-JA• Combination system evaluation
QA teams using IR4QA runs from other teams

Evaluation Metrics• “Manual” evaluation
Nugget pyramid method [Lin/Demnerg multiple
p assessors
Fushman 06] using
for judging nugget matches(Weighted F
F-measure)measure)• Automatic evaluation
POURPRE [Lin/Demner-Fushman[Li /DF h05]
modified for Chinese and JapaneseACLIA: Evaluation EPAN tool

NTC7 OV 2008-12-17

Noriko Kando

21

ACLIA: Evaluation EPAN tool
CCLQA:
Nugget Pyramid

IR4QA:MAP
MS nDCG
Q-Measure((preferencef
based )
NTC7 OV 2008-12-17

Noriko Kando

22

Traditional “ad hoc” IR vs IR4QA
Q• Ad hoc IR (evaluated using Average Precision
etc )
etc.)
- Find as many (partially or marginally) relevant
documents as possible and put them near the
top of the ranked list
using WHAT? )• IR4QA (evaluating using…
- Find relevant documents containing different
correct answers?
- Find multiple documents supporting the same
correct answer to enhance reliability of thatanswer?
- Combine partially relevant documents A and B
to deduce a correct answer?

Average Precision (AP)PrecisionPrcisi nat rank r

Number of 1 iff d
doc att r
relevant
is relevant
docs• Used widely since the advent of TREC
• Mean over topics is referred to as “MAP”
MAP• Cannot handle graded relevance(but many IR researchers just love it)Q measure (Q)Q-measure

PersistenceParameter β
set to 1

Blended ratio at rank r• Generalises AP and
handles graded relevance (Combines Precisionand normalised• Propertiespsimilar to AP
Cumulative Gain)and higher discriminativepowerp
S k i andd Robertson
R b t
EVIA 08• Not widely-used, but Sakai
provides a user model
has been used for Q
QA
for AP and Q
and INEX as well as IR

nDCG (Microsoft version)

Sum of discounted gains
f a systemfortoutputt t

m off discounted g
gains• Fixes a bug of the originalSum
for an ideal output
nDCG• But lacks a parameter that reflects
the user’s persistence• Most popular graded-relevance metricIR4QA evaluation packagepg(Works for ad hoc IR in general)

ComputesAP, Q, nDCG,
RBP, NCU[Sakai and Robertson
EVIA 08] and so on
http://research.nii.ac.jp/ntcir/tools/ir4qa_eval-en• 12 participants from China/Taiwan
China/Taiwan, USA
USA, Japan• 40 CS runs (22 CS-CS, 18 EN-CS)• 26 CT runs (19 CT
CT-CT
CT, 7 EN
EN-CT)CT)• 25 JA runs (14 JA-JA, 11 EN-JA)
Monolingual

CrosslingualOral presentations• CMUJAV (CS(CS-CS
CS, EN
EN-CS
CS, JA
JA-JA
JA, EN-JA)
EN JA)- Proposes Pseudo Relevance Feedback using LexicoSemantic Patterns (LSP-PRF)()• CYUT (EN-CS, EN-CT, EN-JA)- Uses Wikipedia in several ways; post hoc results• MITEL (EN-CS, CT-CT)- SMT and Baidu used for translation; data fusion• RALI (CS-CS, EN-CS, CT-CT, EN-CT)- Uses Wikipedia in several ways; high performanceafter bug fix

Other interesting approaches• BRKLY (JA-JA) A very experienced TREC/NTCIR
participant• HIT (EN-CS) PRF most successful• KECIR (CS(CS-CS)CS) QueryQ
expansioni lengthlth optimisedti i dfor each question type (definition, biography…)• NLPAI (CS-CS) Uses question analyses files fromother teams (next slide)• NTUBROWS ((CT-CT)) Queryy term filtering,g, data fusion• OT (CS-CS, CT-CT, JA-JA) Data fusion-like PRF• TA (EN-JA) SMT document translation from NTCIR-6• WHUCC (CS-CS) Document reranking

Please visit the posters of all 12 IR4QA teams!

NLPAI (CS-CS) used question
analysis files from other teams.
CSWHU-CS-CS-01-T:<KEYTERMS><KEYTERM SCORE="1.0">宇宙大爆炸</KEYTERM><KEYTERM SCORE="0.3">理论</KEYTERM></KEYTERMS>
Apath-CS-CS-01-T:<KEYTERMS><KEYTERM SCORE="1.0">宇宙大爆炸理论</KEYTERM></KEYTERMS>/KEYTERMSCMUJAV-CS-CS-01-T:<KEYTERMS><KEYTERM SCORE="1.0">宇宙</KEYTERM><KEYTERMKEYTERM SCORE="1.0">大</KEYTERM>
SCORE
大 /KEYTERM<KEYTERM SCORE="1.0">爆炸</KEYTERM><KEYTERM SCORE="1.0">理论</KEYTERM><KEYTERM SCORE="1.0">宇宙 大 爆炸 理论</KEYTERM><KEYTERMKEYTERM SCORE="1.0">宇宙大爆炸理论</KEYTERM>SCORE " 0" 宇宙大爆炸理论 /KEYTERM<KEYTERM SCORE="1.0">宇宙 大 爆炸</KEYTERM><KEYTERM SCORE="1.0">宇宙大爆炸</KEYTERM></KEYTERMS>

Different teams
come up withdifferent set ofquery terms withi hdifferent weights.
This clearly affects
retrieval
performance.
pSpecial thanks to
Maofu Liu
(NLPAI)

System ranking
by Q/nDCG vsthat by AP

CS

CT

JA

By
y definition,nDCG is more
forgiving for
low-recall
w
runs
un
than AP and Q.

Forming pseudo
pseudo-qrels
qrels

QUESTION: Can we get away with not doing
any relevancelassessments
t att all?
ll?
1. Sort pooled docs by(1) Number of runs that retrieved it; and then(2) Sum of its ranks within these runs.
runs
2. Take the top 10 docs in the sorted pool andtreat them all as L1-relevant!
L1 relevant!

Very interesting results will bepresented
t d att NTCIR
NTCIR-7!
7!

NTCIR-7: UGC (Blog)( g)

David K Evans (NII ->> Amazon Japan)
Yohei Seki (Toyohashi U Tech -> Columbia U)
LunWei Ku (National Taiwan Univ)
Le Sun ((Chinese Academy
y of Science))
Hsin-Hsi Chen (National Taiwan Univ)
Noriko Kando (NII)NTC7 OV 2008-12-17

Noriko Kando

34

Opinion Analysis - Roadmap
Genre
News
Review
Blog

Subjectivity
Holder Polarity
NTCIR-6
NTCIR-6 NTCIR-6
NTCIR-7
NTCIR-7 NTCIR-7
NTCIR-8
NTCIR-8 NTCIR-8

Strength
NTCIR-7
NTCIR-8

Stakeholder
Temporal Language
GranualityApplication
Chinese single
single-sentSummarization
sentSummarization
NTCIR-7
English clause
QA
NTCIR-8
NTCIR-8
Japanese multi-sent Opinion tracking
CJE
document Consistency checkin
Trend

Chinese, Japanese,
EnglishNTC7 OV 2008-12-17

Noriko Kando

35

NTCIR-7: MOAT ((on News))•Documents:

NEWS CCEJ•CLIR on Blog (CLIRB) Cancelled•Multilingual Opinion Analysis (MOAT)
TraditionalC,Simplifed
Simplifed C,
C J,EJE•TraditionalC• selecting relevant documents from ~25 topics usedin ACLIA• Following Roadmap, but change the genre• Relevant, Opinionated,p
Polarityy (Pos, Neg,g Nue),
Holder, Stakeholder (Object), ??Strength??

NTC7 OV 2008-12-17

Noriko Kando

36

MOAT Participants

Beijing university of posts andtelecomunications
Chinese Academy of
Sciences(NLPR-IACAS)n(NL)
City University of Hong Kong
CUHK（The Chinese University ofHongg Kong)-PolyU(The
g)
y (
Hong
g Kongg
Polythechnic University)Tsinghua(Tsinghua University)
DAEDALUS, S.A.
Dalian University of Technology
Hiroshima City University
Information and Communications
U i
Universityi
Keio University
Louisiana State
U i
University(University
it (U iit offM
Marylandl dCollege Park)

National Taiwan University
NEC
NEU Natural Languageg gProcessing Lab
Peking University
Peking University(ICL)
Pohang University of Science andTechnology
SICS - Swedish Institute ofC
Computer
Science
S i
Technical University ofDarmstadt
Th G
The
Graduated t U
University
i
sit f
for
Advanced Studies(SOKENDAI).
Tornado Technologies Co., Ltd.,
TaiwanTaiwan.
Toyohashi University ofTechnology
University of Neuchatel
University of Sussex
Yuan Ze Univ.

80+ registerd, 30+ resigned when docs were changed, 42
registered to News MOAT, 24 sugmitted
NTC7 OV 2008-12-17

Noriko Kando

37

NTCIR-7: Focused Domain (Patent)()

Atsuhi Fujii
j (Univ Tsukuba)
Taiich Hashimoto (Tokyo Insti Tech)
Makoto Iwayama (Tokyo Insti Tech/ Hitach)
Hidetsugu Nanba (Hiroshima City Univ)
M
Masao
Utiyama
U i(NICT),(NICT)
Mikio Yamamoto, U Tsukuba)
T k hit Uts
Takehito
Utsuro (U Tsukuba)
Ts k b )NTC7 OV 2008-12-17

Noriko Kando

38

NTCIR-7: Focused Domain (Patent)()

Documents:
10 Yrs Japanese Patent Application (NTCIR4-5)
10 Yrs USTPO Patents (NTCIR6)
Parallel Sentence Data (1.8 M sentences JE Pairs)
S i
Scientificifi P
Paper Ab
Abstracts (NTCIR 11-2)
2)
Patent Translation (PATMT) MT is key for CLIR

Training: 1993-2000,
1993 2000 Test: 2001
2001-2002
2002
One Ref Trans good??
Intrinsic Eval. ;BLEU, human assessments
Extrinsic Eval: CLIR task-based

Patent Mining (PATMN)
P(P) Cross-GenreG
PATP& ScientificfClassify Paper Abstracts in to IPC Classes
ML approach:h Classsify
Cl
if Absts
Ab t tot IPC ClassClIR Apprach: use invalidity search system to findrelevant Patent,
Patent then assign IPCs to Paper Absts
Absts.
NTC7 OV 2008-12-17

Noriko Kando

39

Patent classification and mining at
NTCIR
Organizers:
Makotok
Iwayama (Hitachi(h Ltd/Tokyo/ k Institute off Technology)h l)Hidetsugu Nanba (Hiroshima City University)
Taiichi Hashimoto (Tokyo Institute of Technology)
Atsushi Fujii (University of Tsukuba)
Noriko Kando (National Institute of Informatics)
NTC7 OV 2008-12-17

Noriko Kando

40

Goal: Automatic g
generation of patent
p
maps.
p
Example: Blue light-emitting diodes

Given

Problems to be solved
Crystalline

Reliability

S
Solutionns

Structure of active
layer

Long
operatinglife

Emissionstability

Emission
intensity

1998-145000
1998-233554

Electrodecomposition

1998-107318

1998-190063
1998-209498

1998-209495

ElectrodeElt darrangement

1998-215034
1998-223930

1998-173230
1998
1732301998-209499
1998-256602

1998-242515
1998-270757

StructureSttoff lilightht
emitting element

1998-135516
1998
135516
1998-242586
1998-247761

1998-242518

1998-135514
1998-256668

1998-012923
1998
012923
1998-247745
1998-256597

Systems automatically identify rows and columns
NTC7 OV 2008-12-17

Noriko Kando

41

Historyy• NTCIR-4 (2003-2004): Patent-map-creation subtask– Direct approach to creation of patent maps
– Hard tasks and insufficient evaluation• NTCIR-5
NTCIR 5 (2004-2005):(2004 2005): Classification subtask– Categorize patents to pre-defined categories called Fterms (multi-faceted(multi faceted and structured)– Relatively small number of test documents– Evaluate only strict matches in F-term
F term hierarchy• NTCIR-6 (2006-2007): Classification subtask– Increased the number of documents and topics (108 topics)– Evaluate partial matches in F-term hierarchy• NTCIRNTCIR-7
7 (2007(2007-2008):
2008) Mining subtask
NTC7 OV 2008-12-17

Noriko Kando

42

Feasibility
y Study:y automatic patentpmap
pgeneration at NTCIR-4 (2003-2004)
searchtopic
iapplication

retrievalt ildocuments

JAPIO abst
PAJ

topics and documents
in NTCIR-3
NTCIR 3 collection
Patent-map
Patentmap creation =Multi-faceted
patent clustering
NTC7 OV 2008-12-17classificationclass
f cat onvisualization
Noriko Kando

multi-dimensional
matrix43

Classification task overview
Training data

PMGS
(F-term descriptions)

Training

Patents withththemesandd F-terms
F t(1993-1997)

Theme is g
given
5B001
5B001
5B001

Sampling

Test data
Patents withthemeshandd F-terms
F(1998-1999)

Classifier
F-termF
term
assignment
5B001
5B001
5B001
AC04

NTC7 OV 2008-12-17

Noriko Kando

Evaluation
44

Patent mining at NTCIR-7
NTCIR 7 (2007-2008)(2007 2008)
Searches and/or classifying patents and scientific papers into IPC
Research paper written in
Japanese (Japanese / J2E
subtasks))
A Parrticipant System

Machine-translation
module (E2J / J2E)

Text classification module

Research paper written in
English (English / E2J
subtasks))

Japanese, English, and Crosslingualg(J-to-E, E-to-J) subtasks
Patent data
- written
itt ini JapaneseJ(Japanese / J2E)
- written in English(English / E2J)

List of IPC codes
NTC7
OV 2008-12-17
45
Nanba,
Fujii,
Iwayama, and Hashimoto.Noriko“TheKando
Patent Mining Task in the Seventh NTCIR
Workshop”, Patent Information Retrieval Workshop at CIKM 2008 (2008)

Summary of patent classification and mining• Automatic clustering of patents into
“problems”
problems and “solutions”
solutions are quite feasible,
but labeling and controlled evaluation needmore investigation.
• Granularity of F-term is appropriate forpatent map creation and becoming good.
good• Patent minting of scientific papers andp t nts are practicallypatentsp ctic ll needed.
n d d nn-KNN
KNN andndmachine learning have promise– The test collections for classification are available
for research purpose. The one for mining will beavailable to the public after Workshop Meeting•NTC7 OV 2008-12-17

Noriko Kando

46

Patent machine translation at NTCIR
Organizers:
Atsushih Fujii (University(off Tsukuba)k)Masao Utiyama (NICT)
Mikio Yamamoto (University of Tsukuba)
Takehito Utsuro (University of Tsukuba)

OV 2008-12-17
Fujii,NTC7
Utiyama,
Yamamoto, and Utsuro. “Toward the Evaluation of Machine Translation
Noriko Kando
47
Using Patent Information”, AMTA 2008

Historyy of Patent IR at NTCIR• NTCIR-3 (2001-2002)– Technology
T h lsurvey• Applied conventional IR
problems to patent data

• NTCIR-4 (2003-2004)– Invalidity
y search

Addressed patent-specific IR
problems

2 years of JPO
patentt t applications
li ti* JPO = Japanese Patent O

5 years of JPO
patentp
applications
pp sets wereBothdocument
published in 1993-2002• NTCIR-5
NTCIR 5 (2004-2005)(2004 2005)

10 years off JPO– Enlarged invalidity search patent applications• NTCIR-6 (2006-2007)– Added English patents

10 years of USPTO
patents grantedpg* USPTO = US Patent & Trademark Offi

Patent machine translations at
NTCIR-7 (2007-2008)• Patent
P t nt Machine
M chin Tr
Translation
nsl ti n (MT) is rrealistic
listic– Parallel corpora can potentially be produced from
JPO/USPTO patentpatent-document
document sets
– Decoders for statistical MT (SMT) are available• Two types of players– Organizer = Authors of this paper• Providing data, and evaluating participating MT systems– Participants = Researchh groups• They can use e.g., SMT and rule-based MT.

• Utility of patent MT– Cross-lingual patent retrieval– Filing patent applications in foreign countries

NTC7 OV 2008-12-17

Noriko Kando

49

Producing parallel corpora
JPO applicationsJ
ppn
1993-2002
Comparable(3.5-M docs)(not parallel)

J

E

J

E

Sentence-alignmentlmethodh d[Utiyama and Isahara, 2007]
TargetingTi“background” and“description”
description
NTC7 OV 2008-12-17

U
USPTO
grantsgr n
1993-2002(1.3-M docs)

J

E

Patent family
Patent set forsame invention

Sentence pairs
Parallel (alignment accuracy= 90%)
Noriko Kando

50

Extrinsic
evaluation
NTCIR-5NTCIR
5Patent claim

SearchS
rch topict pic
in Japanese

Human

SearchS
rch topic
t pic
in English

Evaluation by BLEU
Training data
1.8-M sentencepairs• System training• ParameterPt tuningt i

MT system

JPO applications
1993-2002

Invalidate patent
Translation in
Japanese

Evaluation by Mean Average
Precision (MAP)
NTC7 OV 2008-12-17

Performed byorganizers

Noriko Kando

IR system
Rankeddoc. list
51

Patent families• Member patents often claim “priority” under
the Parisar s Convention
onvent on– Related patents can easily be identified by priority
numbers– 85 K patentt t families
f ili (J(J-to-E)t E) were id
identified
tifi d• Merit of priority-based patent families– The application date is retroactive
to the original date
– First-to-file
f systemym
J translatingg
in many countries
JPO
Aug 22,22 2008

Free (or inexpensive) bilingual
corpora are growing! ! !

claiming
priority
i it

E

USPTO
Oct 24,24 2008
Aug 22, 2008

Examplep of patentpfamilyyInvention related to Microactuators
JPO

Patent family can be
identified byy prioritypy number

USPTO

J-E sentence p
pairs can beextracted from corresponding fieldEvaluation Methods• Intrinsic evaluation– Automatic evaluation by BLEU
– Manuall evaluationl• Adequacy and Fluency by 5-point rating• Extrinsic evaluation– QueryQ
translation
l i f
for C
Cross-Lingual
Li
lP
PatentRetrieval (CLPR), measured by Average
P
Precisioni i (AP)Patent machine translation• Constructed a large test collection for J/E MT: USTPO
JPO with 10 yyears off fullf textsand J• Large-scale sentence-alignment dataset (E-J sentence pairs)• Statistical MT (SMT)* vs. rule-based MT• Results demonstrated:– SMT is much better for CLIR
– Rule-based
R l bd MT iis good
d for
f h
human evaluationsl ti– Human evaluations and creation of referencetranslations must be carefully done (in the real world,
world
professional patent translators do use MT).
p p
• Test collection will be available for research purpose
after the workshop meeting*SMT : a system automatically learns the translation rules fromtheh given large-scalell sentence pairs.
NTC7 OV 2008-12-17

Noriko Kando

55

MuST: Multimodal Summarization for Trend
Information

Tsuneaki Kato (Tokyoy Univ)
Mitsunori Matsushita (NTTComm Sci Lab Æ Kansei Univ)

Multimodal summarization
for Trend Information
QueriesQ
i on ttrendsd“How the price of gasoline shifted during the year?”“What
What the situation has been in the PC market?
market?”“How terrible the typhoons were last autumn?”

Concise,Ciplain
l i textt t
Information graphics
Multimedia presentationtext including references to graphics
graphicsgpannotated with text

Visualization Platform

NTC7 OV 2008-12-17

Noriko Kando

57

The Roles of Data Set

Information Collected

Articles, Tables and Charts
MultimodalMl i d lSummarization
Annotations

Visualization
softwareft

Summaries,, Reportsp
Textual summaries, Charts and Tables

CLEF2008 2008-09-18

Noriko kando

58

Interactive and Exploratorypy Supportpp
of Information Utilization
Task, Interest

View
Visualization
Graphs, etc

Interaction of Users

Featurerepresentation
Understanding

Summarization& Visualiation

Collected
Infof
CLEF2008 2008-09-18

Re- Retrieval

Noriko kando

59

Multimodal Summarization for
Trend information (MuST)Example: Visualising the Japanesecabinet support rate

Gold standard

System output

Change over timeVisualization Platform

CLEF2008 2008-09-18

Noriko kando

61

Number of Participants by Tasks# of ParticipatingG
Groups120

ACLIA CCLQA

100

Opinion
CLQA
QA
Trend Info80
Summarization

60

Term Extraction
Web Retrieval

40

ChinesePatent MT

Chinese Korean

20

JÆE

JÆE,EÆJ、EÆC

Patent MiningxÆCJEK

Patent Retrieval

0

NonJapanese IR
CLIRJapanese IR

ACLIA
CL
IR4QA
R4Q
NTC7 OV 2008-12-17

Noriko Kando

62

[CCLQA]•Academia Sinica•Beijing
B iji Univ
U i off PostsP t &T l
Telecoms,
China•Carnegie Mellon Univ
•NICT
•NTT Corporation•Shenyang Institute of AeronauticalEngineering•Wuhan Univ
•Yokohama National Univ

•Hiroshima City Univ
•Information
Information and Communications Univ
•Chinese Academy of Sciences(ISCAS)•Keio Univ
•City Univ of Hong Kong
•National Taiwan Univ
•NEC
NEC
•Northeastern Univ
•Peking Univ
•Pohang Univ of Science and Technology•Swedish Institute of Computerp
Science•Technical Univ of Darmstadt•Graduate Univ for Advanced•Tornado Technologies Co., Ltd.,•Toyohashi Univ of Technology
n of Neuchatel•Univ•Univ of Sussex[IR4QA]•Carnegie Mellon Univ
•Chaoyang Univ of Technology•Chinese Academy of Sciences(ICT)
Harbin Institute of Technology +•Harbin
Heilongjiang Institute of Technology•National Taiwan Univ
[Must]•Open Text Corporation•Hiroshima City Univ
•Shenyang Institute of Aeronautical
Keio Univ•Keio
Engineering•Mie Univ•Toyohashi Unive of Technology•NICT•Univ of California, Berkeley•NEC•Univ of Montreal•Ochanomizu Univ (2 Groups)•Wuhan Univ
•Okayama Univ
•WuhanW h Univ
U i off Science
S i
andd Technology
T h l•Osaka Prefecture Univ
•Otaru Univ of Commerce[MOAT]•Tokyo Metropolitan Univ
•Beijing univ
•Chinese Academy of Sciences(NLPR- •Tokyo Denki Univ
U i off Sheffield
Sh ffi ld•Univ
IACAS)•Chinese Univ of Hong Kong + Hong Kong •Yokohama National Univ
Polythechnic Univ+ Tsinghua Univ
•DAEDALUS, S.A.

NTC7 OV 2008-12-17

Noriko Kando[PAT MIN]
Hiroshima City Univ
•Hiroshima•Hitachi, Ltd.,•Huafan Univ
•Nagaoka Univ of Technology•Northeastern Univ
•NTT Corporation•Peking Univ
•Shenyang Institute of AeronauticalEngineering•Toyohashi Univ of Technology•Univ
U i offC
California,lif
i B
Berkeleyk l•Univ of Montreal•Xerox[PAT MT]•Fudan Univ
•Harbin Institute of Technology +Heilongjiang Institute of Technology•Hitachi,Ltd.,p Patent Information Organizationg•Japan•Kyoto Univ
•Massachusetts Institue of Technology•Nara Institute of Science andTechnology + NTT
NICT
•NICT
•National Taiwan Normal Univ
•NTT Corporation•Pohang Univ of Science and Technology•TOSHIBA•Tottori Univ
•Toyohashi Univ of Technology + Hosei
University•Univ of Tsukuba

63

Types of Information Access
Exploratory Search

Look upL
Learn

Investigate

Machionini cacm 2006
Needs behind Queries
Human-Like Document Understanding
NTC7 OV 2008-12-17

Noriko Kando

64

Call ffor NTCIR-8 task p
proposalsp• Let’s workk together
h to construct a
better infrastructure to encourageginformation-access research to move
forward Resources constructed in past
forward.
NTCIRs are also available.
• Due to 30th November 2008
– Write to Noriko Kando
NTC7 OV 2008-12-17

Noriko Kando

65

NTC7 OV 2008-12-17

Noriko Kando

66

NTCIR-7 PC Meeting@NTCIR-6

Mark Sanderson,
Sanderson Doug Oard
Oard, Atsushi Fujii
Fujii, Tatsunori Mori
Mori,Fred Gey, Noriko Kando (and others)
NTC7 OV 2008-12-17

Noriko Kando

67

Acknowledgments• JapanJ
IIntellectualt ll t l P
Propertyt A
Association
i ti(JIPA)• Industrial Property Cooperation Center, Japan• Japan Parent Office• Japan Patent Information Organization (JAPIO)• Mainichi Newspaper• NRI Cyber Patents• PATOLIS• Task organizers• Participantspand test-collections’ users• Information Retrieval Facility
NTC7 OV 2008-12-17

Noriko Kando

68

Thanks
Merci
Danke schön
GracieGracias Ta! TackKöszönöm
KiitosT i
TerimaKasih
K ih
Kh KhunKhap
Kh
Ahsante
Tak
謝謝
ありがとう
http://research.nii.ac.jp/ntcir/
NTC7 OV 2008-12-17

Noriko Kando

69