I t d ti to
Introductiont NTCIRNTCIR-7
7

N k K
NorikoKandodNational Institute of Informatics, Japanh
http://research.nii.ac.jp/ntcir/
//
h ii
j /
i /
kando (at) nii. ac. Jp
NTC intro 2008-12-16

Noriko Kando

1

Road map
• What is NTCIR• Leason learned from past NTCIRs• Brief Introction to NTCIR-7• Conclusion

NTC intro 2008-12-16

Noriko Kando

2

NTCIR: NII Test Collection for Information Retrieval

Research Infrastructure for Evaluating IA

A series of evaluation workshops designed toenhancehresearch
h in
i information-accessi ftit h l i b
technologies
byproviding an infrastructure for large-scale evaluations.
■Data sets, evaluation methodologies, and forum

Project started in late 1997
Once every 18 months

Data sets (Test collections or TCs)
Scientific, news, patents, and web
Chin s K
r n J
p n s and
nd En
lish
Chinese,Korean,Japanese,
EnglishTasks

IR: Cross-lingual tasks, patents, web,
QA：Monolingual tasks, cross-lingual tasks
Summarization, trend info., patent maps
Opinion analysis, text mining

Community-based
Research
Activities
C
it b
d R
h A
ti iti
NTC intro 2008-12-16

Noriko Kando

3
NTCIR-7 participants

82 groups from 15 countries

NTCIR provides;
A scientific basis for understanding theeffectiveness of automated search systems

Document set, a setof topics, and a listof relevant documents
for each topic

Large-scale “Test Collections” or TC•Organizers provide a data set•Participants use the same data set tocompare the effectiveness of their
NTCIR enables:blsystems•TCs are available for research purpose * Cross-systemcomparison
comparson on aFForumoff researcher
h groups
commoninfrastructureShow-case of the State-of-the* SpeedsS
d up R&D
artt technologies
t h l iand technology
Investigations into evaluationtransfers
methodologies and metrics
NTC intro 2008-12-16

Noriko Kando

4

Information retrieval• Retrieve RELEVANT
i f
information
ti fromfvastt collection
ll ti
to meet users’ information
needs
Using
g computerspsince the 1950s
First CS uses human assessments
as success criteria
it i– Judgments vary
– Comparative
C mparative evaluations
evaluati ns onnthe same infrastructure
NTC intro 2008-12-16

Noriko Kando

5

Information access (IA)• Whole process ofpreparing information fromthe vast collection of documents usable byusers.
• For
F example,l IR,
IR ttextt summarization,
i ti
QA
QA,
text mining, and clustering• Use human assessments as success criteria

NTC intro 2008-12-16

Noriko Kando

6

Focus of NTCIR
Lab-type IR Test
Asian Languages/cross-language
Variety of Genre
Parallel/comparable Corpus

N
New
Challenges
Ch llIntersection of IR + NLP
To make information in the
documents more usable forusers!
Realistic eval/user task

Forum forResearchers
Idea Exchange
Discussion/Investigation on
Evaluation methods/metrics

History…
History
Project starts late 1997
Nov ’98 – Sep ’99
J ’00 – Mar
Jun
M ’01
Sep ’01 – Oct ’02
A ’03 – Jun
Apr
J ’04
Oct ’04 – Dec ’05

NTCIR-1
NTCIR 2NTCIR-2
NTCIR-3
NTCIR 4NTCIR-4
NTCIR-5

Apr ’06 – May ’07 NTCIR-6
O t ’07 – Dec
Oct
D ’08 NTCIR-7
NTCIR 7NTCIR-7 Workshop
Meeting Dec 16-19

Tasks Areas)at pastTasks (Researchof NTCIRs
NTCIR Workshops
pJapanese IR
Cross lingual IR
T Cross-linguala Patent Retrievalsmap/classifk Web
W b RetrievalRilsNavigational
Geo
Result Classification
Term Extraction
QuestionAnswering
Info Access Dialog
S
Summ
metricst i s
Cross-Lingual
Text Summarization

1st

2nd

3rd 4th

5th 6thnews

sci

Trend Information
Opinion Analysis
NTC intro 2008-12-16

Noriko Kando

9

NTCIR-7 Clusters

Cluster 2. User-Generated :- Multilingual Opinion Analysis
Cluster 3. Focused Domain : Patent- Patent
P t t TranslationT
sl ti ; English
E li h -> Japanese,J- Patent Mining paper -> IPC
Cluster 4. MuST :- Multi-modal Summarization of Trends
NTC intro 2008-12-16

Noriko Kando

MuuST; V
Visualizationn Challenge

Cluster 1. Advanced CLIA
- Complex CLQA (Chinese, Japanese, English)- IR for QA (Chinese, Japanese, English)

10Number of Participants by Tasks
ACLIA CCLQA

100

Opinion
CLQA
QA
Trend Info80

Summarization
mm

60

Term Extraction

40

Web Retrieval

ChinesePatent MT

Chinese Korean

20

JÆE,EÆJ、EÆCJÆEPatent MiningxÆCJEK

Patent Retrieval
NonJapanese IR

(2

00
7-

8)

7)
7t

h

(2
00

6-

)
-5
h
6t

h
5t

(2

00
4

)
-4
00
3

)
(2
h
4t

(2
0

01
-2

)
01
3r
d

00
(2

2n

d

(1
99

89

)

0

1s
t

# of Pa
articipattingGrou
ups

120

CLIR
Japanese IR

ACLIA
CL
IR4QA
R4Q

NTCIR-7 PC Meeting@NTCIR-6

Mark Sanderson,
Sanderson Doug Oard
Oard, Atsushi Fujii
Fujii, Tatsunori Mori
Mori,Fred Gey, Noriko Kando (and others)

NTCIR-7: Advanced CLIA
Teruko Mitamura (CMU)
Eric Nyberg (CMU)
Ruihua Chen (MSRA)Fred Gey (UCB),
Donghong Ji (Wuhan Univ)
Noriko Kando (NII)
Chin-Yew Lin (MSRA)
Chuan-Jie Lin (Nat Taiwan Ocean Univ)
Tsuneaki Kato (Tokyo Univ)
Tatsunori Mori (Yokohama N Univ)
Tetsuya Sakai (NewsWatch)
Ad i
Advisor:K L K k (Queen
K.L.Kwok(Q
C
College)ll)
CLEF2008 2008-09-18

Noriko kando

14

NTCIR-7: UGC (Blog)( g)

David K Evans (NII ->> Amazon Japan)
Yohei Seki (Toyohashi U Tech -> Columbia U)
LunWei Ku (National Taiwan Univ)
Le Sun ((Chinese Academy
y of Science))
Hsin-Hsi Chen (National Taiwan Univ)
Noriko Kando (NII)

CLEF2008 2008-09-18

Noriko kando

15

NTCIR-7: Focused Domain (Patent)()

Atsuhi Fujii
j (Univ Tsukuba)
Taiich Hashimoto (Tokyo Insti Tech)
Makoto Iwayama (Tokyo Insti Tech/ Hitach)
Hidetsugu Nanba (Hiroshima City Univ)
M
Masao
Utiyama
U i(NICT),(NICT)
Mikio Yamamoto, U Tsukuba)
T k hit Uts
Takehito
Utsuro (U Tsukuba)
Ts k b )CLEF2008 2008-09-18

Noriko kando

16

MuST: Multimodal Summarization for Trend
Information

Tsuneaki Kato (Tokyoy Univ)
Mitsunori Matsushita (NTTComm Sci Lab Æ Kansei Univ)

CLEF2008 2008-09-18

Noriko kando

17

[CCLQA]•Academia Sinica•Beijing
B iji Univ
U i off PostsP t &T l
Telecoms,
China•Carnegie Mellon Univ
•NICT
•NTT Corporation•Shenyang Institute of AeronauticalEngineering•Wuhan Univ
•Yokohama National Univ

•Hiroshima City Univ
•Information
Information and Communications Univ
•Chinese Academy of Sciences(ISCAS)•Keio Univ
•City Univ of Hong Kong
•National Taiwan Univ
•NEC
NEC
•Northeastern Univ
•Peking Univ
•Pohang Univ of Science and Technology•Swedish Institute of Computerp
Science•Technical Univ of Darmstadt•Graduate Univ for Advanced•Tornado Technologies Co., Ltd.,•Toyohashi Univ of Technology
n of Neuchatel•Univ•Univ of Sussex[IR4QA]•Carnegie Mellon Univ
•Chaoyang Univ of Technology•Chinese Academy of Sciences(ICT)
Harbin Institute of Technology +•Harbin
Heilongjiang Institute of Technology•National Taiwan Univ
[Must]•Open Text Corporation•Hiroshima City Univ
•Shenyang Institute of Aeronautical
Keio Univ•Keio
Engineering•Mie Univ•Toyohashi Unive of Technology•NICT•Univ of California, Berkeley•NEC•Univ of Montreal•Ochanomizu Univ (2 Groups)•Wuhan Univ
•Okayama Univ
•WuhanW h Univ
U i off Science
S i
andd Technology
T h l•Osaka Prefecture Univ
•Otaru Univ of Commerce[MOAT]•Tokyo Metropolitan Univ
•Beijing univ
•Chinese Academy of Sciences(NLPR- •Tokyo Denki Univ
U i off Sheffield
Sh ffi ld•Univ
IACAS)•Chinese Univ of Hong Kong + Hong Kong •Yokohama National Univ
Polythechnic Univ+ Tsinghua Univ
•DAEDALUS, S.A.

NTC intro 2008-12-16

Noriko Kando[PAT MIN]
Hiroshima City Univ
•Hiroshima•Hitachi, Ltd.,•Huafan Univ
•Nagaoka Univ of Technology•Northeastern Univ
•NTT Corporation•Peking Univ
•Shenyang Institute of AeronauticalEngineering•Toyohashi Univ of Technology•Univ
U i offC
California,lif
i B
Berkeleyk l•Univ of Montreal•Xerox[PAT MT]•Fudan Univ
•Harbin Institute of Technology +Heilongjiang Institute of Technology•Hitachi,Ltd.,p Patent Information Organizationg•Japan•Kyoto Univ
•Massachusetts Institue of Technology•Nara Institute of Science andTechnology + NTT
NICT
•NICT
•National Taiwan Normal Univ
•NTT Corporation•Pohang Univ of Science and Technology•TOSHIBA•Tottori Univ
•Toyohashi Univ of Technology + Hosei
University•Univ of Tsukuba

18

Oral presentation session

Poster session of past NTCIR Meeting

NTC intro 2008-12-16

Noriko Kando

20

Break out session

NTC intro 2008-12-16

Noriko Kando

21

Breakout session

NTC intro 2008-12-16

Noriko Kando

22

NTCIR-6 (2007) banquet

NTCIR Office members + friends

NTC intro 2008-12-16

Noriko Kando

24

Evaluation Workshops• "evaluation“
It is not an competition!

not an exam!

• Constructs a common data set usable forexperiments.
• provides to participants the data sets and unified
procedures for evaluation– Each participating research group conducts experiments
withh various approaches
h andd can participate withh own
purpose.

• Successful examples; TREC,TREC CLEF,
CLEF DUC,
DUC INEX,
INEX
and TAC, FIRE (new!) Community-based activities• Implications are various
ICCC2007 2007-10-13

Noriko kando

25

NTCIR: NII Test Collection for Information Retrieval
publications
bli
iTask participants
MOU permission

RunsubmitConferences,jjournals, etc.
Reports, papers
publications

evaluate

Data
Providers

MOU

MOUR
documents

Relevance
Assessment(correct answers)

Topics/questions

Research purpose
use

permissioni iReport,bibliographies

Test collections

Experimental results =runs

NII
ICCC2007 2007-10-13

Noriko kando

26

Sample document:<DOC><DOCNO>ctg_xxx_19990110_0001</DOCNO><LANG>EN</LANG><HEADLINE> Asia Urged to Move Faster in Shoring Up Shaky Banks </HEADLINE><DATE>1999-01-10</DATE><TEXT><P>HONG KONG, Jan 10 (AFP) - Bank for International Settlements (BIS) general manager
Andrew Crockett has urged Asian economies to move faster in reforming their shaky banking
sectors, reports said Sunday. Speaking ahead of Monday's meeting at the BIS office here ofinternational central bankers including US Federal Reserve chairman Alan Greenspan,
Crockett said he was encouraged by regional banking reforms but "there is still some way togo " Asian banks shake off their burden of bad debt if they were to be able to finance recovery
go.
in the crisis-hit region, he said according to the Sunday Morning Post. Crockett added that
more stable currency exchange rates and lower interest rates had paved the way for recovery.
"Therefore I believe in the financial area, the crisis has in a sense been contained and that
now it is possible to look forward to real economic recovery," he was quoted as saying by the
Sunday Hong Kong Standard.</P><P>"It would not surprise me, given the interest I know certain governors have, if the subjectof hedge funds was discussed during the meeting
meeting," Crockett said
said. </P>
<P>He reiterated comments by BIS officials here that the central bankers would stay tightlipped about their meeting, the first to be held at the Hong Kong office of the Swiss-based
institution since it opened last July. </P></TEXT></DOC>
ICCC2007 2007-10-13

Noriko kando

27

Sample topic:<TOPIC>
TOPIC
written statement of user’s needs<NUM>013</NUM><SLANG>CH</SLANG><TLANG>EN</TLANG><TITLE>NBA labor dispute</TITLE><DESC>To retrieve the labor disputepbetween the two p
parties of the
US National Basketball Association at the end of 1998 and theagreement that they reached. </DESC><NARR> The content of the related documents should include thecauses of NBA labor dispute, the relations between the players andthe management, main
controversial issues of both sides, compromises after negotiationand content of the new agreement, etc. The document will be
regarded as irrelevant if it only touched upon the influences ofclosing the court on each game of the season. </NARR>
fieldsBasketball
are usable
for retrieval.
run using<CONC> NBAAny(National
Association),
union,A
team,
DESC onlyis andmandatory
mandatory.
league, labor dispute,leagueunion, negotiation, to sign anagreement, salary, lockout, Stern, Bird Regulation. </CONC>
ICCC2007 2007-10-13
Noriko kando
28</TOPIC>Relevance Judgmentsg• Always Multigrades in NTCIR: 3 or 4 grades– [Highly Relvant (S)]– Relevant(A),– Partial Relevant(B),Relevant(B)– Irrelevant(C)• Traditionally
T ditill “bi“binary”” jjudgmentsdt are popular.
l• Contains extracted phrases/passages showing
the reason that the analyst judged it as“relevant” in NTCIR collections.

ICCC2007 2007-10-13

Noriko kando

29

IA Systems Evaluation• Engineering Level:
Level Efficiency• Input Level: ex. Exhaustivity, quality, novelty of DB• ProcessP
Level:
Ll Effectiveness
Eff tiex. recall,precisionlli i• Output Level: Display of output• User Level: ex. Effort that users need
• Social L
Level: ex.. Importance
mp r n

ICCC2007 2007-10-13

Noriko kando((Cleverdon
r n&K
Keen
n 1966)966)

30

Evaluation of IA Effectiveness•Long
L
ttradition
diti off llaboratory-typedbtt
d testingt ti using
i ttestt
collection since Cranfield in 1960s.
•Basic metrics are;
Precision =Recall =… and their variants# of retrieved-relevant
retrieved relevant# of retrieved# of retrieved-relevant#of all relevant docs

“Recall” can be calculated only in the experimental
setting in which all the relevant docs are known.
known
ICCC2007 2007-10-13

Noriko kando

31

Retrieval Difficulty Varies with Topics
1.0000J-J Le ve l1 D auto
Effectiveness
Across SYSTEMS

0.8000Effectiveness Across TOPICS
101
検索システム別の11pt再現率精度
on a System
102
1

Average over
50 topics

103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

A
B
C

0.8

0.6000D

precision

E

precision

F

0.4000

G

0.6

H
I
J

0.4

K
L

0.2000

M
N

0.2

O

0.0000
0

2

4

6

8

0

0.

0.

0.

0.

0.

1.

P

0
0

0.1

recall

ICCC2007 2007-10-13

Noriko kando

0.2

0.3

0.4

0.5
recall

0.6

0.7

0.8

0.9

1

32

Retrieval Difficulty Varies with Topics
J JL
J-J
Le ve l1 D autot

1.0000Effectiveness
Across SYSTEMS

検索システム別の11pt再現率精度
Effectiveness
Across TOPICS 101
102
103
on a System
104“Difficult
Topics”
Vary with
Averageover
50 topics
Systems
10.8000A
B

J-J
J
J Leve l1 D auto

0.8

Mea
an Ave P
Precisionnecisionpre0.60001.0000

A

D

B

E

C

precisioonF

0 80000.80000.4000

D

G

06
0.6

E

H

0.60000.4

0.40000.2000

I

F

J

G

K

H

L

I

M

J

N

0.2

0.20000.0000

O

K

P

L
M

7
1 10 .2011
3
0
1 1 .4611
9
1 20 .
26
12
5
1 20
8.8
13
1
1 31
4.0
13
7
14
0
14
3
14
6
14
9

0

0

10

1

0
1 .

04 0

0.000010

preecisionC

0.1

T opic#Requests#101 150#101-150

recall

ICCC2007 2007-10-13

Noriko kando

0.2

0.3

For0.5N reliable
0.6
0.7
0.8 and0.9
1
recallO
stableevaluation,Pusing substantial #topics is inevitable.
33

0.4

105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

TC usable to evaluate?

Pharmaceutical R & D
Phase I:

Phase II:

Phase III:In Vitro

AnimalExperiments

Clinical Test
Test with
Healthyy Human
Subject

ICCC2007 2007-10-13

Noriko kando

Phase IV:

34

NTCIRTC usable to evaluate what?

Test Collections

Phase I:
Laboratorytype Testing

Phase II:

Levels of
Evaluation

Phase III:

Phase IV:

Phase III:

Phase IV:

Sharing Modules , Controlled
Uncontrolled
PrototypeInteractive
Pre operational
Pre-operational
testing
Testing using
Testing
human Subjects

Pharmeceutical R
Phase II:
&PhaseD I:
In Vitro

Users’ information seeingtasks

AnimalExperiments

Clinical Test
Test with
Healthyy Human
2.Input Level、Subject
6.Social Level

4.User Level、5.Output Levle

1.Engineering Level efficiency
3.Process Level: effectiveness
ICCC2007 2007-10-13

Noriko kando

35

Summary of “What
What is NTCIR”• Providing a scientific basis for understanding
the effectiveness of automated searchsystems• Leveraging the R&D and technology transfer• ReusableR
bl T
Test collectionll
is a key
k component• Evaluatingg search effectiveness is not easy.
yA small-scale or carelessly-designed TCs may
skew the test results

NTC intro 2008-12-16

Noriko Kando

36

Road map
• What is NTCIR• Leason learned from past NTCIRs• Brief Introction to NTCIR-7• Conclusion

NTC intro 2008-12-16

Noriko Kando

37

Lessons Learned from Past
NTCIRsICCC2007 2007-10-13

Noriko kando

38

Information Retrieval
Ad hoc1. Ad hoc & CLIR•••

Scientific Abstracts (NTCIR-1 & -2)
News (NTCIR-3 thurough -7)
Blogs (NTCIR-7)•

Patent (NTCIR-3 through -6)
Æ Translation, Miningg (NTCIR-7)()
WEB (NTCIR-3 through -5)

2 IR in2.
i specificifi Document
Dt G
Genres•ICCC2007 2007-10-13

Noriko kando

39

CLIR at Asian Environment
1.Initial Stage : English & Own language“internationalize” = provide info in English
2 Long (2000 years?) historical relationship,2.
relationshipbut less interaction in 1950-early 1990s
3. Interest increasing rapidlyCommercial/industrial exchange increased:
Cultural/Social interest, Human Exchange,
4. Languages structures are completely
different Character codes are different
ICCC2007 2007-10-13

Noriko kando

40

NTCIR -1 &-2: Japanesep& Englishg
Documents

TopicsMonolingual;J J E E
Monolingual;J-J,E-E

J
J

Japanese
CLIR;J-E, E-J

E
EEnglishMixed CLIR; J-JE,E-JE
J JE E JE
ICCC2007 2007-10-13

J

Noriko kando

J

J Collection

J
J
E

E Collection

E

Sci Abst
Sci.

J

J E
E J

J

J

E

J

SingleLanguageg g

J Collection+

E Collection
E No paired docs

Mixed
Language 41

NTCIR-3 throu -7 CLIR
50 topics ｘ 3 sets
Chi
Chineset d
trad

Documents
J E C
K
CJ J
J
Published
u
in
n
K K
E
C E E C 2000-2001
K
C
J C E
Published inK
1998-1999

KoreanEnglishJapanese- Forcus: NE, OOV- proper nouns vs without PN- domestic/regional/international
d
/
l/
l

ICCC2007 2007-10-13

Noriko kando

42

CLIR: Lessons Learned
•
•
•
•
•

IR Models: majorj IR models were worked
Indexing: bigram vs word vs others, hybrid
Mostly “Query Trans”, but a few “Doc Trans”
Translation disambiguation w/ WEB w/target doc
Out-of-vocabulary (OOV) problem– TransliterationTlii- CognateC– NE identification
– Use
U of
f Web
W b
• Query expansion techniques– Selective application PRF
PRF, Bounce & Throw– Clustering

ICCC2007 2007-10-13

Noriko kando

43Patent Retrieval Tasks
situation & users’’ informationfseekingktaskkNewspaper

PatentP
t tApplications

PatentClaims

5 yrs, 45GB:
10 yrs 90GBNNTCIR-3
R PATENT
P EN
NTCIR-4,-5 PATENT(2001-2002)(2003-2004)(2004-2005)
From a claim of a new
T h l i l Survey:
TechnologicalS
Search patents by newspaper patent application, searchpatents that can
End user: non-experts (ex.
(exinvalidate the new patent
Business manager)application.
User: patent experts
ICCC2007 2007-10-13

Noriko kando

44

NTCIR 3 Patent Collections
NTCIR-3
TOPICS (31)Japanese
E li hEnglishChinesetrad

DOCUMEN
TS(1998,99)
Full text withauthor’s abstract(in Japanese)

Chinesesym

18G bytes

With hierachicalclassification
codes
d

p

KoreanNewspaper
Clips

Searchpatents bynewspaper clipl
ICCC2007 2007-10-13

By professional abstractors(1995-99)((1995-99))
Translation
Ab
Abstract
Abstract(in English)(in Japanese)
4 GB 1.7
1 7 million docs
1.7
docs.
1 7 million docs
docs.
1995-97 are usable for translation
Noriko kando

45

NTCIR-4 thro -6 Patent (20042007)
TOPICS(34 manual +More than 1000)
Japanese
English

DOCUMENTS Search patents by patentCa.7 M docs
- text retrieval + relevant

Ca. 90GB(1993-2002)
Full text withauthor’s abstract(in Japanese)passage pinpointing

Passage Retrieval
F-term Classification(1993-2002)
F ll ttextt withFullithauthor’s abstract
By professional(in Engoish)
abstractors

Patents(claims)

Translation
ICCC2007 2007-10-13

US Patent(1993 2002)(1993-2002)
Abstract(in English)
7 million docs.
docs
5 GB
Noriko kando

46

automatic patent map generation
Egiven l (blue
Example(bl light-emitting
li ht
itti didiode)
d )problems to be solved

crystalline

reliability

sollutionss

structure of active
layeremissionstability

emission
intensity

1998-190063
1998-209498

1998-209495

1998-173230
1998-209499
1998-256602

1998-242515
1998-270757

1998-145000
1998-233554

electrodecomposition

1998-107318

electrodearrangement

1998-215034
1998-223930

structure of lightemitting element

long
operatinglife

1998-135516
1998-242586
1998-247761

1998-242518

1998-135514
1998-256668

1998-012923
1998-247745
1998-256597

participants identify lines and columns
ICCC2007 2007-10-13

Noriko kando

47

Patent full text vs. abstracts vs. Claims
PATENT: <DESCRIPTION>
03
0.3

avera
age precision
n

0.25
0.2

full
abs

0.15

claim
abs+claim
abs
claim

0.1

jsh

0.05

ICCC2007 2007-10-13

25
B
M

f+
dl
tf)
.id

tf)
.id
f

tf)

f
id
tf.

lo
g(

lo
g(

lo
g(

*abs=author abstracts,
jsh=professional abstracts

id
f

tf

lin
e
ba
se

hi
ts

0

Search on patent fulltext
using sophisticated IRmodels worked better thanany other conditions
Noriko kando
48

Retrieval modelprecission

Results (exact match)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
02
0.2
0.1
0

NCS02 (0.4852)
GATE03 (0.4779)
NICT01 (0.4518)
JSPAT01 (0.4381)
NUT05 (0.4101)
RDNDC14 (0.2717)
baseline (0.2821)0

0 1 0.2
0.1
0 2 0.30 3 0.40 4 0.50 5 0.60 6 0.70 7 0.80 8 0.909

1

recall

ICCC2007 2007-10-13

Noriko kando

49Hybrid classifier/Naïve
Results (exact
match) Bayes
Different Classifier for eachc mp n nt in thcomponent
the fulltext
fullt xt

precis
sion

1
0.9
08
0.8

H-SVM forF llt tFulltext

0.7
0.6
05
0.5
0.4
0.3

NCS02 (0(0.4852)4852)
GATE03 (0.4779)
NICT01 (0.4518)

K-NN
K
NN forf
Abstracts &Claims

0.2
0
2
0.1
0
0

SVM
VM forf
Fulltext
recall

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

ICCC2007 2007-10-13

Noriko kando

JSPAT01 (0(0.4381)4381)
NUT05 (0.4101)
RDNDC14 (0.2717)
baseline (0(0.2821)2821)
1
Abstract
&
Claim

50

WEB Retrieval(A) Informational Retrieval Task
(B) Navigational Retrieval Task•

“Known Item Search”. representative pages(C) GeographicalG
hi l Task
T k(D) Topical Classification Task
retrieval result classification
classification, eg
eg.using
using clustering
Documents:– ‘NW100G
NW100G-01
01’ (100GB Web pages crawled in 2001
from “*.jp”)– ‘NW1000G-04’
NW1000G-04 (1.36TB(1 36TB Web pages crawled in
2004 from “*.jp”)

ICCC2007 2007-10-13

Noriko kando

51

Question Answering
1 QA Challenge on a language
1.
• Information Access Dialogues (NTCIR-3,-4, -5)• Natural ((Real)eal) Qs (no answer type llimitation)m tat on)(NTCIR-6)
2. CLQA• Factoid
F t id (NE) (NTCIR(NTCIR-5,-6)
5 6)• Complex Questions (NTCIR-7)

ICCC2007 2007-10-13

Noriko kando

52Series of Question
Situation Settings (User’s Task)
1. Collecting information about a particular
topicp– One (hidden) global topic and series of Qs
on subtopics of the global topic
2. Browsing along transitive interests– Topic
T pi or focus
f
s of
f the
th Qs are shifting
shiftinthrough the interaction of the user ands st msystem.
– Local coherence with the previous Q only
ICCC2007 2007-10-13

Noriko kando

53

Example of Series of Questions• What
Wh t genre
n ddoess ththe "Harry"H
P
Potter"tt " sseries
i sbelong to?
• Who is the author?
• Who are the main characters in that series?
• When was the first volume published?
• What title does it have?
• How many volumes were published by 2001?
• How many languages has it been translated
into?
• How many copies have been sold in Japan?
Series 02: Gathering TypeICCC2007 2007-10-13

Noriko kando

54

Example of Series of Questions• WhereWh
was Wuhan
W h University?
U i
it ?
• Which train station is the nearest?
• Who is the actor who visited the university?
• What is the movie he was featured in thatwas released in the New Year season of 2001?
g Kevin Costner• What is the movie starring
released in the same season?
jmatter of that movie?
• What was the subject• What role did Costner play in that movie?
Series 24: Browsing TypeICCC2007 2007-10-13

Noriko kando

55

QA: Lessons Learned
Tested for Simulated Interaction
anaphora resolution, context
inf gathering >> browsing,browsing but improved
Return one set of All the answer:
Context
Answer Granularity
Level of requiredness : Answer Score
Answer Set
Complex Questions like asking definition, who, how, etc.
More needed to investigate for automatic evaluation
ICCC2007 2007-10-13

Noriko kando

56

Complex QA Evaluation criterion• HumanHevaluationlmeasure– Level A: System answer has almost the samecontents as one off theh correct answers.
– Level B: System answer includes the contents of
one off theth correctt answers.
– Level C: System answer includes some part (not allone) of the contents of the correct answers.
answers– Level D: System answer includes no information of
any of the contents of the correct answers.
answers

ICCC2007 2007-10-13

Noriko kando

57

CLQA : Lessons Learned• Factoid (esp. NE) QA can be a fundamental
module for further CLIA especially among thelanguages with different scripts• Major source of the performance drop waspoor retrieval modules in QA systems. Î
need collabolation with IR groups• OOV

ICCC2007 2007-10-13

Noriko kando

58

Text Summarization
1 Text Summarizastion Challenge on a language1.
• Single document (NTCIR-2,-3)• Multidocument (NTCIR-3, -4)
2. Summarization-based metrics used in QA(NTCIR-6, -7)

ICCC2007 2007-10-13

Noriko kando

59

Text Summarization Challenge
- Two
T
types
t
of
f summarizationi ti -• Extraction

Two lengths:
short longshort,– Extracting importantsentences from document sets
length: # of sentences
A t
Automaticti• Abstraction– Producing summaries fromdocument sets
length: # of characters

ICCC2007 2007-10-13

Noriko kando

Extract
Evaluation Æ
Reusable
Summarization
Test Collection
See. Hirao(COLING
2004)

60

Opinionp
AnalysisyRoadmap
Genre
SubjectivityHolder
News
NTCIR-6
NTCIR-6
Blog
NTCIR-7
NTCIR-7C
Cross-genre
NTCIR 8NTCIR-8
NTCIR 8NTCIR-8
StakeholderTemporal
NTCIR-7
NTCIR-8

NTCIR-8

ICCC2007 2007-10-13

Polarity
NTCIR-6
NTCIR-7
NTCIR 8NTCIR-8

Strength

NTCIR 8NTCIR-8

LanguageGranuality Application
C,J,E
single-sentlSummarization
C,J,E
clauseQA
multi-sent Opinion tracking
CJE
document Consistency checking
Trend

Noriko kando

61

Corpus-Centered Evaluation and
C ll bCollaborative
i R
Researchh

1. TMREC: Term Recognition(NTCIR-1)
2. MuST: Multimodal
Summarization for Trend
Information (NTCIR-5, -6)

ICCC2007 2007-10-13

Noriko kando

62

Multimodal summarization
for Trend Information
Q
Queriesi on trendstd“How the price of gasoline shifted during the year?”“What
What the situation has been in the PC market?
market?”“How terrible the typhoons were last autumn?”

Concise,, plainp
text
Information graphics
Multimedia presentationp
text including references to graphics
graphicsgpannotated with text
ICCC2007 2007-10-13

Noriko kando

63

The Roles of Data Set

Information Collected

Articles, Tables and Charts
MultimodalMl i d lSummarization
Annotations

Visualization
softwareft

Summaries,, Reportsp
Textual summaries, Charts and Tables

ICCC2007 2007-10-13

Noriko kando

64

Road map
• What is NTCIR• Leason learned from past NTCIRs• Brief Introction to NTCIR-7• Conclusion

NTC intro 2008-12-16

Noriko Kando

65

NTCIR-7: Advanced CLIA
Question
Analyzers

CCLQAXML,AP
I Questions

Translation& Retrieval

Q with qtypes

IR for QA

AnswerExtraction &Formatting

Retrieved
documents

CLIR

-Test
T t effectiveness
ff ti
of
f
OOV, PRF, QE in QA

Answers

Eval. By
- IR Effectiveness- QA Effectiveness- Focused Retrieval
ICCC2007 2007-10-13

Noriko kando

66

ACLIA: Test Collection
Language Corpus Name
Time
Span
CS
Xinhua1998-2001
CS
Li h Zaobao
LianheZ b
1998 2001
1998-2001
CT
CIRB020 & CIRB040 1998-2001
JA
Mainichi Shinbun
1998-2001

100 Topics in CS, CT, JA and their English translation
DEFINITION, EVENT, BIO, RELATION
CLQA and IR4QA used the same topics<TITLE> <Q-TYPE> not released<QUEstion><NARRATIVE> released
CLEF2008 2008-09-18

Noriko kando

67

ACLIA: Evaluation

CLEF2008 2008-09-18

Noriko kando

68

ACLIA: Evaluation EPAN tool

CLEF2008 2008-09-18

Noriko kando

69

ACLIA: Evaluation EPAN tool
CCLQA:
Nugget Pyramid

IR4QA:MAP
MS nDCG
Q-Measure((preferencef
based )
CLEF2008 2008-09-18

Noriko kando

70

NTCIR-7: UGC (Blog)( g)

David K Evans (NII ->> Amazon Japan)
Yohei Seki (Toyohashi U Tech -> Columbia U)
LunWei Ku (National Taiwan Univ)
Le Sun ((Chinese Academy
y of Science))
Hsin-Hsi Chen (National Taiwan Univ)
Noriko Kando (NII)

CLEF2008 2008-09-18

Noriko kando

71

Opinion Analysis - Roadmap
Genre
News
Review
Blog

Subjectivity
Holder Polarity
NTCIR-6
NTCIR-6 NTCIR-6
NTCIR-7
NTCIR-7 NTCIR-7
NTCIR-8
NTCIR-8 NTCIR-8

Strength
NTCIR-7
NTCIR-8

Stakeholder
Temporal Language
GranualityApplication
Chinese single
single-sentSummarization
sentSummarization
NTCIR-7
English clause
QA
NTCIR-8
NTCIR-8
Japanese multi-sent Opinion tracking
CJE
document Consistency checkin
Trend

Chinese, Japanese,
EnglishCLEF2008 2008-09-18

Noriko kando

72

NTCIR-7: UGC (Blog)( g)•Documents:

Crawledraw
Blogog posts + Comments
omm nts (Ju
(July
y–S
Sept,
pt, 2007.
7. 6
weeks) CCKEJ•CLIR on Blog (CLIRB) C,C,J,E,(K?) any other q?
• Informational Search Task for Opinion-Focusedsearch requests• 50 topics,i
4 graded relevancel
jjudgmentsd•Multilingual Opinion Analysis (MOAT) TraditionalC,J,E• selectings l ti relevantlt documentsdm ts fromf m ~30
30 topicst i s useds d
in CLIRB.
• Following Roadmap
Roadmap, but change the genre• Relevant, Opinionated, Polarity (Pos, Neg, Nue),
Holder,, Stakeholder (Object),( j), ??Strength??
g
CLEF2008 2008-09-18

Noriko kando

73

NTCIR-7: MOAT ((on News))•Documents:

NEWS CCEJ•CLIR on Blog (CLIRB) Cancelled•Multilingual Opinion Analysis (MOAT)
TraditionalC,Simplifed
Simplifed C,
C J,EJE•TraditionalC• selecting relevant documents from ~25 topics usedin ACLIA• Following Roadmap, but change the genre• Relevant, Opinionated,p
Polarityy (Pos, Neg,g Nue),
Holder, Stakeholder (Object), ??Strength??

CLEF2008 2008-09-18

Noriko kando

74

MOAT Participants

Beijing university of posts andtelecomunications
Chinese Academy of
Sciences(NLPR-IACAS)n(NL)
City University of Hong Kong
CUHK（The Chinese University ofHongg Kong)-PolyU(The
g)
y (
Hong
g Kongg
Polythechnic University)Tsinghua(Tsinghua University)
DAEDALUS, S.A.
Dalian University of Technology
Hiroshima City University
Information and Communications
U i
Universityi
Keio University
Louisiana State
U i
University(University
it (U iit offM
Marylandl dCollege Park)

National Taiwan University
NEC
NEU Natural Languageg gProcessing Lab
Peking University
Peking University(ICL)
Pohang University of Science andTechnology
SICS - Swedish Institute ofC
Computer
Science
S i
Technical University ofDarmstadt
Th G
The
Graduated t U
University
i
sit f
for
Advanced Studies(SOKENDAI).
Tornado Technologies Co., Ltd.,
TaiwanTaiwan.
Toyohashi University ofTechnology
University of Neuchatel
University of Sussex
Yuan Ze Univ.

80+ registerd, 30+ resigned when docs were changed, 42
registered to News MOAT, 24 sugmitted
CLEF2008 2008-09-18

Noriko kando

75

NTCIR-7: Focused Domain (Patent)()

Atsuhi Fujii
j (Univ Tsukuba)
Taiich Hashimoto (Tokyo Insti Tech)
Makoto Iwayama (Tokyo Insti Tech/ Hitach)
Hidetsugu Nanba (Hiroshima City Univ)
M
Masao
Utiyama
U i(NICT),(NICT)
Mikio Yamamoto, U Tsukuba)
T k hit Uts
Takehito
Utsuro (U Tsukuba)
Ts k b )CLEF2008 2008-09-18

Noriko kando

76

NTCIR-7: Focused Domain (Patent)()

Documents:
10 Yrs Japanese Patent Application (NTCIR4-5)
10 Yrs USTPO Patents (NTCIR6)
Parallel Sentence Data (1.8 M sentences JE Pairs)
S i
Scientificifi P
Paper Ab
Abstracts (NTCIR 11-2)
2)
Patent Translation (PATMT) MT is key for CLIR

Training: 1993-2000,
1993 2000 Test: 2001
2001-2002
2002
One Ref Trans good??
Intrinsic Eval. ;BLEU, human assessments
Extrinsic Eval: CLIR task-based

Patent Mining (PATMN)
P(P) Cross-GenreG
PATP& ScientificfClassify Paper Abstracts in to IPC Classes
ML approach:h Classsify
Cl
if Absts
Ab t tot IPC ClassClIR Apprach: use invalidity search system to findrelevant Patent,
Patent then assign IPCs to Paper Absts
Absts.
CLEF2008 2008-09-18

Noriko kando

77Patent classification and mining at
NTCIR
Organizers:
Makotok
Iwayama (Hitachi(h Ltd/Tokyo/ k Institute off Technology)h l)Hidetsugu Nanba (Hiroshima City University)
Taiichi Hashimoto (Tokyo Institute of Technology)
Atsushi Fujii (University of Tsukuba)
Noriko Kando (National Institute of Informatics)
NTC intro 2008-12-16

Noriko Kando

78

Goal: Automatic g
generation of patent
p
maps.
p
Example: Blue light-emitting diodes

Given

Problems to be solved
Crystalline

Reliability

S
Solutionns

Structure of active
layer

Long
operatinglife

Emissionstability

Emission
intensity

1998-145000
1998-233554

Electrodecomposition

1998-107318

1998-190063
1998-209498

1998-209495

ElectrodeElt darrangement

1998-215034
1998-223930

1998-173230
1998
1732301998-209499
1998-256602

1998-242515
1998-270757

StructureSttoff lilightht
emitting element

1998-135516
1998
135516
1998-242586
1998-247761

1998-242518

1998-135514
1998-256668

1998-012923
1998
012923
1998-247745
1998-256597

Systems automatically identify rows and columns
NTC intro 2008-12-16

Noriko Kando

79

Historyy• NTCIR-4 (2003-2004): Patent-map-creation subtask– Direct approach to creation of patent maps
– Hard tasks and insufficient evaluation• NTCIR-5
NTCIR 5 (2004-2005):(2004 2005): Classification subtask– Categorize patents to pre-defined categories called Fterms (multi-faceted(multi faceted and structured)– Relatively small number of test documents– Evaluate only strict matches in F-term
F term hierarchy• NTCIR-6 (2006-2007): Classification subtask– Increased the number of documents and topics (108 topics)– Evaluate partial matches in F-term hierarchy• NTCIRNTCIR-7
7 (2007(2007-2008):
2008) Mining subtask
NTC intro 2008-12-16

Noriko Kando

80

Feasibility
y Study:y automatic patentpmap
pgeneration at NTCIR-4 (2003-2004)
searchtopic
iapplication

retrievalt ildocuments

JAPIO abst
PAJ

topics and documents
in NTCIR-3
NTCIR 3 collection
Patent-map
Patentmap creation =Multi-faceted
patent clustering
NTC intro 2008-12-16classificationclass
f cat onvisualization
Noriko Kando

multi-dimensional
matrix81

Classification task overview
Training data

PMGS
(F-term descriptions)

Training

Patents withththemesandd F-terms
F t(1993-1997)

Theme is g
given
5B001
5B001
5B001

Sampling

Test data
Patents withthemeshandd F-terms
F(1998-1999)

Classifier
F-termF
term
assignment
5B001
5B001
5B001
AC04

NTC intro 2008-12-16

Noriko Kando

Evaluation82

Patent mining at NTCIR-7
NTCIR 7 (2007-2008)(2007 2008)
Searches and/or classifying patents and scientific papers into IPC
Research paper written in
Japanese (Japanese / J2E
subtasks))
A Parrticipant System

Machine-translation
module (E2J / J2E)

Text classification module

Research paper written in
English (English / E2J
subtasks))

Japanese, English, and Crosslingualg(J-to-E, E-to-J) subtasks
Patent data
- written
itt ini JapaneseJ(Japanese / J2E)
- written in English(English / E2J)

List of IPC codes
NTCFujii,
intro 2008-12-16
83
Nanba,
Iwayama, and Hashimoto.Noriko“TheKando
Patent Mining Task in the Seventh NTCIR
Workshop”, Patent Information Retrieval Workshop at CIKM 2008 (2008)

Summary of patent classification and mining• Automatic clustering of patents into
“problems”
problems and “solutions”
solutions are quite feasible,
but labeling and controlled evaluation needmore investigation.
• Granularity of F-term is appropriate forpatent map creation and becoming good.
good• Patent minting of scientific papers andp t nts are practicallypatentsp ctic ll needed.
n d d nn-KNN
KNN andndmachine learning have promise– The test collections for classification are available
for research purpose. The one for mining will beavailable to the public after Workshop Meeting•NTC intro 2008-12-16

Noriko Kando

84

Patent machine translation at NTCIR
Organizers:
Atsushih Fujii (University(off Tsukuba)k)Masao Utiyama (NICT)
Mikio Yamamoto (University of Tsukuba)
Takehito Utsuro (University of Tsukuba)

intro 2008-12-16
Fujii,NTC
Utiyama,
Yamamoto, and Utsuro. “Toward the Evaluation of Machine Translation
Noriko Kando
85
Using Patent Information”, AMTA 2008

Patent machine translations at
NTCIR-7 (2007-2008)• Patent
P t nt Machine
M chin Tr
Translation
nsl ti n (MT) is rrealistic
listic– Parallel corpora can potentially be produced from
JPO/USPTO patentpatent-document
document sets
– Decoders for statistical MT (SMT) are available• Two types of players– Organizer = Authors of this paper• Providing data, and evaluating participating MT systems– Participants = Researchh groups• They can use e.g., SMT and rule-based MT.

• Utility of patent MT– Cross-lingual patent retrieval– Filing patent applications in foreign countries

NTC intro 2008-12-16

Noriko Kando

86

Producing parallel corpora
JPO applicationsJ
ppn
1993-2002
Comparable(3.5-M docs)(not parallel)

J

E

J

E

Sentence-alignmentlmethodh d[Utiyama and Isahara, 2007]
TargetingTi“background” and“description”
description
NTC intro 2008-12-16

U
USPTO
grantsgr n
1993-2002(1.3-M docs)

J

E

Patent family
Patent set forsame invention

Sentence pairs
Parallel (alignment accuracy= 90%)
Noriko Kando

87

Extrinsic
evaluation
NTCIR-5NTCIR
5Patent claim

SearchS
rch topict pic
in Japanese

Human

SearchS
rch topic
t pic
in English

Evaluation by BLEU
Training data
1.8-M sentencepairs• System training• ParameterPt tuningt i

MT system

JPO applications
1993-2002

Invalidate patent
Translation in
Japanese

Evaluation by Mean Average
Precision (MAP)
NTC intro 2008-12-16

Performed byorganizers

Noriko Kando

IR system
Ranked doc.
list
88

Patent machine translation• Constructed a large test collection for J/E MT: USTPO
JPO with 10 yyears off fullf textsand J• Large-scale sentence-alignment dataset (E-J sentence pairs)• Statistical MT (SMT)* vs. rule-based MT• Results demonstrated:– SMT is much better for CLIR
– Rule-based
R l bd MT iis good
d for
f h
human evaluationsl ti– Human evaluations and creation of referencetranslations must be carefully done (in the real world,
world
professional patent translators do use MT).
p p
• Test collection will be available for research purpose
after the workshop meeting*SMT : a system automatically learns the translation rules fromtheh given large-scalell sentence pairs.
NTC intro 2008-12-16

Noriko Kando

89

Multimodal summarization
for Trend Information
QueriesQ
i on ttrendsd“How the price of gasoline shifted during the year?”“What
What the situation has been in the PC market?
market?”“How terrible the typhoons were last autumn?”

Concise,Ciplain
l i textt t
Information graphics
Multimedia presentationtext including references to graphics
graphicsgpannotated with text

Visualization Platform

NTC intro 2008-12-16

Noriko Kando

90

NTCIR-7 Workshop Meeting
December 16-19,
16-19 2008 @ Tokyo

http://research.nii.ac.jp/ntcir/ntcir-ws7/meeting/

Past data: http://research.nii.ac.jp/ntcir/data/data-en.html
Proceedings: http://research.nii.ac.jp/ntcir/publication1-en.html
h
//
h
/
/
l
h l
NTC intro 2008-12-16

Noriko Kando

91Types of Information Access
Exploratory Search

Look up

NTC intro 2008-12-16

L
Learn

Noriko Kando

Investigate

92
Machionini cacm 2006

Call ffor NTCIR-8 task p
proposalsp• Let’s workk together
h to construct a
better infrastructure to encourageginformation-access research to move
forward Resources constructed in past
forward.
NTCIRs are also available.
• Due to 30th November 2008
– Write to Noriko Kando
NTC intro 2008-12-16

Noriko Kando

93

Acknowledgments• JapanJ
IIntellectualt ll t l P
Propertyt A
Association
i ti(JIPA)• Industrial Property Cooperation Center, Japan• Japan Parent Office• Japan Patent Information Organization (JAPIO)• Mainichi Newspaper• NRI Cyber Patents• PATOLIS• Task organizers• Participantspand test-collections’ users• Information Retrieval Facility
NTC intro 2008-12-16

Noriko Kando

94

Thanks
Merci
Danke schön
GracieGracias Ta! TackKöszönöm
KiitosT i
TerimaKasih
K ih
Kh KhunKhap
Kh
Ahsante
Tak
謝謝
ありがとう
http://research.nii.ac.jp/ntcir/
NTC intro 2008-12-16

Noriko Kando

95