NTCIR-7 MOAT
Overview
Yohei Seki, Lun-Wei Ku, David Kirk Evans, Le Sun

1

Opinion Analysis• Given a sentence:• Does it contain any opinions?
• What are the opinions• Polarity? (Positive, Negative, Neutral)• Who expresses the opinion?
• What is the opinion target?
• Is the sentence relevant to the topic?
2

Opinion Analysis
Japan should carefully consider visiting the Yasukuni
Shrine and abide by the solemn statement it has madeso far, Chinese Foreign Ministry spokesman Sun Yuxi
said here Friday.

3Opinion Analysis
Japan should carefully consider visiting the Yasukuni
Shrine and abide by the solemn statement it has madeso far, Chinese Foreign Ministry spokesman Sun Yuxi
said here Friday.

4Opinion Analysis
Japan should carefully consider visiting the Yasukuni
Shrine and abide by the solemn statement it has madeso far, Chinese Foreign Ministry spokesman Sun Yuxi
said here Friday.
Opinion holder: Chinese Foreign Ministry
spokesman Sun Yuxi

5

Opinion Analysis
Japan should carefully consider visiting the Yasukuni
Shrine and abide by the solemn statement it has madeso far, Chinese Foreign Ministry spokesman Sun Yuxi
said here Friday.
Opinion holder: Chinese Foreign Ministry
spokesman Sun Yuxi
Negative, Neutral, Negative, Neutral
6

NTCIR 6→7• Added Simplified Chinese• Added opinion target annotation• Sub-sentence opinion annotation• Relevance judged only for opinionatedsentences• Web-based annotation system for JA
7

Corpus Annotation
Feature

Value

Level

Req’d?

Opinionated

YES, NO

Sentence

Yes

Relevant

YES, NO

Sentence

NoPolarity

Positive, Neutral, Negative

ClauseNoOpinion Holder

StringClauseNo

TargetStringClauseNo8Corpus Sources• Japanese: 1998-2001 Mainichi newspaper• English: 1998-2001 Mainichi Daily News,Korea Times, Xinghua, Hong Kong
Standard, Straights simes

9Corpus Sources• Trad. Chinese: 1998-2001 China Times (+Express), Commercial Times, (United |Central | China | Economic) Daily News,Min Sheng Daily, Star News, United Evening
News• Simp. Chinese: 1998-2001 Xinhua Newsand Lianhe Zaobao

10Corpus Information
TopicsDocuments

Language
Sum

SampleTest

Sum

SampleTest

Trad.
Chinese17314246

58188

Japanese

22418287

38249

English17314167

25

142

Simp.
Chinese16214271

19252

11

Corpus Information
Sentences

Opinion Clauses

Language
SumSample

Test

Sum

SampleTest

Trad.
Chinese

6174

1509

4655

N/A

N/A

4657

Japanese

7163

1278

5885

7569

1348

6221

English

4711

399

4312

4733

404

4329

Simp.
Chinese

5301

242

4877

7523

570

6953

12

Opinion Percentage
Language
T -C hinese
Japanese
E nglishS-C hinese

O pinionatedLenient
46.8
28.9
25.2
38.3

Strict
44.3
21.1
7.5
18.4

Relevant(of O pinionated)
Lenient Strict
82.72
90.16
43.2
22.6
99.4
95.7
95.1
88.7

Polarity(P OS / N E G / N E U)
Lenient
Strict34.1 / 40.3 / 25.6 33.2 / 41.2 / 25.6
5.5 / 15.3 / 79.2
4.3 / 10.2 / 85.5
25.0 / 48.0 / 6.0
18.0 / 46.4 / 0.9
30.7 / 25.8 / 43.5 30.9 / 6.5 / 62.6

Lenient / Strict13

Topic Information

ID

Title

M00
M01
M02
M03
M04
M05
M06
M07
M08
M09
M10
M11
M12
M13
M14
M15
M16
M17
M18
M19
M20
M21
M22
M23
M24

Microsoft Anti-monopoly
Regenerative medicine
American stance on depleted uranium bullets
The impact of 911 terrorist attacks on America’s economy
HIV-tainted blood scandal
Cosovo civil war
Incident with Nepal’s ruling family (royalty)
Attacks toward Chinese Indonesian people
Lawsuit American Government against Microsoft
Nuclear weapons tests
Suriyah in the Middle East Peace Process.
The relationship between AOL and NetscapeEl NinoThe relationship between China and Russia
Greenhouse gasses
The relationship between NATO and Poland
Thailand in the Asian economic crisis
Yasukuni Shrine
Chechin (Chechnia) civil war
Indonesian President Suharto
Nuclear missile abandonment of North Korea
Airplane crashes in AsiaThe floods in the Mainland China
The births of the cloned animals known to the world
The responses of other countries to Lockerbie Air DisasterJapanese
N00
N01
N02
N03
N04
N05
N06
N07
N08
N09
N10
N11
N12
N13
N14
N15
N16
N17
N18
N19
N20
N21

Language ID
EnglishChinese
Traditional Simplified
N01
N02
N03
N04
N05
N06
N07
N08
N09
N10
N11
N12
N13
N14
N15
N16
T01

N02
N03
N05
N07
N09
N10N11
N12
N14
N15
N16
T96
N04
N13
N08
N01
N06

N01
N02
N03
N05
N06
N07
N08
N09
N10N11N12N13N14N15N16N0414

Annotator Agreement
JapaneseTrad.
ChineseSimp.
ChineseEnglishOpinionated 0.7135

0.45810.4362

0.2369Polarity

0.63410.77090.36340.1954

Relevance

0.59050.33290.61850.3459Macro Average
15

Participation
Language

Japanese

EnglishTotal

89J-E-TC-SC
Multi-lingual
participants

1E-SC

1TC-SC

Trad.

Simp.

79

1J-E-TC

E-J

Chinese12416

Annotator Training• JA: 5 annotator pool, 1 training topic, 6 hours ofmeetings• EN: 6 annotator pool, 1 training topic, 6 hours ofmeetings• ZH-TW: 10 annotator pool, 2 hours training, 1training topic, 4th or 5th pass based on kappa• ZH-CN: 12 annotator pool, 1 training topic, 8/4

hours of meetings (4 group leaders / annotator)

17

Some Guidelines• General beliefs, “common sense
knowledge” are not opinions• Expressions of future plans are notopinions• Generally used NTCIR-6 data for examples
18

Evaluation Metrics• Precision, Recall, F-Measure overopinionated, relevant, polarity• Semi-automatic evaluation of opinion

holders and targets (precision, recall, fmeasure)• Multiple approaches used19

Evaluation Metrics• Lenient: 2/3 annotators must agree• Strict: 3/3 annotators must agree• Polarity ZH: Set precision + recall biased• ZH: Rules to select polarity if annotators
do not agree• EN/JA: polarity needs majority agreement#system correct(polar = POS, NEU, NEG) #system correct(polar = POS, NEU, NEG).
#system proposed(opn = Y)#system correct(opn = Y)

20

Holder/Targetevaluation• Semi-automatic evaluation• Match system extracted holders/targets toannotator holder list, automate the process
in some way• Time consuming, only first priority run
evaluated• No JA evaluation (only 1 run by organizer)
21

Simplified Chinese Lenient Results
G roupBUPTICLP K U
NEUNLP
NLCL
NLPR
NTU
T TRD
WIA
IS C A S

O piniona ted
PR
F
0.604
0.39910.48070.48030.80040.60030.4721
0.7116
0.56760.44250.39910.41970.58220.77530.665
0.59390.60890.60130.412
0.9636
0.5772
0.58620.82080.68390.46490.74420.5723

P0.9775
0.963
0.96560.95070.994
0.9703R elevanceR
N/A
0.6559N/A
0.3258N/A
0.76930.69810.50320.9288FPolari ty
S-P

0.785

0.45050.48690.85640.80510.6682
0.94910.49560.43480.7419R ecall-based Polari ty
PRF
N/A
0.21640.36060.2705
N/A
N/A
N/A
0.29440.30180.298
0.17910.41890.251
0.43480.60890.5074
N/A

Simplified Chinese Strict Results
Group
BUPT
ICLPKUNEUNLP
NLCL
NLPR
NTU
TTRD
WIA ∗∗OpinionatedPRF
0.6312 0.4421
0.52
0.4486 0.8207 0.5801
0.4358 0.7339 0.5469
0.38570.402
0.39370.60960.892
0.724
0.6314 0.7517 0.6863
0.3481 0.9699 0.5124
0.6098 0.8964 0.7259

Relevance
PRF
N/A
0.9845 0.6743 0.8004
N/A
0.9736 0.3326 0.4959
N/A
0.9748 0.7859 0.8702
0.9631 0.7006 0.8112
0.99690.524
0.687

Polarity
S-P
0.2836

0.33780.28820.5329Recall-based Polarity
PRF
N/A
0.1272 0.2327 0.1645
N/A
N/A
N/A
0.2133 0.2539 0.2318
0.1003 0.2795 0.1476
0.3250 0.4777 0.3868

22

Traditional Chinese Lenient Results
G roupWIA
C i ty U H K
iclp k u
NLCL
NTU
T TRD
U ni N e

O piniona tedPRF
0.72980.52110.60800.66010.84460.74110.70150.62790.6626
0.53580.2676
0.35700.56480.89690.6931
0.51100.93450.66070.54280.92670.6846

P
0.99490.99430.92400.96150.96730.9614R elevance
R0.5306
N/A
0.67680.18010.71030.84130.8456

F
0.69210.80540.30150.81700.89990.8998Polari ty
S-P
0.69310.5361
0.48100.48750.37470.4293

R ecall-based Polari ty
PRF
0.50580.36110.42140.35390.45280.39730.33740.30200.3187N/A
0.27530.43720.33790.19150.35010.24760.23300.39780.2939Traditional Chinese Strict Results
G roupWIA
C i ty U H K
iclp k u
NLCLNTUT TRD
U ni N e

O piniona tedPRF0.8520
0.60030.70430.83640.90370.86870.85670.69980.77040.62590.29300.39910.70760.93070.80400.64520.93950.76500.69210.93790.7965

P
0.97880.95300.84870.88490.89920.8746R elevanceR
0.4061
N/A
0.56260.14540.64370.80440.8443

F
0.57400.70750.24820.74530.84910.8592

Polari ty
S-P
0.70030.54630.50850.49790.39240.4431

R ecall-based Polari ty
PRF
0.59660.4204
0.49320.45690.49360.47460.43570.35590.3918N/A
0.35230.46340.40030.25310.36860.30020.30670.41560.352923

Japanese Lenient Results
G roupEHBNHCUMIR A CNAKNLCL
TA K
TUT
U ni N e

O piniona ted
PRF
0.49210.73130.58830.619
0.51380.56150.316
0.08940.13940.81150.34160.48080.42550.22340.293
0.51910.27980.3636
0.67420.562
0.613
0.53630.19990.2912

P
0.48190.45450.53670.55270.4147

R elevanceR
0.6354N/A
0.0816
N/A
0.1891
N/A
0.29250.1918F
0.5481P

0.13840.24650.49220.27970.38250.26230.46380.45960.3251

Polari ty
R
N/A
N/A
0.01830.1801
N/A
0.11380.214
0.0548F
0.03410.26370.18280.292
0.0938Japanese Strict Results
G roupEHBNHCUMIR A C
NAK
NLCL
TA K
TUT
U ni N e

O piniona ted
PRF
0.37380.76270.50170.48940.5577
0.52130.2412
0.09360.13490.68850.39790.50430.31350.226
0.26270.41660.30830.35440.5416
0.61990.57810.41640.21310.2819P
0.28080.22330.301
0.30620.1553R elevance
R0.7321N/A
0.0821
N/A
0.2107N/A
0.33570.1464F
0.4059P

0.1201

0.2394
0.49330.24790.32030.15070.51720.4806
0.2914

Polari ty
R
N/A
N/A
0.01660.2154N/A
0.13160.24170.0497F
0.031
0.29990.20980.32160.084924

English Lenient Results
G roup
ICUkleMIR A C L E
NEUNLPNLCL
sics
TUT
U K P07
U ni N e

O piniona ted
PRF
0.24350.36870.29330.35290.72720.47520.59520.01160.02270.35220.77880.48510.37800.10140.15990.41920.61010.49700.31850.40920.35820.33050.90600.48440.33220.69950.4504P
0.27580.37410.12960.20920.4170R elevanceR
0.3648N/A
0.3189N/A
0.0685N/A
0.1755
N/A
0.5992

F
0.3141P
0.2586

0.34440.08960.19090.49180.18380.19430.2028
0.2279Polari ty
R
N/A
0.4301
N/A
N/A
N/A
0.24130.18300.43940.3671F
0.32300.20870.18850.27750.2812English Strict Results
G roup
ICUkleMIR A C L E
NEUNLP
NLCL
sics
TUT
U K P07
U ni N e

O piniona ted
PRF
0.07430.37770.12410.11090.76780.19380.28570.01160.02220.11050.82040.19470.11680.10530.11070.13360.65330.22190.09610.41490.15610.1026
0.94430.18500.10500.74300.1840P
0.09810.08530.0526
0.07400.1614R elevanceR
0.3797
N/A
0.3040N/A
0.0847N/A
0.1853
N/A
0.6768F
0.1559P
0.06870.13330.06490.10570.26070.05240.05690.05700.0637

Polari ty
R
N/A
0.4645
N/A
N/A
N/A
0.27960.21800.50240.4171

F
0.11970.08830.09030.10240.110525

Cross-Lingual Results
Type

Opin

Relevance

PolarityEN
TC-SC-JA-EN

0.15990.0896--TC-JA-EN

0.4504

0.49180.2812JA-EN

0.35820.19090.1885EN

0.4752(0.4918)

0.3230TC
TC-SC-JA-EN

0.35700.3015--

TC-JA-EN

0.6846

0.89980.2939TC-SC

0.60800.69210.4214TC

0.7411(0.9300)

0.3973SC
TC-SC-JA-EN

0.4197

0.4869--TC-SC

0.68390.66820.5074SC

0.68390.66820.507426

English Holder & Target

G roup
ICU
ICU
kle
kle
TUT
TUT

T y peLS
L
S
LS

P
0.40000.13330.39230.1250H olderRNANA
0.5076
0.53220.28330.2829F
0.44740.21320.32900.1735P
0.10590.0374

TargetR
0.17610.1793NA
NA
NA
NA

F
0.13240.061827

Simplified Chinese Holder & TargetGroup
ICLPKU
TTRD
NLPR
NTU
WIA
ICLPKU
TTRD
NLPR
NTU
WIA

L/S
L
L
L
L
L
S
S
S
S
S

P
0.41240.11290.42860.29090.66560.41040.17190.47590.38390.7574

Holder-TR
0.41240.11290.42860.29090.66560.41040.17190.47590.38390.7574

F
0.41240.11290.42860.29090.66560.41040.17190.47590.38390.7574P
0.00330.01510.4505000.5185

Target-TR
0.00330.0151
N/A
N/A
0.450500N/A
N/A
0.5185F
0.00330.01510.4505000.518528

Traditional Chinese Holder & Target

Group
WIA
iclpkuNTU
TTRD
WIA
iclpku
NTU
TTRD

L/S
L
L
L
L
S
S
S
S

P
0.8254
0.58720.50280.56450.82380.5797
0.48250.5496

Holder-TR
0.8254
0.58720.50280.56450.82380.5797
0.48250.5496F
0.8254
0.58720.50280.56450.82380.5797
0.48250.5496P
0.60580.02130.0358
0.6331
0.02280.0372

Target-TR
0.60580.0213N/A
0.03580.6331
0.0228N/A
0.0372F
0.60580.02130.03580.6331
0.02280.037229

Discussion• Relevance < Opinionated < Polarity < Holder• CH, EN, JA corpora have different
annotator agreement: training issue or data
issue?

• How to evaluate when annotators do notagree?

• What do the results MEAN?
30

Difficulties• Annotator agreement (address with better
experiment design?)• Shortened schedule (move to News docs)• Meaningful Cross-language comparison?
• Lack of shared tools for evaluation /processing / etc.

• Relevance this year is suspicious
31

Future Work
• Increase group participation in multiplelanguages• Improve annotator agreement (EN)• Standardization of tools (evaluation andannotation)• Confidence / statistical significance for
evaluation metrics

32

NTCIR-8• Breakout session today from 16:10• Looking for feedback!
• Plans to incorporate a multi-lingual task:• Query in EN, summarize opinion types
for JA, EN, ZH-TW, ZH-CN• New data
33